I owe myself this article for years. Since now I am in the same shoes again, for the umpteenth time, I decided to take some time and write this. I am working in the software industry, in software consulting to be more precise. That means what I (and the companies I work at) do is building software solutions for the unique problems of our customers. These solutions are customized and expensive, made for large enterprises who have deep-enough pockets to pay for the tailor to come to the house.

But every time and again, someone in the company comes up with the idea (or we receive a grant from some innovation fund) of building some software product, that we can create once, then sell it to dozens of customers and make a lot of money. Great idea, sounds terrific, software has the lowest (zero) manufacturing cost once it is designed, after all. Selling it once or a thousand times cost the same from software development perspective. Besides, who else would be more capable of building a successful software product than a software consulting company who has a track record of developing a series of complex, cutting-edge software systems, beating technological challenges, and coming up with novel solutions for the complicated problems of their customers?

Yeah, right, that is all true, but there are some fundamental problems and challenges in this otherwise noble endeavor. I participated in and lead several such initiatives and projects by now, most of them within software consulting companies, and I failed miserably so many times. So I decided to collect, organize and put these experiences down on paper to help myself, my current company, and many others, not to commit the same mistakes as I did.

Let me make a few comments before I start:

• To some extent, I focus on software consulting companies for two reasons. First, this is the industry I know the best. Second, it is frequent and tempting in the life of such firms to venture for product development, since they have the competences and manpower to build any kind of software. Therefore, some of the thoughts I am sharing with you are software-consulting specific, but most of them are more universal. So please don’t stop reading if you happen to work in another industry.
• The sources of my following thoughts are only partly my own experiences. Another significant part is based on books and presentations of industry-leading experts from the field of innovations and startups, like (but not exclusively) the Lean Startup movement from Eric Ries, and the famous Innovator’s Dilemma (and its Solution) from Clayton M. Christensen, to only mention my biggest influencers.

Alright, let’s get started!

The Usual Screen-Play

Let’s take a look at what is the typical, recurring scenario when a consulting company decides to develop and introduce a new product to market.

The most usual scenario goes something like this:

1. Someone (usually from the top) comes up with an idea of a product. Everybody is happy and expectant. The imagination is running wild, we create vivid dreams and crazy commitments. We want to build something that solves half the problems of the world and makes us all live happily ever after.
2. The management charges someone in the company to create some kind of a plan, usually in a shape of a tender backed by some innovation fund. Writing this provides opportunities to shape the idea, the business model, the marketing and business plan, or gives a chance to realize that it is not such a great idea after all and we should do something else instead. But this planning part is usually considered as the annoying, useless but necessary step, so it is done quick and dirty, just-write-something-to-get-the-funds style.
3. The next step gives even more reason for enthusiasm: internally or externally, but we receive the necessary funds to start the project and accomplish our dreams. We pop champagnes and celebrate.
4. A few weeks (or months) pass, and nothing happens. Everybody is quiet, and the euphoric feelings are long gone. There is no project leader, no team, no plan, nothing. Everybody is busy with their current projects where customers are always dissatisfied and want more faster and faster.
5. Sooner (if we received external funds) or later (when the money is raised internally) it starts aching enough to finally do something. The management assigns the project to some senior project manager and gives a few senior people to create some kind of a project plan what really fits the budget. The management hopes that with this they can finally forget about the project for some time.
6. The plan takes weeks to create because all these senior people have critical roles in currently running projects and are extremely busy all the time. Even organizing a 2-hour-long planning meeting takes weeks sometimes. The project-plan is finally completed, but the original vision is mercilessly chopped down to a realistic one that can fit the timeframe and budget. The people feel disappointed as if their dream was just destroyed.
7. Now that the dreaming is over, a real team is needed, and the actual work needs to get done. Since the handful of senior people are busy all the time and the resource allocation decisions are in fact made by the biggest, most profitable, and most demanding customers of the company, they have no chance to even get anywhere near the project. Maybe a few junior people or trainees, who are not that useful and productive yet, are assigned to work on the project, but without proper project management, design, and technological support it is very little they can do. Even if some senior people are allocated, whenever anything happens on a high-priority project of a paying customer (and something always happens sooner or later), or a new prospective customer with deep pockets shows up on the horizon, they are immediately moved away from the innovation project, sometimes for weeks or months.
8. It goes on like this month after month, until the deadline of the project (usually determined by the source of the funds) is approaching dangerously, and the loss (or even necessary payback) of money is threatening. Then all of a sudden the project gets higher priority than anything else. Half the company starts working on it in the last few months, but the goals are even further away from the original vision. Let’s forget about the market, forget about the product, forget about getting rich, just do some shit as quick as possible that is saleable to those who are going to audit the project and the use of the capital. Our only goal is to finish and forget about the project without getting sued and having to pay back everything.
9. If the project is financed from internal money, the previous point can be skipped, and the project is usually just given up halfway, especially if any financial or liquidity issue arises in the main business.

A somewhat better (but still terrible) scenario is when the management is actually committed to the project, a decent team is assigned and allocated to the project, and some usable product gets actually developed in the end. This is great, but now what?

1. Well, we should market and sell our shiny new product. We haven’t thought much about this before. We never did something like this, but let’s give it a try.
2. We send out our sales staff (if we are lucky enough to have one besides the CEO) to try selling it somehow. It turns out that it is more difficult and takes more effort than we ever anticipated.
3. We start with our current customers. They show some interest, but no one seems to care too much about it. ‘Sounds nice,’ they say, ‘let me think about it…’ ‘Maybe next year…’
4. So we have to try new customers. But who? And how? We don’t really have the money, time, and experience to do it.
5. We receive some feedback along the way from here and there, that refers to some fundamental misunderstanding in our assumptions that the project was based on. The problem doesn’t exist at all, or the problem exists but not exactly the way we thought, or our solution is not right to them, or a better and cheaper solution exists already, or we have wildly overshot the price or some other dimension of our product.
6. In the end, we learn something valuable about our customers, about the market, about the problem, and the necessary solution, but it is too little too late. We now see that with this real knowledge and some more inquiry we might be able to actually build something we can sell, but we have already spent all our budget (or maybe even more), so there is very little we can do at this point.
7. We go home sad and disappointed… After all this enthusiasm, effort, and money, nobody seems to care about our product…

(The above scenario very often gets interrupted and finished at earlier points than the 7th.)

Then a few months or years go by, some of the people leave the company, some new people are coming, and we slowly forget about the past and our previous product development project. Then someone comes up with another idea, or a juicy tender is published by some innovation fund, and the whole cycle begins again, and history starts repeating itself.

The Problem

In this segment, I am going to analyze the reasons why this is happening soo often, why we fail over and over again, and how can we handle innovation to have better chances of success.

Resources, Processes, and Values

To analyze the causes of these failures, I need to make a short introduction to the basic building blocks of organizations. Companies and organizations have capabilities independent of the people and other resources in them.

Resources are people in the first place, as well as things like equipment, technology, brand, information, or cash. Resources are pretty flexible and mobile. They are relatively easy to change or develop, hire or fire, buy or sell.

Processes are the patterns of interaction, coordination, communication, and decision making through which a company accomplishes value creation, namely the transformation of inputs of resources into products and services of greater worth. This includes not only manufacturing processes, but product development, procurement, marketing, budgeting, planning, resource allocation, or employee development and compensation. Processes are designed so that employees perform recurrent tasks in a consistent, efficient way, time after time. Processes are meant not to be rigid, repeatable, and hard change. Therefore a process that defines a capability in executing a specific task concurrently defines disabilities in performing different tasks. Inflexible processes are where many organizations’ most serious disabilities with change reside. Processes can be:

• Formal: explicitly defined, written or communicated in other ways
• Informal: implicit or habitual
• Cultural: deeply embedded in the culture of the company, often on a completely unconscious level

Ventures, projects, and initiatives often fail because the wrong processes are used to build it.

Values of an organization are the criteria by which decisions about priorities are made:

• Whether an order is attractive or not
• Whether a customer is more important or less
• Whether an idea for a new product is attractive or not

It is actually a key metric of good management whether clear, consistent, broadly understood values have permeated the organization, at every level. These values usually come down to two main factors:

• How big should a business opportunity be to be interesting?
• How big is the minimal acceptable gross margin on a deal?

These (unlike resources and processes) often define constraints because they define what an organization cannot do: pursue an opportunity with not big enough potential revenue or not thick enough margin.

The bottom line is that capabilities based on resources are easy to change, but capabilities based on processes and values can be extremely difficult to change in an organization.

The Capabilities of Software Consulting Companies

To put it another way: what are we as software consulting companies good at?

We are good at understanding the complex problems of customers, coming up with a solution in the shape of a unique, tailor-made software system, solving the particular pain of the customer. We know what questions to ask, we learn fast, then we build a software system for the problem, hopefully on time and within budget. Then we hope for another round.

These software projects are expensive and unique, built for large enterprises who have deep-enough pockets to pay for the expert to come to the house. By unique I mean that we are lucky if we can reuse even a tiny bit of the code from a previous project because the problems and processes of even very similar customers are always very different in some minor or major ways. So much that it makes more sense to build something entirely new from scratch, especially when the underlying technology evolved in the meantime.

The business model and the pricing is usually very simple, even with big, long, and complex projects: we make an estimation of how many days different people need to work on the project to make it happen. We multiply these days by the standard daily fees of people participating in the project. In fact, we sell the competence and working hours of our employees, wrapped up in a fancy box called project.

The (human) resources of software consulting companies:

• Delivery people, whose working-time is sold (project managers, business analysts, developers, and testers)
• Sales personnel to acquire new projects and new customers (usually some senior delivery people)
• Management people to provide everything is going fine (CEO, HR, marketing, office manager, etc.)

The processes of software consulting companies:

• Processes to acquire prospective new customers and projects
• Processes to execute projects and develop software solutions on time and within budget (with good-enough quality to keep the chance of getting another project from the same customer)
• Processes to allocate resources to current customers and active projects

The values are of software consulting companies:

• The best customers are the ones who hire our people for the most days, and for the highest daily fees (with the thickest margins)
• Assign the best people to the best customers – deal with the best customers first, then deal with the rest if possible
• Find new customers with the best price and for the longest time

Companies’ Mysterious Inability to Put Effort into Product Development Projects

As we have seen in my earlier experiences, software companies seem to be unable to assign the necessary resources to product development projects.

Problem #1: Resource Allocation Decisions

The resource allocation decisions and the values constituting the base of these decisions are actually controlled by the most important customers of the company. The (near and medium-term) future of a software consulting company depends on how well they act on these values, especially because resources must always be a little tight to avoid some people being idle when a project ends or gets suspended.

On the other hand, product development projects have a much longer-term, more uncertain potential to generate actual revenue, and have no customers at all who can influence and pressurize the resource allocation decisions. Because of this, these innovation projects are misfits in the company’s values.

If managers are acting according to the core values of the company, they always handle these innovation projects with the lowest possible priority because in the short and mid-term they only consume precious resources from the most profitable customers and projects. Moreover, if a product is not-yet-profitable, and financial problems arise in the core business, the CEO must shut this venture down to cut costs, because this is the most obvious way to save money. He doesn’t have any other choice, no matter what he promised before.

Problem #2: The Path to Growth

Every successful company has a vision and strategy for future growth. In the case of software consulting companies, this path usually means:

• Acquiring more projects and recruiting more people along with this
• Acquiring bigger projects for higher daily-fees and margins

The larger the organization, the more, bigger, and higher-margin projects they need to acquire to keep up with the growth expectations of owners or shareholders.

When developing an innovative project, future revenues are uncertain and not so tightly related to the invested development cost, if at all. This makes these incentives a misfit for the company’s core business model and the standard, predictable path of future growth.

Introducing a new product usually means an initially small market, whose potential size is yet unknown. These market opportunities can rarely provide a solution to the growth expectations of the company on short and mid-term. The larger the expectation, the more difficult to experiment with new products besides the core business because fewer opportunities seem attractive enough. Rational managers can rarely build a convincing case for entering small, poorly defined low-end markets that offer only lower profitability because prospects for growth and improved profitability upmarket often appear to be so much more attractive.

The pressure from customers (resource allocation) and shareholders (growth expectation) overcome initial enthusiasm simply because these innovation projects don’t make any sense according to the processes and values of a good-functioning, well-managed, successful software consulting company, no matter how big returns they promise in the long-term.

The Different Nature of Product Development

In software consulting projects, there is always a customer who has an aching pain, a defined problem, for which he hasn’t found any solution on the market, so he is looking for a consulting firm to solve it in the shape of a unique software solution. And most importantly, he is willing to pay significant money for this. During the negotiation, we – as a consulting firm – are selling a solution concept for the customer’s problem for an agreed price. So we never have to think or worry about the following questions:

• What problem does the customer have? – “They told us their problem, we don’t have to figure it out.”
• How important is this problem for the customer? Is it really a problem? – “We don’t care about the importance as long as they pay for the solution. By the way, it must be important if they came to us, but if it turns out to be a non-existent problem, it is still not our risk to worry about.”
• How many customers have the same problem? – “We don’t care because this single customer is paying enough for this project to be profitable for us.”
• Is the proposed solution a good solution for the problem? – “We don’t care. Once they accepted our offer with our proposed solution, it is not our responsibility and risk anymore if it turns out they needed something else.”
• How much (if at all) is the customer willing to pay for the solution? – “We don’t have to figure it out. We just give them an offer based on our implementation costs (plus the usual margin), and they can take it or leave it.”
• Are other customers willing to pay for this solution as well? – “We don’t care, we gave an offer for this project to be profitable by itself. If anyone else wants the same solution (which is rarely the case), that’s nice, but we don’t count on that.”

However, when we undertake to develop an innovative product for the market, all these questions above have critical importance, including the following ones:

• Who are the targeted customers for the product?
• Through which channels can we reach them?
• What is the unique value proposition that we should communicate to them?
• What channels or 3rd party sellers do we need to distribute our product?

Analyzing and answering these questions (and many more) are critical to the success and profitability of product development. This is something that we have to deal with from day one, way before building anything.

Sadly, we tend to ignore the importance of these questions because we have never needed to answer such questions before, so we have no expertise and processes to accomplish it. Instead, we often do the following:

• We don’t even think about most of these questions until the product is fully developed
• We answer some of them implicitly, with hypothetical, assumed answers
• We come up with a solution concept based on these assumptions
• We build the solution
• Once the product is developed, we try to sell it, and we face that most our initial assumptions were inaccurate or downright wrong

In reality, the vast majority of successful new business ventures abandon their original business strategies when they begin implementing their initial plans and learn what is and isn’t working in the market. The ones that run out of resources or credibility before they can iterate toward a viable strategy are the ones that fail.

The million dollar question of product development is not the usual “How can we build it?”, but the fundamentally different “What should we build?” Developing a new product and introducing it to a market is rarely a technology or implementation challenge, most often it is a marketing challenge. To overcome this, we need additional expertise and different processes compared to traditional software consulting projects.

The Solution

In this section, we find some answers to overcome the challenges and frequent problems of product development.

Protect the Product Development Project from the Main Organization

The first step is to create a safe space for the innovation project where the team can work without being influenced by the day-to-day events and influences of the main business.

#1 Commitment from the CEO

The CEO is the only person in a company who has the necessary authority to protect the product development project from the main organization, that’s why his or her commitment (and involvement) is essential for the success of an innovation project.

#2 Creating an Independent Organization

The CEO’s job is to create a small and independent organization for the product development project, which is:

• Not affected by the main organization’s resource allocation processes and projects
• Has independent budgeting, planning, and controlling (like a separate company)
• Small enough, so their performance will be meaningfully affected by the revenues, profits and small orders flowing from the new business in its earliest years
• Able to develop and use different processes and values, that are needed for the innovation
• A cross-functional team that has all the necessary resources and skills to act independently: complete autonomy to develop and market new products within their limited mandate, and the ability to conceive and execute experiments without having to gain an excessive number of approvals from the main organization

#3 Finances and Budget

The expectation from this organization should be to be profitable and self-sustaining as soon as possible. Early profitability is a must in order to protect it from the main organization when something unexpected happens in the core business. Without profitability, the project can be shut down in case of a financial problem, in the name of cost-cutting.

On the other hand, the budget needs to be patient and tolerant for failing (quickly and cheaply) and trying again. The same goes for significant growth, which usually takes longer time.

Use Additional Expertise and Different Processes for Product Development

The second step is to apply the right people, processes, values, and strategy for this newly built organization.

#4 The Right Leader and the Right Expertise

It is important to assign someone to the innovation project who has the necessary authority and experiences. When you look at prospective people’s CVs, always look for similar problems solved. For example, a project leader (or CEO), who had launched a venture thinking he had the right strategy, realized it wasn’t working, and then iterated toward a strategy that did work.

Besides the leader, don’t forget about other necessary expertise that the core business might not have. First of all, a marketer with similar experiences. Someone, who had insightfully figured out how a market was structured, had helped to shape a new product and service package that did an important job well for customers.

#5 The Right Processes

We have to analyze and recognize which resources, processes, and values of the main organization are useful for product development, and which are not. Utilize some of the resources, but don’t leverage processes and values that are not suited for innovation.

In the case of a new product, the market application, market size, revenue forecast, cost forecast, budgets and plans are unknown and very often unknowable in advance. This is why the applied processes need to support learning and discovery rather than execution of a rigid plan. We need to iteratively test, validate, and change the key assumptions behind our vision. In a sense, we have to fail early and cheaply over and over again in our search for the right market, the right strategy, the right product with the right technology.

We have to balance carefully between action and analysis. Some action must be taken before detailed plans are made, but these plans should only be made for learning (critical information in the right order) rather than implementation. This is called discovery-driven planning, where the goal is the validation of our fundamental assumptions.

We have to differentiate between a strategy under development and a validated strategy:

• Until our hypotheses are validated, only implement features with the aim of idea validation
• Implement features in production-level high quality only after the assumptions behind it are validated

Summary

Innovation and product development is inherently a challenging, complex, and sometimes unpredictable endeavor. It is a broad and deep topic, with tons of excellent authors and books on the market. These considerations above are only some bare minimum conditions to start a new product development project with some realistic hope of success. Take these into account and don’t let precious human creativity, enthusiasm, energy, and time (plus a lot of money) be wasted on incentives that are destined to fail from the start!

References and Further Reading

1. Clayton M. Christensen (1997). The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail, Harvard Business Review Press (ISBN 0875845851)
2. Clayton M. Christensen (2003). The Innovator’s Solution: Creating and Sustaining Successful Growth, Harvard Business Review Press (ISBN 1578518520)
3. Eric Ries (2011). The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses, Crown Business (ISBN 0307887898)