Share this article

Share on facebook
Share on twitter
Share on linkedin
Share on reddit

The Right Way to Ship Software

This post lists the key insights from this article, by Jocelyn Goldfein, angel investor and former engineering executive at Facebook and VMware.

In a profession where we carry out decade-spanning holy wars over tab widths and capitalization, it’s no surprise that people get attached to their development and release habits. But if shipping so much software has taught us one thing, it’s to be an agnostic. Different methodologies optimize for different goals, and all of them have downsides. If you maximize for schedule predictability, you’ll lose on engineer productivity (as this turns out to be a classic time/space tradeoff). Even when you aren’t dealing with textbook tradeoffs, all investments of effort trade against something else you could be spending the time on, whether it’s building an automated test suite or triaging bugs.

My fellow engineers, please stop asking “Is this process good or bad?” and start asking “Is it well-suited to my situation?”

Consider two of the author’s past lives:

  • When it was a startup, VMware needed to offer predictable dates and high reliability because they had to convince conservative enterprises to buy operating systems from an upstart new vendor. (At the time, virtualization sounded like science fiction!)
  • In Facebook’s startup days, they needed to move quickly because first-mover advantage meant everything for a product based on network effects.

One of them put a ton of engineering emphasis on predictability and reliability; the other put its effort toward driving user engagement. Not hard to guess which was which. As you might imagine, the development practices at these two companies could not have been more different. Neither one was right or wrong — they both made appropriate tradeoffs for what they wanted to accomplish, and each company’s practices would have been ineffective or disastrous if applied to the other company’s products.

First, Take a Look at Who the Customer Is

To determine “the right way to develop software,” you’ve got to understand what matters for your product and how to optimize for that. This isn’t based on personal preference. Ultimately it stems from your company’s mission, and the way you make money is a reasonable proxy for that.

Chances are, if you sell software at a high price tag, you are selling to businesses that are buying your software based on their need. The more expensive your software is, the more mission critical it is for your customers, the more likely you have to optimize for reliability, functionality and a predictable schedule. You might think business customers would like your software as soon as possible, but because they have a lot of dependencies — deployment, training, integration — it is generally much more important to them that you be predictable than you be fast. Larger deal sizes also go hand in hand with fewer customers, meaning that each customer has comparatively more power over you and satisfying their needs is more crucial to your startup’s survival.

As a rule of thumb, expensive software means predictability is key while shipping. Customers need your product. If you have a lower (or no) price tag, focus on UX. Users who don’t need your product have to want it.

Well, what if your customers aren’t demanding enterprises? As you charge less and less (from millions to thousands, to hundreds, to freemium and free), your market goes higher volume and involves smaller businesses or consumers. For these products, schedules can be less important since people will generally accept your latest enhancements whenever they materialize. The influence of a single customer is small, so you might deprioritize a niche platform or bugs that affect only a few people.Read This Next

Stage matters here, too. If you are growing quickly, trading off quality might be acceptable if 80% of your users one year from now will be new and won’t remember your mistakes. On the other hand, if repeat business (aka recurring revenue) is your game, you’d better make sure current customers are delighted.

Next, Assess How You Deploy and How Much You’re Willing to Risk

You want reliability? Instead of weeks of lab-based stress testing, just ship to production and gradually turn up the load. Turn it down and fix the problem when you run into bottlenecks.

You want efficient testing? Well, you can probably catch 80% of the bugs with 20% of the testing, then quickly spot and fix the few that escape.Read This Next

You want design quality? Expose yourself to quick feedback loops by putting prototypes in production for a small number of users and see how it works.

Of course, you don’t have the freedom to choose to deploy in the cloud just because it makes life easier for you. Some products (operating systems or video game consoles) simply can’t exist entirely in the cloud. If you build for consumers on mobile, you’ll probably choose a native app so you can deliver the best UX, because at least in consumer, rich UX trumps engineering productivity.

Facebook’s struggle pivoting to mobile illustrates the potential for trouble. Facebook’s speedy, individualistic and iterative way of designing and shipping software was deeply embedded in product team culture. If you worked in the web tier, the cost of releasing was pretty close to zero and literally everything else about the way you worked was optimized to take advantage of that assumption. As the company’s focus shifted to native mobile apps, the engineers hired for their mobile expertise insisted on a heretofore unknown process like feature and UI freeze, functional specs and QA.

Learning new programming languages and frameworks wasn’t what made it hard for Facebook engineers to pivot to mobile. It was hard because they had to undo all their assumptions about how to make software. I’d like to tell you there was a cool-headed analysis of the merits of various practices given the constraints of native app development and what would be best for Facebook’s user community. What actually happened more closely resembled a discussion of religion or politics over the Thanksgiving dinner table. We were all family but violently disagreed in fundamental ways.

At the heart of that debate were different assumptions about tolerance for risk. Appetite for risk was baked into Facebook’s culture — after all, this company brought you the slogan “Move Fast and Break Things!” Longtime Facebook engineers viewed embracing risk as an essential cultural trait — and at the time, did not realize that mode of operating relied on assumptions about the universe that were true for the web but not for mobile.

Figuring out your own software development style means you have to contemplate your own appetite for risk.

As a rule, startups should be aggressive risk takers, for entirely rational reasons. When you have no customers, revenue or brand, the impact of a mistake is immaterial. You had nothing before, you have nothing afterwards. So who cares? But once you have customers, you have to define the cost of a mistake in terms of the pain you cause. Similar kinds of operational mistakes might cause a 5% decline in growth rate for one startup and a 75% decline for another, based on different business and deployment models. If that’s the case, those founders had better be running those companies differently.

To crystalize how deployment and risk compound: if you happen to sell on-premise system software to enterprises for lots of money, you have magnitude-of-pain and lengthy time-to-update both working against you. You can count on the same kind of mistake costing you two orders of magnitude more than if you provide a free web-based service to consumers.

It’s probably obvious to the world that VMware is substantially more risk averse than Facebook. Realize that it is not because Diane Greene and Mark Zuckerberg have different personalities, nor because one of them is right and one of them is wrong. They are different businesses with different technology stacks and they appropriately ship software in completely different ways based on how much risk their customers can absorb.

How You Ship is One Strand of Your Cultural DNA

Now that you’ve inventoried your business model, deployment model and appetite for risk, you’ve got a good framework for analyzing the release processes you’re using. If you’ve survived long enough to achieve product-market fit and some customer traction, chances are you’ve naturally adapted towards practices that make sense. In startups, as in nature, form follows function. You might surprise yourself and find out that you’ve been wasting energy on goals that don’t matter (like schedule predictability for a consumer business)!

Apart from analyzing your current methodology based on this framework, you can also use it as a filter on whose advice to take, which company’s practices to imitate or even which leaders to hire.

You may find this framework particularly helpful if you struggle with multiple technologies or business models. A startup may very well need to support both web AND native mobile apps, or more than one set of customers (say, a two-sided marketplace with different apps for consumers and businesses). In an ideal world, your release process would just vary to match the product under development. But how you ship is not just process, it’s culture and identity. Swapping out a process is easy. Changing culture is hard. And it’s even harder for a small company to embrace different cultures for different teams.

If you find multiple “shipping cultures” in tension in your company, you’re dealing with one of the fundamentally hard execution challenges of building and shipping software. There are no easy answers when people stake out positions grounded on emotions rather than reason. On the plus side: your team’s emotions are engaged! It can be hard to remember that silver lining when the conflict is raging, but it is good news that your engineers care passionately about your company.

Take a deep breath and remind them that release processes come and go, but your company’s mission and values are immutable. Your team hopefully can agree that what ultimately matters is to ship the best product for your users, and that what remains is negotiable. With any luck, this framework will help you negotiate it.

7 min​ read

Subscribe to our newsletter Weekly Bytes to get our curation of the top 5 articles of the week.

Subscribe to our newsletter Weekly Bytes to get our curation of the top 5 articles of the week.

Need visibility in your software development lifecycle?