Samuel Weiss is a former Product Design Engineer at Apple, now CTO and co-founder of Instrumental.
Now that I lead a software team at Instrumental, it's provided an interesting perspective on why hardware schedules are the way they are -- longer, highly subject to disruption, and difficult to accelerate. This article describes my realizations about hardware development schedules and some opportunities for how hardware development schedules could evolve to look more like software development schedules.
First of all — what’s an “iteration”?
From that first prototype to shipment to customers, hardware development is an iterative process. An “iteration” here refers to the time between “I need to make a well-defined change” and “I know the result of my attempt to produce that change.” The scope of such a change could range from a minor fix to a new feature.
An iteration in hardware development requires building and testing, usually with entirely new parts or processes. Designing and ordering these new parts takes time and coordination, so a hardware program has only a few opportunities for iteration. For software, it may just mean pushing new code.
Hardware iterations take time. A long time.
In hardware and software development, an iteration loosely consists of design, build, and test phases.
In software, many changes in the design and build phases do not introduce new dependencies outside of the engineer’s control, and testing the result of a change is usually as simple as pressing a button to run the code. These factors are partly why most software development is Agile — a strategy that optimizes development speed by emphasizing flexibility and reducing excessive planning.
In an Agile framework, a software iteration — that time between “I need to make a change” and “I know the result of that change” — can take hours or less. A hardware iteration, however, takes about a hundred times longer.
An idealized development iteration
The hardware build phase, which consists of both part manufacturing and product assembly, has the biggest impact on hardware iteration time. To test even a small component change in the context of the larger assembly, tens or hundreds of unique parts need to come together simultaneously. Manufacturing these changed parts at pre-production quantities can take weeks for simpler parts and months for high-complexity parts requiring work from multiple vendors. The final assembly process can last two or more weeks, depending on the number of units built.
Testing each iteration also takes significantly longer for hardware. In addition to functional tests on the assembly line, most hardware products are subjected to mechanical and environmental reliability tests. Long-term environmental or life cycle testing can last months, but most results are available between a few days and two weeks after building.
Because of the lengthy build and test phases, much more must be taken into learning from the previous hardware iteration and designing the next one. The hardware design process consists of brainstorming, modeling, detailed specification, and formal design reviews and can take a couple of weeks. Most importantly, as discussed below, multiple hardware designs are often fully planned in parallel, compounding the effort required during the design phase.
All-in-all, this makes for a months-long hardware iteration cycle. And since this cycle is so long, hardware engineers only get a few iterations to get their product right before it ships to customers.
Hardware iterates on a schedule.
Software development in an Agile framework enables flexible priorities. This can make any schedule fluid, but the timeline for an iteration or even a whole project can be estimated using historical engineering productivity data. Agile software development, therefore, does not follow a schedule — if anything, it predicts one.
On the other hand, hardware development resembles a waterfall model with a prescribed flow and cadence for each iteration phase. Each development build requires parts from many suppliers to come together simultaneously, so timing and coordination are critical. Since component lead times are so long, hardware development builds are scheduled months in advance, and there are fewer opportunities to make changes throughout a program.
When even a single part is delayed, the testing, learning, and redesign of every component can be delayed, too. In many cases, this leads to an overall schedule slip, which has a chain reaction to future part manufacturing, product assembly, and even shipment to customers.
Unfortunately, hardware teams often attempt to compress the design, testing, and learning phases to avoid schedule impact from build delays. Even worse, a program can enter “continuous build mode.” This is a terrible situation for hardware development, with material and time continuously spent building with the latest component revisions. Both outcomes add risk to design and validation and add stress to all engineers who are on the hook.
Hardware ‘iterates’ in parallel.
With relatively few iterations in hardware development, major issues need to be resolved by the next build to keep the schedule on track. But this explicitly violates the notion of ‘iteration’ — that you can try something and, when it doesn’t work, try the next thing.
In Agile software development, backup plans, bug fixes, etc., are scoped as needed. In particular, feature iterations are designed, built, and tested serially. Plan B isn’t usually started until Plan A has failed.
However, if plan A doesn’t pan out when problem-solving on a hardware build schedule, you better have kicked off backup plans B, C, and D already. This adds additional effort to the design phase of each iteration (and to the build and test phases, too), so it’s important to understand which backup plans are worth investing in.
Large and small companies must be judicious about allocating time and money to these backup plans. For critical issues, hardware companies must decide between conducting additional, unscheduled experiments to learn more about a solution’s viability (which often introduces delays) and spending the resources to kick off multiple solutions in parallel.
What can hardware learn from software?
These differences between hardware and software development — long iteration time, a set development schedule, and parallel iteration — stem from the long hardware build and test phases. The result is a hardware development process with significant effort spent on planning and a high sensitivity to delays.
So what can hardware engineers learn from the world of software development? In a way, hardware development is already becoming more “Agile” with the advent of cheap rapid prototyping tools. These tools enable quick iteration of an early concept, but the benefits do not last into later large-scale validation builds.
For these later hardware development builds, we must reduce the iteration time in other ways. Software engineers accelerate issue finding and “failure analysis” with debugging tools and continuous integration, but most hardware engineers have had nothing like that. Instrumental for development is intended to be the debugging tool analog for the hardware world.