I have an Apple hardware background, but I lead a software team at Instrumental. In this role I’ve learned two things: (a) software engineers don’t understand how long it takes to build hardware, and (b) hardware engineers don’t understand how long it takes to build software. We’ll explore that second topic in a future blog post — for now, let’s look at what makes building hardware unique.
First of all — what’s an “iteration”?
In this post I assume an initial scope, design, prototype, code base, or other starting point for a software or hardware product. From that first prototype to shipment to customers, both software and hardware development are iterative processes. An “iteration” here refers to the time between “I need to make a well-defined change” and “I know the result of my attempt to produce that change.” The scope of such a change could range from a minor fix to a new feature.
An iteration in software development can be as short as the time it takes to write a block of code, see what doesn’t work, and figure out how to fix it. New features are often implemented over many such iterations, and significant updates to features are often fast and inexpensive — even after initial release to a customer.
An iteration in hardware development does not benefit from such flexibility. Unlike software (where once something is built, it’s built forever), building and testing a new hardware design requires entirely new parts. Designing and ordering these new parts takes time and coordination, and so there are only a few opportunities for iteration in a hardware program.
To understand the differences between hardware and software development and their respective schedules, we need to dig further into these differences between hardware and software iterations.
Hardware iterations take time. A long time.
In both hardware and software development, an iteration loosely consists of design, build, and test phases.
In software, many changes in the design and build phases do not introduce new dependencies outside of the engineer’s control, and testing the result of a change is usually as simple as pressing a button to run the code. These factors are partly why the majority of software development is Agile — a strategy that optimizes for development speed by emphasizing flexibility and reducing excessive planning.
In an Agile framework, a software iteration — that time between “I need to make a change” and “I know the result of that change” — can take hours or less. A hardware iteration, however, takes about a hundred times longer.
The hardware build phase, which consists of both part manufacturing and product assembly, has the biggest impact on hardware iteration time. In order to test even a small component change in the context of the larger assembly, tens or hundreds of unique parts need to come together at the same time. Manufacturing these changed parts at pre-production quantities can take weeks for simpler parts and months for high-complexity parts requiring work from multiple vendors. The final assembly process can last an additional two weeks or more, depending on the quantity of units built.
Testing each iteration also takes significantly longer for hardware. In addition to functional tests on the assembly line, most hardware products are subjected to a battery of mechanical and environmental reliability tests. Long-term environmental or life cycle testing can last months, but most results are available between a few days and two weeks after building.
Because of lengthy build and test phases, much more care must be put into learning from the previous hardware iteration and designing the next one. The hardware design process consists of brainstorming, modeling, detailed specification, and formal design reviews, and can take a couple of weeks. Most importantly, as we’ll discuss below, multiple hardware designs are often fully planned in parallel, compounding the effort required during the design phase.
All-in-all, this makes for a months-long hardware iteration cycle. And since this cycle is so long, hardware engineers only get a few iterations to get their product right before it ships to customers.
Hardware iterates on a schedule.
Software development in an Agile framework enables flexible priorities. This can make any schedule fluid, but by using historical engineering productivity data, the timeline for an iteration or even a whole project can be estimated. Agile software development therefore does not follow a schedule — if anything, it predicts one.
Hardware development, on the other hand, resembles a waterfall model with prescribed flow and cadence of each iteration phase. Each development build requires parts from many suppliers to come together at once, so timing and coordination is critical. Since component lead times are so long, hardware development builds are scheduled months in advance and there are fewer opportunities to to make changes over the course of a program.
When even a single part is delayed, the testing, learning and redesign of every component can be delayed, too. In many cases, this leads to overall schedule slip, which has a chain reaction to future part manufacturing, product assembly, and even shipment to customers.
Unfortunately, to avoid schedule impact from build delays, hardware teams often attempt to compress the design, testing, and learning phases. Even worse, a program can enter “continuous build mode.” For hardware development this is actually a terrible situation, with material and time continuously spent building with the latest component revisions. Both outcomes add risk to design and validation, and add stress to all engineers who are on the hook.
Hardware ‘iterates’ in parallel.
With relatively few iterations in hardware development, major issues need to be resolved by the next build to keep the schedule on track. But this explicitly violates the notion of ‘iteration’ — that you can try something and, when it doesn’t work, try the next thing.
In Agile software development, backup plans, bug fixes, etc. are scoped on an as-needed basis. In particular, this means that feature iterations are designed, built, and tested serially. Plan B isn’t usually started until Plan A has failed.
When problem-solving on a hardware build schedule, however, if plan A doesn’t pan out, you better have kicked off backup plans B, C, and D already. This adds additional effort to the design phase of each iteration (and to the build and test phases, too), so it’s important to understand which backup plans are worth investing in.
Large and small companies alike must be judicious about how they allocate time and money to these backup plans. For critical issues, hardware companies must decide between conducting additional, unscheduled experiments to learn more about a solution’s viability (which often introduces delays) and spending the resources to kick off multiple solutions in parallel.
What can hardware learn from software?
Each of these differences between hardware and software development — long iteration time, a set development schedule, and parallel iteration — all stem from the long hardware build and test phases. The result is a hardware development process with significant effort spent on planning and a high sensitivity to delays.
So what can hardware engineers learn from the world of software development? In a way, hardware development is already becoming more “Agile” with the advent of cheap rapid prototyping tools. These tools enable quick iteration of an early concept, but the benefits do not last into later large-scale validation builds.
For these later hardware development builds, we must reduce the iteration time in other ways. Software engineers accelerate issue finding and “failure analysis” with debugging tools and continuous integration, but until now most hardware engineers have had nothing of the sort.
A system like Instrumental, however, provides a snapshot like a debugger — view into every unit during each state of assembly that engineers can step through to see where things went wrong. It also helps engineers detect issues earlier in the iteration process like a continuous integration tool, leaving more time to design and implement design changes with lower risk of schedule slip.
To follow along as we share more about how to improve the hardware development process, please sign up below. Or, contact us today to learn how we can help you stay on schedule.
Related Topics
