Consumer brands always struggle behind the scenes to launch new products on schedule. Hardware products are built by managing intricate overlapping dependencies in design and manufacturing over a long development cycle. Because many steps depend on the completion of the preceding one in a decidedly unagile way, any disruption to that process, such as scaling an unproven new technology, shortages of raw materials, factory labor disputes, or transportation challenges, will often cause frustrating delays. COVID-19, the chip shortage, and other supply chain challenges have only made schedules harder to meet and, as a result, more difficult to predict when a product will actually ship. In this article, we will break down the top reasons why hardware products get delayed and why you may want to plan ahead before making a purchase this year.
Supply Chain Complexity
While a lot of attention has been placed on the great chip shortage of 2021, companies have been at the mercy of the supply chain for a long time. The trend toward lean manufacturing in the early 2000s has come with the trade-off of being only a few weeks away from running out of basic parts to feed the factories. In a stable economy, this system works well and saves cost, but sudden changes like the COVID-19 lockdowns have exposed the brittleness of this process we have taken for granted for so long.
Image: Supply chain complexity as products originate in more places than ever before.
For most products, the individual subcomponents can come from many different suppliers from across the globe and often have their own complicated supply chains. A part might say “made in X country” but it might be designed elsewhere and require raw materials from all around the world.
In 2014, I worked at Amazon on the Kindle Voyage which was the first Kindle to have a flush front display. To produce this new module, components were sourced from Japan, China, Taiwan, and Germany. The German component was also modified in Finland, and all of it was ultimately integrated in China. The scheduling logistics for ensuring all the parts made it on time to a build was a nightmare, and this was before anything had even been assembled. I can only imagine how much more challenging that process would have been if I was trying to do this same thing in 2020 or 2021.
Scaling Problems
Typical consumer hardware products follow the EVT/DVT/PVT development model where prototypes are designed and built-in small batches at builds until certain design and manufacturing milestones are met. At each of the stage gates, the engineering and operations teams evaluate their designs, squash bugs, and stabilize processes until the product is ready for mass production. These prototypes are extensively tested under conditions that mimic real-world conditions and are used for developing firmware and software. While builds are carefully overseen by teams of engineers, production is usually run simultaneously on many different lines in different factories around the world according to a locked and stable process. Maintaining the same high level of quality while using materials from multiple vendors across lines in different factories can lead to issues or combinations that were never fully tested in the prototype builds.
Image: As production scales, real issues start to manifest. Tesla’s Model 3 ramp in 2018, or “Production Hell” as Elon Musk put it, is a prime example.
Occasionally, issues that might have been dismissed as a minor anomaly at an early stage become major problems as brands begin to ramp up production quantities. Where engineers may have found one issue in a few hundred or thousand devices, they might uncover hundreds or thousands of instances as they scale to building millions of units. Brands set yield targets at each step in the line and for the overall production and if an issue starts to become an outsized source of yield loss, that is usually an indication of an uncaught material, design, or process issue.
Compounding a brand’s scaling issues is that every component vendor goes through the same growing pains of ramping production capacity. Vendors can be tempted to send cherry-picked “good” parts at a pre-production build to secure contracts. However, they do so at the risk of overcommitting before they have truly finalized their processes. Instead of hitting an expected yield, they may have to make 5 to 10 times the initial quantity and screen out those that meet the highest standards. However, once contracts are locked and high-volume manufacturing starts, those vendors will need to make the economics work by achieving those target yield numbers or reducing part quality.
As an example, in 2017, the new iPhone X was announced in mid-September without an official ship date. Rumors were that the new display module with its novel edge-to-edge design, rounded corners, and large notch could only be produced by a single vendor, Samsung. Yield loss due to quality issues and growing pains in developing this new component likely held production back from ramping at full capacity. Because it was such a signature part of the product, there was nothing Apple could do but wait for enough volume to meet their ramp targets. Apple finally announced pre-orders in late October and began shipping on November 3rd almost 2 months after the announcement.
Software and Platform Challenges
We may think software development is easy with agile development and frequent app updates. We have been conditioned to think that software is fungible and can be fixed after the fact. But device software is often more challenging. This is because unlike apps which use the APIs and output from the operating system, the base layers of software need to stably run sensors, inputs and outputs to achieve consistent and reliable results. Brands often start by building breadboard platforms outside of the actual housing. This allows them to get a head start on creating the firmware required to talk to all the different parts using prototype components before production is set at the semiconductor foundry. Once a platform is up and running, the operating system and application layers of software can start to be built. Of course, moving from the platform to the real product requires extensive tuning and adjustment and maybe even tweaks to the software.
With more complexity built into each product every year, and with consumers expecting new products to just work, brands are under a lot of pressure to test and validate their software. To avoid embarrassing security patches or fatal flaws in production, massive teams of quality assurance engineers will test different configurations under different conditions.
In 2012, during the development of the 3rd generation of the Kindle Fire tablets, the main processor supplier, Texas Instruments, decided to get out of the business of making its OMAP processors. The team was forced to change the hardware platform entirely to Qualcomm chips. This meant that they needed to start from scratch to ensure all the drivers and components could work with the replacement processor. Luckily, this setback did not delay the program significantly, but that was only through heroic efforts by the team. Sometimes, though, making a switch has more drastic consequences. The automobile industry has relied for years on stable, older components and can’t quickly pivot to the latest technology because they don’t know how newer components will perform in 5 or 10 years.
Regulatory Hurdles
Products today are more complex than ever before. Everything from vacuums to cars comes with embedded connectivity in the form of Bluetooth, NFC, UWB, Wi-Fi, or cellular. With more connectivity comes more regulation as governments want to make sure the radiative elements of products don’t emit too much power as they transmit data in close proximity to our vital organs. Launching a product like a new phone or tablet across the globe requires certification in different jurisdictions, each with slightly different requirements. Each unique SKU of a product must undergo compliance testing at the FCC in the US and CE in Europe before it can be declared safe to sell. Other countries have their own regulatory bodies, some of which require more information or more time. You will often see different launch dates in countries with stricter regulations or more bureaucracy such as South Korea or Brazil.
Image: Regulatory body logos on the back of the Kindle Oasis in 2017. Image sourced from TechCrunch
Shipping/Transportation Delays
One of the challenges with launching physical products is that even if the brand does everything right from concept to mass production, they must still submit to the whims of retail distribution. This means entrusting your products to carriers or shippers to send products around the world. Everything from bad weather to longshoreman strikes can severely impact the final delivery of goods from manufacturing hubs in Asia to warehouses and stores across the world. For a brand to have products in-store around the country or world on the same day means having a massive logistics arm to manage that distribution.
Image: Ships anchored outside the ports of Long Beach and Los Angeles courtesy of Bloomberg.
The pandemic has changed customs policies, reduced capacity at the docks, and left many cargo ships floating offshore. This in turn has slowed delivery times while increasing the cost of a single container load. Where a container load in the past might have cost $1000 to send from Asia, today that same size shipment might cost $20,000 to $30,000. Instead, companies may be tempted to ship with air freight as the price is almost at parity. Of course, COVID vaccine shipments are taking high priority on a lot of shipments, and reduced passenger flights have reduced supply there as well.
What can the industry do?
Even before the pandemic occurred, executives at consumer brands had been looking for ways to combat these issues. The challenges with COVID have only served to remind us how delicate our old way of doing things was and further incentivized teams to accelerate the adoption of new technologies. More and more companies are investing in remote real-time oversight at factories up and down the supply chain. With these new streams of data, insights and analysis are being used to better predict and respond to sudden changes. Those that are able to adapt quickly are starting to see the fruits of their investments as they are able to reduce delays and get products out more quickly.
At Instrumental, we have created a Manufacturing Optimization System that can help engineers find problems faster. Many of these topics are covered in-depth in the Build Better Handbook. If you are looking for ways to reduce your time to market, improve transparency, and get ahead of the curve on the next set of tools in manufacturing, contact us at sales@instrumental.com.
What should you do?
2021 has created the perfect storm of challenges for everyone. Supply chain disruptions are happening often making it harder to plan what to build. Some companies are attempting to pivot to using alternate components but will need to write and test with new software. With prototypes being harder to develop due to travel restrictions and regulators having more challenges keeping up with demand, new products are not getting approved as quickly and the shipping and labor shortages have increased costs and reduced throughput. This will be one of the most disjointed holiday seasons on record and we have already seen companies like Amazon try to entice people to buy gifts early with sales starting in October.
Chris Li is a former Amazon Product Design Engineering Manager and current teardown lead for Instrumental. He brought multiple generations of the Kindle to market and advised on the first Echo and Fire TV products. He also worked as the VP of Product Design at the healthtech startup, PillDrill, which shipped an award-winning medication tracking system.
Related Topics
