How do you evaluate whether or not a build was successful? What do you do when it isn’t? In this post, we’ll tackle question #3 of the Shedletsky Test: how to assess your latest build and cover the major pitfalls to avoid when recovering from a failed build.
Success criteria
Before the build, come up with the criteria you’ll use to evaluate your build. What defines success? Setting clear expectations is your first critical step. Without a yardstick to measure results by, it can be more challenging to distinguish a successful build from a failed one.
The purpose of the engineering build process is to ship a high-quality product to your customers at the volumes and yields you expect. To determine what needs to be achieved in development build, we need to work backwards from that goal. Each stage of the development process expects a different level of maturity. Here are some very common high-level development build goals:
- EVT: Looks-like and works-like. Success is a locked design for hard tool release.
- DVT: One line running production-capable parts and process at mass production speeds and yields. Success is one final production configuration.
- PVT: Validation of sustained yields at volume and line replication.
At the end of each build, ask:
- Did this build enable us to achieve both the product maturity we expect and the data we need in order to successfully complete the next build stage?
- Did we collect the data required to give the team confidence that they will be successful in the next build?
- Rather than moving onto the next build, should we spend additional effort on getting more data?
Product maturity and confidence of success will be influenced by:
- Reliability results
- Functional yields
- Functional validation
- User performance
- FCC and regulatory certification
Between a rock and hard place: When you don’t meet your build criteria
If the build has missed the mark, the next step is to contemplate inserting another build. You’re between a rock and a hard place: A new build means a better product but will slip the launch date, bungling channel partners and giving competitors more time to catch up. However, continuing as planned could result in field reliability and customer satisfaction issues.
Temptation 1: Beware the fake schedule
A fake schedule is more damaging to product and team than schedule delays.
At the end of a build, engineering leaders should be especially vigilant for updates that sound like this: “We had some troubles but the team is confident we can make it up in the next build,” or “Let’s have the next build be an XVT and we’ll decide afterwards if we’ve caught up or not.” As a leader, you may feel compelled to suggest the same. This is the situation that breeds the dreaded fake schedule: an unreasonably optimistic schedule that most engineers will privately acknowledge is not plausible. A fake schedule is more damaging to product and team than schedule delays. Hundreds of micro-decisions are being made each day, and if everyone is marching to a fake schedule, your team will be forced to make trade-offs they might not have had to make if the schedule was realistic.
If you’re an engineer: Be honest with yourself, your teammates, and especially your executives. Does anyone on the team feel their component is at risk? Resist the temptation to call out an individual or group. Act on the issue as a team and present it to your leadership as a team. If a new schedule is needed for some extra validation work, DOEs, or another tooling spin, be frank about the need for it.
If you’re an executive: Your team is doing the best they can to deliver — but make sure their good intentions haven’t led them astray. Poke at the upcoming build schedule to make sure it’s not a fake by digging into the remaining engineering risks, lead times, and build planning. Resist the temptation to try to “fix it” by optimizing the schedule down to the hour — that sends the wrong message to the team. Don’t get upset about not meeting the original plan. Your team may feel like they are risking themselves professionally in order to tell you the truth, so create an environment where everyone works through schedule issues together.
Temptation 2: Continuous building
Another temptation in this situation is to throw out the concept of the staged build event and instead to build new revisions of parts as they are available: the “continuous build.” Perhaps more common at larger companies, continuous building is both financially expensive and tiring to the team. Without an end, there’s no pause to evaluate whether success has been achieved. The team is focused on kicking off new parts instead of evaluating the whole. The configurations built are often not fully validated and can create overconfidence in a solution going into the next build stage.
Continuous builds need to be staffed — not only with engineers from your team, but engineers from the factory side (who may not be local). Intense travel and weeks of juggling both their factory job and their responsibilities at headquarters will have a major impact on the productivity of your team. As someone who has been in many continuous build situations, I have never felt that the team got to a positive result faster— and frankly, it was brutal.
Avoid temptations
The best way to handle the challenging situation of a build that doesn’t meet criteria is to be honest about where the product is. Put your foot down and fix issues when you have them before moving onto the next phase.
It’s tempting to say that your team can get it done even though the build criteria wasn’t met — it keeps your executives happy and at least temporarily meets metrics — but your real guiding principle is whether or not the product and customer experience will be at risk. A good product is the best outcome (what you should be working towards) in this tricky situation; let it be your guiding light.
Instrumental tells it how it is. Want more content like this in your inbox? Scroll down to subscribe and get our articles as soon as they are released.
