Leadership Guidance for Failure to Meet Exit Criteria

Item #3 of the Shedletsky Test is: Do your builds have explicit entry and exit criteria — and if a build does not meet exit criteria, do you repeat it? Minus one point if you have ever done an XVT.

Here's how to ensure you have a solid NPI process around build entry and exit criteria and what to do when they aren't met.

Entry & exit criteria for NPI builds

As you make your NPI schedule, you must define your entry and exit criteria for each stage. Our article on hardware stage gates provides some guidelines to get you started.

The NPI process aims to ship a high-quality product to your customers at the volumes and yields you expect. To determine what needs to be achieved in a given development build, work backwards from the exit criteria for maturity. Here are some very common high-level development build goals:

1. EVT: Looks-like and works-like. Success is a locked design for a hard tool release.
2. DVT: One line running production-capable parts and processes at mass production speeds and yields. Success is one final production configuration and one "golden line."
3. PVT: Validation of sustained yields at volume with line replication (if applicable).

At the end of each build, ask:

1. Did this build enable us to achieve both the product maturity we expect and the data we need to complete the next build stage successfully?
2. Did we collect the data required to give the team confidence that they will be successful in the next build?
3. Rather than moving on to the next build, should we spend additional effort getting more data?

Reliability results, functional test yields, cosmetic yields, functional validation, user feedback, and FCC or other regulatory certification will also influence product maturity and confidence in success. Frameworks like digital twin engineering and manufacturing traceability, especially when combined with a visual inspection system, bolster your ability to collect data and manage your build process.

When you don’t meet your build criteria: avoid temptation

At the end of a build, engineering leaders should be especially vigilant for updates that sound like this: “We had some troubles, but the team is confident we can make it up in the next build,” or “Let’s have the next build be an XVT, and we’ll decide afterward if we’ve caught up or not.” As a leader, you may feel compelled to suggest the same.

As a leader, however, not meeting your build criteria presents a significant test of your leadership. Your decisions at this juncture could make you a hero or a failure; they could enable the company to hit its organizational goals or cease to exist (if you are a startup). You will need to be very thoughtful about your next steps.

If you have not met the exit criteria, you must insert a new build into the schedule to try again. Hopefully, you weren't already operating with a fake or overly optimistic schedule, and have some buffer built in. In the worst case, you will have to delay.

Temptation 1: Beware the overly optimistic, fake schedule

This situation breeds the dreaded fake schedule: an unreasonably optimistic schedule that most engineers will privately acknowledge is not plausible. A fake schedule damages the product and hurts the team beyond creating schedule delays. Hundreds of micro-decisions are made each day, and if everyone is marching to a fake schedule, your team will be forced to make trade-offs they could have avoided if the schedule was realistic.

If you’re an engineer: Be honest with yourself, your teammates, and your executives. Does anyone on the team feel their component is at risk? Resist the temptation to call out an individual or group. Act on the issue as a team and present it to your leadership as a team. If a new schedule is needed for some extra validation work, DOEs, or another tooling spin, be frank about the need.

If you’re an executive: Your team is doing their best to deliver — but make sure their good intentions haven’t led them astray. Poke at the upcoming build schedule to ensure it’s not a fake by digging into the remaining engineering risks, lead times, and build planning. Resist the temptation to try to “fix it” by optimizing the schedule to the hour, which sends the wrong message to the team. Don’t get upset about not meeting the original plan. Your team may feel like they are risking themselves professionally to tell you the truth, so create an environment where everyone works through schedule issues together.

Temptation 2: Continuous building or XVT

Another temptation in this situation is to throw out the concept of a staged build event and instead build any new revisions of parts as they are available: the dreaded “continuous build.” Perhaps more common at larger companies, continuous builds are financially expensive and tiring for the team. Without an end, there’s no pause to evaluate whether success has been achieved. The team is focused on kicking off new parts instead of evaluating the whole. The configurations built are often not fully validated and can create overconfidence in a solution going into the next build stage. Continuous builds need to be staffed — not only with engineers from your team, but engineers from the factory side who could be doing other things to improve the product maturity, like reliability testing, failure analysis, and design or process changes. Intense travel and weeks of juggling both their factory job and their responsibilities at headquarters will have a major impact on your team's productivity. As someone who has lived the continuous build as an engineer, I implore leaders in this situation: it's not worth it; you do not get to the finish line faster.

The longer path may be a shortcut

Temporarily meeting metrics to go from "all green" to "all red" overnight is bad for your reputation as a leader and erodes trust with your team and peers. When contemplating the longer path, which may be a delay to the original schedule, stick to a guiding principle: if the product and customer experience will be at risk, then take the hit now. A good product is the best outcome.