Fail Fast or Scale Fast? How to Measure the Value of Your Digital Technology Investment

Once you work with digital technology long enough, the saying “fail fast or scale fast” inevitably comes up. In one of my previous jobs, it served as a guiding principle for new investments. The idea is that you start small and if a pilot either meets or exceeds expectations, you should scale it rapidly across operations (where it makes sense) to maximize returns (read ‘5 Step Phased Approach to Digital Transformation in Manufacturing’ which explains how to effectively enter into a pilot). Otherwise, you should quickly move on to the next idea or option and repeat. The only problem is, accurately measuring success or failure isn’t exactly easy and teams often overlook important considerations. In this post, I’ll explore some of the key things you need to keep in mind to make sure you’re fully understanding the business impacts of digital technology pilots and investments.

Go or no-go?

Your pilot of your latest technology investment at your manufacturing facility is working well and it seems like you’re seeing some improvements that are better than you anticipated. But should you celebrate early? How do you know when you’re ready to tell your boss or the plant manager the outcomes and your recommendations? The only way to really know is to assess your project using a comprehensive business case review. And that requires you to look at both the quantitative (financial) results as well as the qualitative results.

For the sake of an objective evaluation, the same business case that you used to justify the time and expense to pilot the new technology should now be used as the basis for evaluating and scoring the pilot project. And this evaluation is very important to get right. After all, if your pilot was truly successful, you and your stakeholders will quickly begin considering expanding the pilot to other locations and instances within the site or company. If you weren’t successful, then you’ll need to consider whether or not it’s worth trying corrective actions in pursuit of a more favorable outcome or if it makes more sense to start over. Here are three critical things to keep in mind as you move forward:

• Don’t change the bar height. After investing significant amounts of time choosing a technology, finding a vendor and running a pilot, nobody wants to start over if results don’t meet expectations. And that can make it hard to stay objective, often leading decision makers to lower the bar for a project. “Maybe the original goals were lofty… the pilot worked pretty well, even if it didn’t hit all of the numbers,” is something you might think or hear. It’s always best to measure the results based on what you originally stated was important because if you’re willing to negotiate with the stakeholders around the minimally agreed upon performance, you’ll negotiate the next metrics, and the next, until the business case loses its integrity.
• Think quantitatively and qualitatively. Getting the quantitative results for a pilot is usually relatively straightforward. But there’s a lot more to consider than whether or not you moved the needle on some metric; it’s also critical to factor in the impact of soft costs. For example:
  • Can the factory support the technology, or will you need to depend on the vendor? What are the long-term hidden costs if the vendor needs to stay involved?
  • How did the workforce accept the technology? Did it lead to any new and undesirable tasks or does it prevent operations from doing business as usual? Is your workforce on board?
  • Will new skills and training be required? How much time will it take to get the workforce trained and proficient?
  • What does maintenance, IT, electrical and other functional area think of the outcome?
  • If you scale this technology out to different areas in the plant (or different plants), what organizational changes need to take place? Who owns the new technology? Who will support it?
  • How dependent is your process on the new technology? What if it fails? Is it fragile? Have you considered backups and workarounds?
• Make an actionable decision as quickly as possible (fail fast or scale fast). Once you’ve gathered essential information, it’s time to decide if you should move forward, revise the pilot or start over. If you built a strong business case and didn’t move the bar height, pilots generally shouldn’t require more than a few months.

Key reporting steps and ingredients

The challenge with effectively reporting on a pilot is putting together something that is to the point, yet comprehensive enough to accurately illustrate the costs and benefits of a solution. Of course, it’s important to use your company’s standard report format and style, which ideally should include the following:

• Executive Summary—a go or no-go recommendation with a bit of context
• Project/pilot/investment summary— what did you set out to do and what technology and vendor did you go with
• Outcomes:
  • Financial summary
  • Qualitative summary— how were people, process and technology impacted?
    • Suggestions for improving the use of the technology
    • Risks and mitigations

These tips will help you ensure that the content of your report best serves the needs of your company.

• Use the language of your finance/accounting department. Every company has a different process for evaluating capital investments. Your business case and subsequent evaluation should use the process of your current company, not a former employer or some industry recognized process.
• Make sure you include a financial representative on the team. Senior management (or whoever has the power of “go” or “no go”) will likely turn to finance/accounting for guidance. That’s why it pays to involve a finance representative in the evaluation, and especially during the development and evaluation of the business case.
• Capture the costs and benefits through multiple lenses. There are many popular ways to capture the costs and benefits of a technology investment. Most companies that I’ve worked with over the last 10 years like to use payback period because it’s a fairly straightforward measure of value. But it’s still worth exploring return on investment (ROI) and net present value (NPV) in most cases.
• Don’t sugarcoat things. Being straightforward and honest about anything that could be a future issue or headache is key to avoiding unnecessary pain and frustration down the road for you and potentially the entire company. For example, if a vendor seems great but is also small and could be acquired or go out of business, note that along with how you could mitigate the risks.

Skip steps at your own risk

Missing or avoiding key steps in the measurement of a pilot can result in a project blowing up in all kinds of different ways. For example, a pilot project team without a finance person or other objective representatives could get overly enthusiastic and myopic about a technology. Maybe they get excited about moving a key measure from point ‘A’ to ‘B’ and decide to move forward without adequately considering the bigger picture. When the other teams attempt to repeat or scale the solution, they may discover downsides related to integration or support any number of things. Early in my career, I got excited about adding a dedicated automation fiber optic network for the facility of my employer at the time. This would have given us speed and reliability not found on our business LAN. Since the project was my baby, I didn’t want other IT people tampering with it, so I relied on a small and friendly review team for the pilot. It was only after we went to integrate with the larger plant network that we realized it would not only be an enormous burden on the IT team, but we had also overlooked some significant costs associated with software and licenses. In other words, the technology worked great in a vacuum, but in the real world it created unforeseen burdens and costs.

Ultimately, it’s all about data

In some ways, there’s a chicken and egg component to measuring the value of a technology investment. After all, in complex operations you need baseline data about operations to compare against the results of a pilot. And you need a way to capture data related to the pilot at the right intervals, be it every second, minute, day or week. A clipboard or spreadsheet won’t cut it. If you’re looking for a way to capture and analyze data from the shop floor to the top floor, Dploy Solutions can help. We provide the real-time performance insights and analytics capabilities you need to advance operational excellence no matter where you are in your digital journey.

Have you read the previous blogs is this series?

1. Industry 4.0: Avoid the digital manufacturing road to nowhere
2. Is your manufacturing operations ready for Digital Transformation?
3. Digital manufacturing technology culture and its impact on operations
4. 5 steps for selecting the right technology for manufacturing
5. Tips for finding the right manufacturing technology vendor