“Prototype” and “Prototyping” Defined
A test article used for gathering knowledge to guide decision-makers who instigate change—these are the primary attributes of a prototype. Couple them with the general guidelines provided above and the following workable definitions for the terms “prototype” and “prototyping” emerge:
A “prototype” is a test article designed to demonstrate areas of high technical risk that are essential to system success. A prototype need not be a full system, but, in scope and scale, it is tailored to accommodate series of decisions, and as such, can represent a concept, subsystem, or end item according to the decisions to be made. Rather than reflect the final design, prototypes are built with the expectation that, as decisions are made, change will follow.
“Prototyping” is the practice of testing prototypes, of appropriate scope and scale, for the purpose of obtaining knowledge about some requirement, capability, or design approach. The knowledge obtained informs a decision-making process the output of which results in some degree of change. The degree of allowable change is bounded, in inverse proportion, by the scope and scale of the prototype.
These definitions have been refined in different ways to emphasise various aspects of prototyping that, in the multi-faceted decision-making process of today’s acquisition space tend to be missed. Also emphasised are aspects of prototyping that preserve it as a cost-effective alternative to cheaper, less reliable acquisition methods.
Unlike other definitions, these definitions do not exclude production representative prototypes. Rather, they embrace them with the caveat that, with prototypes of a production representative caliber, the scope of allowable change is much less, the range of available decisions more narrow, than for less sophisticated prototypes appearing earlier in the life cycle. Among such prototypes are “demonstrator prototypes” and “advanced development prototypes,” both of which appear between Milestones A and B.
For practitioners charged with prototyping in advance of Milestone B, the distinctions between these two types of prototypes have been confounding Both are properly considered prototypes, but each resides on different sides of the development divide. Technology demonstrators are more closely associated with technology development Advanced development prototypes are more closely associated with systems development.
On visual inspection, their differences are not intuitively obvious. But the latter is much more sophisticated and better suited to informing Milestone B. The former is much less so, but is also much cheaper to design and build. Getting the latter and not the former requires a great deal of judgment. Better policies about the two would also provide a great deal of help, which is part of the reason these definitions and their refinements are reintroduced here.
Their aim is to provide practitioners a rudimentary approach to prototyping so that it remains a viable option to cheaper alternatives. They also provide a baseline upon which more thorough policies can follow. More thorough policies are needed if better acquisition outcomes are to follow. Having working definitions is just a first step in that direction.
Competitive Prototyping and its Dilemmas
When compared to paper studies, prototyping presents an early dilemma: it takes more time and it costs more money, at least in the short term. So prototyping is justified only to the degree it allows for better materiel solutions or future savings. The objective is to spend more in development so that better systems can be fielded more quickly and for less overall cost In practice, prototyping should be a leveraged investment. The dilemma is to prototype in ways that provide a positive return.
When prototypes are evaluated on a competitive basis, the early dilemma of prototyping is exacerbated. Rather than fund one prototyping effort, DoD must fund two or more. Added to the tough decision of determining what to prototype are the tougher decisions of determining how competition will be pursued and the resulting prototypes evaluated. In most cases, the evaluation will support a future down-select decision, but until complete, the funding and management of multiple contractors puts incredible pressure on the procurement budget. Rarely does the addition of a contractor team come with a proportional increase in dollars.
Consequently, for competitive prototyping to be a leveraged investment, certain trade-offs are necessary. Development dollars must be allocated to support the right mix of activities and appropriately controlled so that a positive return might result. Competition, meanwhile, must be harnessed in a way that allows better performing systems to emerge at lower costs. In competitive prototyping, decisions are no longer limited to just those associated with the prototype. With competition, there are additional challenges.
Dealing with Budgetary Pressures
If prototyping is more costly at the front end, then prototyping on a competitive basis is worse. Not only must multiple prototypes be designed and tested as part of a competition, but with competition, the prototypes are usually more advanced .When resources are in scarce supply, this additional demand creates incredible pressure on the research and development budget. It also creates incredible risk for industry in ways that undermine the value of prototyping and hinder competition. The challenge is to find ways to channel these budgetary pressures effectively, not only for the success of the prototypes, but also for the success of DoD.
To channel budgetary pressures effectively requires dealing with risk effectively by allocating resources to what matters most. As Milestone B nears, this means devoting more resources to developing a military system and less to developing technology .
Historically, the aggressive performance goals of military systems have frustrated attempts to allocate resources this way.
To meet performance objectives, immature technology is embraced on the hope that it can be matured at the same time a system is developed . Rare is the program that can successfully develop a system and mature technology at the same time Some plans feature more mature technology at Milestone B. Using more mature technology frees resources for purposes of systems development, which is no small feat when the system is for military use, even when the technology is relatively mature.
Dealing with risk effectively requires making room for more mature technology. In practice this means introducing flexibility in performance objectives, or setting more modest ones, so that the risk profile will align with available funding. Flexibility creates an outlet by which the trade-offs can be made to release some of the budgetary pressure. It also obviates less desirable means by which the pressure can be released: through contributions from contractor independent research.
When competition is introduced to prototyping, tapping contractor independent research and development budgets is incredibly enticing. By tapping these funds, the total budget for prototyping can be increased, more risk can be accommodated, and more performance can be chased. When compared to competitively prototyped programs where the performance goals were much more modest, the difference in total costs is striking. For the Close-Air Support and Lightweight Fighter prototypes, the total budget for prototyping was about five orders of magnitude less.
Competitive prototyping’s ability to absorb large amounts of private capital is not unique to aircraft procurements It is a by-product of competition. It is not irrational for contractors to take a considerable amount of financial risk in development in hopes of winning a production award. Development is not thought of as a profitable venture in military procurement; production is.
So there is an incentive for contractors to take large financial risks in development to win the more lucrative production contract likely to follow. For those in DoD always on the search for better performance, there is an equal incentive to let them do so.
Independent research and development budgets are the primary resource a contractor has for improving its competitive posture in a prototyping phase When that is not enough, pooling resources through teaming, such as occurred in the Advanced Tactical Fighter program, is another means for responding to a prototype’s aggressive needs .
Over the short term, teaming and collaboration among development teams can be a good thing It can spur innovation and increase the flow of ideas. But when few can afford to compete and when failing to win the competition threatens the viability of the industrial base, neither DoD nor industry’s interests are well served controlling these effects is a part of competitive prototyping.
This is not to say contractors should not share in the cost of a prototype’s development, nor is it to say prototyping is bad for industry. Independent research and development, meanwhile, is a valuable tool for expanding the capabilities of a military system. But the pressures of competition, the inescapable lure of a multi-billion dollar defense market, and the combination of aggressive performance goals, immature technology, and inadequate funding, can create a vortex for private capital from which some competitors may never emerge. It may also create an environment that some competitors purposely avoid, all of which is counterproductive to preserving the industrial base and future chances of competition.
At the same time, when too much private capital contributes to the prototype’s development, the prototype’s value as a tool for reducing risk and providing better information is limited At the extreme end, when private funding is wholly responsible for a prototype, the resulting test article has been described as a tool best suited for marketing, not uncovering risks.
DoD procurements do not operate at this extreme. But at the extreme lies the risk of what may happen when a prototyping effort is inadequately funded and controlled. For all its success, the Navy’s Joint Standoff Weapon “JSOW” program encountered this risk as a result of its competitive prototyping effort For JSOW, much of the prototyping was done outside of the contract and on the contractors’ dime. The result: much of the technical risk was carried into Engineering and Manufacturing Development.
To counter these effects, DoD should absorb most of the cost pressures by funding the majority of the competitive prototyping effort. This only makes sense given that the purpose of the prototypes is to inform DoD process of decision-making. After all, the process is a treacherous one. Requirements change. Programs are cancelled. Funds are put on hold. Delays follow. All these things are likely outcomes in major systems acquisition. When they occur, they limit the ability of prototyping investments to provide a positive return. They also are all traceable to DoD If the DoD is responsible for marginalising a prototyping investment, then it should bear the costs for doing so.
This introduces discipline to the decision-making process and makes it more likely resources will be allocated in ways that provide the best return. With only a limited supply of resources, DoD can decide how best to align the risk profile to match the budget. Introducing flexibility in performance requirements so trade-offs can be made is one way of doing this.
Adopting an austere development environment that focuses on the objectives of the prototyping phase is another Austerity controls costs and allows resources to be allocated to the most important activities, such as those that complement the practice of prototyping .
Traditional system models and simulations, for example, are complementary to prototyping because they extend the range of knowledge prototyping provides .
In addition to allocating resources effectively, if DoD is to get a positive return on its prototyping investments, its financial commitment cannot be open-ended. There must be a sensible cap placed on the costs of a prototyping phase to preserve the promise of some expected return. To make limits meaningful, they have been accompanied by suggestions to contract for prototypes on a fixed price basis.
Contracting for competing prototypes on a fixed price basis without absorbing a large amount of private capital counsels for contracting on a best efforts basis. Reining in production expectations is also important. Some have suggested that when competitive prototyping, the future should be wholly in doubt. No production run should be promised or expected It may also be best to limit how much private capital can be contributed to the prototyping phase. All these steps limit how much private capital can shape the prototypes’ baseline and obscure the bottom line.
These are all ways in which resources might be effectively allocated in a competitive prototyping so that the prototype’s status as a risk reduction tool can be preserved. Also preserved is the likelihood of a capturing a positive return on prototyping investments without distorting capital allocations in the industrial base. When excess commitments of private capital are averted, more productive allocations can be made across the industrial base This in turn preserves the competitive landscape of the military arms market, and, over the long term, this is a net benefit for DoD After all, competition may offer plenty of value, but it relies on having enough players to take the field.
With DoD bearing most of the cost pressures for competitive prototyping, other temptations exist besides the allure of private capital. Competition may be a means by which better performing systems can be obtained at lower cost, but competition adds its own costs. In competitive prototyping, the costs come in the form of having to build two or more prototypes instead of one. The temptation is to dispense with competition on the grounds that its benefits will not outweigh its costs. Then, the extra resources competition will require can be redirected to other priorities, such as attaining better performance goals. The challenge is to approach competition creatively so competitive pressures can yield better results. This requires creativity in how objectives are framed, how the benefits of competition are tallied, and how competition is pursued.
On DoD acquisition ledger, benefits are usually tallied in terms of objectives for performance, cost, schedule, and risk When these objectives are defined at the system level, this can mean competition will be pursued likewise. In this approach, the prototypes are full-scale, but not necessarily fully capable, representations of the final system design. The Air Force’s Close Air-Attack-Support, Lightweight Fighter, and Advanced Tactical Fighter programs are all examples of the full-scale competitive prototyping approach. In each case, objectives for the prototypes were defined at the system level, and then evaluated on that basis in a competitive fly-off.
How objectives for a competitive prototyping effort are defined has a large impact on competitive prototyping’s costs. How objectives are defined determines how they will be evaluated, and how they will be evaluated determines how competition will be pursued. When objectives are defined at the system level, they can be particularly costly to address as part of a competition, especially when they are very specific.
Evaluating system-level objectives related to maintenance, operating, and supportability costs, for example, requires prototypes that closely approximate the final design. The Army took this approach in its competition for the Utility Tactical Transport Aircraft System, and as a result, had to field two production representative prototypes to support the competition This required a considerable financial commitment and limited the ability to realise efficiencies over the course of development. Instead, the Army had to realise efficiencies over a much longer term. In the case of the Utility Tactical Transport Aircraft System, it was the entire life cycle.
Tallying the benefits of competition over the long term is probably the best approach, especially for systems likely to be in the field for several decades. But with smaller development budgets, the resources necessary to evaluate production representative designs as part of a competitive effort are not likely to be there. Instead, competition must be approached in a different way, such as at a lower scale or with prototypes having fewer capabilities. What this means is defining system-level objectives with much less specificity, or reducing those objectives to something that can be competed at a lower scale, such as a subsystem.
When system-level objectives can be reduced to terms of subsystem performance, pursuing competition will be cheaper. At the same time, being able to capture system-level returns at lower costs will likely make competition more attractive. Development of the AIM-54 Phoenix missile is one example where the system-level performance largely hinged on a subsystem level competition Rather than devote resources to a full system competition, they were targeted at the subsystem that mattered most. Systems like aircraft have also been highlighted as systems where, better performance at the subsystem level can lead to outsized, system-level returns. For this reason, competition at the subsystem level has been highly encouraged.
One of the best-documented examples of competitive prototyping at the subsystem level is the cannon competition held during the Air Force’s Close-Air-Attack-Support program. Besides the competition between airframes, the Close-Air Attack-Support program pursued two other competitions to attain its life-cycle cost and performance goals
The first competition was for the cannon. The second competition was for the ammunition. The Air Force managed the first. In the second, the Air Force worked through the winning cannon contractor as a surrogate. For the aircraft that later became the A-10, the gun system was critical to providing the desired operational capability. But given the cannon’s high rate of fire, cost objectives were placed on the ammunition so that the capability would remain affordable.
To meet these cost objectives, the Air Force directed that a competition be held at the subcontractor level for ammunition. Additional requirements directed to how competition was to be pursued and the results evaluated were also levied. The result: savings that could be garnered over the life cycle in the form of an eighty percent reduction in unit cost for ammunition.
Successfully pursuing competition at the subcontractor level requires DoD to intervene in a relationship it would often leave alone. To meet DoD needs, additional requirements related to managing a subcontractor must be levied on the prime The prime contractor must be informed of DoD requirements, including source selection criteria, to ensure DoD priorities are what shape the competition. It is also not far-fetched to condition any down-select decision on DoD pre-approval.
Implementing these additional controls will undoubtedly lead to more development costs, but not implementing them may undermine the objectives of having a competition between subcontractors. In the case of the Close-Air-Attack-Support program, directing the prime was surely more expensive, but the gains it reaped have been garnered over the long term.
Framing objectives in non-system specific terms, such as in terms of capability needs rather than materiel needs, is another way let competition work, and at lower costs. The idea behind this approach to competition is that, by allowing competition to work on a wider scale without being confined to a subset of materiel solutions, it will reap greater returns.
Putting It All Together
If the purpose of competition is to obtain better performing systems at lower cost—which is usually referred to as obtaining the best value—then allowing the best value to be found is just as important as letting competition in. Getting right mix of performance and price has been described as the key to securing a technological advantage Surely performance is one aspect of a technological advantage, but so is price. The more systems that can be fielded for a given sum the greater the force multiplier effect. When procuring military systems, the goal is often to find the best mix of the two This is no less true when competitively prototyping. Competition merely postures DoD to make a better choice. The challenge is to structure the competition so the best value shines through.
Competitive prototyping presents DoD with a choice to continue funding one or more of multiple designs Through prototype demonstrations, information about each design is gathered and used to determine whether development will proceed, and if so, which contractor is best situated to pursue it. At Milestone B, the information should reflect that the risks and benefits of a particular design warrant greater financial commitments in Engineering and Manufacturing Development If the benefits outweigh the risks, then the prototypes should further demonstrate which design provides the best value.
Unlike other procurement approaches, though, competitive prototyping does not rely so much on DoD defining the best value as it does allowing competing contractors to find it on their own. Knowing what DoD considers most important in a final design is important, but giving contractors the freedom to experiment within a range of acceptable criteria facilitates better technological solutions at lower cost. Flexibility allows for bolder technological solutions to be tried and tested without renegotiating the contract, and therefore allows more risks and design flaws to be addressed early.
Competition, meanwhile, stokes innovative thought and encourages aggressive design solutions in direct response to identified needs The requirement to prototype allows all this to occur without sacrificing credibility Through competitive prototyping, better technological solutions should emerge based on performance, not promises, and when they do, the results can be surprising.
The outcomes of the Air Force’s Lightweight Fighter competition and the Army’s Advanced Attack Helicopter show how this can be true. The superior handling of the YF-16 prototype in the Lightweight Fighter competition was one reason it was preferred For the Army, the superior handling and simpler design of the YAH-64 prototype is what caused it to prevail In both, operational suitability in the form of airframe handling was a primary discriminator. Airframe handling is difficult to evaluate on paper in advance of design, but it is particularly important to determining an aircraft’s operational suitability. In the cases of the Lightweight Fighter and the Advanced Attack Helicopter programs, the superior operational suitability of the winning designs seemed to carry the day. For the YAH-64 prototype, these results were surprising. All analysis up to the point of flight test suggested its competitor would win.
Allowing contractors the freedom to explore a mix of technological approaches has a second benefit besides providing for more innovative thought: it allows DoD to know how much capability it can afford to buy When approached in this way, competitive prototyping can inform requirements and avert an improvident pursuit for a capability that is just not there. Even when the capability is there, competitive prototyping can show that a cost effective one may be still another generation of prototypes away.
So prototyping uncovers all kinds of valuable information When judging them as part of a competition, all things should be considered, not just performance or cost To make the prototypes more insightful than paper studies, particular attention should be paid to those things that are difficult to capture in advance and on paper, such as operational suitability Also important is past performance. Competitive prototyping can reveal which contractor is more likely to better perform in the next phase In terms of past performance, no gauge is likely to be more reliable than the recent experiences garnered through prototyping.
All the information prototyping provides can and should be used to choose between competing designs. Competition, meanwhile, should be leveraged to spur innovation and presentation of a best solution. The challenge is to put all these things together so the quest for a better solution will be worth the extra costs.
One challenge facing DoD will be to adjust the paradigm by which it approaches prototyping’s results. Sometimes the process of prototyping, whether it is competitive or not, suggests that the benefits of a particular technology are not worth the costs. In such cases, the best decision is probably not to go forward with large commitments of capital. Some may consider such a result to be a prototyping failure. but It is in fact a prototyping success.
Prototyping identifies problems prior to a full commitment of capital when it becomes harder to view a program as anything but “too big to fail” It is better to direct resources to technologies that can get to the war fighter cheaper and more quickly than it is to continue to invest in technologies that have promise but no near term return. Indeed, this is the primary reason one prototypes.
A final challenge is in establishing better prototyping guidelines. These guidelines will need to be more specific than those presented here. They will also have to account for the differences in prototypes among various military systems. Not all systems can be cost effectively prototyped at the full system level. The non-recurring expenses associated with systems like satellites and aircraft carriers are too large to ever be recouped later, and as such, they are best prototyped at something less than full-scale.
Aircraft and munitions, on the other hand, are best prototyped at higher levels of integration. To make competitive prototyping cost effective, there is a balance that has to be struck in terms of cost and capabilities that enables better decisions The balance will vary by system and better guidelines are needed for finding where the balances lie.
Similarly, there are points where prototyping on a competitive basis, as well as prototyping itself, may no longer make sense. To confirm through prototyping what is already known may be reassuring but is not likely a productive investment of capital. To continue to compete when one solution far exceeds the rest is not likely to result in much gain After all, pursuing competition for competition’s sake is not the goal the goal is better value.
1. Take a value view: Delivering the best value and quality in the sustainably shortest lead time requires a fundamental understanding of the economics of the system builder’s mission.
2. Apply systems thinking: In design framework, systems thinking is applied to the organization that builds the system, as well as the system under development, and further, how that system operates in its end user environment.
3. Assume variability; preserve options: design systems developers maintain multiple requirements and design options for a longer period in the development cycle.
4. Empirical data is then used to narrow focus, resulting in a design that creates better economic outcomes.
5. Build incrementally with fast, integrated learning cycles: Increments provide the opportunity for fast customer feedback and risk mitigation, and also serve as minimum viable solutions or prototypes for market testing and validation.
6. Base milestones on objective evaluation of working systems: In design development, each integration point provides an objective milestone to evaluate the solution, frequently and throughout the development life cycle. This objective evaluation assures that a continuing investment will produce a commensurate return.
7. Visualize and limit batch sizes, and manage queue lengths: Three primary keys to implementing flow are to: 1) Visualize and limit the amount of work-in-process so as to limit demand to actual capacity, 2) Reduce the batch sizes of work items to facilitate reliable flow though the system, and 3) Manage queue lengths so as to reduce the wait times for new capabilities.
8. Apply cadence, synchronize with cross-domain planning: Cadence transforms unpredictable events into predictable ones, and provides a rhythm for development. Synchronization causes multiple perspectives to be understood, resolved and integrated at the same time.
9. . Unlock the intrinsic motivation of knowledge workers: Providing autonomy, mission and purpose, and minimizing constraints, leads to higher levels of employee engagement, and results in better outcomes for customers and the enterprise.
10. Decentralise decision-making: Decentralized decision-making reduces delays, improves product development flow and enables faster feedback and more innovative solutions. However, some decisions are strategic, global in nature, and have economies of scale sufficient enough to warrant centralized decision-making.