Competitive prototyping has been highly encouraged, if not mandated, as a preferred approach to major systems acquisition in DoD. Its repeated encouragement is due in part to its description as a best practice by specially formed task forces.. In most instances, competitive prototyping is presented as a tool for stoking creative thought, for improving decision-making, and for leading to better acquisition outcomes.
In other instances, its value has been questioned. In practice, competitive prototyping has not always delivered on its promises. Part of its mixed results has been attributed to widespread confusion over the meaning of terms and how prototyping should be pursued on a competitive basis. Building off lessons learned, this report provides an overview of prototyping accompanied by a description of how competitive prototyping has and could be practiced better within DoD.
Summary
Building off lessons learned, an overview of prototyping is provided and is accompanied by suggestions for doing so better and on a competitive basis within DoD.
The debate over prototyping is not so much over whether prototyping is good but when it provides value. At Milestone C, prototyping as a pre-requisite to a low-rate initial production decision is well accepted. The large commitments of capital that accompany a production award warrant some assurance that the technology to be produced will deliver as promised. Prototyping provides that assurance. At Milestones A and B, however, prototyping has not gained much traction, especially when cheaper alternatives seem to be available.
Paper competitions coupled with systems analysis, models and simulations, and other estimation techniques have been the desired alternative at these earlier stages of the life cycle. These methods are thought to be both cheaper and less time consuming than prototyping, and therefore more cost effective. The critics say otherwise.
Critics argue that if prototyping is good enough to support a production decision, why not use it earlier to justify a formal program start at Milestone B or a comparison of alternatives at Milestone A. Time and time again, they say, paper competitions and capability estimates have shown major systems acquisition to be plagued more by the unknown risks of systems development than the known ones. Experiences learned when developing aircraft illustrate this point: all the analysis in the world cannot reveal what one does not know. Prototyping can.
The response to these criticisms has been that, while prototyping may provide value, there is too much change early in the life cycle to make prototyping worthwhile. Changes in technology, performance objectives, and operational concepts prior to Milestone B marginalize the value of prototyping in the early stages of the life cycle. Besides, the response goes, prototyping can provide little additional knowledge when compared to other acquisition techniques without completing a detailed design. And going through detailed design prior to Milestone B is just setting oneself up to do it all over again in Engineering and Manufacturing Development. The debate, therefore, is over whether prototyping early in the life cycle can ever be cost effective.
A similar debate surrounds the use of competition. All would agree competition is good; not all would agree it is always sensible. Competition for competition’s sake has never been the goal. The goal is to get a better value. Competition may be a means to this end, but, in the defense market, to invite more competition invariably entails more costs.
When these costs exceed their expected returns, competition no longer makes sense. Like the debate over prototyping, with competition, the debate is over how competition should be approached so that it provides enough value to warrant its costs.
Recently, the debates over early prototyping and competition have converged in the context of mandates competitive prototyping for major systems acquisitions up to Milestone B and compels it to be a continued consideration throughout the life cycle. In some ways, competitive prototyping’s resurgence should be no surprise.
The hallmark of competitive prototyping’s ascendance has always been the threat of shrinking defense budgets. With shrinking defense budgets on the horizon and sequestration looming, this is no less true today. But its prevalence as a means for effective reform is counterintuitive. Competitive prototyping not only requires more development dollars up front, it takes more time, and its success in DoD has been mixed.
The issue facing DoD is how to deal with the additional costs of competitive prototyping so that better acquisition outcomes can follow. Fortunately, the lessons from previous periods of competitive prototyping reforms provide some clues. They suggest both how prototyping can remain cost effective and how competition can be sensibly pursued.
Reintroducing these lessons and building from them in ways applicable to today’s acquisition environment is the first step to implementing competitive prototyping reforms. It is also the first step towards obtaining better acquisition outcomes. After all, competitive prototyping does not guarantee such outcomes will follow; it only makes them possible. The goal is to make them possible using fewer dollars than before.
Prototyping Attributes
.
While competitive prototyping can be more valuable than paper competitions, it can also confirm what is already known. The key to uncovering more value and making prototyping more cost effective lies in understanding what is meant by the terms “prototype” and “prototyping.” Despite decades of intermittent prototyping in DoD, settled definitions for these terms have not yet emerged. Conceptually, they are easy to comprehend if not always to explain, at which point a reference is usually handy. but no reference can explain how they relate to DoD acquisition process with its various milestones and decision points. The leap from everyday practice to the highly specialised DoD procurement process is too great.
Lack of terminology has divided the policy of competitive prototyping from its practice in ways that have frustrated realisations of its promises. Practitioners have not understood how prototyping should be approached. Policy-makers have struggled to organise principles around prototyping from which better outcomes can emerge. Settling on definitions for the terms “prototype” and “prototyping” in a way that instructs those who make policy as much as it guides those who must build one is the first step in doing so.
Prototypes Are Test Articles.
Whereas paper studies estimate a technology’s capabilities, prototyping demonstrates those capabilities through testing. Test articles are designed, constructed, and tested to demonstrate the capabilities of some technology or system. In its simplest explication, the test article is the prototype, and as a test article, it can take many forms and represent various states of maturity depending on the aims of the test .Whether the test article represents a concept, subsystem, or end item that is full scale, fully capable, or something that is much less mature, all are forms of prototypes. The process of using these test articles to demonstrate capabilities is the practice of prototyping.
Prototyping’s emphasis on technology demonstration is one reason it has been popular during periods of falling defense budgets. With fewer procurement dollars to spend, there is less appetite for risky expenditures on unproven technologies. To warrant greater investment, technology must prove itself, and more than just in operational terms It also must prove to be affordable.
Prototyping “should allow us to fly—and know how much it will cost—before we buy” But knowing what to fly to justify what to buy has been a recurring difficulty. Conclusions are inconsitent between prototyping something that is production representative and something that is less sophisticated. Of the two, prototyping a production representative test article is the most conservative approach.
Building production representative prototypes in advance of every major program start allows a full understanding of a technology’s costs and benefits. With a production representative approach to prototyping, risks can be contained. Fixed-priced contracts can follow. Programmatic success would be more likely. these are the goals of every prototyping and development effort as it nears production Rarely, however, is such an approach cost effective, especially on a competitive basis prior to Milestone B
Prior to Milestone B, prototyping requires one to be selective, and being selective is where the benefits and difficulties of prototyping lie. It may not be cost effective to build a production representative prototype prior to Milestone B, but building something less sophisticated may be. Whether it is or not depends on whether one can selectively design, construct, and evaluate a prototype in ways that provide more reliable information than paper studies and analysis can provide.
The key to remaining cost effective is to invest no more capability in the prototype than is required to further the prototype’s primary purpose The key to making prototyping more reliable than paper studies is to target those capabilities paper studies struggle to estimate accurately. Perfecting both these aspects of prototyping in a test article of limited capability is extremely difficult Doing both, however, is essential to realising a prototype’s full potential and serving its ultimate end: to generate information and guide future decisions.
Prototypes Guide Decisions.
DoD multi-phased acquisition process has many decisions points all corresponding to individual phases. At different decisions points, the degree and types of knowledge required to support a particular decision varies. But having sufficient knowledge at each point is essential to enabling better acquisition outcomes Where insufficient knowledge exists, resources are committed when not enough about the technology is known.
Technical risk is underestimated; cost increases and schedule slips follow This has been the downside of basing decisions solely on paper studies. They tend to underestimate what is not already known.
Prototyping enables better acquisition outcomes by improving the reliability of available information. Prototyping injects an early dose of realism into the assumptions and conclusions at the core of previous studies and analysis, thereby making them more useful. Realism comes through demonstrated capabilities. As more capabilities are demonstrated, more becomes known, and the more justification there is for the decisions made But the more capabilities are added, the more costs will be incurred, and the more closely one must evaluate whether the information being provided is worth the extra costs.
There is a line where prototyping’s costs begin to exceed its returns. For prototyping to be a productive exercise, prototyping must keep on the positive side of the line. In practice, this requires prototyping with a particular end in mind, investing only in activities that support this end, and then using the information that results to chart a better course.
Charting a better course through the early stages of the acquisition life cycle does not require all the capabilities of a final system to be embedded in a prototype. A production representative prototype at Milestone B is not only overkill, it resembles a waste prototype need have no more capability than is necessary to support the next series of decisions Ensuring the prototypes are more valuable than paper studies, though, requires that certain capabilities be targeted.
Prototypes should target the areas where paper studies are most weak: areas of high technical risk that are essential to system success. This targeting is essential to uncovering the unknowns that plague acquisition programs based on paper and to making prototyping worthwhile. It is also essential to reducing risk in advance of the next phase and positioning an acquisition program to capture efficiencies later on These all make prototyping more cost effective and more desirable than limiting oneself to paper alone.
During the Advanced Tactical Fighter’s prototype phase, a number of fixes for the YF-22 prototype were identified early and incorporated at lower cost as a part of the next phase. The Navy’s A-12 program took a different approach; its early system design was based almost entirely on paper. As Full Scale Development ramped up—which is today’s equivalent to Engineering and Manufacturing Development—a number of technical problems emerged that engulfed all hopes of successfully implementing the paper design.
To a certain degree, the Advanced Tactical Fighter program encountered comparable problems, but not all were technical ones. Problems with funding, work sharing among contractors, and an unstable industrial base hindered efforts to capitalise on promised efficiencies Thus, while the Advanced Tactical Fighter program enjoyed a successful prototype phase, it shows how even a strong start can be overwhelmed by other issues down the road Prototyping may enable better acquisition outcomes, but it does not guarantee they will follow.
The goal with prototyping is to make better outcomes possible, and demonstrating areas of high technical risk is essential to reaching this goal. Demonstrating areas of high technical risk is also essential to making prototyping more cost effective. When areas of high technical risk are demonstrated through prototyping, it presents an opportunity to address problems early, when rates of expenditures are lower, and without risking the success of the next phase.
In development, problems always emerge, and when development is based solely on untested analysis and estimates of a design, problems tend to emerge later in development when expenditure rates are higher. Prototyped programs encounter similar problems, but problems tend to be identified earlier and can be fixed more cheaply, as in the case of the YF-22.
Capturing this efficiency, an example of cost avoidance, bolsters prototyping’s cost effectiveness. Capturing enough of them so that the extra development dollars invested in prototyping can be recouped later is what makes prototyping more worthwhile Sometimes these efficiencies result in reduced cost; most of the time they result in reduced risk.
The Air Force’s Close Air-Attack-Support program, the program that led to the highly successful A-10 aircraft, provides an example of prototyping’s ability to reduce risk and avoid costs. During flight test, the designers of one prototype identified a flaw in wing design while the designers of the other prototype realized the benefits of one critical technology were not worth its costs Fixes were identified and adjustments made so that moving into the next phase, risks for both designs were reduced in ways that also avoided costs. In the testing of both designs, their prototypes served as risk reduction tools.
When areas of high technical risk are not addressed through prototyping, it is not likely to reduce risk or to result in much gain. During early development of the Army’s Anti-Armor Submunition for example, prototypes were constructed and tested with the highest technical risk components excluded from the design . When moving into the next phase, these components became the major risk areas. Without demonstrating those areas of high technical risk that were essential to system success, the prototype’s ability to reduce risk was marginalised. As an acquisition strategy, prototyping did not provide much value.
For any given prototype used within DoD acquisition life cycle, the areas of highest technical risk appropriate for demonstration should vary. A prototype need only provide enough sophistication to address those risks that are most relevant to the next series of decisions These risks tend to vary by phase of the acquisition life cycle. Early in the life cycle, areas of high technical risk relate to technology development. As one nears Milestone B and into Engineering and Manufacturing Development, risks associated with systems development—such as risk in the areas of integration, manufacturability, producibility, and operational suitability—come to the forefront so a prototype’s maturation should vary depending on where it falls within these phases.
So as a matter of acquisition practice is that in terms of reducing risk through technology demonstration, all prototypes are not created equal. It also means that not all risks are suitable for reduction through early prototyping. Some risks, like those appearing later in the life cycle, are just incapable of being reduced without a prototype resembling the final design.
Regardless of the risks that may or may not be reduced in a particular prototype, not to be lost is the fact that all prototypes produce information. This information can and should be used to guide a full range of decision-making, from those occurring in the context of a specific acquisition to those on which the acquisition is based.
For instance, in the acquisition community, prototyping can assist in determining whether the benefits of a new technology outweigh its risks, and thus warrant further investment Prototyping can also be useful for evaluating the merits of a particular design approach, or in the science and technology community to guide the transition of technology outside of the laboratory Prototypes can also be used in the requirements community to evaluate operational concepts and needs When prototyping, the information provided is valuable. All should use it.
Prototyping Leads to Change
The ability for a wide community of users to capitalise on the knowledge prototyping provides is another benefit of prototyping, but it does not always result in efficiency. In the course of acquisition decision-making, some degree of change is expected to result from prototyping.
Indeed, change is what prototyping is all about. When acquisition decisions are based solely on paper studies, one would expect that change would be equally inevitable. It is. The difference is that with paper studies, the need for change is not recognised until greater capital investments have been made and major funding committed. At that point, it becomes more costly to undo what has been done.
Prototyping allows resources to be committed incrementally until the merits of a technology are better understood. When the merits are not there or are simply not worth the costs, prototyping gives reason to change course In this way, by allowing for change, prototyping provides a hedge against uncertainty Sometimes the uncertainty lies in a technology’s maturity. Sometimes the uncertainty lies in something more fundamental, such as an operational concept, requirement, or threat. In each case, prototyping provides an opportunity to change as more that was uncertain becomes known.
But in providing the opportunity to change, prototyping has a weakness. Too much change marginalises the value of prototyping, making it no more useful than the cheaper, less reliable paper studies it supplants. The more a prototype resembles a system’s final configuration, the less change it can tolerate and still provide the expected returns.
When the prototype and final configuration item closely resemble each other, minor changes can be accommodated. Major changes, such as those associated with an operational concept or a fundamental approach, cannot
The experiences encountered on the Air Force’s strategic airlift C-X program underscore this point: too much change in operational concept can marginalize the value of prototyping. The predecessor to the C-X program was the Air Force’s Advanced Medium Short Landing and Take-Off Transport program. The AMST program constructed and tested two full-scale representations of aircraft that emphasized tactical airlift.
Over time, however, the Air Force saw strategic airlift as more important and cancelled the AMST program in favor of the strategic-oriented C-X. The C-X program ultimately led to the development of the C-17 Globemaster III cargo aircraft, but the fundamental change in the operational concept between the two program marginalized the value of the full-scale AMST prototypes The expected benefits of prototyping never emerged. The C-17, for all its low risk technology and preceding AMST prototype phase, still encountered significant challenges going forward .
The Air Force’s first generation Advanced-Medium Range Air-to-Air Missile “AMRAAM” program provides another example of too much change, while at the same time illustrating the perils of too little. In its first generation, AMRAAM was to provide a capability similar to another air-to-air missile of the day, just with shorter range and in a smaller package. To meet the desired form factor, the prototypes used solid-state electronics instead of the conventional tube technology found in its predecessor. The technology could not perform, and when entering Full Scale Development, the design reverted to using the already proven tube technology.
When making the transition, program personnel relied on assumptions that said it could still meet the desired date for initial operational capability without acknowledging the significant step back they were making. AMRAAM essentially started over, and used paper studies to support its optimistic assumptions. Like the C-X program, significant delays and cost overruns ensued. Requirements should have been changed to reflect the new start.
So they were not, and difficulties followed. while prototyping allows and even encourages some amount of changes to be made, a limit must be imposed if prototyping investments are to be preserved There is no bright line rule here. Rather, the scope of allowable change appears to vary in inverse proportion to the scale and sophistication of the prototypes. When the change is significant and the prototypes relatively mature, then proceeding without a new prototype phase could be no more different than starting off with a paper design .
This is one reason the XV-15 tiltrotor prototype was not an effective precursor to the MV-22 Osprey. Though the technology’s feasibility had been demonstrated, its ability to meet an operational need was not, especially the advanced operational needs of the Marine Corps. Building the MV-22 Osprey on the basis of the XV-15 prototype was little more than starting from a paper design.
With prototyping, change may be expected, change may even be encouraged, but not all changes can be accommodated. The amount of allowable change has to be limited to ensure that the prototypes, as a tool for enabling better acquisition outcomes, survive the decision-making process.
1. Prioritise tech prototypes by industry partner more closely with Pentagon to share costs
2. Compare and contrast tech for identifying deployment constraints and develop index for scenario selections and details of return on ops
3. Recommend deployment for ops success and implement given evaluation surround scenario outcomes and distinguish weighting scheme indices/biases
4. Significant questions at Pentagon about defense market size; is issue of strategy and not just acquisition process improvement
5. One of big questions at Pentagon is how to generate unique military advantage from tech widely available on commercial market
6. Negotiate future contracts based on output costs/price risk; benchmarks align demand with supply capacity
7. Defense benchmark repository to find pricing assumptions at odds w/ like projects and embed tough targets into price
8.Pentagon and Congress working to rewrite statutes that are keeping staff busy checking off boxes instead of working on new tech.
9 Pentagon working with Congress to rewrite outdated acquisition rules and remove burdens from overworked programme directors
10. Pentagon acquisition structure charging in several direction at once, pushing initiatives aimed at how think about revamp tech
In other instances, its value has been questioned. In practice, competitive prototyping has not always delivered on its promises. Part of its mixed results has been attributed to widespread confusion over the meaning of terms and how prototyping should be pursued on a competitive basis. Building off lessons learned, this report provides an overview of prototyping accompanied by a description of how competitive prototyping has and could be practiced better within DoD.
Summary
Building off lessons learned, an overview of prototyping is provided and is accompanied by suggestions for doing so better and on a competitive basis within DoD.
The debate over prototyping is not so much over whether prototyping is good but when it provides value. At Milestone C, prototyping as a pre-requisite to a low-rate initial production decision is well accepted. The large commitments of capital that accompany a production award warrant some assurance that the technology to be produced will deliver as promised. Prototyping provides that assurance. At Milestones A and B, however, prototyping has not gained much traction, especially when cheaper alternatives seem to be available.
Paper competitions coupled with systems analysis, models and simulations, and other estimation techniques have been the desired alternative at these earlier stages of the life cycle. These methods are thought to be both cheaper and less time consuming than prototyping, and therefore more cost effective. The critics say otherwise.
Critics argue that if prototyping is good enough to support a production decision, why not use it earlier to justify a formal program start at Milestone B or a comparison of alternatives at Milestone A. Time and time again, they say, paper competitions and capability estimates have shown major systems acquisition to be plagued more by the unknown risks of systems development than the known ones. Experiences learned when developing aircraft illustrate this point: all the analysis in the world cannot reveal what one does not know. Prototyping can.
The response to these criticisms has been that, while prototyping may provide value, there is too much change early in the life cycle to make prototyping worthwhile. Changes in technology, performance objectives, and operational concepts prior to Milestone B marginalize the value of prototyping in the early stages of the life cycle. Besides, the response goes, prototyping can provide little additional knowledge when compared to other acquisition techniques without completing a detailed design. And going through detailed design prior to Milestone B is just setting oneself up to do it all over again in Engineering and Manufacturing Development. The debate, therefore, is over whether prototyping early in the life cycle can ever be cost effective.
A similar debate surrounds the use of competition. All would agree competition is good; not all would agree it is always sensible. Competition for competition’s sake has never been the goal. The goal is to get a better value. Competition may be a means to this end, but, in the defense market, to invite more competition invariably entails more costs.
When these costs exceed their expected returns, competition no longer makes sense. Like the debate over prototyping, with competition, the debate is over how competition should be approached so that it provides enough value to warrant its costs.
Recently, the debates over early prototyping and competition have converged in the context of mandates competitive prototyping for major systems acquisitions up to Milestone B and compels it to be a continued consideration throughout the life cycle. In some ways, competitive prototyping’s resurgence should be no surprise.
The hallmark of competitive prototyping’s ascendance has always been the threat of shrinking defense budgets. With shrinking defense budgets on the horizon and sequestration looming, this is no less true today. But its prevalence as a means for effective reform is counterintuitive. Competitive prototyping not only requires more development dollars up front, it takes more time, and its success in DoD has been mixed.
The issue facing DoD is how to deal with the additional costs of competitive prototyping so that better acquisition outcomes can follow. Fortunately, the lessons from previous periods of competitive prototyping reforms provide some clues. They suggest both how prototyping can remain cost effective and how competition can be sensibly pursued.
Reintroducing these lessons and building from them in ways applicable to today’s acquisition environment is the first step to implementing competitive prototyping reforms. It is also the first step towards obtaining better acquisition outcomes. After all, competitive prototyping does not guarantee such outcomes will follow; it only makes them possible. The goal is to make them possible using fewer dollars than before.
Prototyping Attributes
.
While competitive prototyping can be more valuable than paper competitions, it can also confirm what is already known. The key to uncovering more value and making prototyping more cost effective lies in understanding what is meant by the terms “prototype” and “prototyping.” Despite decades of intermittent prototyping in DoD, settled definitions for these terms have not yet emerged. Conceptually, they are easy to comprehend if not always to explain, at which point a reference is usually handy. but no reference can explain how they relate to DoD acquisition process with its various milestones and decision points. The leap from everyday practice to the highly specialised DoD procurement process is too great.
Lack of terminology has divided the policy of competitive prototyping from its practice in ways that have frustrated realisations of its promises. Practitioners have not understood how prototyping should be approached. Policy-makers have struggled to organise principles around prototyping from which better outcomes can emerge. Settling on definitions for the terms “prototype” and “prototyping” in a way that instructs those who make policy as much as it guides those who must build one is the first step in doing so.
Prototypes Are Test Articles.
Whereas paper studies estimate a technology’s capabilities, prototyping demonstrates those capabilities through testing. Test articles are designed, constructed, and tested to demonstrate the capabilities of some technology or system. In its simplest explication, the test article is the prototype, and as a test article, it can take many forms and represent various states of maturity depending on the aims of the test .Whether the test article represents a concept, subsystem, or end item that is full scale, fully capable, or something that is much less mature, all are forms of prototypes. The process of using these test articles to demonstrate capabilities is the practice of prototyping.
Prototyping’s emphasis on technology demonstration is one reason it has been popular during periods of falling defense budgets. With fewer procurement dollars to spend, there is less appetite for risky expenditures on unproven technologies. To warrant greater investment, technology must prove itself, and more than just in operational terms It also must prove to be affordable.
Prototyping “should allow us to fly—and know how much it will cost—before we buy” But knowing what to fly to justify what to buy has been a recurring difficulty. Conclusions are inconsitent between prototyping something that is production representative and something that is less sophisticated. Of the two, prototyping a production representative test article is the most conservative approach.
Building production representative prototypes in advance of every major program start allows a full understanding of a technology’s costs and benefits. With a production representative approach to prototyping, risks can be contained. Fixed-priced contracts can follow. Programmatic success would be more likely. these are the goals of every prototyping and development effort as it nears production Rarely, however, is such an approach cost effective, especially on a competitive basis prior to Milestone B
Prior to Milestone B, prototyping requires one to be selective, and being selective is where the benefits and difficulties of prototyping lie. It may not be cost effective to build a production representative prototype prior to Milestone B, but building something less sophisticated may be. Whether it is or not depends on whether one can selectively design, construct, and evaluate a prototype in ways that provide more reliable information than paper studies and analysis can provide.
The key to remaining cost effective is to invest no more capability in the prototype than is required to further the prototype’s primary purpose The key to making prototyping more reliable than paper studies is to target those capabilities paper studies struggle to estimate accurately. Perfecting both these aspects of prototyping in a test article of limited capability is extremely difficult Doing both, however, is essential to realising a prototype’s full potential and serving its ultimate end: to generate information and guide future decisions.
Prototypes Guide Decisions.
DoD multi-phased acquisition process has many decisions points all corresponding to individual phases. At different decisions points, the degree and types of knowledge required to support a particular decision varies. But having sufficient knowledge at each point is essential to enabling better acquisition outcomes Where insufficient knowledge exists, resources are committed when not enough about the technology is known.
Technical risk is underestimated; cost increases and schedule slips follow This has been the downside of basing decisions solely on paper studies. They tend to underestimate what is not already known.
Prototyping enables better acquisition outcomes by improving the reliability of available information. Prototyping injects an early dose of realism into the assumptions and conclusions at the core of previous studies and analysis, thereby making them more useful. Realism comes through demonstrated capabilities. As more capabilities are demonstrated, more becomes known, and the more justification there is for the decisions made But the more capabilities are added, the more costs will be incurred, and the more closely one must evaluate whether the information being provided is worth the extra costs.
There is a line where prototyping’s costs begin to exceed its returns. For prototyping to be a productive exercise, prototyping must keep on the positive side of the line. In practice, this requires prototyping with a particular end in mind, investing only in activities that support this end, and then using the information that results to chart a better course.
Charting a better course through the early stages of the acquisition life cycle does not require all the capabilities of a final system to be embedded in a prototype. A production representative prototype at Milestone B is not only overkill, it resembles a waste prototype need have no more capability than is necessary to support the next series of decisions Ensuring the prototypes are more valuable than paper studies, though, requires that certain capabilities be targeted.
Prototypes should target the areas where paper studies are most weak: areas of high technical risk that are essential to system success. This targeting is essential to uncovering the unknowns that plague acquisition programs based on paper and to making prototyping worthwhile. It is also essential to reducing risk in advance of the next phase and positioning an acquisition program to capture efficiencies later on These all make prototyping more cost effective and more desirable than limiting oneself to paper alone.
During the Advanced Tactical Fighter’s prototype phase, a number of fixes for the YF-22 prototype were identified early and incorporated at lower cost as a part of the next phase. The Navy’s A-12 program took a different approach; its early system design was based almost entirely on paper. As Full Scale Development ramped up—which is today’s equivalent to Engineering and Manufacturing Development—a number of technical problems emerged that engulfed all hopes of successfully implementing the paper design.
To a certain degree, the Advanced Tactical Fighter program encountered comparable problems, but not all were technical ones. Problems with funding, work sharing among contractors, and an unstable industrial base hindered efforts to capitalise on promised efficiencies Thus, while the Advanced Tactical Fighter program enjoyed a successful prototype phase, it shows how even a strong start can be overwhelmed by other issues down the road Prototyping may enable better acquisition outcomes, but it does not guarantee they will follow.
The goal with prototyping is to make better outcomes possible, and demonstrating areas of high technical risk is essential to reaching this goal. Demonstrating areas of high technical risk is also essential to making prototyping more cost effective. When areas of high technical risk are demonstrated through prototyping, it presents an opportunity to address problems early, when rates of expenditures are lower, and without risking the success of the next phase.
In development, problems always emerge, and when development is based solely on untested analysis and estimates of a design, problems tend to emerge later in development when expenditure rates are higher. Prototyped programs encounter similar problems, but problems tend to be identified earlier and can be fixed more cheaply, as in the case of the YF-22.
Capturing this efficiency, an example of cost avoidance, bolsters prototyping’s cost effectiveness. Capturing enough of them so that the extra development dollars invested in prototyping can be recouped later is what makes prototyping more worthwhile Sometimes these efficiencies result in reduced cost; most of the time they result in reduced risk.
The Air Force’s Close Air-Attack-Support program, the program that led to the highly successful A-10 aircraft, provides an example of prototyping’s ability to reduce risk and avoid costs. During flight test, the designers of one prototype identified a flaw in wing design while the designers of the other prototype realized the benefits of one critical technology were not worth its costs Fixes were identified and adjustments made so that moving into the next phase, risks for both designs were reduced in ways that also avoided costs. In the testing of both designs, their prototypes served as risk reduction tools.
When areas of high technical risk are not addressed through prototyping, it is not likely to reduce risk or to result in much gain. During early development of the Army’s Anti-Armor Submunition for example, prototypes were constructed and tested with the highest technical risk components excluded from the design . When moving into the next phase, these components became the major risk areas. Without demonstrating those areas of high technical risk that were essential to system success, the prototype’s ability to reduce risk was marginalised. As an acquisition strategy, prototyping did not provide much value.
For any given prototype used within DoD acquisition life cycle, the areas of highest technical risk appropriate for demonstration should vary. A prototype need only provide enough sophistication to address those risks that are most relevant to the next series of decisions These risks tend to vary by phase of the acquisition life cycle. Early in the life cycle, areas of high technical risk relate to technology development. As one nears Milestone B and into Engineering and Manufacturing Development, risks associated with systems development—such as risk in the areas of integration, manufacturability, producibility, and operational suitability—come to the forefront so a prototype’s maturation should vary depending on where it falls within these phases.
So as a matter of acquisition practice is that in terms of reducing risk through technology demonstration, all prototypes are not created equal. It also means that not all risks are suitable for reduction through early prototyping. Some risks, like those appearing later in the life cycle, are just incapable of being reduced without a prototype resembling the final design.
Regardless of the risks that may or may not be reduced in a particular prototype, not to be lost is the fact that all prototypes produce information. This information can and should be used to guide a full range of decision-making, from those occurring in the context of a specific acquisition to those on which the acquisition is based.
For instance, in the acquisition community, prototyping can assist in determining whether the benefits of a new technology outweigh its risks, and thus warrant further investment Prototyping can also be useful for evaluating the merits of a particular design approach, or in the science and technology community to guide the transition of technology outside of the laboratory Prototypes can also be used in the requirements community to evaluate operational concepts and needs When prototyping, the information provided is valuable. All should use it.
Prototyping Leads to Change
The ability for a wide community of users to capitalise on the knowledge prototyping provides is another benefit of prototyping, but it does not always result in efficiency. In the course of acquisition decision-making, some degree of change is expected to result from prototyping.
Indeed, change is what prototyping is all about. When acquisition decisions are based solely on paper studies, one would expect that change would be equally inevitable. It is. The difference is that with paper studies, the need for change is not recognised until greater capital investments have been made and major funding committed. At that point, it becomes more costly to undo what has been done.
Prototyping allows resources to be committed incrementally until the merits of a technology are better understood. When the merits are not there or are simply not worth the costs, prototyping gives reason to change course In this way, by allowing for change, prototyping provides a hedge against uncertainty Sometimes the uncertainty lies in a technology’s maturity. Sometimes the uncertainty lies in something more fundamental, such as an operational concept, requirement, or threat. In each case, prototyping provides an opportunity to change as more that was uncertain becomes known.
But in providing the opportunity to change, prototyping has a weakness. Too much change marginalises the value of prototyping, making it no more useful than the cheaper, less reliable paper studies it supplants. The more a prototype resembles a system’s final configuration, the less change it can tolerate and still provide the expected returns.
When the prototype and final configuration item closely resemble each other, minor changes can be accommodated. Major changes, such as those associated with an operational concept or a fundamental approach, cannot
The experiences encountered on the Air Force’s strategic airlift C-X program underscore this point: too much change in operational concept can marginalize the value of prototyping. The predecessor to the C-X program was the Air Force’s Advanced Medium Short Landing and Take-Off Transport program. The AMST program constructed and tested two full-scale representations of aircraft that emphasized tactical airlift.
Over time, however, the Air Force saw strategic airlift as more important and cancelled the AMST program in favor of the strategic-oriented C-X. The C-X program ultimately led to the development of the C-17 Globemaster III cargo aircraft, but the fundamental change in the operational concept between the two program marginalized the value of the full-scale AMST prototypes The expected benefits of prototyping never emerged. The C-17, for all its low risk technology and preceding AMST prototype phase, still encountered significant challenges going forward .
The Air Force’s first generation Advanced-Medium Range Air-to-Air Missile “AMRAAM” program provides another example of too much change, while at the same time illustrating the perils of too little. In its first generation, AMRAAM was to provide a capability similar to another air-to-air missile of the day, just with shorter range and in a smaller package. To meet the desired form factor, the prototypes used solid-state electronics instead of the conventional tube technology found in its predecessor. The technology could not perform, and when entering Full Scale Development, the design reverted to using the already proven tube technology.
When making the transition, program personnel relied on assumptions that said it could still meet the desired date for initial operational capability without acknowledging the significant step back they were making. AMRAAM essentially started over, and used paper studies to support its optimistic assumptions. Like the C-X program, significant delays and cost overruns ensued. Requirements should have been changed to reflect the new start.
So they were not, and difficulties followed. while prototyping allows and even encourages some amount of changes to be made, a limit must be imposed if prototyping investments are to be preserved There is no bright line rule here. Rather, the scope of allowable change appears to vary in inverse proportion to the scale and sophistication of the prototypes. When the change is significant and the prototypes relatively mature, then proceeding without a new prototype phase could be no more different than starting off with a paper design .
This is one reason the XV-15 tiltrotor prototype was not an effective precursor to the MV-22 Osprey. Though the technology’s feasibility had been demonstrated, its ability to meet an operational need was not, especially the advanced operational needs of the Marine Corps. Building the MV-22 Osprey on the basis of the XV-15 prototype was little more than starting from a paper design.
With prototyping, change may be expected, change may even be encouraged, but not all changes can be accommodated. The amount of allowable change has to be limited to ensure that the prototypes, as a tool for enabling better acquisition outcomes, survive the decision-making process.
1. Prioritise tech prototypes by industry partner more closely with Pentagon to share costs
2. Compare and contrast tech for identifying deployment constraints and develop index for scenario selections and details of return on ops
3. Recommend deployment for ops success and implement given evaluation surround scenario outcomes and distinguish weighting scheme indices/biases
4. Significant questions at Pentagon about defense market size; is issue of strategy and not just acquisition process improvement
5. One of big questions at Pentagon is how to generate unique military advantage from tech widely available on commercial market
6. Negotiate future contracts based on output costs/price risk; benchmarks align demand with supply capacity
7. Defense benchmark repository to find pricing assumptions at odds w/ like projects and embed tough targets into price
8.Pentagon and Congress working to rewrite statutes that are keeping staff busy checking off boxes instead of working on new tech.
9 Pentagon working with Congress to rewrite outdated acquisition rules and remove burdens from overworked programme directors
10. Pentagon acquisition structure charging in several direction at once, pushing initiatives aimed at how think about revamp tech
RSS Feed