​Site Visit Executive job will be to focus on defining requirements as precisely and realistically as possible from the start and then to experiment to see what works before committing big bucks.

There are going to be trade-offs and all this and the reporting lines may change to make sure we get the optimisation. Everybody recognises we have reached a point that we’ve got to just — it’s time to move from this industrial age system to a 21st-century-age system.”

Design Teams are a small group of individuals focused on defining, building, and testing solutions within a short time frame. A Release Train is a self-organizing, long-lived group of Teams, a team of teams, whose purpose is to plan, commit, and execute solutions together. Built around the organization’s core Value Streams, a Release Train exists solely to deliver on promised value by building beneficial solutions for the customer.

Using tools like a common Vision, Roadmap and Program Backlog, aims to complete goals within a specific period of time,.

Program Level is another key concept within design. Simply put, the Program Level is where development teams and other resources are applied to an important, ongoing development mission. Most Program Levels — such as teams, roles, and activities — revolve around a specific Release team, ensuring a constant flow of incremental, value-generating releases.

The first step is to understand and document user requirements and Constraints for system capability so acquisition process can meet requirements.

Availability/reliability parameters must be explained and guide trade-off studies of mission capability and operational support, defining baseline against which the new system will be measured.

Performance factors need to be matched up with user’s needs into clearly defined system parameters and allocate/ integrate parameters to relevant disciplines needed to realise
success.

Systems engineering attempts to optimise effectiveness and affordability as the capability is created. The systems approach makes sure the question What are the user needs and constraints? is answered before designing the answer.

The top-level programme plan for achieving required available/reliable is executed in manner to ensure requirements are achievable. Through understanding user needs and constraints, new capabilities begin to be defined.

Must establish the case for a materiel approach to resolve gaps in capability. The primary focus is to acquire quality products balancing process of satisfying user needs while improving mission capability and operational support, also adhering to scheduling constraints and justifiable acquisition costs.

During capability assessments, time and resources need to be set aside to measure and characterise current operational experience, organise metrics and supply line performance to reach conclusions about the causes of shortfalls.

It is also imperative to understand subsystem design complexity and influence on availability/reliability. Capabilities-based approach leverages the expertise of all service directorate activities defining new capabilities.

Primary focus is to ensure that joint force is properly equipped and supported to perform across disciplines to identify improvements to existing capabilities and crate new warfighting capabilities.

Process defines needed capabilities through characterisation of doctrine, organisation, training, materiel, leadership, and Labour at Job SItes. Availability/reliability levels are defined within this framework, principally in the category of materiel.

So Goal is to inform and share information among decision makers tasked with design, buy, use, and system support. Information to be shared includes user requirements, and how system will be used or potentially miss targets.

Key to any assessments is description of use/support location, constraints on what support is available for system, what information will be available to decision makers, and how that information will be verified.

Army needs to identify what might be missing, and if there is something, they will need to potentially initiate a new program within that portfolio.

In some cases, like the Future Vertical Lift portfolio, it is easier to see the path forward. And the Army is already heading toward using flight demonstrators to help define requirements for a future vertical lift aircraft.

By building and flying demonstrator aircraft, it gives the opportunity for the service to fail early and fail cheaply, and to learn from mistakes and get to a higher level of technical readiness earlier in the program.

But for other portfolios within the new modernization command, more work has to be done, in part due to the nature of the technology involved.

“The network is hard, it’s really, really hard because it’s complex because those types of capabilities or the technology is really in the commercial sector more than the military sector, and it’s moving quickly and yet you can’t just take it from industry and put in the military world because you have to make it secure, it has to be ruggedized and it has to be able to operate in certain environments, and so that is the challenge.

Making a tougher job for future command lead in charge of the network, the Army decided to curb the cornerstone capability of its tactical network, the Warfighting Information Network-Tactical, or WIN-T, system, in favor of other capabilities. The service said it needed to entirely reboot its tactical network to operate against emerging threats on the battlefield.

With all of this, you do have to understand you have to get your requirements right, and for the network … the key, or part of the key going forward, has to be to understand the architecture and to map it out so we have the plan going forward.”

“It’s like building a house — you have to have a blueprint.” Having a blueprint doesn’t necessarily mean deciding who will supply the fixtures or materials or what will be used, but it defines what is needed.

Leadership wants to see the services get away from the idea of filling capabilities with interim, gap-filler solutions that would be scrapped once a next-generation capability comes online.

Leadership does not want to make perfect to be the enemy of the better. Let’s not think as much about interim capabilities. If we made the requirements so high, if we raised the bar so high that we think we need to have an interim, maybe we need to kind of lower that bill, those requirements.

It’s not perfect but it’s better than what we have now. And then we build a system that we can scale, that we can modularize, that is kind of open architecture that we can kind of build upon.”

Congressional spending has become so unpredictable, the defense industrial base is shrinking and the weapons systems of tomorrow are not being developed today.

Defense spending cuts in recent years caused a dramatic number of defense industry suppliers to leave the market while chilling industry’s research and development activities.

“Though the defense budget had been declining in the years leading up to sequestration in FY 2013, the enactment of sequestration and budget caps marked a severe market shock that had a considerable impact on the defense industry.

The cyclical nature of department budgeting, including delays in getting new spending approved, is a problem for all but the largest vendors. With annual unsteady funding, DoD has been unable to “send demand signals to industry.”

“The reality is that the Defense Department does not exist for the purpose of taking care of the industrial base. it’s the other way around, So, what the Department of Defense has to do is to ensure, to the extent that it can while doing its mission, that there is a healthy industrial base to support it.

“In terms of keeping the industrial base healthy, our design teams and capabilities to build cutting-edge, state-of-the-art, ten-years beyond state-of-the-art programs is essential in great power competition. “And it’s been allowed atrophy too much.”

Decades ago Army introduced the M109A1 155mm turreted self-propelled howitzer (SPH), called the Paladin. An artillery piece that could keep up with mobile armored formations and survive counterbattery fire was essential to the Army’s mission of deterring high-end conventional conflict. The Paladin is currently the primary fire support system for the Army’s Armored Brigade Combat Teams (ABCTs).
.
The Paladin was not one of the U.S. Army’s iconic “Big Five” modernization programs. Nevertheless it, along with other major platforms such as the Multiple Launch Rocket System and the High Mobility Multipurpose Wheeled Vehicle or Humvee, has defined the character of the Army’s combat capabilities for decades.

The M109 today is nothing like the system that first saw service in the Army more than fifty years ago. It has been almost continually upgraded. Improvements were made to virtually every Paladin component including the howitzer itself, fire controls, engine and drive train, armor and communications.

Currently, the howitzer is undergoing yet another major upgrade, which is more of a modernization effort. What was called the Paladin Integrated Management program and is now the M19A7 SPH and M992A3 Carrier Ammunition Tracked vehicle (CAT), is intended to provide major improvements to the system’s mobility, reliability and performance.

Both vehicles are essentially being rebuilt, using major components of the Bradley Fighting Vehicle inside a new hull. Commonality of parts between the Paladin and Bradley will improve overall sustainment in the ABCT.

In addition, the M109A7 will incorporate a state-of-the-art digital backbone, enhanced power generation system and electric gun drive and rammer. Notably, several of these new technologies were originally developed under the Non-Line-of-Sight Cannon (NLOS-C) program that was part of the now-canceled Future Combat System.

The Paladin upgrade program is being conducted through a special DoD/Industry partnership splitting skilled labor as well as critical facilities, with engineering support, components and supply chain management.

,Army is looking to leverage this latest set of improvements to the Paladin as the base for another round modernization. The newest Paladin variant can support a larger caliber howitzer that will be able to send projectiles out to 70 km, nearly triple the howitzer’s current range. Even greater ranges are possible with the new artillery rounds being developed for the Army. The improved Paladin, were it also equipped with a longer-range cannon, could help meet the Army’s critical shortfall in long-range fires.

To an Army determined to change the way it pursues modernization, the history of the Paladin program is a cautionary tale. Two efforts at replacing the Paladin, the Crusader and NLOS-C, foundered due to a combination of requirements mess, technology overreach, high costs and changing international threats. Paladin remains and, when upgraded, will operate as an effective part of the ABCT for decades to come.

The Paladin represents both what is right and wrong with the Army’s approach to modernization. What is right about its approach is the ability to continually improve existing platforms and systems. This minimizes technological risk as well as the opportunity costs associated with major changes in equipment.

Through a process of incremental modernization, the Army could soon have a self-propelled howitzer that in virtually all respects is an entirely different system than the one deployed decades ago.

Add to the new platform the latest artillery projectiles, themselves the product of incremental advances, and the result is a major new military capability.

What is wrong with Army acquisition is the penchant of that same system, or at least some of its leaders, to become fixated on the goal of inventing something new, even transformational.

Sometimes this is a function of requirements in search of capabilities to justify a particular vision of future combat. In other cases, it is a reflection of the mistaken notion that technological change demands a response by the acquisition system.

Truly groundbreaking technological change is a rare event. Exploiting technological advances to create a new weapons system or military platform is even rarer. Many things we consider technological revolutions, such as the mobile phone, are the result of a series of incremental advances that are brought together over time in a new piece of hardware.

The leadership of the Army’s new Futures Command must guard against taking their organization’s title literally. They will need to draw a clear line between modernization as a leap ahead and the same outcome resulting from continual technological improvements. Which is the future? The answer is both.

Congress, the defense industry, and Army all believe the Pentagon must fundamentally change the structure and performance of its acquisition enterprise after decades of tweaks and inertia.

Numerous institutional adaptations and reorganizations have been initiated in the past, many of which have led to familiar conditions: cumbersome spans of control; complex communication and procedural structures; difficulty prioritizing competitive programs and budget requirements; decreased accountability and effectiveness; and, disconnects between futures and acquisition procurement strategies, to name a few.

For the Army those conditions materialized into “a lost decade of procurement” marked by, “reductions in modernization, procurement, and RDTE funding”; and a “wave of OSD requirements.

Recent acquisition enterprise efforts to coordinate and create a shared visualization stem from a current state assessment that “acquisition’s underlying problems are exacerbated during conflict, when warfighters are in harm’s way, so natural tendency has been to work around the system rather than fix it.

Army has determined that now is the time to fix the system, as “wartime adaptation against a peer adversary will require capability generation to be exponentially faster than it was for recent operations.

Are we in need of “incremental” or “disruptive” reform? If “disruptive” change is in the cards, the alignment of forces, sustainment, training, combat developments or modernization functions within streamlined commands is one potential course of action. However, what the Army is ready for, what the specific content of the reform will be, and its tolerance levels for disruption while heavily engaged in current operations are yet to be determined.

It is clear that any new modernization command must demonstrate value to industry, research and development unit within and external to the U.S. Army, but, even more so to the warfighters whose equipment readiness is essential to the future of the force.

The new command will be challenged to:

1. Streamline the requirements process and major weapons systems development

2. Simplify current command structure currently designed to approve requirements

3. Overcome a risk-averse acquisition culture optimized for individual and organizational outcomes within stove-piped organizations,

4. Provide a vision-to-victory or futures strategy that alleviates tensions between present requirements and future readiness

5. Improve integration of operational concepts into acquisition strategies, presently determined and developed by multiple unorganized multi-star commands.

6. Create point of contact command with ownership of futures to formulate consensus on a long-term procurement strategy

7. Overcome the usual reforms and existing R&D structure by leverage industries advanced technologies and modernization

8. Increase innovation, improve balance and contrast in approaches to R&D

9. Establish conditions for a “succeed-fast” and “fail-fast” strategy throughout the defense acquisition life cycle.

10. Establish a wartime acquisition enterprise capable of rapid adaptability to threat capabilities today and in the future.

​Marine Warfighting Lab has developed processes that allow senior leaders to make smart decisions about the technologies they need for programs of record, the technologies they do not need and the technologies that may quickly become

obsolete. But making choices can be challenging, he says, because people are swayed by technology and the “bright, shiny new object.

The technological renaissance is providing a lot of options, and with all the technologies you could make a case for each one. But they cannot all be pursued. “The problem is that we can’t afford to buy everything so we have to make an assessment of capabilities and to make recommendations on capabilities that will have the greatest return on our investments.”

“The hardest part is trying to make sure that the people who are in love with their technologies understand what it is that they are in love with. It is difficult to convince people that, ‘Yes, it is a great capability, but is it greater than this other capability over here?’

“Everything we consider has to get a fair shake. There have been technologies that nobody liked that turned out to be pretty effective.” And at the end of the day, when the Marines are kicking in the door, those technologies could make all the difference.

“We’re really quite satisfied with what’s going on there for the Marine Corps. The service pushes for “projects that tend to be more practical, more physical. One promising program is the FVL platform.

When Future Vertical Lift systems enter Army service, new drones and high-speed helicopters won’t be the only things flying through the air. Linking the FVL aircraft will be an invisible network, transmitting data directly from one machine to the next, without the clumsy, slow-moving intermediary of human voices
over the radio or human hands on a keyboard.

This Modular Open Systems Architecture will be essential to FVL’s performance across the board, from combat to maintenance to long-term upgrades.

“Probably one of the hardest things we’re going to do is affordability. Because military systems have to serve for decades, the really big bill here isn’t procurement, it’s long-term maintenance, sustainment, and upgrades to keep the aircraft not only functional but up to date.

Replacing worn-out or obsolete components with new and better ones adds up tremendously over the years. As with other weapons system platforms, percent of our total cost of ownership is the sustainment of the fleet.”

The Army’s worked hard to control those costs with its current fleet of aircraft. But there are hard limits on how efficient you can become when different types of aircraft – and even aircraft of

the same type built at different times – use different electronics, none of which were built with ease of maintenance and upgrades in mind.

Most components don’t have built-in diagnostic chips the way a modern automobile does, for instance, so the only way to check whether they’re working is to unscrew an access hatch and look inside. If something is broken or just out of date, then swapping a single part, even something as simple as a cockpit display, may require laboriously rewriting code on several other systems that interact with it.

“In our current systems, Army Aviation has done a phenomenal job of leading the Army on condition-based maintenance. We are pretty far out there about collecting data and knowing when something is going to fail.”

Using diagnostics this way allows repair crews to intervene when needed, instead of either waiting for a problem to become obvious – which means problems become more dangerous and expensive to fix – or conducting laborious preventive maintenance just-in-case – which consumes countless man-hours of highly trained ground crews.

“However, we did that by platform, and each individual platform uses a different system to be able to do that. As we move forward, the intent is to go ahead and make sure that we have a common platform for condition-based maintenance. We think this will fundamentally change how maintenance is done… which will ultimately drive down cost.”

Those common maintenance diagnostics will be built into the Modular Open System Architecture. Rather than prescribe to contractors how to do it or let each contractor come up with their own, incompatible proprietary solution, the Army will dictate common standards and interfaces, made available to all – ie “open.” That should allow the military to replace a piece of code from one vendor with better code from another, without having to rewrite the rest of the system – this means “modular.”

“In the draft documents… we have put the hooks in there to make sure that industry knows that that is going to be a requirement. When we send out the proposals to industry, we will direct that certain things be common as far as condition-based maintenance is concerned: what they measure, how they measure it, how the ones to zeros are holding, so that the unit…can get that same data, regardless of whether they’re looking at FARA, FLRAA, or one of our enduring systems.”

The Army hopes to backfit at least parts of MOSA onto its existing helicopter fleets, which will serve for decades to come, with the FVL aircraft replacing them over time. Whenever possible, it will upgrade different types of existing aircraft in ways that make them more compatible and use more common systems. But it’s with FVL that MOSA will fully come into its own.

The Army’s determined to enforce common standards and common software in a way it never has before. The number one challenge we have with MOSA is … that discipline and management. “What allowed the enduring fleet of aircraft to wind up with different architectures because there was not a driving central body that said, ‘this is the architecture that you are going to go with.’… With MOSA, we have that.”

“It really comes down to defining that DoD standard, and defining that DoD interphase, and then … holding to it. The PEO has led the charge with the architecture control working group, meeting quarterly, with industry participating.”

Getting MOSA to work is a literal essential to the success of the mission. It won’t just transmit maintenance data, helping forestall breakdowns and prevent accidents. It will also transmit tactical data, helping manned and unmanned FVL aircraft coordinate with each other in combat.

“That system architecture has open systems, interfaces, and gateways, so we can push data” seamlessly across the force, without incompatible systems on different aircraft stopping the flow of information. “We’re refining our data formats to auto-populate nine-lines i.e. calls for urgent evacuation, calls for fires i.e. artillery and air strikes, our production,
exploitation and dissemination of intelligence.”

How soon will elements of this system become available to soldiers? It will enter service before the manned FVL aircraft do. The FVL modernization team is working with their counterparts for the Army network. They’ll field new data-sharing systems as part of the 2023 upgrade, known as Capability Set 23.

But that will just be the beginning of a multi-year effort to build Army aviation’s invisible, digital backbone.

Computations that must be spread out across the entire data set—like finding an average or doing a statistical analysis—must be split up into subjobs, spread out across all of the nodes, and then aggregated when it’s done

Here we introduce a conceptual, yet quantifiable, architecture framework by extending the notion of system modularity in its broadest sense.
Acknowledging that modularity is not a binary feature and comes in various types and levels,

the proposed framework introduces higher levels of modularity that naturally incorporate decentralized architecture on the one hand and autonomy in agents and subsystems on the other.

This makes the framework suitable for modularity decisions in Systems of Systems and for analyzing the impact of modularity on broader surroundings. The stages of modularity in the proposed framework are naturally aligned with the level of variations and uncertainty in the system and its environment, a relationship that is central to the benefits of modularity.

The conceptual framework is complemented with a decision layer that makes it suitable to be used as a computational architecture decision tool to determine the appropriate stage and level of modularity of a system, for a given profile of variations and uncertainties in its environment

The fundamental systemic driving forces and trade-offs of moving from monolithic to distributed architecture are essentially similar to those for moving from integral to modular architectures.

The spectrum, in conjunction with the decision layer, could guide system architects when selecting appropriate parameters and building a system-specific computational tool from a combination of existing tools and techniques.

Environment/Threat Simulation: The ability to create, generate, or replicate ideal, distorted, or “real-life” signals is essential in the design, testing, and operation of electronic warfare and avionics systems.

Building a threat environment involves making or recreating a lifelike radio signal as precisely as possible. This can be especially challenging, because most solutions compromise speed, bandwidth, signal fidelity, and memory.

However, next-generation arbitrary waveform generators are now coming on the market that enable much easier generation of complex signals than in the past.

To apply open standards, you begin at the chassis level, develop the common backplane interfaces, and then move toward replacing modules. Developing commonality at that level will take time, but the military customers are discussing that path in order to align to those goals.

1. New applications that need to be built quickly

2. Enterprise or business applications that need to mirror traditional network departments and processes

3. Teams with inexperienced developers who don’t understand other architectures yet

4. Applications requiring strict maintainability and testability standards

5. Asynchronous systems with asynchronous data flow

6. Individual data blocks interact with only a few of the many modules

7. Tools used by a wide variety of people

8. Clear division between basic routines and higher order rules

9. Fixed set of core routines and a dynamic set of rules that must be updated frequently

10. Development teams that are spread out, often across the globe

​Competitive prototyping has been highly encouraged, if not mandated, as a preferred approach to major systems acquisition in DoD. Its repeated encouragement is due in part to its description as a best practice by specially formed task forces.. In most instances, competitive prototyping is presented as a tool for stoking creative thought, for improving decision-making, and for leading to better acquisition outcomes.

In other instances, its value has been questioned. In practice, competitive prototyping has not always delivered on its promises. Part of its mixed results has been attributed to widespread confusion over the meaning of terms and how prototyping should be pursued on a competitive basis. Building off lessons learned, this report provides an overview of prototyping accompanied by a description of how competitive prototyping has and could be practiced better within DoD.

Summary

Building off lessons learned, an overview of prototyping is provided and is accompanied by suggestions for doing so better and on a competitive basis within DoD.

The debate over prototyping is not so much over whether prototyping is good but when it provides value. At Milestone C, prototyping as a pre-requisite to a low-rate initial production decision is well accepted. The large commitments of capital that accompany a production award warrant some assurance that the technology to be produced will deliver as promised. Prototyping provides that assurance. At Milestones A and B, however, prototyping has not gained much traction, especially when cheaper alternatives seem to be available. 

Paper competitions coupled with systems analysis, models and simulations, and other estimation techniques have been the desired alternative at these earlier stages of the life cycle. These methods are thought to be both cheaper and less time consuming than prototyping, and therefore more cost effective. The critics say otherwise.

Critics argue that if prototyping is good enough to support a production decision, why not use it earlier to justify a formal program start at Milestone B or a comparison of alternatives at Milestone A. Time and time again, they say, paper competitions and capability estimates have shown major systems acquisition to be plagued more by the unknown risks of systems development than the known ones. Experiences learned when developing aircraft illustrate this point: all the analysis in the world cannot reveal what one does not know. Prototyping can.

The response to these criticisms has been that, while prototyping may provide value, there is too much change early in the life cycle to make prototyping worthwhile. Changes in technology, performance objectives, and operational concepts prior to Milestone B marginalize the value of prototyping in the early stages of the life cycle. Besides, the response goes, prototyping can provide little additional knowledge when compared to other acquisition techniques without completing a detailed design. And going through detailed design prior to Milestone B is just setting oneself up to do it all over again in Engineering and Manufacturing Development. The debate, therefore, is over whether prototyping early in the life cycle can ever be cost effective.

A similar debate surrounds the use of competition. All would agree competition is good; not all would agree it is always sensible. Competition for competition’s sake has never been the goal. The goal is to get a better value. Competition may be a means to this end, but, in the defense market, to invite more competition invariably entails more costs.

When these costs exceed their expected returns, competition no longer makes sense. Like the debate over prototyping, with competition, the debate is over how competition should be approached so that it provides enough value to warrant its costs.

Recently, the debates over early prototyping and competition have converged in the context of mandates competitive prototyping for major systems acquisitions up to Milestone B and compels it to be a continued consideration throughout the life cycle. In some ways, competitive prototyping’s resurgence should be no surprise. 

The hallmark of competitive prototyping’s ascendance has always been the threat of shrinking defense budgets. With shrinking defense budgets on the horizon and sequestration looming, this is no less true today. But its prevalence as a means for effective reform is counterintuitive. Competitive prototyping not only requires more development dollars up front, it takes more time, and its success in DoD has been mixed.

The issue facing DoD is how to deal with the additional costs of competitive prototyping so that better acquisition outcomes can follow. Fortunately, the lessons from previous periods of competitive prototyping reforms provide some clues. They suggest both how prototyping can remain cost effective and how competition can be sensibly pursued. 

Reintroducing these lessons and building from them in ways applicable to today’s acquisition environment is the first step to implementing competitive prototyping reforms. It is also the first step towards obtaining better acquisition outcomes. After all, competitive prototyping does not guarantee such outcomes will follow; it only makes them possible. The goal is to make them possible using fewer dollars than before.

Prototyping Attributes
.
While competitive prototyping can be more valuable than paper competitions, it can also confirm what is already known. The key to uncovering more value and making prototyping more cost effective lies in understanding what is meant by the terms “prototype” and “prototyping.” Despite decades of intermittent prototyping in DoD, settled definitions for these terms have not yet emerged. Conceptually, they are easy to comprehend if not always to explain, at which point a reference is usually handy. but no reference can explain how they relate to DoD acquisition process with its various milestones and decision points. The leap from everyday practice to the highly specialised DoD procurement process is too great.

Lack of terminology has divided the policy of competitive prototyping from its practice in ways that have frustrated realisations of its promises. Practitioners have not understood how prototyping should be approached. Policy-makers have struggled to organise principles around prototyping from which better outcomes can emerge. Settling on definitions for the terms “prototype” and “prototyping” in a way that instructs those who make policy as much as it guides those who must build one is the first step in doing so.

Prototypes Are Test Articles.

Whereas paper studies estimate a technology’s capabilities, prototyping demonstrates those capabilities through testing. Test articles are designed, constructed, and tested to demonstrate the capabilities of some technology or system. In its simplest explication, the test article is the prototype, and as a test article, it can take many forms and represent various states of maturity depending on the aims of the test .Whether the test article represents a concept, subsystem, or end item that is full scale, fully capable, or something that is much less mature, all are forms of prototypes. The process of using these test articles to demonstrate capabilities is the practice of prototyping.

Prototyping’s emphasis on technology demonstration is one reason it has been popular during periods of falling defense budgets. With fewer procurement dollars to spend, there is less appetite for risky expenditures on unproven technologies. To warrant greater investment, technology must prove itself, and more than just in operational terms It also must prove to be affordable. 

Prototyping “should allow us to fly—and know how much it will cost—before we buy” But knowing what to fly to justify what to buy has been a recurring difficulty. Conclusions are inconsitent between prototyping something that is production representative and something that is less sophisticated. Of the two, prototyping a production representative test article is the most conservative approach. 

Building production representative prototypes in advance of every major program start allows a full understanding of a technology’s costs and benefits. With a production representative approach to prototyping, risks can be contained. Fixed-priced contracts can follow. Programmatic success would be more likely. these are the goals of every prototyping and development effort as it nears production Rarely, however, is such an approach cost effective, especially on a competitive basis prior to Milestone B 

Prior to Milestone B, prototyping requires one to be selective, and being selective is where the benefits and difficulties of prototyping lie. It may not be cost effective to build a production representative prototype prior to Milestone B, but building something less sophisticated may be. Whether it is or not depends on whether one can selectively design, construct, and evaluate a prototype in ways that provide more reliable information than paper studies and analysis can provide. 

The key to remaining cost effective is to invest no more capability in the prototype than is required to further the prototype’s primary purpose The key to making prototyping more reliable than paper studies is to target those capabilities paper studies struggle to estimate accurately. Perfecting both these aspects of prototyping in a test article of limited capability is extremely difficult Doing both, however, is essential to realising a prototype’s full potential and serving its ultimate end: to generate information and guide future decisions.

Prototypes Guide Decisions.

DoD multi-phased acquisition process has many decisions points all corresponding to individual phases. At different decisions points, the degree and types of knowledge required to support a particular decision varies. But having sufficient knowledge at each point is essential to enabling better acquisition outcomes Where insufficient knowledge exists, resources are committed when not enough about the technology is known. 

Technical risk is underestimated; cost increases and schedule slips follow This has been the downside of basing decisions solely on paper studies. They tend to underestimate what is not already known. 

Prototyping enables better acquisition outcomes by improving the reliability of available information. Prototyping injects an early dose of realism into the assumptions and conclusions at the core of previous studies and analysis, thereby making them more useful. Realism comes through demonstrated capabilities. As more capabilities are demonstrated, more becomes known, and the more justification there is for the decisions made But the more capabilities are added, the more costs will be incurred, and the more closely one must evaluate whether the information being provided is worth the extra costs. 

There is a line where prototyping’s costs begin to exceed its returns. For prototyping to be a productive exercise, prototyping must keep on the positive side of the line. In practice, this requires prototyping with a particular end in mind, investing only in activities that support this end, and then using the information that results to chart a better course.

Charting a better course through the early stages of the acquisition life cycle does not require all the capabilities of a final system to be embedded in a prototype. A production representative prototype at Milestone B is not only overkill, it resembles a waste prototype need have no more capability than is necessary to support the next series of decisions Ensuring the prototypes are more valuable than paper studies, though, requires that certain capabilities be targeted. 

Prototypes should target the areas where paper studies are most weak: areas of high technical risk that are essential to system success. This targeting is essential to uncovering the unknowns that plague acquisition programs based on paper and to making prototyping worthwhile. It is also essential to reducing risk in advance of the next phase and positioning an acquisition program to capture efficiencies later on These all make prototyping more cost effective and more desirable than limiting oneself to paper alone. 

During the Advanced Tactical Fighter’s prototype phase, a number of fixes for the YF-22 prototype were identified early and incorporated at lower cost as a part of the next phase. The Navy’s A-12 program took a different approach; its early system design was based almost entirely on paper. As Full Scale Development ramped up—which is today’s equivalent to Engineering and Manufacturing Development—a number of technical problems emerged that engulfed all hopes of successfully implementing the paper design.
 
To a certain degree, the Advanced Tactical Fighter program encountered comparable problems, but not all were technical ones. Problems with funding, work sharing among contractors, and an unstable industrial base hindered efforts to capitalise on promised efficiencies Thus, while the Advanced Tactical Fighter program enjoyed a successful prototype phase, it shows how even a strong start can be overwhelmed by other issues down the road Prototyping may enable better acquisition outcomes, but it does not guarantee they will follow.

The goal with prototyping is to make better outcomes possible, and demonstrating areas of high technical risk is essential to reaching this goal. Demonstrating areas of high technical risk is also essential to making prototyping more cost effective. When areas of high technical risk are demonstrated through prototyping, it presents an opportunity to address problems early, when rates of expenditures are lower, and without risking the success of the next phase.

In development, problems always emerge, and when development is based solely on untested analysis and estimates of a design, problems tend to emerge later in development when expenditure rates are higher. Prototyped programs encounter similar problems, but problems tend to be identified earlier and can be fixed more cheaply, as in the case of the YF-22. 

Capturing this efficiency, an example of cost avoidance, bolsters prototyping’s cost effectiveness. Capturing enough of them so that the extra development dollars invested in prototyping can be recouped later is what makes prototyping more worthwhile Sometimes these efficiencies result in reduced cost; most of the time they result in reduced risk.

The Air Force’s Close Air-Attack-Support program, the program that led to the highly successful A-10 aircraft, provides an example of prototyping’s ability to reduce risk and avoid costs. During flight test, the designers of one prototype identified a flaw in wing design while the designers of the other prototype realized the benefits of one critical technology were not worth its costs Fixes were identified and adjustments made so that moving into the next phase, risks for both designs were reduced in ways that also avoided costs. In the testing of both designs, their prototypes served as risk reduction tools.

When areas of high technical risk are not addressed through prototyping, it is not likely to reduce risk or to result in much gain. During early development of the Army’s Anti-Armor Submunition for example, prototypes were constructed and tested with the highest technical risk components excluded from the design . When moving into the next phase, these components became the major risk areas. Without demonstrating those areas of high technical risk that were essential to system success, the prototype’s ability to reduce risk was marginalised. As an acquisition strategy, prototyping did not provide much value.

For any given prototype used within DoD acquisition life cycle, the areas of highest technical risk appropriate for demonstration should vary. A prototype need only provide enough sophistication to address those risks that are most relevant to the next series of decisions These risks tend to vary by phase of the acquisition life cycle. Early in the life cycle, areas of high technical risk relate to technology development. As one nears Milestone B and into Engineering and Manufacturing Development, risks associated with systems development—such as risk in the areas of integration, manufacturability, producibility, and operational suitability—come to the forefront so a prototype’s maturation should vary depending on where it falls within these phases.

So as a matter of acquisition practice is that in terms of reducing risk through technology demonstration, all prototypes are not created equal. It also means that not all risks are suitable for reduction through early prototyping. Some risks, like those appearing later in the life cycle, are just incapable of being reduced without a prototype resembling the final design.

Regardless of the risks that may or may not be reduced in a particular prototype, not to be lost is the fact that all prototypes produce information. This information can and should be used to guide a full range of decision-making, from those occurring in the context of a specific acquisition to those on which the acquisition is based.

For instance, in the acquisition community, prototyping can assist in determining whether the benefits of a new technology outweigh its risks, and thus warrant further investment Prototyping can also be useful for evaluating the merits of a particular design approach, or in the science and technology community to guide the transition of technology outside of the laboratory Prototypes can also be used in the requirements community to evaluate operational concepts and needs When prototyping, the information provided is valuable. All should use it.

Prototyping Leads to Change

The ability for a wide community of users to capitalise on the knowledge prototyping provides is another benefit of prototyping, but it does not always result in efficiency. In the course of acquisition decision-making, some degree of change is expected to result from prototyping. 

Indeed, change is what prototyping is all about. When acquisition decisions are based solely on paper studies, one would expect that change would be equally inevitable. It is. The difference is that with paper studies, the need for change is not recognised until greater capital investments have been made and major funding committed. At that point, it becomes more costly to undo what has been done.

Prototyping allows resources to be committed incrementally until the merits of a technology are better understood. When the merits are not there or are simply not worth the costs, prototyping gives reason to change course In this way, by allowing for change, prototyping provides a hedge against uncertainty Sometimes the uncertainty lies in a technology’s maturity. Sometimes the uncertainty lies in something more fundamental, such as an operational concept, requirement, or threat. In each case, prototyping provides an opportunity to change as more that was uncertain becomes known.

But in providing the opportunity to change, prototyping has a weakness. Too much change marginalises the value of prototyping, making it no more useful than the cheaper, less reliable paper studies it supplants. The more a prototype resembles a system’s final configuration, the less change it can tolerate and still provide the expected returns. 
When the prototype and final configuration item closely resemble each other, minor changes can be accommodated. Major changes, such as those associated with an operational concept or a fundamental approach, cannot 

The experiences encountered on the Air Force’s strategic airlift C-X program underscore this point: too much change in operational concept can marginalize the value of prototyping. The predecessor to the C-X program was the Air Force’s Advanced Medium Short Landing and Take-Off Transport program. The AMST program constructed and tested two full-scale representations of aircraft that emphasized tactical airlift. 

Over time, however, the Air Force saw strategic airlift as more important and cancelled the AMST program in favor of the strategic-oriented C-X. The C-X program ultimately led to the development of the C-17 Globemaster III cargo aircraft, but the fundamental change in the operational concept between the two program marginalized the value of the full-scale AMST prototypes The expected benefits of prototyping never emerged. The C-17, for all its low risk technology and preceding AMST prototype phase, still encountered significant challenges going forward .

The Air Force’s first generation Advanced-Medium Range Air-to-Air Missile “AMRAAM” program provides another example of too much change, while at the same time illustrating the perils of too little. In its first generation, AMRAAM was to provide a capability similar to another air-to-air missile of the day, just with shorter range and in a smaller package. To meet the desired form factor, the prototypes used solid-state electronics instead of the conventional tube technology found in its predecessor. The technology could not perform, and when entering Full Scale Development, the design reverted to using the already proven tube technology. 

When making the transition, program personnel relied on assumptions that said it could still meet the desired date for initial operational capability without acknowledging the significant step back they were making. AMRAAM essentially started over, and used paper studies to support its optimistic assumptions. Like the C-X program, significant delays and cost overruns ensued. Requirements should have been changed to reflect the new start.

So they were not, and difficulties followed. while prototyping allows and even encourages some amount of changes to be made, a limit must be imposed if prototyping investments are to be preserved There is no bright line rule here. Rather, the scope of allowable change appears to vary in inverse proportion to the scale and sophistication of the prototypes. When the change is significant and the prototypes relatively mature, then proceeding without a new prototype phase could be no more different than starting off with a paper design .

This is one reason the XV-15 tiltrotor prototype was not an effective precursor to the MV-22 Osprey. Though the technology’s feasibility had been demonstrated, its ability to meet an operational need was not, especially the advanced operational needs of the Marine Corps. Building the MV-22 Osprey on the basis of the XV-15 prototype was little more than starting from a paper design.

With prototyping, change may be expected, change may even be encouraged, but not all changes can be accommodated. The amount of allowable change has to be limited to ensure that the prototypes, as a tool for enabling better acquisition outcomes, survive the decision-making process.

1. Prioritise tech prototypes by industry partner more closely with Pentagon to share costs
​
2. Compare and contrast tech for identifying deployment constraints and develop index for scenario selections and details of return on ops

3. Recommend deployment for ops success and implement given evaluation surround scenario outcomes and distinguish weighting scheme indices/biases

4. Significant questions at Pentagon about defense market size; is issue of strategy and not just acquisition process improvement

5. One of big questions at Pentagon is how to generate unique military advantage from tech widely available on commercial market

6. Negotiate future contracts based on output costs/price risk; benchmarks align demand with supply capacity

7. Defense benchmark repository to find pricing assumptions at odds w/ like projects and  embed tough targets into price

8.Pentagon and Congress working to rewrite statutes that are keeping staff busy checking off boxes instead of working on new tech.

9 Pentagon working with Congress to rewrite outdated acquisition rules and remove burdens from overworked programme directors

10. Pentagon acquisition structure charging in several direction at once, pushing initiatives aimed at how think about revamp tech

​Prototype Breakdown Structure provides basis for communication in all phases of acquisition process. Serves as common link unifies planning, scheduling, cost estimating, budgeting, contracting, configuration control & performance reporting disciplines. Consistent communications permits DoD to evaluate progress in terms of contract performance.

“Prototype” and “Prototyping” Defined

A test article used for gathering knowledge to guide decision-makers who instigate change—these are the primary attributes of a prototype. Couple them with the general guidelines provided above and the following workable definitions for the terms “prototype” and “prototyping” emerge:

A “prototype” is a test article designed to demonstrate areas of high technical risk that are essential to system success. A prototype need not be a full system, but, in scope and scale, it is tailored to accommodate series of decisions, and as such, can represent a concept, subsystem, or end item according to the decisions to be made. Rather than reflect the final design, prototypes are built with the expectation that, as decisions are made, change will follow.

“Prototyping” is the practice of testing prototypes, of appropriate scope and scale, for the purpose of obtaining knowledge about some requirement, capability, or design approach. The knowledge obtained informs a decision-making process the output of which results in some degree of change. The degree of allowable change is bounded, in inverse proportion, by the scope and scale of the prototype.

These definitions have been refined in different ways to emphasise various aspects of prototyping that, in the multi-faceted decision-making process of today’s acquisition space tend to be missed. Also emphasised are aspects of prototyping that preserve it as a cost-effective alternative to cheaper, less reliable acquisition methods.

Unlike other definitions, these definitions do not exclude production representative prototypes. Rather, they embrace them with the caveat that, with prototypes of a production representative caliber, the scope of allowable change is much less, the range of available decisions more narrow, than for less sophisticated prototypes appearing earlier in the life cycle. Among such prototypes are “demonstrator prototypes” and “advanced development prototypes,” both of which appear between Milestones A and B.

For practitioners charged with prototyping in advance of Milestone B, the distinctions between these two types of prototypes have been confounding Both are properly considered prototypes, but each resides on different sides of the development divide. Technology demonstrators are more closely associated with technology development Advanced development prototypes are more closely associated with systems development. 

On visual inspection, their differences are not intuitively obvious. But the latter is much more sophisticated and better suited to informing Milestone B. The former is much less so, but is also much cheaper to design and build. Getting the latter and not the former requires a great deal of judgment. Better policies about the two would also provide a great deal of help, which is part of the reason these definitions and their refinements are reintroduced here.

Their aim is to provide practitioners a rudimentary approach to prototyping so that it remains a viable option to cheaper alternatives. They also provide a baseline upon which more thorough policies can follow. More thorough policies are needed if better acquisition outcomes are to follow. Having working definitions is just a first step in that direction.

Competitive Prototyping and its Dilemmas

When compared to paper studies, prototyping presents an early dilemma: it takes more time and it costs more money, at least in the short term. So prototyping is justified only to the degree it allows for better materiel solutions or future savings. The objective is to spend more in development so that better systems can be fielded more quickly and for less overall cost In practice, prototyping should be a leveraged investment. The dilemma is to prototype in ways that provide a positive return.

When prototypes are evaluated on a competitive basis, the early dilemma of prototyping is exacerbated. Rather than fund one prototyping effort, DoD must fund two or more. Added to the tough decision of determining what to prototype are the tougher decisions of determining how competition will be pursued and the resulting prototypes evaluated. In most cases, the evaluation will support a future down-select decision, but until complete, the funding and management of multiple contractors puts incredible pressure on the procurement budget. Rarely does the addition of a contractor team come with a proportional increase in dollars.

Consequently, for competitive prototyping to be a leveraged investment, certain trade-offs are necessary. Development dollars must be allocated to support the right mix of activities and appropriately controlled so that a positive return might result. Competition, meanwhile, must be harnessed in a way that allows better performing systems to emerge at lower costs. In competitive prototyping, decisions are no longer limited to just those associated with the prototype. With competition, there are additional challenges.

Dealing with Budgetary Pressures

If prototyping is more costly at the front end, then prototyping on a competitive basis is worse. Not only must multiple prototypes be designed and tested as part of a competition, but with competition, the prototypes are usually more advanced .When resources are in scarce supply, this additional demand creates incredible pressure on the research and development budget. It also creates incredible risk for industry in ways that undermine the value of prototyping and hinder competition. The challenge is to find ways to channel these budgetary pressures effectively, not only for the success of the prototypes, but also for the success of DoD.

To channel budgetary pressures effectively requires dealing with risk effectively by allocating resources to what matters most. As Milestone B nears, this means devoting more resources to developing a military system and less to developing technology .
Historically, the aggressive performance goals of military systems have frustrated attempts to allocate resources this way. 

To meet performance objectives, immature technology is embraced on the hope that it can be matured at the same time a system is developed . Rare is the program that can successfully develop a system and mature technology at the same time Some plans feature more mature technology at Milestone B. Using more mature technology frees resources for purposes of systems development, which is no small feat when the system is for military use, even when the technology is relatively mature.

Dealing with risk effectively requires making room for more mature technology. In practice this means introducing flexibility in performance objectives, or setting more modest ones, so that the risk profile will align with available funding. Flexibility creates an outlet by which the trade-offs can be made to release some of the budgetary pressure. It also obviates less desirable means by which the pressure can be released: through contributions from contractor independent research.

When competition is introduced to prototyping, tapping contractor independent research and development budgets is incredibly enticing. By tapping these funds, the total budget for prototyping can be increased, more risk can be accommodated, and more performance can be chased. When compared to competitively prototyped programs where the performance goals were much more modest, the difference in total costs is striking. For the Close-Air Support and Lightweight Fighter prototypes, the total budget for prototyping was about five orders of magnitude less.

Competitive prototyping’s ability to absorb large amounts of private capital is not unique to aircraft procurements It is a by-product of competition. It is not irrational for contractors to take a considerable amount of financial risk in development in hopes of winning a production award. Development is not thought of as a profitable venture in military procurement; production is.

So there is an incentive for contractors to take large financial risks in development to win the more lucrative production contract likely to follow. For those in DoD always on the search for better performance, there is an equal incentive to let them do so.

Independent research and development budgets are the primary resource a contractor has for improving its competitive posture in a prototyping phase When that is not enough, pooling resources through teaming, such as occurred in the Advanced Tactical Fighter program, is another means for responding to a prototype’s aggressive needs . 

Over the short term, teaming and collaboration among development teams can be a good thing It can spur innovation and increase the flow of ideas. But when few can afford to compete and when failing to win the competition threatens the viability of the industrial base, neither DoD nor industry’s interests are well served controlling these effects is a part of competitive prototyping.

This is not to say contractors should not share in the cost of a prototype’s development, nor is it to say prototyping is bad for industry. Independent research and development, meanwhile, is a valuable tool for expanding the capabilities of a military system. But the pressures of competition, the inescapable lure of a multi-billion dollar defense market, and the combination of aggressive performance goals, immature technology, and inadequate funding, can create a vortex for private capital from which some competitors may never emerge. It may also create an environment that some competitors purposely avoid, all of which is counterproductive to preserving the industrial base and future chances of competition.

At the same time, when too much private capital contributes to the prototype’s development, the prototype’s value as a tool for reducing risk and providing better information is limited At the extreme end, when private funding is wholly responsible for a prototype, the resulting test article has been described as a tool best suited for marketing, not uncovering risks.

DoD procurements do not operate at this extreme. But at the extreme lies the risk of what may happen when a prototyping effort is inadequately funded and controlled. For all its success, the Navy’s Joint Standoff Weapon “JSOW” program encountered this risk as a result of its competitive prototyping effort For JSOW, much of the prototyping was done outside of the contract and on the contractors’ dime. The result: much of the technical risk was carried into Engineering and Manufacturing Development.
 
To counter these effects, DoD should absorb most of the cost pressures by funding the majority of the competitive prototyping effort. This only makes sense given that the purpose of the prototypes is to inform DoD process of decision-making. After all, the process is a treacherous one. Requirements change. Programs are cancelled. Funds are put on hold. Delays follow. All these things are likely outcomes in major systems acquisition. When they occur, they limit the ability of prototyping investments to provide a positive return. They also are all traceable to DoD If the DoD is responsible for marginalising a prototyping investment, then it should bear the costs for doing so.

This introduces discipline to the decision-making process and makes it more likely resources will be allocated in ways that provide the best return. With only a limited supply of resources, DoD can decide how best to align the risk profile to match the budget. Introducing flexibility in performance requirements so trade-offs can be made is one way of doing this. 

Adopting an austere development environment that focuses on the objectives of the prototyping phase is another Austerity controls costs and allows resources to be allocated to the most important activities, such as those that complement the practice of prototyping .
Traditional system models and simulations, for example, are complementary to prototyping because they extend the range of knowledge prototyping provides .

In addition to allocating resources effectively, if DoD is to get a positive return on its prototyping investments, its financial commitment cannot be open-ended. There must be a sensible cap placed on the costs of a prototyping phase to preserve the promise of some expected return. To make limits meaningful, they have been accompanied by suggestions to contract for prototypes on a fixed price basis.

Contracting for competing prototypes on a fixed price basis without absorbing a large amount of private capital counsels for contracting on a best efforts basis. Reining in production expectations is also important. Some have suggested that when competitive prototyping, the future should be wholly in doubt. No production run should be promised or expected It may also be best to limit how much private capital can be contributed to the prototyping phase. All these steps limit how much private capital can shape the prototypes’ baseline and obscure the bottom line.

These are all ways in which resources might be effectively allocated in a competitive prototyping so that the prototype’s status as a risk reduction tool can be preserved. Also preserved is the likelihood of a capturing a positive return on prototyping investments without distorting capital allocations in the industrial base. When excess commitments of private capital are averted, more productive allocations can be made across the industrial base This in turn preserves the competitive landscape of the military arms market, and, over the long term, this is a net benefit for DoD After all, competition may offer plenty of value, but it relies on having enough players to take the field.

Approaching Competition

With DoD bearing most of the cost pressures for competitive prototyping, other temptations exist besides the allure of private capital. Competition may be a means by which better performing systems can be obtained at lower cost, but competition adds its own costs. In competitive prototyping, the costs come in the form of having to build two or more prototypes instead of one. The temptation is to dispense with competition on the grounds that its benefits will not outweigh its costs. Then, the extra resources competition will require can be redirected to other priorities, such as attaining better performance goals. The challenge is to approach competition creatively so competitive pressures can yield better results. This requires creativity in how objectives are framed, how the benefits of competition are tallied, and how competition is pursued.

On DoD acquisition ledger, benefits are usually tallied in terms of objectives for performance, cost, schedule, and risk When these objectives are defined at the system level, this can mean competition will be pursued likewise. In this approach, the prototypes are full-scale, but not necessarily fully capable, representations of the final system design. The Air Force’s Close Air-Attack-Support, Lightweight Fighter, and Advanced Tactical Fighter programs are all examples of the full-scale competitive prototyping approach. In each case, objectives for the prototypes were defined at the system level, and then evaluated on that basis in a competitive fly-off.

How objectives for a competitive prototyping effort are defined has a large impact on competitive prototyping’s costs. How objectives are defined determines how they will be evaluated, and how they will be evaluated determines how competition will be pursued. When objectives are defined at the system level, they can be particularly costly to address as part of a competition, especially when they are very specific. 

Evaluating system-level objectives related to maintenance, operating, and supportability costs, for example, requires prototypes that closely approximate the final design. The Army took this approach in its competition for the Utility Tactical Transport Aircraft System, and as a result, had to field two production representative prototypes to support the competition This required a considerable financial commitment and limited the ability to realise efficiencies over the course of development. Instead, the Army had to realise efficiencies over a much longer term. In the case of the Utility Tactical Transport Aircraft System, it was the entire life cycle.

Tallying the benefits of competition over the long term is probably the best approach, especially for systems likely to be in the field for several decades. But with smaller development budgets, the resources necessary to evaluate production representative designs as part of a competitive effort are not likely to be there. Instead, competition must be approached in a different way, such as at a lower scale or with prototypes having fewer capabilities. What this means is defining system-level objectives with much less specificity, or reducing those objectives to something that can be competed at a lower scale, such as a subsystem.

When system-level objectives can be reduced to terms of subsystem performance, pursuing competition will be cheaper. At the same time, being able to capture system-level returns at lower costs will likely make competition more attractive. Development of the AIM-54 Phoenix missile is one example where the system-level performance largely hinged on a subsystem level competition Rather than devote resources to a full system competition, they were targeted at the subsystem that mattered most. Systems like aircraft have also been highlighted as systems where, better performance at the subsystem level can lead to outsized, system-level returns. For this reason, competition at the subsystem level has been highly encouraged.

One of the best-documented examples of competitive prototyping at the subsystem level is the cannon competition held during the Air Force’s Close-Air-Attack-Support program. Besides the competition between airframes, the Close-Air Attack-Support program pursued two other competitions to attain its life-cycle cost and performance goals
The first competition was for the cannon. The second competition was for the ammunition. The Air Force managed the first. In the second, the Air Force worked through the winning cannon contractor as a surrogate. For the aircraft that later became the A-10, the gun system was critical to providing the desired operational capability. But given the cannon’s high rate of fire, cost objectives were placed on the ammunition so that the capability would remain affordable.
 
To meet these cost objectives, the Air Force directed that a competition be held at the subcontractor level for ammunition. Additional requirements directed to how competition was to be pursued and the results evaluated were also levied. The result: savings that could be garnered over the life cycle in the form of an eighty percent reduction in unit cost for ammunition.

Successfully pursuing competition at the subcontractor level requires DoD to intervene in a relationship it would often leave alone. To meet DoD needs, additional requirements related to managing a subcontractor must be levied on the prime The prime contractor must be informed of DoD requirements, including source selection criteria, to ensure DoD priorities are what shape the competition. It is also not far-fetched to condition any down-select decision on DoD pre-approval.

Implementing these additional controls will undoubtedly lead to more development costs, but not implementing them may undermine the objectives of having a competition between subcontractors. In the case of the Close-Air-Attack-Support program, directing the prime was surely more expensive, but the gains it reaped have been garnered over the long term.

Framing objectives in non-system specific terms, such as in terms of capability needs rather than materiel needs, is another way let competition work, and at lower costs. The idea behind this approach to competition is that, by allowing competition to work on a wider scale without being confined to a subset of materiel solutions, it will reap greater returns.

Putting It All Together

If the purpose of competition is to obtain better performing systems at lower cost—which is usually referred to as obtaining the best value—then allowing the best value to be found is just as important as letting competition in. Getting right mix of performance and price has been described as the key to securing a technological advantage Surely performance is one aspect of a technological advantage, but so is price. The more systems that can be fielded for a given sum the greater the force multiplier effect. When procuring military systems, the goal is often to find the best mix of the two This is no less true when competitively prototyping. Competition merely postures DoD to make a better choice. The challenge is to structure the competition so the best value shines through.

Competitive prototyping presents DoD with a choice to continue funding one or more of multiple designs Through prototype demonstrations, information about each design is gathered and used to determine whether development will proceed, and if so, which contractor is best situated to pursue it. At Milestone B, the information should reflect that the risks and benefits of a particular design warrant greater financial commitments in Engineering and Manufacturing Development If the benefits outweigh the risks, then the prototypes should further demonstrate which design provides the best value.

Unlike other procurement approaches, though, competitive prototyping does not rely so much on DoD defining the best value as it does allowing competing contractors to find it on their own. Knowing what DoD considers most important in a final design is important, but giving contractors the freedom to experiment within a range of acceptable criteria facilitates better technological solutions at lower cost. Flexibility allows for bolder technological solutions to be tried and tested without renegotiating the contract, and therefore allows more risks and design flaws to be addressed early.

Competition, meanwhile, stokes innovative thought and encourages aggressive design solutions in direct response to identified needs The requirement to prototype allows all this to occur without sacrificing credibility Through competitive prototyping, better technological solutions should emerge based on performance, not promises, and when they do, the results can be surprising.

The outcomes of the Air Force’s Lightweight Fighter competition and the Army’s Advanced Attack Helicopter show how this can be true. The superior handling of the YF-16 prototype in the Lightweight Fighter competition was one reason it was preferred For the Army, the superior handling and simpler design of the YAH-64 prototype is what caused it to prevail In both, operational suitability in the form of airframe handling was a primary discriminator. Airframe handling is difficult to evaluate on paper in advance of design, but it is particularly important to determining an aircraft’s operational suitability. In the cases of the Lightweight Fighter and the Advanced Attack Helicopter programs, the superior operational suitability of the winning designs seemed to carry the day. For the YAH-64 prototype, these results were surprising. All analysis up to the point of flight test suggested its competitor would win.

Allowing contractors the freedom to explore a mix of technological approaches has a second benefit besides providing for more innovative thought: it allows DoD to know how much capability it can afford to buy When approached in this way, competitive prototyping can inform requirements and avert an improvident pursuit for a capability that is just not there. Even when the capability is there, competitive prototyping can show that a cost effective one may be still another generation of prototypes away.

So prototyping uncovers all kinds of valuable information When judging them as part of a competition, all things should be considered, not just performance or cost To make the prototypes more insightful than paper studies, particular attention should be paid to those things that are difficult to capture in advance and on paper, such as operational suitability Also important is past performance.  Competitive prototyping can reveal which contractor is more likely to better perform in the next phase In terms of past performance, no gauge is likely to be more reliable than the recent experiences garnered through prototyping.

All the information prototyping provides can and should be used to choose between competing designs. Competition, meanwhile, should be leveraged to spur innovation and presentation of a best solution. The challenge is to put all these things together so the quest for a better solution will be worth the extra costs.

One challenge facing DoD will be to adjust the paradigm by which it approaches prototyping’s results. Sometimes the process of prototyping, whether it is competitive or not, suggests that the benefits of a particular technology are not worth the costs. In such cases, the best decision is probably not to go forward with large commitments of capital. Some may consider such a result to be a prototyping failure. but It is in fact a prototyping success.

Prototyping identifies problems prior to a full commitment of capital when it becomes harder to view a program as anything but “too big to fail” It is better to direct resources to technologies that can get to the war fighter cheaper and more quickly than it is to continue to invest in technologies that have promise but no near term return. Indeed, this is the primary reason one prototypes.

A final challenge is in establishing better prototyping guidelines. These guidelines will need to be more specific than those presented here. They will also have to account for the differences in prototypes among various military systems. Not all systems can be cost effectively prototyped at the full system level. The non-recurring expenses associated with systems like satellites and aircraft carriers are too large to ever be recouped later, and as such, they are best prototyped at something less than full-scale.

Aircraft and munitions, on the other hand, are best prototyped at higher levels of integration. To make competitive prototyping cost effective, there is a balance that has to be struck in terms of cost and capabilities that enables better decisions The balance will vary by system and better guidelines are needed for finding where the balances lie.

Similarly, there are points where prototyping on a competitive basis, as well as prototyping itself, may no longer make sense. To confirm through prototyping what is already known may be reassuring but is not likely a productive investment of capital. To continue to compete when one solution far exceeds the rest is not likely to result in much gain After all, pursuing competition for competition’s sake is not the goal the goal is better value.



1. Take a value view: Delivering the best value and quality in the sustainably shortest lead time requires a fundamental understanding of the economics of the system builder’s mission.

2. Apply systems thinking: In design framework, systems thinking is applied to the organization that builds the system, as well as the system under development, and further, how that system operates in its end user environment.

3. Assume variability; preserve options: design systems developers maintain multiple requirements and design options for a longer period in the development cycle. 

4. Empirical data is then used to narrow focus, resulting in a design that creates better economic outcomes.

5. Build incrementally with fast, integrated learning cycles: Increments provide the opportunity for fast customer feedback and risk mitigation, and also serve as minimum viable solutions or prototypes for market testing and validation.

6. Base milestones on objective evaluation of working systems: In design development, each integration point provides an objective milestone to evaluate the solution, frequently and throughout the development life cycle. This objective evaluation assures that a continuing investment will produce a commensurate return.

7. Visualize and limit batch sizes, and manage queue lengths: Three primary keys to implementing flow are to: 1) Visualize and limit the amount of work-in-process so as to limit demand to actual capacity, 2) Reduce the batch sizes of work items to facilitate reliable flow though the system, and 3) Manage queue lengths so as to reduce the wait times for new capabilities.

8. Apply cadence, synchronize with cross-domain planning: Cadence transforms unpredictable events into predictable ones, and provides a rhythm for development. Synchronization causes multiple perspectives to be understood, resolved and integrated at the same time.

9. . Unlock the intrinsic motivation of knowledge workers: Providing autonomy, mission and purpose, and minimizing constraints, leads to higher levels of employee engagement, and results in better outcomes for customers and the enterprise.

10. Decentralise decision-making: Decentralized decision-making reduces delays, improves product development flow and enables faster feedback and more innovative solutions. However, some decisions are strategic, global in nature, and have economies of scale sufficient enough to warrant centralized decision-making.

​More competition in terms of capability needs and during the early stages of the acquisition process will have a greater effect on acquisition outcomes Rather than have industry respond with competing proposals in response to a predetermined materiel need, panels advocated for competing materiel solutions in response to a capability need.

In today’s acquisition practice, this means competition between various materiel solutions prior to Milestone A. Prototyping at this early stage in development takes the form of conceptual prototypes, which are breadboard designs of individual components and subsystems coupled with estimates of system performance provided by traditional modeling, simulation techniques.

Conceptual prototypes provide only basic approximations of system performance and cost, but enough insight to challenge assumptions and open the door for more innovation. They are also less costly to pursue than other types of prototypes, and, considering their timing, provide benefits that can be exploited over a much longer term. Others have picked on this theme of non-system specific prototyping as a means to innovate and have taken it in another direction: technology transition. 

In terms of budget categories, innovation has been found most likely to reside at the applied research and advanced technology development budget categories Technologies that are prototyped at this stage of development are not ready for operational use, but they can be demonstrated and assessed for operational utility.

More prototyping at these early stages of development has been proposed as a means for focusing research, bridging the worlds of technology development and systems development, and expanding operational performance at lower cost prototypes that appear in these budget categories are known as Advanced Technology Demonstrators, and they have become a valuable tool for technology transition. 

Competing prototypes in the context of technology transition can take the form of a competition between system upgrades. A competition between system upgrades is typically more cost effective and less risky than competing entirely new systems The incremental improvements upgrades provide go a long way towards facilitating innovation, and therefore, when still cost effective, likely should be pursued.

Competing system upgrades in a cost effective manner is again a matter of how objectives are framed. When framed in the most general terms, such as in terms of mission effectiveness and expected life-cycle costs, more options for pursuing competition result. 

One option is to compete a combination of upgrades to existing systems to determine which combination provides the most cost effective capability. The Air Force’s Enhanced Tactical Fighter program, which competed a combination of upgrades to the F-15 and F-16 fighter aircraft and led to the F-15E Strike Eagle aircraft, provides an example of this approach. 

Another option is to compete an upgrade to an existing system against the capabilities of a new one. This option is a particularly attractive means for providing competitive pressures when resources are too constrained to compete multiple new designs. What this means for program planners is that competition may be more a question of program planning than anything else. 

Given a limited supply of resources, the objective is always to allocate those resources most effectively to obtain the best mix of performance, cost, schedule and risk. Accommodating competition is one way to obtain a better mix. The challenge is to approach competition in such a way that paying for it is worth the costs.


Workshop is a vehicle for exploring, testing, and hopefully proving an AI-driven tool to generate physical designs. Implementing a divergent design flow, Workshop builds off the traditional flow: define, create, explore visualize, analyze and fabricate. Where that flow departs from the norm is in who — or rather, what — is performing or assisting in those development stages, and the extent of that assistance.

With Workshop, engineers and designers specify design goals, along with parameters such as mass, volume, and engineering constraints, as well as materials and manufacturing processes available for production. 

All that data gets digested by AI deep computational networks to create a model it concludes is the optimal balance of the given constraints. More typically, the network will generate many versions, creating tens, hundreds, or even thousands of variants, all of which balance those constraints in slightly different ways.

AI-based generative design is no replacement for a product designer or engineer. Rather, think of it more as a right-hand man — or better put, a right-hand team — with countless hours to spend on each task. It allows designers the means to explore options far more exhaustively than they otherwise ever could, make effective decisions more quickly and earlier in the process, and eventually implement a design — from concept all the way to manufacturing. 

The computational network proposes options, along with the data to indicate how well that option meets various goals and constraints, but the designer remains the guide of that process, making the key decisions and tradeoffs. Effectively, you tell your team what you’re trying to accomplish and what the limitations are, and that team reports back, “You said you wanted to do this, so here’s the avenue you should probably pursue.”

Ultimately, transformational AI-driven generative design will earn its place in product design toolboxes the same way any now-ubiquitous tools have: by proving its worth in making products faster to design, higher in quality and performance, and most amenable to the optimal means of production. For its part, generative design assists in all those goals by allowing designers to make the most effective and through decisions, and do so earlier in the process.

Workshop delivers that ability for a designer to set objectives and constraints, include limits on geometry, material type and amount, and production methods. With generative design at the fingertips, a designer can then quickly create and integrate hundreds of options, each presenting a different degree of adherence to the constraints given.

This raises a great question: How can the user avoid being overwhelmed by a potentially large number of similar-looking designs, and instead hone in effectively on the one or few best suited to the set goals and priorities? 

To make all those generative design results easier to consume, Workshop includes filters that let you sort designs using the most relevant and important info up front. Filters let you navigate, compare, and contrast the tens or hundreds of options by the degree to which those designs meet the goals and constraints. Filter by strength, mass, cost, manufacturing type, or stiffness, for example, to narrow in on and eventually identify the best option to pursue.

Arriving at the right model, or at least the best one to explore first, shouldn’t end at a creating a conceptual, stand-alone structural representation. Rather, the key is to also allow the means to bridge the gap from that machine-generated model to engineering verification and styling, and eventually on to getting it manufactured.
 
Some simple outputs are of limited value; it yields an interesting shape, but one that can’t serve any use beyond the visual. A productive workflow needs that geometry embedded with all the rich data sets needed to directly feed into simulation, verification, rendering, and on to prototype and manufacturing. 

If the machine integration produced just a shell of geometry, the designer might have to re-create the whole thing manually. Bridging the gap from synthesis to the rest of the workflow was a key design goal for Workshop generative design functionality, and its implementation is unique in the way it automates the synthesized design in a complete, usable, and editable format ready for verification, re-design, and ultimately, physical creation.

A generative design–based workflow should seamlessly culminate in a manufacturable model, be it for prototype or volume production. Workshop  generative design considers the constraints and capabilities of the manufacturing process and materials available. 

The model is particularly relevant in the context of additive methods like 3D printing, especially now given recent advancements in printing with metal. Such methods allow for shapes and structure that conventional manufacturing methods can’t achieve, making it essential for the user to provide generative design guidance on which methods are available or preferred.

But you can’t ask or expect end users to navigate the appropriate performance, features, virtual machine types, or determine when use cases might best for generative design or any other workloads headed to execution space. 

As you might guess, Workshop hides the processing wizard behind the curtain so the user doesn’t have to be concerned with how it gets done. Based on the model and constraints, Workshop assesses the workload at hand, “right-sizes” it to the appropriate machine instances, and when complete, hands it back to you inside the application.

Over time, machine learning will permeate virtually every corner of computing technology and applications. Of that, the majority of us have little doubt. In design computing, uses have already popped up to significantly improve the performance of existing 3D graphics and rendering that professionals require. But most certainly, these uses represent but the tip of the iceberg, and the real impact will come in more revolutionary applications.
The demands and workloads product design represent make it fertile ground for AI infiltration. Expect machine learning advancements to both leverage and transform the existing tried-and-true design/verify/iterate/manufacture workflow. 

Most of the possibilities we’ve likely not yet imagined, but the same would likely have been said ten years ago for generative design, an approach that offers compelling, undeniable appeal. Combine that with its natural pairing with virtual workstations, and we should see momentum increase for both approaches.

It’s time to pay close attention to what AI offers for product design computing — assisting in design creation itself, leveraging the technology to streamline workflows, improving end products, cutting costs, and shortening time to markets. Your competition likely has.

Shifting traditional design work earlier in the design process so as to avoid rework later is difficult.  A number of product development practices that have been characterised as a shift from developing a single-point design to developing a set of possible designs have proven effective at reducing development rework. 

Here we refine the definitions of such “set-based” development practices, which are aimed at early development phases, and shows how they can be applied to the systems engineering process in order to reduce or eliminate the root causes of rework

We use the term rework to specifically mean the work that occurs when a prior decision that was assumed to be final for that project is changed because it was later found to be defective. 

A decision is considered “ final” in the sense that the team does not have any reason to believe that the decision would need to change and therefore expects that development work can proceed assuming that decision will stand for the remainder of the project.

Such rework is distinct from design iterations performed for rapid learning, where the design decisions are understood to be experimental, and thus other development work does not proceed assuming those to be the final decisions.

It is also distinct from establishing rapid project cycles to accelerate customer feedback. However, rework may occur within a design iteration or rapid project cycle if a poor decision would otherwise result in the iteration or project cycle failing to achieve its goals.

Rework has become so commonplace that most development managers seem to consider it inevitable so they schedule multiple prototype build-and-test cycles into their project plans, create elaborate engineering change processes to manage multiple layers of rework, and pad schedules to account for the unknowns. Such practices do not address the root causes so that the rework can be eliminated.

But much of the rework that product development organisations experience is not inevitable. Development organisations can eliminate rework from their product development processes. But to do so requires a very different approach to the systems engineering of new products and technology, all starting with the front end of the product development process.

Several models and tools have been proposed to reduce the need for rework , but these approaches will have limited impact if they do not address the root causes of that rework.

Systems engineers and development managers are all too familiar with the frustration of seeing development teams revisit decisions made earlier in their projects and watching the ripple effects of violated assumptions, associated design changes, and reworked plans, assessments & designs they know are coming as a result. Here we provide several examples:

First, during prototype testing, teams discovers that two of the design specifications are incompatible, meaning one of them must be revised and all the spec creation to date reworked. 

Second, The product team realises to their surprise that customers do not like the trade-offs made among competing objectives, so key product features must be changed late in the game at significant cost to the organisation, or face lower than projected sales.

Third, In developing the manufacturing process, the manufacturing engineering team learns that the current equipment is not capable of manufacturing an important product feature, resulting in the tough decision either to redesign the feature thereby delaying product delivery or to invest much more than planned in updating manufacturing capability.

Finally, After market introduction or delivery to the customer, a latent design flaw results in warranty claims that eat up most of the profitability of that product, possibly even blowing up into a product recall.

The bottom line impact of unfortunate but common occurrences such as these is staggering. The cost to extract such defects can increase significantly the original development costs depending on when they are discovered.

Such defects squander market opportunities, damage brand reputation, and otherwise take huge bites out of return on investment and growth potential. Often missed are the compounding effects of frustrated employees, conflicts across organisations, and related inefficiencies.

Much of life cycle costs of a project are determined by decisions made by the end of the concept phase and most capital is committed before detailed design starts. 

Systems engineers face dilemma that often very little is known about full impact of those decisions when they are being made so it is not a surprise that such decisions are frequently revisited, and the associated development work redone.

We tried many tools to get to the final iteration. You can see in the picture we show two design iterations—the middle one that is still quite similar visually to the original node was our first iteration. And you can see version 2.0, the second iteration is where we really made the big step with the freedom of form.

Using generative design, engineers specify parameters, such as weight-to-strength ratios, efficient material use, and temperature, pressure and force ranges. The generative design engine creates several design options through an iterative approach. Engineers then evaluate and select from among the generatively designed options—more options than would be possible with traditional design tools, and likely many options that the engineers would never have considered.

No one group has complete control over the final results, so it is a lot of different engineers have to move at the same time or at least following in a series, and what you see in these processes is that they all start with great concepts but throughout the phases they become victim to planning pressure and costs so a lot of new ideas and challenges are just designed out.

In a Set-Based Design approach, which has been identified as a preferred approach for the development of future design efforts, discipline-specific designs are done in parallel across a broad design space to improve the flexibility of the design by delaying key decisions until the design space is fully understood the parallel nature of the approach also makes it an ideal fit for high performance computing

Since computers are already widely used and the current level of their usage is very high, simulation building process can lead to greatly reduced manufacturing time when you consider how sequence fits into process. 

Tools need to be verified and validated; problems must be easy to set up and run; structural process generation must be easy and quick; tools must be built to run effectively and efficiently on massive parallel computers; and, results must be timely.

Geometric assembly constraints, are easily identifiable and definable but certain constraints, especially the component constraints and soft constraints, are difficult to identify without a good realistic feel of the assembly process.

In the mixed prototyping concept, several questions need to be clarified: 1. What parts should be real prototypes? 2. What parts should be virtual? 3) How much manipulation of the virtual parts is feasible and needed, etc. Although the answers to these questions are very context dependent, basically we can make a decision based on the following aspects.

For the specific cases, it is difficult to obtain an obvious optimal solution of all these aspects. Must consider the trade-off of these aspects carefully in terms of our application and requirements to define a proper strategy for assembly evaluation based on mixed prototyping.

Initial constraints are imported into the Automatic Assembly Planning System to Generate the feasible Sequences so planners can view and verify the feasible sequences in the virtual reality working space to
Plans identify new constraints and decide on requirements change criteria e.g., cost, number of orientations, etc. Next, the users go back to the Automatic Assembly Planning System to re-plan the sequences. The planners repeat this process until they find a satisfied sequence.

1. Design Strategy must obtain economies of scale in customised productions so standard components of products have become very popular in manufacturing industry

2. In mixed prototyping concept standard parts must normally be real components since they can be found easily in stocks.

3. For some fixed designs that do not need to be changed much, can use real components through conventional rapid prototype technologies.

4. Customised parts must be evaluated and revised many times, virtual prototypes are used since flexible for modification.

5. During an assembly process impossible to connect two real components using a virtual component to obtain realistic feedback-- cannot stack a real component on a virtual component.

6. Using the largest component of an assembly as a virtual part is not ideal if several other real and virtual parts are connected to it.

7. Parts where several components are to be assembled, such as the base part, would serve better if they are real.

8. Some workspace and assembly parts cannot be completely defined and simulation is so must use real components as much as possible.

9. If prototyping cost of some components is very high, try to use virtual prototypes even though designs are already fixed.
​
10. Users can obtain more realistic assembly parts sensory feedback based on real components as compared to virtual components.