Hope is not a plan

In a recent report titled Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes, the GAO addresses a continuing problem within the DOD of new weapon technology programs going seriously off the rails and seriously over budget. They highlight key errors made during project planning, and make suggestions based on best practices as used by successful corporations.

The clear problem is right there in the title, of course. "A Knowledge-Based Funding Approach Could Improve...Outcomes." If you aren't using knowledge now, isn't that an indicator that something is terribly wrong with the current method?

The GAO finds that decision makers within DOD make three big errors when they go about putting money aside for new weapons systems:

• 1. They accept overly optimistic cost predictions
• 2. They then underfund the program below those cost predictions
• 3. They then shift funds to help underfunded programs, leading to problem #2 for other, new programs

So, one by one, why do these happen?

The GAO has reported, over and over again, that procurement is heavily hampered by undue optimism about costs and outcomes. Certainly, a number of factors are at work here. Firstly, as Americans, we are optimistic about outcomes. We believe we can do it, and that technology will appear. Consider the case of the Joint Strike Fighter:

JSF assumed that the commonality between the three variants of aircraft and the use of a joint development program, instead of three separate programs, could cut development costs by about 40 percent. However, after development started, significant design issues forced the program to delay development approximately 18 months to conduct unexpected design work. In addition, the assumed commonality between the variants decreased. As we reported in 2005, these two factors contributed to cost increases that nearly eroded all of the assumed cost savings.

It is likely that enthusiastic designers saw the possibility for efficiency, factored that in, and never considered that they should determine the likelihood that this efficiency might drop out. Similarly, many large DOD projects assume that key technologies will simply appear and be waiting for the project when it reaches a certain point. Once again, it's easy to see how this happens -- you see that, say, radios are becoming more energy efficient each year. You assume that trend will continue, and you will be able to use an "off the shelf" radio in your UAV. Then something changes and radios don't get more efficient in time, and now you are in trouble. You must, at great expense in money and lost time, either make the technology yourself or revamp your project to account for its absence.

As GAO tells us, successful corporations work with this problem by making quantitative assessments of risk:

Conducting quantitative risk and uncertainty analysis provides a way to assess the variability in an estimate. Using this type of analysis, cost estimators can model such effects as a schedule slipping or a key technology failing to materialize, thereby identifying a range of likely costs around an estimate.

In our UAV example, you'd say, "What's the percent chance that we won't have off-the-shelf radios when we get to the point where we need them? What impact does it have on our project if we can't get them?" Then you adjust the project or adjust your cost estimate to provide room for that eventuality.

More generally, this kind of assessment helps curtail the "Well, we must have this at any cost" approach to defense acquisitions. Must you? If your risk analysis says that there's a good chance the Raptor will come in late and massively overpriced, is it still the best approach?

A willingness to change the plan -- or even cancel it entirely -- is critical in making smart acquisitions, and making them affordably.

The second point -- underfunding -- is a curious one. Per DOD policy, a project must be fully funded within the Future Years Defense Program (a sort of "window of time" of DOD funding). Some programs run longer than the FYDP window, and thus specific funding for their later years is not considered during the initial assessment. Other times, however, programs are underfunded even within the FYDP window.

In other words, the DOD procurement folks aren't following DOD policy. This is not a novel concept, as we've seen in recent protests upheld by GAO. Procurement officials often approve projects without considering specific procurement policy that should cover approval of those projects. There's little to do about this other than enforce the policy better, perhaps by having higher-level DOD authorities crack down on the procurement office -- or maybe via the efforts of competing contractors filing protests.

The problem of projects extending past the FYDP is best addressed, thinks the GAO, by adopting the development cycle approach of successful corporations like Motorola. Many DOD weapons systems projects have very long development times, longer than in the world of business where 2-5 years is the ideal. By dint of being so long, they not only extend outside the FYDP window but also become harder and harder to accurately cost out ahead of time. Thus, by angling for shorter development cycles, DOD could simultaneously tackle problems one and two, being able to approach the funding issue with both more accurate estimates and a fairly concise project period.

The third area, of shuffling funding in a panic to keep projects afloat, is best handled via the two-pronged approach of shorter development cycles and more accurate funding estimates. I would offer on top of that the third option of "Canceling the project." Once again, the specter of "We must have this" rises to oppose this idea, but the truth is that open-ended weapons development costs harm our national defense. Excess money spent to make a Joint Strike Fighter is money not going to developing better personal body armor or to the critical task of defense reset. In other words, it is not cost effective to develop the JSF if our unarmored soldiers are dying in broken-down HumVees.

This GAO report boils down to the idea that DOD should make a solid business case for every weapons system development program. Make quantitative assessments of risk. Have realistic horizons for product development. If necessary, subdivide programs into smaller ones so that it's possible to realistically assess likely success or failure. Given that this is an area of vast intersection between DOD and businesses, the suggestion that it ought to be run more like a business is a solid one.