^[1]

The Australian National Audit Office (ANAO) has just released a report into Defence Test and Evaluation (T&E) ^[2] that makes a number of important findings. According to the report, this critical enabler to informed decisionmaking in Defence capability development and acquisition is spread across too many smaller T&E organisations and doesn’t have adequate oversight of the competency of all of those staff managing the T&E. The paper also finds that Defence only recently has a centralised T&E policy that needs much more refinement and governance. Lastly, on the recent T&E of the Landing Helicopter Dock (LHD) ship, Defence was unable to deliver a sufficient number of competent T&E staff to process the test reports and concessions in a timely fashion.

Defence uses T&E to have representative users assess if capabilities deliver effective results when used in representative missions. T&E is Defence’s main measure of capability that goes with cost and schedule to determine if a project is reducing risk and thus succeeding. Good T&E gets done as early as possible; ideally T&E gets done as a real preview before full contract (off-the-shelf systems) and certainly should be done before production is contracted.

ANAO last audited Defence’s T&E in 2001–2002. In the 13 years since, they’ve unfortunately found that there has been little or only late progress on most of their areas of concern. That’s despite the concerns being reiterated in a Defence T&E Roadmap in 2008 and in the Senate Inquiry into Defence Procurement (especially Chapters 2 and 12) in 2012. In this same period US Defense has entrenched independent T&E into Congressional laws (Title 10) that provide mandatory independent annual reporting of all T&E measures to Congress on all major Defense acquisitions ^[3]. Title 10 law also requires T&E competency in all US acquisition projects and T&E agencies overseen by the autonomous Defense Acquisition University. The book ^[4] and movie The Pentagon Wars ^[5] has clearly raised awareness in the US for the value of independent T&E for Defense acquisitions much more than in Australia. There’s a clear option for the Australian government to follow America’s lead with Defence Regulations if Defence doesn’t institute strong, independent, early and adequate T&E for its critical capability decisions.

Most strikingly, the ANAO report’s insights to the project management of the new LHD ships illustrate that Defence didn’t have the necessary T&E staff to manage the contractor’s reports in a timely manner and that this necessitated the transfer of some capability risk to Navy. The project positions for the T&E were planned but not filled, and the project manager repeatedly advised project governance boards of the vacancies. But it appears that the now defunct Defence Materiel Organisation was unable or unwilling to appropriately prioritise what ultimately became a Navy problem. These difficulties reinforce the ANAO’s previous findings that Defence isn’t managing its T&E competency in all areas. That this observation occurred in the maritime domain should be of critical concern given the significant investment in ships and submarines anticipated in the new Defence Capability Plan, most of which require significant development, integration, test, build and more test in Australia.

What isn’t directly mentioned in the ANAO report—but concerns me as a former leader in this area—is that of the 12 Defence organisations dedicated to T&E, not one of them does developmental test of ships or submarines: the Navy only has a dedicated operational T&E agency. Each maritime project has to find the T&E expertise and resources itself. In contrast, the Air Force has a dedicated developmental T&E agency. The Army also has one but theirs was in the now defunct DMO and it’s not clear yet if they’ll take this critical enabler back to the service for early insight into project T&E. The implications of this maritime approach are plain to see in the ANAO coverage of the LHD ship T&E. This approach also places significant risk on the adequacy of preview testing for the submarine project options and is at odds with the long-term decision by this government to invest in continuous build of ships.

The ANAO report and the government’s response today in Parliament point to the First Principles Review and subsequent Defence reorganisation as the means to improve Defence T&E. Little is known of the new structures of this reorganisation other than it aims to reduce acquisition bureaucracy and create a strong strategic centre. If the strategic centre is to be better informed for its acquisition decisionmaking it needs to clearly invest in more structured, resourced and competent T&E. A more complete coverage of what such a first principles reorganisation should consider is in my paper titled ‘Implementing the Defence First Principles Review: two key opportunities to achieve best practice in capability development’ ^[6].

Update: Click here ^[7] to view Keith Joiner’s Strategic Insights paper, published 17 December 2015.