Is the impending decision on Australia’s submarine competition based on any test and evaluation (T&E) report, or will there be any such reports before a development contract? Just as Government’s looking at the submarine record of Japan, Germany and France, it needs to adjust for its own tendencies, especially in inadequate use of T&E to inform decision-making at such key acquisition milestones.

The Collins-class had serious difficulties in its T&E because the majority of the testing occurred too late in the piece. In 2002, when five of the six boats had been built and operational release still not achieved, the Australian National Audit Office (ANAO) found ^[1] test plans weren’t updated, resourced or enforced, that test coordination meetings weren’t being held, and that Defence was struggling to verify safety-critical software. A 1999 special report to the Defence Minister ^[2] stated:

‘To our surprise, deficiencies have occurred in items that should have been relatively straightforward, had testing, even along the lines of that routinely applied to merchant ships, been undertaken.

Prescient to the case for preview activities on SEA 1000 today, the authors also stated:

‘What is required are terms which compel the contractor to demonstrate his capacity to deliver in advance of the execution of the contract itself … a series of tests or trials.’

The ANAO audit in 2001 covered 23 maritime projects as well as Collins, and found the T&E of some, like the Minehunter Coastal ship project, to be exemplary. It recommended Defence take steps similar to the US to better enforce its T&E policy for greater consistency and to avoid release of equipment into service with inadequate testing. Defence rejected this recommendation, claiming it didn’t do the sort of projects requiring such a T&E regime and preferred instead to front-load better planning and resourcing of T&E.

But in the decade following the Collins difficulties, the Australian Senate found ^[3] in 2012 that there’d been insufficient use of preview T&E on many major acquisitions, mainly under the often false premise of acquisitions being off-the-shelf. The Government agreed ^[4] to a policy of conducting preview T&E on all acquisition approvals. The resulting Defence capability planning process began use in 2013 and in 2015 was formally released, and endorsed by the ANAO ^[5] as successfully implemented. The process develops the earliest possible T&E, prior to contract and preferably as part of solicitation, using systematic evaluation of the likely technical and operational risks.

Only last month, the Joint Committee for Parliamentary Accounts and Audit examined yet another multi-billion dollar project that was delayed by 3–4 years when paper-based down-selection proved inadequate. At the hearing Defence officials advised that the lesson on preview T&E had been learnt, the new T&E policy would ensure a repeat was unlikely, and that, interestingly, the current restructure of Defence should only strengthen use of this preview T&E policy.

How the last point would occur wasn’t explained. The two large projects currently undergoing down-selection (SEA 1000 & LAND 400), the outcomes of both possibly to be announced before the election, will seriously test this commitment. Working against the likelihood of the new T&E policy being followed for SEA 1000 is ANAO’s 2015 finding that there are 12 separate test organisations, not one of which is devoted to developmental testing in the maritime domain.

Australian Defence has independent test organisations for development of land and aerospace equipment, but for maritime projects this critical skill and governance is left to individual projects to assemble, with the associated loss of independence and disparity inherent in such a solution.

Given the recently announced investment in maritime capabilities, Defence must address this T&E deficiency. Otherwise, the lack of adherence to test policy observed by the ANAO in maritime projects in 2002, and again in 2011, will continue. Because of this chequered history, I’ve authored a working paper ^[6] (PDF) on how the Australian and American T&E policies ought to apply to SEA 1000 at this critical decision time. The paper proposes:

• close alignment of SEA 1000 to US acquisition lifecycles and T&E governance checks;
  
• five preview test activities that could be done now for the preferred reference submarine design before the first developmental contract is signed; and,
• 11 test goals for the developmental T&E period.

Such activities and goals would shift SEA 1000 towards first building the necessary land-based test infrastructure for real submarine componentry, here in Australia or overseas, to validate the design before building a first-of-class submarine. These many facilities take about 7 years to develop and build, but without them we’ll repeat the Collins’ mistakes and certainly won’t achieve continuous spiral development and build. I commend my full working paper to anyone involved in the SEA 1000 decision and beg them to trust testing as the basis of their decisions and design iterations every step of the way.