{"id":26123,"date":"2016-04-26T09:31:36","date_gmt":"2016-04-25T23:31:36","guid":{"rendered":"http:\/\/www.aspistrategist.ru\/?p=26123"},"modified":"2016-04-26T09:36:14","modified_gmt":"2016-04-25T23:36:14","slug":"sea-1000-test-before-you-buy","status":"publish","type":"post","link":"https:\/\/www.aspistrategist.ru\/sea-1000-test-before-you-buy\/","title":{"rendered":"SEA 1000: test before you buy"},"content":{"rendered":"

\"\"Is the impending decision on Australia\u2019s submarine competition based on any test and evaluation (T&E) report, or will there be any such reports before a development contract? Just as Government’s looking at the submarine record of Japan, Germany and France, it needs to adjust for its own tendencies, especially in inadequate use of T&E to inform decision-making at such key acquisition milestones.<\/span><\/p>\n

The Collins-class had serious difficulties in its T&E because the majority of the testing occurred too late in the piece. In 2002, when five of the six boats had been built and operational release still not achieved, the Australian National Audit Office <\/span>(ANAO)<\/span> found<\/span><\/a> test plans weren\u2019t updated, resourced or enforced, that test coordination meetings weren\u2019t being held, and that Defence was struggling to verify safety-critical software. A <\/span>1999<\/span> special report to the Defence Minister<\/span><\/a> stated:<\/span><\/p>\n

\u2018To our surprise, deficiencies have occurred in items that should have been relatively straightforward, had testing, even along the lines of that routinely applied to merchant ships, been undertaken.<\/span><\/p><\/blockquote>\n

Prescient to the case for preview activities on SEA 1000 today, the authors also stated:<\/span><\/p>\n

\u2018What is required are terms which compel the contractor to demonstrate his capacity to deliver in advance of the execution of the contract itself \u2026 a series of tests or trials.\u2019<\/span><\/p><\/blockquote>\n

The ANAO audit in 2001 covered 23 maritime projects as well as Collins, and found the T&E of some, like the Minehunter Coastal ship project, to be exemplary. It recommended Defence take steps similar to the US to better enforce its T&E policy for greater consistency and to avoid release of equipment into service with inadequate testing. Defence rejected this recommendation, claiming it didn\u2019t do the sort of projects requiring such a T&E regime and preferred instead to front-load better planning and resourcing of T&E.<\/span><\/p>\n

But in the decade following the Collins difficulties, the<\/span> Australian Senate found<\/span><\/a> in 2012 that there’d been insufficient use of preview T&E on many major acquisitions, mainly under the often false premise of acquisitions being off-the-shelf. The<\/span> Government agreed<\/span><\/a> to a policy of conducting preview T&E on all acquisition approvals. The resulting Defence capability planning process began use in 2013 and in 2015 was formally released,<\/span> and endorsed by the ANAO<\/span><\/a> as successfully implemented. The process develops the earliest possible T&E, prior to contract and preferably as part of solicitation, using systematic evaluation of the likely technical and operational risks.<\/span><\/p>\n

Only last month, the Joint Committee for Parliamentary Accounts and Audit examined yet another multi-billion dollar project that was delayed by 3\u20134 years when paper-based down-selection proved inadequate. At the hearing Defence officials advised that the lesson on preview T&E had been learnt, the new T&E policy would ensure a repeat was unlikely, and that, interestingly, the current restructure of Defence should only strengthen use of this preview T&E policy.<\/span><\/p>\n

How the last point would occur wasn’t explained. The two large projects currently undergoing down-selection (SEA 1000 & LAND 400), the outcomes of both possibly to be announced before the election, will seriously test this commitment. Working against the likelihood of the new T&E policy being followed for SEA 1000 is ANAO\u2019s 2015 finding that there are 12 separate test organisations, not one of which is devoted to developmental testing in the maritime domain.<\/span><\/p>\n

Australian Defence has independent test organisations for development of land and aerospace equipment, but for maritime projects this critical skill and governance is left to individual projects to assemble, with the associated loss of independence and disparity inherent in such a solution.<\/span><\/p>\n

Given the recently announced investment in maritime capabilities, Defence must address this T&E deficiency. Otherwise, the lack of adherence to test policy observed by the ANAO in maritime projects in 2002, and again in 2011, will continue. Because of this chequered history, I\u2019ve authored a working paper<\/a>\u00a0(PDF) on how the Australian and American T&E policies ought to apply to SEA 1000 at this critical decision time. The paper proposes:<\/span><\/p>\n