The $1 billion failure of the Super Seasprite helicopter project was a low point in Australian government procurement. It seems incomprehensible that the Department of Defence could ever replicate it.

In detailing a rapid deterioration in regional security, the 2020 defence strategic update highlights that Australia can’t afford to spend time or money on projects that don’t deliver the required warfighting capability efficiently and effectively. The scope and complexity of the capabilities required for Australia’s security are expanding, while the timeframe for procurement is decreasing. Supply-chain issues are bringing new pressures to design and manufacture more in Australia. Defence must be an effective ‘smart buyer’ as envisaged by the first principles review ^[1], which considered it critical that decision-makers assess ‘whether risks and interdependencies have been identified and managed’.

Test and evaluation (T&E) is a key systems engineering tool to identify risk across the capability life cycle, and Defence has long had dedicated policies ^[2] outlining why it’s important and detailing how it should be used in acquisition, sustainment and force generation.

Following multiple reviews, there have been periodic ‘new’ pathways to establish (or recover) and sustain an effective T&E capability. As the various defence procurement and capability manuals have been updated, a consistent theme has been the vital role of T&E in informing risk-based decisions.

In recommending a smart-buyer approach, the first principles review assumes that Defence can use a T&E process to assess whether risks and interdependencies have been identified and managed. T&E has more recently been recognised as one of the 10 initial sovereign defence industry capability priorities.

Given this consistent emphasis, it should surprise taxpayers that almost every review into defence procurement delivers a negative assessment ^[3] of how Defence deals with T&E. Concerns include difficulties in defining, creating and sustaining an experienced workforce; the lag and surge of experience on a project-by-project basis that makes it difficult to apply effective T&E early in the capability life cycle; a lack of coordination between the various entities that are stakeholders in defence T&E (including industry); a lack of coordinated investment in T&E infrastructure; and a lack of accountability to ensure that projects engage T&E and take meaningful account of any subsequent reports.

The first principles review highlighted the need to strengthen and place at arm’s length a continuous contestability function that operates throughout the capability development life cycle from concept to disposal.

It transferred accountability for setting and managing requirements to the vice chief of the defence force and the service chiefs within a regime of strong, arm’s-length contestability.

For contestability to be effective, the risk-identification function must be independent so that assessment is made without bias or influence, intended or unintended. Independence also ensures that the assessor of risk has a voice—not a veto—that is heard at each decision-making level of the capability life cycle. Defence and ultimately the National Security Committee of Cabinet should always make the final risk-based decisions as they are responsible for providing military response options to government.

The first principles review recognised that Defence must ensure that committed people with the right skills are in appropriate jobs. Competence is a matrix of qualifications and experience relevant to the task at hand. Those who performed competently as operational commanders or maintenance engineers may not be competent to assess technical performance, integration and certification risk.

Risk assessors working within Defence face various barriers, individual or organisational, that influence whether their voices are actually heard. Risk must be assessed and the results considered by decision-makers. Given the costs and national security implications, the taxpayer deserves to know that this is occurring, despite the commercial and security considerations of full transparency. There’s already a good model for this. The Office of the Inspector-General of Intelligence and Security conducts regular audits of the national intelligence community as well as specific investigations and reports to the relevant minister and the Parliamentary Joint Committee on Intelligence and Security.

Measures to identify and manage risks and interdependencies must be professional and appropriate to needs across the capability life cycle. This is also true for off-the-shelf products that may be used by an ally. In the Australian mission and environment, an allied capability may be the best option to buy, but those operating and managing it, the government and the taxpayer deserve to know that what is being bought may not be capable of everything hoped of it. It could require additional funding, or a supplementary capability may be required for some tasks. This knowledge is important for operational planning, forecasting funding, and even reputation management.

Facing similar issues, the US Congress legislated for independent T&E providing mandatory annual reporting ^[4] to Congress on all major defence acquisitions. A UK company, Qinetiq, provides technical support to Britain’s Ministry of Defence, including T&E. This model provides independence and highlights that industry can take a lead in determining competence and training requirements, providing training, and managing test ranges and infrastructure.

The defence strategic review should bring a different approach embracing these principles.

It should recommend establishment of a defence capability assurance agency (DCAA), an independent statutory body to assess risks associated with materiel procurement and sustainment, which may include technical risk, systems integration risk, force-integration risk, contractual risk, process risk or even reputational risk. It would be led by a director appointed by a board and its workforce would be qualified and experienced T&E practitioners drawn from defence and industry. The agency would report to the defence minister and parliament.

The DCAA would be required to evaluate risk at agreed points across the capability life cycle and its recommendations would be included in briefs to project managers, assurance bodies, the defence investment committee and the NSC. The agency wouldn’t have a veto but it would ensure that risk-based decisions have a credibly informed basis.

The agency should have a long-term agreement with an Australian industry partner to provide depth of domain expertise and a consistent comprehensive approach to T&E through the life of multiple platforms, environments and systems. The industry partner wouldn’t necessarily conduct all T&E, but as a minimum would oversee the qualifications and professional standards of the workforce conducting T&E.

Defence already has a good model for this approach—the technical airworthiness system. The Defence Aviation Safety Authority (DASA) provides assurance, through assessment of candidate suitability and ongoing audit, that anyone working in this mission-critical and safety-critical area have appropriate qualifications and experience. DASA also provides a range of specialised support services where deep domain expertise is required, such as in aircraft structural integrity.

The industry partner should provide a regulatory function analogous to DASA, in essence acting as the DCAA regulator, and it would be required to sustain an expert workforce providing the capacity to surge, mentor, support and develop T&E practice in support of defence capability. The industry partner could also be responsible for the quality and operation of T&E infrastructure and efficacy of training. Subject to an assessment of probity measures, the industry partner may be limited to selecting, auditing and managing contracted training providers or may also provide a range of in-house T&E training for defence and industry stakeholders.

Scale and flexibility would be achieved by the DCAA drawing on defence personnel or industry providers that meet the qualifications and experience required by the DCAA regulator. This T&E domain expertise would be complemented by the recent operational experience of defence force operators or engineers with relevant T&E training being embedded within DCAA. They could also audit the compliance with qualifications and experience requirements for defence or civilian T&E staff who may be external to the DCAA to comply with operational regulatory requirements.

Defence should be audited to ensure appropriate engagement of the DCAA and transparency of subsequent reporting of identified risks. This should be conducted by a small team with security clearances and subject to protections for commercial-in-confidence information. It would work as part of an independent assurance office.

There are units and individuals within Defence who are professional and competent technical risk assessors, but the organisational inability of Defence to generate, sustain, consistently task and transparently respond to a credible assessment of risk in procurement and sustainment is well known.

The defence strategic review shouldn’t seek to initiate yet another review of T&E within Defence. The strategic imperative for Defence to be a smart buyer, now, should compel the government to instigate legislative reform to establish an independent capability assurance agency.