{"id":23565,"date":"2015-11-25T10:00:17","date_gmt":"2015-11-24T23:00:17","guid":{"rendered":"http:\/\/www.aspistrategist.ru\/?p=23565"},"modified":"2015-12-16T16:33:27","modified_gmt":"2015-12-16T05:33:27","slug":"the-anaos-grim-picture-of-testing-and-evaluation","status":"publish","type":"post","link":"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/","title":{"rendered":"The ANAO’s grim picture of Defence testing & evaluation"},"content":{"rendered":"
\"NUSHIP<\/a><\/figure>\n

The Australian National Audit Office (ANAO) has just released a report into Defence Test and Evaluation (T&E)<\/a> that makes a number of important findings. According to the report, this critical enabler to informed decisionmaking in Defence capability development and acquisition is spread across too many smaller T&E organisations and doesn\u2019t have adequate oversight of the competency of all of those staff managing the T&E. The paper also finds that Defence only recently has a centralised T&E policy that needs much more refinement and governance. Lastly, on the recent T&E of the Landing Helicopter Dock (LHD) ship, Defence was unable to deliver a sufficient number of competent T&E staff to process the test reports and concessions in a timely fashion.<\/p>\n

Defence uses T&E to have representative users assess if capabilities deliver effective results when used in representative missions. T&E is Defence\u2019s main measure of capability that goes with cost and schedule to determine if a project is reducing risk and thus succeeding. Good T&E gets done as early as possible; ideally T&E gets done as a real preview before full contract (off-the-shelf systems) and certainly should be done before production is contracted.<\/p>\n

ANAO last audited Defence\u2019s T&E in 2001\u20132002. In the 13 years since, they\u2019ve unfortunately found that there has been little or only late progress on most of their areas of concern. That\u2019s despite the concerns being reiterated in a Defence T&E Roadmap in 2008 and in the Senate Inquiry into Defence Procurement (especially Chapters 2 and 12) in 2012. In this same period US Defense has entrenched independent T&E into Congressional laws (Title 10) that provide mandatory independent annual reporting of all T&E measures to Congress on all major Defense acquisitions<\/a>. Title 10 law also requires T&E competency in all US acquisition projects and T&E agencies overseen by the autonomous Defense Acquisition University. The book<\/a> and movie The Pentagon Wars<\/em><\/a> has clearly raised awareness in the US for the value of independent T&E for Defense acquisitions much more than in Australia. There\u2019s a clear option for the Australian government to follow America\u2019s lead with Defence Regulations if Defence doesn\u2019t institute strong, independent, early and adequate T&E for its critical capability decisions.<\/p>\n

Most strikingly, the ANAO report\u2019s insights to the project management of the new LHD ships illustrate that Defence didn\u2019t have the necessary T&E staff to manage the contractor\u2019s reports in a timely manner and that this necessitated the transfer of some capability risk to Navy. The project positions for the T&E were planned but not filled, and the project manager repeatedly advised project governance boards of the vacancies. But it appears that the now defunct Defence Materiel Organisation was unable or unwilling to appropriately prioritise what ultimately became a Navy problem. These difficulties reinforce the ANAO\u2019s previous findings that Defence isn\u2019t managing its T&E competency in all areas. That this observation occurred in the maritime domain should be of critical concern given the significant investment in ships and submarines anticipated in the new Defence Capability Plan, most of which require significant development, integration, test, build and more test in Australia.<\/p>\n

What isn\u2019t directly mentioned in the ANAO report\u2014but concerns me as a former leader in this area\u2014is that of the 12 Defence organisations dedicated to T&E, not one of them does developmental test of ships or submarines: the Navy only has a dedicated operational T&E agency. Each maritime project has to find the T&E expertise and resources itself. In contrast, the Air Force has a dedicated developmental T&E agency. The Army also has one but theirs was in the now defunct DMO and it\u2019s not clear yet if they\u2019ll take this critical enabler back to the service for early insight into project T&E. The implications of this maritime approach are plain to see in the ANAO coverage of the LHD ship T&E. This approach also places significant risk on the adequacy of preview testing for the submarine project options and is at odds with the long-term decision by this government to invest in continuous build of ships.<\/p>\n

The ANAO report and the government\u2019s response today in Parliament point to the First Principles Review and subsequent Defence reorganisation as the means to improve Defence T&E. Little is known of the new structures of this reorganisation other than it aims to reduce acquisition bureaucracy and create a strong strategic centre. If the strategic centre is to be better informed for its acquisition decisionmaking it needs to clearly invest in more structured, resourced and competent T&E. A more complete coverage of what such a first principles reorganisation should consider is in my paper titled \u2018Implementing the Defence First Principles Review: two key opportunities to achieve best practice in capability development\u2019<\/a>.<\/p>\n

Update: Click here<\/a> to view Keith Joiner’s Strategic Insights<\/em> paper, published 17 December 2015.<\/p>\n","protected":false},"excerpt":{"rendered":"

The Australian National Audit Office (ANAO) has just released a report into Defence Test and Evaluation (T&E) that makes a number of important findings. According to the report, this critical enabler to informed decisionmaking in …<\/p>\n","protected":false},"author":443,"featured_media":23568,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"footnotes":""},"categories":[1],"tags":[720,33,38,628,1563],"class_list":["post-23565","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-general","tag-anao","tag-capability","tag-department-of-defence","tag-lhd","tag-testing-evaluation"],"acf":[],"yoast_head":"\nThe ANAO's grim picture of Defence testing & evaluation | The Strategist<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The ANAO's grim picture of Defence testing & evaluation | The Strategist\" \/>\n<meta property=\"og:description\" content=\"The Australian National Audit Office (ANAO) has just released a report into Defence Test and Evaluation (T&E) that makes a number of important findings. According to the report, this critical enabler to informed decisionmaking in ...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/\" \/>\n<meta property=\"og:site_name\" content=\"The Strategist\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/ASPI.org\" \/>\n<meta property=\"article:published_time\" content=\"2015-11-24T23:00:17+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-12-16T05:33:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2015\/11\/20151030ran8107437_637.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"800\" \/>\n\t<meta property=\"og:image:height\" content=\"534\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Keith Joiner\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@ASPI_org\" \/>\n<meta name=\"twitter:site\" content=\"@ASPI_org\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Keith Joiner\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.aspistrategist.ru\/#website\",\"url\":\"https:\/\/www.aspistrategist.ru\/\",\"name\":\"The Strategist\",\"description\":\"ASPI's analysis and commentary site\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.aspistrategist.ru\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-AU\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-AU\",\"@id\":\"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/#primaryimage\",\"url\":\"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2015\/11\/20151030ran8107437_637.jpg\",\"contentUrl\":\"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2015\/11\/20151030ran8107437_637.jpg\",\"width\":800,\"height\":534,\"caption\":\"NUSHIP Adelaide is manoeuvred towards her berth at Fleet base East - Garden Island while her sister ship, HMAS Canberra, can be seen near the entrance into the Captain Cook Graving Dock.\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/\",\"url\":\"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/\",\"name\":\"The ANAO's grim picture of Defence testing & evaluation | The Strategist\",\"isPartOf\":{\"@id\":\"https:\/\/www.aspistrategist.ru\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/#primaryimage\"},\"datePublished\":\"2015-11-24T23:00:17+00:00\",\"dateModified\":\"2015-12-16T05:33:27+00:00\",\"author\":{\"@id\":\"https:\/\/www.aspistrategist.ru\/#\/schema\/person\/218fc406e9b1606fe4ee0c0d71e0b63f\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/#breadcrumb\"},\"inLanguage\":\"en-AU\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.aspistrategist.ru\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The ANAO’s grim picture of Defence testing & evaluation\"}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.aspistrategist.ru\/#\/schema\/person\/218fc406e9b1606fe4ee0c0d71e0b63f\",\"name\":\"Keith Joiner\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-AU\",\"@id\":\"https:\/\/www.aspistrategist.ru\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4f24705d45126e71f4e6c77fbf853f6a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/4f24705d45126e71f4e6c77fbf853f6a?s=96&d=mm&r=g\",\"caption\":\"Keith Joiner\"},\"url\":\"https:\/\/www.aspistrategist.ru\/author\/keith-joiner\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The ANAO's grim picture of Defence testing & evaluation | The Strategist","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/","og_locale":"en_US","og_type":"article","og_title":"The ANAO's grim picture of Defence testing & evaluation | The Strategist","og_description":"The Australian National Audit Office (ANAO) has just released a report into Defence Test and Evaluation (T&E) that makes a number of important findings. According to the report, this critical enabler to informed decisionmaking in ...","og_url":"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/","og_site_name":"The Strategist","article_publisher":"https:\/\/www.facebook.com\/ASPI.org","article_published_time":"2015-11-24T23:00:17+00:00","article_modified_time":"2015-12-16T05:33:27+00:00","og_image":[{"width":800,"height":534,"url":"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2015\/11\/20151030ran8107437_637.jpg","type":"image\/jpeg"}],"author":"Keith Joiner","twitter_card":"summary_large_image","twitter_creator":"@ASPI_org","twitter_site":"@ASPI_org","twitter_misc":{"Written by":"Keith Joiner","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebSite","@id":"https:\/\/www.aspistrategist.ru\/#website","url":"https:\/\/www.aspistrategist.ru\/","name":"The Strategist","description":"ASPI's analysis and commentary site","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.aspistrategist.ru\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-AU"},{"@type":"ImageObject","inLanguage":"en-AU","@id":"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/#primaryimage","url":"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2015\/11\/20151030ran8107437_637.jpg","contentUrl":"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2015\/11\/20151030ran8107437_637.jpg","width":800,"height":534,"caption":"NUSHIP Adelaide is manoeuvred towards her berth at Fleet base East - Garden Island while her sister ship, HMAS Canberra, can be seen near the entrance into the Captain Cook Graving Dock."},{"@type":"WebPage","@id":"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/","url":"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/","name":"The ANAO's grim picture of Defence testing & evaluation | The Strategist","isPartOf":{"@id":"https:\/\/www.aspistrategist.ru\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/#primaryimage"},"datePublished":"2015-11-24T23:00:17+00:00","dateModified":"2015-12-16T05:33:27+00:00","author":{"@id":"https:\/\/www.aspistrategist.ru\/#\/schema\/person\/218fc406e9b1606fe4ee0c0d71e0b63f"},"breadcrumb":{"@id":"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/#breadcrumb"},"inLanguage":"en-AU","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.aspistrategist.ru\/the-anaos-grim-picture-of-testing-and-evaluation\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.aspistrategist.ru\/"},{"@type":"ListItem","position":2,"name":"The ANAO’s grim picture of Defence testing & evaluation"}]},{"@type":"Person","@id":"https:\/\/www.aspistrategist.ru\/#\/schema\/person\/218fc406e9b1606fe4ee0c0d71e0b63f","name":"Keith Joiner","image":{"@type":"ImageObject","inLanguage":"en-AU","@id":"https:\/\/www.aspistrategist.ru\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/4f24705d45126e71f4e6c77fbf853f6a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4f24705d45126e71f4e6c77fbf853f6a?s=96&d=mm&r=g","caption":"Keith Joiner"},"url":"https:\/\/www.aspistrategist.ru\/author\/keith-joiner\/"}]}},"_links":{"self":[{"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/posts\/23565"}],"collection":[{"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/users\/443"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/comments?post=23565"}],"version-history":[{"count":6,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/posts\/23565\/revisions"}],"predecessor-version":[{"id":23966,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/posts\/23565\/revisions\/23966"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/media\/23568"}],"wp:attachment":[{"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/media?parent=23565"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/categories?post=23565"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/tags?post=23565"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}