{"id":30668,"date":"2017-02-24T12:30:19","date_gmt":"2017-02-24T01:30:19","guid":{"rendered":"https:\/\/www.aspistrategist.ru\/?p=30668"},"modified":"2017-02-24T13:37:03","modified_gmt":"2017-02-24T02:37:03","slug":"big-data-devils-detail","status":"publish","type":"post","link":"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/","title":{"rendered":"Big Data: the devil\u2019s in the detail"},"content":{"rendered":"<p style=\"text-align: center;\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-30673 size-full\" title=\"Image courtesy of Pixabay user PeteLinforth.\" src=\"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2017\/02\/hacking-2077124_640.jpg\" alt=\"Image courtesy of Pixabay user PeteLinforth.\" width=\"640\" height=\"479\" srcset=\"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2017\/02\/hacking-2077124_640.jpg 640w, https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2017\/02\/hacking-2077124_640-205x153.jpg 205w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><\/p>\n<p>As the government\u2019s <a href=\"https:\/\/www.dpmc.gov.au\/national-security\/2017-independent-intelligence-review\" target=\"_blank\">review<\/a> of the Australian Intelligence Community (AIC) picks up steam, one of the key challenges is to identify and resolve growing gaps in the AIC\u2019s technological capabilities. One such capability is the collection and use of Big Data.<\/p>\n<p>The generally accepted definition of big data casts it as a \u201cproblem\u201d, because it\u2019s characterised by its extreme volume, velocity, and variety, which makes collection, management and analysis rather challenging. The problem stems from a \u2018<a href=\"http:\/\/www.economist.com\/node\/15579717\" target=\"_blank\"><em>data deluge<\/em><\/a>\u2019 of social media posts, photos, videos, purchases, clicks and a burgeoning wave of sensor data from smarter and interconnected appliances and accessories, known as the \u2018<a href=\"https:\/\/www.wired.com\/insights\/2014\/11\/the-internet-of-things-bigger\/\" target=\"_blank\"><em>Internet of Things<\/em><\/a>\u2019. Those sources generated a staggering 4.4 trillion gigabytes of data in <a href=\"https:\/\/www.emc.com\/leadership\/digital-universe\/2014iview\/executive-summary.htm\" target=\"_blank\">2013<\/a>, but that figure is forecasted to reach 44 trillion gigabytes of data by <a href=\"https:\/\/www.emc.com\/leadership\/digital-universe\/2014iview\/executive-summary.htm\" target=\"_blank\">2020<\/a>, which threatens to overwhelm conventional methods for storing and analysing data.<\/p>\n<p>In response to the problem of big data is the \u201cpromise\u201d of big data analytics. Analytics promises to not only manage the data deluge, but also to analyse the data using algorithms to uncover hidden correlations, patterns and links of potential analytical value. Techniques to extract those insights fall under various <a href=\"https:\/\/datasociety.net\/output\/data-civil-rights-technology-primer\/\" target=\"_blank\">names<\/a>: \u2018data mining\u2019, \u2018data analytics\u2019, \u2018data science\u2019, and \u2018machine learning\u2019, among others. That work is expected to yield new insights into a range of puzzles from tracking financial <a href=\"http:\/\/fortune.com\/2016\/03\/23\/palantir-to-track-down-rogue-traders\/\" target=\"_blank\">fraud<\/a> to detecting <a href=\"http:\/\/www.csc.com\/cybersecurity\/publications\/93325\/104033-using_big_data_to_defend_against_cyber_security_threats\" target=\"_blank\">cybersecurity<\/a> incidents through the power of parallel processing hardware, distributed software, new analytics tools and a talented workforce of multidisciplinary data scientists.<\/p>\n<p>However, in order to keep the big data \u201cpromise\u201d, the AIC review needs to address the following challenges:<\/p>\n<ol>\n<li><span style=\"font-weight: 400;\">There need to be mechanisms to ensure that data is manageable in terms of the definitional iron triangle of volume (size), velocity (of data flow), and variety (of data types and formats). While these are challenging enough in the big data <\/span><a href=\"https:\/\/www.ft.com\/content\/21a6e7d8-b479-11e3-a09a-00144feabdc0\" target=\"_blank\"><span style=\"font-weight: 400;\">context<\/span><\/a><span style=\"font-weight: 400;\">, the data analyst must consider the veracity of the datasets they select: the data\u2019s representativeness, reliability, and accuracy. The analyst must also consider how the data will generate insights of value to the end user.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">The current framework for privacy protection, based on the idea of \u2018<\/span><a href=\"https:\/\/www.oaic.gov.au\/media-and-speeches\/speeches\/big-data-and-privacy-a-regulators-perspective\" target=\"_blank\"><span style=\"font-weight: 400;\">notice and consent<\/span><\/a><span style=\"font-weight: 400;\">\u2019, needs updating. Currently, the burden is placed on an individual to make an informed decision about the risks of sharing their information. This is complicated in the context of big data, as this informed decision is, in the words of former President Obama\u2019s council of <\/span><a href=\"https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/microsites\/ostp\/PCAST\/pcast_big_data_and_privacy_-_may_2014.pdf\" target=\"_blank\"><span style=\"font-weight: 400;\">advisors<\/span><\/a><span style=\"font-weight: 400;\">, \u2018defeated by exactly the positive benefits that big data enables: new, non-obvious, unexpectedly powerful uses of data\u2019.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Data-based decisions need to be comprehensible and explicable by maintaining the <\/span><a href=\"http:\/\/journals.sagepub.com\/doi\/abs\/10.1177\/2053951715622512\" target=\"_blank\"><span style=\"font-weight: 400;\">transparency<\/span><\/a><span style=\"font-weight: 400;\"> and \u2018<\/span><a href=\"https:\/\/arxiv.org\/abs\/1606.03490\" target=\"_blank\"><span style=\"font-weight: 400;\">interpretability<\/span><\/a><span style=\"font-weight: 400;\">\u2019. That will become more challenging over time, as explaining the processes and reasoning behind a machine learning algorithm isn\u2019t a simple task, especially as those algorithms become more complex, merge into composite ensembles, and utilise <\/span><a href=\"http:\/\/www.forbes.com\/sites\/gilpress\/2013\/04\/19\/big-data-news-roundup-correlation-vs-causation\/#90c54ec49360\" target=\"_blank\"><span style=\"font-weight: 400;\">correlative<\/span><\/a><span style=\"font-weight: 400;\"> and statistical reasoning.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Regardless of whether data analytics decisions are explicable, a fourth challenge is whether the public will accept algorithmic decisions. The ongoing debate about machine learning in <\/span><a href=\"https:\/\/www.technologyreview.com\/s\/542626\/why-self-driving-cars-must-be-programmed-to-kill\/\" target=\"_blank\"><span style=\"font-weight: 400;\">autonomous vehicles<\/span><\/a><span style=\"font-weight: 400;\"> demonstrates these concerns. As decisions of increasing importance are informed by algorithms, this challenge of \u2018<\/span><a href=\"http:\/\/opim.wharton.upenn.edu\/risk\/library\/WPAF201410-AlgorthimAversion-Dietvorst-Simmons-Massey.pdf\" target=\"_blank\"><span style=\"font-weight: 400;\">Algorithm Aversion<\/span><\/a><span style=\"font-weight: 400;\">\u2019 will intensify.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Big data and analytics security needs to be ensured. That isn\u2019t just limited to the traditional problem of the \u2018<\/span><a href=\"https:\/\/www.cnet.com\/au\/news\/mandatory-data-retention-metadata-honeypot-for-hackers\/\" target=\"_blank\"><span style=\"font-weight: 400;\">honeypot<\/span><\/a><span style=\"font-weight: 400;\">\u2019 allure of big datasets. It also relates to the distributed, parallel, and diverse architectural challenges of big data computing. It also relates to the security of the analytics themselves, with research into \u2018<\/span><a href=\"https:\/\/arxiv.org\/abs\/1312.6199\" target=\"_blank\"><span style=\"font-weight: 400;\">adversarial examples<\/span><\/a><span style=\"font-weight: 400;\">\u2019 showing that small changes to inputs can result in serious misclassification errors in machine learning algorithms.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Big data has both benefited and suffered from fever-pitch hype. According to Gartner\u2019s <\/span><a href=\"http:\/\/www.gartner.com\/technology\/research\/methodologies\/hype-cycle.jsp\" target=\"_blank\"><span style=\"font-weight: 400;\">Hype Cycle<\/span><\/a><span style=\"font-weight: 400;\">, mainstream attention to big data began in 2011, peaked in 2013, and was removed from the hype cycle in 2015, with the <\/span><a href=\"http:\/\/blogs.gartner.com\/nick-heudecker\/big-data-is-now-normal\/\" target=\"_blank\"><span style=\"font-weight: 400;\">explanation<\/span><\/a><span style=\"font-weight: 400;\"> that big data is now \u2018normal\u2019 and several of its aspects are \u2018no longer exotic. They\u2019re common\u2019. While the idea that big data is the \u2018new normal\u2019 is reaching a <\/span><a href=\"http:\/\/www.infoworld.com\/article\/3025931\/big-data\/why-open-source-is-the-new-normal-for-big-data.html\"><span style=\"font-weight: 400;\">growing<\/span><\/a> <a href=\"https:\/\/www.sas.com\/en_us\/insights\/articles\/data-management\/i-see-big-data.html\" target=\"_blank\"><span style=\"font-weight: 400;\">consensus<\/span><\/a><span style=\"font-weight: 400;\">, it\u2019s important to note that some of the early big data promises no longer hold in reality, and the distinction between hype and reality needs to be clear to policymakers before effective big data policy can be developed.<\/span><\/li>\n<\/ol>\n<p>Over the coming months, ASPI, with support from CSC Australia, will undertake an analysis of Big Data in National Security to further explore the policy issues and challenges outlined in this piece, and to stimulate policy discussions around the issue.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>As the government\u2019s review of the Australian Intelligence Community (AIC) picks up steam, one of the key challenges is to identify and resolve growing gaps in the AIC\u2019s technological capabilities. One such capability is the &#8230;<\/p>\n","protected":false},"author":608,"featured_media":30673,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"footnotes":""},"categories":[1],"tags":[343,646,391,95],"class_list":["post-30668","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-general","tag-australian-intelligence-community","tag-big-data","tag-cyber","tag-cyber-security"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v19.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Big Data: the devil\u2019s in the detail | The Strategist<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Big Data: the devil\u2019s in the detail | The Strategist\" \/>\n<meta property=\"og:description\" content=\"As the government\u2019s review of the Australian Intelligence Community (AIC) picks up steam, one of the key challenges is to identify and resolve growing gaps in the AIC\u2019s technological capabilities. One such capability is the ...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/\" \/>\n<meta property=\"og:site_name\" content=\"The Strategist\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/ASPI.org\" \/>\n<meta property=\"article:published_time\" content=\"2017-02-24T01:30:19+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-02-24T02:37:03+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2017\/02\/hacking-2077124_640.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"640\" \/>\n\t<meta property=\"og:image:height\" content=\"479\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Michael Chi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@ASPI_org\" \/>\n<meta name=\"twitter:site\" content=\"@ASPI_org\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Michael Chi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.aspistrategist.ru\/#website\",\"url\":\"https:\/\/www.aspistrategist.ru\/\",\"name\":\"The Strategist\",\"description\":\"ASPI&#039;s analysis and commentary site\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.aspistrategist.ru\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-AU\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-AU\",\"@id\":\"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/#primaryimage\",\"url\":\"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2017\/02\/hacking-2077124_640.jpg\",\"contentUrl\":\"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2017\/02\/hacking-2077124_640.jpg\",\"width\":640,\"height\":479,\"caption\":\"Image courtesy of Pixabay user PeteLinforth.\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/\",\"url\":\"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/\",\"name\":\"Big Data: the devil\u2019s in the detail | The Strategist\",\"isPartOf\":{\"@id\":\"https:\/\/www.aspistrategist.ru\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/#primaryimage\"},\"datePublished\":\"2017-02-24T01:30:19+00:00\",\"dateModified\":\"2017-02-24T02:37:03+00:00\",\"author\":{\"@id\":\"https:\/\/www.aspistrategist.ru\/#\/schema\/person\/bf73a1a4a3c6adc42839e2f406680bb9\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/#breadcrumb\"},\"inLanguage\":\"en-AU\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.aspistrategist.ru\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Big Data: the devil\u2019s in the detail\"}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.aspistrategist.ru\/#\/schema\/person\/bf73a1a4a3c6adc42839e2f406680bb9\",\"name\":\"Michael Chi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-AU\",\"@id\":\"https:\/\/www.aspistrategist.ru\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9308fb9be6733bb4692a599b680cd09c?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9308fb9be6733bb4692a599b680cd09c?s=96&d=mm&r=g\",\"caption\":\"Michael Chi\"},\"url\":\"https:\/\/www.aspistrategist.ru\/author\/michael-chi\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Big Data: the devil\u2019s in the detail | The Strategist","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/","og_locale":"en_US","og_type":"article","og_title":"Big Data: the devil\u2019s in the detail | The Strategist","og_description":"As the government\u2019s review of the Australian Intelligence Community (AIC) picks up steam, one of the key challenges is to identify and resolve growing gaps in the AIC\u2019s technological capabilities. One such capability is the ...","og_url":"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/","og_site_name":"The Strategist","article_publisher":"https:\/\/www.facebook.com\/ASPI.org","article_published_time":"2017-02-24T01:30:19+00:00","article_modified_time":"2017-02-24T02:37:03+00:00","og_image":[{"width":640,"height":479,"url":"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2017\/02\/hacking-2077124_640.jpg","type":"image\/jpeg"}],"author":"Michael Chi","twitter_card":"summary_large_image","twitter_creator":"@ASPI_org","twitter_site":"@ASPI_org","twitter_misc":{"Written by":"Michael Chi","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebSite","@id":"https:\/\/www.aspistrategist.ru\/#website","url":"https:\/\/www.aspistrategist.ru\/","name":"The Strategist","description":"ASPI&#039;s analysis and commentary site","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.aspistrategist.ru\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-AU"},{"@type":"ImageObject","inLanguage":"en-AU","@id":"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/#primaryimage","url":"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2017\/02\/hacking-2077124_640.jpg","contentUrl":"https:\/\/www.aspistrategist.ru\/wp-content\/uploads\/2017\/02\/hacking-2077124_640.jpg","width":640,"height":479,"caption":"Image courtesy of Pixabay user PeteLinforth."},{"@type":"WebPage","@id":"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/","url":"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/","name":"Big Data: the devil\u2019s in the detail | The Strategist","isPartOf":{"@id":"https:\/\/www.aspistrategist.ru\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/#primaryimage"},"datePublished":"2017-02-24T01:30:19+00:00","dateModified":"2017-02-24T02:37:03+00:00","author":{"@id":"https:\/\/www.aspistrategist.ru\/#\/schema\/person\/bf73a1a4a3c6adc42839e2f406680bb9"},"breadcrumb":{"@id":"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/#breadcrumb"},"inLanguage":"en-AU","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.aspistrategist.ru\/big-data-devils-detail\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.aspistrategist.ru\/"},{"@type":"ListItem","position":2,"name":"Big Data: the devil\u2019s in the detail"}]},{"@type":"Person","@id":"https:\/\/www.aspistrategist.ru\/#\/schema\/person\/bf73a1a4a3c6adc42839e2f406680bb9","name":"Michael Chi","image":{"@type":"ImageObject","inLanguage":"en-AU","@id":"https:\/\/www.aspistrategist.ru\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9308fb9be6733bb4692a599b680cd09c?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9308fb9be6733bb4692a599b680cd09c?s=96&d=mm&r=g","caption":"Michael Chi"},"url":"https:\/\/www.aspistrategist.ru\/author\/michael-chi\/"}]}},"_links":{"self":[{"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/posts\/30668"}],"collection":[{"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/users\/608"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/comments?post=30668"}],"version-history":[{"count":4,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/posts\/30668\/revisions"}],"predecessor-version":[{"id":30684,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/posts\/30668\/revisions\/30684"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/media\/30673"}],"wp:attachment":[{"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/media?parent=30668"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/categories?post=30668"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aspistrategist.ru\/wp-json\/wp\/v2\/tags?post=30668"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}