Early in the Covid-19 pandemic, the myGov website was overwhelmed when worried Australians attempted to access the service. In 2016, the online census system was taken offline on census night after problems arising from a small distributed denial-of-service attack (one designed to overwhelm a website with fake traffic).

With the 2021 census coming up on 10 August, we decided to examine what lessons could be learned from these failures, especially the problem of coping with unexpected digital demand. This thinking is captured in a short ‘explainer’ report ASPI’s International Cyber Policy Centre ^[1] has released today ^[2].

The traditional model of government digital service delivery has tried to predict demand and then build a system that’s the right size to cope with it. Ironically, this seemingly sensible approach is destined to fail. There’s a better way to build such services.

A failure to forecast demand correctly is costly and results in either service failure (when real demand is greater than forecast) or expensive overcapacity (when real demand is less than forecast).

Different types of government services will have different expected patterns of demand.

Services such as Centrelink and Medicare might be expected to have relatively constant demand, although with some seasonal and holiday variation over the year. The Taxation Office has a peak quarter, but also significant ongoing activity throughout the year. Some services, such as the census and the electoral roll can reliably be expected to have large peaks in demand at a predictable time.

Emergency services can expect very large spikes in demand, but they don’t know when they’ll occur or how long they’ll last. Being able to satisfy this demand and provide information to citizens during a crisis is an overriding priority for these types of services.

In the case of the census, there’s a long lead time and a set start time, and it only needs to be available for a predetermined amount of time.

However, even ‘good’ forecasts—forecasts that correctly predict regular peak demand— can’t correctly predict demand in a crisis. This was evident in the myGov failure early in the Covid-19 pandemic, but it seems unreasonable to expect every government service to be built to handle the demands of a possible once-in-a-century event like a global pandemic.

The hard truth about needing to correctly size a service to meet demand is that it is guaranteed to fail in the face of an unexpected crisis—which is when we’d most like a service to be working.

Fortunately, there’s a better way that uses technology to avoid having to make upfront guesstimates about expected demand.

‘Cloud’ services provide capacity that can vary gracefully with demand. They can manage ‘peaky’ workloads that are expected to vary by orders of magnitude and have a proven track record in coping with highly variable demand. They are used, for example, by online gambling sites to handle the demand of the Melbourne Cup and by video game companies to handle online concerts with over 10 million concurrent users ^[3].

In addition to the benefits that come with handling unexpected demand, the use of cloud services would allow more efficient and effective use of the public service’s scarce cybersecurity expertise. These skills are spread across various departments and are focused on maintaining and securing each department’s separately built and maintained systems. This involves duplication of effort and silos of expertise that can’t be applied broadly across the public service.

The government’s 2020 cybersecurity strategy ^[4] notes that pooled delivery can make better use of cybersecurity skills by encouraging the use of ‘secure hubs’. Building services across a common infrastructure would develop a depth of expertise and allow the building blocks of capability to be shared beyond a single service.

The government has also formally recognised that cloud services offer the best model for efficiently handling uncertain demand in its secure cloud strategy ^[5]. There are likely budgetary, skills and cultural issues that need to be addressed so that critical government services don’t fail when we need them most.