Data gaps and demographic change: the end of the NZ census will create big blind spots

Ending the New Zealand census as we’ve known it will save money – it was “no longer financially viable”, according to Statistics Minister Shane Reti – but the true cost of those savings could be considerable.

Of course, it’s no secret the two previous censuses raised major questions about the quality of census data and the process. In 2018, an untested experiment with online returns, and a reduced workforce in the field, saw “an unacceptably low response rate”.

In 2023, StatsNZ had to apologise again, this time for failing to keep the collected data safe and for another low response rate, especially for Maori. The problems were compounded by low trust in government and an unwillingness to share private information in the wake of COVID-related misinformation.

It didn’t help that the 2023 census cost NZ$325 million, up from $104 million in 2013 – double the amount per capita, for reasons that remain unclear.

That was enough. Cabinet papers between March and May last year signalled the government was going to move to an alternative system of data collection. The shift was characterised as “modernising the census” – except there will be no census.

But the change has been made without any apparent consideration of how the census is used – specifically, that it is crucial to the management of a modern society and economy – and what will be lost in the process.

Comparison across time

One of the primary functions of a census is to allow comparison with previous censuses over time. And these go back a long way.

The first census, in 1851, collected data on Europeans only, although the Native Secretary provided details of Maori from 1849 to 1850. The Census Act of 1858 required that a national census of all Europeans take place every three years. A new act in 1877 introduced the five-yearly census we’ve become used to.

Data on Maori was collected separately until 1916 when a question on “race” appeared. The 1926 Census and Statistical Act then required all individuals, including Maori, to complete the census forms.

Depression and war meant there were no censuses in 1931 and 1941, and the 2011 census was delayed because of the Christchurch earthquakes. Otherwise, we have had regular updates from nearly all the resident population on a whole range of aspects of life in New Zealand.

This comprehensive picture of New Zealanders and the way we live underpins nearly every aspect of political decision-making and policy development. But no more.

The new approach will use existing administrative data collected by government departments and agencies as part of their normal business. ACC, Inland Revenue, the Ministry of Social Development, Ministry of Education, and Department of Internal Affairs will be key data sources.

The data gaps will be addressed by asking those departments and agencies to change some of what they collect. But the main change will involve surveys – as yet unspecified in terms of sample size or frame, or the questions and topics to be covered – which will “verify data quality and fill gaps”.

As well as saving money, the statistics minister says, this approach will provide “more timely insights”. But this all leaves important questions unanswered.

Inadequate administrative data

Administrative data is collected for specific purposes and in different ways by government departments and agencies. The coverage is incomplete, there is often no consistency in what is collected, and there are issues about data quality and robustness.

Moreover, information management is not a particular strength of most public sector agencies (Inland Revenue might be one of the few exceptions). It will be interesting to see whether the government is prepared to fund new technology options and methods to help improve this data collection.

For example, the Understanding Policing Delivery research project has identified issues with data collection, especially in relation to ethnicity: national intelligence activities collect and hold data on ethnicity, iwi and hapu affiliations, but the process for issuing police infringement notices for offending does not.

As a StatsNZ exercise which looked at ethnicity data collection across the government sector noted:

The question asked for ethnicity differs widely across administrative data sources and often differs within each administrative source depending on the mode of collection or the form used.

Such inconsistencies will need to be rectified if administrative data is to be anything like as comprehensive and consistent as the data provided by the census.

Major demographic change

New Zealand is also undergoing major demographic change, including the following trends:

• fertility has declined and is at sub-replacement levels

• the population is rapidly ageing

• the proportion of population living in the top half of the North Island is increasing

• and immigration has contributed significantly to population growth and diversity.

I am not convinced the new administrative approach will capture these demographic changes, much less good data on the wellbeing of various communities or the nature of families and households.

Administrative data, by definition, is partial and suited to the particular activities and concerns of the agency or department in question. But in a modern, complex society, data is key. We have just lost one of the most powerful tools available for understanding this country in the 21st century.

The author acknowledges Len Cook, former Government Statistician of New Zealand, for his comments and suggestions.

Paul Spoonley has received funding from MBIE and is associated with Koi Tu.