Former Australian Statistician Bill McLennan says the Independent Assurance Panel's 'Report on the Quality of 2106 Census Data' is the flawed outcome of a sloppy process, which glosses over significant failures by the ABS.
I READ the Report on the Quality of 2106 Census Data from the Census Independent Assurance Panel to the Australian Statistician very closely and with great interest.
Here, I will discuss three topics:
- what the Panel did and the conclusions it reached;
- what the panel didn’t do; and, finally,
- the "digital first approach" adopted by the ABS for the 2016 Census.
1. PANEL'S FINDINGS NOT "FIT-FOR-PURPOSE"
My first impression is that the Panel’s Report is the outcome of a very sloppy process.
The terms of reference of the Panel in essence say
‘... the Panel [is] to provide an independent view of the quality of Census outputs from the 2016 Census.'
That’s fair enough, but the panel didn’t see its brief this way.
The panel changed what it was tasked to do, when it said:
‘The panel … was tasked … to provide an independent assessment of the quality and fitness-for-purpose of the Population and Housing data.’
It did so without saying why this change was made or by whom it was made.
Does this matter?
Well, "quality" and "fitness-for-purpose" are two different concepts that have different meanings. To fulfil this revised charter, the Panel would have had to check separately for quality and for fitness-for purpose for every possible use that might be made of the Census outputs! That, of course, would be an almost impossible task to complete.
Nevertheless the Panel continued and concluded, on page iii of their Report, which was part of the executive summary:
‘In determining whether the 2016 Census is fit-for-purpose as required under its Terms of Reference, the Panel assessed this requirement with reference to the purpose and use of the Census in Section 2.1.'
(Image screenshot from 'Report on the Quality of 2106 Census Data' via abs.gov.au)
In making this assessment, the Panel considered whether the 2016 Census data could be used with confidence by stakeholders when compared to the data from previous censuses. It seems, therefore, that it concluded that the Census was fit-for-purpose. However, no information is given about how that conclusion was reached. To me, that is not very comforting.
Further, the Panel’s report is swimming in unsubstantiated claims, or assertions.
For example, in the executive summary the Panel said
‘... the 2016 Census population counts compared well against the preliminary 2016 Estimated Resident Population data.’
Well when you read the section below on demographic analysis, you will find out that the pre-Census Estimated Resident Population for Australia was significantly underestimated.
The sweeping generalisations that abound in this report must be taken with a grain of salt.
The Panel’s Report did add a very significant note:
‘Given the limited time available and data availability, the Panel made this assessment at the national, state and and/or territory levels for selected variables.’
In other words, the Panel noted it hadn’t been able to reach any conclusions about the accuracy of the statistics for small geographic areas, which wasn’t unexpected and the conclusions it did reach related only to the specific set of selected variables. This means that the Panel, nor anyone else, cannot legitimately say, on the basis of the Panels Report, that the Census data is acceptable.
Nevertheless, the Panel in its Report, in the executive summary said:
‘The Panel has concluded that the 2016 Census data is fit for purpose and can be used with confidence. The 2016 data is is of comparable quality the 2011 and 2006 data.’
The Independent Assurance Panel concluded 2016 Census data is fit-for-purpose, can use with confidence. Of comparable quality to 2011, 2006
— Alison Taylor (@AlphaAlison) June 26, 2017
It is very surprising and rather disappointing that the Panel’s Report failed to recognise the facts the Panel itself had put forward. Was the opportunity to provide a quotable quote, even an incorrect one, too good to miss?
In addition, the chair of the Independent Panel, Professor Sandra Harding, said:
‘Prior to census night, public concerns were raised about privacy. Impacts are apparent with more persons reporting age rather than date of birth as well as a large decline in the number of people agreeing to have their Census form archived for 99 years. Further, while it is impossible to know whether some people provided valid names that were not their own, few names were withheld or clearly false names reported. While these outcomes have minimal impact on the accuracy of the Census, incorrect names and lack of date of birth for some respondents can hinder data matching activities.’
I accept this view.
The Australian Statistician, however, said:
‘In our secure facility the data has been comprehensively reviewed and analysed for consistency between Censuses in geographical areas ... This comprehensive range of checks has confirmed the quality of the Census data.'
I cannot comment on this matter as I am not fully aware of the work done in the secure facility. However, I can say that the Panel’s report doesn’t seem to support this contention.
2. THE PLACES THE PANEL WOULD NOT GO
To change the topic, there are several issues that the Panel should have commented on but didn’t.
First, demographic studies, which in the past have been important part of showing the Census has produced reasonable results, have not been undertaken. Such studies are important because they are a source that is independent of the Census process. In days gone by, the ABS had an active demography unit, which would have provided advice, but if the ABS doesn’t have such essential expertise now, some professional help should have been commissioned.
On this note, some thought provoking demographic articles have recently appeared on the Internet. For example, at population expert Glen Capuano at blog.id.com.au provides the distressing news that pre the 2016 census, the ABS had underestimated the Estimated Resident Population (ERP) for Australia by 83,650 and for Victoria by 117,207.
(Source: blog.id.com.au)
The likely conclusion to draw is that that the ERP estimates pre the 2016 Census are incorrect, however it should not be lost from sight that the 2016 Census numbers could be responsible for some of this discrepancy.
I can say from experience this is not a happy result for the ABS, as I was around when it was discovered that the Estimated Resident Population for Queensland was grossly understated. A proper demographic analysis is something the Panel should have insisted on.
Second, no comment has been made in the report about the impact the high non-response rates that resulted for the Post Enumeration Survey (which I understand was around 9.3%) might have on the PES’ usefulness. I consider that a 9.3% non-response rate in the 2016 Post Enumeration Surveyshows its outcomes should be taken with a grain or two of salt.
That this issue has not surfaced is very worrying to me, even if it didn’t surprise me that the ABS has ignored the topic.
I discussed the poor non-response rates for the PES with an ex-colleague, whom I consider knows more about running Censuses and Post Enumeration Surveys than anyone else I know. When I told him that I gathered that the 2016 PES had a 9.3% non-response, his response was that if he had run a Post Enumeration Survey that suffered such a high non-response rate, then he would have got a kick in the xxxx from his boss. I acknowledged he was correct.
3. THE FLAWED NEW "DIGITAL FIRST" APPROACH
Also, the analysis of the digital first approach to the collecting of Census information by the Panel is significantly flawed.
The Report noted [IA emphasis]:
The 2016 Census was conducted with a new ‘digital first’ approach, that centred on the use of a recently established Address Register. Notwithstanding the withdrawal of the online Census form on Census night, the digital approach appears to have been well-received by the public.
Nearly all people in Australia completed Census forms, with the majority completing the Census online. While using an Address Register as the Census frame achieved important efficiencies, there were new challenges in the determination of whether dwellings were occupied on Census night. This impacted on the number of people who were imputed into private dwellings. Furthermore, the response rate in non-private dwellings was lower than in previous Censuses, which put additional pressure on the imputation process.
While challenges were experienced and should be addressed for the 2021 Census, the changed approach led to a more efficient, effective and modern Census operation. The Panel strongly supports the changed approach to the Census and its further refinement in order to secure a high quality and financially sustainable Census into the future.
In my view, this is obviously an incorrect conclusion, which flies in the face of the facts that were in front of the Panel.
Is the #census reliable? Read what the independent panel found. https://t.co/L6tuu1c5Tl pic.twitter.com/LxnjvS5vg4
— Dr Liz Allen (@DrDemography) June 27, 2017
The Panel, in its Report, acknowledged that the ABS'
‘... new digital first approach centred on the use of the recently established Address Register.’
However, from there-on, the Panel seemed to assume that weaknesses in the Register or field operations didn’t exist and that its use of the Register in its first operation, the Census, didn’t incur any operational difficulties.
To be charitable, perhaps, the Panel was only thinking about the electronic Census Form and just lost sight of the overall picture of how the data was collected. I say this because even the possibility of problems with the use of the Register or the field operations has not even been mentioned anywhere in the Panel’s Report.
As a manager, I’d suggest that it was simply foolish on the part of the ABS to even attempt to use the Address Register for the first time in a live situation, particularly in the collection of Census data and while also following new methodology. Also, I assume, without any effective pilot testing .
Does this show an extreme lack of understanding in the ABS concerning the art of collecting statistics?
However, evidence abounds, which in my view shows the Address Register was possibly very flawed. There is also evidence which shows that the management of the field work for collecting the Census was appallingly poor, bordering on being incompetent.
Submission 39 to the Senate Inquiry into the 2016 Census is from .ID the population experts and its author is the same Glenn Capuano. He is a strong daily user of Census demographic data, particularly helping local government and business make evidence-based decisions about their communities. He worked in the ABS for ten years, including on the 2006 Census management, and worked on the past two Censuses as an area supervisor, including for the recent Census.
In brief, Capuano said that the quality of the Address Register was poor and stated in detail why he thought so.
(Screenshot aph.gov.au)
Submission 59 to the Senate Inquiry, titled 'A perspective from the field', was authored by an Area Supervisor for the 2016 Census in Sydney. One of the most telling comments he made was that sometimes directions were changed midstream, revealing the challenges of introducing an 'untried enumeration model' on such a large scale. He was also critical of the numerous errors in the Address Database, in which he said he had over 150 cases in his area alone.
Submission 90 was from several Area Supervisors that presented a well documented case that the management of the collection of the Census data had serious flaws in it and all had difficulties with the Address Register.
Who could possibly believe that these problems didn’t impact on the quality of the Census results? Well, the Independent Panel, for one.
The sum total of just these criticisms means that the Census Review Panel was foolish to ignore the possibility that the Address Register had flaws in its use, some of which maybe even be fatal. It was also poor form not to even give any countenance to the possibility that the collection management system might be flawed. It seems impossible to me, from my long term hands-on experience collecting data from Australians, that no serious impacts would result from this poor Census process.
Rather than ignoring the problem, the ABS should investigate the short falls that became obvious during the collection of the Census and then design, and test replacement procedures. This should be done, while the iron is hot and not left on the shelf for a few years.
That approach would probably mean that the lessons learnt on this occasion might not be applied for the next Census, perhaps because with the passage of time their impact might fade in the minds of the decision makers. This would be a very poor outcome.
Bill McLennan was the ABS Australian Statistician and Director of the UK Central Statistical Office.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Australia License
Monthly Donation
Single Donation
Upload IA. Subscribe today.
