2018 Census External Data Quality Panel: Minutes of Meeting
on 7 December 2018
Date and time
7 December 2018, 9am to 3:30pm
Location
Stats NZ, Auckland Office
48 Greys Avenue,
Auckland Central
Present -
Richard Bedford - Chair
panel members Tahu Kukutai
Donna Cormack
Alison Reid
Thomas Lumley
Len Cook (on video conference from Wel ington from 11:30am)
Barry Milne
Ian Cope (on video conference from UK until 1pm)
Present – Stats
Auckland
NZ
Kathy Connol y, General Manager Census
Vince Galvin, Chief Methodologist
Gareth Meech, Senior Manager Census and Secretariat
Steph Prosser, Senior Analyst Census
Adele Quinn, Manager Census Analytics
Christine Bycroft, Principal Statistician
Christchurch via v/c for selected sessions
Brooke Theyers, Senior Population Insights
Robert Didham, Demographic Analyst
Mike Berry, Manager Census Content and Customer Relations
Digby Carter, Statistical Analyst, Census Content and Customer Relations
Meeting minutes
9-10am – Previous minutes and shared workspace issues
The Chair and secretariat worked through the minutes from the November meeting. Specific
updates are noted in the second half of this document. The secretariat then talked through the
new shared workspace structure and answered questions about the setup. Noted that it looked
like panel members did not have write-access. Also noted that any attachments added should be
appropriately label ed to al ow easier reference and cross reference.
AP4-1: Secretariat to ensure al panel members have shared workspace write-access and items
are label ed appropriately.
AP4-2: Secretariat to ensure al documents have useful headers and footers to ensure it is clear
what panel members are reviewing.
There were several comments from the panel that the late loading of the shared workspace the
previous day made it difficult for the panel members to be prepared. This was acknowledged but
it was pointed out that the newly run data meant many parts of the presentation had only been
created during the previous 2-3 days. There was also a discussion about whether the panel
members should have access to a folder just for their use. Such a folder can be created but it
would not be completely secure given that the manager of the secure site (the secretariat) would
Page 1 of 10
have access. However, the panel was assured that Stats NZ staff would not access the material in
such a folder if it was created. No specific action is required at this stage.
A glossary was also requested for SNZ acronyms.
AP4-15: Gareth to begin compiling a glossary of acronyms that can be added to over time. This
wil be uploaded to the workspace.
The date for the March meeting was confirmed as Wednesday 6 March 2019 in Auckland.
There was a request to add an action to load a document that shows the internet col ection
system screens that respondents completed – in particular the household set-up form. Adele
noted that we do not have a ful copy of every screen, but we could col ate screen shots from
most parts of the system.
AP4-3: Adele to create a load document with Internet Col ection System screen shots for panel
members.
10am – Update about public communications and programme timelines
Kathy talked about the media release published on 27 Nov and the low general media uptake to
date. There has been some interest on social media and from Māori media. Kathy thanked the
panel for their advice as it influenced the media release and supporting messages. The panel
commented that stakeholders were not that surprised by the announcement. Kathy discussed
some complementary strategies being started to better meet the needs for data for Māori. The
panel noted that if Stats NZ know now that iwi data are not going to be of sufficient quality for
publication then they should be letting key Māori stakeholders know this as soon as possible.
An updated 2018 Census Privacy Impact Assessment (PIA) has been published that updates the
public about the data linking and protections being used. There was a discussion that if the PIA
does not contain the ful picture, Stats NZ would be under increased scrutiny and exposure to risk.
There was a level of discomfort about the chain of consent from individuals through data
suppliers to Stats NZ. It was also noted that a discussion at a previous meeting had not created an
action point about what commitments Stats NZ has given to respondents and data suppliers (see
new AP4-6).
AP4-4: Gareth to add link to PIA to shared workspace.
AP4-5: Stats to document how multiple de-identified data points have been protected from
identification once they are assembled.
AP4-6: Stats NZ to document ‘What is the legal basis that it is OK to do what we are doing with
admin data and what are the ethical implications we have considered?’ ‘What commitment has
Stats NZ given to respondents and data suppliers about what we are doing with admin data?
What have they said? What are the key data sources?’
Kathy reviewed the draft critical path for the analysis and release of census data. Comments were
made about the sensitivity of electoral data and questions were raised about the technical papers
Stats NZ intended to publish.
The panel discussed the chal enges of how their report is drafted in terms of the level of technical
information required and the time commitments that wil require, both from the panel and Stats
NZ. The possibility of needing a dedicated writer was suggested. The need to consider the relative
Page 2 of 10
timing of the two external reports in relation to each other and the first release of data was also
discussed.
It was agreed that the panel report wil need to come out around the time of first release of 2018
Census data, as it wil be dependent on the need to refer to published data.
AP4-7: Gareth wil share draft list of technical papers (currently being reviewed internal y) with
the panel to see if this list wil also help the panel produce the independent report. The panel has
also created a list of documents that would be helpful, so these can be compared.
11am – Methods and data update
Christine reviewed briefly the Plan B methods. This session was shortened as the Chair needed to
ensure al items in the agenda for this meeting were covered at least partial y. After presenting
updated data, Kathy and Christine asked the panel ‘what does an acceptable level of under-
coverage look like’ and ‘what is an acceptable level of imputation’? Some members of the panel
felt unprepared to give an answer and others expressed that it ‘was going in the right direction’.
The panel discussed slides Christine presented on alternative sources of iwi information. Examples
included 2013 Census ‘codefile’ data and iwi organisation lists – both were seen to have data
quality issues.
The Chair noted that the iwi data did not look as if they could be improved by drawing on other
sources. At this stage the iwi data did not look fit for use. The panel advised that as it is, iwi data
should not be released as standard output, but could potential y available under certain
conditions to specific groups with caveats. It was acknowledged that this was a panel
recommendation but that it was a Stats NZ decision on whether and how this data would be
released. The panel further advised that Stats NZ consider national vs iwi specific approaches to
how we communicate this. The secretariat noted that Stats NZ wil come back to the panel with
proposals as to how they wil work with iwi on this issue.
Stats NZ participants pointed out that there is stil some work to be done on iwi data, but that
these are unlikely change too much in the coming months. Output options were discussed briefly
and Stats NZ wil come back to the panel with a detailed evaluation and output options paper
once more work had been done.
AP4-8: Gareth and Richard to consult over a when Stats NZ wil return to the panel with proposals
relating to the public release of iwi data.
12pm – Quality Assurance Framework and customer engagement
Gareth presented some of the slides (again truncated due to time constraints) relating to the
quality framework focusing on the principles of decision making. There were questions about
whether the principles had been ordered (not yet specifical y). It was noted that constitutional
and Treaty obligations should be at the top of the lists.
AP4-9: Gareth to update panel once more detail has been completed on the framework including
an ordering of decisions making criteria.
Mike Berry and Digby Carter joined the meeting as the leads of the fit for purpose engagement
team. The panel noted that the example slides did not make reference to Māori stakeholders.
Page 3 of 10
AP4-10: Gareth to upload the ful customer engagement spreadsheets to the workspace.
The panel discussed the publication delays and it was suggested that population estimates and
projections based on these estimates could serve the purposes of some data users. Many other
countries run 10 yearly (rather than 5 yearly) censuses and their key uses of population data need
to live with longer intercensal periods.
There were also concerns raised about a possible judicial review and that Stats NZ needs to be
wel prepared for chal enges stimulated by data quality issues. Mike and Digby left the meeting at
the end of this session.
12:30pm – Post Enumeration Survey (PES) update
The panel was joined by Brooke Theyers and Robert Didham via video conference from
Christchurch. Brooke gave an overview of the progress and issues being faced by the survey
project.
There was discussion about the fact that the 2018 PES would not be able to measure Census
dwel ing under and over coverage. It would stil be able to measure individual coverage. This was
also the case in 2013 but for different reasons. The panel expressed concern about the PES
missing the same dwel ings as the census.
Prior to lunch and the in-committee meeting, the chair summarised some points:
- Ian would be in New Zealand for the February and March meetings.
- The day before the planned 14 Feb panel meeting, the panel would meet with the
external reviewers. This would possibly be a half day and the rest of the day to be devoted
to preparation and planning for the panel report.
- The February meeting needs to address legal and ethical issues.
- More information is needed about the data quality of the variables. The panel suggested
they should be looking at each variable one at a time to be able to give advice.
- Information is needed about our census strategic decisions being made.
- More information is wanted about what users require and whether we can meet their
needs and whether we have talked to enough key users (eg. The Health Quality and Safety
Commission).
AP4-11: Kathy and Len to review whether the document created and loaded into the workspace
on legal issues goes into sufficient detail about Stats NZ‘s legal obligations.
1-2pm – In committee meeting
When the Stats NZ participants were cal ed back after the in-committee meeting, the Chair
summarised the discussions:
- For future meetings, much more time is required in advance for panel members to read
information prepared for each meeting
- The panel has not yet talked about the use of census as a periodic social survey (eg
questions of activity limitation) and that the census is used as a survey frame (eg Te
Kupenga) for other household surveys – and how these uses wil be affected by the data
quality issues.
- The panel would like more information about the quality of the administrative data that
sits behind the variables. This includes the list of possible data sources, the similarity of
Page 4 of 10
the concepts and the hierarchy of steps taken to improve the quality of data relating to
different variables.
AP4-12: Gareth to load a document to workspace that shows variable data sources, concepts and
hierarchy of steps taken to improve quality.
- As the panel is starting to put more pressure on Stats NZ for more information, is the
team able to resource the questions the panel has?
- The panel discussed having an in-committee meeting in late January or early Feb in
Wel ington, before the meeting on 14 February, to discuss the report.
- The need to confirm the April meeting (currently targeting 10/11 or 12 April).
AP4-13: Gareth to create a table to summarise variable measures.
AP4-14: Gareth to facilitate a Jan/early Feb meeting (without Stats people in attendance).
AP4-16: Richard to contact panel about a late Jan/early Feb in-committee meeting and inform the
secretariat.
AP4-17: Gareth to check with panel members about availability for the April meeting before
Christmas.
The next meetings dates are:
- late January/early February, Wel ington
- 13 and 14 February, Wel ington
- 6 March, Auckland
- 10, 11 or 12 April
- May and later dates to be confirmed.
Action log
Ref
Date
Description
Owner
Date
Progress
Date
raised
required
closed
AP 1-
29/8/18 Send independent ABS report
Gareth
Before
23/10 ABS report not yet sent
Close
2
to members.
Oct
given the information is
meeting sensitive. Gareth to work with
Kathy and Richard on how to
give access or present at future
meeting. Update at Nov meeting
9/11. Paper wil be added to
sharepoint folder once up and
running. To close before Dec
meeting.
6/12 Added to
workspace/Shared Documents
AP 1-
29/8/18
Give information on how
Gareth
Before
23/10 Adele has prepared some
Close
5
respondents were to meet
or at Oct information, to be discussed at
their legal obligations.
meeting the informal meeting on 30 Oct.
9/11 Short paper printed and
added to pack at meeting.
Additional paper SNZ presented
to conference to be added to
sharepoint.
6/12 Added to
workspace/Shared Documents
AP 1-
29/8/18
Confirm dates of next
Richard
At Oct
23/10 Nov, Dec, Feb and Mar
Close
7
meetings.
meeting dates confirmed. April date to
be confirmed. Leave open
Page 5 of 10
9/11 Feb and March dates wil
likely include additional
meetings to work on panel
report and meet with external
review team. Now standing
agenda item. Close
AP 1-
29/8/18
Col ate and send out
Gareth
Before
23/10 Summary not provided at
Close
8
information about languages
Oct
Oct meeting. Wil target Nov
(what was considered &
meeting meeting.
priorities).
9/11 Some information was
provided with the October
agenda. To be added to
Sharepoint folder before Dec
meeting. Additional information
wanted on how the languages
were selected.
6/12 Added to shared folder
AP 1-
29/8/18 Find out the response rate of
Gareth
At Oct
23/10 Ran out of time for PES
Close
9
the PES
meeting update. To be prioritised in next
2 meetings.
9/11 Ran out of time at 9 Nov
meeting, but wil be added to
later meeting date.
6/12 PES added to agenda for
Dec meeting
Meeting 2 actions
AP 2-
23/10/18 Revise minutes from the first
Richard
By 9
9/11 To be completed by 23
Open
0
meeting and present to for
and
Nov
Nov.
confirmation at the meeting on Gareth
9 November.
AP 2-
23/10/18 Work out way to give access to Kathy
By 9
9/11 DIA Sharepoint solution
Close
1
sensitive information outside
and
Nov
approved, but not yet
of the meeting room
Gareth
implemented. Target date 16
November.
6/12 Sharepoint solution
created and access given to
panel members
AP 2-
23/10/18 Create and distribute a conflict
Gareth
By 9
9/11 To be completed and
Open
2
of interest register
Nov
added to Sharepoint before Dec
meeting.
6/12 Added to
Sharepoint/Shared Documents –
7/12 Leave action open for
panel members to fil in prior to
14 Feb 2019 meeting
AP 2-
23/10/18 More information requested
Gareth
By Dec
9/11 Information presented that Open
4
about what investigations we
meeting include 2013 response rates and
are doing into 2013 counts and
counts. Gareth to check with
response rates – are we
Richard whether this should be
assuming that 2013 is correct?
closed by 23 Nov.
6/12 Gareth to discuss with
Richard
AP 2-
23/10/18 Send apartment strategies (and Gareth
By 9
9/11 To be completed and
Close
5
other targeted strategies) as
and
Nov
added to Sharepoint before Dec
background to next meeting.
Steph
meeting.
6/12 2018 Census Targeted area
slides added to workspace
AP 2-
23/10/18 Need more detail about the
Gareth
By 9
Sessions at 30 Oct meeting and
Open
6
totality and components of the and
Nov
priority at ful 9 Nov meeting.
imputation methods initial y
Richard
9/11 Information presented at
meeting. Gareth to check with
Page 6 of 10
planned and now being
Richard whether this should be
implemented.
closed by 23 Nov.
6/12 Gareth to discuss with
Richard
AP 2-
23/10/18 Request for data on birthplace. Gareth
By Dec
Topic for December meeting
Open
10
meeting 9/11 To be presented at Dec
meeting.
7/12 Data added to workspace
by 7 Dec
AP 2-
23/10/18 More information needed
Christine
By 9
9/11 Presentation on mitigation
Open
13
about IDI household matching
Nov
including imputations
including whether there wil be
completed at Nov meeting.
any more specific targeting of
Gareth to talk with Richard
areas with the threshold cut-
about whether this should stay
offs (have we considered using
open.
different cut-off rules for
6/12 Gareth to discuss with
different parts of the
Richard
population?).
AP 2-
23/10/18 Re electoral implications, more Kathy
By 9
Discussion planned for 9 Nov
Open
16
information is required about
and
Nov
meeting.
the representativeness of the
Gareth
9/11 No additional electoral
data and impacts of
data was presented (as not
imputations. A range of
complete yet). Updated draft
scenarios, including
electoral calculations to be
constitutional impacts for
presented to panel in Dec
Māori, would be useful
meeting.
6/12 Electoral data has not yet
been run – targeting Feb
meeting update.
AP 2-
23/10/18 Need more information about
Kathy
By Dec
Topic for December meeting.
Close
18
the Post Enumeration Survey
and
meeting 9/11 PES to be included in Dec
methods and outcomes under
Brooke
meeting agenda.
the different imputation
6/12 added to agenda
scenarios.
Meeting 3 actions
AP 3-
9/11/18
Each panel member to ensure
Panel
By 23
6/12 Panel members to update
Close
1
they have a functioning
members Nov
14/12 Al members have access
RealMe username.
AP 3-
9/11/18
Develop and distribute
Gareth
By 23
6/12 Brief guidelines provided
Close
2
guidelines when file sharing
Nov
via email
solution is in place
AP 3-
9/11/18
Agree and distribute future
Richard
By 23
6/12 Feb and March dates now
Close
3
meeting dates by 23 November and
Nov
finalised. Now standing agenda
Gareth
item.
AP 3-
9/11/18
Correct minutes from 23 Oct
Gareth
By 23
6/12 added, new version to be
Open
4
meeting adding a note to the
Nov
uploaded after Dec meeting.
sentence ‘that we could test
the model accuracy’ in addition
to col ecting more data from
respondents.
AP 3-
9/11/18
More information about what
Gareth
By 7 Dec 6/12 Did not have time for this
Open
5
admin data is used in the
item in the Dec agenda. Wil
method and for what purpose.
present information in Feb.
Privacy Impact Assessment
update. Add to agenda for
December.
AP 3-
9/11/18
Check the table stating the
Christine
By 7 Dec To be completed
Open
6
percentage of the IDI_ERP
Page 7 of 10
using different admin sources
as data does not look right.
AP 3-
9/11/18
More information needed
Christine
23 Nov
6/12 included in Dec
Open
7
about the sources and
presentation
component of admin data used
to help fil gaps in Māori
descent data.
AP 3-
9/11/18
For future data presentations,
Gareth
By 7 Dec 6/12 Noted
Close
8
request that any data created
to Territorial Authority level,
that we include Auckland Local
Boards.
AP 3-
9/11/18
More information needed
Gareth
By 21
6/12 Info not yet available, can
Open
9
about the sub-types of Non-
Dec
target Feb or Mar meeting
Private Dwel ings (NPDs).
Check when this detail wil be
available for the panel and add
to sharepoint folder when
ready.
AP 3-
9/11/18
Clarification is needed about
Christine
By 7 Dec 6/12 Leave open, not addressed
Open
10
what data is used in the census and
yet
administration enumeration
Kathy
(referred to as IDI data in the
presentation), and a list of the
current variables being
imputed, where it comes from,
whether the data suppliers are
aware of its use and whether it
uses identifiable or de-
identified person data.
AP 3-
9/11/18
Present criteria about the
Gareth
By 7 Dec 6/12 Some information in Dec
Open
11
thresholds being used to make
and
meeting present, more info to
decisions about methods.
Christine
come after meeting. Leave
open.
AP3-
Panel members to think further Kathy
By 7 Dec 6/12 Check with panel about
Open
12
about going back into the field
whether there was any more
and whether they had any
feedback.
further advice for us (as they
14/12 Panel unable to comment
didn’t think they had time to
on this yet with only a brief
appropriately digest and
discussion in November. The
respond at the meeting).
panel needs more information
from Stats NZ on what sorts of
options exist for col ecting
further data from the field.
Kathy to report back to February
or March 2019 meeting.
AP 3-
9/11/18
Agree priority of topics for Dec
Richard
By 23
Close
Close
13
meeting.
Nov
AP 3-
30/10/18 The Dwel ing Frame evaluation
Gareth
14 Feb
Open
14
report to be shared with the
2019
q/a panel
AP 3-
30/10/18 Provide information on births
Gareth
14 Feb
Open
15
and removals from the
Chris H
2019
Dwel ing frame during the live
operation
AP 3-
30/10/18 Produce an estimate of
Adele
14 Feb
Open
16
population at smal
2019
geographies by applying the
average household size from
2013 census to the number of
Page 8 of 10
dwel ings on the dwel ing
frame.
AP 3-
30/10/18 Produce a matrix showing the
Adele
14 Feb
Open
17
counts of people listed on the
2019
dwel ing questionnaire against
the number of individual
questionnaires
AP 3-
30/10/18 Confirm the source of local
Gareth
14 Feb
Open
18
estimates of those born
2019
overseas (e.g. in Auckland)
AP 3-
30/10/18 Provide the panel with
Gareth
14 Feb
13/12 Gareth: Suggest this is out Open
19
information on staff shortages
2019
of scope but in scope of the
at local levels of geography –
external review and to close
ideal y in a way that identifies
action
the impact on workloads
AP 3-
30/10/18 Report back on how might the
Gareth
14 Feb
13/12 Gareth: Suggest this is out Open
20
problems with the scanning
2019
of scope but in scope of the
field staff recruitment vendors
external review and to close
impact on response rates?
action
AP 3-
30/10/18 Share paper with the panel on
Gareth
14 Feb
Open
21
the imputation approach and
2019
the CPT recommendations on
which admin variables to use
for imputation. Ideal y also to
include a flow chart of the
process.
AP 3-
30/10/18 Elaboration of the options
Christine
14 Feb
Open
22
around imputation – with the
2019
pros and cons of each option
and the impacts on users spelt
out (maybe in a table with a
Red/Amber/Green status?)
Meeting 4 actions
AP 4-
7/12/18
Ensure al panel members have Gareth
By 21
13/12 Panel members now
Open
1
shared workspace write-access
Dec
given write access. Documents
and items are label ed
to be renamed by 21 Dec
appropriately.
AP 4-
7/12/18
Secretariat to ensure al
Gareth
By 21
13/12 Noted for future
Open
2
documents have useful
Dec
attachments. Al current docs to
headers and footers so once
be updated by Xmas
open it is clear what panel
members are reviewing.
AP 4-
7/12/18
Create and load document
Adele
By 14
18/12 Two documents (AP4-3a
Open
3
with Internet Col ection System
Feb
and AP4-3b) uploaded to
screen shots for panel
meeting workspace with screenshots.
members.
Wil leave open until Feb
meeting for panel let know
whether there is enough
information.
AP 4-
7/12/18
Add link to Privacy Impact
Gareth
By 21
13/12 Added to Shared folder
Close
4
Assessment to shared
Dec
workspace.
AP 4-
7/12/18
Document how multiple de-
Gareth/
By 14
Open
5
identified data points in admin
Adele
Feb
sources have been protected
meeting
from identification once they
are assembled.
AP 4-
7/12/18
Document ‘What is the legal
Christine
By 14
Open
6
basis that it is OK to do what
and
Feb
we are doing with admin data
Kathy
meeting
Page 9 of 10
and what are the ethical
implications we have
considered?’ ‘What
commitment has Stats NZ
given to respondents and data
suppliers about what we are
doing with admin data? What
have they said? What are the
key data sources?’
AP 4-
7/12/18
Share draft list of technical
Gareth
By 14
Open
7
reports with the panel to see if
Feb
this list wil also help the panel
meeting
produce the independent
report. The panel has also
created a list of documents
that would be helpful, so these
can be compared.
AP 4-
7/12/18
Consult over a when Stats NZ
Richard
By 14
Open
8
wil return to the panel with a
and
Feb
decision about public release
Gareth
meeting
of iwi data
AP 4-
7/12/18
Update panel once more detail Gareth
By 14
Open
9
has been completed on the
Feb
quality framework including an
meeting
ordering of decision making
criteria.
AP 4-
7/12/18
Upload the ful customer
Gareth
By 21
Open
10
engagement spreadsheets to
Dec
the workspace.
AP 4-
7/12/18
Discuss with Len after the Dec
Kathy
By 21
Open
11
meeting whether the
and Len
Dec
document ‘Information census
legislative obligations – DRAFT
5 Dec 2018’ loaded into the
workspace goes into sufficient
detail about Stats NZ legal
obligations.
AP 4-
7/12/18
Create a table to summarise
Gareth
By 14
Open
12
variable data sources, concepts
Feb
and hierarchy.
meeting
AP 4-
7/12/18
Create a table to summarise
Gareth
By 14
Open
13
variable quality measures.
Feb
meeting
AP 4-
7/12/18
Facilitate set-up of an in-
Gareth
By 21
13/12 Room in Wel ington Stats
Open
14
committee meeting (without
and
Dec
office and travel booked for
Stats people in attendance) in
Sophie
Auckland people coming. Food
late Jan or Feb 2019.
and room access remaining
actions.
AP 4-
7/12/18
Create glossary for Stats NZ
Gareth
By 14
13/12 Started and added to
Close
15
acronyms and load to
Feb
Shared folder
workspace
meeting
AP 4-
7/12/18
Contact panel about a late Jan
Richard
By 21
13/12 Chair has set date of 5
Close
16
or early Feb 2019 in-committee
Dec
Feb and agreed with panel
meeting and inform the
members
secretariat.
AP 4-
7/12/18
Check panel members
Gareth
By 21
Open
17
availability for April meeting.
Dec
Page 10 of 10