From: [redacted - section 9(2)(a)]
Sent: 15/06/2017 8:36:05 a.m.
To: [redacted - section 9(2)(a)] [redacted - section 9(2)(a)] [redacted - section 9(2)(a)]
Subject: RE: Religious affiliation - 2017 Census test results and steps forward
I’ve got no concerns after the discussions yesterday. Challenge will be to get some
meaningful examples to replace what is currently on the paper form. And the write-in sits
above the no religion and object tick boxes. Multiple response would be helpful but I don’t
think there are significant impacts on the overall data with the constraints of the AYT and
ICS versus the paper. Key thing is that we should get much better quality data as respondents
can give more detail on what used to be the tick boxes which was a key criticism that came
through in the recent classification refresh.
[redacted - section 9(2)(a)]
[redacted - section 9(2)(a)]
[redacted - section 9(2)(a)
]
From: [redacted - section 9(2)(a)]
Sent: Thursday, 15 June 2017 7:31 AM
To: [redacted - section 9(2)(a)] [redacted - section 9(2)(a)] [redacted - section 9(2)(a)]
Subject: RE: Religious affiliation - 2017 Census test results and steps forward
Thank you [redacted - section 9(2)(a)]. Not having “as-you-type” options for responses
beyond the first is a pity but understand the problem very well – and the incidence of multiple
response is low enough for this not to be a problem (people committed to adding more than
one response will do so anyway). [redacted - section 9(2)(a)] will confirm if this is likely to
create any problems but as far as I can see it is the best solution in the circumstances. The
“no religion” and the required “object” options are important to retain (the latter is legally
required I believe).
From: [redacted - section 9(2)(a)]
Sent: Wednesday, 14 June 2017 3:38 PM
To: [redacted - section 9(2)(a)] [redacted - section 9(2)(a)] [redacted - section 9(2)(a)]
Subject: RE: Religious affiliation - 2017 Census test results and steps forward
Hi [redacted - section 9(2)(a)]
Thanks for your comments. We had a discussion about this topic at a form development
meeting this morning where [redacted - section 9(2)(a)]was present.
We discussed the option of a paper format closer to the 2013 version – QMD has drafted up a
version of this but it is ‘messy’ with very cramped routing.
We also discussed the option of a question with the write in box with the necessary additions
of ‘no religion’ and ‘object to answer’ response options.
It was decided that the best approach would be to proceed with the write-in as primary
response option, as ultimately this is the information that we want to collect from
respondents. Given that the 2013 format already has issues as you have identified, recreating
a less respondent friendly design of this seems unideal.
We also had to consider some of the constraints of building the online version, with regards
to multiple response for this question. All write-in boxes on the ICS with as-you-type
functionality currently have the option for the respondent to write more text after they have
chosen a response from the drop down suggestions. It was indicated that to build the
functionality to ‘add another religion’ as is available for the iwi question would be very
costly to build. The current default functionality as described would allow multiple response
on the online form, if the respondent wished to do so however as-you-type functionality
would not be available after they had selected their first response.
If you are comfortable with this approach, I will then discuss this with Gareth Meech and
Denise McGregor before QMD and Respondent Interaction begin developing the revised
question format.
Cheers,
[redacted - section 9(2)(a)]
From: [redacted - section 9(2)(a)]
Sent: Wednesday, 14 June 2017 10:44 AM
To: [redacted - section 9(2)(a)] [redacted - section 9(2)(a)] [redacted - section 9(2)(a)]
Subject: RE: Religious affiliation - 2017 Census test results and steps forward
Hi [redacted - section 9(2)(a)]
This is a seriously difficult question, thanks to the paper-size – and it is a pity the pair of
questions failed. Census is an on-line first survey, so the paper form is secondary, but if we
get a large number of paper forms then we need to understand how the two modes perform
differently. The key points have been made in the email trail below – the purpose in
collecting this information is not satisfied by the questions in the test form. I would also
question how you know the data in the first question was of high quality: it is notoriously
difficult to establish data quality in a box-ticking exercise and I would suggest that it is not
good quality because the routing clearly did not work (your call backs will almost always
evoke an “expected” response because people like to appear compliant face-to-face [voice-to-
voice].
The problem is that the level of information on religious affiliation elicited by Q16 is not
what is required since at least two of the tickboxes (Christianity and Other) aggregate many
dozens of very distinct religions. I do agree that the prompts in Q17 should have provided
enough hint of what we need but the damage is already done by the tickbox question
above. Perhaps the wording “more detail” is not helping because I can imagine there would
be a tendency to wonder ”what more detail? Do you want an essay?”. Having ticked
“Christianity” I would imagine that many Presbyterians (as an example) would simply react
that that is not more detail but a different question altogether. It is also more likely to produce
an address rather than affiliation.
Definitely comparability is an extremely important consideration but not the only one. We
are living in a rapidly diversifying society and census is one of the extremely few
opportunities we have to be able to get some measure of this – so we not only need
comparable data but we need contemporarily useful information that we can derive from that
data. Unfortunately q16 does not come close to doing this on its own, and the paired
questions have failed.
The online would be no problem – maybe just use the 2013 question, corrected of course to
use nouns as the Q16 below has. The problem is the paper form and I am wondering whether
one workable solution might be to reformat the 2013 question so that the call-out boxes
become a subset below the headings – but this would lengthen the question significantly to
the detriment of neighbouring questions (especially the 1yr ago question that we discussed
the other day and which is crucially important to get right with a write-in for country). My
main question then is: does the length required to do this exceed the total length used by Q16
and Q17 together?
Given the space available, the potential for drop down categories etc on-line, the best solution
would perhaps be a write-in as Andrew suggested. There will still be the old problems on a
paper form of people who give their religion as “St Stephens” and we can only guess which
temple they mean from their geographic locality (especially where the same names are used
in multiple religions or the same building doubles service as Anglican, Catholic, Muslim
etc). While it will be flying blind in terms of testing I am sure that QMD have experience
with this type of question and know the most effective format for write-in. One thing to
watch for though would be allowing enough characters to adequately capture a descriptor
since we would need to collect up to, ideally, 4 religions (given the current low rate of
multiple response I wonder whether we could drop this to 3? Thoughts, [redacted - section
9(2)(a)]?).
[redacted - section 9(2)(a)]
From: [redacted - section 9(2)(a)]
Sent: Wednesday, 14 June 2017 10:00 AM
To: [redacted - section 9(2)(a)] [redacted - section 9(2)(a)] [redacted - section 9(2)(a)]
Subject: Religious affiliation - 2017 Census test results and steps forward
Hi all,
The 2017 Census Test data has been finalised and the data for the religious affiliation quested
indicates that the question performed poorly in being able to produce comparable data to
previous collections of this concept in the census.
While the data quality collected from question 16 – the first component was of high quality,
the response rate to question 17 for those who stated a religion in question 16 is very low at
33.9%. This is consistent across denominations, for example Anglican responses making up
4.1% of the total respondent population compared to a prevalence of 11.8% in 2013.
We also called back some respondents (6) who did not respond to question 17 to get more
information on why they did not respond. The overall feedback that they had already
responded and were happy with the response they had given, and did not feel the need to
respond to the further optional question.
If we proceeded with this question format, we would be unable to produce comparable data
for any category other than those collected in question 16.
There are obviously space constraints in revising the question to a preferable format.
I have had a think about this issue might be resolved and the factors that need to be
considered. I have copied in our previous discussion on this question design change prior to
the 2017 test.
- The ‘soft’ routing to question 17 is not working successfully – a revised question
needs to have direct routing to the write in box
- If we were to have a combination of tick boxes and write in box, given the limited
space this would likely need to have a mix of concepts collected in the tick boxes
- The other option is to include a write-in only as [redacted - section 9(2)(a)] had
previously mentioned.
It would be good to have a discussion about how we might best proceed with this question in
order to output quality information on the concept. One major constraint is that we don’t have
another major test in order to test any revised format. There will be usability testing
undertaken by QMD however. Happy to hear your thoughts.
Cheers,
[redacted - section 9(2)(a)]
Thanks for the comments.
[redacted - section 9(2)(a)] was not too keen on the Christian tickboxes given the
mixing of levels being collected. We did also discuss a pure write in question which
I agreed would work best in theory – especially if we are asking all religious
respondents to write in detail anyway.
I will forward on the suggestions of the plural wording and the size of the write in
text box – currently the column in which the religion question sits on our marked
up proposed form is very vertically full but until the sign off of topics for the March
2017 Census Test is done, we cannot be certain of the full space constraints!
Cheers,
[redacted - section 9(2)(a)]
From: [redacted - section 9(2)(a)]
Sent: Wednesday, 28 September 2016 8:31 AM
To: [redacted - section 9(2)(a)]
Cc: [redacted - section 9(2)(a)]
Subject: RE: Religion question - Census 3 column IF format
Hi [redacted - section 9(2)(a)]
Is looking much better. Personally I’d be happy with just a write-in box alone – it
isn’t that hard a variable to code, we’ve reduced the codefile by half and
improved the match rate and quality (using the proposed new version that we have
created as part of the current refresh of the classification.) But I would go with
this new design if at all possible.
I would note that the key selected stakeholders that we have consulted during the
current refresh have all made some comment about the inability to collect
multiple responses, and to collect sub-groupings (for want of another word) of
many groupings such as Judaism, Islam, Buddhism as we lose the information by
encouraging the tick-boxes. Religion is not about collecting the information to
create a ranking system of the most popular, it is a measure of our cultural
diversity and social-connectedness which is where a lot of the research is wanting
the better detail. I can only do so much to improve the classification (which I have
at this point) but it sort of becomes pointless if the question doesn’t allow for
some better coverage.
Whatever you do, there will be criticisms – the agnostic/atheist groups will want a
tick-box, the jedis as well, and then there is the whole dump coding to Christianity
which [redacted - section 9(2)(a)] talks about (and which I agree with). Having
Christianity loses the information we want about the emerging and non-traditional
groupings. Nobody really cares about how many Catholics and Anglicans there are
(other than those two groups). So that’s my two cents as Senior Researcher for
S&D and as the person doing the review of Religious Affiliation at the moment..
From a C&S perspective, the proposed change is still in keeping with the spirit and
intent of the Statistical Standard so I can’t see that there are any issues for making
a change,
Cheers
[redacted - section 9(2)(a)]
[redacted - section 9(2)(a)]|
Senior Researcher
Statistics New Zealand - Tatauranga Aotearoa | BNZ Centre, 120 Hereford St, Private Bag 4741,
CHRISTCHURCH 8011
Email: [redacted - section 9(2)(a)]| DDI: [redacted - section 9(2)(a)]
| www.stats.govt.nz
From: [redacted - section 9(2)(a)]
Sent: Wednesday, 28 September 2016 7:35 AM
To: [redacted - section 9(2)(a)] [redacted - section 9(2)(a)]
Subject: RE: Religion question - Census 3 column IF format
Thanks [redacted - section 9(2)(a)]. I think this is a much improved version and is in
fact a better question format than 2013 – the primary thing I would suggest here is
to reinforce that this is a multiple response question so we must provide the ability
for people to provide more than 1 (we do collect up to 4), so change “religion” to
“religion(s)” and add a couple of rows.
I like your options since you have been careful to include a variety of Islam and
two other religions.
However, what really worries me with this format is that people will be likely to
take the easy option of not giving text responses and if they do there is both a
processing overhead and loss of key information. This is why I would have much
preferred to see some tickboxes other than (preferably instead of) “Christianity”
since people are not interested in the high level grouping category but in counts
and characteristics of people who are Catholic, Anglican, etc etc. This would not
actually save space, because we would still have to have write in options for other
religions, but it would simplify processing.
Please keep me in the loop with this one.
Cheers
[redacted - section 9(2)(a)]
From: [redacted - section 9(2)(a)]
Sent: Tuesday, 27 September 2016 3:57 PM
To: [redacted - section 9(2)(a)] [redacted - section 9(2)(a)]
Subject: Religion question - Census 3 column IF format
Hi [redacted - section 9(2)(a)]and [redacted - section 9(2)(a)]
I am aware that [redacted - section 9(2)(a)]has already contacted you about some of the form
changes we are considering for the 2018 Census individual paper form design as a result of
the decision to use a folded A3 paper size.
[redacted - section 9(2)(a)] and myself have been scoping out some of these changes in order
to fit the recommended content into a form this size. In general this has been quite successful
– but an issue with the religious affiliation question has arisen due to reliance on horizontal
spacing for the 2013 format of this question.
If a three column format is used for the 2018 Census individual paper form, the format for
this question must be altered.
While vertical space is not a huge constraint for design of this question in the first instance,
reactions from the Census SLT and Carol Slappendel were negative towards a second
separated question on Christian denominations which would most closely replicate the 2013
format.
[redacted - section 9(2)(a)] and myself had an initial attempt at thinking about how to best
design a question given this constraint in mind, this being below.
Feedback on this approach would be appreciated - with regards to retaining quality and
comparable data for this question within the constraints identified above. [redacted - section
9(2)(a)] – [redacted - section 9(2)(a)] had already passed myself your comment about
collecting the primary Christian denominations as tickboxes within the primary question, let
me know if you think that approach would be most appropriate.
Cheers,
[redacted - section 9(2)(a)]
[redacted - section 9(2)(a)]
| Statistical Analyst/Kaitätari Tatauranga | 2018 Census
Customer Needs and Data
Statistics New Zealand - Tautauranga Aotearoa
Phone: +[redacted - section 9(2)(a)]| Statistics House, The Boulevard, Harbour Quays,
PO Box 2922, WELLINGTON 614
[redacted - section 9(2)(a)]
