Caravel Group (NZ) Ltd
Contents
Interviewees ................................................................................................................................................. 3
Management Summary ................................................................................................................................... 4
Scope of this IQA Review ............................................................................................................................ 4
Statement of Accountability by Senior Responsible Owner ........................................................................ 4
Focus Area Ratings ..................................................................................................................................... 5
Delivery Confidence Rating ......................................................................................................................... 5
Summary of Recommendations .................................................................................................................. 6
Key Comments on Programme Health ........................................................................................................ 8
Electoral Commission Management Comment ......................................................................................... 11
Areas of Focus .............................................................................................................................................. 12
Business Benefits and Target Operating Model ........................................................................................ 12
Programme Organisation........................................................................................................................... 14
Programme Team ...................................................................................................................................... 17
Programme Methodologies........................................................................................................................ 19
Scope Management ................................................................................................................................... 21
Actions & Timescales ................................................................................................................................ 22
Budget & Cost Management ..................................................................................................................... 24
Risk and Issue Management ..................................................................................................................... 25
Quality Management .................................................................................................................................. 27
Communications ........................................................................................................................................ 28
Change Control .......................................................................................................................................... 30
Appendix 1 Comments from Interviewees (unattributed) ............................................................................. 31
Appendix 2 Programme Health Check Assessment Criteria ....................................................................... 32
Appendix 3 Programme Independent Quality Assurance Approach ............................................................ 36
Appendix 4 Description of Delivery Confidence Ratings and Recommendation Priority Ratings ............... 39
Caravel IQA Report GE2023 Programme
Commercial in Confidence
Page 2 of 40
March2023 Final version.docx
Caravel Group (NZ) Ltd
Management Summary
Scope of this IQA Review
The scope of this IQA review of the General Election 2023 Programme includes:
• Communications and Education
• Enrolment and Community Engagement
• Voting Services (including field)
• Technology
• People
• Programme Management
The scope excludes the Māori Electorate Option project.
The review approach is shown in Appendix 3.
Statement of Accountability by Senior Responsible Owner
I accept the findings of this Independent Quality Assurance Report.
Name:
Karl Le Quesne
Title:
Chief Executive and Chief Electoral Officer
Signature:
Date:
Note: IQA is a management and governance review to consider whether everything is in place for the
programme team and the Programme Board to plan, deliver, guide and control the programme. We use
interviews and document appraisal. It is not a forensic audit involving detailed examination and
confirmation of all activities or process compliance. If we find anomalies or gaps, we look for management
or governance causes and solutions so that specific instances can be used to improve the programme as a
whole.
Caravel IQA Report GE2023 Programme
Commercial in Confidence
Page 4 of 40
March2023 Final version.docx
Caravel Group (NZ) Ltd
Summary of Recommendations2
Business Benefits & Target Operating Model
1. The PMO Manager should work with DCE Strategy, Governance and Development to define
requirements for business cases and benefits realisation processes. (Medium Priority)
2. The Programme Director and the DCE Strategy, Governance and Development should finalise the
set of programme benefits metrics and include them in the Programme Management Plan.
(Medium Priority)
3. The Programme Director should consider extending the scope of the programme to include
measurement and reporting of achievement of the programme objectives and benefits. (Medium
Priority)
4. The Programme Director, in coordination with the ELT, should develop a benefits realisation plan
as part of the planning for the 2026 General Election. (High Priority)
5. A programme approach should be taken to the review and update of the Target Operating Model
after GE2023, and the planning and management of the prioritised projects. (High Priority)
Programme Organisation
6. The EC Board should document its expectations of the programme to ensure that the programme
fully aligns with EC governance. (Medium Priority)
7. The Programme Director should standardise the line reporting of the Project Managers for the
GE2026 cycle (Low Priority)
Programme Team
8. The SRO, Programme Director and DCE Operations should explore ways to retain programme,
project and regional/electorate staff throughout the electoral cycle. (High Priority)
9. The Programme Director and Manager PMO in coordination with the ELT, should define and
communicate the relative rights and roles of the Project Managers and the BAU Managers. (High
Priority)
Programme Methodologies
10. Project Managers should all complete their Project Management Plans and receive approval. (High
Priority)
11. The PMO Manager should set expectations of minimum documentation standards for programmes
and projects. (Medium Priority)
12. The Programme Director should restate the case for a process mapper as part of the programme to
deliver the 2026 election. (Medium Priority)
Actions and Timescales
13. The Programme Director should ensure that the Project Schedules are completed and baselined.
(High Priority)
14. The Programme Director should consider having a Project Scheduler to provide support and
mentoring for the Project Managers and Project Coordinators. (High Priority)
Budget and Cost Management
15. When the new FMIS is available, the Programme Director should arrange for the project budgets to
be tightened, while applying a controlled contingency system. (High Priority)
Continued on next page
2 Refer to Appendix 4 for a description of the Recommendation Priority Ratings
Caravel IQA Report GE2023 Programme
Commercial in Confidence
Page 6 of 40
March2023 Final version.docx
Caravel Group (NZ) Ltd
Risk and Issue Management
16. The Programme Director and Project Managers should conduct risk workshops with key
programme/project stakeholders every quarter. (High Priority)
17. The Programme Director and Project Managers should include risk mitigations in the Project
Schedules. (High Priority)
18. The Programme Director and PMO Manager should consider introducing an Issue Summary Form
to inform the governance groups about new issues and the resolution plans. (Low Priority)
Quality Management
19. The Project Managers should each produce a Quality Management Plan for their project. (Medium
Priority)
20. The Programme Director should formalise the process for gathering and prioritising Lessons
Learned during the programme, to reflect that it does not only happen at the end of the Programme.
(Medium Priority)
Communications
21. The Programme Director should ensure that the Communications project is working to approved
communications and stakeholder engagement plans and is reporting progress against those plans.
(High Priority)
22. The Programme Director should issue communications across Business Managers to clarify that
every Project Manager is able to report the true status of their project. (High Priority)
Change Control
23. The Programme Director should create a change control register and populate it with previous
approved changes. (Medium Priority)
Caravel IQA Report GE2023 Programme
Commercial in Confidence
Page 7 of 40
March2023 Final version.docx
Caravel Group (NZ) Ltd
Key Comments on Programme Health
Preface
We have drawn our conclusions from interviews, reading programme documents, and our experience
of programmes and projects. For many of our statements there wasn’t full consensus among those
interviewed. Also, there was some “talking the talk” but reportedly not fully “walking the talk”.
Most people interviewed were very complimentary about the programme and the benefit that it has
provided in planning and management of the election. The programme is well led and well governed.
Overall, there is full commitment to delivering a successful election and supporting the electoral system.
There is a lot of support for the programme management approach, and an understanding that early
planning and managed implementation can help to reduce the “crisis mode” operation that has
frequently been a feature of previous elections. However, there is still some contention with BAU about
the management mechanism and how decisions should be made. In some parts, the Project Managers
are seen as just a reporting mechanism - which misses the point of a lot of the value they can provide.
Governance
The Programme Board (PB) and Steering Group (SG) structure works well, with the SG looking down
at the projects and the PB dealing with strategy and more major issues. While some repetition of
material between the two groups is inevitable, they are presenting to different audiences.
The connection between the Steering Group and Programme Board is not completely clear, raising
some questions about information flow between the two, although judgement seems to be sensibly
applied.
The presence of an independent member provides a valuable addition to the Programme Board by
providing an alternative voice and perspective.
There were previously two Steering Groups. Joining them together was the right thing to do.
“Taking the reports and documents as read” is important to set expectations of the governance
members and to get through the agenda in a reasonable time. It needs to be regularly repeated as the
documents are not always read in advance.
Business Cases
There is no business case for the GE Programme – partly because it is a regular BAU process.
However, there is also no apparent policy about when a business case is applied. For example, the
new Enrolments system was approved with a Project Brief. This means that there is less of a focus on
benefits and benefits realisation than there should be.
Target Operating Model
A programme normally includes a Target Operating Model (TOM) as it supports a prioritised path
towards that future model with managed realisation of benefits along the way. We understand that a
TOM was created for the Electoral Commission and resulted in the organisation re-structure. After the
election the programme should work with the DCE Strategy, Governance and Development to manage
an update to the TOM, and the subsequent gap analysis, prioritisation, and initiation of the initial
projects.
Programme Organisation
The structure of the programme and projects is appropriate - with the projects matched to the key BAU
groups. This mostly works well but there is a need for further definition of the decision-making rights of
the Project Managers and the BAU managers.
Continued on next page
Caravel IQA Report GE2023 Programme
Commercial in Confidence
Page 8 of 40
March2023 Final version.docx
Caravel Group (NZ) Ltd
Programme Team
The programme is very well led and has a group of experienced and dedicated Project Managers, but
the team would be considerably enhanced by the addition of a couple of roles:
• Process mapper: Election processes are documented in an operations manual which needs
continuous improvement. The HQ processes are not documented. Documenting them would
provide the basis for consistent practice and innovation.
• Project scheduler: There is a considerable variation in the Project Schedules (Gantt Charts).
This causes problems and inefficiencies across the programme as there is no master schedule
or project baselines, and there is a manual approach to dependencies and an uncertain critical
path. It means that the programme can not readily do “what if” analyses e.g. ‘if this task is
delayed, what is the flow-on effect?’
The programme team frequently needs to work with subject matter experts (SMEs), but there are few of
them who have experience of previous elections and they have a lot of demands on their time. There
has been substantial staff turnover and strategies are needed to improve this. It is likely that some of
the turnover is due to the high workload and stress in the run-up to elections. Programme management
can reduce this if there is full understanding and acceptance.
Programme and Project methodologies
Methodologies are generally working for the programme, but not all are followed through. For example,
not all of the Project Management Plans have been completed, and some that are substantially
complete are not signed because of the difficulty in getting approval.
There is still considerable scope for tuning existing methods and templates and documenting
expectations as well as implementing new templates, including a Quality Management Plan.
Actions and Timelines
As mentioned earlier, considerable improvements could be made to the Project Schedules. Some of
this is due to a lack of BAU ownership of a planned path and timings, and the value of transparency.
The critical path chart operated at programme level is useful to set expectations about critical
milestones but is not integrated into the scheduling system.
Inter-dependencies are important. They receive a lot of attention through the programme team
meetings. However, there are still occasions where actions with inter-dependencies are changed
without reference to the other project. Also, they are not automated through the Project Schedules and
a master schedule so full “what if” checks cannot be quickly run. This is a significant gap in a fast-
moving programme where a decision in one area can have unanticipated effects in other areas.
Several of the Project Managers run a KanBan board with their BAU teams. This is a visual tool that
gives an overview of the current work status and simplifies team communication. It is effective for
agreeing and communicating forthcoming tasks, and to show what everyone is doing.
Reporting
Apart from the financial reporting, the project and programme reporting is suitable and reflects a fairly
accurate position. It addresses the key items.
However, there are attempts by some BAU managers to water down negative messages – even though
the SG and PB can be engaged to help in the troubled areas.
Financial reporting against the programme budget has been poorly supported by the old FMIS. It is
starting to improve with the incoming FMIS but reports are still not current.
Continued on next page
Caravel IQA Report GE2023 Programme
Commercial in Confidence
Page 9 of 40
March2023 Final version.docx
Caravel Group (NZ) Ltd
Risk Management
There is good attention to risk management. The major risks and their mitigations are regularly
reviewed. For some risks there appears to be too much optimism about how much the risks will be
reduced by the proposed mitigations. This could affect decisions about whether the mitigations are
sufficient or whether more needs to be done.
Over time, we would like to see the mitigation actions “mainstreamed” by inclusion in the Programme
and Project Schedules rather than being managed separately. Also, periodic checking of the more
minor risks is required to make sure their status hasn’t changed and that they are mitigated
appropriately.
Risk workshops at programme and project level should be held quarterly, preferably including regional
or electorate staff.
Information Technology
Critical technology platform upgrades (MIKE, EMS, ERSA) appear to be on track with end-to-end
testing and simulation programmes being implemented. Although not a focus of our review, there
appears to be suitable attention to the security of the systems. The relationship with the key IT supplier
Catalyst has improved recently and their contract has been extended.
A digital strategy development process is underway but there is no timeline for its completion.
Communication
There are several communications and stakeholder strategies and plans in various stages of
completion. Although not complete, these documents provide a fairly extensive view of the
communications and stakeholder engagement elements for GE 2023. However, they don’t provide
assurance that the deliverables are fully planned, resourced and on track to meet agreed dates. The
Project Schedule is incomplete, and we did not see an alternate tracking spreadsheet. We understand
that the communications team is very stretched for resource with roles that have not been able to be
filled.
Quality Management
There is no quality management framework for the Programme or Quality Management Plans for the
projects. This doesn’t mean that quality isn’t considered in the programme, but there is some lack of
formality about setting, approving, and meeting standards and acceptance criteria.
We would like to see a formal process for gathering lessons learned during the programme.
Siloes
We were asked to look at whether organisational siloes were adversely affecting the programme. We
did not see undue evidence of siloes during our review, although they no doubt exist. It seems that
previous silo effects have been reduced by:
• Having programme and project planning and implementation management
• Regular CEO zoom staff meetings
• The ELT supporting a cross-organisational culture
• Internal communications
• Visibility across groups in the Programme Board and Steering Group (and the associated
reports and documents)
• Programme and project team meetings
• Managing dependencies between groups/projects
Interviewee comments
We sometimes find it useful to collect and present unedited (and anonymous) comments that were
made by people during the interviews. These comments can help to provide a flavour of opinions that
get lost in formal reporting. We have included some interviewee comments in Appendix 1.
Caravel IQA Report GE2023 Programme
Commercial in Confidence
Page 10 of 40
March2023 Final version.docx
Caravel Group (NZ) Ltd
Electoral Commission Management Comment
This report is welcomed by the Commission. The main findings and recommendations align with the
Programme view and appear both appropriate and achievable.
The Commission will reprioritise the 23 recommendations, taking into consideration our operating
context and where we are at in the election cycle. We will ensure that those recommendations that
directly contribute to the success of the 2023 election are progressed first, and others that are good
programme management practice that will help us in the future will be progressed post-election.
The Programme Director will create a comprehensive Action Plan and progress towards closing actions
will be reported on monthly in the programme status report.
Caravel IQA Report GE2023 Programme
Commercial in Confidence
Page 11 of 40
March2023 Final version.docx
Caravel Group (NZ) Ltd
Appendix 1 Comments from Interviewees (unattributed)
The following comments were made in the interviews. They don’t all fully align with our overall review
conclusions, but they help to provide an informal flavour of some opinions.
• The 2020 election looks good on the results, but it was a mess internally.
• Emily is great at getting things done or fixed. This has helped a lot to reduce bottlenecks and
emergencies.
• My team is positive about programme management because it keeps other teams on track for work
towards the election.
• Need to have a programme that can deliver transformative change sitting alongside the electoral cycle.
• ELT should do more prioritisation instead of just listing things to be done. No-one says “What won’t we
do if we do this new thing?”. The General Election should be No.1 priority this year.
• Managers don’t want their teams’ actions put into a Gantt chart (or other plan). There is an attitude that
“if we put this in the plan and we don’t deliver, everyone will know that we failed”.
• The risk register is too complex for many BAU people. It may be better to keep the complete register
for the Project Managers but have a simplified view for the BAU teams.
• New people are working across the siloes, older ones not so much.
• The organisation is undergoing significant structural and systems changes – the bandwidth for quality
improvements is limited.
• There are plenty of opportunities for improvement, for example you can enrol for the election online
then a confirmation letter is posted to you (instead of by email).
• There is a new level of comfort that we are going to be given the things we need to do in the field.
• The Electoral Commission has a different definition of critical path i.e., everything they want to have for
the election.
• Project documents should have links to other documents as people have difficulty finding way through
SharePoint and Teams.
• “Everything is a priority”
• People provide dates but don’t have a full commitment to meeting them.
• People change their priorities without finding from the Project Manager the implications of the change.
• Lots of good ideas are floated at ELT, but nothing is removed to make room for them.
Interdependencies and precedents are not always understood or acknowledged.
• There needs to be more prioritisation of IT (and other) work, so that the focus is primarily on the things
that are need for the election.
Caravel IQA Report GE2023 Programme
Commercial in Confidence
Page 31 of 40
March2023 Final version.docx
Caravel Group (NZ) Ltd
Analysis. During and after the document review and interviews we consider the strengths and
4
weaknesses of the programme management and governance. Where potential issues are
uncovered, we will discuss them immediately with the appropriate people to ensure that we
have the facts correct, that we understand the context and the risks, and to provide the opportunity to
explore potential solutions.
The major areas we assess include:
▪ Clarity that the business benefits, objectives and deliverables of the programme is clearly understood
by all stakeholders, including members of governance and programme teams.
▪ Assurance that programme governance, advisory, and management structures are appropriate, fully
documented, and working effectively.
▪ That the programme team is appropriately resourced with the necessary skills, knowledge and
experience, and all roles are documented and understood.
▪ That appropriate programme methodologies are been applied and adhered to in accordance with the
organisation’s toolkit and templates, and review and approval processes.
▪ That the programme scope is clearly defined, documented and managed.
▪ That actions and timescales are planned, managed, communicated and reported.
▪ That budget and cost management principles are embedded in the management process, and all costs
are planned, managed, monitored and reported
▪ That programme risks and issues have been identified, analysed, documented and updated on a
regular basis, with appropriate risk and issue management plans in place.
▪ Quality management actions and planned and implemented.
▪ Communications are planned and applied to support the programme and to reduce risks.
▪ Change control systems are in place to manage any deviations from agreed tolerances for cost, time,
scope and quality.
Caravel has developed a unique health rating system to support our assessment and reporting process.
The ratings help to put our comments and recommendations into an objective context, and provide a basis
for the Delivery Confidence Rating. The ratings also provide a benchmark for subsequent reviews. The
areas that are rated include:
▪ Business Benefits
▪ Programme Organisation
▪ Programme Team
▪ Programme Methodologies3
▪ Scope Management
▪ Actions & Timescales
▪ Budget & Cost Management
▪ Risk and Issue Management
▪ Quality Management
▪ Communications
▪ Change Control
In addition, for go-live readiness reviews, we may assess and rate specific review items such as security,
testing, or change management. The definitions of the attributes that indicate rating scores are shown
below.
Where there are areas that require improvement, we provide prioritised recommendations on how these
improvements could be achieved.
Reporting. We prepare a draft IQA report which describes the programme or project
5
8
elements reviewed, our findings, and our recommendations. We write a report that is
concise and to the point. It includes an assessment of the Delivery Confidence
Rating. We provide the report to the Programme Manager (and others who may be affected by any
issues) to ensure that facts are correct.
Following any agreed modifications, we present the final report to the Programme SRO. In addition, it is
our preference to present the IQA report to the Governance Group, but this will be subject to the
agreement of the Sponsor.
3 Includes procurement methodologies
Caravel IQA Report GE2023 Programme
Commercial in Confidence
Page 37 of 40
March2023 Final version.docx
Caravel Group (NZ) Ltd
We will provide advice as requested on the programme or project assurance findings, options for
improvement, and the recommendations.
Caravel IQA Report GE2023 Programme
Commercial in Confidence
Page 38 of 40
March2023 Final version.docx
