SMS Benefits Realisation review

Advisory and Assurance

Te Kura

Student Management System
Benefits Realisation Review

October 2013

1
Internal Audit
Confidential

link to page 3 link to page 3 SMS Benefits Realisation review


1  Background
Te Aho o Te Kura Pounamu – The Correspondence School (Te Kura) implemented the Student Management
System (SMS) as part of a wider Systems Replacement Project. Te Kura made the decision to replace the
Xtend/ãkona environment fol owing advice received from a number of independent reviews.  The
System Replacement Project was established to meet Te Kura’s need to manage student information (SMS),
deliver learning materials to students, store, retrieve and categorise digital learning resources.
SMS was progressively phased in from March 2012. As a result of concerns relating to the usability of SMS, the
Board and Management have  committed to addressing these  concerns and improving the  usability  of the
system.  The Board, management and staff worked together to initiate  an  SMS  enhancement programme
during July and September 2012.  The Te Kura Board approved the progression of the Action Plan items1 via
the Information Resource Group (IRG) to Adapt IT (SMS Vendor)  for completion by March 2013.  They
specifical y approved:
•  Five of ten Quick win enhancements that were presented.
•  One (the top priority) of ten Te Kura Enrolment System enhancements. The ten varied in importance and
difficulty to achieve.
There were also a number of core functionality items and enhancements2 which are being worked on as part
of the completion of the project and business as usual.
In December 2012, Internal Audit completed a Phase One review of the Action Plan items that found the
action items, which were raised as part of the SMS workshops, were col ated and appropriate plans had been
developed to address the items and these were being monitored, reported on and progressed.  Subsequent to
the review, the Action Plan was completed and the IRG established an ongoing SMS enhancement programme
to address further usability benefits identified.

2  Objective
The objective of this review (Phase Two) was to perform an assessment, against the intended benefits
identified in the Systems Replacement Project Business Case and ratified in the December 2009 Te Kura Board
meeting, of:
•  Benefits that have been realised.
•  Benefits stil  to be delivered.
•  Appropriateness of the arrangements in place to ensure the delivery of remaining benefits.
The Phase Two review did not consider the project costs or other aspects of the Systems Replacement Project
Business Case.


1 Action Plan items – the existing issues that staff most wanted addressed or felt would offer improvement in the future
2 Enhancements – application changes to Adapt IT (SMS Vendor)  which would add new functions or features to SMS
3
Internal Audit
Confidential

SMS Benefits Realisation review


3  Approach
In completing this review we have:
•  Reviewed and analysed the benefits as described in the Systems Replacement Project Business Case and
December 2009 Te Kura Board papers.
•  Interviewed key business owners of benefits to confirm our understanding of the benefits, underlying
assumptions, and benefits realisation actions undertaken by the organisation.
•  Interviewed representatives of key SMS users to assess the achievement of benefits.  The key user groups
included representatives of the Senior Leadership Team, regional and middle management, and help desk
staff.
•  Conducted three workshops with teachers and Student Support Advisors to gain an understanding of
benefits associated with the SMS implementation. One of the three workshops was dedicated to users
with no previous experience with Xtend/ãkona.
•  Attended walkthroughs of three scenarios, where there have been usability issues, with Help Desk support
staff (expert users), users with Xtend/ãkona (the previous SMS) experience, and users that have not used
Xtend/ãkona. The three scenarios included:
Scenario 1: Ordering work for a student.
Scenario 2: Changing a student’s assessment from Achieved to Not Achieved.
Scenario 3: Allocating a student to a different teacher.
•  Analysed financial information  and supporting documentation to validate  the  realisation  of  operational
savings.
The scope of this review excluded a detailed review of the SMS project costs and therefore is not providing an
opinion over the achievement or likely achievement of the expected net present value approved by the Te
Kura Board.

4  Key Findings
One of the key benefits from the Systems Replacement Programme Business Case related to the risk around
the ongoing stability of the previous Akona/Xtend environment.    The implementation of the new SMS and
OTLE has successful y removed this risk.
Tangible Benefits
The System Replacement Project Business Case sets out both tangible benefits and intangible benefits that
were confirmed by the Te Kura Board in December 2009.
The review team verified  the realisation of a number of tangible benefits that  al  related to IRG savings.
Cumulative savings verified between 2009 and the time of our review total ed $1.8M and Te Kura is continuing
to work on realising further benefits.  We were advised by management that the realisation of benefits was
impacted by delays experienced in the implementation of the project.
The IRG savings related to FTEs reduction and associated reduction in licensing costs required to support a less
complex environment.  Other “Operational Cost Savings” were not tracked by the project or the organisation,
nor was a baseline established for benefits realisation measurement.  We are therefore unable to assess the
achievement of these benefits.  Refer to the table below.
Further savings related to “Increased Revenue” were unable to be directly attributed to SMS implementation
due to the lack of a mature benefits realisation plan.  The fol owing table summarises the approved benefits
versus those realised and verified.
4
Internal Audit
Confidential

SMS Benefits Realisation review


Table 1: Tangible benefits approved by the Board vs. Benefits realised and verified
Tangible Benefits ($,000)
2009
2010
2011
2012
2013+

A
V
A
V
A
V
A
V
A
V
Operational Cost Savings

Budget reduction across IRG
250
250
250
250
250
250
250
250
250
250
IRG cost reduction
-
-
-
48
170
126
170
174
170
174
Cost savings as a result of
-
-
-
-
194
-
582
-
874
-
efficiency gains
Fewer manual rol over of re-
-
-
-
-
62
-
186
-
186
-
enrolling students
Improved inventory
-
-
-
-
20
-
20
-
20
-
management

250
250
250
298
696
376  1,208
424
1,500
424
Increased Revenue

Increased student enrolments
-
-
-
-
-
-
415
-
415
-
Increased student numbers
-
-
-
-
277
-
415
-
415
-

-
-
-
-
277
-
830
-
830
-
Total
250
250
250
298
973
376  2,038
424
2,330
424

Key:
A - Approved (Anticipated benefits approved by the Te Kura Board)
V - Verified Benefits realised and verified to appropriate supporting documentation during this review.

The difference between the Approved Benefits and the Verified Benefits is mostly Unverifiable, not Unrealised.

5
Internal Audit
Confidential

SMS Benefits Realisation review


Intangible Benefits
Management has seen  significant improvement in student management and related data quality.
Management has also seen an improvement in the accessibility of  management information that supports
more effective decision-making.  Furthermore, the new SMS represents a platform to enable Te Kura’s
strategic initiatives such as the digitisation of teaching and learning resources.  The users consulted during this
review perceive data quality as an issue. These concerns may be a result of understanding of the changes in
business rules or other technical reasons as management have not identified any specific quality issues.
The review finds a group of users (in particularly from the teacher and SSA roles) that continues to experience
usability issues in the new SMS.  We note that these issues have changed over time from not knowing how to
do basic, day-to-day tasks to how to do more complex tasks at the time of this review.  This group of users
expressed concerns about the appropriateness of the tools to do their jobs.  They also indicated an increase in
staff time to access data and reports, error checking, and in some instances duplication of effort as staff are
entering information into the system and then also capturing information offline in notes and spreadsheets.
Te Kura is working with the users and SMS vendor to address specific usability issues.  The matters raised by
users as part of this review were provided to IRG for further investigation.
The Governance Group is actively overseeing the SMS Improvement Programme – usability benefits are a key
part of this programme.    Overal , the consolidation of the student management environment and the
introduction of consistent processes and business rules have resulted in significant improvement in student
management related data quality, with an ongoing emphasis on enhancing usability benefits.
Other
Other findings noted from this review included:
1  The SMS improvement programme was general y seen as a positive step to address the usability issues.
However, users perceived that progress in this programme has slowed down and further usability benefits
can be achieved.   We understand that work is continuing in the SMS improvement programme and Te
Kura has put considerable effort into internal communications plan (including the intranet, weekly
communication emails, and representatives at team meetings).  We also acknowledge that two out of the
seven phases of usability benefits were approved by the SMS Governance Group and are being progressed.
2  Staff received a nine-day-training when SMS was first introduced.  Subsequently, such intensive training
was  replaced by training videos and tips and tricks sessions.    There are opportunities to review the
approach and training – both from an induction and ongoing perspective.
3  Te Kura currently does not have a mature benefits realisation plan  for SMS.    For future projects, this
should be considered.
4  Some of the business rules within SMS are not aligned to  SMS users understanding of their roles and
responsibilities.  For instance, IRG noted teacher al ocations are designated to be completed by the IRG
team.  However, during our system walkthroughs we noted that teachers are making changes to teacher
al ocations instead of sending requests to IRG.  This reinforces the need for effective change management
when implementing new programmes.

5  Conclusion
The consolidation of the SMS environment from seven separate systems to one has simplified and reduced the
cost of managing the  SMS IT infrastructure and operational environment. The consolidation in the
architecture has also reduced waiting  time in Enrolment and despatching (in the Student Resource Centre)
processes and resulted in improved data quality and reporting.  It has also avoided the risk of systems failure.
6
Internal Audit
Confidential

link to page 7 SMS Benefits Realisation review


The review of tangible benefits identified clear savings in IRG and SRC as a result of this  consolidation  and
supported the view of further benefits to be realised.
Usability and fit for purpose remain issues for  some users.  The review team explored these issues in
workshops with teachers and SSAs, and then assessed these views through walkthroughs, user guides analysis
and review of training videos. We find that certain teacher and SSA tasks are complex and involved several
steps which impacts on how some users perceive the usability of SMS.  Te Kura is working to address these
issues through an ongoing improvement programme.

6  Recommendations and Lessons Learned
6.1  Recommendations
1.  We recommend IRG continues with regular communications and key stakeholder engagement to
ensure key stakeholders remain informed of progress/status.  Review the SMS Governance Group
membership and Terms of Reference to ensure al  key stakeholders are represented and
responsibilities for representation of and communication to the wider SMS users group is clearly
understood.
Management comment: Agree. This was done at the conclusion of the implementation project
but wil  be reviewed.
Responsible position: Chief Operating Officer
Timeframe: March 2014

2.  In addition to the existing SMS Helpdesk, we recommend Te Kura continue to:
•  Encourage SMS users uptake of existing “Tips & Tricks Sessions” and short learning videos on
the intranet. Monitor and report on the effectiveness of the training tools.
•  Refresh the train-the-trainer model  to ensure integration with existing new employee’s
induction programme.
•  Identify SMS users who require additional support and implement an effective support
arrangement to develop usability among these users.
•  Address  any inconsistencies and/or issues  in the  comprehensive list3 of feedback from
Teaching Staff workshops conducted. (This was provided to IRG).
Management comment: Agree
Responsible position: Manager, IRG
Timeframe: Ongoing


3 A summarised version of the feedback from Teaching Staff Workshop is detailed in Appendix 2.
7
Internal Audit
Confidential

SMS Benefits Realisation review


3.  We recommend Te Kura develop a benefits realisation plan for those benefits that are stil  to be
realised, establish clear capability required to realised these benefits, create appropriate baselines
to facilitate their measurement of progress against these.
Management comment: Agree
Responsible position: Chief Operating Officer
Timeframe: March 2014

4.  Consider an adequate change management plan and align business rules with the roles and
responsibilities of SMS users for effective business process refinement.
Management comment: Agree, this wil  be considered in future
Responsible position: Senior Leadership Team
Timeframe: Ongoing

6.2  Lessons Learned
This review has identified lessons learned that Te Kura can bring forward for future projects.
1.  The definition of clear benefits in the Business Case supports the project in maintaining focus and
managing project delivery. However, the project would have benefitted from developing a Benefits
Realisation Plan that describes benefits, their dependencies, expected realisation timescales and
responsibility.
2.  The project actively engaged with the different stakeholder groups and worked to address al  users’
needs; however usability concerns remains an ongoing issue for certain users. The project would
have benefitted from more focus on usability requirements analysis and testing.
3.  The establishment of the ongoing Improvement Programme was widely supported and we would
recommend Te Kura adopt similar arrangements for future projects.

8
Internal Audit
Confidential

SMS Benefits Realisation review


7  Overal  Management Comment
We agree with the overal  findings and recommendations in the report.
However, we believe that two very key points need to be emphasised:
1.  Advice given to Te Kura in 2006-7 by independent consultants was that the existing bespoke systems
(ãkona and Xtend) needed to be replaced. By replacing those systems any risk of failure has been removed
and the single biggest benefit of the SMS project has been achieved.

2.  When the Board decided to the replace the ãkona and Xtend systems it instructed management to move
to a third-party “off the shelf” product rather than develop another bespoke system(s).  The benefit for Te
Kura would come from having a more robust system and no longer needing to commit significant
resources to development and maintenance. However, an implication of this was that Te Kura business
processes would need to change rather than undertaking any large-scale customisation of the new system.
Although a limited amount of customisation has been, and wil  be, completed it was always likely that
usability would be an issue. Management has worked hard to minimise the usability impacts for staff and
wil  continue to do so.

9
Internal Audit
Confidential

Document Outline