Systems Replacement Project
RFP# 22871: Student Management System
Evaluation Team Short-Listing
Recommendations
Prepared by:
Project Administrator
Approved by
Chair SMS Evaluation Panel
Prepared for:
Chair Business Evaluation Team
Process:
To record the process and
Evaluation Team recommendations
Date:
4 November 2008
Version:
Version 1.4
Status:
Final
The approximate value of the LMS tender is
.
5. RFP Responses
Responses to the RFP were due by 4pm 15 September 2008 and were opened by the
Systems Replacement Project Manager in the presence of a Probity Auditor from
Ernst and Young
on 18 September 2008. Two hard copies of
responses were received; one from Integrated Tertiary Software and the other from
, which was not compliant with the required RFP process.
6. Management of Non Compliant Response
After the completion of the inventory of responses and prior to the commencement
of the evaluation process a decision was made, by the Chief Executive, that non-
compliant responses to the three RFPs would only be considered if a determination
was made by the Evaluation Teams that the compliant responses had failed to
adequately meet the evaluation criteria.
7. Action Register
Date
Action
12 August 2008
RFP released on GETS
15 September
Closing date for proposal submissions
18 September
Responses opened by Systems Replacement Project Manager, in the
presence of the Probity Auditor from Ernst and Young.
18 September
Inventory of responses prepared
Week
Précis of responses prepared and distributed to Evaluation Team
commencing 22
Members along with copies of the evaluation sheets, Evaluation
September
Guidelines and copies of responses relevant to the SMS Evaluation
Team.
23 September to Members of the SMS Evaluation Team individual y score and rank
16 October
responses.
17 October
Meeting of SMS Evaluation Team meet to discuss individual scores
and rankings, to determine aggregate scores and rankings and to
prepare a shortlist of vendors for presentation/proof of concept
before the TCS Selection Panel.
21 October
Minutes of 17 October meeting circulated amongst members of the
Evaluation Team.
30 October
Second meeting of SMS Evaluation Team.
Confidential
26/05/2009
Page 3 of 18
30 October
Revised score sheet correcting the aggregate numerator circulated to
members of the Project Team for review.
31 October
Draft Minutes circulated for comment.
8. Conflict of Interest
Conflict of Interest Declarations were completed by all members of the SMS
Functional Evaluation Team, the Systems Replacement Project Manager and the
Project Administrator prior to the Evaluation Team meeting on 17 October 2008.
Prior to the commencement of the meeting of 30 October a probity check list was
provided to all members. The self check probity check list asked members whether
they had been contacted by any of the vendors; whether they had discussed the
responses with anyone other than Team Members and whether as a result of their
review of the responses they had identified any conflicts of interest. Any positive
responses to these questions were to be reported to the Chair. No declarations
were made.
9. Evaluation Process
The evaluation process was conducted in accordance with ‘
Systems Replacement
Project Stage 2 Processes: Evaluation Guidelines, ‘
Systems Replacement Project
Stage 2 Processes: Procurement Plan’ and the RFP rules relating to non compliant
responses which are set out on page 15 in Section 4.2.2 of the RFP document. This
section states that ‘
any non-conforming proposal, including, without limit, any late
proposal, may be excluded from or accepted for consideration by TCS at its sole
discretion’.
At the evaluation meeting of 17 October the responses from
and
Integrated Tertiary Solutions Ltd (I.T.S.) were discussed. As some members of the
Evaluation Team had not had the opportunity of reviewing the responses and
completing their individual assessments it was not possible to complete the
assessment process at this meeting. The SMS Evaluation Team did however agree
that in view of the non compliant response from
and the approaching end
of year that there was sufficient information in the response from Integrated Tertiary
Solutions Ltd to recommend to the Business Evaluation Team that an immediate
invitation to I.T.S. to present their proposal to the Selection Team be issued.
It was further agreed that it would be necessary to schedule a further meeting of the
Functional Evaluation Team before this presentation to agree aggregate scores. This
meeting was held on 30 October 2008.
Confidential
26/05/2009
Page 4 of 18
Subsequent to the meeting of 17 October probity advice was received from Ernst
and Young that an assessment process should be completed prior to the issue of the
invitation to Integrated Tertiary Solutions Ltd to determine whether the response
was of sufficient merit to warrant an invitation to present their proposal and proof of
concept.
10. Summary
Attached to this report are the Minutes of the 17 October meeting, Minutes of the
30 October meeting, a list of issues/questions to be put to I.T.S. at the presentation,
a copy of the probity self check list, and a copy of the aggregate score sheets.
The evaluation process requires that the common and specific criterions be assessed
against a maximum score, which ranges between 10 and 95. The individual ratings
for each criterion would then be aggregated and summed to determine the
aggregate score for each response. One of the ten score sheet responses included a
qualitative assessment (based on an earlier draft of the Evaluation Guidelines)
against the ‘specific’ criterion rather than a numeric assessment; this response was
translated in to a numeric value to enable a uniform methodology to be applied. The
translation process was consistent with the translation process used for aggregation
of the similarly assessed scores for the OTLE RFP.
The individual scores were aggregated and from a possible score of 630 the proposal
from I.T.S. Ltd was assessed as 416.6 out of a maximum of 630.
and
felt that they did not have sufficient technical knowledge to
evaluate the technical and general user requirements. As the maximum possible
score for these two categories is 100 the non-scoring has the effect of distorting the
overall aggregate score. If the scores for these two members are not included in the
aggregate assessment the modified score would be 482.9 out of 630.
Recommendations
The SMS Functional Evaluation Team recommends to the Business Evaluation Team
that it
1. Agrees that Integrated Tertiary Systems Ltd be invited to present their
proposal to TCS selection staff and that the issues/further questions
identified by the Evaluation Team be put to the vendors by the Selection
Staff;
2. Notes the comments set out in the Minutes of the Meetings of 17 and 30
October 2008;
3. Notes that Integrated Tertiary Systems received an aggregate score of 416.6
or of an adjusted 482.9 out of a possible maximum score of 630; and
Confidential
26/05/2009
Page 5 of 18
4. Agrees that a letter be prepared to
advising that their response was
not compliant and in terms of the RFP rules was not considered by the SMS
Evaluation Team.
Name:
Position: Chair SMS Evaluation Team
Date: 2 November 2008
Confidential
26/05/2009
Page 6 of 18
Notes from SMS Functional Evaluation Team Meeting
Date: 17 October 2008
Time: 10:00 – 11:00
Venue: Tawa Room, Hobson Street
Members Present:
(Chair),
,
,
,
,
,
,
,
,
,
,
A
pologies:
,
Absent:
Observers:
,
,
(Probity Advisor, Ernst and
Young)
Notes
1. There are two responses to the RFP, one from ITS which is compliant with the
tender requirements and the other from
which is not compliant. The
objective of the meeting was to decide whether to invite I.T.S. to provide a
presentation/proof of concept and whether to exclude
.
2. Noted that one member had not read the responses and that other members
felt that more time was necessary to read and digest the information before
assigning scores.
3. At the request of a Panel Member the term “open source” used in the
response was explained.
4. Noted that some members felt that it was hard to numerical y rate the
responses without observing the software in operation.
5. Noted that TCS had looked at I.T.S’s product previously (2007) but had not
proceeded to purchase. Members of the previous Evaluation Panel were not
advised as to why, leading to concern that the present evaluation might be
destined for a similar fate. It is believed that the earlier procurement was
halted over process concerns, rather than any identified short-coming of the
product itself.
6. Noted that the evaluators should focus on system functionality not technical
aspects of the vendor’s response.
7. Noted concerns around the provision of ongoing support from I.T.S. given that
they are based in South Africa.
8. Noted that
has been identified
by the vendors as a reference site. There is a protocol to be observed, but it
Confidential
26/05/2009
Page 7 of 18
should be possible to pose such questions as the last (concerning vendor
responsiveness) to them. Additionally, the Evaluation Team could expect to be
offered a demonstration of the
system later in the process, assuming it
proceeds that far.
9. Noted that the response from I.T.S. provides no quantification of ‘speed of
use’. The Project Manager noted that this was not uncommon in responses
from vendors because of the number of variables. Noted that information on
operational performance can be gauged from that available at comparable
reference sites (having a similar size and IT infrastructure). Noted also that
performance cannot be permitted to erode over time (e.g. under increased
load).
10. Noted the view that I.T.S. does not seem to understand the range of functions
of TCS, especially with respect to measures of assessment and the transfer of
data with NZQA. (References to assessment in the proposal are largely framed
in the paradigms and vocabulary of the tertiary sector.) Some concern whether
they could make the necessary adaptations.
11. Noted the need to ensure that all required functionality is explicitly identified
and addressed. Whereas some functions may be forthcoming from the other
system being considered (the OTLE), this cannot be taken for granted.
12. Noted concern about the quality of some data to be migrated to the new
system and the need to give some thought about the transfer of this data.
Experience with the implementation of ākona was referenced as an example
where the conversion process went wel but the quality of the imported data,
particular that concerned with rubrics, was poor. Noted that this issue exists
irrespective of any choice of SMS.
13. Noted that I.T.S. may not understand the difference between the types of
students enrolled with TCS. Noted issues around their proposals for the
allocation of incoming work which does not mention work coming in
electronically or the need to refer all work to the responsible teacher. Noted
that this was an area identified by ITS in 2007 as requiring particular attention.
14. Noted concern that technical specifications for end-user software could cause
difficulties for students and external school staff whose technology may be
older than that at TCS (e.g. the requirement to support the SSL security
protocol). Noted that the requirements for external users appear to be
relatively low (e.g. Internet Explorer v4.0) and that IRG can provide expert
advice to the panel.
15. Noted the need to clarify whether the ITS system can deal with students
working on multiple levels over multiple years and the need to avoid
reentering enrollment data each year.
16. Noted the need to clarify how easily the ITS system can be adapted to cope
with future changes.
Confidential
26/05/2009
Page 8 of 18
17. Noted the need to clarify the interface with Navision.
Where to from here?
18. Agreed, in view of the approaching end of year hiatus to invite I.T.S. to present
their proposal to the Selection Team.
19. Agreed that it would be necessary to schedule a further meeting of the
Functional Evaluation Team before the I.T.S. presentation to agree aggregate
scores and decide whether or not the
proposal should be advanced.
20. Agreed that further questions for the next meeting be referred to
for consolidation.
21. Agreed that the Minutes be circulated.
Confidential
26/05/2009
Page 9 of 18
Notes from SMS Functional Evaluation Team Meeting
Date: 30 October 2008
Time: 3.00 to 4.00
Venue: Tawa Room, Hobson Street
Members Present:
Chair),
,
,
,
,
,
A
pologies:
,
,
,
Absent:
,
,
Observers:
,
,
(Probity Advisor, Ernst and
Young)
Prior to the commencement of the meeting a probity check list was distributed
which asked attendees to disclose whether they had been contacted by any of the
vendors, whether they had discussed their individual assessments with anyone other
than a fellow Panel Member, and whether as a result of the review of the responses
they had become aware of any conflicts of interest. Any positive answers were to be
referred to the Chair of the Evaluation Team. No declarations were made.
Copies of the Minutes of the 17 October meeting and a copy of the aggregate score
sheet was tabled. The score sheet shows an aggregate score of 417 out of a possible
of 630. It was noted that there was some distortion to the scoring as two members
of the Evaluation Team had been unable to assess two technical criteria categories
which had a combined possible maximum score of 100. If these scores were omitted
the aggregate score increases from 417 to 483 out of a possible 630.
The Chair summed up the purpose of the meeting, which was to discuss and agree
the aggregate scoring and to make a decision on whether ITS should be invited to
present their proposal and demonstrate their product to a TCS Selection Team.
There was a general discussion on whether the Evaluation Team felt comfortable
with the aggregate score as a true reflection of the earlier discussion which focused
on qualitative factors and whether the Team had confidence in ITS, and if not should
TCS go back to the market to identify other vendors and other solutions. It was:
Noted that there was some variation between individuals scores and that there
were some issues around the aggregate numerator, especially in those technical
criteria where some members of the Evaluation Team had entered a zero score
to reflect that they felt unable to assess that particular criteria.
Confidential
26/05/2009
Page 10 of 18
Agreed to review the aggregate spreadsheet so that the aggregates for each
category of criteria reflected the number of scores.
Agreed that TCS should first invite ITS to present their proposal.
Agreed that the Chair will review the information sent to the SMS RFP
respondents to identify any gaps and will arrange to provide any necessary
additional or clarifying information.
Noted that while ITS has previously responded to TCS tenders and already has
some knowledge of the School and its operations, there is still concern to ensure
that they have sufficient information about TCS business processes and
methodologies to present properly.
Agreed that, in view of the above, that ITS should have access to Evaluation
Team Members and other appropriate TCS staff prior to the presentation in
order to fully inform themselves of the TCS ways of working and special
requirements.
Agreed that TCS staff prepare a range of scenarios which reflect the wide range
of business undertaken by the School, that these scenarios be referred to
by 14 November 2008, for collating.
Noted that
is available to assist Evaluation Team members prepare
the scenarios.
Noted that questions, issues and scenarios will be referred to ITS prior to the
presentation, and that ITS be offered detailed briefing on matters raised by the
Evaluation Team, so that their presentation can be more fully informed.
A number of such areas were identified in the meeting, and will be included.
They included:
1. assessments
2. allocation of work to teaching staff
3. conduct of the project alongside BAU, and how cut-over of the new
system will be managed
4. clarification of the meaning of certain risks identified in the response
document, particularly that relating to the School’s ‘culture’
Agreed Noted that
and
will brief members of the
Evaluation Team who were unable to attend the meeting on the outcomes and
the need to prepare scenarios.
Confidential
26/05/2009
Page 11 of 18
Agreed that ITS be asked to provide historic issues lists for one or two
comparable clients, (a) as they stood immediately upon going live with the
application, and (b) following a reasonable period (in the order of a year), to
help determine the effectiveness of the System, its implementation and whether
any additional unforeseen capital investment was required to remedy the issues.
Noted that the ideal timeline would be to have all issues clarified by ITS and ITS
having presented before the Christmas break, to enable the commencement of
work as early as possible in 2009.
Questions for I.T.S. at Presentation
1. Is data (like a teacher in-putting marks and ordering work) transferred in real
time and permanently stored or is it batch transferred later? Not doing it in real
time has the potential to lose data when something goes wrong and teachers are
left not knowing whether something has actually been ordered or not.
2. Have any of the ITS systems been hacked into?
3. Can names of actual students be found from a ‘sounds like’ search function?
This is useful when trying to decipher student or supervisor messages from an
answer phone!
4. Can cohorts be automatically updated if that is a chosen parameter? E.g. Can
a cohort of SC150 Ful time Maori students be kept as a cohort if those criteria are
used to create the cohort in the first place or is it only a manual cohort like ākona?
5. What are the organisational culture risks referred to in the RFP response and
how do they see these impacting on the roll out of the new system?
Confidential
26/05/2009
Page 12 of 18
Probity Checklist (Self-Check)
Action
Yes
No
Have you read all of the vendor’s responses to the RFP?
(If no, please advise the Chair of the Evaluation Panel)
Have you read The Evaluation Guidelines?
(If no, please advise the Chair of the Evaluation Panel)
Have you completed your individual assessment of each response?
(If no, please advise the Chair of the Evaluation Panel)
Have any of the vendors contacted you?
(If yes provide details below and copy to Chair of Evaluation Panel)
Have you discussed the responses with anyone other than other
Panel Members?
(If yes provide details below and copy to Chair of Evaluation Panel)
Have you discussed your individual assessments with anyone other
than Panel Members?
(If yes provide details below and copy to Chair of Evaluation Panel)
As a result of your review and assessment of the responses are you
now aware of any potential conflict of interest?
If yes provide details below and copy to Chair of Evaluation Panel)
(
Notes:
Name:
Date: 30 October 2008
Confidential
26/05/2009
Page 13 of 18
Summary of RFP Responses
Respondent
Response
Aggregate
Presentation
Contract Price
Comments
(Address & contact details)
Compliant
Score
Recommended
Integrated Tertiary Software (Pty) Ltd
Yes
Between
and
483
Yes
The Integrated Tertiary Software (ITS) solution includes
Head Office: PO Box 25168,
Million for the
functionality to support student, financial, human
Monument Park, 0105, South Africa
high level phases of the
resources,
payrol and library management business
Tel: 00 27 12 4255600
project depending upon
Web: www its co za
licensing model options.
processes.
While modular, the system is claimed to be
Email: [email address]
These costs do not
fully integrated. ITS will integrate the solution with
include the cost of TCS
other TCS software. One of the core strengths is the
backfilling and third
management of assessments, linking distributed staff
party costs which will
resources to students, bar coded resource handling and
bring the total to
or
more.
distribution of study material through the warehousing
function.
The delivery model that forms the basis for this
proposal (though not necessarily their ideal delivery
model) is based in the system being installed on
servers of TCS (and based at TCS). ITS offers
three Licensing models and TCS can opt for any
one of the models. Pricing for all three models has
been provided. ITS can also supply all the required
Oracle Database products to host the system. A
hosted solution cannot be provided at this stage.
ITS has entered into discussions with
regarding the hosting of the ITS
Integrator system for TCS. This option will be more
cost effective and that this should, although pricing
Confidential
26/05/2009
Page 14 of 18
Respondent
Response
Aggregate
Presentation
Contract Price
Comments
(Address & contact details)
Compliant
Score
Recommended
has not been provided be seriously considered by
TCS.
ITS has recently implemented a Student Management
Systems at
ITS systems have also been
implemented with over 40 tertiary institutions in 12
countries.
No
0
No
This proposal is not compliant with the RFP process,
which
have acknowledged in their response
document.
state that they decided to submit a
non-compliant response to stimulate discussion.
does not have an out-of the-box system but is
interested in conducting an evaluation of open source
Student Management Systems and to then develop a
strategy to extend one of these or create a new open
source package. This evaluation would cost around
$20,000.
Confidential
26/05/2009
Page 15 of 18
