Case Study 25. Moving from OMR to CBA for Summative Exams in Medicine

[Reviewer: Nigel Smellie]

Case Study 25. Moving from OMR to CBA for Summative Exams in Medicine
(University of Nottingham – Medicine, Dentistry and Veterinary Medicine)

1. General Description of Case Study by Reviewer.

• Driven by the need to improve the quality and efficiency of assessments, the Medical School’s Medical Education Unit (MEU) led the change from using optical mark recognition (OMR) to computer-based assessment (CBA).
• Well-planned, rigorously tested and well-executed implementation to reduce risk.
• Clear and tangible benefits for administrators, academics and students.

2. JISC Classification:

a. e-Learning Activity Area: e-Assessment.

b. e-Learning Benefits Model: BOTTOM LEFT (high ROI) part of the graph:

(1) E-Learning Approach (Y-Axis): “Automated”; improve efficiency (JISC 2008a p 29).

(2) Nature of the issue (X-Axis): well defined; rationally driven; measured financially; resource-evidenced.

3. Rationale for Change:

• Increased student throughput.
• Desire to create more realistic questions.

4. Learning, Learners and Staff Involved:

a. Learning and Learners: Medical knowledge of 1,900 registered undergraduate and graduate medical students.

b. Staff: Approximately 1,000 staff registered as users of VLE, but only a proportion of these directly involved - academic staff, SMEs, disability experts, external examiners, IT support personnel, administrators.

5. Methods, Technology and Resources Used:

a. Methods:

• Broad stakeholder engagement and consultation at the design phase.
• Workflow processes as well as questions developed.
• Staff training included half-day workshops and one-to one training.
• Processes for reporting successes and failures.
• Implementation was gradual rather than ‘big bang’ – simulated loading/use of system using volunteers.

b. Technology:

Web-server and VLE (WebCT and Blackboard).
• Original question bank in Authorware but this needed to be transferred.
• Options considered included Test Pilot and Question Mark Perception (QMP), but eventually the MEU used “Touchstone” system developed by Medical School.

c. Resources: Specific costs not identified/recorded in case study report.

6. Possible Innovation Identified:

MEU started summative online exams 3 years ahead of University’s central IT team – “early adopters”.

7. Benefits Claimed:

a. Administrators:

• Marking process/progress better managed.
• Marking time greatly reduced (from 10 hours to 2 seconds for 330 student papers).
• No longer any rejected papers due to mis-reads by OMR.
• Great deal of paper saved.

b. Academics:

• Broadened opportunities for use of multimedia (diagrams reduce cognitive load allowing student to engage more with subject matter).
• More interactivity in questions.
• Easier/faster editing process during question development (multiple inputs often needed).
• Changes to/revisions of questions recorded/tracked on software.

c. Students:

• Simpler to accommodate some disabilities such as dyslexia (personalisation).
• Some have reported that on-line exam “is cleaner”.

8. Lessons Identified:

• Greater involvement of stakeholders at early stages to overcome unfamiliarity with technical aspects/language (at the time – “logins”; “servers” etc).
• Series of small steps rather than “huge leaps”.
• Rigorous testing at every stage is very important – but still some surprises.
• More research needed on CBA to investigate the new interactive question types for validity and reliability.


9. Additional Comments by Reviewer:

• Claims of benefits are credible – and impressive.
• Broad benefits across the organisation, and the tangible results, indicate this type of project is a very good “early candidate” for raising profile of benefits of technology.
• Thorough testing avoided the sorts of disasters encountered in London Heathrow Terminal Five!


(544 words)

10. References:
JISC (2008a), Exploring Tangible Benefits of e-Learning: Does Investment yield interest?