The trials and tribulations of designing and utilising MCQs in HE and for assessing forensic practitioner competency

Communication The trials and tribulations of designing and utilising MCQs in HE... Abstract Multiple Choice Questions (MCQs) are a very well known, traditional and accepted method of assessment. The use of MCQs for testing students has produced numerous debates amongst academics concerning their effectiveness as they are viewed as practical and efficient but also perceived as possibly „too easy‟ and potentially unable to appropriately test the higher order cognitive skills that essay questions can assess.

The use of MCQs in a forensic science context is currently being investigated, not only for use within forensic science education, but also for the testing of competency of qualified forensic practitioners.This paper describes a Higher Education Academy funded project that is investigating the design and the implementation of MCQs for testing forensic practitioners and the lessons that have been learnt so far, that will assist academics in the development of robust MCQ assessments within forensic science degrees to promote and assess deep learning.
Multiple Choice Questions (MCQs) -Friend or Foe?? Multiple Choice Questions (MCQs) are a type of objective test question which involve an answer(s) to be chosen from a list of possible responses 1 .MCQs are commonly used within physical sciences education for both summative and formative assessments as they are a practical and efficient means of assessing large groups of students 2 .Much research has been completed upon MCQ use, design, management and implementation and due to this there are an abundant amount of resources that can be used by academics if they wish to use this type of assessment within their own teaching.
There is a large body of research into MCQ design, for example, studies completed by Carneson et al. 3 , Shultheis 4 , Fellenz 5 and McCoubrie 6 .Within this literature there have been attempts to produce guidelines for academics in the production of MCQs 1,7,8 .Studies by Collins 9 and Lorusso 10 discuss the importance of constructing questions which reflect the material being taught, using consistent writing styles and the correct construction of the stem (the question or statement that leads to the possible answers) and the options (the choices of possible answers which include the correct option and the incorrect options, also known as distractors).In addition to the construction of the MCQ test, the important issue of marking schemes is also well documented in the literature.Whilst standard marking systems (where a correct answer is awarded one point and these are totaled for the final score) are still very popular, alternative scoring processes such as negative marking, confidence based marking, the "hedging" format and value exam format are available to reduce guessing and encourage participants to identify their level of confidence in their answers which promotes deep thinking 11,12 .
A key debate within MCQ use is the reliability of this assessment method, with critics claiming that they are "too easy", they allow students to pass through guesswork and do not test higher order cognitive skills that other assessments such as essays and laboratory write-ups are able to 5,13 .Although these comments are relevant, there are published ideas and methods that address these issues.Bloom 14 in 1959 published a taxonomy of hierarchical cognitive learning which is regularly used by MCQ designers to test knowledge, comprehension, application, analysis, synthesis and evaluation.Bloom"s Taxonomy is commonly simplified to just three levels; knowledge (the easiest MCQs to construct), combined comprehension and application (understanding the meaning of material and then being able to apply it to concepts and theories) and problem solving (transferring existing knowledge to new problems and situations) 14 .With this in mind, MCQs can now be designed to assess a range of different module objectives beyond just the recollection of facts.

The trials and tribulations of designing and utilising MCQs in HE and for assessing forensic practitioner competency
The use of MCQs in a forensic science context is currently being investigated, not only for use within forensic science education, but also for the testing of competency of qualified forensic practitioners.

MCQs in the Field of Forensic Science
With the number of students reading for a forensic science degree having increased over the past 5 years 15,16

Concealed
This MCQ is purely asking the student to remember the meaning of the word "latent" in terms of marks found at a crime scene, this is obviously important but does not reflect the skills needed of someone involved with mark development or interpretation.An MCQ that asks for this deeper thinking could involve a description of a scenario in which a fingermark is present on a particular object and then ask "what is the best method for enhancing a mark on this surface?"This would then require the student to have knowledge of latent marks, the development techniques available and their relative effectiveness and to then apply this knowledge for this particular scenario.Although care must be made in the construction of this type of MCQ, for example, to make sure that the stem is clear and contains all the relevant information needed to correctly answer the question and the distractors are all plausible and accurate but do not fully meet the criteria for the correct answer, the production of these are not too onerous 9 .Problematically, the nature of forensic science is that many crime scenarios involve complex relationships between evidence, require research to answer particular questions and interpretation is generally not straightforward; therefore a more sophisticated set of MCQs is required.
The generation of questions which feed from one or more particular crime scenarios, that combine evidence, procedures and ideas and require the student to have completed research prior to their answer are recognised to be time consuming to produce 17 .Currently, there are no protocols or "toolkits" for the production of MCQs specifically for forensic science topics.
Although there are readily accessible databanks of MCQs for other subjects, such as Question Bank 18 and OCTAVE database 19 (for veterinary science education), there is no "example bank" of forensic science related MCQs available for academics to use.This current project ultimately aims to develop a series of resources (a "toolkit") for the development and implementation of MCQs for academics that teach forensic science to undergraduate and postgraduate students.These resources will include the generation of a bank of example MCQs, guidelines for MCQ management and an associated workshop in the design and implementation of MCQs in forensic HE assessment.To enable this, contribution to a large scale MCQ testing scheme for forensic practitioners was carried out and observations made upon the potential problems that can be encountered when designing MCQs in forensic science topics.

MCQs for Competency Testing of Forensic Practitioners
There is a widely known need to ensure that the quality of forensic and expert evidence is of a suitable standard."Competency" is a key performance criterion within the forensic investigative process for all forensic practitioners and proving competency in a UK Court is an important aspect of the legal system.There is currently a government initiative for improving employee skills, including in the scientific community 20 .Many professionals within the forensic arena have a variety of qualifications but they all need to demonstrate "continuing professional development" and continuous professional competency.Since the Council for the Registration of Forensic Practitioners (CRFP) ceased operating in March 2009, there has been an imperative to provide a quality control system, which will help maintain public confidence in forensic practice in the UK and to provide assurance to the UK Criminal Justice System.This is especially important for forensic practitioners that are from smaller laboratories that do not have ISO standard accreditation or who are infrequently required to act as a forensic expert due to the very specialist nature of their expertise.
The trials and tribulations of designing and utilising MCQs in HE... ...the nature of forensic science is that many crime scenarios involve complex relationships between evidence, require research to answer particular questions and interpretation is generally not straightforward; therefore a more sophisticated set of MCQs is required.

Communication
Where the CRFP essentially based their accreditation of forensic experts by peer review, a new method for assessing competency in forensic experts has been developed by the Forensic Science Society (FSSoc), which includes a pre-assessment evaluation of the practitioner, a practical based proficiency test and an MCQ test 21 .As the outcomes of such assessments allow a practitioner to describe themselves as competent, the MCQ tests were required to be a robust assessment strategy that allowed the level of knowledge and understanding across a broad range of forensic science topics to be identified and quantified and allow the use of learning outcomes and assessment criteria to provide quality assurance and academic rigour.The quality assurance of such tests was of upmost importance which highlighted the mechanisms required for MCQ test management and implementation.
The aim of this paper is not to discuss the competency of forensic practitioners but to utilise the initial experiences of implementing a pilot testing scheme for practitioners to identify issues involved in designing and managing MCQs particularly in forensic science topics and to identify any considerations required to use MCQs to test higher levels of learning in students studying forensic science in HE.

MCQ Testing the Experts
The main purpose of the testing scheme for forensic experts is to get a rounded-view of the competence and professional understanding of applicants as forensic practitioners in their various specialisms.Consequently, MCQs do not represent the full assessment process as prior experience and practical abilities in using particular equipment and methods also need to be considered.This project has only focussed upon the MCQ element of assessment therefore no evaluation of any other parts of the process have been considered.Two rounds of MCQ testing were conducted for forensic podiatrists in the UK; a pilot test and an actual test.The MCQs used for both the pilot and actual test consisted of 25 MCQ questions covering general principles of forensic procedure, evidence interpretation and relevant legal issues and 25 MCQ questions covering subject-specific topics relating to the relevant specialism of the applicant, in this case forensic podiatry.The general questions were all designed to primarily test knowledge and application/comprehension whereas the subject-specific MCQs were designed to focus upon problem solving.The participants were informed that for each correct response they achieved one mark but for each incorrect response, half a mark would be deducted.As part of the quality assurance process each MCQ was reviewed by three MCQ developers from the Department of Forensic and Crime Science at Staffordshire University and FSSoc.The final test formats were checked by a committee within the Forensic Science Society.The pilot-testing scheme was carried-out on two forensic practitioners.The second MCQ test was implemented in the same format as the pilot test but with revised and validated questions and was completed by four forensic podiatrists.The small number of participants was partly due to the small number of forensic podiatrists practicing in the UK.Forensic Podiatry was the initial specialism used, as experts from this area of forensic science are usually not affiliated with an accredited laboratory and therefore require other means to show competency such as the MCQ testing scheme being discussed in this paper.Feedback from the participants was gathered by one-to-one interviews and focussed on their understanding of the questions, their thought processes on choosing their answers and their opinions on the appropriateness of the MCQ format.This was to gauge whether the design of the questions tested beyond just knowledge and had climbed up Bloom"s ladder of hierarchy of educational objectives 14 and to understand how MCQs could be used to encourage deep learning.The 25 general MCQs were also investigated in terms of which levels of Bloom"s ladder of hierarchy were addressed and whether there was any difference in the participants" performance between these different levels.
For the purposes of question confidentiality, only general descriptions of the questions used are given to allow the feedback from participants to be understood.

Observations from the MCQ Test Process -Lessons to be Learned Development of Learning Objectives for Testing Competency in Forensic Science Skills
Prior to the generation of the MCQs for the testing scheme, learning outcomes were required.As stated by Collins 9 , these learning objectives should be written in terms of specific learner behaviour and should define the important skills and knowledge required to be tested.
When defining this knowledge, it must be clear to the examiner what a forensic expert must know to be considered as an expert.For appropriate MCQs to be generated this knowledge must be unambiguous but in the world of forensic science, there are grey areas of knowledge that require some experts to understand but not others.Consequently, instead of defining specific facts that must be known, the learning outcomes were directly linked with Bloom"s taxonomy 14 and described in terms of what an expert must demonstrate in general and subject specific knowledge, comprehension, application and problem solving within a forensic science context.This is also an effective way to test specific areas of The trials and tribulations of designing and utilising MCQs in HE...

MCQ Design for Forensic Science Topics
When designing the MCQs, it became apparent that a designer should not become bogged-down with attempting to make every question test higher cognitive levels.There is a tendency, once fully aware of the potential to test skills such as problem solving, to design all of the questions to meet this level.In reality, the level must take into account the subject matter as well as the level of understanding that is required from participants.For example, the general forensic science questions in the pilot test were centred mainly on factual issues such as expert witness duties, continuity of evidence and court conduct.As stated by MCQ researchers such as Glaser 22 and McCoubrie 6 , to ascertain expertise in this type of general area of forensic information, demonstration of pure knowledge of a domain is "the single best determinant of expertise."Figure 1, shows the number of general MCQs testing knowledge, application and comprehension and problem solving.
As stated above, the majority of questions were knowledge based only, which was appropriate for the topics being tested, but to enable a participant to demonstrate their understanding of certain forensic procedures and protocols, MCQs designed to test application and comprehension were also incorporated.These questions invariably asked for the participants to evaluate certain court case scenarios and evidence handling ideas and identify which was "the best" or "most appropriate".In these types of question, all of the options are correct but the participant must use their court going/court report writing experience to identify which is the preferred answer.In addition to these questions, a problem solving type MCQ was also included which described a particular scenario and then asked the participant to use their knowledge and experience to evaluate the best course of action.The success of the four participants in these different question types was variable.
Figure 2 shows the percentage number of correct answers for each of the four participants in each of the knowledge, application/comprehension and problem solving questions.
It is apparent that each of the four participants performed the best in the lower level of Bloom"s taxonomy 14 , with all participants correctly answering 50% or more of the knowledge based MCQs.
No real trends can be determined from the higher level problem solving MCQ as there was only one question of this nature in the general forensic science topic test but it was clear that this question was possible to answer with two participants giving a correct response.The most interesting results were seen for the MCQs that tested application and comprehension of knowledge.Although participant 1 performed very similarly in these questions, the other three participants were not as competent.This could have been due to the subjective style of some of these questions which lead to participants interpreting the question in different manners.This could especially be seen in questions that asked for the participant to use their own experience of court cases which may have differed.Of course, it is also possible that these questions have accurately depicted the participants" competency in this topic, where they all have sufficient knowledge of the topic but differ in their ability to fully comprehend some of the forensic processes being questioned.Some of the design features of the MCQs that could have contributed to the participants" performance in application and comprehension are discussed below.
With the subject-specific questions, the emphasis was to test the skills in which an expert would need to be proficient to conduct analyses and interpretation of a particular evidence type.For this, the lower level learning was inadequate, as this would not demonstrate a forensic scientist"s ability to use particular information to answer bigger questions about a crime scene or evidence.The subject specific questions were a mix of knowledge, comprehension, application and problem solving, many of which provided a case scenario with relevant information and then a series of questions that were related to the crime and evidence.The main difficulty identified when designing MCQs that provided a lot of prior information as part of a crime scene scenario and are lead-ins for more than one question, was to avoid producing questions that cued a response to subsequent questions relating to that crime.Careful writing of the stems and ensuring that each question is as independent as possible reduced the risk of giving the participants "clues" to the subsequent questions.The effect of cueing has also been discussed in MCQ use for clinical nursing practice, which also attempts to provide "real-life" scenarios to test participants" higher learning skills 23 .Test timing must be increased for this type of question as the participant has a large amount of material to read (and re-read) before making a response.This type of MCQ could be expanded further, instead of only providing a case The trials and tribulations of designing and utilising MCQs in HE...

Figure 2: Percentage number of correct responses for each participant for the knowledge, application/comprehension and problem-solving MCQs
Communication summary or a description of the evidence, photos of exhibits or even video recordings, e.g.simulated CCTV scenarios, could be used as a lead-in to the questions.
If this type of question was used for HE students, the resources such as photos and video recordings could be provided to the students prior to the test so that they can carry out analysis and research on the evidence to prepare for the MCQ assessment.Research conducted by Williams and Clark 24 observed that students rated the effort they exert before an MCQ test higher than their actual ability or teacher input.This study showed that students perceive their input towards the preparation of an MCQ test, such as note-taking, reading etc as being very important leading to deeper learning prior to the tests.By providing case scenarios and information about forensic evidence before MCQ tests, this will encourage the desired deeper learning as an outcome.
The participants in the pilot study were able to provide insight into the thought processes of a student when deciding which option to choose in an MCQ.These thought processes are highly important in aiding an MCQ creator in the identification of potential pitfalls in question design.
When creating a stem for an MCQ that asks for "the best" or the "the most important" option to be chosen, it became apparent that the designer must be aware of the affect of any differences in the participants experiences in their ability to choose the correct answer.For example, on a question relating to laboratory procedure, one participant stated "I struggled with "the most important reason" aspect of the question as I could see that some other options offered as being rather important too" It appeared that in some situations, distractors can appear appropriate in their balance of plausibility and not fulfilling the criteria for the correct answer to the MCQ producer but depending on a participant"s experience, for example, work place protocols, the same balance is not seen by the participant.Awareness of differences present in the forensic field is essential for this balance to be true for all participants.The idea that options should not be biased towards a particular group of people due to their background has been discussed by Collins 9 .
One general question asking what the functions of a particular forensic science body were, caused issues as the options indicated the answer should be the body"s main "concern" rather than depicting the main "role" which the participant had memorised.They stated; "I struggled with this one a bit.I went for option (a) as the described "role" of this body represents half of option a, but I would not have considered the second part of this option as being correct, therefore I am not 100% confident in my answer because of this" The problem with this question appeared to lie not only in poor stem wording but also in the fact that the question had different possible answers depending on how the participant interpreted the published information (readily available to forensic practitioners) describing this particular forensic body.
In these situations, if appropriate, the wording of the options must fit with the stem or this type of question should be avoided completely if there is not solely one answer.Keeping in mind the possible ambiguity of some areas within forensic science, the choice of question is highly significant.Although the participant had reduced confidence in their answer, this did not stop them making a response even though there were penalties for incorrect answers.In fact this participant described themselves as having a lack of confidence or doubt with 10 out of the 25 responses he gave to the general MCQs but still attempted all questions regardless of the negative marking.This shows that negative marking is potentially only a deterrent when the participant has no idea of the answer at all and an element of "hedging one"s bets" occurs when the participant can narrow it down to two possible answers.

MCQ Management
When designing MCQs that must potentially stand-up to scrutiny in a UK Court, any quality assurance procedures used must be transparent and easy to follow by those considering it.When designing the MCQs for the forensic practitioners, it became apparent that when there are a large number of questions that need to be reviewed and updated by a group of designers, a continuity trail is desirable.This trail can most easily be provided by utilising a database which states any changes to questions that have been made, the name of the designer making the changes and when and how the questions have been used.In HE, a MCQ database containing forensic questions would benefit from the ability to be updated by multiple designers and to audit the evolution of questions over time.This is especially useful when showcasing a course"s assessment methods to University External Examiners.From these observations a software programme which provides a useful interface linked to an MCQ database has been designed and created.This software programme combines both a testing facility and also a management system.It allows the user to not only store and search for MCQs based on topic, type and level of cognitive thinking being tested, which is similar in style with other extant databases of MCQs but also allows the evolution and quality control procedures to be viewed for each individual question.The programme allows multiple users to view developing MCQs and provide revisions and comments regarding design and validity, all of which is logged providing a trail of continuity.Multiple Choice Question tests can also be created using the software, similar to using the test function in Blackboard but in this case, the test creator is not limited to the structure of Blackboard and can make bespoke tests for different purposes without the need for copying and pasting MCQs from other areas.This is particularly useful for HE institutions who deliver bespoke courses to external companies.This software programme is currently being piloted at Staffordshire University and the FSSoc but will be made available to HE institutions in the future.
The trials and tribulations of designing and utilising MCQs in HE...

Conclusions
The generation of a "toolkit" in MCQ design and implementation is a useful resource for academics wanting to use this type of assessment in their teaching.This toolkit could be developed from observing MCQ use in other disciplines but this project has also utilised the experience of developing a professional competency testing scheme for forensic practitioners.Observing an MCQ testing scheme for forensic practitioners to provide information and ideas for academics has been invaluable.Higher Education is constantly aiming to develop assessment types which are not only robust but also so that their actual creation may be mapped in terms of quality control and validation.In examining a system which must be sufficiently robust to stand up in a court, the smallest details in design are crucial, and therefore most useful for academics trying to create MCQ tests for students.This preliminary study has identified test design issues that must be considered by academics when developing their own MCQs and has also been invaluable for the initiation of the MCQ "toolkit".A limitation of this study has been the small cohort of participants that took part in the tests to date.This project will develop as further MCQ tests are generated for the FSSoc in forensic subject areas other than forensic podiatry.Feedback from future participants will be gathered and analysed for subsequent publication and ideas and issues raised incorporated in the final MCQ toolkit.Currently, the toolkit has been initiated by the creation of example MCQs in a range of forensic topics for academics to use within their teaching.Further to this, an MCQ design and implementation workshop will be delivered at Staffordshire University in September 2011.
As an assessment process for forensic practitioner competency, MCQs have demonstrated that they can be robust if designed with knowledge of the requirements of the expected audience and of the design philosophy of MCQs which appropriately test hierarchical cognitive learning.
MCQs... can be robust if designed with knowledge of the requirements of the expected audience and of the design philosophy of MCQs which appropriately test hierarchical cognitive learning.

Figure
Figure 1: Number of general forensic science MCQs testing the three main levels of Bloom"s taxonomy