Impacts of assignment in problem-based learning : A case study from chemistry

The use of problem-based learning (PBL) within undergraduate chemistry courses is increasing in popularity. Despite several previous reports describing the impacts of PBL in terms of students‟ motivation and interest in chemistry, evaluations of its impact with respect to student learning are virtually absent. Here, an evaluation of PBL case studies in chemistry is made by consideration of assessment performance data over a six year period. The performance data are considered at different stages of the undergraduate courses and are compared against related data from laboratory work and closed-book examinations. These performance data are complemented by student feedback. The analysis reveals that, regardless of level, students find PBL case studies enjoyable and motivating. In contrast, performance in assessed work is found to depend strongly on assessment criteria. Students perform comparably with other modes of assessment when the PBL case study assessment criteria are familiar to them. In contrast, when the assessment criteria demand wider consideration of PBL outcomes, typical of those appropriate for the latter stages of degree courses, lack of familiarity with such criteria appears to result in lower performance in assessments, despite careful counselling from the tutor. Problem-based learning (PBL) PBL is an approach to teaching and learning that has received increasing attention since it was first described formally in the early 1980s (Barrows and Tamblyn, 1980). The underlying philosophy of PBL is that students advance their knowledge and understanding of a topic by tackling problems related to it. In all cases, the problems cannot simply be solved by application of a series of known algorithms or by reference to a previous or related example. The problems are always placed in an applied context to provide relevance and they require students to work as part of a team towards a common goal. Since the specific problems/contexts are designed in such a way as to be unfamiliar to the students, the tasks are necessarily „problem-based‟. This is different from traditional problem-solving workshops or tutorials where individual students may tackle themed activities as either problems or exercises depending on their experience at the time. There is a growing database of literature describing the use of PBL in HE, though the majority of these focus on descriptions of the principles of PBL and „best practice‟ in terms of PBL implementation (Duch et al., 2001; Savin-Baden and Major, 2004). PBL in Chemistry There are still relatively few reports of specific illustrations of PBL in chemistry. Most examples involve the use of case studies (e.g. Belt et al., 2002; Grant et al., 2004; Heaton et al., 2006; Potter and Overton, 2006; Belt and Overton, 2007) and laboratory work (McGarvey, 2004; McDonnell et al., 2007; Kelly and Finlayson, 2007). A number of years ago, colleagues and I developed a series of PBL case studies suitable for all stages of the undergraduate curriculum (Belt and Phipps, 1998; Belt et al., 1999, 2002, 2005; Summerfield et al., 2003; Belt and Overton, 2007). The case studies were written around a series of applied chemical themes including environmental, industrial and pharmaceutical chemistry. Students routinely described the „real world aspects‟, „opportunity to put theory into practice‟, „working with others‟ and „getting my own opinions across‟ as key features of the case study approach and they welcomed the opportunity to develop professional skills alongside subject-specific skills. The popularity of the PBL case study approach prompted us to write further case study material for areas of core chemistry and these too proved popular with students (Belt et al., 2005). Obtaining positive feedback of this type is, of course, gratifying from a developers pointof-view, and reporting such positive comments likely provides some encouragement for others to consider adopting PBL in their own teaching. Indeed, requests for copies of case study material continue to be received from UK-based chemistry teachers and from The underlying philosophy of PBL is that students advance their knowledge and understanding of a topic by tackling problems related to it. In all cases, the problems cannot simply be solved by application of a series of known algorithms or by reference to a previous or related example. Impacts of assessment in problem-based learning


Problem-based learning (PBL)
PBL is an approach to teaching and learning that has received increasing attention since it was first described formally in the early 1980s (Barrows and Tamblyn, 1980).The underlying philosophy of PBL is that students advance their knowledge and understanding of a topic by tackling problems related to it.In all cases, the problems cannot simply be solved by application of a series of known algorithms or by reference to a previous or related example.The problems are always placed in an applied context to provide relevance and they require students to work as part of a team towards a common goal.Since the specific problems/contexts are designed in such a way as to be unfamiliar to the students, the tasks are necessarily "problem-based".This is different from traditional problem-solving workshops or tutorials where individual students may tackle themed activities as either problems or exercises depending on their experience at the time.There is a growing database of literature describing the use of PBL in HE, though the majority of these focus on descriptions of the principles of PBL and "best practice" in terms of PBL implementation (Duch et al., 2001;Savin-Baden and Major, 2004).

PBL in Chemistry
There are still relatively few reports of specific illustrations of PBL in chemistry.Most examples involve the use of case studies (e.g.Belt et al., 2002;Grant et al., 2004;Heaton et al., 2006;Potter and Overton, 2006;Belt and Overton, 2007) and laboratory work (McGarvey, 2004;McDonnell et al., 2007;Kelly and Finlayson, 2007).A number of years ago, colleagues and I developed a series of PBL case studies suitable for all stages of the undergraduate curriculum (Belt and Phipps, 1998;Belt et al., 1999Belt et al., , 2002Belt et al., , 2005;;Summerfield et al., 2003;Belt and Overton, 2007).The case studies were written around a series of applied chemical themes including environmental, industrial and pharmaceutical chemistry.Students routinely described the "real world aspects", "opportunity to put theory into practice", "working with others" and "getting my own opinions across" as key features of the case study approach and they welcomed the opportunity to develop professional skills alongside subject-specific skills.The popularity of the PBL case study approach prompted us to write further case study material for areas of core chemistry and these too proved popular with students (Belt et al., 2005).Obtaining positive feedback of this type is, of course, gratifying from a developers pointof-view, and reporting such positive comments likely provides some encouragement for others to consider adopting PBL in their own teaching.Indeed, requests for copies of case study material continue to be received from UK-based chemistry teachers and from The underlying philosophy of PBL is that students advance their knowledge and understanding of a topic by tackling problems related to it.In all cases, the problems cannot simply be solved by application of a series of known algorithms or by reference to a previous or related example.

Impacts of assessment in problem-based learning
Communication lecturers overseas.However, despite the clear value of such feedback, it offers little in terms of providing evidence of student learning or, more prosaically, whether problem-based case studies offer a more (or at least as) effective teaching style compared to other approaches.
Research into the effectiveness of PBL has been reported previously, although the majority of detailed investigations have been in medicine (Gijbers et al., 2005).
Using the perceptions of students as the principal body of evidence for the effectiveness of PBL in medicine, Mackenzie et al. (2003) showed that the PBL approach encourages a more critical and self-directed approach to learning, and that these features would probably be of benefit to science students.Other reviewers have considered the effectiveness of PBL in terms of impacts on student attitude and/or performance (Albanese and Mitchell, 1993;Berkson, 1993;Vernon and Blake, 1993;Colliver, 2000).The outcomes of these analyses are mixed, with somewhat contradictory evidence for improvements over traditional methods.However, Dochy et al. (2003) have reported positive effects of PBL on students" abilities in knowledge application at the expense of their knowledge base, while Gijbers et al. (2005) present further evidence that PBL is beneficial in terms of promoting the understanding of principles that link concepts.
Reports of PBL in HE chemistry teaching have focussed largely on the rationale for its introduction, descriptions of the specific teaching activities and various logistical issues.In contrast, analysis of the effectiveness of PBL in chemistry teaching has been largely confined to analyses of student feedback data.Students consistently describe the PBL approach as enjoyable and they particularly like studying chemistry in context.In addition, they describe how PBL provides (a) an opportunity for developing key skills (Belt et al., 2002;Belt and Overton, 2007;Heaton et al., 2006); (b) a way of developing confidence with new disciplines or scenarios (Potter and Overton, 2006); (c) a greater appreciation and awareness of the chemical industry (Grant et al., 2004).In the laboratory, students usually find the PBL approach motivating although sometimes frustrating and demanding in the first instance (Ram, 1999;McGarvey, 2004;Kelly and Finlayson, 2007).Perceptions improve, however, with greater experience, and McDonnell et al. (2007) have noted enhanced student learning and better preparation for subsequent project work through PBL laboratory work.
The purpose of this paper is to provide some assessment performance data for students who have completed PBL case studies at the University of Plymouth, with the aim of providing some insights into the effectiveness of this form of PBL on student learning.This paper includes: i. a presentation of PBL coursework marks for (up to) 6 consecutive years of study (2001)(2002)(2003)(2004)(2005)(2006)(2007); ii. an analysis of student performance for PBL coursework at different stages of the undergraduate curriculum; iii. a comparison of student performance in PBL coursework against other (related) coursework and closed book examinations; iv.A summary of student feedback data.

Stage 2 BSc (Hons) Analytical/Applied Chemistry
Background Three case studies have been used with Stage 2 Analytical/ Applied Chemistry students since 2001-02.The Titan Project is a case study that requires students to research two different manufacturing process for the industrial scale production of TiO 2 and, having considered a range of factors (chemical, economic, environmental, neighbouring industries, etc), make proposals for future development.A second case study, New Drugs for Old, involves devising short-and long-term investigations of a potentially new analgesic drug isolated from a natural source, determination of the structures of a series of extracted chemicals using spectroscopic data and physical properties, and proposal of a method of making the most active compound based on some suggested synthetic procedures, economics and scale-up considerations.A third case study, Tales of the Riverbank, requires students to consider some basic principles of analytical measurements within the applied context of pollutant species within a river system, together with selection and evaluation of appropriate methodology.These three case studies are described in more detail elsewhere (Belt and Overton, 2007).

Assessment criteria
For each of the three case studies, student groups are assessed through a combination of oral presentations and reports.The assessment criteria, which are the same in each case, focus on the accuracy of solutions to the various problems given the data which is available, together with clarity of presentation.The type of information that students work with ranges from datasets that they will be familiar with (e.g analytical and spectroscopic data) to cases where they have to make "best-guess" estimates to parameters whose strict correct values are unknown.These too can be wide ranging and can include having to estimate scale-up costs in organic synthesis or evaluation of flow rates in rivers for environmental sampling.In all cases, however, the form of the assessment criteria is well known to the students.

Student performance
An evaluation of the effectiveness of the case study approach is given here by consideration of student performance in oral presentations and reports, together with a comparison with performance in other coursework (laboratory reports) and in examinations.These data (Table 1) have been compiled, uninterrupted, since 2001-02, though examination data is only available for 2005-2006 and 2006-2007.A couple of points emerge from the analysis.Firstly, students perform consistently well in the case study work.Although there is some annual variation in the mean marks for the case study , the combined mean mark (66.0 ± 7.9%) is very typical for coursework at this mid-point of the degree program.Secondly, both the annual and the combined mean marks for the case study assessments (25% of coursework) are extremely similar to the accompanying marks for the laboratory reports (75% of coursework) within the same module.The corresponding marks for the end-of-year examinations (50% of module) are available only for the last two years, with the combined mean mark being slightly lower than those of the two coursework components, although the spread of examination marks is somewhat higher.Both of these features are common for examination-based assessments.
Student feedback is largely anecdotal.Since parallel modules address transferable and professional skills in a more overt manner, the students are acutely aware of the significance of such skills and the importance placed on their development throughout the course.They particularly appreciate the opportunities that case studies give them to advance their skills, commonly citing group work, time management and presentation skills as those areas of greatest significance for them.A more detailed account of this qualitative feedback has been reported previously (Belt et al., 2002).

Background
The Pale Horse is a PBL case study that has been used as part of a Stage 3 or final year module entitled "Forensic Analysis" since 2001-02.The case study is carried out in small groups (4-5 students per group) over 4 sessions and the groups are assessed via oral presentations and a group report.The remainder of the module is assessed through traditional laboratory reports, which represent 75% of the coursework component, and an examination (50% overall module mark).Thus, the division of assessment marks is the same as for the Stage 2 module described previously.
A detailed description of The Pale Horse, including qualitative student feedback, can be found elsewhere (Belt et al., 2002), but it is worth considering an overview here.The case study sets analytical chemistry within the context of a forensic investigation of a (fictitious) suspicious death.The case study begins by setting the scene and introducing the characters involved, which permits a preliminary assessment of any motives for committing a crime.The role of the groups is also defined at the outset together with the intended learning outcomes of the case study in terms of acquisition of subject specific knowledge and development of scientific and transferable skills.Significant attention is also placed on the importance of the assessment criteria at this point.

Assessment criteria
Students are assessed by a combination of an oral presentation and a written report.The assessment criteria (which are identical for both components) require the students to provide a step-by-step rationale for their various analytical requests (evidence/technique), a fully justified solution to the problem (cause of death), an account of their problem-solving strategy, and the role of analytical chemistry in the case study.Thus, the students not only need to work towards a "best-fit" answer to the problem which is consistent with all of their data, but also to think carefully on the wider implications of their problem-solving.Emphasis is given to these features during the first briefing session including highlighting the importance of continuous and detailed note-taking to evidence planning, changes of direction, and a detailed rationale for specific requests as the case study progresses.

Student performance
The mean marks for students" assessment marks for the forensic case study, laboratory reports and end-of-module examination are summarised in Table 2. Analysis of these data reveals a number of outcomes.Firstly, the mean marks for the case study, when considered on an annual basis, show little deviation from the overall mean value of 57.5% (2001)(2002)(2003)(2004)(2005)(2006)(2007) indicative of little variation due to differences in abilities of the student cohorts.(There is an even smaller difference between the individual marks for oral presentations and for reports, as expected due to the identical assessment criteria).Consideration of standard deviations from the mean value (57.5 ± 9.6%; n=120) gives a better indication of the range of individual student performances.Similar (small) variations between individual years" marks and collective marks can also be seen for the mean marks from laboratory reports (74.6 ± 11.0%; n=118) and examinations (53.2 ± 13.3%; n=120) although the spread of marks for the latter is greater than that for either of the two coursework components as is common for other modules.Secondly, these annual consistencies observed for mean marks from individual assessment components, conveniently permits comparison between them.Thus, with the exception of 2001-2002 (the first year that the module was run), the mean mark for the case study has always been higher than that of the mean exam mark, and by approximately the same margin (ca.5-8%).For the collective 6-year period, the case study mark (57.5 ± 9.6%) is 4.3% higher than the mean examination mark (53.2 ± 13.3%); these observations are consistent with common relative performances in coursework and examinations observed in other modules.Thirdly, the mean annual case study marks are found to be always lower than the marks for the laboratory reports, despite the common theme of the module (forensic analysis) and, overall, there is a substantial difference (17.1%) between the mean case study mark (57.5 ± 9.6%) and the mean laboratory report mark (74.6 ± 11.0%) for the whole dataset.Fourthly, with the exception of 2001-2002, the (average) performance falls in the sequence: Laboratory Report > Case Study > Examination, with this sequence corresponding to 61% of students when considered individually.
A qualitative assessment of student feedback has been reported previously (Belt et al., 2002) and this is made more quantitative here.Briefly, 95% of students enjoyed the case study and agreed that it had enabled them to make more sense of theory.Consistent with these figures, 85% disagreed that the case study had not taught them anything new.In terms of process, no students said that their approach was the same at the end of the case study compared to the beginning, with the majority (80%) stating that it was clearly different.When asked about possible changes, 60% of students thought they would have achieved more given more time, although 90% of students claimed to have finished the work on-time.Half the students would have preferred to have gathered their own data via laboratory work and this might make an alternative adaptation of the case study.

Discussion
The collation of the assessment performance data together with its evaluation has been carried out retrospectively so, for the latter, there are no data corresponding to a "control" experiment.The study is not (strictly speaking) a longitudinal one, and there has not been an initial hypothesis, with experiments designed to test it from the beginning of the analysis.Despite these potential "research failings", the study possibly benefits from the absence of any preconceived bias towards expected outcomes, and the teaching within the modules has remained largely constant throughout -there is some benefit to teaching the same material every year !In Issue 5 Communication for other modes of assessment within the same module.
Performance in case study work is slightly better than for examinations, a feature which is common to the majority of accompanying modules, and one that has been reported previously (Potter and Overton, 2006).The popularity of the case study approach together with the good performance of students is, of course, pleasing from a tutor"s point-of-view, but is perhaps expected given the assessment criteria used for each of the case studies namely, the quality of solutions to problems and clarity of presentations and reports.Thus, although the contextual element and some open-ended nature to the problems within the case studies may not be entirely familiar to the students, the assessment methods almost certainly are.
The performance data for the Stage 3 students reveal some differences.Firstly, the uniformity in performance across different assessment components is not observed.Students perform particularly well when reporting on laboratory work, but achieve lower grades during examinations, a common observation for modules that have both coursework and examination components.However, performance in Stage 3 case study work is not only lower than in examinations as might be expected, but is lower than for laboratory reports, despite the focus of the module on a single theme (Forensic Analysis).This poorer performance in case study work can be seen both annually and overall (Table 2).In addition, the mean case study mark for Stage 3 (57.5 ± 9.6%) is markedly lower than for the corresponding Stage 2 mark (66.0 ± 7.9%), although this difference is not always seen on an annual basis.
It is possible that the lower Stage 3 case study marks might be attributable to students taking the assessed work less seriously than the other elements due to its lower weighting (12.5% of module mark).However, since there is an identical assessment weighting for the Stage 2 module, for which performance in the case studies is good, this explanation seems unlikely.Instead, it is suggested that the relatively poor Stage 3 case study performances are linked to the assessment criteria and the students" responses to them.The Stage 3 case study assessment requires students to provide "solutions" to the problem and most students respond well in this area by providing good evidence to support their conclusions (c.f.Stage 2 observations).Such abilities to solve problems, even when they are complex or trans-disciplinary, has been identified as one of the most positive learning experiences of PBL in medicine (Gijbers et al., 2005).However, the Stage 3 assessment criteria also require students to evaluate aspects of group methodology and planning, rationales for decision making and, in a wider sense, the role of analytical methods in applied disciplines such as forensic science.Such criteria are probably less familiar to the students, who give insufficient attention to them, despite careful counselling by the tutor in the first session.Of course, it is important that assessment methods align with the teaching approach employed.For PBL, it is recommended that assessments should permit students to "demonstrate understanding of the influence of contextual factors on problem analysis as well as problem solving" (Birenbaum and Dochy, 1996), "argue for their ideas on the basis of various relevant perspectives" (Segers, 1997), and that "the test items ask for more than the knowledge of separate concepts: Integrative knowledge, requiring the integration of relevant ideas and concepts, is stressed" (Segers, 1997).Thus, the case study assessment criteria would appear appropriate given these principles.
In summary, the impact of PBL, particularly within the natural sciences, has focussed traditionally on anecdotal evidence and/or student feedback with little attention placed on quantitative measures.The data presented here indicate that students" performance may be strongly linked to PBL assessment criteria, an observation seen previously with medical students (Dochy et al., 2003;Gijbers et al., 2005).
For PBL, it is recommended that assessments should permit students to "demonstrate understanding of the influence of contextual factors on problem analysis as well as problem solving"
Impacts of assessment in problem-based learning