Measuring the Impact of Context and Problem Based Learning Approaches on Students ’ Perceived Levels of Importance of Transferable & Workplace Skills

Year-one chemistry students at two different institutions were asked to rate the importance of a series of discipline-specific, transferable and laboratory skills by responding to a series of Likert-type questions. The students at both institutions had studied similar curricula but had different levels of experience of the Context and Problem Based Learning (C/PBL) approach. Analysis of the responses to the questionnaire was conducted by determining the “Level of Importance” (reported as the percentage of students at each Institution rating each skills as “Important” or “Very Important”). Both cohorts assigned a very similar level of importance to all disciplinespecific and laboratory skills (i.e. under 10% difference). There were larger (>15%) differences between responses from the two institutions to statements on two transferable skills: Team-working Skills and Oral Presentation Skills, the cohort with exposure to C/PBL giving the higher level of importance in each case. This study has revealed some potentially important differences in the perceived level of importance chemistry students place on the development of oral presentation and team-working skills which may be related to the use of C/PBL in the early stages of degree programmes. The study has also shown that the level of importance students assign certain transferable skills (such as Problem Solving Skills) may be independent of exposure to C/PBL.


Introduction
In the last fifteen years, a number of studies have investigated the skills development opportunities afforded by physical science degree programmes (Hanson & Overton 2010;Galloway 2017).One of the key findings of this work was the existence of a "skills gap": a significant difference between employer expectations of graduate skill sets and the skill sets that many graduates actually possess when they enter the workplace (Hanson & Overton 2010;Kondo & Fair 2017).Physical science degree programmes appeared to be under developing the transferable skills that employers valued.These studies made a number of recommendations relating to how physical science degree programmes should better support the development of students' transferable skills throughout degree programmes.Subsequent work in chemistry and other disciplines has focused on the use of student-centred active learning experiences, teamwork and authentic assessment practices to develop these skills (Blumhof, Hall et al. 2001;Carmona-Murillo, González-Macías et al. 2013;Freeman, Eddy et al. 2014;Williams & Handa 2016).
In order to build on previous work investigating the development of chemistry student perceptions of their skills development in years one and two of degree programmes (Williams & Handa 2016), a comparison of year one chemistry student perceptions at two different UK higher education institutions was undertaken.The two different institutions both use a variety of teaching and learning approaches in year one of their respective chemistry degree programmes, but one institution (Institution A) makes substantially more use of the open-ended Context and Problem Based Learning (C/PBL) approach.Previous research has suggested the C/PBL approach is an effective method for supporting the development of transferable skills (Kelly & Finlayson 2007;Carvalho 2016).The approach has already been used in undergraduate chemistry programmes on numerous occasions (Summerfield, Overton et al. 2003;Williams, Woodward et al. 2010).Institution B adopts a more conventional approach in year one: most teaching is lectureand laboratory-based.
The C/PBL approach works by providing students with open-ended problems based on real world contexts (Engel 1997;Raine & Symons 2005).These problems are typically solved in small teams (usually consisting of between four and eight students).The approach requires students to take ownership of much of the learning process through time management, working with other team members and communicating science effectively.C/PBL is usually assessed authentically, requiring students to work on assignments which are framed within the context of the problem.(For example, a problem that requires students to work as government advisors may be assessed by writing a policy report or giving a short presentation to a politician who may not be an expert on the subject they have researched).
The aim of this study was to investigate whether the method of instruction has any effect on the level of Importance that first-year chemistry students perceive transferable and workplace skills to have for graduates in their discipline.The key research question was framed as: Does the use of Context & Problem Based Learning approaches in Year One of a Chemistry Degree programme lead to students placing a higher "Level of Importance" on specific transferable and workplace skills?

Institutional Context
This section will provide a review of the teaching approaches used in year one at the two UK higher education institutions involved in this study (anonymised as Institution A and Institution B).The level of content and student ability of both programmes can be assumed to be approximately the same as both institutions have similar entry requirements and tariff.Both programmes deliver MChem and BSc programmes which are accredited by the Royal Society of Chemistry.The most significant difference between the institutions is the intake size.Institution A typically expects to recruit approximately 120 students per year into year one, whereas institutions B typically expects to recruit approximately 70 students per year into year one.
Table 1 summarises the core teaching approaches used in year one of the chemistry programmes at both institutions.Lecture and laboratory teaching constitute the two single largest elements of contact time at both institutions.
The inclusion of C/PBL elements in the core curriculum at Institution A allows authentic assessment to be embedded throughout year one teaching.This means students must work on teams to produce written reports, prepare oral presentations (in the form of a PowerPoint presentation and a debate presentation against another team) and present scientific posters numerous times during the academic year.

The inclusion of regular group work at
Institution A in the form of weekly C/PBL, tutorial and problem class sessions helps ensure that students get significant experience of working in teams and discussing science with peers.In addition to the above regularly scheduled teaching, year one students at Institution A are given three workshops from the institution's careers tutors and five training sessions on science communication delivered by chemistry academics.

Table 1 Comparison of teaching approaches used in year one at the two institutions
Although beyond the scope of this study, it is worth noting that students in Institution B get many opportunities to develop their transferable skills during years two and three of their degree programmes.

Research Methodologies
A previously developed and validated questionnaire (Williams & Handa 2016) was distributed to year one students at both institutions in the closing weeks of the 2016-17 academic year.The questionnaire was distributed in a compulsory laboratory class at Institution A and was distributed in workshops at Institution B. The response rate at Institution A (72 out of 103 students) was better than that at Institution B (24 out of 70 students).
The questionnaire took the form of a series of Likert-type questions which asked students to rate how important they believed a number of skills and competencies were for chemistry students to develop by graduation.The questionnaire was based on previous research conducted by Hanson and Overton (2010) on behalf of the Higher Education Academy (Hanson & Overton 2010).The questionnaire included the same skills and competencies as that used in Hanson and Overton's.It was hoped that analysis of student responses to this questionnaire would help establish whether the use of C/PBL approaches in year one teaching approaches does transform student perceptions of the importance of the skills and competencies that graduate employers value.The questionnaire included transferable skills as well as discipline-specific and laboratory skills and knowledge (see List 1).
List 1.The skills and competencies included in the research questionnaire (Williams & Handa 2016).

Name of skill/competency:
Analytical

Results and Discussion
For the purposes of the analysis, the level of importance has been defined as the sum of the percentages of students who rated a particular skill or area of knowledge as important or very important.A comparison of the level of importance assigned to statements relating to theoretical discipline-specific skills and knowledge showed a high level of consistency between the two cohorts (the difference in response for all but one skill was <5%.See Figure 1).The biggest difference between the two cohorts was a perceived 6.9% difference in the importance of analytical techniques (the Institution A cohort assigned this a higher level of importance).Additionally, it is worth noting that the minimum level of importance for any discipline-specific theoretical disciplinespecific skill is 82.6%.
The responses to the question on analytical techniques also showed the largest difference between the two cohorts' responses on discipline-specific skills (the level of importance assigned by the Institution A cohort was 6.9% higher).This data suggests that the majority of first-year chemistry students (>82%) place a high level of importance on discipline-specific skills and that this appears to be independent of the teaching methods that students are exposed to.
The level of importance reported for transferable skills also shows a good level of consistency (i.e. the difference is <10%) between the two cohorts with two exceptions: team-working skills and oral presentation skills (see Figure 2).The Institution A cohort assigned a higher level of importance to oral presentation skills (18% higher) and teamworking skills (19.4% higher) than the Institution B cohort.This suggests that a greater percentage of the Institution B cohort fail to recognise the importance of these skills to chemistry graduates compared to the Institution A cohort.This may be a reflection of the strong emphasis placed on C/PBL at Institution A. This approach places a focus on team-work (all problems are worked on in small teams) and makes use of authentic assessment practice which includes opportunities for students to demonstrate their understanding and interpretation in the form of oral presentations.
It is interesting to note that the level of importance of numeracy and computational skills was almost 10 % higher for Institution B than Institution A. It is possible that this difference occurred due to the delivery of a Symmetry and Simulation module at Institution B in the late stages of year one.This module illustrates how mathematical concepts are related to the chemical principles of spectroscopy and bonding and how computational simulation can be used in chemistry and biochemistry.The equivalent content is not taught until the opening weeks of year two at Institution A.
The level of importance placed on problemsolving skills is the Institution B cohort is 8.5% greater than the Institution A cohort.Although this appears to be a surprising result, it is worth noting that both cohorts have assigned this skill a high level of importance (87.3% at Institution A and 95.8% at Institution B).This result suggests that curricula that do not include C/PBL experience can still provide students with learning experiences which reinforce the importance of some transferable skills.Students at Institution B gain experience of problem solving through regular workshop and tutorial sessions or in-lecture problems.Although these problems are different in nature from those encountered by students working on C/PBL problems, questionnaire respondents are very unlikely to recognise that difference.

Skill Transferable Skills Comparison
Institution A (N=72) Institution B (N=24) The final set of skills and knowledge analysed related to laboratory (practical) teaching (see Figure 3).Like the theoretical skills, there is a very high level of consistency between responses from the two cohorts.This appears entirely reasonable due to the similarities in the structures of the year one laboratory programmes at the two institutions.Both institutions teach a similar number of hours of laboratory chemistry and both programmes teach a comparable range of core skills (aligned to the expectations of the Royal Society of Chemistry who accredit these programmes).
The lowest recorded values at both institutions were for planning and design of experiments which can be justified by the level of the students responding to the questionnaire (the scope for students to design their own experiments is usually fairly limited in year one of chemistry degree programmes in the UK).
Responses to the statement on manipulative practical skills showed the largest difference between the two cohorts (8%) however it is worth noting that only 22 respondents at Institution B submitted responses to this statement.
Figure 4 summarises the magnitude of the differences in levels of importance recorded at both institutions.The similarities in the levels of importance of the discipline-specific and practical skills are likely to be due to the similarities in the lecture and laboratory curricula of the two programmes.The differences in levels of importance assigned to team-working skills and oral presentation skills may be due to the use of C/PBL approaches at Institution A. The institution B cohort rated the level of significance of problem-solving skills and numeracy and computational skills between 5-10% higher than cohort A suggesting that other types of learning experience also contribute to students' perceived levels of importance of certain skills.

Conclusions
A number of potentially important differences between the cohorts worthy of further study were observed.Students who had been exposed to C/PBL approaches in year one of their degree programme appeared to be more likely to believe team-working skills and oral presentation skills were either important or very important skills for chemistry graduates to have developed during their degree.It is possible that the use of C/PBL approaches which are built around a framework of teams working through open ended problems, and are assessed using authentic approaches (including oral presentations) help reinforce the importance of certain transferable skills in a context which highlights how professional chemists use them.
Cohort B reported higher levels of importance for both numeracy and computational skills and problem solving skills.This suggests that students who gain other experience of problem solving (e.g.regular tutorials, workshops or inlecture problems) may recognise the importance of many transferable skills to professional chemists as well as (or even better than) students who have been exposed to the C/PBL approach.

Figure 1
Figure1Percentage of year 1 respondents at Institution A (N=72) and Institution B (N=24) who stated that the indicated discipline-specific skills and/or knowledge were important or very important for chemistry graduates.

Figure 2
Figure 2 Percentage of year 1 respondents at Institution A (N=72) and Institution B (N=24)who stated that the indicated transferable skills were important or very important for chemistry graduates.

Figure 3 Figure 4
Figure 3 Percentage of year 1 respondents at Institution A (N=72) and Institution B (N=24)who stated that the indicated practical skills and/or knowledge were important or very important for chemistry graduates.
Lectures -Between six and eight per week 50 minute long lectures per week.Often include some interactive components.