Assessing the Status of Fundamental Chemistry Knowledge Using a Visually Displayed Chemistry Diagnostic Test

Students in the classroom may possess varying levels of knowledge and understanding of fundamental chemical concepts so it is necessary to ascertain if any misalignment exists with their expected prior knowledge; if left un-addressed, such misalignment may create difficulties for students beyond the first year of their undergraduate study. The aim of this initial diagnostic test study is to assess students' knowledge of basic concepts in chemistry that underpin the science of patient safety in pharmacy practice using a novel approach which enables a variety of question types. A diagnostic test using Microsoft PowerPoint consisting of 40 individually timed questions was presented to an entire cohort of Master of Pharmacy (MPharm) degree programme undergraduate students in both the first year (n = 163) and third year (n = 118). The questions ranged from basic chemical nomenclature to more complex areas such as stereochemistry. Our results showed that the third year undergraduates performed significantly better than those in their first year (p ≤ 0.004) with both cohorts performing well in the basic questions such as recognition of elements and bonding. However, a more in-depth analysis of the questions indicated areas such as chemical structures and mole calculations that caused difficulty for both cohorts. This test highlights problem areas in fundamental chemistry concepts which students find difficult either to grasp or to solve, and as such it serves as a useful


Introduction
Fundamental chemistry knowledge is essential to understanding more complex concepts and applications and as such is not only restricted to Pharmacy.Students enter degree programmes in the UK via a variety of routes: traditional secondary school, Further Education and those with previous qualifications.With so diverse a cohort, academics may find themselves faced with students of whom some may either struggle with the basic concepts or, conversely, have a strong foundation in these topics.Meaningful learning is enabled when new knowledge is related to concepts which the learner currently holds and knows (Ausubel et al. 1957).
There is an expectation of the knowledge base students possess based on the entry requirements for the Master of Pharmacy (MPharm) degree programme in the UK (a grade B or equivalent at A-Level Chemistry).However, it has become apparent to staff that some mismatch exists between what students are expected to know and what they actually do know and understand.Academics can address any mismatch only when it has been identified.Chemistry Diagnostic Tests have been used predominantly to predict grades or for student placements (McFate & Olmsted 1999).For example, Russell (1994) reported the use of a diagnostic test as a useful predictor of success in general chemistry where questions were designed to examine the problem solving ability using triads of quantitative chemistry 'word problems', quantitative mathematics problems and chemistry concept questions not requiring mathematical calculations.The quantitative chemistry questions were superior to the quantitative mathematics questions in predicting success in general chemistry.Prior knowledge and practice in applying mathematical concepts to chemistry questions are important.Students were permitted to use calculators in this test and it was found that those who did so performed significantly better than those who did not.Kennepohl et al. (2010) used an online, self-diagnostic instrument to predict university students' success in an introductory General Chemistry course.The self-diagnostic test analyses the areas of student background, conceptual basics, critical thinking, mathematical skills and problem solving.This work focuses primarily on identifying at-risk students with the objective of addressing individual needs, placement of students in appropriate cohorts and protecting students from a de-motivating experience in chemistry which might impact on other courses or modules.Chemistry education research has also been aimed at investigating what it is that students do not understand or, equally, that they misunderstand.Bodner (1991) discusses misunderstandings and answers from a conceptual knowledge exam given to entering graduate students.Interviews are a common method used to obtain further insights into students' misconceptions and level of understanding.This is a successful approach but highly time consuming and one that requires substantial training to ensure interviews are effective.A two-tier Multiple Choice Question diagnostic test has been used by Peterson & Treagust (1989) with high-school students to determine misconceptions in covalent bonding and structure.This instrument was also used by Birk & Kurtz (1999) to determine the retention of specific misconceptions over time with students from high school to graduate school and to Chemistry faculty.There was no time limit to complete the exam.Cros (1988) used a diagnostic test to evaluate fundamental chemistry knowledge after one year of undergraduate study where the focus was on the misconceptions and misunderstandings regarding aspects of the constitution of matter and acid/base chemistry.The extent of improvements in students' conceptions with university teaching was reported as disappointingly low, with a similar outcome for questions applying chemistry concepts to everyday life.Hassan et al. (2004) conducted a study to measure first year university student understanding in four key concept areas: 1) the nature of the covalent bond; 2) bond polarity; 3) stereochemistry; and 4) the importance of molecular shape and functionality.The test was not timed and an area highlighted as a specific weakness was bond polarity.This study emphasises that an appreciation of students' prior knowledge when entering Higher Education is important in order to build new concepts based upon a solid foundation.
A Chemistry Concepts Inventory (CCI 2013) with 22 one-and two-tiered non-mathematical conceptual multiple choice questions is available to indicate the level of chemistry misconceptions generally covered in the first semester of a college chemistry course.It was administered to over 1,400 students studying majors in Science and Engineering during the first week of an autumn semester and repeated during the first week of the following spring semester.The average grade on the CCI was 45% (in the autumn) and 50% (the following spring) and highlights the gaps in fluency with chemistry concepts.Specific to Pharmacy undergraduate students, Sharif et al. (2007) describe a diagnostic test in Chemistry, Mathematics, Physics, Biology and English administered as a Multiple Choice Questions (MCQ) paper test.Paper-based diagnostic tests have also been used to assess numeracy skills in Pharmacy undergraduates.Malcolm & McCoy (2007) carried out a seven-year study during which students were tested on entry and again after a basic numeracy course in the first term.It is important to appreciate the depth of knowledge students have already acquired in their secondary education on entry to undergraduate degree programmes.
A chemistry diagnostic test is presented here to address fundamental chemistry knowledge as applied in a UK Pharmacy degree programme.Some features of the test have been specifically selected: the test is paper-based, which enables students to draw chemical structures, a feature that is not available when using Electronic Voting Systems (EVS) or MCQ diagnostic tests; the use of calculators was not permitted.The professional body in the UK, the General Pharmaceutical Council, does not permit the use of calculators for its Pre-Registration exam for Pharmacists, so, for Pharmacy students, a feature of the diagnostic test needed to be calculations which could be performed easily without the use of a calculator.
Chemistry forms a substantial proportion of the MPharm degree programme at the University of Hertfordshire with almost a quarter of the course consisting of Chemistry modules.Other modules integrate Chemistry in an applied setting within the use of formulations, drug delivery, biochemistry, pharmacology and therapeutics and a strong foundation in chemistry is required to understand fully these key areas in Pharmacy.Without this fundamental knowledge, students cannot fully integrate the pharmaceutical importance of drug structures, drug interactions and side effects.Nakhleh (1992) postulates that one reason why some students struggle to learn chemistry is the failure to construct an understanding of fundamental chemistry concepts from the very beginning of their studies.Such understanding is paramount to ensure patient safety.A typical example where chemistry knowledge is essential in the treatment of medical conditions involves the use of a single enantiomer of a drug.In the treatment of Parkinson's disease, racemic dopa resulted in adverse effects, such as nausea, vomiting, granulocytopenia and involuntary movements.However, the use of a single enantiomer, L-dopa, resulted in a reduction of the adverse effects including no observation of granulocytopenia, and the required dose was halved with added improvements to patient health (Valentová & Hutt 2004).Other available chemistry diagnostic tests (e.g.CCI) do not include such topics, which are of paramount importance to patient safety, and this provides the rationale for our study.
In order to apply chemistry to clinical pharmacy cases, students must be able to know and recall fundamental concepts as required.This skill was included in the design of the test by limiting the time allocated for answering each question, rather than by imposing an overall limit of time where students could return to any question at a later stage in the test.The aim of this research was to develop a simple, non-paper-based chemistry diagnostic test, which included an element of limitation of time on each question and was relevant to problem solving and fundamental chemistry concepts and which would be easy to administer to an entire cohort of students during a teaching session.Such a format has also been used in assessing numeracy skills in Pharmacy undergraduates (Hitch et al. 2010).

Method
Using Microsoft PowerPoint a test was designed that consisted of 40 questions on fundamental chemistry delivered to an entire cohort of undergraduate Pharmacy students (Year 1: n = 163; Year 3: n = 118).Six categories of questions comprising Structures, Bonding, Chemistry Calculations, Acids and Bases, Organic Chemistry and Stereochemistry were assessed in the diagnostic test.These categories of fundamental chemistry concepts were identified from the Indicative Syllabus required by the UK General Pharmaceutical Council for all UK Master of Pharmacy degree programmes (Pharmacy Education 2013).A 40-item diagnostic test was used in this initial study to ensure a sufficient spread of questions in each category for assessment.
The questions ranged from an easier start on structures and bonding and progressed to more complex areas in organic chemistry and stereochemistry.Each question was allocated a specific display time ranging from 6 s for the recognition of chemical symbols to 25 s for more complex problems.The questions were displayed on a projector screen for the specific length of time during which students wrote an answer down on a supplied answer sheet.Each displayed question in the test was followed by a blank screen for 4 s.
The test was initially piloted among academic and postgraduate members of the faculty so that appropriate timings for the display of each question could be determined.One question was adjusted to avoid misunderstanding of its wording; a total of five questions were allocated a longer display time.The answers were scored as follows: correct (3), incorrect (−1), no answer (0).The students were not given prior warning of this diagnostic test and therefore no specific preparation was expected of them.It was anonymised as it was designed to indicate areas of strength and weaknesses among the whole cohort and not specifically to indicate individual student performance.
The test results were analysed according to the following categories: 1. comparison of performance between Year 1 and Year 3; 2. performance of both cohorts in six different categories of questions.
In order to determine whether there was any significant difference in the diagnostic test questions between Year 1 and Year 3 undergraduate students, a one-tailed Mann-Whitney U test was carried out (CI = 0.05).This test is used to compare the medians of non-normally distributed data sets.A test for comparing two proportions was carried out to determine whether there was a significant difference in the individual diagnostic test questions between both cohorts (CI = 0.01).
The mean score per question for each cohort was determined as shown in Figure 1 and details against each question item are available in Supplementary file 1 (supporting material on line).A one-tailed Mann-Whitney U test confirmed that Year 3 students, not surprisingly, performed significantly better than Year 1 students overall (p ≤ 0.004).This was to be expected, but, as explained previously, even at Year 3 some students continue to struggle with knowledge of fundamental chemistry, a difficulty that impacts their contextual application of this knowledge.It is evident that certain questions were answered poorly by both cohorts from the categories as shown in Figure 2. Some questions overlap more than one category but were placed in a single category for simplicity, which gives an instant visual representation of the areas assessed in this diagnostic test.

Discussion
Comparison of performance between Year 1 and Year 3 Both Year 3 and Year 1 students performed well on basic questions requiring the recognition and recall of chemical symbols and bonding and structure.
Students embarking on MPharm degrees are expected to have good competency with these fundamental questions and it is therefore an expected result from both cohorts.Examples of questions with overall percentage of students giving the correct answer are shown in Table 1.
Year 3 students performed significantly better than Year 1 students (Table 2) in questions relating to functional groups and solubility (p < 0.005) and on stereochemistry (p < 0.007).A possible reason for this may be that Year 3 students had revisited these concepts during their MPharm studies in a laboratory practical on drug absorption and functional groups towards the end of their first year.
Students also covered stereochemistry of drugs in their first year and more extensively in their second year.

Performance of both cohorts in different categories of questions
The diagnostic test highlighted common areas of difficulty for both groups of students.This is most informative in terms of feedback for teaching and learning developments.Figure 3 illustrates questions that elicited a high overall percentage of incorrect answers and/or questions not attempted.
On examining these questions in more detail, it was evident that there was a similar format underlying their composition.For example, with questions 8, 10 and 26, students had difficulty in translating text  Only 10% of students did not attempt question 10, on the chemical structure of the amide functional group, but 80% of the answers were incorrect.This shows that the majority of students is unable to recall the correct structure of an amine bond but may possibly understand that it contains a nitrogen atom.Question 8, which also requires the translation of text into a chemical structure, shows a similar result in the diagnostic test with 12.7% and 2.5% correct answers from Year 3 and Year 1 students respectively.There was a significant difference between the two cohorts on this question (p ≤ 0.0008).In the category Acids and Bases, question 26 requires the translation of a name into a chemical formula with the added complexity in this problem regarding an acid base reaction and balancing of the equation.The number of Year 1 students who answered this question correctly was higher than for Year 3: 19% correct answers from Year 1 compared to 4.2% from Year 3 (p ≤ 0.0001).This general topic is covered in the A-level syllabus so illustrating that the recall ability of these Year 1 students is higher than that of Year 3 students.However, a large number of students  overall who attempted this question answered it incorrectly (81%).It may be that the time allocation for this question was insufficient for students to work through the problem, or that they simply did not know how to apply the basic concepts.In the Organic Chemistry section Q34 there were 5.2% correct answers.
Q34: Write the structure of the product that is formed from the following reaction: Interestingly, there was a greater number of attempts (76%) but a higher level of incorrect answers for this question (70% of students).There was an increased complexity in Q34 as it was testing understanding of acid/base chemistry and also required the students to draw the chemical structure of the final product.Questions that require the translation of text into a chemical structure demonstrate that, while many students may have knowledge of the structure and of the chemical terminology, they have difficulty with the translation between the two aspects and are therefore likely to make mistakes.
In the calculations category, questions 30 and 32 show a larger number of students who simply did not even attempt to answer the questions.For Q30, 53% of the students overall did not attempt this question with only 4.3% of the cohort giving correct answers and over 42% of the cohort answering the question incorrectly.
Q30: To prepare 250 mL of 0.400 M H 2 SO 4 requires that _____________ mol of H 2 SO 4 is dissolved in enough water to make the final volume 250 mL.
To solve Q30 a student would need to carry out the following operations: 1. Recognise Molarity which is given as 0.400 M.
2. Recall the relationship between moles and volume: 0.400 M represents 0.400 moles per litre of solution.
3. Calculate the ratio of 250 mL to 1 L, which is one quarter.
4. Divide the number of moles (0.400) by four to obtain 0.100 moles as the final answer.The high percentage of students not attempting the question demonstrates either that the students were not able to break down the more complex information load in this question into smaller "chunks" or steps as indicated, or that they required more time.Q32 shows a similar trend with 39% of students not attempting this question and only 4.7% correct answers.
Q32: When 100 mL of 0.1 M hydrochloric acid are neutralised by 0.01 M sodium hydroxide, what volume of the base is needed?
These questions, which are multistep in nature, elicited a higher percentage of students who either lacked understanding or could not break down the problem into steps in order to progress to the final answer.Barriers relating to the learning of chemistry may arise from the language, symbols and chemical structures used.Rather as in learning a language, a lack of understanding of specific terminology creates a barrier to learning when students subsequently encounter such terms.
Students must first understand how to convert a symbol into its meaningful representation (Robinson 2003).Bodner & Anderson (2008) examined the difficulties faced by a good student in learning organic chemistry.Although the student attended class and read the recommended textbooks, there was a significant barrier to learning.This was attributed to the use of chemical structures in that it required "conscious thought and effort to interpret these structures correctly'.The concepts and theories were explained in both the lectures and textbooks; however, it was the format of the explanations, which consisted of lines, letters and curved arrows, that did not correspond or link into meaningful reality for the student.
The second key area of difficulty demonstrated from the results relates to questions which are multistep in nature.Questions that require students to retain a number of pieces of information at any one time in order to process and solve a problem have the potential to overload the working memory for some students.Most adults have a capacity of seven 'chunks' or units of information, which are defined by the individual based on their previous knowledge (Miller 1956).It is this capacity that is the rate-determining step in much learning.It has been shown that when a task is greater than the working memory capacity, effects on performance are dramatic (Reid 2008).
For chemistry problems information is not simply held in the short-term memory; it must be processed and manipulated.Chemistry uses specific terminology; a failure to understand will also impact the available working memory space.Novice chemistry learners can use up considerable working memory space very quickly leaving little space for understanding and processing (Hussein & Reid 2009).Johnstone & Kellett (1980) have shown that in order for a student to complete a given task successfully, the amount of information the student must retain must be less than the capacity of the student's working memory.When the working memory is overloaded, there is consistent evidence to show that learning decreases and the student fails even to tackle the task (Johnstone & El-Banna 1986, 1989).It has been shown that curriculum intervention to reduce working memory load does impact positively on performance (Danili & Reid 2004).This involves a stepwise approach in presenting information in order to reduce the topic into more manageable bite sizes.The syllabus was not changed, but simply decreasing the amount of material which students needed to process at the same time had a significant effect.Without this approach there is a danger of content overload, whereby students cannot distinguish between the important message and the additional information included to illustrate the bigger picture.

Future implications
The key areas of drawing chemical structures and reducing working memory load, highlighted as problematic for both Year 3 and Year 1 undergraduate students, need to be considered in the design and delivery of the curriculum.This diagnostic test is an initial study that has provided some insights into the level of students' knowledge of fundamental chemistry.The question items reported have not undergone a detailed validation in terms of identifying the specific factors involved that account for significant incorrect answers to specific questions.
The stepwise chunking down of material can significantly help students to learn chemistry and not overload their working memory.Using prelectures has been shown to be a successful approach in assisting the less well-performing students in Chemistry and in tailoring the delivery to meet their needs (Sirhan et al. 1999, Sirhan & Reid 2001).The better performing students in this research were involved in assisting the students who were struggling and, during this process, reinforcing and clarifying their own learning.This "develops reciprocity and cooperation amongst students", one of Chickering & Gamson's seven principles of good practice in Higher Education (1987).It is also necessary that students draw out chemical structures more frequently during lectures, workshops and practicals so that the process becomes routine and helps ensure familiarity.This is particularly important where students are given handouts of PowerPoint presentations and the drawing of structures by hand is not required.
The diagnostic test can be used as formative assessment with peer marking of the test directly afterwards.This provides immediate feedback and an opportunity for each student to evaluate their own learning.Analysis of the questions will indicate any students who have difficulties in specific areas of fundamental chemistry.This strategy of formative assessment with feedback is aimed at improving and accelerating learning (Nicol & Macfarlane-Dick 2006).The purpose is to enable students to selfregulate and take ownership of their learning.Feedback is essential in order to give the student information on their present state and how this relates to their goals and expected standards.The student then occupies a central role in the feedback and evaluation process.Future investigations will focus on more in-depth analysis of the question items to help elucidate the factors affecting students' understanding of fundamental topics in chemistry and their solving of problems.

Conclusion
This chemistry diagnostic test is a simple-to-use tool for understanding and evaluating student learning in chemistry.Its format allows for a variety of chemistry question styles and provides timely feedback even when used with large cohorts of students.The test highlights specific areas of fundamental chemistry which may be poorly understood, or lacking, in a cohort of students and therefore can be targeted directly in teaching activities and feedback.

Figure 1
Figure 1 Year 3 and Year 1 MPharm Students Diagnostic Chemistry Test Performance.
structure or recalling chemical structures from nomenclature.Q8: Nitrous acid has the chemical formula HNO 2 .Write its structural formula.Q10: Draw the chemical structure of an amide functional group.Q26: Write the net equation for the reaction: sodium bicarbonate with hydrochloric acid.

Figure 2
Figure 2 Students' performances in the different categories of questions.

Figure 3
Figure3Diagnostic test questions with high percentage of incorrect answers or un-attempted questions.

Table 1
Percentage of correct answers illustrating good performance on basic chemistry questions.

Table 2
Percentage of correct answers illustrating better performance from Year 3 students.