Testing conceptual understanding in introductory astronomy

Understanding students‟ prior beliefs about the nature of the Universe is a first step towards improving astronomy instruction. This article describes results from two diagnostic surveys testing understanding of astronomy concepts given to first, second and third-year St Andrews students taking astronomy and astrophysics modules. We highlight results pertaining to the phases of the Moon, the cause of the seasons, planet temperatures and properties of comets, and discuss possible underlying reasons for student difficulties. We find that some misconceptions remain at higher levels, and that new knowledge may be incorporated into prior beliefs without a substantial conceptual change. from the Solar System part of the level 1 AS1001 course in the first and the last lecture of this course (pre- and post-test design). The survey consisted of nine multiple-choice questions on planet temperatures, phases of celestial objects, cause of the seasons, properties of comets, common features of satellites, and common properties of planets. Students were asked to choose one or more answers. An additional question asked about prior astronomy school courses. To analyse the survey results, we determined moments of the distribution of total scores and gains and normalised gains for each question and averaged over the survey 8 . The Solar System survey covered only one part of the AS1001 course, and students rated only their overall confidence, so that it was not possible to determine students‟ confidence in particular incorrect choices. This led us to develop a conceptual survey covering all parts of the AS1001 course, including 11 questions on the Solar System, 6 questions on stars and elementary astrophysics and 5 questions on galaxies and cosmology. The survey (called Astronomy Conceptual Survey, ACS, in what follows) was developed using known misconceptions from the literature and discussions with course lecturers and tutors. The survey was validated by trialling it with postgraduate astronomy

This work investigates understanding of astronomy concepts for physics and astrophysics students at the University of St Andrews. We describe the development of two surveys, one administered to level 1 (first-year) students in an introductory astronomy and astrophysics course, the other given to level 1, 2 and 3 (first, second and third year) students enrolled in astronomy and astrophysics modules. We describe selected results from these surveys, their implications and possible future work.

Methodology
In 2008/09 and 2009/10, students completed a diagnostic survey (called Solar System survey in what follows) covering topics from the Solar System part of the level 1 AS1001 course in the first and the last lecture of this course (pre-and post-test design). The survey consisted of nine multiple-choice questions on planet temperatures, phases of celestial objects, cause of the seasons, properties of comets, common features of satellites, and common properties of planets. Students were asked to choose one or more answers. An additional question asked about prior astronomy school courses. To analyse the survey results, we determined moments of the distribution of total scores and gains and normalised gains for each question and averaged over the survey 8 . The Solar System survey covered only one part of the AS1001 course, and students rated only their overall confidence, so that it was not possible to determine students" confidence in particular incorrect choices. This led us to develop a conceptual survey covering all parts of the AS1001 course, including 11 questions on the Solar System, 6 questions on stars and elementary astrophysics and 5 questions on galaxies and cosmology. The survey (called Astronomy Conceptual Survey, ACS, in what follows) was developed using known misconceptions from the literature and discussions with course lecturers and tutors. The survey was validated by trialling it with postgraduate astronomy  Although these are small student numbers, these results may show that instruction does not necessarily lead to the revision of prior beliefs, but instead may lead to an inconsistent set of ideas. It would seem that misconceptions need to be explicitly challenged in order for students to Testing conceptual understanding in introductory astronomy students and staff. The survey was administered via WebCT (a virtual learning environment) as an anonymous survey to level 1, 2 and 3 students taking astronomy modules and to higher-level astronomy students on paper. Students were asked to choose one or more answers for each question, and to rate their certainty for each question.
To analyse the results, we determined moments of the distribution of total scores, performed t-tests to test for a significant difference in means between levels, and calculated effect sizes (defined as the difference in means divided by the average standard deviation) for significant differences. Due to small numbers, we grouped data from different levels where the distribution of responses was not significantly different. We determined discriminatory power of individual questions using the item difficulty index, item discrimination index and point biserial coefficient, and whole test reliability using the Kuder-Richardson reliability index and Ferguson"s delta 9 , and found the values of these indicators to be satisfactory. Of particular interest were those questions that were answered with a high degree of certainty but a low percentage of correct answers.

Student Survey outcomes
For the Solar System survey, in 2008/09, 59 students took the pre-and 48 students took the post-test. In 2009/10, the numbers were 53 (pre-test) and 41 (post-test).
For the ACS, 22 of 49 (45%) level 1 students, 17 of 22 (77%) level 2 students and 21 of 28 (75%) students in levels 3 and above completed the survey. Of those students at level 1 that completed the survey, 5 were astrophysics students, 15 are physics students and 2 are studying for other science degrees. At levels 2 and above, all students are studying for an astrophysics degree. The mean total score for the ACS Communication overcome them, in particular when these prior beliefs are deep-rooted in everyday experience. The ACS contained a slightly revised version of the above question (including an additional choice C), allowing comparisons with astrophysics students in levels 2 and above (see Figure 1). 21% (8 of 38) of higher-level students stated choice A is correct, two students in this group stated that both A) and B) are correct. All these students were certain of their answer, excepting one student who stated "somewhat certain".
Cause of the seasons Question 1 of the ACS was as follows:

Which of the following predominantly causes the seasons on Earth? A) The changing distance from the Earth to the Sun B) The tilt of the Earth"s axis causing one hemisphere to be closer to the Sun than the other C) The tilt of the Earth"s axis inducing changes in sunlight intensity and day length
While there is a difference in temperatures due to the axis tilt (choice B), it is exceedingly small (0.02 K), so cannot be the cause of the seasons. Figure 2 shows the results. No students chose response A), which is not shown in the figure. Results were similar for level 1, 2 and 3 students, leading us to sum the data for all levels. No student chose "not at all confident" as confidence rating. 37% of students chose B). The confidence of students choosing B) was quite high, but lower than for the correct response C).
These results coincide with the results from the AS1001 exam in 2010: almost all students were able to calculate correctly the difference in equilibrium temperatures at Earth"s perihelion and aphelion (the points nearest and furthest away from the Sun on its elliptical orbit), 4K for a difference in distance of 5 Testing conceptual understanding in introductory astronomy million km. In contradiction to this, 9 of 46 students (19.6%) stated in the exam that the seasons are caused by the hemispheres alternately being closer to the Sun, with all but one of these students getting full marks or almost full marks on the previous calculation.
How can one explain these results? Comins 10 argues that over-generalisation of a general principle based on our everyday experience may be an underlying cause: in this case, the principle would be that the closer we are to a heat source, the more heat we feel. This would agree with the level 1 results that pre-instruction, 54% of students state that Mercury is the hottest planet, not Venus.
When asked about common properties of all known Solar System satellites, post-instruction 32% of level 1 to 3 students combined (81 students in total) state "they are smaller than the smallest planet" (Ganymede is larger than Mercury), 20% state "they are practically spherical" (e.g., Mars" satellites aren"t), and 9% state "they have no atmosphere" (Titan does).
Here, the underlying cause may be the overgeneralisation of the properties of our Moon. Another underlying principle may be the belief in permanence of celestial objects: In a question on comet properties, pre-instruction only 31% of level 1 students agreed with the statement "Comets disintegrate after 100 to 200 passages close to the Sun" (2008/09 and 2009/10 data combined). Post-instruction, this number rose to 79%.

Discussion and outlook
Our results show that some misconceptions in astronomy persist to higher levels, and are not easily corrected. They also illustrate the importance of stressing conceptual understanding in astronomy instruction, and getting students to relate results of calculations to astronomy concepts.

Communication Testing conceptual understanding in introductory astronomy
We found that the distribution of responses to the ACS questions was similar across the levels, and that the increase in average score with increasing level was only modest. This may be due to the fact that higher-level astronomy courses do not stress the concepts tested in the ACS in detail.
Limitations of this work are relatively small student numbers, lack of free text explanations of reasoning or student interviews to uncover underlying reasons for incorrect choices, and the fact that this investigation was only carried out at a single institution. Knowing students" astronomy misconceptions can only be a first step in the ultimate aim of helping students to come to a correct understanding of the nature of the Universe. Future work includes extending this study to other institutions, gaining more insight into underlying reasons for misconceptions in astronomy, and developing and evaluating course material to target specific misconceptions.