Report on Assessment Goals and Methods
Department of Physics and Astronomy, October 2008
Student Achievement Goals
We expect that after completion of a physics or astronomy major,
- Students will understand the important conceptual models used in the core subject areas of physics and demonstrate their ability to correctly draw logical conclusions from these models and use them to make accurate quantitative predictions in realistic situations.
- Students will understand a broad range of experimental and data-analysis techniques and demonstrate their ability to use these techniques in both designing and conducting scientific experiments and observations.
- Students will develop certain fabrication skills useful in the field, such as the ability to design and construct electronic circuits and other experimental devices.
- Students will demonstrate their ability to read, understand, and critically analyze the physical ideas presented in published textbooks and journal articles.
- Students will demonstrate their ability to present information clearly, logically, and critically, both orally and in writing.
- Students will demonstrate both understanding and the practical application of the ethical standards implicit in science, such as appropriate attribution of ideas, good recordkeeping, and truthful presentation of data and conclusions.
- Students will be fully prepared for graduate study in physics or astronomy and/or careers in scientifically oriented jobs in the public or private sector.
The courses in our major are explicitly designed to address these goals. Some of the courses focus strongly on developing one or two specific skills in the list above, while others (particularly the introductory course) focus on providing students with a solid foundation in most of the items on the list.
Assessment Tools and Methods
Generally, we test many of these skills using in-class testing. Exams in our classes are not generally knowledge-based so much as skill-based: even in our introductory course, students are given problems asking for essay solutions that require application of logical reasoning skills, quantitative modeling skills, and technical reading and writing skills to complete successfully. They therefore will not get a good grade if they lack these skills at the appropriate level. Students at all levels practice technical writing skills by writing lab reports and problem solutions.
Specific courses use other evaluation methods as well. In the introductory course, we test students' capabilities using standardized tests, such as the Force Concept Inventory and the Basic Electricity and Magnetism Assessment. These tests focus on students' ability to reason using core physical principles, and allow us to compare students' performance both before and after the course calibrated to other students nationwide For the FCI in particular, we have data for many years that show that our students consistently perform at the very top of students nationwide and display a large value added with the introductory course compared to comparable courses nationwide. Other courses involve student-initiated projects which end in oral and written reports. In some courses these reports involve peer evaluation and feedback.
Students practice (and we evaluate their mastery of) experimental techniques in a number of courses. In the introductory course, we use group-interview methods to evaluate students' use of experimental techniques and statistical analysis, and groups are fundamentally not allowed to leave until they have displayed skills at the appropriate level. There are similar laboratory components to our sophomore-level Atomic and Nuclear Physics and Electronics courses. Students learn advanced experimental techniques in a junior-level course (Contemporary Experimental Physics) designed explicitly for the purpose of preparing them for senior-thesis research, Experimental design methods are modeled in all of these courses, and students are expected to contribute more and more to the actual design of experiments as they go on in the major. Fabrication skills are also emphasized in the Electronics and Contemporary Experimental Physics classes: in both courses, students have required projects involving the design and fabrication of devices.
Our senior comprehensive exercise has three major components (required of all majors): a senior seminar class, taking the GRE physics subject exam, and preparing a senior thesis. Using a journal-club format, the seminar explicitly teaches technical reading and oral presentation skills (with explicit instruction and feedback). The GRE subject exam tests students' mastery of reasoning and modeling skills as well as subject knowledge, and provides a nationally standardized means of comparing all of our graduates to graduate-school-bound students from other institutions. Since we just instituted the GRE subject exam requirement a few years ago, we are still trying to determine what level of performance would be appropriate to demand from all of our graduates, but our intention is to require a certain minimal performance on this exam to graduate.
The senior thesis project itself involves nontrivial experimental or theoretical research (usually in a faculty members' research program), a 20-to-50-page paper presenting the results of this research, and a short oral summary of the research to faculty and students. The paper is carefully read and graded by the student's thesis advisor and at least one other faculty member, and is discussed by all faculty at a departmental meeting. It is expected that seniors will work on their theses during their entire senior year (perhaps even using research done the summer before). While students can elect to do a "library thesis" that does not involve research, no major has elected this route in a number of years. The students' research advisor in particular gets a very good sense of how well his or her student has mastered the skills and techniques that they should have learned as majors, and we encourage students to display most if not all of the listed skills in their paper and presentation.
Finally, we regularly solicit feedback from alumni about the value of the skills they have gained at Pomona. This feedback is sometimes informal, but we have conducted formal alumni surveys for departmental reviews. The alumni generally report being very satisfied with the utility in their lives of the skills they have learned from our department.
Making Students Aware of these Goals
The best way to make students aware of these goals is designing course requirements and grading structures so that mastery of the appropriate skills is necessary to get a good grade in the class. It is well known in the physics education research community that mere statements in catalogs, on web sites, in syllabi or during class do very little to motivate (or even attune students to recognize) the skills that they need to learn. Consciously and carefully designing course structures to require and reward the behaviors that you want them to master, in contrast, is very effective. We try to do this very explicitly and carefully when developing both courses and the sequence of courses required in the major.
The second most important way we make students aware of these goals is to mentor students while they are doing research. Doing real research with faculty provides an ideal environment for explicitly discussing as well as modeling and insisting upon excellent analytical and experimental technique, good reporting skills, and sound scientific ethics. As a department, we are currently trying very hard to involve all majors in research (either here or off-campus) during one of the three summers between their academic years at Pomona.
This being said, we could probably do a better job of explicitly discussing these skills in the catalog, on the website, and in course syllabi. A few of these skills are explicitly discussed in the introductory physics syllabus and text, and some are briefly mentioned in the catalog. Preparing this document is an important first step in clarifying and coming to consensus about our goals for the major, and will give us some language to incorporate in future revisions of catalog copy, web information, and syllabi.
Toward the end of improving our teaching and our assessment of student achievement with respect to these goals, we are planning over the next few years to focus on analyzing and discussing our department's success with regard to one or two of the listed goals per year. Results from our self-assessment will appear in the annual reports prepared by the chair.