Reviewers' Notebook: ETS Major Field Test
From CSWiki
When the combined Department of Mathematics and Computer Science formulated its initial Computer Science Assessment Plan in 1997, we hoped to take advantage of the ETS Major Field Test in Computer Science. Three elements from the plan seem relevant here:
- The ETS Major Field Test in Computer Science will be used to help evaluate majors' understanding of problem-solving paradigms and fundamental principles.
- Correlations will be run between scores on the ETS Computer Science Major Field Test and student grade point averages (GPAs) in their Computer Science Major to clarify expectations and validity for both the ETS test and student grades.
- Scores on the ETS Computer Science Major Field Test will be reviewed in accordance with the recommendations of the Liberal Arts Computer Science Consortium, using as a base the article, "A Revised Model Curriculum for a Liberal Arts Degree in Computer Science" by Henry M. Walker and G. Michael Schneider, Communications of the ACM, December 1996, pp. 85-95. Following those recommendations, strong weight will be given to scores in programming methodology and in theory and computational mathematics, with only moderate weight to software systems, computer organization, and architecture.
As part of this effort, the College agreed to cover the costs for administering the ETS Major Field Test for each CS major. In addition, the taking of this test was listed as a requirement of the CS Major in the college Catalog for four years: 1997-1998 through 2000-2001. The test was actually given three times in 1998, 1999, and 2000.
Results on the ETS Major Field Test in Computer Science
The following commentary comes largely from a 1999 report, supplemented by results of the test administered in June 2000.
ETS reports a composite score for each student and both subfield and composite scores for the entire group. Nine computer science majors took the test in 1998, ten took the test in 1999, and 13 took the test in 2000. The composite results for individuals are as follows:
| 1998 Test | 1999 Test | 2000 Test | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Low | Median | High | Low | Median | High | Low | Median | High | |||
| Individual Scores | 79% | 87% | 97% | 70% | 83-90% | 99% | 23% | 96% | 99% | ||
While the scores were fairly evenly distributed for the 1998 exam, the 1999 results indicated four students scoring in the 70s and five in the 90s. Of the 9 computer science majors taking the test in 1998, 4 were juniors, and 5 were seniors. Of the 10 computer science majors taking the test in 1999, 7 were juniors, and 3 were seniors. In 2000, the two of the three bottom scores were from juniors and the third did not complete a computer science major. Among graduating seniors, the lowest score from the 2000 test was at the 80% level.
The departmental scores were as follows:
| 1998 Test | 1999 Test | 2000 Test | |
|---|---|---|---|
| Percentile | Percentile | Percentile | |
| Ranking | Ranking | Ranking | |
| Composite | 99% | 99% | 96% |
| Programming Methodology | 99% | 88% | 92% |
| Software Systems | 75% | 77% | 96% |
| Comp. Org. & Org. | 87% | 93% | 99% |
| Theory and Comp. Math. | 99% | 99% | 89% |
Analysis
The department's analysis of the 1998 and 1999 test results indicated:
While the raw 1998 scores indicate that our students got a relatively low percentage of the problems correct in the areas of software systems and computer organization, the comparative percentile scores are quite reasonable in these areas. Our overall curriculum emphasizes theory and computational mathematics, and in this area our composite institutional rating was at the 99th percentile, with a very small range. The pattern is quite similar for the 1999 test, although our 1999 students did not do quite as well on the "programming methods" assessment indicator. We noted that the fraction of our juniors taking the test in 1999 was considerably higher than in 1998.
With such core courses as CSC 211, 213, and 341 being offered in alternate years, some unevenness should be expected in the preparation of our students in these areas. Also, liberal arts colleges, such as Grinnell, typically emphasize computer architecture less than other types of schools, and this emphasis is reflected somewhat in the test results.
Even so, architecture scores were lower than we might hope, and some students complained that the test used different terminology than the textbooks. Mr. Walker confirmed this after a survey of the two most commonly used textbooks. While our courses follow standard terminology, this does not seem to provide full preparation for students taking the ETS Major Field Test or the GRE. Thus, we considered ways in which we might supplement courses to provide more effective ways to prepare students for the GRE. One possibility would be to provide special sessions for coaching or practice tests.
We found no particular correlation between the scores of individual students and our perceptions of their abilities. Generally, a student will perform better in the senior year than the junior year, but both juniors and seniors had among the best and among the lowest scores.
While we considered to what extent the ETS Major Field Test may help students prepare for the GRE, we did not reach a conclusion about this. The GRE is apparently much harder than the Major Field tests, but the tests are developed by the same testing committee, and they cover similar topics.
Dropping the ETS Major Field Test
Finally, in 2000, the CS faculty, in consultation with our mathematician/statistician colleagues and with Dean Swartz, decided to drop the ETS Major Field Test from our CS Major requirements and from our Assessment Plan. A note to Dean Swartz in Summer 2001 summarizes our conclusions:
When the CS faculty developed its initial strategy for assessment several years ago, we had hoped that the ETS Major Field Test in Computer Science would provide helpful information to assess our program and guide further development. Unfortunately, as discused in our August 1999 Assessment report, this instrument proved largely worthless.
- We found no particular correlation between the scores of our students and our perceptions of their abilities from classes, independent projects, and research experiences.
- While students generally performed better in their senior year than in their junior year, both juniors and seniors had among the best and among the lowest scores.
- Our composite institutional ratings were consistently at or near the 99th percentile, possibly showing the strength of our program, but not giving helpful guidance.
- As a liberal arts college, Grinnell emphasizes theory, principles, and problem solving, and the ETS test subscores simply document this expected focus.
- Similarly, Grinnell deemphasizes computer architecture and low-level programming tricks, and the ETS test also documents this expected result.
- Since Grinnell offers core courses CSC 211, 213, and 341 in alternative years, student preparation in these areas is uneven, and subscores do not distinguish between those students who have taken a course and those planning to take it later.
- Very few schools of Grinnell's type or stature administer this ETS Major Field Test in Computer Science, so comparative data from the test is not clearly relevant or applicable.
- At the Spring 2000 meeting of the Iowa Undergraduate Computer Science Consortium, the session on assessment indicated similar experiences to the ETS test from schools across Iowa. All schools using the test found results did not correlate with perceived student ability, and subject subscores only confirmed known emphases of the curriculum.
In short, while the CS faculty had hoped the ETS Major Field Test in Computer Science would yield helpful information, the results provided no new insights of any kind. Without any identifiable benefits, the administration of the test was discontinued in Fall 2000.
