IMSA ORE

SAT Scores, Weighting, Prediction and the Student Descriptive Questionnaire

ETS Plans to Weight Scores

In September 1999, Educational Testing Services (ETS) announced that it would be developing two weighted versions of its SAT score in addition to its traditional unweighted one. The first will utilize only socioeconomic and educational factors, while the second will add racioethnicity. While such a move may have political implications (e.g., Affirmative Action-related concerns), the logic behind improving the predictive ability of the test is straightforward. If the purpose of the test is, for example, to predict success in college as measured by freshman GPA, then it should be easy to see whether the weighting enhances such predictive ability.

Predictive Value of the SAT at IMSA

At the Illinois Mathematics and Science Academy, we use the SAT I (both mathematics and verbal scores) as one part of the screening criteria for the incoming students of our residential high school (grades 10-12) program, which serves Illinois students talented in mathematics and science. While the SAT is not designed for predictive value at the high school level, the academic level of IMSA students and their learning environment are college-like (e.g., highly intelligent students from all parts of the state in a residential setting).

The entering composite SAT I score is a useful predictor, explaining 12.2% of the variance in IMSA GPA. However, prior school GPA, as shown at right, predicts a much higher 32.8% of the variance in overall IMSA grades. The SAT mathematics score (SAT-M) which by itself would predict 11.6% of the variance, adds 5.0% to the variance predicted by GPA in a regression analysis (hence, 37.8% total). Four other statistically significant predictors would add an additional 4.3% (gender, SAT verbal score [SAT-V], and two of the subscores [communications and science] from the student portfolio ratings).

Model		R^2
1	GPA	.328
2	GPA, SATM	.378
3	GPA, SATM, S	.402
4	GPA, SATM, S, COM	.415
5	GPA, SATM, S, COM, SATV	.418
6	GPA, SATM, S, COM, SATV, SC	.421

The best prediction provided by the entering SAT is for IMSA mathematics grades only, with the SAT-M score predicting 21.1% of the variance in scores. However, prior GPA predicts 21.9% of the variance in IMSA mathematics grades. The SAT-M score adds 13.5% to the variance predicted by GPA (hence, 35.4% total). Other portfolio subscores, and whether or not the student was a member of an underrepresented group (i.e., Black, Hispanic, or Native American contrasted with White or Asian), were nonsignificant when prior school GPA was entered into the regression equation.

As a predictor of graduation from IMSA, SAT scores are minimally useful. In a recent study, SAT-M predicted only 2.1% in the variance in graduation rates, whereas prior school GPA predicted 8.7%. Inasmuch as these are academic predictors and withdrawal motivations may have substantial nonacademic components, such lower levels of prediction are not surprising.

Please note that all of the above and following analyses inherently are limited to students accepted at IMSA. If we wanted to test the actual predictive value of screening by the select predictors above, we would admit applicants across a full range of scores for each variable and track their success. In this way, we could determine the level of success for those who normally would have been eliminated from enrollment consideration by their lower score on a given variable. An additional consideration is that by restricting the range of scores, we reduce the potential magnitude of the relationships found.

SDQ Data and Their Potential Use as SAT Score Weights

The Student Descriptive Questionnaire (SDQ) is administered during the SAT in order to provide information to colleges about the student's background, interests, plans, and activities, along with their test scores. The information provided by the SDQ includes demographic variables such as sex, race, household income, and parental education level, which are the data that could be used for weighting. It also includes academic variables such as student grade level, GPA, and intended field of study, as well as literally hundreds of other variables.

Data from the SDQ seldom is analyzed by schools for two reasons. First, the data cost $250, so the cost is prohibitive. Second, the analysis of a database with almost 500 variables is a complicated task requiring technical skills, statistical software, and a lot of time.

An additional problem is that such data does not provide a student identifier so that a researcher can match the SDQ database to our students. Therefore, our analysis was limited to examining the SDQ data as predictors of SAT scores for IMSA students who took the test during the 1997-1998 academic year. This is not the same as our specific interest, the use of SDQ variables as weights for the SAT scores in the explanation of student success. However, an examination of the predictive value of SDQ data for SAT scores will, in turn, provide valuable information for determining their potential as valid SAT weights for the prediction of overall academic success (particularly with respect to GPA and the completion of academic goals).

The depth of these analyses were limited by the sample size of 209, since the use of more than two independent variables in a single analysis can reduce subgroup size to excessively small levels. Factorial ANOVA techniques were used when possible to examine contrasts where multiple predictors were of interest. However, the creation of a multiple regression prediction model using the above and other information was infeasible. Incidentally, SDQ categories were collapsed when appropriate (e.g., for racioethnicity, Black and Hispanic were combined).

Findings from the SDQ Information

Two hundred nine students, comprised of 100 class of 1998 seniors and 109 class of 1999 juniors, took the SAT during the academic year. As expected, these students were demographically representative of the academy as a whole. The table below lists the racioethnic and gender characteristics of the test-takers, with IMSA overall enrollment percentage figures for the two classes in parentheses. Please note that 14 students did not report race.

RACE	n	Percentage	GENDER	n	percentage
Asian	60	30.8 (28.1)	Male	102	48.8 (50.1)
African American	14	7.2 (9.7)	Female	107	51.2 (49.9)
White	106	54.4 (52.3)
Hispanic	10	5.1 (6.4)
Other	5	2.6 (3.5)
Total Reported	195

The composite SAT scores ranged from 980 to 1600 with a mean of 1412.55. The SATV mean was 682.93, ranging from a low of 420 to 12 scores of 800. The SATM mean was 729.62, ranging from a low of 560 to 46 scores of 800. National and IMSA overall mean scores are reported in the table below.

SAT MEAN SCORES	1997-1998 IMSA	1997-1998 National
SAT-V	682.93	505
SAT-M	729.62	512
SAT	1412.55	1017

Significant differences in scores were found with respect to several demographic variables. We used the standard p-value of .05, meaning that a finding is not considered significant unless there is a 5% or less probability that the difference found could have occurred by chance. Significant findings are boldfaced. Each column also contains the eta squared (w2), which like the r2 used in the initial table gives the amount of variance explained by the relationship. For example, in the table on page 4, the difference between mother's education level for SAT-V explains a statistically significant but fairly small 7.2% of the variance in SAT-M scores, leaving 92.8% unexplained.

Males scored slightly higher on both SAT tests, but only the SAT-M differences were significant (and just barely, p = .035). This is consistent with most findings in the literature. The difference in SAT-M scores is smaller than usually found between genders, but given the high levels of the scores and the resulting ceiling effects, such smaller differences are expected.

As is consistently found in the literature, demographic differences with respect to race were highly significant. Again, the differences were smaller than those found in the literature, probably due to the high levels of the scores.

Parental education, as shown below, explained a small but usually significant amount of variance in SAT scores.

SAT MEAN SCORES BY MOTHER'S EDUCATION LEVEL	SAT-V	SAT-M	SAT
High School or less	633.89	702.78	1336.67
Trade School-Comm. College	662.08	703.33	1365.42
Bachelor or Grad level	693.05	735.97	1429.03
significance level (p-value)	.001	.004	<.0005
eta squared	.072	.057	.093

As seen from the table above and subsequent post hoc tests, SAT scores were significantly higher for students whose mothers had achieved a bachelor's degree or above than for either of the other groups (no significant differences were found between the high school and trade school groups).

SAT MEAN SCORES BY FATHER'S EDUCATION LEVEL	SAT-V	SAT-M	SAT
High School or less	646.36	706.82	1353.18
Trade School-Comm. College	658.18	718.18	1376.36
Bachelor or Grad level	694.77	733.20	1427.97
significance level (p-value)	.001	.084	.001
eta squared	.074	.029	.078

As shown above, p-values for verbal and total were highly significant. However, due to considerable differences in sample sizes, and unequal variances within those samples, a more conservative post hoc test was conducted. These tests produced marginally nonsignificant levels for comparisons between fathers who achieved a bachelor's degree or above and either of the groups that had not. However, when the cells were collapsed so that the bachelor-level and above group was contrasted with community college and below, the t-tests demonstrated that all of the differences were significant (p = .002, .034, and .002, respectively).

One unfortunate omission from the SDQ is a designation for whether each parent was born in the United States. Recent immigrants who had no access to education in their native lands but had the motivation to emigrate are very different from their American-born education-level counterparts.

The family income data is of questionable validity, since the level to which students are aware of their parents' income is suspect. Also, there may be a systematic underreporting, since students are aware of its ramifications on financial aid. This will be compounded if the proposed weighting is based upon such reporting, since students will be aware of the ramifications. Finally, some cost-of-living adjustment for regions of the United States also would be desirable.

Given such limitations, family income was a useful predictor of SAT scores, as shown below. In all cases post hoc tests showed that the under $50,000 group were significantly lower than the other two.

SAT MEAN SCORES BY INCOME	SAT-V	SAT-M	SAT
Under $50,000	653.96	706.67	1360.63
$50,000 - $100,000	687.44	735.00	1422.44
Over $100,000	709.79	742.77	1452.55
significance level (p-value)	<.0005	.003	.001
eta squared	.082	.062	.104

Furthermore, a two-way analysis demonstrated that the initially identified differences in SAT scores due to racioethnicity became nonsignificant when income was considered.

Incidentally, one of the most significant factors of score differences is the student's educational level. As demonstrated in the table below, 11th grade students performed significantly higher in both the verbal and math scores. In fact, juniors earned all seven of the perfect 1600 scores. Students planning to apply to colleges and universities that have more rigorous admittance procedures, or students seeking acceptance into early admission programs, often would need to apply prior to their senior year. Additionally, students earning a very high score on the test during their junior year would be unlikely to retake the exam. On the other hand, those that did not fare as well would, as expected, continue taking the exam in the hopes of raising their score.

SAT MEAN SCORES BY GRADE LEVEL	SAT-V	SAT-M	SAT
11th Grade	701.01	740.37	1441.38
12th Grade	663.03	717.78	1380.81
significance level (p-value)	<.0005	.004	<.0005
eta squared	.075	.039	.084

Summary

There are obvious reasons for considering the weighting of SAT scores according to educational and socioeconomic data. Racioethnic data also may add additional information. Of course, institutions often already factor such information into their screening of students, so such weighting may be redundant and therefore useless. Hopefully, SAT will devise a better assessment of family income, safeguard the accuracy of the reporting of other data that may be used in the weighting process, and provide all score variants to the educational institutions so that each can determine which SAT test variant is the best predictor for its students.