黑料不打烊

In many countries, take into account the average progress their students make on standardised tests relative to their peers. This is sometimes expressed as the value-added or progress-score of a school. The national databases containing students’ standardised test scores have also been used in educational research for studying or

In this blog post, I demonstrate how analyses of relationships between achievements can reveal insights on how entrance-level scores relate to the predictability of learning progress. I then examine how this causes potential biases in school evaluations that rely on average progress scores.

Each year in the UK, the relationship between students’ key stage 4 (KS4) scores (taken at ages 15–16) and their key stage 2 (KS2) results (from ages 10–11) is calculated. This allows each student’s KS4 score to be compared with the average KS4 scores of students who had similar KS2 achievements. Averaged deviations from expected achievements on a school combine into a school’s , which is used as a metric to compare school performances.

Figure 1 shows the average relation between KS4 English and KS4 mathematics scores as a function of KS2 attainment.

Figure 1: KS4 English and maths vs KS2

Figure legend: Average Relation between KS4 English 2023 and KS4, Mathematics 2023 scores with KS2 attainment scores. Numbers published by . The data are compared with models using standard statistics parameters as described in the text. First, realistic standard statistic parameters were obtained from the best linear match around KS2 = 105 and using for the correlation r_xy reported values of 0,55 for English and 0,70 for . The other values for Average (Standard Deviation) in the Model curves were: KS2: 105 (6,16); KS4_English: 5,0 (2,1); KS4_maths: 4,47(1,965). Note that s_x and s_y appear as s_y/s_x in the formulas and have no individual meaning. The same obtained from the linear model were applied in the nonlinear model. For comparison free polynomial fits of the data using Excel are depicted.

Correlation values of 0.55 for English and 0.70 for mathematics have been for these relationships. As the correlation value squared reflects the prediction percentage, KS2 outcomes predict for 30 per cent the variation in students’ KS4 outcomes for English and for 50 per cent the variation in KS4 mathematics.

However, the relationship between average KS2 and KS4 scores show clear deviations from linearity for both subjects. Since more variation in outcomes flattens a curve, whereas more predictability steepens a curve, these curves could depict that the ability of KS2 scores to predict KS4 outcomes varies across levels of achievement, with the higher scores being more predictive.

To understand how this nonlinear trend would affect Progress 8 evaluation, a model description that can be tested on the data would be helpful. In the framework below (figure 2), I translate my thoughts into a mathematical relationship. I begin with the formula for a linear relationship and then adjust this by normalising predictability on achievement level.

Figure 2: Linear model based on linear regression

Figure 2 illustrates how the model’s relationship closely aligns with the average relationship between KS2 and KS4 English. For mathematics some disparity between model and data remains. Still, it can be observed that prediction from the nonlinear model clearly provides a much better data approximation than the linear model.

Based on first derivatives, it can now be estimated that the predicted values of KS2 scores are about 1.5 times lower at KS2 = 95 compared to KS2 = 115. Consequently, this would correspond to predictability percentages ranging from 19 to 44 per cent for English and from 31 per cent to 71 per cent for mathematics.

What does this mean for Progress 8?

‘Variations in the deviation of expected performance will be much smaller for schools that select students with higher KS2 scores.’

Variation in KS4 outcomes that are not predicted by KS2 scores must be affected by other factors. Therefore, variations in the deviation of expected performance will be much smaller for schools that select students with higher KS2 scores. When in addition students at selective schools are more privileged in other ways, profit doubling would ensure not only high KS4 achievement but also more frequently above expectation. This trend is seen in the

In contrast, schools that admit students with lower KS2 scores will see outcomes influenced more strongly by other factors, including school quality, but also social demographic factors. As a result, variation for KS4 and Progress 8 outcomes will be greater. This could very well explain why schools with both a lower average entrance level and a less privileged student population have lower Progress 8 scores, as reported for

It is important to note that if the full achievement range is not covered with sufficient datapoints, such as in datasets from individual schools, deviation from linearity usually is not visible. However, this will not affect that it is of importance for studies in which comparisons are made on the school level.

Considering all these points, one should be aware that focusing on average deviations from expected performances, as done with Progress 8, can be misleading. Scientifically, it could be more informative to compare how achievement scores relate to each other for different schools or for different sociodemographic groups.