What is scale anchoring?

Scale anchoring is a methodology which categorizes questions, in an assessment, based on the Ability level of students who are likely to answer them correctly. Each question in the assessment is anchored to a particular ability level on a scale of different ability levels. This process gives an understanding of the kind of questions that are likely to be answered correctly by only high ability students and those that are likely to be answered correctly by even low ability students. Once the questions are categorized, a team of experts analyzes them and determines the learning each ability group has acquired.

Need for Scale Anchoring

• Insights on Student Learning

Consider a post assessment report mentioning actual scores of a student, percentile scores (relative ranking based on actual performance) and even scaled scores(ranking considering the difficulty of the assessment). These would give a comparison of the student abilities but would not provide information about the concepts mastered at different ability levels for each skill. Even a skill-wise average performance report or a report indicating the Number of questions students answered per skill will not reveal exactly what the student knows or does not know makes sin that skill. Scale Anchoring would provide this information – what students of different ability know in each skill.

Imagine a school receiving a report indicating that a many of its students are of low ability. The school, though wanting to take steps to rectify this, might not know what to focus on to improve the students’ ability level. Now imagine if the same school had an understanding as to what the ability report means in terms of the actual learning the students have! They would find it extremely useful to direct their teaching strategies to rectify any learning inefficiencies. For example, a report stating that a student is unable to identify a number represented in a non-standard form can lead to a more focused remediation than a report just stating that the student is weak in number concepts.

• Indicator of Difficulty

The anchoring of a question at different ability levels is a better indicator of the difficulty of a question when compared to the average performance of the question.This is because a lot of factors can contribute to the performance of a question, especially in a multiple choice test. For instance, there may be a small percentage of lower ability students correctly answering a question anchored at a high ability level. This might increase the overall performance of the question itself. Two questions anchoring at different ability levels might have the same average performance. There may even be cases where some questions in the assessment do not anchor even at the high ability level. What this could indicate is that students answering questions that are anchored above their ability level (or questions that are not anchored) might have done so without understanding the concept thoroughly.

Scale Anchoring and ASSET

ASSET is an ideal assessment to use the scale anchoring methodology for two reasons. Firstly ASSET is a diagnostic test which aims to provide insights into student learning. Secondly, a large number of students (more than ten thousand per class) take the assessment which means that the data set is large enough to enable accurate anchoring of questions.

The steps involved in the scale anchoring for an ASSET class 5 Math paper are as follows.

• Setting Ability Ranges

Based on their scaled scores in the assessment, the students were ranked in order of percentiles. Four percentile ranges, in terms of ability, and their corresponding scaled score ranges were classified as Low, Intermediate, High and Advanced. Further, the number of students falling in each range was identified.  The total number of students in these four ranges was found to be 3960. The total number of students who took the test was 22927.

In the lowest ability range, which represents the percentile scores from 23 to 27 and the corresponding scaled scores from 424 to 433, there were 1021 students.

In the intermediate ability range, which represents the percentile scores from 48 to 52 and the corresponding scaled scores from 480 to 489, there were 1009 students.

In the high ability range, which represents the percentile scores from 73 to 77 and the corresponding scaled scores from 547 to 561, there were 979 students.

In the advanced ability range, which represents the percentile scores from 88 to 92 and the corresponding scaled scores from 615 to 642 there were 951 students.

• Anchoring of Questions

Each question in the assessment is then anchored to a particular ability range if a certain criteria of student performance on the question is met across these ranges. The questions are then grouped skill-wise which gives a better picture about the difficulty of questions across skills.

• Arriving at Learning Insights for Different Ability Levels

Once the questions are anchored and grouped skill-wise a group of experts view these questions independently and list down the knowledge and/or competencies required to answer each of them.

The table shows the learning insights arrived at for the Number Sense skill.

Conclusion

Scale anchoring is a very valuable tool for educators who can use it to shape teaching instruction in classrooms.

• How can schools use the reports based on Scale Anchoring?

Students in a school can be classified in terms of their percentile scores and the percentage segregation of students in each range can be compared with the overall percentage segregation. Getting to know the ability range where most of the students in a classroom lie can help teachers focus on concepts that will enable these students to move up to the next ability level.

School A has very few students cross even the 25th percentile. This school might be better off focusing on teaching those concepts which anchor at the 25th percentile before proceeding to advanced concepts. For the Number Sense skill, this would mean helping students interpret simple verbal statements involving subtraction and teach them basic subtraction without regrouping. A list of different questions anchoring at the 25th percentile across many assessments, along with the concepts understood by the students of that level, will help teachers decide on what to teach.

On the other hand, it might be better for School C to focus on concepts that anchor at the higher percentiles as their students already do well on the easier set of questions. This would mean helping students interpret complex framing of questions and going beyond algorithmic thinking.

This kind of systematic and focused approach is necessary in the current educational environment where time is limited and teaching needs to be efficient.

The post Insights on Student Learning from ASSET with Scale Anchoring appeared first on EI blog.