The SAT is to standardized testing what floppy disks are to data storage.
As new AI tools challenge traditional approaches to probing student knowledge and enable new ways of administering and scoring tests, companies that provide some of the most popular standardized tests are rethinking their offerings.
For example, ETS, one of the oldest and largest organizations in the field of standardized testing, has moved away from traditional college admissions exams like the SAT to focus on new approaches that measure students’ skills and persistence.
It’s been a tumultuous time for academic testing in general, and for ETS, a 75-year-old nonprofit, in particular. During the pandemic, at least 1,600 colleges and universities decided to make the SAT and other admissions tests optional, at least temporarily, due to concerns about equity and access. And earlier this year, ETS SAT administration canceled College Board spokeswoman Holly Stepp said the organization has transitioned to a fully digital format and “now develops and administers SAT- and PSAT-related assessments in person.”
ETS began a rebranding in April to focus on “people solutions” rather than just academic testing, then scaled back to recalibrate and offered many of its employees buyouts earlier this year. Lay off 6% of employees Last September.
“The assessments that ETS offers in the future will be more behavioral than cognitive,” says Kara McWilliams, vice president of product innovation and development at ETS. “That means creating experiences that measure user behavior, not answers to questions,” she adds. “So we want to look at things like perseverance, and when we think about how to build these assessments, [assessment] In our experience, we create nudges. [so] “Did you ask for a hint? Did you reach out to a friend? Did you ask for more time? So what actions did you take to get to the answer? It doesn’t matter what the answer is, it matters how you got there.”
One example of that effort is the group’s new focus. Skills for the future This is a collaborative effort with the Carnegie Foundation for the Advancement of Teaching to rethink assessment methods.
The goal of the effort is to move away from having students drop all their work, sit in a room, and answer questions for a few hours, says Timothy Knowles, president of the Carnegie Foundation. Instead, he says, the group is experimenting with using data schools have on students, including from after-school activities like sports, clubs, and internships, to measure and track progress in skills like communication, collaboration, and critical thinking.
“The idea is to build an insight system that’s useful for kids, families and educators,” he says, “so we can understand where people are in terms of developing skills that are predictive of success. So we’re figuring out how to visualize this in a way that’s not punitive or problematic for kids.”
Schools and school systems already have a wealth of data that’s not being used much, he said. The question is, “Can we look at that data in different ways and infer from that data how well young people are mastering certain skills?”
The effort has partnered with education leaders in five states — Indiana, Nevada, North Carolina, Rhode Island and Wisconsin — to help pilot test the approach starting in January, Knowles said. ETS and Carnegie officials plan to use the new form of AI to review and tag existing student work, analyze state education data and run interactive assessments, though not all of those uses will be ready by January.
But experts urge caution, especially when AI is used to analyze data or write test questions.
“We still have a lot to learn about whether bias is built into the use of AI,” said Nicole Turner Lee, director of the Center for Innovation at the Brookings Institution. “AI is only as good as the training data. If the training data is biased toward advantaged students who have more resources than students in disadvantaged schools, it will hurt them.”
She’s controversial 2020 ExperimentAs the pandemic peaked, many schools were closed and operating remotely, and many students were unable to take the end-of-year in-person International Baccalaureate exams, so the research group decided to build a model to predict student performance based on historical data.
“They developed an algorithm that basically predicts which schools are more likely to produce tertiary-level graduates,” she says.
Thousands of students have voiced dissatisfaction with their result scores, and some governments have launched formal investigations. “The algorithm itself didn’t take into account the location of the school or the resources of the school,” Turner-Lee says.
The researchers said that ETS staff She was invited to give a lecture. At a recent event, she shared her views and concerns about approaches using AI for testing and assessment.
“Think about how hard we’ve worked to address inequities in standardized testing,” she says, “and we have to be cautious about going all in because the datasets we use to train AI are themselves likely historically biased.”
Other test providers are experimenting with using AI to create new kinds of test questions.
Next year’s Programme for International Student Assessment (PISA) exams (international tests measuring reading, mathematics and science literacy among 15-year-olds) will include: A new kind of “performance task”” is designed to see how students approach problems and is graded by AI.
ETS’s McWilliams said the past year has “changed” thinking about AI in testing.
Whereas last year the focus was on using AI to create traditional multiple-choice questions, now, she says, “What I’m really focused on is generating content dynamically on the fly. Rather than multiple-choice questions, I’m focusing on more experiential tasks where individuals can most meaningfully demonstrate their knowledge and abilities.”
One example is a new AI tool called Authentic Interview Prep, which uses AI to help people hone their job interview skills.
“A lot of people get nervous when they have an interview,” she says. “So what we’re trying to do is create an experience that helps people understand how to have a more meaningful interview. The AI does a lot of different things, like giving me feedback on my tone of voice, the speed at which I speak, the eye contact I make with you, and then instantly you get haptics on your watch that tell you, ‘Cara, calm down. You’re talking too quickly,’ or, ‘Make more eye contact.'”
Of course, such tests aren’t for college or grad school admissions; they’re a different kind of assessment than the SAT. She says the SAT will likely have some role for the foreseeable future. “What I’m thinking about now is, ‘What content can I use to inform the experiences that people have every day?'”
