The transition from paper-based to digital national tests in Sweden creates a unique set of opportunities and challenges. For many students, ranging from year three of compulsory school to upper secondary school, the Swedish national tests are compulsory, and furthermore important in the Swedish context. The high-stakes tests are assessed by the teachers and the result is used as a component of grading. In this kind of transition, it is key to preserve aspects of validity, fairness and reliability of the tests, and also try to enhance them. The authors belong to a research and test development group at Stockholm University. In this text we give examples of possibilities and challenges in the transition towards digital national tests. The group, PRIM, is responsible for development and research of national tests in mathematics in Sweden. As test developers we recognize several possibilities of transitioning to digital tests.

For example, the inclusion of complex dynamic items in order to assess new cognitive performances. Thereby, there is an opportunity for a broader construct to be assessed. Along with this, we face several challenges connected to limitations within the test platform, students´ experiences with digital assessments and the students´ opportunities to present their mathematical reasoning.

A challenge in computer-based assessment, especially when introducing specific digital tools, is balancing mathematics assessment with students’ computer skills. This exploratory qualitative study aims to investigate aspects of student-tool-interac-tion, essential to consider when designing digital tools for assessment. Eight 15-year-old students worked in pairs with three digital test items employing interactive spreadsheet tools, followed by semi-structured interviews. The combined analysis of observations of students’ interactions with the items and the interviews indicates dif-ficulties in using basic spreadsheet functions – despite their conceptual understand-ing of the tools. The main implications of the findings emphasize the importance of integrating digital tools into learning situations before assessment, and not taking basic tool-related techniques for granted.

With the ongoing digital development of society, an increasing number of digital tools are being utilised when learning mathematics. Therefore, assessment situations should also incorporate digital tools. International large-scale assessments, such as the PISA study, are recent examples of this trend. However, a significant problem – still to be explored – concerns validity issues that emerge when new digital tools are included in assessments. In PISA 2022, digital items that mandated students to sort data and perform calculations directly in a spreadsheet tool were introduced. This study investigates the technical and conceptual aspects students must develop to use this tool productively, the training provided to students, and the implications for assessment validity. By analysing the training in relation to the tool use required to solve the task, we identify a critical conceptual tool-related aspect that is missing. Drawing on the instrumental approach, we argue that this aspect is fundamental to the emergence of an instrumental genesis. The gap between the training provided and what is needed to solve the task is interpreted as a potential compromise to assessment validity, which we further elaborate on in the discussion. This chapter addresses the broader issue of validity related to technology use in large-scale assessments.

Given the rise of large-scale assessment digitalisation with the increasing use of technology in the teaching and learning of mathematics, it has become important to design items that properly capture mathematical knowledge. However, empirical evidence is mixed concerning students prior experience with information and communication technology generally and mathematics performance. The aim of this study, therefore, was to investigate if there was an effect of prior experience with digital tools on students’ mathematical performance, by focusing on students’ prior experience with spreadsheets. 

To do this, we modelled the probability of a correct response to mathematics assessment items with and without spreadsheet functionality. Items with spreadsheet functionality include interactive features that simulate common spreadsheet operations such as sorting and basic calculations. We included a measure of students’ familiarity with spreadsheets to see if this changed the association between spreadsheet functionality and item difficulty. Therefore, we used two statistical approaches. First, we used Rasch analysis to assess the measurement of spreadsheet familiarity in the PISA 2022 student surveys. Second, using explanatory item response modelling, we used the presence of spreadsheet functionality of items and students’ spreadsheet familiarity to predict the probability of a correct response

PISA 2022 mathematics assessment items with spreadsheet functionality were significantly more difficult than items without spreadsheet functionality, and increased familiarity with spreadsheets did not alter this association. Students with greater spreadsheet familiarity tended to perform slightly less well on items with spreadsheet functionality, but this was not significant.

Students with higher spreadsheet familiarity may be at an disadvantage, raising concerns about assessment validity. These findings caution against assuming a positive association between prior digital tool use and mathematical performance. General spreadsheet familiarity does not appear to provide an advantage on items requiring spreadsheet functionality. This emphasises the importance of providing students opportunities to practice in advance with the specific digital tool used in the assessment.