Breaking the Magic of Automation and Augmentation in Swedish Classrooms

Data production

Data were produced with the aim of exploring enactments of AI technologies and teachers. Due to the COVID-19 pandemic, which hindered much of the planned fieldwork, different methods were employed. The produced data included field notes from four classroom observations in Years 2 and 5 that took place during the second and third week of the first of the two interventions. Seven online, semi-structured interviews with three experienced lower secondary school teachers and four representatives from the project team (a project manager, a development teacher at the local authority, a teaching-aid author and an education researcher) were carried out between May and June 2021. The interviews, ranging from 45 to 90 minutes, were each recorded and then transcribed. These interviews were primarily seen as a way to capture accounts of enactments that were not directly observable (Mazzei, 2013).

Data production

In this study, data were produced to explore enactments of AI technologies and teachers. The materials derive from a range of different methods for data production in order to broaden the analysis: observations in the different spaces of these schools, formal and informal interviews with teachers, blog posts, webpages, YouTube tutorials and social media. Approximately 80 hours of classroom observations, observations during three co-design workshops and observations at a trade fair were documented using fieldnotes and anonymised photos. Thirteen in-depth interviews were held with seven primary, lower and upper secondary teachers from the participating schools, one principal and one representative from the local school authority, and three representatives from the EdTech company (a UX designer, a product developer and the company founder responsible for the LMS). The interviews ranged from 50 to 90 minutes and aimed to provide a comprehensive understanding of the co-design process and the implementation of the LMS from multiple perspectives.

Event 1: Adaptations

The maths lesson for a group of 20 Year 2 pupils is drawing to a close, marking the onset of a 10-minute individual exercise with the AI Engine. The pupils sit in front of their computer screens and enter numbers in the free-pacing training module. The AI Engine displays the exercise 64–56 on several computer screens. The pupils type in their answers, upon which similar number combinations follow. Some pupils exclaim, ‘hey, this is too difficult,’ yet the classroom atmosphere is one of intense concentration and perseverance. The AI Engine keeps on delivering new exercises. The pupils assist one another and raise their hands to seek help from their teacher Elsa. She is in constant movement, circulating around the classroom and addressing each pupil who raises their hand. She keeps on instructing pupils that get stuck and makes sure that everyone has understood the strategies presented during the lesson and that are necessary to progress.

This event illustrates how pupils aged 8–9, along with their teacher Elsa and the AI Engine, co-produce automation. While the AI Engine is designed for individualised use and the pupils earnestly strive to solve the delivered exercises correctly, some pupils complain, seeking assistance from their peers and the attentive teacher. For automation to emerge, significant adaptation is required from Elsa, as manifested through the instructions specifically addressing the level of difficulty delivered by the AI Engine, as well as active monitoring and scaffolding during the 10-minute exercise. Rather than the AI Engine automating the work of Elsa by adapting the exercises to every individual pupil, Elsa must adapt her actions to the algorithmic decision-making for the exercise session to succeed. Thus, the co-production—symmation—where the teachers and pupils in collaboration with the AI Engine create the misleading illusion of automation. In the next event we highlight a slightly different symmation process where teachers need to experiment when faced with situations where the AI Engine does not deliver exercises as anticipated.

Event 2: Experimentation

Henry: This engine is great! It just keeps going. The exercises don’t really end. The issue, however, was that they couldn’t type the answer. And then there was this… what was the module that came after the ‘Number Chase’ called?

Interviewer: ‘Exercise more’? The one with a lot of exercises?

Henry: Yes, we worked with it. […] It was a bit too difficult for quite a few, but really great for those who are more proficient, as they were challenged. However, I told some pupils, ‘This looks a bit tough; you should go back to the “Number Chase” module.’ So, we experimented a bit.

From this event, which illustrates a conversation between Henry, a primary school teacher, and the researcher, we understand that Henry had to direct his pupils back and forth between different modules to help them accomplish exercises with the AI Engine. When some of the modules were not working as expected, he tried out new ones created as a response by the teaching-aid author to compensate for emerging errors. Henry recounted that these modules suited some pupils well but other pupils that struggled needed to be directed back to the old modules. This constant monitoring of which pupil was getting which exercise was demanded of the teacher to create the impression of automation. While Henry holds the belief that the AI Engine ‘just keeps going’ and can adapt to the pupils’ abilities, he gave an account of doing an extensive amount of work related to monitoring and personalising during the exercise sessions. Rather than just adapting, Henry needed to try out new modules and direct the pupils in a way that he described as experimental. Thus, his experimentation with various available modules, the pupils who complied with trying out different modules and the AI Engine performing ‘great’ in some cases and not at all in others, co-produced something that looked like automation, at least for some of the more proficient pupils. In reality, symmation necessitates considerable extra work, where the teacher, while trusting the AI Engine to work, needs to be attentive to the breakdown of the system and find different strategies for the pupils to train with the AI Engine. This work, as well as the pupils’ compliance with Henry’s experimentation, was entangled with the machine-learning algorithms of the AI Engine, in what may have looked like automation. A new module, ‘Exercise More,’ was created by the teaching-aid author to mitigate some of the encountered problems. While this module included more exercises, it proved effective primarily for the most proficient pupils. Other less-proficient pupils required redirection to the previous modules by the teacher.

In contrast to the narrative of AI in education as a time-saving measure, the first two events manifested how teachers’ adaptations and experimentation not only served as compensatory measures but also demanded significant—yet hidden—labour and effort. The enacted automation emerged as a messy and entangled symmation process, involving teachers, pupils, the technology and the entire learning environment.

Event 3: Compensations

Alfred: So, here you can also see the analytics [referring to the LAD], and I find it really interesting. First, I notice that they [the pupils] have a very high motivation during the lessons, their attendance is extremely high and there’s a great level of focus. The lesson satisfaction level is 96%, but they [the pupils] still think that the degree of lesson difficulty shouldn’t be like, ‘it was very easy, and we just watched a movie’ because then you can achieve very high results. Rather, it’s about, and I talk a lot with them [the pupils], that they have to work hard and in a varied way, and that the right level is individual, not too easy but not too difficult either. […] So, for me, it’s a confirmation that the time I spend on analysis and conversations with the pupils, etc., also yields results.

Interviewer: But when you see this analysis [points to the stars in the corner with alerts on low score], what do you do?

Alfred: I usually check who it is, and there’s usually an explanation […] With low lesson satisfaction it can be like, ‘well, I already knew this, so I thought I hadn’t learned much’ […] So, they don’t always understand exactly what they’re filling in either, you know.

15 minutes later

Interviewer: Is there anything else you’d like to share, anything that comes to mind about the LMS, any concerns perhaps?

Alfred: Well, the concern is that we might rely too much on it [learning analytics] […] Numbers are one thing, but aspects like, ‘I have seen a pupil because I’ve seen their data,’ it’s not really the same. So, that’s a concern, that you must not forget that just because I’m sitting here at my place and the pupils are inside the system, then we are connected. But we are not, you know, the most important part is outside all of this, and one must never forget that.

In this conversation with Alfred, a primary school teacher, symmation—as opposed to mere augmentation—came into existence through the assembly of diverse visual elements, including the LAD, computers, the pupils’ compliance, (mis)interpretations of the questions and the teacher’s compensatory actions. At first, Alfred explained what he could tell from a specific view of the LAD that referred to a lesson a few months ago. He seemed satisfied with the figures displayed in the colourful pie charts, attributing the high average score for lesson satisfaction (96%) to his systematic use of the LMS and the work invested in interpreting the various visual elements of the dashboard (numbers, headings, pie charts, stars), both during and after class. Alfred’s work also entailed making sure that the answers his pupils, aged 10–11, provided through the automatically generated lesson questionnaires corresponded to his concept of effective teaching and learning. This demonstrates not only that Alfred needed to become involved in analysing the data but also that he had to prepare detailed instructions and oversee responses to ensure that the pupils answered in ways he deemed adequate. Thus, Alfred compensated by carrying out substantial work, such as teaching the pupils how to interpret (and respond to) the questions, to enable the augmentation to emerge. The co-production of augmentation—symmation—emerged from Alfred’s compensatory work together with the pupils’ compliance with the data collection and the data visualisations of the LAD. Symmation was also reflected in Alfred’s compensatory work as a response to the stars positioned in the lower right-hand corner of the LAD’s interface (Figure 1). The stars signify low scores that the teacher should ‘take action on,’ prompting Alfred to make sure that these numbers were not indicators of pupils falling behind or feeling unwell. However, drawing on his past experience, Alfred asserted that the stars were not necessarily indicative of a genuine problem but were, in most cases, indicative of the pupils misunderstanding the questions. His statement, ‘So they don’t always understand exactly what they’re filling in either you know,’ together with his compensatory work of producing the LA data contradicts the narrative suggesting that the gathered, analysed and visualised data helps teachers to act less on intuition and more on evidence. As illustrated in this event, Alfred needed to compensate for the limitations of the LAD by continuously instructing and monitoring the pupils, which also included disregarding responses resulting from pupils misinterpreting the questions. At the same time, the data analytics practice could impose different ethical dilemmas. For example, Alfred might have overlooked the possibility that the pupils, when questioned about their negative responses, might have hesitated to take responsibility for their answers. Thus, when contradicting the teacher’s knowledge about his pupils, the LAD still instilled doubt as to whether the pupils were truly being honest or not. The event, therefore, also highlights how the practice required Alfred’s reflexive understanding of the risks involved in translating pupils into data without having a close relationship with and prior knowledge of the pupils. This understanding underscores again how symmation, and not augmentation, is at play. Here, the teacher’s relational knowledge about his pupils’ well-being and progress compensated for the limited insights that the LAD provided. In a similar vein, the next and final event shows how symmation depended on the teacher Kim confirming the LAD with already existing knowledge.

Event 4: Confirmations

Kim: Because if you look at this list on reds…it’s not…the ones that have a lot of red, I’m not surprised that I see a lot of red there. Because I know that as I see them in the classroom too. But what I see in the classroom is also part of a gut feeling. Here, it is [points to the screen] visible. Here they have also put their… [does not finish the sentence].

Interviewer: Mmm. But you also said [earlier in the conversation] that some of them just check the boxes?

Kim: Well, some just check them, yes. But I don’t think everyone, I think it’s only a small group that checks, at least that is what I hope in the groups that we looked at. In those groups, only some of them check the boxes, mmm.

Interviewer: Right, and you know that because…?

Kim: Because I know who they are. Because I have seen them in the classroom, every week for almost a year […] I spend a lot of my class time walking around and talking to my students and seeing how they’re doing when they’re working. Mmm.

Interviewer: So you don’t really need this?

Kim: But it’s a good confirmation, to know that it’s not my gut feeling.

The red squares with emojis (Figure 2) seemed to play a key role in this conversation where Kim, an upper secondary teacher, rather than dismissing them confirmed their relevance. However, something that at first may have looked like augmentation, here represented by the LAD’s visual representation of the students’ responses, instead turned out to be more of a ‘sym,’ a ‘with.’ Here Kim needed to support the red emojis of the LAD, which are intended to be indicators of students not doing well. However, Kim appeared neither surprised nor troubled by their appearance, which may stem from the teacher’s observations in the classroom that some students, aged 16–17, merely checked the boxes. This suggests that the teacher compensated for the different source of errors impacting on the data by constantly paying attention to what was happening in the classroom. Without this work, so significant for the teaching profession, Kim would have struggled to comprehend the importance of visually representing the students’ well-being. Thus, in this example, symmation means that the teacher’s knowledge about the students’ well-being, cultivated through time spent in the classroom walking around and talking to them combined with the confirmation of the visual representations of the LAD, needed to be in place for augmentation to emerge. Rather than the LAD supporting and confirming the teacher’s prior knowledge, Kim needed to act confirmatively in relation to the relevance of the statistics, even when it was obviously not relevant as some of the students demonstratively disengaged with the data collection. This confirmation was further emphasised when Kim was confronted by the researcher who wondered what the LAD could possibly offer in terms of new insights. Rather than dismissing the value of measuring and representing well-being with red emojis, Kim confirmed their validity by arguing that ‘gut feeling’ was less significant without the numbers and statistical representations to support it. Again, the teacher’s confirmation was a key ingredient in the symmation process, while also creating a paradox where other ways of knowing in the classroom were being challenged by the LAD and were simultaneously pivotal for the co-production of the augmentation of teachers.