Guest post by Dr Martin Johnson and Tori Coleman.
We work for Cambridge Assessment (https://www.cambridgeassessment.org.uk), a university department that develops and researches educational assessments. In this research project we wanted to consider how teachers’ assessment knowledge might be influenced by their prior experiences – including their work as professional examiners of students’ academic tests. To do this we used a concept mapping narrative method to elicit teachers’ complex and interconnected conceptualisations of Assessment Literacy. We used MAXQDA to analyse the teachers’ maps and we report the project in more detail in the International Journal of Research & Method in Education https://www.tandfonline.com/doi/full/10.1080/1743727X.2020.1804542).
What is Assessment Literacy?
In common with some other countries, teachers in the UK can work as examiners of students’ academic tests. In line with ideas about how experience informs development, we assume that these examiner experiences influence teachers’ understanding of assessment and their Assessment Literacy. The most recent research into Assessment Literacy suggests that it is a complex, holistic concept that includes teachers’ assessment knowledge and skills, as well as their beliefs and feelings about assessment (Adie, 2013; Looney et al., 2017; Xu & Brown, 2016). To find out about how prior examining experience influenced AL we carried out an exploratory concept mapping study with 8 teachers. Following an initial brainstorm exercise (on the theme of ‘What words come into your mind when you think about the word ‘Assessment’?’), the teachers were introduced to a definition of Assessment Literacy. We then provided instructions about the concept map task and gave the teachers examples of other concept maps on unrelated topics. The teachers then completed their own concept map around the question of ‘How has examining developed or changed your Assessment Literacy?’ The teachers varied in how they approached the mapping task. Some talked about their thoughts as they constructed their map and others worked largely in silence. We recorded the teachers during the concept mapping exercise and added this data to the information that we collected in a semi-structured interview immediately following the mapping task. Analysing these maps and narratives with MAXQDA allowed us insight into how concept mapping methods help participants unpack and communicate complex interacting concepts.
‘Getting into the issue’
We used MAXQDA to identify the common components of the different concept maps. Using the image coding function, we were able to carry out deep analysis of the maps alongside each other so that we could see their common features. We were also able to link each map to the corresponding teacher narrative that we captured during the concept map construction process. This linking process was facilitated by the ability to easily collate the linked documents in the MAXQDA Document System (e.g. each teacher’s interview recording, transcript, and concept map file). This ensured that the teachers’ rationalisations grounded our analysis. By linking the ordering of concepts as they arose in the narrative, we were able to see which elements of the map emerged first. We observed that the concept mapping process allowed all the teachers to engage in a first order activity of organising their thoughts, effectively giving the teachers ‘a way into the issue’. In this concept map we can see how one teacher uses high level values (assessment purpose) to divide up their assessment understandings.
Breaking concepts into sub-concepts
Our analysis also showed that a common feature of all the concept maps was for teachers to break AL down into sub-concepts. By reflecting on their maps, the teachers were able to signify linkages across these sub-concepts – and this feature allows us insight into the complexity of assessment as understood by the teachers. The image coding function allowed us to pay attention to the graphic elements of the teachers’ maps so that we could explore their meaning in the accompanying narrative texts. In the concept map example below the teacher indicates how their understanding of the nuances of assessment questions (from having the experience of marking lots of these questions as an examiner) helps them to teach their students how better to interpret assessment questions.
Showing what is most important
All of the teachers employed iconic representation, such as ‘arrows’ in their maps. These indicators allowed the teachers to make explicit the most important elements of their assessment experience. In the concept map below, we were able to use the image file coding function to show how the teacher highlights the salience of understanding ‘mark schemes’ for their AL. Having access to test papers is relatively meaningless without this understanding, the use of command words in assessment questions links to the valued ideas in a mark scheme; and examining experience allows more familiarity with these mark schemes.
Linking pictures and words
A key affordance of MAXQDA for analysis for us was the way that it allowed us to link our teacher narrative data to the concept map content. This feature is a key to validating the outcomes of our analysis. The graphic and verbal aspects of the teachers’ mapping process unfolded alongside each other as part of the same elicitation process. Having access to a simultaneous view of the map and the transcribed narrative reinforces the point that we need to consider how these data relate to each other – with neither existing in isolation.
How does MAXQDA support the analysis of concept maps?
We found that our use of MAXQDA allowed us to systematically consider how concept maps work as an elicitation method. The software supported our analysis by allowing us to develop a common coding method to consider the interplay of teachers’ verbal narratives alongside their graphic representations. We were also able to develop several insights into the benefits of using concept maps for research processes. We found that concept maps are both representation and communication – they convey ideas in a way that makes them visible to others. Maps are also simultaneously simple and complex – they tend to start with a few high-level words that allow the mapper to begin a process of more complex articulation.
For a more extended discussion on our project and the use of concept maps, please see our paper in the International Journal of Research & Method in Education https://www.tandfonline.com/doi/full/10.1080/1743727X.2020.1804542
Adie, L. (2013). The development of teacher assessment identity through participation in online moderation. Assessment in Education: Principles, Policy & Practice, 20(1), 91–106. https://doi.org/10.1080/0969594X.2011.650150
Looney, A., Cumming, J., Kleij, F. van D., & Harris, K. (2017). Reconceptualising the role of teachers as assessors: Teacher assessment identity. Assessment in Education: Principles, Policy and Practice, 25(5), 442–467. https://doi.org/10.1080/0969594X.2016.1268090
Xu, Y., & Brown, G. T. L. (2016). Teacher assessment literacy in practice: A reconceptualization. Teaching and Teacher Education, 58(Supplement C), 149–162. https://doi.org/10.1016/j.tate.2016.05.010
About the Authors
Dr. Martin Johnson is a Senior Researcher at Cambridge Assessment. The focus of much of his work is on the interaction between assessment, learning and curriculum issues, and on how to better understand assessment as enacted practice.
Tori Coleman is an Education Researcher at Cambridge Assessment, with a background in Psychology. She is involved in research on education, curriculum, and assessment using mixed qualitative and quantitative research methods.