Alternative Kappa calculation via table output?

09.05.2016, 09:57

Our research team has moved from Nvivo to MaxQDA recently (I have received formal training in MaxQDA, but have not used it as extensively in my work as I have NVivo in the past) for various reasons. However, we have a problem with a current project that we cannot resolve as of yet.

Project summary: We are coding for themes in a set of texts (25 in total, long texts of about 20 pages each), and each theme is simply coded for presence (i.e., themes may overlap). For a subset of the data, we had two coders apply the same codebook as an inter-rater reliability check.

The problem: We are trying to figure out how to calculate Kappa's for several codes on content coded by two individuals. The built-in Kappa calculation does not work for our purposes, and we are trying to figure out how to get output that will allow us to calculate our own Kappas using a different Kappa formula (than the built-in one) with different assumptions.

The problem we have is that the built-in Kappa coefficient in MaxQDA seems to assume that one is trying to code for X vs. Y vs. Z, as opposed to simply coding for the presence (or lack of presence) of X (then the same for Y, Z, etc.). Both the modified Kappa formula that MaxQDA gives in its documentation, as well as the approach described (i.e., only allowing for coded segments as the unit of analysis) make this the case (as far as I understand - and I am fairly confident that I am thinking about this right). However, in our case, we want to calculate a simple 'present-not present’ Kappa for each code, for two coders, with the paragraphs as a unit of analysis (each paragraph is one observation, which will be either present/not present for each coder).

We have tried to figure out an adequate work-around in MaxQDA (I am personally much better at figuring out workarounds in NVivo). For example, I have tried to produce a cross-tab for each code where the cells of the table indicate the number of paragraphs for which X was 1) present for Coder 1 and also for Coder 2, 2) present for Coder 1 but not coder 2, 3) not present for Coder 1 but present for coder 2, and 4) not present for either, and if we could produce such a table, we could calculate our own Kappas for each code, but I have hit a wall in figuring out how to do this with this project.

In sum, I have two questions for this forum, one methodological question and one technical one:

1) Do you have any thoughts on the reliability method, i.e., using each paragraph in each text as a unit of analysis, and using the presence/no presence for both coders in a 2x2 table to calculate a Kappa for each code?

2) Can any of you point us to a potential workaround that would allow us to produce such output in MaxQDA? A table that gives a total count of paragraphs, with codes as rows, and columns indicating paragraph counts of the overlap of Coder 1 and Coder 2—the 4 categories above (1&2 yes, 1 yes 2 no, 2 yes 1 no, 1&2 no) would be ideal.

We have about 20-30 codes, and there are 25 or so texts. Both coders coded the texts independently, and we have imported the second coders codes as additional documents into MaxQDA. Its not outside the realm of possibilities to do this manually, but we would really like to figure out a reasonable workaround in MaxQDA, both to save having to do it manually and to get a better grip on thinking through such issues in the logic of how MaxQDA is structured.

Any thoughts or suggestions would be much appreciated.

Version: MAXQDA 12
System: Mac OS X 10.11 (El Capitan)
Posts: 1
Joined: 20.04.2016, 15:48

Re: Alternative Kappa calculation via table output?

09.05.2016, 10:42

Hi jacobrhickman,

a first short question: have you already tried to only activate one code and use the built in Kappa output? In this case MAXQDA will produce a table that shows coded vs. not-coded for segments for two documents.

You will then have to look at the line "In case of missing values or in case of comparing one code:"

MAXQDA Kappa for single code
kappa-for-single-code.png (11.44 KiB) Viewed 2909 times

Best regards,
Dr Stefan Rädiker
Training, Coaching, Analysis for Research & Evaluation
MAXQDA Professional Trainer
Stefan Rädiker
Posts: 254
Joined: 12.02.2007, 10:18

Return to MAXQDA in Research

Who is online

Users browsing this forum: No registered users and 1 guest