Blog

This is a guest article written by Matthew Loxton, a Principal Healthcare Analyst and Professional MAXQDA Trainer.

In this blog post, I will expand the scope to the left of the survey analysis process, and draw your attention to a source of data that can enrich the coding and code memos in a way that is efficient, effective, and uses existing data to enhance the coding and analysis of survey responses.

Using Survey Questionnaire Handbooks to speed up the development of Codes, and to enhance analysis of survey data with informative Code Memos

Although it has always been possible to collect participant feedback via open-ended survey questions, the advent of online surveys has made them ubiquitous. However, although online questionnaires typically have several open-ended questions, almost all the analysis work has typically gravitated to the closed answers that are more easily quantifiable.

For example, I am sent a weekly survey of patients at a U.S. ambulatory care clinic, and each week, and at the end of every month, quarter, and year, the Likert scale questions are used to generate trend graphs, pie charts, and bar charts. In comparison, over several years of the survey being sent out, the open-ended responses have never resulted in more than an occasional “why did this complaint happen” query from the hospital management team.

Part of this disparity between reporting on closed and open responses is due to the difficulty in analyzing open-ended questions. However, as MAXQDA Professional Trainer Graham Hughes shows in his recent webinar tutorial on Analyzing Answers to Open-Ended Questions in Surveys, MAXQDA has very powerful methods to import, analyze, and report on open-ended survey responses. Likewise, the MAXQDA team created an online tutorial for Working with Surveys in MAXQDA, which shows methods to streamline the process of importing and analyzing survey data. A link to the videos can be found at the end of this article.

Questionnaire Handbook

In many cases, a survey questionnaire handbook will have been developed in parallel with the questionnaire. The handbook documents the research approach, including the constructs that are to be measured, the population and sampling methodology to be used, as well as details of the question items, their scales and interpretation, and the logic and context for each question.

Some of the metadata can be used to bolster analysis in MAXQDA by using the questionnaire item titles as codes, and question item details as part of the code memos. Having these details can greatly enhance the thickness of the coding and depth of analysis in MAXQDA.

A questionnaire handbook may contain the following sections:

• Overview: A high-level description of the scope, goals, and stakeholders.
• Background: A description of the problems and context of why the research question is relevant and important.
• Measurement Approach: A description of how the sampling is envisaged, and what process, tools, and populations will be used.
• Working Group: A list of who was involved in the survey design.
• Goals: A description of the research questions the survey aims to answer, or what problems it seeks to resolve or explain.
• Metric Definition Key: A multi-level detailed explanation of the measurement structure.

Metric Definition Key

Each metric will be defined in terms of a formal structure. For example, the Agency for Healthcare Research and Quality (AHRQ) Measurement Clearinghouse measurement framework includes the following dimensions.

1. 1. Description: Provides a concise statement of the specific aspects of health, the target population, providers, setting(s), and period that the measure addresses.
   2. Rationale: Explains the significance and purpose of the measure (i.e., why it is used, and how it will be used).
   3. Denominator: Provides the general specifications of any descriptive component that is the basis for inclusions and exclusions in the denominator.
   4. Numerator: Provides the general specifications of any descriptive component that is the basis for inclusions and exclusions in the numerator.
   5. Denominator Time Window: Classifies the period in which occurrences identified as potentially eligible for inclusion in the denominator are reviewed to determine whether they are finally included.
   6. Data Source: Identifies the data source(s) necessary to implement the measure.
   7. Instruments Used: Identifies all instruments, such as a standardized survey instrument or flow sheet, used in implementing the measure.
   8. Scoring: Identifies the method used to score the measure.
   9. Interpretation of Score: Explains the meaning and implication of the various possible score ranges, and describes any desired value, and implications of the delta between actual and desired score.
   10. Measure Availability: Identifies contact information for requesting the measure documentation. Where possible, information regarding electronic (including hypertext links to the original measure documentation) and print copies.

Each construct is typically documented along with the measures and metrics that make it up. For example, in Table 1, the construct of “Self Advocacy” for patients is broken out into the measure “Health Self-Management” (one of many), which is composed of several metrics. The handbook defines each measure, and how the data is to be collected and interpreted. Abbreviated metric titles are typically used to identify the metric in spreadsheets to avoid long names. E.g. Metric 1.4.1 “Documented prevention preferences” has an abbreviated title of “Prev-Pref”, which is very suitable for a MAXQDA Code.

Table 1: Benefit Construct Structure

Likewise, in this handbook, each metric is listed again as a question item with a preamble that will be used to populate the online survey. For example, in Table 2, “Self-Advocacy” is broken out into facets (metrics) as previously, but now shows the question that will be presented, along with an introductory preamble.

Table 2: Effectiveness

Constructing the Import Sheet

The preamble and questions can be easily combined in an Excel sheet by concatenating the cell for the question preamble and each of the question items, as well as any other details that will inform the qualitative analysis. In Table 3, I have used the question title for the code column and combined the question components as memos.

Table 3: Excel Import Sheet

Analysis with MAXQDA 2018

Importing the resulting code structure is done by clicking on the “Codes” tab in MAXQDA 2018, and then on “Import Codes and Memos from Excel Spreadsheet”, as shown in Figure 1.

Figure 1: Code Import from Excel

The result of importing the code system from Excel is shown in Figure 2. The codes for each question are now in MAXQDA, and the mouse hovering over the memo for the “Clin-Plan” code shows that the question details have been successfully imported.

Figure 2: Imported Code System

Following the establishment of the code system, survey results can be imported by using the MAXQDA Import function. By clicking on the “Import” tab, and “Documents from Excel Spreadsheet” (Figure 3), we can add the respondent feedback and auto-code with the imported questionnaire item titles as a single step.

Figure 3: Import Documents from Excel

After the import is done, we can see that the codes have been associated with the related documents (Figure 4).

Figure 4: Imported and Auto-Coded Documents

Conclusion

The use of the questionnaire codebook as the source of MAXQDA codes and memos enables us to reuse existing data, stay aligned with the original survey terminology, and assist analysis by having a ready reference in the code memo to remind us of the code meaning and intent without having to retype or create memos manually.

In this example, I have only pulled in the preamble and question items as memos, but the same technique can be used to add scoring interpretation or any other pertinent details to assist in analysis.

Want to learn more about Survey Analysis with MAXQDA? Check out these 2 videos on the MAXQDA YouTube channel or download the chapter from the online Manual:

About the Author

Matthew Loxton is a Principal Analyst at Whitney, Bradley, and Brown Inc. focused on healthcare improvement, serves on the board of directors of the Blue Faery Liver Cancer Association, and holds a master’s degree in KM from the University of Canberra. Matthew is the founder of the Monitoring & Evaluation, Quality Assurance, and Process Improvement (MEQAPI) organization, and regularly blogs for Physician’s Weekly. Matthew is active on social media related to healthcare improvement and hosts the weekly #MEQAPI chat. You can also read another guest post by Mathew Loxton on “Mixed Methods Research: Surmounting Obstacles with MAXQDA and R” here in the MAXQDA Research Blog.

Mixed Methods with MAXQDA