Balance and rigour in qualitative analysis frameworks

Training researchers to use qualitative software and helping people who get stuck with Quirkos, I get to see a lot of people’s coding frameworks. Most of the time they are great, often they are fine but have too many codes, but sometimes they just seem to lack a little balance.

In good quality quantitative research, you should see the researchers have adopted a ‘null hypothesis’ before they start the analysis. In other words, an assumption that there is nothing significant in the data. So statisticians play a little game where they make a declaration that there should be no correlation between variables, and try and prove there is nothing there. Only if they try their hardest and can’t possibly convince themselves there is no relationship are they allowed to go on and conclude that there may be something in the data. This is called rejecting the null hypothesis and may temper the excitement of researchers with big data sets, over enthusiastic for career making discoveries.

Unfortunately, it’s rare to see this approach described in published quantitative analysis. But there’s no reason that a similar approach can’t be used in qualitative research to provide some balance from the researcher’s interpretation and prejudices. Most of the time the researcher will have their own preconception of what they are going to find (or would like to find) in the data, and may even have a particular agenda they are trying to prove. Whether a quantitative or qualitative methodology, this is not a good basis for conducting good impartial research. (Read more about the differences between qualitative and quantitative approaches.)

Steps like reflexivity statements, and considering unconscious biases can help improve the neutrality of the research, but it’s something to consider closely during the analysis process itself. Even the coding framework you use to tag and analyse your qualitative data can lead to certain quotes being drawn from the data more than others.

It’s like trying to balance standing in the middle of a seesaw. If you stand over to one end, it’s easy to keep your balance, as you will just be rooted to the ground on one side. However, standing in the middle is the only way you are challenged, and it’s possible to be influenced by sways and wind from one side to another. Before starting your analysis, researchers should ideally be in this zen like state, where they are ready to let the data tell them the story, rather than trying to tell their own data from selective interpretations.

When reading qualitative data, try to have in your head the opposite view to your research hypothesis. Maybe people love being unemployed, and got rich because of it! The data should really shout out a finding regardless of bias or cherry picking.

When you have created a coding framework, have a look through at the tone and coverage. Are there areas which might show any bias to one side of the argument, or a particular interpretation? If you have a code for ‘hates homework’ do you have a code for ‘loves homework’? Are you actively looking for contrary evidence? Usually I try and find a counter example to every quote I might use in a project report. So if I want to show a quote where someone says ‘Walking in the park makes me feel healthy and alive’ I’ll see if there is someone else saying ‘The park makes me nervous and scared’. If you can’t, or at least if the people with the dissenting view is in a minority, you might just be able to accept a dominant hypothesis.

Your codes should try and reflect this, and in the same way that you shouldn’t have leading questions “Does your doctor make you feel terrible?” be careful about leading coding topics with language like “Terrible doctors”. There can be a confirmation bias, and you may start looking too hard for text to match the theme. In some types of analysis like discourse or in-vivo coding, reflecting the emotive language your participants use is important. But make sure it is their language and not yours that is reflected in strongly worded theme titles.

All qualitative software (Quirkos included) allows you to have a longer description of a theme as well as the short title. So make sure you use it to detail what should belong in a theme, as if you were describing it to someone else to do the coding. When you are going through and coding your data, think to yourself: “Would someone else code in the same way?”

Even when topics are neutral (or balanced with alternatives) you should also make sure that the text you categorise into these fields is fair. If you are glossing over opinions from people who don’t have a problem with their doctor to focus on the shocking allegations, you are giving primacy to the bad experiences, perhaps without recognising that the majority were good.

However, qualitative analysis is not a counting game. One person in your sample with a differing opinion is a significant event to be discussed and explained, not an outlier to be ignored. When presenting the results of qualitative data, the reader has to put a great deal of trust in how the researcher has interpreted the data, and if they are only showing one view point or interpretation they can come across as having a personal bias.

So before you write up your research, step back and look again at your coding framework. Does it look like a fair reflection of the data? Is the data you’ve coded into those categories reflective? Would someone else have interpreted and described it in the same way? These questions can really help improve the impartiality, rigour and balance of your qualitative research.

A qualitative software tool like Quirkos can help make a balanced framework, because it makes it much easier than pen and Post-It notes to go back and change themes and recode data. Download a free trial and see how it works, and how software kept simple can help you focus on your qualitative data.