If you’re new to commissioning research, you may find yourself wondering “what comes next?” after all the RFPs have been sent, a research partner has been chosen, and surveys have been fielded. Typically, your research provider will give you the deliverables outlined in their proposal – usually a report. Sometimes, the deliverable will also include computer tables (frequently called “tabs”) of the processed/compiled data. If you want to be very hands-on with the data, you may also receive data files in excel or SPSS so that you can manipulate the data yourself.
No matter what sort of deliverable you’re receiving, there are three key things our experts look for to make sure your supplier has checked over the data and provide reassurance that you’ve been presented with correct, high quality data.
We always look at the demographic profile of your survey respondents. Does that profile match the demographics of the intended sample target? If it doesn’t quite match up, we may need to weight your data to bring it in line with your sample target. Weighting the data will lessen the impact of the results from any group you have too many of and heighten the impact of results from groups you have too few of. Too much weight on any specific segment, however, will lead to skewed results, where one (or a few) respondent(s) impacts the results far more than any one person should. “Too much” depends on the study, and our research experts are adept at ensuring the weightings are correct based on your needs.
Many times, it’s important to have multiple cells (distinct groups) in a test, which allows you to test multiple concepts. Is each cell properly balanced? When testing two concepts you’ll want to make sure the demographic profile in each cell matches, so that differences in the responses aren’t due to unbalanced samples. These differences may be hidden if you only look at the total sample demographics. For example, let’s say you want a 50/50 split on gender for the total test. If Cell A has a 60/40 males/females split and Cell B as a 40/60 males/females split, the total will show a 50/50 split. While the total sample is the 50/50 split you’re looking for, the two cells are very different, which could impact your results. This could throw off your data interpretation when comparing the two concepts to one another. In both cells, you need the same 50/50 split to ensure that differences in ratings of the two concepts relate to the concepts, and aren’t driven by demographic differences in the cells’ samples.
Speaking of sample, make sure that your sub groups’ sample sizes all add back to the total sample size? Do this by ensuring the sample sizes in the final report match the tabs. (For example, do the number of male respondents plus the number of female respondents equal the total sample size? If they don’t, ask your provider why not). There may be valid reasons why the sample sizes don’t match. For example, was the question asked a “skip” type question, where not every respondent was asked, or it was a non-mandatory response question? The important thing to confirm is that everyone who should have been asked the question got asked. If you find that a specific questions’ sample size is higher than the total sample size it may be double counted. If the sample for that question is smaller, hopefully it’s ok, but confirm that there wasn’t a programming error.
When examining the data itself, here are some of the main things we double check. These include:
For a question with several attributes to be rated on a scale (i.e. rate X on a scale of 1-5), check for patterns in how respondents answer. For example, be sure they’re not simply entering the same response for every answer (like all 5s), or that there isn’t an obvious pattern to their answers. Common patterns are the classic “Christmas tree” (1, 2, 3, 4, 5, 1, 2, 3, 4, 5) and straight-lining (1,1,1,1,1,1). These are likely indications that the respondent has not truly read the question and answered thoughtfully. You may want to throw out suspect responses like that.
Time spent on the survey
If your survey is approximately 15 minutes long, and a respondent finishes it in 3 minutes, odds are good that they didn’t actually read the questions and just sped through it. While there will be some variation in terms of how long it takes respondents to take a survey, there is a point where the time spent is too short to get thoughtful responses. In a scenario like our example above, we would likely discard the respondent’s entire survey.
If a question uses a 1-5 scale and each number is also associated with a worded response, make sure the correct number goes with the correct verbiage in data processing. For example, if you look at a standard purchase interest question:
5) Definitely will purchase
4) Probably will purchase
3) Might or might not purchase
2) Probably will not purchase
1) Definitely will not purchase
When the data are processed, simply confirm a score of 5 (and not a 1) was given to the “Definitely will purchase” responses, a 4 (and not a 2) to the “Probably will purchase” responses, etc. This may seem obvious, but it can and does get mixed up on occasion. A strong hint that this has happened is if the results for a question just don’t seem right to you. Trust your gut and check with your supplier.
If you have a question with a list of attributes or have several concepts to be evaluated in your survey, these lists should probably be rotated or randomized so they are being shown in a different order for different respondents (more on this in an upcoming blog about survey writing). Like the scale-inversion example above, you need to be certain that each attribute or concept is getting the correct data from individual respondents attributed to it when all the data is compiled.
So there you have it. While this list is by no means comprehensive, it’s a good starting point for what to examine when it comes to the data you collect in your surveys. Being able to double check your data will help you to feel confident that not only are you receiving quality data but that you picked an excellent data provider. Are you curious about other ways to quality-check your data? Reach out to our Custom Research team any time, we’ll be glad to talk with you one on one.