I recently had a student submit one of the research surveys and all of the answers were reversed – he or she had not clearly read the anchor choices. This led me to wanting to share with you a reminder of the need for being a compulsive researcher – we cannot control what our study’s participants do, so we must not only check their responses but also check what we do several times.
You should check surveys when someone hands them in to make sure all blanks are filled and do a quick look to make sure that they make sense. If they don’t make sense, or there are blanks – it is perfectly ok to ask them to double check that everything is answered as they wish. They may WANT to leave something blank – which is perfectly acceptable or they may have accidentally left it blank. Once they leave the room/ you can no long double check. Why do you care? Depending on the particular survey, you have to toss out their survey if they have left answers blank.
We also have to check our own work, we too make errors and it is our responsibility to try to correct as many as we can. Think about that when you are entering data into SPSS and go back and check it again, or better yet, ask someone else to check you. The implications for errors become great in research. You are going to make claims based upon your data and you want them to be correct and replicable by others.
For those of you that are going into academics and will be doing research with students, remember you are responsible for their work. Double check everything, because when the results fail to replicate, it will be you that others will be looking at not the students that helped you!
Be compulsive, be professional in research, and be proud of what you have done!
Lee Stadtlander
Sage advice. Thank you for the reminder.
ReplyDeleteDr. Lee brought up a good point, but also one that we should think about as researchers.
ReplyDeleteThe first survey in our packet used a Likert scale with the negative end on the right and the positive end on the left, and the second a Likert scale that reversed the order. When I crossed over to answering the second survey, I answered the first few questions according to the scheme of the first survey; I had to stop and think about what I was doing to catch the reversal. This created a level of cognitive dissonance in me that made answering the second survey more distressing, even though its content was fairly neutral.
As researchers, we want to eliminate any cognitive "bumps" that might influence or confuse our respondents. Making sure that surveys we give them use the same ordering scheme, or giving them a task break for an interviewing segment, or specific instructions for the change would ease the transition between the two. Especially when dealing with the oldest old, inducing confusion may be completely counterproductive to getting the information we are seeking.
I don't know which of the students submitted their survey with answers reversed, but I posit to you that it may well have resulted from the cognitive discord of the reversed scales. As subjects get in the "habit" of answering one direction, shifting to the other without clear warning may cause them to simply continue marking the way they had been without noticing the change. As researchers, we need to carefully examine such surveys, and think about whether such compulsiveness may be a factor in a set of answers that appear "off." But thinking about that possibility in the design stage may be easier (i.e., cheaper) than finding it in the sample data.
Or am I just being a geeky software architect again?
Michael
Michael,
ReplyDeleteGood points! And I totally agree I would never do this with the oldest old. However, none of our students are in that category :)
There is another philosophy on survey administration, that you do want to cause such dissonance. You want to make sure that people are actually reading the instructions and the survey questions. That is why so many surveys have a few negative responses thrown in (e.g., "My involvement with research will take time away from leisure activities."). It gives the researcher a check to make sure that people did not just choose the same response for all of the questions.
Thanks for your thoughts!
Lee Stadtlander