October 13, 2010, 10:27 am
What’s Wrong With These Survey Questions?
By Kirsten Robinson
I’m often called upon to design surveys and also to take them. One of my pet peeves is surveys that are released to the wild with design errors. Just as authors need editors, survey designers need reviewers to check for editorial and logical errors. Here are a few problem questions I’ve encountered recently. Can you spot the problems? If so, please post a comment.
Question 1.
Question 2.
Question 3.
Have you seen a poorly designed survey question recently? If you design surveys, what steps do you take to avoid errors?
Comments
An update: I wrote to the survey designer who was responsible for questions 1 and 3. It turns out he had his survey tested by five people before he released it. So, he did the right things…. Survey design is hard!
Posted by Kirsten on October 14, 2010 at 8:48 am
For question 1, if the radio buttons function as normal, you can only select one option, where the question tells you to select all the answers that apply.
Question 2 I’m not sure of
Question 3 perhaps should be two options, so you know which of the two answers applies? Consultant as one y/n, self-employed as another?
Definitely looking forward to what your answers are.
Jon
Posted by jonmcrawford on October 14, 2010 at 9:32 am
I’d agree with Jon, and add for Q2:
If the checkboxes function as normal, it’s possible for someone to select ‘follow’ without selecting ‘aware’. Some users might think checking ‘follow’ is enough while others check both ‘aware’ and ‘follow’ to mean the same thing.
Posted by Michael Niggel on October 14, 2010 at 10:24 am
Regards #2 an expectation is that if you follow someone then you are aware of them. And so there should be three exclusive choices: unaware, aware, and follow.
Posted by Andrew Gilmartin on October 14, 2010 at 10:32 am
Thanks for your astute comments! Jon, you nailed #1 and #3. I didn’t know how to respond to #3 as I’m a consultant who is not self-employed. (I interpreted the / as a Boolean OR and answered yes.) As for #2, my main objection is the same as Andrew’s: there was no option for those who are not aware of the twitter accounts (a response was required – leaving all checkboxes blank generated an error). Michael also makes an excellent point about #2.
Posted by Kirsten Robinson on October 14, 2010 at 1:32 pm
ah, good point, can hardly follow without being aware of the account.
Posted by jonmcrawford on October 14, 2010 at 1:33 pm