Surveys are complex. There is a lot that you can do wrong. Actually, looking at many surveys around, there is a lot that is done wrong. Time after time.
Sometime it’s the big stuff. Sometimes small.
Sometimes the surveys are not done to get answers, but to show what you want the answers to be, by skewing the questions. That’s bad for academic research, but very popular in politics.
And sometimes you really do want answers. Which is hard to do right. Ask the wrong question, ask them in the wrong way, or give the wrong options for answers, and the results may not say what you think they do, or may be impossible to analyse properly. That’s why there are those who deal professionally with surveys, know the theory (and, hopefully, statistics), have done it many times before, and should be able to avoid most of the mistakes.
They usually don’t do the really big mistakes.
They do, however, often do small, or really incomprehensible, mistakes. I guess finding a professional can be a problem as well.
Take, for example, a survey currently being run by iPerceptions , for InforWorld and ComputerWorld.
Both these clients are one client, belonging to the same company. And they do very similar things. So the surveys are practically identical (I did the ComputerWorld one originally, and just now noticed they also run it for InfoWorld. I progressed a little bit, and they’re the same questions in the same order with the same possible answers. Just the name of the company in the survey changed).
This survey has some strange points.
One main problem was that they apparently forgot that some questions may not apply. There was one (maybe 2-3) question where they did have an option to indicate the question is not relevant, or that I don’t know or can’t judge. For all the rest, and there were many of them, I was asked to rank the sites on many criteria, some of which really didn’t interest me and I didn’t know. But the options were just to rank.
Assuming that I’m not the only person who goes through a site that has many different sections, and doesn’t know (or use) all sections, this means that the answers they receive are worthless. What do you pick when you don’t know, or can’t rank? Do you say that it was excellent, since you don’t know it’s bad, causing a potential problematic part to appear good? Do you rank it as very lousy, since it didn’t do anything for you, thereby causing a potentially excellent service to appear bad? Do you rank it in the middle, trying not to judge either way, but still making anything really good, or really bad, seem more average and undeserving of attention?
This is why these things usually contain an answer to state that this question isn’t relevant for you, and you don’t have a real answer for it. But here, no. Good luck to them in the later analysis.
A second point is much less severe, but far more amusing and baffling.
In the personal details, at some point they asked for year of birth. And provided a field to type the year number in. With a maximum of 500 characters. Yes, you read that right, 500 characters to answer the question “In what year where you born?”. They also made the text box large enough to type a small essay in.
What sort of an answer where they expecting? Hmm… Maybe…
That’s a tough question, there. I don’t know what year I was born in. It was a cold, harsh, dark year for my family. My parents were moving a lot. I don’t have no birth certificate, ’cause they were always running from them cops. Who need a stinking certificate? My mom knew I was born. And I had a tough childhood, so people tell me I look 40, but I bet I’m younger. I don’t remember much from those years, really can’t say. Is this important? If it is, I can try remembering, just let me know. Yes?
That’s not a true story (for me, anyway), but it does have exactly 500 characters. For comparison, writing something like 2008, or 1912, takes 4 characters (as does “NOYB“). They could have even been generous, cover all their bases, and give 5-6 characters (You know, for time travellers, or for really really old people). Maybe 3 digits more, for a space followed by “AD”, in case they’re actually worried? A little longer still, so they can get “year of the dragon”? Why the heck 500 characters?
You want to know what’s even more strange? This is in the third part of the survey. In the first part they already asked most personal questions (gender, business, people working in same company, etc), including one about age. But there they just gave several age groups (e.g. 24-35 or something like that), so I suppose they really needed the birth year too.
Then again, if they ask for birth year, why bother asking for age? Odd, that.
Well, I was in a nice mood (this was more amusing than annoying), so I decided I’ll let them know. At the last page of the survey they had a link to provide feedback. I was actually impressed with that, since sometimes I have comments, and nobody official to tell them to. This was nice. Or so I thought.
It was a mailto style link, that contained an email address, and a prepared subject line with the code/number of this survey (Good idea, so they won’t have to wonder what survey it was, and I won’t have to try and describe it too much to ensure they identify it).
There was just one main problem with it. The email address they provided? It wasn’t correct.
I sent a message. I got back a bounce.
<info1@iperceptions.com>:
208.65.144.12 does not like recipient.
Remote host said: 553 mailbox info1@iperceptions.com is restricted (Mode: normal)
Giving up on 208.65.144.12.
Impressive. This is a company that specializes in running surveys. In getting feedback from people for their clients. Except they can’t seem to arrange to get feedback for themselves.
It seems like a typo. The “1″ in the email address does not belong. I checked later on their site, and this address is listed there, without the “1″, in their contact page. But, well, by that time I was out of the helpful mood, and into the annoyed and unimpressed one. Which I think is perfectly understandable.