This is why you should let someone experienced do surveys. Or, well, not.

Surveys are complex. There is a lot that you can do wrong. Actually, looking at many surveys around, there is a lot that is done wrong. Time after time.

Sometime it’s the big stuff. Sometimes small.

Sometimes the surveys are not done to get answers, but to show what you want the answers to be, by skewing the questions. That’s bad for academic research, but very popular in politics.

And sometimes you really do want answers. Which is hard to do right. Ask the wrong question, ask them in the wrong way, or give the wrong options for answers, and the results may not say what you think they do, or may be impossible to analyse properly. That’s why there are those who deal professionally with surveys, know the theory (and, hopefully, statistics), have done it many times before, and should be able to avoid most of the mistakes.

They usually don’t do the really big mistakes.

A very long birth yearThey do, however, often do small, or really incomprehensible, mistakes. I guess finding a professional can be a problem as well.

Take, for example, a survey currently being run by iPerceptions , for InforWorld and ComputerWorld.

Both these clients are one client, belonging to the same company. And they do very similar things. So the surveys are practically identical (I did the ComputerWorld one originally, and just now noticed they also run it for InfoWorld. I progressed a little bit, and they’re the same questions in the same order with the same possible answers. Just the name of the company in the survey changed).

This survey has some strange points.

One main problem was that they apparently forgot that some questions may not apply. There was one (maybe 2-3) question where they did have an option to indicate the question is not relevant, or that I don’t know or can’t judge. For all the rest, and there were many of them, I was asked to rank the sites on many criteria, some of which really didn’t interest me and I didn’t know. But the options were just to rank.

Assuming that I’m not the only person who goes through a site that has many different sections, and doesn’t know (or use) all sections, this means that the answers they receive are worthless. What do you pick when you don’t know, or can’t rank? Do you say that it was excellent, since you don’t know it’s bad, causing a potential problematic part to appear good? Do you rank it as very lousy, since it didn’t do anything for you, thereby causing a potentially excellent service to appear bad? Do you rank it in the middle, trying not to judge either way, but still making anything really good, or really bad, seem more average and undeserving of attention?

This is why these things usually contain an answer to state that this question isn’t relevant for you, and you don’t have a real answer for it. But here, no. Good luck to them in the later analysis.

A second point is much less severe, but far more amusing and baffling.

In the personal details, at some point they asked for year of birth. And provided a field to type the year number in. With a maximum of 500 characters. Yes, you read that right, 500 characters to answer the question “In what year where you born?”. They also made the text box large enough to type a small essay in.

What sort of an answer where they expecting? Hmm… Maybe…

That’s a tough question, there. I don’t know what year I was born in. It was a cold, harsh, dark year for my family. My parents were moving a lot. I don’t have no birth certificate, ’cause they were always running from them cops. Who need a stinking certificate? My mom knew I was born. And I had a tough childhood, so people tell me I look 40, but I bet I’m younger. I don’t remember much from those years, really can’t say. Is this important? If it is, I can try remembering, just let me know. Yes?

That’s not a true story (for me, anyway), but it does have exactly 500 characters. For comparison, writing something like 2008, or 1912, takes 4 characters (as does “NOYB“). They could have even been generous, cover all their bases, and give 5-6 characters (You know, for time travellers, or for really really old people). Maybe 3 digits more, for a space followed by “AD”, in case they’re actually worried? A little longer still, so they can get “year of the dragon”? Why the heck 500 characters?

You want to know what’s even more strange? This is in the third part of the survey. In the first part they already asked most personal questions (gender, business, people working in same company, etc), including one about age. But there they just gave several age groups (e.g. 24-35 or something like that), so I suppose they really needed the birth year too.

Then again, if they ask for birth year, why bother asking for age? Odd, that.

Well, I was in a nice mood (this was more amusing than annoying), so I decided I’ll let them know. At the last page of the survey they had a link to provide feedback. I was actually impressed with that, since sometimes I have comments, and nobody official to tell them to. This was nice. Or so I thought.

It was a mailto style link, that contained an email address, and a prepared subject line with the code/number of this survey (Good idea, so they won’t have to wonder what survey it was, and I won’t have to try and describe it too much to ensure they identify it).

There was just one main problem with it. The email address they provided? It wasn’t correct.

I sent a message. I got back a bounce.

<>: does not like recipient.
Remote host said: 553 mailbox is restricted (Mode: normal)
Giving up on

Impressive. This is a company that specializes in running surveys. In getting feedback from people for their clients. Except they can’t seem to arrange to get feedback for themselves.

It seems like a typo. The “1″ in the email address does not belong. I checked later on their site, and this address is listed there, without the “1″, in their contact page. But, well, by that time I was out of the helpful mood, and into the annoyed and unimpressed one. Which I think is perfectly understandable.

4 Responses to “This is why you should let someone experienced do surveys. Or, well, not.”

  1. Jonathan Levitt says:

    I just wanted to personally thank you for your feedback. In the thousands of website optimization surveys we serve every day, it’s rare that we come across a respondent who is so engaged with our methodology and questionnaire structure; it’s actually quite refreshing.

    You raise an interesting point re: the website Attribute ratings that are posed in the second portion of our research. On the surface, your point appears to be sound: precluding the visitor from indicating no opinion would seem to be an invitation for wonky data. But the numbers tell a different story. Time and again, our website Attributes evince mesokurtic scoring distributions and don’t in fact show any evidence of the kind of skews you intimated would exist using our scale.

    I can’t really offer such a scientifically reasoned defense of our 500-charater limit on the date of birth question (though, if our character limits ever were to become a competitive disadvantage, I’m sure our clients would certainly let us know).

    Thanks again for taking the time to blog, I wish I could evoke this kind of engagement and interest from everyone :)
    Keep up the good work.

    Jonathan Levitt
    Vice President Marketing
    iPerceptions Inc.

  2. Post author comments:

    Hello Jonathan,

    I think I was a bit too impolite to actually deserve thanks, but you’re welcome.

    The fact that the distribution of scores seem mesokurtic doesn’t really say that the results are all meaningful. Consider:
    1. The data does, in fact, contain meaningless scores. Because you do ask people to rate things they can’t reliably rate. If there were sections of the site I didn’t feel I can rate correctly, but had to, then my ratings/scores are not meaningful. And I’m not the only one. When you know that some answers are meaningless, it shouldn’t really matter how the junk data is distributed compared to the real data.
    2. If some people just answered in “random”, the scores will still have a mesokurtic distribution (especially taking into account point #3 bellow).
    3. For most sites I would expect a somewhat consistent quality across different sections. So it’s quite possible that people who can’t give a real score, will give a score matching what they gave a section that they actually were knowledgeable about. But by not giving people the option to not score, you’re forcing the results to match this expectation, instead of just looking at a real data and see if it matches it or not.

    Or, to make it simple, even if the score distribution is nice, it’s no reason to force people to give you false data, and then to claim that the false data looks nice and so you’re happy with it.

    I don’t know you well enough to guess whether your response to the year of birth issue was supposed to be funny or not. I’ll give you the benefit of the doubt and assume it was. Even though it does somewhat matches the attitude of the previous response, in that you claim not to really care if something is wrong as long as you don’t notice any obvious consequences.

    Getting engagement and interest is good. Great even, considering that you’re in marketing, and that’s pretty much what you’re supposed to do. But don’t forget that the job of the company you work for, for which you’re supposed to provide this marketing, is to get good survey data for the company’s customers. If you get engagement and interest by making mistakes in the way you get this data for the company’s customers, then you (as marketing) may still be doing you job well, but the company you work for isn’t.

    * I tell you, as someone who took the survey, that I had to give meaningless responses to some questions, because I didn’t have a way not to. And the form of the question didn’t give me, or you (as iPerceptions), and way to make sure this “noise” won’t get into the data.
    * You have no potential survey takers, at all, who would need more than a 4 digit number to provide their year of birth. Providing a 500 characters text field for that is just wrong, and leaves an opening for (even encourages) junk data there too.

    None of these are good for the core business of your company, in any way you can look at it. Even if it means I, or other people, noticed your survey more than I have noticed other surveys. Good marketing, by providing bad service to existing customers, isn’t a good strategy.

  3. Candace says:

    I have to say I thoroughly your posts and your dry sense of humor.
    As someone with a marketing background, I can attest to the fact that often such surveys are done purely to prove a predetermined point and not to indicate real data or honest results. I notice that Jonathon from iPerceptions did not address your point regarding the invalid email address. I believe this is because that was done on purpose. They really don’t want your feedback during the survey.
    For instance, if I were to perform a survey regarding dental emarketing, I would probably already have decided on what results I wanted to see and would create the forms based on this. It’s much easier to give the client the results he expects, or those which make the marketing firm more money!

  4. Post author comments:

    Hello Candace,

    Sadly, you’re probably correct. If they already know what they want the results to be, then it doesn’t matter how well they collect the data, or don’t.
    It just seems like a shame. That’s an actual opportunity to get real feedback from potential customers/users, to know what they want, and what they think needs to be improved. Wasting this chance, and wasting the time and good will of people who agreed to try to help by participating in a survey, is stupid.

Leave a Reply

You must be logged in to post a comment.