Comments on: This is why you should let someone experienced do surveys. Or, well, not.

By: Yaron

Yaron — Wed, 25 Feb 2009 07:59:05 +0000

Hello Candace,

Sadly, you’re probably correct. If they already know what they want the results to be, then it doesn’t matter how well they collect the data, or don’t.
It just seems like a shame. That’s an actual opportunity to get real feedback from potential customers/users, to know what they want, and what they think needs to be improved. Wasting this chance, and wasting the time and good will of people who agreed to try to help by participating in a survey, is stupid.

By: Candace

Candace — Wed, 25 Feb 2009 02:16:09 +0000

I have to say I thoroughly your posts and your dry sense of humor.
As someone with a marketing background, I can attest to the fact that often such surveys are done purely to prove a predetermined point and not to indicate real data or honest results. I notice that Jonathon from iPerceptions did not address your point regarding the invalid email address. I believe this is because that was done on purpose. They really don’t want your feedback during the survey.
For instance, if I were to perform a survey regarding dental emarketing, I would probably already have decided on what results I wanted to see and would create the forms based on this. It’s much easier to give the client the results he expects, or those which make the marketing firm more money!

By: Yaron

Yaron — Wed, 02 Jul 2008 17:06:29 +0000

Hello Jonathan,

I think I was a bit too impolite to actually deserve thanks, but you’re welcome.

The fact that the distribution of scores seem mesokurtic doesn’t really say that the results are all meaningful. Consider:
1. The data does, in fact, contain meaningless scores. Because you do ask people to rate things they can’t reliably rate. If there were sections of the site I didn’t feel I can rate correctly, but had to, then my ratings/scores are not meaningful. And I’m not the only one. When you know that some answers are meaningless, it shouldn’t really matter how the junk data is distributed compared to the real data.
2. If some people just answered in “random”, the scores will still have a mesokurtic distribution (especially taking into account point #3 bellow).
3. For most sites I would expect a somewhat consistent quality across different sections. So it’s quite possible that people who can’t give a real score, will give a score matching what they gave a section that they actually were knowledgeable about. But by not giving people the option to not score, you’re forcing the results to match this expectation, instead of just looking at a real data and see if it matches it or not.

Or, to make it simple, even if the score distribution is nice, it’s no reason to force people to give you false data, and then to claim that the false data looks nice and so you’re happy with it.

I don’t know you well enough to guess whether your response to the year of birth issue was supposed to be funny or not. I’ll give you the benefit of the doubt and assume it was. Even though it does somewhat matches the attitude of the previous response, in that you claim not to really care if something is wrong as long as you don’t notice any obvious consequences.

Getting engagement and interest is good. Great even, considering that you’re in marketing, and that’s pretty much what you’re supposed to do. But don’t forget that the job of the company you work for, for which you’re supposed to provide this marketing, is to get good survey data for the company’s customers. If you get engagement and interest by making mistakes in the way you get this data for the company’s customers, then you (as marketing) may still be doing you job well, but the company you work for isn’t.

Again:
* I tell you, as someone who took the survey, that I had to give meaningless responses to some questions, because I didn’t have a way not to. And the form of the question didn’t give me, or you (as iPerceptions), and way to make sure this “noise” won’t get into the data.
* You have no potential survey takers, at all, who would need more than a 4 digit number to provide their year of birth. Providing a 500 characters text field for that is just wrong, and leaves an opening for (even encourages) junk data there too.

None of these are good for the core business of your company, in any way you can look at it. Even if it means I, or other people, noticed your survey more than I have noticed other surveys. Good marketing, by providing bad service to existing customers, isn’t a good strategy.

By: Jonathan Levitt

Jonathan Levitt — Wed, 02 Jul 2008 02:52:31 +0000

I just wanted to personally thank you for your feedback. In the thousands of website optimization surveys we serve every day, it’s rare that we come across a respondent who is so engaged with our methodology and questionnaire structure; it’s actually quite refreshing.

You raise an interesting point re: the website Attribute ratings that are posed in the second portion of our research. On the surface, your point appears to be sound: precluding the visitor from indicating no opinion would seem to be an invitation for wonky data. But the numbers tell a different story. Time and again, our website Attributes evince mesokurtic scoring distributions and don’t in fact show any evidence of the kind of skews you intimated would exist using our scale.

I can’t really offer such a scientifically reasoned defense of our 500-charater limit on the date of birth question (though, if our character limits ever were to become a competitive disadvantage, I’m sure our clients would certainly let us know).

Thanks again for taking the time to blog, I wish I could evoke this kind of engagement and interest from everyone :)
Keep up the good work.

Jonathan Levitt
Vice President Marketing
iPerceptions Inc.