Tribal HQ

There are several reasons why I pull out the 2004 Ralph Magazine Readers’ Survey in my Survey Design Masterclass.

Many of our Survey Tips have been on what people do wrong when they design a survey. This time I’m sharing what was done well. While it isn’t perfect, no survey is, it has many strengths.

Appealing to your audience

Ralph Magazine describes itself as catering to the fun loving Australian male who has a fine appreciation for the good looking women of the world.

The survey design did this in several ways

• The cover image was a good looking female
• The incentive was winning a year’s supply of Coopers beer
• The language of the survey is suitable to the market; eg: How long have you been reading Ralph?
  • Not long (less than one year)
  • Quite a while (one to three years)
  • Yonks (more than three years)
  • Ever since Golden Books got boring

Survey is created like a story

The survey starts with easy questions to draw the participants in with them thinking ‘this isn’t too bad, I can help them out by answering this’. A question like, How often do you read Ralph? is an easy to answer first question.

The questions in the middle require more thinking. They include some open responses and matrix questions, but the participant is already in the flow of answering the questions so willing to answer questions that require a little more thought.

It ends with personal questions – Gender, age, marital status, employment status, occupation and personal income. Participants at this stage are more engaged, they’ve almost finished and have an understanding of what the survey is about so are more willing to provide that kind of information.

Had a clear aim so it isn’t too long

There are three pages of questions, while not a mini survey, it isn’t too long for participants to think it is too much work to complete. The three pages also don’t look too full, they have included white space, making it easier to complete.

The aim was clear to them, and you can see that in the survey design so they didn’t extend it with wasted questions. It was a reader survey. They wanted to know about:

• views about Ralph content, the way it is read and how often
• advertising take up and buying interests
• demographics of readers

Keep participants engaged to the end

The survey ends with this…

End notes: Tribe Research didn’t design the survey and don’t necessarily agree with the magazine’s content or all the language in the survey, but it is a good example of a survey in terms of the participants they wanted to complete the survey.

Looking to design your own survey and want to make sure that it is well designed? Our Survey Design Review will have experts review the survey before it’s live.

A while ago we asked on our Facebook page ‘What are your poor survey design stories?’.

Lara Solomon (@LaRoo) said:

I have to say that I shamelessly often fill in survey’s in hotels and restaurants when I get bad service, and there is that box where you can choose if you want to be contacted about your comments, I always tick yes…. needless to say I am still waiting on the calls…. To me these are really badly designed, as obviously no one reads them, so it tells me a lot about the company, I think some companies do it because they think they have to, but if nothing is done with it what is the point?

Often a core problem with research is that asking the questions is thought to be the research. There can be a process in the back engine of the business that looks at the feedback, but if you don’t complete the feedback cycle – communicate back to the customer – then they’re left dissatisfied no matter how well the survey is designed. They don’t know you’ve listened if you don’t tell them.

This is why the feedback cycle so important:

• Know Your Tribe – Explore (ask questions) and Uncover (analyse the answers)
• Tribal Know-How – Clear-head (planning session) and Drive Change (implement)
• Tribal Voice – Communicate back

When you do a survey, ensure that you’ve got the resources (time, staff, procedure) to follow up the feedback. Not following up the feedback afterwards will cause a bigger problem than the customer not feeling they can provide feedback in the first place or the problem they had.

Worried your survey is going to send you down the wrong path? Our Customer Feedback Survey Design Masterclasses for small business will help you design a personalised survey ready for implementation.

An invitee of the Vanguard Investor Survey sent us the link so we could provide comment. We thought it was perfect for our Survey Tips.

This is a screen dump of the survey:

Tribe Research’s feedback

We didn’t see the email invitation, but according to the start of the survey their aim is to better understand the views and needs of their investors.

How do they know only investors are participating? I participated twice (more about that below) and I’m not an investor. Although the link couldn’t be easily found on their website and it seems that you probably need to be provided with the link to find it. Do they know that only investors are accessing the survey? It is on their website, not an external one.

Technical issues

• Several of the questions have default selected answers. The screen dump above is as the survey loads. Having default responses, results in leading the participant to answer in a certain way, mainly because participants do the easiest option. This is similar to an earlier video post on survey bias.
• The survey allows you to click submit without changing anything on the survey. I submitted to see where I’d be redirected. They could work out if people have submitted the survey like this because the the first question on recommendation goes from 0-10 (where 0 is unlikely and 10 is likely) whereas the other questions start from positive and go to negative. The defaults are all the first option so it goes against natural thinking that someone would rate 0 for recommend (unlikely and default) then rate excellent (the default) for overall client services rating and quality of newsletter.
• The survey can be submitted more than once, from the same browser, in the same session. This makes me think that that the survey doesn’t collect cookies, in keeping with their statement ‘completely anonymous’, but also means that they could get erroneous responses without knowing.
• On a smart phone the survey doesn’t have a submit button. This is an increasingly important issue and the investment in considering various browsers needs to be incorporated in survey design.

Questions

• First question – On a scale of 0-10 (ten being very likely and zero being not at all likely) please tell us how likely it is that you would recommend Vanguard Investments to a family member, friend or colleague? – There are several issues with this question:
  • It is based on the Net Promoter question, but not asked in the same way so comparisons to their benchmarks will be corrupted by the different question format.
    

    Although on the survey you wouldn't put Detracters, Passives, Promoters.

  • The scale should be after the question, so that the participant finds it easier to orientate their thinking. They need to know the question then the scale to answer about.
  • The scale and the descriptive text about the scale are reversed making it complicated for the participant to easily answer:
    • On a scale of 0-10
    • ten being very likely and zero being not at all likely
  • If you want to include a default, then make it ’5′, the mid point so that participants are thinking about recommendation being above or below it, rather than thinking relative to an extreme like 0 or 10.
  • We don’t know the background to the survey, but it is important to think about the order of the survey questions. Do you want to ask about recommendation before or after they’ve considered the aspects about their experience that you’ve included in the rest of the survey?
• Second question is just ‘please elaborate’. This is very open and therefore not encouraging response. “Please elaborate about how we could improve or why you’d recommend us?”
• Third question “Overall how yould you rate Vanguard client services?”
  • The scale goes from positive to negative, we recommend negative to positive as that is the direction that the mind naturally goes.
  • The default is on ‘Excellent’ so participants will natually leave it there unless they strongly feel another way. This will bias their results in terms of business planning but will look great for marketing.
• Fourth question is just ‘please elaborate’. Again we wonder how many useful responses they get to this very open question.
• For the following two questions, ‘How would you rate the quality and content in the Helm newsletter? Followed by ‘Please elaborate’ has the same comments as above. In addition:
  • ‘quality’ and ‘content’ are two different things. While to some they may appear the same, if Helm is printed on flimsy paper but has great content then the quality could be rated as poor but the content could be high. Alternatively, if the content is poor but printed on great paper then the alternative could be true. This is just an example, we haven’t seen Helm, but are wary of questions that have ‘and’ in them.
  • They could also ask useful other questions, such as:
    • How often do you read Helm?
    • Do you give your copy of Helm to anyone else? Or, do you share the link of Helm on social media?
• The following four questions are started with ‘Thinking about each of the forms of communication you engage in with Vanguard, please provide a rating beside each of the following’
  • While these questions are consistent with the previous in terms of the scale used, they go in a positive to negative  approach that isn’t consistent to natural thinking.
  • Does the mail and print communcations include Helm, that was previously asked about? In reviewing the survey it was still in our minds so the rating could be blurred by the participants similar flow of thinking.
  • The online question assumes that they are not thinking about the current survey. An additional question about browser use or having cookies set to collect the browser they are using would assist in understanding the browser use of their participants.
• The question “What is the current value of your investment with Vanguard?” is set as a default of $0-$50,000. The default indicates that they think the majority of their participants are at that level, and those that don’t want to report the level of their investment won’t change the default to another response. This will skew their perception of respondents to the lowest level of investment. Were these segments set up from an understanding of behaviour from different levels of benefit for Vanguard or an arbitary convenience?
• The final question “Please provide any comments on how Vanguard can improve our service to you, or please add any general comments below“, is a good question, although a little long. We recommend: “If there is one thing we could improve what would it be?” as this give a direction to one thing and is about improvement.

Where is the feedback cycle?

What do participants get for giving their time to Vanguard? Do they get a response from Vanguard about what they are going to do with the feedback? An incentive isn’t always necessary, but just getting a screen saying ‘Many thanks for taking the time to fill out this survey’ says nothing about how it will be analysed or if the feedback will be used in marketing or business planning. Does that lack of feedback help customer engagement? We think ‘no’.

A short survey is good, but if the questions in the short survey are not well designed then it isn’t useful, no matter how many participate.

If you’d like us to comment on a live survey you’ve seen, then contact us through our website, twitter or facebook.

Wondering if your survey is going to send you down the wrong path? Our Survey Design Review could help or our Survey Design masterclasses for small business and member organisations if you’re in Sydney, Brisbane or Melbourne. If you’re outside these areas and interested then contact us so we can plan for expanding the program.

Have you considered the different ways your survey participants will look back and then answer the questions in your survey?

Recall bias is an often forgotten issue when designing a survey.

According to wikipedia:

recall bias is a type of systematic bias which occurs when the way a survey respondent answers a question is affected not just by the correct answer, but also by the respondent’s memory. This can affect the results of the survey.

The recall bias in this example is based on a focus group story from a colleague.

Would you get a different, and more accurate response, if you asked a question slightly differently?

If you would lie to find out more about recall bias or other common issues that occur in surveys, then sign up for either of our masterclasses on survey design for small business and member organisations.

Do you love or hate matrix questions?

There is good way to use them and mistakes that people make.

Consider how enjoyable it is for participants to complete the survey as they might not like them as much as you do.

• Don’t fill your survey with matrix questions, it will just cause people to not complete it.
• Make sure a matrix is not too big (ask for rating of too many aspects) as it is very intimidating to answer so people will either not complete it or ‘flat-line’ (answer the same rating for each question going down the page or screen).

A mindset of avoiding them, can result in not getting some very useful information for analysis and helping your organisation. So here are some good things to remember about matrix questions.

• They are very useful for finding out ratings of aspects on two scales so you can establish priorities. These could be a range of things you want to understand with two ratings: importance and satisfaction, or preference and availability, or importance and accessibility.
• They can have scales that are categorical (low, medium, high) or numerical (1, 2, 3, 4, 5).
• You can then focus on particular aspects easily to develop priorities. For example which aspect has the highest percentage of people rating both high importance and high satisfaction. This is something you should be marketing as a strength. While, the aspect that has the highest percentage of high importance and low satisfaction is something that you need to set as a priority for improvement. This can be established easily without sophisticated statistics software.
• If the scale is numerical then you can work out the average for each aspect on both scales then find out the ‘gap’ to work out where the largest gaps between the two for each aspect. This is another way to work out priorities for driving change.

Data from numerical scales can also be imported into our ActionMap giving you easy to understand priorities – imagine the growth your business could achieve if you had an action plan that allowed you to:

• Know what to market as your unique selling points, to differentiate your business
• Know what to educate your customers about, to increase your perceived value
• Know what to improve, to increase your customers’ happiness and loyalty

Having a matrix in your survey can be useful, if they are used well.

If you are unsure about how you can use matrix questions well in your survey, our Survey Design Review can assist or contact us for further options.

When should you invite people to participate in your survey and what impact will the timing have on the results you get?

A natural disaster like the floods in Australia is a reminder for us professionally about the importance of timing survey distribution and the impact on your results.

There are probably a few of you that are thinking ‘der’, and while this example is fairly obvious, the timing of sending out a survey and what that does to the results is often forgotten.

Firstly, it isn’t only businesses or residents in Queensland that have been affected by floods in January 2011. There is also flooding in NSW and Victoria, and in other countries. People in those communities are going to be more focused on the direct impact of the floods than your survey – regardless of whether it takes 1 or 20 minutes to complete. Right now, they don’t care about your survey.

There are secondary impacts to those outside of the regions directly impacted, such as decreased supply while the affected businesses recover. This causes increased costs and business interruptions to those not directly affected. This will change their priorities in 2011.

In considering this:

• You need to be aware of other factors, such as a flood and other natural and economic disasters, when designing your survey.
• Questions need to be included in your survey that relate to the impact of these as they probably relate to the overall subject of your survey.
• Geographical information is still useful. While more and more business is happening online, we are still physically living in a location that can be affected locally.
• Creating a survey where you can compare the results or the proportion of participants to either national standards (such as Australian Bureau of Statistics data) or your own database, will allow you to understand if a lower proportion of people in affected areas have participated. Your analysis then needs to account for this.
• Acting on the results needs to be fairly fast. Views and opinions can change fast so using the results of a survey you conducted months ago, might not be relevant still. This is why a smaller more targeted survey is probably more useful for your business planning and to drive change in your organisation.

If you are unsure about when you should distribute your survey, or are looking for better solutions, our Survey Design Review can assist or please contact us for further options.

Photo courtesy of Tatters:)

Have you said thank you to your participants?

Often called the two most important words, the simple thank you is often forgotten in a survey as well as other occasions. There are several places where you can thank your survey participants:

• Thank them in the invitation letter or email.
• Thank them when they have almost reached the end.
• Thank them at the end of the survey – before they click submit in an online survey or fold it for posting a paper survey back to you.
• Thank them after they have submitted an online survey so they know they have completed it.
• Thank them afterwards when you are communicating the results back to them.

You thank clients at the end of the year, so why don’t you thank participants at the end of a survey?

You can say thank you too often, but from my observation, this is rarely the issue.

If you are unsure if you have sufficiently thanked your participants, or are looking for better solutions, our Survey Design Review can assist or please contact us for further options.

Want to properly understand employment status when designing a survey?

Employment status is regularly included in a survey as it is useful for many organisations to know this type of information.

It is often asked incorrectly, making the information difficult to analyse or incorrect. And, that isn’t useful to you.

If the participant can only select one response, it is fine if there is only one possible response for each participant. If you are not sure, you can make it a multiple response question, or break the options up to a few questions.

• ‘Student‘ is not an employment status. It can be a separate question if you want to know this information, but they are not the same question.
• Employment is becoming less traditional and therefore more thought needs to be made when designing these questions. Otherwise, they will just select one, regardless of accuracy.

How to ask about these:

• Which of the following best describes your employment status?
  • Working full time paid employment (35 or more hours per week)
  • Working part time paid employment (less than 35 hours per week)
  • Self employed (35 or more hours per week)
  • Self employed (less than 35 hours per week)
  • Casual employment
  • Other form of paid employment
  • Not currently in paid employment
• Are you currently studying?
  • Yes, studying full time
  • Yes, studying part time
  • No, I am not currently undertaking formal study

If you are unsure about the wording of your employment status questions, or are looking for better solutions, our Survey Design Review can assist or contact us for further options.

Are your open questions getting meaningless responses?

Ending your survey with just ‘Other Comments’ doesn’t entice your participants to provide you with feedback that is useful for you to drive change in your business or organisation.

Give your participants some direction so that the feedback is beneficial. Here are some ideas:

• If we could make one implementable change, that would improve your satisfaction, what is it?
• What is one thing we could do differently?
• When you think of [ organisation name ] what are the first 3 words that come to mind?
• What is the main reason you utilise our services?

There are many more, these are just some ideas to get you started.

If you are unsure about how you can word your open response questions for better insights, or are looking for better solutions, check out our Survey Design Review or contact us for further options.

Does your survey introduction cover these essentials? You don’t want this kind of response from someone you invited, that doesn’t become a participant:

So the first main point is, no matter how enticing your survey introduction is, if you contradict it later you will loose all the survey participant’s trust.

Checklist for an introduction:

• Who is conducting the research – you or a market research company?
• What is the aim of the research?
• Why were they invited to participate?
• What is the closing date?
• Is it anonymous or not? It isn’t anonymous if you are sending them a long personalised link or if they have to include their personal details.
• What are you going to do with the results? Is personally identifying information going to be distributed or only grouped anonymous results? Are you going to merge the responses with other data you have about them?
• Who will see their responses?
• Is there an incentive for them to participant? It isn’t always applicable, but should be clear.
• Thank them for their time!

If you are unsure about your introduction and if your survey follows it, or are looking for better solutions, check out our Survey Design Review or contact us for further options.