Tribal HQ

An invitee of the Vanguard Investor Survey sent us the link so we could provide comment. We thought it was perfect for our Survey Tips.

This is a screen dump of the survey:

Tribe Research’s feedback

We didn’t see the email invitation, but according to the start of the survey their aim is to better understand the views and needs of their investors.

How do they know only investors are participating? I participated twice (more about that below) and I’m not an investor. Although the link couldn’t be easily found on their website and it seems that you probably need to be provided with the link to find it. Do they know that only investors are accessing the survey? It is on their website, not an external one.

Technical issues

• Several of the questions have default selected answers. The screen dump above is as the survey loads. Having default responses, results in leading the participant to answer in a certain way, mainly because participants do the easiest option. This is similar to an earlier video post on survey bias.
• The survey allows you to click submit without changing anything on the survey. I submitted to see where I’d be redirected. They could work out if people have submitted the survey like this because the the first question on recommendation goes from 0-10 (where 0 is unlikely and 10 is likely) whereas the other questions start from positive and go to negative. The defaults are all the first option so it goes against natural thinking that someone would rate 0 for recommend (unlikely and default) then rate excellent (the default) for overall client services rating and quality of newsletter.
• The survey can be submitted more than once, from the same browser, in the same session. This makes me think that that the survey doesn’t collect cookies, in keeping with their statement ‘completely anonymous’, but also means that they could get erroneous responses without knowing.
• On a smart phone the survey doesn’t have a submit button. This is an increasingly important issue and the investment in considering various browsers needs to be incorporated in survey design.

Questions

• First question – On a scale of 0-10 (ten being very likely and zero being not at all likely) please tell us how likely it is that you would recommend Vanguard Investments to a family member, friend or colleague? – There are several issues with this question:
  • It is based on the Net Promoter question, but not asked in the same way so comparisons to their benchmarks will be corrupted by the different question format.
    

    Although on the survey you wouldn't put Detracters, Passives, Promoters.

  • The scale should be after the question, so that the participant finds it easier to orientate their thinking. They need to know the question then the scale to answer about.
  • The scale and the descriptive text about the scale are reversed making it complicated for the participant to easily answer:
    • On a scale of 0-10
    • ten being very likely and zero being not at all likely
  • If you want to include a default, then make it ’5′, the mid point so that participants are thinking about recommendation being above or below it, rather than thinking relative to an extreme like 0 or 10.
  • We don’t know the background to the survey, but it is important to think about the order of the survey questions. Do you want to ask about recommendation before or after they’ve considered the aspects about their experience that you’ve included in the rest of the survey?
• Second question is just ‘please elaborate’. This is very open and therefore not encouraging response. “Please elaborate about how we could improve or why you’d recommend us?”
• Third question “Overall how yould you rate Vanguard client services?”
  • The scale goes from positive to negative, we recommend negative to positive as that is the direction that the mind naturally goes.
  • The default is on ‘Excellent’ so participants will natually leave it there unless they strongly feel another way. This will bias their results in terms of business planning but will look great for marketing.
• Fourth question is just ‘please elaborate’. Again we wonder how many useful responses they get to this very open question.
• For the following two questions, ‘How would you rate the quality and content in the Helm newsletter? Followed by ‘Please elaborate’ has the same comments as above. In addition:
  • ‘quality’ and ‘content’ are two different things. While to some they may appear the same, if Helm is printed on flimsy paper but has great content then the quality could be rated as poor but the content could be high. Alternatively, if the content is poor but printed on great paper then the alternative could be true. This is just an example, we haven’t seen Helm, but are wary of questions that have ‘and’ in them.
  • They could also ask useful other questions, such as:
    • How often do you read Helm?
    • Do you give your copy of Helm to anyone else? Or, do you share the link of Helm on social media?
• The following four questions are started with ‘Thinking about each of the forms of communication you engage in with Vanguard, please provide a rating beside each of the following’
  • While these questions are consistent with the previous in terms of the scale used, they go in a positive to negative  approach that isn’t consistent to natural thinking.
  • Does the mail and print communcations include Helm, that was previously asked about? In reviewing the survey it was still in our minds so the rating could be blurred by the participants similar flow of thinking.
  • The online question assumes that they are not thinking about the current survey. An additional question about browser use or having cookies set to collect the browser they are using would assist in understanding the browser use of their participants.
• The question “What is the current value of your investment with Vanguard?” is set as a default of $0-$50,000. The default indicates that they think the majority of their participants are at that level, and those that don’t want to report the level of their investment won’t change the default to another response. This will skew their perception of respondents to the lowest level of investment. Were these segments set up from an understanding of behaviour from different levels of benefit for Vanguard or an arbitary convenience?
• The final question “Please provide any comments on how Vanguard can improve our service to you, or please add any general comments below“, is a good question, although a little long. We recommend: “If there is one thing we could improve what would it be?” as this give a direction to one thing and is about improvement.

Where is the feedback cycle?

What do participants get for giving their time to Vanguard? Do they get a response from Vanguard about what they are going to do with the feedback? An incentive isn’t always necessary, but just getting a screen saying ‘Many thanks for taking the time to fill out this survey’ says nothing about how it will be analysed or if the feedback will be used in marketing or business planning. Does that lack of feedback help customer engagement? We think ‘no’.

A short survey is good, but if the questions in the short survey are not well designed then it isn’t useful, no matter how many participate.

If you’d like us to comment on a live survey you’ve seen, then contact us through our website, twitter or facebook.

Wondering if your survey is going to send you down the wrong path? Our Survey Design Review could help or our Survey Design masterclasses for small business and member organisations if you’re in Sydney, Brisbane or Melbourne. If you’re outside these areas and interested then contact us so we can plan for expanding the program.