Todd Hollander Market Research

As an incentive for online survey participation, some companies offer to share the survey results with respondents. Here are the pros and cons for this type of incentive.

Pros
Offering to share some or all of the survey results with respondents can sometimes be effective with business-to-business research. For employees of certain companies, incentives such as guaranteed rewards and prize drawings are considered conflicts of interest and are therefore not permitted. Thus, the ability to share in the insight gleaned from the survey can be an extra incentive.

Cons
For consumer research, offering to share the results with respondents is rarely an effective incentive because most consumers prefer guaranteed rewards or prize drawings.

For business-to-business research, offering to share the results can increase the likelihood that some respondents who are not qualified to participate in the research will provide dishonest responses to demographic questions in order to be assured that they will be able to complete the survey and receive the results. This can skew the data.

The bottom line: For consumer research, offering to share the results is rarely an incentive for completing the survey. However, for certain business-to-business research projects, this can be an effective incentive.

See also:

• Online survey incentives - Guaranteed rewards
• Online survey incentives - Prize drawings

For online survey incentives, a common alternative to a guaranteed reward is a prize drawing or sweepstakes. Here are the pros and cons for this type of incentive.

Pros
This type of incentive offers several advantages over one-to-one incentives, including:

• Lower overall cost
• Ease of administration
• Little or no shipping expense
• Ability to increase the incentive effect by increasing the odds of winning

Another advantage of the drawing is that it limits the financial exposure of the research organization by setting a fixed cost for the incentive. Even if the number of respondents greatly exceeds expectations, the payout can remain the same.

Cons
If only one or two respondents are to be selected in the prize drawing, it is critically important to select a prize that appeals to potential respondents. For example, a drawing for a $10 gift card probably isn’t much of an incentive compared to a drawing for a $100 prepaid gas card.

Similarly, it is very important to establish an attractive odds of winning the drawing. In my experience, odds that tend to resonate with survey respondents fall in the 1-in-100 to 1-in-250 range.

Many experts caution against using this type of incentive for certain types of research, particularly projects in which respondents can be self-referred. For example, if a Web site has a banner inviting all visitors to take part in a survey and offering them a chance to win a prize for participating, it runs the risk of attracting “professional respondents” (people who complete the survey just to win the reward). The responses of such respondents are often dishonest.

The bottom line: Although there is no definitive research proving that prize drawings are always more effective than one-to-one incentives, it is logical to conclude that more times than not the chance to win a significant prize motivates more people than a guarantee to receive a cheap trinket or bauble.

See also:

• Online survey incentives - Guaranteed rewards
• Online survey incentives - Results sharing

Incentives offered to everyone who completes the survey are known as “one-to-one rewards” or “direct incentives.” Here are the pros and cons for this type of incentive.

Pros
The argument in favor of this type of incentive is that it increases the response rate because every respondent is rewarded for participating. When surveying professionals (doctors, attorneys, etc.), a fairly high cash reward for each respondent is generally the norm.

Cons
For consumer surveys, many projects lack a sufficient budget to offer an incentive that actually appeals to potential respondents. In the U.S., the average budget for a consumer incentive is approximately one dollar. Thus, the preponderance of stress balls, golf tees, etc.

Another problem with one-to-one incentives is the lack of an effective method for making small cash payments to large numbers of people. Many panel providers have worked around this problem by creating their own micropayment systems, usually based on respondents collecting points that can be exchanged for cash or merchandise.

The requirement to ship the incentive to each respondent significantly reduces the list of possible incentive items because only very lightweight items can be affordably mailed to respondents. As a rule, if the total cost for a one-to-one incentive (product cost plus shipping cost) exceeds one dollar, this approach generally loses its cost advantage.

Many experts caution against using incentives for some kinds of research, particularly projects in which respondents can be self-referred. For example, if a Web site has a banner inviting all visitors to take part in a survey and guaranteeing them a prize for participating, it runs the risk of attracting “professional respondents” (people who complete the survey just to win the reward). The responses of such respondents are often dishonest.

A final consideration with one-to-one incentives is that it can be important to take preventative measures to safeguard the budget from an unexpectedly high response rate. If the survey programming includes a limit on the number of interviews that will be completed, the survey is closed to respondents as soon as the quota is achieved.

The bottom line: one-to-one incentives can be effective at increasing response rates, particularly among high-income respondents. For consumer surveys, however, this type of incentive can be less effective.

See also:

• Online survey incentives - Prize drawings
• Online survey incentives - Results sharing

There is ample evidence that respondent incentives increase response rates for online surveys. What is less clear is what type of offer produces the best response.

The three most common types of respondent incentives for online surveys are:

1. One-to-one incentives: A small reward for everyone
2. Prize drawings: A significant reward for one or two people
3. Results sharing: A partial or complete report of results for everyone

Each type of incentive has a unique set of advantages, disadvantages, and applications. Each of the next three installments will examine one of the following:

1. Guaranteed rewards
2. Prize drawings
3. Results sharing

To make sure you receive this information, you can subscribe to this feed in the reader of your choice or bookmark the blog.

Many websites request visitors to take part in an online survey. These invitations generally fall into one of three formats: static text, banners, or pop-ups. Here are the pros and cons of using these types of invitations to find survey respondents.

These types of requests are inexpensive to administer but include two inherent drawbacks:

1. They tend to produce very low response rates
2. Because the sample of respondents is entirely self-selecting, the results are not representative of a larger audience (e.g., all customers)

Thus, although these invitations can provide useful feedback, they are not a good substitute for more traditional methods of locating survey respondents. Think of the information gleaned from this type of survey as qualitative/directional rather than quantitative/conclusive.

For more information about finding respondents, see these posts.

A common mistake made by those conducting online surveys is failing to send reminder emails to potential respondents. Another common mistake is to send out too many reminders. Here is some advice that will help you develop an effective reminder system.

Failing to send reminder emails to potential survey participants can result in unacceptable response rates. Conversely, annoying respondents by sending too many reminders can adversely affect both the response rate and the quality of the data collected. Therefore, it is important to send the right number of reminders with the right wording at the right time.

One of the reasons some researchers fail to send reminders is that they think the reminder should be sent only to those who have not yet completed the survey. The problem is they don’t know how to do it. Although it is possible to identify those who have already completed the survey, the tactics for doing this are risky and can be viewed as underhanded (see “How to calculate response rates for online surveys“).

The good news is that you can send an email reminder to everyone on the mailing list without annoying those who have already completed the survey. Here’s how…

1. Forward the original invitation
2. Change the Subject Line to: Reminder: [original invitation subject line]
3. Add this text at the top of the message: If you have already completed the survey, thank you. If you have not, please see below.

That’s all you need to do. Nine times out of 10, one reminder at the mid-point of the data collection period does the trick. Occasionally, a second reminder is needed. Important: Never send more than two reminder emails - it will only annoy people.

The bottom line: Some people will take your survey as soon as they receive the invitation email. Others will need a follow-up email reminder. One reminder at the mid-point of the data collection period is best. Never send more than two reminder emails.

Calculating the response rate for an online survey is an important but sometimes difficult process. Here are some tips to help you calculate an accurate response rate.

For surveys in which potential respondents are invited by email, the traditional practice of calculating response rate as the number of completed interviews divided by the number of invitations generally understates the response rate because it assumes that all invitation were received and opened. Because of the prevalence of spam and spam filters, and the simple fact that many people are inundated with email messages, a more accurate way of measuring response rate is needed.

A truer measure of response rate is calculated as the number of completed interviews divided by a) the number of invitations sent minus b) the number of bounce backs. However, even this calculation can understate the response rate because:

1. Not all failed emails generate a bounce back
2. It still assumes that all emails that were received were seen by the recipients

Although it is possible to count the number of email invitations that were seen (not just received, but opened), the process by which this is done can be detrimental to the study.

Return Receipts

The option of requesting a return receipt is included in many email clients, but is not a good option because it is so intrusive. Generally, when a sender requests a return receipt the recipient’s email client posts a message such as: “This user has requested a return receipt. Allow? Yes/No”. If the recipient clicks “No” (and many do as a standard response), that is the end of that.

Using an Image

This method is more effective but a lot sneakier. In this process, the survey invitation is formatted in HTML and includes a tiny image tag (often only one pixel) such as:

<img src='http://yoursite.com/UserOpenedEmail.aspx?UID=username@domain.com'>

As soon as the email is opened or displayed in the preview pane, the website receives a page request and extracts the email address from the querystring to confirm that the email sent to that address was opened.

Sounds great, right? Well, maybe not. One problem is that not all recipients receive email in HTML format (many use text only). Another problem is that many email clients now default to not automatically downloading images. But the biggest problem is that this method seems very underhanded. If potential recipients discover that you are doing it, they are likely to feel you are trying to deceive them. As a result, they are less inclined to complete the survey or provide frank responses.

The bottom line: the best practice for calculating the response rate is to divide the number of completed interviews by the number of invitations sent minus the number of bounce backs. Although it is possible to achieve a more accurate calculation, the risks of doing so generally outweigh the benefits.

In addition to controlling access to online surveys, it is also important to ensure that your results are not skewed by respondents who did not answer questions seriously or truthfully. Here are a few tips to ensure that your final data contains quality responses from valid respondents.

The ever-increasing use of online surveys for market research has produced the unfortunate consequence of some survey respondents who are more interested in receiving a reward (incentive) for completing the survey than they are with providing accurate, thoughtful survey responses. These people are often referred to as “Professional Respondents.” Although they represent only a small minority of survey respondents, it only takes a few such people to skew your results.

Here are some methods you can use to reduce the odds of having respondents try to “game the system.”

1. Use a reputable panel with a strong (and published) set of verification standards. (For more information, see
   How to find respondents for your online survey (Part 2: Survey Panels)
2. Implement deterrents such as a legal/fraudulent access agreement that a respondent agree to comply with before they completing the survey.
3. Examine the data to identify fraudulent or irresponsible responses. For example, on a series of rating scale questions a pattern such as 7,7,7,7,7 or 1,2,3,4,5,1,2,3,4,5 may indicate this. Note that many full-featured survey systems (as opposed to the free or low-cost ones) use computer programs to identify and remove such responses.
4. Thoroughly check the final data (e.g., compare questions where the answers should not conflict, and review responses to open-ended questions).

The bottom line: a little extra effort to prevent fraudulent or irresponsible respondents from skewing your online survey data will significantly increase your confidence in the validity and accuracy of the results.

When conducting online surveys, researchers generally want to prevent unauthorized access by:

1. Restricting participation solely to those who were invited to respond.
2. Allowing each respondent to complete the survey only once.

Here is some practical advice for achieving these important objectives.

(more…)

Online surveys can be configured so a respondent is required to provide a response to a question before proceeding to the next question. This practice is known as forced completion. Here are the arguments for and against this practice.

(more…)