Todd Hollander Market Research

Many websites request visitors to take part in an online survey. These invitations generally fall into one of three formats: static text, banners, or pop-ups. Here are the pros and cons of using these types of invitations to find survey respondents.

These types of requests are inexpensive to administer but include two inherent drawbacks:

1. They tend to produce very low response rates
2. Because the sample of respondents is entirely self-selecting, the results are not representative of a larger audience (e.g., all customers)

Thus, although these invitations can provide useful feedback, they are not a good substitute for more traditional methods of locating survey respondents. Think of the information gleaned from this type of survey as qualitative/directional rather than quantitative/conclusive.

For more information about finding respondents, see these posts.

A common mistake made by those conducting online surveys is failing to send reminder emails to potential respondents. Another common mistake is to send out too many reminders. Here is some advice that will help you develop an effective reminder system.

Failing to send reminder emails to potential survey participants can result in unacceptable response rates. Conversely, annoying respondents by sending too many reminders can adversely affect both the response rate and the quality of the data collected. Therefore, it is important to send the right number of reminders with the right wording at the right time.

One of the reasons some researchers fail to send reminders is that they think the reminder should be sent only to those who have not yet completed the survey. The problem is they don’t know how to do it. Although it is possible to identify those who have already completed the survey, the tactics for doing this are risky and can be viewed as underhanded (see “How to calculate response rates for online surveys“).

The good news is that you can send an email reminder to everyone on the mailing list without annoying those who have already completed the survey. Here’s how…

1. Forward the original invitation
2. Change the Subject Line to: Reminder: [original invitation subject line]
3. Add this text at the top of the message: If you have already completed the survey, thank you. If you have not, please see below.

That’s all you need to do. Nine times out of 10, one reminder at the mid-point of the data collection period does the trick. Occasionally, a second reminder is needed. Important: Never send more than two reminder emails - it will only annoy people.

The bottom line: Some people will take your survey as soon as they receive the invitation email. Others will need a follow-up email reminder. One reminder at the mid-point of the data collection period is best. Never send more than two reminder emails.

Calculating the response rate for an online survey is an important but sometimes difficult process. Here are some tips to help you calculate an accurate response rate.

For surveys in which potential respondents are invited by email, the traditional practice of calculating response rate as the number of completed interviews divided by the number of invitations generally understates the response rate because it assumes that all invitation were received and opened. Because of the prevalence of spam and spam filters, and the simple fact that many people are inundated with email messages, a more accurate way of measuring response rate is needed.

A truer measure of response rate is calculated as the number of completed interviews divided by a) the number of invitations sent minus b) the number of bounce backs. However, even this calculation can understate the response rate because:

1. Not all failed emails generate a bounce back
2. It still assumes that all emails that were received were seen by the recipients

Although it is possible to count the number of email invitations that were seen (not just received, but opened), the process by which this is done can be detrimental to the study.

Return Receipts

The option of requesting a return receipt is included in many email clients, but is not a good option because it is so intrusive. Generally, when a sender requests a return receipt the recipient’s email client posts a message such as: “This user has requested a return receipt. Allow? Yes/No”. If the recipient clicks “No” (and many do as a standard response), that is the end of that.

Using an Image

This method is more effective but a lot sneakier. In this process, the survey invitation is formatted in HTML and includes a tiny image tag (often only one pixel) such as:

<img src='http://yoursite.com/UserOpenedEmail.aspx?UID=username@domain.com'>

As soon as the email is opened or displayed in the preview pane, the website receives a page request and extracts the email address from the querystring to confirm that the email sent to that address was opened.

Sounds great, right? Well, maybe not. One problem is that not all recipients receive email in HTML format (many use text only). Another problem is that many email clients now default to not automatically downloading images. But the biggest problem is that this method seems very underhanded. If potential recipients discover that you are doing it, they are likely to feel you are trying to deceive them. As a result, they are less inclined to complete the survey or provide frank responses.

The bottom line: the best practice for calculating the response rate is to divide the number of completed interviews by the number of invitations sent minus the number of bounce backs. Although it is possible to achieve a more accurate calculation, the risks of doing so generally outweigh the benefits.

In addition to controlling access to online surveys, it is also important to ensure that your results are not skewed by respondents who did not answer questions seriously or truthfully. Here are a few tips to ensure that your final data contains quality responses from valid respondents.

The ever-increasing use of online surveys for market research has produced the unfortunate consequence of some survey respondents who are more interested in receiving a reward (incentive) for completing the survey than they are with providing accurate, thoughtful survey responses. These people are often referred to as “Professional Respondents.” Although they represent only a small minority of survey respondents, it only takes a few such people to skew your results.

Here are some methods you can use to reduce the odds of having respondents try to “game the system.”

1. Use a reputable panel with a strong (and published) set of verification standards. (For more information, see
   How to find respondents for your online survey (Part 2: Survey Panels)
2. Implement deterrents such as a legal/fraudulent access agreement that a respondent agree to comply with before they completing the survey.
3. Examine the data to identify fraudulent or irresponsible responses. For example, on a series of rating scale questions a pattern such as 7,7,7,7,7 or 1,2,3,4,5,1,2,3,4,5 may indicate this. Note that many full-featured survey systems (as opposed to the free or low-cost ones) use computer programs to identify and remove such responses.
4. Thoroughly check the final data (e.g., compare questions where the answers should not conflict, and review responses to open-ended questions).

The bottom line: a little extra effort to prevent fraudulent or irresponsible respondents from skewing your online survey data will significantly increase your confidence in the validity and accuracy of the results.

When conducting online surveys, researchers generally want to prevent unauthorized access by:

1. Restricting participation solely to those who were invited to respond.
2. Allowing each respondent to complete the survey only once.

Here is some practical advice for achieving these important objectives.

(more…)

This table summarizes the top 20 countries by Internet users as of June 30, 2007.

NOTES

1. The table above was downloaded from Internetworldstats.com on January 22, 2008.
2. Internet Usage Statistics are for June 30, 2007.
3. User information comes from data published by Nielsen//NetRatings, International Telecommunications Union, Official country reports, and other trustworthy research sources.

This table summarizes Internet populations and penetrations by World Region as of November 30, 2007.

NOTES

1. The table above was downloaded from Internetworldstats.com on January 18, 2008.
2. Internet Usage and World Population Statistics are for November 30, 2007.
3. Demographic (Population) numbers are based on data from the US Census Bureau.
4. Internet usage information comes from data published by Nielsen//NetRatings, by the International Telecommunications Union, by local NIC, and other reliable sources.

Looking for free data on internet usage? Here are some good sources of worldwide data�

There are multiple sources that measure the Internet-using population of any given country. Due to methodological differences, data for the same location can vary widely depending on the source. Here are some of the better ones.

Online surveys can be configured so a respondent is required to provide a response to a question before proceeding to the next question. This practice is known as forced completion. Here are the arguments for and against this practice.

(more…)

In Part 1 of this 3-part series, we learned that 87 percent of online surveys conducted in the United States in 2006 were deployed using a preexisting survey panel. In Part 2, we explored why panels are the preferred method for online interviewing and how to select a panel provider for a market research project.

In this final installment, we will discuss the pros and cons of another popular method of recruiting respondents for online surveys: Customer Lists.

(more…)