How to calculate response rates for online surveys

Calculating the response rate for an online survey is an important but sometimes difficult process. Here are some tips to help you calculate an accurate response rate.

For surveys in which potential respondents are invited by email, the traditional practice of calculating response rate as the number of completed interviews divided by the number of invitations generally understates the response rate because it assumes that all invitation were received and opened. Because of the prevalence of spam and spam filters, and the simple fact that many people are inundated with email messages, a more accurate way of measuring response rate is needed.

A truer measure of response rate is calculated as the number of completed interviews divided by a) the number of invitations sent minus b) the number of bounce backs. However, even this calculation can understate the response rate because:

  1. Not all failed emails generate a bounce back
  2. It still assumes that all emails that were received were seen by the recipients

Although it is possible to count the number of email invitations that were seen (not just received, but opened), the process by which this is done can be detrimental to the study.

Return Receipts

The option of requesting a return receipt is included in many email clients, but is not a good option because it is so intrusive. Generally, when a sender requests a return receipt the recipient’s email client posts a message such as: “This user has requested a return receipt. Allow? Yes/No”. If the recipient clicks “No” (and many do as a standard response), that is the end of that.

Using an Image

This method is more effective but a lot sneakier. In this process, the survey invitation is formatted in HTML and includes a tiny image tag (often only one pixel) such as:

<img src='http://yoursite.com/UserOpenedEmail.aspx?UID=username@domain.com'>

As soon as the email is opened or displayed in the preview pane, the website receives a page request and extracts the email address from the querystring to confirm that the email sent to that address was opened.

Sounds great, right? Well, maybe not. One problem is that not all recipients receive email in HTML format (many use text only). Another problem is that many email clients now default to not automatically downloading images. But the biggest problem is that this method seems very underhanded. If potential recipients discover that you are doing it, they are likely to feel you are trying to deceive them. As a result, they are less inclined to complete the survey or provide frank responses.

The bottom line: the best practice for calculating the response rate is to divide the number of completed interviews by the number of invitations sent minus the number of bounce backs. Although it is possible to achieve a more accurate calculation, the risks of doing so generally outweigh the benefits.

About Todd Hollander

Hollander is Founder and President of Todd Hollander Market Research. He has over 25 years of experience in the design and analysis of strategic market research, is an instructor in the University of Georgia’s "Principles of Marketing Research" course, and is the author of "We’re Killing Our Kids," a highly-acclaimed book on the childhood obesity epidemic, and “The Online Survey Doctor,” a weblog dispensing expert advice about internet survey research.
This entry was posted in Response Rate and tagged , , , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>