Open-ended questions are those in which respondents type a response in a text box rather than selecting from a predefined set of responses (radio buttons, checkboxes, drop-down lists, etc). When open-ended questions are used properly, they can provide a number of benefits. However, when used improperly, they can be the kiss of death for your survey.
As noted in a previous entry, one common mistake is to use too many open-ended questions in a survey, which may cause many respondents to abandon the interview. Another common mistake made by inexperienced researchers is to force responses to an open-ended question even when it is likely that many or most respondents will have no valuable response to offer.
An example of this would be the question:
Do you have any other comments or suggestions about our company?
If the survey software is programmed to require a response to this question, it is likely that many respondents will be frustrated when they receive an error message telling them they have to type a response before they can proceed to the next question. When this mistake is made, the open-ended responses often include a rash of expletives.
The bottom line: As a rule, a response to an open-ended question should only be required when it is reasonable to think that every respondent will have a meaningful response to offer. Otherwise, a response should be optional.
In online surveys, open-ended questions are those in which the respondents type their responses in a text box rather than selecting from a set of predefined responses such as radio buttons, checkboxes, or drop-down lists. When used properly, open-ended questions can provide several benefits. However, when used improperly, they can be the kiss of death for your survey.
Pros
Open-ended questions are useful when the range of responses is not tightly defined and/or detailed responses are desired. Advantages include a wider variety and greater depth of responses, and a higher likelihood of receiving unexpected and insightful information. They can also be used to capture any important ideas or opinions that you may not have thought to ask about.
Cons
Open-ended questions have several disadvantages. First, the responses must be read individually because there is no effective way to automatically tabulate or perform statistical analyses on them. This is more expensive and time consuming than the analysis of closed-ended questions and may not be practical for low-budget or time-sensitive research.
These responses are also open to the interpretation of the reader because different people are likely to interpret the meaning of a response in different ways. This potential problem can be avoided by using a single analyst, but a large number of responses can make this impractical.
The most significant drawback of using open-ended questions is that it may significantly increase the number of incompleted interviews (when respondents exit the survey before responding to all of the questions). This is because open-ended questions require more thought, time, and effort from the respondent.
Inexperienced researchers often make the mistake of including too many open-ended questions in a survey. It is important to understand that in most cases respondents are less passionate about the subject matter than the sponsor of the research. If your survey exceeds their tolerance of time and effort, they are likely to abandon the interview.
So how many open-ended questions should be included in a survey?
This is an important question. Unfortunately, there is no simple response. The number of open-ends your respondents will tolerate generally depends on:
The total length of the interview
The incentive for completing the survey
How much they care about the subject matter
The extent to which they believe their responses will be read and considered
The bottom line: Open-ended questions can be highly beneficial but should be used sparingly. Using too many can significantly decrease the response rate for your survey.
When conducting online research, it is essential to have a well-defined set of objectives. Surveys designed in the absence of clear objectives frequently suffer from one or both of the following flaws:
Wasting the sponsor’s time and money by failing to include necessary questions
Wasting the respondent’s time by asking unnecessary questions
The problems stemming from poorly-defined research objectives do not end with data collection but continue on to the analysis stage. Simply stated:
It is impossible to develop meaningful insights if you don’t know what you were expecting to observe.
Step 1: Define the marketing problem
The first step in the process of defining the research objectives is for managers and researchers to discuss and clarify the current situation. This discussion should focus on why the research is needed (i.e., the marketing problem should be defined).
Step 2: Determine the decisions that will be made
Having defined the marketing problem, the next step is for managers and researchers to discuss and agree upon the specific decisions that will be guided by the results of the research. The word “guided” in the previous sentence is an important one. Although some managers hope or expect that the research results will tell them what to do (i.e., make the decisions for them), it is important to remember that:
Effective marketing research is an aid to decision-making, not a substitute.
Step 3: Establish the research objectives
The research objectives should link the marketing problem with the decisions to be made. To this end, a properly constructed set of objectives should outline:
What kind of information will be provided by the research
How much information will be provided
How the research information will be translated into management decisions.
In a sense, these objectives serve as the researcher’s promise to deliver information that can be acted upon.
Step 4: Divide and conquer
While all research projects are designed to gather information, rarely can a single project provide all of the information associated with a product or marketing issue. Thus, rather than falling into the trap of trying to cram 20 gallons of research into a 5-gallon hat, it may be necessary to divide the objectives across two or more research studies.
The bottom line: To define effective research objectives, follow these steps and increase the odds of making the right decisions as a result of the research.
A leading question (also known as a “loaded question”) is one that suggests an answer by the way in which the question or response options are worded. Here’s an example of leading the respondent by the way the question is worded:
How high would you rate this product?
Asking the question in this way leads the respondent to offer a high rating. That is, it sets the expectation that a high rating is the “correct” response. To ask this question in a neutral (i.e., non-leading) way, the word “high” should be eliminated to render the question: “How would you rate this product?” Often a survey question is loaded by an unbalanced set of response options. Take this question for example:
How would you rate this product
Excellent
Great
Good
Fair
Poor
Notice that the question itself is not leading but the response options are. This is because four of the five possible responses are positive and only one (”Poor”) is negative. A closed-ended question should include response options that not only cover the whole range of responses, but that are also equally distributed throughout the range. That is, all responses should be equally likely. Thus, a more appropriate scale would be:
Very Good
Good
Neutral
Bad
Very Bad
This balanced set of response options does not imply an expectation of either a positive or negative response. Nor does it stack the odds that responses will cluster at one end of the spectrum.
The bottom line: To avoid the common mistake of asking leading questions, make sure that none of your survey questions suggest an answer by the way in which the question is worded, and that each question provides response options that not only cover the whole range of responses, but are also equally distributed throughout the range.
Ambiguous questions (those that could be understood in more than one way) are one of the more common mistakes in questionnaire design.
Take this question for example:
How often do you visit our website?
O Very Often
O Often
O Sometimes
O Rarely
O Never
With the exception of “Never,” these response options may mean different things to different people. If some respondents consider once a week as “often” and others consider once a week as “rarely,” the data won’t mean much. Therefore, whenever possible, it is prudent to quantify the choices, such as:
O More than once a day
O Every day
O 2-6 Times a week
O Once a Week
O Less than once a week
O Never
The goal is to eliminate any chance that a question will mean different things to different people. Failing to do this runs the risk that respondents essentially will be answering different questions.
To this end, it is best to use the following guidelines:
Phrase your questions empirically
Avoid the use of adjectives
Avoid colloquial or ethnic expressions
Avoid technical terms that assume preexisting knowledge or experience
The bottom line: To avoid ambiguity, make sure that every question is clear, succinct, and has only one possible meaning.
One of the most common mistakes I see in questionnaire design is the problem of overlapping scales. Take this question for example:
Which of the following includes your age?
O 18-25
O 25-35
O 35-45
O 45-55
O 55+
If the respondent is 35 years old, which response option should she select: “25-35″ or “35-45″? This seems like common sense, yet I see this mistake made time and time again.
There are several significant problems caused by overlapping scales. First, respondents may conclude that the survey sponsor is inept or unsophisticated. Second, this type of error can lead to above-average rates of survey abandonment (when respondents exit the survey before completing it). Finally, and perhaps most importantly, it can significantly skew the survey data.
Here are two bits of advice to make sure you don’t fall into the trap of this common mistake:
Always make sure that single-selection response options are mutually exclusive
Always pretest the survey
The bottom line: the problem with overlapping scales is one of the most common mistakes of questionnaire design. Ensuring that your response options are mutually exclusive will not only producemore reliable data, but will alsoprovide a better experience for your respondents .
Closed-ended questions should be one-dimensional. That is, they should ask about one and only one topic at a time. Questions that use more that one dimension are called “double-barreled”
Double-barreled questions often leave the respondent with no way to respond accurately. Take this question for example:
Do you think your manager is friendly and honest?
O Yes
O No
Although only two response options are offered (”Yes” or “No”), there are actually four possible responses to this question:
I think my manager is both friendly and honest
I think my manager is neither friendly nor honest
I think my manager is friendly but not honest
I think my manager is honest but not friendly
Because of the double-barreled nature of the question, the respondent can be put in a situation where she has to select one of the response options provided, even though neither may accurately represent her opinion.
In this example, the double-barreled question could have been avoided by using two questions instead of only one:
“Do you think your manager is friendly?”
O Yes
O No
“Do you think your manager is honest?”
O Yes
O No
The bottom line: To avoid the common mistake of asking double-barreled questions, make sure that each question in your survey asks about one and only one topic at a time.
As an incentive for online survey participation, some companies offer to share the survey results with respondents. Here are the pros and cons for this type of incentive.
Pros
Offering to share some or all of the survey results with respondents can sometimes be effective with business-to-business research. For employees of certain companies, incentives such as guaranteed rewards and prize drawings are considered conflicts of interest and are therefore not permitted. Thus, the ability to share in the insight gleaned from the survey can be an extra incentive.
Cons
For consumer research, offering to share the results with respondents is rarely an effective incentive because most consumers prefer guaranteed rewards or prize drawings.
For business-to-business research, offering to share the results can increase the likelihood that some respondents who are not qualified to participate in the research will provide dishonest responses to demographic questions in order to be assured that they will be able to complete the survey and receive the results. This can skew the data.
The bottom line: For consumer research, offering to share the results is rarely an incentive for completing the survey. However, for certain business-to-business research projects, this can be an effective incentive.
For online survey incentives, a common alternative to a guaranteed reward is a prize drawing or sweepstakes. Here are the pros and cons for this type of incentive.
Pros
This type of incentive offers several advantages over one-to-one incentives, including:
Lower overall cost
Ease of administration
Little or no shipping expense
Ability to increase the incentive effect by increasing the odds of winning
Another advantage of the drawing is that it limits the financial exposure of the research organization by setting a fixed cost for the incentive. Even if the number of respondents greatly exceeds expectations, the payout can remain the same.
Cons
If only one or two respondents are to be selected in the prize drawing, it is critically important to select a prize that appeals to potential respondents. For example, a drawing for a $10 gift card probably isn’t much of an incentive compared to a drawing for a $100 prepaid gas card.
Similarly, it is very important to establish an attractive odds of winning the drawing. In my experience, odds that tend to resonate with survey respondents fall in the 1-in-100 to 1-in-250 range.
Many experts caution against using this type of incentive for certain types of research, particularly projects in which respondents can be self-referred. For example, if a Web site has a banner inviting all visitors to take part in a survey and offering them a chance to win a prize for participating, it runs the risk of attracting “professional respondents” (people who complete the survey just to win the reward). The responses of such respondents are often dishonest.
The bottom line: Although there is no definitive research proving that prize drawings are always more effective than one-to-one incentives, it is logical to conclude that more times than not the chance to win a significant prize motivates more people than a guarantee to receive a cheap trinket or bauble.
Incentives offered to everyone who completes the survey are known as “one-to-one rewards” or “direct incentives.” Here are the pros and cons for this type of incentive.
Pros
The argument in favor of this type of incentive is that it increases the response rate because every respondent is rewarded for participating. When surveying professionals (doctors, attorneys, etc.), a fairly high cash reward for each respondent is generally the norm.
Cons
For consumer surveys, many projects lack a sufficient budget to offer an incentive that actually appeals to potential respondents. In the U.S., the average budget for a consumer incentive is approximately one dollar. Thus, the preponderance of stress balls, golf tees, etc.
Another problem with one-to-one incentives is the lack of an effective method for making small cash payments to large numbers of people. Many panel providers have worked around this problem by creating their own micropayment systems, usually based on respondents collecting points that can be exchanged for cash or merchandise.
The requirement to ship the incentive to each respondent significantly reduces the list of possible incentive items because only very lightweight items can be affordably mailed to respondents. As a rule, if the total cost for a one-to-one incentive (product cost plus shipping cost) exceeds one dollar, this approach generally loses its cost advantage.
Many experts caution against using incentives for some kinds of research, particularly projects in which respondents can be self-referred. For example, if a Web site has a banner inviting all visitors to take part in a survey and guaranteeing them a prize for participating, it runs the risk of attracting “professional respondents” (people who complete the survey just to win the reward). The responses of such respondents are often dishonest.
A final consideration with one-to-one incentives is that it can be important to take preventative measures to safeguard the budget from an unexpectedly high response rate. If the survey programming includes a limit on the number of interviews that will be completed, the survey is closed to respondents as soon as the quota is achieved.
The bottom line: one-to-one incentives can be effective at increasing response rates, particularly among high-income respondents. For consumer surveys, however, this type of incentive can be less effective.
Entries (RSS)