Todd Hollander Market Research

Open-ended questions are those in which respondents type a response in a text box rather than selecting from a predefined set of responses (radio buttons, checkboxes, drop-down lists, etc). When open-ended questions are used properly, they can provide a number of benefits. However, when used improperly, they can be the kiss of death for your survey.

As noted in a previous entry, one common mistake is to use too many open-ended questions in a survey, which may cause many respondents to abandon the interview. Another common mistake made by inexperienced researchers is to force responses to an open-ended question even when it is likely that many or most respondents will have no valuable response to offer.

An example of this would be the question:

Do you have any other comments or suggestions about our company?

If the survey software is programmed to require a response to this question, it is likely that many respondents will be frustrated when they receive an error message telling them they have to type a response before they can proceed to the next question. When this mistake is made, the open-ended responses often include a rash of expletives.

The bottom line: As a rule, a response to an open-ended question should only be required when it is reasonable to think that every respondent will have a meaningful response to offer. Otherwise, a response should be optional.

See also: How and when to use text boxes in online surveys

In online surveys, open-ended questions are those in which the respondents type their responses in a text box rather than selecting from a set of predefined responses such as radio buttons, checkboxes, or drop-down lists. When used properly, open-ended questions can provide several benefits. However, when used improperly, they can be the kiss of death for your survey.

Pros
Open-ended questions are useful when the range of responses is not tightly defined and/or detailed responses are desired. Advantages include a wider variety and greater depth of responses, and a higher likelihood of receiving unexpected and insightful information. They can also be used to capture any important ideas or opinions that you may not have thought to ask about.

Cons
Open-ended questions have several disadvantages. First, the responses must be read individually because there is no effective way to automatically tabulate or perform statistical analyses on them. This is more expensive and time consuming than the analysis of closed-ended questions and may not be practical for low-budget or time-sensitive research.

These responses are also open to the interpretation of the reader because different people are likely to interpret the meaning of a response in different ways. This potential problem can be avoided by using a single analyst, but a large number of responses can make this impractical.

The most significant drawback of using open-ended questions is that it may significantly increase the number of incompleted interviews (when respondents exit the survey before responding to all of the questions). This is because open-ended questions require more thought, time, and effort from the respondent.

Inexperienced researchers often make the mistake of including too many open-ended questions in a survey. It is important to understand that in most cases respondents are less passionate about the subject matter than the sponsor of the research. If your survey exceeds their tolerance of time and effort, they are likely to abandon the interview.

So how many open-ended questions should be included in a survey?
This is an important question. Unfortunately, there is no simple response. The number of open-ends your respondents will tolerate generally depends on:

• The total length of the interview
• The incentive for completing the survey
• How much they care about the subject matter
• The extent to which they believe their responses will be read and considered

The bottom line: Open-ended questions can be highly beneficial but should be used sparingly. Using too many can significantly decrease the response rate for your survey.

See also: How and when to use text boxes in online surveys

When conducting online research, it is essential to have a well-defined set of objectives. Surveys designed in the absence of clear objectives frequently suffer from one or both of the following flaws:

1. Wasting the sponsor’s time and money by failing to include necessary questions
2. Wasting the respondent’s time by asking unnecessary questions

The problems stemming from poorly-defined research objectives do not end with data collection but continue on to the analysis stage. Simply stated:

It is impossible to develop meaningful insights if you don’t know what you were expecting to observe.

Step 1: Define the marketing problem
The first step in the process of defining the research objectives is for managers and researchers to discuss and clarify the current situation. This discussion should focus on why the research is needed (i.e., the marketing problem should be defined).

Step 2: Determine the decisions that will be made
Having defined the marketing problem, the next step is for managers and researchers to discuss and agree upon the specific decisions that will be guided by the results of the research. The word “guided” in the previous sentence is an important one. Although some managers hope or expect that the research results will tell them what to do (i.e., make the decisions for them), it is important to remember that:

Effective marketing research is an aid to decision-making, not a substitute.

Step 3: Establish the research objectives
The research objectives should link the marketing problem with the decisions to be made. To this end, a properly constructed set of objectives should outline:

• What kind of information will be provided by the research
• How much information will be provided
• How the research information will be translated into management decisions.

In a sense, these objectives serve as the researcher’s promise to deliver information that can be acted upon.

Step 4: Divide and conquer
While all research projects are designed to gather information, rarely can a single project provide all of the information associated with a product or marketing issue. Thus, rather than falling into the trap of trying to cram 20 gallons of research into a 5-gallon hat, it may be necessary to divide the objectives across two or more research studies.

The bottom line: To define effective research objectives, follow these steps and increase the odds of making the right decisions as a result of the research.

A leading question (also known as a “loaded question”) is one that suggests an answer by the way in which the question or response options are worded. Here’s an example of leading the respondent by the way the question is worded:

How high would you rate this product?

Asking the question in this way leads the respondent to offer a high rating. That is, it sets the expectation that a high rating is the “correct” response. To ask this question in a neutral (i.e., non-leading) way, the word “high” should be eliminated to render the question: “How would you rate this product?” Often a survey question is loaded by an unbalanced set of response options. Take this question for example:

How would you rate this product

Excellent

Great

Good

Fair

Poor

Notice that the question itself is not leading but the response options are. This is because four of the five possible responses are positive and only one (”Poor”) is negative. A closed-ended question should include response options that not only cover the whole range of responses, but that are also equally distributed throughout the range. That is, all responses should be equally likely. Thus, a more appropriate scale would be:

Very Good

Good

Neutral

Bad

Very Bad

This balanced set of response options does not imply an expectation of either a positive or negative response. Nor does it stack the odds that responses will cluster at one end of the spectrum.

The bottom line: To avoid the common mistake of asking leading questions, make sure that none of your survey questions suggest an answer by the way in which the question is worded, and that each question provides response options that not only cover the whole range of responses, but are also equally distributed throughout the range.

Ambiguous questions (those that could be understood in more than one way) are one of the more common mistakes in questionnaire design.

Take this question for example:

How often do you visit our website?

O Very Often
O Often
O Sometimes
O Rarely
O Never

With the exception of “Never,” these response options may mean different things to different people. If some respondents consider once a week as “often” and others consider once a week as “rarely,” the data won’t mean much. Therefore, whenever possible, it is prudent to quantify the choices, such as:

O More than once a day
O Every day
O 2-6 Times a week
O Once a Week
O Less than once a week
O Never

The goal is to eliminate any chance that a question will mean different things to different people. Failing to do this runs the risk that respondents essentially will be answering different questions.

To this end, it is best to use the following guidelines:

• Phrase your questions empirically
• Avoid the use of adjectives
• Avoid colloquial or ethnic expressions
• Avoid technical terms that assume preexisting knowledge or experience

The bottom line: To avoid ambiguity, make sure that every question is clear, succinct, and has only one possible meaning.

One of the most common mistakes I see in questionnaire design is the problem of overlapping scales. Take this question for example:

Which of the following includes your age?

O 18-25
O 25-35
O 35-45
O 45-55
O 55+

If the respondent is 35 years old, which response option should she select: “25-35″ or “35-45″? This seems like common sense, yet I see this mistake made time and time again.

There are several significant problems caused by overlapping scales. First, respondents may conclude that the survey sponsor is inept or unsophisticated. Second, this type of error can lead to above-average rates of survey abandonment (when respondents exit the survey before completing it). Finally, and perhaps most importantly, it can significantly skew the survey data.

Here are two bits of advice to make sure you don’t fall into the trap of this common mistake:

1. Always make sure that single-selection response options are mutually exclusive
2. Always pretest the survey

The bottom line: the problem with overlapping scales is one of the most common mistakes of questionnaire design. Ensuring that your response options are mutually exclusive will not only produce more reliable data, but will also provide a better experience for your respondents .

Closed-ended questions should be one-dimensional. That is, they should ask about one and only one topic at a time. Questions that use more that one dimension are called “double-barreled”

Double-barreled questions often leave the respondent with no way to respond accurately. Take this question for example:

Do you think your manager is friendly and honest?
O Yes
O No

Although only two response options are offered (”Yes” or “No”), there are actually four possible responses to this question:

1. I think my manager is both friendly and honest
2. I think my manager is neither friendly nor honest
3. I think my manager is friendly but not honest
4. I think my manager is honest but not friendly

Because of the double-barreled nature of the question, the respondent can be put in a situation where she has to select one of the response options provided, even though neither may accurately represent her opinion.

In this example, the double-barreled question could have been avoided by using two questions instead of only one:

1. “Do you think your manager is friendly?”
   O Yes
   O No
2. “Do you think your manager is honest?”
   O Yes
   O No

The bottom line: To avoid the common mistake of asking double-barreled questions, make sure that each question in your survey asks about one and only one topic at a time.