Questions About Cancer? 1-800-4-CANCER

Pink Book - Making Health Communication Programs Work

Communication Research Methods

Types of Communication Research
Differences Between Qualitative and Quantitative Research Methods
Qualitative Research Methods
Quasi-Quantitative Research Methods: Pretesting Messages and Materials
Quantitative Research Methods
Additional Research Methods

Types of Communication Research

Research into intended audiences’ culture, lifestyle, behaviors and motivations, interests, and needs is a key component to a health communication program’s success. This section describes communication research methods commonly used throughout program planning. See the chart Types of Research and Evaluation for more detail about research conducted in each of the stages of health communication program planning.

Most programs use more than one research method. For example, conducting exploratory focus groups with an intended audience at the start of program planning can orient you to the types of approaches, messages, and channels that are most likely to be successful with a particular group. In some cases, focus groups might be augmented with in-depth interviews to learn more about intended audience members’ motivations. Later, messages and materials might be pretested approximate how an individual would encounter them in "real life." Theater-style testing also approximates reality, using a simulated television-viewing environment. Clearly, some methods are better suited to specific purposes than others. Using multiple methods can help ensure that you get an accurate picture of your intended audience members and their likely responses to your program.

Differences Between Qualitative and Quantitative Research Methods

There are two basic types of research you might conduct with intended audiences: qualitative and quantitative. You will use methods from one of these two types depending upon what you want to learn. See the sidebar below, Qualitative Versus Quantitative Methods, for common distinctions between qualitative and quantitative research.

Qualitative Versus Quantitative Methods

Qualitative

  • Provides depth of understanding
  • Asks "Why?"
  • Studies motivations
  • Is subjective; probes individual reactions to discover underlying motivations
  • Enables discovery
  • Is exploratory
  • Allows insights into behavior and trends
  • Interprets

Quantitative

  • Measures level of occurrence
  • Asks "How many?" and "How often?"
  • Studies actions
  • Is objective; asks questions without revealing a point of view
  • Provides proof
  • Is definitive
  • Measures levels of actions and trends
  • Describes

 

Note. From Methodological Review: A Handbook for Excellence in Focus Group Research by M. Debus. Copyright 1988 by The Academy for Educational Development, Washington, DC. Reprinted with permission.

In this section, you will learn when to use each type of research, how to conduct research with members of your intended audience, and how you can use the data you collect to inform your project. Qualitative, quasi-quantitative, and quantitative research methods are discussed separately.

Qualitative Research

Use qualitative research methods when:

  • You are planning a communication program and developing materials for it
  • When the goal of your research is to explore a topic or idea
  • When the goal of your research is to gain insights into an intended audience’s lifestyle, culture, motivations, behaviors, and preferences

Conduct qualitative research by:

  • Selecting a small group of people chosen for particular characteristics
  • Convening a discussion (i.e., a focus group or in-depth interview) or observing individuals’ behaviors in schools, malls, supermarkets, etc.
  • Keeping the discussion fairly unstructured, so that participants are free to make any response and are not required to choose from a list of possible responses
  • Choosing which question to ask next based on your participants’ previous responses

Qualitative research results cannot be:

  • Quantified or subjected to statistical analysis
  • Projected to the population from which the respondents were drawn because participants are not selected randomly (to be representative of the population as a whole) and because not all participants are asked precisely the same questions

Quantitative Research

Use quantitative research methods when:

  • You are planning a communication program (e.g., to measure the prevalence of a particular behavior) or assessing a program already in place
  • The goal of your research is measurement of particular variables Conduct quantitative research by:
  • Selecting a large group or groups of people
  • Using a structured questionnaire containing predominantly forced-choice or closed-ended questions

Quantitative research results can be:

  • Analyzed using statistical techniques
  • Considered representative of the population from which the respondents were drawn if each person in the population had an equal chance of being included

Qualitative Research Methods

Use qualitative research methods during the following parts of your program:

  • Stage 1—to find out more about your intended audiences and to learn what the priorities and approach should be for trying to influence their awareness, knowledge, attitudes, intentions, or behaviors
  • Stage 2—to determine whether your materials communicate the intended messages effectively and persuasively
  • Stage 3—to understand why the program is or isn’t working as expected
  • Stage 4—to learn more about what worked and what didn’t, and why certain outcomes occurred

 

Types of Research and Evaluation
StageType of Research/EvaluationBenefits
Stage 1Consumer Research, Market Research 
Planning and Strategy DevelopmentProvides information on the problem, intended audiences, and barriers to and opportunities for changeAnswers questions such as:

What dimensions of the health problem do we need to address?

How should the population be segmented and which intended audience should be targeted?

What are the best ways of reaching the intended audience?

What benefits would be credible and appealing to the intended audience?

What images should be conveyed?

What barriers need to be overcome?

What actions can the intended audience take?
Stage 2Pretesting 
Developing and Pretesting Concepts, Messages, and MaterialsAssesses reactions to proposed messages or materialsTests and refines messages and materials prior to production
 Pilot or Field Testing 
 Assesses program activities in limited areas and/or time periodsTests and refines activities prior to full-scale implementation
Stage 3Process Evaluation 
Implementing the ProgramDocuments and assesses implementation; quantifies what was done; when, where, and how it was done; and who was reachedIdentifies areas for improvement as implementation proceeds

Documents progress of implementation
Stage 4Outcome Evaluation 
Assessing Effectiveness and Making RefinementsMeasures whether, and to what extent, a program or activity had the planned effectsDocuments the extent of the campaign’s success or failure

Documents success to support replication

Determines any need to improve the existing program or future efforts
 Impact Evaluation 
 Measures whether, and to what extent, a program contributed to long-term goalsIs not often used for health communication activities (improving health status usually requires multifaceted approaches—e.g., communication plus changes in health care service delivery and relevant policies—and it is generally not possible to isolate a particular communication program’s contribution to achieving longer-term goals)

Focus groups and in-depth interviews are the most common methods used in qualitative communication research. However, there are many innovative methods, some described here, that can help you learn about an audience. Because the methodologies for each are very similar, they are discussed together here, using instructions for focus groups as a guide.

About Focus Groups

Working from a discussion guide, a skilled moderator facilitates a 1- to 2-hour discussion among 6 to 10 participants, which can be conducted either in person or by telephone (ideally in person). The moderator keeps the session on track while participants talk freely and spontaneously. As new topics related to the material emerge, the moderator asks additional questions to learn more.

Common Uses

  • Developing a communication strategy:
    • Learning about feelings, motivators, and past experiences related to a health topic
    • Exploring the feasibility of various potential actions (from the intended audience’s viewpoint)
    • Identifying barriers to those actions
    • Exploring what benefits the intended-audience members find compelling and what results they expect from taking a particular action
    • Learning about the intended-audience’s use of settings, channels, and activities
    • Capturing the language used by the intended audience to discuss issues
  • Exploring reactions to message concepts (concept testing):
    • Identifying concepts that do or do not resonate and understanding why
    • Triggering the creative thinking of communication professionals
    • Illustrating to others how the intended audience thinks and talks about a health issue
  • Developing hypotheses (or broad issues) for quantitative studies and identifying the range of responses that should be included in closed-ended questionnaires
  • Providing insights into the results of quantitative studies by obtaining in-depth information from individuals typical of the intended audience to help understand why individuals responded in certain ways

Pros

  • Group interaction can help elicit in-depth thought and discussion.
  • Group interaction can help with brainstorming because respondents can build off one another's ideas.
  • Moderators have considerable opportunity to probe responses.
  • Focus groups yield richer data than surveys about the complexities of an intended audience’s thinking and behavior.

Cons

  • Findings are not generalizable to the population.
  • Focus groups can be labor intensive and expensive, especially if sessions are conducted in multiple locations.
  • Group responses do not necessarily reflect individuals’9 opinions because some individuals in the group may dominate the discussion or may influence others’ opinions.
  • Each person is limited to about 10 minutes of talking.

Working With Market Research Professionals

You may need to hire or contract with two kinds of market research professionals as you design, conduct, and analyze your concept and materials testing:

1. Someone to design the research and data instruments (e.g., questionnaires, discussion guides, screeners), to analyze the results, and to prepare a report

2. A vendor to handle the fieldwork (i.e., recruiting and hosting focus groups; administering telephone, mail, or in-person surveys)

Ideally, these professionals will have a background in health communication or, if not, a background in marketing or advertising research. You can get the best service from these professionals by:

  • Providing clear research objectives and appropriate background information, including the creative brief.
  • Learning enough about common communication research methods to understand their strengths and limitations, so that you don’t ask for more than a given method can deliver (e.g., asking, "What percentage of the American public does that represent?" when a focus group study was conducted).
  • Letting market researchers’ expertise guide your selection of methods. Rather than saying, "We want to focus test this," explain your research objectives, timing, budgetary constraints, and any additional factors (such as the need for a publication to be tested with people from a wide range of cultures). Then let the experts propose methods to you and explain their rationale.
  • Being realistic about timelines, quantity of information, materials to be tested at one time, and the level of "proof" you need. Pretesting is diagnostic; it can provide guidance on what needs to be improved, but it can’t tell you how successful something will be. Other factors, such as the final production of your message, the number of people who see it, the frequency with which it is seen, and the presence of competing messages will all influence your message’s success.
  • Recognizing that there are inherent differences between testing advertising and other commercial communication materials versus testing health communication materials, even if the channel and activity (e.g., a television spot) are the same. Individuals trained in commercial concept development and copy testing will be able to draw on their commercial experience for selecting the appropriate methodology. However, they often have little experience assessing reactions to complex health messages; they are more familiar with assessing efforts to direct an existing behavior toward use of a particular product brand than with assessing efforts to completely change a behavior.

Sometimes, one individual or organization can play both roles; at other times, you may have internal staff, a consultant, or staff at a health communication firm to handle the first role but contract externally for the second. The American Marketing Association’s Green Book lists suppliers and services geographically throughout the United States. Other sources include the Marketing Research Association, the Association of Public Opinion Researchers, the Qualitative Research Consultants Association, and faculty at university departments of marketing, communication, health education, psychology, and sociology.

About In-Depth Interviews

The process, benefits, and drawbacks of in-depth interviews are similar to those of focus groups, except that the interviewer speaks with one person at a time. In-depth interviews can take place at a central facility or at the participant’s home or place of business. As with focus groups, when individual interviews cannot be conducted in person, they can be conducted by phone or over the Internet. Although these interviews are more time intensive, one of their key benefits is that each respondent is isolated from other respondents and therefore not influenced by what others say.

How to Design and Conduct a Focus Group or In-Depth Interview Study

To design and conduct a qualitative research study, complete the following steps.

Plan the Study

Determine the following:

  • What you want to learn. Determine the objectives of your study at the outset, and then check to make sure that the moderator’s/interviewer’s guide includes lines of questioning that will provide the answers.You may also use the objectives to help analyze the results of the discussions and to organize the focus group or in-depth interview report.
  • When you need to have that information.
  • How you will apply what you learn. It is important to decide how you will use your focus group or in-depth interview results before you conduct your study.
  • Your budget.
  • Your criteria for who should participate.
    Select people who are:
    • Typical of your intended audience (the same behavioral, demographic, and psychographic characteristics).You may want to conduct separate groups with "doers" (those who already engage in the desired behavior) and "nondoers" (those who do not) to help identify what actions the doers take, and why, so that those approaches can be explored with the nondoers.
    • Not experts. Exclude market researchers and advertising professionals (because of their familiarity with the methodology) and those who have, or might be perceived by other group members as having, expertise in the subject matter (e.g., exclude health professionals from focus groups when the topic is a health issue).
    • Relative newcomers to focus groups or interviews, so that their reactions will be spontaneous. This will help you avoid "professional" respondents (i.e., those who have participated in many previous focus groups or individual interviews before) who may lead or monopolize the discussion.
  • The number of groups you will convene.
    • Divide participants into different focus groups based on their gender, race, age, level of formal education, or any other variable likely to hinder their freedom of expression (e.g., teenage girls will be more comfortable discussing sexual activity if teenage boys or college-age women are not in the group).
    • Conduct a minimum of two focus groups with each intended audience segment (e.g., if you are conducting separate groups with men and women, you will need at least four groups–two with men, two with women). If intended audience perceptions vary or the audience feedback is unclear, you may want to conduct additional groups with each segment, especially if you revise the discussion guide to more fully explore unresolved issues.
    • If you are using in-depth interviews, conduct approximately 10 interviews per intended audience segment. If common themes do not emerge or the interviews, especially if you revise the interview guide in between interviews.

Choose the Location

You can convene focus group discussions or in-depth interviews in a variety of ways:

  • Commercial focus group facilities can recruit participants for you (for both focus groups and interviews) and offer audio and video recording equipment as well as observation rooms with one-way mirrors for viewing. However, these facilities are usually available only in larger metropolitan areas.
  • Teleconference services can set up telephone focus groups for you. Most allow observers to listen without being heard, and some provide remote viewing programs to allow the moderator to see a list of participant names (with a symbol next to the one currently speaking) or notes sent in by a technician from observers listening to the call. Some teleconference services can recruit participants; with others, you will have to recruit participants or contract with a recruiter separately.
  • You can also conduct focus groups or in-depth interviews in meeting rooms at churches, office buildings, or other locations. If an observation room with a one-way mirror is not available, staff may still listen in through speakers hooked up in a nearby room or by audiotaping or videotaping the session. In some cases, one or two quiet observers may be allowed in the room to take notes.

See the sidebar Pros and Cons of Different Formats below for the advantages and disadvantages of different formats for focus group and in-depth interview research.

Pros and Cons of Different Formats

  1. Face-to-Face
  2. Moderator/interviewer and participants are in one room, usually around a table; observers (members of the research team) are behind a one-way mirror.

    Pros
    • Can assess body language If videotaped, can share with others who couldn’t attend
    • Have participants’ undivided attention
    Cons
    • Responders lose some anonymity
    • Higher travel expenses due to multiple locales
    • Usually excludes people in rural areas or small towns
  3. Telephone
  4. Moderator/interviewer and participants are on a conference call; observers listen in

    Pros
    • More convenient for participants and observers
    • Can easily include people in rural areas, in small towns, and who are homebound
    • For professional groups, may be easier to gain participation because it is less likely participants will know each other
    • Relative anonymity may result in more frank discussion of sensitive issues
    Cons
    • Can’t assess nonverbal reactions
    • More difficult to get reactions to visuals (although they can be sent ahead of time)
    • Participants can be distracted by their surroundings
  5. Internet Chat Sessions
  6. Moderator and participants "chat"; observers watch

    Pros
    • Complete record of session instantly available
    • Relative anonymity may result in more frank discussion of sensitive issues
    Cons
    • Only useful for participants comfortable with this mode of communication
    • Relatively slow pace limits topics that can be covered
    • No way to assess if participants meet recruitment criteria
    • Can’t assess body language or tone of voice
    • More difficult to get reaction to visuals

Draft a Recruitment Screener

A recruitment screener is a short questionnaire that is administered to potential participants, typically by telephone, to ensure that they meet the criteria you developed during step 1. Ideally, the screener should help you exclude participants who are already familiar with the specific subject of the sessions, or who know each other. If participants know the subject in advance, they may formulate ideas or may study to become more knowledgeable about the subject than the typical intended audience member. If participants know each other, they may speak less freely. See Appendix A for a sample screener.

Recruit Participants

Recruit participants by telephone one to three weeks in advance of the sessions. You can identify potential participants in different ways depending upon the type of people you are seeking and the resources you have available. Identify members of the public through focus group facility databases, by running an ad in a local publication, by working with community organizations, or by using the phone book (although the latter will be extremely time consuming if you have stringent recruitment criteria). Identify professionals through a relevant association or mailing list service or through a focus group facility’s recruiting databases. Depending on your budget and internal resources, you may choose to recruit in one of the following ways:

  • Use your own organization’s staff to recruit participants.
  • Hire a focus group facility or independent recruiter (to identify such facilities, consult a directory such as the American Marketing Association’s Green Book or the Marketing Research Association’s Blue Book). If you have many facilities and recruiters to choose from, consider getting recommendations from local corporations or organizations that conduct qualitative research.
  • Get help from a university marketing research or advertising class.
  • Work through gatekeepers such as teachers (for students), health care providers (for patients, physicians, or nurses), religious institutions or community organizations (a small donation may encourage them to participate), and instructors of English as a second language.

Regardless of how the recruiting is done, ensure that the screener is followed carefully so that only individuals who qualify for participation will be included.

Contracting With Commercial Facilities

Before you contract with a commercial facility to conduct in-depth interviews or focus groups, prepare a specification sheet detailing all of the services you need and, if you will be asking the facility to recruit, a profile of your intended audience. Vendors will use this information to estimate the cost of the project and to develop bids.

Decide whom to approach by using the following checklist to determine which vendors will fit your needs. Each vendor should provide the following information:

  • Descriptions of past projects
  • Descriptions of, or a list of, clients
  • Location of facility (Is the facility conveniently located? Is parking available? Is the facility accessible by public transportation? If not, does vendor provide transportation assistance, such as taxi money or van service? What does this add to cost?)
  • Diagram of table/seating arrangement (What size and shape are tables? Rooms?)
  • Description of the size and features of observation rooms
  • Details about audio and video recording arrangements and costs
  • Details about food arrangements for participants and clients
  • Description of moderator services
  • Description of recruitment methods and geographic coverage
  • Recommendations for participant incentives
  • Reasonable rates for the services they will provide (ask for nonprofit rates, if appropriate)

Getting People to Show Up

To ensure that enough people show up, offer an incentive (usually money) and recruit two or three more people than you actually need. If all show up, select those who best fit the screening criteria, thank the extra participants, give them the agreed-upon incentive, and dismiss them. Other ways to increase participation include:

  • Scheduling sessions at times that are convenient for your potential participants (e.g., at lunch or after work)
  • Choosing a safe and convenient site with easy parking
  • Providing transportation (or reimbursement for agreed-upon transportation costs)
  • Arranging for childcare, if necessary
  • Letting them know you will be serving refreshments

Recruiting Patients and Their Families

Recruiting patients and their families requires special consideration. Contact clinics, hospitals, or local HMOs for help and make adequate plans to ensure that the participants and their family members are not inconvenienced or upset. Some organizations may require institutional review board (IRB) approval of your research. Gaining IRB approval is often a long process, so be sure you check with the organization early in the planning stage of your study to find out whether you will need IRB approval.

Recruiting for Telephone Interviews

If you are recruiting for in-depth interviews to be conducted on the telephone, create a spreadsheet that includes spaces for the following information about each potential participant:

  • Time zone in which the person is located
  • Date, time, and telephone number at which he or she should be called for the interview
  • Disposition of each call (e.g., scheduled an interview, no answer, busy, refused)

Develop a Moderator’s Guide

The moderator’s guide tells the moderator what information you want from the groups and helps him or her keep the discussion on track and on time. It is only a guide, however. During the focus groups, experienced moderators flow with the conversation, asking questions in the prescribed language and sequence when possible but sometimes deviating from the guide to avoid awkward transitions or unnecessary back-and-forth between topics.

Before you draft the moderator's guide, answer the following questions:

  • What do we want to learn from the focus group?
  • How will we apply what we learn?
  • What tools (e.g., descriptive information, message concepts, or other draft creative work) will we need to provide?

Then, write questions for the guide that relate to the purposes you have identified. Make most questions open-ended so that participants can provide more in-depth responses than just "yes" or "no," but make sure the questions are not leading. This will help you get answers that reflect participants’ true feelings and not what they think you would like to hear. The amount of time and depth of questions devoted to each issue should reflect the value of the issue to the research. See Appendix A for an example of a moderator’s guide.

Do not include questions for group discussion when you need individual responses. However, you can have the moderator give self-administered questionnaires to each participant to be completed prior to conducting a focus group, or participants can be asked to individually rank items on paper—such as potential actions, benefits, or message concepts—during a group to obtain both individual and group reactions.

Working With Community Organizations to Conduct Focus Groups

Once you have identified potentially cooperative community groups (see the following sidebar for a list of groups you might approach), contact an official within each group (e.g., the president or program director) to request cooperation. You may make these initial contacts by telephone and follow up with a formal written request that includes the following:

  • Description of your agency or organization
  • Description of the material/topic to be discussed and its purpose
  • Details regarding the participants to be recruited and how you will protect their confidentiality
  • Outline of the activities involved<</li>
  • Incentives you are offering the organization and the participants
  • Detailed explanation of why the organization official should not reveal details about the nature of the discussion to participants in advance, unless the organization is to recruit participants
  • If and how you will share the information learned

Once you have an agreement with a community organization, decide how you will recruit participants. One possibility is to conduct your research as part of the group's regularly scheduled meeting.

The advantages of this approach are:
  • Little extra effort is required to recruit participants.
  • You may need to provide only minimal or no incentives.
  • The group’s regular and familiar meeting place can be used.
The disadvantages of this approach are:
  • You have little control over the number of people who will come or the composition of the group.
  • It is difficult to place a 1- to 2-hour focus group on the agenda of a regularly scheduled meeting.
  • Many organizations set their calendars months ahead of time (it may be difficult to schedule the focus group within a reasonable time frame).

An alternative is to recruit the group’s members to a special meeting. Schedule this meeting immediately before or after the group’s regular meeting to make it most convenient for the participants. If you use this alternative, contact members in advance on behalf of the group and ask them to participate. A person from the community group can also ask others to participate. To ensure that participants attend and stay through the whole meeting, let them know in advance that you will be providing refreshments (assuming that you are doing so).

The advantages of recruiting participants to a special meeting are:

  • It provides an opportunity to screen participants on relevant characteristics and to eliminate market researchers or other experts who should not participate.
  • It helps reduce participant fatigue because the entire meeting will be devoted to your research.

If you recruit participants yourself, you will have more control over what the participants are told about the focus group, and you will be able to screen potential participants to make certain that they fit special intended audience characteristics. However, recruiting participants takes a significant amount of time, and it is possible that people would be more likely to participate if asked by someone whom they know.

If the member organization recruits participants, it is essential that you provide the recruiter with detailed instructions for carrying out the task. These instructions must include a written description of the focus group topic, which should be read to potential participants verbatim, and a questionnaire to screen participants on relevant criteria.

Conduct the Focus Groups

Focus groups typically begin with the moderator welcoming participants and briefing them on the process (e.g., all opinions welcome—there are no right or wrong answers; the presence of audio- and videotaping and observers; the importance of speaking one at a time; confidentiality). Participants introduce themselves to the group by first name, usually including some information relevant to the topic of discussion (e.g., number of years with glaucoma, amount/type of insulin used each day). Next, the moderator asks a few simple "ice-breaker" questions to help participants get used to the group process and to reduce participant anxiety. This also helps the moderator develop rapport with the participants.

Continuing to follow the moderator’s guide, the moderator manages the group and ensures that all topics are covered without overtly directing the discussion. Participants are encouraged to express their views and even disagree with each other about the discussion topics. The moderator does not simply accept what participants say but probes to learn more about participants’ underlying thinking and attitudes. The moderator also seeks out opinions from all participants so that all are heard and a few do not dominate the discussion.

Near the end of the discussion, the moderator will often give participants an activity or simply excuse him- or herself from the room for a moment to check with the observers and obtain any additional questions. Alternatively or additionally, notes can be sent in to the moderator while the group is in process if the observers would like different questions asked or other changes made to the group.

One advantage of focus group methodology is that the moderator’s guide, and any materials presented, can be revised between groups if necessary.

Analyze Results

The easiest and most thorough way to analyze focus groups is by reviewing transcripts, although groups can also be analyzed (albeit less thoroughly) by reviewing notes taken during the discussion. In many analyses, the goal is to look for general trends and agreement on issues. At the same time, it is important to note divergent opinions. Don’t ignore individual comments that raise interesting ideas or concerns such as lack of cultural sensitivity or difficulty in comprehension. In some instances, the goal is to capture the range of opinions about an issue, rather than to look for evidence of agreement or consensus.

Avoid counting or quantifying types of responses (e.g., "75 percent of participants preferred concept A"). Attempting to quantify the results—or suggesting in other ways that they represent the opinions of the intended audience as a whole—is inappropriate for qualitative research.

Quasi-Quantitative Research Methods: Pretesting Messages and Materials

Some commonly used communication research methods, such as central-location intercept interviews and theater tests, are best termed quasi-quantitative. While these methods are used in situations in which the goal is measurement and typically involve a questionnaire with mostly forced-choice questions, the results cannot be projected to the population as a whole (as with true quantitative surveys) because of the way in which participants are selected. For centrallocation intercept interviews, the only people who have a chance to participate are those who go to the location where the interviews are being held and who go there during the times they are conducted; this is not a truly representative sample of the intended audience. For theater tests, the only people who have a chance to participate are those who are recruited for the test, and recruitment does not follow a truly representative sampling design.

Quasi-quantitative methods are most often used during Stage 2 to pretest messages and materials. If your intended audience is geographically dispersed or it is difficult for them to get to a central facility, you can use telephone interviews and send participants any materials in advance. This type of pretest typically resembles an in-depth interviewing project in price and number of interviews, although there may be more closed-ended questions and the question sequence may be adhered to more closely.

The Moderator's Role

The moderator does not need to be an expert in the subject of your research but must have experience facilitating group discussions. A good moderator builds rapport and trust and probes, without reacting to or influencing, participants' opinions. The moderator must be able to lead the discussion and not be led by the group. He or she must emphasize that there are no right or wrong answers to the questions that are posed. A good moderator understands the process of eliciting comments, keeps the discussion on track, and figures out other ways of approaching a topic if the first way is unproductive. Good moderators understand what you are looking for and what you need to do with the information, and they are able to probe and guide the discussion accordingly. Go over the guide with the moderator to point out any topics or concerns you want emphasized or discussed in depth. By the end of the focus group or interview, the moderator should ensure that all agreed-upon topics are covered sufficiently.

If your organization plans to conduct focus groups regularly, consider hiring a skilled, experienced moderator to train your internal staff to moderate focus groups. Use local advertising agencies, the American Marketing Association's Focus Group Directory, or the Qualitative Research Consultants Association to identify a good moderator.

Using Software to Analyze Qualitative Data

If you have conducted a large number of focus groups or interviews with many intended audience subsets-and are interested in analyzing results by different cultural groups, age groups, or economic groups within the overall respondent population-you may want to use computer software to do a comparative analysis of your results. If your results are from only a few groups, however, computer analysis will be too time consuming to benefit your program. Be sure to supplement computer analysis with "human" analysis, since the strength of qualitative research is that it can uncover unexpected human reactions that software cannot properly capture or weigh.

Before you decide to use computer software to analyze your qualitative data, assess the following advantages and drawbacks of this type of analysis.

Advantages

  • The ability to highlight sections of the transcript that are important to the project and to eliminate "noise" or sections of the transcript that are not important to answering your research questions (of course, a wordprocessor's cut-and-paste functions can also accomplish this).
  • The ability to quickly access and compare information on one topic or questions across several transcripts.

Disadvantages

  • You must tape all interviews
  • It costs time and money to transcribe focus group sessions and in-depth interviews.
  • Coding the transcripts is both time intensive and expensive. If more than one person will be coding, you will need to train the coders, periodically assess intercoder reliability, and retrain as necessary.
  • While all comments on a particular topic can be gathered, they are taken out of context in the process. Sarcasm and other tonal characterizations may be lost.
  • Analysis by software will help you organize information, but will also produce overwhelming amounts of paper.

Steps

If you decide to use software to analyze your qualitative data, follow these steps:

  1. Transcribe the focus group discussions or interviews into an electronic format that can be read by the analysis software. If you decide to use qualitative data analysis software, check the requirements of the package you choose.
  2. Develop and apply the codes you will use to organize the information in the transcripts. A code is a word or number that represents a research objective, research question, theory, or idea you are testing. The codes you develop will be unique to your research. For example, if your first research question is to find out how many vaccine shots a parent is willing to give his or her infant, you might code all lines of your transcript that include comments on that question as "A,"' signifying that they correspond to your first research question. Applying the code requires that you review the transcripts and use a mouse to indicate on the computer screen all the lines of the transcript that pertain to that code word.
  3. Generate reports using the codes you developed. To do this, you instruct the program to search for a particular code or a combination of codes. The program then presents a single report showing all the lines of the transcript bearing the requested codes.

For a complete review of qualitative data analysis software, refer to Computer Programs for Qualitative Data Analysis: A Software Sourcebook, by Eben A. Weitzman and Matthew B. Miles, 1995, Thousand Oaks, CA: Sage. A new edition of this book is due out in 2002.

Central-Location Intercept Interviews

Central-location intercept interviews consist of stationing interviewers at a point frequented by individuals from your intended audience and asking the individuals to participate in a study. If they agree, they are asked specific screening questions to see whether they fit the study criteria. If so, the interviewer takes them to the interviewing station (a quiet spot at a shopping mall or other site), shows the pretest materials, and then administers the pretest questionnaire.

For intercept interviews to be effective, you must obtain results from a minimum of 60 to 100 respondents from each intended audience segment you want to test.

Pros

  • You can connect with harder-to-reach respondents in locations convenient and comfortable for them.
  • The interviews can be conducted quickly. (The interview should be no longer than 15 to 20 minutes.)
  • The interviews are a cost-effective means of gathering data in a relatively short time.
  • If you choose an appropriate location, you will increase your chance to interview respondents who are among your intended audience.

Cons

  • You must train interviewers.
  • Your results are not representative or generalizable.
  • Intercept interviews are not appropriate for sensitive issues or potentially threatening questions.
  • Intercept interviews do not allow you to probe easily for additional information.

Central-location intercept interviews should not be used if respondents must be interviewed in depth or on emotional or sensitive subjects. The intercept approach also may not be suitable if respondents are likely to be resistant to being interviewed on the spot. In cases in which central-location intercepts will not work well, schedule interviews with respondents instead.

 

Estimated Costs of Focus Groups and In-Depth Interviews, 2002

These estimated costs are included to suggest how you should budget for focus groups and in-depth interviews if you are using commercial research firms. Your actual costs will vary depending upon your geographic location, the intended audience to be recruited, and the amount of time donated by staff, companies, and participants. Be sure you do not jeopardize the quality of your results with a budget that is too small.

The focus group cost estimate in the table assumes that you conduct two groups, each composed of 10 members of the general public. A group size of 6 to 8 is sometimes preferred, because it is easier to engage all participants in the conversation. This estimate is also based on the assumption that each group session is 2 hours long, is conducted in English, and includes audiotapes. Staff travel, food for participants, and videotaping are not included.

The in-depth interview estimate assumes a total of ten 30-minute interviews conducted in English and audiotaped.

 2 Focus Groups10 In-Depth Interviews
Develop screener$ 920 - 1,380$ 920 - 1,380
Develop discussion guide$ 920 - 1,840$ 920 - 1,840
Recruit$1,725 - 2,875$ 860 - 1,725
Rent facility$ 705 - 1,380$ 0 - 1,150
Provide respondent incentives$ 690 - 1,380$ 0 - 575
Compensate moderator or interviewer to conduct$1,000 - 2,200$ 500 - 1,000
Analyze and report results$1,840 - 2,760$1,840 - 2,760
Total$7,800 - 13,815*$5,040 - 10,430**
*Add an additional $400-600 for transcribing focus group audiotapes (optional).
**Add an additional $300-400 for transcribing in-depth interview audiotapes (optional).


Estimated Costs of Central-Location Intercept Interviews, 2002
The central-location intercept interview cost estimate below assumes that you question 100 respondents from the general population for 15 to 20 minutes each.
Develop questionnaire$ 460 - 4,025
Produce questionnaire, schedule facility, phones, and mail$ 400 - 600
Screen/conduct interviews$1,955 - 4,025
Provide respondent incentives$ 345 - 575
Code/enter data/tabulate$ 805 - 1,495
Analyze and report results$1,495 - 4,025
Total$5,460 - 14,745


Questionnaire Contents

Unlike focus groups or in-depth interviews, the questionnaire used in central-location intercept pretesting is highly structured and contains primarily multiple choice or closedended questions to permit quick response. Open–ended questions, which allow free–flowing answers, should be kept to a minimum because they take too much time for the respondent to answer and for the interviewer to record. Questions that assess the intended audience's comprehension and perceptions of the pretest materials form the core of the questionnaire. A few additional questions, tailored to the specific item or items being tested ("Do you prefer this picture—or this one?"), may also be included to meet your program planners' particular needs. The questionnaire should be pretested before it is used in the field. See Appendix A for a sample questionnaire.

Interview Setup

A number of market research companies throughout the country conduct central–location intercept interviews in shopping malls.You can also conduct these interviews in clinic waiting rooms, religious institutions, Social Security offices, schools, work sites, train stations, and other locations frequented by members of your intended audience. Be sure to obtain permission well in advance of the time you want to set up interviewing stations in these locations.

If you are using a market research company to conduct the interviews, you will need to provide screening criteria, test materials, and the questionnaire. In some cases, market research companies have offices in shopping malls, and you can watch the testing through a one-way mirror.

Participant Recruitment

If you or someone in your organization is recruiting the participants, you will need to develop a script and provide training in approaching members of the intended audience. For example, if you are recruiting participants in a clinic waiting room, the interviewer should be familiar with the screening criteria (e.g., women under 60 years of age) and approach only those people who appear to fit the criteria. When, after screening, individuals do not qualify to participate, the interviewer should thank them for their time and indicate that this study is not the right fit for them but that their willingness to participate is appreciated. If they do qualify, the interviewer can bring them to a designated location (e.g., another room or corner of the waiting room) and proceed with the study.

University and college departments of marketing, communication, or health education may be able to provide interviewer training or trained student interviewers. Pretesting is an excellent real–world project for a faculty member to adopt as a class project or for a master's student to adopt as a thesis project. However, this approach may mean that it takes longer to accomplish the research, and you could compromise the quality of the results if the individuals are not experienced in this type of research.

Examples of Community Organizations to Contact for Help Recruiting Participants

  • American Legion
  • B’nai B’rith Women
  • Business groups
  • Jaycees
  • Junior League
  • Knights of Columbus
  • League of Women Voters
  • Lions Club
  • NAACP
  • National Council of Negro Women
  • National Urban League
  • Parent Teacher Associations
  • Religious organizations
  • Rotary Club
  • Schools
  • Senior citizen centers
  • Unions
  • Veterans of Foreign Wars
  • YWCA/YMCA

Theater-Style Tests

Theater testing is often used in the commercial arena to test advertisements for products and services. Theater testing can also be used to test the effectiveness of PSAs. In this methodology, participants are invited to a central location to respond to a pilot for a new television show; in the midst of viewing the TV pilot, they are shown your PSA or advertisement along with other ads. Participants complete a questionnaire following the presentation, first answering questions about the show and then answering questions about how effectively your message was communicated to them and what their overall reactions were.

Common Uses

Theater–style tests are most commonly used to test TV advertisements and PSAs. For theater–style tests to be effective, you must obtain results from 50 to 100 respondents from each segment you want to test.

Pros

  • You can obtain responses from a large number of respondents at the same time.
  • Theater–style tests more closely replicate what goes on in someone's home when they are watching TV, so you can accurately judge people's reactions to your message.
  • Theater–style tests can be cost–effective if you use donated facilities and equipment.

Cons

  • It is expensive to rent a facility and equipment (if necessary).
  • Your results are not representative of the general population.

General Format

Individuals typical of your intended audience are invited to a conveniently located meeting room. The room should be set up for screening a television program. Participants should not be told the real purpose of the session, only that their reactions to a television program are being sought.

At the session, participants watch a television program. The program can be any entertaining, nonhealth video approximately 15 to 30 minutes in length. The videotape is interrupted about halfway through by a sequence of four commercials. Your message should be inserted between the second and third commercials. See Appendix A for a description of how to create a roughcut video for theater-testing your message.

At the end of the program, participants receive a questionnaire and answer questions designed to gauge their reactions, first to the program and then to the advertisements. Finally, your ad is played again and participants complete several questions about your ad. The majority of these questions should be closed-ended to enable an easy and accurate summary of participant responses.

In more sophisticated theater testing, participants use automated intended audience response systems to answer questions. Participants are provided with a small device that has response keys. Once a question is asked, they push a key to respond and the data are automatically tabulated.You have instant access to the numbers using this system. In addition, questions can be instantly added or deleted from the questionnaire based on the previous responses. Using an automated system is much more costly than using a standard paper-and-pencil questionnaire.

Other Media You Can Test

This methodology can also be used to test videos by asking participants to view a series of videos in which yours has been included. Examples of videos that might be tested include a 15- to 30-minute breast cancer awareness video that will be played in a clinic or a "how-to" video on administering epinephrine. These testing sessions will, of course, last longer than those testing ads. Participants evaluate the videos as described above.

Print advertisements can also be tested using a variation of this methodology. Several ads, including yours, are inserted into a magazine. Participants are given an adequate amount of time to read through the article, which includes your ad and others. After reading the article, participants receive a questionnaire and answer questions designed to gauge their reactions, first to the article and then to the advertisements. Finally, your ad is displayed alone and participants complete several additional questions.

Designing and Conducting a Theater-Style Pretest

The process for conducting a theater–style test includes the following steps:

  1. Planning the pretest
  2. Developing the questionnaire
  3. Recruiting respondents
  4. Preparing for the pretest
  5. Conducting the pretest
  6. Analyzing the pretest

You may find step 2 also useful for central–location intercept interviews.

Plan the Pretest

Determine:

  • The purpose of the study (e.g., what do you want to learn?)
  • When you need to produce results
  • What your budget is
  • The type and number of people who should participate in the pretest
  • The locations where the pretest will be conducted

To conduct theater testing, you must have a large enough space to accommodate all of your participants at the same time. You must also ensure that you have several video monitors so that all participants can adequately view the program. Space constraints may be overcome by seeking out low-cost facilities such as a school auditorium or church hall.You may be able to borrow the audiovisual equipment from these facilities as well. You can also rent space, such as a hotel ballroom, if you want to test a large number of people. Hotels often rent audiovisual equipment as well. Reserve facilities and equipment well in advance of your pretest.

Develop the Questionnaire

To gather useful information from the pretest, you must carefully construct the questionnaire. See the sidebar Components Used in Most Questionnaires on the next page for general guidelines. Once you have written your questionnaire, be sure to test and revise it before you use it with a large number of respondents.

Recruit Respondents

Participants may be recruited through a market research facility or through local community organizations. In either case, you will need to provide an incentive for participants. If using a market research facility, you will also incur recruiting expenses. If you are working with a community organization, you may choose to make a donation.

ESTIMATED COSTS OF THEATER TESTING, 2002*
Develop questionnaire$ 460 – 2,760
Produce questionnaire$ 400 – 600
Recruit$ 0 – 5,750
Rent audiovisual equipment$ 0 – 2,300
Conduct theater test$ 0 – 920
Provide respondent incentives$ 285 – 2,875
Code/enter data/tabulate$ 920 – 3,680
Analyze and report results$1,840 – 3,680
Total$3,905 – $22,565+
* Estimates assume 50 participants. They exclude facility rental costs. The costs of large facilities (e.g., hotel ballrooms) vary widely by geographic location. Check with local facilities for approximate costs.

Components Used in Most Questionnaires

Although the following components should be used in most questionnaires, these descriptions are specific to a theater-style pretest.

Program Questions
Program questions elicit general intended audience reactions to the program viewed. Use the questions in Part I of the sample Theater–Style Pretest Questionnaire in Appendix A as the first page of your questionnaire.

Recall, Main Idea, and General Reaction Questions
These three standard questions that assess your message’s ability to attract attention, convey its main point, and create a positive response should appear on the second page of your questionnaire. Use the questions in Part II of the sample questionnaire in Appendix A.

These standard questions should be incorporated into the pretest for several reasons. The questions address the most important indicators of a message’s potential effectiveness: 1) whether it attracts intended audience attention (recall), 2) whether it communicates your main point (main idea), and 3) what respondents thought and felt when they viewed the ad (reaction).

If you test many ads and always use the same questions, you can develop a database of results to allow you to assess the relative strength of various ads.

Specific intended audience Reaction Questions
These provide answers to specific questions you have about your message.

Develop questions that address specific concerns you have about your message. For example, suppose your message asks viewers to call a toll-free number for more information. You may want to include a question that asks, "What action, if any, does the message ask you to take?" A related question may be, "Did the telephone number appear on the screen long enough for you to write it down (or remember it)?"

It is best to develop one or more questions addressing each characteristic of your message. The list below includes various characteristics commonly found in messages. Note the characteristics that apply to your test message and then develop questions that focus on those characteristics.

  • Use of music (with or without lyrics)
  • Use of famous spokesperson
  • Use of telephone number/Web site address
  • Request for a particular action
  • Instructions for adopting a specific health behavior
  • Presentation of technical or medical information
  • Presentation of new information
  • Promotion of a sponsoring organization or event
  • People intended to be typical of the intended audience
  • Use of a voice-over announcer
  • Presentation of controversial or unpleasant information

See Part III of the sample in Appendix A for examples of questions you can use for each message characteristic listed above. These questions are just examples and should be adapted to your needs. Remember that the objective of pretesting is to uncover any problems with your ad before final production.

Demographic Questions
These questions help to record the characteristics of the participants (e.g., their sex, age, level of education, health status).

Once you have written your questionnaire, be sure to test it before you use it with a large number of respondents. After any necessary revisions, you are ready to make copies for the pretest participants. You will need a cover page that instructs participants not to open their questionnaires until they are asked to do so by the meeting host. Place a cover sheet between each part of the questionnaire and instruct participants not to continue until they are asked to do so by the meeting host.

Prepare for the Pretest

Rehearse the testing session at your own office to anticipate and avoid any problems before actual pretesting. Review the following questions to be sure that your session will go as smoothly as possible:

  • Is the pretesting videotape ready for use? Are the video and audio portions clear?
  • Is the videotape equipment—recorder and television monitors—functioning properly?
  • Is the facility set up? Is the room reserved? Are there enough chairs? Are extra chairs available in case more people show up than you expect? Do you need another monitor so that everyone will be able to see the program? Is the heat or air conditioning working properly? Do you know where the light switches are? If a microphone is needed, is it set up and functioning properly?
  • Have you made all the necessary staffing arrangements? Are your assistants coming to the session? Do they have transportation and directions for getting there?
  • Have you made enough copies of your pretest questionnaire (including some extras)? Is each questionnaire complete (with no pages missing)? Do you have pencils for participants? Will they need clipboards or pads?
  • Has participant recruitment taken place as scheduled? Did you call and remind participants to attend? Do they have transportation and directions?
  • Has the moderator rehearsed?

Conduct the Pretest

The procedures to follow during the pretest session are relatively simple. The keys to a successful testing session are to:

  • Be friendly and courteous to participants from the moment they arrive until they leave (remember to say "thank you").
  • Keep calm and cool-headed throughout the session.
  • Anticipate problems in advance (conducting a rehearsal to make sure that both equipment and timing work is a good idea).

The test session should take no more than 1 hour and 15 minutes if you are organized and well prepared. Follow the steps below to conduct your test:

  1. Encourage participants to take a seat as they arrive. Close the doors no later than 10 minutes after the
    scheduled starting time.
  2. When everyone is seated, introduce yourself by your name only (assuming you are the host). Do not
    tell participants the name of your organization during the session because it might bias their responses to your test ad.
  3. Thank participants for coming and assure them that the evening should be enjoyable and that they will have
    a chance to give their views to the producers of "new" television program material. Discourage participants from talking to one another during the session. Tell them you are interested in their own individual views and that there are no right or wrong answers. Also, encourage them to write their answers clearly in the space provided on the questionnaire.
  4. After your introductory remarks, have your assistants hand out the questionnaires (see Appendix A for a sample), pencils, and clipboards (if needed). Instruct the participants not to open the questionnaire until you ask them to do so. Turn on the video recorder and monitor to begin the test session.
  5. Be attentive and watch for any problems with the sound or picture on the monitor. Be sure that the equipment is functioning properly throughout the program.
  6. Be prepared to stop the recorder when the television program has ended. Introduce the questions, and thank the participants for their help so far. Ask them to open their questionnaires and complete the questions on the first page.
  7. When the participants have finished Part I of the questionnaire, tell them that you would like to gather their reactions to the messages/PSAs that were shown during the program. Have them turn to Part II and instruct them to fill out the questions about the messages.When they have completed these questions, tell them that you want to obtain their reactions to one particular message in the series of messages they viewed.
  8. Start the video. (Note: To avoid an awkward pause in the session’s pace, be sure there is not too much lead tape before the message starts.) After your PSA/ad has been replayed, ask participants to turn to the next page of the questionnaire and complete the remaining questions. Encourage them to answer every question and to avoid giving more than one answer, except when this option is indicated on the questionnaire.
  9. Circulate through the room to monitor progress and to be sure participants are not discussing their responses. Collect the questionnaires as participants finish.
  10. Thank participants for their cooperation. If you have incentives or token gifts, distribute them to participants as they leave. If you have provided a donation to a group in lieu of payment to participants, mention that you hope the group will find the donation helpful.

Analyze the Pretest

Analyze the questionnaires in two steps. First, tabulate or count how many participants gave each possible response to each question. Next, look for patterns in the responses to both closed– and open–ended questions. The patterns will help you draw conclusions about the effectiveness of your message. See Appendix A for detailed instructions on tabulating closed- and open-ended questions and for a table of average ratings to help interpret standard question responses.

At this point, look at the overall results:

  • What did you learn from the pretest?
  • Did your message receive a favorable and appropriate intended audience reaction?
  • Did your message fulfill its communication objectives?
  • What are your message's strengths? Weaknesses?
  • Did answers to any particular question stand out?

Use your answers to these questions to decide whether your message is both effective and appropriate and whether you need to revise your message prior to program implementation.

Diaries and Activity Logs

Other tools you can use to evaluate your program are diaries and activity logs. If you plan to use these tools to gauge the quality of program planning or execution, be sure to start keeping the diaries and activity logs as soon as you begin program planning. For each activity, request information in a specific format from program managers or participants. This information may cover issues such as the quality of program components or track how your intended audience uses the components.

Common Uses

  • Track program implementation
  • Assess effectiveness of program implementation
  • Monitor whether planned activities are being conducted on schedule and within budget
  • Learn what questions program participants had
  • Learn what technical assistance was needed by program staff
  • Track intended audience exposure to program components

Pros

  • Allow respondents flexibility in their responses
  • Enable researchers to observe behavior over time, rather than only once

Cons

  • Require considerable effort on respondents' parts (for this reason, consider offering incentives for completion of the diaries/logs)
  • Require staff able to code voluminous and challenging incoming data
  • Are not appropriate for respondents who have low literacy skills or who have poor writing skills or penmanship

Instituting Diary/Activity Log Use

Steps in instituting the keeping of diaries and activity logs are:

  1. Planning the study
  2. Identifying who will participate
  3. Developing and pretesting the form you will use
  4. Collecting data
  5. Analyzing results

Follow the steps below to institute the keeping of diaries and activity logs.

Plan the Study

Determine the following:

  • What you want to learn
  • How much information you need to collect
  • How you will apply what you learn
  • When you need the information
  • What your budget is
  • Your criteria for who should participate

Identify Who Will Participate

The sample you select depends on the goals of your study. If you are focusing on program implementation, you will want the diaries/logs to be completed by program staff (e.g., nurses in a clinic). In this case, you may have some control over the quality of responses you receive.

When planning the study, you must obtain permission from a manager or supervisor on site for staff to complete the diaries/logs during the study.You should provide an estimate of the amount of time and effort participation will entail (e.g., 15 minutes per day, 1 hour per day). Share drafts of the diaries/logs and get input from the supervisor prior to the study. This will help to ensure cooperation during the study.

Before the study begins, you should train staff to complete the diaries/logs. Even if it seems obvious to you, it is essential that you explain exactly what you want recorded in the diary/log. (See the sample log in Appendix A.) In addition, you should provide detailed, written instructions for future reference. These instructions can be used in lieu of training if you cannot physically get to the study site.

If you are focusing on participant experience with a program, you will want the diaries/logs to be completed by people who were exposed to program components. In this case, you will have much less control over the quality and quantity of responses.

Obtaining cooperation from participants may also be more difficult in this situation. For example, people attending an educational program on nutrition might be recruited to complete a diary of what they eat for a week and send it back to the researchers.You will likely need to provide an incentive (e.g., a gift certificate upon receipt of the completed diary), and you may also need to remind participants to send back the diaries at the end of the study period.

Develop and Pretest the Form You Will Use

Once you have identified what you want to learn and who will complete the diaries/logs, you must create a user-friendly document to collect the necessary information.

Create questions. Write questions that are specific to your study objectives.

Examples of the types of information you might collect include:

  • For a toll-free hotline activity log:
    • Date?
    • What is the gender of the caller?
    • Where did the caller get the number?
    • Did the caller request any educational materials?
    • What questions did the caller ask?
  • For a health education program diary:
    • Date?
    • Which module was used?
    • Which of the suggested activities were completed?
    • How long did it take to complete the suggested activities?
    • Would you use these activities again?
    • Why or why not?
  • For a clinic observation activity log:
    • Date?
    • How many people passed by the display containing the new breast cancer information?
    • How many stopped to look at the information?
    • How many took a brochure?

Pretest the diary/log. Once you have created the draft diary/log, you must pretest it with individuals who represent your intended audience. Describe the scenario for them before the pretest. For example, in the case of a hotline, you might say, "You are an operator on a hotline. People will be calling in, and you will need to fill out this activity log as you complete each call." Sit together with them and ask them to read each question aloud and tell you what they think they are supposed to do. Do not correct them if they do not say what you intended. This probably means that your diary/log is unclear. Continue through the entire diary/log and then ask them if there was anything that they found confusing or unclear. Pretest the diary/log with everyone as planned before you make any changes.

Revise the diary/log. Revise questions that people found confusing during the pretest. If a question was confusing only to one person, use your judgment to decide whether to change the question. Ask yourself whether there is something you can easily fix that would have helped that one person understand the question (e.g., providing an example). If so, you may be able to make a simple change or addition to clarify the question. Also consider whether this respondent found many of the questions confusing while other respondents had no problem with them. If this is the case, you may not want to make changes.You will have to decide on a case-by-case basis. If you make substantial changes to the diary/log, you should conduct another pretest before finalizing the form.

Collect Data

Produce diaries/logs in sufficient quantities so that respondents have several extra forms in case they make errors or need more space. Deliver the diaries/logs to respondents, along with detailed written instructions, prior to training (if applicable) or at least 1 week before the study begins. If you are asking program participants rather than program staff to complete diaries/logs for you, you will have to distribute the materials on site. Give respondents a fixed time frame to complete the diaries/logs and provide them with a means (envelope/postage) to return the data to you. If your study is longer than a week or two, you may want to ask respondents to ship the first week of data to you so that you can review the logs for accuracy and completeness and even begin to tally some of the information.

Analyze Results

In the planning phase, you determined what you wanted to learn from the study. Now you will have the chance to look through the diaries/logs to answer these questions. Diaries generally contain qualitative information (e.g., how food choices were made that day, evaluation of programs completed). Activity logs may contain several types of information—quantitative information you can tabulate easily (e.g., how many people called a hotline each day, whether people picked up a brochure) as well as qualitative information (e.g., reasons that students liked or participated in an activity).

Analyzing qualitative responses. The best way to analyze qualitative information is to read through the information, searching for similarities and differences between diaries. You will need to consider all of the questions that you determined were important in the planning phase. Once you have reviewed several diaries, you should be able to pull out general themes or patterns from the information. The best way to analyze these themes is to develop categories for the responses. For example, if you want to know why teachers thought their students liked or disliked a certain educational module in your program, you might group responses into categories such as "challenging," "fun," "too much work," "boring." Continue reading through the remaining diaries and see how many responses fall into these categories. As you go along, you may come up with additional categories or decide to collapse several categories together.You can certainly make inferences (e.g., "Teachers liked the module becau...") about diary information, but resist the temptation to quantify this information.

Analyzing quantitative responses. The easiest way to analyze these types of responses is to create a coding sheet for each quantitative question. Use a separate sheet for each question, writing the question at the top and creating columns for each possible response. For example, for a question about how many people picked up particular brochures, you could create columns for the following categories: 0, 1–5, 6–10, 11–15, 16–20, >20.

Use the following procedure to record the responses:

  1. Take the first activity log and record the response by making a check mark in the appropriate column.
  2. Repeat this procedure for every questionnaire.
  3. Tally the total number of check marks in each column and then calculate the percentage of participants who gave each type of response.

Quantitative Research Methods

Use quantitative research methods during the following parts of your program:

  • Stage 1—to obtain information on prevalence of relevant knowledge, attitudes, behaviors, and behavioral intentions
  • Stage 3—to monitor usage of materials and intended audience awareness of the communication program and its various tactics
  • Stage 4—to measure progress toward objectives

Two different quantitative research methods, surveying and readability testing, can be used.

Surveys

Surveys are characterized by large numbers of respondents (100 or more) and questionnaires that contain predominantly forced-choice (closed-ended) questions.

Common Uses

Used in planning and assessment to obtain baseline and tracking information on knowledge, attitudes, behaviors, and behavioral intentions

Pros

  • Provides generalizable results (to generalize to a broader population, you must have a statistically valid random sample)
  • Can be anonymous (useful for sensitive topics)
  • Can incorporate visual material (e.g., can pretest prototype materials)

Cons

  • Limited ability to probe answers
  • Potential bias from possible respondent self-selection

Most surveys are custom studies that are designed to answer a specific set of research questions. Some surveys, however, are omnibus studies, in which you add questions about your topic to an already existing survey. A number of national and local public opinion polls offer this option.

Pros and Cons of Various Survey Formats
  1. Mail

  2. Pros
    • Can be used to cost–effectively access difficult–to–reach populations (e.g., the homebound, rural populations)
    • Respondents can answer questions when most convenient for them
    Cons
    • Not appropriate for respondents with limited literacy skills
    • Low response rate diminishes value of results
    • May require extensive/ expensive followup by mail or telephone to increase response rate
    • Respondents may return incomplete questionnaires
    • Can be difficult to read responses>
    • May take long time to receive sufficient numbers of responses
    • Postage may be expensive if sample is large, questionnaire is long, or multiple reminder cards are needed
  3. Telephone (With interviewer using paper-and-pencil questionnaires)

  4. Pros
    • Appropriate for those with limited literacy skills
    • Results in more complete responses because interviewer fills out questionnaires
    • Can control question sequence
    Cons
    • Potential respondents without phones cannot participate
    • Respondents may hang up if they believe the survey is part of a solicitation call or they don’t want to take time to participate
  5. Using computer-assisted telephone interviewing (CATI)

  6. Pros
    • Useful for complex questionnaires because "skip patterns" can be programmed in
    • Data entry is eliminated
    Cons
    • Requires CATI software and computers
    • Requires extensive interviewer training
    • Requires time to program questionnaire into CATI
  7. In Person (Interviewer-administered)

  8. Pros
    • Face-to-face persuasion tactics can be used to increase response rates
    • Can be used with those with limited literacy skills
    • Useful with difficult-to-reach populations (e.g., homeless, rural) or when intended audience cannot be sampled using other data collection methods
    • Interviewer can clarify questions for respondents
    • Results in more complete responses because interviewer fills out questionnaires
    Cons
    • More expensive than self–administered or telephone data collection
    • Not appropriate for sensitive, threatening, or controversial questions (respondents may not answer as truthfully in person)
  9. Written, self-administered
    (Respondents are asked to complete survey at a location frequented by the target population (e.g., during a conference, in a classroom, after viewing an exhibit at a health fair)

  10. Pros
    • Can connect with harder-to-reach respondents in locations convenient and comfortable for them
    • Can be conducted quickly
    • Cost-effective means of gathering data in relatively short time
    • Can result in increased number of respondents within intended audience if appropriate location chosen
    Cons
    • Must be able to reach respondents in person at a central location or a gathering
    • Respondents must have adequate literacy skills and be self-motivated to respond
  11. Computerized, self–administered
    (Questionnaire is displayed on computer screen and respondents key in their answers)

  12. Pros
    • Useful for complex questionnaires because computerized "skip patterns" can be used
    • Can control question sequence
    • Eliminates data entry and provides quick summary or analysis of results
    Cons
    • Not appropriate for intended audiences with limited literacy skills or those unfamiliar or uncomfortable with computers
    • Requires expensive technical equipment that may not be readily available or may be cumbersome in many settings
  13. Internet (Computerized, self–administered Questionnaire is displayed on respondent’s computer screen via a Web site)

  14. Pros
    • Useful for complex questionnaires because computerized "skip patterns" can be used
    • Can control question sequence
    • Eliminates data entry and provides a quick summary or analysis of results
    Cons
    • Not appropriate for audiences with limited literacy skills or those unfamiliar with or uncomfortable with computers
    • Respondents must have access to the Internet and be somewhat familiar with using computers
    • No way to confirm the validity of information provided by respondents

Follow these steps to conduct a survey:

  1. Plan the study
  2. Determine how the sample will be obtained and contacted
  3. Develop and pretest the questionnaire
  4. Collect the data
  5. Analyze results

Sampling size and composition, questionnaire design, and analysis of quantitative data are complex topics beyond the scope of this book. If you are planning a quantitative study, see the reference list at the end of this book for additional information.

Additional Research Methods

Gatekeeper Reviews

Public and patient education materials are often routed to their intended audiences through health professionals or other individuals or organizations that can communicate with these audiences for you. These intermediaries act as gatekeepers, controlling the distribution channels that reach your intended audiences. Their approval or disapproval of materials can be a critical factor in your program’s success. If they do not like a poster or a booklet or do not believe it to be credible or scientifically accurate, it may never reach your intended audience.

Common Uses

Gatekeeper review of rough materials is important and should be considered part of the pretesting process, although it is not a substitute for pretesting materials with intended audience members. Neither is it a substitute for obtaining clearances or expert review for technical accuracy; these should be completed before pretesting is undertaken. Sometimes, telling gatekeepers that technical experts have reviewed the material for accuracy will reassure them and may speed their approval of your material.

Methodology

The methodology you should use for gatekeeper review depends upon your available resources, time, and budget. Common methods include:

  • Self-administered questionnaires—
    Participants are sent the materials and the questionnaire at the same time.
  • Interviewer-administered questionnaires—
    Typically, an appointment for the interview is scheduled with the gatekeeper, and the materials are sent for review prior to the interview.

Develop questionnaires that ask about overall reactions to the materials and for assessment of the information’s appropriateness and usefulness.

In some cases, you might not use a formal questionnaire (especially if you don’t think the reviewer will take the time to fill it out) but will instead schedule a telephone conversation or a meeting about the materials. If you are not using a questionnaire, consider in advance what kind of questions you want to ask in the meeting or interview and determine whether you need formal approval of the materials. A discussion with gatekeepers (e.g., a television public service director, the executive director of a medical society) at this point can also be used to solicit their involvement in a variety of ways that extend beyond materials development.

Readability Testing*

Readability formulas often are used to assess the reading level of materials. Fry, Flesch, FOG, and SMOG are among the most commonly used. Applying these formulas is a simple process that can be done manually or by using a computer software program. Each method takes only a few minutes.

Typically, readability formulas measure the difficulty of the vocabulary used and the average sentence length. In addition, computer software programs analyze a document’s grammar, style, word usage, and punctuation, and assign a reading level. These formulas, however, do not measure the reader’s level of comprehension.

Readability software programs are available at computer stores. Some software programs, such as Microsoft Word, include a readability-testing function. (Note: Mention of software products does not constitute an endorsement by the National Cancer Institute.)

Researchers James Pichert and Peggy Elam suggest three principles for using readability formulas effectively:

  1. Use readability formulas only in concert with other means of assessing the effectiveness of the material.
  2. Use a formula only when the text’s intended readers are similar to those on whom the formula was validated.
  3. Do not write a text with readability formulas in mind.

Before you choose a readability testing method, decide on an appropriate reading level for the materials you’ve written. Then use readability testing to determine whether your text corresponds to the reading level you want.

The term reading level refers to the number of years of education required for a reader to understand a written passage. Some experts suggest aiming for a level that is two to five grades lower than the highest average grade level of your intended audience to account for a probable decline in reading skills over time. Others note that a third- to fifth-grade level is frequently appropriate for low-literacy readers. Keep publications as simple as possible to increase reader comprehension of the material.

Readability Testing Methods

You can test readability easily using a formula such as Fry, Flesch, FOG, or SMOG. These tests can be done quickly to indicate any problems with the drafted text. They do not involve the intended audience.

SMOG

To calculate the SMOG reading grade level of a written sample, begin with the entire written work that is being assessed, and follow these four steps:

  1. Count off 10 consecutive sentences each near the beginning, in the middle, and near the end of the text.
  2. From this sample of 30 sentences, circle all of the words containing three or more syllables (polysyllabic), including repetitions of the same word, and total the number of words circled.
  3. Estimate the square root of the total number of polysyllabic words counted. Do this by finding the nearest perfect square and taking its square root.
  4. Finally, add a constant of three to the square root. The resulting number is the SMOG grade or the reading grade level that a person must have reached to fully understand the text being assessed.

A few additional guidelines will help to clarify these instructions:

  • A sentence is defined as a string of words punctuated with a period, an exclamation point, or a question mark.
  • Hyphenated words are considered one word.
  • Numbers that are written out should also be considered, and if in numeric form in the text, they should be pronounced to determine whether they are polysyllabic.
  • Proper nouns, if polysyllabic, should be counted, too.
  • Abbreviations should be read as unabbreviated to determine whether they are polysyllabic.

Not all pamphlets, fact sheets, or other printed materials contain 30 sentences. To test a text that has fewer than 30 sentences:

  1. Count all of the polysyllabic words in the text.
  2. Count the number of sentences.
  3. Find the average number of polysyllabic words per sentence as follows:
    Average = Total # of polysyllabic words
                           Total # of sentences
  4. Subtract the total number of sentences from 30.
  5. Multiply that number by the average.
  6. Add that figure to the total number of polysyllabic words.
  7. Find the square root and add a constant of 3.

Perhaps the quickest way to administer the SMOG test is by using the SMOG conversion table. Simply count the number of polysyllabic words in 30 sentences (chains of 10 each from the beginning, middle, and end of the text) and look up the approximate grade level on the chart. See the sidebar Example Using the SMOG Readability Formula on the next page for an example of how to use the SMOG Readability Formula and the SMOG conversion table. In the sidebar, each of the 3 sets of 10 sentences is marked with brackets.

Readability Testing With the Intended Audience*

Other methods of evaluating reading levels and comprehension include having your intended audience pretest your materials. The WRAT or the Cloze technique can be used to do this. These types of testing are useful when you suspect that the intended audience may encounter difficulties with the material. Including pretest participants who have the same characteristics as the lowliteracy intended audience you are trying to reach is critical to the validity of your pretest results. Recruiting participants through groups or settings that include people with limited literacy skills is a logical starting point. But the only way to be sure your pretest volunteers read at the same level as your intended audience is to test their reading skills.

The Wide Range Achievement Test (WRAT) is used to measure reading levels, and the Cloze technique is used to measure comprehension. To avoid offending or causing discomfort to those whose reading ability you are testing, you can integrate a WRAT or a Cloze test into the pretest interview. For example, in a recent pretest conducted by the National Cancer Institute, the interviewers introduced the WRAT test as the last part of the pretest. They stated, "Thank you for helping with the questions on the chemotherapy booklet.We need your help with one last part—a word list. This will take only a few minutes. The word list will help us understand how difficult the words are in the chemotherapy booklet." This integrated approach spared participants the pressure or potential embarrassment of "failing a reading test."

The WRAT is based on word recognition and does not measure comprehension or vocabulary. The WRAT is an efficient way to determine reading levels and takes only a short time to administer.

The WRAT involves listening to the participant read from a prepared list of words, arranged in increasing order of difficulty. Pronouncing the word correctly shows that the reader recognizes the word. The WRAT focuses on recognition because, at the most basic level, if a person does not recognize a word, comprehension is impossible.

The test is over after the reader mispronounces 10 words. The test administrator notes the level at which the last mispronunciation occurred. The "stop" level equates to a grade level of reading skills.You can compare this level with the reading level of your intended audience to see if your pretest readers are a representative match for that audience.

The Cloze technique measures the reader’s ability to comprehend a written passage. Because it requires readers to process information, it may take up to 30 minutes to administer.

In a Cloze test, text appears with every fifth word omitted. The reader tries to fill in the blanks. This task demonstrates how well he or she understands the text. The reader’s ability to supply the correct word also reflects his or her familiarity with sentence structure.

While packaged Cloze tests are available, Leonard and Cecilia Doak’s Teaching Patients with Low Literacy Skills explains how to make up and score a Cloze test yourself, based on the materials you are pretesting. The book also discusses use of the WRAT to assess reading levels.

Example Using the SMOG Readability Formula

Benign Prostate Hyperplasia (BPH)
[Benign prostatic hyperplasia (BPH) is an enlarged prostate. Benign means noncancerous and hyperplasia means excessive growth of tissue. BPH is the result of small, noncancerous growths inside the prostate. It is not known what causes these growths, but they may be related to hormone changes that occur with aging. By age 60, more than half of all American men have microscopic signs of BPH, and by age 70, more than 40 percent will have enlargement that can be felt on physical examination.

The prostate normally starts out about the size of a walnut. By the time a man is age 40, the prostate may already have grown to the size of an apricot; by the age of 60, it may be as big as a lemon.

BPH, which usually does not affect sexual function, is a troublemaker because the prostate, as it enlarges, presses against the bladder and the urethra, blocking the flow of urine.

A man with BPH may find it difficult to initiate a urine stream or to maintain more than a dribble. He also may need to urinate frequently, or he may have a sudden, powerful urge to urinate.] Many men are forced to get up several times a night; others have an annoying feeling that the bladder is never completely empty.

Straining to empty the bladder can make matters worse; the bladder stretches, the bladder wall thickens and loses its elasticity, and the bladder muscles become less efficient. The pool of urine that collects in the bladder can foster urinary tract infections, and trying to force a urine stream can produce back pressure that eventually damages the kidneys. The kidneys are where urine is formed as waste products are filtered from the blood.

BPH sometimes leads to problems. [For instance, a completely blocked urethra is a medical emergency requiring immediate catheterization, a procedure in which a tube called a catheter is inserted through the penis into the bladder to allow urine to escape. Other serious potential complications of BPH include bladder stones and bleeding.

Diagnosing BPH
A detailed medical history focusing on the urinary tract—kidneys, ureters (the pair of tubes that carry urine from the kidneys to the bladder), the bladder, and the urethra— allows the doctor to identify symptoms and to evaluate the possibility of infection or other urinary problems.

The initial medical workup typically includes a physical exam called a digital rectal examination (DRE), a urinalysis to check for infection or bleeding, and a blood test to measure kidney function. Some physicians may also check the amount of prostate-specific antigen (PSA) using a PSA test to help rule out the likelihood of cancer. PSA is a protein that is produced by the cells of the prostate gland.

In addition, other tests may help a urologist—a doctor who specializes in disorders of the urinary tract and the male reproductive tract—to determine if BPH has affected the bladder or kidneys. These include tests that measure the speed of urine flow, pressure in the bladder during urination, and how much urine is left in the bladder after urinating.

Some other tests that are widely used, according to an expert panel sponsored by the United States Public Health Service (USPHS) practice guidelines, are expensive, sometimes risky, and, for most men, unnecessary. These include cystoscopy, in which the doctor inserts a viewing tube up the urethra to get a direct look at the bladder; an x-ray called a urogram, in which urine is made visible on an x-ray after dye is injected into a vein; and ultrasound, which obtains images of the kidneys and bladder after a probe is placed on the abdomen.]

Treating BPH
About half of men with BPH develop symptoms serious enough to warrant treatment. BPH cannot be cured, but its symptoms can be relieved by surgery or by drugs in many cases.

BPH does not necessarily grow worse. According to one review, mild to moderate symptoms worsened in only about 20 percent of the cases. They improved (without any specific treatment) in another 20 percent and stayed about the same in the rest.

[Men whose symptoms are mild enough often opt for an approach called watchful waiting. This means that they report for regular checkups and have further treatment only if and when their symptoms become too bothersome.

The USPHS Clinical Practice Guidelines call watchful waiting "an appropriate treatment strategy for the majority of patients." Men who choose watchful waiting should have regular, perhaps annual, checkups, including DREs and laboratory tests.

For those who choose watchful waiting, a number of simple steps may help to reduce bothersome symptoms. These include limiting fluid intake in the evening, especially beverages containing alcohol or caffeine, which can trigger the urge to urinate and can interfere with sleep; taking time to empty the bladder completely; and not allowing long intervals to pass without urinating.

Men monitoring prostate conditions should also be aware that certain medications they are taking for other ailments may make their symptoms worse. These include some over-the-counter cough and cold remedies, prescribed tranquilizers, antidepressants, and drugs to control high blood pressure. Switching to a different prescription may help.

Watchful waiting, of course, is not always enough for BPH, and surgery or drug therapy may be required.]

Readability Test Calculations
Total Number of Polysyllabic Words= 104
Nearest Perfect Square= 100
Square Root= 10
Constant= 3
SMOG Reading Grade Level= 13
We have calculated the reading grade level for this example. Compare your results to ours, and then check both with the SMOG conversion table:
SMOG Conversion Table*
Total Polysyllabic Word CountsApproximate Grade Level
(± 1.5 Grades
)
0 — 2  4
3 — 6  5
7 — 126
13 — 20 7
21 — 30 8
31 — 42 9
43 — 56 10
57 — 72 11
73 — 90 12
91  — 11013
111 — 13214
133 — 15615
157 — 18216
183 — 21017
211 — 24018
* Developed by Harold C. McGraw, Office of Educational Research, Baltimore County Schools, Towson, Maryland.

 

* Adapted from Clear and Simple: Developing Effective Print Materials for Low-Literate Readers (NIH Publication No. 95-3594), by the National Cancer Institute, 1994. Bethesda, MD. In the public domain.