5.1 Designing surveys and questionnaires
4 min read•august 16, 2024
Surveys and questionnaires are crucial tools in impact evaluation, helping researchers gather data to assess program effectiveness. Designing these instruments requires careful planning to ensure data validity and reliability, from choosing methods to developing strategies and addressing ethical concerns.
Crafting effective questions is key to obtaining accurate, useful information. This involves selecting appropriate question types, wording items clearly, and structuring the survey logically. Techniques like randomization and attention checks help minimize bias and improve response quality, enhancing the overall evaluation process.
Survey Design Considerations
Planning and Methodology
Top images from around the web for Planning and Methodology
Planning, monitoring and evaluation - supporting constructive change and social learning View original
Is this image relevant?
The Planning Cycle | Principles of Management View original
Is this image relevant?
EEP impacts 1 - Framework for impact evaluation - HAMK Unlimited View original
Is this image relevant?
Planning, monitoring and evaluation - supporting constructive change and social learning View original
Is this image relevant?
The Planning Cycle | Principles of Management View original
Is this image relevant?
1 of 3
Top images from around the web for Planning and Methodology
Planning, monitoring and evaluation - supporting constructive change and social learning View original
Is this image relevant?
The Planning Cycle | Principles of Management View original
Is this image relevant?
EEP impacts 1 - Framework for impact evaluation - HAMK Unlimited View original
Is this image relevant?
Planning, monitoring and evaluation - supporting constructive change and social learning View original
Is this image relevant?
The Planning Cycle | Principles of Management View original
Is this image relevant?
1 of 3
- Survey design critically impacts impact evaluation by ensuring data validity and reliability
- Choose between quantitative and qualitative methods based on evaluation objectives and required information
- Develop sampling strategy to affect data representativeness and generalizability
- Select survey administration mode (face-to-face, telephone, online) considering response rates, data quality, and cost-effectiveness
- Address ethical considerations including and data privacy
- Conduct pilot testing to identify and rectify potential issues before full implementation
- Consider timing and frequency of survey administration to influence data quality and usefulness
Practical Implementation
- Tailor questions to target population's literacy level, cultural context, and subject familiarity
- Include filter questions to direct respondents to relevant follow-up questions, improving survey efficiency
- Implement skip logic and branching to reduce respondent fatigue and improve question relevance
- Provide clear instructions and definitions for complex terms to improve response accuracy
- Include attention check questions to identify and filter out low-quality responses
- Employ cognitive interviewing techniques during development to identify potential response errors
- Use visual aids (progress bars, estimated completion times) to manage expectations and reduce dropout rates
Outcome Measurement Questions
Question Types and Formats
- Craft (multiple choice, Likert scales) for standardized, easily quantifiable responses
- Utilize for detailed and nuanced responses, considering analysis resources
- Avoid to ensure clarity and accuracy in responses
- Select appropriate response scales (nominal, ordinal, interval, ratio) to capture intended measurement level
- Incorporate a mix of positively and negatively worded items to detect response sets and improve data quality
- Utilize reverse-coded items to detect and mitigate acquiescence bias (tendency to agree regardless of content)
Question Wording and Structure
- Prioritize clear and concise question wording to directly impact respondent comprehension and response accuracy
- Apply the funnel approach, moving from general to specific questions, to ease respondents into the survey
- Group related questions in thematic sections to enhance logical flow and coherence
- Strategically place sensitive or controversial questions later in the survey to build rapport
- Incorporate transition statements between sections to improve narrative flow and maintain interest
- Vary question types and formats throughout the survey to maintain engagement and reduce monotony
- Craft a clear, concise introduction explaining the survey's purpose to motivate high-quality responses
Minimizing Response Bias
Randomization and Balancing Techniques
- Randomize question order to reduce order effects and primacy/recency bias
- Balance the use of positively and negatively worded items to detect response sets
- Implement counterbalancing of question blocks to control for order effects across different survey versions
- Utilize randomized response techniques for sensitive questions to encourage honest responses
- Employ item randomization within matrix questions to reduce straight-lining behavior
Response Quality Improvement
- Use attention check questions to identify and filter out low-quality responses (inattentive or disengaged participants)
- Implement soft and hard validation checks to ensure data quality and consistency
- Utilize forced-choice questions to reduce satisficing behavior
- Employ branching logic to personalize question paths and improve response relevance
- Use hover-over definitions or tooltips for complex terms to ensure consistent understanding across respondents
Survey Structure and Engagement
Logical Flow and Organization
- Apply the funnel approach to question ordering (general to specific)
- Group related questions into thematic sections for improved coherence
- Place sensitive or controversial questions strategically later in the survey
- Incorporate clear transition statements between sections
- Implement a modular design to allow for flexible survey administration (core and optional modules)
Respondent Experience Enhancement
- Vary question types and formats to maintain engagement and reduce monotony
- Use visual aids (progress bars, completion time estimates) to manage expectations
- Craft a clear, concise introduction explaining the survey's purpose and importance
- Implement mobile-friendly design for surveys administered on smartphones or tablets
- Provide estimated time for completion at the beginning of the survey
- Offer breaks or save-and-resume functionality for longer surveys
- Use conditional display logic to show only relevant questions based on previous responses
Key Terms to Review (18)
Closed-ended questions: Closed-ended questions are survey items that provide respondents with a limited set of predefined answers to choose from, usually in the form of yes/no options or multiple-choice responses. These types of questions streamline data collection and analysis by allowing for easier quantification of responses, making them useful in surveys and questionnaires that aim to gather specific information efficiently.
Confidentiality: Confidentiality refers to the ethical principle of keeping sensitive information private and secure, ensuring that data related to individuals or organizations is not disclosed without permission. This concept is crucial in various research methodologies, as it fosters trust between participants and researchers, promoting honest and open responses. Maintaining confidentiality is not just a legal requirement; it is also vital for ethical research practices and the integrity of data collection processes.
Construct Validity: Construct validity refers to the extent to which a test or instrument accurately measures the theoretical construct it is intended to assess. This concept is crucial in ensuring that surveys and questionnaires effectively capture the underlying traits, attitudes, or behaviors they are designed to evaluate, rather than irrelevant factors. Establishing strong construct validity involves careful design and testing to confirm that the instrument is both reliable and valid in its measurement of the desired constructs.
Cross-sectional survey: A cross-sectional survey is a research method that collects data at a single point in time from a sample of participants, allowing researchers to analyze the relationships between variables within a population. This approach provides a snapshot view of the population's characteristics, attitudes, or behaviors without tracking changes over time. By examining different groups simultaneously, it helps identify patterns and associations that may inform future studies or interventions.
Descriptive Statistics: Descriptive statistics refers to methods for summarizing and organizing data to provide a clear and concise overview of its main features. These techniques include measures such as mean, median, mode, and standard deviation, which help researchers to understand the basic characteristics of their data set. By employing descriptive statistics, researchers can efficiently present data collected from surveys and questionnaires, making it easier to interpret the results and draw preliminary conclusions.
Dichotomous questions: Dichotomous questions are survey questions that offer respondents only two possible answer options, typically 'yes' or 'no'. This binary format simplifies data collection and analysis, making it easier for researchers to quantify responses and identify patterns. They are commonly used in surveys and questionnaires to gather clear and straightforward information.
Double-barreled questions: Double-barreled questions are survey or questionnaire items that ask respondents to address two different issues within a single question, making it difficult for them to provide a clear and accurate answer. This type of question can lead to confusion and skewed data, as it forces respondents to consider multiple aspects at once instead of focusing on a single issue. Properly designed surveys should avoid these questions to ensure clarity and accuracy in responses.
Inferential Statistics: Inferential statistics is a branch of statistics that allows researchers to make conclusions and predictions about a population based on a sample of data drawn from that population. This process involves using sample data to estimate population parameters, test hypotheses, and derive insights that extend beyond the immediate data set. In the context of designing surveys and questionnaires, inferential statistics is crucial for interpreting the results and ensuring that findings are representative of the larger population.
Informed Consent: Informed consent is the process by which individuals voluntarily agree to participate in research or interventions after being fully informed about the nature, risks, benefits, and implications of their involvement. This principle is crucial in ensuring ethical standards in research and impact evaluations, emphasizing the respect for participants' autonomy and decision-making.
Internal consistency: Internal consistency refers to the degree to which different items or questions within a survey or questionnaire measure the same concept or construct. It is crucial for ensuring that a measurement tool is reliable, meaning that it produces stable and consistent results over time. High internal consistency indicates that the items are well correlated and contribute effectively to the overall score of the survey, which enhances the validity of the conclusions drawn from the data.
Leading Questions: Leading questions are inquiries that suggest a particular answer or contain information that biases the respondent toward a specific response. They can significantly influence the outcome of surveys and questionnaires by steering participants toward an expected answer, which can compromise the objectivity of the data collected.
Likert scale: A Likert scale is a psychometric scale commonly used in surveys and questionnaires to measure attitudes or opinions by asking respondents to indicate their level of agreement or disagreement with a series of statements. This scale typically offers a range of response options, such as 'strongly agree,' 'agree,' 'neutral,' 'disagree,' and 'strongly disagree,' allowing for nuanced insights into respondents' views. The structure of the Likert scale is key in designing effective surveys, as it helps researchers quantify subjective data.
Longitudinal survey: A longitudinal survey is a research method that involves repeated observations of the same variables over a period of time, often years or even decades. This type of survey allows researchers to track changes and developments in the subject being studied, providing insights into trends and causal relationships. Longitudinal surveys are particularly valuable for understanding how variables interact and evolve over time.
Open-ended questions: Open-ended questions are inquiries that allow respondents to provide unrestricted answers in their own words, rather than selecting from predefined options. This format encourages deeper insights and richer qualitative data, as respondents can express their thoughts and feelings freely. Open-ended questions are particularly valuable in surveys and questionnaires for capturing complex ideas and personal experiences that may not fit into standardized response categories.
Qualtrics: Qualtrics is a cloud-based software platform designed for creating, distributing, and analyzing surveys and questionnaires. It offers users a range of tools to design custom surveys that can gather data efficiently and effectively, enabling organizations to make data-driven decisions based on the feedback collected.
Response Bias: Response bias refers to the tendency of survey respondents to answer questions in a manner that is not truthful or accurate, often due to various influences such as social desirability, question wording, or the context of the survey. This can lead to distorted data and results that do not truly reflect the opinions or behaviors of the population being studied. Understanding and mitigating response bias is crucial for designing effective surveys and questionnaires that yield reliable insights.
Sampling: Sampling is the process of selecting a subset of individuals or observations from a larger population to draw conclusions about that population. This technique is crucial for making inferences, as it allows researchers to gather insights without examining every single member of the population. By using sampling, one can analyze data effectively and visualize results, making it a foundational element in research design and data analysis.
Surveymonkey: SurveyMonkey is an online survey development cloud-based software that allows users to create, distribute, and analyze surveys. It provides tools to design questionnaires with a range of question types, enabling researchers to gather data efficiently and make informed decisions based on the feedback collected.