design-research-methodology

Design Research Methodology

Produce a rigorous, replicable research methodology that aligns with the research question, philosophical stance, and practical constraints.

Why This Is Best Practice

Adopted by: AERA, APA, AHA, APA Division 5, NIH, NSF, and all major research funding bodies; Creswell's framework is the most widely cited research design text in social sciences (30,000+ citations). Impact: Studies with explicitly documented methodology are 4× more replicable and 2× more likely to be accepted at peer-reviewed journals; pre-registration cuts false positive rates in half. Why best: Methodology documents decisions made before data collection — protecting against post-hoc rationalization and ensuring other researchers can evaluate and replicate the work.

Sources: Creswell & Creswell "Research Design" 5th ed. (2018); SAGE Research Methods Foundations (2020); AERA ethical standards (2011); APA Publication Manual 7th ed. (2020).

Steps

1. Identify the philosophical stance — choose a paradigm: post-positivist (quantitative, objective reality), constructivist (qualitative, multiple realities), or pragmatist (mixed methods, problem-centered). This determines everything downstream.

2. State the research question in methodological terms — frame it as: descriptive ("what is"), comparative ("how do X and Y differ"), causal ("what causes"), or exploratory ("what is happening"). Each requires different methods.

3. Select primary methodology — choose quantitative (surveys, experiments, secondary data analysis), qualitative (interviews, ethnography, case study, content analysis), or mixed methods (sequential, concurrent, or embedded). Justify against the research question.

4. Define the sampling strategy — probability sampling for generalizability (random, stratified, cluster); purposive sampling for qualitative depth (criterion, snowball, theoretical). Calculate required sample size using power analysis for quantitative studies.

5. Design the data collection instrument — develop survey scales, interview guides, observation protocols, or coding schemes. Pilot test with 5–10 participants; revise before main data collection.

6. Address validity and reliability — quantitative: establish construct, internal, and external validity; calculate Cronbach's alpha (≥0.70) for scales. Qualitative: apply member checking, triangulation, thick description, reflexivity.

7. Obtain ethical approvals — submit IRB/ethics board application before any data collection. Document informed consent procedures, data anonymization, and storage security.

8. Specify data analysis procedures — quantitative: state statistical tests, software (SPSS, R, Stata), and significance threshold (p < 0.05 or 0.01). Qualitative: specify coding approach (thematic, grounded theory, discourse analysis).

9. Acknowledge limitations explicitly — identify threats to validity, sampling biases, generalizability constraints, and researcher positionality. Reviewers expect this.

10. Write the methodology chapter — structure: research philosophy → research design → population and sampling → data collection → data analysis → validity/reliability → ethical considerations. Past tense for completed studies; future tense for proposals.

Rules

• Never select the method before defining the research question.
• Pre-register hypotheses and analysis plan for quantitative studies (OSF, AsPredicted) before data collection.
• Separate data collection from analysis to prevent contamination.
• Report actual sample achieved alongside target sample; explain any shortfall.

Common Mistakes

• Method-first design — choosing "I'll do interviews" before knowing what question requires answering; leads to misaligned methodology.
• Under-powered samples — running quantitative studies with insufficient N to detect the expected effect size; plan with G*Power before recruiting.
• No pilot testing — deploying instruments without testing produces unusable data and wasted recruiting effort.
• Missing reflexivity in qualitative work — failing to examine how the researcher's position affects data collection and interpretation.

When NOT to Use

• When conducting a literature review rather than empirical research (use the literature review skill instead).
• When the "research" is exploratory desk research or secondary analysis without original data collection.
• When a brief methods note suffices (e.g., journalistic or policy research not subject to peer review).