content-based-validation

Content-based validation

Evidence that the content of a selection procedure is a representative sample of important work behaviors, activities, and/or worker KSAOs defined by an analysis of work. The inference rests on the fidelity of the procedure's content to the work — not on an empirical predictor–criterion correlation. Often the strategy of choice when a job is unique, samples are too small for a criterion study, or the procedure is built directly from the work (e.g., work samples, job-knowledge tests).

When content evidence is and isn't the right tool

• Strong fit when procedure content is closely linked to job content.
• A construct labeled an ability or personality characteristic does not by itself rule out a content strategy — but when content is less clearly linked to job content, other sources of evidence take precedence (e.g., criterion-related, internal structure).
• Content evidence supports construct interpretation and generalization to job performance/ training when the procedure samples the work well.

SMEs define the domain — qualify them

The study's success hinges on SME qualifications. SMEs:

• define the work domain (important behaviors, activities, worker KSAOs),
• participate in the analysis of work, and
• judge the link between selection-procedure content and work/worker requirements.

SMEs must be competent to perform the work and knowledgeable about relevant organizational characteristics (shift, location, equipment, software). For behavior definition, include people fully knowledgeable about the work. Standardize and document the method for translating SME judgments into the procedure; if SME ratings evaluate the content–work match, standardize the rating procedures and criteria for each aspect.

Process

1. Confirm feasibility. Are the work and worker requirements reasonably stable? Can irrelevant content be minimized? Are qualified, unbiased SMEs available? Do cost/time allow the needed fidelity?
2. Define the content domain from accurate, thorough work information (behaviors, activities, responsibilities, prerequisite KSAOs). The domain need not include everything done on the job. State which important elements are included, how each links to the procedure, and why any parts were excluded. Ground the definition in organizational needs, labor markets, and specifications.
3. Decide what the candidate must already have vs. will be trained. Base the procedure on the analysis that defines the balance between KSAOs expected at entry and training the organization provides. (Read-the-manual example: if employees are trained on the technical manuals, test reading ability to the required level — not manual-specific knowledge.)
4. Develop or choose the procedure; sample the domain. Random sampling of the domain is usually neither feasible nor appropriate. Instead, sample the most important behaviors/KSAOs or those prerequisite to others, or a smaller set predictive of critical outcomes. Document the rationale in a test plan specifying which KSAOs are measured by which methods.
5. Set the level of specificity deliberately. More fidelity to exact job components → stronger content evidence, but less resilience to change and less transferability. If the work is likely to change, build less job-specific content (e.g., software principles/techniques rather than one program). Measure skills and knowledge that are job-relevant, never irrelevant content, even when a full simulation isn't feasible (a single-step problem may substitute for a multistep work sample).
6. Establish administration and scoring guidelines in advance — especially when the procedure includes work behaviors or activities that raise administration/scoring challenges.
7. Evaluate the content evidence and document everything.

Evaluating content evidence

Content evidence rests on showing the procedure adequately samples and is linked to the important work behaviors/activities/KSAOs from the work analysis. The documented development methods are the primary evidence. The sufficiency of the match is a matter of professional judgment based on evidence collected in the effort. Determine reliability of performance on the procedure when feasible, choosing a reliability type that fits the measurement design and intended use (e.g., pass/fail vs. rank ordering).

Experience and training requirements as content

A procedure may include evidence of specific prior training, experience, or achievement — judged by the relationship between the content of that experience and the content of the work. Superficial resemblance is not enough: course titles and job titles may poorly indicate actual content or proficiency. Evaluate the similarity of behaviors/activities/processes/KSAOs, not labels.

Competency modeling

A competency model can found a content study only if it is detailed and rigorous — as rigorous as a traditional work-analysis project. Judge that rigor before relying on it. See work-analysis.

Pitfalls

• Including content not part of the work domain (standardization artifacts aside) or, worse, irrelevant content.
• Over-specifying content for work that is about to change.
• Resting on a thin competency model.
• Treating face validity as content evidence (acceptability ≠ validity).
• Failing to document why domain elements were excluded.

Checklist

• Work/worker requirements reasonably stable; feasibility confirmed
• Content domain defined from thorough work information
• Entry-vs.-trained KSAOs distinguished
• Sampling rationale (importance/prerequisite) documented in a test plan
• Specificity level chosen against likelihood of work change
• SMEs qualified; linkage method standardized and documented
• Administration/scoring guidelines set in advance
• Reliability assessed where feasible
• Excluded domain elements and their rationale recorded

See also

work-analysis (defines the domain) · validation-planning · internal-structure-validation · generalizing-validity-evidence · technical-validation-report

Source: Principles (5th ed., 2018), "Sources of Validity Evidence → Evidence for Validity Based on Content."