administration-documentation

Administration documentation

The operational counterpart to the technical-validation-report. It tells administrators and users how to deliver the procedure so that scores mean what the validation says they mean. Keep it separate from the technical report so confidential research information is protected and administrators get only what they need. A publisher's manual often must be supplemented with organization-specific addenda for local decisions.

A test is valid for an inference only if administered, scored, and used as it was validated. Deviations from prescribed procedures threaten the inference.

Who administers — qualifications & training

State the qualifications required for each administrative responsibility (administering, scoring, maintaining scoring algorithms, handling retest requests). Train administrators on the specific procedure and on why standardized administration matters — failures to follow protocols can render research results and operational scores partly meaningless. Build observational checks/ quality control; periodically review pass rates / mean scores for spikes or dips signaling a compromised key or administration problems. Administrators should be aware of personal limitations (physical, perceptual, cognitive) and decline to administer/score when they can't meet role demands.

What to document

• Introduction & overview — purpose; why the org uses validated procedures; importance of security and consistency. Don't oversell the program's applicability.
• Contact information — whom to contact for questions/problems.
• Selection procedures — names, editions, forms; ordering and security; sample items (not full secure content); for proprietary tools, a description of items and constructs.
• Applicability — to whom the procedure applies, exceptions/exemptions, and rules about when it's administered (e.g., only when a vacancy exists vs. periodic pooling). Include applicability to candidates with disabilities and different cultural/linguistic groups.
• Administration responsibilities & guidelines — detailed administration instructions; rules/tips for an appropriate testing environment and verifying candidate identity; for remote/unproctored internet testing (UIT/RPIT), the administrator still must inform test takers of environmental factors and security expectations.
• Administration environment — workspace, lighting, quiet, freedom from distractions, and effects of technology (browser, monitor size, touch screen). At minimum, an environment responsive to candidate concerns and dignity; inform test takers of conditions that may affect performance.
• Scoring instructions & interpretation guidelines — how to score and interpret; keep scoring keys out of manuals/technical reports (release only to those who score). Document any administrator role in interpreting scores; ensure accuracy in scoring/recording (responsibility can't be delegated away). For computer-based scoring/automated feedback, give detailed decision-use instructions.
• Reporting and using scores — how scores are reported and used, who has access; caution against using scores for unintended purposes (e.g., later retention decisions) without evidence; if scores drive differential job assignments, require evidence linking scores to those jobs.

Security

Public disclosure of content/scoring is a serious threat to reliability, validity, and subsequent use. Retain all materials at a security level allowing access only to those with a need to know. In UIT/RPIT, adopt mechanisms to reduce content compromise and cheating and to confirm identity. Communicate and enforce protections (verification codes, content rotation, supervised destruction of confidential documents); document a retention period for confidential information; consult evolving international data-privacy law.

Candidate-facing communications

• Information provided to candidates — clear, pertinent, timely: intended use, procedures, format/ interface, completion strategies (e.g., can they skip/return), time parameters, feedback/score access, confidentiality, accommodation request process, retesting policy, conduct expectations. Standardize content and delivery for consistent treatment.
• Candidate feedback — if feasible/appropriate, give clear, understandable feedback without compromising test security or scoring.

Special situations

• Nonstandard administrations — document how to handle accommodations and disruptions (power failures, emergencies, illness), errors (mismatched booklet/sheet, timing mistakes). Establish a clear process to document and explain any deviation in administration, scoring, or handling. Where reporting a nonstandard event (e.g., dropped internet connection) is left to the test taker, explain how and when to report.
• Reassessing candidates — generally provide reassessment opportunities when feasible; state clearly whether and how candidates may be reassessed (required time intervals, prerequisite developmental activities). Some one-time examinations may preclude it.
• Corrective reassessment — define when it's appropriate (administrator errors like timing mistakes/nonmatching materials; extraordinary disturbances). Document procedures, required timing, and documentation; inform remote test takers of the conditions under which they may request it.

Data retention & test-score databases

Decide what administration records/scores to keep; document record-keeping and retention policies (duration, security, accessibility) consistent with federal/state/local law, industry best practice, and recent court rulings. Retain raw item data and scores (derived scales limit future research). Secure personally identifying data to need-to-know access. Maintain databases long enough to support periodic audits and ongoing evaluation of the selection system.

Review & updating

Periodically review operational use — including timeliness of normative data — to decide whether more research is needed. The greater the changes to test format, mode of administration, instructions, or language, the more likely score interpretations no longer hold. Revise documentation (or add an addendum) whenever changes make any statement or instruction incorrect or misleading. Changes in organizational demands (new assessments, cutoff adjustments) require further study of the procedure.

Pitfalls

• Putting scoring keys or live items in the manual.
• One-size communications that aren't standardized → inconsistent candidate treatment.
• No documented process for deviations/disruptions.
• Vague or absent retest/reassessment rules.
• Letting the manual drift out of date after format/mode/language changes.
• Using operational scores for purposes never validated.

Checklist

• Administrator qualifications & training (incl. standardization rationale) documented
• Testing environment and identity-verification guidance (incl. UIT/RPIT) provided
• Scoring/interpretation documented; keys kept secure and out of the manual
• Security mechanisms and retention period specified; privacy law considered
• Candidate communications standardized; feedback approach set
• Nonstandard-administration and corrective-reassessment processes defined
• Reassessment/retest policy stated
• Data-retention and database-audit policy set
• Review/update triggers and addendum process in place

See also

technical-validation-report (keep separate) · candidate-accommodations (nonstandard administrations) · fairness-and-bias-analysis (equitable treatment) · selection-decisions-and-scoring (how scores are used)

Source: Principles (5th ed., 2018), "Operational Considerations → Administration Information and Other Circumstances Regarding the Validation Effort and Use of Selection Procedures."