Validity of results

What the published 2016 guidance says

Validity has a practical edge. Results taken at different times can be compared only while the scope and context stay the same. A new tool, a changed sampling method or a redefined population usually breaks the series, and the organization may need a parallel run or another specific test before it joins the old and new lines. Where judgement enters the method, as with a questionnaire, the scoring rule has to be written down. Some observations are too variable on their own and become valid only when pooled.

What it means in practice

Comparability is the quiet failure mode. A trend line drawn across a change of scope, tool or method is not a trend. Record the scope and the method with each series, and when either changes, say so on the chart or run the old and new methods in parallel before you join the line.

What good looks like

Record the scope and method beside every series. When either changes, mark the chart and write a note. If continuity matters, run the old and new methods in parallel for one cycle and record the comparison.

Evidence an assessor asks for

• Performance evaluationMeasurement definitions that record the scope and context they assume
• Data qualityRecords of parallel runs or other comparability tests when a method changed
• Performance evaluationDefined criteria for evaluating subjective inputs such as questionnaires

The question an assessor will ask

An assessor will ask for measurement definitions that record the scope and context they assume. The test for 5.3 is whether that record can be produced for a live sample, not whether the guidance is described in a procedure.

Common gaps

• Trend lines drawn across a scope change or a method change as if the series were continuous
• Questionnaire results reported as measures with no scoring criteria
• Single subjective observations reported where only an aggregate is valid

Before the next review

Before the next review, test this against one live record: trend lines drawn across a scope change or a method change as if the series were continuous.

What to check when the revision publishes

When DIS 27004 is published, compare the 5.3 Validity of results record with this page. Ask whether the information need, the formula, the target, the frequency or the owner changes. Update the measure register and the kit, not the requirement, because the clause 9.1 duty stays as it is.