Applying Reliability
- This skill asks you to judge how reliable a described study is and suggest improvements.
- Ask whether the study would give the same results if it were run again in the same way.
- Then suggest a change that would make it more consistent.
- Reliability improves when everything is kept the same each time.
- Standardising the procedure is usually the strongest fix.
A Worked Example
- Two observers sit in a busy UK shopping centre and rate how helpful passers-by are when someone drops their bags.
- They score helpfulness from 1 to 5, but their scores often differ by two or three points.
- That gap shows low inter-observer reliability.
- The likely cause is that "helpful" has not been clearly defined, so each observer judges it differently.
- Replace the vague 1-to-5 scale with observable actions, such as "stops", "picks up a bag" or "speaks to the person".
- With clear categories, the two observers now agree far more often.
Ways to Improve Reliability
- Standardise the procedure so every participant is treated in exactly the same way.
- Operationalise the categories or scoring so they are applied consistently.
- Train the observers or raters together first, so they record in the same way.
- Do not suggest an improvement that does not actually target consistency.
- Do not confuse improving reliability with improving validity.
A Step-by-Step Routine
- Decide whether the study would give the same results if repeated the same way.
- Name the exact source of inconsistency in the scenario, such as a vague rating observers judge differently.
- Suggest a matched fix (standardise the procedure, operationalise the categories, or train the observers); the fix tied to the problem is a separate mark.
- How can you tell a study has low reliability?
- Name two ways to improve the reliability of an observation.
- Why does standardising a procedure improve reliability?
- If two observers disagree a lot, what should you improve?