What reliability is

1. Reliability is the consistency of a measure or a procedure.
2. A reliable measure gives the same result when it is repeated under the same conditions.
3. It is about consistency rather than about being correct, so a measure can be reliable and still measure the wrong thing.
4. A bathroom scale that always reads 2 kg heavy is perfectly reliable and still inaccurate.
5. Unreliable results cannot be trusted, because there is no way to tell which run to believe.
6. There are three types: internal, external and inter-rater.

Internal reliability

1. Internal reliability is whether a measure is consistent within itself.
2. On a questionnaire it means every item is tapping the same thing rather than pulling in different directions.
3. It is checked by comparing the answers on one half of the items with the answers on the other half.
4. An item that disagrees with the rest is reworded or dropped.

External reliability

1. External reliability is whether a measure gives consistent results on different occasions.
2. The usual check is test-retest, giving the same test to the same people twice.
3. A memory test that produces similar scores a week apart has good external reliability.
4. Standardised instructions and matching conditions on each occasion are what protect it.

Inter-rater reliability

1. Inter-rater reliability is whether two people scoring or observing the same thing agree on what they record.
2. Both watch the same event, record separately, and their totals are then compared.
3. It matters most in observations, where a judgement is being made in real time.
4. Clear behavioural categories, and training both raters on the same examples, are what raise it when they disagree.

Test-retest

1. Test-retest means giving the same test to the same people on two separate occasions.
2. Everything else is kept identical, including the wording, the setting and the time allowed.
3. The gap is left long enough that nobody is still remembering their earlier answers.
4. The two sets of scores are then compared, and a close match shows good reliability.
5. With too short a gap people recall their old answers, so the match is falsely high.
6. With too long a gap the thing being measured may genuinely have changed, so a poor match is not the fault of the test.
7. The match is judged by laying the two sets of scores side by side and seeing how closely each person's pair agrees.

Raising the reliability of a study

1. Write standardised instructions and read them out word for word to every participant.
2. Hold the conditions steady, using the same room, the same time limit and the same materials each time.
3. Define behavioural categories tightly, so two observers cannot reasonably disagree about what counts.
4. Use closed questions and rating scales where possible, because a fixed set of options removes the guesswork from scoring.
5. Add a second rater and compare totals, then rewrite whatever they disagreed about.

Judging reliability in a described study

1. A procedure described tightly enough for someone else to run again has good external reliability.
2. Data that two different recorders would have written down the same way has good inter-rater reliability.
3. A measure that would land in the same place on another day is consistent across occasions.
4. A vague behavioural category, an unscripted instruction or a single observer is the feature that lowers reliability in most study descriptions.

Turning this into marks

1. Name the type of reliability and then point at the feature of the study that causes the problem, keeping the two as separate points.
2. Say what the problem does to the findings, because naming a type of reliability on its own does not secure the knowledge mark.
3. Describe the check in full, such as a second observer tallying the same lesson and the two totals being compared.
4. The mark most often lost is calling a study unreliable without tying the claim to anything in its procedure.