What you'll learn

• What reliability means in psychological research.
• How reliability applies across experiments, observations, questionnaires, interviews, correlations, case studies and content analysis.
• How to assess reliability using test-retest and inter-observer methods.
• How to improve reliability without confusing it with validity.

The basic idea: reliability means consistency

In research methods, a method of investigation is the way psychologists collect data: for example, an experiment, observation, questionnaire, interview, case study, correlation or content analysis.

Reliability is about whether that method produces consistent results. If the same thing is measured again under similar conditions, a reliable method should give similar findings.

A bathroom scale gives a simple analogy. If it gives wildly different readings every time you step on it, it is unreliable. In psychology, the same issue applies to a depression questionnaire, an observation of aggression, or a content analysis of media messages.

Here is the big picture of the two main AQA reliability checks and how researchers improve them.

Reliability is not the same as validity

A method can be reliable but not valid.

For example, a questionnaire might give very consistent scores each time it is completed, so it has good reliability. But if it is supposed to measure anxiety and actually measures social desirability — how much someone wants to look good to others — then it has poor validity.

Reliability across methods of investigation

Reliability matters in all psychological methods, but it looks slightly different depending on the method.

Experiments

In an experiment, the researcher manipulates an independent variable, which is the factor being changed, and measures a dependent variable, which is the outcome.

Reliability improves when the procedure is standardised, meaning every participant is treated in the same way as far as possible. This includes the same instructions, same timing, same materials and same scoring rules.

For example, Loftus and Palmer’s 1974 study on leading questions used controlled film clips and standardised questions. This helps reliability because another researcher could repeat the procedure more easily. However, reliability does not automatically mean the task is realistic, so validity still needs evaluating.

Observations

In an observation, researchers record behaviour as it happens. Reliability depends heavily on whether observers agree about what they saw.

If one observer records “aggression” but another does not, the problem may be that the behavioural categories are unclear. Researchers improve reliability by making categories precise, such as “hits another child with hand or object” rather than “acts aggressively”.

Questionnaires and interviews

A questionnaire is a set of written questions. An interview is a spoken research conversation.

Structured questionnaires and structured interviews are usually more reliable because everyone receives the same questions in the same order. Unstructured interviews may be less reliable because the interviewer can ask different follow-up questions each time.

Correlations

A correlation investigates whether two co-variables are related. Reliability matters because both variables must be measured consistently. If one measure is unreliable, the correlation may be weaker or misleading.

For example, if a stress scale gives unstable scores from one day to the next for no clear reason, it may be hard to judge whether stress is genuinely related to sleep.

Case studies and content analysis

A case study is an in-depth investigation of one individual, group or institution. Case studies can be difficult to replicate exactly, so reliability may be lower.

A content analysis is a method for systematically coding communication, such as adverts, newspapers, films or interview transcripts. Reliability improves when coders use clear categories and independently code the same material.

Measuring reliability: test-retest reliability

The logic is simple: if the measure is reliable, participants should get similar scores both times, assuming the thing being measured has not genuinely changed.

Researchers often compare the two sets of scores using a correlation coefficient, which is a number showing the strength and direction of a relationship between two variables. A strong positive correlation suggests good test-retest reliability.

In AQA Psychology, you do not usually need to calculate this from raw data, but you should understand the decision: if scores at Time 1 and Time 2 are very similar, reliability is high.

Choosing the time gap

The time gap in test-retest reliability needs to be sensible.

If the gap is too short, participants may remember their previous answers. This can create artificially high reliability due to practice effects or memory.

If the gap is too long, the psychological characteristic may genuinely change. For example, stress may change across an exam period, and mood can change from day to day.

Measuring reliability: inter-observer reliability

This is especially important in observations and content analysis. It can also apply when researchers code interview transcripts.

The usual procedure is:

1. Create clear behavioural categories or coding categories.
2. Train two or more observers to use the same categories.
3. Ask them to record the same behaviour or code the same material independently.
4. Compare their records to see how much they agree.

A simple method is percentage agreement, which calculates the percentage of observations where the observers gave the same code.

$$percentage\; agreement=\frac{number\; of\; agreements}{total\; observations}\times 100$$

More advanced research may use Cohen’s kappa, a statistic that adjusts for agreement that could occur by chance. For A-Level Psychology, it is usually enough to know that high agreement between observers suggests good inter-observer reliability.

Improving reliability

Reliability is not just measured after the study. Good researchers build it into the design from the start.

Operationalise variables clearly

“Anxiety” is vague. “Score on a 20-item anxiety questionnaire” is operationalised. “Aggression” is vague. “Number of times a child hits, kicks or pushes another child in 10 minutes” is clearer.

Clear operationalisation makes it more likely that another researcher could repeat the study in the same way.

Standardise the procedure

Standardisation reduces the influence of extraneous variables, which are unwanted factors that might affect the results. For example, if some participants receive extra encouragement and others do not, differences in scores may be due to the researcher rather than the variable being studied.

Train observers and interviewers

Observers should practise using the same coding system before the main study. Interviewers should use an agreed interview schedule if reliability is important.

This reduces observer bias, which occurs when an observer’s expectations influence what they record.

Use pilot studies

A pilot study is a small trial run before the main investigation. It helps researchers spot unclear questions, confusing instructions, overlapping categories or practical problems.

For example, a pilot observation might show that “physical aggression” and “rough play” are too difficult to separate. The researcher can then refine the categories before collecting the main data.

Use clear scoring rules

Questionnaires, interviews and content analyses are more reliable when researchers use a scoring manual. This explains exactly how answers or behaviours should be coded.

However, there is a trade-off. Closed questions and rigid coding can improve reliability, but they may reduce richness and validity if they oversimplify complex experiences.

AO3 evaluation: why reliability matters

Strength: supports scientific replication

A major strength of reliability is that it supports replication, which means repeating a study to see whether similar findings are obtained. Psychology aims to be scientific, so reliable methods help researchers check whether findings are dependable rather than accidental.

Strength: improves confidence in findings

If a questionnaire gives similar scores across time, or observers agree closely, researchers can be more confident that the data reflect a real pattern. Low reliability adds “noise” to the data and can hide genuine effects.

Limitation: reliability does not prove validity

A reliable measure may consistently measure the wrong thing. For example, a memory test might produce consistent scores because participants learn the task format, not because it accurately measures everyday memory.

This is important in essay evaluation: never treat reliability as the only mark of good research.

Limitation: high standardisation may reduce realism

Highly standardised procedures can improve reliability, but they may reduce ecological validity, which is the extent to which findings reflect real-life behaviour. For example, a tightly controlled laboratory task may be easy to replicate but unlike real social interaction.

Ethical angle

Improving reliability should not come at the expense of ethics. Researchers still need appropriate consent, avoidance of unnecessary deception, the right to withdraw, protection from harm, confidentiality and a proper debrief.

This is especially relevant in observations. Covert observation may improve natural behaviour and reliability, but it can raise consent and privacy issues.