Observation: a guide for use in evaluation

Observations can be made in just about any situation: you see and hear things all the time, and it’s all data that can be turned into evidence.

Observation provides the opportunity to monitor or assess a process or situation and document evidence of what is seen and heard. Seeing actions and behaviours within a natural context, or as they usually occur provides insights and understanding of the event, activity or situation being evaluated. The key to using observational data as evidence in an evaluation is to take a systematic and consistent approach as you collect, organise and analyse what is observed. These notes should be read alongside the general advice on understanding quantitative data and qualitative data, as observational data can include both types.

Advantages

• Observation is a flexible approach to data collection, suitable for a broad range of contexts.
• Observation can produce a mix of qualitative and quantitative data. For example, when observing people in a group situation, you might count up how many times certain behaviours or interactions occur (quantitative), while also taking freehand notes about the nature of the group dynamics (qualitative).
• Structured observation helps provide measures or records of behaviours, without relying on people’s (those being observed) capacity to report what they do or estimate how often they do it.
• Observation can be a low impact way to collect data. When planned appropriately, the observer may have only a minor effect on the activities or blend into the observation setting.
• The discussion of feedback from observation can lead to valuable reciprocal professional learning conversations.
• Teachers can also observe teaching and learning in their own classes using digital recording technologies, such as 360 degree cameras or other appropriately placed recording devices.

Relationship to other methods

Observation can be used as stand-alone data collection tool. Often, however, observational data is used in conjunction with other approaches as part of an evaluation design. For example:

• Initial focus groups, interviews or surveys might identify a set of behaviours that are of interest. Observation then allows the evaluation team to assess how common the behaviours are, or to look for patterns in the circumstances or triggers that give rise to them.
• Conversely, an evaluation might start with some exploratory observations, and then follow these up with interviews where participants are asked to comment on their experiences in the situation.
• Data from interviews or surveys (about classroom management practices, for example) might be used in conjunction with data from observations (of classroom management) to build a more complete assessment of the effectiveness of a strategy being used in the classroom to improve student engagement.
• Observations can be used in conjunction with other data, such as administrative records or document analysis. After reviewing a mathematics program (document analysis), the evaluation team may use observations to support or challenge claims found in the documents about the effectiveness of the program.

Limitations, and how to manage them

Observation needs to be carefully planned, with a clear understanding of the questions to be answered and the particular behaviours or attributes of interest. This helps avoid being overwhelmed by a vast amount of data, or getting stuck at the analysis stage wishing ‘If only I had kept an eye out for…’.

It is valuable to have two or more people undertake observations, as one person alone might miss things that someone else would see.

Conducting observations can be labour intensive, in preparation, data collection and analysis.

Depending on the rigour required for the evaluation there may be a need to allocate time for pre-observation discussion to ‘calibrate’ the observers. Time may also be needed at the end of the observation for discussion, checking consistency between observers and reflection of what was collected.

Like most measurement in a social setting, the process of collecting observational data will have an influence on what is being measured and can result in unintentional biases that we need to be mindful of.

Skilled observers are good at being unobtrusive or ‘fading into the background’ when they need to. Early data may need to be discarded while subjects are still getting used to the observer, especially if using video to record the activities. The more familiar people are with observation – the more ‘normal’ it becomes – the less this problem arises.

Keep reading

• Step 1: Set the scope and choose an approach
• Step 2: Prepare the tools and the people
• Step 3: Conduct observations
• Step 4: Organise and analyse the data

External links