What to evaluate

New initiatives, special programmes and new ways of working are the prime targets for evaluation because we need to be clear that the new ways are more effective than the old. However, there is also a need to evaluate existing practice with the same rigour. Such evaluations do not need to be on a large scale, and there is great value in small-scale evaluations that have clear objectives and appropriate methods. For instance, individuals who work with offenders evaluating their own one-to-one work can highlight important issues. It is often possible to make small modifications to existing data collection or reporting processes to provide the basis for more systematic evaluation of working methods.

The reduction of offending is generally the ultimate aim of work with offenders, but this is usually addressed by focussing on one or more aspects or issues that are related to offending, such as substance abuse or antisocial attitudes. In order to better understand the relationship between such factors and changes in offending behaviour it is important that the immediate objective(s) of such work is assessed. These are often known as intermediate outcomes. Without knowledge of the achievement of the programme objectives it will not be possible to explain how a change in offending has been achieved.
Evaluation can be undertaken about any aspect of practice, and can be done at an individual case level as well as with groups and in relation to special programmes. For effective practice to generate knowledge it is important that such evaluation is informed by ethical and methodological rigour. The key is to know the ‘what’ – the precise aspect of practice being evaluated, and the ‘why’ – the theoretical reasoning that explains why it is expected to be effective.

It is also important to consider the evaluation of evaluation methods and instruments – often referred to as validation. Much evaluation of work with offenders has been undertaken in the Western world, but little of it has been replicated in other contexts, though there is an increase in ways of working crossing geographical boundaries. There is considerable scope for re-testing previously used methods and instruments, as well as developing new ones. When taking a way of working from one context to another, such as from one country to another, it is important to evaluate the work in the new context to ensure that it retains its effectiveness, such as the transfer of the SEED model from England to Romania as described in workstream 2 of the STREAM project.

When to evaluate

Ideally, decisions about evaluation should be made at the planning stage of any piece of practice, as this will improve both the practice and the evaluation: consideration of the needs of evaluation during the early stages will assist the design of the programme, and knowledge of the underlying thinking behind a programme will improve the evaluation.

Some sorts of evaluation can only be undertaken prospectively, that is, by collecting data as events occur, for example before and after measures. Such data is most reliably collected during the routines of practice. The integration of data collection procedures into the operation of programmes, or the use of data generated during the practice of a programme, can be ‘designed-in’ much more easily than it can be ‘tacked on’ later in the process.

Retrospective evaluation (collecting data after the event) can be illuminative, often clarifying the unarticulated objectives of a piece of work. It can generate knowledge to inform the development of practice. For instance, an interesting piece of evaluation for an individual who works with offenders would be to look at cases worked with over a two year period and identify those which were thought to be particularly successful. Looking for common aspects of these cases can help to gain a more detailed understanding of what worked, for whom, and in what circumstances. A similar process was carried out as a component of the two pilot projects.

Who to evaluate

Good evaluation requires some understanding of the work being evaluated, as well as the skills to determine and apply appropriate evaluation methods. Those who deliver the work with offenders have the detailed knowledge and understanding of the practice being evaluated and insights into the theoretical underpinnings of that work, but do not necessarily possess the skills or the time to undertake evaluation of their work.

Evaluators have the skills but often do not have the detailed understanding of practice. With appropriate guidance and support, however, those who work with offenders can and do undertake successful evaluations. This guidance will provide some help to those wishing to evaluate practice, and evaluators are often willing to provide some support and advice to individuals doing their own work. Conversely, evaluators should draw upon the skills and knowledge of service staff in designing evaluations. Partnership between the evaluator, the service provider and the manager is key. Good evaluation will be integrated with practice and is linked to practice aims and definitions.

Skills in evaluation may be available from local universities. This can be undertaken on a consultancy basis or by contracting a full evaluation. Evaluation can be designed to be undertaken by service delivery staff and managers too, with guidance from experienced evaluators enabling them to develop skills in this area.