Types of design


Experimental

This is the ‘ultimate’ research design, which would enable definitive statements to be made about effectiveness if it were possible to implement in practice. It requires that all factors of influence, other than the one that is the focus of the programme, be controlled. Individuals are allocated to one of two groups: the ‘experimental’ group, which receives the programme, and the ‘control’ group, which does not. The two groups are determined by means of statistically random allocation, a method that ensures that an individual has an equal chance of being allocated to either group. Comparison of the outcomes of the two groups enables any improvements to be attributed to the programme. The social world is not so easy to control, and random allocation is frequently not possible or desirable, thus true experimental research and evaluation rarely happen outside the science laboratory. Striving for experimental control has, however, led to the development of a range of designs that emulate experimental status – these are called quasi-experimental.

Quasi-experimental

This is the design most frequently used in assessing the effectiveness of a particular method of working with offenders. Comparison of groups is the essential feature of the quasi-experimental design. Naturally occurring groups are frequently used, but the validity of the comparisons and the conclusions drawn depends of the quality of the comparison group chosen. The comparison group should be the same as the ‘treatment’ group in all aspects except the programme (treatment). A discussion of the impact of different sorts of comparison groups and reconviction can be found in section 4.4.1.

The ability to be able to say that ‘like is being compared with like’ is fundamental. This is particularly true when comparing reconviction rates of programme and comparison groups, where Offender Group Reconviction Scale (OGRS) scores can be used as a means of controlling for risk of reconviction (see Table 4.6 in section 4.4.1).

There are numerous techniques for obtaining a control group, including retrospective designs. Sometimes numbers are large enough to enable choice in the selection of subjects in the groups for comparison, in which case ‘matched’ samples can be chosen. A good example of this is the work of Mair and Brockington (1988) where matched samples of male and female offenders were chosen to enable evaluation of the impact of sex on sentencing practice.

Survey

The survey design is one with which we are all familiar. Surveys aim to obtain information in a standard form from large samples. There are several large government surveys which are regularly conducted, for instance the Census and the British Crime Survey. Analysis of this data assesses trends, looks for patterns and makes comparisons between subgroups within the survey. An important feature of surveys is the aim for representativeness of coverage, and therefore the sampling procedures chosen are critical. Data is frequently collected by questionnaire, is usually quantitative and is pre-coded and amenable to analysis by computer. The large survey as such is relatively rare in probation evaluation, although section 6.4 describes a Home Office survey of offenders. The systematic recording of ordinary practice can be a useful evaluation method. An example is assessment, a task routinely undertaken by probation officers in their day-to-day work. If a standard procedure is used across a number of cases this data can be summarised and sub-groups compared to provide more knowledge about the characteristics of those being assessed. If detail about the result of that assessment is also collected in a standard way, whether as a supervision plan, or as a proposal for a certain type of disposal, or acceptance on to a drug programme, this can provide information about the assessment features of those with different results. This can be a very simple and powerful way of identifying potential discriminatory practice. For instance, results that showed that black drink drive offenders were referred proportionately less than white offenders for a drink driver programme would suggest possible discriminatory practice that required further investigation. It would not prove there was discrimination, but it would indicate something worthy of further investigation.

Single case design (case study)

The case study is a detailed examination and study of one case. The case can be a person, a group or a programme, which is chosen as an illustrative example of a way of working. Sometimes several case studies are undertaken and compared to enable greater insights into the feature under study. This is essentially an exploratory approach allowing themes, patterns and the interrelationship of factors to be explored. A case study is methodically planned and data is collected systematically. This approach allows the study of a level of detail that is not possible in large scale work such as a survey. It is frequently undertaken in the early stages of development of a programme. Another common use for case study work is alongside a survey, where the triangulation of method allows the insights from case studies to be linked to more quantitative characteristics. Single case designs applied to one-to-one work are presented in Chapter 6.

Action research

Action research is a means of using evaluation actively to assist the achievement of an outcome rather than to assess whether or not it has been achieved. Its aim is considered change and improvement: the examination of process and outcome with regular feedback into the process which forms the basis of change. It is often a feature of the developmental stage of a piece of practice. It is a systematised reflective practice, a practical problem solving approach applied over a period of time. Although all evaluation is about systematic data collection and analysis as part of the review and improvement of practice, action research is a designed cycle that includes routine evaluation, and reporting and change based on the outcome of that evaluation. The action and change also relate to the methods of evaluation used. The evaluation is regularly reviewed to decide what sort of data is needed for the next phase and how best to obtain it. For instance, questionnaire surveys can be supplemented by case studies, observations or interviews with participants.

Mixed methods


Frequently an evaluation requires a multi-method approach. The three main means of mixing approaches are the following.

Multi-level design

This is about the use of different designs and methods at different stages of the evaluation and for different purposes. For instance, in the exploratory stage of an evaluation the case study design can be very illuminating, identifying key questions that should be asked of all participants. The next stage of the evaluation could consist of a questionnaire to everyone involved in the programme. After this data has been analysed, the next stage may be to convene a focus group of some of the individuals to share the results and ascertain their views.

Triangulation

A term used in map reading which neatly describes the process of approaching an issue from several angles in order to gain a better understanding of it. Three useful kinds of triangulation have been identified by Denzin (1989).

- The first type of triangulation is data triangulation. This is where data is collected from different groups, or different times, or different places, or combinations, each being used to illuminate the others.
-The second type is investigator triangulation where more than one person undertakes the evaluation. This is a way of reducing the possibly biased view of one evaluator.
-The most frequently used type of triangulation is methodological triangulation, where more than one method is used to obtain information about a topic, for instance observation of a group session and interviews with the participants.

Pluralistic approach

This term refers to having an evaluation design that takes account of the range of stakeholders in a programme and trying to obtain information from each, for instance assessing the impact of a programme by asking the offender, the programme leader and the supervising officer. This can be seen as a sort of data triangulation. If the picture obtained from all three is one of improvement it is likely that some real improvement has occurred, even if there is a difference of opinion about the extent of that improvement.