Using existing sources of data (secondary analysis)

It is frequently worthwhile considering the use of existing records within the probation service before designing a method to collect data from scratch. Probation is an information-based operation which requires the recording and transfer of data at a number of levels in day-to-day work. There are two ways of using such processes as a source of data for evaluation. The first is by extracting data from the material generated routinely in these processes; the second is by organising a slight modification of these processes to enable the evaluation. The first method is usually preferable because the material already exists, is available for immediate analysis and has not been affected by the evaluation process. There is the potentially significant disadvantage that such material does not contain exactly the information required for the evaluation. In these circumstances it can be advisable to modify systems and to wait for the data to be generated. Information collected and processed in computer systems provides a valuable source of data for evaluation.

Documents as a source of data (content analysis)

Much information about the work of the probation service is collected and stored in textual documents. This data can be accessed by content analysis, which is the systematic analysis of the content of documents on a quantitative basis. It is essentially asking questions of a document rather than a person, and is governed by the same principles as designing a questionnaire. The range of sources available includes minutes and agendas of meetings, court lists, case records and written reports. Data of two broad sorts can be obtained from such sources:

1. concrete – what is said, i.e. the actual information on the page, ‘facts’, e.g. age
– how it is said, i.e. the language and style of writer

2. abstract – inferences from what said, e.g. values and opinions of writer; this is more difficult and less reliable but can be useful, particularly in qualitative work.
Sometimes a document is the best or only source of certain kinds of data, for instance the proposal in a PSR and the way in which it is presented to the court. This is particularly the case where an evaluation contains a quality assurance component, such as an evaluation of supervision plans. This exercise could produce information at three different levels: a) about the quality of supervision plans, b) about the sorts of issues that officers address regularly within them, and c) a description of the offenders being supervised by the officers.

Advantages

- Material is easily accessible and usually readily available for immediate analysis.
- It does not need the co-operation of lots of people.
- The evaluator does not influence events (no observer effect).
- It is cheap relative to other methods. - Documents are sometimes the only source of information (e.g. historical.

Constraints

- The main constraint is the document being analysed, the contents of which are determined by the purpose for which it created. It is important to remember that where something is not mentioned that does not mean that it does not exist or did not happen, but that it may not have been important to mention in this context.
- The evaluator has no control over the information available, and therefore cannot decide in advance the questions to which answers will be provided.
- The information can be inconsistent. Many different observers may have created the documents they observe and record different things, and certain information may be present in some and not others.
- The document can be quite distant from what actually happened, e.g. minutes of meetings, which will affect the validity of the data extracted.
- Content analysis of documents can be laborious.

Data from existing monitoring systems


Most probation services will have a range of monitoring systems which routinely collect information about the work of the service and the offenders being dealt with. Frequently the use of this information is limited to routine management monitoring reports. These systems can be a valuable source of data for evaluation. Coverage of the data tends to be comprehensive and thus can provide information about relatively large samples of individuals. The biggest problem in such exercises can be the limitations of the system for undertaking non-routine analyses and the extraction of sub-sets of data for analysis using other computer programmes. Systems are improving all the time, however, and it is becoming easier to carry out this sort of evaluation.

Again, focus is the important factor in deciding what information is required from such an exercise. The sorts of analysis that can be undertaken will be constrained by the data that is recorded within the system and the way in which it is recorded.

For example, an evaluation of groupwork might start with a review of the statistical data about all offenders given an order with a particular groupwork requirement over a one-year period. To assess the outcomes of these orders it is necessary to allow time for the orders to be completed, which would suggest that the orders should be chosen by their completion date. From this data it should be possible to look at the outcomes of the order, and of the groupwork requirement and to compare sub-groups within the sample. Very simple analyses using data such as age, sex, race, type of offence and previous conviction can help to identify groups where a groupwork condition seems to have a more successful outcome. If the monitoring system collects such information, it may be possible to analyse in more depth by the type of group that the offender attended. Where this data is not available within the monitoring system a sample of cases can be identified for the collection of additional data to supplement the original monitoring information.