Questionnaires and forms

A questionnaire is a written document that is used to obtain information directly from a respondent. Respondents have to be able to read and write in the language of the questionnaire so it is important to ensure that this is a valid assumption for the population of interest, for instance there is quite a lot of illiteracy among offenders. Where the information is being collected indirectly, such as from a probation officer about an offender, the document in frequently referred to as a form, but effectively it is a questionnaire and the same issues apply to the design of forms as to the design of questionnaires. For simplicity, the following description will refer to questionnaires.
The following sections give guidance on how to create a questionnaire or form. Before creating an instrument from scratch it is preferable to assess the value of using appropriate instruments from the wide range presented in the catalogue of instruments in Chapter 7. Although the use of a pre-existing instrument may compromise the precise assessment of programme objectives, it is likely that this will be outweighed by the value of obtaining data directly comparable with that from other studies. The evaluation of Pathfinder programmes being undertaken by the Home Office uses a range of instruments, and these instruments should be the first choice wherever possible, as the comparable data will be of high quality from substantial numbers. These instruments are identified in Chapter 7. More detailed guidance about specific techniques can be found in Oppenheim (1992) and Robson (1993).

Designing questionnaires

This is discussed in four stages: types of question, designing questions, combining them in a questionnaire, and ways of improving the reliability and validity of questionnaires.

Types of question

There are two broad types of question: open and closed. Open questions are those where the respondent is presented with a blank space after the question and is able to write whatever they wish in answer. Closed questions present the respondent with a list of alternative answers after the question.

Questionnaire construction

Questionnaires and structured interview schedules are basically the same. Some important guidelines are as follows.
- Only include questions that relate directly to the issue being studied.
- Keep instructions clear.
- The questionnaire should appear neat, professional and easy to complete.
- Opening questions are important for arousing interest. It is sometimes useful to include ‘warm-up’ questions.
- Questions should flow logically.
- Put any sensitive questions towards the middle or end of the questionnaire.
- Arrange the questionnaire so it is easy to see any missing sections.
- Think about how you will analyse the answers.

Increasing reliability and validity of responses

To be reliable each respondent must understand the question in the same way. Piloting the questionnaire will help to ensure consistency. It is possible to ask a question in a number of different ways. Sets of questions are often used for attitudes. Anonymity can sometimes improve honesty, but be careful as you will not be able to link the responses with other information.

Designing questions

It is important that a question will obtain the required information (validity) in a reliable way from all respondents.
- Ask clear factual questions wherever possible.
- Each question should contain only one idea.
- Use simple language.
- Avoid ambiguous questions – try them out.
- Avoid the use of double negatives.
- Avoid leading questions.
- Multiple response questions must have responses for all possible answers including ‘don’t know’ or ‘not applicable’.
To be valid the questions must address the real issues. Measuring validity for attitudes is a skilled task and it is preferable to use tried and tested instruments.

Pilot runs can improve reliability and validity. Try out the questions on your friends or a small number of respondents, and then conduct a more formal pilot.

Low response rates can reduce the overall validity of the exercise. A range of techniques can increase the response rate to questionnaires
1. Explanation and invitation, e.g. in a covering letter
2. Engaging the interest of respondent, in the letter and via the questions
3. Credibility, e.g. university sponsored, or stressing the value to a respected organisation
4. Neat appearance and presentation, smart and professional
5. Easy to complete (and as short as possible)
6. Easy means of return, e.g. SAE, or accessible collection point
7. Confidentiality, but do not promise what you cannot provide; anonymity sometimes helps
8. Reminders and cut off dates
9. Incentives – free gifts may not be appropriate for the probation service but other less tangible incentives may be effective, e.g. the respondent has been specially selected, or promising feedback – but be sure this can be fulfilled.
Maitland and Nickalls (1985) is a useful reference on questionnaire design.

Questionnaires and forms

Attitude measures are usually a series of special pre-categorised questions that collectively represent a respondent’s attitude. They consist of a series of statements about the topic of interest, and the respondent is required to state the extent to which he or she agrees with each statement. Creating valid and reliable attitude measures involves substantial fieldwork with potential respondents, and extensive pilot work and statistical testing. This refines and reduces a list of items to the minimum that can reliably distinguish between respondents. Scales usually include several items that correlate, demonstrating the measurement of an underlying common theme.

The design of attitude measures is a specialist task not to be undertaken lightly. There are real problems of reliability and validity that require considerable investment of time and expertise to address. It is usually wiser to use an established instrument wherever possible.

There are many existing scales that have been usefully employed in evaluation. It is important, however, to check out the requirements for valid use of these psychometric instruments. They may be copyrighted and require permission and payment before use. Some will require specific qualifications or training before they can be administered and/or analysed and interpreted. These two factors could affect the feasibility of using certain instruments.

Literacy and understanding

A very important consideration in the use of self-completion questionnaires, especially for offenders, is the ability to understand and answer the questions. The questionnaire cannot be completed as intended if the respondent does not understand the questions. They must be able to read and write in the language of the questionnaire at the appropriate level, which raises questions of literacy, language and intellectual ability. If it is not possible to ensure this understanding it may be more appropriate to use the questionnaire as an interview schedule and ask the questions face-to-face, with the use of an interpreter if necessary.