To be valid the questions must address the real issues. Measuring validity for attitudes is a skilled task and it is preferable to use tried and tested instruments.
Pilot runs can improve reliability and validity. Try out the questions on your friends or a small number of respondents, and then conduct a more formal pilot.
Low response rates can reduce the overall validity of the exercise. A range of techniques can increase the response rate to questionnaires
1. Explanation and invitation, e.g. in a covering letter
2. Engaging the interest of respondent, in the letter and via the questions
3. Credibility, e.g. university sponsored, or stressing the value to a respected organisation
4. Neat appearance and presentation, smart and professional
5. Easy to complete (and as short as possible)
6. Easy means of return, e.g. SAE, or accessible collection point
7. Confidentiality, but do not promise what you cannot provide; anonymity sometimes helps
8. Reminders and cut off dates
9. Incentives – free gifts may not be appropriate for the probation service but other less tangible incentives may be effective, e.g. the respondent has been specially selected, or promising feedback – but be sure this can be fulfilled.
Maitland and Nickalls (1985) is a useful reference on questionnaire design.