Evaluation of SEED in Romania and England

Angela SorsbyJoanna Shapland

University of Sheffield

Funded by National Offender Management Service (England) and European Commission

SEED aims to improve practitioner/ convicted person engagement.

It focuses on what practitioners do with convicted persons in supervision sessions and aims to enhance the effectiveness of one-to-one work.

In the evaluation we were interested in:

•practitioners’ views about the training they had received, particularly in terms of whether it had improved their engagement skills

•managers’ views

•views of convicted persons about their supervision, for those supervised by SEED trained staff and those supervised by staff that are not SEED trained

•whether there is any impact on compliance.

Evaluation of impact requires trained and comparison groups.

Research design and methods to some extent determined by resources and practicalities.

Random assignment not possible

• Random assignment of probation teams would require excessively large number of participating offices.

• Random allocation of practitioners to training does not fit with the SEED model of training teams (or most of the team) together.

• Random assignment of convicted persons to practitioners not practical for a number of reasons e.g. geographical location.

In England the whole team trained together.

For each SEED trained team we had a comparison team that was not SEED trained.

The disadvantage was that, almost inevitably, the profile of convicted persons was not identical for trained and comparison teams.

Multivariate statistics were used to statistically take account of differences.

In Romania, originally conceived as feasibility study but decided to do scaled-down replication of English Evaluation.

Most but not all the practitioners in each team took part in SEED training.

Those that did not take part formed the comparison group.

Likely that convicted person profile will be more similar for the two groups, compared to England.

Disadvantage possible contamination.

SEED evaluation methods

Capturing staff views

•Observation of training and informal talks with staff.

•Questionnaires to staff undergoing training at every training session.

•Interviews with managers, practitioners and trainers.

SEED evaluation methods

Capturing convicted persons’ views

•Questionnaires to convicted persons in SEED and comparison groups asking them about the supervision they are receiving. Is there any difference between the groups and between the countries?

•In-depth interviews with convicted persons near the beginning and end of their order.

SEED evaluation methods

Assessing whether there is any impact on outcomes

•Comparison of available compliance data for SEED and comparison groups.

Practical observations in relation to some of the methods

Observation of training

•Extremely useful, gave us a direct overall impression of practitioners’ reactions and their views in relation to potential issues, especially as training includes SWOT analyses, focus groups and other forms of feedback.

•Resource intensive, observed almost all initial training and follow-up training in England, in Romania could only observe initial training in Bucharest.

Practical observations in relation to some of the methods

Questionnaires to practitioners participating in training

•Fairly straightforward, easy to administer, gave us more detailed quantifiable data about staff reactions.

•Indicated slightly different things appreciated in each of the two countries.

Practical observations in relation to some of the methods

Questionnaires for convicted persons - issues for staff

Substantial administrative burden for probation staff in terms of getting them to the right people at the right time - only to be given to people who commenced orders after staff completed their initial training and to be administered at a certain point in the order.

Issues in relation to local availability of data about who commenced orders after training and who was at the right stage in their order.

In England local offices had to obtain lists from centralised IT departments, problems with delays and lists including people outside of the relevant office or parameters of the study, hence required sifting.

Practical observations in relation to some of the methods

Questionnaires for convicted persons issues in relation to convicted persons

Potential literacy issue in some jurisdictions. Completion rate and written comments indicated this may not have been too much of an issue in England and Romania.

Study in Romania intended as a feasibility study of whether evaluation can be applied to another EU jurisdiction. Likert type scales

Never o------o------o------o------o Almost every session

did not seem to be so clearly understood in Romania.

Practical observations in relation to some of the methods

Compliance data

Lots of practical issues around what data is collected, how it is stored and how it can be provided for research purposes.

Case management systems tend not to be designed for research purposes.

Practical observations in relation to some of the methods

Compliance data

In England, data fields that can be used for queries were overwritten when there were changes. For example, if someone was recalled to prison, the release date was removed from the release date field, so if you tried to extract people released between certain dates using the release date field, that person wouldn’t appear in the data.

Regular downloads therefore required to avoid losing people.

Queries also tended to capture people outside the parameters of the project so considerable work for evaluators in determining who should and shouldn’t be in the data set.

Practical observations in relation to some of the methods

Compliance data

In Romania systems set up to run off cases by office but not individual probation counsellors.

As SEED trained and comparison probation counsellors were in the same office, we needed cases organised by probation counsellor or by whether trained or not trained. This was done by probation staff so required quite a bit of work by them.

Probation orders in Romania are long, an average of five years, so within the timescale of the project we can only look at compliance over a small proportion of the order.

In conclusion

We have obtained a lot of worthwhile data but this has involved considerable effort from SEED trainers, probation counsellors, administrators, volunteers, IT departments, the National Offender Management Service in England and the Ministry of Justice in Romania, all of whom I am grateful to.

Questions and discussion