Collating and scoring interim priorities

Steering Group members will need to have agreed who will be responsible for coordinating, collating and listing the interim prioritised uncertainties (indicative questions). This process is closely monitored by the JLA Adviser to ensure transparency and minimisation of bias.

Completed interim prioritisation results should be grouped into patients and/or carers, and clinicians, and separate scores kept to ensure a fair weighting of the different constituent groups. Where people have been asked to choose and rank 10 questions, the most straightforward approach is to apply a reverse scoring system to each submission:

In order to ensure equal influence of all stakeholder groups, regardless of actual numbers of responses, totals for both patients/carers and clinicians should be recorded, ranking the questions and giving them a score based on their position in each groups' list. It is those position scores that are then combined, to work out the final interim prioritised list. This means that where a low number of patients, for example, has responded compared with clinicians, their scores will still be given equal weighting, in line with the JLA's principle of equal involvement of patients and clinicians.

However, there have been cases of such disparity between the patient/carer priorities and those of the clinicians that the combined scoring has led to questions that are important to one group being left out of the shortlist for the workshop. In this situation the Steering Group should consider whether an alternative working should be applied. For example, the Adult Social Work PSP created a shortlist for the workshop based on including the separate Top 10s for each of its three stakeholder groups which, taking into account the overlaps, led to 21 questions being taken to the priority setting workshop.

The JLA Adviser will work with the Steering Group to agree how many of the prioritised indicative questions to take to the workshop. Generally, this will be around 25 questions for an in-person workshop (please see Chapter 8 for adaptations made for the online format). The JLA advises a maximum of 30, otherwise the workshop process can become unmanageable, with too little time available to be able to meaningfully discuss every question. However, for some groups it will be more appropriate to have a smaller number nearer to 20. It is important to consider the number of people who will be at the workshop and the ease with which they are likely to be able to review large numbers of questions on the day. The nature and needs of the participants should also be considered, as should the topic area. Narrower topics, such as single diagnostic areas, may mean participants have a shared understanding, experience and vocabulary, the impact of which could be a more easily established group dynamic and a faster-flowing discourse. Broader topics may need more time to be understood and discussed, and therefore bringing fewer questions to the workshop may be sensible. There may be an obvious place in the list of questions to draw the line, for example, where scores drop off rapidly. There may be a reason for including questions that are on the margins, for example, a question may be about an important aspect of the condition or may have been particularly important to one group of respondents or was of clear concern to a group who might be less likely to engage with the process.

The shortlist of questions should be sorted into random order, i.e. not the ranked order, and each question assigned a letter of the alphabet as a reference.

Here are some examples of the numbers of questions that PSPs have taken to the priority setting workshop: