Reflections on Rapid Priority Setting and the James Lind Alliance method

Introduction

The James Lind Alliance (JLA) method is a way to set priorities for research. With over 150 JLA Priority Setting Partnerships (PSPs) having completed the process, it is a particularly well-documented approach that has been undertaken internationally. In general, the JLA priority setting process takes around 12-18 months to complete. The JLA’s experience is that it takes time to conduct an inclusive, responsive and consultative process, in which a diverse range of patients and clinicians are actively involved, strategically and operationally.

In recent years the JLA has fielded several questions about the feasibility of speeding up its method. This paper describes three alternative priority setting approaches undertaken in a shorter timeframe than a typical JLA PSP, compares them with the full JLA PSP approach, and assesses the extent to which JLA principles can be met with these approaches.

The Covid-19 pandemic exposed a need for fast but robust decision-making about priorities for research. Health and Care Research Wales (Welsh Government) and Social Care Wales (a Welsh Government Sponsored Body) sought to work in association with the JLA to develop robust and transparent ways to set priorities using a structured approach that involved practitioners and service users but which would be more able to meet their circumstances and short timeframes than a typical full JLA PSP.

This report describes the methods used in the two priority setting exercises and examines how the approaches diverge from the JLA PSP approach and what learning there is for JLA methodology. It also features a third rapid priority setting exercise, carried out independently of the JLA, but which drew heavily on JLA methods and published its findings, to draw out further learning and insight into the implications of making the priority setting process rapid.

It should be noted that for the purpose of this paper, the JLA has not defined what is meant by rapid, other than to say that the assumption is that it would take less than a year from start to finish.

Examples of priority setting exercises with rapid approaches

The three projects that will be examined are:

1. Social Care Prioritisation in Wales – care and support of older people (in association with JLA)
2. Social Care Prioritisation in Wales – family support services (in association with JLA)
3. Prioritisation of UK adult social care and social work innovations suitable for evaluation (not in association with JLA)

The following table describes the key stages of the standard JLA PSP process, and how the three rapid priority setting projects employed, adapted or removed those.

These are described in more detail below.

1. Social Care Prioritisation in Wales – care and support of older people

This project was initiated in late April 2020 by Health and Care Research Wales and Social Care Wales. Set up as a pilot, the goal was to identify a process to consult people with lived experience and practitioners in setting research priorities. The aim was to develop an approach that would be repeatable, rapid and resource efficient. The topic agreed was the social care of older people in Wales.

• A working group was set up to include members with relevant experience and networks, with a view to meeting once a month on average. This only included professionals and no lay members. The project was supported by the JLA Senior Adviser and an experienced Information Specialist.
• The survey was designed to gather ideas on research priorities from older people, their carers and social care practitioners. This was based on the JLA Adult Social Work PSP approach (which produced separate surveys for service users and practitioners). It was piloted, and Welsh language, printable and easy read versions were produced. A promotion guide was also produced and distributed to around 100 organisations. This generated 294 responses, including 80 from older people. A Zoom discussion group was also facilitated to generate input from under-represented diverse groups. The survey ran for five weeks.
• The Information Specialist developed overarching questions with involvement of the Working Group, creating a long list of 42 indicative research questions.
• An interim priority setting survey ran for four weeks, but only received 89 responses, of which just seven were from older people. The shortlisting of questions for the workshop still took account of both groups equally.
• An online workshop to determine the top 10 priorities was facilitated over two half-days, bringing together 20 older people, carers and social care practitioners in a range of settings, experience and job roles.

2. Social Care Prioritisation in Wales – family support services

In March 2022 Social Care Wales and Health and Care Research Wales launched a project to identify research priorities relating to care and support for families and children. The aim was to develop a social care research agenda to explore how Strengths-Based Approaches may be able to help solve issues that are priorities for parents, families and young people who are in contact with family support services and the practitioners who support them. The project was coordinated by the same team as the previous project looking at research priorities for the care and support of older people, and modifications were made based on the learning from that experience.

• A working group was established which included representatives of policy, research and the voluntary sector, as well as practitioners and parents who had been in contact with family support services.
• The first survey was open for five weeks and generated 72 responses, of which 19 were parents.
• After analysis of the data, 42 questions were put forward for interim prioritisation. The survey was live for five weeks. In addition, two focus groups were carried out with young people who had been in contact with family support services. This stage resulted in the shortlisting of 16 questions.
• An online priority setting workshop determined the final top 10 priorities. There were 15 participants, all of which were practitioners, although two disclosed that they also had lived experience of using family support services.

Both Welsh projects described above were conducted under the banner of the JLA Lab, a conceptual space where methodological experimentation can occur, without compromising the integrity of the JLA’s established PSP model.

3. BRACE/RSET – prioritisation of adult social care and social work innovations for evaluation

The BRACE/RSET project was a collaboration between the National Institute for Health and Care Research (NIHR) rapid evaluation centres, BRACE (The Birmingham, RAND and Cambridge Evaluation Centre) and RSET (The Rapid Service Evaluation Team). It was initiated by NIHR in July 2019, with the aim of identifying, by November 2019, a prioritised shortlist of UK adult social care and social work innovations suitable for evaluation by the two rapid evaluation teams, and to inform the NIHR Health and Social Care Delivery Research (HSDR) Programme and other NIHR programmes in relation to their commissioning priorities for adult social care and social work. The team chose to adapt the JLA PSP approach because of its established method for involving stakeholders, and because of the opportunity to draw on recent learning from the JLA Adult Social Work PSP. Here is the project report (.PDF) and here is the paper.

The timeframe did not allow for the recruitment and convening of a stakeholder-led Steering Group, so the process was led by the project team. The JLA’s Senior Adviser at the time, Katherine Cowan, was a member of this group, and advised on method, but not in her JLA capacity, so the project was not affiliated with the JLA. The following four steps were followed:

• Identification of innovations through a survey distributed to a network of 182 relevant individuals and organisations. A four-week deadline was given. No individual service users or carers took part in this.
• Development of criteria for shortlisting by the project team.
• Grouping and sifting innovations to generate a workshop shortlist, done by the project team.
• Prioritisation of innovations in a JLA-style workshop setting involving 23 people who use adult social care services, practitioners, academics/researchers, commissioners/policy makers and carers.

Drawing on these three examples, the section below summarises the factors which facilitated a rapid approach, and the compromises that resulted from being rapid compared with a full JLA PSP approach.

Facilitators of a rapid approach

Drawing on existing materials

Where a similarly themed JLA PSP had already completed its process, having existing materials to draw on expedited the process. For example, the first Welsh social care project drew heavily on the JLA Adult Social Work PSP’s survey materials, which had originally been developed and refined over several months with direct input from people with lived experience and practitioners, both on the Steering Group and in an external pilot. It is unlikely that the rapid timescale could have been achieved for this challenging topic without that existing experience to draw on.

Focusing on a small population

Concentrating on a smaller, well-networked geographical area helped speed up the process. For both Welsh social care projects, strong networks and existing working relationships enabled a faster and more effective dissemination of the survey and project materials than if the process had been UK-wide.

Access to JLA-specific experience and expertise

Input from experienced Information Specialists expedited the process for both Welsh projects too. Their existing knowledge of the JLA, having worked previously on PSPs, meant that time was not required to induct them or to establish how they would work inclusively with the project team and Steering Groups. For all projects, strong, dedicated project management was key to remaining on task and to time.

Access to engaged consumer networks

In terms of seeking consumer input, for the family support services project, parents were able to be rapidly recruited to the working group because they were already part of an existing consumer panel and were known to members of the project team.

The social care innovations example was the most rapid, partly because they did not involve consumers until the final workshop stage and did not need to plan in time to consult with a multi-stakeholder steering group. Of course, this is one reason why this would not have been able to be a formal JLA-endorsed exercise.

Comparisons with the JLA PSP approach

Steering Group engagement

A rapid timeframe will impact on the number of opportunities there are to engage with a Steering Group. A JLA PSP Steering Group usually meets monthly and may, for example, meet at least two or three times while the first survey is live. If a survey is only live for a short time, it may not be possible to convene the group during that time to discuss responses to date and strategies for improving reach.

Being able to short-cut the production of materials, such as surveys, by drawing on previous PSP outputs, may be advantageous in terms of time. However, this comes with a risk. Not only is it possible that the materials are not perfectly transferable to the topic, but it may also be the case that Steering Group members are less engaged and have less of a sense of ownership of the project if they are less involved in the development of materials. In a standard JLA PSP, Steering Groups look to other PSP’s materials for inspiration, but take time to tailor their own to accommodate the needs of their specific audiences. Members’ investment in those materials tends to increase their willingness to promote them and remain engaged in the process as it progresses.

Similarly, securing Steering Group input on the question development stages is limited by being rapid. In a PSP, the Steering Group plays a crucial role in reviewing and critiquing the Information Specialist’s work, ensuring that the survey data is being appropriately interpreted and that the summary questions are being worded accessibly for all audiences. They may spend several meetings, as well as time outside meetings, carrying out this work and coming to a consensus on the long list of questions for prioritisation. Again, this scrutiny from the different stakeholder groups is key to the transparent production of good quality questions, and to the Steering Group’s ownership of the PSP’s outputs. This may be compromised by being rapid.

Working exclusively online, as both Welsh projects did, was in some ways more efficient than convening multiple meetings requiring travel. Online working may allow the involvement of stakeholders who may not have been able to join an in-person meeting. However, not meeting in person may affect a way that a Steering Group gels when working rapidly. While many JLA PSPs have now completed a fully virtual process, they have avoided overloading their Steering Groups by allowing time to stagger the various tasks and ensuring there is time for team building within meetings. Working rapidly, under pressure and constantly fielding deadlines impedes this.

Stakeholder consultation

Rapid priority setting exercises may not achieve the number of survey responses or the diversity or respondent types that a full PSP would typically expect to see, which may impact on the range of submissions they receive. With a tighter timeframe, there is less opportunity to tailor communications or build the relationships required to connect with diverse communities. For the older people’s project, while an additional virtual discussion group generated input from older people of African and Caribbean heritage, there was insufficient time to run other discussions with under-represented groups or further promote the survey to them. Similarly with the family support project, excellent input was gained through focus groups with young people at the interim prioritisation stage, but these were organised by a member of the Steering Group, whose colleague facilitated them, and the timeframe was significantly extended to accommodate their availability.

Flexibility

Priority setting exercises are ideally able to respond to unanticipated challenges and opportunities by building flexibility into their timeframe and not rushing their consultation periods. This might include extending a survey period to target an under-represented group of respondents. A rapid timescale does not allow for this. The time of year may also cause issues when being rapid. One project found their interim prioritisation survey going live during August, with no option to extend the timeframe. A low response resulted as it is likely that many of the people who had cascaded the first survey were away during that time and did not see the email request. There was no opportunity to redress this.

Communication

Communication of the different stages and opportunities for engagement in a PSP may also be impacted by being rapid. In one of the projects, running the interim prioritisation survey so soon after the first survey caused some confusion, with some people mistaking it for a re-dissemination of the original survey, and therefore failing to fill it out or promote it, because they thought they had already done it.

Evidence check

JLA PSPs carry out a relatively high-level evidence check before putting questions forward for prioritisation. It is limited and pragmatic, but it is carried out according to agreed criteria set by the Steering Group. It will require time on the part of the Information Specialist to carry out searches and document findings transparently. Being rapid will also impact on the time available to carry out an evidence check, which may lead to evidence being missed depending on methods used. In the social care innovations project, for example, the short period of time available for the horizon scanning meant that certain innovations may have been missed from the final shortlist. Similarly, some innovations that were included may have already been, or currently were being evaluated. In future, new methods, for example, using automation, may improve the opportunities for finding evidence accurately and quickly.

Broad, complex topics

The breadth and complexity of the topic, and the nature of the stakeholder groups have implications for being rapid, especially for the involvement of patients, carers and people who use services, particularly those who are marginalised or vulnerable. In the social care innovations project, achieving buy-in was hard to do rapidly for such a broad topic, without a set audience or narrowly defined patient community. Securing engagement may be easier for rapid priority setting on narrower topics. Family support services in Wales was a challenging topic to conceptualise, plus people who use family support services are likely to be experiencing personal crisis, potential housing vulnerability and chaotic support needs. Involving them in a priority setting exercise is not impossible but needs time and care. Similarly, in the same project, it was challenging to engage practitioners in a service under extreme pressure, including dealing with stressful and complex workloads, resourcing challenges and staff shortages. More broadly, it can be useful to draw public, patient and service user representatives from existing networks, but they will still need support and encouragement to engage, particularly if they are individuals who balance other demands and have limited time to dedicate to the project. Building these relationships and ensuring that everyone understands the complexities of the project and feels genuinely involved takes time.

Culture change

On a higher level, JLA PSPs are an opportunity to foster a culture change, where true collaboration between patients and clinicians occurs. They gain a strong sense of their mutual stake in the future of research. PSPs aim to change research culture and the nature of funding decisions, and have a transformative effect on policy, organisations and people, as described in More Than a Top 10 (.PDF). Getting to this point necessitates a challenging process of partnership and collaboration, to understand different perspectives, develop trust and respect, and come to a consensus. It is doubtful that this sustained, embedded culture change is achievable within the constraints of a rapid version of the JLA method. It is probable that those involved in a rapid exercise may not find the experience as rewarding or feel as invested in the outputs, compared with those who are involved in a full JLA PSP.

Conclusion and implications for the JLA PSP approach

How a rapid priority setting exercise may impact on JLA’s core principles:

• A rapid exercise is likely to mean that the level of scrutiny and therefore transparency that a Steering Group can provides to their process and outputs is more limited than in a typical JLA PSP.
• Being rapid requires greater input from research and project teams to carry out the work quickly, and impacts on the equal involvement of patients, carers and clinicians in that process.
• Being rapid means that an evidence check is likely to be more compromised (or eliminated) compared with a JLA PSP, impacting on the project’s ability to draw on and contribute to the evidence base.

The principle of excluding those with a commercial interest and researchers with no current clinical practice is not necessarily impacted by being rapid.

Of particular note is the effect of rapidity on the quality and effectiveness of the patient involvement and engagement aspect of the JLA method which is a fundamental feature of a JLA PSP. The degree to which this aspect is compromised in a rapid process is likely to prevent a rapid application of the JLA method from being described as a JLA PSP.

It is important for the JLA to protect the integrity of its method. The individuals who instigate PSPs make a significant commitment when they undertake the JLA process, both in terms of resource and reputation. They do this to generate a set of priorities that are robust and meaningful, to have an impact on their broad stakeholder communities. Their PSPs can demonstrate that they have upheld the JLA’s principles, have followed a documented method and as a result have genuinely had input from patients and clinicians at all stages. This enhances the quality of their outputs. It is not in the JLA’s interest to compromise the integrity of what they achieve by endorsing rapid PSPs and the compromises therein, and by giving them equal weighting to standard PSPs.

The JLA recognises the compelling arguments for rapid priority setting. Setting priorities quickly involving the end users of research in some way is possible. All three examples in this paper selectively replicated some of the JLA method’s characteristics and components. However, for those who do choose to undertake priority setting using a rapid approach, it is important to be aware of the compromises, and to be open about what these will mean and how they will impact.

Key learning points for a rapid approach:

• Time of year is an important consideration when planning a survey period, particularly when working to a tight timescale which does not allow for an extension of the time that the survey is open. Avoiding summer holidays and any other times when many people are likely to be away is key.
• Be clear about communications, especially when working to a compressed timescale which may lead to the initial consultation survey and the second shortlisting survey appearing within a short space of time. It may be easy for audiences to get confused about the need to complete both of them and to assume that they are the same.
• Concentrating on a smaller, already well-networked geographical area helped to speed up the process in the two Welsh examples, as did access to an existing consumer panel for one of them.
• Having strong, dedicated project management is important, ideally with the support of people experienced in priority setting.
• A rapid timeframe may limit the engagement of the Steering Group in discussing the consultation responses to date and strategies for improving reach.
• Basing materials too closely on existing examples without engaging the Steering Group fully in tailoring them may lead to them feeling less engaged in the promotion of the work.
• Scrutiny of the summary research questions by the Steering Group is key, and time for this may be limited in a rapid priority setting exercise.
• Consider the positives and drawbacks of working online, taking into account the stakeholders that the exercise needs to hear from.
• Successful stakeholder involvement in rapid priority setting exercises may be influenced by the breadth and complexities of the topic.
• Rapid priority setting exercises may not build a sense of partnership and collaboration in the same way that in-depth involvement in a longer process may lead to.