Up to Standard - the NHS Performance Assessment Framework

A report of an IFM Healthcare Study Day held on 15 June 2000 at the University of York

by Julie Glanville, NHS Centre for Reviews and Dissemination, University of York

Introduction
Merlin Wilce
Giles Wilmore
John Sargent
Elaine Rodger
Russell Mannion
The workshops
References

Introduction

In April 1999 the NHS Performance Assessment Framework (PAF) was published, providing a structure for assessing achievements across the NHS in terms of efficiency and performance.  It focuses on the following indicators:

• health gain
• fair access
• effective delivery of services
• efficient delivery of services
• health outcomes
• patient / carer experience

Clearly, implementing the framework so as to achieve better performance is a complex undertaking and the many issues involved were explored during the Study Day by a range of speakers presenting from very different perspectives.

Back to the top

Merlin Wilce

The scene was set by the Chair, Merlin Wilce of Leeds Metropolitan University, who noted the Government's emphasis on performance, as signalled by the use of 'performance' 88 times in the white paper, The New NHS (1).

Giles Wilmore

Giles Wilmore of the NHS Executive in Leeds then discussed the PAF from the point of view of the NHS Executive.  He suggested that the PAF can be seen within a wider context of other quality initiatives such as the work of NICE, the National Service Frameworks, clinical governance and CHI.  The PAF is important for benchmarking and for spreading good practice, as well as for improving and accounting for performance.  The PAF also emphasizes the involvement of the public and the importance of taking account of patients' and carers' experiences of health care.

Giles noted that implicit within the PAF is an assumption that performance measurement needs to be broadened out to look at performance rather than just efficiency.  The latter, he felt, had distorted service priorities and rewarded volume rather than quality.  Aggregating the indicators would present a meaningful whole system measure of performance that could be followed over time.  Performance on the aggregated indicators would attract rewards and local flexibility would be facilitated by allowing local decisions on where performance could be most improved.  There would be minimum requirements against key indicators and areas of weakness would be monitored.

Giles pointed out that whereas the concepts of performance assessment have been well received within the health service, the robustness of the high-level performance indicators has been questioned.  Questions have also been voiced about the indicators' comprehensiveness, their applicability within the planning process, and what weightings should be applied to them (do all domains have equal weight?).  Issues also surround methods of cascading indicators from health authorities to primary care groups and trusts.  However, the most important question seems to be about the quality and availability of supporting data.  Indicators need to avoid creating disincentives, to be robust and responsive, usable and timely and give a sense of public accountability.

Back to the top

John Sargent

John Sargent, on secondment with the NHS Executive North West from Trafford Health Care NHS Trust, explored the practical issues of implementing the PAF.  He gave examples of specific projects and the lessons he had learned.  He noted that the complex weightings and changes in the performance indicators are difficult to explain to the public.  Performance to date has been based on finished consultant episodes, outpatient visits, inputs to the NHS and available financial resources.  When implementing the PAF issues arose about what successful implementation would look like and the nature of a performance culture.  He felt it was important not to drift into processes and inputs, but to define milestones based on available evidence.  For example, if we are seeking to improve coronary heart disease deaths by 10%, how are we going to do it, what is the long-term approach and what is the long-term evidence to inform us about any changes?  An example given of one way to measure improvement was to follow up families of coronary heart disease patients and look at lifestyle changes.

The issues involved in creating a performance culture are many.  What are the determinants of good organizational performance?  We can look at management evidence, we can focus on good leadership and demand clear organizational vision and shared objectives.  We also want strategies that fit with the external environment.  We need good information systems and we need to understand that good performance is multi-dimensional.  Because of this we need to see the Performance Assessment Framework in context.  There are many different areas of policy development on which it impinges and that will make it successful.  John used a pyramid analogy with the National Service Frameworks at the apex of the pyramid, the PAF in the middle layer of the pyramid and clinical governance on the bottom of the pyramid.  The inter-relationships and interdependency of these initiatives is very important and can be helpful to illustrate the relationships of the quality initiatives.  Successful implementation needs to involve the right people and be aware of their personal time and commitment issues.  There is a trade-off between national consistency and measuring improvement locally and it is important when implementing locally not to get hung-up on high quality national data.  A pragmatic approach to data collection seems essential and it is possible to use local data that people have already been collecting so that, at least locally, data collection will be consistent.

Back to the top

Elaine Rodger

Elaine Rodger of Sunderland North Primary Care Group followed up with the primary care perspective on the PAF.  Whilst the PAF is relevant to primary care, she suggested that there need to be more indicators worked up for this sector, and at the PCG Trust level.  The initiative needs to be underpinned by the development of a performance management culture.  Elaine discussed six aspects of the national primary care performance indicators: health improvement, fair access to health care, effective delivery, efficiency, health outcomes and the patient care experience.  She then explored ways in which primary care could deliver on each of these aspects.  In terms of health improvement there are many targets which could be developed.  In terms of fair access, the number of GPs per head of the population and their geographical location could be an issue.

Immunisation and 'flu vaccination were typical aspects of effective delivery that primary care could focus on.  In terms of efficiency, generic prescribing and overall financial targets could be set.  From the viewpoint of Sunderland Health Authority, targets for the patient-carer experience and for health outcomes had not yet been devised.  Elaine noted the information sources available to people in primary care: the Exeter System (based on payments to GMS practitioners); E/PACT (prescribing data); hospital data (to indicate the outcomes of primary care); and practice level data.  With the latter there are many systems with little consistency and obtaining consistent and comparable data from these various systems will require a lot of development effort.  There is still a lack of hard data on community programmes.

Elaine moved on to describe the NHS bench-marking club project "Primary Care Indicators for Health Authorities and PCGs/LHGs".  A preliminary report on the project was published in September 1999 in the Health Service Journal (2).  The project's objective was to act as a tool kit to assess the state of primary care across participating health authorities, PCGs and LHGs in Scotland.  Twenty health authority club members were submitting data which represents 101 PCGs and LHGs.  The bench-marking club collects data on variables such as average practice size, number of single-handed GPs as a percentage of total practices, percentage of practices with female GP sessions and so on, and then compares how different health authorities perform to identify outliers.

Back to the top

Russell Mannion

The final speaker for the morning session was Russell Mannion from the Centre for Health Economics at the University of York.  Russell presented a research perspective on the PFA.  He questioned what the Performance Assessment Framework was measuring and how, and concluded that it was measuring processes and outcomes.  He was concerned that performance analysis should be seen within the context of  local circumstances which might explain difference between performances of different organisations.  He questioned how analysis of the performance assessment feedback would be fed into policy development.

He identified many factors which might undermine the information we receive on performance.  For example, how much resource does a hospital have to allocate to data collection and how widespread are coding errors? It is unclear whether all organisations will be measuring the same things and that classifications of severity, for example, will be consistent across trusts.  Performance assessment can be either weakened or strengthened by the incentive structures that flow from it.  The unintended consequences of performance assessment were then entertainingly described:

• tunnel vision, where a focus on things that have been measured means that other important areas are ignored
• measure fixation, where success is as measured rather than as intended such as having 'hello nurses' in A&E who ensure that people have very quick contact with a nurse but then still wait for treatment
• suboptimisation, where different incentives are given to different people and meeting one target conflicts with other targets
• myopia, where short term issues are followed and the long term perspective, such as preventive health, is forgotten
• complacency; if your organisation is not an outlier in the performance assessment process then you may feel you don't need to improve
• misrepresentation, in the form of outright fraud and manufactured figures
• misinterpretation; type I and type II errors
• gaming, where behaviour is altered to gain strategic advantage.  For example, an organisation may not try very hard to improve performance in the first few years (although it could improve) but choose to improve later when it has lower barriers to success
• falsification, where the performance assessment indicator stifles innovation.

1. Parliament.  The new NHS : modern, dependable.  London : Stationery Office, 1997

2. Rodger, E., Watkins, S.  Variations enigma.  Health Service Journal, 1999, 109 (5670) : 20-23

Back to the top

Back to Inform 12 (1)

Last updated February 2001.