Performance Measurement in the NHS

Mr. Paul Burstow (Sutton and Cheam) (LD): This is proving an interesting and useful debate. In his response, the Secretary of State answered some of the questions, but gave rise to several others that I hope that the Minister will be able to address. Not long ago, on another Conservative Opposition day, we debated the target-setting culture in the national health service and the way in which it provokes changes in behaviour in the NHS that are not always those that are intended-indeed, poor target setting can be corrosive of NHS morale and lead to perverse outcomes.

During that debate, we focused on the way in which targets and performance indicators can distort clinical priorities and lead to unintended consequences; today, we are focusing on the important expose" in the Health Service Journal and the information that it helpfully brought into the public domain. I hope that the Secretary of State will agree, even at this late stage, that that process should go further to ensure that we have all the information about the star ratings process and the build-up to its publication in 2002. A few bits have been teased out today, but there is still more to come.

There is a fine line between what one might call the fine-tuning, data checking and reality checking of performance indicators and star ratings and the fiddling of figures. On the basis of the evidence that was published in the Health Service Journal before Christmas, one could conclude that that line has been crossed. I want to return to the letters that the Secretary of State mentioned, because they raise further questions.

The basis on which the 2002 star ratings were calculated was changed-that is accepted. The Secretary of State tells us that that is part of an ongoing, iterative process. They were changed at the last minute, and the change had a real effect. Nine trusts went up from two to three stars and six went down from three to two stars. As a consequence, those six trusts were, in effect, robbed of up to £1 million each for service improvements that would have been available to them had they been three-star trusts. In addition, they were denied a range of modest, but nevertheless welcome, freedoms and flexibilities that are part of the star rating system.

The Health Service Journal documents the e-mail exchanges between the office of the former Secretary of State and the head of the performance development unit in the Department, Mr. Wilmore, who warned on 12 July 2002-this has been quoted before, but it is important to my argument:

Unfortunately even if time allowed it, further revisions to the methodology to promote these trusts would inevitably lead to other individual results we had not expected, as well as making the scoring system more complicated."

The Secretary of State has laboured the point that so far the debate has focused on South Durham. That will not be the thrust of my argument, because the key question is not whether the methodology was changed to fix it for one trust, but whether it was changed inappropriately, at the last minute, with unforeseen consequences for a number of trusts that hitherto would have had three stars and all the benefits that flow from that. That is why Basildon and Thurrock general hospital was one of those mentioned in the official's report to the then Secretary of State. He said:

Adjustments to the methodology would have to be severe to move the trust to three stars and would inevitably demote other high profile trusts in the process."

In his response to the hon. Member for South Suffolk (Mr. Yeo), the Secretary of State said, effectively, that the main changes to the methodology concerned catering services and information management. Yet the paper that was sent to the Secretary of State at the time, which was the basis of the article in the Health Service Journal, says that the reason why Basildon and Thurrock general hospital was not going to secure a three-star rating was because of the patient-focused element in the methodology, and specifically because of the results from the patient survey. I wonder, therefore, how that methodology was changed to reflect a high three-star rating for that trust in the final run of the star rating calculations for 2002. The Secretary of State has not yet dealt with that.

Dr. John Reid: I hope that this will be helpful to the hon. Gentleman. As regards his suggestion that things were done late, they were done continually; some were done late. My hon. Friend the Minister will respond to that. The key word in his argument is inappropriate", because we believed then, and believe now, that the changes were appropriate, and such changes continue. It is important to remember that changing one or two indicators will not affect the majority of trust results, but it will inevitably affect some-not only one-because the star rating system uses the relative position of trusts against a wide range of balanced scorecard indicators to help to distinguish between two-star and three-star trusts. That means that a trust's position can be improved or worsened by changes to the indicator scores of other trusts. That is the essence of the relativity effect whereby changing one indicator can affect a series of trusts, which, in turn, affects other trusts because of their relative performances.

Mr. Burstow: I am grateful to the Secretary of State. That implies that the removal from the methodology of catering and information management did constitute a severe change in terms of that official's advice and warning to the then Secretary of State on 12 July 2002. I will look at what the Secretary of State says, because I accept that he is trying to be helpful.

Mr. Wilmore asked the Secretary of State's office for comments by 15 July, and received an e-mail that stated that the

Secretary of State would also identify South Durham as a high profile Trust"-

we heard earlier that the Secretary of State's office had used the words high-profile trust"-

given it serves the Prime Minister's constituency. Why has it fallen from three stars last year to two stars?"

Mr. Paul Tyler (North Cornwall) (LD): That further suggests that there are some low-profile trusts. I hope that my hon. Friend agrees that there should be no such trusts. A quite different set of categories, which appears to be political rather than based on any clinical need, is being introduced into the discussion.

Mr. Burstow: My hon. Friend makes an important point. I hope that the Minister will accept that high-profile trusts" is an unfortunate use of terminology that should not have crept even into an e-mail that was not expected to become as widely disseminated as it has.

In response to the e-mail, Mr. Wilmore stated:

Alterations to the methodology have been made, resulting in the changes to individual trusts that were requested. This makes the scoring methodology more difficult to explain and less transparent."

He continued:

An explanation of the revised methodology has yet to be written up."

Will the Secretary of State put into the public domain the detailed write-up of the methodology that followed the further recalculation that led to that exchange of e-mails? That would be a useful document in terms of understanding what the Secretary of State has said to me and to the hon. Member for South Suffolk.

Mr. Wilmore then notes that Basildon and Thurrock and South Durham

now receive a three-star rating".

His paper offers no further explanation of why South Durham was originally downgraded from three to two stars and does not attempt to answer the Secretary of State's perfectly legitimate question about why that change occurred. It will be interesting to have the answer so that we can be clear about the way in which the methodology and other processes that contribute to a star rating were affected to produce the outcome in 2002.

We also know that the exchange of e-mails was copied to the Prime Minister's health adviser and we are told that the relevant e-mail was not acted upon. Does that mean that the e-mail was not read? Given that the Prime Minister's health adviser apparently routinely receives copies of e-mails between Health Ministers' private offices and the Department, what does he do with them? I am told that he is dynamic, proactive and hands on. The nature of e-mail drives people to respond and I am therefore puzzled that the health adviser did not respond and possibly chose to open it and do no more.

A question, which I hope the Minister who responds to the debate can answer, about the Prime Minister's knowledge of events has not been asked. It is not whether his adviser, on the basis of the e-mail exchanges, notified him of the specific anxiety about the high-profile trust, but when he became aware of the South Durham star rating. It would useful to have that specific piece of chronology.

Since the publication of the e-mail trail, the official line has comprised two elements. First, what happened is part of the normal process of signing off the star ratings, which would be subject to adjustments and corrections every year as part of the iterative process. Secondly, the responsibility for finalising and publishing star ratings is now that of the Commission for Healthcare Audit and Inspection-as if that makes okay anything that was inappropriate in the past.

The unofficial line that the right hon. Member for Darlington (Mr. Milburn) has taken is that it is all a load of tosh. I hope that that is a parliamentary term and that I can therefore use it. How can it be tosh for a senior official to warn the Secretary of State that changes to the method of calculating the 2002 star ratings made them more difficult to explain and less transparent?

I want to ask a few questions of the Minister who will reply to the debate. First, why were the changes made so late in the day, after they had been reality checked? That is especially relevant to the trusts that were part of the system in 2002. Secondly, why were the warnings of the head of the performance development unit about the difficulties of making the changes at such a late stage ignored? Thirdly, what changes to the methodology led to such a big shift in the star ratings but did not affect the trusts with zero star ratings? Again, that relates to the detailed write-up of the methodology that I requested earlier. I hope that the Secretary of State or the Minister who replies to the debate can provide that.

The handling of the 2002 star ratings must give rise to questions about that of the 2003 ratings. In 2003, for the first time, the Commission for Health Improvement was responsible for publishing them, but it would be strange if the Department had not maintained a close interest in the way in which CHI was taking the work forward. As well as publishing all the Department's material that was relevant to the 2002 star ratings, will the Secretary of State undertake to publish all papers and e-mails about the methodology and weightings for each element of the 2003 star ratings, in the interest of restoring public confidence in the star ratings system, which he clearly wishes to achieve?

The way in which the 2002 star ratings were handled raises wider questions about performance monitoring in the NHS and the way in which data are collected and published. In November last year, the Nuffield Trust published its mid-term review of the Government's progress in delivering improvement in the NHS. The report was positive-I am sure that that pleased Ministers. However, there was a sting in the tail. One of the report's authors, Professor Sheila Leatherman said:

There are significant data and analytic weaknesses in the NHS which mean carrying out a comprehensive, robust, definitive, transparent and defensible assessment is impossible.

The unrelenting and distracting problem of inconsistent and highly contested data throws the whole of the quality agenda into a confusing fray."

Until the data are collected, audited and published independently of the Government, how can the public have confidence that their money is being spent wisely?

In its report, Performance Indicators: Good, Bad and Ugly", the Royal Statistical Society calls for performance indicators to be accorded the same status as national statistics. In other words, the process should be clearly independent and at arm's length from the Government, to an even greater extent than the new Commission for Healthcare Audit and Inspection allows.

The report detailed several pitfalls that we discussed in the Chamber when we previously considered target setting, not least that performance measurement can create unintended consequences and lead to manipulation of data, gaming or fraud by service providers. Indeed, the Audit Commission and the National Audit Office found that in reports that they published last year.

Performance measurement changes behaviour. The Royal Statistical Society stated:

Behaviour change is a factor because no performance measurement scheme can be viewed in isolation from the incentives-designed or accidental-that exist alongside it. Designed incentives often take the form of targets, and set of consequences associated with performance. If the assessment of management functions in the NHS depends centrally on whether explicit waiting time targets are secured, then this can affect such things as patient handling among health care professionals."

In other words, what gets measured gets done.

The Secretary of State recently told the Select Committee on Health that the Department was reviewing its data collection.

Mr. Stephen McCabe (Birmingham, Hall Green) (Lab): I am following the hon. Gentleman's argument with interest. What does he want to have done but not measured? Can he give some examples?

Mr. Burstow: I am about to deal with the review of data collection, about which the Secretary of State told the Health Committee. [Interruption.] I shall answer the hon. Gentleman's point in my own way and in due course, if he does not mind.

The Secretary of State told the Health Committee that the Department was reviewing its data collection with a view to rationalising what it collects. The NHS confederation has expressed concern for some time about the burden of reporting requirements on the NHS. In December, after a year of consultation and discussions, the Confederation published its report, Smarter Reporting". The report found that more than half the information requests from the Department were perceived not to be useful for managing NHS trusts, either because it asked for duplicate information or because the data were of questionable value.

The survey also found that a quarter of returns that the Department required were wholly or partly duplications. Much could therefore be stripped out of data collection without materially affecting the value of the data that the Department is currently gathering. Simply cutting back the duplication would make a difference. Rationalising data collection helps to improve the quality of the data that are being collected.

In the conclusion to its report, the NHS Confederation warns that the exercise that the Department is currently undertaking entails a risk of losing the value of data collection in the existing system. I hope that the Secretary of State can assure us through the Minister who responds that an extensive dialogue is going on with the NHS Confederation and others to ensure that we get the best data collection and fill the gaps when that will add something to our understanding of service development, policy development and performance in the NHS.

Liberal Democrat Members will support the Opposition motion because we believe that there is a need to bring into the public domain information, which is not there, about what happened in 2002. We are critical of the way in which the star rating and performance management system has been rolled out in the past few years. It is clear from the events that surrounded the 2002 star rating that the process and reporting have damaged the credibility of the star ratings system. Indeed, they have dealt it a fatal blow. How can the public have confidence in the current star ratings system? What messages have been sent to NHS staff, especially those in trusts that lost their third star because of the recalculations? They were told one minute that they were an excellent trust, then suddenly that they were considered middle rating. What message does that convey?

We will support the motion. The NHS is improving, but we need a reliable system that enables us to know that it is improving. That currently does not exist. Until it does, we cannot support the star ratings system.

To read the debate in full take the following link:

http://pubs1.tso.parliament.uk/pa/cm200304/cmhansrd/cm040107/debtext/40107-04.htm#40