Many executives contributing to this paper noted the constant tension between how they felt their organization should be held to account and how their health systems actually judge them. There was virtually unanimous concern over the increasing number of measures, most of which are felt to be largely irrelevant. Leaders acknowledge the rights of patients and payers to know the outcomes that matter to them, yet also feel that the incessant demands for information can actually hold back rather than stimulate transparency and accountability.
Any organization basing its clinical measurements on inadequate internal administrative data and external regulatory requirements – rather than on intermediate and final clinical, cost, and service outcomes built around specific clinical care processes – will fail in its attempts to manage care delivery. - Dr. Brent James, Chief Quality Officer, Intermountain Healthcare, US
These views were largely consistent across different healthcare organizations and geographies. “We have to report on well over 300 measures, a number that is rapidly expanding each year,” says the Mayo Clinic’s Mike Harper, referring to requests coming from different sources such as regulators, accreditation agencies and state departments of health. “We play the game, but the regulators and payers often do not coordinate their efforts nor focus on the things that we think represent ‘value.’ It takes a lot of manpower to cobble together the information.” This includes time spent working with on-site inspections and survey teams that aim to dive deeper into the data, to discover whether the organization is compliant with accreditation standards and other regulations.
Dr. David Rosser, Executive Medical Director of University Hospitals Birmingham, UK, commented on the effort that goes into reporting hundreds of measures to the clinical quality commissions, regulators, the General Medical Council, and local governing bodies: “The majority of these are measuring the wrong thing, such as mortality measures and the overall performance of hospitals. Yet overall mortality simply is far too limited a measure when the quality of hospital care is concerned.”
The leaders interviewed felt that, at best, half of the information they reported was meaningful – and some felt that none was of any use. In addition to the administrative costs, these tasks send out confusing and potentially demotivating messages to healthcare professionals, as Mary Jo Haddad of Toronto’s SickKids Hospital points out: “We’ve been working on measurement for a long time. Too often, policy-makers come up with new measures that do not have any relevance to us. In Ontario, for example, our new legislation ‘Excellent Care for All,’ requires six key measures, of which four do not apply to a children’s hospital. We have to measure some indicators, that are not relevant in our children’s hospital. I would rather report on the key indicators that are relevant to us, yet those won’t be taken into account. What message does that send to front line professionals, having to measure meaningless items and to report them? This may actually hurt what we are trying to do, by diminishing staff engagement and even undermining the board’s authority, which is perceived as micromanaging. The unintended consequences of such wrongly focused messages are huge.”
What message does that send to front line professionals, having to measure meaningless items and to report them? - Mary Jo Haddad, President and CEO, SickKids Hospital, Toronto
A further criticism is the low level of granularity of many measures, requiring reports, audits and inspections on outcomes of a particular treatment in great detail. Many providers feel that such data does not paint a picture of the overall value of care being delivered, a point emphasized by Haddad: “We are dwelling in all these overly detailed measures, while we should be focusing at the right level. I’d like the regulators to demand that we establish, for all our fields, the key outcome measures and linked targets and then publish the outcomes reliably and verifiably.”
“One UK hospital asked an expert statistician to determine the most important performance indicator for each of its services. The response was completely different to that which was being reported. When boards or regulators have too limited a focus on measures, the whole dashboard may be green while the house is on fire.” - Neil Thomas, Audit Partner, KPMG in the UK
Professionals and scientists are used to discussing those outcomes – known as ‘primary endpoints’ – that really matter to patients with a specific or multiple conditions.
These endpoints are an excellent starting point for measuring broader outcomes. For stroke care, for example, the status 90 days after the onset of stroke is seen as the ‘primary outcome measure,’ on the road to optimum recovery. For rheumatoid arthritis patients, the most important intermediate goal – a strong predictor of long-term outcomes – is controlling the disease activity, as measured by the disease activity score, achievable through a few questions and one blood test.1,2
1 Contemporary outcome measures in acute stroke research: choice of primary outcome measure, European Stroke Organization Outcomes Working Group, Lees, K. R., et al. Stroke, 43(4), 1163-70, 2012.
2 Outcome measures in inflammatory rheumatic diseases, Fransen, J, & Riel, PL van. Arthritis 11, 244., 2009.