
	» SIMPATIE » Results of the workpackages of the project » Hidden folder » 3. External auditing methods

SIMPATIE

About the project

Council of Europe Recommendation (2006)7

Interactive Recommendation(2006)7

Agenda

Forum

Mapping Europe

Search in Framework

Mapping Europe

Best Practices

SEARCH THIS SITE

- Login

3. External auditing methods

3.1 Models of external evaluation: variety and convergences

There are numerous methods usable for external evaluation of HCOs. The ExpeRT (External Peer Review Technique) project 11 , funded by EU, showed that methods of developing and assessing organisational standards range from the medical speciality driven 'visitation/visitatie') (the Netherlands and UK), through traditional accreditation (developed in North America, Australia, UK, Spain, The Netherlands, Finland, Italy, France, Portugal, Sweden, Bulgaria, Poland, Germany, and Switzerland) and European Quality Awards (Scandinavian nations, The Netherlands, Spain) to industrial certification ISO standards (Germany, UK, Switzerland, and Spain). The multiplication of methods, often coexistent in the same country, does not facilitate a global and consistent safety process in Europe. Although a process of convergence is expected, the main barriers to adopt a unique method for external evaluation remain cultural (the cost of evaluation -who pays: private or public-, the bureaucratic tendencies to too much focus on public perception and local efficiency compared with attention to medical effectiveness, and the impact of various national systems of responsibility and accountability).

Nevertheless, all the models share a minimum set of common features, such as:

use of standards,

external evaluation by auditors who are variably trained, polyvalent or specialized, and who may or may not be peers,

on site visit to check compliance (at different degrees: meetings, documents analysis, direct observation),

decision committee,

report with mention of axes to be improved,

need for external recognition,

a process of dissemination whether public or restricted to professionals

defined cycle-duration inter procedures,

little based-evidence regarding their impact on effectiveness on health system performance and especially on patient safety improvement.

3.2 The families of methods

Accreditation:

ISO 9001 certification (International Organization for Standardization)

link

Audit by professional peers (Peer review, Dutch visitatie):

.
The Quality Management Model (Malcolm Baldrige type)

Indicators monitoring

.
Inspections of healthcare services

Evaluations done by patients

ReCertification, registration and licensing:

3.3 Barriers to implementation and points for discussion irrespective of audit method

Voluntary versus compulsory audit

There is a lot of debate on whether an audit should be a free choice or imposed to some extent. The response seems to be mainly related to a country's political system, but there is doubtless a cultural dimension as well. However, the previous section has clearly pointed to the growing development of mandatory systems in all Western countries.

If the process of auditing becomes mandatory for all medical establishments, the number of auditors will have to increase in a similar proportion. The human resources cost of external auditing may rapidly become exponential. For example, France has 800 accreditation surveyors to accreditate all 2900 HCOs in the country.

As mentioned above, the compulsory nature of the process automatically drags the objectives towards minimum standards able to be applicable to most establishments.

Profile of surveyors

In most programs, auditors are peers who have little exposure and practice in auditing and for whom surveying represents an auxiliary and occasional job.

There is also a need to guarantee the independence of judgment that may restrict the number of available candidates. Some countries have recruited foreign surveyors (the Irish use Canadian and Australian experts). Others recruit surveyors from structures relatively close to the HCO (a body representing health insurance funds, an inter-hospital group organisation, etc).

More needs and less specialized resources may lead to wider eligibility criteria for surveyors, thus resulting in a huge disparity of these surveyors and an instability of results. The disparity of judgment and the disputes among experts grows with the number of experts, the previous professional experience, the reduction of training, and the use of qualitative indicators.

Conversely, the disparity may represent a rich mix of experience, particularly adapted to the diversity of the activities of HCOs.

Finally, four suggestions may reduce conflicts:

Audit frequency and procedure

Most methods are based on periodic audits notified in advance, for which (significant) preparatory work is carried out by the HCO being audited. The preparation is often considered to be a particularly valuable opportunity for acquiring a culture of safety.

But increasingly, in spite of the initial advantages of scheduled surveys (with the preparation itself constituting a 'learning system'), disadvantages become apparent in a system involving too much preparation (artificiality, concern for the image presented rather than the reality); this is particularly true for accreditation. Several systems now recommend moving on to unscheduled external auditing (JCAHO 2006, Health Care Commision (UK), Australian Council for Health Standards).

A number of bodies (JCAHO, HCC) have suggested or practised a mixture of (i) periodic self-assessments (a sector currently developing rapidly, which includes the benefits of a learning system and the acquisition of a culture of safety) and (ii) unscheduled external audits for validation, sometimes even limited to a few HCOs chosen at random (HCC).

Maximum versus minimum requirements: setting the objectives

A number of authors have emphasised the risk of working to optimum standards: they are generally very hard to achieve, and the level of complexity and cost involved is such that in the end, the optimum standard may be adhered to less than an actually achievable minimum standard (Australian Council for Safety and Quality in Health Care, 2003; Ovretveit, 2005).

The absence of mutually-agreed standards often makes the work of external audit difficult or open to criticism.

~ Demonstration of scientific proof of the efficacy of actions undertaken for patients (burden of proof), balanced by the cost and simplicity of implementation, guides the drafting of external audit standards (the first to be implemented are the most effective and the easiest to put in place and to monitor and the least expensive) (Ovretveit, 2005, pp 22-25).

~ But the more systemic aspects in the longer term (e.g. promoting an organisation and culture within an HCO), although recognised by the literature in the field as essential for achieving and maintaining a high level of safety, are difficult to impose as they are not supported by scientific proof.

Too many standards can be just as harmful as having no agreed standards; a battery of rules, especially when they differ from the HCO's or country's economic and cultural model, create strong pressure to violate them. The Australian reference document on accreditation considers that having too many standards is one of the greatest risks of an accreditation as it may lead to a 'paper policy' with a high level of virtual safety going hand in hand with a very high number of violations in the real world 14 and 15 .

The economic conditions of the countries and HCOs concerned are not sufficiently taken account of in the range of requirements. This is particularly the case in Europe, and is a very sensitive issue for the SIMPATIE project (safety is known to be closely related to a country's GDP).

At the European Level, a reasonable platform for action should inevitably adopt an achievable standard for all countries. Such a minimum platform could include:

A series of patient safety goals, supposed to be revised and improved on periodic cycles.

And a set of standards for external evaluation that have the following properties.

An objective to be attained

Realistic/economic

Pertinent: Evidence-based or consensus-based

Recognized by professionals and the public

Evolving by close linkage between the processes of definition of standards and feedback on their utilisation by HCOs and surveyors

Measurable by mean of easy-to-gather, easy-to-compare safety indicators.

Comparative and public audit results

Publishing the results to give the public confidence is problematical. Publicity about risk is poorly managed, and often poorly interpreted; there can be both too much and too little publicity.

Many studies and guidelines emphasise the benefit of a clear decision on the quality observed, and of credible corrective follow-up actions addressing all weak points.

But the actual publishing of these weak points is controversial. The accepted wisdom is that a culture of safety is built in a 'blame-free' setting. If this is the case, insisting that faults in the system are publicised will make a system purely reactive and defensive, and the harmful effect will be greater than the anticipated benefit (there is a very interesting discussion of this in the Australian Council's 2003 publication, "Standard setting and accreditation systems in healthcare").

However, the recent French experience of publishing on the web site the full details of the second round of accreditation has apparently not been detrimental to public confidence, and has not provoked a new legal pressure on the medical system.

Role of self-assessment

Most accreditation programs include a self evaluation made prior to the visit of surveyors (this is e.g. the case of the French accreditation program). This self evaluation has numerous objectives. First, the process of evaluation is by itself an educational process for the medical staff; it can clarify safety objectives, encourages installing or updating reporting systems and dedicated protocols possibly using external assistance; in any case it may facilitate the development of a safety culture. Second, many complex indicators are not directly accessible within the time frame of the visit by surveyors; the self assessment, less constrained in time, may serve to feed these series of missing indicators for the final evaluation. Third and not least, self-assessment permits the HCOs to make corrections prior to the external auditing process and to demonstrate the ability of the HCO to react and improve.

As mentioned (3.3 Audit frequency), some accreditation programs go beyond the use of self evaluation as a pre visit, considering that a joint negotiated selection of objectives between authorities and HCO, followed by self-evaluation and communication of results to the external agency may replace the external auditing process. External controls by surveyors may then take place randomly with a much lower pace.

The development of indicators and performance measurements

A very wide range of indicators can be used. Only indicators that focus on immediate deficiencies of patient care are easy to compare and doubtless to standardise.
A good review of the literature is provided by WP4, with a special value of the section on the evaluation of patient safety culture.

These are the five families of safety indicators recognised as generally accepted and published by OECD (Quality indicator project,2006) 16 .

1.	Hospital acquired infections	Ventilator pneumonia Wound infection Infection due to medical care Decubitus ulcer
2.	Operative and post-operative complications	Complications of anaesthesia Postoperative hip fracture Postoperative pulmonary embolism or deep vein thrombosis Post operative sepsis Technical difficulty with procedure
3.	Sentinel events	Transfusion reaction Wrong blood type Wrong-site surgery Foreign body left in during procedure Medical equipment-related adverse event Medication errors
4.	Obstetrics	Birth trauma injury to neonate Obstetric trauma vaginal delivery Obstetric trauma caesarean section Problems with childbirth
5.	Other care-related adverse events	Patient falls In-hospital hip fracture or fall

The indicators can be strategically prioritized and used for developing a solid method for evaluating and improving patient safety strategies.

IHI's Whole System Measures Tool Kit: IHI's Pursuing Perfection and IMPACT Network aim is to make fundamental improvements in the performance of participating health systems. The changes in the design of these systems are expected to improve major outcomes identified in the IOM's Chasm Report so that care is more safe, effective, patient-centered, timely, efficient and equitable. Version 2.0 recommends the following ten measures:

Adverse Events - inpatient and outpatient
Work Days Lost

Hospital Standardized Mortality Ratio (HSMR)

Unadjusted Raw Mortality

Functional outcomes

30 Day readmission

Patient Satisfaction - inpatient and outpatient

Patient Days spent in the Hospital during the last six months of life

Days to 3rd Next Available Appointment - primary care and specialty care

Health Care Costs per Capita for Region.

The indicators focusing on safety culture are distinctly less consistent between countries, and over time. Another disadvantage is that they are much harder to collect objectively.

However, there is growing tendency to evaluate the culture. A series of in-depth reviews of existing and validated methods has been recently published by Pronovost & Sexton, 2005; Kho, Carbone, Lucas & al, 2005; and Sexton, Thomas, Helmreich and al, 2004 17 .

The integration of strategies to imbed specific consensual or evidenced-based professional practices

Beyond the utilization of well established indicators for internal improvement, Agencies are setting up global programs for HCOs to embed safe and well evidenced professional practices. These include the implementation of Patient safety solutions and the measurement of system wide indicators of process and outcome.

The US JCAHO proposes for 2007 a National Patient Safety Goals made of 15 goals (Improve the accuracy of patient identification, improve the effectiveness of communication among caregivers, improve the safety of using medications, reduce the risk of healthcare associated infection, accurately and completely
reconcile medications across the continuum of care, reduce the risk of patient harm resulting from falls, reduce the risk of surgical fires, reduce the risk of influenza and pneumococcal disease in institutionalized adults, etc).

The JCAHO's is also recommending the promotion of Patient Tracer Methodology aiming at adopting a whole and systemic approach of all aspects of patient safety 18 , including:

Priority focus areas identified by priority focus process

Traces patients through entire inpatient experience

Patient records are selected in several
departments.

Surveyors trace the analytical procedures

Supporting activities reviewed:

Diagnostic related activities (i.e.: evaluation of needs, ordering,
specimen collection , preservation, transportation, etc.),

Care administration activities (i.e. Procedure written, approved, implemented, specimen analysis, assuring competence of staff, etc.), and

Post-analytical processes (i.e.: review of results prior to reporting, assuring an acceptable process for correcting erroneous results, etc.).

Instrument calibration and maintenance

Multiple patients followed

Issues may be identified in one or more steps of a process or in interfaces between processes (numerous consultancies are available to follow the process) 19 .

The Canadian Council for Health Services Accreditation (CCHSA) adopted in 2004 a global plan to implement Canadian's Patient Safety Goals and Required Organizational Practices that came into effect in January 2007 after an experimentation of one year in 2006.. The outcome is the creation of five Patient Safety areas (culture, communication, medications, workforce, infection control), six Patient/Client Safety Goals, and 21 Required Organizational Practices (ROPs).

The World Health Organization (WHO) Collaborating Centre on Patient Safety (Solutions), the World Alliance for Patient Safety are presently working on the definition of safety solutions: these include look-alike, sound-alike medications, patient identification, communication during hand-overs, prevention of wrong site/wrong procedure/wrong person surgical errors, prevention of continuity of medication errors or medication reconciliation, prevention of high concentration drug errors and promotion of effective hand hygiene practices.

Some clusters of indicators have been successfully developed by the Institute for Healthcare Improvement in Boston (IHI) and may serve safety self assessment by means of triggers tools 20 . These triggers based
on an risk-related aggregation of indicators (in most cases related to bundles of best practices) permit rapid identification of adverse events through simplified chart review.