Introductory remarks

Summary

The objective of this work package of the SImPatIE-project was the development of a vocabulary and an internal indicator set for patient safety. This report describes the work done to establish the patient safety indicators (PSIs).

The ESQH-office for Quality Indicators in Aarhus, Denmark, was the lead partner of WP4. An expert group consisting of European representatives of project partners, stakeholders and external experts was established for achieving the aims of WP4.

The PSIs were derived through a formalised consensus process based on literature review, targeted information gathering, and expert consultation taking into account previous work done by the project partners and international quality and patient safety organisations. A literature search was performed. We searched for nationwide and international indicators programs known in a cross-European context. Literature for developing new indicators was also identified.

We developed a Stepwise Assessment Framework Approach (SAIFA) to select, characterise and evaluate the new PSIs and the existing indicators we wanted to review. The framework was based on the definition of the term "patient safety indicator" and the vocabulary framework developed by SImPatIE, WP4 and was followed by a charaterisation and an evaluation of the PSI. Indicators were characterised using the developed "Schemes for Characterisation of Indicators". Eight experts from six nations evaluated the PSIs on a scale ranging from 1 to 9 for "Relevance", "Validity and Reliability", and "Feasibility". Statistics for each dimension of the indicator formed the basis of recommendations in four categories from "recommended to be used throughout EU" to "not recommendable for implementation in EU".

A number of 28 known indicators, which have been clinically applied are described and evaluated. Also 14 new PSIs are characterised and evaluated by the expert group. Description of the PSIs can be found on www.simpatie.org. The PSIs are related to risk reduction and harm reduction and cover the dimensions; process and outcome. The PSIs are devided into subsets: "Institution Wide Measures", "Specific Measures" and "Theme Related Measures" covering the themes: "infection control", "surgical complications", "medication errors", "Obstetrics", and "Fall".

The existence of differences including aspects of organisational and clinical culture and sub cultures related to e.g. specialities and professions and also differences related to national, regional and local aspects led us to discommend a common "package" of PSIs for implementation in the EU at present. The consensus process was successfully completed leading to a recommendation of nine of 12 new SImPatIE PSIs whereas 16 of 30 PSIs from existing programmes were recommended for implementation in parts or throughout EU. Implementing the PSIs must be based upon thorough assessment of suitable data, considerations of interpretation, and publication of results.

Monitoring and surveillance of patient safety using PSIs depend on data that are varied along a number of dimensions. However, as the quality of administrative data vary across Europe embarking on the actual patient safety assessment activities using the PSIs across Europe entails additional work.

Introduction to WP4

Measures of aspects of quality of care have been developed and they are increasingly used in Europe being the promising answer to the demands for increased transparency and accountability while creating the basis for improvement and prioritisation in health care systems.

Patient safety has long been recognised as a dimension of quality of care and organisational performance. Care has become more and more complex over time, as the uses of technology as well as the number of effective treatments available have increased. Healthcare professionals are thus involved in increasingly complex care and the possibilities for adverse events have increased.

Patient safety is an outcome of many factors. While patient safety is the ultimate goal, belonging to “good outcomes” what ultimately determines safety is a safer care environment during the patients’ process of care.

Adverse events are systematically and frequently reported in European countries and analyses are made in order to learn from incidents and improve safety. This is resource consuming work, which creates a major need to measure dimensions of safety on an ongoing systematic basis, implement learning organisations, demonstrate ongoing safety improvement, determine when lapses in patient safety occur, and document positive effect of the efforts made. The quality improvement work is the role model of such an ongoing systematic monitoring, as it has proven to make the quality of process, structure and outcome visible.

Quality and patient safety studies providing evidence of the frequency of use of different tools are very few. In Western health care the most commonly used continuous quality improvement tools reported across a range of studies are: brainstorming, cause-effect diagrams, flow diagrams, and data collection tools such as forms for recording observations, and data display or analysis tools. Guidelines, protocols, and organisational procedures has been found to be the most commonly reported generic tools either for implementing the latest research or as part of continuous quality improvement projects to institutionalise changes tested by project teams (1) . The World Health Organization (WHO) synthesis found incident report data collection and analysis, root cause analysis, and crew resource management to be the most often reported in health care (1) .

Thus there is a major need to assess patient safety on an ongoing basis, systematic collection and analyses of PSIs can help prevent future “unsafe” actions of care and, in the long term, their adverse effect.

The SImPatIE project was made possible by a grant from the European Commission on ‘Public Health/Directorate General for Health and Consumer Affairs in 2004’ (OJ 2004/C52, 27 February 2004). This report represents the views of its authors and the expert group of WP4. These views have not been approved by the European Commission and should not be relied upon as a statement of the European Commission.

Organisation of WP4

The ESQH-office for Quality Indicators in Aarhus, Denmark was the led partner of this work package (WP4). It was led by Prof. Jan Mainz, medical director of the Danish Institute for Quality and Accreditation in Healthcare, supported by Dr. Paul Bartels, medical director of the Danish Indicator Project. Overall project management and scientific work was executed by Master of Health Science Solvejg Kristensen.

The expert group consisted of representatives of project partners and external experts, with as members:

- MD, PhD. A. Bourek, University Center for Healthcare Quality, Masaryk University, Czech Republic
- Dr. I. Callanan, Vice President of the Irish Society for Quality and Safety in Healthcare
- Dr. K. Essinger, President, HOPE Subcommittee on Co-ordination (Appointed by HOPE)
- Dr. J. van Everdingen, chief medical officer of CBO (Appointed by CBO)
- Dr. J. Hansen, The Danish National Board of Health
- Dr. M. Kallewaard of the Association of Medical Specialists (Appointed by CBO)
- Dr. B. Lilja, Director of the Danish Society for Patient Safety, Denmark
- Dr. G. Maguerez (Appointed by HAS)
- Prof. Dr. med. G. Ollenschläger, Guidelines International Network (G-I-N)
- MD, PhD R. Suñol, Director of Avedis Donabedian Foundation (FAD).

Overall approach

The work of WP4 was initiated and coordinated by the ESQH-office for Quality Indicators in Aarhus.

The expert group came together February 2006 with the purpose of introducing SImPatIE and WP4. SimPatIE project manager Benno van Beek, CBO took part in the meeting. The following presentations were given:

• Overall SImPatIE Project Plan and Organisation, by project manager Benno van Beek, CBO


• Related Work of Work Package 5, by Dr. Georges Marguerez


• Introduction to Indicators Including Qualifications and Characterisation by Prof. Jan Mainz


• Proposed Schemes for Classification and Evaluation of Indicators by Prof. Jan Mainz


• Patient Safety Indicators, by Dr. Paul Bartels


• Specific Issues of Definition and Methodology and an Overview of Available Materials and Ongoing Work on Taxonomy/Vocabulary, by Dr. Paul Bartels

A detailed work plan for WP4 was established and tasks were assigned.

The overall working method in the expert group has been telephone conferences. Development has been initiated and decisions made in a formalised consensus process. The method of developing the indicators is described in details later.

Prior to the meeting of the expert group an extensive literature search was initiated using the search terms: “Patient safety”, “Vocabulary”, “Glossary”, “Taxonomy” and “Indicator”. PubMed http://scholar.google.dk/ were searched. The literature search was repeated and extended in the process of the work and finalised towards the end of 2006. It was based on a review of similar studies and carried out by the Danish ESQH-office.

A literature review was carried out by the by the Danish ESQH-office in order to identify all relevant sources for the description of concepts and terminology related to patient safety and indicators. The review included work by the Council of Europe (CoE), European Comminities (EC), Organisation for Economic Co-operation and Development (OECD), Agency for Healthcare Research and Quality (AHRQ), European Community Health Indicator Monitoring (ECHIM), The Nordic Indicator Group etc. It was decided to take all identified available material – European and international - into account in the development of the indicators. Details of the literature/background sources related to the development of the PSIs are given later. A full reference list can be found at the end of the report (see here).

Introduction to indicators

The assessment of patient safety can be carried out through both qualitative and quantitative methods (2) . The quantitative approach uses indicators and epidemiological methods of analysis to systematically quantify distinct aspects of patient safety.

Qualitative analysis of adverse events and organisational practice in patient safety has proved to be a rich source of detailed information which has increased knowledge of causation, preventability, and safe practices. The quantitative approach is necessary; it enables comparisons over time, between providers, and of effectiveness of interventions.

In terms of methodological demands for selection, validation, and characterisation of PSIs, they must be considered as a specific type of quality indicators which focus on aspects of patient safety.

Defining an indicator

Indicators can be defined in different ways (3) :

• As measures assessing a particular healthcare process or outcome
,

• As quantitative measures used to monitor and evaluate the quality of health care provider institutions including clinical and support functions
, and

• As measuring tools, screens or flags used as guides to monitor, evaluate, and improve the quality of care, clinical support services and organisational functions affecting patient outcomes

Purpose of the use of indicators

Indicators provide a quantitative basis for clinicians, organisations and planners aiming at achieving improvements in care and the processes by which care is provided.
Indicator measuring and monitoring serve many purposes making it possible to:

• Document the quality of care
,

• Make comparisons and benchmarking over time between places (e.g. units, hospitals)
,

• Make judgments and set priorities (e.g. choosing a hospital or surgery or organising medical care)
,

• Support accountability, regulation, and accreditation
,

• Support quality improvement
, and

• Support patients’ choice of providers

The use of indicators enables professionals and organisations to monitor and evaluate what happens to patients as a consequence of how well professionals and organisational systems function to answer the needs of patients. However, indicators are not a direct measure of quality. As quality is multi-dimensional, understanding quality requires many different measures.

Indicators are based on standards of care. These can be evidence-based and derive from academic literature (e.g. COCHRANE Collaboration, literature syntheses, meta-analyses or randomised-controlled trials). When scientific evidence is lacking, indicators can also be determined by an expert panel of health professionals in a consensus process based on their experience. Thus, indicators and standards can be described according to the strengths of scientific evidence of their ability to predict outcomes (4;5) .

Key characteristics of an ideal indicator

An ideal indicator has the following key characteristics:

• Is based on agreed definitions and is described exhaustively and exclusively
,

• Is highly or optimally specific and sensitive, i.e. detecting few false positives and false negatives
,

• Is valid and reliable
,

• Discriminates well
,

• Relates to clearly identifiable events for the user (e.g. if meant for clinical providers it is relevant to clinical practice)
,

• Permits useful comparisons
, and

• Is evidence-based

.
Each indicator must be defined in detail with explicit data specifications in order to be specific and sensitive.

Indicators may vary in their validity and reliability. Validity is the degree to which the indicator measures what it is intended to measure, i.e. the result of a measurement corresponds to the true state of the phenomenon being measured. A valid indicator discriminates between care otherwise known to be of good or bad quality and concurs with other measures intended to measure the same dimension of quality.

Reliability is the extent to which repeated measurements of a stable phenomenon by different data collectors, judges or instruments at different times and places get similar results. Reliability is important when using an indicator to make comparisons among or within groups over time. A valid indicator must be reproducible and consistent.

Indicators should be based on the best available evidence which can be described as the integration of the best research evidence with clinical expertise and patient values. The strengths of evidence of an indicator will determine its scientific soundness or the likelihood that improvement in the indicator will produce consistent and credible improvements in the quality of care (6) .

Rate-based versus sentinel indicators

A rate-based indicator uses data on events expected to occur with some frequency. These can be expressed as proportions or rates (proportions within a given time period), ratios or mean values for a sample population.

To permit comparisons among providers or trends over time, proportions or rate-based indicators need both a numerator and a denominator specifying the population at risk for an event and the period of time over which the event may take place.

A sentinel indicator identifies individual events or phenomena that are intrinsically undesirable and such indicators always trigger further analysis and investigations. Each incident will trigger an investigation. Sentinel events represent the extreme of poor performance and they are generally used for risk management (7) .

Indicators related to structure, process, and outcome

Indicator can be related to structure, process or outcome of healthcare. Structure denotes the attributes of the settings in which care occurs. This includes the attributes of material resources (such as facilities, equipment and financing), of human resources (such as the number and qualifications of staff), and of organisational structure (such as medical staff, organisation, methods of pure review, and methods of reimbursement).

Process denotes what is actually done in giving and receiving care, i.e. the practitioners’ activities in making a diagnosis, recommending or implementing treatment or other interaction with the patient.

Outcome measures attempt to describe the effects of care on the health status of patients and populations. Improvements in the patient’s knowledge and salutary changes in the patient’s behaviour may be included under a broad definition of outcome and some may represent the degree of the patient’s satisfaction with care.

For a process indicator to be valid its use must previously have been demonstrated to produce a better outcome. Similarly, using structural indicators for quality assessment is only possible if structural components have been shown to increase the likelihood of either a good outcome or a process that has previously been shown to yield better outcomes. Therefore it is necessary to establish such relationships between any particular component of structure or process that is used to assess quality. These linkages may be based on scientific literature. If little evidence exists professional experience concerning these linkages can be distilled using consensus message. Only clinical indicators which are evidence-based have had the linkage between structure or process and patient health outcomes confirmed. The ability to assess the quality of medical, technical care is bound to the strengths and weaknesses of clinical science (8) .

Generic and disease-specific indicators>

Generic indicators measure aspects of care relevant to most patients while disease-specific indicators are diagnosis-specific and measure particular aspects of care related to specific diseases. Most generic and disease-specific indicators focus on structure, process or outcome.

Generic indicators may be difficult to interpret – especially when making comparisons among hospitals or providers as there may be profound differences in patient mix. Disease-specific outcome indicators can be used to compare hospitals and plans when data are risk-adjusted. Confounding factors such as prognostic factors for specific diseases are likely to be found in the scientific literature for these diseases thereby indicating the need for risk adjustment (9-11) .

Indicators related to type of care, function, and modality

Indicators can be classified according to type of care, function, and modality. Indicators classified by type of care may be preventive, acute or chronic. Function of care can relate to screening, diagnoses, treatment, and follow-up. The modality by which care can be delivered relates to physical examination of the patient, laboratory or radiology studies or prescription of medication (11-13) .

Risk adjustment

In most cases, multiple factors contribute to a patient’s survival and health outcome. Therefore, outcome measures must be adjusted for factors outside the health system influence if fair comparisons are to be made. In quality assessment, components relating to the medical care system should be isolated. This is accomplished by controlling for significant confounding factors that contribute to the outcome. Factors that are frequently included in risk adjustment models include patient demographic, psycho-social characteristics (such as age, sex and functional status), lifestyle factors (smoking and alcohol consumption), and severity of the illness that is the focus of measurement, health status and co-morbid conditions. Risk adjustment is essential prior to comparing patient outcomes across hospitals or providers.

Risk adjustments may be most important for outcome indicators. There are also other methods to ensure that other differences among patient groups are not influencing comparisons of process or outcome indicators – e.g. the population of patients for whom the indicator is measured can be carefully restricted. Alternatively, stratified analyses can be performed to examine specific types of patients within a small and overall sample (11;14;15) .

Definitions aims and interpretation of Patient Safety Indicators

PSIs are defined by their purpose, which is monitoring preventable adverse events – directly or indirectly (16;17).

PSIs are thus quality indicators (18), which prove to be valid within the specific framework of interpretation and refer to preventable events or medical errors.

• In terms of structure and process indicators: are healthcare organisational features, practices or interventions, with evidence of effects on exposure to preventable risk factors (e.g. safety culture, hand washing practices, screening of schizophrenic patients for suicidal risk)
, and


• In terms of outcome: harm which is or may be conceived to be caused by preventable events: death, temporary or permanent disability (19;20)

.
The most important contribution of indicator monitoring to patient safety is the potential for quantitative surveillance. Currently assessment of patient safety problems and appropriate interventions relies mainly on case-based qualitative methodology as reporting systems and root-cause analysis. While these widely used tools have proved to be essential for establishing and supporting a clinical culture of safety, evidence of their effectiveness and efficacy is still uncertain (21;22).

Monitoring of relevant indicators such as mortality rates, true rates of adverse drug events or safety culture is a necessary prerequisite to rational priority decisions concerning patient safety interventions (23).

The interpretation of PSIs as absolute measures of patient safety for benchmarking or accountability purposes is demanding in terms of method development. Firstly because of the probabilistic nature of the relationship between exposure (structure-process) and harm (outcome), and secondly because of the interactions between system and patient factors influences outcome values as mortality rates.

The few categories of directly measurable preventable adverse events (wrong sided surgery, wrong blood type, in-hospital suicide) occurs so infrequently, that corresponding indicators has the characteristics of sentinel indicators (24) with very limited value in monitoring changes over time or for benchmarking purposes at the clinical (hospital) level. However, these indicators have been found to be useful at the system (national-regional) level (e.g. as part of the recommended PSIs from OECD) (25).

Therefore assessment of patient safety status by indicators requires access to multiple sources and kinds of data (26) and has the characteristics of screening for safety problems, rather than making a definitive diagnosis (27;28).

Example: Rate-based indicator

Death due to anaesthesia has become rare, by contrast morbid events, i.e. complications related to anaesthetic care such as anaesthetic overdose, reaction, or endotracheal tube misplacement, are much more prevalent causing harm to the patient to a different extent. Safety should be assessed at the aggregated patient level.

Example: Sentinel Indicator

Although surgeons and operating room teams rely on the practice of counts of sponges, sharp and instrument as a means to eliminate detained foreign bodies, practices are not standardised and every single event may signal a serious system failure that should be addressed. Thus safety should be assessed at the individual patient level.

Areas for development and use of Patient Safety Indicators

As with quality indicators in general the effectiveness of PSIs depends on selection of proper clinical areas of measurement in terms of frequency, severity as well as the existence of evidence-based interventions towards patient safety problems.

When these factors are taken into account (29;30), four main areas for hospital-related PSIs can be identified:

• Generic indicators which monitors organisation-wide patient safety characteristics
,

• Indicators monitoring hospital-acquired infections
,

• Indicators monitoring preventable surgical complications
, and

• Indicators monitoring medication safety

Supplemented by a number of condition-specific areas (e.g. bedsores, falls).

Definition of a 'Reporting System' according to WP4:

A system which is designed to contain reports on adverse events. On the basis of reports, analysis and communication of known causes and risk situations is possible. The system can contain reports on human and technical errors as well as organisational circumstances, which affect the occurrence of adverse events in the health care process. Reporting systems include input from all stakeholders - providers and service users.

Example of a developmental area - Reporting systems of adverse events as a basis for developing PSIs:

A reporting system which is used systematically for reports of adverse events, will contain a wide variety of information on reports of different events - different in type, frequency, severity, and actions resulting from analysis.
At first, each report will represent an incident perhaps even a sentinel event. Events which are rated rare but severe could be candidate sentinel indicator event.
Some events in the reporting system database, are alike: such events form thematical clusters and they provide essential information for the selection of adequate rate-based patient safety indicators.

Institution-wide patient safety indicators:

Consists of indicators which address general patient safety characteristics in all types of healthcare organisations and covers both primary care and hospitals:

• Measures of staff safety culture (structural indicators)
,

• Standardized mortality rates (outcome indicators)
, and

• Fraction of patients experiencing adverse events (outcome indicators)

.
Because of their generic nature these indicators are mainly useful in surveillance of organisational interventions or characteristics.

Standardized mortality rates are well-established and validated as indicators of patient safety problems (31;32), but require access to developed administrative healthcare databases and sophisticated analysis of data.

Safety culture assessment and patient experience of adverse events seem promising both in terms of relevance and feasibility, but evidence of validity of these indicators is still limited (33-36).


Indicators monitoring hospital-acquired infections:

Indicators monitoring hospital-acquired infections consist of:

• Rate-based indicators based on registration (direct or through patient administrative data collection systems) of various types of hospital acquired infections (37;38) e.g.
:


• Surgical wound infections
,

• Ventilator pneumonia
, and

• Catheter based UTI
,

• Probability function assessment of hospital acquired infections
, and

• Structural indicators covering infection control measures (e.g. rate of consumption of hand –disinfectants per 1000 bed days) (39)

.
The registration-based indicators are well characterized in terms of limitations of validity and feasibility (40;41).

Probability function assessment of hospital-acquired infections is validated and seems less prone to bias than direct registration, but requires advanced health care IT systems (42-44).


Indicators monitoring preventable surgical complications:

Indicators monitoring preventable surgical complications (45), consist of:

Risk adjusted mortality rates,

Postoperative infection rates

,

Postoperative rates of haemorrhage or thrombosis

,

Readmission rates

, and

Rate of processes which reduces risk of complications (peroperative antibiotics, antithrombotic prophylaxis)

.
These indicators are validated, well-established, and characterized.


Indicators monitoring medication safety:

Construction of relevant and valid indicators monitoring medication safety has posed specific problems because of the very high incidence of medication errors (46).

Presently published indicators consist of:

• Rate of adverse drug events assessed by trigger tool methodology (47-50)
.

This type of outcome indicators has been validated for surveillance of the effects of interventions to improve medication safety within an organisation. The indicators are less useful for benchmarking purposes because of standardisation problems.


• Patients understanding of purpose for taking medications
.

This questionnaire based indicator is a part of the Care Transitions Measure and is closely related to the process of medication reconciliation at discharge

.

Example: Generic indicator:

Communication and transfer of information between healthcare settings and between professionals and patients, are essential aspects of patient safety. Especially care transition processes are known to be vulnerable regarding patient safety.
Understanding the purpose of the medication can impact on compliance. Adherence to medications is important, since lack of compliance can have fatal consequences for the patient. Thus, the quality of staff-patient communication regarding patients' understanding of the purpose of their medication when leaving hospital, is an adequate generic measure of patient.

Example: Disease-specific indicator:

Schizophrenic patients have a known higher risk of suicidal behaviour, especially in the period right after discharge form hospital. Thus, assessment of suicidal risk at discharge is an adequate disease-specific measure of patient safety.

Specific issues of data quality, validity, and reliability of Patient Safety Indicators

In terms of reliability the value of PSIs in organisational learning seems dependent on a high sensitivity of the indicator, while specificity may be of main concern if the indicator is used for public accountability (51;52). The evaluation of PSIs is therefore strongly dependent on the intended use.

The level of the PSIs which monitors harm (e.g. rates of mortality and infections) is usually sensitive to bias caused by the severity of the disease, comorbidities and lifestyle factors. Validity of these indicators is therefore depended on simultaneous collection of patient-related data appropriate for risk adjustment. Empirical evidence suggests that administrative data are not satisfactory in this respect (53;54) especially when used at clinical (hospital) level.

Experience with direct collection of medical record-based data (primary data) seems to partly alleviate this problem. There is however still ambiguity in the clinical definitions with resulting risk of bias in e.g. recording of wound infections which requires caution in interpretation of benchmarking studies. Moreover demands in terms of resources and organisation in direct collection of clinical data limit the scale of this approach (55).

Newer developments in the field of medical informatics in the form of automated detection systems based on primary data from laboratory- pharmacy-, and hospital administrative systems show promising results in unbiased assessment of infection rates (56) and medication safety (automated trigger tool methodology) (57). The methodology requires sophisticated resources in terms of informatics and reliable system wide patient identification.