Objectives of European Patient Safety Indicators
The patient safety indicators (PSIs) should be useful in:
- Surveillance and monitoring of the impact of patient safety activities – e.g. monitoring effects of interventions by reduced harm
.
- Identify unsafe processes of care and eventually also unsafe patient outcomes
.
- Monitor safety consequences of organisational changes in processes and/or structures on the health care system.
Furthermore the PSIs should capture important safety aspects, be scientifically sound and feasible, as recommended in the literature
(58) and from the expert group.
The PSIs are meant for use by professionals. The results achieved when the PSIs have been applied can be of use for both professionals (e.g. clinicians, organisations and planners aiming at achieving improvements in care and the processes by which patient care is provided), and laymen e.g. patient and potential patients.
Organisation
The work presented was developed in interplay between the ESQH-office for Quality Indicators in Aarhus and the members of the established expert group of WP4. Participating expert group members and external experts assigned by an expert group member in the work concerning the PSIs were:
* MD, PhD. A. Bourek, University Center for Healthcare Quality, Masaryk University, Czech Republic
* Dr. I. Callanan, Vice President of the Irish Society for Quality and Safety in Healthcare
* Dr. K. Essinger, President, HOPE Subcommittee on Co-ordination (Appointed by HOPE) in cooperation with Dr. J. Ahlberg, Medical director, LÖF, Patientforsakring, Sweden
* R. Gijsen, MSc., Health care researcher, epidemiologist in cooperation with J. de Koning, MPH, PhD, Senior health care researcher, project leader (Appointed by CBO, Holland)
* Dr. M. Kallewaard of the Association of Medical Specialists (Appointed by CBO)
* Dr. B. Lilja, Director of the Danish Society for Patient Safety, Denmark in cooperation with Dr. J. Anhoej, the Danish Society for Patient Safety
* Dr. G. Maguerez (Appointed by HAS, France), Dr. med C. Thomeczek, Senior Researcher, Managing Director, the German Agency for Quality in Medicine (AQuMed / AEZQ) Joint Institution of The German Medical Association and the National Association of the Statutory Health Insurance Physicians representing Prof. Dr. med. G. Ollenschläger of the Expert Group
Prof. Jan Mainz, medical director of the Danish Institute for Quality and Accreditation in Healthcare and Dr. Paul Bartels, medical director of the Danish Indicator Project from the ESQH-office in Aarhus participated in the expert group. Project management and scientific work was executed by Solvejg Kristensen, Master of Health Science.
Methodology
The PSIs/outcome measures were derived through a formalised consensus process beginning at the expert group meeting in February 2005 followed by mail contact and telephone conferences.
A step wise – adjusted – method known from development of clinical indicators was chosen in developing the PSIs (59). The planning phase was: choosing areas to evaluate and organising the measurement team, whereas the development phase was: providing an overview of existing indicators and evidence, selecting indicators, and designing measurement specification.
During the initial expert group meeting consensus was reached on a definition of a PSI, it was also decided to focus on a set of indicators for risk and harm reduction both rewieving excisting indicators and developing new ones.The expert group decided on specific areas for the PSIs, and tasks were assigned.
An extensive literature search was initiated using the search terms: “Patient safety”, “Indicator”, “Risk” and “Harm” to identity nationwide and international PSIs or indicator programs. The search terms were used single and in all possible combinations. To know if the indicators had been used and evaluated in a clinical setting we extended the search using the terms; “Test*”, “Usage”/”Use”, “Apply” and “Valid*”. PubMed and http://scholar.google.dk/ were searched. The literature search was started in the beginning of 2006, and repeated and extended in the process of the work, it was finalised by the end of 2006. It was carried out by the ESQH-office in Aarhus.
Indicators or indicator programmes of organisations mentioned below were reviewed to determine whether they were suitable for characterisation and evaluation according to the “Stepwise Assessment Framework Approach”:
- Australian Council for Safety and Quality in Health Care (ACSQHC)
- Agency for Healthcare Research and Quality (AHRQ)
- The Good Medical Department, Denmark (DGMA)
- International Compendium of Health Indicators (WHO, OECD, Eurostat and ECHIM
- Institute of Healthcare Improvements (IHI)
- Joint Commission on accreditation in Health Care (JCAHO)
- Nordic Indicators (NI)
- Performance Assessment Tool for Quality improvement in Hospitals (PATH)
- The Danish National Indicator Project (NIP)
- Performance Indicators on Patient Safety and effectiveness for Dutch Hospitals.
.
The literature search
identified the following methods suitable for the basing of indicators:
- Measuring Hospital Standardised Mortality Rates
,
- Hospital Acquired-Infection Registration
,
- Quantitative Patient Safety Cultural Assessment
,
- Patients Satisfactions’ Surveys
, and
- Electronic Trigger Tools
.
The literature search supported by the initial expert group meeting
identified the following themes suitable for basing indicators upon:
- Hand hygiene
- Transition of care
- Medication errors
- In-hospital fall
- Infection control
- Surgical complication
We developed a “
Stepwise Assessment Framework Approach” to select, characterise, and evaluate new PSIs as well as reviewing existing indicators. The “Stepwise Assessment Framework Approach” is described in details later.
To assess whether an indicator qualified, it had to match the definition of an indicator. Also it had to fit into one of the boxes in the table of the core terms of the vocabulary, stating the relation to process or outcome as well as to preventability. Further details can be seen in the report of WP4 on the development of the SImPatIE Patient Safety Vocabulary.
To characterise the indicators (both the reviewed and new) a Scheme for Characterisation of Indicators as well as was a Brief Assessment Instrument was developed for external application. Both instruments were developed by the ESQH-office in Aarhus according to recommendations in the literature, experience, and similar schemes
(60;61). The Scheme for Characterisation of Indicators and the Brief Assessment Instrument were pilot tested on a few PSIs. Discussion, comments, and evaluation of the performance of the two methods was made in an expert telephone conference and adjustments were made accordingly. In the section describing the Stepwise Assessment Indicator Framework Approach both instruments are shown.
Characterised indicators were discussed in a telephone conference of the expert group; suggested changes leading to consensus were made. The indicators were then redistributed to the expert group for evaluation.
Indicators (both existing ones and new) were evaluated by individual members of the expert group using the “Scoring Sheet” integrated in the Brief Assessment Instrument. Evaluation of the PSIs was carried out according to three dimensions (Relevance and Appropriateness, Validity and Reliability and Feasibility) of the indicator on a scale ranging from 1 to 9. Scores were divided into; 1-3 Low degree, 4-6 Medium degree and 7-9 High degree. Ratings on each of the three dimensions of each PSI were added and percentiles, mode, minimum and maximum scores calculated and frequencies of scores displayed graphically. On the basis of these statistics the expert group discussed recommendations for application in Europe in telephone conferences. Making these recommendations, the expert group focused on the scientific properties mentioned; the group discussed aspects such as resources available, organisation in individual EU countries, legal systems etc. in connection with the dimension Feasibility. However these aspects are not systematically uncovered Europe wide.
Patients perspectives were represented in WP4's work in different ways; the organisation “Action against Medical Accidents” (AWA), an independent English charity promoting better patient safety and justice for people who have been adversely affected by a medical accident, were represented in the over all project meetings. A representative of AWA also reviewed and commented the characterisation of the PSIs, a set number of five PSIs, and a draft of this report.
The Organisation HOPE commented on a draft of this report.
To coordinate between the work of WP4 and the other work packages at least one representative of the ESQH-office in Aarhus continuously took part in telephone conferences, overall project meetings, steering group meetings, and congresses.
The Stepwise Assessment Indicator Framework Approach
The developed Stepwise Assessment Indicator Framework Approach to select, characterise, and evaluate appropriate indicators is illustrated below. If the indicator does not fulfil all the required demands of a step, it is deleted and not taken any further. The steps are described below.
Step 1. Selecting the indicator
Making sure the indicator addresses relevant and significant patient safety issues: it qualifies as an indicator according to the definition in Table 1, and it can be placed within the bold frame in Table 2.
Table 1. Definition of a Patient Safety Indicator
If the indicator does not fulfil the required demands of step 1, it is deleted.
Table 2 pdf-file here
Step 2. Characterising the indicator
To characterise the indicators Scheme for Characterisation of Indicators is used. It is vital for this step that both the numerator and the denominator are defined.
Step 3. Evaluating the indicator - a brief rating assessment instrument for external application
Indicators were evaluated by each individual member of the expert group using the scoring sheet integrated in the Brief Assessment Instrument.
Table 4 pdf-file here
If the indicator does not fulfil the required demands of step 3 it is deleted. Only indicators meeting all demands in the “Stepwise Assessment Framework Approach” have been fully characterised, evaluated, and recommended.
Step 4. Recommendations for use of the indicator in Europe
Registering and analysing the PSIs has to be feasible. It is crucial, therefore, to aim for a limited set of PSIs that when possible can draw upon existing registries or databases within hospitals and be used for benchmarking. It would be an illusion to think that all aspect of patient safety could be adequately captured in such a set of PSIs.
As the expert group was not able to make definite statements about exact data availability and quality for all the PSIs in all European countries it strived to take cross-cultural registration practices into account in making recommendations for PSI application. The PSIs are divided into four sets dependent on whether the expert group found them:
1. Immediately workable throughout the European health care systems
2. Immediately workable in parts of the European health care systems
3. At present not workable for implementation in Europe – Recommendation for future decision on implementation or
4. Not suitable as a PSI for recommendation in Europe
Table 3. Step 2: Characterising the Indicator
PSI (number): (Title)
Overall origin of the PSI relating to the organisation
Dimension Description
Description of Specific
Aspects of Patient Safety
Provides a concise statement of the specific aspects of patient safety, the
patient population, providers, setting(s) of care, and time period that the
measure addresses.
Aim of the PSI Gives a concise description of the aim of the specific PSI.
Level of Determination of
Patient Safety
Identifies the level at which safety can be assessed (i.e., at the individual
patient level or the aggregate patient level).
Source(s) Describes the specific sources of the PSI
Evidence Supporting the
Criterion of Patient Safety
Describes the type(s) of supporting evidence appropriate for the measure
domain.
Extent of Clinically
Testing
Describes the extent of testing of the measure including reliability and/or
validity testing.
Evidence of Clinically use
of Standards
Describes the extent of standards used clinically, and which standards
were used according to the objective of the indicator.
PSI Category Specifies whether the indicator is:
Institution-wide PSIs
Theme Related PSIs:
“Infections Control”
“Surgical Complication”
“Medication Errors”
“Obstetrics”
“In-Hospital Fall”
Diagnose Specific as well as other Specific PSIs
Data Definitions Describes the data definition in detail
Denominator Description Provides the general specifications of any clinical component that is the
basis for inclusions and exclusions in the denominator.
Numerator Description Provides the general specifications of any clinical component that is the
basis for inclusions and exclusions in the numerator.
Data Source Identifies the data source(s) necessary to implement the measure
Identifying the
institutional context
Placing the indicator in an institutional context. E.g. it is also relating to
quality improvement work and/or accreditation
Care Setting Classifies the settings for which the measure applies
Professionals Responsible
for Health Care Classifies the professional(s) who is/are responsible for health care
Lowest Level of Health
Care Delivery Addressed
Classifies the lowest level of health care delivery to which the measure
(in its current use) applies
SImPatIE WP4 – Establishing a set of Patients Safety Indicators
21
Allowance for Patient
Factors
Identifies the type of analytic considerations made for the measure based
on patient factors or characteristics (e.g., High-risk/vulnerable subgroups,
Other subgroups [e.g., age cohort], Case-mix adjustment, Paired data at
the patient level, Risk adjustment).
Stratification by
Vulnerable Populations
Describes the populations vulnerable to health care patient safety
problems that are separately identified for sampling
Standard of Comparison Classifies the type and time frame of the comparison according to
whether the comparison is external (at a given point-in-time or of a time
trend), internal, or to a prescriptive standard.
Scoring Identifies the method used to score the measure (e.g., Categorical,
Continuous Variable, Count, Frequency Distribution, Non-weighted
Score/Composite/Scale, Rate, Ratio, Weighted Score/Composite/Scale).
Table 4. Step 3: Evaluating the indicators using the Brief Assessment Instrument
SCORING MATRIX
Dimension Definition Score
Relevance and
Appropriateness
Are areas of significance covered (severity and
frequency) in terms of patient safety within its
specified domain (population and/or
organisation)?
1-3 Low degree of relevance
4-6 Medium degree of relevance
7-9 High degree of relevance
Validity and
Reliability
Is the instrument satisfactory in terms of:
- construct validity (evidence based)
- Internal consistency
- Exhaustiveness/exclusiveness
- Reliability
1-3 Low degree of validity
4-6 Medium degree of validity
7-9 High degree of validity
Feasibility How is the:
- Availability of data
- Clinical burden of data collection
1-3 Low degree of feasibility
4-6 Medium degree of feasibility
7-9 High degree of feasibility
SCORING SHEET :
Title (of the evaluated instrument):
Scores
Relevance and
Appropriateness
Validity and
Reliability Feasibility
Additional Comments/
Overall Evaluation
Score from 1-9
Score from 1-9
Score from 1-9
Free text
Table 2. Step 1: Identifying the type of indicator
PROCESS Actual event Near miss (sub-event)
Non preventable event Preventable event (Adverse event)
OUTCOME Harm: Adverse outcome No Harm
Table 1. Definition of a Patient Safety Indicator:
Patient safety indicators are measures that directly or
indirectly monitor preventable adverse events
