Methodology

The PSIs/outcome measures were derived through a formalised consensus process beginning at the expert group meeting in February 2005 followed by mail contact and telephone conferences.

A step wise – adjusted – method known from development of clinical indicators was chosen in developing the PSIs (59). The planning phase was: choosing areas to evaluate and organising the measurement team, whereas the development phase was: providing an overview of existing indicators and evidence, selecting indicators, and designing measurement specification.

During the initial expert group meeting consensus was reached on a definition of a PSI, it was also decided to focus on a set of indicators for risk and harm reduction both rewieving excisting indicators and developing new ones.The expert group decided on specific areas for the PSIs, and tasks were assigned.

An extensive literature search was initiated using the search terms: “Patient safety”, “Indicator”, “Risk” and “Harm” to identity nationwide and international PSIs or indicator programs. The search terms were used single and in all possible combinations. To know if the indicators had been used and evaluated in a clinical setting we extended the search using the terms; “Test*”, “Usage”/”Use”, “Apply” and “Valid*”. PubMed and http://scholar.google.dk/ were searched. The literature search was started in the beginning of 2006, and repeated and extended in the process of the work, it was finalised by the end of 2006. It was carried out by the ESQH-office in Aarhus.

Indicators or indicator programmes of organisations mentioned below were reviewed to determine whether they were suitable for characterisation and evaluation according to the “Stepwise Assessment Framework Approach”:

- Australian Council for Safety and Quality in Health Care (ACSQHC)
- Agency for Healthcare Research and Quality (AHRQ)
- The Good Medical Department, Denmark (DGMA)
- International Compendium of Health Indicators (WHO, OECD, Eurostat and ECHIM
- Institute of Healthcare Improvements (IHI)
- Joint Commission on accreditation in Health Care (JCAHO)
- Nordic Indicators (NI)
- Performance Assessment Tool for Quality improvement in Hospitals (PATH)
- The Danish National Indicator Project (NIP)
- Performance Indicators on Patient Safety and effectiveness for Dutch Hospitals.

.
The literature search identified the following methods suitable for the basing of indicators:

• Measuring Hospital Standardised Mortality Rates
,

• Hospital Acquired-Infection Registration
,

• Quantitative Patient Safety Cultural Assessment
,

• Patients Satisfactions’ Surveys
, and

• Electronic Trigger Tools

.
The literature search supported by the initial expert group meeting identified the following themes suitable for basing indicators upon:

• Hand hygiene


• Transition of care


• Medication errors


• In-hospital fall


• Infection control


• Surgical complication

We developed a 'Stepwise Assessment Framework Approach' to select, characterise, and evaluate new PSIs as well as reviewing existing indicators. The “Stepwise Assessment Framework Approach” is described in details later.

To assess whether an indicator qualified, it had to match the definition of an indicator. Also it had to fit into one of the boxes in the table of the core terms of the vocabulary, stating the relation to process or outcome as well as to preventability. Further details can be seen in the report of WP4 on the development of the SImPatIE Patient Safety Vocabulary.

To characterise the indicators (both the reviewed and new) a Scheme for Characterisation of Indicators as well as was a Brief Assessment Instrument was developed for external application. Both instruments were developed by the ESQH-office in Aarhus according to recommendations in the literature, experience, and similar schemes (60;61). The Scheme for Characterisation of Indicators and the Brief Assessment Instrument were pilot tested on a few PSIs. Discussion, comments, and evaluation of the performance of the two methods was made in an expert telephone conference and adjustments were made accordingly. In the section describing the Stepwise Assessment Indicator Framework Approach both instruments are shown.

Characterised indicators were discussed in a telephone conference of the expert group; suggested changes leading to consensus were made. The indicators were then redistributed to the expert group for evaluation.

Indicators (both existing ones and new) were evaluated by individual members of the expert group using the “Scoring Sheet” integrated in the Brief Assessment Instrument. Evaluation of the PSIs was carried out according to three dimensions (Relevance and Appropriateness, Validity and Reliability and Feasibility) of the indicator on a scale ranging from 1 to 9. Scores were divided into; 1-3 Low degree, 4-6 Medium degree and 7-9 High degree. Ratings on each of the three dimensions of each PSI were added and percentiles, mode, minimum and maximum scores calculated and frequencies of scores displayed graphically. On the basis of these statistics the expert group discussed recommendations for application in Europe in telephone conferences. Making these recommendations, the expert group focused on the scientific properties mentioned; the group discussed aspects such as resources available, organisation in individual EU countries, legal systems etc. in connection with the dimension Feasibility. However these aspects are not systematically uncovered Europe wide.

Patients perspectives were represented in WP4's work in different ways; the organisation “Action against Medical Accidents” (AWA), an independent English charity promoting better patient safety and justice for people who have been adversely affected by a medical accident, were represented in the over all project meetings. A representative of AWA also reviewed and commented the characterisation of the PSIs, a set number of five PSIs, and a draft of this report.

The Organisation HOPE commented on a draft of this report.

To coordinate between the work of WP4 and the other work packages at least one representative of the ESQH-office in Aarhus continuously took part in telephone conferences, overall project meetings, steering group meetings, and congresses.