Substantial capital and high expectations are presently invested in the implementation of health information systems and electronic health records (EHRs) in the USA [1], and in European countries such as the UK [2], Austria [3], and Norway [4] (for a comprehensive overview of studies of EHRs, see [5]). In Denmark, efforts to develop and implement EHRs have been part of national IT strategies since the mid-1990s, with relatively high ambitions of cross-professional, structured records that enhance clinical work, planning, and research [6], [7]. Presently, all Danish hospitals have electronic patient administration systems (PAS), medication modules, and modules for ordering tests and receiving test results, and some also have electronic modules for clinical documentation and booking.
In 2009, Central Denmark Region, one of the five Danish regions, decided to implement a comprehensive EHR at an entire hospital, to test its large-scale clinical applicability. The EHR had been developed by the regional administration and a software company over the preceding six years, and comprises modules for booking, ordering tests and receiving test results, prescribing and administering medication, patient administration, and documenting clinical work. The EHR modules are integrated, use the same database, and exchange information seamlessly, and since the modules encompass almost all functionality needed by health care professionals in their daily work, the EHR may be called “comprehensive”. The medication, booking and ordering/receiving test modules had already been implemented at other hospitals in the region some years previously, but the PAS was to be replaced by a new module, and another entirely new module was also introduced, to replace all paper-based records documenting the work of physicians, nurses, occupational and physiotherapists, midwives, and so on. The existing picture archiving and communication systems for x-rays and MR scans, microbiology lab reports system, and other electronic systems were not integrated into this version of the EHR, which is why it was labeled a “comprehensive”, rather than a “complete” EHR. The EHR is based on structured data so that entries by all professions are primarily structured and linked to standard nomenclatures, so that, for example, the EHR automatically reports Diagnose Related Group codes to the National Board of Health for statistical and reimbursement purposes. Free texts field exist, but most entries are done by ticking of radio buttons or check boxes.
This was the first time the region implemented this broad range of functionality, and earlier versions of the EHR had been heavily criticized by clinical staff because of slow performance and lack of functionality. At the same time, the region was to decide whether to continue or abandon the development of this EHR. Successful implementation of the EHR was crucial to Central Denmark Region, which had invested much capital in the development of this EHR, approx. 45 million Euro at this point. Depending on the results of an evaluation, the EHR was to be implemented at the region's other five hospitals, including a large university hospital. Once implemented, use of the comprehensive EHR would be mandatory for all health care staff in the region, amounting to approx. 10,000 different daily users and covering 1.17 million patients with registered data.
Randers Regional Hospital (RRH) was designated as the implementation site for testing the EHR's large-scale clinical applicability. RRH has a staff of approximately two thousand. The hospital has 360 beds, and admits 30,000 patients annually, of which 25,000 are acute cases. Yearly, the outpatient clinics treat 110,000 patients. The hospital is medium-sized, and ranks nationally as one of the most productive measured on treatments per staff. Implementation was scheduled to take place in three stages during February, March, and April 2010, and because of the region's need for a rapid evaluation, focus was on the Department of Internal Medicine and the Emergency Medical Ward, which participated in the first stage of implementation in February 2010. Between them, these departments employ about half of the personnel at the hospital.
Shortly before its implementation at RRH, the regional administration commissioned an evaluation of the EHR, with the broad aim of evaluating the EHR's immediate consequences to health care work, and to identify potential barriers to further enrollment. The Institute of Public Health, which is part of the region, and Aarhus University (Denmark) were chosen as evaluators, and established a steering committee that included six representatives of health care staff. The region commissioned and received the evaluation report, which became publicly available, but did not otherwise participate in the evaluation. The region's decision regarding further deployment was urgently needed, and the evaluation was to be handed in four months after implementation.
Few comprehensive EHRs have been adopted in European and U.S. hospitals [8], [9]. In 2009, fewer than 3% of all U.S. hospitals had a comprehensive EHR, though Kaiser Permanente and the Veteran Health Administration being important exceptions have implemented such EHRs. Comprehensive EHRs may be found in European hospitals (e.g. at Hospital da Luz, Portugal) but there is no accurate estimate of their number. Few evaluations of comprehensive EHRs have been published, and are either based on interviews [10] or administrative data [11], so experiences with evaluation results and how to evaluate comprehensive EHRs are sparse.
This paper presents the results of an evaluation of a comprehensive EHR, shortly after its implementation at a medium-sized hospital. The paper makes three contributions to medical informatics: First, it provides an example of how comprehensive EHRs can be evaluated in the shake-down phase; second, it describes system performance data and experiences with the implementation of a comprehensive EHR, shortly after its implementation, third, it describes possible factors contributing to the relative success of the implementation.
Evaluation is a recommended part of the development and implementation of information systems and EHRs, to ensure system functionality, fit with work processes, and decisions regarding future design and deployment [12], [13]. Evaluations may be either formative or summative. A formative evaluation is intended to improve the EHR by providing system providers with feedback, so changes may be made to the EHR. Formative evaluations can also guide subsequent implementations in other departments or
In the interviews, the data pointed toward a generally positive, though not uncritical assessment of the EHR across departments and professions. Although staff assessed the EHR to not entail less work, it was seen as supporting their tasks, and they had high expectations of its benefits to patients, staff, and the hospital in the future. A summary of the results of the questionnaire across professions and the two departments also reflects this general satisfaction (Table 2). (The questionnaire
The results of the evaluation were impatiently anticipated, since the region had to decide whether to implement the EHR at its other five hospitals, or abandon the ten-year-long development project entirely. Furthermore, physicians had been severely critical of the poor performance of previous versions of the EHR, prior to implementation of the EHR version in focus in this paper. Therefore, the generally positive evaluation results were a welcome relief to the region, system providers, and RRH,
We have presented a formative evaluation of a comprehensive EHR based on a mixed-methods case study, designed to be interactive and concurrent, and conducted in two hospital departments, based on the updated DeLone and McLean framework. The evaluation was based on system performance data, a questionnaire, single-person and group interviews, and ethnographic observation. It focused on four selected groups: physicians, nurses, medical secretaries, and physiotherapists. Additionally, focus group
All three authors have contributed equally to the generation of data, analysis of data and writing of this article. However, individual author has specifically contributed to the following: ethnographic fieldwork was conducted by Claus Bossen; summary of quantitative survey and analysis of system performance data were conducted by Flemming Witt Udsen.
The authors have no conflicts of interest. Summary points What was already known Evaluation during the shakedown phase of implementation is difficult, but may be necessary. Few evaluations of comprehensive EHRs have been conducted. Comprehensive EHR systems are difficult to evaluate, owing to their complexity, and because various professions use them. Because EHRs are complex, mixed-methods approaches are recommended. The updated DeLone and McLean framework has been used to measure information systems
Sincere thanks to everyone who took time and effort to contribute to the evaluation.
