The Working Group is working on a bibliography of the some important papers and books on the evaluation of health information systems. Here are first results, ordered by:
Do you have any suggestions for the bibliography? Please send the complete references, together with a 3-line abstract (!), to elske.ammenwerth@umit.at. Thank you.
Anderson J G, Aydin C E, Jay S J, editors. Evaluating Health Care Information Systems - Methods and Applications. London, New Delhi: Sage Publications; 1994.
Van Bemmel JH and Musen M (eds.), Handbook of Medical Informatics. Springer Verlag, Houten Diegem, 1997.
Chapter 30: Evaluation of clinical information systems. Overview of evaluation approaches, methods and guidance on how to perform an evaluation study. Additionally Chap 32: Costs and benefits of information systems, chap 33: Security in medical information systems and chap 34: Standards in health care informatics and telematics in Europe.
Brender J. Methodology for Assessment of Medical IT-based Systems in an Organisational Context. Studies in Health Technology and Informatics, volume 42. Amsterdam: IOS Press; 1997.
Brender J. Handbook of Evaluation Methods for Health Informatics. Academic Press: 2005.
Flagle C, Gremy F and Perry S (eds.), Assessment of medical informatics technology. Proceedings of the IMIA-ISTAHC Joint Working Conference, Montpellier, France, Oct 1990. Editions ENSP, Rennes, 1991.
Many interesting articles covering topics: Dimensions of medical technology assessment, status of assessment in medical informatics, domains of applications of tecghnology assessment applied to medical informatics.
Friedman CP, Wyatt J C. Evaluation Methods in Medical Informatics. New York: Springer; 1997.
Van Gennip EMSJ, Talmon J, editors. Assessment and Evaluation of Information Technologies in Medicine. Studies in Health Technology and Informatics, volume 17. Amsterdam: IOS Press; 1995.
Morgan Mayhew DJ. The Usability Engineering Lifecycle: A Practitioner's Handbook for User Interface Design. Kaufman Publishers; 1999.
Nielsen J, Usability engineering, Academic Press, USA, 1993
A classic in usability testing. Presents methods and issues to be considered in usability testing, including much referenced Nielsen's 10 usability rules.
Rossi PH and Freeman HE, Evaluation. A systematic approach. Fifth edition. Sage Publications, USA, 1993.
Presents a broad set of activities related to evaluation: Evaluation research, role of evaluators, tailoring evaluations, strategies for impact assessment, randomized designs, measuring efficiency and social context of evaluation. The book is targeted for evaluation of social programs, but even in IT and health care area we can learn about presented principles and theories of evaluation.
Spil TAM, Schuring RW (Hrsg.): E-Health Systems - Diffusion and Use. The Innovation, The User and the Use IT Model. Hershey: Idea Group Publishing. 212-236. ISBN 1-591-40424-X.
Contains several articles on quantitative and qualitative evaluation of e-health systems as well as a lot of information on information systems adoption theories.
Buss B, Friesdorf W: Sicherheit im intensivmedizinischem Arbeitsfeld: ein Optimierungsproblem zukünftiger Entwicklungen. In: M. Grandt (Hrsg.): Verlässlichkeit der Mensch-Maschine-Interaktion. DGLR-Bericht 2004 (3), S. 107 - 120. Bonn: Deutsche Gesellschaft für Luft- und Raumfahrt. 2004.
Brender J; Talmon J, Nykänen P, McNair P, Demeester M and Beuscart R, On the evaluation of system integration. In: van Gennip EMSJ and Talmon JL (eds), Assessment and evaluation of information technologies in medicine. IOS Press, Amsterdam, 1995, 189-208.
The VATAM guidelines are applied on evaluation of the integration process. The integration process aims at integration of several decision support systems developed in EU TAP R&D projects with the hospital information system in a real user environment.
Clarke K, O'Moore R, Smeets R, Talmon J, Brender J, McNair P, Nykänen P, Grimson J, Barber B. A Methodology for Evaluation of Knowledge-Based Systems in Medicine. In: van Bemmel JH, McCray AT, eds. Yearbook of Medical Informatics 1995:513-527.
Evaluation is critical to the development and successful integration of knowledge-based systems into their application environment. This is of particular importance in the medical domain- not only for reasons of safety and correctness, but also to reinforce the users' confidence in these systems. This paper describes an iterative, four-phased development evaluation cycle covering the following areas: (i) early prototype development, (ii) validity of the system, (iii) functionality of the system, and (iv) impact of the system.
Nykänen P and Karimaa E, Evaluation during design of a regional seamless network of social and health care services - information technology perspective. In: Surjan G, Engelbrecht R and McNair P (eds.), Health data in the information society. Technology and Informatics 90, IOS Press, Amsterdam 2002, 539 -542.
Reports results from a constructive study applying the VATAM evaluation approach. Models of health care processes were studied and the needs and requirements of the stakeholders. The results show the importance to focus on the regional information model that would support interoperability of the legacy systems and decribe concepts and their relations. The study also emphasises application of theoretical frameworks of health informatics in design and development of health information systtems and integration of constructive evaluation with the development process.
Special Issue Int Journal of Medical Informatics: Organisational issues and technology assessment in health care, vol 56, nos 1-3, 1999
Reports papers and discussions from the Joint IMIA Working Conference of Working Groups 13 (Organisational Impacts of Medical Informatics) and 15 (Technology Assessment in Health Care), Helsinki, February 1998.
Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care: Trends in evaluation research 1982 - 2002. Methods of Information in Medicine. 2005;44:44-56.
Analysed 1.035 abstracts of evaluation studies published in Medline between 1982 and 2002 and describes developments with regard to e.g. methods, evaluation approaches, setting, study design, and evaluation criteria.
Barbour R. S. The case for combining qualitative and quantitative approaches in health services research. 1999; 4(1):39-43.
Argues in favour of a combination of qualitative and quantiative methods in evaluation studies and discusses various ways how to combine both approaches in a multi-method approach.
Brender J. Trends in Assessment of IT-Based Solutions in Healthcare and Recommendations for the Future. Int. J. of Med. Informatics 1998;52(1-3):217-227.
State-of-the-Art with respect to assessment of IT-based solutions in Healthcare is discussed. Special emphasis is put on the human and organisation-centred perspective at the development of IT-based solutions. Based on this and basic conditions for system analysis and design reported in the literature, requirements for a methodology for user-driven, constructive assessment during the entire life-cycle of the IT-based system are synthesised.
Brender J. Methodology for Constructive Assessment of IT-based Systems in an Organisational Context. Int. J. of Med. Informatics 1999;56:67-86.
Even if painted in black & white, there is no doubt that the assumptions for application of traditional approaches for user requirements specification are more or less unfulfilled. This indicates a need for evolutionary system development combined with constructive assessment throughout the life-cycle of an IT-based solution. A methodology for constructive technology assessment is presented, which 1) covers the entire system life-cycle; 2) have users from the application domain of the future system - or their representatives - as the target users of the assessment methodology; 3) enables constructive assessment during the development of an IT-based solution; 4) is applicable independently of the system development approach; and 5) provides users of the IT-based solution with information enabling them to decide whether or not to take the system into real-life clinical usage.
Egmont-Petersen M, Talmon J, Brender J, McNair P. On the Quality of Neural Net Classifiers. AIM 1994;6 (5):359-381.
This paper describes several concepts and metrics that may be used to assess various aspects of the quality of neural net classifiers. Each concept describes a property that may be taken into account by both designers and users of neural net classifiers when assessing their utility. Besides metrics for assessment of the correctness of classifiers we also introduce metrics that address certain aspects of the misclassifications. We show the applicabi1ity of the introduced quality concepts for selection among several neural net classifiers in the domain of thyroid disorders.
Fraser JM, Smith PJ. A catalogue of errors. Int J Man-Machine Studies 1992;37:265-307.
This paper reviews and discusses various biases and errors that have been described in the literature by comparing human behaviour to the norms of probability, causal connection and logical deduction.
van Gennip EM, Bakker AR: Assessment of effects and costs of information systems. Int J Biomed Comput 1995;39:67-72.
Description of a case study on assessment, where different characteristics of the involved departments - and implied second order effects - are carefully addressed, while handling the confounding effects within matching control groups during the design and interpretation of the study outcome.
Goodhue DL. Understanding user evaluations of information systems. Manage Sci 1995; 41(12):1827-44.
Presents his task-technology-fit theory (TTF) which describes that users can both be supported or hindered by information technology, based on the 'fit' between user characteristics, tasks characteristics, and technology characteristics.
Goodman C. It's time to rethink health care technology assessment. Int J Technol Assess Health Care 1992;8:335-358.
This paper puts health care technology assessment into perspective, discussing its concepts, role and a number of it key issues, among others the role of RCT and the distinction between efficacy, effectiveness and efficiency measures.
Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998; 316:1959-61.
Analyses recent problems in quantiative evaluation of information technology in health care, and argues in favour of a more qualitative, multi-method evaluation approach.
Jaeschke R, Sackett DL. Research methods for obtaining primary evidence. Int J Technology Assessment 1989;5:503-519.
Some of the methods and the rationale for their use in assessment of therapeutic and diagnostic technologies are discussed in the perspective of dealing with placebo effects, confounders and biases. The benefits and risks of RCT, case control and cohort studies are reviewed.
Kaplan B, Duchon D. Combining qualitative and quantitative approaches in information systems research: a case study. MIS Quarterly 1988; (4):571-86.
Reports on a case study on the combination of quantiative and qualitative methods in a longitudinal study of an information system. Thorough description of the chances, but also the practical problems, in this triangulation approach.
Klein HK, Myers MD. A Set of Principles for Conducting and Evaluating Field Studies in Information Systems. MIS Quarterly 1999; 23(1):67-93.
Discusses the conduct and evaluation of interpretive (qualitative) field studies in the evaluation of information systems. He illustrates the usefulness of the principles by three field studies.
Moehr J. R. Evaluation: salvation or nemesis of medical informatics? Comput Biol Med 2002; 32(3):113-25.
Discusses the shortcomings of the objectivist evaluation approach, and then presents the subjectivist approach as well as extensions to both approaches to circumvent the analysed problems. .
Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P. Qualitative research methods in health technology assessment: a review of literature. Health Technology Assessment 2001; 3(16): 1 - 276.
Comprehensive analysis of theories and methods of qualitative research, and their strengths and weaknesses.
Nykänen P, Boran G, Pince H, Clarke C, Yearworth M, Willems JL and O'Moore R. Interpretative reporting and alarming based on laboratory data. Clinica Chimica Acta 222; 1-2; 1993, 37-48
The paper reports development and evaluation of post-analytical systems for interpretation of laboratory results in the domains of clinical chemistry and intensive care. The developed systems are evaluated from the perspectives of the users of post-analytical services.
Nykänen P, Chowdhury S and Wigertz O, Evaluation of decision support systems in medicine. Int J Computer Methods and Programs in Biomedicine 34; 2/3; 1991, 229 - 238. Reprinted in: van Bemmel JH and McCray T (eds.), IMIA Yearbook of Medical Informatics 1992, Schattauer, Heidelberg, 1992, 301-310
The paper presents an approach to evaluation of medical decision supportsystems. The approah emphasises the need to approach evaluation from three perspectives: Knowledge acquisition, system development lifecycle and user-system integrated behavior. The three perspectives help to manage the development of decision support systems through considering the development divided into phases and assessing the validity at each phase.
Nykänen P and Nuutila P; Validation and evaluation of a system for thyroid disorders. Int J Expert Systems with Applications 3; 2; 1991, 289-295.
The paper reports development ane evaluation of a decision support system for interpretation of thyroid disorders. The purpose of the evaluation was to assess the validity of the system in the real user environment and to find specifications for the next generation interpretative systems. The evaluation methodology applied relates evaluation with the system development lifecycle comprising of a series of iterative phases and emphasising the importance of feedback to health professionals and end users. Special emphasis in this study is on the system's validity in application and on acceptability of the system by users.
Nykänen P, Enning J, Talmon J, Hoyer D, Sanz F, Thayer C, Roine R, Vissers M and Eurlings M, Inventory of validation approaches in selected health telematics projects. Int J of Medical Informatics 56; 1-3; 1999, 87-96
The paper presents results from an inventory on evaluation methods and approaches applied in health telematics projects. The purpose of the inventory was to analyse the methodologies and their application assumptions in order to identify possibilities for harmonisation and consolidation. The inventory was made in five dimensions: IT-development, quality, user, technology assessment and marketing. The results show that there exist possibilities to synthetisise methodologies and to provide practical guidance and support for projects that are developing health telematics applications.
Sittig D F. Work Sampling: A Statistical Approach to Evaluation of the Effect of Computers on Work Patterns in The Healthcare Industry. Meth Inform Med 1993; 32(2):167-174.
Techniques have been developed to evaluate effects of computer systems on the work patterns of health-care workers including: time-motion analysis, subjective evaluations, review of departmental statistics, personal activity records, and work-sampling. This study reviews these techniques, discusses both positive and negative aspects, and presents a detailed step-by-step description of work-sampling.
Wyatt J and Spiegelhalter D, Evaluating medical expert systems: What to test and how? Int J Medical Informatics 15, 1990, 205-217
This paper presents a methodology to evaluate clinical decision support systems. A comprehensive discussion on what to test and methods how to test.
Review papers
van der Loo RP, van Gennip EMSJ, Bakker AR, Hasman A, Rutter FFH, Evaluation of automated information systems in health care:An approach to classifying comparative studies. Int J Computer Methods and Programs in Biomediicne 48; 1995, 45-52.
In this paper 76 evaluative studies of IT systems in health care were analysed for the criteria used in evaluation. The three most often studied criteria were performance of the user when using the system, time changes in personnel workload and the performance of the information system itself. Only 10 of the studied 76 studies had included some kind of economic evaluation.
Lorenzi NM, Riley RT, Blyth AJC, Southon G, Dixon BJ. Antecedents of the People and Organizational Aspects of Medical Informatics: Review of the Literature. 1997; 4(2):79-93.
People and organizational issues are critical in both implementing medical informatics systems and in dealing with the altered organizations that new systems often create. This article reviews the behavioral and business referent disciplines that can potentially contribute to improved implementations and on-going management of change in the medical informatics arena
Kaplan B, Addressing organisational issues into the evaluation of medical systems. Journal of American Medical Informatics Association 4; 2; 1997, 94-101.
These two (Kaplan, Lorenzi et al. ) are really not evaluation papers, but they focus on consideration of organisational issues in developing and evaluating medical information systems. These issues are often forgotten and therefore these two papers might be important to emphasise the need to see health information systems as socio-technical systems.
Kaplan B. Evaluating informatics applications - some alternative approaches: theory, social interactionism, and call for methodological pluralism. International Journal of Medical Informatics 2001; 64:39-56.
A review of evaluation literature concerning clinical decision support systems indicates that randomized controlled clinical trials (RCTs) are the 'gold standard' for evaluation. This paper critiques RCT and experimental evaluation approaches and presents alternative approaches to evaluation that address questions outside the scope of the usual RCT and experimental designs.
Reisman Y, Computer-based clinical decision aids. A review of methods and assessment of systems. Int J of Medical Informatics 21; 3; 1996, 179-197.
A review article on developed decision support systems and their evaluation. Useful, but focuses only on decision support systems.
Brender J, Talmon J, McNair P. Framework for quality assessment of knowledge. In: Gordon C, Christensen JP, eds. Health Telematics for Clinical Guidelines and Protocols. Amsterdam: IOS Press. Studies in Health Technology and Informatics 1995;16:219-227. (Reprint from: Albrechtsen H, Oernager S, eds. Knowledge Organisation and Quality Management. Frankfurt/Main:Indeks Verlag. Advances in Knowledge Organisation 1994;4:42-51).
One of the key issues in the development (and subsequent application) of medical knowledge - be it in terms of a KBS or otherwise - is the assessment of its quality. The authors present a framework for how to manage and make measurable the quality of the semantic as well as pragmatic aspects of the knowledge embedded in classification models during the development of such models.
Brender J. Review of pitfalls and perils in evaluation of IT-Based Solutions (MUP-IT Project). In: Brown A, Remenyi D, eds. Ninth European Conference on Information Technology Evaluation, Reading:MCIL, ISBN 0-9540488-5-7. 2002:79-85.
The scope is to provide a means for meta-analysis of assessment studies with emphasis on methodological and methodical issues, specifically on the identification of pitfalls and perils in assessing medical IT-based solutions. In the present contribution, the foundation in terms of a list of experimental pitfalls and perils is established on the basis of the literature on experimental issues from medical science, natural science and social science.
Brender J, McNair P. Tools for Constructive Assessment of bids to a Call for Tender - some experiences. In: Surján G, Engelbrecht R, McNair P, eds.Health Data in the Information Society, Proceedings of the MIE2002. Amsterdam:IOS Press. Studies in Health Technology and Informatics 2002;90:527-532.
The paper describes the experiences from a full-scale case study applying a number of novel assessment techniques for selecting among income bids at a call for tender, based on a User Requirements Document that comprises non-prescriptive, goal-oriented requirements.
Brender J, Schou-Christensen J, McNair P. A case study on Constructive Assessment of bids to a Call for Tender. In: Surján G, Engelbrecht R, McNair P, eds. Health Data in the Information Society, Proceedings of the MIE2002. Amsterdam:IOS Press. Studies in Health Technology and Informatics 2002;90:533-538.
A case study based on constructive assessment methodology is taking place at the purchase of a Laboratory Information System. The paper describes the strategy and approach for the selection process, together with some results and lessons learned.
Enning J, Talmon J, Nykänen P, Marz J, SAnz F and Thayer C, VATAM - Developing consensus in validation of health telematics applications. In: Pappas C, Maglaveras N and Scherrer J-R (eds.), Medical Informatics Europe '97, Technology and informatics 43, IOS Press, Amsterdam 1997, 771-775.
The paper introduces the VATAM-approach through desription of the three project phases: Inventory phase, dissemination phase and experiences phase. The results of the inventory phase are reported as a framework for validation: Actors, Use, Phase. Actors describe the stakeholders for validation and can be found at three levels: Macro, meso and micro. Use refers to the type of the system under asessment, and phase refers to the application lifecycle.
Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. Proceedings of AMIA Annual Fall Symposium; 1997. p. 218-222.
Triangulation of methods in this contribution clearly indicates the problem of interviewees' biased judgement when applying a questionnaire approach and thus illustrates some of the psychological biases in questionnaires.
Wyatt J, The evaluation fo clinical decision support systems: A discussion of the methodology ised in the ACORN project. In: Fox J, Fieschi M and Engelbrecht R (eds.), Proceedings of the Artificial Intelligence in Medicine 1987, Lecture Notes in Medical Informatics 33, Springer Verlag, Berlin, 1987, 15-24.
This paper presents an approach where evaluation of an information system is compared the methodology of evaluating drugs. A discussion of the evaluation findings is included.
Wyatt J, Spiegelhalter D. Field trials of medical decision-aids: potential problems and solutions. In: Clayton P, editor. Proceedings of the Annual Symposium on Computer Applications in Medical Care; 1991. p. 3-7.
The authors review a number of biases related to the assessment of medical decision support systems based on cases from the literature.
Olve N-G, Vimarlund V. Elderly Healthcare, Collaboration and ICT. Enabling the Benefit of an Enabling Technology. Vinnova Report VR 2006:05, Linköping University. 2006.
Various reports of the VATAM project (VATAM = Validation of Health Telematics Applications) are available at the VATAM homepage.
O'Moore R, Clarke K, Smeets R, Brender J, Nykänen P, McNair P, Grimson J, Barber B. Items of relevance for evaluation of knowledge-based systems and influence from domain characteristics. Technical Report of the KAVAS (A1021) AIM Project 1990; EM-1.1. (publicly available, contact Jytte Brender).
This report comprises a comprehensive review of the literature on various items considered to be relevant for the evaluation of knowledge based/expert systems (KBS) and influence from the domain characteristics.
O'Moore R, Clarke K, Brender J, McNair P, Nykänen P, Smeets R, Talmon J, Grimson J, Barber B. Methodology for evaluation of knowledge based systems. Technical Report of the KAVAS (A1021) AIM Project 1990; EM-1.2. (publicly available).
This report describes a methodology for the evaluation of knowledge-based systems. The approach has been to adopt a four-phased development and evaluation process, from a Preliminary Exploration through to Final Evaluation of the system's impact. Each of these phases has also its own methodology. The relevant literature characteristically describes issues of relevance in reasonable detail but contains only limited information on the appropriate metrics (measurements) and even less on methods for evaluation. This report comprises a synthesis of a methodology.
Beuscart R, Blassel N, Roussel D, Beuscart M-C, Menager I, Bossard B, Brender J, McNair P, Demeester M. Evaluation of the Integration Exercise: Technical, User and Clinical Aspects. Technical Report of the ISAR (A2052) AIM Project. 1996; Deliverable 8. (publicly available, contact Jytte Brender)
A case study of evaluation applying a slightly refined KAVAS's Evaluation Methodology, based on EM-1.2.
Brender J. Methodological and methodical perils and pitfalls within assessment studies performed on IT-based solutions in healthcare. Technical Report of the MUP-IT Project. Aalborg University, Virtual Centre for Health Informatics, April 2002 (publicly available, contact Jytte Brender).
The scope of the report is to provide a means for performing meta-analysis of assessment studies on IT-based solutions, focused at the validity of such studies, based on a review of pitfalls and perils reported in the scientific literature. The emphasis is on methodological and methodical issues within assessment studies and the interpretation of their outcome, and specifically on identifying analogues in assessing IT-based solutions. Example cases were identified within the medical informatics literature on assessment of IT-based systems and solutions for most of the candidate error types.
Hailey D, Roine R and Ohinmaa A, Assessments of telemedicine applications - an update. Alberta Foundation for Medical Research and Finnish Office for Health Technology Assessment, September 2001. www.ahfmr.ab.ca/hta/hta-publications/joint/telemedicine-update.pdf (26.11.2002). (paper copy available from Pirkko Nykanen)
The report presents results of a study where a systematic review of telemedicine assessments based on searches of the electronic data bases (bet Nov 1998 - Dec 2000) identified 38 scientifically credible studies. These studies are further classified and analysed in the report based on two classification criteria. First criteria considers studies in terms of area of application, objectives, approach taken, the setting and results and conclusions, including any economic analysis. The second criteria provides further context for the studies considering their potential effects on decision making in respect to telemedicine services, any methodological limitations and suggestions made for future work. The overall results show that useful data are emerging on some telemedicine applications, but good quality studies are scarce and generalisability of most assessment findings may be limited.
Nykänen P (ed.), Issues in evaluation of computer-based support to clinical decision making. Report of the SYDPOL-5 Working Group. Oslo University, Institute of Informatics, Research Report 127, 1990.
The report gives health professionals and users of computer-based support systems information on the importance of evaluation, and guidance on how to perform evaluation studies. The report includes guidance on how to evaluate the system's objective and role, sources and characteristics of knowledge, inference mechanisms, user interfaces and man-machine interaction and environmental impacts. Additionally the report introduces evaluation methods and gives guidance for a user on how to design and perform an evaluation study.
A list of original publications by Jeremy Wyatt on evaluation theory and practice.
Collection of evaluation papers from Maryati Yusof
Evaluation Database including health IT evaluation studies since 1982 up to now
Health IT Survey Compendium of AHRQ