Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation.

Publisher: Elsevier Country of Publication: United States NLM ID: 100970413 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1532-0480 (Electronic) Linking ISSN: 15320464 NLM ISO Abbreviation: J Biomed Inform Subsets: MEDLINE

Publication: Orlando : Elsevier
Original Publication: San Diego, CA : Academic Press, c2001-

Information Technology* ; Medical Informatics*; Commerce ; Electronic Health Records ; Humans ; Technology

Objective: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation.
Methods: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies.
Results: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment.
Discussion: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation.
Conclusion: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.
(Copyright © 2022 The Author(s). Published by Elsevier Inc. All rights reserved.)

Am J Prev Med. 2016 Nov;51(5):843-851. (PMID: 27745684)
J Am Med Inform Assoc. 2020 Aug 1;27(8):1225-1234. (PMID: 32719880)
Int J Med Inform. 2020 Oct;142:104247. (PMID: 32871491)
Child Youth Serv Rev. 2014 Apr 1;39:177-182. (PMID: 24729650)
J Biomed Inform. 2018 Sep;85:1-9. (PMID: 30017975)
Healthcare (Basel). 2018 Aug 31;6(3):. (PMID: 30200307)
J Med Internet Res. 2019 Feb 01;21(2):e12902. (PMID: 30707097)
Methods Inf Med. 2021 Jun;60(S 01):e32-e43. (PMID: 33975376)
PLoS One. 2015 Aug 07;10(8):e0134977. (PMID: 26252889)
Int J Med Inform. 2017 Jan;97:195-209. (PMID: 27919378)
J Healthc Qual. 2020 Sep/Oct;42(5):e66-e74. (PMID: 31923009)
J Hosp Med. 2018 Feb 8;13(8):531-536. (PMID: 29444195)
BMJ Qual Saf. 2016 Dec;25(12):986-992. (PMID: 26369893)
J Biomed Inform. 2005 Feb;38(1):75-87. (PMID: 15694887)
J Am Med Inform Assoc. 1994 Jan-Feb;1(1):28-33. (PMID: 7719785)
Methods Inf Med. 2012;51(2):122-30. (PMID: 22311125)
JAMIA Open. 2020 Sep 08;3(3):326-331. (PMID: 33215066)
Int J Technol Assess Health Care. 2012 Jan;28(1):44-51. (PMID: 22617736)
Proc AMIA Symp. 2002;:265-9. (PMID: 12463828)
J Hosp Med. 2015 Dec;10(12):780-6. (PMID: 26218366)
Open Forum Infect Dis. 2020 Oct 19;7(11):ofaa497. (PMID: 33269294)
Int J Med Inform. 2004 Jun 30;73(6):479-91. (PMID: 15171977)
Pediatrics. 2017 Mar;139(3):. (PMID: 28196932)
Milbank Q. 2010 Jun;88(2):256-76. (PMID: 20579285)
Stud Health Technol Inform. 2015;216:406-9. (PMID: 26262081)
Yearb Med Inform. 2006;:20-5. (PMID: 17051290)
Implement Sci. 2019 Jan 5;14(1):1. (PMID: 30611302)
Ann Intern Med. 2020 Jun 2;172(11 Suppl):S73-S78. (PMID: 32479174)
Stud Health Technol Inform. 2004;107(Pt 2):1323-7. (PMID: 15361029)
J Am Coll Radiol. 2021 Mar;18(3 Pt A):413-424. (PMID: 33096088)
Int J Med Inform. 2004 Nov;73(11-12):781-95. (PMID: 15491929)
J Am Med Inform Assoc. 2021 Feb 15;28(2):197-208. (PMID: 33587750)
Ann Intern Med. 2020 Jun 2;172(11 Suppl):S101-S109. (PMID: 32479177)
JAMA. 2021 Jun 1;325(21):2149-2150. (PMID: 33818587)
Circ Cardiovasc Qual Outcomes. 2020 Feb;13(2):e006286. (PMID: 32069091)
Implement Sci. 2021 Apr 1;16(1):34. (PMID: 33794956)
JAMA. 2005 Mar 9;293(10):1197-203. (PMID: 15755942)
J Med Internet Res. 2020 Oct 29;22(10):e19676. (PMID: 33118943)
Implement Sci. 2013 Dec 01;8:139. (PMID: 24289295)
Int J Med Inform. 2008 Jun;77(6):386-98. (PMID: 17964851)
J Am Med Inform Assoc. 2016 Sep;23(5):899-908. (PMID: 26911829)
Cell Syst. 2015 Jul;1(1):8-13. (PMID: 26339683)
Implement Sci. 2018 Jun 26;13(1):88. (PMID: 29941011)
BMC Health Serv Res. 2019 Nov 21;19(1):868. (PMID: 31752846)
J Biomed Inform. 2001 Feb;34(1):52-66. (PMID: 11376543)
Yearb Med Inform. 2016 Nov 10;(1):61-69. (PMID: 27830232)
Implement Sci. 2009 Aug 07;4:50. (PMID: 19664226)
Int J Med Inform. 2011 Dec;80(12):815-27. (PMID: 21920809)
AMIA Annu Symp Proc. 2017 Feb 10;2016:1179-1188. (PMID: 28269915)
AMIA Annu Symp Proc. 2020 Mar 04;2019:562-571. (PMID: 32308850)
Adm Policy Ment Health. 2011 Jan;38(1):4-23. (PMID: 21197565)
J Am Med Inform Assoc. 2015 Jan;22(1):223-35. (PMID: 25324556)
Implement Sci. 2019 Jun 6;14(1):58. (PMID: 31171014)
Ochsner J. 2014 Summer;14(2):195-202. (PMID: 24940129)
BMC Psychol. 2015 Sep 16;3:32. (PMID: 26376626)
JAMIA Open. 2020 Apr 21;3(2):261-268. (PMID: 32734167)
J Am Med Inform Assoc. 2021 Jun 12;28(6):1298-1307. (PMID: 33619519)
J Hosp Med. 2016 May;11(5):348-54. (PMID: 26843272)
BMJ. 2017 Mar 6;356:i6795. (PMID: 28264797)
Transl Behav Med. 2022 Feb 16;12(2):187-197. (PMID: 34424342)
Am J Public Health. 1999 Sep;89(9):1322-7. (PMID: 10474547)
Pediatrics. 2004 Jul;114(1):297-316. (PMID: 15231951)
Int J Med Inform. 2009 Jan;78(1):1-9. (PMID: 18930696)
JAMIA Open. 2021 Jul 31;4(3):ooab041. (PMID: 34345802)
JAMA. 1996 Aug 28;276(8):637-9. (PMID: 8773637)
Value Health. 2013 Mar-Apr;16(2):231-50. (PMID: 23538175)
J Biomed Inform. 2008 Dec;41(6):982-90. (PMID: 18462999)
Healthc Q. 2010;13 Spec No:53-8. (PMID: 20959731)
JAMA Intern Med. 2021 Aug 1;181(8):1065-1070. (PMID: 34152373)
J Med Internet Res. 2020 Aug 6;22(8):e18388. (PMID: 32759098)
J Biomed Inform. 2011 Dec;44(6):1056-67. (PMID: 21867774)
Adm Policy Ment Health. 2011 Mar;38(2):65-76. (PMID: 20957426)
Int J Med Inform. 2003 Sep;71(2-3):125-35. (PMID: 14519405)
JAMA Netw Open. 2019 Nov 1;2(11):e1915343. (PMID: 31730181)
J Biomed Inform. 2004 Feb;37(1):56-76. (PMID: 15016386)
Int J Med Inform. 2001 Nov;64(1):39-56. (PMID: 11673101)
J Med Internet Res. 2017 Nov 01;19(11):e367. (PMID: 29092808)
Ann Intern Med. 2014 Jan 7;160(1):48-54. (PMID: 24573664)
BMC Med Inform Decis Mak. 2014 May 29;14:43. (PMID: 24884588)
Int J Technol Assess Health Care. 2020 Jun;36(3):204-216. (PMID: 32297588)
Int J Med Inform. 2008 Jun;77(6):377-85. (PMID: 17904898)
Int J Med Inform. 2005 Dec;74(11-12):908-16. (PMID: 16099711)
J Am Med Inform Assoc. 2003 Nov-Dec;10(6):523-30. (PMID: 12925543)
J Am Med Inform Assoc. 2018 May 1;25(5):496-506. (PMID: 29045651)
NPJ Digit Med. 2019 May 13;2:38. (PMID: 31304384)

R18 HS026198 United States HS AHRQ HHS; T15 LM007124 United States LM NLM NIH HHS

Keywords: Clinical decision support; Evaluation framework; Health information technology; Health technology assessment; Human factors engineering; Implementation science

Date Created: 20220215 Date Completed: 20220316 Latest Revision: 20230302

20240105

PMC8959015

10.1016/j.jbi.2022.104014

35167977