View source for Journal:An extract-transform-load process design for the incremental loading of German real-world data based on FHIR and OMOP CDM: Algorithm development and validation

{{Infobox journal article
|name         = 
|image        = 
|alt          = <!-- Alternative text for images -->
|caption      = 
|title_full   = An extract-transform-load process design for the incremental loading of German real-world data based on FHIR and OMOP CDM: Algorithm development and validation
|journal      = ''JMIR Medical Informatics''
|authors      = Henke, Elisa; Peng, Yuan; Reinecke, Ines; Zoch, Michéle; Sedlmayr, Martin; Bathelt, Franziska
|affiliations = Technische Universität Dresden
|contact      = Email: elisa dot henke at tu dash dresden dot de
|editors      = Lovis, Christian
|pub_year     = 2023
|vol_iss      = '''11'''
|at           = e47310
|doi          = [https://doi.org/10.2196/47310 10.2196/47310]
|issn         = 2291-9694
|license      = [http://creativecommons.org/licenses/by/4.0/ Creative Commons Attribution 4.0 International]
|website      = [https://medinform.jmir.org/2023/1/e47310 https://medinform.jmir.org/2023/1/e47310]
|download     = [https://medinform.jmir.org/2023/1/e47310/PDF https://medinform.jmir.org/2023/1/e47310/PDF] (PDF)
}}
{{ombox	 
| type      = notice	 
| image     = [[Image:Emblem-important-yellow.svg|40px]]	 
| style     = width: 500px;	 
| text      = This article should be considered a work in progress and incomplete. Consider this article incomplete until this notice is removed.	 
}}
==Abstract==
'''Background''': In the Medical Informatics in Research and Care in University Medicine (MIRACUM) consortium, an IT-based [[Clinical trial registry|clinical trial recruitment]] support system was developed based on the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM). Currently, OMOP CDM is populated with German [[Fast Healthcare Interoperability Resources]] (FHIR) data using an extract-transform-load (ETL) process, which was designed as a bulk load. However, the computational effort that comes with an everyday full load is not efficient for daily recruitment.

'''Objective''': The aim of this study is to extend our existing ETL process with the option of incremental loading to efficiently support daily updated data.

'''Methods''': Based on our existing bulk ETL process, we performed an analysis to determine the requirements of incremental loading. Furthermore, a literature review was conducted to identify adaptable approaches. Based on this, we implemented three methods to integrate incremental loading into our ETL process. Lastly, a test suite was defined to evaluate the incremental loading for data correctness and performance compared to bulk loading.

'''Results''': The resulting ETL process supports bulk and incremental loading. Performance tests show that the incremental load took 87.5% less execution time than the bulk load (2.12 minutes compared to 17.07 minutes) related to changes of one day, while no data differences occurred in OMOP CDM.

'''Conclusions''': Since incremental loading is more efficient than a daily bulk load, and both loading options result in the same amount of data, we recommend using bulk load for an initial load and switching to incremental load for daily updates. The resulting incremental ETL logic can be applied internationally since it is not restricted to German FHIR profiles.

'''Keywords''': extract-transform-load, ETL, incremental loading, OMOP CDM, FHIR, interoperability, Observational Medical Outcomes Partnership Common Data Model; Fast Healthcare Interoperability Resources 

==Introduction==
===Background and significance===

==References==
{{Reflist|colwidth=30em}}

==Notes==
This presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added.

<!--Place all category tags here-->
[[Category:LIMSwiki journal articles (added in 2023)]]
[[Category:LIMSwiki journal articles (all)]]
[[Category:LIMSwiki journal articles on clinical research]]
[[Category:LIMSwiki journal articles on data analysis]]
[[Category:LIMSwiki journal articles on health informatics]]