Journal:Mapping hierarchical file structures to semantic data models for efficient data integration into research data management systems

Full article title	Mapping hierarchical file structures to semantic data models for efficient data integration into research data management systems
Journal	Data
Author(s)	tom Wörden, Henrik; Spreckelsen, Florian; Luther, Stefan; Parlitz, Ulrich; Schlemmer, Alexander
Author affiliation(s)	Indiscale GmbH, Max Planck Institute for Dynamics and Self-Organization, Georg-August-Universität, German Center for Cardiovascular Research (DZHK) Göttingen, University Medical Center Göttingen
Primary contact	Email: alexander dot schlemmer at ds dot mpg dot de
Editors	Sedig, Kamran
Year published	2024
Volume and issue	9(2)
Page(s)	24
DOI	10.3390/data9020024
ISSN	2306-5729
Distribution license	Creative Commons Attribution 4.0 International
Website	https://www.mdpi.com/2306-5729/9/2/24
Download	https://www.mdpi.com/2306-5729/9/2/24/pdf (PDF)

Abstract

Although other methods exist to store and manage data using modern information technology (IT), the standard solution is file systems. Therefore, maintaining well-organized file structures and file system layouts can be key to a sustainable research data management infrastructure. However, file structures alone lack several important capabilities for FAIR (findable, accessible, interoperable, and reusable) data management, the two most significant being insufficient visualization of data and inadequate possibilities for searching and obtaining an overview. Research data management systems (RDMSs) can fill this gap, but many do not support the simultaneous use of the file system and RDMS. This simultaneous use can have many benefits, but keeping data in an RDMS in synchrony with the file structure is challenging.

Here, we present concepts that allow for keeping file structures and semantic data models (found in RDMSs) synchronous. Furthermore, we propose a specification in YAML format that allows for a structured and extensible declaration and implementation of a mapping between the file system and data models used in semantic research data management. Implementing these concepts will facilitate the re-use of specifications for multiple use cases. Furthermore, the specification can serve as a machine-readable and, at the same time, human-readable documentation of specific file system structures. We demonstrate our work using the open-source RDMS LinkAhead (previously named “CaosDB”).

Keywords: research data management, FAIR, file structure, file crawler, metadata, semantic data model

Introduction

Data management for research is part of an active transformation in science, with effective management required in order to meet the needs of increasing amounts of complex data. Furthermore, the FAIR guiding principles [1] for scientific data—which are an elementary part of numerous data management plans, funding guidelines, and data management strategies of research organizations [2,3], requiring that research objects be more findable, accessible, interoperable, and reusable—require scientists to review and enhance their established data management workflows.

One particular focus of this endeavor is the introduction and expansion of research data management systems (RDMSs). These systems help researchers organize their data during the whole data management life cycle, especially by increasing findability and accessibility. [4] Furthermore, semantic data management approaches [5] can increase the reuse and reproducibility of data that are typically organized in file structures. As has been pointed out by Gray et al. [4], one major shortcoming of file systems is the lack of rich metadata features, which additionally limits search options. Typically, RDMSs employ database management systems (DBMSs) to store data and metadata, but the degree to which data is migrated, linked, or synchronized into these systems can vary substantially.

The import of data into an RDMS typically requires the development of data integration procedures that are tied to the specific workflows at hand. While very few standard products exist [6], in practice, mostly custom software written in various programming languages and making use of a high variety of different software packages are used for data integration in scientific environments. There are two main workflows for integrating data into RDMSs: manually inputting data (e.g., using forms [7]) or facilitating the batch import of data sets. The automatic methods often include data import routines for predefined formats, like tables in Excel or CSV format. [8,9] Some systems include plugin systems to allow for a configuration of the data integration process. [10] Sometimes, data files have to be uploaded using a web front-end [11] and are afterwards attached to objects in the RDMSs. In general, developing this kind of software can be considered very costly [6], as it is highly dependent on the specific environment. Data import can still be considered one of the major bottlenecks for the adaption of an RDMS.

There are several advantages to using an RDMS over organization of data in classical file hierarchies. There is a higher flexibility in adding metadata to data sets, while these capabilities are limited for classical file systems. The standardized representation in an RDMS improves the comparability of data sets that possibly originate from different file formats and data representations. Furthermore, semantic information can be seamlessly integrated, possibly using standards like RDF [12] and OWL. [13] The semantic information allows for advanced querying and searching, e.g., using SPARQL. [14] Concepts like linked data [15,16] and FAIR digital objects (FDO [17]) provide overarching concepts for achieving more standardized representations within RDMSs and for publication on the web. Specifically, the FDO concept aims at bundling data sets with a persistent digital identifier (PID) and its metadata to self-contained units. These units are designed to be machine-actionable and interoperable, so that they have the potential to build complex and distributed data processing infrastructures. [17]

Using file systems and RDMSs simultaneously

Despite the advantages mentioned above, RDMSs have still failed to gain a widespread adoption. One of the key problems in the employment of an RDMS in an active research environment is that a full transition to such a system is very difficult, as most digital scientific workflows are in one or multiple ways dependent on classical hierarchical file systems. [4] Examples include data acquisition and measurement devices, data processing and analysis software, and digitized laboratory notes and material for publications. The complete transition to an RDMS would require developing data integration procedures (e.g., extract, transform, load [ETL] [6,18] processes) for every digital workflow in the lab and to provide interfaces for input and output to any other software involved in these workflows.

As files on classical file systems play a crucial role in these workflows, our aim is to develop a robust strategy to use file systems and an RDMS simultaneously. Rather than requiring a full transition to an RDMS, we want to make use of the file system as an interoperability layer between the RDMS and any other file-based workflow in the research environment.

There are two important tasks that need to be solved and that are the main focus of this article:

1. There must be a method to keep data and metadata in the RDMS synchronized with data files on the file system. Using that method, the file system can be used as an interoperability layer between the RDMS and other software and workflows. Our approach to solving this issue is discussed in detail in the results section. One key component of the synchronization method is defining the concept of "identity" for data in the RDMS, also discussed in the results section about identifiables.
2. The high variety of different data structures found on the file system needs an adaptive and flexible approach for data integration and synchronization into the RDMS. We discuss our solution for this task in the results section concerning YAML, where we present a standardized but highly configurable format for mapping information from files to a semantic data model.

Apart from the main motivation, described above, we have identified several additional advantages of using a conventional folder structure simultaneous to an RDMS: standard tools for managing the files can be used for backup (e.g., rsync), versioning (e.g., git), archiving, and file access (e.g., SSH). Functionality of these tools does not need to be re-implemented in the RDMS. Furthermore, the file system can act as a fallback in cases where the RDMS might become unavailable. This methodology, therefore, increases robustness. As a third advantage, existing workflows relying on storing files in a file system do not need to be changed, while the simultaneous findability within an RDMS is available to users.

The concepts described in this article can be used independent of a specific RDMS software. However, as a proof-of-concept, we implemented the approach as part of the file crawler framework that belongs to the open-source RDMS LinkAhead (recently renamed from CaosDB). [19,20] The crawler framework is released as open-source software under the AGPLv3 license (see Appendix A).

Example data set

We will illustrate the problem of integrating research data using a simplified example that is based on the work of Spreckelsen et al. [21] This example will be used in the results section to demonstrate our data integration concepts. Examples for more complex data integration, e.g., for data sets found in the neurosciences (BIDS [22] and DICOM [23]) and in the geosciences, can be found online (see Appendix B). Although the concept is not restricted to data stored on file systems, in this example we will assume for simplicity that the research data are stored on a standard file system with a well-defined file structure layout:

ExperimentalData/

2020_SpeedOfLight/

2020-01-01_TimeOfFlight

README.md

...

2020-01-02_Cavity

README.md

...

2020-01-03

README.md

...

The above listing replicates an example with experimental data from Spreckelsen et al. [21] using a three-level folder structure:

• Level 1 (ExperimentalData) stores rough categories for data, in this data acquired from experimental measurements.
• Level 2 (2020_SpeedOfLight) is the level of project names, grouping data into independent projects.
• Level 3 stores the actual measurement folders, which can also be referred to as “scientific activity” folders in the general case. Each of these folders could have an arbitrary substructure and store the actual experimental data along with a README.md file, containing meta data.

The generic use case of integrating data from file systems involves the following sub tasks:

1. Identify the required data for integration into the RDMS. This can possibly involve information contained in the file structure (e.g., file names, path names, or file extensions) or data contained in the contents of the files themselves.
2. Define an appropriate (semantic) data model for the desired data.
3. Specify the data integration procedure that maps data found on the file system (including data within the files) to the (semantic) data in the RDMS.

A concrete example for this procedure, including a semantic data model, is provided in the results section. As previously state, there are already many use cases that can benefit from the simultaneous use of the file system and RDMS. Therefore, it is important to implement reliable means for identifying and transferring the data not only once, as a single “data import,” but also while allowing for frequent updates of existing or changed data. Such an update might be needed if an error in the raw data has been detected. It can then be corrected on the file system, with the changes needing to be propagated to the RDMS. Another possibility is that data files that are actively worked on have been inserted into the RDMS. A third-party software is used to process these files and, consequently, the information taken from the files has to be frequently updated in the RDMS.

We use the term “synchronization” here to refer to the possible insertion of new data sets and to update existing data sets in the same procedure. To avoid confusion, we want to explicitly note here that we are not referring to bi-directional synchronization. Bi-directional synchronization means that information from RDMS that is not present in the file system can be propagated back to the file system, which is not possible in our current implementation. Although ideas exist to implement bi-directional synchronization in the future, in the current work (and also the current software implementation), we focus on the uni-directional synchronization from the file system to the RDMS. The outlook for adding extensions to bi-directional synchronization will be discussed in the discussion section.

About LinkAhead

LinkAhead was designed as an RDMS mainly targeted at active data analysis. So, in contrast to electronic laboratory notebooks (ELNs), which have a stronger focus on data acquisition and data repositories, which are used to publish data, data in LinkAhead are assumed to be actively worked on by scientists on a regular basis. Its scope for single instances (which are usually operated on-premises) ranges from small work groups to whole research institutes. Independent of any RDMS, data acquisition typically leads to files stored on a file system. The LinkAhead crawler synchronizes data from the file system into the RDMS. LinkAhead provides multiple interfaces for interacting with the data, such as a web-based graphical user interface (GUI) and an application programming interface (API) that can be used for interfacing the RDMS from multiple programming languages. LinkAhead itself is typically not used as a data repository, but the structured and enriched data in LinkAhead serves as a preparation for data publication, and data can be exported from the system and published in data repositories. The semantic data model used by LinkAhead is described in more detail in the next subsection. LinkAhead is open-source software, released under the AGPLv3 license (see Appendix A).

Data models in LinkAhead

The LinkAhead data model is basically an object-oriented representation of data which makes use of four different types of entities: RecordType, Property, Record and File RecordTypes, and Properties. These four entities define the data model, which is later used to store concrete data objects, which are represented by Records. In that respect, RecordTypes and Properties share a lot of similarities with ontologies, but have a restricted set of relations, as described in more detail by Fitschen et al. [19] Files have a special role within LinkAhead as they represent references to actual files on a file system, but allow for linking them to other LinkAhead entities and, e.g., adding custom properties.

Properties are individual pieces of information that have a name, description, optionally a physical unit, and can store a value of a well-defined data type. Properties are attached to RecordTypes and can be marked as “obligatory,” “recommended,” or “suggested.” In case of obligatory Properties, each Record of the respective RecordType is enforced to set the respective Properties. Each Record must have at least one RecordType and RecordTypes can have other RecordTypes as parents. This is known as (multiple) inheritance in object-oriented programming languages.

In Figure 1, an example data model is shown in the right column in a UML-like diagram. There are three RecordTypes (Project, Person, and Experiment), each with a small set of Properties (e.g., an integer Property called “year” or a Property referring to records of type Person called “responsible”). The red lines with a diamond show references between RecordTypes, i.e., where RecordTypes are used as Properties in other RecordTypes.

Figure 1. Mapping between file structure and data model. Blue lines indicate which pieces of information from the file structure and file contents are mapped to the respective properties in the data model.

Results

We solved the issues described prior about using file systems and RDMSs simultaneously by using a modular crawler system, which is discussed extensively in this section. The main task of the crawler is to automatically synchronize information found in the file system to the semantic RDMS. The modularity of the crawler is achieved by providing a flexible configuration of synchronization and mapping rules in a human- and machine-readable YAML format. Using these configuration files (which we will refer to as CFoods), it is possible to adapt the crawler to heterogeneous use cases. (These crawler definitions will be described a bit later in this section.) We assumed that a semantic data model that is appropriate for the data structures had been created. As an example, we discuss the data model that is shown in Figure 1 in the right column.

In order to be able to synchronize information from the file system with the RDMS, a specification of the identity of the objects is needed. This will allow us to check which objects are already present in the RDMS and, therefore, need an update instead of an insert operation. Our concept, which is similar to unique keys in relational database management systems (RDBMSs), will be discussed later in this section. Our implementation of the procedure, the LinkAhead crawler, makes use of these concepts in order to integrate data into the RDMS. Based on the example we introduced in the prior section, we will illustrate in the following subsections how the information from the file system will be mapped onto semantic data in the RDMS.

The LinkAhead crawler

Figure 2 provides an overview of the complete data integration procedure with the LinkAhead crawler. In Step 1, the scanner uses the YAML crawler definition to match converters to a given file system tree. From the information that is matched, a list of LinkAhead Records with Properties is created. In Step 2, the crawler checks which of these Records are already contained in LinkAhead in order to separate the list of Records into a list of new Records and a list of changed Records. In order to complete this check, it makes use of a given definition of Identifiables. In Step 3 both lists are synchronized with the LinkAhead server, i.e., all Records from the list of new Records are inserted into LinkAhead and all Records from the list of changed Records are updated. The data model is needed to write a valid YAML crawler definition and to create the definition of Identifiables. Furthermore, it is used by the LinkAhead server directly.

Figure 2. Overview of the complete data integration procedure.

Mapping files and layouts into a data model

Suppose we have a hierarchical structure of some kind that contains certain information and we want to map this information onto our object-oriented data model. The typical example for the hierarchical structure would be a folder structure with files, but hierarchical data formats like HDF5 [24,25] files (or a mixture of both) would also fit the use case.

The prior Figure 1 illustrates what such a mapping could look like in practice using the prior stated file structure:

• ExperimentalData contains one subfolder 2020_SpeedOfLight, storing all data from this experimental series. The experimental series is represented in a RecordType called Project. The Properties “year” (2020) and “identifier” (SpeedOfLight) can be directly filled from the directory name.
• Each experiment of the series has its dedicated subfolder, so one Record of type Experiment will be created for each one. The association to its Project, which is implicitly clear in the folder hierarchy, can be mapped to a reference to the Project Record created in the previous step. Again, the Property “date” can be set from the directory name.
• Each experiment contains a file called README.md storing text and metadata about the experiment, according to the described standard. In this case, the “description” Property for the Experiment Record created in the previous step will be set from the corresponding YAML value. Furthermore, a Person Record with Properties "firstName = Florian" and "lastName = Spreckelsen" will be created. Finally the Person Record will be set as the value for the property “responsible” of the Experiment Record.

YAML definitions

Figure 3 illustrates the modular design of the crawler. Data acquisition and other scientific activities can lead to heterogeneous file structures involving very different data formats. The crawler uses crawler specifications (CFoods) that define how these structures are interpreted and synchronized with LinkAhead. We explain this description using an example of CFood in this section. CFood is designed to make use of the example file structure discussed in the introduction.

Figure 3. Illustration of the data synchronization procedure using a crawler. Data acquisition (possibly including computer simulations or data produced during data analysis) leads to a variety of files in different formats on the file system. Crawler plugins (CFoods) are designed in a way that they understand the local structures on the file system. The file tree is traversed, possibly opening files and extracting (meta) data in order to transform them into a semantic data model that is then synchronized with the RDMS. The figure was previously published by Schlemmer. [26]

As we pointed out in the introduction, one major challenge is allowing for the adaptation of data integration to very different and potentially very heterogeneous use cases. Therefore, we designed a special syntax in YAML format for configuring the mapping which were just described in a machine-readable form. We call this description of rules for the crawler the crawler definition or CFood. For the example file structure, the corresponding YAML file would look like this:

ExperimentalData_Dir:

type: Directory

match: ExperimentalData

subtree:

Project_Dir:

type: Directory

match: (?P<year>[0-9]{4,4})_(?P<name>.*)

records:

Project:

year: $year

name: $name

subtree:

Experiment_Dir:

type: Directory

match: >- (?P<date>[0-9]{4,4}_[0-9]{2,2}_[0-9]{2,2})(_(?P<identifier>.*))?

records:

Experiment:

Project: $Project

date: $date

identifier: $identifier

subtree:

Readme_File:

type: MarkdownFile

match: README.md

subtree:

description:

type: DictTextElement

match_value: (?P<description>.*)

match_name: description

records:

Experiment:

description: $description

responsible_single:

type: DictTextElement

match_name: responsible

match_value: ((?P<first_name>.+) )?(?P<last_name>.+)

records:

Person:

first_name: $first_name

last_name: $last_name

Experiment:

Person: $Person

The YAML structure mimics the hierarchical structure of a file tree. The crawler operates by successively matching converters to files, folders, and possibly other StructureElements (A StructureElement specifies any piece of information that is derived from some part of the file structure and that can be matched by a converter.) If a converter matches, Records are created corresponding to the entries under the Records in the crawler definition. Afterwards, the crawler proceeds by processing sub-elements, like sub-folders, using the converters defined under the subtree.

In the given example, the following converters can be found:

• ExperimentalData_Dir, Project_Dir, Experiment_Dir, which are all of type Directory. These converters match names of folders against the regular expression given by match. When matching, converters of this type yield sub-folders and -files for processing of converters in the subtree section.
• Readme_File is of type MarkdownFile and allows for processing of the contents of the YAML header by converters given in the subtree section.
• description and responsible_single are two converters of type DictTextElement and can be used to match individual entries in the YAML header contained in the markdown file.

There are many more types of standard converters included in the LinkAhead crawler. Examples include converters for interpreting tabular data (in Excel or CSV format), JSON [27,28] files, or HDF5 [24,25] files. Custom converters can be created in Python using the LinkAhead crawler Python package. There is a community repository online where community extensions are collected and maintained (see Appendix C).

Variables

Variables begin with a dollar sign and can be used in multiple occasions within the CFood, e.g., for setting properties of Records. These variables and their values are set during multiple operations within the crawling procedure; variables that are part of a matched regular expression will be available. For example, $year is set from the match entry of the converter Project_Dir. Record definitions within the crawler definition will be available as variables, e.g., the records section of Project_Dir creates a variable $Project, which will later be used as the value of a Property of Record Experiment created in the records section of Experiment_Dir. More details about the syntax are made available in the official documentation of the LinkAhead crawler software (see Appendix D).

Scanner

We call the subroutines that gather information from the file system by applying the crawler definition to a file hierarchy during the process of scanning. The corresponding module of the crawler software is called Scanner. The result of a scanning process is a list of LinkAhead Records, where the values of the Properties are set to the information found. In the example shown in Figure 1, the list will contain at least two Project Records with names “ElementaryCharge” and “SpeedOfLight”; three Experiment Records with dates “2020-01-01,” “2020-01-02,” and “2020-01-03”; and one Person Record for “Florian Spreckelsen.” The remaining steps in the synchronization procedure are to split this list into a list of Records that need to be newly inserted and a list of Records that need an update. The next subsection describes how the updates are distinguished from the insert operations. Then the subsequent subsection describes the final process of carrying out the transactions.

Identifiables

In order to determine which of the Records that have been generated during the scanning procedure are already present in the RDMS, the crawler needs information on how to determine the identity of Records. For this purpose, we implemented a concept that is similar to unique keys, which are used in the context of RDBMSs. For each RecordType, we define a set of Properties that can be used to uniquely identify a single Record.

Using the example from Figure 1, we can claim that each Person in our RDMS can be uniquely identified by providing a “firstName” and a “lastName.” It is important to point out that the definition of identities for RecordTypes can vary highly depending on the usage scenario and environment. There might, of course, be many cases where this example definition is not sufficient, because people can have the same first and last names.

Appendices

Appendix A. Supporting software

The following software projects can be used to implement the workflows described in the article:

• Repository of the LinkAhead open-source project: https://gitlab.com/linkahead
• Repository of the LinkAhead crawler: https://gitlab.com/linkahead/linkahead-crawler

The installation procedures for LinkAhead and the crawler framework are provided in their respective repositories. Also a Docker container is available for the instant deployment of LinkAhead.

Appendix B. Example crawlers

There is a documented example available online ( at https://gitlab.com/linkahead/crawler-extensions/documented-crawler-example) that demonstrates the application of the crawler to example data. This can also be used as a template for the development of custom crawlers. We are currently aware of one public instance of LinkAhead, which makes use of a complex crawler based on the crawler framework described in this article. It is provided by ZMT-Leibniz Centre for Tropical Marine Research and can be accessed online at https://dataportal.leibniz-zmt.de/.

Appendix C. Community repository for crawler extensions

In order to foster the re-usability of crawler definitions, we are building a community repository for crawler extensions, which can be found at https://gitlab.com/linkahead/crawler-extensions.

Appendix D. Software documentation

The official documentation for LinkAhead, including an installation guide, can be found at https://docs.indiscale.com. The documentation for the crawler framework that is presented in this article can be found at https://docs.indiscale.com/caosdb-crawler/.

Notes

This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.