Journal:Semantics for an integrative and immersive pipeline combining visualization and analysis of molecular data

Full article title	Semantics for an integrative and immersive pipeline combining visualization and analysis of molecular data
Journal	Journal of Integrative Bioinformatics
Author(s)	Trellet, Mikael; Férey, Nicolas; Flotyński, Jakub; Baaden, Marc; Bourdot, Patrick
Author affiliation(s)	Bijvoet Center for Biomolecular Research, Université Paris Sud, Poznań Univ. of Economics and Business, Laboratoire de Biochimie Théorique
Primary contact	Email: m dot e dot trellet at uu dot nl
Year published	2018
Volume and issue	15(2)
Page(s)	20180004
DOI	10.1515/jib-2018-0004
ISSN	1613-4516
Distribution license	Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International
Website	https://www.degruyter.com/view/j/jib.2018.15.issue-2/jib-2018-0004/jib-2018-0004.xml
Download	https://www.degruyter.com/downloadpdf/j/jib.2018.15.issue-2/jib-2018-0004/jib-2018-0004.xml (PDF)

Abstract

The advances made in recent years in the field of structural biology significantly increased the throughput and complexity of data that scientists have to deal with. Combining and analyzing such heterogeneous amounts of data became a crucial time consumer in the daily tasks of scientists. However, only few efforts have been made to offer scientists an alternative to the standard compartmentalized tools they use to explore their data and that involve a regular back and forth between them. We propose here an integrated pipeline especially designed for immersive environments, promoting direct interactions on semantically linked 2D and 3D heterogeneous data, displayed in a common working space. The creation of a semantic definition describing the content and the context of a molecular scene leads to the creation of an intelligent system where data are (1) combined through pre-existing or inferred links present in our hierarchical definition of the concepts, (2) enriched with suitable and adaptive analyses proposed to the user with respect to the current task and (3) interactively presented in a unique working environment to be explored.

Keywords: virtual reality, semantics for interaction, structural biology

Introduction

Recent years have seen a profound change in the way structural biologists interact with their data. New techniques that try to capture the structure and dynamics of bio-molecules have reached an extraordinary high throughput of structural data. Scientists must try to combine and analyze data flows from different sources to draw their hypotheses and conclusions. However, despite this increasing complexity, they tend to rely mainly on compartmentalized tools to only visualize or analyze limited portions of their data. This situation leads to a constant back and forth between the different tools and their associated environments. Consequently, a significant amount of time is dedicated to the transformation of data to account for the heterogeneous input data types each tool is allowing.

The need for platforms capable of handling the intricate data flow is then strong. In structural biology, the numerical simulation process is now able to deal with very large and heterogeneous molecular structures. These molecular assemblies may be composed of several million particles and consist of many different types of molecules, including a biologically realistic environment. This overall complexity raises the need to go beyond common visualization solutions and move towards integrated exploration systems where visualization and analysis can be merged.

Immersive environments play an important role in this context, providing both a better comprehension of the three-dimensional structure of molecules, and offering new interaction techniques to reduce the number of data manipulations executed by the experts (see Figure 1). A few studies took advantage of recent developments in virtual reality to enhance some structural biology tasks. Visualization is the first and most obvious task that was improved through new adaptive stereoscopic screens and immersive environments, plunging experts into the very center of their molecules. Structure manipulations during specific docking experiments have been improved thanks to the use of haptic devices and audio feedback to drive a simulation. However, if 3D objects can rather easily be represented and manipulated in such environments, the integration of analytical values (energies, distance to reference, etc.)—2D by nature—leads to a certain complexity and is not a solved problem yet. As a consequence, no specific development has been made to set up an immersive platform where the expert could manipulate data coming from different sources to accelerate and improve the development of new hypotheses.

Figure 1. Immersive, augmented reality, and screen wall environments used for molecular visualization: (A) EVE platform, a multi-user CAVE-system composed of 4 screens (LIMSI-CNRS/VENISE team, Orsay), (B) Microsoft Hololens and (C) screen wall of 8.3 m2 composed of 12 screens at full HD resolution with 120 Hz refresh rate in stereoscopy (IBPC-CNRS/LBT, Paris).

This lack of development can also be partly explained by the significant differences between the data handled by the 3D visualization software packages and the analytical tools. On one side, 3D visualization solutions such as PyMol, VMD, and UnityMol explore and manipulate 3D structure coordinates composing the molecular complex that will be displayed. The scene seen by the user is composed of 3D objects reporting the overall shape of a particular molecule and its environment at a particular state. This scene is static if we are interested in only one state of a given molecule, but is often dynamic when a whole simulated trajectory of conformational changes over time is considered. Analysis tools, on the other side, handle raw numbers, vectors, and matrices in various formats and dimensions, from various input sources depending on the analysis pipeline used to generate them. Their outputs are graphical representations of trends or comparisons between parameters or properties in 1 to N dimensions formatted in a way that experts can quickly understand and use such information to guide their hypotheses.

Some of the aforementioned software do provide tools to gather analyses as static plots aside the 3D visualization space. Interactivity is limited and flexibility mainly depends on the user capability to create and tune scripts to improve the information displayed. We believe that a major improvement of tools available today would bring into play a scenario where the 3D visualization of a molecular event is coupled to monitoring the evolution of analytical properties, e.g., sub-elements such as distance variations and progression of simulation parameters, into a single working environment. The expert would be able to see any action performed in one space (either 3D visualization or analysis) with a coherent graphical impact on the second space to filter or highlight the parameter or sub-ensemble of objects targeted by the expert.

We have developed a pipeline that aims to bring within the same immersive environment the visualization and analysis of heterogeneous data coming from molecular simulations. This pipeline addresses the lack of integrated tools efficiently combining the stereoscopic visualization of 3D objects and the representation/interaction with their associated physicochemical and geometric properties (both 2D and 3D) generated by standard analysis tools and that are either combined to the 3D objects (shape, colour, etc.) or displayed on a dedicated space integrated in the working environment (second mobile screen, 2D integration in the virtual scene, etc.).

In this pipeline, we systematically combine structural and analytical data by using a semantic definition of the content (scientific data) and the context (immersive environments and interfaces). Such a high-level definition can be translated into an ontology from which instances or individuals of ontological concepts can then be created from real data to build a database of linked data for a defined phenomenon. On top of the data collection, an extensive list of possible interactions and actions defined in the ontology and based on the provided data can be computed and presented to the user.

The creation of a semantic definition describing the content and the context of a molecular scene in immersion leads to the creation of an intelligent system where data and 3D molecular representations are (1) combined through pre-existing or inferred links present in our hierarchical definition of the concepts, (2) enriched with suitable and adaptive analyses proposed to the user with respect to the current task, and (3) manipulated by direct interaction allowing to both perform 3D visualization and exploration as well as analysis in a unique immersive environment.

Our method narrows the need for complex interactions by considering what actions the user can perform with the data he is currently manipulating and the means of interaction his immersive environment provides.

We will highlight our developments and the first outcomes of our work through three main sections: the first section attempts to provide a complete background of the usage of semantics in the fields of VR/AR systems and structural biology. In the second section we will describe and justify our implementation choices and how we linked the different technologies highlighted in the previous section. Finally, in a third section, we will show several applications of our platform and its capabilities to address the issues raised previously.

Related works

We present here the state of the art in the two fields related to this paper: the semantic formalism chosen to represent the data and how semantic representations are applied in bioinformatics.

Semantic modeling formalism and semantic web

From classical logic to description logic, from which was derived the "conceptual graph" representation introduced by Sowa, many semantic formalisms were used to embed knowledge into applications in order to query and perform reasoning about them.

The conceptual graph formalism represents concepts and properties such as connected graphs and allows complex operations on them. However, it quickly reaches some limitations in terms of performances and implementation flexibility. Classical logic is another well-known formalism but is not broadly used in biology and suffers a lack of implementation tools and libraries. A semantic network limits itself to the representation of concepts and their relations through directed or undirected graphs. It is lacking the possibility to reason over the concepts and their links, reasoning that our intended platform needs. The different requirements of our platform, coupled with our aim to make it as generic as possible, made us choose to use description logics as a formalism for knowledge representation and more precisely the semantic web as underlying standard for the creation of our ontology and the associated knowledge base.

The semantic web has been created by the World Wide Web Consortium under the lead of Tim Berners-Lee, with the aim to share semantic data on the web. It is broadly used by the biggest web companies to uniformly store and share data. It belongs to the family of description logics that use the notions of concepts, roles, and individuals. The concepts are represented by the sub-ensemble of elements in a specific universe, the roles are the links between the elements, and the individuals are the elements of the universe. Each layer of the semantic web (ontology, experimental data, querying process, etc.) has been associated to a language or a format.

The following four standards create the core of the semantic web and act as the layers evoked previously: the Resource Description Framework (RDF), the Resource Description Framework Schema (RDFS), the Web Ontology Language (OWL), and SPARQL. Whereas the first three standards enable semantic descriptions of data in the form of ontologies and knowledge bases, the last standard enables queries to ontologies and knowledge bases (see Figure 2).

Figure 2. Web semantics and its different layers. This figure describes the main format classically used for each layer: RDF, RDFS, OWL, SPARQL, etc. Source : http://www.w3.org/2001/sw/

RDF is a data model, which allows the creation of statements to describe resources. Each statement is a triple comprised of: a subject (resource described by the statement), a predicate (property of the subject), and an object (literal value or resource identified by a URI, which describes the subject). An example of a triple is: <#Molecule, #has-charge, -1>

RDFS and OWL are semantic web standards that extend the expressiveness of RDF by providing additional concepts. RDFS provides hierarchies of classes and properties as well as property domains and ranges. OWL, built upon RDF and RDFS, provides symmetry, transitivity, equivalence, and restrictions of properties as well as operations on sets of resources. In turn, SPARQL is a query language for ontologies and knowledge bases built using RDF, RDFS, and OWL. Conceptually, in terms of possible operations on data, SPARQL is similar to SQL, as it enables data selection, insertion, update, and removal.

In the semantic web, two types of statements are distinguished. Terminological statements (T-Box) specify conceptualization, classes and properties of resources, without describing any particular resources. Assertion statements (A-Box) specify utilization, particular resources (also called individuals or objects), which are instances of classes described by properties with particular values assigned. For example, a T-Box specifies different classes of molecules (different chemical compounds) and properties that can be used to describe them (e.g., charge and the number of neutrons), while an A-Box specifies particular molecules (instances of the classes) with given charges. In this paper, an ontology is a T-Box, while a knowledge base is the union of a T-Box and an A-Box. Ontologies and knowledge bases constitute the foundation of the semantic web across diverse domains and applications. In particular, ontologies can specify schemes of molecular descriptions, while knowledge bases—particular descriptions (instances of such schemes) with individual objects—are used for analysis and visualization. Due to the use of the standards encoded in XML or equivalent formats, ontologies and knowledge bases are interpretable to software, making them intelligible to users. Moreover, since RDFS and OWL are built upon description logics, which are formal knowledge representation techniques, ontologies and knowledge bases can be subject to reasoning, which is a process of inferring implicit (tacit) properties of resources (which have not been explicitly specified by the author) on the basis of their explicitly specified properties.

For instance, from the following triples explicitly specified by the content author:

<my:is-composed-of> <my:is-a> <owl:TransitiveProperty>

<my:Protein> <my:is-composed-of> <my:Amino-acid>

<my:Amino-acid> <my:is-composed-of> <my:Atom>

the following statement can be inferred by software:

<my:Protein> <my:is-composed-of> <my:Atom>

Here, thanks to the definition of property “is-composed-of” as transitive, we can infer that atoms, that compose amino acids, compose as well a protein since amino acids compose proteins. The second statement does not need to be added to the ontology since automatically inferred. This reduces significantly the number of statements to store in the database and potentially allows for more complex inferences.

Ontologies in bioinformatics

On the application side, the use of ontologies in order to standardize knowledge in scientific fields underwent an important and spontaneous growth at the end of the 1990s. Bioinformatics, tightly anchored in structural biology, has used ontologies for a long time. The most significant example is the fast-growing genomic field, in which it became impossible to handle data flow without a proper and standardized organization of the data. The tool Gene Ontology regroups genomic data into a uniform format and a knowledge base. Currently, it is one of the most referred to ontologies in the literature. Rabattu et al. propose an approach to spatio-temporal reasoning on semantic descriptions of an evolving human embryo. Several biological databases or organizations such as UniProtKB1 and the Open Biomedical Ontologies provide ways to access data or ontologies under RDF or OWL format to allow their use in expert tools or specific pipelines. One can also note the open-source project Bio2RDF that aims to build and provide the largest network of "Linked Data for the Life Sciences" using semantic web approaches.

Only a few expert software packages based on ontologies have been developed for structural biology. Avogadro and DIVE appear as exceptions, implementing, in different ways, a semantic description of data that can be manipulated in these environments. Avogadro uses the Chemical Markup Language (CML) as the format for describing data semantics, and it adds a semantic description layer on top of the data being described. However, the tool leverages neither ontologies nor other knowledge representation formalisms, thus it does not permit reasoning on the described data.

DIVE partially creates ontologies and datasets derived from the input data upon loading. Pre-formatted input in a row/column representation are converted into a SQL-like structure where rows are individuals and columns properties. This data representation conforms to a common data model that the software libraries use. Therefore, creation of links between data values and concepts are possible, and different DIVE components for data presentation (analyses, 3D visualization, etc.) as well as links and relationships between dataset elements can be queried. In addition, DIVE includes a powerful and generic ontology creator directly depending on the type of the input data. However, reasoning on ontologies in DIVE is limited to inheritance between classes. Consequently, only a few ontological relationships are available: is-a, contains, is-part-of, and bound-by. There is no notion of cardinality or logical operators to define the concept classes. Then, it is not possible, for instance, to force the presence of a property, or to impose that only a fixed number of values are associated to a specific property (e.g., a molecule must have at least one atom, an Alanine side-chain has a minimum of three atoms and a maximum of four atoms, etc.). These limitations render the DIVE environment insufficient to solve the problem stated in this paper.

Using a semantic representation to efficiently store, query, and link heterogeneous structural biology data

Several important choices have been made to integrate the different technologies required for the establishment of a platform that would allow a proper 3D immersion of users together with an accurate and intelligent way to interact with their data. Our platform heavily relies on the ontology/knowledge base couple. The way to represent and access the data present in the databases is of a crucial importance, and this point led us to ask ourselves the question of the most appropriate formalism for the data representation.

Knowledge formalism choice

The formalism of knowledge representation used in our approach must address the following three rules to properly fit our platform needs:

1. Hierarchical data representation via concepts and properties
2. Advanced reasoning possibility in order to extend the ontology or the dataset ruled by the ontology
3. Efficient query time on the data to stay within interaction time

We mentioned previously that several formalisms exist to create ontologies and define databases. A quick comparison of these formalisms, complementary to their introduction in the previous section, can be found in Table 1.

Table 1. Comparison of different knowledge representation formalisms with respect to key criteria Formalism Domain description Reasoning on knowledge Big data management Efficient Implementation flexibility Conceptual graphs X X - X - Semantic networks X - X X - Classical logics X X X X - Description logics X X X X -

Our first implementation of a semantic representation of knowledge in molecular biology was applied through conceptual graphs (CG) within Cogitant’s software. The use of CGs through the Cogitant API quickly proved to be incompatible with the constraints of the interactive context. This limitation had already been highlighted by the work of Yannick Dennemont with the Prolog CG API, limitations confirmed by our own experience with the Cogitant library in C++. The need for high performance imposed by the interactive context has led us to the path of description logic and semantic web for the representation of knowledge and the efficient extraction of information within a massive fact base to support Visual Analytics functionalities in molecular biology.

Notes

This presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added. The original references after 27 were slightly out of order in the original; due to the way this wiki works, references are listed in the order they appear. Nothing else was changed in accordance with the NoDerivatives portion of the license.