View source for Journal:Semantics for an integrative and immersive pipeline combining visualization and analysis of molecular data

{{Infobox journal article
|name         = 
|image        = 
|alt          = <!-- Alternative text for images -->
|caption      = 
|title_full   = Semantics for an integrative and immersive pipeline combining visualization and analysis of molecular data
|journal      = ''Journal of Integrative Bioinformatics''
|authors      = Trellet, Mikael; Férey, Nicolas; Flotyński, Jakub; Baaden, Marc; Bourdot, Patrick
|affiliations = Bijvoet Center for Biomolecular Research, Université Paris Sud, Poznań Univ. of Economics and Business, Laboratoire de Biochimie Théorique
|contact      = Email: m dot e dot trellet at uu dot nl
|editors      = 
|pub_year     = 2018
|vol_iss      = '''15'''(2)
|pages        = 20180004
|doi          = [http://doi.org/10.1515/jib-2018-0004 10.1515/jib-2018-0004]
|issn         = 1613-4516
|license      = [https://creativecommons.org/licenses/by-nc-nd/4.0/ Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International]
|website      = [https://www.degruyter.com/view/j/jib.2018.15.issue-2/jib-2018-0004/jib-2018-0004.xml https://www.degruyter.com/view/j/jib.2018.15.issue-2/jib-2018-0004/jib-2018-0004.xml]
|download     = [https://www.degruyter.com/downloadpdf/j/jib.2018.15.issue-2/jib-2018-0004/jib-2018-0004.xml https://www.degruyter.com/downloadpdf/j/jib.2018.15.issue-2/jib-2018-0004/jib-2018-0004.xml] (PDF)
}}
==Abstract==
The advances made in recent years in the field of structural biology significantly increased the throughput and complexity of data that scientists have to deal with. Combining and [[Data analysis|analyzing]] such heterogeneous amounts of data became a crucial time consumer in the daily tasks of scientists. However, only few efforts have been made to offer scientists an alternative to the standard compartmentalized tools they use to explore their data and that involve a regular back and forth between them. We propose here an integrated pipeline especially designed for immersive environments, promoting direct interactions on semantically linked 2D and 3D heterogeneous data, displayed in a common working space. The creation of a semantic definition describing the content and the context of a molecular scene leads to the creation of an intelligent system where data are (1) combined through pre-existing or inferred links present in our hierarchical definition of the concepts, (2) enriched with suitable and adaptive analyses proposed to the user with respect to the current task and (3) interactively presented in a unique working environment to be explored.

'''Keywords''': virtual reality, semantics for interaction, structural biology

==Introduction==
Recent years have seen a profound change in the way structural biologists interact with their data. New techniques that try to capture the structure and dynamics of bio-molecules have reached an extraordinary high throughput of structural data.<ref name="ZhaoMature13">{{cite journal |title=Mature HIV-1 capsid structure by cryo-electron microscopy and all-atom molecular dynamics |journal=Nature |author=Zhao, G.; Perilla, J.R.; Yufenyuy, E.L. et al. |volume=497 |issue=7451 |pages=643–6 |year=2013 |doi=10.1038/nature12162 |pmid=23719463 |pmc=PMC3729984}}</ref><ref name="ZhangStructure17">{{cite journal |title=Structure of phycobilisome from the red alga Griffithsia pacifica |journal=Nature |author=Zhang, J.; Ma, J.; Liu, D. et al. |volume=551 |issue=7678 |pages=57–63 |year=2017 |doi=10.1038/nature24278 |pmid=29045394}}</ref> Scientists must try to combine and analyze data flows from different sources to draw their hypotheses and conclusions. However, despite this increasing complexity, they tend to rely mainly on compartmentalized tools to only [[Data visualization|visualize]] or analyze limited portions of their data. This situation leads to a constant back and forth between the different tools and their associated environments. Consequently, a significant amount of time is dedicated to the transformation of data to account for the heterogeneous input data types each tool is allowing.

The need for platforms capable of handling the intricate data flow is then strong. In structural biology, the numerical simulation process is now able to deal with very large and heterogeneous molecular structures. These molecular assemblies may be composed of several million particles and consist of many different types of molecules, including a biologically realistic environment. This overall complexity raises the need to go beyond common visualization solutions and move towards integrated exploration systems where visualization and analysis can be merged.

Immersive environments play an important role in this context, providing both a better comprehension of the three-dimensional structure of molecules, and offering new interaction techniques to reduce the number of data manipulations executed by the experts (see Figure 1). A few studies took advantage of recent developments in virtual reality to enhance some structural biology tasks. Visualization is the first and most obvious task that was improved through new adaptive stereoscopic screens and immersive environments, plunging experts into the very center of their molecules.<ref name="vanDamImmersive00">{{cite journal |title=Immersive VR for scientific visualization: A progress report |journal=IEEE Computer Graphics and Applications |author=van Dam, A.; Forsberg, A.S.; Laidlaw, D.H. et al. |volume=20 |issue=6 |pages=26–52 |year=2000 |doi=10.1109/38.888006}}</ref><ref name="StoneImmersive10">{{cite journal |title=Immersive molecular visualization and interactive modeling with commodity hardware |journal=Proceedings of the 6th International Conference on Advances in Visual Computing |author=Stone. J.E.; Kohlmeyer, A.; Vandivort, K.L.; Schulten, K. |pages=382–93 |year=2010 |doi=10.1007/978-3-642-17274-8_38}}</ref><ref name="ODonoghueVisual10">{{cite journal |title=Visualization of macromolecular structures |journal=Nature Methods |author=O'Donoghue, S.I.; Goodsell, D.S.; Frangakis, A.S. et al. |volume=7 |issue=3 Suppl. |pages=S42–55 |year=2010 |doi=10.1038/nmeth.1427 |pmid=20195256}}</ref><ref name="HirstMolec14">{{cite journal |title=Molecular simulations and visualization: Introduction and overview |journal=Faraday Discussions |author=Hirst, J.D.; Glowacki, D.R.; Baaden, M. et al. |volume=169 |pages=9–22 |year=2014 |doi=10.1039/c4fd90024c |pmid=25285906}}</ref><ref name="GoddardUCSF18">{{cite journal |title=UCSF ChimeraX: Meeting modern challenges in visualization and analysis |journal=Protein Science |author=Goddard, T.D., Huang, C.C.; Meng, E.C. et al. |volume=27 |issue=1 |pages=14–25 |year=2018 |doi=10.1002/pro.3235 |pmid=28710774 |pmc=PMC5734306}}</ref> Structure manipulations during specific docking experiments have been improved thanks to the use of haptic devices and audio feedback to drive a simulation.<ref name="FéreyMulti09">{{cite journal |title=Multisensory VR interaction for protein-docking in the CoRSAIRe project |journal=Virtual Reality |author=Férey, N.; Nelson, J.; Martin, C. et al. |volume=13 |pages=273 |year=2009 |doi=10.1007/s10055-009-0136-z}}</ref> However, if 3D objects can rather easily be represented and manipulated in such environments, the integration of analytical values (energies, distance to reference, etc.)—2D by nature—leads to a certain complexity and is not a solved problem yet. As a consequence, no specific development has been made to set up an immersive platform where the expert could manipulate data coming from different sources to accelerate and improve the development of new hypotheses.


[[File:Fig1 Trellet JOfIntegBioinfo2018 15-2.jpg|600px]]
{{clear}}
{| 
 | STYLE="vertical-align:top;"|
{| border="0" cellpadding="5" cellspacing="0" width="600px"
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;"| <blockquote>'''Figure 1.''' Immersive, augmented reality, and screen wall environments used for molecular visualization: (A) EVE platform, a multi-user CAVE-system composed of 4 screens (LIMSI-CNRS/VENISE team, Orsay), (B) Microsoft Hololens and (C) screen wall of 8.3 m<sup>2</sup> composed of 12 screens at full HD resolution with 120 Hz refresh rate in stereoscopy (IBPC-CNRS/LBT, Paris).</blockquote>
 |- 
|}
|}

This lack of development can also be partly explained by the significant differences between the data handled by the 3D visualization software packages and the analytical tools. On one side, 3D visualization solutions such as PyMol<ref name="DeLanoThePy00">{{cite web |url=http://pymol.sourceforge.net/overview/index.htm |title=The PyMOL Molecular Graphics System |author=DeLano, W. |date=04 September 2000}}</ref>, VMD<ref name="HumphreyVMD96">{{cite journal |title=VMD: Visual molecular dynamics |journal=Journal of Molecular Graphics |author=Humphrey, W.; Dalke, A.; Schulten, K. et al. |volume=14 |issue=1 |pages=33–8 |year=1996 |doi=10.1016/0263-7855(96)00018-5}}</ref>, and UnityMol<ref name="LvGame13">{{cite journal |title=Game on, science - How video game technology may help biologists tackle visualization challenges |journal=PLoS One |author=Lv, Z.; Tek, A.; Da Silva, F. et al. |volume=8 |issue=3 |pages=e57990 |year=2013 |doi=10.1371/journal.pone.0057990 |pmid=23483961 |pmc=PMC3590297}}</ref> explore and manipulate 3D structure coordinates composing the molecular complex that will be displayed. The scene seen by the user is composed of 3D objects reporting the overall shape of a particular molecule and its environment at a particular state. This scene is static if we are interested in only one state of a given molecule, but is often dynamic when a whole simulated trajectory of conformational changes over time is considered. Analysis tools, on the other side, handle raw numbers, vectors, and matrices in various formats and dimensions, from various input sources depending on the analysis pipeline used to generate them. Their outputs are graphical representations of trends or comparisons between parameters or properties in 1 to ''N'' dimensions formatted in a way that experts can quickly understand and use such [[information]] to guide their hypotheses.

Some of the aforementioned software do provide tools to gather analyses as static plots aside the 3D visualization space. Interactivity is limited and flexibility mainly depends on the user capability to create and tune scripts to improve the information displayed. We believe that a major improvement of tools available today would bring into play a scenario where the 3D visualization of a molecular event is coupled to monitoring the evolution of analytical properties, e.g., sub-elements such as distance variations and progression of simulation parameters, into a single working environment. The expert would be able to see any action performed in one space (either 3D visualization or analysis) with a coherent graphical impact on the second space to filter or highlight the parameter or sub-ensemble of objects targeted by the expert.

We have developed a pipeline that aims to bring within the same immersive environment the visualization and analysis of heterogeneous data coming from molecular simulations. This pipeline addresses the lack of integrated tools efficiently combining the stereoscopic visualization of 3D objects and the representation/interaction with their associated physicochemical and geometric properties (both 2D and 3D) generated by standard analysis tools and that are either combined to the 3D objects (shape, colour, etc.) or displayed on a dedicated space integrated in the working environment (second mobile screen, 2D integration in the virtual scene, etc.).

In this pipeline, we systematically combine structural and analytical data by using a semantic definition of the content (scientific data) and the context (immersive environments and interfaces). Such a high-level definition can be translated into an ontology from which instances or individuals of ontological concepts can then be created from real data to build a database of linked data for a defined phenomenon. On top of the data collection, an extensive list of possible interactions and actions defined in the ontology and based on the provided data can be computed and presented to the user.

The creation of a semantic definition describing the content and the context of a molecular scene in immersion leads to the creation of an intelligent system where data and 3D molecular representations are (1) combined through pre-existing or inferred links present in our hierarchical definition of the concepts, (2) enriched with suitable and adaptive analyses proposed to the user with respect to the current task, and (3) manipulated by direct interaction allowing to both perform 3D visualization and exploration as well as analysis in a unique immersive environment.

Our method narrows the need for complex interactions by considering what actions the user can perform with the data he is currently manipulating and the means of interaction his immersive environment provides.

We will highlight our developments and the first outcomes of our work through three main sections: the first section attempts to provide a complete background of the usage of semantics in the fields of VR/AR systems and structural biology. In the second section we will describe and justify our implementation choices and how we linked the different technologies highlighted in the previous section. Finally, in a third section, we will show several applications of our platform and its capabilities to address the issues raised previously.

==Related works==
We present here the state of the art in the two fields related to this paper: the semantic formalism chosen to represent the data and how semantic representations are applied in [[bioinformatics]].

===Semantic modeling formalism and semantic web===
From classical logic to description logic, from which was derived the "conceptual graph" representation introduced by Sowa<ref name="SowaConcept84">{{cite book |title=Conceptual structures: Information processing in mind and machine |author=Sowa, J.F. |publisher=Addison-Wesley Longman Publishing Co |year=1984 |isbn=0201144727}}</ref>, many semantic formalisms were used to embed knowledge into applications in order to query and perform reasoning about them.

The conceptual graph formalism represents concepts and properties such as connected graphs and allows complex operations on them. However, it quickly reaches some limitations in terms of performances and implementation flexibility. Classical logic is another well-known formalism but is not broadly used in biology and suffers a lack of implementation tools and libraries. A semantic network limits itself to the representation of concepts and their relations through directed or undirected graphs. It is lacking the possibility to reason over the concepts and their links, reasoning that our intended platform needs. The different requirements of our platform, coupled with our aim to make it as generic as possible, made us choose to use description logics as a formalism for knowledge representation and more precisely the semantic web as underlying standard for the creation of our ontology and the associated knowledge base.

The semantic web has been created by the World Wide Web Consortium under the lead of Tim Berners-Lee, with the aim to share semantic data on the web.<ref name="Berners-LeeTheSemantic01">{{cite journal |title=The Semantic Web |journal=Scientific American |author=Berners-Lee, T.; Hendler, J.; Lassila, O. |volume=284 |pages=28–37 |year=2001}}</ref> It is broadly used by the biggest web companies to uniformly store and share data. It belongs to the family of description logics that use the notions of concepts, roles, and individuals. The concepts are represented by the sub-ensemble of elements in a specific universe, the roles are the links between the elements, and the individuals are the elements of the universe. Each layer of the semantic web (ontology, experimental data, querying process, etc.) has been associated to a language or a format.

The following four standards create the core of the semantic web and act as the layers evoked previously: the Resource Description Framework (RDF)<ref name="CyganiakRDF14">{{cite web |url=https://www.w3.org/TR/rdf11-concepts/ |title=RDF 1.1 Concepts and Abstract Syntax |editor=Cyganiak, R.; Wood, D.; Lanthaler, M. |publisher=World Wide Web Consortium |date=25 February 2014}}</ref>, the Resource Description Framework Schema (RDFS)<ref name="BrickleyRDF14">{{cite web |url=https://www.w3.org/TR/rdf-schema/ |title=RDF Schema 1.1 |editor=Brickley, D.; Guha, R.V. |publisher=World Wide Web Consortium |date=25 February 2014}}</ref>, the Web Ontology Language (OWL)<ref name="MotikOWL12">{{cite web |url=https://www.w3.org/TR/owl2-syntax/ |title=OWL 2 Web Ontology Language |editor=Motik, B.; Patel-Schneider, P.F.; Parsia, B. |publisher=World Wide Web Consortium |date=11 December 2012}}</ref>, and SPARQL.<ref name="HarrisSPARQL13">{{cite web |url=https://www.w3.org/TR/sparql11-query/ |title=SPARQL 1.1 Query Language |editor=Harris, S.; Seaborne, A. |publisher=World Wide Web Consortium |date=21 March 2013}}</ref> Whereas the first three standards enable semantic descriptions of data in the form of ontologies and knowledge bases, the last standard enables queries to ontologies and knowledge bases (see Figure 2).


[[File:Fig2 Trellet JOfIntegBioinfo2018 15-2.jpg|600px]]
{{clear}}
{| 
 | STYLE="vertical-align:top;"|
{| border="0" cellpadding="5" cellspacing="0" width="600px"
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;"| <blockquote>'''Figure 2.''' Web semantics and its different layers. This figure describes the main format classically used for each layer: RDF, RDFS, OWL, SPARQL, etc. Source : http://www.w3.org/2001/sw/</blockquote>
 |- 
|}
|}

RDF is a data model, which allows the creation of statements to describe resources. Each statement is a triple comprised of: a subject (resource described by the statement), a predicate (property of the subject), and an object (literal value or resource identified by a URI, which describes the subject). An example of a triple is: <code><#Molecule, #has-charge, -1></code>

RDFS and OWL are semantic web standards that extend the expressiveness of RDF by providing additional concepts. RDFS provides hierarchies of classes and properties as well as property domains and ranges. OWL, built upon RDF and RDFS, provides symmetry, transitivity, equivalence, and restrictions of properties as well as operations on sets of resources. In turn, SPARQL is a query language for ontologies and knowledge bases built using RDF, RDFS, and OWL. Conceptually, in terms of possible operations on data, SPARQL is similar to SQL, as it enables data selection, insertion, update, and removal.

In the semantic web, two types of statements are distinguished. Terminological statements (T-Box) specify conceptualization, classes and properties of resources<ref name="GiacomoTBox96">{{cite journal |title=TBox and ABox Reasoning in Expressive Description Logics |journal=Proceedings of the Fifth International Conference on Principles of Knowledge Representation and Reasoning |author=De Giacomo, G.; Lenzerini, M. |pages=316–27 |year=1996 |isbn=1558604219}}</ref>, without describing any particular resources. Assertion statements (A-Box) specify utilization, particular resources (also called individuals or objects), which are instances of classes described by properties with particular values assigned. For example, a T-Box specifies different classes of molecules (different chemical compounds) and properties that can be used to describe them (e.g., charge and the number of neutrons), while an A-Box specifies particular molecules (instances of the classes) with given charges. In this paper, an ontology is a T-Box, while a knowledge base is the union of a T-Box and an A-Box. Ontologies and knowledge bases constitute the foundation of the semantic web across diverse domains and applications. In particular, ontologies can specify schemes of molecular descriptions, while knowledge bases—particular descriptions (instances of such schemes) with individual objects—are used for analysis and visualization. Due to the use of the standards encoded in [[XML]] or equivalent formats, ontologies and knowledge bases are interpretable to software, making them intelligible to users. Moreover, since RDFS and OWL are built upon description logics, which are formal knowledge representation techniques, ontologies and knowledge bases can be subject to reasoning, which is a process of inferring implicit (tacit) properties of resources (which have not been explicitly specified by the author) on the basis of their explicitly specified properties.

For instance, from the following triples explicitly specified by the content author:

<code><my:is-composed-of> <my:is-a> <owl:TransitiveProperty></code>

<code><my:Protein> <my:is-composed-of> <my:Amino-acid></code>

<code><my:Amino-acid> <my:is-composed-of> <my:Atom></code>

the following statement can be inferred by software:

<code><my:Protein> <my:is-composed-of> <my:Atom></code>

Here, thanks to the definition of property “is-composed-of” as transitive, we can infer that atoms, that compose amino acids, compose as well a protein since amino acids compose proteins. The second statement does not need to be added to the ontology since automatically inferred. This reduces significantly the number of statements to store in the database and potentially allows for more complex inferences.

===Ontologies in bioinformatics===
On the application side, the use of ontologies in order to standardize knowledge in scientific fields underwent an important and spontaneous growth at the end of the 1990s.<ref name="Schulze-KremerOnto02">{{cite journal |title=Ontologies for molecular biology and bioinformatics |journal=In Silico Biology |author=Schulze-Kremer, S. |volume=2 |issue=3 |pages=179–93 |year=2002 |pmid=12542404}}</ref> Bioinformatics, tightly anchored in structural biology, has used ontologies for a long time. The most significant example is the fast-growing [[Genomics|genomic field]], in which it became impossible to handle data flow without a proper and standardized organization of the data.<ref name="SchuurmanOnto08">{{cite journal |title=Ontologies for bioinformatics |journal=Bioinformatics and Biology Insights |author=Schuurman, N.; Leszcynski, A. |volume=2 |pages=187—200 |year=2008 |pmid=19812775 |pmc=PMC2735951}}</ref> The tool Gene Ontology<ref name="GOCGene00">{{cite journal |title=Gene ontology: Tool for the unification of biology |journal=Nature Genetics |author=The Gene Ontology Consortium, Ashburner, M.; Ball, C.A. et al. |volume=25 |issue=1 |pages=25–9 |year=2000 |doi=10.1038/75556 |pmid=10802651 |pmc=PMC3037419}}</ref> regroups genomic data into a uniform format and a knowledge base. Currently, it is one of the most referred to ontologies in the literature. Rabattu ''et al.''<ref name="RabattuMyCorporis15">{{cite journal |title=My Corporis Fabrica Embryo: An ontology-based 3D spatio-temporal modeling of human embryo development |journal=Journal of Biomedical Semantics |author=Rabattu, P.Y.; Massé, B.; Ulliana, F. et al. |volume=6 |pages=36 |year=2015 |doi=10.1186/s13326-015-0034-0 |pmid=26413258 |pmc=PMC4582726}}</ref> propose an approach to spatio-temporal reasoning on semantic descriptions of an evolving human embryo. Several biological databases or organizations such as [http://www.uniprot.org/uniprot/ UniProtKB] and the Open Biomedical Ontologies<ref name="SmithTheOBO07">{{cite journal |title=The OBO Foundry: Coordinated evolution of ontologies to support biomedical data integration |journal=Nature Biotechnology |author=Smith, B.; Ashburner, M.; Rosse, C. et al. |volume=25 |issue=11 |pages=1251–5 |year=2007 |doi=10.1038/nbt1346 |pmid=17989687 |pmc=PMC2814061}}</ref> provide ways to access data or ontologies under RDF or OWL format to allow their use in expert tools or specific pipelines. One can also note the open-source project Bio2RDF<ref name="BelleauBio2RDF">{{cite journal |title=Bio2RDF: towards a mashup to build bioinformatics knowledge systems |journal=Journal of Biomedical Informatics |author=Belleau, F.; Nolin, M.A;. Tourigny, N. et al. |volume=41 |issue=5 |pages=706–16 |year=2008 |doi=10.1016/j.jbi.2008.03.004 |pmid=18472304}}</ref> that aims to build and provide the largest network of "Linked Data for the Life Sciences" using semantic web approaches.

Only a few expert software packages based on ontologies have been developed for structural biology. Avogadro<ref name="HanwellAvo12">{{cite journal |title=Avogadro: An advanced semantic chemical editor, visualization, and analysis platform |journal=Journal of Cheminformatics |author=Hanwell, M.D.; Curtis, D.E.; Lonie, D.C. et al. |volume=4 |issue=1 |pages=17 |year=2012 |doi=10.1186/1758-2946-4-17 |pmid=22889332 |pmc=PMC3542060}}</ref> and DIVE<ref name="RysavyDIVE14">{{cite journal |title=DIVE: A Graph-Based Visual-Analytics Framework for Big Data |journal=IEEE Computer Graphics and Applications |author=Rysavy, S.J.; Bromley, D.; Daggett, V. |volume=34 |issue=2 |pages=26–37 |year=2014 |doi=10.1109/MCG.2014.27}}</ref> appear as exceptions, implementing, in different ways, a semantic description of data that can be manipulated in these environments. Avogadro uses the Chemical Markup Language (CML)<ref name="RzepaCML12">{{cite web |url=http://www.xml-cml.org/ |title=Chemical Markup Language |author=Rzepa, H. |publisher=CMLC |date=2012}}</ref> as the format for describing data semantics, and it adds a semantic description layer on top of the data being described. However, the tool leverages neither ontologies nor other knowledge representation formalisms, thus it does not permit reasoning on the described data.

DIVE partially creates ontologies and datasets derived from the input data upon loading. Pre-formatted input in a row/column representation are converted into a SQL-like structure where rows are individuals and columns properties. This data representation conforms to a common data model that the software libraries use. Therefore, creation of links between data values and concepts are possible, and different DIVE components for data presentation (analyses, 3D visualization, etc.) as well as links and relationships between dataset elements can be queried. In addition, DIVE includes a powerful and generic ontology creator directly depending on the type of the input data. However, reasoning on ontologies in DIVE is limited to inheritance between classes. Consequently, only a few ontological relationships are available: is-a, contains, is-part-of, and bound-by. There is no notion of cardinality or logical operators to define the concept classes. Then, it is not possible, for instance, to force the presence of a property, or to impose that only a fixed number of values are associated to a specific property (e.g., a molecule must have at least one atom, an Alanine side-chain has a minimum of three atoms and a maximum of four atoms, etc.). These limitations render the DIVE environment insufficient to solve the problem stated in this paper.

==Using a semantic representation to efficiently store, query, and link heterogeneous structural biology data==
Several important choices have been made to integrate the different technologies required for the establishment of a platform that would allow a proper 3D immersion of users together with an accurate and intelligent way to interact with their data. Our platform heavily relies on the ontology/knowledge base couple. The way to represent and access the data present in the databases is of a crucial importance, and this point led us to ask ourselves the question of the most appropriate formalism for the data representation.

===Knowledge formalism choice===
The formalism of knowledge representation used in our approach must address the following three rules to properly fit our platform needs:

# Hierarchical data representation via concepts and properties
# Advanced reasoning possibility in order to extend the ontology or the dataset ruled by the ontology
# Efficient query time on the data to stay within interaction time

We mentioned previously that several formalisms exist to create ontologies and define databases. A quick comparison of these formalisms, complementary to their introduction in the previous section, can be found in Table 1.

{| 
 | STYLE="vertical-align:top;"|
{| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="80%"
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" colspan="6"|'''Table 1.''' Comparison of different knowledge representation formalisms with respect to key criteria
 |-
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"|Formalism
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"|Domain description
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"|Reasoning on knowledge
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"|Big data management
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"|Efficient
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"|Implementation flexibility
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Conceptual graphs
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|-
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Semantic networks
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|-
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Classical logics
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|-
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Description logics
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|X
  | style="background-color:white; padding-left:10px; padding-right:10px;"|-
 |-
|}
|}

Our first implementation of a semantic representation of knowledge in molecular biology was applied through conceptual graphs (CG) within Cogitant’s software.<ref name="HuangTheSPHINX93">{{cite journal |title=The SPHINX-II speech recognition system: An overview |journal=Computer Speech & Language |author=Huang, X.; Alleva, F.; Hsiao-Wuen, H. et al. |volume=7 |issue=2 |pages=137–148 |year=1993 |doi=10.1006/csla.1993.1007}}</ref> The use of CGs through the Cogitant API quickly proved to be incompatible with the constraints of the interactive context. This limitation had already been highlighted by the work of Yannick Dennemont<ref name="GenestAPlat98">{{cite journal |title=A platform allowing typed nested graphs: How CoGITo became CoGITaNT |journal=Proceedings from the 1998 International Conference on Conceptual Structures |author=Genest, D.; Salvat, E. |pages=1154–61 |year=1998 |doi=10.1007/BFb0054912}}</ref> with the Prolog CG API, limitations confirmed by our own experience with the Cogitant library in C++. The need for high performance imposed by the interactive context has led us to the path of description logic and semantic web for the representation of knowledge and the efficient extraction of information within a massive fact base to support Visual Analytics functionalities in molecular biology.

===Ontology for modeling of structural biology concepts===
An OWL-based ontology was implemented as the core of the platform, thereby creating a broad description of concepts an expert has to interact with during his/her visualization and analysis activities. We previously mentioned that several bio-ontologies already exist. We extended one of them, a bio-ontology [http://bioportal.bioontology.org/ontologies/AMINO-ACID describing amino acids] and their biophysical and geometrical properties to define the molecular objects and principles manipulated in structural biology. Each component structuring molecular complexes and each associated property coming from various common bio-informatics tools have been systematically defined and added to this ontology. However, since needs may vary, we have designed this ontology such that it could easily be updated and enriched with new concepts. A tiny subpart of our ontology is illustrated in Figure 3. Our ontology has been designed around five categories, addressing five different parts of our platform:

* Biomolecular knowledge – Field-related concepts and objects in structural biology
* 3D structure representation – Concepts related to the representation and visualization of 3D molecular complexes
* 2D data representation – Concepts related to the representation of numerical analyses and their results
* 3D interactions – Concepts related to the interactions in 3D environments
* 2D interactions – Concepts related to the interactions in 2D environments


[[File:Fig3 Trellet JOfIntegBioinfo2018 15-2.jpg|1000px]]
{{clear}}
{| 
 | STYLE="vertical-align:top;"|
{| border="0" cellpadding="5" cellspacing="0" width="1000px"
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;"| <blockquote>'''Figure 3.''' A part of our structural biology ontology used in our application</blockquote>
 |- 
|}
|}

The separation of the categories does not induce the absence of relationships between them. For instance, the “Atom” concept belongs to the "Biomolecular" knowledge category but is directly linked to the “Sphere” concept from 3D structure representation. The whole network of connections will then permit reasoning on the ontology in order to support the advanced interactivity level required in our platform.

Concepts and properties among the 3D structure representation and 2D data representation categories gather the graphical elements that allow for the representation of the "Biomolecular" knowledge category. Shape, colors, but also graph types are notions defined in these two categories. It is worth noting that analytical concepts are defined by graphical or abstract elements that play a role in the creation and visualization of an analytical result. However, we voluntarily chose not to define the different calculations and analyses related to molecular simulation data because of their high complexity and their heterogeneous nature varying significantly between the range of available specialized tools. This choice does not imply that the results of such analyses will not be used among the platform, merely that it is not relevant to include their definition in the ontology. The values of their results are nevertheless defined in the ontology under the form of properties of the individuals they bring to play.

In addition to the biomolecular concepts and representations previously cited, we also defined every concept around the interaction between the user and the data they will directly or indirectly manipulate. These interactions include commands proposed by most of the common visualization software packages and analysis tools.

Our full ontology is [https://raw.githubusercontent.com/mtrellet/PyMol_Interactive_Plotting/master/data/visual_analytics_owl.ttl publicly available online].

===Storing molecular data linked by a structural biology ontology===
Once we set up our ontology, it was possible to feed the database by adding biological information gathered by the expert. The new information has to fit the vocabulary and classification defined by the rules present in the ontology in order to be adequately stored in the database. This combination of ontology and knowledge base will form the RDF database (as illustrated later in this article in Figure 6).

The description of a molecular system is constructed from the analysis of any biological information that can be described by a character chain or a value and that corresponds to a concept or property identified in the ontology. Each information will be exhaustively gathered in the RDF database as triples. Within the scope of our study, we focused on numerical molecular simulations. These simulations output time series of static snapshots of the molecular system at a regular time step. The Hamiltonian of the simulated model will drive the system towards specific states that experts try to decipher in order to understand underlying molecular mechanisms. The whole simulation creates a trajectory where each state, at a precise time, is associated to a snapshot. Our ontology defines a snapshot by the "model" concept. A model gathers all the atom coordinates of the molecular system at a defined time step. In order to distinguish the different components of a system, these components are identified by "chain," another concept of our ontology. Each chain in the system is composed of a sequence of "residues" (also known as amino-acids in proteins). The different inference rules present in the ontology save us to specify all the links between the different hierarchical components of a specific model explicitly. As a result, a residue that belongs to a specific chain will be automatically associated to the corresponding model where the chain appears. Similarly, group of atoms, the smallest entities of a molecular structure at our scale, constitute residues and are then directly linked to chains and models.

Every geometrical property (position, angle, distance, etc.), physicochemical property (solvent accessibility, partial charge, bond, etc.), or analytical property (interaction energy, RMSD, temperature, etc.) is then integrated in the database and associated to individuals created from 3D structures (Model/Chain/Residue/Atom) for each step of the simulation. As a reminder, any individual is an instance of concepts defined in the ontology. Individuals and their properties form the population of the molecular database.

==Using semantic queries to support direct interactions for a new generation of molecular visualization applications==
Once all data has been integrated in the RDF database, it is necessary to set up an interrogation system able to retrieve the data for visualization and processing following interaction events in the working space. Our implementation of the query system mainly relies on the usage of SPARQL, as introduced before, and provides several ways to address the different needs of our platform.

===From vocal keywords to application command===
The richness and flexibility of SPARQL queries allowed us to design a keyword to command interpretation engine that aims to transform a list of keywords into a comprehensive application command triggering an action in the working space.

One of the most-widely used interactive techniques in immersive environments is the vocal command. Based on a vocal recognition process, it consists in translating a sentence or a group of words said by the user into an application command. Vocal commands have the strong advantage that they can be associated with gestures to express complex multimodal commands.

Most of the actions identified in our platform involve a structural group designated by the expert. These structural groups can be characterized by identifiers having a biological meaning (for example residue ids are, by convention, numbered from one extremity of the chain to the other), unique identifiers in the RDF database, or via their properties. The interpretation of commands vocalized by the expert with natural language using a specific field-related vocabulary requires a representation carrying the complexity of the knowledge and linking the objects targeted by the user to the virtual objects involved in the interaction.

For this purpose, we set up a process that takes as input a vocal command of the user and translates it into an application command for the operating system. This procedure can be divided in three main parts:

# Recognition of keywords from a vocal command
# Keyword classification into a decomposed command structure
# Creation of the final and operational command

Our conceptualization effort and the use of the ontology mainly focused on the second part. Parts one and three are more implementation oriented and will not be deeply described.

====Keyword recognition====
We are using the keyword spotting capability of [https://cmusphinx.github.io/ Sphinx]<ref name="DennemontUneAss13">{{cite web |url=https://www.semanticscholar.org/paper/Une-assistance-%C3%A0-l'interaction-3D-en-r%C3%A9alit%C3%A9-par-un-Dennemont/254289782f5feb44e0a0db19ea2f7661578241a1 |title=Une assistance à l'interaction 3D en réalité virtuelle par un raisonnement sémantique et une conscience du contexte |author=Dennemont, Y. |publisher=Allen Institute for Artificial Intelligence |date=2013}}</ref>, a vocal recognition toolkit, to recognize keywords. Based on a dictionary created from the ontology list of concepts, it aims to detect any word said by the user that would match a word present in the dictionary.

====Keyword classification====
Each keyword recognized in the previous step is assigned to a category. This classification is based on our ontology splitting, which identifies five categories of words that can be found in a vocal command, semantically modeled as:

* Action
* Component
* Identifier
* Property
* Representation

This classification is achieved through successive SPARQL queries to the ontology. Action, Component, Property, and Representation categories have their own concepts and can be identified by a unique word (“Hide,” “Chain,” “Charged,” “Sphere,” etc.). At the opposite, the Identifier category is linked to a concept instance from the Component category. A biological identifier is very likely to be redundant because of the repetition of the molecular system at each time step. Therefore it is mandatory to pair an identifier with a component in the keywords in order to validate its presence. Without component, any identifier is withdrawn from the list. If the identifier and the associated component exist in the database, the couple is validated.

SPARQL commands use the ASK operator to define whether a keyword belongs to a category or not. This operator takes one or several triples and returns a boolean that reflects whether the ensemble of triples is true or not with respect to the database. Some examples of queries can be found below:

<code>ASK {my:Hide rdfs:subClassOf my:Action}</code>

<code>ASK {my:Alanine rdfs:subClassOf my:Biological_component}</code>

<code>ASK {my:Cartoon rdfs:subClassOf my:Representation}</code>

<code>ASK {my:Aliphatic rdfs:subClassOf my:Property}</code>

Reasoning and inference rules are automatically used in SPARQL queries. For instance, the following query:

<code>ASK {my:Alanine rdfs:subClassOf my:Biological_component}</code>

will output true despite the absence of an explicit direct link between the two concepts (Alanine and Biological_component) since AminoAcid, Residue and Molecule are located between the two concepts (see Figure 4).


[[File:Fig4 Trellet JOfIntegBioinfo2018 15-2.jpg|1000px]]
{{clear}}
{| 
 | STYLE="vertical-align:top;"|
{| border="0" cellpadding="5" cellspacing="0" width="1000px"
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;"| <blockquote>'''Figure 4.''' Extract from our OWL ontology for the Alanine concept</blockquote>
 |- 
|}
|}

====Command creation====
Once each keyword is validated and associated to a category, e.g., identified as a concept of the database (or as an individual for identifiers) and eventually grouped with another keyword, it forms a syntactic group. Each syntactic group carries an information corresponding to a specific part of the application command.

In our platform, a vocal command is composed by a succession of syntactic groups linked between them to create an action query to the immersive platform. It is possible to describe the type of command that was defined in the following manner:

<tt>action [parameter]<sup>+</sup>, ( structural_group [identifier]<sup>+</sup> )<sup>+</sup></tt>

Syntactic groups between [] are optional, whereas others are mandatory. The + indicates the possibility to have 0, 1, or several occurrences of the syntactic group. Finally, () indicates a bloc of syntactic groups. This command architecture is present in our ontology under the form of pre-required concepts associated to the action concepts. For instance, the action concept "Color" requires a property of "Colors" type and a structural component to work with. These elements of information are then stored in the ontology, rendering them automatically checkable by the engine to detect whether all requirements are fulfilled for a specific action. This feature simplifies the definition of other actions in the ontology as the changes that have to be applied to the engine are minimal, typically either no or minor changes. The checking process will stay the same as long as the action is well-defined within the ontology.

At the same level as for an action, a structural group is always mandatory to trigger a command. The different ways to obtain a structural sub-ensemble are:

# Component only: every individual that belongs to the concept will be taken into account
# Combination of a component and an ensemble of identifiers: coherency checking between component and identifiers
# Property only: every individual that possesses the property will be taken into account
# Combination of a component and a property: coherency checking between component and property

The structural group always refers to a group of individuals in order to disambiguate the results between the commands. This disambiguation implies that final commands are more complex. The hierarchical classification between structural components (Model/Chain/Residue/Atom) has a significant impact on the results of a given command. Indeed, the nature of structural components targeted by an action will be compared to the nature of the structural components currently studied. Depending on whether the command individual will be of a higher or lower hierarchical order, the command might trigger an action either on a subpart of the displayed scene (for lower classified individuals) or as a scene composition changer (for higher or equal classified individuals). For instance, if only two models are studied when a vocal command is transmitted, putative amino acids individually targeted by an action will be the ones that belong to the two displayed models. If the individuals targeted by the command action would have been models, different from the displayed ones, an update of the displayed molecular complexes would have occurred first.

Once the different checks for the command coherency and validity have been carried out, the command is sent to both spaces (visualization and analysis) in order to synchronize the visual results.

====Performances====




==References==
{{Reflist|colwidth=30em}}

==Notes==
This presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added. The original references after 27 were slightly out of order in the original; due to the way this wiki works, references are listed in the order they appear. Footnotes were turned into inline URLs. Nothing else was changed in accordance with the NoDerivatives portion of the license.

<!--Place all category tags here-->
[[Category:LIMSwiki journal articles (added in 2019)‎]]
[[Category:LIMSwiki journal articles (all)‎]]
[[Category:LIMSwiki journal articles (with rendered math)‎]]
[[Category:LIMSwiki journal articles on data analysis]]
[[Category:LIMSwiki journal articles on data visualization]]