Journal:Handling metadata in a neurophysiology laboratory - Revision history

Shawndouglas: Finished adding rest of content.

2017-12-22T18:41:35Z

Finished adding rest of content.

← Older revision		Revision as of 18:41, 22 December 2017
Line 19:		Line 19:
	\|download = [https://www.frontiersin.org/articles/10.3389/fninf.2016.00026/pdf https://www.frontiersin.org/articles/10.3389/fninf.2016.00026/pdf] (PDF)		\|download = [https://www.frontiersin.org/articles/10.3389/fninf.2016.00026/pdf https://www.frontiersin.org/articles/10.3389/fninf.2016.00026/pdf] (PDF)
	}}		}}
	~~{{ombox~~
	~~\| type = content~~
	~~\| style = width: 500px;~~
	~~\| text = This article should not be considered complete until this message box has been removed. This is a work in progress.~~
	}}
	==Abstract==		==Abstract==
	To date, non-reproducibility of neurophysiological research is a matter of intense discussion in the scientific community. A crucial component to enhance reproducibility is to comprehensively collect and store metadata, that is, all information about the experiment, the data, and the applied preprocessing steps on the data, such that they can be accessed and shared in a consistent and simple manner. However, the complexity of experiments, the highly specialized analysis workflows, and a lack of knowledge on how to make use of supporting software tools often overburden researchers to perform such a detailed documentation. For this reason, the collected metadata are often incomplete, incomprehensible for outsiders, or ambiguous. Based on our research experience in dealing with diverse datasets, we here provide conceptual and technical guidance to overcome the challenges associated with the collection, organization, and storage of metadata in a neurophysiology [[laboratory]]. Through the concrete example of managing the metadata of a complex experiment that yields multi-channel recordings from monkeys performing a behavioral motor task, we practically demonstrate the implementation of these approaches and solutions, with the intention that they may be generalized to other projects. Moreover, we detail five use cases that demonstrate the resulting benefits of constructing a well-organized metadata collection when processing or analyzing the recorded data, in particular when these are shared between laboratories in a modern scientific collaboration. Finally, we suggest an adaptable workflow to accumulate, structure and store metadata from different sources using, by way of example, the odML metadata framework.		To date, non-reproducibility of neurophysiological research is a matter of intense discussion in the scientific community. A crucial component to enhance reproducibility is to comprehensively collect and store metadata, that is, all information about the experiment, the data, and the applied preprocessing steps on the data, such that they can be accessed and shared in a consistent and simple manner. However, the complexity of experiments, the highly specialized analysis workflows, and a lack of knowledge on how to make use of supporting software tools often overburden researchers to perform such a detailed documentation. For this reason, the collected metadata are often incomplete, incomprehensible for outsiders, or ambiguous. Based on our research experience in dealing with diverse datasets, we here provide conceptual and technical guidance to overcome the challenges associated with the collection, organization, and storage of metadata in a neurophysiology [[laboratory]]. Through the concrete example of managing the metadata of a complex experiment that yields multi-channel recordings from monkeys performing a behavioral motor task, we practically demonstrate the implementation of these approaches and solutions, with the intention that they may be generalized to other projects. Moreover, we detail five use cases that demonstrate the resulting benefits of constructing a well-organized metadata collection when processing or analyzing the recorded data, in particular when these are shared between laboratories in a modern scientific collaboration. Finally, we suggest an adaptable workflow to accumulate, structure and store metadata from different sources using, by way of example, the odML metadata framework.
Line 315:		Line 311:

	Indeed, while we believe that using a metadata framework such as odML represents an important step toward better data and metadata management, it is also clear that this approach has a number of potential improvements. A natural step would be for odML to become an intrinsic file format for the commercially available data acquisition systems, such that their metadata are instantly available for inclusion in the user's hierarchical tree. In this context, the odML format so far has been adopted in the [https://www.nitrc.org/projects/cff/ Connectome] file format, the [http://www.relacs.net/ Relacs] data acquisition and stimulation software, and the [https://eegdatabase.kiv.zcu.cz/ EEGBase]] database for EEG/ERP data.		Indeed, while we believe that using a metadata framework such as odML represents an important step toward better data and metadata management, it is also clear that this approach has a number of potential improvements. A natural step would be for odML to become an intrinsic file format for the commercially available data acquisition systems, such that their metadata are instantly available for inclusion in the user's hierarchical tree. In this context, the odML format so far has been adopted in the [https://www.nitrc.org/projects/cff/ Connectome] file format, the [http://www.relacs.net/ Relacs] data acquisition and stimulation software, and the [https://eegdatabase.kiv.zcu.cz/ EEGBase]] database for EEG/ERP data.

			Likewise, the availability of interfaces for easy odML export in popular experiment control suites, vendor-specific hardware or generic software, such as LabVIEW, would greatly speed up metadata generation. Additionally, a repository for popular terminologies and hardware devices has been initiated by the [http://portal.g-node.org/odml/terminologies/v1.0/terminologies.xml German Neuroinformatics Node], which is open for any extensions by the community. More importantly, in conditions where the details of recording or post-processing steps may change over time, it is essential to keep track also of the versions of the code that generated a certain post-processing result, including the corresponding libraries used by the code and installed on the computer executing the code. Popular solutions to version control (e.g., Git) or provenance tracking (e.g., Sumatra<ref name="DavisonSumatra14" />) offer mechanisms to keep track of this information. However, it is up to the user to make sure that information about the correct version numbers or hash values provided by these systems are saved to the file containing the metadata collection to guarantee that metadata can be linked to its provenance trail. The more direct integration of support for metadata recording into the various tools performing the post-processing would allow to automatize this process, leading to a more robust metadata collection, paired with enhanced usability.

			A perhaps more challenging problem is devising mechanisms that link the metadata to the actual data objects they refer or relate to. This issue becomes particularly clear in the context of the Neo library<ref name="GarciaNeo14" />, which is an open-source Python package that provides data objects for storing electrophysiological data, along with file input/output for common file formats. The Neo library could be used to read a particular spike train that relates to a certain unit ID in the recording, while the corresponding odML file would contain for each unit ID the information about the assigned unit type (SUA, MUA, or noise) and the signal-to-noise ratio (SNR) obtained by the spike sorting preprocessing step. A common task would now be to link these two pieces of information in a generic way. Currently, it is up to the user to manually extract the SNR of each neuron ID from the metadata, and then annotate the spike train data with this particular piece of metadata. This is a procedure that is time-consuming and again may be performed differently by partners in a collaboration causing incoherence in the workflow. Recently, the NWB format<ref name="TeetersNeurodata15">{{cite journal \|title=Neurodata Without Borders: Creating a Common Data Format for Neurophysiology \|journal=Neuron \|author=Teeters, J.L.; Godfrey, K.; Young, R. et al. \|volume=88 \|issue=4 \|pages=629-34 \|year=2015 \|doi=10.1016/j.neuron.2015.10.025 \|pmid=26590340}}</ref> was proposed as a file format to store electrophysiological data with a detailed, use-case specific data and metadata organization. In contrast, the [http://www.g-node.org/nix NIX] file format<ref name="StoewerFile14">{{cite journal \|title=File format and library for neuroscience data and metadata \|journal=Frontiers in Neuroinformatics \|author=Stoewer, A.; Keller, C.J.; Benda, J. et al. \|year=2014 \|doi=10.3389/conf.fninf.2014.18.00027}}</ref> was proposed as a more general solution to link data and metadata already on the file level. With this approach, metadata are organized hierarchically, as in odML, but can be linked to the respective data stored in the same file. This enables relating data and metadata meaningfully to facilitate and automate data retrieval and analysis.

			Finally, convenient manual metadata entry is an important requirement for collecting metadata of an experiment. In the case of odML, while the existing editor enables researchers to fill in odML files, a number of convenience features would not only reduce the amount of time required to collect the information but could also provide further incentives to store metadata in the odML format. An example for the former could be editor support for templates such that new files with default values may be created quickly, while an example for the latter could be to provide more flexible ways to display metadata in the editor. The odML-tables library, which can transform the hierarchical structure of an odML file to an editable flat table in the Excel or CSV format, is one current attempt to solve these issues in the odML framework. The conversion of odML to commonly known spreadsheets increases the accessibility of odML for collaborators with little programming knowledge. Another particularly notable project aimed to improve the manual entries of metadata during an experiment is the odML mobile app<ref name="LeFrancMobile14">{{cite journal \|title=Mobile metadata: bringing Neuroinformatics tools to the bench \|journal=Frontiers in Neuroinformatics \|author=Le Franc, Y.; Gonzalez, D.; Mylyanyk, I. et al. \|year=2014 \|doi=10.3389/conf.fninf.2014.18.00053}}</ref> that runs on mobile devices that are easy to carry around in a lab situation.

			The odML framework by itself is of a general nature and can easily be used in other domains of neuroscience than electrophysiology, or even other fields of science. An obvious use case would be to store metadata of neuroscientific simulation experiments. As we witness a similar increase in the complexity of ''in silico'' data, providing adequate metadata records to describe these data gains importance. However, datasets emerging from simulations differ in the composition of their metadata in terms of the more advanced technical descriptions required to capture the mathematical details of the employed models (e.g., descriptions using NeuroML<ref name="GleesonNeuroML10" /><ref name="CrookCreating12" />), and by the fact that simulations are often described on a procedural rather than a declarative level (e.g., descriptions based on PyNN<ref name="DavisonPyNN09">{{cite journal \|title=PyNN: A Common Interface for Neuronal Network Simulators \|journal=Frontiers in Neuroinformatics \|author=Davison, A.P.; Brüderle, D.; Eppler, J. et al. \|volume=2 \|pages=11 \|year=2009 \|doi=10.3389/neuro.11.011.2008 \|pmid=19194529 \|pmc=PMC2634533}}</ref>). How these can be best linked to more generic standards for metadata capture and representation, and what level of description of metadata is adequate in this scenario, remains a matter of investigation supported by use-cases, as performed here for experimental data. A common storage mechanism for metadata of experimental and simulated data would simplify their comparison, a task that is bound to become increasingly important for the future of computational neuroscience. Using a well-defined, machine-readable format for metadata brings the potential for integration of the information across heterogeneous datasets, for example in larger databases or data repositories.

			In summary, the complexity of current electrophysiological experiments forces the scientific community to reorganize their workflow of data handling, including metadata management, to ensure reproducibility in research.<ref name="StoddenImplem14">{{cite book \|title=Implementing Reproducible Research \|editor=Stodden, V.; Leisch, F.; Peng, R.D. \|publisher=CRC Press \|pages=448 \|year=2014 \|isbn=9781466561595}}</ref> Readily available tools to support metadata management, such as odML, are a vital component in constructing such workflows. It is our responsibility to propagate and incorporate these tools into our daily routines in order to improve workflows through the principle of co-design between scientists and software engineers.

			==Author contributions==
			LZ, MD, and SG designed the research. LZ, TB, and AR performed the research. LZ, MD, JG, FJ, ASo, ASt, and TW contributed to unpublished software tools. All authors participated in writing the paper.

			==Conflict of interest statement==
			The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

			==Acknowledgments==
			We commemorate Paul Chorley and thank him for his valuable input during the development of the metadata structures and templates. We thank Benjamin Weyers and Christian Kellner for valuable discussions. This work was partly supported by Helmholtz Portfolio Supercomputing and Modeling for the Human Brain (SMHB), Human Brain Project (HBP, EU Grant 604102), German Neuroinformatics Node (G-Node, BMBF Grant 01GQ1302), BrainScaleS (EU Grant 269912), and DFG SPP Priority Program 1665 (GR 1753/4-1 and DE 2175/1-1). ANR-GRASP, CNRS, and Riken-CNRS Research Agreement.

			==Supplementary material==
			The supplementary material for this article can be found [https://www.frontiersin.org/articles/file/downloadfile/195830_supplementary-materials_presentations_1_pdf/octet-stream/Presentation%201.pdf/2/195830 here].

	==References==		==References==

Shawndouglas: Saving and adding more.

2017-12-22T18:12:04Z

Saving and adding more.

← Older revision		Revision as of 18:12, 22 December 2017
Line 311:		Line 311:

	As a result of our experience with complex analysis of experiments in systems neuroscience as reported in this paper, one of our recommendations is to record as much information about the experiment as possible. This may seem in contrast to efforts specifying minimal information guidelines in the life sciences (MIBBI<ref name="TaylorPromoting08">{{cite journal \|title=Promoting coherent minimum reporting guidelines for biological and biomedical investigations: The MIBBI project \|journal=Nature Biotechnology \|author=Taylor, C.F.; Field, D.; Sansone, S.A. et al. \|volume=26 \|issue=8 \|pages=889-96 \|year=2008 \|doi=10.1038/nbt.1411 \|pmid=18688244 \|pmc=PMC2771753}}</ref> and MINI<ref name="GibsonMinimum09">{{cite journal \|title=Minimum Information about a Neuroscience Investigation (MINI): Electrophysiology \|journal=Nature Precedings \|author=Gibson, F.; Overton, P.G.; Smulders, T.V. et al. \|year=2009 \|url=http://precedings.nature.com/documents/1720/version/2}}</ref>). However, those initiatives target the use case where data are uploaded to a public database, and their goal is to achieve a balance of information detail such that the minimally sufficient information is provided to make the data potentially useful for the community. They are not meant as guidelines for procedures in the laboratory. To ensure reproducibility of the primary analysis, but also in the interest of future reuse of the data, we argue that it is highly desirable to store all potentially relevant information about an experiment.		As a result of our experience with complex analysis of experiments in systems neuroscience as reported in this paper, one of our recommendations is to record as much information about the experiment as possible. This may seem in contrast to efforts specifying minimal information guidelines in the life sciences (MIBBI<ref name="TaylorPromoting08">{{cite journal \|title=Promoting coherent minimum reporting guidelines for biological and biomedical investigations: The MIBBI project \|journal=Nature Biotechnology \|author=Taylor, C.F.; Field, D.; Sansone, S.A. et al. \|volume=26 \|issue=8 \|pages=889-96 \|year=2008 \|doi=10.1038/nbt.1411 \|pmid=18688244 \|pmc=PMC2771753}}</ref> and MINI<ref name="GibsonMinimum09">{{cite journal \|title=Minimum Information about a Neuroscience Investigation (MINI): Electrophysiology \|journal=Nature Precedings \|author=Gibson, F.; Overton, P.G.; Smulders, T.V. et al. \|year=2009 \|url=http://precedings.nature.com/documents/1720/version/2}}</ref>). However, those initiatives target the use case where data are uploaded to a public database, and their goal is to achieve a balance of information detail such that the minimally sufficient information is provided to make the data potentially useful for the community. They are not meant as guidelines for procedures in the laboratory. To ensure reproducibility of the primary analysis, but also in the interest of future reuse of the data, we argue that it is highly desirable to store all potentially relevant information about an experiment.

			We have exemplified the practical issues of metadata management using the odML metadata framework as one particular method. Similar results could be obtained using other formats. [https://www.w3.org/RDF/ RDF] is a powerful standard approach specifically designed for semantic annotation of data. Many libraries and tools are available for this format, and in combination with ontologies it is highly suitable for standardization. However, efficiently utilizing this format requires elaborate technology that is not easy to use. Simpler formats like [http://www.json.org/ JSON] or [http://yaml.org/ YAML] are in many respects similar to the XML schema used for odML, and we would expect that they have been used in individual labs to realize approaches similar to the one presented here. We are, however, not aware of specific tools available for the collection of metadata that use these formats. In addition, none of these alternative formats provide specific support for storing measured quantities as odML does. Using a combination of XML and [https://www.hdfgroup.org/HDF5/ HDF<sub>5</sub>] has been proposed to define a format for scientific data<ref name="MillardAdapt11">{{cite journal \|title=Adaptive informatics for multifactorial and high-content biological data \|journal=Nature Methods \|author=Millard, B.L.; Niepel, M.; Menden, M.P. et al. \|volume=8 \|issue=6 \|pages=487-93 \|year=2011 \|doi=10.1038/nmeth.1600 \|pmid=21516115 \|pmc=PMC3105758}}</ref>, which could in principle be used for metadata collection. However, requiring extensive schema definitions it is much less flexible and lightweight than odML. Solutions for the management of scientific workflows, like [http://www.taverna.org.uk/ Taverna], [https://kepler-project.org/ Kepler], [http://www.vistrails.org/ VisTrails], [https://www.knime.org/knime/ KNIME], and [http://www.wings-workflows.org/ Wings]<ref name="INCF13">{{cite web \|url=https://space.incf.org/index.php/s/kuFRRGBezreEYPJ#pdfviewer \|title=INCF Program on Standards for Data Sharing: New perspectives on workflows and data management for the analysis of electrophysiological data \|author=INCF \|date=4 December 2013}}</ref> are targeted toward standardized and reproducible data processing workflows. We are focusing here on the management of experimental metadata, and a consideration of data processing would go beyond this scope. Workflow management systems could be utilized for managing metadata, but their suitability for the collection of metadata during the experiment seems limited. Nevertheless, the approach we have described here must ultimately be combined with such systems, as the development of automated workflows depends critically on the availability of a metadata collection. Likewise, our approach is suitable for combination with provenance tracking solutions like Sumatra.<ref name="DavisonSumatra14">{{cite book \|chapter=Sumatra: A toolkit for reproducible research \|title=Implementing Reproducible Research \|author=Davison, A.P.; Mattioni, M.; Samarkanov, D.; Teleńczuk, B. \|editor=Stodden, V.; Leisch, F.; Peng, R.D. \|publisher=CRC Press \|pages=57–79 \|year=2014 \|isbn=9781466561595}}</ref>

			Indeed, while we believe that using a metadata framework such as odML represents an important step toward better data and metadata management, it is also clear that this approach has a number of potential improvements. A natural step would be for odML to become an intrinsic file format for the commercially available data acquisition systems, such that their metadata are instantly available for inclusion in the user's hierarchical tree. In this context, the odML format so far has been adopted in the [https://www.nitrc.org/projects/cff/ Connectome] file format, the [http://www.relacs.net/ Relacs] data acquisition and stimulation software, and the [https://eegdatabase.kiv.zcu.cz/ EEGBase]] database for EEG/ERP data.

	==References==		==References==

Shawndouglas: Saving and adding more.

2017-12-21T21:36:21Z

Saving and adding more.

← Older revision		Revision as of 21:36, 21 December 2017
Line 309:		Line 309:

	While the storage of metadata to improve the overall experimental and analysis workflow is technically feasible, it cannot solve the intrinsic problem of identifying the metadata in an experiment, collecting the individual metadata, and pooling them in an automatic way. All of these steps are not trivial and are time-consuming by nature, in particular when organizing metadata for a particular experiment for the first time. Currently, no funding is granted for such tasks, and software tools that support the automated generation and usage of metadata are largely missing. Therefore, rightly the question may arise whether it is truly worth the effort. Through the illustration of use cases, we hope to have convinced the reader that indeed numerous advantages are associated with a well-maintained metadata collection that accompanies the data. For the individual researcher these can possibly be best summarized by the ease of organizing, searching, and selecting datasets based on the metadata information. Even more advantages are gained for collaborative work. First, an easily accessible central source for metadata information ensures that all researchers access the exact same information. This is particularly important if metadata are difficult to access from the source files, hand-written notebooks or via a specialized program code, or if the metadata need to be collected from different locations. Second, the communication between collaborators becomes more precise by the strict use of defined property-value pairs, lowering the probability for unintended confusion. Last but not least, once the metadata collection structure, content, and method of creation are defined by the experimenter, metadata entry and compilation will in general run very smoothly, be less error prone, and will even save time in comparison to traditional methods. Thus, we believe that there are a number of significant advantages that justify the initial investment of including automated metadata handling as part of the workflow of an electrophysiological experiment. The survey results presented above reveal that these advantages are also increasingly recognized by the community. Furthermore, as scientists we have the obligation to properly document our scientific work in a clear fashion that enables the highest degree of reproducibility. Reproducibility becomes increasingly recognized by publishers and funding agencies<ref name="MorrisonTime14">{{cite journal \|title=Time to do something about reproducibility \|journal=eLife \|author=Morrison, S.J. \|volume=4 \|pages=e039841 \|year=2014 \|doi=10.7554/eLife.03981 \|pmid=25493617 \|pmc=PMC4260475}}</ref><ref name="CandelaData15">{{cite journal \|title=Data journals: A survey \|journal=Journal of the Association for Information Science and Technology \|author=Candela, L.; Castelli, D.; Manghi, P.; Tani, A. \|volume=66 \|issue=9 \|pages=1747–1762 \|year=2015 \|doi=10.1002/asi.23358}}</ref><ref name="OSCPsychology15" /><ref name="PulvererRepro15">{{cite journal \|title=Reproducibility blues \|journal=EMBO Journal \|author=Pulverer, B. \|volume=34 \|issue=22 \|pages=2721-4 \|year=2015 \|doi=10.15252/embj.201570090 \|pmid=26538323 \|pmc=PMC4682652}}</ref>, such that appropriate resources of time and man-power allocated to produce more sophisticated data management will become a necessity. It is thus not only important to gain experience in how to document data properly but also to produce better tools that reduce the time investment required for these steps.		While the storage of metadata to improve the overall experimental and analysis workflow is technically feasible, it cannot solve the intrinsic problem of identifying the metadata in an experiment, collecting the individual metadata, and pooling them in an automatic way. All of these steps are not trivial and are time-consuming by nature, in particular when organizing metadata for a particular experiment for the first time. Currently, no funding is granted for such tasks, and software tools that support the automated generation and usage of metadata are largely missing. Therefore, rightly the question may arise whether it is truly worth the effort. Through the illustration of use cases, we hope to have convinced the reader that indeed numerous advantages are associated with a well-maintained metadata collection that accompanies the data. For the individual researcher these can possibly be best summarized by the ease of organizing, searching, and selecting datasets based on the metadata information. Even more advantages are gained for collaborative work. First, an easily accessible central source for metadata information ensures that all researchers access the exact same information. This is particularly important if metadata are difficult to access from the source files, hand-written notebooks or via a specialized program code, or if the metadata need to be collected from different locations. Second, the communication between collaborators becomes more precise by the strict use of defined property-value pairs, lowering the probability for unintended confusion. Last but not least, once the metadata collection structure, content, and method of creation are defined by the experimenter, metadata entry and compilation will in general run very smoothly, be less error prone, and will even save time in comparison to traditional methods. Thus, we believe that there are a number of significant advantages that justify the initial investment of including automated metadata handling as part of the workflow of an electrophysiological experiment. The survey results presented above reveal that these advantages are also increasingly recognized by the community. Furthermore, as scientists we have the obligation to properly document our scientific work in a clear fashion that enables the highest degree of reproducibility. Reproducibility becomes increasingly recognized by publishers and funding agencies<ref name="MorrisonTime14">{{cite journal \|title=Time to do something about reproducibility \|journal=eLife \|author=Morrison, S.J. \|volume=4 \|pages=e039841 \|year=2014 \|doi=10.7554/eLife.03981 \|pmid=25493617 \|pmc=PMC4260475}}</ref><ref name="CandelaData15">{{cite journal \|title=Data journals: A survey \|journal=Journal of the Association for Information Science and Technology \|author=Candela, L.; Castelli, D.; Manghi, P.; Tani, A. \|volume=66 \|issue=9 \|pages=1747–1762 \|year=2015 \|doi=10.1002/asi.23358}}</ref><ref name="OSCPsychology15" /><ref name="PulvererRepro15">{{cite journal \|title=Reproducibility blues \|journal=EMBO Journal \|author=Pulverer, B. \|volume=34 \|issue=22 \|pages=2721-4 \|year=2015 \|doi=10.15252/embj.201570090 \|pmid=26538323 \|pmc=PMC4682652}}</ref>, such that appropriate resources of time and man-power allocated to produce more sophisticated data management will become a necessity. It is thus not only important to gain experience in how to document data properly but also to produce better tools that reduce the time investment required for these steps.

			As a result of our experience with complex analysis of experiments in systems neuroscience as reported in this paper, one of our recommendations is to record as much information about the experiment as possible. This may seem in contrast to efforts specifying minimal information guidelines in the life sciences (MIBBI<ref name="TaylorPromoting08">{{cite journal \|title=Promoting coherent minimum reporting guidelines for biological and biomedical investigations: The MIBBI project \|journal=Nature Biotechnology \|author=Taylor, C.F.; Field, D.; Sansone, S.A. et al. \|volume=26 \|issue=8 \|pages=889-96 \|year=2008 \|doi=10.1038/nbt.1411 \|pmid=18688244 \|pmc=PMC2771753}}</ref> and MINI<ref name="GibsonMinimum09">{{cite journal \|title=Minimum Information about a Neuroscience Investigation (MINI): Electrophysiology \|journal=Nature Precedings \|author=Gibson, F.; Overton, P.G.; Smulders, T.V. et al. \|year=2009 \|url=http://precedings.nature.com/documents/1720/version/2}}</ref>). However, those initiatives target the use case where data are uploaded to a public database, and their goal is to achieve a balance of information detail such that the minimally sufficient information is provided to make the data potentially useful for the community. They are not meant as guidelines for procedures in the laboratory. To ensure reproducibility of the primary analysis, but also in the interest of future reuse of the data, we argue that it is highly desirable to store all potentially relevant information about an experiment.

	==References==		==References==

Shawndouglas: Saving and adding more.

2017-12-21T21:26:32Z

Saving and adding more.

Show changes

Shawndouglas: Cite fix

2017-12-21T00:19:44Z

Cite fix

← Older revision		Revision as of 00:19, 21 December 2017
Line 35:		Line 35:

	==Organizing metadata in neurophysiology==		==Organizing metadata in neurophysiology==
	Metadata are generally defined as data describing data.<ref name="BacaIntro08">{{cite book \|url=http://www.getty.edu/publications/intrometadata/ \|title=Introduction to Metadata \|editor=Baca, M. publisher=Getty Publications \|edition=3rd \|year=2016}}</ref><ref name="MWMetadata">{{cite web \|url=https://www.merriam-webster.com/dictionary/metadata \|title=Metadata \|work=Merriam-Webster.com \|publisher=Merriam-Webster \|accessdate=06 July 2016}</ref> More specifically, metadata are information that describe the conditions under which a certain dataset has been recorded.<ref name="GreweABottom11" /> Ideally all metadata would be machine-readable and available at a single location that is linked to the corresponding recorded dataset. The fact that such central, comprehensive metadata collections are not common practice already today is by no means a sign of negligence on the part of the scientists, but is explained by the fact that in the absence of conceptual guidelines and software support, such a situation is extremely difficult to achieve given the high complexity of the task. Already, the fact that an electrophysiological setup is composed of several hardware and software components, often from different vendors, imposes the need to handle multiple files of different formats. Some files may even contain metadata that are not machine-readable and -interpretable. Furthermore, performing an experiment requires the full attention of the experimenters, which limits the amount of metadata that can be manually captured online. Metadata that arise unexpectedly during an experiment, e.g., the cause of a sudden noise artifact, are commonly documented as handwritten notes in the laboratory notebook. In fact, handwritten notes are often unavoidable, because the legal regulations of some countries, e.g., [http://www.cnrs.fr/infoslabos/cahier-laboratoire/docs/cahierlabo.pdf France], require the documentation of experiments in the form of a handwritten laboratory notebook.		Metadata are generally defined as data describing data.<ref name="BacaIntro08">{{cite book \|url=http://www.getty.edu/publications/intrometadata/ \|title=Introduction to Metadata \|editor=Baca, M. publisher=Getty Publications \|edition=3rd \|year=2016}}</ref><ref name="MWMetadata">{{cite web \|url=https://www.merriam-webster.com/dictionary/metadata \|title=Metadata \|work=Merriam-Webster.com \|publisher=Merriam-Webster \|accessdate=06 July 2016}}</ref> More specifically, metadata are information that describe the conditions under which a certain dataset has been recorded.<ref name="GreweABottom11" /> Ideally all metadata would be machine-readable and available at a single location that is linked to the corresponding recorded dataset. The fact that such central, comprehensive metadata collections are not common practice already today is by no means a sign of negligence on the part of the scientists, but is explained by the fact that in the absence of conceptual guidelines and software support, such a situation is extremely difficult to achieve given the high complexity of the task. Already, the fact that an electrophysiological setup is composed of several hardware and software components, often from different vendors, imposes the need to handle multiple files of different formats. Some files may even contain metadata that are not machine-readable and -interpretable. Furthermore, performing an experiment requires the full attention of the experimenters, which limits the amount of metadata that can be manually captured online. Metadata that arise unexpectedly during an experiment, e.g., the cause of a sudden noise artifact, are commonly documented as handwritten notes in the laboratory notebook. In fact, handwritten notes are often unavoidable, because the legal regulations of some countries, e.g., [http://www.cnrs.fr/infoslabos/cahier-laboratoire/docs/cahierlabo.pdf France], require the documentation of experiments in the form of a handwritten laboratory notebook.

	To present the concept of metadata management in a practical context, we introduce in the following a selected electrophysiological experiment. For clarity, a graphical summary of the behavioral task and the recording setup is provided in Figure 1, and a list of task-related abbreviations (e.g., behavioral conditions and events) is listed in Table 1. First described in Riehle ''et al.''<ref name="RiehleMapping13" /> and Milekovic ''et al.''<ref name="MilekovicLocal15">{{cite journal \|title=Local field potentials in primate motor cortex encode grasp kinetic parameters \|journal=Neuroimage \|author=Milekovic, T.; Truccolo, W.; Grün, S. et al. \|volume=114 \|pages=338–55 \|year=2015 \|doi=10.1016/j.neuroimage.2015.04.008 \|pmid=25869861 \|pmc=PMC4562281}}</ref>, the experiment employs high density multi-electrode recordings to study the modulation of spiking and LFP activities in the motor/pre-motor cortex of monkeys performing an instructed delayed reach-to-grasp task. In total, three monkeys (Macaca mulatta; two females, L, T; one male, N) were trained to grasp an object using one of two different grip types (side grip, SG, or precision grip, PG) and to pull it against one of two possible loads requiring either a high (HF) or low (LF) pulling force (cf. Figure 1). In each trial, instructions for the requested behavior were provided to the monkeys through two consecutive visual cues (C and GO) which were separated by a one-second delay and generated by the illumination of specific combinations of five LEDs positioned above the object (cf. Figure 1). The complexity of this study makes it particularly suitable to expose the difficulty of collecting, organizing, and storing metadata. Moreover, metadata were first organized according to the laboratory internal procedures and practices for documentation, while the organization of the metadata into a comprehensive machine-readable format was done a-posteriori, after the experiment was completed. To work with the data and metadata of this experiment, one has to handle on average 300 recording sessions per monkey, with data distributed over three files per session, metadata distributed over five files per implant, at least 10 files per recording, and one file for general information of the experiment (cf. Table 2). This situation imposed an additional complexity to reorganize the various metadata sources.		To present the concept of metadata management in a practical context, we introduce in the following a selected electrophysiological experiment. For clarity, a graphical summary of the behavioral task and the recording setup is provided in Figure 1, and a list of task-related abbreviations (e.g., behavioral conditions and events) is listed in Table 1. First described in Riehle ''et al.''<ref name="RiehleMapping13" /> and Milekovic ''et al.''<ref name="MilekovicLocal15">{{cite journal \|title=Local field potentials in primate motor cortex encode grasp kinetic parameters \|journal=Neuroimage \|author=Milekovic, T.; Truccolo, W.; Grün, S. et al. \|volume=114 \|pages=338–55 \|year=2015 \|doi=10.1016/j.neuroimage.2015.04.008 \|pmid=25869861 \|pmc=PMC4562281}}</ref>, the experiment employs high density multi-electrode recordings to study the modulation of spiking and LFP activities in the motor/pre-motor cortex of monkeys performing an instructed delayed reach-to-grasp task. In total, three monkeys (Macaca mulatta; two females, L, T; one male, N) were trained to grasp an object using one of two different grip types (side grip, SG, or precision grip, PG) and to pull it against one of two possible loads requiring either a high (HF) or low (LF) pulling force (cf. Figure 1). In each trial, instructions for the requested behavior were provided to the monkeys through two consecutive visual cues (C and GO) which were separated by a one-second delay and generated by the illumination of specific combinations of five LEDs positioned above the object (cf. Figure 1). The complexity of this study makes it particularly suitable to expose the difficulty of collecting, organizing, and storing metadata. Moreover, metadata were first organized according to the laboratory internal procedures and practices for documentation, while the organization of the metadata into a comprehensive machine-readable format was done a-posteriori, after the experiment was completed. To work with the data and metadata of this experiment, one has to handle on average 300 recording sessions per monkey, with data distributed over three files per session, metadata distributed over five files per implant, at least 10 files per recording, and one file for general information of the experiment (cf. Table 2). This situation imposed an additional complexity to reorganize the various metadata sources.

Shawndouglas: Saving and adding more.

2017-12-21T00:18:36Z

Saving and adding more.

← Older revision		Revision as of 00:18, 21 December 2017
Line 140:		Line 140:

	In summary, compared to Scenario 1, Scenario 2 improved the workflow of selecting datasets according to certain criteria in two aspects: (i) to check for the selection criteria, only one metadata file per recording needs to be loaded, and (ii) to extract metadata stored in a standardized format, a loading routine is already available. Thus, in Scenario 2 the scripts for automatized data selection are less complicated, which improved the reproducibility of the operation.		In summary, compared to Scenario 1, Scenario 2 improved the workflow of selecting datasets according to certain criteria in two aspects: (i) to check for the selection criteria, only one metadata file per recording needs to be loaded, and (ii) to extract metadata stored in a standardized format, a loading routine is already available. Thus, in Scenario 2 the scripts for automatized data selection are less complicated, which improved the reproducibility of the operation.

			====Use Case 4: Metadata screening====
			<blockquote>Sometimes it can be helpful to gain an overview of the metadata of an entire experiment. Such a screening process is often negatively influenced by the following aspects: (i) Some metadata are stored along with the actual electrophysiological data, (ii) metadata are often distributed over several files and formats, and (iii) some metadata need to be computed from other metadata. All three aspects slow down the screening procedure and complicate the corresponding code. Use Case 4 demonstrates how a comprehensive metadata collection improves the speed and reproducibility of a metadata screening procedure.</blockquote>

			After generating for each weekday a corresponding list of recording file names (see Use Case 3, Section 3.3), Bob would like to generate an overview figure summarizing the set of metadata that best reflect the performance of the monkey during the selected recordings (Figure 3). This set includes the following metadata: the RTs of the trials for each recording (Figure 3B), the number of correct vs. error trials (Figure 3C), and the total duration of the recording (Figure 3D). To exclude a bias due to a variable distribution of the different task conditions, Bob also wants to include the trial type combinations and their sequential order (Figure 3A).


			[[File:Fig3 Zehl FrontInNeuro2016 10.jpg\|922px]]
			{{clear}}
			{\|
			\| STYLE="vertical-align:top;"\|
			{\| border="0" cellpadding="5" cellspacing="0" width="922px"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\| <blockquote>'''Figure 3.''' Overview of reach-to-grasp metadata summarizing the performance of monkey L on each weekday. '''(A)''' displays the order of trial sequences of the first 50 trials (small bars along the x-axis) for each of all sessions (rows along the y-axis) recorded on the corresponding weekday. Each small bar corresponds to a trial and its color to the requested trial type of the trial (see color legend below subplot (A); for trial type acronyms see main text). '''(B)''' summarizes the median reaction times (RTs) for all trials of the same trial type (see color legend) of all sessions recorded on the corresponding weekday. '''(C)''' shows for each weekday the total number of trials of all sessions differentiating between correct and error trials (colored in black and gray) and between sessions where the monkey was first informed about the grip type (grip first) compared to sessions where the first cue informed about the force type (force first). '''(D)''' displays for each weekday the median recording duration of the grip first and force first sessions.</blockquote>
			\|-
			\|}
			\|}

			To create the overview figure in Scenario 1, Bob has to load the .nev data file (Label 5 in Figure 1 and Table 2) in which most selection criteria are stored, and additionally the .ns2 data file (Label 6b in Figure 1 and Table 2) to extract the duration of the recording. For both files, Bob is able to use available loading routines, but for accessing the desired metadata, he always has to load the complete data files. Depending on the data size this processing can be very time consuming. In Scenario 2, Bob is able to directly and efficiently extract all metadata from one comprehensive metadata file without having to load the neuronal data in parallel.

			In summary, compared to Scenario 1, the workflow of creating an overview of certain metadata of an entire experiment is improved in Scenario 2 by reducing the number of metadata files which need to be screened to one per recording, and by drastically lowering the run time to collect the criteria used in the figure. In addition, Bob benefits from better reproducibility as in Use Case 3.

			===Use Case 5: Metadata queries for data selection===
			<blockquote>It is common that datasets are analyzed not only by members of the experimenter's lab but also collaborators. Two difficulties may arise in this context. First, the partners will often base their work on different workflow strategies and software technologies, making it difficult to share their code, in particular code that is used to access data objects. Second, the geographical separation represents a communication barrier resulting from infrequent and impersonal communication by telephone, chat, or email, requiring extra care in conveying relevant information to the partner in a precise way. Use Case 5 demonstrates how a standard format used to save the comprehensive metadata collection improves cross-lab collaborations by formalizing the communication process through the use of queries on the metadata.</blockquote>

			Alice has detected a systematic noise artifact in the LFP signals of some channels in recording sessions performed in July (possibly due to additional air conditioning). As a consequence, Alice decided to exclude recordings performed in July from her LFP spectral analyses. Alice collaborates with Carol to perform complementary spike correlation analyses on an identical subset of recordings in order to find out if the network correlation structure is affected by task performance. To ensure that they analyze exactly the same datasets, the best data selection solution is to rely on metadata information that is located in the data files, rather than error-prone measures such as interpreting the file name or file creation date.

			In Scenario 1, since Alice and Carol use different programming languages (MATLAB and Python), they need to ensure that their routines extract the same recording date. This procedure will require providing corresponding cross-validations between their MATLAB and Python routines to extract identical dates.

			In Scenario 2, Alice and Carol agree on a concrete formal specification of the dataset selection (Figure 4) via the comprehensive metadata collection. In such a specification, metadata are stored in a defined format that reflects the structure of key-value pairs: in our example, Alice would specify the data selection by telling Carol to allow for only those recording sessions where the key month has the exact value "July." For the chosen standardized format of the collection, the metadata query can be handled by an application program interface (API) available both at the MATLAB and Python levels for Alice and Carol, respectively. Therefore, the query is guaranteed to produce the same result for both scientists. The formalization of such metadata queries will result in a more coherent and less error-prone synchronization between the work in the two laboratories.


			[[File:Fig4 Zehl FrontInNeuro2016 10.jpg\|400px]]
			{{clear}}
			{\|
			\| STYLE="vertical-align:top;"\|
			{\| border="0" cellpadding="5" cellspacing="0" width="400px"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\| <blockquote>'''Figure 4.''' Schematic workflow of selecting data across laboratories based on available API for a comprehensive metadata collection. In this optimized workflow, Alice and Carol are both able to identically select data from the common repository by screening the comprehensive metadata collection for the selection criteria (month == July) via the available APIs for MATLAB and Python. Such a formalized query in data selection supports a well-defined collaborative workflow across laboratories.</blockquote>
			\|-
			\|}
			\|}

	==References==		==References==

Shawndouglas: Saving and adding more.

2017-12-20T23:52:02Z

Saving and adding more.

← Older revision		Revision as of 23:52, 20 December 2017
Line 72:		Line 72:
	\|}		\|}
	\|}		\|}

			To give a more concrete impression of the painstaking detail that needs to be considered while planning and organizing a comprehensive metadata collection, we provide an exhaustive description of the experiment in the supplementary material. To illustrate the level of complexity of metadata management in this example, Figure 1 outlines the different components of the experimental setup, the signal flow, the task, and the trial scheme. In addition, the heterogeneous pieces of metadata and the corresponding files that contain them in the absence of a comprehensive metadata collection are listed and described in Table 2. Here, all metadata source files are labeled by numbers and appear in Figure 1 wherever they were generated. If labels appear multiple times, they were iteratively enriched with information obtained from the corresponding components of the setup.

			In such a complex experiment, the relevance of some metadata does not become immediately apparent and is sometimes underestimated. For example, the immediate relevance of each minute detail of the experimental setup are of little interest for the interpretation of a single recording, and only when one attempts to rebuild the experiment do these metadata become valuable. Apart from that, any piece of metadata can become highly relevant for screening and selecting datasets according to specific criteria at a later stage. Different experiments produce different sets of metadata, and it is therefore not possible to provide a detailed list of metadata that needs to be collected from any given type of experiment. Nevertheless, the general principle should be to keep as much information about an experiment as possible from the beginning on, even if information seems to be trivial or irrelevant at the time. Based on our experience gathered from various collaborations, we compiled a list of guiding questions to help scientists define their optimal collection of metadata for their use scenario, e.g., reproducing the experiment and/or the analysis of the data, or sharing the data:

			* What metadata would be required to replicate the experiment elsewhere?
			* Is the experiment sufficiently explained, such that someone else could continue the work?
			* If the recordings exhibit spurious data, is the signal flow completely reconstructable to find the cause?
			* Are metadata provided which may explain variability (e.g., between subjects or recordings) in the recorded data?
			* Are metadata provided which enable access to subsets of data to address specific scientific questions?
			* Is the recorded data described in sufficient detail, such that an external collaborator could understand them?

			For the example experiment presented in Figure 1, we used these guiding questions to control if the content of the various metadata sources was sufficient, and enriched these where necessary. The process of planning the relevant metadata content and organizing the metadata into a comprehensive collection can be time-consuming, especially if the process happened after the experiment was performed.

			In the following, we will outline use cases that illustrate why the time for creating a comprehensive metadata collection is well spent, nevertheless.

			==Advantage of a comprehensive metadata collection: Use cases==
			Based on the example experiment (Figure 1), we present five use cases that demonstrate the common scenarios in which access to metadata becomes important. For the implementation of each use case, we contrast the scenarios before and after having organized metadata in a comprehensive collection:

			* ''Scenario 1'': Metadata are organized in different files and formats which are stored in a metadata source directory.
			* ''Scenario 2'': Metadata sources of Scenario 1 are compiled into a comprehensive collection, stored in one file per recording using a standard file format.

			We introduce the protagonists acting in the use cases and characterize their relationship:

			* ''Alice'': She is the experimenter who built up the setup, trained the monkeys, and carried out the experimental study. She has programming experience in MATLAB and performs the preprocessing step of spike sorting.
			* ''Bob'': He is a data analyst and a new member of Alice's laboratory. He has programming experience in MATLAB and Python. His task is to support Alice in implementing the preprocessing and first analysis of the data.
			* ''Carol'': She is a theoretical neuroscientist and an analyst of an external group who collaborates with Alice's laboratory. She is an experienced Python programmer. Information exchange with Alice's group is limited to phone, email, and video calls, and few in-person meetings. She is not an experimentalist and therefore not used to work with real neuronal data. She never participated in an experiment and thus never experienced the workflow of a typical recording session.

			Each of the use cases below illustrates a different aspect of the analysis workflow: enrichment of metadata information, metadata accessibility, selection and screening of datasets, and formal queries to reference metadata.

			===Use Case 1: Enrichment of the metadata collection===
			<blockquote>A difficult aspect of metadata management is that one cannot know all metadata in advance that are necessary to ensure the reproducibility of the experiment and the data analysis. Thus, the metadata collection needs to be enriched when new and updated information becomes available (see Figure 2). An example for such an enrichment of a metadata collection is the integration of results and parameters of preprocessing steps applied to the recorded data, since these are usually performed long after the primary data acquisition (cf. Table 2). Use Case 1 demonstrates the advantage of using a standard file format when a metadata collection is enriched with information from successive preprocessing steps.</blockquote>


			[[File:Fig2 Zehl FrontInNeuro2016 10.jpg\|966px]]
			{{clear}}
			{\|
			\| STYLE="vertical-align:top;"\|
			{\| border="0" cellpadding="5" cellspacing="0" width="966px"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\| <blockquote>'''Figure 2.''' Enrichment of the reach-to-grasp metadata collection over time. Small gray and labeled boxes represent files which contain metadata (cf. Table 2). The metadata collection is first generated at the beginning of the experiment and contains only information constant for the complete experiment. With each monkey, the metadata collection is enriched by monkey-specific information that remains constant for the time of the experiment. With each array implantation, information about that array is included in the metadata collection. During or directly after the recording of each session the metadata collection is enriched by information specific to the recording. Several preprocessing steps are then performed on the recorded data (here spike sorting, event detection, and quality check). The newly generated metadata are integrated step-by-step into the metadata collection and enrich the information available for subsequent data analysis.</blockquote>
			\|-
			\|}
			\|}

			In one of the first meetings with Alice, Bob learned that a number of preprocessing steps are performed by Alice. One of these is to identify and extract time points of the behavioral events describing the object displacement in each trial (OT, HS, OR, see Table 1) from the continuous force and displacement signals which were saved in the ns2-datafile (label 6b in Figure 1 and Table 2). For this, the signals were loaded by a program Alice developed in MATLAB to perform this preprocessing step. The extracted behavioral events are saved for each recording session along with the processing parameters in a respective .mat file (labels 11 in Figure 1 and Table 2). It is important that these preprocessing results are readily accessible to all collaboration partners, including Bob, because they are needed to correctly interpret the timeline and behavior in each trial.

			In Scenario 1, Alice does not create a comprehensive metadata collection, and Bob has to deal with the fact that the behavioral events are not only not stored along with the digitally available trial events (e.g., TS, see Table 1) in the recorded nev-file (label 5 in Figure 1 and Table 2) but also saved in the custom-made mat-file, which requires him to write a corresponding loading routine. In Scenario 2, Alice saved the digital, directly available trial events into one comprehensive metadata collection per recording. With the preprocessing of each recording, she enriched then the corresponding primary collection with all results and parameters of the behavioral trial event extraction. As a result, the access to all trial events becomes easier for Bob, so that he is able to quickly scan the behavior in each trial.

			In summary, a comprehensive metadata collection has the advantage that it bundles metadata that conceptually belongs together, and it simplifies access by combining metadata into one single, standard file format. Moreover, the flexible enrichment of a comprehensive metadata collection simplifies the organization of metadata that originate from multiple versions of performing a preprocessing step, e.g., as in the case of offline spike sorting (see Supplementary Material). With this, a comprehensive metadata collection not only simplifies the reproducibility of Alice's work (e.g., repetition of preprocessing steps), but also guarantees better reproducibility for the collaboration with Bob (e.g., standardized access to parameters and results of preprocessing steps).

			===Use Case 2: Metadata accessibility===
			<blockquote>Complex experimental studies usually include organizational structures on the level of the file system (e.g., directories) and file format to store all data and metadata. Even in cases where all data are very well organized, metadata are usually distributed over several files and formats (see Table 2). Use Case 2 demonstrates how a standard file format of a comprehensive metadata collection improves the accessibility and readability of metadata.</blockquote>

			When Bob first arrived in the lab, Alice explained to him the structure of the data and metadata of the experimental study. Unfortunately, Bob could not start working on the data directly after the meeting with Alice. Of course he made several notes during this meeting, but due to the complexity of the information, he forgot where to find the individual pieces of metadata in sufficient detail.

			In Scenario 1, Bob starts to scan all different metadata sources to regain an overview of what can be found where. This means that he has not only to go through several files, but also that he needs to deal with several formats, and that he needs to understand the different design of each file. In Scenario 2, Bob knows that all metadata are collected and stored in one file per recording, which is readable using standard software, e.g., a text editor. To access again an overview of the available metadata, he can open a file of any recording, screen its content or search for specific metadata.

			Thus, the metadata organization in Scenario 1 has the following disadvantages: (i) it is difficult to keep track of which metadata source contains which piece of information; (ii) not every metadata source is readable with a simple, easily accessible software tool (e.g., the binary data files, .nev and .nsX; labels 5, 6a, and 6b in Figure 1 and Table 2, respectively); (iii) not every software tool offers the option to search for specific metadata content. In contrast, in Scenario 2 a comprehensive, standardized, and searchable organization of the metadata guarantees complete and easy access to all metadata related to an experiment, even over a long period of time. This simplifies the long-term comprehensibility of Alice's experiment not only for herself, but also for all further group members and collaboration partners.

			===Use Case 3: Selection of datasets===
			<blockquote>Data of an experiment can usually be used to address multiple scientific questions. In this context, it is necessary to select datasets according to certain criteria, which are defined by the requirements and constraints of the scientific question. These criteria are usually represented by metadata related to one recording session and generated during or after the recording period (see Table 2). Use Case 3 demonstrates how a comprehensive metadata collection with a standard file format supports the automatic selection of datasets according to defined selection criteria.</blockquote>

			Alice reported that she had the feeling that the weekend break influences the monkey's performance on Mondays and asked Bob to investigate her suspicion. To solve this new task, Bob first wants to identify the recording sessions for each weekday (Criterion 1) which were recorded with standard task settings (Criterion 2) and under the standard experimental paradigm (instructed delayed reach-to-grasp task with two cues; Criterion 3). To automatize the data selection, Bob writes a Python program where he loops through all available recording sessions, uses Criterion 1 to identify the weekday, and adds each session name to a corresponding list if in the session Criteria 2 and 3 were fulfilled.

			To write this program, Bob uses in Scenario 1 the knowledge he gained from the manual inspection in Use Case 2 to identify for the chosen selection criteria the different metadata source files. Criterion 1 was stored in the .nev data file (Label 5 in Figure 1 and Table 2) which Bob can access via the data loading routine. Criteria 2 and 3 are instead stored in the recording-specific spreadsheet (Label 7 in Figure 1 and Table 2). To automatize their extraction, Bob additionally needs to write a spreadsheet loading routine, because there is no standard loading routine available for the homemade structure of the spreadsheet. In contrast, in Scenario 2, Bob extracts all three criteria for each recording from one comprehensive metadata file using an available loading routine of the chosen standard file format.

			In summary, compared to Scenario 1, Scenario 2 improved the workflow of selecting datasets according to certain criteria in two aspects: (i) to check for the selection criteria, only one metadata file per recording needs to be loaded, and (ii) to extract metadata stored in a standardized format, a loading routine is already available. Thus, in Scenario 2 the scripts for automatized data selection are less complicated, which improved the reproducibility of the operation.

	==References==		==References==

Shawndouglas: Saving and adding more.

2017-12-20T23:24:05Z

Saving and adding more.

← Older revision		Revision as of 23:24, 20 December 2017
Line 6:		Line 6:
	\|title_full = Handling metadata in a neurophysiology laboratory		\|title_full = Handling metadata in a neurophysiology laboratory
	\|journal = ''Frontiers in Neuroinformatics''		\|journal = ''Frontiers in Neuroinformatics''
	\|authors = Zehl, Lyuba; Jaillet, Florent; Stoewer, Adrian; Grewe, Jan; Sobolev, Andrey; Wachtler, Thomas; Brochier, Thomas G.; Riehle, Alexa; Denker, Michael; Grün, Sonja		\|authors = Zehl, Lyuba; Jaillet, Florent; Stoewer, Adrian; Grewe, Jan; Sobolev, Andrey; Wachtler, Thomas;<br />Brochier, Thomas G.; Riehle, Alexa; Denker, Michael; Grün, Sonja
	\|affiliations = Jülich Research Centre, Aix-Marseille Université, Ludwig-Maximilians-Universität München, Eberhard-Karls-Universität Tübingen, Aachen University		\|affiliations = Jülich Research Centre, Aix-Marseille Université, Ludwig-Maximilians-Universität München,<br />Eberhard-Karls-Universität Tübingen, Aachen University
	\|contact = Email: l dot zehl at fz-juelich dot de		\|contact = Email: l dot zehl at fz-juelich dot de
	\|editors = Luo, Qingming		\|editors = Luo, Qingming
Line 38:		Line 38:

	To present the concept of metadata management in a practical context, we introduce in the following a selected electrophysiological experiment. For clarity, a graphical summary of the behavioral task and the recording setup is provided in Figure 1, and a list of task-related abbreviations (e.g., behavioral conditions and events) is listed in Table 1. First described in Riehle ''et al.''<ref name="RiehleMapping13" /> and Milekovic ''et al.''<ref name="MilekovicLocal15">{{cite journal \|title=Local field potentials in primate motor cortex encode grasp kinetic parameters \|journal=Neuroimage \|author=Milekovic, T.; Truccolo, W.; Grün, S. et al. \|volume=114 \|pages=338–55 \|year=2015 \|doi=10.1016/j.neuroimage.2015.04.008 \|pmid=25869861 \|pmc=PMC4562281}}</ref>, the experiment employs high density multi-electrode recordings to study the modulation of spiking and LFP activities in the motor/pre-motor cortex of monkeys performing an instructed delayed reach-to-grasp task. In total, three monkeys (Macaca mulatta; two females, L, T; one male, N) were trained to grasp an object using one of two different grip types (side grip, SG, or precision grip, PG) and to pull it against one of two possible loads requiring either a high (HF) or low (LF) pulling force (cf. Figure 1). In each trial, instructions for the requested behavior were provided to the monkeys through two consecutive visual cues (C and GO) which were separated by a one-second delay and generated by the illumination of specific combinations of five LEDs positioned above the object (cf. Figure 1). The complexity of this study makes it particularly suitable to expose the difficulty of collecting, organizing, and storing metadata. Moreover, metadata were first organized according to the laboratory internal procedures and practices for documentation, while the organization of the metadata into a comprehensive machine-readable format was done a-posteriori, after the experiment was completed. To work with the data and metadata of this experiment, one has to handle on average 300 recording sessions per monkey, with data distributed over three files per session, metadata distributed over five files per implant, at least 10 files per recording, and one file for general information of the experiment (cf. Table 2). This situation imposed an additional complexity to reorganize the various metadata sources.		To present the concept of metadata management in a practical context, we introduce in the following a selected electrophysiological experiment. For clarity, a graphical summary of the behavioral task and the recording setup is provided in Figure 1, and a list of task-related abbreviations (e.g., behavioral conditions and events) is listed in Table 1. First described in Riehle ''et al.''<ref name="RiehleMapping13" /> and Milekovic ''et al.''<ref name="MilekovicLocal15">{{cite journal \|title=Local field potentials in primate motor cortex encode grasp kinetic parameters \|journal=Neuroimage \|author=Milekovic, T.; Truccolo, W.; Grün, S. et al. \|volume=114 \|pages=338–55 \|year=2015 \|doi=10.1016/j.neuroimage.2015.04.008 \|pmid=25869861 \|pmc=PMC4562281}}</ref>, the experiment employs high density multi-electrode recordings to study the modulation of spiking and LFP activities in the motor/pre-motor cortex of monkeys performing an instructed delayed reach-to-grasp task. In total, three monkeys (Macaca mulatta; two females, L, T; one male, N) were trained to grasp an object using one of two different grip types (side grip, SG, or precision grip, PG) and to pull it against one of two possible loads requiring either a high (HF) or low (LF) pulling force (cf. Figure 1). In each trial, instructions for the requested behavior were provided to the monkeys through two consecutive visual cues (C and GO) which were separated by a one-second delay and generated by the illumination of specific combinations of five LEDs positioned above the object (cf. Figure 1). The complexity of this study makes it particularly suitable to expose the difficulty of collecting, organizing, and storing metadata. Moreover, metadata were first organized according to the laboratory internal procedures and practices for documentation, while the organization of the metadata into a comprehensive machine-readable format was done a-posteriori, after the experiment was completed. To work with the data and metadata of this experiment, one has to handle on average 300 recording sessions per monkey, with data distributed over three files per session, metadata distributed over five files per implant, at least 10 files per recording, and one file for general information of the experiment (cf. Table 2). This situation imposed an additional complexity to reorganize the various metadata sources.


			[[File:Fig1 Zehl FrontInNeuro2016 10.jpg\|1000px]]
			{{clear}}
			{\|
			\| STYLE="vertical-align:top;"\|
			{\| border="0" cellpadding="5" cellspacing="0" width="1000px"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\| <blockquote>'''Figure 1.''' Overview of the reach-to-grasp setup. Bottom: Time line of the experiment indicating the sequence of presented stimuli (WS, C-ON, C-OFF, GO, RW), the behavioral epochs (SP, WP, C, DP, RT, MT, PT, MP) and registered behavioral events (TS, SR, OT, HS). Fixed durations of behavioral epochs are specified in the time line. The abbreviations used in the trial scheme are explained in Table 1. The sketch of the experimental setup above illustrates the involved hard and software components (boxes) as well as the signal flow. The signal flow is composed of two streams of signals: Recording and processing of the neuronal signals (yellow arrows) and the signals related behavior (green and blue arrows). The controller of each component is indicated by black arrows. The gray boxes indicate files that contain metadata generated or updated at this stage (cf. numbers in boxes to labels in Table 2 for details). The signal flow of the neuronal data starts at the level of the Utah array, continues with the headstage, the Front-End Amplifier, the Neural Signal Processor (NSP), and ends with saving the data as three Blackrock specific data file formats (.nev, .ns2 and .ns5/.ns6) using Central Suite running on the data acquisition PC. The stream of behavioral signals is split into two parallel pathways. One of them contains the analog signals (force and displacement sensors), colored in blue, and the other one the digital signals (LEDs, table switch, and reward control), colored in green. The correct sequence of trial events (presentation of stimuli), the setting of the load force of the object and the performed behavior (timing, movement) are monitored (blue and green arrows) and controlled (black arrows) online via LabVIEW on a second setup control PC. The digitization of the signals is performed via an Analog-Digital converter (ADC) of National Instruments (NI) which is upstream to the setup control PC. The ADC is also upstream to NSP which is used to save all digital or digitized signals into the nev data file via Central Suite. In parallel the analog signals of the load and displacement sensors of the object are fed directly into the NSP and saved into the .ns2 file. Consecutive preprocessing of the neuronal as well as the behavioral signals (flow of preprocessed signals marked by dashed lines) is performed offline on separate PCs, such as the spike sorting with the Plexon offline Spike Sorter or other preprocessing steps with custom programs (e.g., “Quality Check” or “Event Detection”). Image of Utah array courtesy of Blackrock Microsystems.</blockquote>
			\|-
			\|}
			\|}

			[[File:Tab1 Zehl FrontInNeuro2016 10.jpg\|1000px]]
			{{clear}}
			{\|
			\| STYLE="vertical-align:top;"\|
			{\| border="0" cellpadding="5" cellspacing="0" width="1000px"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\| <blockquote>'''Table 1.''' Trial scheme parameters of the reach-to-grasp experiment</blockquote>
			\|-
			\|}
			\|}

			[[File:Tab2 Zehl FrontInNeuro2016 10.jpg\|1000px]]
			{{clear}}
			{\|
			\| STYLE="vertical-align:top;"\|
			{\| border="0" cellpadding="5" cellspacing="0" width="1000px"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\| <blockquote>'''Table 2.''' Metadata sources of the reach-to-gasp experiment</blockquote>
			\|-
			\|}
			\|}

	==References==		==References==

Shawndouglas: Saving and adding more.

2017-12-20T22:47:52Z

Saving and adding more.

← Older revision		Revision as of 22:47, 20 December 2017
Line 33:		Line 33:

	In order to provide metadata in an organized, easily accessible, but also machine-readable way, Grewe ''et al.''<ref name="GreweABottom11">{{cite journal \|title=A Bottom-up Approach to Data Annotation in Neurophysiology \|journal=Frontiers in Neuroinformatics \|author=Grewe, J.; Wachtler, T.; Benda, J. \|volume=5 \|pages=16 \|year=2011 \|doi=10.3389/fninf.2011.00016 \|pmid=21941477 \|pmc=PMC3171061}}</ref> introduced odML (open metadata Markup Language) as a simple file format in analogy to SBML in systems biology<ref name="HuckaTheSys03">{{cite journal \|title=The systems biology markup language (SBML): A medium for representation and exchange of biochemical network models \|journal=Bioinformatics \|author=Hucka, M.; Finney, A.; Sauro, H.M. et al. \|volume=19 \|issue=4 \|pages=524–31 \|year=2003 \|doi=10.1093/bioinformatics/btg015 \|pmid=12611808}}</ref>, or NeuroML in neuroscientific simulation studies.<ref name="GleesonNeuroML10">{{cite journal \|title=NeuroML: a language for describing data driven models of neurons and networks with a high degree of biological detail \|journal=PLoS Computational Biology \|author=Gleeson, P.; Crook, S.; Cannon, R.C. et al. \|volume=6 \|issue=6 \|pages=e1000815 \|year=2010 \|doi=10.1371/journal.pcbi.1000815 \|pmid=20585541 \|pmc=PMC2887454}}</ref><ref name="CrookCreating12">{{cite journal \|title=Creating, documenting and sharing network models \|journal=Network \|author=Crook, S.M.; Bednar, J.A.; Berger, S. et al. \|volume=23 \|issue=4 \|pages=131-49 \|year=2012 \|doi=10.3109/0954898X.2012.722743 \|pmid=22994683}}</ref> However, lacking to date is a detailed investigation on how to incorporate metadata management in the daily lab routine in terms of (i) organizing the metadata in a comprehensive collection, (ii) practically gathering and entering the metadata, and (iii) profiting from the resulting comprehensive metadata collection in the process of analyzing the data. Here we address these points, both conceptually and practically, in the context of a complex behavioral experiment that involves neuronal recordings from a large number of electrodes that yield massively parallel spike and local field potential (LFP) data.<ref name="RiehleMapping13">{{cite journal \|title=Mapping the spatio-temporal structure of motor cortical LFP and spiking activities during reach-to-grasp movements \|journal=Frontiers in Neural Circuits \|author=Riehle, A.; Wirtssohn, S.; Grün, S.; Brochier, T. \|volume=7 \|pages=48 \|year=2013 \|doi=10.3389/fncir.2013.00048 \|pmid=23543888 \|pmc=PMC3608913}}</ref> To illustrate how to organize a comprehensive metadata collection (i), we introduce in the next section the concept of metadata and demonstrate the rich diversity of metadata that arise in the context of the example experiment. To demonstrate why the effort of creating a comprehensive metadata collection is time well spent, we next describe five use cases that summarize where the access to metadata becomes relevant when working with the data (iii). Afterwards we provide detailed guidelines and assistance on how to create, structure, and hierarchically organize comprehensive metadata collections (i and ii). Complementing these guidelines, we provide in the supplementary material section a thorough practical introduction on how to embed a metadata management tool, such as the odML library, into the experimental and analysis workflow. Finally, we close by critically contrasting the importance of proper metadata handling against its difficulties and deriving future challenges.		In order to provide metadata in an organized, easily accessible, but also machine-readable way, Grewe ''et al.''<ref name="GreweABottom11">{{cite journal \|title=A Bottom-up Approach to Data Annotation in Neurophysiology \|journal=Frontiers in Neuroinformatics \|author=Grewe, J.; Wachtler, T.; Benda, J. \|volume=5 \|pages=16 \|year=2011 \|doi=10.3389/fninf.2011.00016 \|pmid=21941477 \|pmc=PMC3171061}}</ref> introduced odML (open metadata Markup Language) as a simple file format in analogy to SBML in systems biology<ref name="HuckaTheSys03">{{cite journal \|title=The systems biology markup language (SBML): A medium for representation and exchange of biochemical network models \|journal=Bioinformatics \|author=Hucka, M.; Finney, A.; Sauro, H.M. et al. \|volume=19 \|issue=4 \|pages=524–31 \|year=2003 \|doi=10.1093/bioinformatics/btg015 \|pmid=12611808}}</ref>, or NeuroML in neuroscientific simulation studies.<ref name="GleesonNeuroML10">{{cite journal \|title=NeuroML: a language for describing data driven models of neurons and networks with a high degree of biological detail \|journal=PLoS Computational Biology \|author=Gleeson, P.; Crook, S.; Cannon, R.C. et al. \|volume=6 \|issue=6 \|pages=e1000815 \|year=2010 \|doi=10.1371/journal.pcbi.1000815 \|pmid=20585541 \|pmc=PMC2887454}}</ref><ref name="CrookCreating12">{{cite journal \|title=Creating, documenting and sharing network models \|journal=Network \|author=Crook, S.M.; Bednar, J.A.; Berger, S. et al. \|volume=23 \|issue=4 \|pages=131-49 \|year=2012 \|doi=10.3109/0954898X.2012.722743 \|pmid=22994683}}</ref> However, lacking to date is a detailed investigation on how to incorporate metadata management in the daily lab routine in terms of (i) organizing the metadata in a comprehensive collection, (ii) practically gathering and entering the metadata, and (iii) profiting from the resulting comprehensive metadata collection in the process of analyzing the data. Here we address these points, both conceptually and practically, in the context of a complex behavioral experiment that involves neuronal recordings from a large number of electrodes that yield massively parallel spike and local field potential (LFP) data.<ref name="RiehleMapping13">{{cite journal \|title=Mapping the spatio-temporal structure of motor cortical LFP and spiking activities during reach-to-grasp movements \|journal=Frontiers in Neural Circuits \|author=Riehle, A.; Wirtssohn, S.; Grün, S.; Brochier, T. \|volume=7 \|pages=48 \|year=2013 \|doi=10.3389/fncir.2013.00048 \|pmid=23543888 \|pmc=PMC3608913}}</ref> To illustrate how to organize a comprehensive metadata collection (i), we introduce in the next section the concept of metadata and demonstrate the rich diversity of metadata that arise in the context of the example experiment. To demonstrate why the effort of creating a comprehensive metadata collection is time well spent, we next describe five use cases that summarize where the access to metadata becomes relevant when working with the data (iii). Afterwards we provide detailed guidelines and assistance on how to create, structure, and hierarchically organize comprehensive metadata collections (i and ii). Complementing these guidelines, we provide in the supplementary material section a thorough practical introduction on how to embed a metadata management tool, such as the odML library, into the experimental and analysis workflow. Finally, we close by critically contrasting the importance of proper metadata handling against its difficulties and deriving future challenges.

			==Organizing metadata in neurophysiology==
			Metadata are generally defined as data describing data.<ref name="BacaIntro08">{{cite book \|url=http://www.getty.edu/publications/intrometadata/ \|title=Introduction to Metadata \|editor=Baca, M. publisher=Getty Publications \|edition=3rd \|year=2016}}</ref><ref name="MWMetadata">{{cite web \|url=https://www.merriam-webster.com/dictionary/metadata \|title=Metadata \|work=Merriam-Webster.com \|publisher=Merriam-Webster \|accessdate=06 July 2016}</ref> More specifically, metadata are information that describe the conditions under which a certain dataset has been recorded.<ref name="GreweABottom11" /> Ideally all metadata would be machine-readable and available at a single location that is linked to the corresponding recorded dataset. The fact that such central, comprehensive metadata collections are not common practice already today is by no means a sign of negligence on the part of the scientists, but is explained by the fact that in the absence of conceptual guidelines and software support, such a situation is extremely difficult to achieve given the high complexity of the task. Already, the fact that an electrophysiological setup is composed of several hardware and software components, often from different vendors, imposes the need to handle multiple files of different formats. Some files may even contain metadata that are not machine-readable and -interpretable. Furthermore, performing an experiment requires the full attention of the experimenters, which limits the amount of metadata that can be manually captured online. Metadata that arise unexpectedly during an experiment, e.g., the cause of a sudden noise artifact, are commonly documented as handwritten notes in the laboratory notebook. In fact, handwritten notes are often unavoidable, because the legal regulations of some countries, e.g., [http://www.cnrs.fr/infoslabos/cahier-laboratoire/docs/cahierlabo.pdf France], require the documentation of experiments in the form of a handwritten laboratory notebook.

			To present the concept of metadata management in a practical context, we introduce in the following a selected electrophysiological experiment. For clarity, a graphical summary of the behavioral task and the recording setup is provided in Figure 1, and a list of task-related abbreviations (e.g., behavioral conditions and events) is listed in Table 1. First described in Riehle ''et al.''<ref name="RiehleMapping13" /> and Milekovic ''et al.''<ref name="MilekovicLocal15">{{cite journal \|title=Local field potentials in primate motor cortex encode grasp kinetic parameters \|journal=Neuroimage \|author=Milekovic, T.; Truccolo, W.; Grün, S. et al. \|volume=114 \|pages=338–55 \|year=2015 \|doi=10.1016/j.neuroimage.2015.04.008 \|pmid=25869861 \|pmc=PMC4562281}}</ref>, the experiment employs high density multi-electrode recordings to study the modulation of spiking and LFP activities in the motor/pre-motor cortex of monkeys performing an instructed delayed reach-to-grasp task. In total, three monkeys (Macaca mulatta; two females, L, T; one male, N) were trained to grasp an object using one of two different grip types (side grip, SG, or precision grip, PG) and to pull it against one of two possible loads requiring either a high (HF) or low (LF) pulling force (cf. Figure 1). In each trial, instructions for the requested behavior were provided to the monkeys through two consecutive visual cues (C and GO) which were separated by a one-second delay and generated by the illumination of specific combinations of five LEDs positioned above the object (cf. Figure 1). The complexity of this study makes it particularly suitable to expose the difficulty of collecting, organizing, and storing metadata. Moreover, metadata were first organized according to the laboratory internal procedures and practices for documentation, while the organization of the metadata into a comprehensive machine-readable format was done a-posteriori, after the experiment was completed. To work with the data and metadata of this experiment, one has to handle on average 300 recording sessions per monkey, with data distributed over three files per session, metadata distributed over five files per implant, at least 10 files per recording, and one file for general information of the experiment (cf. Table 2). This situation imposed an additional complexity to reorganize the various metadata sources.

	==References==		==References==
Line 38:		Line 43:

	==Notes==		==Notes==
	This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version — by design — lists them in order of appearance.		This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version — by design — lists them in order of appearance. What were originally footnotes have been turned into inline external links.

	<!--Place all category tags here-->		<!--Place all category tags here-->

Shawndouglas: Saving and editing more.

2017-12-20T22:25:50Z

Saving and editing more.

← Older revision		Revision as of 22:25, 20 December 2017
Line 29:		Line 29:
	==Introduction==		==Introduction==
	Technological advances in neuroscience during the last decades have led to methods that nowadays enable researchers to simultaneously record the activity from tens to hundreds of neurons simultaneously, in vitro or in vivo, using a variety of techniques<ref name="NicolelisMulti02">{{cite journal \|title=Multielectrode recordings: The next steps \|journal=Current Opinion in Neurobiology \|author=Nicolelis, M.A.; Ribeiro, S. \|volume=12 \|issue=5 \|pages=602–6 \|year=2002 \|doi=10.1016/S0959-4388(02)00374-4 \|pmid=12367642}}</ref><ref name="VerkhratskyFrom06">{{cite journal \|title=From Galvani to patch clamp: The development of electrophysiology \|journal=Pflügers Archiv \|author=Verkhratsky, A.; Krishtal, O.A.; Petersen, O.H. \|volume=453 \|issue=3 \|pages=233–47 \|year=2006 \|doi=10.1007/s00424-006-0169-z \|pmid=17072639}}</ref><ref name="ObienRevealing15">{{cite journal \|title=Revealing neuronal function through microelectrode array recordings \|journal=Frontiers in Neuroscience \|author=Obien, M.E.; Deligkaris, K.; Bullmann, T. et al. \|volume=8 \|pages=423 \|year=2015 \|doi=10.3389/fnins.2014.00423 \|pmid=25610364 \|pmc=PMC4285113}}</ref> in combination with sophisticated stimulation methods, such as optogenetics.<ref name="DeisserothEngineer13">{{cite journal \|title=Engineering approaches to illuminating brain structure and dynamics \|journal=Neuron \|author=Deisseroth, K.; Schnitzer, M.J. \|volume=80 \|issue=3 \|pages=568–77 \|year=2013 \|doi=10.1016/j.neuron.2013.10.032 \|pmid=24183010 \|pmc=PMC5731466}}</ref><ref name="MiyamotoTheFiber16">{{cite journal \|title=The fiber-optic imaging and manipulation of neural activity during animal behavior \|journal=Neuroscience Research \|author=Miyamoto, D.; Murayama, M. \|volume=103 \|pages=1–9 \|year=2016 \|doi=10.1016/j.neures.2015.09.004 \|pmid=26427958}}</ref> In addition, recordings can be performed in parallel from multiple brain areas, together with behavioral measures such as eye or limb movements.<ref name="MaldonadoSynchro08">{{cite journal \|title=Synchronization of neuronal responses in primary visual cortex of monkeys viewing natural images \|journal=Journal of Neurophysiology \|author=Maldonado, P.; Babul, C.; Singer, W. et al. \|volume=100 \|issue=3 \|pages=1523-32 \|year=2008 \|doi=10.1152/jn.00076.2008 \|pmid=18562559}}</ref><ref name="VargasIrwinDecoding10">{{cite journal \|title=Decoding complete reach and grasp actions from local primary motor cortex populations \|journal=Journal of Neuroscience \|author=Vargas-Irwin, C.E.; Shakhnarovich, G.; Yadollahpour, P. et al. \|volume=30 \|issue=29 \|pages=9659-69 \|year=2010 \|doi=10.1523/JNEUROSCI.5443-09.2010 \|pmid=20660249 \|pmc=PMC2921895}}</ref> Such recordings enable researchers to study network interactions and cross-area coupling, and to relate neuronal processing to the behavioral performance of the subject.<ref name="BerenyiLarge14">{{cite journal \|title=Large-scale, high-density (up to 512 channels) recording of local circuits in behaving animals \|journal=Journal of Neurophysiology \|author=Berényi, A.; Somogyvári, Z.; Nagy, A.J. et al. \|volume=111 \|issue=5 \|pages=1132-49 \|year=2014 \|doi=10.1152/jn.00785.2013 \|pmid=24353300 \|pmc=PMC3949233}}</ref><ref name="LewisRecording15">{{cite journal \|title=Recording of brain activity across spatial scales \|journal=Current Opinion in Neurobiology \|author=Lewis, C.M.; Bosman, C.A.; Fries, P. et al. \|volume=32 \|pages=68–77 \|year=2015 \|doi=10.1016/j.conb.2014.12.007 \|pmid=25544724}}</ref><ref name="LismanTheChall15">{{cite journal \|title=The challenge of understanding the brain: Where we stand in 2015 \|journal=Neuron \|author=Lisman, J. \|volume=86 \|issue=4 \|pages=864-882 \|year=2015 \|doi=10.1016/j.neuron.2015.03.032 \|pmid=25996132 \|pmc=PMC4441769}}</ref> These approaches lead to increasingly complex experimental designs that are difficult to parameterize, e.g., due to multidimensional characterization of natural stimuli<ref name="GeislerVisual08">{{cite journal \|title=Visual perception and the statistical properties of natural scenes \|journal=Annual Review of Psychology \|author=Geisler, W.S. \|volume=59 \|pages=167–92 \|year=2008 \|doi=10.1146/annurev.psych.58.110405.085632 \|pmid=17705683}}</ref> or high-dimensional movement parameters for almost freely behaving subjects.<ref name="SchwarzChronic14">{{cite journal \|title=Chronic, wireless recordings of large-scale brain activity in freely moving rhesus monkeys \|journal=Nature Methods \|author=Schwarz, D.A.; Lebedev, M.A.; Hanson, T.L. et al. \|volume=11 \|issue=6 \|pages=670-6 \|year=2014 \|doi=10.1038/nmeth.2936 \|pmid=24776634 \|pmc=PMC4161037}}</ref> It is a serious challenge for researchers to keep track of the overwhelming amount of metadata generated at each experimental step and to precisely extract all the information relevant for data analysis and interpretation of results. Various aspects such as the parametrization of the experimental task, filter settings and sampling rates of the setup, the quality of the recorded data, broken electrodes, preprocessing steps (e.g., spike sorting), or the condition of the subject need to be considered. Nevertheless, the organization of these metadata is of utmost importance for conducting research in a reproducible manner, i.e., the ability to faithfully reproduce the experimental procedures and subsequent analysis steps.<ref name="LaineReproduc07">{{cite journal \|title=Reproducible research: Moving toward research the public can really trust \|journal=Annals of Internal Medicine \|author=Laine, C.; Goodman, S.N.; Griswold, M.E.; Sox, H.C. \|volume=146 \|issue=6 \|pages=450–3 \|year=2007 \|doi=10.7326/0003-4819-146-6-200703200-00154 \|pmid=17339612}}</ref><ref name="PengReproduc11">{{cite journal \|title=Reproducible research in computational science \|journal=Science \|author=Peng, R.D. \|volume=334 \|issue=6060 \|pages=1226-7 \|year=2011 \|doi=10.1126/science.1213847 \|pmid=22144613 \|pmc=PMC3383002}}</ref><ref name="TomaselloMethod11">{{cite journal \|title=Methodological challenges in the study of primate cognition \|journal=Science \|author=Tomasello, M.; Call, J. \|volume=334 \|issue=6060 \|pages=1227-8 \|year=2011 \|doi=10.1126/science.1213443 \|pmid=22144614}}</ref> Moreover, detailed knowledge of the complete recording and analysis processes is crucial for the correct interpretation of results, and is a minimal requirement to enable researchers to verify published results and build their own research on the previous findings.		Technological advances in neuroscience during the last decades have led to methods that nowadays enable researchers to simultaneously record the activity from tens to hundreds of neurons simultaneously, in vitro or in vivo, using a variety of techniques<ref name="NicolelisMulti02">{{cite journal \|title=Multielectrode recordings: The next steps \|journal=Current Opinion in Neurobiology \|author=Nicolelis, M.A.; Ribeiro, S. \|volume=12 \|issue=5 \|pages=602–6 \|year=2002 \|doi=10.1016/S0959-4388(02)00374-4 \|pmid=12367642}}</ref><ref name="VerkhratskyFrom06">{{cite journal \|title=From Galvani to patch clamp: The development of electrophysiology \|journal=Pflügers Archiv \|author=Verkhratsky, A.; Krishtal, O.A.; Petersen, O.H. \|volume=453 \|issue=3 \|pages=233–47 \|year=2006 \|doi=10.1007/s00424-006-0169-z \|pmid=17072639}}</ref><ref name="ObienRevealing15">{{cite journal \|title=Revealing neuronal function through microelectrode array recordings \|journal=Frontiers in Neuroscience \|author=Obien, M.E.; Deligkaris, K.; Bullmann, T. et al. \|volume=8 \|pages=423 \|year=2015 \|doi=10.3389/fnins.2014.00423 \|pmid=25610364 \|pmc=PMC4285113}}</ref> in combination with sophisticated stimulation methods, such as optogenetics.<ref name="DeisserothEngineer13">{{cite journal \|title=Engineering approaches to illuminating brain structure and dynamics \|journal=Neuron \|author=Deisseroth, K.; Schnitzer, M.J. \|volume=80 \|issue=3 \|pages=568–77 \|year=2013 \|doi=10.1016/j.neuron.2013.10.032 \|pmid=24183010 \|pmc=PMC5731466}}</ref><ref name="MiyamotoTheFiber16">{{cite journal \|title=The fiber-optic imaging and manipulation of neural activity during animal behavior \|journal=Neuroscience Research \|author=Miyamoto, D.; Murayama, M. \|volume=103 \|pages=1–9 \|year=2016 \|doi=10.1016/j.neures.2015.09.004 \|pmid=26427958}}</ref> In addition, recordings can be performed in parallel from multiple brain areas, together with behavioral measures such as eye or limb movements.<ref name="MaldonadoSynchro08">{{cite journal \|title=Synchronization of neuronal responses in primary visual cortex of monkeys viewing natural images \|journal=Journal of Neurophysiology \|author=Maldonado, P.; Babul, C.; Singer, W. et al. \|volume=100 \|issue=3 \|pages=1523-32 \|year=2008 \|doi=10.1152/jn.00076.2008 \|pmid=18562559}}</ref><ref name="VargasIrwinDecoding10">{{cite journal \|title=Decoding complete reach and grasp actions from local primary motor cortex populations \|journal=Journal of Neuroscience \|author=Vargas-Irwin, C.E.; Shakhnarovich, G.; Yadollahpour, P. et al. \|volume=30 \|issue=29 \|pages=9659-69 \|year=2010 \|doi=10.1523/JNEUROSCI.5443-09.2010 \|pmid=20660249 \|pmc=PMC2921895}}</ref> Such recordings enable researchers to study network interactions and cross-area coupling, and to relate neuronal processing to the behavioral performance of the subject.<ref name="BerenyiLarge14">{{cite journal \|title=Large-scale, high-density (up to 512 channels) recording of local circuits in behaving animals \|journal=Journal of Neurophysiology \|author=Berényi, A.; Somogyvári, Z.; Nagy, A.J. et al. \|volume=111 \|issue=5 \|pages=1132-49 \|year=2014 \|doi=10.1152/jn.00785.2013 \|pmid=24353300 \|pmc=PMC3949233}}</ref><ref name="LewisRecording15">{{cite journal \|title=Recording of brain activity across spatial scales \|journal=Current Opinion in Neurobiology \|author=Lewis, C.M.; Bosman, C.A.; Fries, P. et al. \|volume=32 \|pages=68–77 \|year=2015 \|doi=10.1016/j.conb.2014.12.007 \|pmid=25544724}}</ref><ref name="LismanTheChall15">{{cite journal \|title=The challenge of understanding the brain: Where we stand in 2015 \|journal=Neuron \|author=Lisman, J. \|volume=86 \|issue=4 \|pages=864-882 \|year=2015 \|doi=10.1016/j.neuron.2015.03.032 \|pmid=25996132 \|pmc=PMC4441769}}</ref> These approaches lead to increasingly complex experimental designs that are difficult to parameterize, e.g., due to multidimensional characterization of natural stimuli<ref name="GeislerVisual08">{{cite journal \|title=Visual perception and the statistical properties of natural scenes \|journal=Annual Review of Psychology \|author=Geisler, W.S. \|volume=59 \|pages=167–92 \|year=2008 \|doi=10.1146/annurev.psych.58.110405.085632 \|pmid=17705683}}</ref> or high-dimensional movement parameters for almost freely behaving subjects.<ref name="SchwarzChronic14">{{cite journal \|title=Chronic, wireless recordings of large-scale brain activity in freely moving rhesus monkeys \|journal=Nature Methods \|author=Schwarz, D.A.; Lebedev, M.A.; Hanson, T.L. et al. \|volume=11 \|issue=6 \|pages=670-6 \|year=2014 \|doi=10.1038/nmeth.2936 \|pmid=24776634 \|pmc=PMC4161037}}</ref> It is a serious challenge for researchers to keep track of the overwhelming amount of metadata generated at each experimental step and to precisely extract all the information relevant for data analysis and interpretation of results. Various aspects such as the parametrization of the experimental task, filter settings and sampling rates of the setup, the quality of the recorded data, broken electrodes, preprocessing steps (e.g., spike sorting), or the condition of the subject need to be considered. Nevertheless, the organization of these metadata is of utmost importance for conducting research in a reproducible manner, i.e., the ability to faithfully reproduce the experimental procedures and subsequent analysis steps.<ref name="LaineReproduc07">{{cite journal \|title=Reproducible research: Moving toward research the public can really trust \|journal=Annals of Internal Medicine \|author=Laine, C.; Goodman, S.N.; Griswold, M.E.; Sox, H.C. \|volume=146 \|issue=6 \|pages=450–3 \|year=2007 \|doi=10.7326/0003-4819-146-6-200703200-00154 \|pmid=17339612}}</ref><ref name="PengReproduc11">{{cite journal \|title=Reproducible research in computational science \|journal=Science \|author=Peng, R.D. \|volume=334 \|issue=6060 \|pages=1226-7 \|year=2011 \|doi=10.1126/science.1213847 \|pmid=22144613 \|pmc=PMC3383002}}</ref><ref name="TomaselloMethod11">{{cite journal \|title=Methodological challenges in the study of primate cognition \|journal=Science \|author=Tomasello, M.; Call, J. \|volume=334 \|issue=6060 \|pages=1227-8 \|year=2011 \|doi=10.1126/science.1213443 \|pmid=22144614}}</ref> Moreover, detailed knowledge of the complete recording and analysis processes is crucial for the correct interpretation of results, and is a minimal requirement to enable researchers to verify published results and build their own research on the previous findings.

			To achieve reproducibility, experimenters have typically developed their own lab procedures and practices for performing experiments and their documentation. Within the lab, crucial [[information]] about the experiment is often transmitted by personal communication, through handwritten [[laboratory notebook]]s, or implicitly by trained experimental procedures. However, at least when it comes to data sharing across labs, essential information is often missed in the exchange.<ref name="HinesSorting14">{{cite journal \|title=Sorting out the FACS: A devil in the details \|journal=Cell Reports \|author=Hines, W.C.; Su, Y.; Kuhn, I. et al. \|volume=6 \|issue=5 \|pages=779-81 \|year=2014 \|doi=10.1016/j.celrep.2014.02.021 \|pmid=24630040}}</ref><ref name="OSCPsychology15">{{cite journal \|title=Estimating the reproducibility of psychological science \|journal=Science \|author=Open Science Collaboration \|volume=349 \|issue=6251 \|pages=aac4716 \|year=2015 \|doi=10.1126/science.aac4716 \|pmid=26315443}}</ref> Moreover, if collaborating groups have different scientific backgrounds, for example experimenters and theoreticians, implicit domain-specific knowledge is often not communicated or is communicated in an ambiguous fashion that leads to misunderstandings. To avoid such scenarios, the general principle should be to keep as much information about an experiment as possible from the beginning on, even if information seems to be trivial or irrelevant at the time. Furthermore, one should annotate the data with these metadata in a clear and concise fashion.

			In order to provide metadata in an organized, easily accessible, but also machine-readable way, Grewe ''et al.''<ref name="GreweABottom11">{{cite journal \|title=A Bottom-up Approach to Data Annotation in Neurophysiology \|journal=Frontiers in Neuroinformatics \|author=Grewe, J.; Wachtler, T.; Benda, J. \|volume=5 \|pages=16 \|year=2011 \|doi=10.3389/fninf.2011.00016 \|pmid=21941477 \|pmc=PMC3171061}}</ref> introduced odML (open metadata Markup Language) as a simple file format in analogy to SBML in systems biology<ref name="HuckaTheSys03">{{cite journal \|title=The systems biology markup language (SBML): A medium for representation and exchange of biochemical network models \|journal=Bioinformatics \|author=Hucka, M.; Finney, A.; Sauro, H.M. et al. \|volume=19 \|issue=4 \|pages=524–31 \|year=2003 \|doi=10.1093/bioinformatics/btg015 \|pmid=12611808}}</ref>, or NeuroML in neuroscientific simulation studies.<ref name="GleesonNeuroML10">{{cite journal \|title=NeuroML: a language for describing data driven models of neurons and networks with a high degree of biological detail \|journal=PLoS Computational Biology \|author=Gleeson, P.; Crook, S.; Cannon, R.C. et al. \|volume=6 \|issue=6 \|pages=e1000815 \|year=2010 \|doi=10.1371/journal.pcbi.1000815 \|pmid=20585541 \|pmc=PMC2887454}}</ref><ref name="CrookCreating12">{{cite journal \|title=Creating, documenting and sharing network models \|journal=Network \|author=Crook, S.M.; Bednar, J.A.; Berger, S. et al. \|volume=23 \|issue=4 \|pages=131-49 \|year=2012 \|doi=10.3109/0954898X.2012.722743 \|pmid=22994683}}</ref> However, lacking to date is a detailed investigation on how to incorporate metadata management in the daily lab routine in terms of (i) organizing the metadata in a comprehensive collection, (ii) practically gathering and entering the metadata, and (iii) profiting from the resulting comprehensive metadata collection in the process of analyzing the data. Here we address these points, both conceptually and practically, in the context of a complex behavioral experiment that involves neuronal recordings from a large number of electrodes that yield massively parallel spike and local field potential (LFP) data.<ref name="RiehleMapping13">{{cite journal \|title=Mapping the spatio-temporal structure of motor cortical LFP and spiking activities during reach-to-grasp movements \|journal=Frontiers in Neural Circuits \|author=Riehle, A.; Wirtssohn, S.; Grün, S.; Brochier, T. \|volume=7 \|pages=48 \|year=2013 \|doi=10.3389/fncir.2013.00048 \|pmid=23543888 \|pmc=PMC3608913}}</ref> To illustrate how to organize a comprehensive metadata collection (i), we introduce in the next section the concept of metadata and demonstrate the rich diversity of metadata that arise in the context of the example experiment. To demonstrate why the effort of creating a comprehensive metadata collection is time well spent, we next describe five use cases that summarize where the access to metadata becomes relevant when working with the data (iii). Afterwards we provide detailed guidelines and assistance on how to create, structure, and hierarchically organize comprehensive metadata collections (i and ii). Complementing these guidelines, we provide in the supplementary material section a thorough practical introduction on how to embed a metadata management tool, such as the odML library, into the experimental and analysis workflow. Finally, we close by critically contrasting the importance of proper metadata handling against its difficulties and deriving future challenges.

	==References==		==References==