Journal:A metadata-driven approach to data repository design - Revision history

Shawndouglas: /* Notes */ Cats

2024-04-29T16:31:58Z

Notes: Cats

← Older revision		Revision as of 16:31, 29 April 2024
Line 191:		Line 191:

	<!--Place all category tags here-->		<!--Place all category tags here-->
	[[Category:LIMSwiki journal articles (added in 2017)‎]]		[[Category:LIMSwiki journal articles (added in 2017)]]
	[[Category:LIMSwiki journal articles (all)‎]]		[[Category:LIMSwiki journal articles (all)]]
	[[Category:LIMSwiki journal articles on big ~~data‎‎~~]]		[[Category:LIMSwiki journal articles on big data]]
	[[Category:LIMSwiki journal articles on ~~informatics‎‎~~]]		[[Category:LIMSwiki journal articles on FAIR data principles]]
			[[Category:LIMSwiki journal articles on informatics]]

Shawndouglas: /* Abstract */ Keyword

2024-04-29T16:31:42Z

Abstract: Keyword

← Older revision		Revision as of 16:31, 29 April 2024
Line 22:		Line 22:
	The design and use of a metadata-driven data repository for research data management is described. Metadata is collected automatically during the submission process whenever possible and is registered with DataCite in accordance with their current metadata schema, in exchange for a persistent digital object identifier. Two examples of data preview are illustrated, including the demonstration of a method for integration with commercial software that confers rich domain-specific [[data analysis\|data analytics]] without introducing customization into the repository itself.		The design and use of a metadata-driven data repository for research data management is described. Metadata is collected automatically during the submission process whenever possible and is registered with DataCite in accordance with their current metadata schema, in exchange for a persistent digital object identifier. Two examples of data preview are illustrated, including the demonstration of a method for integration with commercial software that confers rich domain-specific [[data analysis\|data analytics]] without introducing customization into the repository itself.

	'''Keywords''': Data repository, metadata-driven, DataCite, data preview, Mpublish		'''Keywords''': Data repository, metadata-driven, DataCite, data preview, FAIR, Mpublish

	==Background==		==Background==

Shawndouglas: Finished adding rest of content.

2017-02-08T20:36:41Z

Finished adding rest of content.

← Older revision		Revision as of 20:36, 8 February 2017
Line 18:		Line 18:
	\|website = [http://jcheminf.springeropen.com/articles/10.1186/s13321-017-0190-6 http://jcheminf.springeropen.com/articles/10.1186/s13321-017-0190-6]		\|website = [http://jcheminf.springeropen.com/articles/10.1186/s13321-017-0190-6 http://jcheminf.springeropen.com/articles/10.1186/s13321-017-0190-6]
	\|download = [http://jcheminf.springeropen.com/track/pdf/10.1186/s13321-017-0190-6?site=jcheminf.springeropen.com http://jcheminf.springeropen.com/track/pdf/10.1186/s13321-017-0190-6] (PDF)		\|download = [http://jcheminf.springeropen.com/track/pdf/10.1186/s13321-017-0190-6?site=jcheminf.springeropen.com http://jcheminf.springeropen.com/track/pdf/10.1186/s13321-017-0190-6] (PDF)
	}}
	~~{{ombox~~
	~~\| type = content~~
	~~\| style = width: 500px;~~
	~~\| text = This article should not be considered complete until this message box has been removed. This is a work in progress.~~
	}}		}}
	==Abstract==		==Abstract==
Line 144:		Line 139:

	We believe this feature demonstrates a powerful incentive for using the repository. By enabling the use of custom software on submitted datasets, the repository becomes more than a passive silo for data, actively enabling depositors and viewers to interact with datasets in a rich, domain-specific way. Furthermore, it is accomplished without the need to develop format-specific enhancements into the repository itself.		We believe this feature demonstrates a powerful incentive for using the repository. By enabling the use of custom software on submitted datasets, the repository becomes more than a passive silo for data, actively enabling depositors and viewers to interact with datasets in a rich, domain-specific way. Furthermore, it is accomplished without the need to develop format-specific enhancements into the repository itself.

			===LiveView===
			The most generic solution to data preview is the deployment of HTML in conjunction with appropriated visualization routines. One example of this can be illustrated with the following components.

			* The HTML document is assigned the reserved name <tt>index.html</tt>.

			* When a document with this name is deposited, it is automatically transcluded into the landing page of the deposition using an HTML iframe: <tt><nowiki><iframe name=“liveview” src=“/resolve/?doi=1248&file=13&access=“ width=“100%” height=“600”></iframe></nowiki></tt>, where the string <tt>doi=1248&file=13&access=</tt> references the appropriate database entry for the object assigned the DOI:[http://dx.doi.org/10.14469/hpc/1248 10.14469/hpc/1248].

			* The preview functionality is then enabled by author-specified inclusion of JavaScript containing utility functions hosted on the repository into the <tt>index.html</tt> document: <tt><nowiki><script src=“ https://data.hpc.imperial.ac.uk/js/utilities.js “></nowiki></tt>. These serve to invoke an open-source molecular visualiser JSmol<ref name="HansonJSmol13">{{cite journal \|title=JSmol and the Next-Generation Web-Based Representation of 3D Molecular Structure as Applied to ''Proteopedia'' \|journal=Israel Journal of Chemistry \|author=Hanson, R.M.; Prilusky, J.; Renjian, Z. et al. \|volume=53 \|issue=3-4 \|pages=207–216 \|year=2013 \|doi=10.1002/ijch.201300024}}</ref> which as the name implies is based purely on JavaScript.

			* A further script is loaded at this stage, along with a formatting stylesheet: <tt><nowiki><script> insertFile(“resolve-doi.js”); insertFile(“table.css”); </script></nowiki></tt>. The resolve-doi.js script invokes procedures which accept a dataset DOI as input. Querying the metadata associated with that DOI using the form <tt>http://data.datacite.org/10.14469/ch/192018</tt> allows the path to the ORE or METS resource manifests to be identified (https://spectradspace.lib.imperial.ac.uk:8443/metadata/handle/10042/196268/ore.xml in this instance) and parsing of this manifest then allows the direct path to the data to be extracted and passed through to the JSmol visualization script.

			* An author-initiated entry of the following type in the <tt>index.html</tt> document then conflates these various actions, the result being a live view of the retrieved dataset in the browser window (Fig. 4): <tt><nowiki>javascript:handle_jmol(‘10.14469/ch/192018’,’;display script;’)”>anchored text</a></nowiki></tt>


			:[[File:Fig4 Harvey JoCheminformatics2017 9.gif\|794px]]
			{{clear}}
			{\|
			\| STYLE="vertical-align:top;"\|
			{\| border="0" cellpadding="5" cellspacing="0" width="794px"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\| <blockquote>'''Figure 4.''' Liveview<ref name="RzepaFAIR16">{{cite web \|url=https://data.hpc.imperial.ac.uk/resolve/?doi=1248 \|title=FAIR Data table. Computed relative reaction free energies (kcal/mol-1) of Obtusallene derived oxonium and chloronium cations \|work=Imperial College High Performance Computing Service Data Repository \|author=Rzepa, H.; White, A.; Braddock, D.C. et al. \|publisher=Imperial College London \|doi=10.14469/hpc/1248 \|date=10 August 2016 \|accessdate=07 November 2016}}</ref> of a dataset collection expressed using HTML and integrated visualization package with data retrieved by script-driven DOI-based data retrieval</blockquote>
			\|-
			\|}
			\|}

			* The document <tt>index.html</tt> is itself available for download from the repository and hence can act as a useful template for other authors.

			The metadata-driven approach described here is directed towards enabling the FAIR<ref name="WilkinsonTheFAIR16" /> attributes of the data. Registering rich metadata associated with, for example, a primary research object; in molecular sciences the molecule, in turn allows rich searches (the F of FAIR) to be constructed using the generic resources available at the metadata aggregator, DataCite. This approach makes for an interesting contrast with that adopted by the publisher Elsevier<ref name="DS">{{cite web \|url=https://datasearch.elsevier.com/ \|title=DataSearch \|publisher=Elsevier B.V \|accessdate=07 September 2017}}</ref> for their recently introduced DataSearch site. This initially appears to be based on the content provided by their journal base, ScienceDirect. The metadata here is simply the likely (but not guaranteed) presence of data within containers such as images or tables. In this approach, a user-driven data search culminates in the user being directed to the data source, which is in fact the article itself as an object. It is then very much up to the user to identify the data of interest within the article, whether in the text or images of the article itself or any associated supporting information. Such an approach is clearly not based on metadata describing data objects such as molecular entities and leaves the burden on the user to identify and extract any such information themselves. It would be itself fair to suggest that such a process does not fully adhere to the principles of FAIR data.

			A clear emerging trend is that journal publication is starting to be associated with procedures for identifying associated data as a primary research object in its own right. The extent to which such data is rendered fully open, in the sense of being compliant with all of the FAIR principles, remains uncertain. It seems likely that journal publishers, who will retain full control over the complete workflows involving data, may not necessarily wish to expose the data as openly FAIR or at the granularity which may be most useful to the researcher. Here we have outlined an alternative metadata-driven mechanism for achieving finely-grained FAIR data exposures in association with journal publication which can be utilized by authors themselves as the creators of research data, and where authors can retain control over the type of metadata captured. In this alternative model, open FAIR data is published at the research institutional level and the associated metadata aggregated at the global level by agencies such as DataCite without a need for intervention by journal-publisher workflows.

			==Declarations==
			===Authors' contributions===
			All authors contributed to this work. All authors read and approved the final manuscript.

			====Acknowledgements====
			We are grateful to MestreLab Research for providing instructions for signing their datasets and for countersigning the public key for the repository described here.

			====Competing interests====
			The authors declare that they have no competing interests.

	==Footnotes==		==Footnotes==

Shawndouglas: Saving and adding more.

2017-02-08T19:19:17Z

Saving and adding more.

Show changes

Shawndouglas: Saving and adding more.

2017-02-08T18:28:16Z

Saving and adding more.

← Older revision		Revision as of 18:28, 8 February 2017
Line 69:		Line 69:
	* The repository is designed to be operable through a command line and programmatic web API. This allows scripted integration of the deposition process into other workflows such as [[electronic laboratory notebook]]s.<ref name="HarveyDigital14">{{cite journal \|title=Digital data repositories in chemistry and their integration with journals and electronic laboratory notebooks \|journal=Journal of Chemical Information and Modeling \|author=Harvey, M.J.; Mason, N.J.; Rzepa, H.S. \|volume=54 \|issue=10 \|pages=2627–2635 \|year=2014 \|doi=10.1021/ci500302p}}</ref>		* The repository is designed to be operable through a command line and programmatic web API. This allows scripted integration of the deposition process into other workflows such as [[electronic laboratory notebook]]s.<ref name="HarveyDigital14">{{cite journal \|title=Digital data repositories in chemistry and their integration with journals and electronic laboratory notebooks \|journal=Journal of Chemical Information and Modeling \|author=Harvey, M.J.; Mason, N.J.; Rzepa, H.S. \|volume=54 \|issue=10 \|pages=2627–2635 \|year=2014 \|doi=10.1021/ci500302p}}</ref>

	* The repository to be integrated with the widely-used source code management website ~~Github~~, and can automatically allocate DOIs to software releases made through that platform. This extends the benefit of DOI citability to software projects without requiring additional effort on behalf of the developer, once the initial configuration has been made.		* The repository to be integrated with the widely-used source code management website GitHub, and can automatically allocate DOIs to software releases made through that platform. This extends the benefit of DOI citability to software projects without requiring additional effort on behalf of the developer, once the initial configuration has been made.

	* The repository is registered via the registry of research data repositories.<ref name="RE3">{{cite web \|url=http://www.re3data.org/about \|title=About \|work=Registry of Research Data Repositories \|publisher=Karlsruhe Institute of Technology \|accessdate=07 September 2016}}</ref> This involves populating a schema template provided by re3data with the appropriate attributes, which is then processed to create a repository record. This results in the metadata describing the repository itself being assigned a DOI.<ref name="ImperialCollege">{{cite web \|url=http://www.re3data.org/repository/r3d100011965 \|title=Imperial College High Performance Computing Service Data Repository \|work=Registry of Research Data Repositories \|publisher=Karlsruhe Institute of Technology \|doi=10.17616/R3K64N \|accessdate=07 September 2016}}</ref> The repository schema is available as an XML file<ref name="XMLReg">{{cite web \|url=https://data.hpc.imperial.ac.uk/resolve/?doi=1369 \|title=XML registration with re3data \|work=Imperial College High Performance Computing Service Data Repository \|publisher=Imperial College London \|doi=10.14469/hpc/1369 \|date=07 September 2016 \|accessdate=07 September 2016}}</ref>, with further data and metadata information deposited for inspection.<ref name="DRP">{{cite web \|url=https://data.hpc.imperial.ac.uk/resolve/?doi=1088 \|title=Data Repository Project \|work=Imperial College High Performance Computing Service Data Repository \|publisher=Imperial College London \|doi=10.14469/hpc/1088 \|date=25 July 2016 \|accessdate=07 September 2016}}</ref>		* The repository is registered via the registry of research data repositories.<ref name="RE3">{{cite web \|url=http://www.re3data.org/about \|title=About \|work=Registry of Research Data Repositories \|publisher=Karlsruhe Institute of Technology \|accessdate=07 September 2016}}</ref> This involves populating a schema template provided by re3data with the appropriate attributes, which is then processed to create a repository record. This results in the metadata describing the repository itself being assigned a DOI.<ref name="ImperialCollege">{{cite web \|url=http://www.re3data.org/repository/r3d100011965 \|title=Imperial College High Performance Computing Service Data Repository \|work=Registry of Research Data Repositories \|publisher=Karlsruhe Institute of Technology \|doi=10.17616/R3K64N \|accessdate=07 September 2016}}</ref> The repository schema is available as an XML file<ref name="XMLReg">{{cite web \|url=https://data.hpc.imperial.ac.uk/resolve/?doi=1369 \|title=XML registration with re3data \|work=Imperial College High Performance Computing Service Data Repository \|publisher=Imperial College London \|doi=10.14469/hpc/1369 \|date=07 September 2016 \|accessdate=07 September 2016}}</ref>, with further data and metadata information deposited for inspection.<ref name="DRP">{{cite web \|url=https://data.hpc.imperial.ac.uk/resolve/?doi=1088 \|title=Data Repository Project \|work=Imperial College High Performance Computing Service Data Repository \|publisher=Imperial College London \|doi=10.14469/hpc/1088 \|date=25 July 2016 \|accessdate=07 September 2016}}</ref>

			==Engineering==
			The repository is intended for use by affiliates of the deploying hosting institution. Deposition first requires requires authentication performed against an institutional authentication and authorization (A&A) LDAP service. As a matter of policy, the repository also requires the depositing user to provide their ORCID identifier, obtained via an Oauth transaction<ref name="RicherUser">{{cite web \|url=https://oauth.net/articles/authentication/ \|title=User Authentication with OAuth 2.0 \|author=Richer, J. \|work=OAuth.net \|accessdate=07 September 2016}}</ref> with the ORCID web service.

			The repository is accessed via interfaces designed to function both as a human-friendly UI (accessed via a web browser) and as a programmable API. The latter is essential for integrating deposition into higher level tools and workflows and exposes all the capabilities of the repository. In order to deposit, command-line tools or other programs using the API must also authenticate and the repository is able to provide delegated access to a user’s account for such tools through a transaction similar to Oauth. This allows automated use performed by a third party tool on behalf of a user to be clearly delineated from actions performed by the user and furthermore allows selective revocation of access to the third-party. Current integrations include a computational science portal which manages the execution of quantum chemistry calculations on Imperial College HPC resources. This portal is able to directly publish results into the repository, automatically passing on dataset data files and descriptive metadata.

			Data files stored within a repository are maintained on a local filesystem on the server hosting the repository. As data burdens grow to multi-terabyte levels, we expect to migrate this data to remote filesystems. The internal database representation of a dataset deposition allows the files to reside on independent web server, in which case the repository will resolve any requests for them to an HTTP redirect. This would facilitate any future extension of the repository to use a third-party storage solution (e.g., Amazon Web Services S3 object store), or a content distribution network.

			The repository automatically generates and publishes metadata records conforming to the DataCite Medadata Schema 4.0.<ref name="DMWG">{{cite web \|url=http://schema.datacite.org/meta/kernel-4.0/ \|title=DataCite Metadata Schema 4.0 \|publisher=DataCite Metadata Working Group \|date=19 September 2016 \|accessdate=16 January 2017}}</ref> The metadata records are automatically updated whenever a user updates an entry, such as, for example, including a subsequently obtained DOI to a related journal article. At the present time there is a latency of approximately two days before the DataCite search engine index incorporates any updates.

			For the GitHub integration, the repository end-user first associates the repository with GitHub, again using an OAuth transaction.<ref name="RicherUser" /> Thereafter, the repository maintains a list of the user’s GitHub projects, both public and private, for which DOI creation may be selectively enabled. Once activated, a GitHub “webhook”<ref name="GHWebhooks">{{cite web \|url=https://developer.github.com/webhooks/ \|title=Webhooks \|work=API \|publisher=GitHub, Inc \|accessdate=07 September 2016}}</ref> is created which automatically makes an HTTP request to the repository whenever a software release is created. This request contains sufficient metadata about the release to allow the repository to create a DOI and automatically populate its metadata. The DOI is recorded within the repository and also added to the release description held within GitHub.

			The repository is implemented in PHP hosted within an Apache web server and depends on a [[PostgreSQL\|Postgres]] database. The source code is available on GitHub.<ref name="HPCRepo">{{cite web \|url=https://github.com/ICHPC/hpc-repo \|title=ICHPC/hpc-repo \|author=Harvey, M.J. \|publisher=GitHub, Inc \|doi=10.14469/hpc/1487 \|accessdate=07 September 2016}}</ref>

	==References==		==References==

Shawndouglas: Saving and adding more.

2017-02-08T18:11:15Z

Saving and adding more.

← Older revision		Revision as of 18:11, 8 February 2017
Line 58:		Line 58:

	* Our approach to metadata collection is to automate the process whenever possible. In the case of a molecule as an object, there are algorithms which can be used to generate appropriate metadata, the most useful and prominent of which is the InChI (International Chemical Identifier).<ref name="InChI">{{cite web \|url=http://www.inchi-trust.org/ \|title=InChI Trust \|publisher=InChI Trust \|accessdate=07 September 2016}}</ref> The task of creating such an identifier is effectively accomplished using the OpenBabel program library<ref name="OBoyleOpen11">{{cite journal \|title=Open Babel: An open chemical toolbox \|journal=Journal of Cheminformatics \|author=O'Boyle, N.M.; Banck, M.; James, C.A. et al. \|volume=3 \|pages=33 \|year=2011 \|doi=10.1186/1758-2946-3-33 \|pmid=21982300 \|pmc=PMC3198950}}</ref> or via Javascript-based resources.<ref name="InChI.js">{{cite web \|url=https://metamolecular.com/inchi-js/ \|title=InChI for the Web Browser with InChI.js \|publisher=Metamolecular, LLC \|accessdate=07 September 2016}}</ref> These can accept as input a variety of chemical documents and generate an appropriate InChI identifier and InChI key uniquely describing them. The repository workflow automatically processes any uploaded data file through this algorithm and records all successful outputs. Such metadata is then associated with the <tt>Subject</tt> element in the DataCite schema (Fig. 1b).		* Our approach to metadata collection is to automate the process whenever possible. In the case of a molecule as an object, there are algorithms which can be used to generate appropriate metadata, the most useful and prominent of which is the InChI (International Chemical Identifier).<ref name="InChI">{{cite web \|url=http://www.inchi-trust.org/ \|title=InChI Trust \|publisher=InChI Trust \|accessdate=07 September 2016}}</ref> The task of creating such an identifier is effectively accomplished using the OpenBabel program library<ref name="OBoyleOpen11">{{cite journal \|title=Open Babel: An open chemical toolbox \|journal=Journal of Cheminformatics \|author=O'Boyle, N.M.; Banck, M.; James, C.A. et al. \|volume=3 \|pages=33 \|year=2011 \|doi=10.1186/1758-2946-3-33 \|pmid=21982300 \|pmc=PMC3198950}}</ref> or via Javascript-based resources.<ref name="InChI.js">{{cite web \|url=https://metamolecular.com/inchi-js/ \|title=InChI for the Web Browser with InChI.js \|publisher=Metamolecular, LLC \|accessdate=07 September 2016}}</ref> These can accept as input a variety of chemical documents and generate an appropriate InChI identifier and InChI key uniquely describing them. The repository workflow automatically processes any uploaded data file through this algorithm and records all successful outputs. Such metadata is then associated with the <tt>Subject</tt> element in the DataCite schema (Fig. 1b).

			* Other metadata describing any individual collection or items within the collection can be used to link to other data repositories via the appropriate persistent identifier (DOI) as well as associated journal publications where relevant, again using the DOI. These linkages can of course be made bidirectional by including a citation to the data at the remote site. Such inclusion of bidirectional linking data is currently less automated, but one might envisage future methods for automation involving the ORCID identifier and the ORCID resources as a possible aggregator.

			* When a collection or an individual dataset is deposited, the item is immediately issued with a reserved DataCite DOI to allow the authors to quote it in any articles being prepared. Its status is defined as embargoed with an associated access code to allow collaborators to view the item and if necessary to also forward to a journal editor so that they can arrange access for referees. The embargo can be released at a time agreed by the authors, either in advance of the submission of any resulting article, or at the time of open publication of that article. The embargo release is not recursive to any members.

			* The repository incorporates an ORE resource map<ref name="ORE">{{cite web \|url=http://www.openarchives.org/ore/1.0/datamodel \|title=ORE Specification - Abstract Data Model \|publisher=Open Archives Initiative \|accessdate=07 September 2016}}</ref>, with appropriate metadata descriptors collected to describe the location of this resource map in the repository. This in turn allows a query of DataCite using just the assigned DOI to retrieve the ORE map (Fig. 1d) and facilitates automated retrieval of any individual file contained within a dataset based just on its DOI and if necessary its media type. We have described applications of this procedure termed DOI2Data.<ref name="HarveyStand15">{{cite journal \|title=Standards-based metadata procedures for retrieving data for display or mining utilizing persistent (data-DOI) identifiers \|journal=Journal of Cheminformatics \|author=Harvey, M.J.; Mason, N.J.; McLean, A.; Rzepa, H.S. \|volume=7 \|pages=37 \|year=2015 \|doi=10.1186/s13321-015-0081-7 \|pmid=26257829 \|pmc=PMC4528360}}</ref> Such procedures effectively remove any need to navigate from the landing page associated with the DOI to find and recover data and open up possibilities for large scale automated data mining procedures based just on, for example, top-level collection DOIs. We have also implemented the metadata required to allow the procedure DataCite calls content negotiation<ref name="HarveyStand15" /><ref name="ConNeg">{{cite web \|url=http://citation.crosscite.org/docs.html \|title=DOI Content Negotiation \|work=DOI Citation Formatter \|accessdate=07 September 2016}}</ref> (Fig. 1f). An example of date retrieval involving such negotiation might be <tt>http://data.datacite.org/chemical/x-mnpub/10.14469/hpc/1280</tt>. This queries whether the item with assigned the DOI <tt>10.14469/hpc/1280</tt> has any content associated with the specified media type "chemical/x-mnpub" and if so retrieves the first instance of such data. If there are multiple such instances in the dataset, then the ORE<ref name="ORE" /> (or METS)<ref name="HarveyStand15" /> method must be used to select them.

			* An emerging feature of data repositories is data preview which can be used as a navigational metaphor. When repositories were largely focused on storing journal articles, preview of the most common document type, the PDF format, was the most important requirement. Most data however is not (certainly should not be) contained in such a document. Clearly, data preview is going to be largely dependent on the discipline associated with the data, and it will be difficult to generalize such procedures. We will describe two specific implementations of preview below, but it is important in the initial design of a repository to recognize the need for such rich preview.

			* The repository is designed to be operable through a command line and programmatic web API. This allows scripted integration of the deposition process into other workflows such as [[electronic laboratory notebook]]s.<ref name="HarveyDigital14">{{cite journal \|title=Digital data repositories in chemistry and their integration with journals and electronic laboratory notebooks \|journal=Journal of Chemical Information and Modeling \|author=Harvey, M.J.; Mason, N.J.; Rzepa, H.S. \|volume=54 \|issue=10 \|pages=2627–2635 \|year=2014 \|doi=10.1021/ci500302p}}</ref>

			* The repository to be integrated with the widely-used source code management website Github, and can automatically allocate DOIs to software releases made through that platform. This extends the benefit of DOI citability to software projects without requiring additional effort on behalf of the developer, once the initial configuration has been made.

			* The repository is registered via the registry of research data repositories.<ref name="RE3">{{cite web \|url=http://www.re3data.org/about \|title=About \|work=Registry of Research Data Repositories \|publisher=Karlsruhe Institute of Technology \|accessdate=07 September 2016}}</ref> This involves populating a schema template provided by re3data with the appropriate attributes, which is then processed to create a repository record. This results in the metadata describing the repository itself being assigned a DOI.<ref name="ImperialCollege">{{cite web \|url=http://www.re3data.org/repository/r3d100011965 \|title=Imperial College High Performance Computing Service Data Repository \|work=Registry of Research Data Repositories \|publisher=Karlsruhe Institute of Technology \|doi=10.17616/R3K64N \|accessdate=07 September 2016}}</ref> The repository schema is available as an XML file<ref name="XMLReg">{{cite web \|url=https://data.hpc.imperial.ac.uk/resolve/?doi=1369 \|title=XML registration with re3data \|work=Imperial College High Performance Computing Service Data Repository \|publisher=Imperial College London \|doi=10.14469/hpc/1369 \|date=07 September 2016 \|accessdate=07 September 2016}}</ref>, with further data and metadata information deposited for inspection.<ref name="DRP">{{cite web \|url=https://data.hpc.imperial.ac.uk/resolve/?doi=1088 \|title=Data Repository Project \|work=Imperial College High Performance Computing Service Data Repository \|publisher=Imperial College London \|doi=10.14469/hpc/1088 \|date=25 July 2016 \|accessdate=07 September 2016}}</ref>

	==References==		==References==

Shawndouglas: /* Data repository design features */ TT

2017-02-07T22:29:22Z

Data repository design features: TT

← Older revision		Revision as of 22:29, 7 February 2017
Line 57:		Line 57:
	: Basing the repository design on collections also reflects the manner in which much modern science is conducted, often via multi-disciplinary collaborations in which each group can generate its own data collections. Collections also greatly facilitate data citation in journal articles. For example, the persistent identifier (DOI) of just the highest collection level of datasets associated with an article can be therein cited, avoiding citation blight. If a particular object (a molecule in our case) is being discussed in the text of the article, it might nevertheless be more appropriate to reference the specific DOI at that stage. Individual citation is also useful in, for example, tables of results or figures. The metadata for any individual cited dataset will also contain the attribute "is member of," so that the hierarchy can be both tracked upwards, and via the attribute "has members" downwards (Fig. 1d). This hierarchy also introduces via such metadata further semantics into the citation process itself; each item is placed into appropriate context. Lack of such semantics/context are arguably one of the most deficient aspects of current citation practices in journal articles.		: Basing the repository design on collections also reflects the manner in which much modern science is conducted, often via multi-disciplinary collaborations in which each group can generate its own data collections. Collections also greatly facilitate data citation in journal articles. For example, the persistent identifier (DOI) of just the highest collection level of datasets associated with an article can be therein cited, avoiding citation blight. If a particular object (a molecule in our case) is being discussed in the text of the article, it might nevertheless be more appropriate to reference the specific DOI at that stage. Individual citation is also useful in, for example, tables of results or figures. The metadata for any individual cited dataset will also contain the attribute "is member of," so that the hierarchy can be both tracked upwards, and via the attribute "has members" downwards (Fig. 1d). This hierarchy also introduces via such metadata further semantics into the citation process itself; each item is placed into appropriate context. Lack of such semantics/context are arguably one of the most deficient aspects of current citation practices in journal articles.

	* Our approach to metadata collection is to automate the process whenever possible. In the case of a molecule as an object, there are algorithms which can be used to generate appropriate metadata, the most useful and prominent of which is the InChI (International Chemical Identifier).<ref name="InChI">{{cite web \|url=http://www.inchi-trust.org/ \|title=InChI Trust \|publisher=InChI Trust \|accessdate=07 September 2016}}</ref> The task of creating such an identifier is effectively accomplished using the OpenBabel program library<ref name="OBoyleOpen11">{{cite journal \|title=Open Babel: An open chemical toolbox \|journal=Journal of Cheminformatics \|author=O'Boyle, N.M.; Banck, M.; James, C.A. et al. \|volume=3 \|pages=33 \|year=2011 \|doi=10.1186/1758-2946-3-33 \|pmid=21982300 \|pmc=PMC3198950}}</ref> or via Javascript-based resources.<ref name="InChI.js">{{cite web \|url=https://metamolecular.com/inchi-js/ \|title=InChI for the Web Browser with InChI.js \|publisher=Metamolecular, LLC \|accessdate=07 September 2016}}</ref> These can accept as input a variety of chemical documents and generate an appropriate InChI identifier and InChI key uniquely describing them. The repository workflow automatically processes any uploaded data file through this algorithm and records all successful outputs. Such metadata is then associated with the Subject element in the DataCite schema (Fig. 1b).		* Our approach to metadata collection is to automate the process whenever possible. In the case of a molecule as an object, there are algorithms which can be used to generate appropriate metadata, the most useful and prominent of which is the InChI (International Chemical Identifier).<ref name="InChI">{{cite web \|url=http://www.inchi-trust.org/ \|title=InChI Trust \|publisher=InChI Trust \|accessdate=07 September 2016}}</ref> The task of creating such an identifier is effectively accomplished using the OpenBabel program library<ref name="OBoyleOpen11">{{cite journal \|title=Open Babel: An open chemical toolbox \|journal=Journal of Cheminformatics \|author=O'Boyle, N.M.; Banck, M.; James, C.A. et al. \|volume=3 \|pages=33 \|year=2011 \|doi=10.1186/1758-2946-3-33 \|pmid=21982300 \|pmc=PMC3198950}}</ref> or via Javascript-based resources.<ref name="InChI.js">{{cite web \|url=https://metamolecular.com/inchi-js/ \|title=InChI for the Web Browser with InChI.js \|publisher=Metamolecular, LLC \|accessdate=07 September 2016}}</ref> These can accept as input a variety of chemical documents and generate an appropriate InChI identifier and InChI key uniquely describing them. The repository workflow automatically processes any uploaded data file through this algorithm and records all successful outputs. Such metadata is then associated with the <tt>Subject</tt> element in the DataCite schema (Fig. 1b).

	==References==		==References==

Shawndouglas: Saving and adding more.

2017-02-07T22:27:53Z

Saving and adding more.

← Older revision		Revision as of 22:27, 7 February 2017
Line 34:		Line 34:
	One feature of the curation phase<ref name="HarveyStandards15">{{cite journal \|title=Standards-based curation of a decade-old digital repository dataset of molecular information \|journal=Journal of Cheminformatics \|author=Harvey, M.J.; Mason, N.J.; McLean, A. et al. \|volume=7 \|pages=43 \|year=2015 \|doi=10.1186/s13321-015-0093-3 \|pmid=26322133 \|pmc=PMC4550659}}</ref> of the project aimed to explore the capabilities of the DataCite metadata schemas to improve the discoverability of the deposited data. The metadata can then be exploited to create rich search queries.<ref name="RzepaInChl16">{{cite journal \|title=InChI as a research data management tool \|journal=Chemistry International \|author=Rzepa, H.S.; Mclean, A.; Harvey, M.J. \|volume=38 \|issue=3–4 \|pages=24–26 \|year=2015 \|doi=10.1515/ci-2016-3-408}}</ref> As a result of the experiences gained from this project, we became aware that one limiting factor to the effective use of metadata was the repository design itself. The next stage therefore was to explore whether what we considered the essential requirements for a data repository could be incorporated into a new design. Here we report the principles used to create such a repository and some of the applications in chemistry that have resulted. These principles may in turn assist researchers wishing to deposit data in identifying the repository attributes that can best expose the discoverability and re-use of their data.		One feature of the curation phase<ref name="HarveyStandards15">{{cite journal \|title=Standards-based curation of a decade-old digital repository dataset of molecular information \|journal=Journal of Cheminformatics \|author=Harvey, M.J.; Mason, N.J.; McLean, A. et al. \|volume=7 \|pages=43 \|year=2015 \|doi=10.1186/s13321-015-0093-3 \|pmid=26322133 \|pmc=PMC4550659}}</ref> of the project aimed to explore the capabilities of the DataCite metadata schemas to improve the discoverability of the deposited data. The metadata can then be exploited to create rich search queries.<ref name="RzepaInChl16">{{cite journal \|title=InChI as a research data management tool \|journal=Chemistry International \|author=Rzepa, H.S.; Mclean, A.; Harvey, M.J. \|volume=38 \|issue=3–4 \|pages=24–26 \|year=2015 \|doi=10.1515/ci-2016-3-408}}</ref> As a result of the experiences gained from this project, we became aware that one limiting factor to the effective use of metadata was the repository design itself. The next stage therefore was to explore whether what we considered the essential requirements for a data repository could be incorporated into a new design. Here we report the principles used to create such a repository and some of the applications in chemistry that have resulted. These principles may in turn assist researchers wishing to deposit data in identifying the repository attributes that can best expose the discoverability and re-use of their data.

			==Data repository design features==
			Here we describe the requirements we identified for a metadata-driven repository, an instance of which is deployed by the Imperial College HPC Service at [https://data.hpc.imperial.ac.uk https://data.hpc.imperial.ac.uk]:

			* In our design, we have focused on enhancing the FAIR<ref name="WilkinsonTheFAIR16">{{cite journal \|title=The FAIR Guiding Principles for scientific data management and stewardship \|journal=Scientific Data \|author=Wilkinson, M.D.; Dumontier, M.; Aalbersberg, I.J. et al. \|volume=3 \|pages=160018 \|year=2016 \|doi=10.1038/sdata.2016.18 \|pmid=26978244 \|pmc=PMC4792175}}</ref> attributes of the data. The first attribute F means the data must be findable and practically this means making the metadata descriptors as rich and complete as possible to enable this. A = Accessibility is achieved by assigning persistent identifiers to the datasets and again associating them with appropriate metadata to enable automated retrieval processes if appropriate. This in turn helps ensure that the data can be accessed in a standard manner to enable its inter-operability in various software environments. R = Re-usability is related to understanding and trusting its provenance and the license terms under which it can be processed.

			* The provenance of the deposited data is established from the unique ORCiD identifier of the depositor(s). On the first occasion that the repository is used after initial institutional-based authentication, a redirection to the ORCiD site occurs. There the depositor creates an account or authenticates an existing account, followed by authorising the repository request. The retrieved ORCiD is then added to the metadata manifest for the deposition as a depositor attribute. This initial depositor can then add further ORCiDs as co-authors to the entry; these again are validated automatically from the ORCiD site. This information is then collected and sent to DataCite for aggregation (Fig. 1e).


			[[File:Fig1 Harvey JoCheminformatics2017 9.gif\|709px]]
			{{clear}}
			{\|
			\| STYLE="vertical-align:top;"\|
			{\| border="0" cellpadding="5" cellspacing="0" width="709px"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\| <blockquote>'''Figure 1.''' Metadata registered with DataCite for doi:10.14469/hpc/1280, with individual items (a)–(f) discussed in the section on metadata expression</blockquote>
			\|-
			\|}
			\|}

			* The structure of the repository is based on hierarchical collections. Although collections have been a feature of early repositories such as DSpace, relatively little use has been made of them. We first identified the need for such structures from our early project<ref name="DowningSPECTRa08" /> involving individual deposition of >168,000 items. This was deemed necessary since we considered that each item would benefit from having its own unique metadata descriptors, but within the context of a complete collection described using separate metadata. This is illustrated by assigning metadata both to individual entries<ref name="DC10.14469ch153690">{{cite web \|url=https://data.datacite.org/10.14469/ch/153690 \|title=doi:10.14469/ch/153690 \|work=DataCite Content Service Beta \|publisher=DataCite Association \|accessdate=07 September 2016}}</ref> and to the collection which the individual items are members of.<ref name="DC10.14469ch2">{{cite web \|url=https://data.datacite.org/10.14469/ch/2 \|title=doi:10.14469/ch/2 \|work=DataCite Content Service Beta \|publisher=DataCite Association \|accessdate=07 September 2016}}</ref> Such hierarchical structures allow a research group to assign collections to project themes and within these to identify sub-collections associated with individual researchers or teams. The sub-collections can be further structured into types of data, other research objects such as software, presentations on the topic and other media such as video. The granularity of this approach is likely to depend very much on the discipline associated with the data. Thus in molecular sciences, the basic object naturally maps to the molecule, since this is the smallest object for which a dataset can be normally be generated and which can usefully be described by its own metadata. It would be less useful or convenient, for example, to disassemble the molecule into individual atoms as metadata carriers.

			: Basing the repository design on collections also reflects the manner in which much modern science is conducted, often via multi-disciplinary collaborations in which each group can generate its own data collections. Collections also greatly facilitate data citation in journal articles. For example, the persistent identifier (DOI) of just the highest collection level of datasets associated with an article can be therein cited, avoiding citation blight. If a particular object (a molecule in our case) is being discussed in the text of the article, it might nevertheless be more appropriate to reference the specific DOI at that stage. Individual citation is also useful in, for example, tables of results or figures. The metadata for any individual cited dataset will also contain the attribute "is member of," so that the hierarchy can be both tracked upwards, and via the attribute "has members" downwards (Fig. 1d). This hierarchy also introduces via such metadata further semantics into the citation process itself; each item is placed into appropriate context. Lack of such semantics/context are arguably one of the most deficient aspects of current citation practices in journal articles.

			* Our approach to metadata collection is to automate the process whenever possible. In the case of a molecule as an object, there are algorithms which can be used to generate appropriate metadata, the most useful and prominent of which is the InChI (International Chemical Identifier).<ref name="InChI">{{cite web \|url=http://www.inchi-trust.org/ \|title=InChI Trust \|publisher=InChI Trust \|accessdate=07 September 2016}}</ref> The task of creating such an identifier is effectively accomplished using the OpenBabel program library<ref name="OBoyleOpen11">{{cite journal \|title=Open Babel: An open chemical toolbox \|journal=Journal of Cheminformatics \|author=O'Boyle, N.M.; Banck, M.; James, C.A. et al. \|volume=3 \|pages=33 \|year=2011 \|doi=10.1186/1758-2946-3-33 \|pmid=21982300 \|pmc=PMC3198950}}</ref> or via Javascript-based resources.<ref name="InChI.js">{{cite web \|url=https://metamolecular.com/inchi-js/ \|title=InChI for the Web Browser with InChI.js \|publisher=Metamolecular, LLC \|accessdate=07 September 2016}}</ref> These can accept as input a variety of chemical documents and generate an appropriate InChI identifier and InChI key uniquely describing them. The repository workflow automatically processes any uploaded data file through this algorithm and records all successful outputs. Such metadata is then associated with the Subject element in the DataCite schema (Fig. 1b).

	==References==		==References==

Shawndouglas: Created stub. Saving and adding more.

2017-02-07T21:41:09Z

Created stub. Saving and adding more.

New page

{{Infobox journal article
|name =
|image =
|alt = 
|caption =
|title_full = A metadata-driven approach to data repository design
|journal = ''Journal of Cheminformatics''
|authors = Harvey, Matthew J.; McLean, Andrew; Rzepa, Henry S.
|affiliations = Imperial College London
|contact = Email: rzepa at imperial dot ac dot uk
|editors =
|pub_year = 2017
|vol_iss = '''9'''
|pages = 4
|doi = [http://doi.org/10.1186/s13321-017-0190-6 10.1186/s13321-017-0190-6]
|issn = 1758-2946
|license = [http://creativecommons.org/licenses/by/4.0/ Creative Commons Attribution 4.0 International]
|website = [http://jcheminf.springeropen.com/articles/10.1186/s13321-017-0190-6 http://jcheminf.springeropen.com/articles/10.1186/s13321-017-0190-6]
|download = [http://jcheminf.springeropen.com/track/pdf/10.1186/s13321-017-0190-6?site=jcheminf.springeropen.com http://jcheminf.springeropen.com/track/pdf/10.1186/s13321-017-0190-6] (PDF)
}}
{{ombox
| type = content
| style = width: 500px;
| text = This article should not be considered complete until this message box has been removed. This is a work in progress.
}}
==Abstract==
The design and use of a metadata-driven data repository for research data management is described. Metadata is collected automatically during the submission process whenever possible and is registered with DataCite in accordance with their current metadata schema, in exchange for a persistent digital object identifier. Two examples of data preview are illustrated, including the demonstration of a method for integration with commercial software that confers rich domain-specific [[data analysis|data analytics]] without introducing customization into the repository itself.

'''Keywords''': Data repository, metadata-driven, DataCite, data preview, Mpublish

==Background==
Turnkey institutional repositories based on platforms such as [[DSpace]]<ref name="DSpace">{{cite web |url=http://www.dspace.org/ |title=DSpace |publisher=DuraSpace Organization |accessdate=07 September 2016}}</ref> were introduced more than 10 years ago, with early applications directed largely towards archival of publication preprints and postprints. The recent increasing requirement for research data management emerging from funding agencies means that the focus is now shifting to the use of repositories as part of the data management processes. More recent data-centric tools such as Figshare<ref name="Figshare">{{cite web |url=https://figshare.com/ |title=Figshare |publisher=Figshare LLP |accessdate=07 September 2016}}</ref> and Zenodo<ref name="Zenodo">{{cite web |url=https://zenodo.org/ |title=Zenodo |publisher=CERN Data Centre |accessdate=07 September 2016}}</ref> reflect these changes. Such services rely on the minting of persistent identifiers or DOIs for the depositions using the DataCite agency.<ref name="DataCite">{{cite web |url=https://www.datacite.org/ |title=DataCite |publisher=DataCite Association |accessdate=07 September 2016}}</ref> Metadata describing the deposited material is supplied to DataCite and a DOI is returned. An early example of such research data management is illustrated by a DSpace-based project to produce and, 10 years later, curate a library of quantum-mechanically-optimised molecular coordinates derived from a computable subset of the [[National Cancer Institute]]'s (NCI) collection of small molecules.<ref name="DowningSPECTRa08">{{cite journal |title=SPECTRa: The deposition and validation of primary chemistry research data in digital repositories |journal=Journal of Chemical Information and Modeling |author=Downing, J.; Murray-Rust, P.; Tonge, A.P. et al. |volume=48 |issue=8 |pages=1571–1581 |year=2008 |doi=10.1021/ci7004737}}</ref>

One feature of the curation phase<ref name="HarveyStandards15">{{cite journal |title=Standards-based curation of a decade-old digital repository dataset of molecular information |journal=Journal of Cheminformatics |author=Harvey, M.J.; Mason, N.J.; McLean, A. et al. |volume=7 |pages=43 |year=2015 |doi=10.1186/s13321-015-0093-3 |pmid=26322133 |pmc=PMC4550659}}</ref> of the project aimed to explore the capabilities of the DataCite metadata schemas to improve the discoverability of the deposited data. The metadata can then be exploited to create rich search queries.<ref name="RzepaInChl16">{{cite journal |title=InChI as a research data management tool |journal=Chemistry International |author=Rzepa, H.S.; Mclean, A.; Harvey, M.J. |volume=38 |issue=3–4 |pages=24–26 |year=2015 |doi=10.1515/ci-2016-3-408}}</ref> As a result of the experiences gained from this project, we became aware that one limiting factor to the effective use of metadata was the repository design itself. The next stage therefore was to explore whether what we considered the essential requirements for a data repository could be incorporated into a new design. Here we report the principles used to create such a repository and some of the applications in chemistry that have resulted. These principles may in turn assist researchers wishing to deposit data in identifying the repository attributes that can best expose the discoverability and re-use of their data.

==References==
{{Reflist|colwidth=30em}}

==Notes==
This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. In one case, the original citation was incomplete (#6) and was corrected here.


[[Category:LIMSwiki journal articles (added in 2017)‎]]
[[Category:LIMSwiki journal articles (all)‎]]
[[Category:LIMSwiki journal articles on big data‎‎]]
[[Category:LIMSwiki journal articles on informatics‎‎]]