Journal:Risk assessment for scientific data

From LIMSWiki
Revision as of 22:15, 14 December 2020 by Shawndouglas (talk | contribs) (Saving and adding more.)
Jump to navigationJump to search
Full article title Risk assessment for scientific data
Journal Data Science Journal
Author(s) Mayernik, Matthew S.; Breseman, Kelsey; Downs, Robert R.; Duerr, Ruth; Garretson, Alexis; Hou, Chung-Yi,
EDGI and ESIP Data Stewardship Committee[a]
Author affiliation(s) National Center for Atmospheric Research, Environmental Data & Governance Initiative, Columbia University,
Ronin Institute for Independent Scholarship, George Mason University
Primary contact Email: mayernik at ucar dot edu
Year published 2020
Volume and issue 19(1)
Article # 10
DOI 10.5334/dsj-2020-010
ISSN 1683-1470
Distribution license Creative Commons Attribution 4.0 International
Website https://datascience.codata.org/articles/10.5334/dsj-2020-010/
Download https://datascience.codata.org/articles/10.5334/dsj-2020-010/galley/944/download/ (PDF)
Emblem-important-yellow.svg This article should be considered a work in progress and incomplete. Consider this article incomplete until this notice is removed.

Abstract

Ongoing stewardship is required to keep data collections and archives in existence. Scientific data collections may face a range of risk factors that could hinder, constrain, or limit current or future data use. Identifying such risk factors to data use is a key step in preventing or minimizing data loss. This paper presents an analysis of data risk factors that scientific data collections may face, and a data risk assessment matrix to support data risk assessments to help ameliorate those risks. The goals of this work are to inform and enable effective data risk assessment by: a) individuals and organizations who manage data collections, and b) individuals and organizations who want to help to reduce the risks associated with data preservation and stewardship. The data risk assessment framework presented in this paper provides a platform from which risk assessments can begin, and a reference point for discussions of data stewardship resource allocations and priorities.

Keywords: risk assessment, data preservation, data stewardship, metadata

Introduction

At the “The Rescue of Data At Risk” workshop held in Boulder, Colorado on September 8 and 9, 2016[b], participants were asked the following question: “How would you define ‘at-risk’ data?” Discussions on this point ranged widely and touched on several challenges, including lack of funding or personnel support for data management, natural and political disasters, and metadata loss. One participant’s organization’s definition of risk, however, stood out: “data were considered to be at-risk unless they had a dedicated plan to not be at-risk.” This simple statement vividly depicts how data’s default state is being in a state of risk. In other words, ongoing stewardship is required to keep data collections and archives in existence.

The risk factors that a given data collection or archive may face vary, depending on the data’s characteristics, the data’s current environment, and the priorities and resources available at the time. Many risks can be reduced or eliminated by following best practices codified as certifications and guidelines, such as the CoreTrustSeal Data Repository Certification[1], as well as the ISO 16363:2012 standard, which defines audit and certification procedures for trustworthy digital repositories.[2] Both the CoreTrustSeal certification and ISO 16363:2012 are based on the ISO 14721:2012 standard that defines the reference model for an open archival information system (OAIS).[3] But these certifications can be large and complex. Additionally, many of the organizations that hold valuable scientific data collections may not be aware of these standards, even if the organizations are potentially resourced to tackle the challenge.[4] Further, the attainment of such certifications does not necessarily reduce the risks to data that are outside of the scope of a particular certification instrument.

This paper presents an analysis of data risk factors that stakeholders of scientific data collections and archives may face, and a matrix to support data risk assessments to help ameliorate those risks. The three driving questions for this analysis are:

  • How do stakeholders assess what data are at risk?
  • How do stakeholders characterize what risk factors data collections and/or archives face?
  • How do stakeholders make the associated risks more transparent, internally and/or externally?

The goals of this work are to inform and enable effective data risk assessment by: a) individuals and organizations who manage data collections, and b) individuals and organizations who want to help to reduce the risks associated with data preservation and stewardship. Stakeholders for these two activities include producers, stewards, sponsors, and users of data, as well as the management and staff of the institutions that employ them.

Background

This project was coordinated through the Data Stewardship Committee within the Earth Science Information Partners (ESIP), a non-profit organization that exists to support collection, stewardship, and use of earth science data, information, and knowledge.[c] The immediate motivation for the project stemmed from the Data Stewardship Committee members engaging with groups who were undertaking grass-roots “data rescue” initiatives after the 2016 U.S. presidential election. At that time, a number of loosely organized and coordinated efforts were initiated to duplicate data from U.S. government organizations to prevent potential politically motivated data deletion or obfuscation.[5][6] In many cases, these initiatives specifically focused on duplicating government-hosted earth science data.

ESIP Data Stewardship Committee members wrote a white paper to provide the earth science data centers’ perspective on these grassroots “data rescue” activities.[7] That document described essential considerations within the day-to-day work of existing federal and federally-funded earth science data archiving organizations, including data centers’ constant focus on documentation, traceability, and persistence of scientific data. The white paper also provided suggestions for how those grassroots efforts might productively engage with the data centers themselves.

One point that was emphasized in the white paper was that the actual risks faced by the data collections may not be transparent from the outside. In other words, “data rescue” activities may have in fact been duplicating data that were at minimal risk of being lost.[8] This point, and the white paper in general, was well received by people inside and outside of these grass-roots initiatives.[9][10] Questions then came back to the ESIP Data Stewardship Committee about how to understand what data held by government agencies were actually at-risk.

The analysis presented in this paper was initiated in response to these questions. Since then, these grassroots “data rescue” initiatives have had mixed success in sustaining and formalizing their efforts.[11][12][13]The intention of our paper is to enable more effective data risk assessment broadly. Rescuing data after they have been corrupted, deleted, or lost can be time- and effort-intensive, and in some cases it may be impossible.[14] Thus, we aim to provide guidelines to any individual or organization that manages and provides access to scientific data. In turn, these individuals and organizations can better assess the risks that their data face and characterize those risks.

When discussing risk and, in particular, data risk, it is useful to ask "what is the objective that is being challenged by the possible risk factors?" With regard to data, in general, discussions of risk might presume that “risks” threaten the current or future access to data by the potential data users. Currently, continuing public access to and use of scientific data is particularly relevant in light of recent open data and open science initiatives. In this regard, risks for scientific data include factors that could hinder, constrain, or limit current or future data use. Identifying such data use risk factors offers further analysis opportunities to prevent, mitigate, or eliminate the risks.

Data risk assessment

Risk assessment is a regular activity within many organizations. In a general sense, risk management plans are complementary to project management plans. (Cervone 2006) Organizational assessment of digital data and information collections is likewise not new. (Maemura, Moles & Becker 2017) The analysis presented in this paper builds on prior work in a number of areas: 1) research on data risks, 2) data rescue initiatives within government agencies and specific disciplines, 3) CODATA and RDA working groups and meetings, 4) trusted repository certifications, and 5) knowledge and experience of the ESIP Data Stewardship Committee members. Table 1 summarizes data risk factors that emerge from these knowledgebases. The list of risk factors shown in Table 1 is not meant to be exhaustive. Rather, it provides a useful illustration of the diverse ways in which data sets, collections, and archives might encounter risks to data usability and accessibility. The rest of this section details further key insights from the five areas of prior work noted above.

Table 1. Risk factors for scientific data collections
Risk factor Description
1. Lack of use Data are rarely accessed and dubbed "unwanted," thus getting thrown away.
2. Loss of funding for archive The whole archive loses its funding source.
3. Loss of funding for specific datasets Specific datasets lose their funding source.
4. Loss of knowledge around context or access Data owners lose individuals—e.g., due to retirement or death—who know how to access the data or know the metadata associated with these data that make the data useable to others.
5. Lack of documentation and metadata Data cannot be interpreted due to lack of contextual knowledge.
6. Data mislabeling Data are lost because they are poorly identified (either physically or digitally).
7. Catastrophes Fires, floods, wars, human conflicts, etc. destroy data and/or their owners.
8. Poor data governance Uncertain or unknown decision making processes impede effective data management.
9. Problems with legal status for data ownership and use Uncertain, unknown, or restrictive legal status limits the possible uses of data.
10. Media deterioration Physical media deterioration prevents data from being accessed (paper, tape, or digital media).
11. Missing files Data files are lost without any known reason.
12. Overdependence on a single service provider Problems arise from having a single point of failure, particularly if a vital service provider goes out of business.
13. Accidental deletion Data are accidentally deleted by staff error.
14. Lack of planning Lack of planning puts data collections at risk of being susceptible to unexpected events.
15. Cybersecurity breach Data are intentionally deleted or corrupted via a security breach, e.g., via malware.
16. Overabundant data Difficulty dealing with too much data results in a reduction in value or quality of whole collections.
17. Political interference Data is deleted or made inaccessible due to uncontrollable political decisions.
18. Lack of provenance information Data cannot be trusted or understood because of a lack of information about data processing steps, or about data stewardship chains of trust.
19. File format obsolescence Data cannot be accessed due to lack of knowledge, equipment, or software for reading a specific file format.
20. Storage hardware breakdown Data is lost due to a sudden and catastrophic malfunction of storage hardware.
21. Bit rot and data corruption Digital data on storage hardware gradually becomes corrupted due to an accumulation of non-critical failures (bits flipping) in a data storage device.

Research on data risks

A range of studies have explored the kinds of risks that scientific data may face, and potential ways to mitigate specific risk factors. Many of these studies touch on practices that are typical of scientific data archives. Metadata, for example, can be considered both a risk factor and a mitigation strategy. Insufficient metadata is itself a potential factor that can reduce the discoverability, usability, and preservability of data, particularly in situations where direct human knowledge of the data is absent.[15] In fact, many data rescue projects find that the “rescue” efforts must be targeted much more toward metadata than data.[16][17] This might be the case for a couple of reasons. First, insufficient or missing metadata might prevent data from being usable regardless of the condition of the data themselves. Examples include missing column headers in tabular data that prevent a user from knowing what the data are representing, and insufficient provenance metadata that prevent users from trusting the data due to lack of context about data collection and quality control. Second, metadata are also central to documenting and mitigating risks as they manifest, while preventing risks from becoming problematic in the future. (Anderson et al. 2011) For example, documenting data ownership and usage rights is an essential step in mitigating risk factor #9, “Problems with legal status for data ownership and use,” from Table 1.


Footnotes

  1. ↑ We list EDGI and the ESIP Data Stewardship Committee as authors due to the contributions of many individuals from both organizations to the work described in this paper. The named authors are the individuals involved in each organization who contributed directly to the paper’s text.
  2. ↑ The workshop was organized under the auspices of the Research Data Alliance (RDA) and the Committee on Data (CODATA) within the International Science Council.
  3. ↑ See https://wiki.esipfed.org/Preservation_and_Stewardship.

References

  1. ↑ CoreTrustSeal Standards and Certification Board (2020). "CoreTrustSeal". https://www.coretrustseal.org/. 
  2. ↑ "ISO 16363:2012 - Space data and information transfer systems — Audit and certification of trustworthy digital repositories". International Organization for Standardization. February 2012. https://www.iso.org/standard/56510.html. 
  3. ↑ "ISO 14721:2012 - Space data and information transfer systems — Open archival information system (OAIS) — Reference model". International Organization for Standardization. September 2012. https://www.iso.org/standard/56510.html. 
  4. ↑ Maemura, E.; Moles, N.; Becker, C. (2017). "Organizational assessment frameworks for digital preservation: A literature review and mapping". JASIST 68 (7): 1619–37. doi:10.1002/asi.23807. 
  5. ↑ Dennis, B. (13 December 2016). "Scientists are frantically copying U.S. climate data, fearing it might vanish under Trump". The Washington Post. https://www.washingtonpost.com/news/energy-environment/wp/2016/12/13/scientists-are-frantically-copying-u-s-climate-data-fearing-it-might-vanish-under-trump/. 
  6. ↑ Varinsky, D. (11 February 2017). "Scientists across the US are scrambling to save government research in 'Data Rescue' events". Business Insider. https://www.businessinsider.com/data-rescue-government-data-preservation-efforts-2017-2. 
  7. ↑ Mayernik, M.S.; Downs, R. R.; Duerr, R. et al. (4 April 2017). "Stronger together: The case for cross-sector collaboration in identifying and preserving at-risk data". FigShare. https://esip.figshare.com/articles/journal_contribution/Stronger_together_the_case_for_cross-sector_collaboration_in_identifying_and_preserving_at-risk_data/4816474/1. 
  8. ↑ Lamdan, S. (2018). "Lessons from DataRescue: The Limits of Grassroots Climate Change Data Preservation and the Need for Federal Records Law Reform". University of Pennsylvania Law Review Online 166 (1). https://scholarship.law.upenn.edu/penn_law_review_online/vol166/iss1/12. 
  9. ↑ Cornelius, K.B.; Pasquetto, I.V. (2018). "‘What Data?’ Records and Data Policy Coordination During Presidential Transitions". Proceedings from iConference 2018: Transforming Digital Worlds: 155–63. doi:10.1007/978-3-319-78105-1_20. 
  10. ↑ McGovern, N.Y. (2017). "Data rescue: Observations from an archivist". ACM SIGCAS Computers and Society 47 (2): 19–26. doi:10.1145/3112644.3112648. 
  11. ↑ Allen, L.; Stewart, C.; Wright, S. (2017). "Strategic open data preservation: Roles and opportunities for broader engagement by librarians and the public". College & Research Libraries News 78 (9): 482. doi:10.5860/crln.78.9.482. 
  12. ↑ Chodacki, J. (2017). "Data Mirror-Complementing Data Producers". Against the Grain 29 (6): 13. doi:10.7771/2380-176X.7877. 
  13. ↑ Janz, M.M. (2017). "Maintaining Access to Public Data: Lessons from Data Refuge". Against the Grain 29 (6): 11. doi:10.7771/2380-176X.7875. 
  14. ↑ Pienta, A.M.; Lyle, J. (2017). "Retirement in the 1950s: Rebuilding a Longitudinal Research Database". IASSIST Quarterly 42 (1): 12. doi:10.29173/iq19. 
  15. ↑ Kichener, W.K.; Brunt, J.W.; Helly, J.J. et al. (1997). "Nongeospatial metadata for the ecological sciences". Ecological Applications 7 (1): 330–42. doi:10.1890/1051-0761(1997)007[0330:NMFTES]2.0.CO;2. 
  16. ↑ Knapp, K.R.; Bates, J.J.; Barkstrom, B. et al. (2007). "Scientific Data Stewardship: Lessons Learned from a Satallite–Data Rescue Effort". Bulletin of the American Meteorological Society 88 (9): 1359–62. doi:10.1175/BAMS-88-9-1359. 
  17. ↑ Hsu, L.; Lehnert, K.A.; Goodwillie, A. et al. (2015). "Rescue of long-tail data from the ocean bottom to the Moon: IEDA Data Rescue Mini-Awards". GeoResJ 6: 108–114. doi:10.1016/j.grj.2015.02.012. 

Notes

This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article lists references in alphabetical order; however, this version lists them in order of appearance, by design.

Retrieved from "https://www.limswiki.org/index.php?title=Journal:Risk_assessment_for_scientific_data&oldid=41158"