Journal:Risk assessment for scientific data - Revision history

Shawndouglas: Fixed citation issue

2021-09-19T20:39:03Z

Fixed citation issue

← Older revision		Revision as of 20:39, 19 September 2021
Line 150:		Line 150:

	===Research on data risks===		===Research on data risks===
	A range of studies have explored the kinds of risks that scientific data may face, and potential ways to mitigate specific risk factors. Many of these studies touch on practices that are typical of scientific data archives. Metadata, for example, can be considered both a risk factor and a mitigation strategy. Insufficient metadata is itself a potential factor that can reduce the discoverability, usability, and preservability of data, particularly in situations where direct human knowledge of the data is absent.<ref name="~~MichenerNongeo97~~">{{cite journal \|title=Nongeospatial metadata for the ecological sciences \|journal=Ecological Applications \|author=Kichener, W.K.; Brunt, J.W.; Helly, J.J. et al. \|volume=7 \|issue=1 \|pages=330–42 \|year=1997 \|doi=10.1890/1051-0761(1997)007[0330:NMFTES]2.0.CO;2}}</ref> In fact, many data rescue projects find that the “rescue” efforts must be targeted much more toward metadata than data.<ref name="KnappScien07">{{cite journal \|title=Scientific Data Stewardship: Lessons Learned from a Satallite–Data Rescue Effort \|journal=Bulletin of the American Meteorological Society \|author=Knapp, K.R.; Bates, J.J.; Barkstrom, B. et al. \|volume=88 \|issue=9 \|pages=1359–62 \|year=2007 \|doi=10.1175/BAMS-88-9-1359}}</ref><ref name="HsuRescue15>{{cite journal \|title=Rescue of long-tail data from the ocean bottom to the Moon: IEDA Data Rescue Mini-Awards \|journal=GeoResJ \|author=Hsu, L.; Lehnert, K.A.; Goodwillie, A. et al. \|volume=6 \|pages=108–114 \|year=2015 \|doi=10.1016/j.grj.2015.02.012}}</ref> This might be the case for a couple of reasons. First, insufficient or missing metadata might prevent data from being usable regardless of the condition of the data themselves. Examples include missing column headers in tabular data that prevent a user from knowing what the data are representing, and insufficient provenance metadata that prevent users from trusting the data due to lack of context about data collection and [[quality control]]. Second, metadata are also central to documenting and mitigating risks as they manifest, while preventing risks from becoming problematic in the future.<ref name="AndersonMeta11">{{cite web \|url=https://repositories.lib.utexas.edu/handle/2152/20056 \|title=Metadata for Data Rescue and Data at Risk \|author=Anderson, W.L.; Faundeen, J.L.; Greenberg, J. et al. \|work=Texas ScholarWorks \|date=22 September 2011}}</ref> For example, documenting data ownership and usage rights is an essential step in mitigating risk factor #9, “Problems with legal status for data ownership and use,” from Table 1.		A range of studies have explored the kinds of risks that scientific data may face, and potential ways to mitigate specific risk factors. Many of these studies touch on practices that are typical of scientific data archives. Metadata, for example, can be considered both a risk factor and a mitigation strategy. Insufficient metadata is itself a potential factor that can reduce the discoverability, usability, and preservability of data, particularly in situations where direct human knowledge of the data is absent.<ref name="KichenerNongeo97">{{cite journal \|title=Nongeospatial metadata for the ecological sciences \|journal=Ecological Applications \|author=Kichener, W.K.; Brunt, J.W.; Helly, J.J. et al. \|volume=7 \|issue=1 \|pages=330–42 \|year=1997 \|doi=10.1890/1051-0761(1997)007[0330:NMFTES]2.0.CO;2}}</ref> In fact, many data rescue projects find that the “rescue” efforts must be targeted much more toward metadata than data.<ref name="KnappScien07">{{cite journal \|title=Scientific Data Stewardship: Lessons Learned from a Satallite–Data Rescue Effort \|journal=Bulletin of the American Meteorological Society \|author=Knapp, K.R.; Bates, J.J.; Barkstrom, B. et al. \|volume=88 \|issue=9 \|pages=1359–62 \|year=2007 \|doi=10.1175/BAMS-88-9-1359}}</ref><ref name="HsuRescue15>{{cite journal \|title=Rescue of long-tail data from the ocean bottom to the Moon: IEDA Data Rescue Mini-Awards \|journal=GeoResJ \|author=Hsu, L.; Lehnert, K.A.; Goodwillie, A. et al. \|volume=6 \|pages=108–114 \|year=2015 \|doi=10.1016/j.grj.2015.02.012}}</ref> This might be the case for a couple of reasons. First, insufficient or missing metadata might prevent data from being usable regardless of the condition of the data themselves. Examples include missing column headers in tabular data that prevent a user from knowing what the data are representing, and insufficient provenance metadata that prevent users from trusting the data due to lack of context about data collection and [[quality control]]. Second, metadata are also central to documenting and mitigating risks as they manifest, while preventing risks from becoming problematic in the future.<ref name="AndersonMeta11">{{cite web \|url=https://repositories.lib.utexas.edu/handle/2152/20056 \|title=Metadata for Data Rescue and Data at Risk \|author=Anderson, W.L.; Faundeen, J.L.; Greenberg, J. et al. \|work=Texas ScholarWorks \|date=22 September 2011}}</ref> For example, documenting data ownership and usage rights is an essential step in mitigating risk factor #9, “Problems with legal status for data ownership and use,” from Table 1.

	Different kinds of metadata might be necessary to reduce specific data risks. For example, specifications of file format structures are a critical type of metadata for mitigating risks associated with digital file format obsolescence. Open specifications complement other critical mitigation practices and tools related to file format obsolescence. As one example, keeping rendering software available is an important way to retain access to particular file formats, but this typically also requires maintaining documentation of how the rendering software works.<ref name="RyanOccam14">{{cite journal \|title=Occam’s Razor and File Format Endangerment Factors \|journal=Proceedings of the 11th International Conference on Digital Preservation \|author=Ryan, H. \|pages=179–88 \|year=2014 \|url=https://www.nla.gov.au/sites/default/files/ipres2014-proceedings-version_1.pdf \|format=PDF}}</ref>		Different kinds of metadata might be necessary to reduce specific data risks. For example, specifications of file format structures are a critical type of metadata for mitigating risks associated with digital file format obsolescence. Open specifications complement other critical mitigation practices and tools related to file format obsolescence. As one example, keeping rendering software available is an important way to retain access to particular file formats, but this typically also requires maintaining documentation of how the rendering software works.<ref name="RyanOccam14">{{cite journal \|title=Occam’s Razor and File Format Endangerment Factors \|journal=Proceedings of the 11th International Conference on Digital Preservation \|author=Ryan, H. \|pages=179–88 \|year=2014 \|url=https://www.nla.gov.au/sites/default/files/ipres2014-proceedings-version_1.pdf \|format=PDF}}</ref>

Shawndouglas: Finished adding rest of content.

2020-12-15T18:57:09Z

Finished adding rest of content.

Show changes

Shawndouglas: Saving and adding more.

2020-12-15T17:58:36Z

Saving and adding more.

← Older revision		Revision as of 17:58, 15 December 2020
Line 297:		Line 297:

	In summary, the matrix was very useful as “something to think with.” In other words, it jump-started the process for doing the risk assessment because the NCAR Library staff did not need to spend time developing a comprehensive list of risk factors that may apply for these data, or brainstorm about how to categorize those risks. The risk factor matrix provided a ready-made starting point for the assessment. Because the matrix does not dictate how the cells should be filled in, the NCAR Library staff made decisions about how to apply the matrix for each categorization method that was chosen. The matrix structure could potentially be applied or customized to create a prioritization rubric, by supporting the creation of a numeric scoring process for categories where that is appropriate.		In summary, the matrix was very useful as “something to think with.” In other words, it jump-started the process for doing the risk assessment because the NCAR Library staff did not need to spend time developing a comprehensive list of risk factors that may apply for these data, or brainstorm about how to categorize those risks. The risk factor matrix provided a ready-made starting point for the assessment. Because the matrix does not dictate how the cells should be filled in, the NCAR Library staff made decisions about how to apply the matrix for each categorization method that was chosen. The matrix structure could potentially be applied or customized to create a prioritization rubric, by supporting the creation of a numeric scoring process for categories where that is appropriate.

			===Case 2 – Mohonk Preserve Daniel Smiley Research Library===
			Mohonk Preserve is a land trust and nature preserve in New Paltz, New York covering more than 8,000 acres of a northern section of the Appalachian Mountains known as the Shawangunk Mountains. Mohonk Preserve’s conservation science division, the Daniel Smiley Research Center (DSCR), is affiliated with the Organization of Biological Field Stations (OBFS) and acts as a NOAA Climate Observation Center. DSRC staff and citizen scientists carry out a variety of long-term monitoring projects and manage an extensive archive of historical observations. The archive houses 60,000 physical items, 9,000 photographs, 86 years of natural history observations, 123 years of daily weather data, and a research library of legacy titles. The physical items include more than 3,000 herbarium specimens, 107 bird specimens, 140 butterfly specimens, 139 mammal specimens, 400 arthropod specimens, and over 14,000 index cards with handwritten and typed observations. The digitization process of the archive holdings is ongoing, but the packaging and publishing of datasets in the Environmental Data Initiative is a priority.<ref name="MohonkHistory18">{{cite web \|url=https://portal.edirepository.org/nis/mapbrowse?packageid=edi.225.2 \|title=History of Acid Precipitation on the Shawangunk Ridge: Mohonk Preserve Precipitation Depths and pH, 1976 to Present \|author=Mohonk Preserve; Belardo, C.; Feldsine, N. et al. \|work=Environmental Data Initiative Data Portal \|date=02 August 2018 \|doi=10.6073/pasta/734ea90749e78613452eacec489f419c}}</ref><ref name="MohonkMohonk18">{{cite web \|url=https://portal.edirepository.org/nis/mapbrowse?packageid=edi.230.1 \|title=Mohonk Preserve Ground Water Springs Data, 1991 to Present \|author=Mohonk Preserve; Forester, A.; Huth, P. et al. \|work=Environmental Data Initiative Data Portal \|date=02 August 2018 \|doi=10.6073/pasta/928feed7ee748509ab065de7e3791966}}</ref><ref name="MohonkMohonkPres19">{{cite web \|url=https://portal.edirepository.org/nis/mapbrowse?packageid=edi.398.1 \|title=Mohonk Preserve Amphibian and Water Quality Monitoring Dataset at 11 Vernal Pools from 1931-Present \|author=Mohonk Preserve; Feldsine, N.; Forester, A. et al. \|work=Environmental Data Initiative Data Portal \|date=10 July 2019 \|doi=10.6073/pasta/864aea25998b73c5d1a5b5f36cb6583e}}</ref> These data and natural history collections underpin the Mohonk Preserve’s land management and stewardship and have been crucial to an increasing number of scientific publications<ref name="CookACross08">{{cite journal \|title=A cross‐taxa phenological dataset from Mohonk Lake, NY and its relationship to climate \|journal=International Journal of Climatology \|author=Cook, B.I.; Cook, E.R.; Huth, P.C. et al. \|volume=28 \|issue=10 \|pages=1369-1383 \|year=2008 \|doi=10.1002/joc.1629}}</ref><ref name="CookAHomo10">{{cite journal \|title=A Homogeneous Record (1896–2006) of Daily Weather and Climate at Mohonk Lake, New York \|journal=Journal of Applied Meteorology and Climatology \|author=Cook, B.I.; Cook, E.R.; Anchukaitis, K.J. et al. \|volume=49 \|issue=3 \|pages=544–555 \|year=2010 \|doi=10.1175/2009JAMC2221.1}}</ref><ref name="CharifsonHist15">{{cite journal \|title=History of Fish Presence and Absence Following Lake Acidification and Recovery in Lake Minnewaska, Shawangunk Ridge, NY \|journal=Northeastern Naturalist \|author=Charifson, D.M.; Huth, P.C.; Thompson, J.E. et al. \|volume=22 \|issue=4 \|pages=762-781 \|year=2015 \|doi=10.1656/045.022.0411}}</ref><ref name="RichardsonReconst17">{{cite journal \|title=Reconstructing a trophic cascade following unintentional introduction of golden shiner to Lake Minnewaska, New York, USA \|journal=Inland Waters \|author=Richardson, D.C.; Charifson, D.M.; Stanson, V.J. et al. \|volume=6 \|issue=1 \|pages=29–33 \|year=2017 \|doi=10.5268/IW-6.1.915}}</ref>, but the collections remain underutilized.

Shawndouglas: Saving and adding more.

2020-12-15T17:22:57Z

Saving and adding more.

← Older revision		Revision as of 17:22, 15 December 2020
Line 185:		Line 185:
	Through subsequent discussion and analysis of the data risk assessment literature noted above, we identified at least ten different ways that data risk factors could be assessed. Many of these categorization methods are applicable to risk assessments of any kind.<ref name="CervoneProject06">{{cite journal \|title=Project risk management \|journal=OCLC Systems & Services: International digital library perspectives \|author=Cervone, H.F. \|volume=22 \|issue=4 \|pages=256-262 \|year=2006 \|doi=10.1108/10650750610706970}}</ref> The list below is not meant to be exhaustive, and some methods are likely related. Data risk factors could be categorized or prioritized according to the methods listed in Table 2.		Through subsequent discussion and analysis of the data risk assessment literature noted above, we identified at least ten different ways that data risk factors could be assessed. Many of these categorization methods are applicable to risk assessments of any kind.<ref name="CervoneProject06">{{cite journal \|title=Project risk management \|journal=OCLC Systems & Services: International digital library perspectives \|author=Cervone, H.F. \|volume=22 \|issue=4 \|pages=256-262 \|year=2006 \|doi=10.1108/10650750610706970}}</ref> The list below is not meant to be exhaustive, and some methods are likely related. Data risk factors could be categorized or prioritized according to the methods listed in Table 2.

			{\|
			\| STYLE="vertical-align:top;"\|
			{\| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="80%"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;" colspan="2"\|'''Table 2.''' Methods for categorizing data risks
			\|-
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Categorization method
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Description
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Severity of risk
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|How much impact could this risk factor have on the data itself, regardless of the current importance of data to the user?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Likelihood of occurrence
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|How likely is the risk factor to occur?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Length of recovery time
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|How long would it take to recover data or re-establish data accessibility?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Impact on user
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|How significantly would data users be impacted by data loss or loss of data accessibility?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Who is responsible for addressing the problem
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Who has the expertise and responsibility to mitigate or respond to particular risk factors?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Cause of problem
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|What caused a data risk factor to occur?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Degree of control
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|How much control does an organization or individual have over whether a risk factor is present or will occur?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Proactive vs. reactive response
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Should risk factors be mitigated via preventative measures, or should they be responded to upon occurrence?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Nature of mitigation
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|What steps must be taken or processes put in place to prevent a risk, or mitigate a risk after it has occurred?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Resources required for mitigation
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|What time, money, or personnel resources will be necessary to mitigate risk factors?
			\|-
			\|}
			\|}

			The lists shown in Tables 1 and 2 offer characteristics on which data risk assessments can be built. Combining the categorization methods from Table 2 with the selected risk factors from Table 1 leads to a risk assessment matrix, as shown in Table 3. This figure shows an example of a selection of specific data risk factors and the categorization methods. Depending on the situation or data collection being assessed, different risk factors and/or categorization methods may be more applicable than the ones shown in Table 3. Those conducting a data risk assessment can then use the matrix as a way to organize, prioritize, or potentially quantify the selected risks according to the categorization methods that are most relevant for the specific case at hand. (The next section provides more detailed illustrations of the use of the data risk assessment matrix. Appendix I, at the end of this article, shows the full data risk assessment template, with all risks and categorization methods from Tables 1 and 2.)

			{\|
			\| STYLE="vertical-align:top;"\|
			{\| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="100%"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;" colspan="5"\|'''Table 3.''' Example of a blank data risk assessment matrix, after selection of specific risk factors and categorization methods of interest
			\|-
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" rowspan="2"\|''Risk factors''
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" colspan="4"\|''Categorization methods''
			\|-
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Severity of risk
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Likelihood of occurrence
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Cause of problem
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Required mitigation resources
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of use
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Loss of knowledge
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of docs and metadata
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Catastrophes
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Poor data governance
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Media deterioration
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\|-
			\|}
			\|}

			==Application of the data risk assessment matrix==
			Three case studies are described below, in which the data risk assessment matrix was used to develop a better understanding of data risks for particular resources. These cases enable evaluation of the data risk assessment framework presented in this paper, clarifying its strengths and weaknesses, and pinpointing the situations in which it can be most useful.<ref name="BeckerTheDesign20">{{cite journal \|title=The Design and Use of Assessment Frameworks in Digital Curation \|journal=JASIST \|author=Becker, C.; Maemura, E.; Moles. N. \|volume=71 \|issue=1 \|pages=55–68 \|year=2020 \|doi=10.1002/asi.24209}}</ref>

			===Case 1 – NCAR Library analog data collection===
			The National Center for Atmospheric Research (NCAR) Library maintains an analog data collection that consists of about 300 datasets in support of atmospheric and meteorological research conducted by NCAR scientists. These assets are largely compilations of measurements and statistics published by national and international meteorological services and other kinds of government entities. Many of these assets have been in the NCAR Library’s collections for decades, and most were minimally cataloged when they were first brought into the collection. As such, the current usage of the collection is minimal. A prior assessment done by the NCAR Library and a student assistant sought to identify individual assets that were of higher potential value and interest for current science. This assessment effort resulted in a modernization prioritization based on a geographic and temporal framework, and improved metadata records for about five percent of the collection.<ref name="MayernikModern18">{{cite journal \|title=Modernizing Library Metadata for Historical Weather and Climate Data Collections \|journal=Journal of Library Metadata \|author=Mayernick, M.S.; Huddle, J.; Hou, C.-Y. et al. \|volume=17 \|issue=3–4 \|pages=219–39 \|year=2017 \|doi=10.1080/19386389.2018.1440927}}</ref> This effort did not, however, include any kind of risk assessment related to the physical assets themselves.

			The data risk assessment matrix was therefore helpful in conducting a second-level priority analysis for these NCAR Library analog data assets. We used the matrix as a way to identify which risk factors were most important for these materials, and to characterize the mitigation efforts that were needed for each risk factor. In particular, we focused the risk assessment on the data assets that were previously identified as having high geospatial and temporal interest. The NCAR Library use of the matrix involved a series of steps:

			:''Step 1'' – A number of risk factors listed in the matrix were identified as being of most importance, with the focus being on factors that prevented or impeded the use of these data within current scientific studies. The most immediate risk factors were identified to be the “lack of use” and the “lack of documentation/metadata” for these assets. Other risks that were secondary in immediacy, but still potentially important, were data mislabeling, the questionable legal status for ownership and use, media deterioration, lack of planning, and poor data governance.

			:''Step 2'' – The second step was to identify which categorization methods shown in the matrix were most applicable/appropriate for the NCAR Library’s management and maintenance of this collection. The methods selected were: a) length of recovery time, b) who is responsible for addressing the problem, c) nature of mitigation, and d) resources required for mitigation.

			:''Step 3'' – The third step was to fill in the boxes in the matrix for the risk factors and categorization methods. For example, for the “length of recovery time” question, we used a simple 1–3 scale to indicate relative differences in how long it would take to mitigate the two most important risk factors: “lack of use” and the “lack of documentation/metadata.” As one example, some data assets were published by international agencies and therefore have title pages and documentation that are not in English. In turn, due to the lack of relevant foreign language expertise in the NCAR Library staff, developing new metadata for these resources will take more effort than for those assets that were published by English-speaking countries. For the “resources required for mitigation” categorization method, a numerical scale was not as appropriate. Instead, we filled in the matrix with text descriptions of the resources required to mitigate the risk factors. An example entry under the “lack of documentation and metadata” risk factor was: “We would need to create new metadata for the library catalog, then transform to ISO for inclusion in NCAR DASH Search, with the added challenge of needing to look at microfilm files (no current working reader in Library).”

			In summary, the matrix was very useful as “something to think with.” In other words, it jump-started the process for doing the risk assessment because the NCAR Library staff did not need to spend time developing a comprehensive list of risk factors that may apply for these data, or brainstorm about how to categorize those risks. The risk factor matrix provided a ready-made starting point for the assessment. Because the matrix does not dictate how the cells should be filled in, the NCAR Library staff made decisions about how to apply the matrix for each categorization method that was chosen. The matrix structure could potentially be applied or customized to create a prioritization rubric, by supporting the creation of a numeric scoring process for categories where that is appropriate.

Shawndouglas: Saving and adding more.

2020-12-15T16:56:04Z

Saving and adding more.

← Older revision		Revision as of 16:56, 15 December 2020
Line 177:		Line 177:

	==Developing a data risk assessment matrix==		==Developing a data risk assessment matrix==
			Risk assessment is a well-established field, with 30 to 40 years of history.<ref name="NRCRisk83">{{cite book \|url=https://www.nap.edu/catalog/366/risk-assessment-in-the-federal-government-managing-the-process \|title=Risk Assessment in the Federal Government: Managing the Process \|author=National Research Council \|publisher=National Academies Press \|year=1983 \|isbn=9780309033497 \|doi=10.17226/366}}</ref><ref name="AvenRisk16">{{cite journal \|title=Risk assessment and risk management: Review of recent advances on their foundation \|journal=European Journal of Operational Research \|author=Aven, T. \|volume=253 \|issue=1 \|pages=1–13 \|year=2016 \|doi=10.1016/j.ejor.2015.12.023}}</ref> However, the practice of applying risk assessment methodologies to scientific data collections is less formally established, though regular audits and reviews of data management systems are common in some organizations.<ref name="RamapriyanNASA17">{{cite web \|url=https://esip.figshare.com/articles/presentation/NASA_s_EOSDIS_Trust_and_Certification/5258047/1 \|title=NASA's EOSDIS, Trust and Certification \|author=Ramapriyan, H.K. \|work=FigShare \|date=29 July 2017}}</ref>

			The starting point for this project was to establish a process for categorizing the data risk factors shown in Table 1. The initial idea of our effort was that if data risk factors could be categorized into a logical structure, it would allow data managers to assess the risks to their data collections via a set of predefined and consistent categories. To develop a logical categorization, we held a session to conduct a “card sorting” exercise at the 2018 ESIP Summer Meeting, which took place in July 2018 in Tucson, Arizona. “Card sorting” is an established method for developing categorizations of concepts, vocabulary terms, or web sites.<ref name="ZimmermanAGroup02">{{cite journal \|title=A group card sorting methodology for developing informational Web sites \|journal=Proceedings of the 2002 IEEE International Professional Communication Conference \|author=Zimmerman, D.E.; Akerelrea, C. \|pages=437-445 \|year=2002 \|doi=10.1109/IPCC.2002.1049127}}</ref><ref name="GSACard19">{{cite web \|url=https://www.usability.gov/how-to-and-tools/methods/card-sorting.html \|title=Card Sorting \|work=Usability.gov \|publisher=U.S. General Services Administration \|date=2019}}</ref> Following the card sorting methodology, participants in the 2018 ESIP meeting session were provided the list of data risks in Table 1 and asked to complete the following task: “Looking at the list of data risk factors, how would you group these factors, based on the categories you would define?”

			Approximately 15 attendees engaged in the exercise. We used a combination of an online card sorting tool and hand-written recommendations to collect the completed card sorting categorizations. Following the completion of the exercise, the results were displayed in front of the session participants and a group discussion took place. The outcome of the card sorting exercise and subsequent discussion was a clear recognition that there could be many valid and useful ways of categorizing data risks. No single method for categorizing the risk factors would be sufficient to cover the diverse organizations and situations within which data collections exist. Depending on the situation(s) a data curation organization or individual is facing, they may need to categorize data risks in different ways. This characteristic is common in risk assessments generally, as risk prioritization and categorizations are dependent on the phenomena being assessed, the characteristics of the situation, and the goals of the organizations or people performing the assessment.<ref name="SlovicTrust99">{{cite journal \|title=Trust, emotion, sex, politics, and science: Surveying the risk-assessment battlefield \|journal=Risk Analysis \|author=Slovic, P. \|volume=19 \|pages=689–701 \|year=1999 \|doi=10.1023/A:1007041821623}}</ref>

			Through subsequent discussion and analysis of the data risk assessment literature noted above, we identified at least ten different ways that data risk factors could be assessed. Many of these categorization methods are applicable to risk assessments of any kind.<ref name="CervoneProject06">{{cite journal \|title=Project risk management \|journal=OCLC Systems & Services: International digital library perspectives \|author=Cervone, H.F. \|volume=22 \|issue=4 \|pages=256-262 \|year=2006 \|doi=10.1108/10650750610706970}}</ref> The list below is not meant to be exhaustive, and some methods are likely related. Data risk factors could be categorized or prioritized according to the methods listed in Table 2.

Shawndouglas: Saving and adding more.

2020-12-15T00:01:00Z

Saving and adding more.

← Older revision		Revision as of 00:01, 15 December 2020
Line 164:		Line 164:

	Mitigating risks, of whatever kind, takes effort and resources. The time required to create metadata, re-format files, create contingency plans, and communicate these efforts to user communities can be considerable. This time investment can be the greatest barrier to performing risk assessment and mitigation activities.<ref name="ThompsonWhere14">{{cite journal \|title=Where Have All the Scientific Data Gone? LIS Perspective on the Data-At-Risk Predicament \|journal=College & Research Libraries \|author=Thompson, C.A.; Robertson, D.; Greenberg, J. \|volume=75 \|issue=6 \|pages=842-861 \|year=2014 \|doi=10.5860/crl.75.6.842}}</ref> Putting focus on assessment of data risk factors may mean that “certain priorities need to be re-ordered, new skills acquired and taught, resources redirected, and new networks constructed.”<ref name="GriffinWhen15">{{cite journal \|title=When are Old Data New Data? \|journal=GeoResJ \|author=Griffin, R.E.; CODATA Task Group ‘Data At Risk’ (DAR-TG) \|volume=6 \|pages=92–97 \|year=2015 \|doi=10.1016/j.grj.2015.02.004}}</ref> It can be possible to automate some components of risk assessment<ref name="GrafADec16">{{cite journal \|title=A Decision Support System to Facilitate File Format Selection for Digital Preservation \|journal=Libellarium \|author=Graf, R.; Ryan, H.M.; Houzanme, T. et al. \|volume=9 \|issue=2 \|pages=267–74 \|year=2016 \|doi=10.15291/libellarium.v9i2.274}}</ref>, but most of the steps require human effort. This intensive effort is vividly illustrated by the many data rescue initiatives that have taken place within government agencies and other kinds of organizations over the past few decades.		Mitigating risks, of whatever kind, takes effort and resources. The time required to create metadata, re-format files, create contingency plans, and communicate these efforts to user communities can be considerable. This time investment can be the greatest barrier to performing risk assessment and mitigation activities.<ref name="ThompsonWhere14">{{cite journal \|title=Where Have All the Scientific Data Gone? LIS Perspective on the Data-At-Risk Predicament \|journal=College & Research Libraries \|author=Thompson, C.A.; Robertson, D.; Greenberg, J. \|volume=75 \|issue=6 \|pages=842-861 \|year=2014 \|doi=10.5860/crl.75.6.842}}</ref> Putting focus on assessment of data risk factors may mean that “certain priorities need to be re-ordered, new skills acquired and taught, resources redirected, and new networks constructed.”<ref name="GriffinWhen15">{{cite journal \|title=When are Old Data New Data? \|journal=GeoResJ \|author=Griffin, R.E.; CODATA Task Group ‘Data At Risk’ (DAR-TG) \|volume=6 \|pages=92–97 \|year=2015 \|doi=10.1016/j.grj.2015.02.004}}</ref> It can be possible to automate some components of risk assessment<ref name="GrafADec16">{{cite journal \|title=A Decision Support System to Facilitate File Format Selection for Digital Preservation \|journal=Libellarium \|author=Graf, R.; Ryan, H.M.; Houzanme, T. et al. \|volume=9 \|issue=2 \|pages=267–74 \|year=2016 \|doi=10.15291/libellarium.v9i2.274}}</ref>, but most of the steps require human effort. This intensive effort is vividly illustrated by the many data rescue initiatives that have taken place within government agencies and other kinds of organizations over the past few decades.

			===Data rescue initiatives within government agencies and specific disciplines===
			Legacy data are data collected in the past with different technologies and data formats than those used today. These data often face the largest numbers of risk factors that could lead to data loss. A wide range of government agencies and other organizations have conducted legacy data rescue initiatives to modernize data and make them more accessible and usable for today’s science. Each data rescue project typically faces many different kinds of data risks. For example, a recent satellite data rescue effort had to address the “loss of datasets, reconciliation of actual media contents with metadata available, deviation of the actual data format from expectations or documentation, and retiring expertise.”<ref name="PoliRecent17">{{cite journal \|title=Recent Advances in Satellite Data Rescue \|journal=Bulletin of the American Meteorological Society \|author=Poli, P.; Dee, D.P.; Saunders, R. et al. \|volume=98 \|issue=7 \|pages=1471–1484 \|year=2017 \|doi=10.1175/BAMS-D-15-00194.1}}</ref> Data rescue projects typically involve work to prevent future risk factors from manifesting, in addition to modernizing data for accessibility and usability. For example, data rescue projects migrate data to less endangered data formats, and create new metadata and quality control documentation.<ref name="LevitusTheUNESCO12">{{cite journal \|title=The UNESCO-IOC-IODE "Global Oceanographic Data Archeology and Rescue" (GODAR) Project and "World Ocean Database" Project \|journal=Data Science Journal \|author=Levitus, S. \|volume=11 \|pages=46–71 \|year=2012 \|doi=10.2481/dsj.012-014}}</ref>

			===CODATA/RDA working groups & meetings===
			Relevant professional organizations, including the International Council for Science (ICSU) Committee on Data for Science and Technology (CODATA) and the Research Data Alliance (RDA), also have been actively identifying improvements for data stewardship practices that can reduce potential risks to data. For example, the former Data At Risk Task Group (DAR-TG), of CODATA, raised awareness about the value of heritage data and described the benefits obtained from several data rescue projects.<ref name="GriffinWhen15" /> This group also organized the 2016 “Rescue of Data At Risk” workshop mentioned in the introduction of this paper. That workshop led to a document titled “Guidelines to the Rescue of Data At Risk.”<ref name="RDAGuide17">{{cite web \|url=https://www.rd-alliance.org/guidelines-rescue-data-risk \|title=Guidelines to the Rescue of Data At Risk \|author=Research Data Alliance \|date=24 March 2017}}</ref> Subsequently, the Data Rescue Interest Group<ref name="RDAData19">{{cite web \|url=https://rd-alliance.org/groups/data-rescue.html \|title=Data Rescue IG \|author=Research Data Alliance \|date=14 August 2019}}</ref> of the Research Data Alliance (RDA), spawned from the CODATA DAR-TG, also focuses on efforts to increase awareness of data rescue projects.

			===Repository certifications and maturity assessment===
			Many data repositories have conducted self-assessments and external assessments to document their compliance with the standards for trusted repositories and attain certification of their capabilities and practices for managing data. In addition to emphasizing organizational issues, repository certification instruments, such as ISO 16363<ref name="ISO16363_12" /> and CoreTrustSeal<ref name="CTSCore" /> certification, also focus on digital object management and infrastructure capabilities. Engaging in such assessments offers benefits to repositories and their stakeholders. A key benefit is the identification of areas where improvements have been completed or need to be completed to reduce risks to data.<ref name="CTSCore" /> In an examination of perceptions of repository certification, Donaldson ''et al.''<ref name="PoliRecent17">{{cite journal \|title=The Perceived Value of Acquiring Data Seals of Approval \|journal=International Journal of Digital Curation \|author=Donaldson, D.R.; Dillo, I.; Downs, R. et al. \|volume=12 \|issue=1 \|pages=130–51 \|year=2017 \|doi=10.2218/ijdc.v12i1.481}}</ref> found that process improvement was often reported by repository staff as a benefit of repository certification.

			In addition to (or complementary to) formal certifications, data repositories may conduct data stewardship maturity assessment exercises to help in identifying data risks and informing data risk mitigation strategies.<ref name="FaundeenDevelop17">{{cite journal \|title=Developing Criteria to Establish Trusted Digital Repositories \|journal=Data Science Journal \|author=Faundeen, J. \|volume=16 \|at=22 \|year=2017 \|doi=10.5334/dsj-2017-022}}</ref> “Maturity” is used in the sense presented by Peng ''et al.''<ref name="PengAUnif15">{{cite journal \|title=A Unified Framework for Measuring Stewardship Practices Applied to Digital Environmental Datasets \|journal=Data Science Journal \|author=Peng, G.; Privette, J.L.; Kearns, E.J. et al. \|volume=13 \|pages=231–53 \|year=2015 \|doi=10.2481/dsj.14-049}}</ref>, refering to the level of performance attained to ensure preservability, accessibility, usability, transparency/traceability, and sustainability of data, along with the level of performance in data quality assurance, data quality control/monitoring, data quality assessment, and data integrity checks. Maturity at the institutional (or archive) level in areas such as policy, funding, and infrastructure does not necessarily translate to comprehensive maturity at the dataset level.<ref name="PengTheState18">{{cite journal \|title=The State of Assessing Data Stewardship Maturity – An Overview \|journal=Data Science Journal \|author=Peng, G. \|volume=17 \|at=7 \|year=2018 \|doi=10.5334/dsj-2018-007}}</ref> Data stewardship maturity assessment should therefore be performed both at the institutional level and at the dataset level. It is recognized that performing stewardship maturity assessments can be time-consuming and resource-intensive. However, the stewardship organizations are encouraged to perform self-assessment using a “stage by stage” or “a la carte” approach.<ref name="PengPractic19">{{cite journal \|title=Practical Application of a Data Stewardship Maturity Matrix for the NOAA OneStop Project \|journal=Data Science Journal \|author=Peng, G.; Milan, A.; Ritchey, N.A. et al. \|volume=18 \|issue=1 \|at=41 \|year=2019 \|doi=10.5334/dsj-2019-041}}</ref> Ultimately, both formal certifications and informal maturity assessments help organizations not only gain self-awareness, but also identify better solutions for their data that might be at risk of being lost or rendered unusable.

			==Developing a data risk assessment matrix==




	==Footnotes==		==Footnotes==

Shawndouglas: Saving and adding more.

2020-12-14T23:31:11Z

Saving and adding more.

← Older revision		Revision as of 23:31, 14 December 2020
Line 157:		Line 157:
	A range of studies have explored the kinds of risks that scientific data may face, and potential ways to mitigate specific risk factors. Many of these studies touch on practices that are typical of scientific data archives. Metadata, for example, can be considered both a risk factor and a mitigation strategy. Insufficient metadata is itself a potential factor that can reduce the discoverability, usability, and preservability of data, particularly in situations where direct human knowledge of the data is absent.<ref name="MichenerNongeo97">{{cite journal \|title=Nongeospatial metadata for the ecological sciences \|journal=Ecological Applications \|author=Kichener, W.K.; Brunt, J.W.; Helly, J.J. et al. \|volume=7 \|issue=1 \|pages=330–42 \|year=1997 \|doi=10.1890/1051-0761(1997)007[0330:NMFTES]2.0.CO;2}}</ref> In fact, many data rescue projects find that the “rescue” efforts must be targeted much more toward metadata than data.<ref name="KnappScien07">{{cite journal \|title=Scientific Data Stewardship: Lessons Learned from a Satallite–Data Rescue Effort \|journal=Bulletin of the American Meteorological Society \|author=Knapp, K.R.; Bates, J.J.; Barkstrom, B. et al. \|volume=88 \|issue=9 \|pages=1359–62 \|year=2007 \|doi=10.1175/BAMS-88-9-1359}}</ref><ref name="HsuRescue15>{{cite journal \|title=Rescue of long-tail data from the ocean bottom to the Moon: IEDA Data Rescue Mini-Awards \|journal=GeoResJ \|author=Hsu, L.; Lehnert, K.A.; Goodwillie, A. et al. \|volume=6 \|pages=108–114 \|year=2015 \|doi=10.1016/j.grj.2015.02.012}}</ref> This might be the case for a couple of reasons. First, insufficient or missing metadata might prevent data from being usable regardless of the condition of the data themselves. Examples include missing column headers in tabular data that prevent a user from knowing what the data are representing, and insufficient provenance metadata that prevent users from trusting the data due to lack of context about data collection and [[quality control]]. Second, metadata are also central to documenting and mitigating risks as they manifest, while preventing risks from becoming problematic in the future. (Anderson et al. 2011) For example, documenting data ownership and usage rights is an essential step in mitigating risk factor #9, “Problems with legal status for data ownership and use,” from Table 1.		A range of studies have explored the kinds of risks that scientific data may face, and potential ways to mitigate specific risk factors. Many of these studies touch on practices that are typical of scientific data archives. Metadata, for example, can be considered both a risk factor and a mitigation strategy. Insufficient metadata is itself a potential factor that can reduce the discoverability, usability, and preservability of data, particularly in situations where direct human knowledge of the data is absent.<ref name="MichenerNongeo97">{{cite journal \|title=Nongeospatial metadata for the ecological sciences \|journal=Ecological Applications \|author=Kichener, W.K.; Brunt, J.W.; Helly, J.J. et al. \|volume=7 \|issue=1 \|pages=330–42 \|year=1997 \|doi=10.1890/1051-0761(1997)007[0330:NMFTES]2.0.CO;2}}</ref> In fact, many data rescue projects find that the “rescue” efforts must be targeted much more toward metadata than data.<ref name="KnappScien07">{{cite journal \|title=Scientific Data Stewardship: Lessons Learned from a Satallite–Data Rescue Effort \|journal=Bulletin of the American Meteorological Society \|author=Knapp, K.R.; Bates, J.J.; Barkstrom, B. et al. \|volume=88 \|issue=9 \|pages=1359–62 \|year=2007 \|doi=10.1175/BAMS-88-9-1359}}</ref><ref name="HsuRescue15>{{cite journal \|title=Rescue of long-tail data from the ocean bottom to the Moon: IEDA Data Rescue Mini-Awards \|journal=GeoResJ \|author=Hsu, L.; Lehnert, K.A.; Goodwillie, A. et al. \|volume=6 \|pages=108–114 \|year=2015 \|doi=10.1016/j.grj.2015.02.012}}</ref> This might be the case for a couple of reasons. First, insufficient or missing metadata might prevent data from being usable regardless of the condition of the data themselves. Examples include missing column headers in tabular data that prevent a user from knowing what the data are representing, and insufficient provenance metadata that prevent users from trusting the data due to lack of context about data collection and [[quality control]]. Second, metadata are also central to documenting and mitigating risks as they manifest, while preventing risks from becoming problematic in the future. (Anderson et al. 2011) For example, documenting data ownership and usage rights is an essential step in mitigating risk factor #9, “Problems with legal status for data ownership and use,” from Table 1.

			Different kinds of metadata might be necessary to reduce specific data risks. For example, specifications of file format structures are a critical type of metadata for mitigating risks associated with digital file format obsolescence. Open specifications complement other critical mitigation practices and tools related to file format obsolescence. As one example, keeping rendering software available is an important way to retain access to particular file formats, but this typically also requires maintaining documentation of how the rendering software works.<ref name="RyanOccam14">{{cite journal \|title=Occam’s Razor and File Format Endangerment Factors \|journal=Proceedings of the 11th International Conference on Digital Preservation \|author=Ryan, H. \|pages=179–88 \|year=2014 \|url=https://www.nla.gov.au/sites/default/files/ipres2014-proceedings-version_1.pdf \|format=PDF}}</ref>

			Other risk factors (listed in Table 1) relate to the sustainability and transparency of the archiving organization. These factors are important in ensuring the accessibility of the data and the trustworthiness of the archive. As Yakel ''et al.''<ref name="YakelTrust13">{{cite journal \|title=Trust in Digital Repositories \|journal=International Journal of Digital Curation \|author=Yakel, E.; Faniel, I.; Krisberg, A. et al. \|volume=8 \|issue=1 \|pages=143–56 \|year=2013 \|doi=10.2218/ijdc.v8i1.251}}</ref> note, “[t]rust in the repository is a separate and distinct factor from trust in the data.” For people outside of the repository, “institutional reputation appears to be the strongest structural assurance indicator of trust.”<ref name="YakelTrust13" /> In essence, effective communication about data risks and steps taken to eliminate problems is helpful in ensuring users that the archive is trustworthy.<ref name="YoonData17">{{cite journal \|title=Data reusers' trust development \|journal=JASIST \|author=Yoon, A. \|volume=68 \|issue=4 \|pages=946-956 \|year=2016 \|doi=10.1002/asi.23730}}</ref>

			Data that face extreme or unusual risks, however, may not be manageable via typical data curation workflows. Downs and Chen<ref name="DownsCuration17">{{cite book \|chapter=Chapter 12: Curation of Scientific Data at Risk of Loss: Data Rescue and Dissemination \|title=Curating research data - Volume one: Practical strategies for your digital repository \|author=Downs, R.R.; Chen, R.S. \|publisher=Association of College and Research Libraries \|pages=263–77 \|year=2017 \|doi=10.7916/D8W09BMQ}}</ref> note that dealing with some data risk factors “may well require divergence from regular data curation procedures, as tradeoffs may be necessary.” For example, Gallaher ''et al.''<ref name="GallaherTheProc15">{{cite journal \|title=The process of bringing dark data to light: The rescue of the early Nimbus satellite data \|journal=GeoResJ \|author=Gallaher, D.; Campbell, G.G.; Meier, W. et al. \|volume=6 \|pages=124–34 \|year=2015 \|doi=10.1016/j.grj.2015.02.013}}</ref> undertook an extensive project to recover, reconstruct, and reprocess data from early satellite missions into modern formats that are usable by modern scientists. This project involved dealing with degrading and fragile magnetic tapes, extracting data from the tapes’ unusual format, and recreating documentation for the data. Additionally, natural disasters, fires, and floods also present unpredictable risk factors to data collections of all kinds. While these kinds of events can be planned for and steps can be taken to prevent the occurrence of some of them (e.g., fires), they can still cause major data loss and/or require significant recovery effort.

			Mitigating risks, of whatever kind, takes effort and resources. The time required to create metadata, re-format files, create contingency plans, and communicate these efforts to user communities can be considerable. This time investment can be the greatest barrier to performing risk assessment and mitigation activities.<ref name="ThompsonWhere14">{{cite journal \|title=Where Have All the Scientific Data Gone? LIS Perspective on the Data-At-Risk Predicament \|journal=College & Research Libraries \|author=Thompson, C.A.; Robertson, D.; Greenberg, J. \|volume=75 \|issue=6 \|pages=842-861 \|year=2014 \|doi=10.5860/crl.75.6.842}}</ref> Putting focus on assessment of data risk factors may mean that “certain priorities need to be re-ordered, new skills acquired and taught, resources redirected, and new networks constructed.”<ref name="GriffinWhen15">{{cite journal \|title=When are Old Data New Data? \|journal=GeoResJ \|author=Griffin, R.E.; CODATA Task Group ‘Data At Risk’ (DAR-TG) \|volume=6 \|pages=92–97 \|year=2015 \|doi=10.1016/j.grj.2015.02.004}}</ref> It can be possible to automate some components of risk assessment<ref name="GrafADec16">{{cite journal \|title=A Decision Support System to Facilitate File Format Selection for Digital Preservation \|journal=Libellarium \|author=Graf, R.; Ryan, H.M.; Houzanme, T. et al. \|volume=9 \|issue=2 \|pages=267–74 \|year=2016 \|doi=10.15291/libellarium.v9i2.274}}</ref>, but most of the steps require human effort. This intensive effort is vividly illustrated by the many data rescue initiatives that have taken place within government agencies and other kinds of organizations over the past few decades.

	==Footnotes==		==Footnotes==

Shawndouglas: Saving and adding more.

2020-12-14T22:15:15Z

Saving and adding more.

← Older revision		Revision as of 22:15, 14 December 2020
Line 153:		Line 153:
	\|}		\|}
	\|}		\|}

			===Research on data risks===
			A range of studies have explored the kinds of risks that scientific data may face, and potential ways to mitigate specific risk factors. Many of these studies touch on practices that are typical of scientific data archives. Metadata, for example, can be considered both a risk factor and a mitigation strategy. Insufficient metadata is itself a potential factor that can reduce the discoverability, usability, and preservability of data, particularly in situations where direct human knowledge of the data is absent.<ref name="MichenerNongeo97">{{cite journal \|title=Nongeospatial metadata for the ecological sciences \|journal=Ecological Applications \|author=Kichener, W.K.; Brunt, J.W.; Helly, J.J. et al. \|volume=7 \|issue=1 \|pages=330–42 \|year=1997 \|doi=10.1890/1051-0761(1997)007[0330:NMFTES]2.0.CO;2}}</ref> In fact, many data rescue projects find that the “rescue” efforts must be targeted much more toward metadata than data.<ref name="KnappScien07">{{cite journal \|title=Scientific Data Stewardship: Lessons Learned from a Satallite–Data Rescue Effort \|journal=Bulletin of the American Meteorological Society \|author=Knapp, K.R.; Bates, J.J.; Barkstrom, B. et al. \|volume=88 \|issue=9 \|pages=1359–62 \|year=2007 \|doi=10.1175/BAMS-88-9-1359}}</ref><ref name="HsuRescue15>{{cite journal \|title=Rescue of long-tail data from the ocean bottom to the Moon: IEDA Data Rescue Mini-Awards \|journal=GeoResJ \|author=Hsu, L.; Lehnert, K.A.; Goodwillie, A. et al. \|volume=6 \|pages=108–114 \|year=2015 \|doi=10.1016/j.grj.2015.02.012}}</ref> This might be the case for a couple of reasons. First, insufficient or missing metadata might prevent data from being usable regardless of the condition of the data themselves. Examples include missing column headers in tabular data that prevent a user from knowing what the data are representing, and insufficient provenance metadata that prevent users from trusting the data due to lack of context about data collection and [[quality control]]. Second, metadata are also central to documenting and mitigating risks as they manifest, while preventing risks from becoming problematic in the future. (Anderson et al. 2011) For example, documenting data ownership and usage rights is an essential step in mitigating risk factor #9, “Problems with legal status for data ownership and use,” from Table 1.

Shawndouglas: Saving and adding more.

2020-12-14T21:57:27Z

Saving and adding more.

← Older revision		Revision as of 21:57, 14 December 2020
Line 55:		Line 55:

	==Data risk assessment==		==Data risk assessment==
			[[Risk assessment]] is a regular activity within many organizations. In a general sense, [[risk management]] plans are complementary to project management plans. (Cervone 2006) Organizational assessment of digital data and information collections is likewise not new. (Maemura, Moles & Becker 2017) The analysis presented in this paper builds on prior work in a number of areas: 1) research on data risks, 2) data rescue initiatives within government agencies and specific disciplines, 3) CODATA and RDA working groups and meetings, 4) trusted repository certifications, and 5) knowledge and experience of the ESIP Data Stewardship Committee members. Table 1 summarizes data risk factors that emerge from these knowledgebases. The list of risk factors shown in Table 1 is not meant to be exhaustive. Rather, it provides a useful illustration of the diverse ways in which data sets, collections, and archives might encounter risks to data usability and accessibility. The rest of this section details further key insights from the five areas of prior work noted above.

			{\|
			\| STYLE="vertical-align:top;"\|
			{\| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="80%"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;" colspan="3"\|'''Table 1.''' Risk factors for scientific data collections
			\|-
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Risk factor
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Description
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|1.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of use
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data are rarely accessed and dubbed "unwanted," thus getting thrown away.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|2.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Loss of funding for archive
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|The whole archive loses its funding source.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|3.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Loss of funding for specific datasets
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Specific datasets lose their funding source.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|4.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Loss of knowledge around context or access
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data owners lose individuals—e.g., due to retirement or death—who know how to access the data or know the metadata associated with these data that make the data useable to others.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|5.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of documentation and metadata
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data cannot be interpreted due to lack of contextual knowledge.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|6.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data mislabeling
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data are lost because they are poorly identified (either physically or digitally).
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|7.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Catastrophes
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Fires, floods, wars, human conflicts, etc. destroy data and/or their owners.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|8.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Poor data governance
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Uncertain or unknown decision making processes impede effective data management.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|9.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Problems with legal status for data ownership and use
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Uncertain, unknown, or restrictive legal status limits the possible uses of data.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|10.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Media deterioration
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Physical media deterioration prevents data from being accessed (paper, tape, or digital media).
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|11.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Missing files
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data files are lost without any known reason.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|12.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Overdependence on a single service provider
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Problems arise from having a single point of failure, particularly if a vital service provider goes out of business.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|13.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Accidental deletion
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data are accidentally deleted by staff error.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|14.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of planning
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of planning puts data collections at risk of being susceptible to unexpected events.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|15.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|[[Cybersecurity]] breach
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data are intentionally deleted or corrupted via a security breach, e.g., via malware.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|16.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Overabundant data
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Difficulty dealing with too much data results in a reduction in value or quality of whole collections.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|17.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Political interference
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data is deleted or made inaccessible due to uncontrollable political decisions.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|18.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of provenance information
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data cannot be trusted or understood because of a lack of information about data processing steps, or about data stewardship chains of trust.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|19.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|File format obsolescence
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data cannot be accessed due to lack of knowledge, equipment, or software for reading a specific file format.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|20.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Storage hardware breakdown
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data is lost due to a sudden and catastrophic malfunction of storage hardware.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|21.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Bit rot and data corruption
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Digital data on storage hardware gradually becomes corrupted due to an accumulation of non-critical failures (bits flipping) in a data storage device.
			\|-
			\|}
			\|}

Shawndouglas: Saving and adding more.

2020-12-14T21:36:37Z

Saving and adding more.

← Older revision		Revision as of 21:36, 14 December 2020
Line 46:		Line 46:
	This project was coordinated through the Data Stewardship Committee within the Earth Science Information Partners (ESIP), a non-profit organization that exists to support collection, stewardship, and use of earth science data, [[information]], and knowledge.{{Efn\|See https://wiki.esipfed.org/Preservation_and_Stewardship.}} The immediate motivation for the project stemmed from the Data Stewardship Committee members engaging with groups who were undertaking grass-roots “data rescue” initiatives after the 2016 U.S. presidential election. At that time, a number of loosely organized and coordinated efforts were initiated to duplicate data from U.S. government organizations to prevent potential politically motivated data deletion or obfuscation.<ref name="DennisScien16">{{cite web \|url=https://www.washingtonpost.com/news/energy-environment/wp/2016/12/13/scientists-are-frantically-copying-u-s-climate-data-fearing-it-might-vanish-under-trump/ \|title=Scientists are frantically copying U.S. climate data, fearing it might vanish under Trump \|author=Dennis, B. \|work=The Washington Post \|date=13 December 2016}}</ref><ref name="VarinskyScien16">{{cite web \|url=https://www.businessinsider.com/data-rescue-government-data-preservation-efforts-2017-2 \|title=Scientists across the US are scrambling to save government research in 'Data Rescue' events \|author=Varinsky, D. \|work=Business Insider \|date=11 February 2017}}</ref> In many cases, these initiatives specifically focused on duplicating government-hosted earth science data.		This project was coordinated through the Data Stewardship Committee within the Earth Science Information Partners (ESIP), a non-profit organization that exists to support collection, stewardship, and use of earth science data, [[information]], and knowledge.{{Efn\|See https://wiki.esipfed.org/Preservation_and_Stewardship.}} The immediate motivation for the project stemmed from the Data Stewardship Committee members engaging with groups who were undertaking grass-roots “data rescue” initiatives after the 2016 U.S. presidential election. At that time, a number of loosely organized and coordinated efforts were initiated to duplicate data from U.S. government organizations to prevent potential politically motivated data deletion or obfuscation.<ref name="DennisScien16">{{cite web \|url=https://www.washingtonpost.com/news/energy-environment/wp/2016/12/13/scientists-are-frantically-copying-u-s-climate-data-fearing-it-might-vanish-under-trump/ \|title=Scientists are frantically copying U.S. climate data, fearing it might vanish under Trump \|author=Dennis, B. \|work=The Washington Post \|date=13 December 2016}}</ref><ref name="VarinskyScien16">{{cite web \|url=https://www.businessinsider.com/data-rescue-government-data-preservation-efforts-2017-2 \|title=Scientists across the US are scrambling to save government research in 'Data Rescue' events \|author=Varinsky, D. \|work=Business Insider \|date=11 February 2017}}</ref> In many cases, these initiatives specifically focused on duplicating government-hosted earth science data.

			ESIP Data Stewardship Committee members wrote a white paper to provide the earth science data centers’ perspective on these grassroots “data rescue” activities.<ref name="MayernikStronger17">{{cite web \|url=https://esip.figshare.com/articles/journal_contribution/Stronger_together_the_case_for_cross-sector_collaboration_in_identifying_and_preserving_at-risk_data/4816474/1 \|title=Stronger together: The case for cross-sector collaboration in identifying and preserving at-risk data \|author=Mayernik, M.S.; Downs, R. R.; Duerr, R. et al. \|work=FigShare \|date=04 April 2017}}</ref> That document described essential considerations within the day-to-day work of existing federal and federally-funded earth science [[Wikipedia:Data library\|data archiving]] organizations, including data centers’ constant focus on documentation, traceability, and persistence of scientific data. The white paper also provided suggestions for how those grassroots efforts might productively engage with the data centers themselves.

			One point that was emphasized in the white paper was that the actual risks faced by the data collections may not be transparent from the outside. In other words, “data rescue” activities may have in fact been duplicating data that were at minimal risk of being lost.<ref name="LamdanLessons18">{{cite journal \|title=Lessons from DataRescue: The Limits of Grassroots Climate Change Data Preservation and the Need for Federal Records Law Reform \|journal=University of Pennsylvania Law Review Online \|author=Lamdan, S. \|volume=166 \|issue=1 \|year=2018 \|url=https://scholarship.law.upenn.edu/penn_law_review_online/vol166/iss1/12}}</ref> This point, and the white paper in general, was well received by people inside and outside of these grass-roots initiatives.<ref name="CorneliusWhat18">{{cite journal \|title=‘What Data?’ Records and Data Policy Coordination During Presidential Transitions \|journal=Proceedings from iConference 2018: Transforming Digital Worlds \|author=Cornelius, K.B.; Pasquetto, I.V. \|pages=155–63 \|year=2018 \|doi=10.1007/978-3-319-78105-1_20}}</ref><ref name="McGovernData17">{{cite journal \|title=Data rescue: Observations from an archivist \|journal=ACM SIGCAS Computers and Society \|author=McGovern, N.Y. \|volume=47 \|issue=2 \|pages=19–26 \|year=2017 \|doi=10.1145/3112644.3112648}}</ref> Questions then came back to the ESIP Data Stewardship Committee about how to understand what data held by government agencies were actually at-risk.

			The analysis presented in this paper was initiated in response to these questions. Since then, these grassroots “data rescue” initiatives have had mixed success in sustaining and formalizing their efforts.<ref name="AllenStrat17">{{cite journal \|title=Strategic open data preservation: Roles and opportunities for broader engagement by librarians and the public \|journal=College & Research Libraries News \|author=Allen, L.; Stewart, C.; Wright, S. \|volume=78 \|issue=9 \|at=482 \|year=2017 \|doi=10.5860/crln.78.9.482}}</ref><ref name="ChodackiData17">{{cite journal \|title=Data Mirror-Complementing Data Producers \|journal=Against the Grain \|author=Chodacki, J. \|volume=29 \|issue=6 \|at=13 \|year=2017 \|doi=10.7771/2380-176X.7877}}</ref><ref name="JanzMaint17">{{cite journal \|title=Maintaining Access to Public Data: Lessons from Data Refuge \|journal=Against the Grain \|author=Janz, M.M. \|volume=29 \|issue=6 \|at=11 \|year=2017 \|doi=10.7771/2380-176X.7875}}</ref>The intention of our paper is to enable more effective data risk assessment broadly. Rescuing data after they have been corrupted, deleted, or lost can be time- and effort-intensive, and in some cases it may be impossible.<ref name="PientaRetire18">{{cite journal \|title=Retirement in the 1950s: Rebuilding a Longitudinal Research Database \|journal=IASSIST Quarterly \|author=Pienta, A.M.; Lyle, J. \|volume=42 \|issue=1 \|at=12 \|year=2017 \|doi=10.29173/iq19}}</ref> Thus, we aim to provide guidelines to any individual or organization that manages and provides access to scientific data. In turn, these individuals and organizations can better assess the risks that their data face and characterize those risks.

			When discussing risk and, in particular, data risk, it is useful to ask "what is the objective that is being challenged by the possible risk factors?" With regard to data, in general, discussions of risk might presume that “risks” threaten the current or future access to data by the potential data users. Currently, continuing public access to and use of scientific data is particularly relevant in light of recent open data and open science initiatives. In this regard, risks for scientific data include factors that could hinder, constrain, or limit current or future data use. Identifying such data use risk factors offers further analysis opportunities to prevent, mitigate, or eliminate the risks.

			==Data risk assessment==

← Older revision		Revision as of 17:22, 15 December 2020
Line 185:		Line 185:
	Through subsequent discussion and analysis of the data risk assessment literature noted above, we identified at least ten different ways that data risk factors could be assessed. Many of these categorization methods are applicable to risk assessments of any kind.<ref name="CervoneProject06">{{cite journal \|title=Project risk management \|journal=OCLC Systems & Services: International digital library perspectives \|author=Cervone, H.F. \|volume=22 \|issue=4 \|pages=256-262 \|year=2006 \|doi=10.1108/10650750610706970}}</ref> The list below is not meant to be exhaustive, and some methods are likely related. Data risk factors could be categorized or prioritized according to the methods listed in Table 2.		Through subsequent discussion and analysis of the data risk assessment literature noted above, we identified at least ten different ways that data risk factors could be assessed. Many of these categorization methods are applicable to risk assessments of any kind.<ref name="CervoneProject06">{{cite journal \|title=Project risk management \|journal=OCLC Systems & Services: International digital library perspectives \|author=Cervone, H.F. \|volume=22 \|issue=4 \|pages=256-262 \|year=2006 \|doi=10.1108/10650750610706970}}</ref> The list below is not meant to be exhaustive, and some methods are likely related. Data risk factors could be categorized or prioritized according to the methods listed in Table 2.

			{\|
			\| STYLE="vertical-align:top;"\|
			{\| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="80%"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;" colspan="2"\|'''Table 2.''' Methods for categorizing data risks
			\|-
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Categorization method
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Description
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Severity of risk
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|How much impact could this risk factor have on the data itself, regardless of the current importance of data to the user?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Likelihood of occurrence
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|How likely is the risk factor to occur?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Length of recovery time
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|How long would it take to recover data or re-establish data accessibility?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Impact on user
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|How significantly would data users be impacted by data loss or loss of data accessibility?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Who is responsible for addressing the problem
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Who has the expertise and responsibility to mitigate or respond to particular risk factors?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Cause of problem
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|What caused a data risk factor to occur?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Degree of control
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|How much control does an organization or individual have over whether a risk factor is present or will occur?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Proactive vs. reactive response
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Should risk factors be mitigated via preventative measures, or should they be responded to upon occurrence?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Nature of mitigation
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|What steps must be taken or processes put in place to prevent a risk, or mitigate a risk after it has occurred?
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Resources required for mitigation
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|What time, money, or personnel resources will be necessary to mitigate risk factors?
			\|-
			\|}
			\|}

			The lists shown in Tables 1 and 2 offer characteristics on which data risk assessments can be built. Combining the categorization methods from Table 2 with the selected risk factors from Table 1 leads to a risk assessment matrix, as shown in Table 3. This figure shows an example of a selection of specific data risk factors and the categorization methods. Depending on the situation or data collection being assessed, different risk factors and/or categorization methods may be more applicable than the ones shown in Table 3. Those conducting a data risk assessment can then use the matrix as a way to organize, prioritize, or potentially quantify the selected risks according to the categorization methods that are most relevant for the specific case at hand. (The next section provides more detailed illustrations of the use of the data risk assessment matrix. Appendix I, at the end of this article, shows the full data risk assessment template, with all risks and categorization methods from Tables 1 and 2.)

			{\|
			\| STYLE="vertical-align:top;"\|
			{\| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="100%"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;" colspan="5"\|'''Table 3.''' Example of a blank data risk assessment matrix, after selection of specific risk factors and categorization methods of interest
			\|-
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" rowspan="2"\|''Risk factors''
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" colspan="4"\|''Categorization methods''
			\|-
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Severity of risk
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Likelihood of occurrence
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Cause of problem
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Required mitigation resources
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of use
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Loss of knowledge
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of docs and metadata
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Catastrophes
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Poor data governance
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Media deterioration
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|
			\|-
			\|}
			\|}

			==Application of the data risk assessment matrix==
			Three case studies are described below, in which the data risk assessment matrix was used to develop a better understanding of data risks for particular resources. These cases enable evaluation of the data risk assessment framework presented in this paper, clarifying its strengths and weaknesses, and pinpointing the situations in which it can be most useful.<ref name="BeckerTheDesign20">{{cite journal \|title=The Design and Use of Assessment Frameworks in Digital Curation \|journal=JASIST \|author=Becker, C.; Maemura, E.; Moles. N. \|volume=71 \|issue=1 \|pages=55–68 \|year=2020 \|doi=10.1002/asi.24209}}</ref>

			===Case 1 – NCAR Library analog data collection===
			The National Center for Atmospheric Research (NCAR) Library maintains an analog data collection that consists of about 300 datasets in support of atmospheric and meteorological research conducted by NCAR scientists. These assets are largely compilations of measurements and statistics published by national and international meteorological services and other kinds of government entities. Many of these assets have been in the NCAR Library’s collections for decades, and most were minimally cataloged when they were first brought into the collection. As such, the current usage of the collection is minimal. A prior assessment done by the NCAR Library and a student assistant sought to identify individual assets that were of higher potential value and interest for current science. This assessment effort resulted in a modernization prioritization based on a geographic and temporal framework, and improved metadata records for about five percent of the collection.<ref name="MayernikModern18">{{cite journal \|title=Modernizing Library Metadata for Historical Weather and Climate Data Collections \|journal=Journal of Library Metadata \|author=Mayernick, M.S.; Huddle, J.; Hou, C.-Y. et al. \|volume=17 \|issue=3–4 \|pages=219–39 \|year=2017 \|doi=10.1080/19386389.2018.1440927}}</ref> This effort did not, however, include any kind of risk assessment related to the physical assets themselves.

			The data risk assessment matrix was therefore helpful in conducting a second-level priority analysis for these NCAR Library analog data assets. We used the matrix as a way to identify which risk factors were most important for these materials, and to characterize the mitigation efforts that were needed for each risk factor. In particular, we focused the risk assessment on the data assets that were previously identified as having high geospatial and temporal interest. The NCAR Library use of the matrix involved a series of steps:

			:''Step 1'' – A number of risk factors listed in the matrix were identified as being of most importance, with the focus being on factors that prevented or impeded the use of these data within current scientific studies. The most immediate risk factors were identified to be the “lack of use” and the “lack of documentation/metadata” for these assets. Other risks that were secondary in immediacy, but still potentially important, were data mislabeling, the questionable legal status for ownership and use, media deterioration, lack of planning, and poor data governance.

			:''Step 2'' – The second step was to identify which categorization methods shown in the matrix were most applicable/appropriate for the NCAR Library’s management and maintenance of this collection. The methods selected were: a) length of recovery time, b) who is responsible for addressing the problem, c) nature of mitigation, and d) resources required for mitigation.

			:''Step 3'' – The third step was to fill in the boxes in the matrix for the risk factors and categorization methods. For example, for the “length of recovery time” question, we used a simple 1–3 scale to indicate relative differences in how long it would take to mitigate the two most important risk factors: “lack of use” and the “lack of documentation/metadata.” As one example, some data assets were published by international agencies and therefore have title pages and documentation that are not in English. In turn, due to the lack of relevant foreign language expertise in the NCAR Library staff, developing new metadata for these resources will take more effort than for those assets that were published by English-speaking countries. For the “resources required for mitigation” categorization method, a numerical scale was not as appropriate. Instead, we filled in the matrix with text descriptions of the resources required to mitigate the risk factors. An example entry under the “lack of documentation and metadata” risk factor was: “We would need to create new metadata for the library catalog, then transform to ISO for inclusion in NCAR DASH Search, with the added challenge of needing to look at microfilm files (no current working reader in Library).”

			In summary, the matrix was very useful as “something to think with.” In other words, it jump-started the process for doing the risk assessment because the NCAR Library staff did not need to spend time developing a comprehensive list of risk factors that may apply for these data, or brainstorm about how to categorize those risks. The risk factor matrix provided a ready-made starting point for the assessment. Because the matrix does not dictate how the cells should be filled in, the NCAR Library staff made decisions about how to apply the matrix for each categorization method that was chosen. The matrix structure could potentially be applied or customized to create a prioritization rubric, by supporting the creation of a numeric scoring process for categories where that is appropriate.

← Older revision		Revision as of 21:57, 14 December 2020
Line 55:		Line 55:

	==Data risk assessment==		==Data risk assessment==
			[[Risk assessment]] is a regular activity within many organizations. In a general sense, [[risk management]] plans are complementary to project management plans. (Cervone 2006) Organizational assessment of digital data and information collections is likewise not new. (Maemura, Moles & Becker 2017) The analysis presented in this paper builds on prior work in a number of areas: 1) research on data risks, 2) data rescue initiatives within government agencies and specific disciplines, 3) CODATA and RDA working groups and meetings, 4) trusted repository certifications, and 5) knowledge and experience of the ESIP Data Stewardship Committee members. Table 1 summarizes data risk factors that emerge from these knowledgebases. The list of risk factors shown in Table 1 is not meant to be exhaustive. Rather, it provides a useful illustration of the diverse ways in which data sets, collections, and archives might encounter risks to data usability and accessibility. The rest of this section details further key insights from the five areas of prior work noted above.

			{\|
			\| STYLE="vertical-align:top;"\|
			{\| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="80%"
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;" colspan="3"\|'''Table 1.''' Risk factors for scientific data collections
			\|-
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Risk factor
			! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;"\|Description
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|1.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of use
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data are rarely accessed and dubbed "unwanted," thus getting thrown away.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|2.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Loss of funding for archive
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|The whole archive loses its funding source.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|3.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Loss of funding for specific datasets
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Specific datasets lose their funding source.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|4.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Loss of knowledge around context or access
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data owners lose individuals—e.g., due to retirement or death—who know how to access the data or know the metadata associated with these data that make the data useable to others.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|5.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of documentation and metadata
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data cannot be interpreted due to lack of contextual knowledge.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|6.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data mislabeling
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data are lost because they are poorly identified (either physically or digitally).
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|7.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Catastrophes
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Fires, floods, wars, human conflicts, etc. destroy data and/or their owners.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|8.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Poor data governance
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Uncertain or unknown decision making processes impede effective data management.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|9.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Problems with legal status for data ownership and use
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Uncertain, unknown, or restrictive legal status limits the possible uses of data.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|10.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Media deterioration
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Physical media deterioration prevents data from being accessed (paper, tape, or digital media).
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|11.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Missing files
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data files are lost without any known reason.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|12.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Overdependence on a single service provider
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Problems arise from having a single point of failure, particularly if a vital service provider goes out of business.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|13.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Accidental deletion
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data are accidentally deleted by staff error.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|14.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of planning
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of planning puts data collections at risk of being susceptible to unexpected events.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|15.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|[[Cybersecurity]] breach
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data are intentionally deleted or corrupted via a security breach, e.g., via malware.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|16.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Overabundant data
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Difficulty dealing with too much data results in a reduction in value or quality of whole collections.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|17.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Political interference
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data is deleted or made inaccessible due to uncontrollable political decisions.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|18.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Lack of provenance information
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data cannot be trusted or understood because of a lack of information about data processing steps, or about data stewardship chains of trust.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|19.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|File format obsolescence
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data cannot be accessed due to lack of knowledge, equipment, or software for reading a specific file format.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|20.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Storage hardware breakdown
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Data is lost due to a sudden and catastrophic malfunction of storage hardware.
			\|-
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|21.
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Bit rot and data corruption
			\| style="background-color:white; padding-left:10px; padding-right:10px;"\|Digital data on storage hardware gradually becomes corrupted due to an accumulation of non-critical failures (bits flipping) in a data storage device.
			\|-
			\|}
			\|}