Difference between revisions of "Journal:Towards a contextual approach to data quality"

From LIMSWiki
Jump to navigationJump to search
(Saving and adding more.)
(→‎Notes: Cats)
 
(2 intermediate revisions by the same user not shown)
Line 19: Line 19:
|download    = [https://www.mdpi.com/2306-5729/5/4/90/pdf https://www.mdpi.com/2306-5729/5/4/90/pdf] (PDF)
|download    = [https://www.mdpi.com/2306-5729/5/4/90/pdf https://www.mdpi.com/2306-5729/5/4/90/pdf] (PDF)
}}
}}
{{ombox
 
| type      = notice
| image    = [[Image:Emblem-important-yellow.svg|40px]]
| style    = width: 500px;
| text      = This article should be considered a work in progress and incomplete. Consider this article incomplete until this notice is removed.
}}
==Abstract==
==Abstract==
This essay delves into the need for a framework for approaching [[data quality]] in the context of scientific research. First, the concept of "quality" as a property of [[information]], evidence, and data is presented, and research on the philosophy of information, science, and [[Biomedical sciences|biomedicine]] is reviewed. Based on this review, the need for a more purpose-dependent and contextual approach to data quality in scientific research is argued, whereby the quality of a dataset is dependent on the context of use of the dataset as much as the dataset itself. The rationale to the approach is then exemplified by discussing current critiques and debates of scientific quality, thus showcasing how data quality can be approached contextually.
This essay delves into the need for a framework for approaching [[data quality]] in the context of scientific research. First, the concept of "quality" as a property of [[information]], evidence, and data is presented, and research on the philosophy of information, science, and [[Biomedical sciences|biomedicine]] is reviewed. Based on this review, the need for a more purpose-dependent and contextual approach to data quality in scientific research is argued, whereby the quality of a dataset is dependent on the context of use of the dataset as much as the dataset itself. The rationale to the approach is then exemplified by discussing current critiques and debates of scientific quality, thus showcasing how data quality can be approached contextually.
Line 55: Line 50:
The third line of philosophical research discussed here has focuses on quality as a property of scientific evidence, especially in the [[Biomedical sciences|biomedical]] context. This research has partly been a reaction to the rise of evidence-based medicine (EBM), an approach to medical research and practice that is based on a specific categorization and ranking of evidence. Since the 1980s, as a movement to reform medical practice and research, EBM has aimed to improve decision-making by removing the influence of subjective preferences from different stages of the process. As formulated by Sackett and colleagues, the central idea of EBM has been “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.”<ref name="SackettEvidence96">{{cite journal |title=Evidence based medicine: What it is and what it isn't |journal=BMJ |author=Sackett, D.L.; Rosenberg, W.M.C.; Gray, J.A.M. et al. |volume=312 |page=71 |year=1996 |doi=10.1136/bmj.312.7023.71}}</ref> Practically, EBM proponents have introduced “evidence hierarchies,” which describe the assumed quality of different types of evidence and are supposed to help decision makers to project some order in the available evidence.<ref name="BluhmFromHier05">{{cite journal |title=From hierarchy to network: A richer view of evidence for evidence-based medicine |journal=Perspectives in Biology and Medicine |author=Bluhm, R. |volume=48 |issue=4 |pages=535–47 |year=2005 |doi=10.1353/pbm.2005.0082 |pmid=16227665}}</ref> This order aligns better support for the efficacy of different interventions with better evidence types, which in the EBM context consists of evidence from randomized controlled trials or systematic reviews and meta-analyses of randomized controlled trials.
The third line of philosophical research discussed here has focuses on quality as a property of scientific evidence, especially in the [[Biomedical sciences|biomedical]] context. This research has partly been a reaction to the rise of evidence-based medicine (EBM), an approach to medical research and practice that is based on a specific categorization and ranking of evidence. Since the 1980s, as a movement to reform medical practice and research, EBM has aimed to improve decision-making by removing the influence of subjective preferences from different stages of the process. As formulated by Sackett and colleagues, the central idea of EBM has been “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.”<ref name="SackettEvidence96">{{cite journal |title=Evidence based medicine: What it is and what it isn't |journal=BMJ |author=Sackett, D.L.; Rosenberg, W.M.C.; Gray, J.A.M. et al. |volume=312 |page=71 |year=1996 |doi=10.1136/bmj.312.7023.71}}</ref> Practically, EBM proponents have introduced “evidence hierarchies,” which describe the assumed quality of different types of evidence and are supposed to help decision makers to project some order in the available evidence.<ref name="BluhmFromHier05">{{cite journal |title=From hierarchy to network: A richer view of evidence for evidence-based medicine |journal=Perspectives in Biology and Medicine |author=Bluhm, R. |volume=48 |issue=4 |pages=535–47 |year=2005 |doi=10.1353/pbm.2005.0082 |pmid=16227665}}</ref> This order aligns better support for the efficacy of different interventions with better evidence types, which in the EBM context consists of evidence from randomized controlled trials or systematic reviews and meta-analyses of randomized controlled trials.


Philosophers of medicine have analyzed and criticized various tenets of EBM, including the theoretical and methodological basis of the choice of specific types of evidence as high-quality evidence<ref name="WorrallWhatEv02">{{cite journal |title=What Evidence in Evidence‐Based Medicine? |journal=Philosophy of Science |author=Worrall, J. |volume=69 |issue=S3 |pages=S316–30 |year=2002 |doi=10.1086/341855}}</ref> , the exclusion and denigration of some types of evidence<ref name="ClarkeMech13">{{cite journal |title=Mechanisms and the Evidence Hierarchy |journal=Topoi |author=Clarke, B.; Gillies, D.; Illari, P. et al. |volume=33 |pages=339–60 |year=2013 |doi=10.1007/s11245-013-9220-9}}</ref>, and the ways in which hierarchies of evidence are delineated in evidence-based approaches.<ref name="CampanerEvidence12">{{cite journal |title=Evidence and the Assessment of Causal Relations in the Health Sciences |journal=International Studies in the Philosophy of Science |author=Campaner, R.; Galavotti, M.C. |volume=26 |issue=1 |pages=27–45 |year=2012 |doi=10.1080/02698595.2012.653113}}</ref> While these analyses have not explicitly taken issue with notions of quality per se, their results are significant for this discussion on how to approach data quality. The ways in which evidence is classified and its quality is assessed in EBM seem to apply an intrinsic and universalistic approach to evidence, whereby, e.g., evidence collected through randomized controlled trials (RCTs) is “gold standard.” This means that RCTs are normally given the highest level of quality, although this may be lowered in case of methodological problems; instead, evidence from other methods such as observational studies could be ranked as high-quality, but are automatically given a lower rank as the starting point.<ref name="KerryCausation12">{{cite journal |title=Causation and evidence‐based practice: An ontological review |journal=Journal of Evaluation in Clinical Practice |author=Kerry, R.; Eriksen, T.E.; Lie. S.A.N. et al. |volume=18 |issue=5 |pages=1006–12 |year=2012 |doi=10.1111/j.1365-2753.2012.01908.x}}</ref> In other words, certain methods are considered to be ''prima facie'' and epistemically superior, with a gold-like, higher value compared to alternatives.<ref name="StegengaIsMeta11">{{cite journal |title=Is meta-analysis the platinum standard of evidence? |journal=Studies in History and Philosophy of Science Part C |author=Stegenga, J. |volume=42 |issue=4 |pages=497–507 |year=2011 |doi=10.1016/j.shpsc.2011.07.003}}</ref> The problem with this approach to the classification of evidence quality is that it is applied to most areas of biomedical research, with no consideration for specifics and different research contexts. In many areas of biomedical research, the gold standard evidence hailed by EBM often cannot be produced, but this does not necessarily mean that the evidence produced is of low quality. For example, Saana Jukola has shown that in nutrition research, RCTs cannot be conducted because of practical, ethical, and methodological aspects of this line of research.<ref name="JukolaOnThe19">{{cite journal |title=On the evidentiary standards for nutrition advice |journal=Studies in History and Philosophy of Science Part C |author=Jukola, S. |volume=73 |pages=1–9 |year=2019 |doi=10.1016/j.shpsc.2018.05.007}}</ref> Differently from the EBM approach, the quality of biomedical evidence is used to meet specific rather than universal hierarchies, depending on the aims and the context in which it is to be used.
Philosophers of medicine have analyzed and criticized various tenets of EBM, including the theoretical and methodological basis of the choice of specific types of evidence as high-quality evidence<ref name="WorrallWhatEv02">{{cite journal |title=What Evidence in Evidence‐Based Medicine? |journal=Philosophy of Science |author=Worrall, J. |volume=69 |issue=S3 |pages=S316–30 |year=2002 |doi=10.1086/341855}}</ref>, the exclusion and denigration of some types of evidence<ref name="ClarkeMech13">{{cite journal |title=Mechanisms and the Evidence Hierarchy |journal=Topoi |author=Clarke, B.; Gillies, D.; Illari, P. et al. |volume=33 |pages=339–60 |year=2013 |doi=10.1007/s11245-013-9220-9}}</ref>, and the ways in which hierarchies of evidence are delineated in evidence-based approaches.<ref name="CampanerEvidence12">{{cite journal |title=Evidence and the Assessment of Causal Relations in the Health Sciences |journal=International Studies in the Philosophy of Science |author=Campaner, R.; Galavotti, M.C. |volume=26 |issue=1 |pages=27–45 |year=2012 |doi=10.1080/02698595.2012.653113}}</ref> While these analyses have not explicitly taken issue with notions of quality per se, their results are significant for this discussion on how to approach data quality. The ways in which evidence is classified and its quality is assessed in EBM seem to apply an intrinsic and universalistic approach to evidence, whereby, e.g., evidence collected through randomized controlled trials (RCTs) is “gold standard.” This means that RCTs are normally given the highest level of quality, although this may be lowered in case of methodological problems; instead, evidence from other methods such as observational studies could be ranked as high-quality, but are automatically given a lower rank as the starting point.<ref name="KerryCausation12">{{cite journal |title=Causation and evidence‐based practice: An ontological review |journal=Journal of Evaluation in Clinical Practice |author=Kerry, R.; Eriksen, T.E.; Lie. S.A.N. et al. |volume=18 |issue=5 |pages=1006–12 |year=2012 |doi=10.1111/j.1365-2753.2012.01908.x}}</ref> In other words, certain methods are considered to be ''prima facie'' and epistemically superior, with a gold-like, higher value compared to alternatives.<ref name="StegengaIsMeta11">{{cite journal |title=Is meta-analysis the platinum standard of evidence? |journal=Studies in History and Philosophy of Science Part C |author=Stegenga, J. |volume=42 |issue=4 |pages=497–507 |year=2011 |doi=10.1016/j.shpsc.2011.07.003}}</ref> The problem with this approach to the classification of evidence quality is that it is applied to most areas of biomedical research, with no consideration for specifics and different research contexts. In many areas of biomedical research, the gold standard evidence hailed by EBM often cannot be produced, but this does not necessarily mean that the evidence produced is of low quality. For example, Saana Jukola has shown that in nutrition research, RCTs cannot be conducted because of practical, ethical, and methodological aspects of this line of research.<ref name="JukolaOnThe19">{{cite journal |title=On the evidentiary standards for nutrition advice |journal=Studies in History and Philosophy of Science Part C |author=Jukola, S. |volume=73 |pages=1–9 |year=2019 |doi=10.1016/j.shpsc.2018.05.007}}</ref> Differently from the EBM approach, the quality of biomedical evidence is used to meet specific rather than universal hierarchies, depending on the aims and the context in which it is to be used.
 
==Developing a contextual approach to data quality==
Where does the previous review leave us? It appears that discussions of quality in relatively distinct areas of the philosophical literature can be integrated into an overarching approach to quality in the context of scientific data, which has contextuality as its core principle. According to this approach, quality is a contextual feature of data: it is a result of the relations established between a dataset and the questions, aims, and tools employed in the context of the use of data. However, the assessment of the quality of a dataset needs to focus on the features of this context as much as the dataset itself.
 
As such, the contextual approach is a development of contemporary accounts of data and information. Leonelli has defended a view of scientific data according to which data is a relational entity, whose evidential value is not a given and intrinsic component, but is rather a result of the relations established between the questions, claims, and purposes involved in scientific practices and the objects that are used as data.<ref name="LeonelliDataCent16" /> Similarly, Floridi has discussed information in relational terms, according to which something counts as information only for a certain type of agent and use—to the point that misinformation does not count as information for an agent interested in the production of knowledge.<ref name="FloridiThePhil11">{{cite book |title=The Pholosophy of Information |author=Floridi, L. |publisher=University of Oxford Press |year=2011 |doi=10.1093/acprof:oso/9780199232383.001.0001 |isbn=9780199232383}}</ref>
 
More specifically, the main difference-maker that determines the relevant features and dimensions of data quality in a specific context is the use of data, including the goals, assumptions, chains of inference, and evidential reasoning involved. Depending on the features of the context where a dataset is used, some dimensions will become more important than others. For example, in cases of urgency and the need to use data for immediate policy measures, the timeliness and accuracy of data might be preferred over its completeness. In turn, the context of use is shaped by the evidential reasoning that determines the use of a dataset as a representation of certain phenomena and is thus based on various chains of inference and the mobilization of other evidence and knowledge, as well as specific assumptions, warrants, and goals.<ref name="CanaliMaking20">{{cite journal |title=Making evidential claims in epidemiology: Three strategies for the study of the exposome |journal=Studies in History and Philosophy of Science Part C |author=Canali, S. |volume=82 |at=101248 |year=2020 |doi=10.1016/j.shpsc.2019.101248}}</ref> This means that, for instance, when mobilizing and integrating different datasets, users might determine and assess the quality of a dataset for its relevance and compatibility with available evidence, as opposed to its objectivity. Therefore, in a specific context, the quality of a dataset is determined by the use of the dataset and in particular the alignment of its dimensions with the contextual properties of use. In this sense, conceptualizing the quality of a dataset in contextual terms is a move beyond seeing quality as a “static” and universal component of datasets, which can be determined independently and on the basis of their intrinsic characteristics only.
 
At the same time, emphasizing the role of context entails that quality will have to be assessed differently in different contexts. Yet, this does not mean that the categorization of quality into different dimensions and components is subjective, or that the development of approaches, tools, and standards for the evaluation of data quality is unnecessary. These need to be encouraged as attempts at developing more local and situated approaches to quality, which includes critically evaluating the use of data and explicitly reflecting on its relation with specific dimensions of quality. In this sense, the contextual approach indicates quality criteria and assessment approaches according to which:
 
* the specific elements and dimensions of quality will be different depending on the features and goals of data use;
* some dimensions will be more important than others in different contexts; and,
* since each dimension of quality needs a specific type of measurement, different measurement tools and techniques will be used in different contexts.
 
==The contextual approach in practice==
The contextual approach is a way of conceptualizing quality as a relation of scientific data with other components of a research context and therefore thinking about how quality assessment should be structured and implemented. But how does the approach translate in research practice? Three cases follow, exemplifying where the contextual approach can be seen in practical areas of research and showing how the approach can work in the context of specific applications, as well as noting the issues and different directions the approach points to.
 
===FAIR data priniciples===
This first case addresses one of the current attempts at improving the quality of research, by making data FAIR, i.e., findable, accessible, interoperable. and reusable.<ref name="WilkinsonTheFAIR16" /> The [[Journal:The FAIR Guiding Principles for scientific data management and stewardship|FAIR data principles]] aim to raise the quality of data collected and produced as a result of research practices and related [[Data analysis|analytic]] and processing tools. Both this specification of FAIR's four components and the contribution of the FAIR movement can be informed and interpreted from the perspective of the contextual approach to data quality. First, the four FAIR guiding principles can be conceptualized as dimensions of quality as an overarching (desired) feature of a dataset; e.g., accessibility was already present in some of the categorizations of quality in the information quality debates of the 1990s.<ref name="WangToward95" /> Second, as such, the guiding principles presented by FAIR are contextual dimensions of quality. Namely, the findability, accessibility, interoperability, and reusability of a dataset are features that determine the quality of a dataset, yet arise and can only be evaluated in specific contexts. Whether a dataset is FAIR will depend on the specific features of a dataset, and indeed the FAIR guiding principles are presented with concrete suggestions for how to deliver these principles. For instance, in this context, the findability of a dataset can be achieved through, e.g., [[metadata]] that can assign a unique identifier to the dataset, describe it, and are registered in searchable resources and repositories. The digital object identifier (DOI) is a good example of a way in which the findability of publications and datasets can be applied and their quality therefore improved as a result.
 
Worthy of highlighting here is that these elements, such as metadata and repositories, pertain to the context of a dataset, rather than the intrinsic properties of the dataset itself. The evaluation of the findability of a dataset comes down to these elements and, more generally, the context in which the data are actually used, as much as of the dataset in itself. Certainly, in individual cases, a dataset will either be found or not, but the realization of findability as a property of a dataset and a dimension of its quality will be due to the specific context where the use of a dataset is realized. As such, the FAIR guidelines largely instantiate the approach to data quality presented in this essay. At the same time, seeing the features indicated by FAIR in contextual terms also suggests that they will be highly dependent on the context of the specific sciences where the data are used, which means that their meaning and application will change, as research practices are different. For instance, although findability is a relatively generalizable feature, whether a dataset is findable by researchers is not an absolute or intrinsic feature of a dataset that can be easily scaled to any discipline. Different researchers will approach and potentially access a dataset coming from the perspective of their discipline and its specific data, which might be different from those of the context of data productions. This is why the work of data curation is so important in current data-intensive research, because it provides data with information on its original context, and in this way it enables new users to judge its reliability, quality, and relevance for new and different uses.<ref name="LeonelliOnTheLoc09">{{cite journal |title=On the Locality of Data and Claims about Phenomena |journal=Philosophy of Science |author=Leonelli, S. |volume=76 |issue=5 |pages=737–49 |year=2009 |doi=10.1086/605804}}</ref>
 
Another implication of this contextual interpretation of FAIR is that each re-use of data will lead to a different quality assessment and that a dataset used in one research project could have a different level of quality in another. Yet, one might argue that quality ratings need to be independent of context, especially for new users and different communities, who have not used the data yet and want to know about its quality. Is this approach, thus, unhelpful and not applicable to actual research practice? Quality ratings are necessary, in particular for data re-use, but here the contextual approach can give different suggestions. Approaching quality in contextual terms suggests that researchers should take into consideration that initial quality ratings are highly dependent on the original context of the production and use of the data. This does not strip them of their value; rather, it pushes new users to consider the situated nature of data production and make full use of the results of data curation. In addition, as we have seen, viewing quality ratings in context-independent terms is problematic, considering the evidential underdetermination of quality assessment tools, their low operability, and failures at being used in research practice.<ref name="LeonelliGlobal17" /> More generally, in contemporary and data-intensive research, a single dataset is often used as different types of evidence, depending on how the dataset is analyzed, interpreted, and used. For example, a dataset about dietary practices of a population could be used as evidence for diverse types of studies on, e.g., food practices and culinary culture, socioeconomic status and the [[epidemiology]] of obesity, etc. As a consequence, data quality could similarly differ depending on this use: what counts as a certain level of quality for a research project in epidemiology might be different in the case of research on food history.
 
===Reproducibility===
The second case relates to the various issues connected to the reproducibility of research. The possibility of reproducing results has traditionally been considered a crucial feature of the scientific method, as a way of ensuring that experimental design and methods can be tested to deliver the same results of the original study and therefore prove the reliability of the researcher. As famously argued by Karl Popper, “non-reproducible single occurrences are of no significance to science.”<ref name="PopperTheLog59">{{cite book |title=The Logic of Scientific Discovery |author=Popper, K. |publisher=Routledge |page=64 |year=1959}}</ref> The issue has recently gained prominence in meta-scientific discussions as a result of the failures to reproduce various studies in different areas of the sciences.<ref name="NatureChallenges18">{{cite web |url=https://www.nature.com/collections/prbfkwmwvz/ |title=Challenges in irreproducible research |work=Nature |date=18 October 2018 |accessdate=19 May 2020}}</ref> These and other issues have therefore been framed in terms of a “reproducibility crisis” of the sciences, which has been attributed to a number of problems and features of contemporary research management and practice, such as failures in quality checks, lack of transparency of research methods and data, cognitive and other forms of bias, and the problematic use of, e.g., p-hacking, etc.<ref name="RomeroPhilo19">{{cite journal |title=Philosophy of science and the replicability crisis |journal=Philosophy Compass |author=Romero, F. |volume=14 |issue=11 |at=e12633 |year=2019 |doi=10.1111/phc3.12633}}</ref>
 
The reproducibility crisis is usually connected to discussions on a general decrease in the quality of scientific research and in particular the quality of research outputs, such as specific types of data. This is where the reproducibility and quality of data can be connected, in the sense that reproducibility is often taken to be an overarching epistemic value of science and, among other things, a sign of the quality of data.<ref name="IoannidisImp13">{{cite journal |title=Implausible results in human nutrition research |journal=BMJ |author=Ioannidis, J.P.A. |volume=347 |at=f6698 |year=2013 |doi=10.1136/bmj.f6698 |pmid=24231028}}</ref> In this sense, the contextual approach can be used to critically analyze current discussions on the reproducibility crisis, suggesting that we should look at reproducibility as a contextual property of research practices, as opposed to requiring it as a general indicator of quality that can be equally applied in any research context. Recently, Felipe Romero has argued that reproducibility and replication do not necessarily apply to all of the sciences or work as regulatory ideals and indicators of research quality. What counts as reproducible is actually highly variable and depends on<ref name="RomeroNovely17">{{cite journal |title=Novelty versus Replicability: Virtues and Vices in the Reward System of Science |journal=Philosophy of Science |author=Romero, F. |volume=84 |issue=5 |pages=1031–43 |year=2017 |doi=10.1086/694005}}</ref>:
 
* the research object that is supposed to be reproducible;
* the type, methods, and area of the research; and
* its reliance on and use of statistical tools and the degree of control that can be exerted on the research environment.
 
As a consequence, what reproducibility means, what can and should be reproduced, and the degree to which reproducibility is possible—if at all—change significantly in connection to the specific context of research. With a focus of experimental practices, Uljana Feest has argued that the role of reproducibility is not particularly central<ref name="FeestWhyRep">{{cite journal |title=Why Replication Is Overrated |journal=Philosophy of Science |author=Feest, U. |volume=86 |issue=5 |pages=895–905 |year=2019 |doi=10.1086/705451}}</ref>, while Sabina Leonelli has claimed that reproducibility requirements should be tailored to the contextual features, circumstances, and goals of a specific scientific project or area of research, as well as the assumptions, values, and judgements that are involved in practice.<ref name="LeonelliRethink18">{{cite book |chapter=Rethinking Reproducibility as a Criterion for Research Quality |title=Research in the History of Economic Thought and Methodology: Including a Symposium on Mary Morgan: Curiosity, Imagination, and Surprise |author=Leonelli, S. |editor=Fiorito, L.; Scheall, S.; Suprinyak, C.E. |publisher=Emerald Publishing |volume=36B |pages=129-46 |year=2018 |isbn=9781787564237}}</ref> This direction in the literature on the reproducibility crisis further intersects with the contextual approach so far presented. The push for more documentation, coordination, and transparency of local variants of reproducibility are ways to document the use of data and frame the quality and reproducibility of data and results as contextual properties of research practices. This is also a move beyond the application of general requirements and protocols, which are supposed to apply universally and thus largely independently of the specific contexts of research. As has been previously argued, the contextual approach suggests to move away from viewing quality in universal terms and in the direction of locality and dependency, also in the context of the current debate on reproducibility. At the same time, similarly to the previous discussion on re-use, this brings up the question of how and when to apply reproducibility standards.<ref name="GuttingerTheLim20">{{cite journal |title=The limits of replicability |journal=European Journal for Philosophy of Science |author=Guttinger, S. |volume=10 |at=10 |year=2020 |doi=10.1007/s13194-019-0269-1}}</ref> Here, the contextual approach follows a “local” approach to the issue, meaning that requiring reproducibility as a general standard is highly problematic and discussing the issue on a case-by-case basis is the direction to follow.
 
===Non-experimental data collection===
A third case further explicates the interplay between the previously mentioned universal and local standards, in the context of current critiques of the quality of data collected through non-experimental methods. For instance, the use of observational studies in areas of research such as the life and health sciences has been significantly criticized because of the low degree of quality and reproducibility that they yield.<ref name="JukolaOnThe19" /> In EBM evidential hierarchies, data from observational research are ranked at a very low level of quality compared to clinical studies. In this context, various issues of quality connected with bias, internal and external validity, and reproducibility are condensed into more general critiques of observational data, to the point that the collection and employment of other types of data are often encouraged.<ref name="CanaliEval19">{{cite journal |title=Evaluating evidential pluralism in epidemiology: Mechanistic evidence in exposome research |journal=History and Philosophy of the Life Sciences |author=Canali, S. |volume=41 |at=4 |year=2020 |doi=10.1007/s40656-019-0241-6}}</ref>
 
One of the problems with this line of argument is that this does not do justice to methodologies and traditions of research that employ observational methods and are not primarily experimental, nor can have similar levels of control on the experimental environment and setup; in this sense, these critiques do not consider the contextual nature of data quality. In many areas of the sciences, observational studies are the primary means of data collection and not just on the basis of practical constraints but also for epistemic reasons. For example, in epidemiology, observational data can deliver epistemic goals that would be difficult—if not impossible—to get through experimental studies.<ref name="JukolaOnThe19" /> While one can acknowledge that EBM evidential hierarchies are designed with specific requirements and reasons for classifying types of evidence in the ways they do, the problem is that these hierarchies are often considered to have a universal value and are applied to a variety of different contexts, without considering local and contextual features. Following the contextual approach, the quality of observational data depends on the specific ways and context in which the data are used and cannot be only evaluated on the basis of the methods used to collect the data. Decisions on whether a dataset can be used as a source of high-quality evidence depend on compatibility with other datasets, and their evidential significance is produced and evaluated as part of data practices, not just on the basis of the intrinsic property of a dataset or its compliance with experimental and universal standards.<ref name="JukolaCasuistic">{{cite journal |title=Casuistic Reasoning, Standards of Evidence, and Expertise on Elite Athletes’ Nutrition |journal=Philosophies |author=Jukola, S. |volume=4 |issue=2 |at=19 |year=2020 |doi=10.3390/philosophies4020019}}</ref>
 
==Conclusions==
This essay has focused on quality from a conceptual perspective, with the aim of developing and showcasing a contextual way of approaching data quality for scientific research. The philosophical literature on quality in relation to information, evidence, and data was reviewed, and the results of that review were integrated to highlight a specific way of framing quality in relational, purpose-dependent, and contextual terms. A contextual approach to data quality was then presented, based on the premise that the quality of a dataset is determined by relations between the specific features of a dataset and the questions, aims, and tools employed in the context of the use of data, and that the assessment of the quality of a dataset needs to focus on the features of this context as much as the dataset itself. This approach can effectively make sense of issues and critiques of quality, such as EBM, and promote movements and guidelines concerning data that have recently been put forward, such as FAIR. However, discussions of quality should also take into account specific contexts and be flexible in connection to these contexts, as opposed to setting up categorizations and hierarchies that are intended be applied to all and any contexts of research practices.
 
This is an initial presentation and application of the approach. However, this essay points to future research that looks at the specific aspects of research context that determine quality when data are used, how these may change in different areas of scientific practice and disciplines, and, conceptually, how quality can connect notions of data, evidence, and information.
 
==Acknowledgements==
I am indebted to the graduate research group “Integrating Ethics and Epistemology of Scientific Research” ([https://grk2073.org/ http://grk2073.org]) for all the support during the years of my PhD research and for the feedback received at an early presentation of this material in December 2019. Thanks to two anonymous referees for their helpful feedback and for pushing me to improve the article.


===Funding===
This research was partly funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) Project 254954344/GRK2073, as part of the graduate research group “Integrating Ethics and Epistemology of Scientific Research” and by the Institute of Philosophy of Leibniz University Hannover.


===Conflic of interest===
The author declares no conflict of interest.


==References==
==References==
Line 70: Line 116:
[[Category:LIMSwiki journal articles on data management and sharing]]
[[Category:LIMSwiki journal articles on data management and sharing]]
[[Category:LIMSwiki journal articles on data quality]]
[[Category:LIMSwiki journal articles on data quality]]
[[Category:LIMSwiki journal articles on FAIR data principles]]
[[Category:LIMSwiki journal articles on research]]
[[Category:LIMSwiki journal articles on research]]

Latest revision as of 16:27, 29 April 2024

Full article title Towards a contextual approach to data quality
Journal Data
Author(s) Canali, Stefano
Author affiliation(s) Leibniz University Hannover
Primary contact Email: stefano dot canali at philos dot uni-hannover dot de
Year published 2020
Volume and issue 5(4)
Article # 90
DOI 10.3390/data5040090
ISSN 2306-5729
Distribution license Creative Commons Attribution 4.0 International
Website https://www.mdpi.com/2306-5729/5/4/90/htm
Download https://www.mdpi.com/2306-5729/5/4/90/pdf (PDF)

Abstract

This essay delves into the need for a framework for approaching data quality in the context of scientific research. First, the concept of "quality" as a property of information, evidence, and data is presented, and research on the philosophy of information, science, and biomedicine is reviewed. Based on this review, the need for a more purpose-dependent and contextual approach to data quality in scientific research is argued, whereby the quality of a dataset is dependent on the context of use of the dataset as much as the dataset itself. The rationale to the approach is then exemplified by discussing current critiques and debates of scientific quality, thus showcasing how data quality can be approached contextually.

Keywords: research data management, scientific epistemology, data quality, FAIR, reproducibility crisis

Introduction

Determining the quality of scientific data is a task of key importance for any research project and involves considerations at conceptual, practical, and methodological levels. The task has arguably become even more pressing in recent years, as a result of the ways in which the volume, variety, value, volatility, veracity, and validity of scientific data have changed with the rise of data-intensive methods in the sciences.[1] At the start of the last decade, many commentators argued that these changes would bring dramatic shifts to the scientific method and would per se make science better, thanks to fully automated reasoning, more data-driven methods, less theorizing, and more objectivity.[2] However, analyses of the use of data-intensive methods in the sciences have shown that the feasibility and benefits of these methods are not automatic results of these changes, but crucially rest upon the transparency, validity, and quality of data practices.[3] As a consequence, there are currently various attempts at implementing guidelines to maintain and promote the quality of datasets, developing ways and tools to measure it, and conceptualizing the notion of quality.[4][5][6]

This essay focuses on the latter line of research and discusses the following question: what are high-quality data? At the essay's core is a framework for data quality that suggests a contextual approach, whereby quality should be seen as a result of the context where a dataset is used, and not only of the intrinsic features of the data. This approach is based on the integration of philosophical discussions on the quality of data, information, and evidence. The next section begins by reviewing analyses of quality in different areas of philosophical research, particularly in the philosophy of information, science, and biomedicine. Then, shared results from this review are identified and integrated, with those results arguably pointing towards the need for a contextual approach. A discussion of what the approach entails and how it can be used in practice follows, looking at current debates on quality in the scientific and philosophical literature. Finally, in the conclusion, a discussion of the commentary is made and future research is proposed.

Quality as a property of information, data, and evidence

Quality has been discussed in areas of philosophical work highly engaged with research practices and debates in the sciences.[7] In this context, three main areas of research were identified, whose results are particularly significant for conceptualizations of quality and yet have only partially been applied to issues in data quality. These results and their integration as important contributions for more general and interdisciplinary discussions on data quality are worthy of discussion. As such, this essay proposes that quality can be discussed as a property of three closely related notions: information, data, and evidence.

Information

First, research on quality has traditionally focused on information quality, which became prominent in computer science in the 1990s. In this context, an influential line of research started to move beyond traditional interpretations of quality in terms of solely accuracy, developing a multi-dimensional and purpose-dependent view whereby a piece of information is of high quality insofar as it is fit for a certain purpose.[8] This line of research has developed into two main approaches since the 1990s: surveying opinions and definitions of academics and practices from an “empirical” point of view; and studying the different dimensions of quality and interrelations between these from a theoretical and “ontological” perspective.[9] The empirical approach has expanded conceptualizations of information quality to include not only traditional dimensions such as accuracy, but also objectivity, completeness, relevance, security, access. and timeliness; here, the goal has primarily been to categorize these dimensions, rather than to define them.[10] On the other hand, the goal of the ontological approach has been to understand how to connect different dimensions of information quality (such as those surveyed through the empirical approach[11]) and conceptualize and measure potential disconnections as errors.[12]

These discussions have been picked up and analyzed in the area of research known as "philosophy of information." According to Phyllis Illari and Luciano Floridi, computer science has not fully embraced the purpose-dependent approach to information quality in all of its implications, and theoretical understandings of information quality are still in search of a way of applying the approach to concrete contexts.[6] With these problems and goals in mind, Illari has suggested that information quality suffers from a "rock-and-a-hard-place" problem.[13] While information quality is defined as information that is fit for purpose, many still think that some aspects and dimensions of information quality should be independent of specific purposes (the rock). At the same time, there is a sense in which quality should make information fit for multiple if not all purposes; a piece of information that is fit for a specific purpose, but not for others, will not be considered of high quality (the hard place). As a way of going beyond the impasse, Illari has argued that we should classify information quality on the basis of a relational model, which links the different dimensions of quality to specific purposes and uses.[13] Therefore, Illari conceives of quality as a property of information that is highly dependent on its context, i.e., the specific uses, aims, and purposes we want to employ a piece of information for. In other words, quality cannot be independent of fit for a specific purpose and cannot consist in a single fit-for-any purpose.

Data

Here, a similar push for the purpose-dependent and contextual approach has been identified in a second area of philosophical analyses, which have more specifically focused on the use of data in the context of scientific practice. The increasing volume and variety of data used in the sciences—with related and different levels of veracity, validity, volatility, and value—have created a number of potential benefits as well as challenges for scientific epistemology.[14] Determining and assessing quality is one of the main challenges of data-intensive science because of the diversity of sources of data and integration practices, the often short “timespan” and relevance of data, the difficulties of providing quality assessments and evaluations in a timely manner, and the overall lack of unified standards.[4]

Partly as a result of these shifts, philosophers of science have recently expanded their focus on data as an important component of scientific epistemology.[15] In this context, some analyses have focused on the tools that are used to calibrate, standardize, and assess the quality of data in the sciences. For instance, data quality assessment tools are often applied to clinical studies, in the form of scales or checklists about specific aspects of the study, with the goal of checking whether the study, e.g., makes use of specific statistical methods, sufficiently describes subject withdrawal, etc. According to Jacob Stegenga, there are two main issues affecting the use of these tools in the biomedical context: a poor level of inter-rating operability, i.e., different users of the tools achieve different instead of similar results; and a low level of inter-tool operability, i.e., different types of tools give different instead of similar results when assessing the same study.[16] Stegenga has argued that this can be conceptualized as a result of the underdetermination of the evidential significance of data: there is no uniquely correct way of estimating information quality, and different results will always be obtained in relation to the context, users, and type of study. These results can be interpreted in similar terms to the aforementioned analysis by Illari[13], as pointing to the crucial role that the context where data are analyzed and used plays in determination of its quality. Quality is not an intrinsic property of data that only depends on the characteristics of the data itself: quality will differ depending on contextual features, such as the tools used to assess quality, who uses them, their purposes, etc.

Further support for this point comes from Sabina Leonelli’s studies of data practices—particularly assessment methods—in the life sciences.[17] Leonelli has argued that existing approaches to data quality assessment mostly fail at delivering on their objectives or being actually used in standard practice, to the point that, currently, new and more recently developed technologies and techniques of data collection are used as unofficial markers for data quality. This leads to a problematic situation for the following reasons. Using technologies as markers of quality creates problematic relations with industry, whose economic interests in pushing specific and new technologies do not necessarily align with the epistemic aims of research communities. In particular, when quality standards are locked in and tied to specific technologies, researchers without access to those technologies cannot meet those standards. In this way, using technologies as proxies reduces diversity by creating systematic disadvantages towards researchers who have little access to the latest technologies, often excluding their contributions. To overcome these issues, Leonelli has argued for a different approach to quality: the quality of data is determined by the alignment and relations between data and other components of scientific research, including not only technologies but also research questions, methods, and infrastructures. This can be interpreted as another point for a purpose-driven and local approach to quality, which takes into account the contextual features of data use as much as the intrinsic characteristics of the data themselves.

These discussions align with other and close areas of philosophical research, which are focused on the history and epistemology of experimentation[18][19] and the role of measurement practices, concepts, and quantity terms.[20] In this context, measurement has been discussed as an inferential process that starts from instrument indications and results in outcomes, in the form of claims about the status of the object that is measured. In this sense, Bas van Fraassen has interpreted measurement outcomes as regions of the space of possible values identified by measurement practices, whose dependence on theory is involved at the stage of the interpretation of the outcomes as much as for their capacity of representing the objects of interest.[21] More recently, Luca Mari has argued that measurement should be discussed as a form of information gathering; on this basis, measurement and standardization practices should be seen as producers of knowledge, and their quality can be measured as the quality of the types of knowledge they produce.[22] In this direction, standardization is a type of modeling, whereby the calibration of measurement as a system of practices and conceptualizations is obtained by the specific modeling and representation of the elements involved in a specific context of those measurements.[23][24]

Evidence

The third line of philosophical research discussed here has focuses on quality as a property of scientific evidence, especially in the biomedical context. This research has partly been a reaction to the rise of evidence-based medicine (EBM), an approach to medical research and practice that is based on a specific categorization and ranking of evidence. Since the 1980s, as a movement to reform medical practice and research, EBM has aimed to improve decision-making by removing the influence of subjective preferences from different stages of the process. As formulated by Sackett and colleagues, the central idea of EBM has been “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.”[25] Practically, EBM proponents have introduced “evidence hierarchies,” which describe the assumed quality of different types of evidence and are supposed to help decision makers to project some order in the available evidence.[26] This order aligns better support for the efficacy of different interventions with better evidence types, which in the EBM context consists of evidence from randomized controlled trials or systematic reviews and meta-analyses of randomized controlled trials.

Philosophers of medicine have analyzed and criticized various tenets of EBM, including the theoretical and methodological basis of the choice of specific types of evidence as high-quality evidence[27], the exclusion and denigration of some types of evidence[28], and the ways in which hierarchies of evidence are delineated in evidence-based approaches.[29] While these analyses have not explicitly taken issue with notions of quality per se, their results are significant for this discussion on how to approach data quality. The ways in which evidence is classified and its quality is assessed in EBM seem to apply an intrinsic and universalistic approach to evidence, whereby, e.g., evidence collected through randomized controlled trials (RCTs) is “gold standard.” This means that RCTs are normally given the highest level of quality, although this may be lowered in case of methodological problems; instead, evidence from other methods such as observational studies could be ranked as high-quality, but are automatically given a lower rank as the starting point.[30] In other words, certain methods are considered to be prima facie and epistemically superior, with a gold-like, higher value compared to alternatives.[31] The problem with this approach to the classification of evidence quality is that it is applied to most areas of biomedical research, with no consideration for specifics and different research contexts. In many areas of biomedical research, the gold standard evidence hailed by EBM often cannot be produced, but this does not necessarily mean that the evidence produced is of low quality. For example, Saana Jukola has shown that in nutrition research, RCTs cannot be conducted because of practical, ethical, and methodological aspects of this line of research.[32] Differently from the EBM approach, the quality of biomedical evidence is used to meet specific rather than universal hierarchies, depending on the aims and the context in which it is to be used.

Developing a contextual approach to data quality

Where does the previous review leave us? It appears that discussions of quality in relatively distinct areas of the philosophical literature can be integrated into an overarching approach to quality in the context of scientific data, which has contextuality as its core principle. According to this approach, quality is a contextual feature of data: it is a result of the relations established between a dataset and the questions, aims, and tools employed in the context of the use of data. However, the assessment of the quality of a dataset needs to focus on the features of this context as much as the dataset itself.

As such, the contextual approach is a development of contemporary accounts of data and information. Leonelli has defended a view of scientific data according to which data is a relational entity, whose evidential value is not a given and intrinsic component, but is rather a result of the relations established between the questions, claims, and purposes involved in scientific practices and the objects that are used as data.[15] Similarly, Floridi has discussed information in relational terms, according to which something counts as information only for a certain type of agent and use—to the point that misinformation does not count as information for an agent interested in the production of knowledge.[33]

More specifically, the main difference-maker that determines the relevant features and dimensions of data quality in a specific context is the use of data, including the goals, assumptions, chains of inference, and evidential reasoning involved. Depending on the features of the context where a dataset is used, some dimensions will become more important than others. For example, in cases of urgency and the need to use data for immediate policy measures, the timeliness and accuracy of data might be preferred over its completeness. In turn, the context of use is shaped by the evidential reasoning that determines the use of a dataset as a representation of certain phenomena and is thus based on various chains of inference and the mobilization of other evidence and knowledge, as well as specific assumptions, warrants, and goals.[34] This means that, for instance, when mobilizing and integrating different datasets, users might determine and assess the quality of a dataset for its relevance and compatibility with available evidence, as opposed to its objectivity. Therefore, in a specific context, the quality of a dataset is determined by the use of the dataset and in particular the alignment of its dimensions with the contextual properties of use. In this sense, conceptualizing the quality of a dataset in contextual terms is a move beyond seeing quality as a “static” and universal component of datasets, which can be determined independently and on the basis of their intrinsic characteristics only.

At the same time, emphasizing the role of context entails that quality will have to be assessed differently in different contexts. Yet, this does not mean that the categorization of quality into different dimensions and components is subjective, or that the development of approaches, tools, and standards for the evaluation of data quality is unnecessary. These need to be encouraged as attempts at developing more local and situated approaches to quality, which includes critically evaluating the use of data and explicitly reflecting on its relation with specific dimensions of quality. In this sense, the contextual approach indicates quality criteria and assessment approaches according to which:

  • the specific elements and dimensions of quality will be different depending on the features and goals of data use;
  • some dimensions will be more important than others in different contexts; and,
  • since each dimension of quality needs a specific type of measurement, different measurement tools and techniques will be used in different contexts.

The contextual approach in practice

The contextual approach is a way of conceptualizing quality as a relation of scientific data with other components of a research context and therefore thinking about how quality assessment should be structured and implemented. But how does the approach translate in research practice? Three cases follow, exemplifying where the contextual approach can be seen in practical areas of research and showing how the approach can work in the context of specific applications, as well as noting the issues and different directions the approach points to.

FAIR data priniciples

This first case addresses one of the current attempts at improving the quality of research, by making data FAIR, i.e., findable, accessible, interoperable. and reusable.[5] The FAIR data principles aim to raise the quality of data collected and produced as a result of research practices and related analytic and processing tools. Both this specification of FAIR's four components and the contribution of the FAIR movement can be informed and interpreted from the perspective of the contextual approach to data quality. First, the four FAIR guiding principles can be conceptualized as dimensions of quality as an overarching (desired) feature of a dataset; e.g., accessibility was already present in some of the categorizations of quality in the information quality debates of the 1990s.[8] Second, as such, the guiding principles presented by FAIR are contextual dimensions of quality. Namely, the findability, accessibility, interoperability, and reusability of a dataset are features that determine the quality of a dataset, yet arise and can only be evaluated in specific contexts. Whether a dataset is FAIR will depend on the specific features of a dataset, and indeed the FAIR guiding principles are presented with concrete suggestions for how to deliver these principles. For instance, in this context, the findability of a dataset can be achieved through, e.g., metadata that can assign a unique identifier to the dataset, describe it, and are registered in searchable resources and repositories. The digital object identifier (DOI) is a good example of a way in which the findability of publications and datasets can be applied and their quality therefore improved as a result.

Worthy of highlighting here is that these elements, such as metadata and repositories, pertain to the context of a dataset, rather than the intrinsic properties of the dataset itself. The evaluation of the findability of a dataset comes down to these elements and, more generally, the context in which the data are actually used, as much as of the dataset in itself. Certainly, in individual cases, a dataset will either be found or not, but the realization of findability as a property of a dataset and a dimension of its quality will be due to the specific context where the use of a dataset is realized. As such, the FAIR guidelines largely instantiate the approach to data quality presented in this essay. At the same time, seeing the features indicated by FAIR in contextual terms also suggests that they will be highly dependent on the context of the specific sciences where the data are used, which means that their meaning and application will change, as research practices are different. For instance, although findability is a relatively generalizable feature, whether a dataset is findable by researchers is not an absolute or intrinsic feature of a dataset that can be easily scaled to any discipline. Different researchers will approach and potentially access a dataset coming from the perspective of their discipline and its specific data, which might be different from those of the context of data productions. This is why the work of data curation is so important in current data-intensive research, because it provides data with information on its original context, and in this way it enables new users to judge its reliability, quality, and relevance for new and different uses.[35]

Another implication of this contextual interpretation of FAIR is that each re-use of data will lead to a different quality assessment and that a dataset used in one research project could have a different level of quality in another. Yet, one might argue that quality ratings need to be independent of context, especially for new users and different communities, who have not used the data yet and want to know about its quality. Is this approach, thus, unhelpful and not applicable to actual research practice? Quality ratings are necessary, in particular for data re-use, but here the contextual approach can give different suggestions. Approaching quality in contextual terms suggests that researchers should take into consideration that initial quality ratings are highly dependent on the original context of the production and use of the data. This does not strip them of their value; rather, it pushes new users to consider the situated nature of data production and make full use of the results of data curation. In addition, as we have seen, viewing quality ratings in context-independent terms is problematic, considering the evidential underdetermination of quality assessment tools, their low operability, and failures at being used in research practice.[17] More generally, in contemporary and data-intensive research, a single dataset is often used as different types of evidence, depending on how the dataset is analyzed, interpreted, and used. For example, a dataset about dietary practices of a population could be used as evidence for diverse types of studies on, e.g., food practices and culinary culture, socioeconomic status and the epidemiology of obesity, etc. As a consequence, data quality could similarly differ depending on this use: what counts as a certain level of quality for a research project in epidemiology might be different in the case of research on food history.

Reproducibility

The second case relates to the various issues connected to the reproducibility of research. The possibility of reproducing results has traditionally been considered a crucial feature of the scientific method, as a way of ensuring that experimental design and methods can be tested to deliver the same results of the original study and therefore prove the reliability of the researcher. As famously argued by Karl Popper, “non-reproducible single occurrences are of no significance to science.”[36] The issue has recently gained prominence in meta-scientific discussions as a result of the failures to reproduce various studies in different areas of the sciences.[37] These and other issues have therefore been framed in terms of a “reproducibility crisis” of the sciences, which has been attributed to a number of problems and features of contemporary research management and practice, such as failures in quality checks, lack of transparency of research methods and data, cognitive and other forms of bias, and the problematic use of, e.g., p-hacking, etc.[38]

The reproducibility crisis is usually connected to discussions on a general decrease in the quality of scientific research and in particular the quality of research outputs, such as specific types of data. This is where the reproducibility and quality of data can be connected, in the sense that reproducibility is often taken to be an overarching epistemic value of science and, among other things, a sign of the quality of data.[39] In this sense, the contextual approach can be used to critically analyze current discussions on the reproducibility crisis, suggesting that we should look at reproducibility as a contextual property of research practices, as opposed to requiring it as a general indicator of quality that can be equally applied in any research context. Recently, Felipe Romero has argued that reproducibility and replication do not necessarily apply to all of the sciences or work as regulatory ideals and indicators of research quality. What counts as reproducible is actually highly variable and depends on[40]:

  • the research object that is supposed to be reproducible;
  • the type, methods, and area of the research; and
  • its reliance on and use of statistical tools and the degree of control that can be exerted on the research environment.

As a consequence, what reproducibility means, what can and should be reproduced, and the degree to which reproducibility is possible—if at all—change significantly in connection to the specific context of research. With a focus of experimental practices, Uljana Feest has argued that the role of reproducibility is not particularly central[41], while Sabina Leonelli has claimed that reproducibility requirements should be tailored to the contextual features, circumstances, and goals of a specific scientific project or area of research, as well as the assumptions, values, and judgements that are involved in practice.[42] This direction in the literature on the reproducibility crisis further intersects with the contextual approach so far presented. The push for more documentation, coordination, and transparency of local variants of reproducibility are ways to document the use of data and frame the quality and reproducibility of data and results as contextual properties of research practices. This is also a move beyond the application of general requirements and protocols, which are supposed to apply universally and thus largely independently of the specific contexts of research. As has been previously argued, the contextual approach suggests to move away from viewing quality in universal terms and in the direction of locality and dependency, also in the context of the current debate on reproducibility. At the same time, similarly to the previous discussion on re-use, this brings up the question of how and when to apply reproducibility standards.[43] Here, the contextual approach follows a “local” approach to the issue, meaning that requiring reproducibility as a general standard is highly problematic and discussing the issue on a case-by-case basis is the direction to follow.

Non-experimental data collection

A third case further explicates the interplay between the previously mentioned universal and local standards, in the context of current critiques of the quality of data collected through non-experimental methods. For instance, the use of observational studies in areas of research such as the life and health sciences has been significantly criticized because of the low degree of quality and reproducibility that they yield.[32] In EBM evidential hierarchies, data from observational research are ranked at a very low level of quality compared to clinical studies. In this context, various issues of quality connected with bias, internal and external validity, and reproducibility are condensed into more general critiques of observational data, to the point that the collection and employment of other types of data are often encouraged.[44]

One of the problems with this line of argument is that this does not do justice to methodologies and traditions of research that employ observational methods and are not primarily experimental, nor can have similar levels of control on the experimental environment and setup; in this sense, these critiques do not consider the contextual nature of data quality. In many areas of the sciences, observational studies are the primary means of data collection and not just on the basis of practical constraints but also for epistemic reasons. For example, in epidemiology, observational data can deliver epistemic goals that would be difficult—if not impossible—to get through experimental studies.[32] While one can acknowledge that EBM evidential hierarchies are designed with specific requirements and reasons for classifying types of evidence in the ways they do, the problem is that these hierarchies are often considered to have a universal value and are applied to a variety of different contexts, without considering local and contextual features. Following the contextual approach, the quality of observational data depends on the specific ways and context in which the data are used and cannot be only evaluated on the basis of the methods used to collect the data. Decisions on whether a dataset can be used as a source of high-quality evidence depend on compatibility with other datasets, and their evidential significance is produced and evaluated as part of data practices, not just on the basis of the intrinsic property of a dataset or its compliance with experimental and universal standards.[45]

Conclusions

This essay has focused on quality from a conceptual perspective, with the aim of developing and showcasing a contextual way of approaching data quality for scientific research. The philosophical literature on quality in relation to information, evidence, and data was reviewed, and the results of that review were integrated to highlight a specific way of framing quality in relational, purpose-dependent, and contextual terms. A contextual approach to data quality was then presented, based on the premise that the quality of a dataset is determined by relations between the specific features of a dataset and the questions, aims, and tools employed in the context of the use of data, and that the assessment of the quality of a dataset needs to focus on the features of this context as much as the dataset itself. This approach can effectively make sense of issues and critiques of quality, such as EBM, and promote movements and guidelines concerning data that have recently been put forward, such as FAIR. However, discussions of quality should also take into account specific contexts and be flexible in connection to these contexts, as opposed to setting up categorizations and hierarchies that are intended be applied to all and any contexts of research practices.

This is an initial presentation and application of the approach. However, this essay points to future research that looks at the specific aspects of research context that determine quality when data are used, how these may change in different areas of scientific practice and disciplines, and, conceptually, how quality can connect notions of data, evidence, and information.

Acknowledgements

I am indebted to the graduate research group “Integrating Ethics and Epistemology of Scientific Research” (http://grk2073.org) for all the support during the years of my PhD research and for the feedback received at an early presentation of this material in December 2019. Thanks to two anonymous referees for their helpful feedback and for pushing me to improve the article.

Funding

This research was partly funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) Project 254954344/GRK2073, as part of the graduate research group “Integrating Ethics and Epistemology of Scientific Research” and by the Institute of Philosophy of Leibniz University Hannover.

Conflic of interest

The author declares no conflict of interest.

References

  1. ↑ Leonelli, S. (2020). "Scientific Research and Big Data". Stanford Encyclopedia of Philosophy Archive (Summer 2020). https://plato.stanford.edu/archives/sum2020/entries/science-big-data/. 
  2. ↑ Canali, S. (2016). "Big Data, epistemology and causality: Knowledge in and knowledge out in EXPOsOMICS". Big Data & Society 3 (2). doi:10.1177/2053951716669530. 
  3. ↑ Leonelli, S. (2014). "What difference does quantity make? On the epistemology of Big Data in biology". Big Data & Society 1 (1). doi:10.1177/2053951714534395. 
  4. ↑ 4.0 4.1 Cai, L.; Zhu, Y. (2015). "The Challenges of Data Quality and Data Quality Assessment in the Big Data Era". Data Science Journal 14: 2. doi:10.5334/dsj-2015-002. 
  5. ↑ 5.0 5.1 Wilkinson, M.D.; Dumontier, M.; Aalbersberg, I.J. et al. (2016). "The FAIR Guiding Principles for scientific data management and stewardship". Scientific Data 3: 160018. doi:10.1038/sdata.2016.18. PMC PMC4792175. PMID 26978244. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4792175. 
  6. ↑ 6.0 6.1 Illari, P.; Floridi, L. (2014). "Chapter 2: Information Quality, Data and Philosophy". In Floridi, L., Illari, P.. The Philosophy of Information Quality. Springer International Publishing. pp. 5–23. doi:10.1007/978-3-319-07121-3. ISBN 9783319071213. 
  7. ↑ Boumans, M.; Lenelli, S. (2013). "Introduction: On the Philosophy of Science in Practice". Journal for General Philosophy of Science 44: 259–61. doi:10.1007/s10838-013-9232-6. 
  8. ↑ 8.0 8.1 Wang, R.Y.; Reddy, M.P.; Kon, H.B. (1995). "Toward quality data: An attribute-based approach". Decision Support Systems 13 (3–4): 349–72. doi:10.1016/0167-9236(93)E0050-N. 
  9. ↑ Wang, R.Y. (1998). "A product perspective on total data quality management". Communications of the ACM 41 (2): 58–65. doi:10.1145/269012.269022. 
  10. ↑ Batini, C.; Scannapieca, M. (2006). Data Quality: Concepts, Methodologies and Techniques. Springer. ISBN 9783540331728. 
  11. ↑ Wand, Y.; Wang, R.Y. (1996). "Anchoring data quality dimensions in ontological foundations". Communications of the ACM 39 (11): 86-95. doi:10.1145/240455.240479. 
  12. ↑ Primiero, G. (2014). "Chapter 7: Algorithmic Check of Standards for Information Quality Dimensions". In Floridi, L., Illari, P.. The Philosophy of Information Quality. Springer International Publishing. pp. 107–34. doi:10.1007/978-3-319-07121-3. ISBN 9783319071213. 
  13. ↑ 13.0 13.1 13.2 Illari, P. (2014). "Chapter 14: IQ: Purpose and Dimensions". In Floridi, L., Illari, P.. The Philosophy of Information Quality. Springer International Publishing. pp. 281–301. doi:10.1007/978-3-319-07121-3. ISBN 9783319071213. 
  14. ↑ Leonelli, S.; Tempini, N. (2020). Data Journeys in the Sciences. Springer. doi:10.1007/978-3-030-37177-7. ISBN 9783030371777. 
  15. ↑ 15.0 15.1 Leonelli, S. (2016). Data-Centric Biology: A Philosphical Study. University of Chicago Press. ISBN 9780226416502. 
  16. ↑ Stegenga, J. (2013). "Down with the Hierarchies". Topoi 33: 313–22. doi:10.1007/s11245-013-9189-4. 
  17. ↑ 17.0 17.1 Leonelli, S. (2017). "Global Data Quality Assessment and the Situated Nature of “Best” Research Practices in Biology". Data Science Journal 16: 32. doi:10.5334/dsj-2017-032. 
  18. ↑ Hacking, I. (1993). Representing and Intervening: Introductory Topics in the Philosophy of Natural Science. Cambridge University Press. doi:10.1017/CBO9780511814563. ISBN 9780511814563. 
  19. ↑ Rheinberger, H.-J. (2010). An Epistemology of the Concrete: Twentieth-Century Histories of Life. Duke University Press. ISBN 9780822345756. 
  20. ↑ Chang, H.; Cartwright, N. (2008). "Chapter 34: Measurement". In Curd, M.; Psillos, S.. The Routledge Companion to Philosophy of Science (1st ed.). Routledge. pp. 367–75. doi:10.4324/9780203000502. ISBN 9780203000502. 
  21. ↑ van Fraassen, B.C. (2008). Scientific Representation: Paradoxes of Perspective. Oxford University Press. doi:10.1093/acprof:oso/9780199278220.001.0001. ISBN 9780199278220. 
  22. ↑ Mari, L. (2003). "Epistemology of measurement". Measurement 34: 17–30. doi:10.1016/S0263-2241(03)00016-2. 
  23. ↑ Boumans, M. (2007). "Chapter 9: Invariance and Calibration". In Boumans, M.. Measurement in Economics: A Handbook. Academic Press. pp. 231–47. ISBN 9780123704894. 
  24. ↑ Tal, E. (2013). "Old and New Problems in Philosophy of Measurement". Philosophy Compass 8 (12): 1159–73. doi:10.1111/phc3.12089. 
  25. ↑ Sackett, D.L.; Rosenberg, W.M.C.; Gray, J.A.M. et al. (1996). "Evidence based medicine: What it is and what it isn't". BMJ 312: 71. doi:10.1136/bmj.312.7023.71. 
  26. ↑ Bluhm, R. (2005). "From hierarchy to network: A richer view of evidence for evidence-based medicine". Perspectives in Biology and Medicine 48 (4): 535–47. doi:10.1353/pbm.2005.0082. PMID 16227665. 
  27. ↑ Worrall, J. (2002). "What Evidence in Evidence‐Based Medicine?". Philosophy of Science 69 (S3): S316–30. doi:10.1086/341855. 
  28. ↑ Clarke, B.; Gillies, D.; Illari, P. et al. (2013). "Mechanisms and the Evidence Hierarchy". Topoi 33: 339–60. doi:10.1007/s11245-013-9220-9. 
  29. ↑ Campaner, R.; Galavotti, M.C. (2012). "Evidence and the Assessment of Causal Relations in the Health Sciences". International Studies in the Philosophy of Science 26 (1): 27–45. doi:10.1080/02698595.2012.653113. 
  30. ↑ Kerry, R.; Eriksen, T.E.; Lie. S.A.N. et al. (2012). "Causation and evidence‐based practice: An ontological review". Journal of Evaluation in Clinical Practice 18 (5): 1006–12. doi:10.1111/j.1365-2753.2012.01908.x. 
  31. ↑ Stegenga, J. (2011). "Is meta-analysis the platinum standard of evidence?". Studies in History and Philosophy of Science Part C 42 (4): 497–507. doi:10.1016/j.shpsc.2011.07.003. 
  32. ↑ 32.0 32.1 32.2 Jukola, S. (2019). "On the evidentiary standards for nutrition advice". Studies in History and Philosophy of Science Part C 73: 1–9. doi:10.1016/j.shpsc.2018.05.007. 
  33. ↑ Floridi, L. (2011). The Pholosophy of Information. University of Oxford Press. doi:10.1093/acprof:oso/9780199232383.001.0001. ISBN 9780199232383. 
  34. ↑ Canali, S. (2020). "Making evidential claims in epidemiology: Three strategies for the study of the exposome". Studies in History and Philosophy of Science Part C 82: 101248. doi:10.1016/j.shpsc.2019.101248. 
  35. ↑ Leonelli, S. (2009). "On the Locality of Data and Claims about Phenomena". Philosophy of Science 76 (5): 737–49. doi:10.1086/605804. 
  36. ↑ Popper, K. (1959). The Logic of Scientific Discovery. Routledge. p. 64. 
  37. ↑ "Challenges in irreproducible research". Nature. 18 October 2018. https://www.nature.com/collections/prbfkwmwvz/. Retrieved 19 May 2020. 
  38. ↑ Romero, F. (2019). "Philosophy of science and the replicability crisis". Philosophy Compass 14 (11): e12633. doi:10.1111/phc3.12633. 
  39. ↑ Ioannidis, J.P.A. (2013). "Implausible results in human nutrition research". BMJ 347: f6698. doi:10.1136/bmj.f6698. PMID 24231028. 
  40. ↑ Romero, F. (2017). "Novelty versus Replicability: Virtues and Vices in the Reward System of Science". Philosophy of Science 84 (5): 1031–43. doi:10.1086/694005. 
  41. ↑ Feest, U. (2019). "Why Replication Is Overrated". Philosophy of Science 86 (5): 895–905. doi:10.1086/705451. 
  42. ↑ Leonelli, S. (2018). "Rethinking Reproducibility as a Criterion for Research Quality". In Fiorito, L.; Scheall, S.; Suprinyak, C.E.. Research in the History of Economic Thought and Methodology: Including a Symposium on Mary Morgan: Curiosity, Imagination, and Surprise. 36B. Emerald Publishing. pp. 129-46. ISBN 9781787564237. 
  43. ↑ Guttinger, S. (2020). "The limits of replicability". European Journal for Philosophy of Science 10: 10. doi:10.1007/s13194-019-0269-1. 
  44. ↑ Canali, S. (2020). "Evaluating evidential pluralism in epidemiology: Mechanistic evidence in exposome research". History and Philosophy of the Life Sciences 41: 4. doi:10.1007/s40656-019-0241-6. 
  45. ↑ Jukola, S. (2020). "Casuistic Reasoning, Standards of Evidence, and Expertise on Elite Athletes’ Nutrition". Philosophies 4 (2): 19. doi:10.3390/philosophies4020019. 

Notes

This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.

Retrieved from "https://www.limswiki.org/index.php?title=Journal:Towards_a_contextual_approach_to_data_quality&oldid=63370"