Journal:Designing a knowledge management system for naval materials failures

From LIMSWiki
Jump to navigationJump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
Full article title Designing a knowledge management system for naval materials failures
Journal MATEC Web of Conferences
Author(s) Melanitis, Nikolaos; Giannakopoulos, George; Stamatakis, Konstantinos; Mouzakis, Dionysios; Koutsomichalis, Aggelos
Author affiliation(s) Hellenic Naval Academy, Institute of Informatics & Telecommunications
Primary contact Email: melanitis at hna dot gr
Year published 2021
Volume and issue 349
Article # 03006
DOI 10.1051/matecconf/202134903006
ISSN 2261-236X
Distribution license Creative Commons Attribution 4.0 International
Website https://www.matec-conferences.org/articles/matecconf/abs/2021/18/matecconf_iceaf2021_03006/matecconf_iceaf2021_03006.html
Download https://www.matec-conferences.org/articles/matecconf/pdf/2021/18/matecconf_iceaf2021_03006.pdf (PDF)

Abstract

Implemented materials fail from time to time, requiring failure analysis. This type of scientific analysis expands into forensic engineering for it aims not only to identify individual and symptomatic reasons for failure, but also to assess and understand repetitive failure patterns, which could be related to underlying material faults, design mistakes, or maintenance omissions. Significant information can be gained and studied from carefully documenting and managing the data that comes from failure analysis of materials, including in the naval industry.

The NAVMAT research project, presented herein, attempts an interdisciplinary approach to materials informatics by integrating materials engineering and informatics under a platform of knowledge management. Our approach utilizes a focused, common-cause failure analysis methodology for the naval and marine environment. The platform's design is dedicated to the effective recording, efficient indexing, and easy and accurate retrieval of relevant information, including the associated history of maintenance and secure operation concerning failure incidents of marine materials, components, and systems in an organizational fleet. Based on a materials failure ontology, utilizing artificial intelligence (AI) algorithms and modern approaches to data handling, NAVMAT aims to optimize naval materials failure analysis management and support decision making in maintenance and repair operations (MRO), materials supply management, and staff training. It will eventually support decision making through appropriate AI and natural language processing methods.

Keywords: materials informatics, failure analysis, forensic engineering, naval and marine materials, information management

Introduction: Addressing the need

Some of the most important considerations in materials management and the necessary failure analysis of those materials for a large organization, industry, or service provider include the meticulous recording and indexing of an incident, the assessment and identification of causes, and the proposal of feasible solutions. To best handle these considerations, quick access to and retrieval of knowledge is a prerequisite. The related information is recorded at the platform level, but its integration at a central level and redistribution as knowledge and guidelines is not an easy process. As a result, problem solving operations may repeat when the same or similar problem appears in a different area of the organization, resulting in human and financial resources being “wasted” for re-interpreting the incident and operational readiness potentially being compromised. The disruption in the flow of knowledge (i.e., the knowledge gap) may be further intensified when taking into account:

  • The geographically distributed nature of platforms and units of a large organization (such as a naval fleet). Direct and seamless communication and knowledge transfer is a mark of efficiency.
  • The career path of staff, which are frequently transferred and reallocated (as with the military). The job requirements of the new post might be completely different, and the original knowledge acquired is not easily passed on during the traditional transfer of information from the predecessor to the newcomer.
  • The early retirement of individuals due to the specific characteristics of some professions (such as the navy officer). This may deprive the organization of critical knowledge transfer. Institutional and legal obstacles may further impede knowledge transfer from retired to active personnel.

These types of issues arise in the maritime trades. For example, a marine engine shaft failure (together with the knowledge acquired by the staff involved in failure analysis and the resulting repairs), remains a faint trace of information in the memory of the ship, when after three years, all staff may have been re-allocated.

To cope with the above challenges, we have developed, promoted, and established NAVMAT, a platform for the recording, indexing, comparing, assessment, and retrieval of all information relevant to failure analysis in the defined environment, including history of operations and maintenance, scientific evidence, and testimony. NAVMAT accounts for the variability of language use when reporting, together with the multi-modal assets attached to such events (e.g., measurements, photos, etc.), fulfilling the requirement for an intelligent system beyond simple keyword-based indexing and retrieval. Its design is based on an interdisciplinary approach to materials informatics by integrating materials engineering and informatics under a platform of knowledge management.

State of the art

Knowledge management has been linked to artificial intelligence (AI) for more than 20 years. Such management has a strong organizational component, beyond technical requirements. Early on, the challenge of forming a knowledge management culture within an organization was reported to be estimated as “90% of the effort” by business leaders.[1] Today, with the increased digitization of business and the transfer of information in everyday practice, the cultural gaps have been somewhat reduced. However, a number of settings may still have only limited access to effective, user-friendly knowledge management systems. As such, and despite the fact that today we discuss and implement concepts such as Industry 4.0, the internet of things (IoT)[2], and condition monitoring and diagnostic engineering management (COMADEM)[3], even reactive (vs. proactive) use of pre-existing knowledge at the human expert level for material failures is still an open challenge.[4]

In the literature, one can find approaches and systems that utilize data mining and analysis of text in manufacturing to improve “maintenance knowledge intelligence.”[5] Through analyses of maintenance reports, operators’ workbooks, and digital logbooks, such systems aim to identify hidden patterns and relations between settings, incidents, and human reactions. These approaches, however, may act in an a posteriori manner, not facilitating the user in the data entry process (as NAVMAT does). Approaches using expert systems have also been applied in the past[6] for settings such as onshore pipelines. However, again the focus is on the post-entry analysis and organization of documents.

Other approaches focus on the analysis of machine maintenance and use in a cloud setting to estimate the remaining useful life of components.[7] In such cases, however, the aim is to optimize predictors and not to allow for use and re-use of human knowledge as part of a collaborative system.

The development of a knowledge management system for automobile system failures[8] was founded of the creation of a failure analysis ontology from maintenance experience, which allows connecting the pre-existing human understanding of the domain to the knowledge system at hand. In the NAVMAT setting, we utilize a similar view of preexisting knowledge by initializing a related ontology in the lexical and semantic resource components, which can evolve over time based on everyday system use.

Case-based reasoning techniques are considered to be the most suitable for generic failure analysis due to the complexity of knowledge required.[9] Such techniques have been employed to support users in hard disk failure analysis[10], using a reverse-index based approach (e.g., Lucene.NET[11]) and a cosine-similarity metric to identify similar use cases. In our setting, we take advantage of the expert ontology to not only allow for intelligent and agile search and retrieval, but also to facilitate [12]data entry.

In the medical field, the need for image interpretation (e.g., in bone fractures) has triggered the development of algorithms based on artificial neural network (ANN) architectures.[12] Such systems may involve unsupervised or supervised learning.[13][14] We see similar potential in our implementation.

Given these advancements, NAVMAT aims to provide a point of reference for human-centric, intelligent knowledge management of naval materials failure, throughout the full lifecycle of such information, from knowledge encoding to incident data gathering and entry, and then to its search and reuse for decision support.

The concept and tools

The main scientific challenges in NAVMAT reside in (a) knowledge engineering and (b) information retrieval and summarization. Knowledge engineering aims to develop the underlying common understanding of terms, senses, and their relations. This, in turn, supports concise definition and storage of incidents. Finally, indexing, retrieval, and summarization can be empowered by these definitions and the underlying ontology knowledge to fine-tune incident search and retrieval, allowing users to better focus on the desired information.

Converging to a common vocabulary and set of relations between terms is critical: such (meta-) information guides storage, indexing, and retrieval, touching all main aspects of the system. The ontology encapsulates concepts of the systems, processes, and security setting, forming a universally understood language within the design. The process-centric approach in the creation and evolution of the system ontology[15][16] uses ontology engineering patterns where appropriate.[17] Established tools, such as Protégé, are employed to support the editing and evolution of the ontology over time till convergence, providing (a) support for editing ontologies, (b) full change tracking and revision history, (c) collaboration tools, and (d) multiple file formats supported for upload and download of ontologies (RDF/XML, Turtle, OWL/XML, OBO, and others).

On the other hand, text-based data mining tools (e.g. keyword extraction methods) are used early on to identify possible terms/keywords and facilitate the ontology creation process dynamically during its evolution. Automatic text classification is employed[18] to suggest tags to new incidents, facilitating the user and the uptake of the system by experts. To this end, the classification takes into account previously labelled incidents as training instances to form a consistent understanding of existing labels. Over time, newly entered incidents and related work will be provided to the system for more training, creating a virtuous circle of improvement.

Efficient indexing and retrieval (cf., Apache Lucene - SolR / Elastic Search) further enhance the process, allowing the user to leverage the strength of the ontology to get meaningful matches beyond keyword and Boolean search.

System architecture

NAVMAT builds its architecture taking into account the main operational workflows in maritime incident management and response. Figure 1 provides a system overview, both summarizing the flow of information and depicting the main system components of NAVMAT (in the NAVMAT Platform box).


Fig1 Melanitis MATECWebConf21 349.png

Figure 1. System overview

In regards to the components in the NAVMAT Platform box:

  • Ontology engineering UI: This component is essentially a front-end for domain experts, allowing them to define the main concepts of the domain and their interrelations (Figure 2, below). This process essentially encodes the domain knowledge in a machine-usable format, exploitable by the "Document analysis" and the "Knowledge-based interface" components.
  • Lexical and semantic resources: These resources combine the output of the "Ontology engineering UI" with other external resources (e.g., dictionaries or pre-existing ontologies/thesauri) to complement the domain knowledge with encoded real-world knowledge such as synonyms lists, established encodings of failures, etc. (Figure 2, below)
  • Document analysis: This component is one of the two main components that brings the power of AI into the NAVMAT platform. It allows the automatic annotation of input documents with labels/tags (i.e., classification) that facilitate the indexing of the documents and summarization of the related information. The resulting tagged documents are then fed to the repository for later retrieval and re-use, as appropriate.
  • Repository: This storage component allows the efficient storage and indexing of documents—and in general resources, e.g., images, audio, and text—that constitute the main body of information within NAVMAT. These documents are input by the "Document analysis" component to be later sought and retrieved by the "Knowledge-based interface" requests. Published[19][20][21][22][23] and unpublished work of some of the authors provide the test-bed and initial content of the system.
  • Knowledge-based interface: This interface allows the users to implement two different workflows. First, incident data entry workflow supports the user when a new incident description is to be added to the system. The component follows the text and documents entered by the user and suggests labels/keywords (provided by the document analysis component) for the efficient indexing of the document. The user can confirm the appropriate labels to finalize the storage. Second, workflows for querying previous data allows an efficient search of previously stored incidents, by means of an appropriate query. This query can be an incident description in itself or a term-/concept-based query. This query results in a number of related pre-entered incidents (from the "Repository" component) which are selected and prioritized to be provided to the user. This flow also essentially utilizes the summary of information per incident to facilitate the user when skimming through all appropriate incidents to find the most related information. Such information is meant to support the user for the final assessment and/or decision, as appropriate.


Fig2 Melanitis MATECWebConf21 349.png

Figure 2. Ontology and taxonomy with the semantic resources for one indicative class (i.e., "Failure Types") of the NAVMAT system

Key features

System strengths

Multi-linguality

Protégé[24][25], a well-established, open-source framework supporting multi-lingual ontology editing, handles the creation and maintenance of ontologies. Various stakeholders have adopted Protégé as its output can be easily integrated with rule systems or other problem solvers to construct a wide range of intelligent systems. Also, in our case, the NAVMAT lexical and semantic resources—a combination of Protégé's output with other possible external resources—supports the analysis and extraction tasks performed by the "Document analysis" component.

Scalability

The NAVMAT platform aims to include various sources of incidents related to materials and systems failure:

  • Previously stored incidents;
  • Relevant publications or technical analyses proposed by our experts (i.e., the system users responsible for the assessment reports); and/or
  • Other public failure incident information and open-access publications, identified and suggested by our semantic content analysis and information extraction component.

Additionally, multi-modal assets, such as documents (e.g., diagnostics, publications, reports) and media files (e.g., photos, videos, audio recordings) can be attached to incidents. Therefore, every single incident of failure can have various sources of supporting documentation.

Adaptivity/personalization

The system provides varying levels of access, supporting a hierarchy of users, with specific permissions and access levels to information according to their role. Additionally, the NAVMAT system intends to enable communication and interaction among its users, as well as to facilitate the management of knowledge and documents related to incidents. As such user interactions and suggestions multiply over time, they will be used as training instances by our expert system, creating a virtuous circle of improvements in user experience.

System interface and security

The web interface of the NAVMAT system essentially refers to one or more thin clients (a web or mobile app). Indicative workflows include:

  • CRUD (create/read/update/delete) operations on reports, incidents, and documents;
  • Request suggestion from the "Document analysis" component concerning main concepts in the text;
  • Enrichment of inserted documents through the "Document analysis" component;
  • Storage of the enriched document into the repository; and
  • Efficient searching (based on ontology) for previous related incidents and/or related resources (e.g., publications, videos, etc.).

Additionally, the system will evaluate the use of well-established practices for sufficient security. Indicatively, this implies multiple things:

  • Use of end-to-end encryption: Protocols such as https will be used to ascertain encrypted, secure communication.
  • Secure login mechanisms: Login could be coupled with improved security mechanisms, such as two-factor authentication.
  • Data isolation: This involves flexibility in deciding on isolated hosting of data per organization vs. cloud data storage.

System potential

The NAVMAT system has the ambition to integrate the currently existing and fast growing body of information concerning the failure of materials and components with decision making on maintenance, repair, overhaul/operations (MRO), using informatics and telecommunication tools. The development of an information system of such nature does not concern only the naval materials value chain, however. The open-source nature of several elements of the system allows a degree of knowledge transfer to other industrial activities that may benefit: in the same thematic field (e.g., other industrial materials failure), similar thematic fields (e.g., forensics), or adjacent thematic fields (e.g., failure of electrical/electronic components).

Expected outcomes

After completion of development; the feeding of the repository with currently available published and unpublished proprietary documents of the partner organizations (beneficiaries); the full operation of the platform for authorized users; and the continuous enrichment of the system with new incidents, assessments and reports; the NAVMAT platform is expected to:

  • Erase disruptions in the flow of knowledge (due to the high mobility of personnel and the distribution of knowledge across the beneficiary organizations);
  • Improve the management of knowledge of critical component failures;
  • Support the decision making in the maintenance and supply of materials for marine environment operations;
  • Train and further educate the technical and scientific personnel of the beneficiary organizations;
  • Benefit operational readiness; and
  • Adopt and explore the potential of the semantic web for enriching information and knowledge beyond the beneficiaries (e.g., provide access to public failure incident information and open-access publications).

Acknowledgements

The research project NAVMAT is supported by the Hellenic Foundation for Research and Innovation (H.F.R.I.) under the “1st Call for H.F.R.I. Research Projects to support Faculty Members & Researchers and the Procurement of High-and the procurement of high-cost research equipment grant” (Project Number: 822).

References

  1. ↑ Liao, Shu-hsien (1 August 2003). "Knowledge management technologies and applications—literature review from 1995 to 2002" (in en). Expert Systems with Applications 25 (2): 155–164. doi:10.1016/S0957-4174(03)00043-5. https://linkinghub.elsevier.com/retrieve/pii/S0957417403000435. 
  2. ↑ Radanliev, Petar; De Roure, David; Walton, Rob; Van Kleek, Max; Montalvo, Rafael Mantilla; Maddox, La’Treall; Santos, Omar; Burnap, Peter et al. (1 November 2020). "Artificial intelligence and machine learning in dynamic cyber risk analytics at the edge" (in en). SN Applied Sciences 2 (11): 1773. doi:10.1007/s42452-020-03559-4. ISSN 2523-3963. https://link.springer.com/10.1007/s42452-020-03559-4. 
  3. ↑ K. Nagaraja Rao, B. (2021). "The Role of Artificial Intelligence (AI) in Condition Monitoring and Diagnostic Engineering Management (COMADEM): A Literature Survey" (in en). American Journal of Artificial Intelligence 5 (1): 17. doi:10.11648/j.ajai.20210501.12. ISSN 2639-9717. http://www.sciencepublishinggroup.com/journal/paperinfo?journalid=542&doi=10.11648/j.ajai.20210501.12. 
  4. ↑ Talamo, C; Paganin, G; Rota, F (1 July 2019). "Industry 4.0 for failure information management within Proactive Maintenance". IOP Conference Series: Earth and Environmental Science 296 (1): 012055. doi:10.1088/1755-1315/296/1/012055. ISSN 1755-1307. https://iopscience.iop.org/article/10.1088/1755-1315/296/1/012055. 
  5. ↑ Ansari, Fazel (1 March 2020). "Cost-based text understanding to improve maintenance knowledge intelligence in manufacturing enterprises" (in en). Computers & Industrial Engineering 141: 106319. doi:10.1016/j.cie.2020.106319. https://linkinghub.elsevier.com/retrieve/pii/S036083522030053X. 
  6. ↑ Castellanos, V.; Albiter, A.; Hernández, P.; Barrera, G. (1 September 2011). "Failure analysis expert system for onshore pipelines. Part – I: Structured database and knowledge acquisition" (in en). Expert Systems with Applications 38 (9): 11085–11090. doi:10.1016/j.eswa.2011.02.153. https://linkinghub.elsevier.com/retrieve/pii/S0957417411003769. 
  7. ↑ Mourtzis, Dimitris; Vlachou, Ekaterini; Milas, Nikolaos; Xanthopoulos, Nikitas (2016). "A Cloud-based Approach for Maintenance of Machine Tools and Equipment Based on Shop-floor Monitoring" (in en). Procedia CIRP 41: 655–660. doi:10.1016/j.procir.2015.12.069. https://linkinghub.elsevier.com/retrieve/pii/S2212827115011488. 
  8. ↑ James, Ajith Tom; Gandhi, O.P.; Deshmukh, S.G. (2 October 2017). "Knowledge management of automobile system failures through development of failure knowledge ontology from maintenance experience" (in en). Journal of Advances in Management Research 14 (4): 425–445. doi:10.1108/JAMR-02-2017-0024. ISSN 0972-7981. https://www.emerald.com/insight/content/doi/10.1108/JAMR-02-2017-0024/full/html. 
  9. ↑ Graham-Jones, P.J; Mellor, B.G (1 June 1995). "Expert and knowledge-based systems in failure analysis" (in en). Engineering Failure Analysis 2 (2): 137–149. doi:10.1016/1350-6307(95)00009-F. https://linkinghub.elsevier.com/retrieve/pii/135063079500009F. 
  10. ↑ Wichawong, Parinya; Chongstitvatana, Prabhas (1 June 2017). "Knowledge management system for failure analysis in hard disk using case-based reasoning". 2017 18th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD) (Kanazawa: IEEE): 1–6. doi:10.1109/SNPD.2017.8332383. ISBN 978-1-5090-5504-3. https://ieeexplore.ieee.org/document/8332383/. 
  11. ↑ "Lucene,NET". The Apache Software Foundation. 2021. https://lucenenet.apache.org/. 
  12. ↑ 12.0 12.1 Tang, An; Tam, Roger; Cadrin-Chênevert, Alexandre; Guest, Will; Chong, Jaron; Barfett, Joseph; Chepelev, Leonid; Cairns, Robyn et al. (1 May 2018). "Canadian Association of Radiologists White Paper on Artificial Intelligence in Radiology". Canadian Association of Radiologists Journal 69 (2): 120–135. doi:10.1016/j.carj.2018.02.002. ISSN 0846-5371. https://doi.org/10.1016/j.carj.2018.02.002. 
  13. ↑ Erickson, Bradley J.; Korfiatis, Panagiotis; Akkus, Zeynettin; Kline, Timothy L. (1 March 2017). "Machine Learning for Medical Imaging" (in en). RadioGraphics 37 (2): 505–515. doi:10.1148/rg.2017160130. ISSN 0271-5333. PMC PMC5375621. PMID 28212054. http://pubs.rsna.org/doi/10.1148/rg.2017160130. 
  14. ↑ Rainey, Clare; McConnell, Jonathan; Hughes, Ciara; Bond, Raymond; McFadden, Sonyia (2021). "Artificial intelligence for diagnosis of fractures on plain radiographs: A scoping review of current literature" (in en). Intelligence-Based Medicine 5: 100033. doi:10.1016/j.ibmed.2021.100033. https://linkinghub.elsevier.com/retrieve/pii/S2666521221000090. 
  15. ↑ Zablith, Fouad; Antoniou, Grigoris; d'Aquin, Mathieu; Flouris, Giorgos; Kondylakis, Haridimos; Motta, Enrico; Plexousakis, Dimitris; Sabou, Marta (1 January 2015). "Ontology evolution: a process-centric survey" (in en). The Knowledge Engineering Review 30 (1): 45–75. doi:10.1017/S0269888913000349. ISSN 0269-8889. https://www.cambridge.org/core/product/identifier/S0269888913000349/type/journal_article. 
  16. ↑ Zablith, Fouad (2009), Aroyo, Lora; Traverso, Paolo; Ciravegna, Fabio et al.., eds., "Evolva: A Comprehensive Approach to Ontology Evolution", The Semantic Web: Research and Applications (Berlin, Heidelberg: Springer Berlin Heidelberg) 5554: 944–948, doi:10.1007/978-3-642-02121-3_87, ISBN 978-3-642-02120-6, http://link.springer.com/10.1007/978-3-642-02121-3_87. Retrieved 2022-07-16 
  17. ↑ Presutti, V.; Daga, E.; Gangemi, A. et al. (2009). "eXtreme design with content ontology design patterns". Proceedings of the 2009 International Conference on Ontology Patterns 516: 83–97. doi:10.5555/2889761.2889768. https://dl.acm.org/doi/10.5555/2889761.2889768. 
  18. ↑ Giannakopoulos, George; Mavridi, Petra; Paliouras, Georgios; Papadakis, George; Tserpes, Konstantinos (2012). "Representation models for text classification: a comparative analysis over three web document types" (in en). Proceedings of the 2nd International Conference on Web Intelligence, Mining and Semantics - WIMS '12 (Craiova, Romania: ACM Press): 1. doi:10.1145/2254129.2254148. ISBN 978-1-4503-0915-8. http://dl.acm.org/citation.cfm?doid=2254129.2254148. 
  19. ↑ Karalis, D.G.; Melanitis, N.Ε.; Yannoulis, Y.G. (1 December 2018). "Failure analysis of a Cu 12Mn mechanical fastener in marine environment" (in en). Engineering Failure Analysis 94: 69–77. doi:10.1016/j.engfailanal.2018.07.035. https://linkinghub.elsevier.com/retrieve/pii/S1350630718301419. 
  20. ↑ Lekatou, Angeliki G.; Mpalanou, Maria; Lentzaris, Konstantinos; Karantzalis, Alexander E.; Melanitis, Nikolaos (2018). Pantelakis, S.; Koubias, S.. eds. "Microstructure and surface degradation of Al reinforced by Al x W intermetallic compounds via different fabrication routes". MATEC Web of Conferences 188: 03001. doi:10.1051/matecconf/201818803001. ISSN 2261-236X. https://www.matec-conferences.org/10.1051/matecconf/201818803001. 
  21. ↑ Karalis, D. G.; Melanitis, N. Ε. (1 April 2014). "Analysis of a Premature Failure of a Hub from a Diesel Generator of a High-Speed Motor Ship" (in en). Journal of Failure Analysis and Prevention 14 (2): 236–246. doi:10.1007/s11668-014-9788-4. ISSN 1547-7029. http://link.springer.com/10.1007/s11668-014-9788-4. 
  22. ↑ Paipetis, A.S.; Aggelis, D.G.; Barkoula, N.M. et al. (2012). "Damage monitoring of composite laminates using ultrasonics". In Paipetis, Alkis S.; Matikas, Theodore E.. Emerging technologies in non-destructive testing V: proceedings of the Fifth Conference on Emerging Technologies in NDT, Ioannina, Greece, 19 - 21 September 2011. Boca Raton, Fla.: CRC Press, Balkema. pp. 281-286. ISBN 978-0-415-62131-1. OCLC 795699933. https://www.worldcat.org/title/mediawiki/oclc/795699933. 
  23. ↑ Karalis, D.G.; Melanitis, N.E.; Pantelis, D.I. (1 January 2012). "Failure analysis of a rock anchor made of stainless steel in marine environment" (in en). Engineering Failure Analysis 19: 123–130. doi:10.1016/j.engfailanal.2011.09.011. https://linkinghub.elsevier.com/retrieve/pii/S1350630711002263. 
  24. ↑ "Protégé". Board of Trustees of the Leland Stanford Junior University. 2021. https://protege.stanford.edu/. 
  25. ↑ Musen, Mark A. (16 June 2015). "The protégé project: a look back and a look forward" (in en). AI Matters 1 (4): 4–12. doi:10.1145/2757001.2757003. ISSN 2372-3483. PMC PMC4883684. PMID 27239556. https://dl.acm.org/doi/10.1145/2757001.2757003. 

Notes

This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. Some grammar and spelling were cleaned up for better readability.

Retrieved from "https://www.limswiki.org/index.php?title=Journal:Designing_a_knowledge_management_system_for_naval_materials_failures&oldid=48556"