Journal:Human–information interaction with complex information for decision-making

Abstract

Human–information interaction (HII) for simple information and for complex information is different because people's goals and information needs differ between the two cases. With complex information, comprehension comes from understanding the relationships and interactions within the information and factors outside of a design team's control. Yet, a design team must consider all these within an HII design in order to maximize the communication potential. This paper considers how simple and complex information requires different design strategies and how those strategies differ.

Keywords: human–information interaction, decision-making, complex communication, information design

Introduction

A primary goal of human–information interaction (HII) for complex information is to communicate concepts and ideas to be used for decisions. That is, information that a reader uses to develop an understanding of a situation and to make decisions about a complex situation.

In any complex communication situation, the proper amount of information should result in maximum communication; of course, too little, too much, poor organization, or inappropriate content reduces the communication. Stating that the communication needs to provide the proper amount of information could considered tautological. It is easy to claim — and adds little to a discussion — a text should have the proper information (I use text within the article in a generic sense: paper, web pages, etc). However, defining and measuring the communication value of a document (and the information it contains) proves to be a much more difficult task.

The interaction and presentation needs of complex information are very different from those of simple information, where a person looks for single information elements. For example, consider the difference in looking up the score of yesterday’s game (simple information) and considering house remodeling to help care for a chronically sick parent (complex information). When we move from a communication viewpoint to an HII viewpoint, we also bring into play issues of how a person can manipulate the information. The base text may contain the “proper information” but for maximum communication, it may need to be reordered or otherwise manipulated to meet individual needs.

A large percentage of the HII literature either looks at simple information search and interaction or conflates simple and complex information. It also tends to take a web-based focus with the person having an essentially infinite number of sources to explore. While this is true in many cases, within this article, I take a much narrower focus on HII and look at the construction of content within a single set of information that is under a content development team’s control, such as the production of corporate reports or public information for a specific goal (an example presented later will be on communicating hurricane information). A view focused on the design team’s task needs to consider what information to create and how people will interact with it. The more commonly discussed view of people searching the Internet (or other sources) for existing information will not be directly considered since a design team has no control over how that search proceeds or what information the person may view.

Explanation of terms

In the introduction, I used the terms simple and complex information. Here I provide a working definition for each.

Simple information

Simple information can be characterized by existing as a single information element. A high percentage of the “look it up on Google” amounts to simple information. For example, a person can look up when a movie was released or a recipe for a carrot cake. Both fit the definition of simple information. Factors that distinguish simple information include:

• Single path. One path can be fully defined that will result in the answer.
• Right/wrong answer. The correctness can be tested and the information declared correct or not correct.
• Complete information. The completeness can be tested and the information declared complete or not complete.
• Closed system. All of the factors that might influence the answer can be defined and accounted for.

Complex information

Complex information exists at the opposite end from simple answers. There is no single “answer”. Addressing complex questions requires using and integrating multiple information elements, which often conflict. People have a complex web of information needs and interactions to fill those needs. For example, analyzing monthly business reports and making decisions about this month’s production, or making health care choices. Factors that distinguish complex information include:

• Multiple paths. There is no single path to an answer. A person can take many different paths and all will work. The effectiveness of the paths may, of course, vary.
• Open-ended. The idea of “complete” is undefined. A person can continue to collect information and refine their understanding with an essentially infinite amount of information. Instead, a person has to pick a stopping point and make a decision.
• Needs cannot be predefined. The information that a person needs cannot be predefined. Of course the major or essential information can be predefined, but the many smaller information elements that can exert a strong influence vary too much between individuals.
• History. The information exists within a continuum and that history influences how it gets interpreted and used.
• Non-linear. The overall situation shows a non-linear response with small differences in some information elements resulting in very large differences in appropriate decisions.
• Open system. All of the factors that might influence the decision cannot be defined. There are too many and the information is dynamic, changing on time scales relevant to the decision situation.

Most of the simple information that gets communicated tends to be single facts or procedural type of information. Complex information, on the other hand, deals with a broader scope.

• Complex information communicates concepts and ideas.
• Complex information communicates an understanding of a situation.
• Complex information communicates relationships and interactions.

Clearly, the HII required to maximize the communication of complex information is very different between the two. Most writing and UX guidelines deal with communicating simple information. But to really address people’s information needs, we need to consider why and how they interact with complex information. Consider the case of electronic health records — they clearly represent complex information — and represent an example where the design team has control over a significant part of how the HII occurs. The design has to consider many potentially conflicting sets of priorities (development time, maintenance cost, hospital administration goals, government regulations, medical needs, etc.), which, in turn, affect how well the users can interact with the information. The input methods must be efficient for data entry, but presenting the information the same way to a nurse or physician may not be the best for integrating it into a clear view of the patient’s condition. The relationships of information (connecting the results of various medical tests, etc.) should reflect the initial diagnosis and should also reflect the changing patient status as treatment progresses.

In previous work, I have defined a situation as “the current world state which the user needs to understand. An underlying assumption is that the user needs to interact with a system to gain the necessary information in order to understand the situation” (p. 11). The information relevant to the HII exists both inside and outside any circle drawn to enclose “the situation” (Figure 1). Honestly, the difficulty of drawing that enclosing circle is a characteristic of a complex situation.

Figure 1. Complex situation with information both inside and outside of a design team’s control.
Human–information interaction (HII) only directly applies to the information a design team controls.

Once an upfront analysis has defined the people’s goals and information needs, then the HII factors which support manipulating the information and revealing the relationships can be considered and entered into the design.

Typically, the failure of these technical documents [used here in a generic sense for any information source] comes not from a lack of information; the text probably contains an excess of information. Post hoc studies of communication failures find many sources to blame: poor information architecture, poor organization, wrong grade level or writing style, or poor presentation. But instead of seeing these problems as a root cause, let’s consider them as symptoms of a more fundamental problem: a problem stemming from the underlying complexity of the situational context and a failure of the information presentation to match that complexity.(p. 110)

The task assigned to an HII design team is to ensure the underlying complexity is not overly simplified and that the information presentation and manipulation meets the needs of the people and the demands of the situation. As others have discussed, the HII problem for maximizing the communication for complex information is not a tools question, but one of understanding what people need and how they come to understand the information.

Information relationships

Understanding complex information depends on understanding relationships, but the potential relationships are essentially infinite. Comprehension of a complex situation occurs when people can mentally integrate those relationships into their view of the situation.

Information integration lies, not in a text element itself, but in the relationships between those elements. A reader needs to figure out what information is relevant and how to connect it to the current problem. Without proper information relationships, the reader does not gain an integrated understanding of information, but instead gains a collection of facts. Without relationships, information exists as a bunch of interesting factoids which do not help a person form an adequate mental picture of the situation. Collections of facts are less than useful for understanding and working with the open-ended problems that people encounter in complex situations. Without the relationships, a person learns about X and Y, but not how X and Y relate to each other or to Z in terms of their current problem or situation. The text fails to communicate because the reader can’t form the necessary information relationships.(p. 111)

A patient trying to understand a medical problem often has difficulty understanding the situation because they don’t understand the relationships. They can look at a set of lab results and know the numbers, but lack the medical practitioner’s knowledge of what the values should be for their condition. The patient may be worried because some values are high, but the physician is satisfied with them because the specific condition often has even higher values. Likewise, a junior manager needs to develop the knowledge of how to interpret multiple values that may appear on a collection of monthly reports. Good HII can help by providing a more integrated view and providing help with the integration, without interfering with more experienced people’s information interaction.

Building relationships means developing a deep-level understanding of the text rather than just a surface-level knowledge. Deep-level knowledge involves seeing the macrostructure of the text and being able to apply prior knowledge to it and fit the text within the reader’s prior knowledge framework. Surface-level knowledge involves knowing the basic text. People with surface-level knowledge can quote the text and if asked recall-type questions would respond with answers very close to the text language, but they would not be able to elaborate on the answer or connect it to other information. People with deep-level understanding would be able to place the text’s information into their own words. A major element in the difference between people being able to develop deep-level understanding versus a surface-level understanding is their prior knowledge.

Communicating complex information for decision-making can be viewed as working to help build relationships within the information. Unfortunately, building relationships is a great theoretical concept, but one that does not lend itself to a direct operational definition. One way to define judging or qualifying relationships can be how they reduce the uncertainty a person has about the situation. This reshapes an HII design team’s goal to one of focusing content creation in terms of what information does the reader need to reduce their uncertainty and, consequently, to build a web of relationships between the information elements.

Contextual awareness

Except for training material, the readers of technical information typically understand the basics but need to know specific information about the current situation (as opposed to the general situation) in order to make decisions. The understanding of a complex situation needed to make informed decisions comes when people can distinguish the information structure, grasp the relationships within it, and make inferences on the future evolution of the situation.

Building on Endsley’s (1995) situation awareness work, we can call this contextual awareness. Contextual awareness is the understanding of the information within an informational situation which forms the basis for how to interpret new information and how to make decisions for interacting with that situation. With poor contextual awareness, people can know something is occurring or that a particular piece of information exists, but they cannot easily find relevant related information or they have the information but do not understand how it relates to the overall situation. On the other hand, good contextual awareness does not guarantee a person will form the proper intention or make the proper decisions; the error analysis literature is filled with cases where people understood a situation, but still made incorrect choices. Unfortunately for design teams, the concept of relevance itself is highly nuanced and multifaceted, creating a complex interplay that must be understood to engineer high quality HII.

Elements of good contextual awareness are (Figure 2): (1) Understands how the information fits within the current situation. (2) Understands the information relationships. Information comprehension requires knowing how information relates to other information. (3) Understands the future development of the situation and can make predictions about the ripple effects of any decision across the entire situation. HII that supports complex information needs to provide the interactions that support people developing high quality contextual awareness.

Kain, de Jong, and Smith (2010) study into how to communicate hurricane risks and warnings highlighted the issues of how people interpret information and make decisions about how they will react. The hurricane experts had their view of what information was needed and how it should be presented, but the research showed the people wanted/needed a different presentation. Their mental methods of forming relationships and of interpreting complex information differed from what the experts thought. The process of building contextual awareness differs and a design team’s analysis must capture those differences.

A design team working on developing the HII for a country’s hurricane awareness plan needs to balance both the expert’s “here is what the people must hear” against the more pragmatical “here’s what I want to know” as well as the local people’s opinion of how they react to hurricane warnings. Many of them have been through multiple hurricanes and have strongly held views that often conflict with the authorities. Notice how in this case, the design team has control of the information. A general search on hurricane warnings will have sentences such as “consult your local authorities for evacuation routes” but, here, the design team will be tasked with providing that evacuation route information. At the same time, they need to ensure the relationships between evacuation route, getting ready for evacuation, and planning on returning are all clearly laid out and connected.

Figure 2. Stages of developing contextual awareness

HII is information interaction, not data interaction

People who are engaged with decision-making and complex information need to be presented with information, not data (Figure 3). Quick definition of my terms here, which I discuss in more detail elsewhere.

• Data: Raw numbers, facts, and figures.
• Information: Information is data in context. It relates to the situation and contains the relationships that connect the information to the situation.
• Knowledge: Interconnected web of the relevant information and the relationships linking the information within the situation.

Figure 3. Data, information, and knowledge hierarchy. The higher the HII works in the hierarchy,
the better it fits building an understanding of the situation.

As an example, I heard a presentation that looked at the effect of sea level rise on the Norfolk, VA area. A software program had nice manipulation that let a person dynamically see the effects of different amounts of sea level rise from global warming and how it would affect the city. The problem was not with the implementation, but with the underlying assumptions of a design team. They were assuming that by providing a tool and letting people see how different sea level changes would affect the area that it would bring about understanding and increase long-term preparedness. But the tool was presenting content at the data level. Yes, I can change the sliders and see how the sea level changes, but it was devoid of supporting content (in the metaphor of this paper, the tool was a single puzzle piece). It not only didn’t connect with other pieces, those other pieces were not provided. As a result, a global warming denier could play with the model, agree that a four foot rise would be a catastrophic problem, but then reject it as something that would never happen. One of the software’s goals was to help people prepare, but without giving them the content and relationships to build their contextual awareness; it was a single data point and not part of a coherent presentation of information.

The transition from information to knowledge is important since it involves comprehending the relationships within the data and placing them within the context of the situation. Moving to knowledge is essential to building contextual awareness and must be the goal of an HII design team.

Communication of simple and complex information

Many discussions on communication start with Shannon information theory (1948) (Figure 4), which works for analyzing simple information. With clearly defined information to be conveyed, a design team can concentrate on efficient communication methods. Unfortunately, Shannon was really discussing the minimum information to convey a message, not how to communicate in a natural language. It works well for computer-computer communication, but is not as applicable to human-human communication, especially for complex information, with or without a computer mediator.

Figure 4. Basic block diagram of Shannon information theory. Information moves from the source
to the final destination with the overall goal to be minimizing both the noise and total amount of required content.

Driven in part by a Shannon-based desire to efficiently communication information, design teams try to break the communication into individual components. This fits with our reductionist nature to break any problem into individual components. Standard approaches for communicating complex information (and analyzing any complex system) are to assume it is assembled from smaller systems. Thus, it is hardly a surprising statement that we see a complex system as the sum of its components: 1 + 1 + 1 + 1 + ….. = THE SYSTEM.

Unfortunately, assuming a complex system is the sum of its components at best redefines it as a simple system. The complexity that people need to understand and the HII must support is not just a sum of individual components, but the interactions and relationships between components. The redefinition to a sum of individual components ignores the complex interactions that give rise to more than the sum of the parts: 1 + 1 + 1 + 1 + …..+ i_n >> Ʃ_n.

The communication problem, and consequently the HII problem, is that redefining the problem to be a simple problem fundamentally changes the problem itself. As a result, the communication itself changes and the relationships and interactions that are essential to understanding get lost. The communication-related literature seems to generally miss this important concept, although the computer science literature consistently contains quotes such as “the fundamental reason today’s software engineering cannot effectively manage inherent complexity is that its basis is in developing individual programs rather than in interacting systems.” (p. 73). In designs that strive to communicate complex information, an over-privileging of developing and/or organizing content while discounting the interactions and relationships within and between content leads to results similar to those critiqued by Sommerville.

In the introduction, I described most simple information as being single information elements that can be looked up (Figure 5). Complex information on the other hand, has a multitude of information elements that a person must understand, some of which are outside of the bounds of a HII system. Factors such as asking a person down the hall, having heard of a bad experience with product X, etc. (p. 115)

Figure 5. Relationships and understanding a situation

Figure 5 shows content as nice rectangles. But they could be better viewed as jigsaw pieces with the interlocks on the pieces representing the relationships (Figure 6), how the pieces interconnect with each other into a coherent understanding.

Figure 6. Relationships and the complexities of assembling information into a coherent understanding.
The understanding comes from both the content and knowing how it fits together.

Unfortunately, the pieces in Figure 6 are not static jigsaw shapes. They are better viewed as a collection of puzzle pieces that change shape as they move away/toward other pieces, and as they merge together (Figure 7). As a result, changes to one piece can ripple through the entire information web. A change to a piece in the upper left corner results in many of the pieces changing in some relevant manner. The information and HII appropriate and relevant for each point of a situation can be different. Situations are not static; they have a beginning, middle, and end. The puzzle pieces of Figure 7 are not just dynamic from a person’s interaction, but also change over time and over the situation’s evolution.

Figure 7. Dynamic evolution of information. The relationships between information elements
can change over both time and by the act of connecting them.

Let’s reexamine the six characteristics of complex information with respect to the puzzle metaphor.

Multiple paths: People have multiple paths through the information. The order in which they move between the puzzle pieces cannot be predefined. Each HII with a piece can change how it interconnects; thus, different paths through the information change how people build the relationships and, consequently, how the pieces fit together.

Open-ended: No clear point of enough information. The HII cannot move toward a predefined point of “now you have all the information.” With a goal of communicating concepts and ideas, how many pieces a person needs, cannot be defined. Of course, the issue of information needs versus information wants also comes into play. Coupled with this is the fact that information search and problem-solving are sufficing processing. People stop once they are comfortable with their understanding of the found information. Unfortunately, people are poor judges of knowing they have found adequate information.

Cannot be predefined: How many pieces a person will interact with remains unknowable to a design team. Interestingly, the size (content) of individual pieces is dynamic; some people need smaller/larger pieces to effectively comprehend the overall situation and tend to make choices in terms of immediate, rather than long-term, efficiency and effort of the HII.

History: Past history of a situation affects how the pieces will evolve and the past history of the people interacting with the information affects how they interpret it. Two situations may appear to be the same but the past strongly influences how the piece will change. For example, six months ago sales in the southeast region were down by 30%. A set of decisions was made and sales have steadily increased since them. They are still 10% below desired levels, but are increasing at an acceptable rate. This situation is much different than if sales are 10% below the desired level and not conforming to predictions.

Non-linear: The relationships between pieces can show non-linear response to changes. Minor differences in past history or the HII path can result in widely different — but appropriate — end points. In individual piece can morph into very different final shapes, even though they started from similar initial conditions.

Open system: As the system evolves, the overall content within the system changes. Some new information gets introduced and some information drops out. The number of pieces, their content, and their shape can change.

When the information is viewed as dynamic puzzle pieces, how those pieces change shape helps to emphasize how many solutions to problems introduce other problems, sometimes worse than the one they were intended to solve. For example, many environmental solutions inevitably end up doing more harm than good. The future changes to relationships are not understood when decisions are made, so the evolution of the situation is poorly predicted (poor contextual awareness). Worse, too many people make assumptions that they can change one piece with nothing else changing. In complex ecosystems — biological, educational or industrial — actions always have consequences that are hard to predict; a change to one piece ripples through all the others. The “hard to predict” is a hallmark of a complex system. HII that supports interacting with the information must consider those ramifications.

Beyond the need for the HII to handle dynamic puzzle pieces, it must also allow for easy trimming of the information space. All of the available information is not relevant to any specific situation. Instead, a person needs a select collection of information (Figure 8).

Figure 8. Information relevant to a situation. Each situation (or class of situations) only needs
a subset of the information. High quality HII helps to keep the focus on only the relevant subset.

HII for decision-making

The overall scope of the information flow for the HII of complex information moves to prominence when the interaction goal is decision-making. People need to collect information, analyze it, and make a decision; a process that depends on the flow of information both within and outside the control of an HII design team.

Decision-making occurs as a result of comparing what is perceived in the environment and what is known by the decision-maker. Essentially all of our current literature discusses considering the needs of people when designing information. However, current practice does not typically consider how people’s questions depend on how they make decisions and interact with the system. HII attempts to take the entire cycle of information interpretation and decision-making and place it within people’s current situation.

Mirel (1998) follows the same line of thinking as Conklin’s (2003) wicked problems when she points out that analyzing complex tasks requires seeing more than a single path:

This broader view is necessary to capture the following traits of complex tasks: paths of action that are unpredictable, paths that are never completely visible from any one vantage point, and nuance judgments and interpretations that involve multiple factors and that yield many solutions. (p. 14)

Mirel may have focused on the analysis aspects, but that is an early and vital step in any information interaction, especially for HII of complex information. Unless a design team clearly understands the people’s goals and information needs, there is minimal chance for the HII to support it.

The issue of wicked problems and unpredictable paths that both Conklin and Mirel discuss brings to light an interesting point. As design teams work toward HII of complex information, they encounter an interesting and confounding recursion: the HII of complex information is itself a complex problem and, as such, does not lend itself to easy answers. Yet, many design teams try to address those issues with the same process that works for simple information, which leads to an easy-answer mentality. As a result, they reduce the scope and avoid confronting the complex information HII issues head-on. Design teams who try to reduce all problems to a collection of individual parts or who assume they should consider only the information they control fall into this trap. They have redefined the complex HII problem into a simple problem.

Decision-making strategies

People use the available information to make a decision about how to proceed in their current situation; information use is always based upon the context within which it is used. This hard connection between use and context greatly complicates a design team’s task since they must understand both factors. And, depending on audience, the base information may exist within multiple contexts.

Based on the results of the decision, people take action and then need to re-analyze the situation. Closely related to decision-making is monitoring the situation to ensure it continues along the predicted path. If the information fits the expected progress, then a decision of “do nothing” is correct. However, if the information indicates that it is deviating, then people cycle back into decision-making.

There are two broad decision-making strategies that people use.

Experience-based

Experience-based decision-making is used the most and is a person’s default strategy. It is a rule-based strategy: “if X happens, then do Y.” Most decision-making, at least in the corporate world, is experienced-based. The reader is skilled in the overall area (such as a CFO making financial decisions), but does need specific information for the current situation. They need to know how this month’s production or sales numbers differ from last month’s or between different regions, but they do have a set of mental rules on how to interpret those numbers.

Based on past experience, people develop a rule-based strategy so they can reliably expect that “when X happens, then do Y.” Rule-based decisions reduce cognitive effort and work well within normal situations because the rules themselves evolved from past experience. However, that same rule-based strategy can cause problems if a person jumps to a decision based on incomplete information. X can happen in multiple situations but other factors differ—the relationships of what/how X is interconnected within the situation—which cause solution Y to only be an appropriate response in some situations.

Klein’s (1999) recognition-primed model of decision-making posits that experienced people do little problem solving, especially in routine situations. Instead, once they recognize a situation, they form a possible intention very rapidly, mentally evaluate it, and, if no major problems are evident, take action on it. Alternatives are not considered. In other words, they tend to use their prior knowledge and initial assessment to immediately pick a solution, and if it seems workable, then that is considered the answer. (Concerns about whether or not the solution is optimal are not part of the mental evaluation.)

Post-failure analysis tends to highlight that the decision-makers had the proper information, but improperly interpreted it. In many cases, it is a result of using experience-based decision-making and not realizing the “if x, then y” rule did not apply to the situation.

Knowledge-based

Experience-based decision-making can break down, typically when the situation contains unexpected factors or relationships. When people recognize their rules no longer apply, they have to shift to knowledge-based decision-making. Unfortunately, the shift to knowledge-based decision-making often happens after the monitoring phase of an experienced-based strategy reveals the situation is not developing as predicted.

In knowledge-based decision, a person knows their experience-based rule set has broken down and they are no longer working within a normal situation. Something is not right in the current situation and they are trying to figure out what to change to fix the problem. In other words, the expected relationships are not being found and a person needs to understand why. It tends to happen when people do not understand what caused the base problem. Unlike decision-making based on rules of “when X happens, then do Y,” a person has to fully develop their contextual awareness so they can make a decision.

The HII of complex information applies more for knowledge-based strategies than it does for experience-based ones. Knowledge-based strategies require people to engage in a systematic interaction with the information and work to build up their contextual awareness.

Someone working on a global warming problem is working at the knowledge-based and not experienced-based level. Allowing people to manipulate data as if it was a simple information — a collection of individual components — can misguide them into thinking it’s an experience-based problem and they can draw on their prior knowledge. Worse, defining the problem as “if they can work this sea level rise model, then they will understand the problem” redefines the problem from a complex problem to a simple problem. Unfortunately, but inevitable with the history component of complex information, prior knowledge and biases strongly influence how information gets interpreted. Of course, with global warming—and many other situations—the political aspects, factors outside of an HII design team’s control, come to the forefront.

Conclusions

Providing people with any collection of information, especially complex information, can easily result in information overload. With the large amounts of information and its dynamic nature, high quality HII is essential for people to comprehend that information.

Rather than a lack of information, the failure to anticipate people’s needs forms the basis of most information problems and poor decision-making; an issue design teams must explicitly address. Managers typically suffer from a shortage of decision-relevant information and a simultaneous overabundance of irrelevant information. People are forced to make decisions based on an unfiltered avalanche of information that may or may not be relevant to their needs.

Understanding a complex situation comes from understanding the relationships within the information. Building that understanding results not in knowing the individual pieces of information but in knowing how those pieces of information interconnect. That, in turn, requires looking at the system as not just a large collection of information, but as a highly dynamic integration of information and relationships between information. As a design team works on developing the system HII, they must ensure they capture those relationships and interconnections so they can be explicitly made to the people using the system. As such, a design team must:

• Define the situations that must be understood
• Determine the information people need to understand the situation
• Determine how that information is connected and how people see those connections as they build their contextual awareness
• Understand how the information and relationships change as the situation evolves
• Understand the biases, interaction expectations, and decision-making styles of the audience
• Determine the best HII for presenting the information to achieve the best information communication while allowing for those biases, interaction expectations, and decision-making styles

HII plays a fundamental part in ensuring the overall information content is usable and that people can form the relationships needed to build their contextual awareness. As that understanding develops, people develop their contextual awareness of the situation and can make informed decisions.

Conflict of interests

The author declares there is no conflict of interest.

Notes

This presentation is faithful to the original, with only a few minor changes to presentation and content. The two most notable changes involve citations. Albers lists the following citation in his references as citation 20:

"20. Albers, M. Contextual awareness as measure of human-information interaction in usability and design. In Proceedings of the HCI International, Orlando, FL, USA, 12–14 July 2011."

However, he never referenced that citation number in the paper. We have thus omitted it. Additionally, under the subsection "Experience-based", he references "Klein’s (1999)" but seemingly uses the incorrect citation number of 12, for Mirel 1998. We've corrected the citation number.