Decision-Making Schemas in Rapidly Changing Situations

Research Sponsored by the Office of Naval Research
Grant No. N00014-93-1-0525
Program Officer: Gerald Malecki
Continued from Page 1

Study 1: Expert Decision Makers

This research project builds on the cognitive theory about the acquisition and use of schema knowledge as described in Schemas in Problem Solving (S, Marshall, Cambridge University Press, 1995). It extends the schema theory from its original emphasis on problem solving to the related domain of tactical decision making. A central goal was to study the feasibility of using schema theory as the cognitive basis for understanding how officers use decision support systems as they make decisions. To this end, we have studied the performance of six teams of officers as they engaged in various computer simulation tasks as part of the TADMUS Program. Each team consisted of the Commanding Officer and his Tactical Action Officer. These officers were currently assigned to ships, were highly expert in their performance, and the data collected as they participated in the tasks have yielded a number of important findings.

The data were analyzed to determine two things: (1) the types of schema knowledge that were most used by the experts and (2) the times in the decision-making process at which the different kinds of knowledge were most important. The investigation initially focused on the identification knowledge component of a schema because it is of primary importance in tactical decision making. Identification knowledge is the store of information in memory that allows the individual (or team) to make rapid recognition of patterns as they occur in the environment. For an officer in this study, the patterns that needed to be recognized were the characteristics that described the many aircraft and vessels (i.e., tracks) surrounding his ship. It was clear from the officers' performance that an important early step in the decision-making process was the reliable reduction of all tracks to a manageable set of possible problems and/or potential threats. One of the demands of tactical situations such as those studied here is that tracks of interest need to be recognized as rapidly as possible.

Under schema theory, identification knowledge is modeled by a neural network and is considered to be chiefly pattern recognition. The decision maker sees a number of features in the situation, and the particular configuration that results is recognized as a pattern, not as isolated features. Details about context of the situation are not needed for this recognition. It is fast and automatic, requiring few additional cognitive resources. The advantages of such recognition in complex tasks with time constraints are obvious; the decision maker will be able to make more evaluations and to make them rapidly if the process is context independent.

To conduct the study, an empirical analysis of the officers' use of identification knowledge was first carried out. The results were then verified with computer models. All teams showed a high degree of pattern-matching performance, and each team was well modeled by a neural network model. However, it is significant that each team required its own unique network, which meant that the patterns generated different responses from the teams. This variety of response was reflected in the empirical data as well, with over 20 percent of the patterns eliciting the full range over all possible responses from the teams.

From the teams' conversations during the scenarios, it was apparent that all teams recognized a common set of features because these features were frequently mentioned by them all. Thus, their identification knowledge consisted of the same elements. However, the ways in which these elements were combined and weighted to form patterns obviously differed because the teams did not necessarily reach the same conclusions when given the same input information. Some teams found a specific pattern to be highly important and demanding of immediate attention while other teams found the same pattern to be relatively unimportant and easy to ignore. The most likely reason for such differences is that each team had its own store of knowledge about air contacts in tactical situations. This knowledge developed as a result of actual experiences of the team during sea duty and was necessarily unique to a team.

This research effort clarified the role of identification knowledge and addressed some of the important issues about how it is used in tactical decision making. The data analyses suggest that the use of identification knowledge is largely automatic and consistent. All teams apparently recognized patterns in many instances without additional, more elaborate cognitive processing of the situational details surrounding them.
 
 

Study 2: Less Experienced Decision Makers

.Study 2 was a replication and validation of Study 1, involving officers who had much less tactical experience than the experts of the original research. The data for the study were collected by Dr. Kim Jentsch and her associated at NAWC-TSD, and we are grateful to Dr. Jentsch for making the data available to us.

In her original 1995 research, Dr. Jentsch videotaped the performance of teams as they responded to three scenarios as part of the TADMUS Leadership Study. In contrast to the experts of Study 1, these teams were composed of moderately experienced officers who were currently undergoing training.

Data from 10 teams in the Leadership Study were analyzed in the same way that we previously analyzed the expert teams. The empirical data were encoded, and neural network models were constructed.

The results of the validation study confirmed the original findings:

• A team does not allocate the same level of attention to every track displayed on the screen.
• Each team is highly consistent in the way it responds to specific track profiles.
• A track profile may elicit different responses from expert teams.
• Team responses to the profiles can be successfully modeled.
• A neural network is sufficient for modeling the identification knowledge of expert teams.

We compared our cognitive error analysis with the performance analysis of the NAWC-TSD subject matter experts who rated the 10 teams according to how well they performed during the 3 scenarios. The level of agreement between the two analyses was high, with an average correlation of -.68. Teams receiving high scores from the subject matter experts (i.e., teams who performed well) made relatively few identification errors according to our analysis. For teams rated low by the expert, we found large numbers of errors. The importance of this analysis is in the degree to which our cognitive analysis supports the SME ratings. The SMEs rated only small snapshots of performance (called "events") during each scenario. As shown by our more exhaustive analysis, these snapshots tended to be highly accurate, and they appear to be sufficient for evaluating team performance.
 
 
 
 

Study 3: Feasibility of Eye Tracking

A second area of research focused on evaluating the usefulness of eye tracking in an applied setting and was carried out in collaboration with the TADMUS group at SPAWAR in San Diego.

A participating naval officer wearing the old eye tracking apparatus

Subjects participated in teams, with each team consisting of two TAOs. One officer took the role of Commanding Officer and the other served as the Tactical Action Officer for the study. Each officer wore a special headband on which was mounted a small optics module, a small camera, and a sensor for tracking head movements. Traditional earphones were placed on top of the headband to allow normal communications among the officers and the confederates who also participated in the study.

Each experimental session took about 5-6 hours. Typically, the officers were briefed by SPAWAR personnel for about 1-2 hours on the use of the DSS1 at the start of the session. At the conclusion of this briefing, the eye-tracking headbands were positioned and initial eye calibrations were carried out. The officers were observed for three tasks: two scenarios that ran for 18 minutes and 25 minutes, respectively, and a set of evaluative questions. During the scenarios, the officers had the opportunity to view both the new DSSI and a traditional display. At the conclusion of each scenario, the officers answered a set of questions posed by SPAWAR personnel and were given the opportunity to remove the headband and earphones. For the final task, each officer was asked to respond to 30 questions designed by the SPAWAR team to evaluate the officers’ ability to locate specific information on the DSS1 display. This task consists of six different questions which were each presented five times, focusing on different air tracks with each presentation.

A GazeTraceTM showing the search pattern used by one U.S. Naval Officer as he searched the display.

The eye-tracking study was undertaken to answer two important questions: first, was it feasible to use eye tracking in an applied setting such as the DEFTT lab? And, second, did eye movement data provide meaningful results? The answer to both questions was affirmative.

The usability of the data can be seen from the types of analyses we have thus far created in this study. Three types of analyses have been developed: (1) a comparison of the overall usage of the DSS1 versus the traditional display; (2) the calculation of the percentage of usage of the various areas of the DSS1 display during the scenarios; and (3) scan patterns over defined screen regions during the question task. These analyses allow evaluation of large issues, such as the relative amount of time the DSS1 was used compared to other displays, as well as fine-grained issues, such as the patterns of scanning that are exhibited when one officer searches repeatedly for a single piece of information.

Study 4: Patterns of Communication Among CIC Members

As a continuation of the research effort to examine the schema knowledge required in tactical decision making, we analyzed the communications of the teams of Study 1 and Study 2. For this study, only the communications of the two main decision makers—i.e., the Commanding Officer and the Tactical Action Officer—were examined. The objective was to determine how often the four types of schema knowledge (identification, elaboration, planning, and execution) were evident as the officers spoke to each other and to the other members of the Combat Information Center.

From transcripts of each scenario run, research assistants coded every utterance made by the two officers. These were subsequently coded according to whether they showed use of identification knowledge, elaboration knowledge, planning knowledge, or execution knowledge. Many utterances contained more than one type of knowledge and were coded for all types present. Other utterances show no evidence of schema knowledge and were ignored.

The coded communications were then grouped into 30-second intervals, and the analysis focused on each interval with its immediate preceding and succeeding interval. Thus, the actual time analyzed is 90 seconds. The analysis for each pair of officers focused on whether the schema knowledge types appeared at all in their communications and if so, on the different combinations of knowledge that occurred in the 90 seconds under study.

Results showed that regardless of experience, teams made more identification communications than any other. However, the experienced team made significantly more communications involving planning and execution knowledge than did the less experienced teams.

A key question in this research was the extent to which a team demonstrated integrated schema knowledge. The evidence of integration was the use of several types of knowledge in the same time interval. Coding the occurrence of the different types from 0-4 for each of the 90-second intervals, we compared the frequency of multiple schema knowledge utterances for the two groups of officers. The results were striking: the more experienced officers were significantly more likely than the less experienced officers to have three or four types of knowledge present in an interval. Conversely, the less experienced officers were more likely to have none or a single type of knowledge present in their utterances.

Examination of the initial communications among the team members as a scenario begins points to the main differences observed in this study. A total of 24 scenarios were coded for the more experienced officers. We found that 23 of the 24 scenarios opened with three types of information being communicated to the full team; these were identification, planning, and execution. In contrast, not one of the 30 scenarios coded for the less experienced officers had this initial pattern of communication. Instead, the opening 90 seconds contained either no schema communication (17 instances) or a single identification (8 instances). The more experienced officers began the scenario by setting the stage for the rest of their team, indicating in their communications what they saw in a situation (the identification knowledge), how they intended to proceed (the planning knowledge), and what the various team members should do (the execution knowledge).

These analyses show that the level of experience influences the extent to which team members communicate their schema knowledge. Teams differed by experience in the ways they initiated the communication and the extent to which they communicated schema knowledge.

The analysis of the use of schema knowledge by teams of officers will have significant impact in training. These results show that highly experienced officers communicate more schema knowledge than less experienced officers and that they do so in similar ways. In particular, the more experienced teams tend to integrate their schema knowledge, so that their communication contains important aspects of identification, elaboration, planning, and execution that are linked together. In contrast, less experienced teams tended to verbalize isolated aspects of schema knowledge, especially identification knowledge. Less experienced teams also tended to have long periods of time in which the team leaders were not sharing critical schema knowledge with other team members.

The research has two important training implications. First, the differences in communication may have arisen because the experienced teams had deeper schemas for tactical decision making than the less experienced teams. In that case, the less experienced teams may need additional explicit schema training to provide them with a richer knowledge base. Second, the differences may have arisen because the less experienced teams did not understand the value of communicating this knowledge immediately to their teammates. In this case, their schema knowledge may already be quite strong; they need additional training about effective communication.

Study 5: Eye Tracking Evaluation of the DSS2 (ongoing)

This study extends the use of eye-tracking technology described above in Study 3. The two main participants of TADMUS, i.e., the research groups at NAWC-TSD in Orlando and at SPAWAR in San Diego, are currently collaborating to study the impact of training with and without a newly designed decision support system (DSS2). As part of the performance evaluation in that collaboration, we are monitoring the eye movements of officers as they use the DSS2.

The study is being carried out at the Surface Warfare Officers School in Newport, RI. The officers are Tactical Action Officers enrolled in Department Head Training at SWOS. The study began in 1998 and will continue through 1999.
 
 

References and Reports