Learning in Tactical Decision-Making Situations:
A Hybrid Model of Schema Development
1. OVERVIEW OF SCIENTIFIC PROGRESS
During the period covered by this report, we have focused on two areas: developing a baseline set of measures with which to evaluate subject performance and modeling the learning that takes place as the subject interacts with the experimental task. We have had successful results in both areas. A set of highly complex scenarios were designed and used with a substantial number of novice subjects, yielding valuable information about how subjects learn to carry out this command and control task. The data from this experiment, together with data from several previous experiments carried out as part of this research effort, have been used as input to the cognitive model. The model performs very well thus far, capturing very satisfactorily the performance of the subjects we have chosen to study. We expect to make significant progress on the model during the next six months.
Our main accomplishments have been (1) the construction and evaluation of baseline scenarios for the CIC task to be used by all projects working on the CIC task and (2) the development of procedures to evaluate performance in the scenarios, and (3) construction of the schema model of human learning and performance.
Scenarios
We have created a set of three new scenarios that are considerably more complex that the initial set of nine that were created earlier for this project. Each scenario is set in a potentially threatening environment that novice decision makers will recognize, namely, the Persian Gulf, the Korean coast, and the Cuban coast. For each scenario, we created a general briefing that describes current geo-political events and that explicitly alerts the subject to the potential threats that exist in the current setting.
After evaluating the efficacy of the three scenarios with pilot subjects, we made the scenarios available to the other projects funded under ONR's Hybrid Architecture Initiative to study the CIC task. Included in the materials shared were the detailed program files that enable the scenarios to run on the CIC together with the briefing descriptions and playback files for each scenario of an expert. These files and playbacks have been useful to researchers of the other projects in creating their hybrid models of cognition.
We also carried out a learning experiment in which we observed the performance of 30 subjects who worked through the three scenarios, presented in counterbalanced random order. Each subject was observed at two different times. On the first day, he received the basic orientation about the nature of the experiment, read the instructions, and worked through one scenario. On the second day, following a brief review of the interface, he worked through the remaining two scenarios and had a debriefing interview.
Core Profiles
One of the central problems facing researchers working on the CIC task is performance evaluation. The nature of the task precludes a simple measure of right or wrong activities because a number of actions might be satisfactory at several extended points in time. Our approach has been to take a set of core profiles that we purposely included in each of the new scenarios. Each instance of a core profile is a set of 15 well-defined characteristics that captures important information about an air contact. A core profile typically occures more than a single time in a scenario, resulting in multiple opportunities for subject responses. The specific profiles were selected on the basis of my previous research with ONR's TADMUS Program. In that work, I observed a group of highly experienced experts, namely Commanding Officers and Tactical Action Officers currently assigned to ship duty. These experts participated in four scenarios as part of the TADMUS baseline study, and in the course of so doing, they responded to a large number of track profiles. The core profiles selected for use in the Hybrid Architecture Program were ones to which all experts responded identically. Thus, the experts' performance gives us a baseline against which to compare our novice subjects. Two measures have been applied to the core profiles: the number of inappropriate actions taken and the number of failures to recognize or take action. We are currently analyzing the experimental study described above with respect to subjects' response to the core profiles.
Schema Model
The model of schema development that is the focus of this research builds directly on the schema models of problem solving (described in my book, Schemas in Problem Solving, Cambridge University Press, 1995) and on the schema models of decision making currently being developed under ONR Grant No. N00014-93-1-0525 in relation to the TADMUS Program. Our progress to date includes integration of the programs for the neural network component of the model with the programs for the rule-based components.
The subjects we are modeling are novices to tactical decision making. Nevertheless, it is assumed that they bring considerable expertise (and schemas) from commonsense domains to the CIC task. The pertinent schemas that can be applied to the CIC task involve asking and answering questions, selecting menu items on the screen and observing the consequences, understanding threats and avoidance responses, and carrying out actions of self defense. In building the knowledge base that makes up the initial state of the model, structures have been built using the ACT-R architecture to model these commonsense schemas. To represent the performance and learning of a specific subject, the model requires input data from the subject's reading of the task instructions as well as input data resulting from the subject's performance during a scenario. The instruction input data consists of several eye tracking variables that provide information about the way the subject responded to the several pages of instructions. Included here are fixations, scan patterns, and total dwell times. The data from the scenario are the various button selections for actions as well as eye movement data indicating that a subject noticed a particular track at a given point in time. Our objective is to create a full model for each subject based on his initial scenario and then to determine how the model adapts to subsequent scenarios, comparing the model's performance with the subject's responses. We also intend to examine the similarities and differences of the model across subjects.
The model currently has just over 250 productions which appear to be sufficient for most subjects. Modeling a subject's behavior is characterized as prediction and confirmation of each of the subject's actions in the course of the scenario. Much of the modeling thus far has focused on the feature space and the production rule set. We are currently developing measures of model performance based on the number of times that each production is selected by the model as well as the number of times the production also agrees with the action taken by a subject.
Publications: none
(Most of this year was devoted to creating the experimental scenarios and to establishing a methodology evaluating subject performance. A key experiment has been carried out using the new procedures and scenarios. The results are now being analyzed, and we anticipate publication of these results within the next year. We also anticipate publication of a paper describing the model structure and performance in the coming year.)
Presentations:
Allred, L. E., & Marshall, S. P. (1997, May). Pupillary changes as indicators of understanding computer-based instruction. Poster presentation at the Annual Meeting of the American Psychological Society. Washington, DC.
Masters Thesis:
Allred, L. E. (1997). Pupillary changes as indicators of understanding computer-based instruction. (Expected completion: September, 1997).
Awards/Honors:
On May 9, 1997, I was one of approximately 20 researchers invited to be part of an exhibition for the U.S. House of Representatives. The exhibit, held on Capitol Hill and entitled "Basic Research in the National Defense," was intended to showcase university research funded by the Department of Defense. I was asked by the Public Policy Office of the American Psychological Association to present my ONR-funded research. My presentation, "Tactical Decision Making: How Do You Know When You Need to Act?", described my cognitive modeling from the Hybrid Architecture Program (N00014-95-1-0237) and my eye-tracking studies and modeling under the TADMUS Program (N00014-93-1-0525).
The research from this project already is being shared with two large ongoing ONR projects: the TADMUS Program, carried out by research personnel at NRaD (San Diego) and NAWCTSD (Orlando), and the Advanced Embedded Training ATD, also at NAWCTSD. The aspects of my research that are pertinent to both programs are the cognitive models and the eye tracking procedures. For example, in the next phase of TADMUS development, I expect to work with the Orlando team to implement some of the model components in their assessment procedures to evaluate the impact of tactical training instruction. I also share with both NAWCTSD and NRaD the results of my eye tracking research. I have met with members of the AET-ATD (Peter Rosenfeld & Allen Hale (Lockheed Martin) and Joan Ryder (CHI Systems) to discuss both issues of modeling and eye tracking methodology. I expect to continue working with them in an effort to share 6.1 results with 6.3 applications.