Capitalizing on chance

In doing simulation experiments it would be very easy to capitalize on chance, e.g., by running simulations over which do not work, or reporting select subsets of tests. The primary set of simulations reported in the thesis involving 50 simulated depressed and 50 simulated nondepressed individuals was run 15 times, using various network parameters, training sets, activation rules, and learning rules before the final network and parameter configuration which were reported in this thesis was arrived upon. Each iteration involved the refinement of the simulation according to a rationally motivated rapid-prototyping process. Adjustments included steps such as changing the error thresholds and learning rates, changing the number of hidden units in the network, manipulating the amount of noise in the network, manipulating the number of stimuli used during overtraining on negative exemplars, etc. The current simulation was the first one using the final configuration of parameters. Subsequent simulations in which results from a single simulation are reported are the only such simulations which were run. None of these simulations were run multiple times.

Just as replication is important to the statistical conclusion validity of experiments relying on human data, replication is also important in validating computational simulations, especially those with relatively few simulated subjects. Future attempts to replicate the current results will be extremely important.