Significance of Association in Tool Mark Characterization
L.S. Chumbley and M. Morris
See also the instructional videos on "Firearms and Tool Mark Evidence"
Abstract
One weakness that currently exists in the field of comparative examination of evidence is the general failure of current approaches to adequately assess the significance of association through quantitative measures that provide a statistical evaluation of evidence. While various efforts have been made and methodologies employed over the years, such as the measurement of consecutive matching striations, tool mark comparisons remain difficult to quantify in a robust statistically valid sense. While the desire to develop a methodology that allows the examiner to assign confidence levels and predict error rates is universal, the unfortunate truth is that such a description (similar to what is possible in the field of DNA) is an unattainable goal in toolmark analysis since the population will continually increase and the variability cannot be satisfactorily defined. However, this is not to say that statistical relevance cannot be assigned to toolmark examination. Relevance can be assigned if one is careful about the structure of the study attempted.
In a recent study of tool marks produced by sequentially made screwdriver tips the authors developed a computer algorithm that was able to reliably separate matching tool marks from those that do not match using an analysis based on Mann-Whitney U-statistics applied to data files containing 2-dimensional information obtained using an optical profilometer. These successful results indicate that significance of association can be accomplished by statistical evaluation of the data files. The work carried in the present project (and discussed in this report) built upon this success by providing additional statistical information that will increase the relevance of the measurements obtained. Thus, the overall goal of this work was to increase the statistical relevance of toolmark analysis. To achieve this goal two distinct objectives were identified:
1) Extend the previously developed statistical methodology to allow for self-calibration to control rates of false non-matches.
Our previous work (Chumbley et al., 2010) has focused on the use of the Mann-Whitney U- statistic as an index for assessing the similarity of toolmarks. While it has been empirically shown to be useful in sorting mark-pairs made by the same tool from mark-pairs made with different tools, it is also influenced by many other aspects of the toolmark structure, hence a single value cannot be used as objective evidence (with quantifiable risk) for or against a match. The current work has focused on overcoming this difficulty by using multiple test marks made in the laboratory, in a ``self-calibrated’’ analysis. In short, comparison values between lab marks (that are known to match) form the basis for comparisons between lab marks and evidence marks, eliminating the need for “universal” critical values (i.e. single sets of constant references values such as those found in commonly used statistical tables) for the comparison index. A formal statistical analysis based on likelihood functions have been developed to allow for control of false non-match calls.
2) Empirically validate the methodology developed by performing experiments using a different type of tool mark.
The first part of the project dealt with objective one. In this task the variability of marks from screwdriver tips by characterized examining multiple marks made by a trained forensic examiner. The data was used to establish the variation in U-statistic values inherent in the system and allowed likelihood analysis to be conducted denoting significance of association between lab-lab comparisons and lab-field comparisons made at the same angle. With the initial model established, modifications were then undertaken to generalize the model to be applicable to marks made at all angles. In this analysis it was shown that the angle at which mark was made could be deduced to a fairly high level of accuracy.
In the second part of the project the second objective was attained by applying the algorithm developed in the previous study to an entirely new system separate and apart from the screwdriver tool marks studied initially. In this case, markings produced by shear cutting metal wire using the shear face on pliers were analyzed and tested to determine the applicability of the approach. Since the marks produced are not regularly striated, this study represented a significant extension concerning the performance of the algorithm. The study found that with adjustment of the analysis parameters used, known matched sets of data from these quasi-striated marks, i.e. marks characterized by groups of striations instead of regular striae, could be successfully differentiated from known non-match sets. Areas for improvement were also identified that will make the system even more reliable. Successful validation of the methodology has created a wide range of possible future applications for the developed statistical algorithm that could revolutionize comparative tool mark analysis.