Study Identifies Ways to Improve ATF Ballistic Evidence Program
Nancy Ritter
National Institute of Justice
An NIJ-funded examination of NIBIN offers recommendations for improving the program's tactical and strategic value.
NIJ-funded researchers have concluded that the nation's ballistic evidence program has significant "untapped potential" to solve crimes that involve a firearm.
The NIJ-funded examination of the National Integrated Ballistic Information Network — referred to simply as "NIBIN" by everyone in law enforcement — revealed that the timeliness of processing evidence varies greatly from jurisdiction to jurisdiction. Some identified hits against the national gun-crime evidence database within a few days of a crime, and others produced hit reports so slowly that they had no investigative value.
It is important to understand that this study did not examine the science of firearm and tool mark examination itself; rather, the study looked only at the operations of the NIBIN program as an investigative tool. For the latest on the accuracy, reliability and validity of firearm and tool mark examinations, see "The Science Behind Firearm and Tool Mark Examination."
NIBIN is a program operated by the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF) in which firearms examiners at state and local crime laboratories compare tool marks on spent cartridges or bullets found at a crime scene to digitized images of ballistic evidence in the database. In essence, it is a grant-in-aid program that makes ballistic imaging technology available, via the database, to state and local law enforcement agencies (called "NIBIN sites") that generally would not be able to acquire the technology on their own. ATF provides the equipment, and the NIBIN sites are responsible for entering data into the nationwide database and then, of course, following up on hits.
In their final report, the NIBIN study researchers make extensive recommendations for improving the program's tactical and strategic value.
How Does NIBIN Work?
Firearms have numerous metal parts. During the manufacturing of a firearm, the machining process leaves unique, microscopic markings (called tool marks) on parts of the firearm. When most firearms are fired, the tool marks — which, in terms of their distinctiveness, can be thought of as somewhat analogous to fingerprints — are transferred to the "spent" bullet or cartridge case, which can be collected from the scene of a crime (such as a homicide or shooting) and then compared with a test-fired firearm that, for example, has been confiscated from a suspect.
Ballistic imaging technology converts these tool marks into a two- or three-dimensional digitized image. These images are uploaded into the NIBIN database, which searches for possible matches or hits. Hit information is sent back to the local jurisdiction, where a firearms examiner views the possible-match images, side by side, on a computer screen. If they appear to match, it is called an "unconfirmed hit." It then becomes necessary to "confirm" the hit. To do this, the actual piece of evidence (the spent cartridge or bullet from a test-fire or from a crime scene) must be manually reviewed using a comparison microscope. If evidence is stored at another laboratory or law enforcement agency, the laboratory originating the unconfirmed hit must secure the evidence and examine it. Only after this visual examination can a hit be confirmed, at which time a "hit report" is generated and sent by the crime laboratory to police investigators.
Law enforcement can use a confirmed hit in two ways: tactically or strategically. On the tactical level, a NIBIN hit can link crimes that were not previously known to be related and, in turn, can help identify suspects. On the strategic level, NIBIN hits can help law enforcement understand larger patterns of gun crime, including gun usage; gun sharing; and the gun-related criminal activities of street gangs, drug cartels, outlaw motorcycle gangs and other organized crime entities.
See "How ATF Is Working to Improve NIBIN" by ATF Senior Special Agent John Risenhoover, national coordinator for the NIBIN program.
How Was the Study Conducted?
After a competitive solicitation process, NIJ awarded $341,807 to researchers at four universities: William King and William Wells, from Sam Houston State University; Charles Katz, from Arizona State University; Edward Maguire, from American University; and James Frank, from the University of Cincinnati. This research team collected data from four sources:
- Input and confirmed hit data from June 2006 through July 2012 from all 150 NIBIN sites.
- Detailed data from 2007 to 2012 on 8,231 NIBIN hits from 19 sample sites. These sites were not randomly selected; rather, the researchers chose them in an effort to ensure a range of geographic representation and practices.
- A 2012 survey of publicly funded crime laboratories in the U.S., with a 33 percent response rate from laboratory directors and a 49 percent response rate from firearms sections in crime laboratories.
- Interviews of laboratory directors, firearms section personnel, and all levels of law enforcement at 10 of the 19 sample sites, including detectives who investigated 65 serious violent crimes linked to NIBIN hit reports.
The fourth data source was important because one of the study's goals was to determine how useful NIBIN hits are for investigators. Ultimately, NIBIN is a criminal investigation and intelligence tool. Firearm and tool mark examiners in crime laboratories produce the hits, but they do not act on the hits — law enforcement investigators do.
Prior to the NIJ-funded study, two other entities had looked at NIBIN's operation: the National Research Council (NRC) of the National Academy of Sciences (2008) and the U.S. Department of Justice's Office of the Inspector General, which conducted an audit in 2005. Both of those studies noted the lack of research on how law enforcement actually uses NIBIN information. The NRC report stated that "a full evaluation of the program's performance would consider what happens after a 'hit' is made using NIBIN — whether the information leads to an arrest or a conviction and how large a role the ballistics evidence 'hit' played in achieving those results." The recently released NIJ study addresses this issue.
It is important to note that the NIJ-funded researchers faced some challenges in obtaining data due, in part, to a 50-percent budget cut to the NIBIN program in 2011. That said, they determined that the data were adequate for examining the operation and performance of NIBIN in the 19 sample sites, and independent peer reviewers determined that the research methods were sound and the findings and recommendations for improving the operability and effectiveness of NIBIN were well-grounded.
Does NIBIN "Work"?
Again, keep in mind that there are really two parts to this question: (1) the science of firearm and tool mark examinations and (2) NIBIN's operation in solving crimes. The NIJ NIBIN study addressed only the second part. (To learn more about the science of firearm and tool mark examinations, see "The Science Behind Firearm and Tool Mark Examination.")
Based on their study of 19 of the 150 nationwide NIBIN sites, the researchers made three overarching findings:
- The implementation of NIBIN varies greatly with respect to staffing, data input and hits. Some NIBIN sites were relatively unproductive with respect to the quantity and type of data entered into the system and the hits produced; others were highly productive. Even when hit reports were produced, however, they often lacked the contextual information that could have made them more useful in investigations.
- Generally, there was no feedback from investigators (the "end users" of a NIBIN hit) back to the crime laboratory. This means that hit reports were rarely used strategically to assist in the identification, investigation and prosecution of criminal groups; however, the researchers said that in jurisdictions where this feedback loop occurred, the results were impressive.
- The timeliness of processing evidence and identifying hits varies greatly.
Certainly, as the number of ballistic images in the NIBIN database increases — and as more law enforcement agencies turn to the database for investigative support — the tactical and strategic value of NIBIN will also increase. However, the researchers found in an in-depth look at 65 cases that the suspect had already been identified by law enforcement before the NIBIN hit was produced in half of those cases; in 34 percent of the cases, the suspect had been arrested before the hit.
The bottom line seems to be that criminal investigators rarely used NIBIN hit reports to identify unknown suspects. Police credited a NIBIN hit with helping them identify a suspect in about 10 percent of the cases and assisting in an arrest in about 2 percent. They reported, however, that NIBIN hit reports were useful as background or to confirm (or disconfirm) information provided by suspects, witnesses and informants. The researchers said that this means that police are currently using NIBIN hits and other forms of forensic intelligence in similar ways.
Given all these data, the question is fairly asked: Why isn't NIBIN doing a better job of helping investigators identify and arrest suspects in violent crimes involving a firearm? The researchers explore a number of explanations but primarily point to the often considerable delays in identifying hits.
"Delays in processing ballistic evidence are the single greatest threat to the utility of NIBIN as an investigative tool," the researchers said, noting that in the 19 sample sites, the median elapsed time between a crime and identification of a NIBIN hit was 101 days; the mean was 337 days.
There are a variety of reasons for these delays. Some are imposed on laboratories by outside agencies; others result from cumbersome laboratory procedures. For example, in some crime laboratories, procedures for routing firearms and evidence to the firearms section create delays that the firearms examiners do not control.
"At present, the data in NIBIN are bottlenecked within the NIBIN system," the researchers said. "Labs can only access NIBIN data by viewing it on a screen or printing paper reports."
Also compounding delays is that investigators who want to search NIBIN must go through the firearms section in the crime laboratory; the researchers noted that "[t]his access is not easy for investigators to attain and requests for searches create additional work for firearms personnel."
In addition, the researchers found that there are not enough ATF personnel to analyze and disseminate NIBIN data to local agencies. They also noted that the only metrics currently used to evaluate the performance of NIBIN and NIBIN sites are the numbers of data inputs and hits/matches. Although this information is certainly useful for gauging workload, it fails to offer meaningful outcomes such as arrests, clearances or successful prosecutions. Therefore, among their many recommendations, the researchers state that the swiftness with which hits are produced should be included in a suite of new performance measures.
How Can NIBIN Be Improved?
The final report offers a robust discussion on ways to improve the value of NIBIN. Although most NIBIN sites are not taking full advantage of the program's strategic capabilities, some are. Basically, the researchers found that sites successfully using NIBIN — both tactically and strategically — shared two practices:
- The sites viewed NIBIN as a process involving people, interorganizational relationships and information sharing and not merely as a piece of technology.
- They added information to hit reports (such as geocodes and information from criminal records databases) that improved hits' investigative value.
Among measures that would improve the tactical and strategic value of NIBIN in solving gun crimes, the researchers recommend:
- Creating standardized measures (beyond the numbers of inputs and hits) for evaluating the performance of local NIBIN sites.
- Developing a "force-multiplier" software program to add other criminal intelligence, such as eTrace data, to NIBIN reports.
- Establishing demonstration projects to highlight successful strategic uses of NIBIN.
- Addressing delays in processing ballistic evidence and identifying hits through better cooperation among all stakeholders (that is, beyond crime laboratories).
- Establishing an ATF research and development program to discover, cultivate and test innovative practices at local NIBIN sites, particularly practices to remove impediments to identifying hits in a timely manner.
- Establishing regional NIBIN Centers of Excellence to provide training and technical assistance to local jurisdictions, host regional conferences, create websites and publications to raise the level of professional activity, and create networking among criminal investigators, firearms examiners and firearms technicians.
The bottom line, the researchers said, is that thinking of NIBIN only as a tool that can produce hits "places unnecessary limits upon its potential as a strategic intelligence source." As with many technologies, they added, NIBIN's success "is ultimately a function not only of its technological capacity but also of the interplay between the technology and the organizations and human systems in which it is situated" — that is, the entities entering data into the NIBIN database, the entities using NIBIN to process the inputs and the entities using the hit reports.
For example, the researchers found that hit reports are rarely routed to crime analysis units where they could be used for strategic analyses. "There is generally no information feedback from investigators and end users of NIBIN hits back to labs," they said. "Lab personnel rarely collect systematic information on how NIBIN hits are used or the level of utility of hits for end users."
The researchers also found that although NIBIN contains an incredible amount of information on crimes and weapons that goes well beyond what is contained in a hit report, "the majority of evidence in NIBIN is never connected with other offenses, events or weapons."
For already overburdened firearms examiners to make such connections, the researchers said, is easier said than done. But adding more information to NIBIN — such as geocodes, the names of suspects and victims, and possible gang affiliations — could, they said, help realize NIBIN's full strategic value.
From the
NIJ Journal No. 274, posted October 2014
NCJ 247878
Cite this article as:
Ritter, Nancy, “Study Identifies Ways to Improve ATF Ballistic Evidence Program,” NIJ Journal 274 (2014): 12-18, available at http://nij.gov/journals/274/Pages/ways-to-improve-nibin.aspx.
About the Author
Nancy Ritter is a writer and editor at NIJ.
For More Information
Read the final report, Opening the Black Box of NIBIN: A Descriptive Process and Outcome Evaluation of the Use of NIBIN and Its Effects on Criminal Investigations (pdf, 116 pages).
The Crime Scene Investigator Network gratefully acknowledges the U.S. Department of Justice, Office of Justice Programs, National Institute of Justice, for allowing us to reproduce, in part or in whole, the article Study Identifies Ways to Improve ATF Ballistic Evidence Program. The opinions, findings, and conclusions or recommendations expressed in this article are those of the author and do not necessarily represent the official position or policies of the U.S. Department of Justice.