The Police Chief, the Professional Voice of Law Enforcement
Advanced Search
July 2014HomeSite MapContact UsFAQsSubscribe/Renew/UpdateIACP

Current Issue
Search Archives
Web-Only Articles
About Police Chief
Advertising
Editorial
Subscribe/Renew/Update
Law Enforcement Jobs
buyers Your Oppinion

 
IACP
Back to Archives | Back to May 2008 Contents 

The Next Step in Accountability Driven Leadership: "CompStating" the CompStat Data

By Ronal W. Serpas, Chief of Police; and Matthew Morley, Research Analyst II, Metropolitan Nashville, Tennessee, Police Department


here can be little debate that CompStat processes in police agencies across the United States have revolutionized crime fighting and made communities safer. For more than a decade, these innovations have brought about greatly enhanced accountability and empowerment to police officers at the neighborhood level. But CompStat is not without its critics and detractors, including some who allege that the reported crime reductions are a result of “cooking the books.” As CompStat has evolved over the years, the next logical step is to identify aggressive and thorough ways to analyze the outcomes of a department’s use of data to ensure its integrity, reliability, and validity.1 By “CompStating” the CompStat data, police executives can bring into play powerful tools that protect the integrity of crime data and, most importantly, the integrity of the agency as a whole.

For all the positive attributes of CompStat, there have been unfortunate examples in some jurisdictions where allegations of “numbers fixing” or crime data falsification have been found to be valid. The refrain among those who shun the accountability that professionally led CompStat, or accountability-driven leadership, can bring has been unrelenting: field commanders are under unrealistic pressure to reduce crime; therefore, their only option is to manipulate the data. There is an equally powerful group that believes for academic, political, social, or economic reasons—that police cannot make a difference in crime; therefore, any reduction in reported crime must be artificial. When these two forces are combined, they create a perfect media storm directed straight at those who embrace the CompStat philosophy. Unfortunately, the few examples of unethical behavior by a handful of departments add even more fuel to this fire.

Further complicating this debate is that there are many different sources of data concerning crime in the United States—another prime opportunity to “cloud the issue.” It is clear that when the public and the media are educated to understand the differences between Uniform Crime Reports (UCR), the National Incident-Based Reporting System, and the National Crime Victimization Survey, the perceptions of inaccurate reporting can be ameliorated. Thus police chiefs should be very aggressive in explaining these differences to internal and external audiences.

A key part of such public discussions should be information gleaned from the accurate and timely analysis of the CompStat data. Using internal and external data sources as well as quantitative and qualitative analysis of these data, police chiefs are able to turn back erroneous accusations made against their officers and leaders. It is far too easy for a critic to point to one, or a handful, of examples of honest errors on the part of officers and supervisors in reporting crimes and then cascade those examples into a general indictment against the department’s crime reporting and investigative outcomes as illicit. Without carefully designed audits to confirm the accuracy of an agency’s efforts, these allegations can gain traction and cause irreparable harm to the agency’s reputation.

Agencies that cannot substantiate the efforts of their officers can become too focused on how the findings of a police investigation changing a UCR Part I crime to a Part II crime will “appear” to the public.2 It would be an abomination of U.S. policing to relegate its efforts to writing reports instead of criminal investigation and fighting crime. Policing in the United States should never consign itself simply to writing reports as the primary mission. Crime fighting and investigations are the most important charges. Aggressive, thorough, and multifaceted data analysis can provide the support chiefs need to show that crime has in fact been reduced by law enforcement efforts or, conversely, to ferret out illegal or unethical behavior of a department member.

Police agencies worldwide have clearly demonstrated that when focused on the job at hand, irrespective of economy, social ills, and so on, they can make and have made a difference in reducing crime rates. Certainly, there is no doubt that solid educational systems, robust economies, effective offender reentry strategies, and stable social organizations within communities and neighborhoods are key components in a permanent solution to the problem of crime. Many of these issues are out of the direct control of police. Nevertheless, the evidence is overwhelming in many large and small U.S. cities: when law enforcement agencies focus on controlling the things they can control, crime rates fall. The reduction is directly attributable to the efforts of dedicated officers who do not “cook the books” or use deceit and unethical behavior. Accountability-driven leadership provides for “CompStating” the CompStat data; in doing so, agencies can collect, analyze, and report the evidence necessary to support resoundingly the fact that they have made a difference.

Police must hold on to their duties of crime fighting and investigation of crimes with the appropriate outcome to follow, irrespective of UCR Part I or Part II offense classification. A major component of accountability-driven leadership is auditing CompStat data throughout a given agency, which leads to analysis that further supports the mission of the agency. Over the past several years, the Metropolitan Nashville Police Department (MNPD) has directed its crime analysis staff to develop many audits shared in this article. These audits review data under the control of field commanders but also, more importantly, analyze many data points that are not under the control of field personnel. As stated above, if the point of CompStat criticism is that field commanders are under such pressure to reduce crime that they may be willing to falsify their results, audits completed on data compiled outside of their control are key to documenting that their efforts are valid and professional or to providing evidence of unprofessional conduct.

About the MNPD

The Metropolitan Government of Nashville and Davidson County was the first consolidated city/county government in the United States, establishing this form of government in 1963. The MNPD is the chief law enforcement agency of the government, policing a geographical area of 533 square miles, with a resident population of approximately 610,000. Nashville, also known as “Music City USA,” is home to the Country Music Hall of Fame, Tennessee Titans National Football League franchise, Nashville Predators National Hockey League franchise, and several noteworthy national landmarks and hosts upwards of 10 million visitors per year. The MNPD has an authorized sworn strength of 1,312, with approximately 500 full- and part-time professional support staff. Nashville has recorded overall crime declines each of the four years from 2004 through 2007. The overall crime rate is currently at an 18-year low; the violent crime rate is the lowest since 1990, and the property crime rate is the lowest in nearly 30 years.

The MNPD uses a three-pronged approach in auditing the efforts and actions of the department: actions reviewed by field commanders, data points beyond the control of field commanders, and perceptions and impressions of the public on the performance of the MNPD. The combination of these three different views of data analysis can serve to provide solid evidence either to substantiate allegations of misconduct (such as underreporting crime or overreporting departmental outcomes) or to exonerate, fully, the conduct called into question.

The MNPD is responsible for maintaining an enormous amount of data essential to the Nashville criminal justice system. Analyses are performed daily on these data to provide decision makers in the MNPD with vital information in order to allocate resources according to specific and immediate needs. This data analysis also seeks to review and authenticate the outcomes of investigations and the performance of the department. To increase data integrity and accuracy in the reporting and analysis of crimes and employee behavior, the Crime Analysis Unit has implemented several processes for evaluating these data.

Lieutenant Survey Analysis

The Lieutenant Survey is the most basic of these audit processes and uses sound methodological approaches to obtain and analyze data. This analysis may be considered under the control of field commanders, as it is completed by personnel under their direct supervision. Each month, a random list of victims of crime is generated, and individuals on the list are selected to participate in the survey. The main purpose of this process is to survey victims of criminal offenses randomly to determine if the police report on file accurately depicts the events that occurred, as reported to the original officer by the victim.

It is an important audit link for the shift lieutenant to have direct conversations with citizens, not only to ensure the integrity of the reports as submitted by the officers and approved by their direct supervisors but also to hear directly from a large number of citizens how their employees are performing their duties. Since 2004, an average of 86,418 Part I and Part II criminal offenses have been reported each year to the MNPD. The MNPD attempts to contact 1,296 victims each year (18 contacts per six precincts per month). (Juvenile victims as well as victims of rape and domestic abuse are excluded.) To ensure reliability of the survey results, the MNPD needs to contact at least 382 victims a year in order to obtain a confidence level of 95 percent with a confidence interval of ±5 percent, an industry standard. The survey has maintained a confidence level of 99 percent, with a confidence interval of ±4.2 percent, by successfully contacting over 2,600 victims of criminal offenses.

Since January 2005, MNPD patrol lieutenants have successfully contacted 70.8 percent of the targeted population, who have confirmed at a 99 percent rate that the police report, as read to them, describes in accurate detail the events surrounding the crimes that were committed against them. One constant among critics of CompStat and accountability-driven leadership is that officers, under the duress or direction of supervisors, can purposefully “downgrade” the crime as reported by the victim. In this survey, the actual crime report is read to the victim to confirm that the report is an accurate reflection of the victim’s statement. Clearly there may be differences in what type of crime offense classification may derive from a police investigation, but by ensuring that the victim’s statements are accurately recorded, there is less opportunity for misrepresentation (purposeful or mistaken) of statements to go undetected.

In addition, victims are polled on their opinions of the interaction with the responding police officer and/or detective. Such questions are asked so that the MNPD can monitor and assess victims’ perceptions of the behavior of department officers. After the interview has been conducted, the surveying patrol lieutenant provides additional input on whether the officer accurately filed the facts of the incident and acted appropriately both toward the victim and within the policies and procedures of the MNPD. All of this information is then forwarded to Crime Analysis for further review and storage of the data.

Critically important to this assessment by the patrol lieutenant is whether the crime is classified as consistent with the facts as presented by the statements of victims and the investigative outcomes of the officer or detective. If an error is found, the patrol lieutenant is then responsible for ensuring that the appropriate actions follow immediately. Finally, it has been the experience of the MNPD that these follow-up phone contacts by patrol lieutenants serve as a valuable opportunity for police leadership to interact with the public and for the public to interact with police leadership.


Virtual Audit Teams

As many police departments in the United States have suffered funding reductions over the last several years, internal auditing processes have in many cases been cut or reduced. Inspection units are often cut to ensure that staff members remain focused on calls for service and criminal investigations. However, with the use of virtual audit teams, departments can maintain aggressive efforts at auditing the work of units and commands.

Virtual audit teams in the MNPD replicate those created in the Washington State Patrol between 2002 and 2003.3 A captain assigned to the chief’s office oversees a group of temporarily assigned police lieutenants and civilian equivalents who are dispatched in teams of five to seven, usually for one week, to conduct full audits of organizational units throughout the department. Virtual audit team members are not regularly assigned members of units under the control of the field commanders or the units being audited. The virtual audit team uses Commission on Accreditation for Law Enforcement Agencies (CALEA) standards and department policies to create audit items. In the case of field units and investigative components, audit teams review random samples of police reports as well as supplemental police reports that are submitted to change a crime’s investigative status or Part I/Part II offense status. In doing so, another audit or review of the outcome of police investigations is analyzed—in this case beyond the control of the field commanders.

Team members are provided with training on how to conduct the audits by the captain assigned to the chief’s office. Beyond serving the obvious purpose of conducting thorough audits of units, managers taking part in the audit teams gain valuable experience to further their career aspirations.

Send-Back Reports

Another line of audit occurs when officers submit a report to their supervisors for review. This process falls under the control of field commanders, who are responsible for ensuring that any identified error is corrected. However, once the supervisor reviews and signs all submitted reports, they are then sent to the Records Division, whose staff members review each report manually. If any errors are detected, the Records Division enters the report into the system as a “send-back” report and consequently sends a notice to the officer to return to headquarters to correct the errors. Records Division personnel are trained to ensure that crime elements, statements, investigative action, and offense classification are correctly documented. Crime analysis staff members track these data to identify any trends in (1) the number of send-back reports identified each month; (2) the number of send-back reports outstanding at the end of each month; (3) the number of reports sent back to reclassify an offense; and (4) the average number of days it takes an officer to correct a sendback report. The MNPD also monitors send-back reports to learn if there are any consistencies in officers or supervisors who are disproportionately represented in the send-back population and if there is any consistency in the reason reports are sent back. An officer or supervisor who is maliciously changing reports will be easily identified in this analysis, as well as officers or supervisors needing remedial training. In this case, the Records Division and the Crime Analysis Unit provide another review of data without the involvement of field commanders or supervisors. (Figure 1 provides data on send-back reports in 2007.)

Computer-Aided Dispatch Data

As evidence of downgraded crimes, CompStat critics will often cite the variation, based upon a cursory comparison, of the difference in call-for-service (CFS) data and UCR Part I crime or Part II crime reporting data. In other words, if there is not 100 percent consistency between a CFS for a robbery call and a UCR report filed as a robbery, then the data must have been illicitly manipulated. In an effort to ensure that this practice is not occurring in Nashville, crime analysis staff members compare on a global level the CFSs to police reports filed as a result of officer responses to the original CFSs. In Nashville, it is important to note that the dispatch function is not under the control of the MNPD. Trends tracking back to 2003 hold steady that approximately 80 percent of all Part I crime CFSs are subsequently followed by a police report that is filed as a Part I crime offense report. In addition, the same trend holds steady when analyzing Part II crime offenses: 80 percent of all Part II crime CFSs are subsequently entered after a police investigation as a Part II crime report when the offense report is filed. This consistency also demonstrates a predictable degree of difference in what call takers and dispatchers classify for dispatch as crimes, based upon the original information exchange, versus the actual outcome of a police investigation. Because these trends are monitored, sudden variations would prompt further analysis to determine the cause.

Without variation of these trends over time, this additional level of analysis further demonstrates consistency and reliability in the MNPD’s offense-reporting practices. This analysis is based on decisions made in the field but analyzed beyond the field commanders’ involvement.

Impact of Original and Supplemental Reports on Part I and Part II Crime Reports

Another allegation often used to support critics’ contentions that crimes are being statistically downgraded rather than actually reduced in number is the difference between the “type” of CFSs (such as robbery, assault, burglary, and so on) and what the department actually reports through original offense reports for a given call record. In other words, the allegation would be that officers are purposefully changing reports for Part I crimes, such as robbery, to miscellaneous reports for Part II crimes, such as possession of stolen property.

Unlike the analysis in the previous section, which looked globally at all CFSs and subsequent reports filed, this analysis drills down to the type of CFS and reports filed as linked to that CFS record. This analysis also captures reports that are filed according to the categories that are beyond those the CFS registered originally.

As the charts in figure 2 demonstrate for the year 2007, the MNPD’s reporting of crimes, as compared by Part I crime type with the CFS, tracks very closely. In particular, the MNPD reports more assaults and robberies than the CFS rates indicate for these calls—the result of officers filing charges and reports based on follow-up investigations of these crimes or new investigations. The point here is obvious: if an agency can point to data showing that it files reports of more crimes of a particular type than CFS data indicate, it soundly defeats the contention that crimes are being purposely downgraded. This analysis provides another excellent example of agency data that clearly would not support accusations of inappropriate or unethical downgrading of crimes. This is an example of a check on the outcomes of decisions made by field commanders that is beyond their involvement in data analysis.

Another useful analysis of this type of data, one that is straightforward and easy to compile, is comparing the actual number of Part I reports that are changed to Part II crime reports, and vice versa, to the total number of Part I and Part II crimes reported. For example, it is easy and common for a critic to point to a number of Part I offense reports that were changed to Part II offenses through supplemental reports and argue that sinister actions are the cause. As can be seen in table 1, for the calendar year 2007, a total of 118 supplemental reports were filed to change a Part I crime to a Part II crime, and 42 supplemental reports were filed to change a Part II crime to a Part I crime, for a net difference of 76 cases. However, in the same time frame, 42,455 Part I crime reports were filed. In other words, police investigations that led to supplemental reports that reclassified Part I crimes to Part II crimes accounted for less than one-fifth of one percent (0.18 percent) of the Part I crimes reported to the community and to the FBI’s Uniform Crime Reports total. The negligible percentage that comes out of this analysis, readily available and ongoing in nature, should immediately thwart any attempt to argue to the public that there is evidence of supplemental reports used to systematically downgrade crime reports unethically. Agencies finding errors during their review of supplemental reports have ways of detecting and dealing with those errors. (Suggested analysis techniques to search for such errors are included in the next section.) But in this analysis, erroneously completed supplemental reports would have no impact on the agency’s overall crime reporting data, since the total number of supplemental reports is so small. This analysis reviews actions under the control of commanders but is completed by the Crime Analysis Unit.

Analysis of Supplemental Reports

Supplemental reports submitted by officers and detectives and approved by supervisors can be analyzed beyond the influence of field commanders. This analysis provides additional review of the reasons for and trends of the submission of supplement reports to change classifications of original reports, such as offense change (Part I to Part II crimes and so on), status change (unfounded, cleared by arrest, and so on), or both. The first two charts in table 2 show these data.

Critics argue that officers, detectives, or supervisors can submit falsified supplemental reports to alter crime statistics to deflate crime offenses or inflate the activities of their agency by artificial means. Information such as whether the classification of an offense was changed from a Part I to Part II crime, or if the status was updated to a clearance (by arrest, exception, or an unfounded charge) is carefully monitored throughout the agency (this type of analysis is also conducted as part of the Virtual Audit Team analysis). The number of officers and supervisors submitting supplements is also analyzed. The third chart shows the analysis of the reports of officers and detectives submitting a significantly greater number (greater than three standard deviations) of supplements. These officer or detective reports are then scrutinized even further to validate the accuracy of the reports used to justify changes in offense or status classification. Depending on the total number of reports, all reports, or a random sample of reports, are printed and reviewed by the Crime Analysis Unit Captain to verify manually the validity of the reports and actions of those who submitted, as well as the actions of supervisors who approved those reports. This analysis is also beyond the influence of field or investigative commanders.

Felony Charge Movement

Yet another analysis performed outside of the control of field commanders is a review of arrest reports and arrest files transferred to the office of the district attorney (DA) upon completion of the related investigations. In the event a person is charged with the wrong crime (meaning that lesser offenses might result in crime downgrading, or higher offenses might indicate overreporting the efforts of the department), the DA must file additional paperwork to charge the suspect with the correct offense. The MNPD Crime Analysis Unit has designed an analysis that tracks any updates made to felony charges throughout the life of each case (including grand jury, indictment, and disposition) in the Criminal Justice Information System, a database outside the MNPD’s database system. It is understood that many changes are made due to prosecutorial decisions and needs. However, this analysis is ideally situated to learn of patterns of behavior that could signal inappropriate police action as well.

The analysis also tracks the disposition of charges that do not undergo changes. There has been a very slight decrease (1.0) in the percent of charges changed at indictment from 2006 to 2007. There was no difference (0.0) in the percent of charges changed at disposition from 2006 to 2007. This analysis seems to indicate that additional examination would not be over- or undercharging suspects to inflate or deflate arrest statistics, as well as Part I or Part II crime reports, in the FBI’s Uniform Crime Reports.

Criminal Court Case Flow: Dismissal Rate and Review

Detailed analysis of data over the last three years provides input into where felony cases may be stalling in the criminal justice system and where the efficiency of case processing is optimal. In Nashville, all felonies, excluding cases resolved through plea bargaining, flow through the grand jury. In 2005, the year this audit was implemented, the MNPD discovered that it was losing a significant number of cases to dismissals before the indictments were issued by the grand jury (see figure 3). As a result, the department created a sergeant’s position to serve as a liaison with the grand jury section in the DA’s office. One of this sergeant’s most important functions is to help the DA’s office prepare cases going to the grand jury. In addition, this sergeant was to identify cases dismissed out of the grand jury and provide an analysis each month on the reason each case was dismissed. This position has provided much-needed assistance to the grand jury section of the DA’s office as well as a great deal of insight to the MNPD on training topics for case preparation. As a result, the number of warrants dismissed out of the grand jury dropped by 24.0 percent in 2006. In 2007, the number increased slightly, by 6.8 percent, over 2006, but it was still well below the total for 2005 (see figure 4).

The analysis of all felony cases as they flow through the criminal justice process provides another excellent opportunity for evaluation outside of the control of field or investigative commanders of investigations and reports submitted by officers or detectives. Trends can be identified quickly if one component of the process demonstrates significant and unexplained variation in arrest charges compared with the amount of expected variation.

As the final assessment of felony cases, each MNPD division commander is provided with a monthly list detailing the prosecuting officer, offense, name of defendant, case number, and final disposition of each case. Routinely, senior command staff members request an accountability report from component divisions. In particular, field and investigative commanders are expected to provide insight and documented evaluation to senior commanders on each Part I criminal case that was dismissed throughout the year. This is another analysis that could provide information or evidence of officers or supervisors inappropriately downgrading or upgrading charges and arrest decisions as well as reporting. Furthermore, by analyzing each dismissed case, the MNPD has been able to learn lessons on how best to improve its performance.

Since this practice was implemented in 2005, the MNPD’s conviction rate has increased at an average rate of 1.5 percent each year, while dismissals have decreased at an average rate of 1.7 percent each year (see figure 5 for details). Total dispositions in criminal court have increased at an average annual rate of 2.3 percent since 2005.

Polling Data of Davidson County Residents and Businesses

Agencies can use sophisticated and professional polling services to assess the perceptions and observations of the public at large—still another type of audit that can be performed beyond the control of field commanders. For any organization, public or private, to survive, it must be seen as having the confidence of the audience served. Such confidence is vitally important to the MNPD. Changes in the department brought about through accountability-driven leadership have raised and maintained the community’s positive perception, as shown through semiannual citizen satisfaction surveys conducted by Wilson Research Strategies, based in Oklahoma City. These surveys, which began in June 2005, consistently show that approximately 80 percent of adults and businesses are satisfied or very satisfied with the overall competence of the department and the quality of service delivered.

Perhaps most importantly, the surveys have shown that the changes incorporated in the MNPD from accountability-driven leadership have raised the positive perception of the department’s competence. More than 80 percent of adults and 82 percent of businesses over this series of surveys are satisfied or very satisfied with the overall competence of the MNPD. This point corresponds with the 99 percent of the 2,600 crime victims over the last four years who confirmed to patrol lieutenants that the incident report filed matched their recollection of the incident. The public’s satisfaction with the MNPD’s competence would be negatively affected (significantly) if the department reported events that did not match the victims’ descriptions of incidents.

This series of surveys also revealed that 81 percent of adults and 83 percent of businesses feel safe in their neighborhoods or businesses. These survey findings reflect an overall confidence and trust in the MNPD that has been built through hard work, cooperation, honesty, and involvement with the community over the years. It is highly unlikely that the MNPD would experience such significant and continuing support from the citizens and businesses of Nashville if members of the department schemed to systematically downgrade or overreport its efforts.

Conclusion

The use of CompStat to focus agencies on crime fighting has proven successful throughout the United States. Accountability-driven leadership, which builds on the foundation of CompStat, provides for the critical step of “CompStating” the CompStat data—the next step in further professionalizing policing in the United States. ■   

Notes:

1See Ronal W. Serpas, “Beyond CompStat: Accountability-Driven Leadership,” The Police Chief 71, no. 1 (January 2004): 17–23.
2Generally speaking, UCR Part I crimes are serious, whereas Part II crimes are minor.
3One of the authors (R.W.S.), then chief of the Washington State Patrol, initiated the creation of this unit.


Top

 

From The Police Chief, vol. LXXV, no. 5, May 2008. Copyright held by the International Association of Chiefs of Police, 515 North Washington Street, Alexandria, VA 22314 USA.








The official publication of the International Association of Chiefs of Police.
The online version of the Police Chief Magazine is possible through a grant from the IACP Foundation. To learn more about the IACP Foundation, click here.

All contents Copyright © 2003 - International Association of Chiefs of Police. All Rights Reserved.
Copyright and Trademark Notice | Member and Non-Member Supplied Information | Links Policy

44 Canal Center Plaza, Suite 200, Alexandria, VA USA 22314 phone: 703.836.6767 or 1.800.THE IACP fax: 703.836.4543

Created by Matrix Group International, Inc.®