Police in the Armed Response Teams failed to record their callouts properly on almost every occasion during the trial's first two months.
One of the vehicles used in the trial. Photo: RNZ / Liu Chen
The six-month experiment ended in April. The trial involved a group of officers in three regions - Counties Manukau, Waikato and Canterbury - equipped with guns on their hips at all times.
Officers were expected to record and submit data on every single call-out. In the first two months, data from five out of every six callouts was missing.
Police did not provide the total rate of responses for the remainder of the trial when asked by RNZ. Instead a spokesperson said the evaluation of the trial would "only be one of the factors taken into consideration as part of our decision making".
Documents to police, authored by the Evidence Based Policing Centre (EBPC), state that records from the End of Deployment (EOD) forms should be filled out after "each operation or call for service" and would be "essential" to understanding how the patrols had been used.
Two months into the trial, just 17 percent of forms were completed.
"There is still a level of underreporting that, while not posing an immediate problem, means that a complete picture is not available. As a fraction of the number of incidents ARTs have been deployed to, the number of EOD forms received by the EBPC is still quite low (17%)," an EBPC document from January this year said.
A report from December stated: "These discrepancies likely reflect a general under-reporting of deployment activity and selective reporting practices across each district."
The December report went on to show concern that officers were categorising EOD submissions wrongly, which meant some information was not being collected.
"From an evaluation point of view this is problematic because complete data is not available for a large number of deployments... If such practices are being undertaken, this compromises the fullness of the data available to the EBPC and will undermine the evaluation."
Thomas Lumley. Photo: RNZ /Dom Thomas
Auckland University professor of biostatistics Thomas Lumley said this level of reporting damaged the integrity of the data.
"Having a low response rate like that, to forms that were supposed to be filled out 100 percent of the time, and where the police were the ones who had made the choices as to which forms were filled out, completely undermines this as an evidence-based summary of what was happening in the trials," Dr Lumley said.
"If they're going to choose which ones are reported, they might selectively report ones where they felt that having armed police there was helpful. They might be less likely to report ones where they didn't feel it was helpful, or ones where they didn't think it was relevant, like traffic stops, but where the other people involved might have thought it was relevant that the police had guns."
In addition to the general lack of data, the vast majority of the most attended jobs - vehicle stops - were categorically ignored.
There were so few EOD forms for vehicle or people stops - with the Police code 3T turnover - that the Evidence Based Policing Centre asked whether the small amount of data that did exist should be left out of the analysis altogether.
"Principally, a decision needs to be made regarding the absence of reporting on 3T incidents. As previously mentioned, there are pragmatic reasons for not submitting an EOD form for every 3T incident, though clarity needs to be established on whether these incidents form part of the deployment analysis or should be omitted altogether."
Lowest number of reports in Waikato
The Waikato officers turned in the lowest proportion of the reports in the first two months - just more than eight percent, or one in every 12 events.
The Waikato ARTs attended more than 800 turnover (3T) events, and close to 200 bail checks (incident code 5K) - the two most attended incident codes for the region. However, fewer than 10 EOD forms were filled out for each of these codes.
This was despite the forms being made accessible on the police's mobile devices.
A former police officer and investigator, Tim McKinnel, said it was now difficult for police to properly evaluate the trial given there were such major holes in the data.
"It's bizarre in the circumstances, given police knew how closely scrutinised they would be on the deployment of these teams, that there are such substantial gaps in information."
Evaluation one of several factors - police
RNZ asked police what the rate of End of Deployment responses was for the entire trial, but police did not provide this, nor any indication if it had improved after the first two months.
"No single short-term evaluation can answer the question about the right style of policing, and any conversations we have about this must include the communities we serve, as well as police staff. How the public feels is important as we police with consent of the public, and that is a privilege," an unsigned statement from a police spokesperson said.
"We have acknowledged that the evaluation will only be one of the factors taken into consideration as part of our decision making."
When asked if this was an acknowledgement the evaluation was flawed due to a lack of data, the police employee sent back lines from a previous media release.
"The trial of Armed Response Teams ended on 26 April and a review is under way. The short trial was approved to explore whether it could improve operational responsiveness, particularly to critical and life-threatening incidents, and the safety of both frontline officers and the public.
"The ART's primary purpose was to respond to high-risk critical events. When they weren't attending these incidents, they undertook routine or prevention activities such as 3Ts.
"NZ police remains a routinely unarmed organisation. No officer in any of the Armed Response Teams fired a shot during the trial."
The evaluation is due to be completed this month.
One of the trial's main purposes was to gauge the team's effectiveness during life-threatening incidents. Photo: RNZ / Liu Chen
