6 Nov 2019

Raft of problems with IT systems overhaul revealed

8:48 am on 6 November 2019

Documents show big problems surrounding the overhaul of the country's over-taxed IT systems for managing highway crashes, snarl-ups and roadworks.

Traffic camera images

The Transport Agency identified ageing technology infrastructure as a continued risk. Photo: RNZ / Alison Ballance

The Transport Agency ditched a multi-million dollar contract soon after signing it, then the alternative system failed at its first deployment.

The agency has identified its "ageing technology infrastructure and applications as a continued risk", in the face of an increasing volume of "unpredictable incidents" on the roads.

So it set out in 2016 to build a national incident and event management system, or Niems, to consolidate the way its three big-city operations centres (Auckland, Wellington and Christchurch) manage traffic using highway cameras and sensors. The plan was to extend this to all councils later.

The system would be key, for instance, to providing the right information to traffic light operators - to ensure they can phase the lights if there is a crash, to keep intersections safe and quickly and safely direct traffic elsewhere.

One goal of Niems is cutting the time to get things moving again - each minute is worth a couple of million dollars to the economy.

The first hurdle was when Auckland Transport pulled out of a joint Niems in 2016, to build its own incident management system using a different IT system.

But NZTA went ahead, undertaking what it called "an exhaustive and rigourous process" to find a supplier.

It signed up Castle Rock Associates in Oregon - a "clear winner", documents say - after sending five agency staff on a $50,000, three-city trip to the US to vet the company.

Some initial systems tests were being done by November 2016, and there were numerous meetings with police, fire and transport operations centre staff in preparation.

But six months later, the agency scrapped the Castle Rock contract entirely.

"With four months left to deliver, $3.3m of the total $5m budget has been used, and there has been slow delivery of a usable product to date," the agency's IT governance group was told on June 5, 2017.

This advice came in a short presentation from Martin McMullan, director of the agency's own high-tech unit Connected Journeys, who said an independent assessment had also faulted the project.

Martin McMullan

Martin McMullan, director of Connected Journeys. Photo: Supplied

NZTA has not released this assessment to RNZ.

Sources close to the project say Castle Rock's product was not state-of-the-art and did not have a great user interface, but it was good enough for what operators, working under pressure and not so interested in interfaces, need.

Mr McMullan's unit was later castigated within NZTA for having "duplicate functions, a lack of clarity in the long-term direction, no workplan and with funding all set up for the short term". It had 40 projects on the go when it was disbanded this year, with others having fallen by the wayside, according to an investigation.

It also had very loose financial controls.

The governance group endorsed an immediate pullout from Castle Rock - even though, the documents show, the group did not know the scope, cost or benefits of an alternative system promised by Connected Journeys.

By December 2017, the governance group still did not know the nature of what it had approved six months before.

"Approval of the rescoped project was dependent on the group reviewing and approving the relevant project documentation ... [on Connected Journeys] coming back with a clear scope, financial, costs/benefits, and do-ability," the group's minutes say.

But by then the agency had already told the incoming Transport Minister Phil Twyford it would begin the Niems rollout by mid-2018.

A Deloitte review this year found Connected Journeys lacked accountability because "governance teams lacked visibility" over it, however, in this case the group was informed about and endorsed the Niems u-turn. Two of the five voting members of that governance group are still in senior posts at NZTA.

Having dropped Castle Rock, the agency adapted the old system to work in the Cloud.

After a series of delays, it tried to roll it out for the first time, via the capital's transport operations centre on Wellington roads on June 5, 2018 - even though a severe weather warning was in place.

It was dealing with real-time incidents on the roads, but the documents suggest it was a mess.

The launch debrief reads: "Details not saving or saving in the wrong event"; "Overall system slowness"; "Event initiator unable to reopen event".

NZTA Chairman Sir Brian Roche

NZTA chairperson Sir Brian Roche. Photo: RNZ / Dom Thomas

And because of poor planning, repairs were held up so problems went on from the Friday until the Monday.

"The developer tried reducing the number of servers but this caused the SCATS team to lose access to ILS for a time."

This meant teams operating traffic lights could not access the crucial Incident Logging System.

"With a severe weather warning in place for the next 24 hours, alternative options were considered."

It was deployed without the usual systems integration, performance testing, rollback or disaster recovery plans.

With incidents or events being duplicated or overwritten, if police needed any information for investigation then that would have been at risk.

This was meant to be the system's big launch, but they had to pull out.

A debrief indicates they tried again in late June 2018, with better results, though the system's technology remained of limited use.

It is not clear just where the promised national rollout of the Niems system has got to, nor its integration with roadwork planning systems Myworksites and Forward Works Viewer, or crucially, Auckland Transport's separate system.

This integration was the number one factor that Martin McMullan stressed was crucial in his case against Castle Rock, even though the Castle Rock project team had aimed to integrate Myworksites.

NZTA has not disclosed to RNZ what it has written off from Castle Rock, or if it is carrying it on its books even though it is not a usable asset.

Figure 'incorrectly entered'

Its annual reviews show the contract was worth $4.5m.

But after RNZ began asking questions, the agency said that figure "was incorrectly entered" into parliamentary documents, and it was in fact worth $1.5m.

The agency also told RNZ in recent days that the whole Niems project budget was $2.4m.

Yet its own documents give a figure of $6m, and show that by mid-2017 it had already spent $3.3m.

The agency has repeatedly put the caveat "to the best of our knowledge" around its answers.

The Deloitte review of Connected Journeys found that "budgets are not closely monitored or adhered to. In some instances, funds from one project have been transferred to another project".

There was "dominance and confusion" around key jobs, such as communicating incidents, roadworks and closures, which are "particularly critical".

The new NZTA chairperson Sir Brian Roche said he was not aware of the Castle Rock matter, that the agency had been reviewed enough and he was focused on the future.

"I could spend the next two years redefining the problems that we've already sought to address or I could spend the next two years trying to create a future which creates a different environment," he said.

He suggested RNZ ask the Auditor-General. The AG said it did not audit Connected Journeys in its 2017-18 audit of parts of NZTA, which pronounced its financial controls as "good".

In that audit, the AG said NZTA had "a 'keep the lights on' programme to remediate ageing infrastructure ... This is a multi-year/ongoing programme of work to ensure that ... systems remain fit for purpose. A more strategic/long-term approach has been developed to remediate ageing technology infrastructure".

Numerous reviews have been held into the agency this year, and one is still going on into contracts awarded by Martin McMullan, that is overdue since mid-2019.

Minister for Urban Development Phil Twyford

Transport Minister Phil Twyford Photo: VNP / Daniela Maoate-Cox

The Minister Phil Twyford told RNZ last month he had received no advice about hi-tech write-offs at NZTA.

At the controls

A national incident and event management system would coordinate:

  • the active traffic system SCATS that controls traffic lights.
  • the Dynac system that runs the overhead message boards, speed advisory signs and estimated journey time signs.
  • the DVTEL that controls CCTV cameras on highways.
  • while calling on information from police, other emergency services, and the MetService.
  • and putting out real-time advisories on social media.
  • Read the OIA here:

Get the RNZ app

for ad-free news and current affairs