The "For Lease" signs on shop windows in Nelson have become a predominant window display.
There were now about 30 vacant shops and offices in the central city - a higher number than usual - at least one commercial building owner said.
About 50 retailers joined the Nelson City Council, the inner city promotions group and the economic development agency at a forum this week to figure out how to stem the retreat, and explore ways they might help future-proof the city centre
Renee Wilson, who attended the forum, belongs to a new wave of local retailers keen to keep locals shopping at home.
The 27-year-old recently quit a corporate career to buy her own retail business - an up-market women's clothing store Palm.
She was aware of the current climate, but did her research.
"It was a little bit concerning for me, however I'm young, I do have a bit of business nous and I thought now is a good time in my life to venture in to something like this.
"I've seen the business work and I'm hoping that with my fresh approach I can keep it performing well."
The senior council staff member who hosted the forum, Clare Barton said while it was easy in the past to blame the decline in retail on the rise in online shopping, it was not always the case.
She said feedback from retailers showed the shop floor was still where the money was - but there was another problem:
"We're finding that Nelsonians are shopping around the rest of New Zealand, so how do we get them to buy local?"
Nelson retailers have long blamed rising rents for part of their troubles. Building owners, like Gaire Thompson who owned properties around the country and 10 in central Nelson, looked elsewhere for the root cause.
"The rates issue is one where the central city has been screwed really, in my view, for the last number of years. Commercial inner city rates are about three times residential rates," Mr Thompson said.
The Nelson City Council has made central city enhancement one of four key priorities in its long term plan.
It followed Nelson's drop on the ASB regional economic scoreboard from first to 12th place in just over a year. And a Colliers International retail report that showed commercial rents in Nelson were among the highest in provincial New Zealand.
Clare Barton said the council was responding by altering the rating differential which would reduce rates on commercial property.
"One of the levers that we can influence directly...which we've got control over is the rate component."
Simon Duffy of the city's inner city promotions group, Uniquely Nelson, said despite the empty stores, Nelson still had more going for it, than against.
"Nelson's kind of left field. We're not on the main trunk line - we're not Auckland, Wellington, Christchurch or Dunedin and there's something special about that. There's quite a boutique feel to Nelson."
Mr Duffy said its heritage buildings, some dating back 160 years, created a point of difference.
But Mr Thompson said older buildings had fuelled the exit among national chain stores. He said earthquake strengthening was not tax deductible and building owners had to either increase rent, or wear the cost.
"We get blamed for causing the problem but in fact we're carrying a lot of the problem, to try and keep the place full and looking right."
Keeping it looking right was also high up on Renee Wilson's list. She wanted standards in place for owners and landlords of empty premises, to ensure they did not leave them to disintegrate.
"It's not a nice look in town to have empty buildings, and to be littered with signs. It's a really hard thing to see when you work really hard to present your building well."
Ms Wilson said based on her experience as a consumer, and now as a business owner, she wanted to see access to the city opened up.
"Part of that's around parking and it's quite inconvenient for people."
The forum revealed some good signs that inner city businesses were willing to work together, Mr Duffy said.
"Because people talk about division, whether that's between the authorities and retail and the commercial people. Let's now put the focus back on ourselves and working together as a collective group and take the silos down.
"Humans love silos - take them down and let's get on with it," Mr Duffy said.