Race Relations commissioner Susan Devoy says Sir Peter Leitch's comments this week were offensive and need to be 'fixed up', after earlier saying Sir Peter was the least racist person she knew.
Sir Peter - also known as the Mad Butcher - told a woman with Māori ancestry that Waiheke Island was a "white man's island" and that she should get off it.
In a now-removed video on her Facebook page, 23-year-old Lara Wharepapa Bridger cried as she described the incident with Sir Peter.
She said he approached her and her whānau and started talking, before telling her Waiheke Island was a white man's island.
Sir Peter said he was extremely disappointed she misinterpreted some light-hearted banter.
In a statement released this morning, Dame Susan condemned casual racism.
"Many of us have said or done things that are hurtful to others without really realising what we were saying is offensive: but that's not the end of the story. The important thing is being able to recognise when we've offended someone, to work to resolve it with mana and to make sure we never do it again.
"The thing about so-called casual racism is that it doesn't feel very casual if it happens to you or your family as Lara has shown us."
She said she knew Sir Peter and thought he was a "very good person at heart", but his actions were offensive to Ms Wharepapa Bridger and it "needed to be fixed up".
"I am confident he will do this and it's great to hear the local iwi are able to help mediate as are our own commission mediators. It is pretty clear he had no idea how offensive his words were but he will definitely know this now."
Earlier, Dame Susan told RNZ News it was unlikely his comments were meant to offend.
"I know he's the least racist person I know in the world and yet what he said was obviously taken as offence by that young woman.
"But I wasn't there and I wasn't part of the conversation.
"It's grown real legs, hasn't it?"
Dame Susan said Sir Peter often used light-hearted banter, which could be misinterpreted.
"The last thing he would have wanted in the world was to offend someone, I know that.
"Let's not forget he's done a lot of great work in terms of race relations in New Zealand - providing opportunity and building bridges between different cultures.
"I think it's generational and culturally different these days and he's probably licking his wounds today."
Dame Susan said what was acceptable 40 years ago was not now.
"We at the Commission launched a campaign about casual racism, getting people to stand up and address it.
"The thing to remind ourselves is it's good to have conversations about these issues but also to remind ourselves that we have to reasonable and rational in our discussions about it too.
"Everyone's entitled to have an opinion, but respect each other when you're doing that."
Dame Susan said the mediation service at the Human Rights Commission was the best way to resolve a situation in a neutral environment.
'How do we pull the community together?'
The island's principal iwi said Sir Peter was welcome to sit down with them to make sure all communities on Waiheke Island are looked after.
Ngāti Pāoa spokesperson Hauauru Rawiri said Sir Peter's comments came out of left field, considering his history with rugby league and the Māori and Pasifika communities.
He said he usually judged Sir Peter by his involvement with those groups but the latest comments were surprising.
"Waiheke Island is a really important place for Māori, for Ngāti Pāoa.
"But also it's actually a really unique community and how do we pull that community together?"
Mr Rawiri said those with a high profile should maintain their mana and integrity by being careful about what they say.
Former comedian and personality Mike King stood behind his mate Sir Peter.
"She had every right to call the butcher out but by the same token the butcher made a mistake.
"It doesn't make him a racist and he doesn't deserve the condemnation that some people online are making," Mr King said.
"He didn't kill a child."
Mr King said it was a lesson for everyone to watch what they say.
A statement released by the Vodafone Warriors rugby league team also backed Sir Peter.
In it, managing director Jim Doyle said Sir Peter had been an ardent supporter of the team since 1995 and they had the greatest admiration for him.