Immigration Minister says he would have made a different decision to the Immigration Department, if the case of Sultan Ali Abduri Ali Akbari had been brought to his attention.
Akbari was released on parole two months ago, after he was sentenced to two years' imprisonment on five counts of indecent acts with a girl under 12 and one of the indecent assault of a woman.
The 58-year-old had previously been convicted, in 2013, of indecently assaulting a woman.
But Immigration New Zealand didn't deport Akbari, saying he could stay in the country provided he didn't re-offend over the next five years.
It made that call using powers directly delegated to 11 or 12 senior managers within the department by the immigration minister, Michael Woodhouse.
But Mr Woodhouse has temporarily suspended those powers, saying that decision did not stack up with the values he sets.
"If that decision had been brought to my attention, I would have made a different [decision]."
Akbari's probation conditions included undergoing counselling.
He was not allowed to have contact with anyone under the age of 16, unless approved by his probation officer.
Immigration New Zealand said the risk of Akbari reoffending, and his family connections and support in New Zealand, contributed to the decision not to deport him.
However, an immigration lawyer Simon Laurent said Mr Woodhouse's decision showed the government was highly embarrassed.
"It sounds like in this situation he is effectively cancelling a whole class of those decision-making powers."
"Is speaks to me that the minister is not only not particularly happy about it, but he fears it's going to be a considerable embarrassment."
While Mr Woodhouse stopped short of saying he was embarrassed when he found out, he said it became clear as more information came to light that he would have to take action.
"I was disappointed, surprised, and the more I found out about it, the more I felt it was necessary for me to take action in order to ensure it doesn't happen again. "
Over the next fortnight he would personally handle all deportation-related matters, and would have discussions with upper management within Immigration to make sure they were on the same page, he said.
"Two things will happen. I will reach a shared understanding about when it's appropriate for a decision-maker to make a decision on my behalf and when it's appropriate for the case to be referred to me personally."
"Secondly, I want to make sure we are absolutely lock-step around those areas that I think are unambiguously cases for deportation."
Mr Woodhouse said those cases included foreign residents who had committed sexual crimes.