Dominic Cummings says he "obviously" will not be quitting as the UK Prime Minister Boris Johnson's chief adviser over claims he broke coronavirus lockdown rules.
He said he did the "right thing" by travelling 260 miles with his family to be near relatives when his wife developed Covid-19 symptoms.
Downing Street said he wanted to ensure he had childcare if he got symptoms.
Labour and the SNP say he flouted the government's own advice and are calling for an urgent inquiry into his conduct.
It comes as the government announced 282 more people had died with coronavirus since Friday, across all settings, bringing the total to 36,675.
Cummings told reporters he "behaved reasonably and legally" when asked about the trip from London to Durham.
Asked whether it looked good, he said: "Who cares about good looks? It's a question of doing the right thing. It's not about what you guys think."
He was later asked by reporters whether he would consider his position, he said: "Obviously not."
"You guys are probably all about as right about that as you were about Brexit: do you remember how right you all were about that," he added.
Cummings masterminded the 2016 Vote Leave campaign before being made Prime Minister Boris Johnson's chief political adviser.
Transport Secretary Grant Shapps said the prime minister - who has not commented so far - had "full confidence" in Cummings, following calls from the SNP and the Scottish Labour Party for him to quit or be fired.
A statement by Downing Street earlier denied that police spoke to Cummings or his family, after the Guardian and the Daily Mirror newspapers first reported the adviser had been seen near his parents' home in Durham.
The papers quoted police saying they had spoken to a family in Durham on 31 March.
In an updated statement, Durham Police said officers learned of his trip on 31 March and spoke to Cummings' father the following day.
"During that conversation, Cummings' father confirmed that his son had travelled with his family from London to the North-East and was self-isolating in part of the property.
"Durham Constabulary deemed that no further action was required. However, the officer did provide advice in relation to security issues," a statement from the police force said.
Durham Police and Crime Commissioner Steve White said it had been "most unwise" for Cummings to make the journey, "given the whole ethos" of the government's guidance.
Speaking at the daily coronavirus briefing, Shapps said Cummings went to Durham because "that's where the family was".
"As we all do in moments of crisis, we always seek to have our family, those who can assist us, around us and I think that's all that has happened in this case."
He said he did not know the "personal circumstances" of the Cummings family that prevented them seeking help closer to home.
Quizzed about how this fitted with the government's guidance, which is to stay at home, he said: "The important thing is that everyone remains in the same place whilst they are on lockdown."
He added: "The guidance says if you're living with children keep following this advice to the best of your ability.
"However, we are aware that not all these measures will be possible depending therefore on circumstances."
He said it was "for an individual to make the decision" on how best to practice lockdown measures.
The deputy chief medical officer for England, Jenny Harries, said: "All of the guidance has a common sense element to it, which includes safeguarding around adults or children."
And she added that travelling despite lockdown measures could be justified if there was an "extreme risk to life".
Labour and the SNP have both written to the UK's most senior civil servant, Sir Mark Sedwill, to call for an urgent inquiry into Cummings' conduct.
Labour's Rachel Reeves said: "The British people do not expect there to be one rule for them and another rule for the prime minister's most senior adviser."
Labour has stopped short of calling for Cummings' resignation - but it says Downing Street's explanations for his behaviour "raised more questions than they answer" including when the PM was made aware of his aide's decision to travel from London to Durham during lockdown.
Downing Street earlier put out a statement on Cummings' actions.
It said: "Owing to his wife being infected with suspected coronavirus and the high likelihood that he would himself become unwell, it was essential for Dominic Cummings to ensure his young child could be properly cared for.
"His sister and nieces had volunteered to help so he went to a house near to, but separate from, his extended family in case their help was needed. His sister shopped for the family and left everything outside.
"At no stage was he or his family spoken to by the police about this matter, as is being reported.
"His actions were in line with coronavirus guidelines."
Ministers are rallying around Dominic Cummings and it's clear the PM does not want to lose a trusted adviser with whom he first worked closely on the Vote Leave campaign.
The hope is that this will be seen as a "Westminster bubble" story and that the bubble will soon burst.
Also, after a U-turn on the NHS surcharge for migrants this week, No 10 will not want to look as though it is on the run.
But this is an issue that has resonance way beyond Westminster.
Other parents will have been sick during the pandemic and stuck to what they thought were the guidelines to self-isolate for up to 14 days.
So, the danger for the government isn't just the prospect of political attacks from opponents.
It is the perception the rules that apply in your street don't apply in Downing Street.
And if No 10 are interpreting the guidelines flexibly enough to allow people with Covid-19 symptoms to travel substantial distances to isolate away from their principal residence, the wider public may follow suit.
That may, in turn, make the government's task of pushing the reproduction rate of the virus down all the more difficult.
On Tuesday, 31 March, when Dominic Cummings was in Durham with his sick wife, UK government advice on essential travel included:
* Not visiting second homes, whether for isolation purposes or holidays
* Not leaving your home - the place you live - to stay at another home
* Remaining at your primary residence, to avoid putting additional pressure on communities and services at risk
* The government's essential travel advice remains in place, as does its advice on self-isolating.
On 30 March, it was reported that Cummings had developed symptoms of coronavirus and was self-isolating at home.
His wife, journalist Mary Wakefield, described in the Spectator magazine how he collapsed and was bed-ridden for 10 days.
Durham Police confirmed officers had spoken to the owners of an address after its officers were made aware on 31 March of reports a person had travelled there from London.
A spokesman said that owners of the address confirmed the individual "was present and self-isolating in part of the house", and officers "explained to the family the guidelines around self-isolation and reiterated the appropriate advice around essential travel".