Boris Johnson has stood by his remarks about the burka after the Conservative Party chairman told him to apologise.
The former foreign secretary has been criticised for saying Muslim women wearing burkas "look like letter boxes" and comparing them to "bank robbers".
UK Prime Minister Theresa May has backed calls for Mr Johnson to apologise, saying the remarks have "clearly caused offence".
But a source close to Mr Johnson said he "won't be apologising", adding it was "ridiculous" to attack his views.
"We must not fall into the trap of shutting down the debate on difficult issues," the source added.
"We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists."
Mr Johnson's remarks, in a Daily Telegraph article, have provoked criticism from Muslim groups, some Tory MPs and opposition parties.
In the article, Mr Johnson said full-face veils should not be banned but looked "ridiculous".
However, the founder of the Conservative Muslim Forum said the remarks would harm community relations.
Conservative Party chairman Brandon Lewis said he agreed with Foreign Office Minister Alistair Burt that there had been degree of offence in Mr Johnson's comments, and called on the former London mayor to apologise.
Former Conservative chairwoman Baroness Warsi, the first Muslim woman to sit in a British cabinet, welcomed Mr Lewis's intervention and called for disciplinary action against Mr Johnson if he did not apologise.
She described the remarks as "offensive and deliberately provocative, but very clever politics".
A debate about the burka should be had "in a serious way", she said, rather than "trying to get airtime and attention on an issue which he knows will resonate with a certain part of the Tory Party".
Shazia Awan-Scully, a Muslim who ran as a Conservative candidate in 2010, said she did not agree with wearing the burka but it was up to women to make up their own minds.
She told BBC Radio 4's PM programme: "Some women choose to wear it, some women are conditioned to wear it, but it's certainly out of the question for me to tell a woman what to wear."
She added that the language used by Mr Johnson showed he was "pandering to this Islamaphobic attitude bubbling away in the Conservative Party", which she said she had experienced herself.
Some MPs oppose the banning of the burka, but want to banish grid girls from existence. You cannot expect a society that celebrates gay pride and embraces gay marriage to live harmoniously when condoning the suppression of women forced to cover up, segregate and become invisible— Nadine Dorries (@NadineDorries) August 7, 2018
Conservative backbench MP Andrew Bridgen said Mr Johnson had raised an important subject in a "light-hearted way".
The reaction "says a lot about internal Conservative Party politics", he told the BBC.
Dr Qanta Ahmed, a British-American Muslim doctor who supports banning face veils, said Mr Johnson had been "tactless".
The 50-year-old said wearing a full-face veil was a "new phenomenon" and she didn't consider it a religious right, adding that she would be "very opposed" to it becoming normalised in British society.
What Boris Johnson said
In his Daily Telegraph column, Mr Johnson - who last month quit the government in protest at UK Prime Minister Theresa May's Brexit policy - was commenting on the introduction of a burka ban in Denmark.
He said he felt "fully entitled" to expect women to remove face coverings when talking to him at his MP surgery - and schools and universities should be able to take the same approach if a student "turns up... looking like a bank robber".
"If you tell me that the burka is oppressive, then I am with you," he said.
"If you say that it is weird and bullying to expect women to cover their faces, then I totally agree - and I would add that I can find no scriptural authority for the practice in the Koran.
"I would go further and say that it is absolutely ridiculous that people should choose to go around looking like letter boxes."
He said businesses and government agencies should be able to "enforce a dress code" that allowed them to see customers' faces.
But he said: "Such restrictions are not quite the same as telling a free-born adult woman what she may or may not wear, in a public place, when she is simply minding her own business."
He said a total ban on face-covering veils would give a boost to radicals who said there was a "clash of civilisations" between Islam and the West, and could lead to "a general crackdown on any public symbols of religious affiliation".