The US has accused Iran of a "crude and transparent attempt to extort payments from the international community" by violating the 2015 nuclear deal.
The American ambassador to the International Atomic Energy Agency said there was no credible reason for Iran to breach two key commitments on uranium enrichment.
Iran has said the steps were a response to the sanctions the US reinstated when it abandoned the deal last year.
It has vowed to reverse them if it is given compensation for economic losses.
European powers still party to the deal have set up a mechanism for facilitating legitimate trade without direct financial transactions that they hope will circumvent the US sanctions. However, Iran has said it does not meet its needs.
Iran's representative to the UN in New York told the BBC that the Europeans could do more, and that if they did not Iran would take further steps.
If nothing happens in the next 60 days we will have to go to the third phase. The elements of the third phase are not known yet, but when it comes to that we will announce what we are going to do," Majid Takht-Ravanchi warned.
At Wednesday's special meeting of the IAEA board of governors in Vienna, diplomats were reportedly told that the global watchdog's inspectors had verified Iran was enriching uranium to 4.5 percent concentration - above the 3.67 percent limit set by the nuclear deal.
The country announced the step three days ago, saying it wanted to be able to produce fuel for the Bushehr nuclear power plant.
The IAEA was also said to have verified that Iran's stockpile of low-enriched uranium had grown since the 300kg (660lb) limit was exceeded on 1 July.
Low-enriched uranium, which typically has a 3-5 percent concentration, can be used to produce fuel for commercial nuclear power plants. Weapons-grade uranium is 90 percent enriched or more.
Experts have said the breach of the stockpile limit does not pose a near-term proliferation risk, but that enriching uranium to a higher concentration would begin to shorten Iran's so-called "break-out time" - the time required for it to produce enough fissile material for a bomb.
Iran insists it has never sought to develop a nuclear weapon. But the international community does not believe Iran, and negotiated a nuclear deal to prevent it from doing so.
US President Donald Trump said the deal did not go far enough to restrict Iran's nuclear programme and unilaterally withdrew from the accord in May 2018.
He wants to replace it with one that would also curb Iran's ballistic missile programme and its involvement in regional conflicts. But Iran has so far refused.
At the IAEA meeting, US ambassador Jackie Wolcott said Iran's recent actions and statements were deeply concerning, and affected security and stability.
"Iran's current nuclear posture is clearly aimed at escalating tensions rather than defusing them, and underscores the serious challenges Iran continues to pose to international peace and security," she said.
"Such brinkmanship and extortion tactics will neither resolve the current impasse nor bring Iran sanctions relief. The path the regime is now on will only deepen its international isolation and raise the dangers it faces."
Ms Wolcott said it was imperative that Iran's "misbehaviour" not be rewarded.
"For if it is, Iran's demands and provocations will only escalate - as has happened all too often in the past," she warned.
The ambassador called on Iran to reverse its nuclear steps and halt any plans for further advancements in the future.
"The United States has made clear that we are open to negotiation without preconditions, and that we are offering Iran the possibility of a full normalisation of relations," she added.
Iranian Foreign Minister Mohammad Javad Zarif said Iran's actions were "lawful" under the deal, which allows one party to "cease performing its commitments… in whole or in part" in the event of "significant non-performance" by other parties.
President Trump later alleged in a tweet that Iran had been "secretly 'enriching' uranium in total violation" of the deal made by his predecessor, Barack Obama, and promised that sanctions would "soon be increased, substantially".
Mr Trump did not give any further details. But the IAEA had repeatedly verified Iran's compliance with the deal until this month and Iran's ambassador to the agency, Kazim Gharib Abadi, said it had "nothing to hide."
Meanwhile, the top diplomatic adviser to French President Emmanuel Macron held talks in Tehran with Iranian officials to try to avoid further escalation.
Emmanuel Bonne was told by Iranian President Hassan Rouhani that Iran had "kept the path of diplomacy and talks completely open" and that it hoped other parties to the nuclear deal would be able to "use this opportunity properly".