Amateur video shot in Bulgaria shows vigilantes arresting three refugees or migrants and tying their hands behind their backs as they lie on the ground in a wood.
One of the vigilantes tells them: "No Bulgaria - go back [to] Turkey."
The video was shot in Strandja, a mountainous area near the Turkish border, the Bulgarian BTV news website reported.
Bulgarian border police chief Antonio Angelov, quoted by BTV, said such an arrest was illegal.
Bulgaria has erected a razor-wire fence along 95km of its 269km border with Turkey.
According to Mr Angelov, the three refugees or migrants shown in the video said they were Afghans. The video was the main story across Bulgarian TV on Monday.
The vigilantes - at least one of them armed with a machete - stood over the three and one was heard telling them in broken English to return to Turkey. The migrants appeared frightened and were not resisting.
Bulgaria's neighbour Greece was embroiled in a row with Macedonia on Monday over rough treatment of migrants at the Idomeni border crossing on Sunday.
Medical charity MSF said 260 people were hurt when Macedonian security forces fired tear gas as hundreds of refugees and migrants tried to cross the border. Thirty were hit by rubber bullets, including three children under 10, MSF said. Tensions were still high at the border on Monday.
Greek Prime Minister Alexis Tsipras condemned the use of tear gas and rubber bullets, describing the violence as "a big disgrace for European civilisation".
But Macedonia insisted its security forces had not used rubber bullets, adding that 23 of its security forces had been wounded. Police had repeatedly asked Greek officials to secure the border, the foreign affairs ministry said.
More than 11,000 refugees and migrants have been camping at Idomeni for weeks in wretched conditions.
Greece remains the chief Balkan transit country for migrants - many of them fleeing the Syrian war - who hope to reach northern Europe.
There is concern that Greek deportations of migrants to Turkey, approved by the EU, may be violating the rights of some refugees.
Praise for vigilantes
Vigilantes became involved in the migrant crisis in Bulgaria earlier this year. Dinko Valev, a Bulgarian trader in spare parts for buses, became a national celebrity in February after starting to patrol the Turkish border "hunting" for migrants.
He won national praise for subduing a group of 12 Syrian men, three women and a child. Mobile phone footage showed the migrants lying on the ground, waiting for the police, while he insulted them.
Last week Bulgaria's border police gave an award to a volunteer "border patrol" which detained 23 refugees near the Turkish border, the Balkan Insight news website reported.
Mr Angelov said the vigilantes had acted "very appropriately", but he cautioned that such patrols were "very risky" and urged them to inform the police about any future patrols.
More than a million undocumented refugees and other migrants have entered the EU by boat from Turkey to Greece since January 2015, generating an unprecedented crisis for the EU's 28 member states.