Australia's Asylum Seeker Resource Centre says the Manus Island processing centre in Papua New Guinea is designed to make the inmates so fearful that they voluntarily return home.
129 asylum seekers at the Australian-run centre have been officially declared refugees and are being offered the choice of staying in PNG.
More than 400 others have left the processing centre to return to their home countries.
Pamela Curr of the Asylum Seeker Resource Centre says the men in the camp are terrified of being put outside the detention centre because of fears of violence.
"The situation in Manus is really designed to break people. I've heard that expression over and over from different people who'd worked there. It is designed to get people to agree to go home."
But Porou Papi, one of the principal landowners at the detention camp site, rejects reports the refugees face threats from locals.
He says most Manus people want to encourage those who are skilled to stay and contribute to the province.
"The locals want them to be part and parcel of their workforce - our locals in Manus, but other provinces, I don't know. But if they settle some in Manus, we don't mind. They are Manusans, okay? We were telling them, if you want to stay here, you'll be a Manus Citizen."
PNG's government says it is undertaking extensive public awareness campaigns before integrating the refugees.