Residents of a south Wales town are under investigation after they mistook a youth group visiting a summer camp for an influx of asylum seekers.