This expression is not very pleasant to hear. It has its roots in the Campo das Hortas, where the city's wastewater and waste used to flow, since it was an area outside and below the city.
So it means that you want to send someone to an unpleasant place...