Officers detained 22-year-old Hassan Saada, a Moroccan athlete, after allegations of rape were made.
The alleged sexual assaults took place on Wednesday, police said in a statement.
There are conflicting reports as to whether the complainants worked as cleaners or waitresses at the village.
Saada was arrested on Friday and is being held in a prison cell - a day before his first fight was scheduled to take place.
Police said he would remain in jail for 15 days while they investigate the claims.
Under Brazilian law, suspects can be held for a lengthy period while a case is examined.
The light heavyweight boxer was due to fight Mehmet Nadir of Turkey in the preliminary rounds on Saturday.