Kasumigaseki Country Club, the course set to host the golf competition at the 2020 Olympics in Japan, has come under fire regarding its policies towards women. The club’s protocol was up for revision in early February, and it was believed Kasumigaseki would vote full playing rights for both sexes. Alas, the vote was surprisingly postponed, raising further doubt of the venue’s participation in the Summer Games.

Overnight, the International Olympic Committee delivered its response to Kasumigaseki: abandon the discriminatory practices, or another course will be picked.

“At some point there has to be a cut off,” IOC vice president John Coates said, according to the AP. “Image-wise, our position is clear. We will only go to a club that has non-discrimination.”

Coates did remain optimistic about Kasumigaseki changing its ways, saying it is heading “in the right direction”. Still, a new vote may not surface until June.

At the moment, Kasumigaseki is supposed to host both the men’s and women’s golf contests in the 2020 Olympics. Golf remains a trial sport in the Summer Games, with a guaranteed spot in 2020 and a likely appearance in 2024, but the IOC has not committed further.