Its only an issue if im left to challange a xenos player. There are 4 imperial challengers and 2 non imperial. So the 4 imp challengers will take 4 from the 3 chaos and 2 xenos. As long as they don't take all 3 chaos, theres no problem.
The 2 xenos challengers will need to challenge 1 imperial (as no one else left can challenge them anyways) and 1 chaos