Exactly this. Because of the way we currently organise international sport with athletes representing a nation state, it is inherently and unavoidably political, even if we wish it wasn't. That also isn't going to change any time soon.
This means there is no mechanism available to us for managing such situations that isn't political, complex or the potential start of a slippery slope. Allowing selected Russian athletes in as unaffiliated is certainly that. All we get to do is pick which political, complex possible slippery slope we prefer. I understand arguments for letting Russian athletes compete, but nobody should be in any doubt that they are all of these things.
The FIG in some ways have a harder job than the IOC here not just because the IOC has the Charter, which simplifies, but also because they've already had to contend with the issue of visa denial. If you look at last quad, the UK simply wasn't going to let Russian or Belarussian athletes in for 2022 worlds. Wasn't happening. Whereas that hasn't yet come up for the Olympics, and my guess is it's less likely.