Add app.bsky.graph.rejected containing the handles of removed followers and use that to consolidate the follower list (and count) ?
(It would still leave us in the "following" lists of bots and other undesirables, but nobody looks there for similar accounts to follow)