This also fixes it so that ipvs.CleanupLeftovers only deletes ipvs/ipset stuff once, rather than first deleting all of it on behalf of the IPv4 Proxier and then no-op "deleting" it all again on behalf of the IPv6 Proxier.