kube-controller-manager and kube-scheduler do not use the configured loopback clients. Drop them. Fix up scheduler test server to not depend on the loopback client.