mirror of
https://github.com/k3s-io/kubernetes.git
synced 2025-11-02 06:47:34 +00:00
63d4b85bf4
Automatic merge from submit-queue (batch tested with PRs 54316, 53400, 55933, 55786, 55794). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Add Amazon NLB support **What this PR does / why we need it**: This adds support for AWS's NLB for `LoadBalancer` services. **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes # Fixes #52173 **Special notes for your reviewer**: This is NOT yet ready for merge, but I'd love any feedback before it is. This requires at least `v1.10.40` of the [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go), which is not yet included in Kubernetes. Per @justinsb, I'm waiting on possibly #48314 to update to `v1.10.40` or some other PR. I tried to make the change as easy to review as possible, so some LoadBalancer logic is duplicated in the `if isNLB(annotations)` blocks. I can refactor that and sprinkle more `isNLB()` switches around, but it might be harder to view the diff. **Other Notes:** * NLB's subnets cannot be modified after creation (maybe look for public subnets in all AZ's?). Currently, I'm just using `c.findELBSubnets()` * Health check uses TCP with all the NLB default values. I was thinking HTTP health checks via annotation could be added later. Should that go into this PR? * ~~`externalTrafficPolicy`/`healthCheckNodePort` are ignored. Should those be implemented for this PR?~~ * `externalTrafficPolicy` and subsequent `healthCheckNodePort` are handled properly. This may come with uneven load balancing, as NLB doesn't support weighted backends. * With classic ELB, you have a security group the ELB is inside of to associate Instance (k8s node) SG rules with a LoadBalancer (k8s Service), but NLB's don't have a security group. Instead, I use the `Description` field on [`ec2.IpRange`](https://docs.aws.amazon.com/sdk-for-go/api/service/ec2/#IpRange) with the following annotations. Is this ok? I couldn't think of another way to associate SG rule to the NLB * Node SG gets an rule added for VPC cidr on NodePort for Health Check with annotation in description `kubernetes.io/rule/nlb/health=<loadBalancerName>` * Node SG gets an rule added for `loadBalancerSourceRanges` to NodePort for client traffic with annotation in description `kubernetes.io/rule/nlb/client=<loadBalancerName>` * **Note: if `loadBalancerSourceRanges` is unspecified, this opens instance security groups to traffic from `0.0.0.0/0` on the service's nodePorts** * Respects internal annotation * Creates a TargetGroup per frontend port: simplifies updates when you have same backend port for multiple front end ports. * Does not (yet) verify that we're under the NLB limits in terms of # of listeners * `UpdateLoadBalancer()` basically just calls `EnsureLoadBalancer` for NLB's. Is this ok? **Areas for future improvement or optimization**: * A new annotation indicating a new security group should be created for NLB traffic and instances would be placed in this new SG. (Could bump up against the default limit of 5 SG's per instance) * Only create a client health check security group rule when the VPC cidr is not a subset of `spec.loadBalancerSourceRanges` * Consolidate TargetGroups if a service has 2+ frontend ports and the same nodePort. * A new annotation for specifying TargetGroup Health Check options. **Release note**: ```release-notes Add Amazon NLB support - Fixes #52173 ``` ping @justinsb @bchav