mirror of
https://github.com/k3s-io/kubernetes.git
synced 2025-11-04 07:49:35 +00:00
63d4b85bf4
Automatic merge from submit-queue (batch tested with PRs 54316, 53400, 55933, 55786, 55794). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Add Amazon NLB support **What this PR does / why we need it**: This adds support for AWS's NLB for `LoadBalancer` services. **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes # Fixes #52173 **Special notes for your reviewer**: This is NOT yet ready for merge, but I'd love any feedback before it is. This requires at least `v1.10.40` of the [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go), which is not yet included in Kubernetes. Per @justinsb, I'm waiting on possibly #48314 to update to `v1.10.40` or some other PR. I tried to make the change as easy to review as possible, so some LoadBalancer logic is duplicated in the `if isNLB(annotations)` blocks. I can refactor that and sprinkle more `isNLB()` switches around, but it might be harder to view the diff. **Other Notes:** * NLB's subnets cannot be modified after creation (maybe look for public subnets in all AZ's?). Currently, I'm just using `c.findELBSubnets()` * Health check uses TCP with all the NLB default values. I was thinking HTTP health checks via annotation could be added later. Should that go into this PR? * ~~`externalTrafficPolicy`/`healthCheckNodePort` are ignored. Should those be implemented for this PR?~~ * `externalTrafficPolicy` and subsequent `healthCheckNodePort` are handled properly. This may come with uneven load balancing, as NLB doesn't support weighted backends. * With classic ELB, you have a security group the ELB is inside of to associate Instance (k8s node) SG rules with a LoadBalancer (k8s Service), but NLB's don't have a security group. Instead, I use the `Description` field on [`ec2.IpRange`](https://docs.aws.amazon.com/sdk-for-go/api/service/ec2/#IpRange) with the following annotations. Is this ok? I couldn't think of another way to associate SG rule to the NLB * Node SG gets an rule added for VPC cidr on NodePort for Health Check with annotation in description `kubernetes.io/rule/nlb/health=<loadBalancerName>` * Node SG gets an rule added for `loadBalancerSourceRanges` to NodePort for client traffic with annotation in description `kubernetes.io/rule/nlb/client=<loadBalancerName>` * **Note: if `loadBalancerSourceRanges` is unspecified, this opens instance security groups to traffic from `0.0.0.0/0` on the service's nodePorts** * Respects internal annotation * Creates a TargetGroup per frontend port: simplifies updates when you have same backend port for multiple front end ports. * Does not (yet) verify that we're under the NLB limits in terms of # of listeners * `UpdateLoadBalancer()` basically just calls `EnsureLoadBalancer` for NLB's. Is this ok? **Areas for future improvement or optimization**: * A new annotation indicating a new security group should be created for NLB traffic and instances would be placed in this new SG. (Could bump up against the default limit of 5 SG's per instance) * Only create a client health check security group rule when the VPC cidr is not a subset of `spec.loadBalancerSourceRanges` * Consolidate TargetGroups if a service has 2+ frontend ports and the same nodePort. * A new annotation for specifying TargetGroup Health Check options. **Release note**: ```release-notes Add Amazon NLB support - Fixes #52173 ``` ping @justinsb @bchav
Deprecation Notice: This directory has entered maintenance mode and will not be accepting new providers. Cloud Providers in this directory will continue to be actively developed or maintained and supported at their current level of support as a longer-term solution evolves.
Overview:
The mechanism for supporting cloud providers is currently in transition: the original method of implementing cloud provider-specific functionality within the main kubernetes tree (here) is no longer advised; however, the proposed solution is still in development.
Guidance for potential cloud providers:
- Support for cloud providers is currently in a state of flux. Background information on motivation and the proposal for improving is in the github proposal.
- In support of this plan, a new cloud-controller-manager binary was added in 1.6. This was the first of several steps (see the proposal for more information).
- Attempts to contribute new cloud providers or (to a lesser extent) persistent volumes to the core repo will likely meet with some pushback from reviewers/approvers.
- It is understood that this is an unfortunate situation in which 'the old way is no longer supported but the new way is not ready yet', but the initial path is unsustainable, and contributors are encouraged to participate in the implementation of the proposed long-term solution, as there is risk that PRs for new cloud providers here will not be approved.
- Though the fully productized support envisioned in the proposal is still 2 - 3 releases out, the foundational work is underway, and a motivated cloud provider could accomplish the work in a forward-looking way. Contributors are encouraged to assist with the implementation of the design outlined in the proposal.
Some additional context on status / direction:
- 1.6 added a new cloud-controller-manager binary that may be used for testing the new out-of-core cloudprovider flow.
- Setting cloud-provider=external allows for creation of a separate controller-manager binary
- 1.7 adds extensible admission control, further enabling topology customization.