github/kubernetes
1
0
Fork 0
mirror of https://github.com/k3s-io/kubernetes.git synced 2025-07-21 02:41:25 +00:00
Code Issues Projects Releases Wiki Activity

dual stack services (#91824)

Browse Source 
* api: structure change

* api: defaulting, conversion, and validation

* [FIX] validation: auto remove second ip/family when service changes to SingleStack

* [FIX] api: defaulting, conversion, and validation

* api-server: clusterIPs alloc, printers, storage and strategy

* [FIX] clusterIPs default on read

* alloc: auto remove second ip/family when service changes to SingleStack

* api-server: repair loop handling for clusterIPs

* api-server: force kubernetes default service into single stack

* api-server: tie dualstack feature flag with endpoint feature flag

* controller-manager: feature flag, endpoint, and endpointSlice controllers handling multi family service

* [FIX] controller-manager: feature flag, endpoint, and endpointSlicecontrollers handling multi family service

* kube-proxy: feature-flag, utils, proxier, and meta proxier

* [FIX] kubeproxy: call both proxier at the same time

* kubenet: remove forced pod IP sorting

* kubectl: modify describe to include ClusterIPs, IPFamilies, and IPFamilyPolicy

* e2e: fix tests that depends on IPFamily field AND add dual stack tests

* e2e: fix expected error message for ClusterIP immutability

* add integration tests for dualstack

the third phase of dual stack is a very complex change in the API,
basically it introduces Dual Stack services. Main changes are:

- It pluralizes the Service IPFamily field to IPFamilies,
and removes the singular field.
- It introduces a new field IPFamilyPolicyType that can take
3 values to express the "dual-stack(mad)ness" of the cluster:
SingleStack, PreferDualStack and RequireDualStack
- It pluralizes ClusterIP to ClusterIPs.

The goal is to add coverage to the services API operations,
taking into account the 6 different modes a cluster can have:

- single stack: IP4 or IPv6 (as of today)
- dual stack: IPv4 only, IPv6 only, IPv4 - IPv6, IPv6 - IPv4

* [FIX] add integration tests for dualstack

* generated data

* generated files

Co-authored-by: Antonio Ojea <aojea@redhat.com>
This commit is contained in:
Khaled Henidak (Kal) 2020-10-26 13:15:59 -07:00 committed by GitHub
parent d0e06cf3e0
commit 6675eba3ef
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
84 changed files with 11170 additions and 3514 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
api/openapi-spec/swagger.json generated
View File

@ -10257,6 +10257,14 @@
"description": "clusterIP is the IP address of the service and is usually assigned randomly by the master. If an address is specified manually and is not in use by others, it will be allocated to the service; otherwise, creation of the service will fail. This field can not be changed through updates. Valid values are \"None\", empty string (\"\"), or a valid IP address. \"None\" can be specified for headless services when proxying is not required. Only applies to types ClusterIP, NodePort, and LoadBalancer. Ignored if type is ExternalName. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies",
"type": "string"
},
"clusterIPs": {
"description": "ClusterIPs identifies all the ClusterIPs assigned to this service. ClusterIPs are assigned or reserved based on the values of service.spec.ipFamilies. A maximum of two entries (dual-stack IPs) are allowed in ClusterIPs. The IPFamily of each ClusterIP must match values provided in service.spec.ipFamilies. Clients using ClusterIPs must keep it in sync with ClusterIP (if provided) by having ClusterIP matching first element of ClusterIPs.",
"items": {
"type": "string"
},
"type": "array",
"x-kubernetes-list-type": "atomic"
},
"externalIPs": {
"description": "externalIPs is a list of IP addresses for which nodes in the cluster will also accept traffic for this service. These IPs are not managed by Kubernetes. The user is responsible for ensuring that traffic arrives at a node with this IP. A common example is external load-balancers that are not part of the Kubernetes system.",
"items": {
@ -10277,8 +10285,16 @@
"format": "int32",
"type": "integer"
},
"ipFamily": {
"description": "ipFamily specifies whether this Service has a preference for a particular IP family (e.g. IPv4 vs. IPv6) when the IPv6DualStack feature gate is enabled. In a dual-stack cluster, you can specify ipFamily when creating a ClusterIP Service to determine whether the controller will allocate an IPv4 or IPv6 IP for it, and you can specify ipFamily when creating a headless Service to determine whether it will have IPv4 or IPv6 Endpoints. In either case, if you do not specify an ipFamily explicitly, it will default to the cluster's primary IP family. This field is part of an alpha feature, and you should not make any assumptions about its semantics other than those described above. In particular, you should not assume that it can (or cannot) be changed after creation time; that it can only have the values \"IPv4\" and \"IPv6\"; or that its current value on a given Service correctly reflects the current state of that Service. (For ClusterIP Services, look at clusterIP to see if the Service is IPv4 or IPv6. For headless Services, look at the endpoints, which may be dual-stack in the future. For ExternalName Services, ipFamily has no meaning, but it may be set to an irrelevant value anyway.)",
"ipFamilies": {
"description": "IPFamilies identifies all the IPFamilies assigned for this Service. If a value was not provided for IPFamilies it will be defaulted based on the cluster configuration and the value of service.spec.ipFamilyPolicy. A maximum of two values (dual-stack IPFamilies) are allowed in IPFamilies. IPFamilies field is conditionally mutable: it allows for adding or removing a secondary IPFamily, but it does not allow changing the primary IPFamily of the service.",
"items": {
"type": "string"
},
"type": "array",
"x-kubernetes-list-type": "atomic"
},
"ipFamilyPolicy": {
"description": "IPFamilyPolicy represents the dual-stack-ness requested or required by this Service. If there is no value provided, then this Service will be considered SingleStack (single IPFamily). Services can be SingleStack (single IPFamily), PreferDualStack (two dual-stack IPFamilies on dual-stack clusters or single IPFamily on single-stack clusters), or RequireDualStack (two dual-stack IPFamilies on dual-stack configured clusters, otherwise fail). The IPFamilies and ClusterIPs assigned to this service can be controlled by service.spec.ipFamilies and service.spec.clusterIPs respectively.",
"type": "string"
},
"loadBalancerIP": {

13
cmd/kube-apiserver/app/options/validation.go
View File

@ -55,9 +55,12 @@ func validateClusterIPFlags(options *ServerRunOptions) []error {
}
// Secondary IP validation
// while api-server dualstack bits does not have dependency on EndPointSlice, its
// a good idea to have validation consistent across all components (ControllerManager
// needs EndPointSlice + DualStack feature flags).
secondaryServiceClusterIPRangeUsed := (options.SecondaryServiceClusterIPRange.IP != nil)
if secondaryServiceClusterIPRangeUsed && !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
errs = append(errs, fmt.Errorf("--secondary-service-cluster-ip-range can only be used if %v feature is enabled", string(features.IPv6DualStack)))
if secondaryServiceClusterIPRangeUsed && (!utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) || !utilfeature.DefaultFeatureGate.Enabled(features.EndpointSlice)) {
errs = append(errs, fmt.Errorf("secondary service cluster-ip range(--service-cluster-ip-range[1]) can only be used if %v and %v feature is enabled", string(features.IPv6DualStack), string(features.EndpointSlice)))
}
// note: While the cluster might be dualstack (i.e. pods with multiple IPs), the user may choose
@ -68,14 +71,14 @@ func validateClusterIPFlags(options *ServerRunOptions) []error {
// Should be dualstack IPFamily(PrimaryServiceClusterIPRange) != IPFamily(SecondaryServiceClusterIPRange)
dualstack, err := netutils.IsDualStackCIDRs([]*net.IPNet{&options.PrimaryServiceClusterIPRange, &options.SecondaryServiceClusterIPRange})
if err != nil {
errs = append(errs, errors.New("error attempting to validate dualstack for --service-cluster-ip-range and --secondary-service-cluster-ip-range"))
errs = append(errs, fmt.Errorf("error attempting to validate dualstack for --service-cluster-ip-range value error:%v", err))
}
if !dualstack {
errs = append(errs, errors.New("--service-cluster-ip-range and --secondary-service-cluster-ip-range must be of different IP family"))
errs = append(errs, errors.New("--service-cluster-ip-range[0] and --service-cluster-ip-range[1] must be of different IP family"))
}
if err := validateMaxCIDRRange(options.SecondaryServiceClusterIPRange, maxCIDRBits, "--secondary-service-cluster-ip-range"); err != nil {
if err := validateMaxCIDRRange(options.SecondaryServiceClusterIPRange, maxCIDRBits, "--service-cluster-ip-range[1]"); err != nil {
errs = append(errs, err)
}
}

88
cmd/kube-apiserver/app/options/validation_test.go
View File

@ -54,10 +54,11 @@ func makeOptionsWithCIDRs(serviceCIDR string, secondaryServiceCIDR string) *Serv
func TestClusterSerivceIPRange(t *testing.T) {
testCases := []struct {
name string
options *ServerRunOptions
enableDualStack bool
expectErrors bool
name string
options *ServerRunOptions
enableDualStack bool
enableEndpointSlice bool
expectErrors bool
}{
{
name: "no service cidr",
@ -66,10 +67,11 @@ func TestClusterSerivceIPRange(t *testing.T) {
enableDualStack: false,
},
{
name: "only secondary service cidr, dual stack gate on",
expectErrors: true,
options: makeOptionsWithCIDRs("", "10.0.0.0/16"),
enableDualStack: true,
name: "only secondary service cidr, dual stack gate on",
expectErrors: true,
options: makeOptionsWithCIDRs("", "10.0.0.0/16"),
enableDualStack: true,
enableEndpointSlice: true,
},
{
name: "only secondary service cidr, dual stack gate off",
@ -78,16 +80,18 @@ func TestClusterSerivceIPRange(t *testing.T) {
enableDualStack: false,
},
{
name: "primary and secondary are provided but not dual stack v4-v4",
expectErrors: true,
options: makeOptionsWithCIDRs("10.0.0.0/16", "11.0.0.0/16"),
enableDualStack: true,
name: "primary and secondary are provided but not dual stack v4-v4",
expectErrors: true,
options: makeOptionsWithCIDRs("10.0.0.0/16", "11.0.0.0/16"),
enableDualStack: true,
enableEndpointSlice: true,
},
{
name: "primary and secondary are provided but not dual stack v6-v6",
expectErrors: true,
options: makeOptionsWithCIDRs("2000::/108", "3000::/108"),
enableDualStack: true,
name: "primary and secondary are provided but not dual stack v6-v6",
expectErrors: true,
options: makeOptionsWithCIDRs("2000::/108", "3000::/108"),
enableDualStack: true,
enableEndpointSlice: true,
},
{
name: "valid dual stack with gate disabled",
@ -96,16 +100,33 @@ func TestClusterSerivceIPRange(t *testing.T) {
enableDualStack: false,
},
{
name: "service cidr to big",
expectErrors: true,
options: makeOptionsWithCIDRs("10.0.0.0/8", ""),
enableDualStack: true,
name: "service cidr is too big",
expectErrors: true,
options: makeOptionsWithCIDRs("10.0.0.0/8", ""),
enableDualStack: true,
enableEndpointSlice: true,
},
{
name: "dual-stack secondary cidr to big",
expectErrors: true,
options: makeOptionsWithCIDRs("10.0.0.0/16", "3000::/64"),
enableDualStack: true,
name: "dual-stack secondary cidr too big",
expectErrors: true,
options: makeOptionsWithCIDRs("10.0.0.0/16", "3000::/64"),
enableDualStack: true,
enableEndpointSlice: true,
},
{
name: "valid v6-v4 dual stack + gate on + endpointSlice gate is on",
expectErrors: false,
options: makeOptionsWithCIDRs("3000::/108", "10.0.0.0/16"),
enableDualStack: true,
enableEndpointSlice: true,
},
{
name: "valid v4-v6 dual stack + gate on + endpointSlice is off",
expectErrors: true,
options: makeOptionsWithCIDRs("10.0.0.0/16", "3000::/108"),
enableDualStack: true,
enableEndpointSlice: false,
},
/* success cases */
{
@ -115,22 +136,25 @@ func TestClusterSerivceIPRange(t *testing.T) {
enableDualStack: false,
},
{
name: "valid v4-v6 dual stack + gate on",
expectErrors: false,
options: makeOptionsWithCIDRs("10.0.0.0/16", "3000::/108"),
enableDualStack: true,
name: "valid v4-v6 dual stack + gate on",
expectErrors: false,
options: makeOptionsWithCIDRs("10.0.0.0/16", "3000::/108"),
enableDualStack: true,
enableEndpointSlice: true,
},
{
name: "valid v6-v4 dual stack + gate on",
expectErrors: false,
options: makeOptionsWithCIDRs("3000::/108", "10.0.0.0/16"),
enableDualStack: true,
name: "valid v6-v4 dual stack + gate on",
expectErrors: false,
options: makeOptionsWithCIDRs("3000::/108", "10.0.0.0/16"),
enableDualStack: true,
enableEndpointSlice: true,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, tc.enableDualStack)()
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.EndpointSlice, tc.enableEndpointSlice)()
errs := validateClusterIPFlags(tc.options)
if len(errs) > 0 && !tc.expectErrors {
t.Errorf("expected no errors, errors found %+v", errs)

8
cmd/kube-controller-manager/app/core.go
View File

@ -110,14 +110,14 @@ func startNodeIpamController(ctx ControllerContext) (http.Handler, bool, error)
return nil, false, err
}
// failure: more than one cidr and dual stack is not enabled
if len(clusterCIDRs) > 1 && !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
return nil, false, fmt.Errorf("len of ClusterCIDRs==%v and dualstack feature is not enabled", len(clusterCIDRs))
// failure: more than one cidr and dual stack is not enabled and/or endpoint slice is not enabled
if len(clusterCIDRs) > 1 && (!utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) || !utilfeature.DefaultFeatureGate.Enabled(features.EndpointSlice)) {
return nil, false, fmt.Errorf("len of ClusterCIDRs==%v and dualstack or EndpointSlice feature is not enabled", len(clusterCIDRs))
}
// failure: more than one cidr but they are not configured as dual stack
if len(clusterCIDRs) > 1 && !dualStack {
return nil, false, fmt.Errorf("len of ClusterCIDRs==%v and they are not configured as dual stack (at least one from each IPFamily", len(clusterCIDRs))
return nil, false, fmt.Errorf("len of ClusterCIDRs==%v and they are not configured as dual stack (at least one from each IPFamily)", len(clusterCIDRs))
}
// failure: more than cidrs is not allowed even with dual stack

5
pkg/apis/core/helper/helpers.go
View File

@ -267,7 +267,10 @@ func IsIntegerResourceName(str string) bool {
// IsServiceIPSet aims to check if the service's ClusterIP is set or not
// the objective is not to perform validation here
func IsServiceIPSet(service *core.Service) bool {
return service.Spec.ClusterIP != core.ClusterIPNone && service.Spec.ClusterIP != ""
// This function assumes that the service is semantically validated
// it does not test if the IP is valid, just makes sure that it is set.
return len(service.Spec.ClusterIP) > 0 &&
service.Spec.ClusterIP != core.ClusterIPNone
}
var standardFinalizers = sets.NewString(

70
pkg/apis/core/helper/helpers_test.go
View File

@ -294,3 +294,73 @@ func TestIsOvercommitAllowed(t *testing.T) {
}
}
}
func TestIsServiceIPSet(t *testing.T) {
testCases := []struct {
input core.ServiceSpec
output bool
name string
}{
{
name: "nil cluster ip",
input: core.ServiceSpec{
ClusterIPs: nil,
},
output: false,
},
{
name: "headless service",
input: core.ServiceSpec{
ClusterIP: "None",
ClusterIPs: []string{"None"},
},
output: false,
},
// true cases
{
name: "one ipv4",
input: core.ServiceSpec{
ClusterIP: "1.2.3.4",
ClusterIPs: []string{"1.2.3.4"},
},
output: true,
},
{
name: "one ipv6",
input: core.ServiceSpec{
ClusterIP: "2001::1",
ClusterIPs: []string{"2001::1"},
},
output: true,
},
{
name: "v4, v6",
input: core.ServiceSpec{
ClusterIP: "1.2.3.4",
ClusterIPs: []string{"1.2.3.4", "2001::1"},
},
output: true,
},
{
name: "v6, v4",
input: core.ServiceSpec{
ClusterIP: "2001::1",
ClusterIPs: []string{"2001::1", "1.2.3.4"},
},
output: true,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
s := core.Service{
Spec: tc.input,
}
if IsServiceIPSet(&s) != tc.output {
t.Errorf("case, input: %v, expected: %v, got: %v", tc.input, tc.output, !tc.output)
}
})
}
}

80
pkg/apis/core/types.go
View File

@ -3510,8 +3510,13 @@ type LoadBalancerIngress struct {
Hostname string
}
const (
// MaxServiceTopologyKeys is the largest number of topology keys allowed on a service
MaxServiceTopologyKeys = 16
)
// IPFamily represents the IP Family (IPv4 or IPv6). This type is used
// to express the family of an IP expressed by a type (i.e. service.Spec.IPFamily)
// to express the family of an IP expressed by a type (e.g. service.spec.ipFamilies).
type IPFamily string
const (
@ -3519,8 +3524,29 @@ const (
IPv4Protocol IPFamily = "IPv4"
// IPv6Protocol indicates that this IP is IPv6 protocol
IPv6Protocol IPFamily = "IPv6"
// MaxServiceTopologyKeys is the largest number of topology keys allowed on a service
MaxServiceTopologyKeys = 16
)
// IPFamilyPolicyType represents the dual-stack-ness requested or required by a Service
type IPFamilyPolicyType string
const (
// IPFamilyPolicySingleStack indicates that this service is required to have a single IPFamily.
// The IPFamily assigned is based on the default IPFamily used by the cluster
// or as identified by service.spec.ipFamilies field
IPFamilyPolicySingleStack IPFamilyPolicyType = "SingleStack"
// IPFamilyPolicyPreferDualStack indicates that this service prefers dual-stack when
// the cluster is configured for dual-stack. If the cluster is not configured
// for dual-stack the service will be assigned a single IPFamily. If the IPFamily is not
// set in service.spec.ipFamilies then the service will be assigned the default IPFamily
// configured on the cluster
IPFamilyPolicyPreferDualStack IPFamilyPolicyType = "PreferDualStack"
// IPFamilyPolicyRequireDualStack indicates that this service requires dual-stack. Using
// IPFamilyPolicyRequireDualStack on a single stack cluster will result in validation errors. The
// IPFamilies (and their order) assigned to this service is based on service.spec.ipFamilies. If
// service.spec.ipFamilies was not provided then it will be assigned according to how they are
// configured on the cluster. If service.spec.ipFamilies has only one entry then the alternative
// IPFamily will be added by apiserver
IPFamilyPolicyRequireDualStack IPFamilyPolicyType = "RequireDualStack"
)
// ServiceSpec describes the attributes that a user creates on a service
@ -3565,6 +3591,36 @@ type ServiceSpec struct {
// +optional
ClusterIP string
// ClusterIPs identifies all the ClusterIPs assigned to this
// service. ClusterIPs are assigned or reserved based on the values of
// service.spec.ipFamilies. A maximum of two entries (dual-stack IPs) are
// allowed in ClusterIPs. The IPFamily of each ClusterIP must match
// values provided in service.spec.ipFamilies. Clients using ClusterIPs must
// keep it in sync with ClusterIP (if provided) by having ClusterIP matching
// first element of ClusterIPs.
// +optional
ClusterIPs []string
// IPFamilies identifies all the IPFamilies assigned for this Service. If a value
// was not provided for IPFamilies it will be defaulted based on the cluster
// configuration and the value of service.spec.ipFamilyPolicy. A maximum of two
// values (dual-stack IPFamilies) are allowed in IPFamilies. IPFamilies field is
// conditionally mutable: it allows for adding or removing a secondary IPFamily,
// but it does not allow changing the primary IPFamily of the service.
// +optional
IPFamilies []IPFamily
// IPFamilyPolicy represents the dual-stack-ness requested or required by this
// Service. If there is no value provided, then this Service will be considered
// SingleStack (single IPFamily). Services can be SingleStack (single IPFamily),
// PreferDualStack (two dual-stack IPFamilies on dual-stack clusters or single
// IPFamily on single-stack clusters), or RequireDualStack (two dual-stack IPFamilies
// on dual-stack configured clusters, otherwise fail). The IPFamilies and ClusterIPs assigned
// to this service can be controlled by service.spec.ipFamilies and service.spec.clusterIPs
// respectively.
// +optional
IPFamilyPolicy *IPFamilyPolicyType
// ExternalName is the external reference that kubedns or equivalent will
// return as a CNAME record for this service. No proxying will be involved.
// Must be a valid RFC-1123 hostname (https://tools.ietf.org/html/rfc1123)
@ -3626,24 +3682,6 @@ type ServiceSpec struct {
// +optional
PublishNotReadyAddresses bool
// ipFamily specifies whether this Service has a preference for a particular IP family (e.g.
// IPv4 vs. IPv6) when the IPv6DualStack feature gate is enabled. In a dual-stack cluster,
// you can specify ipFamily when creating a ClusterIP Service to determine whether the
// controller will allocate an IPv4 or IPv6 IP for it, and you can specify ipFamily when
// creating a headless Service to determine whether it will have IPv4 or IPv6 Endpoints. In
// either case, if you do not specify an ipFamily explicitly, it will default to the
// cluster's primary IP family.
// This field is part of an alpha feature, and you should not make any assumptions about its
// semantics other than those described above. In particular, you should not assume that it
// can (or cannot) be changed after creation time; that it can only have the values "IPv4"
// and "IPv6"; or that its current value on a given Service correctly reflects the current
// state of that Service. (For ClusterIP Services, look at clusterIP to see if the Service
// is IPv4 or IPv6. For headless Services, look at the endpoints, which may be dual-stack in
// the future. For ExternalName Services, ipFamily has no meaning, but it may be set to an
// irrelevant value anyway.)
// +optional
IPFamily *IPFamily
// topologyKeys is a preference-order list of topology keys which
// implementations of services should use to preferentially sort endpoints
// when accessing this Service, it can not be used at the same time as

1
pkg/apis/core/v1/BUILD
View File

@ -27,7 +27,6 @@ go_library(
"//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/validation/field:go_default_library",
"//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library",
"//vendor/k8s.io/utils/net:go_default_library",
"//vendor/k8s.io/utils/pointer:go_default_library",
],
)

49
pkg/apis/core/v1/defaults.go
View File

@ -27,7 +27,6 @@ import (
utilfeature "k8s.io/apiserver/pkg/util/feature"
"k8s.io/kubernetes/pkg/features"
utilnet "k8s.io/utils/net"
)
func addDefaultingFuncs(scheme *runtime.Scheme) error {
@ -137,32 +136,36 @@ func SetDefaults_Service(obj *v1.Service) {
obj.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeCluster
}
// if dualstack feature gate is on then we need to default
// Spec.IPFamily correctly. This is to cover the case
// when an existing cluster have been converted to dualstack
// i.e. it already contain services with Spec.IPFamily==nil
if utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) &&
obj.Spec.Type != v1.ServiceTypeExternalName &&
obj.Spec.ClusterIP != "" && /*has an ip already set*/
obj.Spec.ClusterIP != "None" && /* not converting from ExternalName to other */
obj.Spec.IPFamily == nil /* family was not previously set */ {
if utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
// Default obj.Spec.IPFamilyPolicy if we *know* we can, otherwise it will
// be handled later in allocation.
if obj.Spec.Type != v1.ServiceTypeExternalName {
if obj.Spec.IPFamilyPolicy == nil {
if len(obj.Spec.ClusterIPs) == 2 || len(obj.Spec.IPFamilies) == 2 {
requireDualStack := v1.IPFamilyPolicyRequireDualStack
obj.Spec.IPFamilyPolicy = &requireDualStack
}
}
// there is a change that the ClusterIP (set by user) is unparsable.
// in this case, the family will be set mistakenly to ipv4 (because
// the util function does not parse errors *sigh*). The error
// will be caught in validation which asserts the validity of the
// IP and the service object will not be persisted with the wrong IP
// family
// If the user demanded dual-stack, but only specified one family, we add
// the other.
if obj.Spec.IPFamilyPolicy != nil && *(obj.Spec.IPFamilyPolicy) == v1.IPFamilyPolicyRequireDualStack && len(obj.Spec.IPFamilies) == 1 {
if obj.Spec.IPFamilies[0] == v1.IPv4Protocol {
obj.Spec.IPFamilies = append(obj.Spec.IPFamilies, v1.IPv6Protocol)
} else {
obj.Spec.IPFamilies = append(obj.Spec.IPFamilies, v1.IPv4Protocol)
}
ipv6 := v1.IPv6Protocol
ipv4 := v1.IPv4Protocol
if utilnet.IsIPv6String(obj.Spec.ClusterIP) {
obj.Spec.IPFamily = &ipv6
} else {
obj.Spec.IPFamily = &ipv4
// Any other dual-stack defaulting depends on cluster configuration.
// Further IPFamilies, IPFamilyPolicy defaulting is in ClusterIP alloc/reserve logic
// NOTE: strategy handles cases where ClusterIPs is used (but not ClusterIP).
}
}
}
// any other defaulting depends on cluster configuration.
// further IPFamilies, IPFamilyPolicy defaulting is in ClusterIP alloc/reserve logic
// note: conversion logic handles cases where ClusterIPs is used (but not ClusterIP).
}
}
func SetDefaults_Pod(obj *v1.Pod) {
// If limits are specified, but requests are not, default requests to limits

391
pkg/apis/core/v1/defaults_test.go
View File

@ -29,15 +29,16 @@ import (
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/util/diff"
"k8s.io/apimachinery/pkg/util/intstr"
utilfeature "k8s.io/apiserver/pkg/util/feature"
featuregatetesting "k8s.io/component-base/featuregate/testing"
"k8s.io/kubernetes/pkg/api/legacyscheme"
corev1 "k8s.io/kubernetes/pkg/apis/core/v1"
"k8s.io/kubernetes/pkg/features"
utilpointer "k8s.io/utils/pointer"
// ensure types are installed
_ "k8s.io/kubernetes/pkg/apis/core/install"
utilfeature "k8s.io/apiserver/pkg/util/feature"
featuregatetesting "k8s.io/component-base/featuregate/testing"
"k8s.io/kubernetes/pkg/features"
)
// TestWorkloadDefaults detects changes to defaults within PodTemplateSpec.
@ -1023,140 +1024,6 @@ func TestSetDefaultService(t *testing.T) {
}
}
func TestSetDefaultServiceIPFamily(t *testing.T) {
svc := v1.Service{
Spec: v1.ServiceSpec{
SessionAffinity: v1.ServiceAffinityNone,
Type: v1.ServiceTypeClusterIP,
},
}
testCases := []struct {
name string
inSvcTweak func(s v1.Service) v1.Service
outSvcTweak func(s v1.Service) v1.Service
enableDualStack bool
}{
{
name: "dualstack off. ipfamily not set",
inSvcTweak: func(s v1.Service) v1.Service { return s },
outSvcTweak: func(s v1.Service) v1.Service { return s },
enableDualStack: false,
},
{
name: "dualstack on. ipfamily not set, service is *not* ClusterIP-able",
inSvcTweak: func(s v1.Service) v1.Service {
s.Spec.Type = v1.ServiceTypeExternalName
return s
},
outSvcTweak: func(s v1.Service) v1.Service { return s },
enableDualStack: true,
},
{
name: "dualstack off. ipfamily set",
inSvcTweak: func(s v1.Service) v1.Service {
ipv4Service := v1.IPv4Protocol
s.Spec.IPFamily = &ipv4Service
return s
},
outSvcTweak: func(s v1.Service) v1.Service {
ipv4Service := v1.IPv4Protocol
s.Spec.IPFamily = &ipv4Service
return s
},
enableDualStack: false,
},
{
name: "dualstack off. ipfamily not set. clusterip set",
inSvcTweak: func(s v1.Service) v1.Service {
s.Spec.ClusterIP = "1.1.1.1"
return s
},
outSvcTweak: func(s v1.Service) v1.Service {
return s
},
enableDualStack: false,
},
{
name: "dualstack on. ipfamily not set (clusterIP is v4)",
inSvcTweak: func(s v1.Service) v1.Service {
s.Spec.ClusterIP = "1.1.1.1"
return s
},
outSvcTweak: func(s v1.Service) v1.Service {
ipv4Service := v1.IPv4Protocol
s.Spec.IPFamily = &ipv4Service
return s
},
enableDualStack: true,
},
{
name: "dualstack on. ipfamily not set (clusterIP is v6)",
inSvcTweak: func(s v1.Service) v1.Service {
s.Spec.ClusterIP = "fdd7:7713:8917:77ed:ffff:ffff:ffff:ffff"
return s
},
outSvcTweak: func(s v1.Service) v1.Service {
ipv6Service := v1.IPv6Protocol
s.Spec.IPFamily = &ipv6Service
return s
},
enableDualStack: true,
},
{
name: "dualstack on. ipfamily set (clusterIP is v4)",
inSvcTweak: func(s v1.Service) v1.Service {
ipv4Service := v1.IPv4Protocol
s.Spec.IPFamily = &ipv4Service
s.Spec.ClusterIP = "1.1.1.1"
return s
},
outSvcTweak: func(s v1.Service) v1.Service {
ipv4Service := v1.IPv4Protocol
s.Spec.IPFamily = &ipv4Service
return s
},
enableDualStack: true,
},
{
name: "dualstack on. ipfamily set (clusterIP is v6)",
inSvcTweak: func(s v1.Service) v1.Service {
ipv6Service := v1.IPv6Protocol
s.Spec.IPFamily = &ipv6Service
s.Spec.ClusterIP = "fdd7:7713:8917:77ed:ffff:ffff:ffff:ffff"
return s
},
outSvcTweak: func(s v1.Service) v1.Service {
ipv6Service := v1.IPv6Protocol
s.Spec.IPFamily = &ipv6Service
return s
},
enableDualStack: true,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, tc.enableDualStack)()
tweakedIn := tc.inSvcTweak(svc)
expectedSvc := tc.outSvcTweak(svc)
defaulted := roundTrip(t, runtime.Object(&tweakedIn))
defaultedSvc := defaulted.(*v1.Service)
if expectedSvc.Spec.IPFamily != nil {
if defaultedSvc.Spec.IPFamily == nil {
t.Fatalf("defaulted service ipfamily is nil while expected is not")
}
if *(expectedSvc.Spec.IPFamily) != *(defaultedSvc.Spec.IPFamily) {
t.Fatalf("defaulted service ipfamily %v does not match expected %v", defaultedSvc.Spec.IPFamily, expectedSvc.Spec.IPFamily)
}
}
if expectedSvc.Spec.IPFamily == nil && defaultedSvc.Spec.IPFamily != nil {
t.Fatalf("defaulted service ipfamily is not nil, while expected service ipfamily is")
}
})
}
}
func TestSetDefaultServiceSessionAffinityConfig(t *testing.T) {
testCases := map[string]v1.Service{
"SessionAffinityConfig is empty": {
@ -1935,3 +1802,253 @@ func TestSetDefaultEnableServiceLinks(t *testing.T) {
t.Errorf("Expected enableServiceLinks value: %+v\ngot: %+v\n", v1.DefaultEnableServiceLinks, *output.Spec.EnableServiceLinks)
}
}
func TestSetDefaultIPFamilies(t *testing.T) {
preferDualStack := v1.IPFamilyPolicyPreferDualStack
requireDualStack := v1.IPFamilyPolicyRequireDualStack
testCases := []struct {
name string
expectedIPFamiliesWithGate []v1.IPFamily
svc v1.Service
}{
{
name: "must not set for ExternalName",
expectedIPFamiliesWithGate: nil,
svc: v1.Service{
Spec: v1.ServiceSpec{
Type: v1.ServiceTypeExternalName,
},
},
},
{
name: "must not set for single stack",
expectedIPFamiliesWithGate: nil,
svc: v1.Service{
Spec: v1.ServiceSpec{
Type: v1.ServiceTypeClusterIP,
},
},
},
{
name: "must not set for single stack, even when a family is set",
expectedIPFamiliesWithGate: []v1.IPFamily{v1.IPv6Protocol},
svc: v1.Service{
Spec: v1.ServiceSpec{
Type: v1.ServiceTypeClusterIP,
IPFamilies: []v1.IPFamily{v1.IPv6Protocol},
},
},
},
{
name: "must not set for preferDualStack",
expectedIPFamiliesWithGate: []v1.IPFamily{v1.IPv6Protocol},
svc: v1.Service{
Spec: v1.ServiceSpec{
Type: v1.ServiceTypeClusterIP,
IPFamilyPolicy: &preferDualStack,
IPFamilies: []v1.IPFamily{v1.IPv6Protocol},
},
},
},
{
name: "must set for requireDualStack (6,4)",
expectedIPFamiliesWithGate: []v1.IPFamily{v1.IPv6Protocol, v1.IPv4Protocol},
svc: v1.Service{
Spec: v1.ServiceSpec{
Type: v1.ServiceTypeClusterIP,
IPFamilyPolicy: &requireDualStack,
IPFamilies: []v1.IPFamily{v1.IPv6Protocol},
},
},
},
{
name: "must set for requireDualStack (4,6)",
expectedIPFamiliesWithGate: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
svc: v1.Service{
Spec: v1.ServiceSpec{
Type: v1.ServiceTypeClusterIP,
IPFamilyPolicy: &requireDualStack,
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
},
}
for _, test := range testCases {
// run with gate
t.Run(test.name, func(t *testing.T) {
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, true)()
obj2 := roundTrip(t, runtime.Object(&test.svc))
svc2 := obj2.(*v1.Service)
if len(test.expectedIPFamiliesWithGate) != len(svc2.Spec.IPFamilies) {
t.Fatalf("expected .spec.IPFamilies len:%v got %v", len(test.expectedIPFamiliesWithGate), len(svc2.Spec.IPFamilies))
}
for i, family := range test.expectedIPFamiliesWithGate {
if svc2.Spec.IPFamilies[i] != family {
t.Fatalf("failed. expected family %v at %v got %v", family, i, svc2.Spec.IPFamilies[i])
}
}
})
// run without gate (families should not change)
t.Run(fmt.Sprintf("without-gate:%s", test.name), func(t *testing.T) {
obj2 := roundTrip(t, runtime.Object(&test.svc))
svc2 := obj2.(*v1.Service)
if len(test.svc.Spec.IPFamilies) != len(svc2.Spec.IPFamilies) {
t.Fatalf("expected .spec.IPFamilies len:%v got %v", len(test.expectedIPFamiliesWithGate), len(svc2.Spec.IPFamilies))
}
for i, family := range test.svc.Spec.IPFamilies {
if svc2.Spec.IPFamilies[i] != family {
t.Fatalf("failed. expected family %v at %v got %v", family, i, svc2.Spec.IPFamilies[i])
}
}
})
}
}
func TestSetDefaultServiceIPFamilyPolicy(t *testing.T) {
singleStack := v1.IPFamilyPolicySingleStack
preferDualStack := v1.IPFamilyPolicyPreferDualStack
requireDualStack := v1.IPFamilyPolicyRequireDualStack
testCases := []struct {
name string
expectedIPfamilyPolicy *v1.IPFamilyPolicyType
svc v1.Service
}{
{
name: "must not set for ExternalName",
expectedIPfamilyPolicy: nil,
svc: v1.Service{
Spec: v1.ServiceSpec{
Type: v1.ServiceTypeExternalName,
},
},
},
{
name: "must not set for ExternalName even with semantically valid data",
expectedIPfamilyPolicy: nil,
svc: v1.Service{
Spec: v1.ServiceSpec{
Type: v1.ServiceTypeExternalName,
ClusterIPs: []string{"1.2.3.4", "2001::1"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
},
},
},
{
name: "must set if there are more than one ip",
expectedIPfamilyPolicy: &requireDualStack,
svc: v1.Service{
Spec: v1.ServiceSpec{
ClusterIPs: []string{"1.2.3.4", "2001::1"},
},
},
},
{
name: "must set if there are more than one ip family",
expectedIPfamilyPolicy: &requireDualStack,
svc: v1.Service{
Spec: v1.ServiceSpec{
IPFamilies: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
},
},
},
{
name: "must not set if there is one ip",
expectedIPfamilyPolicy: nil,
svc: v1.Service{
Spec: v1.ServiceSpec{
ClusterIPs: []string{"1.2.3.4"},
},
},
},
{
name: "must not set if there is one ip family",
expectedIPfamilyPolicy: nil,
svc: v1.Service{
Spec: v1.ServiceSpec{
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
},
{
name: "must not override user input",
expectedIPfamilyPolicy: &singleStack,
svc: v1.Service{
Spec: v1.ServiceSpec{
IPFamilyPolicy: &singleStack,
},
},
},
{
name: "must not override user input/ preferDualStack",
expectedIPfamilyPolicy: &preferDualStack,
svc: v1.Service{
Spec: v1.ServiceSpec{
IPFamilyPolicy: &preferDualStack,
},
},
},
{
name: "must not override user input even when it is invalid",
expectedIPfamilyPolicy: &preferDualStack,
svc: v1.Service{
Spec: v1.ServiceSpec{
IPFamilies: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
IPFamilyPolicy: &preferDualStack,
},
},
},
}
for _, test := range testCases {
// with gate
t.Run(test.name, func(t *testing.T) {
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, true)()
obj2 := roundTrip(t, runtime.Object(&test.svc))
svc2 := obj2.(*v1.Service)
if test.expectedIPfamilyPolicy == nil && svc2.Spec.IPFamilyPolicy != nil {
t.Fatalf("expected .spec.PreferDualStack to be unset (nil)")
}
if test.expectedIPfamilyPolicy != nil && (svc2.Spec.IPFamilyPolicy == nil || *(svc2.Spec.IPFamilyPolicy) != *(test.expectedIPfamilyPolicy)) {
t.Fatalf("expected .spec.PreferDualStack to be set to %v got %v", *(test.expectedIPfamilyPolicy), svc2.Spec.IPFamilyPolicy)
}
})
// without gate. IPFamilyPolicy should never change
t.Run(test.name, func(t *testing.T) {
obj2 := roundTrip(t, runtime.Object(&test.svc))
svc2 := obj2.(*v1.Service)
if test.svc.Spec.IPFamilyPolicy == nil && svc2.Spec.IPFamilyPolicy != nil {
t.Fatalf("expected .spec.PreferDualStack to be unset (nil)")
}
if test.svc.Spec.IPFamilyPolicy != nil && (svc2.Spec.IPFamilyPolicy == nil || *(svc2.Spec.IPFamilyPolicy) != *(test.expectedIPfamilyPolicy)) {
t.Fatalf("expected .spec.PreferDualStack to be set to %v got %v", *(test.expectedIPfamilyPolicy), svc2.Spec.IPFamilyPolicy)
}
})
}
}

8
pkg/apis/core/v1/zz_generated.conversion.go generated
View File

@ -7582,6 +7582,7 @@ func autoConvert_v1_ServiceSpec_To_core_ServiceSpec(in *v1.ServiceSpec, out *cor
out.Ports = *(*[]core.ServicePort)(unsafe.Pointer(&in.Ports))
out.Selector = *(*map[string]string)(unsafe.Pointer(&in.Selector))
out.ClusterIP = in.ClusterIP
out.ClusterIPs = *(*[]string)(unsafe.Pointer(&in.ClusterIPs))
out.Type = core.ServiceType(in.Type)
out.ExternalIPs = *(*[]string)(unsafe.Pointer(&in.ExternalIPs))
out.SessionAffinity = core.ServiceAffinity(in.SessionAffinity)
@ -7592,8 +7593,9 @@ func autoConvert_v1_ServiceSpec_To_core_ServiceSpec(in *v1.ServiceSpec, out *cor
out.HealthCheckNodePort = in.HealthCheckNodePort
out.PublishNotReadyAddresses = in.PublishNotReadyAddresses
out.SessionAffinityConfig = (*core.SessionAffinityConfig)(unsafe.Pointer(in.SessionAffinityConfig))
out.IPFamily = (*core.IPFamily)(unsafe.Pointer(in.IPFamily))
out.IPFamilies = *(*[]core.IPFamily)(unsafe.Pointer(&in.IPFamilies))
out.TopologyKeys = *(*[]string)(unsafe.Pointer(&in.TopologyKeys))
out.IPFamilyPolicy = (*core.IPFamilyPolicyType)(unsafe.Pointer(in.IPFamilyPolicy))
return nil
}
@ -7607,6 +7609,9 @@ func autoConvert_core_ServiceSpec_To_v1_ServiceSpec(in *core.ServiceSpec, out *v
out.Ports = *(*[]v1.ServicePort)(unsafe.Pointer(&in.Ports))
out.Selector = *(*map[string]string)(unsafe.Pointer(&in.Selector))
out.ClusterIP = in.ClusterIP
out.ClusterIPs = *(*[]string)(unsafe.Pointer(&in.ClusterIPs))
out.IPFamilies = *(*[]v1.IPFamily)(unsafe.Pointer(&in.IPFamilies))
out.IPFamilyPolicy = (*v1.IPFamilyPolicyType)(unsafe.Pointer(in.IPFamilyPolicy))
out.ExternalName = in.ExternalName
out.ExternalIPs = *(*[]string)(unsafe.Pointer(&in.ExternalIPs))
out.LoadBalancerIP = in.LoadBalancerIP
@ -7616,7 +7621,6 @@ func autoConvert_core_ServiceSpec_To_v1_ServiceSpec(in *core.ServiceSpec, out *v
out.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyType(in.ExternalTrafficPolicy)
out.HealthCheckNodePort = in.HealthCheckNodePort
out.PublishNotReadyAddresses = in.PublishNotReadyAddresses
out.IPFamily = (*v1.IPFamily)(unsafe.Pointer(in.IPFamily))
out.TopologyKeys = *(*[]string)(unsafe.Pointer(&in.TopologyKeys))
return nil
}

3
pkg/apis/core/validation/BUILD
View File

@ -7,7 +7,6 @@ load(
go_library(
name = "go_default_library",
srcs = [
"conditional_validation.go",
"doc.go",
"events.go",
"validation.go",
@ -49,7 +48,6 @@ go_library(
go_test(
name = "go_default_test",
srcs = [
"conditional_validation_test.go",
"events_test.go",
"validation_test.go",
],
@ -63,7 +61,6 @@ go_test(
"//staging/src/k8s.io/api/events/v1beta1:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/diff:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/validation:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/validation/field:go_default_library",

110
pkg/apis/core/validation/conditional_validation.go
View File

@ -1,110 +0,0 @@
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package validation
import (
"fmt"
"net"
"strings"
"k8s.io/apimachinery/pkg/util/validation/field"
utilfeature "k8s.io/apiserver/pkg/util/feature"
api "k8s.io/kubernetes/pkg/apis/core"
"k8s.io/kubernetes/pkg/features"
netutils "k8s.io/utils/net"
)
// ValidateConditionalService validates conditionally valid fields. allowedIPFamilies is an ordered
// list of the valid IP families (IPv4 or IPv6) that are supported. The first family in the slice
// is the cluster default, although the clusterIP here dictates the family defaulting.
func ValidateConditionalService(service, oldService *api.Service, allowedIPFamilies []api.IPFamily) field.ErrorList {
var errs field.ErrorList
errs = append(errs, validateIPFamily(service, oldService, allowedIPFamilies)...)
return errs
}
// validateIPFamily checks the IPFamily field.
func validateIPFamily(service, oldService *api.Service, allowedIPFamilies []api.IPFamily) field.ErrorList {
var errs field.ErrorList
// specifically allow an invalid value to remain in storage as long as the user isn't changing it, regardless of gate
if oldService != nil && oldService.Spec.IPFamily != nil && service.Spec.IPFamily != nil && *oldService.Spec.IPFamily == *service.Spec.IPFamily {
return errs
}
// If the gate is off, setting or changing IPFamily is not allowed, but clearing it is
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
if service.Spec.IPFamily != nil {
if oldService != nil {
errs = append(errs, ValidateImmutableField(service.Spec.IPFamily, oldService.Spec.IPFamily, field.NewPath("spec", "ipFamily"))...)
} else {
errs = append(errs, field.Forbidden(field.NewPath("spec", "ipFamily"), "programmer error, must be cleared when the dual-stack feature gate is off"))
}
}
return errs
}
// PrepareCreate, PrepareUpdate, and test cases must all set IPFamily when the gate is on
if service.Spec.IPFamily == nil {
errs = append(errs, field.Required(field.NewPath("spec", "ipFamily"), "programmer error, must be set or defaulted by other fields"))
return errs
}
// A user is not allowed to change the IPFamily field, except for ExternalName services
if oldService != nil && oldService.Spec.IPFamily != nil && service.Spec.Type != api.ServiceTypeExternalName {
errs = append(errs, ValidateImmutableField(service.Spec.IPFamily, oldService.Spec.IPFamily, field.NewPath("spec", "ipFamily"))...)
}
// Verify the IPFamily is one of the allowed families
desiredFamily := *service.Spec.IPFamily
if hasIPFamily(allowedIPFamilies, desiredFamily) {
// the IP family is one of the allowed families, verify that it matches cluster IP
switch ip := net.ParseIP(service.Spec.ClusterIP); {
case ip == nil:
// do not need to check anything
case netutils.IsIPv6(ip) && desiredFamily != api.IPv6Protocol:
errs = append(errs, field.Invalid(field.NewPath("spec", "ipFamily"), *service.Spec.IPFamily, "does not match IPv6 cluster IP"))
case !netutils.IsIPv6(ip) && desiredFamily != api.IPv4Protocol:
errs = append(errs, field.Invalid(field.NewPath("spec", "ipFamily"), *service.Spec.IPFamily, "does not match IPv4 cluster IP"))
}
} else {
errs = append(errs, field.Invalid(field.NewPath("spec", "ipFamily"), desiredFamily, fmt.Sprintf("only the following families are allowed: %s", joinIPFamilies(allowedIPFamilies, ", "))))
}
return errs
}
func hasIPFamily(families []api.IPFamily, family api.IPFamily) bool {
for _, allow := range families {
if allow == family {
return true
}
}
return false
}
func joinIPFamilies(families []api.IPFamily, separator string) string {
var b strings.Builder
for i, family := range families {
if i != 0 {
b.WriteString(separator)
}
b.WriteString(string(family))
}
return b.String()
}

266
pkg/apis/core/validation/conditional_validation_test.go
View File

@ -1,266 +0,0 @@
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package validation
import (
"reflect"
"strings"
"testing"
"k8s.io/apimachinery/pkg/util/diff"
utilfeature "k8s.io/apiserver/pkg/util/feature"
featuregatetesting "k8s.io/component-base/featuregate/testing"
api "k8s.io/kubernetes/pkg/apis/core"
"k8s.io/kubernetes/pkg/features"
)
func TestValidateServiceIPFamily(t *testing.T) {
ipv4 := api.IPv4Protocol
ipv6 := api.IPv6Protocol
var unknown api.IPFamily = "Unknown"
testCases := []struct {
name string
dualStackEnabled bool
ipFamilies []api.IPFamily
svc *api.Service
oldSvc *api.Service
expectErr []string
}{
{
name: "allowed ipv4",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv4Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &ipv4,
},
},
},
{
name: "allowed ipv6",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv6Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &ipv6,
},
},
},
{
name: "allowed ipv4 dual stack default IPv6",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv6Protocol, api.IPv4Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &ipv4,
},
},
},
{
name: "allowed ipv4 dual stack default IPv4",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &ipv4,
},
},
},
{
name: "allowed ipv6 dual stack default IPv6",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv6Protocol, api.IPv4Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &ipv6,
},
},
},
{
name: "allowed ipv6 dual stack default IPv4",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &ipv6,
},
},
},
{
name: "allow ipfamily to remain invalid if update doesn't change it",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &unknown,
},
},
oldSvc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &unknown,
},
},
},
{
name: "not allowed ipfamily/clusterip mismatch",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &ipv4,
ClusterIP: "ffd0::1",
},
},
expectErr: []string{"spec.ipFamily: Invalid value: \"IPv4\": does not match IPv6 cluster IP"},
},
{
name: "not allowed unknown family",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv4Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &unknown,
},
},
expectErr: []string{"spec.ipFamily: Invalid value: \"Unknown\": only the following families are allowed: IPv4"},
},
{
name: "not allowed ipv4 cluster ip without family",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv6Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "127.0.0.1",
},
},
expectErr: []string{"spec.ipFamily: Required value: programmer error, must be set or defaulted by other fields"},
},
{
name: "not allowed ipv6 cluster ip without family",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv4Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "ffd0::1",
},
},
expectErr: []string{"spec.ipFamily: Required value: programmer error, must be set or defaulted by other fields"},
},
{
name: "not allowed to change ipfamily for default type",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv4Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &ipv4,
},
},
oldSvc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &ipv6,
},
},
expectErr: []string{"spec.ipFamily: Invalid value: \"IPv4\": field is immutable"},
},
{
name: "allowed to change ipfamily for external name",
dualStackEnabled: true,
ipFamilies: []api.IPFamily{api.IPv4Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeExternalName,
IPFamily: &ipv4,
},
},
oldSvc: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeExternalName,
IPFamily: &ipv6,
},
},
},
{
name: "ipfamily allowed to be empty when dual stack is off",
dualStackEnabled: false,
ipFamilies: []api.IPFamily{api.IPv4Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "127.0.0.1",
},
},
},
{
name: "ipfamily must be empty when dual stack is off",
dualStackEnabled: false,
ipFamilies: []api.IPFamily{api.IPv4Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
IPFamily: &ipv4,
ClusterIP: "127.0.0.1",
},
},
expectErr: []string{"spec.ipFamily: Forbidden: programmer error, must be cleared when the dual-stack feature gate is off"},
},
{
name: "ipfamily allowed to be cleared when dual stack is off",
dualStackEnabled: false,
ipFamilies: []api.IPFamily{api.IPv4Protocol},
svc: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
ClusterIP: "127.0.0.1",
},
},
oldSvc: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
ClusterIP: "127.0.0.1",
IPFamily: &ipv4,
},
},
expectErr: nil,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, tc.dualStackEnabled)()
oldSvc := tc.oldSvc.DeepCopy()
newSvc := tc.svc.DeepCopy()
originalNewSvc := newSvc.DeepCopy()
errs := ValidateConditionalService(newSvc, oldSvc, tc.ipFamilies)
// objects should never be changed
if !reflect.DeepEqual(oldSvc, tc.oldSvc) {
t.Errorf("old object changed: %v", diff.ObjectReflectDiff(oldSvc, tc.svc))
}
if !reflect.DeepEqual(newSvc, originalNewSvc) {
t.Errorf("new object changed: %v", diff.ObjectReflectDiff(newSvc, originalNewSvc))
}
if len(errs) != len(tc.expectErr) {
t.Fatalf("unexpected number of errors: %v", errs)
}
for i := range errs {
if !strings.Contains(errs[i].Error(), tc.expectErr[i]) {
t.Errorf("unexpected error %d: %v", i, errs[i])
}
}
})
}
}

307
pkg/apis/core/validation/validation.go
View File

@ -4139,13 +4139,16 @@ var supportedSessionAffinityType = sets.NewString(string(core.ServiceAffinityCli
var supportedServiceType = sets.NewString(string(core.ServiceTypeClusterIP), string(core.ServiceTypeNodePort),
string(core.ServiceTypeLoadBalancer), string(core.ServiceTypeExternalName))
var supportedServiceIPFamily = sets.NewString(string(core.IPv4Protocol), string(core.IPv6Protocol))
var supportedServiceIPFamilyPolicy = sets.NewString(string(core.IPFamilyPolicySingleStack), string(core.IPFamilyPolicyPreferDualStack), string(core.IPFamilyPolicyRequireDualStack))
// ValidateService tests if required fields/annotations of a Service are valid.
func ValidateService(service *core.Service, allowAppProtocol bool) field.ErrorList {
allErrs := ValidateObjectMeta(&service.ObjectMeta, true, ValidateServiceName, field.NewPath("metadata"))
specPath := field.NewPath("spec")
isHeadlessService := service.Spec.ClusterIP == core.ClusterIPNone
if len(service.Spec.Ports) == 0 && !isHeadlessService && service.Spec.Type != core.ServiceTypeExternalName {
if len(service.Spec.Ports) == 0 && !isHeadlessService(service) && service.Spec.Type != core.ServiceTypeExternalName {
allErrs = append(allErrs, field.Required(specPath.Child("ports"), ""))
}
switch service.Spec.Type {
@ -4160,16 +4163,25 @@ func ValidateService(service *core.Service, allowAppProtocol bool) field.ErrorLi
allErrs = append(allErrs, field.Invalid(portPath, port.Port, fmt.Sprintf("may not expose port %v externally since it is used by kubelet", ports.KubeletPort)))
}
}
if service.Spec.ClusterIP == "None" {
allErrs = append(allErrs, field.Invalid(specPath.Child("clusterIP"), service.Spec.ClusterIP, "may not be set to 'None' for LoadBalancer services"))
if isHeadlessService(service) {
allErrs = append(allErrs, field.Invalid(specPath.Child("clusterIPs").Index(0), service.Spec.ClusterIPs[0], "may not be set to 'None' for LoadBalancer services"))
}
case core.ServiceTypeNodePort:
if service.Spec.ClusterIP == "None" {
allErrs = append(allErrs, field.Invalid(specPath.Child("clusterIP"), service.Spec.ClusterIP, "may not be set to 'None' for NodePort services"))
if isHeadlessService(service) {
allErrs = append(allErrs, field.Invalid(specPath.Child("clusterIPs").Index(0), service.Spec.ClusterIPs[0], "may not be set to 'None' for NodePort services"))
}
case core.ServiceTypeExternalName:
if service.Spec.ClusterIP != "" {
allErrs = append(allErrs, field.Forbidden(specPath.Child("clusterIP"), "must be empty for ExternalName services"))
// must have len(.spec.ClusterIPs) == 0 // note: strategy sets ClusterIPs based on ClusterIP
if len(service.Spec.ClusterIPs) > 0 {
allErrs = append(allErrs, field.Forbidden(specPath.Child("clusterIPs"), "may not be set for ExternalName services"))
}
// must have nil families and nil policy
if len(service.Spec.IPFamilies) > 0 {
allErrs = append(allErrs, field.Forbidden(specPath.Child("ipFamilies"), "may not be set for ExternalName services"))
}
if service.Spec.IPFamilyPolicy != nil {
allErrs = append(allErrs, field.Forbidden(specPath.Child("ipFamilyPolicy"), "may not be set for ExternalName services"))
}
// The value (a CNAME) may have a trailing dot to denote it as fully qualified
@ -4185,7 +4197,7 @@ func ValidateService(service *core.Service, allowAppProtocol bool) field.ErrorLi
portsPath := specPath.Child("ports")
for i := range service.Spec.Ports {
portPath := portsPath.Index(i)
allErrs = append(allErrs, validateServicePort(&service.Spec.Ports[i], len(service.Spec.Ports) > 1, isHeadlessService, allowAppProtocol, &allPortNames, portPath)...)
allErrs = append(allErrs, validateServicePort(&service.Spec.Ports[i], len(service.Spec.Ports) > 1, isHeadlessService(service), allowAppProtocol, &allPortNames, portPath)...)
}
if service.Spec.Selector != nil {
@ -4206,11 +4218,8 @@ func ValidateService(service *core.Service, allowAppProtocol bool) field.ErrorLi
}
}
if helper.IsServiceIPSet(service) {
if ip := net.ParseIP(service.Spec.ClusterIP); ip == nil {
allErrs = append(allErrs, field.Invalid(specPath.Child("clusterIP"), service.Spec.ClusterIP, "must be empty, 'None', or a valid IP address"))
}
}
// dualstack <-> ClusterIPs <-> ipfamilies
allErrs = append(allErrs, validateServiceClusterIPsRelatedFields(service)...)
ipPath := specPath.Child("externalIPs")
for i, ip := range service.Spec.ExternalIPs {
@ -4338,6 +4347,7 @@ func ValidateService(service *core.Service, allowAppProtocol bool) field.ErrorLi
}
}
// external traffic fields
allErrs = append(allErrs, validateServiceExternalTrafficFieldsValue(service)...)
return allErrs
}
@ -4446,13 +4456,16 @@ func ValidateServiceCreate(service *core.Service) field.ErrorList {
func ValidateServiceUpdate(service, oldService *core.Service) field.ErrorList {
allErrs := ValidateObjectMetaUpdate(&service.ObjectMeta, &oldService.ObjectMeta, field.NewPath("metadata"))
// ClusterIP should be immutable for services using it (every type other than ExternalName)
// which do not have ClusterIP assigned yet (empty string value)
if service.Spec.Type != core.ServiceTypeExternalName {
if oldService.Spec.Type != core.ServiceTypeExternalName && oldService.Spec.ClusterIP != "" {
allErrs = append(allErrs, ValidateImmutableField(service.Spec.ClusterIP, oldService.Spec.ClusterIP, field.NewPath("spec", "clusterIP"))...)
}
}
// User can upgrade (add another clusterIP or ipFamily)
// can downgrade (remove secondary clusterIP or ipFamily)
// but *CAN NOT* change primary/secondary clusterIP || ipFamily *UNLESS*
// they are changing from/to/ON ExternalName
upgradeDowngradeClusterIPsErrs := validateUpgradeDowngradeClusterIPs(oldService, service)
allErrs = append(allErrs, upgradeDowngradeClusterIPsErrs...)
upgradeDowngradeIPFamiliesErrs := validateUpgradeDowngradeIPFamilies(oldService, service)
allErrs = append(allErrs, upgradeDowngradeIPFamiliesErrs...)
// allow AppProtocol value if the feature gate is set or the field is
// already set on the resource.
@ -6094,3 +6107,255 @@ func ValidateSpreadConstraintNotRepeat(fldPath *field.Path, constraint core.Topo
}
return nil
}
// validateServiceClusterIPsRelatedFields validates .spec.ClusterIPs,, .spec.IPFamilies, .spec.ipFamilyPolicy
func validateServiceClusterIPsRelatedFields(service *core.Service) field.ErrorList {
// ClusterIP, ClusterIPs, IPFamilyPolicy and IPFamilies are validated prior (all must be unset) for ExternalName service
if service.Spec.Type == core.ServiceTypeExternalName {
return field.ErrorList{}
}
allErrs := field.ErrorList{}
hasInvalidIPs := false
specPath := field.NewPath("spec")
clusterIPsField := specPath.Child("clusterIPs")
ipFamiliesField := specPath.Child("ipFamilies")
ipFamilyPolicyField := specPath.Child("ipFamilyPolicy")
// Make sure ClusterIP and ClusterIPs are synced. For most cases users can
// just manage one or the other and we'll handle the rest (see PrepareFor*
// in strategy).
if len(service.Spec.ClusterIP) != 0 {
// If ClusterIP is set, ClusterIPs[0] must match.
if len(service.Spec.ClusterIPs) == 0 {
allErrs = append(allErrs, field.Required(clusterIPsField, ""))
} else if service.Spec.ClusterIPs[0] != service.Spec.ClusterIP {
allErrs = append(allErrs, field.Invalid(clusterIPsField, service.Spec.ClusterIPs, "element [0] must match clusterIP"))
}
} else { // ClusterIP == ""
// If ClusterIP is not set, ClusterIPs must also be unset.
if len(service.Spec.ClusterIPs) != 0 {
allErrs = append(allErrs, field.Invalid(clusterIPsField, service.Spec.ClusterIPs, "must be empty when clusterIP is empty"))
}
}
// ipfamilies stand alone validation
// must be either IPv4 or IPv6
seen := sets.String{}
for i, ipFamily := range service.Spec.IPFamilies {
if !supportedServiceIPFamily.Has(string(ipFamily)) {
allErrs = append(allErrs, field.NotSupported(ipFamiliesField.Index(i), ipFamily, supportedServiceIPFamily.List()))
}
// no duplicate check also ensures that ipfamilies is dualstacked, in any order
if seen.Has(string(ipFamily)) {
allErrs = append(allErrs, field.Duplicate(ipFamiliesField.Index(i), ipFamily))
}
seen.Insert(string(ipFamily))
}
// IPFamilyPolicy stand alone validation
//note: nil is ok, defaulted in alloc check registry/core/service/*
if service.Spec.IPFamilyPolicy != nil {
// must have a supported value
if !supportedServiceIPFamilyPolicy.Has(string(*(service.Spec.IPFamilyPolicy))) {
allErrs = append(allErrs, field.NotSupported(ipFamilyPolicyField, service.Spec.IPFamilyPolicy, supportedServiceIPFamilyPolicy.List()))
}
}
// clusterIPs stand alone validation
// valid ips with None and empty string handling
// duplication check is done as part of DualStackvalidation below
for i, clusterIP := range service.Spec.ClusterIPs {
// valid at first location only. if and only if len(clusterIPs) == 1
if i == 0 && clusterIP == core.ClusterIPNone {
if len(service.Spec.ClusterIPs) > 1 {
hasInvalidIPs = true
allErrs = append(allErrs, field.Invalid(clusterIPsField, service.Spec.ClusterIPs, "'None' must be the first and only value"))
}
continue
}
// is it valid ip?
errorMessages := validation.IsValidIP(clusterIP)
hasInvalidIPs = (len(errorMessages) != 0) || hasInvalidIPs
for _, msg := range errorMessages {
allErrs = append(allErrs, field.Invalid(clusterIPsField.Index(i), clusterIP, msg))
}
}
// max two
if len(service.Spec.ClusterIPs) > 2 {
allErrs = append(allErrs, field.Invalid(clusterIPsField, service.Spec.ClusterIPs, "may only hold up to 2 values"))
}
// at this stage if there is an invalid ip or misplaced none/empty string
// it will skew the error messages (bad index || dualstackness of already bad ips). so we
// stop here if there are errors in clusterIPs validation
if hasInvalidIPs {
return allErrs
}
// must be dual stacked ips if they are more than one ip
if len(service.Spec.ClusterIPs) > 1 /* meaning: it does not have a None or empty string */ {
dualStack, err := netutils.IsDualStackIPStrings(service.Spec.ClusterIPs)
if err != nil { // though we check for that earlier. safe > sorry
allErrs = append(allErrs, field.InternalError(clusterIPsField, fmt.Errorf("failed to check for dual stack with error:%v", err)))
}
// We only support one from each IP family (i.e. max two IPs in this list).
if !dualStack {
allErrs = append(allErrs, field.Invalid(clusterIPsField, service.Spec.ClusterIPs, "may specify no more than one IP for each IP family"))
}
}
// match clusterIPs to their families, if they were provided
if !isHeadlessService(service) && len(service.Spec.ClusterIPs) > 0 && len(service.Spec.IPFamilies) > 0 {
for i, ip := range service.Spec.ClusterIPs {
if i > (len(service.Spec.IPFamilies) - 1) {
break // no more families to check
}
// 4=>6
if service.Spec.IPFamilies[i] == core.IPv4Protocol && netutils.IsIPv6String(ip) {
allErrs = append(allErrs, field.Invalid(clusterIPsField.Index(i), ip, fmt.Sprintf("expected an IPv4 value as indicated by `ipFamilies[%v]`", i)))
}
// 6=>4
if service.Spec.IPFamilies[i] == core.IPv6Protocol && !netutils.IsIPv6String(ip) {
allErrs = append(allErrs, field.Invalid(clusterIPsField.Index(i), ip, fmt.Sprintf("expected an IPv6 value as indicated by `ipFamilies[%v]`", i)))
}
}
}
return allErrs
}
// specific validation for clusterIPs in cases of user upgrading or downgrading to/from dualstack
func validateUpgradeDowngradeClusterIPs(oldService, service *core.Service) field.ErrorList {
allErrs := make(field.ErrorList, 0)
// bail out early for ExternalName
if service.Spec.Type == core.ServiceTypeExternalName || oldService.Spec.Type == core.ServiceTypeExternalName {
return allErrs
}
newIsHeadless := isHeadlessService(service)
oldIsHeadless := isHeadlessService(oldService)
if oldIsHeadless && newIsHeadless {
return allErrs
}
switch {
// no change in ClusterIP lengths
// compare each
case len(oldService.Spec.ClusterIPs) == len(service.Spec.ClusterIPs):
for i, ip := range oldService.Spec.ClusterIPs {
if ip != service.Spec.ClusterIPs[i] {
allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "clusterIPs").Index(i), service.Spec.ClusterIPs, "may not change once set"))
}
}
// something has been released (downgraded)
case len(oldService.Spec.ClusterIPs) > len(service.Spec.ClusterIPs):
// primary ClusterIP has been released
if len(service.Spec.ClusterIPs) == 0 {
allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "clusterIPs").Index(0), service.Spec.ClusterIPs, "primary clusterIP can not be unset"))
}
// test if primary clusterIP has changed
if len(oldService.Spec.ClusterIPs) > 0 &&
len(service.Spec.ClusterIPs) > 0 &&
service.Spec.ClusterIPs[0] != oldService.Spec.ClusterIPs[0] {
allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "clusterIPs").Index(0), service.Spec.ClusterIPs, "may not change once set"))
}
// test if secondary ClusterIP has been released. has this service been downgraded correctly?
// user *must* set IPFamilyPolicy == SingleStack
if len(service.Spec.ClusterIPs) == 1 {
if service.Spec.IPFamilyPolicy == nil || *(service.Spec.IPFamilyPolicy) != core.IPFamilyPolicySingleStack {
allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "clusterIPs").Index(0), service.Spec.ClusterIPs, "`ipFamilyPolicy` must be set to 'SingleStack' when releasing the secondary clusterIP"))
}
}
case len(oldService.Spec.ClusterIPs) < len(service.Spec.ClusterIPs):
// something has been added (upgraded)
// test if primary clusterIP has changed
if len(oldService.Spec.ClusterIPs) > 0 &&
service.Spec.ClusterIPs[0] != oldService.Spec.ClusterIPs[0] {
allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "clusterIPs").Index(0), service.Spec.ClusterIPs, "may not change once set"))
}
// we don't check for Policy == RequireDualStack here since, Validation/Creation func takes care of it
}
return allErrs
}
// specific validation for ipFamilies in cases of user upgrading or downgrading to/from dualstack
func validateUpgradeDowngradeIPFamilies(oldService, service *core.Service) field.ErrorList {
allErrs := make(field.ErrorList, 0)
// bail out early for ExternalName
if service.Spec.Type == core.ServiceTypeExternalName || oldService.Spec.Type == core.ServiceTypeExternalName {
return allErrs
}
oldIsHeadless := isHeadlessService(oldService)
newIsHeadless := isHeadlessService(service)
// if changed to/from headless, then bail out
if newIsHeadless != oldIsHeadless {
return allErrs
}
// headless can change families
if newIsHeadless {
return allErrs
}
switch {
case len(oldService.Spec.IPFamilies) == len(service.Spec.IPFamilies):
// no change in ClusterIP lengths
// compare each
for i, ip := range oldService.Spec.IPFamilies {
if ip != service.Spec.IPFamilies[i] {
allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "ipFamilies").Index(0), service.Spec.IPFamilies, "may not change once set"))
}
}
case len(oldService.Spec.IPFamilies) > len(service.Spec.IPFamilies):
// something has been released (downgraded)
// test if primary ipfamily has been released
if len(service.Spec.ClusterIPs) == 0 {
allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "ipFamilies").Index(0), service.Spec.IPFamilies, "primary ipFamily can not be unset"))
}
// test if primary ipFamily has changed
if len(service.Spec.IPFamilies) > 0 &&
service.Spec.IPFamilies[0] != oldService.Spec.IPFamilies[0] {
allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "ipFamilies").Index(0), service.Spec.ClusterIPs, "may not change once set"))
}
// test if secondary IPFamily has been released. has this service been downgraded correctly?
// user *must* set IPFamilyPolicy == SingleStack
if len(service.Spec.IPFamilies) == 1 {
if service.Spec.IPFamilyPolicy == nil || *(service.Spec.IPFamilyPolicy) != core.IPFamilyPolicySingleStack {
allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "clusterIPs").Index(0), service.Spec.ClusterIPs, "`ipFamilyPolicy` must be set to 'SingleStack' when releasing the secondary ipFamily"))
}
}
case len(oldService.Spec.IPFamilies) < len(service.Spec.IPFamilies):
// something has been added (upgraded)
// test if primary ipFamily has changed
if len(oldService.Spec.IPFamilies) > 0 &&
len(service.Spec.IPFamilies) > 0 &&
service.Spec.IPFamilies[0] != oldService.Spec.IPFamilies[0] {
allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "ipFamilies").Index(0), service.Spec.ClusterIPs, "may not change once set"))
}
// we don't check for Policy == RequireDualStack here since, Validation/Creation func takes care of it
}
return allErrs
}
func isHeadlessService(service *core.Service) bool {
return service != nil &&
len(service.Spec.ClusterIPs) == 1 &&
service.Spec.ClusterIPs[0] == core.ClusterIPNone
}

933
pkg/apis/core/validation/validation_test.go
View File

File diff suppressed because it is too large Load Diff

20
pkg/apis/core/zz_generated.deepcopy.go generated
View File

@ -5255,6 +5255,21 @@ func (in *ServiceSpec) DeepCopyInto(out *ServiceSpec) {
(*out)[key] = val
}
}
if in.ClusterIPs != nil {
in, out := &in.ClusterIPs, &out.ClusterIPs
*out = make([]string, len(*in))
copy(*out, *in)
}
if in.IPFamilies != nil {
in, out := &in.IPFamilies, &out.IPFamilies
*out = make([]IPFamily, len(*in))
copy(*out, *in)
}
if in.IPFamilyPolicy != nil {
in, out := &in.IPFamilyPolicy, &out.IPFamilyPolicy
*out = new(IPFamilyPolicyType)
**out = **in
}
if in.ExternalIPs != nil {
in, out := &in.ExternalIPs, &out.ExternalIPs
*out = make([]string, len(*in))
@ -5270,11 +5285,6 @@ func (in *ServiceSpec) DeepCopyInto(out *ServiceSpec) {
*out = make([]string, len(*in))
copy(*out, *in)
}
if in.IPFamily != nil {
in, out := &in.IPFamily, &out.IPFamily
*out = new(IPFamily)
**out = **in
}
if in.TopologyKeys != nil {
in, out := &in.TopologyKeys, &out.TopologyKeys
*out = make([]string, len(*in))

39
pkg/controller/endpoint/endpoints_controller.go
View File

@ -216,22 +216,49 @@ func (e *Controller) addPod(obj interface{}) {
func podToEndpointAddressForService(svc *v1.Service, pod *v1.Pod) (*v1.EndpointAddress, error) {
var endpointIP string
ipFamily := v1.IPv4Protocol
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
// In a legacy cluster, the pod IP is guaranteed to be usable
endpointIP = pod.Status.PodIP
} else {
ipv6Service := endpointutil.IsIPv6Service(svc)
//feature flag enabled and pods may have multiple IPs
if len(svc.Spec.IPFamilies) > 0 {
// controller is connected to an api-server that correctly sets IPFamilies
ipFamily = svc.Spec.IPFamilies[0] // this works for headful and headless
} else {
// controller is connected to an api server that does not correctly
// set IPFamilies (e.g. old api-server during an upgrade)
if len(svc.Spec.ClusterIP) > 0 && svc.Spec.ClusterIP != v1.ClusterIPNone {
// headful service. detect via service clusterIP
if utilnet.IsIPv6String(svc.Spec.ClusterIP) {
ipFamily = v1.IPv6Protocol
}
} else {
// Since this is a headless service we use podIP to identify the family.
// This assumes that status.PodIP is assigned correctly (follows pod cidr and
// pod cidr list order is same as service cidr list order). The expectation is
// this is *most probably* the case.
// if the family was incorrectly indentified then this will be corrected once the
// the upgrade is completed (controller connects to api-server that correctly defaults services)
if utilnet.IsIPv6String(pod.Status.PodIP) {
ipFamily = v1.IPv6Protocol
}
}
}
// find an ip that matches the family
for _, podIP := range pod.Status.PodIPs {
ipv6PodIP := utilnet.IsIPv6String(podIP.IP)
if ipv6Service == ipv6PodIP {
if (ipFamily == v1.IPv6Protocol) == utilnet.IsIPv6String(podIP.IP) {
endpointIP = podIP.IP
break
}
}
if endpointIP == "" {
return nil, fmt.Errorf("failed to find a matching endpoint for service %v", svc.Name)
}
}
if endpointIP == "" {
return nil, fmt.Errorf("failed to find a matching endpoint for service %v", svc.Name)
}
return &v1.EndpointAddress{

17
pkg/controller/endpoint/endpoints_controller_test.go
View File

@ -1256,8 +1256,8 @@ func TestPodToEndpointAddressForService(t *testing.T) {
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIP: v1.ClusterIPNone,
IPFamily: &ipv4,
ClusterIP: v1.ClusterIPNone,
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
@ -1289,7 +1289,7 @@ func TestPodToEndpointAddressForService(t *testing.T) {
},
},
expectedEndpointFamily: ipv4,
expectedEndpointFamily: ipv6,
},
{
name: "v6 service, in a dual stack cluster",
@ -1320,33 +1320,32 @@ func TestPodToEndpointAddressForService(t *testing.T) {
expectedEndpointFamily: ipv6,
},
{
name: "v6 headless service, in a dual stack cluster",
name: "v6 headless service, in a dual stack cluster (connected to a new api-server)",
enableDualStack: true,
ipFamilies: ipv4ipv6,
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIP: v1.ClusterIPNone,
IPFamily: &ipv6,
ClusterIP: v1.ClusterIPNone,
IPFamilies: []v1.IPFamily{v1.IPv6Protocol}, // <- set by a api-server defaulting logic
},
},
expectedEndpointFamily: ipv6,
},
{
name: "v6 legacy headless service, in a dual stack cluster",
name: "v6 legacy headless service, in a dual stack cluster (connected to a old api-server)",
enableDualStack: false,
ipFamilies: ipv4ipv6,
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIP: v1.ClusterIPNone,
ClusterIP: v1.ClusterIPNone, // <- families are not set by api-server
},
},
// This is not the behavior we *want*, but it's the behavior we currently expect.
expectedEndpointFamily: ipv4,
},

1
pkg/controller/endpointslice/BUILD
View File

@ -27,6 +27,7 @@ go_library(
"//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/types:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/errors:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/runtime:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library",

25
pkg/controller/endpointslice/endpointslice_controller_test.go
View File

@ -340,7 +340,6 @@ func TestSyncServiceFull(t *testing.T) {
client, esController := newController([]string{"node-1"}, time.Duration(0))
namespace := metav1.NamespaceDefault
serviceName := "all-the-protocols"
ipv6Family := v1.IPv6Protocol
pod1 := newPod(1, namespace, true, 0)
pod1.Status.PodIPs = []v1.PodIP{{IP: "1.2.3.4"}}
@ -364,8 +363,8 @@ func TestSyncServiceFull(t *testing.T) {
{Name: "udp-example", TargetPort: intstr.FromInt(161), Protocol: v1.ProtocolUDP},
{Name: "sctp-example", TargetPort: intstr.FromInt(3456), Protocol: v1.ProtocolSCTP},
},
Selector: map[string]string{"foo": "bar"},
IPFamily: &ipv6Family,
Selector: map[string]string{"foo": "bar"},
IPFamilies: []v1.IPFamily{v1.IPv6Protocol},
},
}
esController.serviceStore.Add(service)
@ -491,8 +490,9 @@ func TestPodAddsBatching(t *testing.T) {
esController.serviceStore.Add(&v1.Service{
ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: ns},
Spec: v1.ServiceSpec{
Selector: map[string]string{"foo": "bar"},
Ports: []v1.ServicePort{{Port: 80}},
Selector: map[string]string{"foo": "bar"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
Ports: []v1.ServicePort{{Port: 80}},
},
})
@ -624,8 +624,9 @@ func TestPodUpdatesBatching(t *testing.T) {
esController.serviceStore.Add(&v1.Service{
ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: ns},
Spec: v1.ServiceSpec{
Selector: map[string]string{"foo": "bar"},
Ports: []v1.ServicePort{{Port: 80}},
Selector: map[string]string{"foo": "bar"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
Ports: []v1.ServicePort{{Port: 80}},
},
})
@ -758,8 +759,9 @@ func TestPodDeleteBatching(t *testing.T) {
esController.serviceStore.Add(&v1.Service{
ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: ns},
Spec: v1.ServiceSpec{
Selector: map[string]string{"foo": "bar"},
Ports: []v1.ServicePort{{Port: 80}},
Selector: map[string]string{"foo": "bar"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
Ports: []v1.ServicePort{{Port: 80}},
},
})
@ -810,8 +812,9 @@ func createService(t *testing.T, esController *endpointSliceController, namespac
UID: types.UID(namespace + "-" + serviceName),
},
Spec: v1.ServiceSpec{
Ports: []v1.ServicePort{{TargetPort: intstr.FromInt(80)}},
Selector: map[string]string{"foo": "bar"},
Ports: []v1.ServicePort{{TargetPort: intstr.FromInt(80)}},
Selector: map[string]string{"foo": "bar"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
}
esController.serviceStore.Add(service)

61
pkg/controller/endpointslice/reconciler.go
View File

@ -27,6 +27,7 @@ import (
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
utilerrors "k8s.io/apimachinery/pkg/util/errors"
"k8s.io/apimachinery/pkg/util/sets"
clientset "k8s.io/client-go/kubernetes"
corelisters "k8s.io/client-go/listers/core/v1"
@ -56,12 +57,62 @@ type endpointMeta struct {
// slices for the given service. It creates, updates, or deletes endpoint slices
// to ensure the desired set of pods are represented by endpoint slices.
func (r *reconciler) reconcile(service *corev1.Service, pods []*corev1.Pod, existingSlices []*discovery.EndpointSlice, triggerTime time.Time) error {
addressType := discovery.AddressTypeIPv4
slicesToDelete := []*discovery.EndpointSlice{} // slices that are no longer matching any address the service has
errs := []error{} // all errors generated in the process of reconciling
slicesByAddressType := make(map[discovery.AddressType][]*discovery.EndpointSlice) // slices by address type
if endpointutil.IsIPv6Service(service) {
addressType = discovery.AddressTypeIPv6
// addresses that this service supports [o(1) find]
serviceSupportedAddressesTypes := getAddressTypesForService(service)
// loop through slices identifying their address type.
// slices that no longer match address type supported by services
// go to delete, other slices goes to the reconciler machinery
// for further adjustment
for _, existingSlice := range existingSlices {
// service no longer supports that address type, add it to deleted slices
if _, ok := serviceSupportedAddressesTypes[existingSlice.AddressType]; !ok {
slicesToDelete = append(slicesToDelete, existingSlice)
continue
}
// add list if it is not on our map
if _, ok := slicesByAddressType[existingSlice.AddressType]; !ok {
slicesByAddressType[existingSlice.AddressType] = make([]*discovery.EndpointSlice, 0, 1)
}
slicesByAddressType[existingSlice.AddressType] = append(slicesByAddressType[existingSlice.AddressType], existingSlice)
}
// reconcile for existing.
for addressType := range serviceSupportedAddressesTypes {
existingSlices := slicesByAddressType[addressType]
err := r.reconcileByAddressType(service, pods, existingSlices, triggerTime, addressType)
if err != nil {
errs = append(errs, err)
}
}
// delete those which are of addressType that is no longer supported
// by the service
for _, sliceToDelete := range slicesToDelete {
err := r.client.DiscoveryV1beta1().EndpointSlices(service.Namespace).Delete(context.TODO(), sliceToDelete.Name, metav1.DeleteOptions{})
if err != nil {
errs = append(errs, fmt.Errorf("Error deleting %s EndpointSlice for Service %s/%s: %v", sliceToDelete.Name, service.Namespace, service.Name, err))
} else {
r.endpointSliceTracker.Delete(sliceToDelete)
metrics.EndpointSliceChanges.WithLabelValues("delete").Inc()
}
}
return utilerrors.NewAggregate(errs)
}
// reconcileByAddressType takes a set of pods currently matching a service selector and
// compares them with the endpoints already present in any existing endpoint
// slices (by address type) for the given service. It creates, updates, or deletes endpoint slices
// to ensure the desired set of pods are represented by endpoint slices.
func (r *reconciler) reconcileByAddressType(service *corev1.Service, pods []*corev1.Pod, existingSlices []*discovery.EndpointSlice, triggerTime time.Time, addressType discovery.AddressType) error {
slicesToCreate := []*discovery.EndpointSlice{}
slicesToUpdate := []*discovery.EndpointSlice{}
slicesToDelete := []*discovery.EndpointSlice{}
@ -70,7 +121,7 @@ func (r *reconciler) reconcile(service *corev1.Service, pods []*corev1.Pod, exis
existingSlicesByPortMap := map[endpointutil.PortMapKey][]*discovery.EndpointSlice{}
numExistingEndpoints := 0
for _, existingSlice := range existingSlices {
if existingSlice.AddressType == addressType && ownedBy(existingSlice, service) {
if ownedBy(existingSlice, service) {
epHash := endpointutil.NewPortMapKey(existingSlice.Ports)
existingSlicesByPortMap[epHash] = append(existingSlicesByPortMap[epHash], existingSlice)
numExistingEndpoints += len(existingSlice.Endpoints)
@ -106,7 +157,7 @@ func (r *reconciler) reconcile(service *corev1.Service, pods []*corev1.Pod, exis
if err != nil {
return err
}
endpoint := podToEndpoint(pod, node, service)
endpoint := podToEndpoint(pod, node, service, addressType)
if len(endpoint.Addresses) > 0 {
desiredEndpointsByPortMap[epHash].Insert(&endpoint)
numDesiredEndpoints++

349
pkg/controller/endpointslice/reconciler_test.go
View File

@ -70,7 +70,10 @@ func TestReconcileEmpty(t *testing.T) {
// a slice should be created
func TestReconcile1Pod(t *testing.T) {
namespace := "test"
ipv6Family := corev1.IPv6Protocol
noFamilyService, _ := newServiceAndEndpointMeta("foo", namespace)
noFamilyService.Spec.ClusterIP = "10.0.0.10"
noFamilyService.Spec.IPFamilies = nil
svcv4, _ := newServiceAndEndpointMeta("foo", namespace)
svcv4ClusterIP, _ := newServiceAndEndpointMeta("foo", namespace)
svcv4ClusterIP.Spec.ClusterIP = "1.1.1.1"
@ -80,9 +83,17 @@ func TestReconcile1Pod(t *testing.T) {
svcv4BadLabels.Labels = map[string]string{discovery.LabelServiceName: "bad",
discovery.LabelManagedBy: "actor", corev1.IsHeadlessService: "invalid"}
svcv6, _ := newServiceAndEndpointMeta("foo", namespace)
svcv6.Spec.IPFamily = &ipv6Family
svcv6.Spec.IPFamilies = []corev1.IPFamily{corev1.IPv6Protocol}
svcv6ClusterIP, _ := newServiceAndEndpointMeta("foo", namespace)
svcv6ClusterIP.Spec.ClusterIP = "1234::5678:0000:0000:9abc:def1"
// newServiceAndEndpointMeta generates v4 single stack
svcv6ClusterIP.Spec.IPFamilies = []corev1.IPFamily{corev1.IPv6Protocol}
// dual stack
dualStackSvc, _ := newServiceAndEndpointMeta("foo", namespace)
dualStackSvc.Spec.IPFamilies = []corev1.IPFamily{corev1.IPv4Protocol, corev1.IPv6Protocol}
dualStackSvc.Spec.ClusterIP = "10.0.0.10"
dualStackSvc.Spec.ClusterIPs = []string{"10.0.0.10", "2000::1"}
pod1 := newPod(1, namespace, true, 1)
pod1.Status.PodIPs = []corev1.PodIP{{IP: "1.2.3.4"}, {IP: "1234::5678:0000:0000:9abc:def0"}}
@ -98,26 +109,56 @@ func TestReconcile1Pod(t *testing.T) {
}
testCases := map[string]struct {
service corev1.Service
expectedAddressType discovery.AddressType
expectedEndpoint discovery.Endpoint
expectedLabels map[string]string
service corev1.Service
expectedAddressType discovery.AddressType
expectedEndpoint discovery.Endpoint
expectedLabels map[string]string
expectedEndpointPerSlice map[discovery.AddressType][]discovery.Endpoint
}{
"ipv4": {
service: svcv4,
expectedAddressType: discovery.AddressTypeIPv4,
expectedEndpoint: discovery.Endpoint{
Addresses: []string{"1.2.3.4"},
Conditions: discovery.EndpointConditions{Ready: utilpointer.BoolPtr(true)},
Topology: map[string]string{
"kubernetes.io/hostname": "node-1",
"topology.kubernetes.io/zone": "us-central1-a",
"topology.kubernetes.io/region": "us-central1",
"no-family-service": {
service: noFamilyService,
expectedEndpointPerSlice: map[discovery.AddressType][]discovery.Endpoint{
discovery.AddressTypeIPv4: {
{
Addresses: []string{"1.2.3.4"},
Conditions: discovery.EndpointConditions{Ready: utilpointer.BoolPtr(true)},
Topology: map[string]string{
"kubernetes.io/hostname": "node-1",
"topology.kubernetes.io/zone": "us-central1-a",
"topology.kubernetes.io/region": "us-central1",
},
TargetRef: &corev1.ObjectReference{
Kind: "Pod",
Namespace: namespace,
Name: "pod1",
},
},
},
TargetRef: &corev1.ObjectReference{
Kind: "Pod",
Namespace: namespace,
Name: "pod1",
},
expectedLabels: map[string]string{
discovery.LabelManagedBy: controllerName,
discovery.LabelServiceName: "foo",
},
},
"ipv4": {
service: svcv4,
expectedEndpointPerSlice: map[discovery.AddressType][]discovery.Endpoint{
discovery.AddressTypeIPv4: {
{
Addresses: []string{"1.2.3.4"},
Conditions: discovery.EndpointConditions{Ready: utilpointer.BoolPtr(true)},
Topology: map[string]string{
"kubernetes.io/hostname": "node-1",
"topology.kubernetes.io/zone": "us-central1-a",
"topology.kubernetes.io/region": "us-central1",
},
TargetRef: &corev1.ObjectReference{
Kind: "Pod",
Namespace: namespace,
Name: "pod1",
},
},
},
},
expectedLabels: map[string]string{
@ -127,7 +168,25 @@ func TestReconcile1Pod(t *testing.T) {
},
},
"ipv4-clusterip": {
service: svcv4ClusterIP,
service: svcv4ClusterIP,
expectedEndpointPerSlice: map[discovery.AddressType][]discovery.Endpoint{
discovery.AddressTypeIPv4: {
{
Addresses: []string{"1.2.3.4"},
Conditions: discovery.EndpointConditions{Ready: utilpointer.BoolPtr(true)},
Topology: map[string]string{
"kubernetes.io/hostname": "node-1",
"topology.kubernetes.io/zone": "us-central1-a",
"topology.kubernetes.io/region": "us-central1",
},
TargetRef: &corev1.ObjectReference{
Kind: "Pod",
Namespace: namespace,
Name: "pod1",
},
},
},
},
expectedAddressType: discovery.AddressTypeIPv4,
expectedEndpoint: discovery.Endpoint{
Addresses: []string{"1.2.3.4"},
@ -149,7 +208,25 @@ func TestReconcile1Pod(t *testing.T) {
},
},
"ipv4-labels": {
service: svcv4Labels,
service: svcv4Labels,
expectedEndpointPerSlice: map[discovery.AddressType][]discovery.Endpoint{
discovery.AddressTypeIPv4: {
{
Addresses: []string{"1.2.3.4"},
Conditions: discovery.EndpointConditions{Ready: utilpointer.BoolPtr(true)},
Topology: map[string]string{
"kubernetes.io/hostname": "node-1",
"topology.kubernetes.io/zone": "us-central1-a",
"topology.kubernetes.io/region": "us-central1",
},
TargetRef: &corev1.ObjectReference{
Kind: "Pod",
Namespace: namespace,
Name: "pod1",
},
},
},
},
expectedAddressType: discovery.AddressTypeIPv4,
expectedEndpoint: discovery.Endpoint{
Addresses: []string{"1.2.3.4"},
@ -173,7 +250,25 @@ func TestReconcile1Pod(t *testing.T) {
},
},
"ipv4-bad-labels": {
service: svcv4BadLabels,
service: svcv4BadLabels,
expectedEndpointPerSlice: map[discovery.AddressType][]discovery.Endpoint{
discovery.AddressTypeIPv4: {
{
Addresses: []string{"1.2.3.4"},
Conditions: discovery.EndpointConditions{Ready: utilpointer.BoolPtr(true)},
Topology: map[string]string{
"kubernetes.io/hostname": "node-1",
"topology.kubernetes.io/zone": "us-central1-a",
"topology.kubernetes.io/region": "us-central1",
},
TargetRef: &corev1.ObjectReference{
Kind: "Pod",
Namespace: namespace,
Name: "pod1",
},
},
},
},
expectedAddressType: discovery.AddressTypeIPv4,
expectedEndpoint: discovery.Endpoint{
Addresses: []string{"1.2.3.4"},
@ -195,21 +290,25 @@ func TestReconcile1Pod(t *testing.T) {
corev1.IsHeadlessService: "",
},
},
"ipv6": {
service: svcv6,
expectedAddressType: discovery.AddressTypeIPv6,
expectedEndpoint: discovery.Endpoint{
Addresses: []string{"1234::5678:0000:0000:9abc:def0"},
Conditions: discovery.EndpointConditions{Ready: utilpointer.BoolPtr(true)},
Topology: map[string]string{
"kubernetes.io/hostname": "node-1",
"topology.kubernetes.io/zone": "us-central1-a",
"topology.kubernetes.io/region": "us-central1",
},
TargetRef: &corev1.ObjectReference{
Kind: "Pod",
Namespace: namespace,
Name: "pod1",
service: svcv6,
expectedEndpointPerSlice: map[discovery.AddressType][]discovery.Endpoint{
discovery.AddressTypeIPv6: {
{
Addresses: []string{"1234::5678:0000:0000:9abc:def0"},
Conditions: discovery.EndpointConditions{Ready: utilpointer.BoolPtr(true)},
Topology: map[string]string{
"kubernetes.io/hostname": "node-1",
"topology.kubernetes.io/zone": "us-central1-a",
"topology.kubernetes.io/region": "us-central1",
},
TargetRef: &corev1.ObjectReference{
Kind: "Pod",
Namespace: namespace,
Name: "pod1",
},
},
},
},
expectedLabels: map[string]string{
@ -218,21 +317,67 @@ func TestReconcile1Pod(t *testing.T) {
corev1.IsHeadlessService: "",
},
},
"ipv6-clusterip": {
service: svcv6ClusterIP,
expectedAddressType: discovery.AddressTypeIPv6,
expectedEndpoint: discovery.Endpoint{
Addresses: []string{"1234::5678:0000:0000:9abc:def0"},
Conditions: discovery.EndpointConditions{Ready: utilpointer.BoolPtr(true)},
Topology: map[string]string{
"kubernetes.io/hostname": "node-1",
"topology.kubernetes.io/zone": "us-central1-a",
"topology.kubernetes.io/region": "us-central1",
service: svcv6ClusterIP,
expectedEndpointPerSlice: map[discovery.AddressType][]discovery.Endpoint{
discovery.AddressTypeIPv6: {
{
Addresses: []string{"1234::5678:0000:0000:9abc:def0"},
Conditions: discovery.EndpointConditions{Ready: utilpointer.BoolPtr(true)},
Topology: map[string]string{
"kubernetes.io/hostname": "node-1",
"topology.kubernetes.io/zone": "us-central1-a",
"topology.kubernetes.io/region": "us-central1",
},
TargetRef: &corev1.ObjectReference{
Kind: "Pod",
Namespace: namespace,
Name: "pod1",
},
},
},
TargetRef: &corev1.ObjectReference{
Kind: "Pod",
Namespace: namespace,
Name: "pod1",
},
expectedLabels: map[string]string{
discovery.LabelManagedBy: controllerName,
discovery.LabelServiceName: "foo",
},
},
"dualstack-service": {
service: dualStackSvc,
expectedEndpointPerSlice: map[discovery.AddressType][]discovery.Endpoint{
discovery.AddressTypeIPv6: {
{
Addresses: []string{"1234::5678:0000:0000:9abc:def0"},
Conditions: discovery.EndpointConditions{Ready: utilpointer.BoolPtr(true)},
Topology: map[string]string{
"kubernetes.io/hostname": "node-1",
"topology.kubernetes.io/zone": "us-central1-a",
"topology.kubernetes.io/region": "us-central1",
},
TargetRef: &corev1.ObjectReference{
Kind: "Pod",
Namespace: namespace,
Name: "pod1",
},
},
},
discovery.AddressTypeIPv4: {
{
Addresses: []string{"1.2.3.4"},
Conditions: discovery.EndpointConditions{Ready: utilpointer.BoolPtr(true)},
Topology: map[string]string{
"kubernetes.io/hostname": "node-1",
"topology.kubernetes.io/zone": "us-central1-a",
"topology.kubernetes.io/region": "us-central1",
},
TargetRef: &corev1.ObjectReference{
Kind: "Pod",
Namespace: namespace,
Name: "pod1",
},
},
},
},
expectedLabels: map[string]string{
@ -250,41 +395,61 @@ func TestReconcile1Pod(t *testing.T) {
r := newReconciler(client, []*corev1.Node{node1}, defaultMaxEndpointsPerSlice)
reconcileHelper(t, r, &testCase.service, []*corev1.Pod{pod1}, []*discovery.EndpointSlice{}, triggerTime)
if len(client.Actions()) != 1 {
t.Errorf("Expected 1 clientset action, got %d", len(client.Actions()))
if len(client.Actions()) != len(testCase.expectedEndpointPerSlice) {
t.Errorf("Expected %v clientset action, got %d", len(testCase.expectedEndpointPerSlice), len(client.Actions()))
}
slices := fetchEndpointSlices(t, client, namespace)
if len(slices) != 1 {
t.Fatalf("Expected 1 EndpointSlice, got %d", len(slices))
if len(slices) != len(testCase.expectedEndpointPerSlice) {
t.Fatalf("Expected %v EndpointSlice, got %d", len(testCase.expectedEndpointPerSlice), len(slices))
}
slice := slices[0]
if !strings.HasPrefix(slice.Name, testCase.service.Name) {
t.Errorf("Expected EndpointSlice name to start with %s, got %s", testCase.service.Name, slice.Name)
for _, slice := range slices {
if !strings.HasPrefix(slice.Name, testCase.service.Name) {
t.Fatalf("Expected EndpointSlice name to start with %s, got %s", testCase.service.Name, slice.Name)
}
if !reflect.DeepEqual(testCase.expectedLabels, slice.Labels) {
t.Errorf("Expected EndpointSlice to have labels: %v , got %v", testCase.expectedLabels, slice.Labels)
}
if slice.Labels[discovery.LabelServiceName] != testCase.service.Name {
t.Fatalf("Expected EndpointSlice to have label set with %s value, got %s", testCase.service.Name, slice.Labels[discovery.LabelServiceName])
}
if slice.Annotations[corev1.EndpointsLastChangeTriggerTime] != triggerTime.Format(time.RFC3339Nano) {
t.Fatalf("Expected EndpointSlice trigger time annotation to be %s, got %s", triggerTime.Format(time.RFC3339Nano), slice.Annotations[corev1.EndpointsLastChangeTriggerTime])
}
// validate that this slice has address type matching expected
expectedEndPointList := testCase.expectedEndpointPerSlice[slice.AddressType]
if expectedEndPointList == nil {
t.Fatalf("address type %v is not expected", slice.AddressType)
}
if len(slice.Endpoints) != len(expectedEndPointList) {
t.Fatalf("Expected %v Endpoint, got %d", len(expectedEndPointList), len(slice.Endpoints))
}
// test is limited to *ONE* endpoint
endpoint := slice.Endpoints[0]
if !reflect.DeepEqual(endpoint, expectedEndPointList[0]) {
t.Fatalf("Expected endpoint: %+v, got: %+v", expectedEndPointList[0], endpoint)
}
expectTrackedResourceVersion(t, r.endpointSliceTracker, &slice, "100")
expectMetrics(t,
expectedMetrics{
desiredSlices: 1,
actualSlices: 1,
desiredEndpoints: 1,
addedPerSync: len(testCase.expectedEndpointPerSlice),
removedPerSync: 0,
numCreated: len(testCase.expectedEndpointPerSlice),
numUpdated: 0,
numDeleted: 0})
}
if !reflect.DeepEqual(testCase.expectedLabels, slice.Labels) {
t.Errorf("Expected EndpointSlice to have labels: %v , got %v", testCase.expectedLabels, slice.Labels)
}
if slice.Annotations[corev1.EndpointsLastChangeTriggerTime] != triggerTime.Format(time.RFC3339Nano) {
t.Errorf("Expected EndpointSlice trigger time annotation to be %s, got %s", triggerTime.Format(time.RFC3339Nano), slice.Annotations[corev1.EndpointsLastChangeTriggerTime])
}
if len(slice.Endpoints) != 1 {
t.Fatalf("Expected 1 Endpoint, got %d", len(slice.Endpoints))
}
endpoint := slice.Endpoints[0]
if !reflect.DeepEqual(endpoint, testCase.expectedEndpoint) {
t.Errorf("Expected endpoint: %+v, got: %+v", testCase.expectedEndpoint, endpoint)
}
expectTrackedResourceVersion(t, r.endpointSliceTracker, &slice, "100")
expectMetrics(t, expectedMetrics{desiredSlices: 1, actualSlices: 1, desiredEndpoints: 1, addedPerSync: 1, removedPerSync: 0, numCreated: 1, numUpdated: 0, numDeleted: 0})
})
}
}
@ -404,13 +569,13 @@ func TestReconcileEndpointSlicesSomePreexisting(t *testing.T) {
// have approximately 1/4 in first slice
endpointSlice1 := newEmptyEndpointSlice(1, namespace, endpointMeta, svc)
for i := 1; i < len(pods)-4; i += 4 {
endpointSlice1.Endpoints = append(endpointSlice1.Endpoints, podToEndpoint(pods[i], &corev1.Node{}, &svc))
endpointSlice1.Endpoints = append(endpointSlice1.Endpoints, podToEndpoint(pods[i], &corev1.Node{}, &svc, discovery.AddressTypeIPv4))
}
// have approximately 1/4 in second slice
endpointSlice2 := newEmptyEndpointSlice(2, namespace, endpointMeta, svc)
for i := 3; i < len(pods)-4; i += 4 {
endpointSlice2.Endpoints = append(endpointSlice2.Endpoints, podToEndpoint(pods[i], &corev1.Node{}, &svc))
endpointSlice2.Endpoints = append(endpointSlice2.Endpoints, podToEndpoint(pods[i], &corev1.Node{}, &svc, discovery.AddressTypeIPv4))
}
existingSlices := []*discovery.EndpointSlice{endpointSlice1, endpointSlice2}
@ -460,13 +625,13 @@ func TestReconcileEndpointSlicesSomePreexistingWorseAllocation(t *testing.T) {
// have approximately 1/4 in first slice
endpointSlice1 := newEmptyEndpointSlice(1, namespace, endpointMeta, svc)
for i := 1; i < len(pods)-4; i += 4 {
endpointSlice1.Endpoints = append(endpointSlice1.Endpoints, podToEndpoint(pods[i], &corev1.Node{}, &svc))
endpointSlice1.Endpoints = append(endpointSlice1.Endpoints, podToEndpoint(pods[i], &corev1.Node{}, &svc, discovery.AddressTypeIPv4))
}
// have approximately 1/4 in second slice
endpointSlice2 := newEmptyEndpointSlice(2, namespace, endpointMeta, svc)
for i := 3; i < len(pods)-4; i += 4 {
endpointSlice2.Endpoints = append(endpointSlice2.Endpoints, podToEndpoint(pods[i], &corev1.Node{}, &svc))
endpointSlice2.Endpoints = append(endpointSlice2.Endpoints, podToEndpoint(pods[i], &corev1.Node{}, &svc, discovery.AddressTypeIPv4))
}
existingSlices := []*discovery.EndpointSlice{endpointSlice1, endpointSlice2}
@ -621,7 +786,7 @@ func TestReconcileEndpointSlicesRecycling(t *testing.T) {
if i%30 == 0 {
existingSlices = append(existingSlices, newEmptyEndpointSlice(sliceNum, namespace, endpointMeta, svc))
}
existingSlices[sliceNum].Endpoints = append(existingSlices[sliceNum].Endpoints, podToEndpoint(pod, &corev1.Node{}, &svc))
existingSlices[sliceNum].Endpoints = append(existingSlices[sliceNum].Endpoints, podToEndpoint(pod, &corev1.Node{}, &svc, discovery.AddressTypeIPv4))
}
cmc := newCacheMutationCheck(existingSlices)
@ -663,7 +828,7 @@ func TestReconcileEndpointSlicesUpdatePacking(t *testing.T) {
slice1 := newEmptyEndpointSlice(1, namespace, endpointMeta, svc)
for i := 0; i < 80; i++ {
pod := newPod(i, namespace, true, 1)
slice1.Endpoints = append(slice1.Endpoints, podToEndpoint(pod, &corev1.Node{}, &svc))
slice1.Endpoints = append(slice1.Endpoints, podToEndpoint(pod, &corev1.Node{}, &svc, discovery.AddressTypeIPv4))
pods = append(pods, pod)
}
existingSlices = append(existingSlices, slice1)
@ -671,7 +836,7 @@ func TestReconcileEndpointSlicesUpdatePacking(t *testing.T) {
slice2 := newEmptyEndpointSlice(2, namespace, endpointMeta, svc)
for i := 100; i < 120; i++ {
pod := newPod(i, namespace, true, 1)
slice2.Endpoints = append(slice2.Endpoints, podToEndpoint(pod, &corev1.Node{}, &svc))
slice2.Endpoints = append(slice2.Endpoints, podToEndpoint(pod, &corev1.Node{}, &svc, discovery.AddressTypeIPv4))
pods = append(pods, pod)
}
existingSlices = append(existingSlices, slice2)
@ -724,7 +889,7 @@ func TestReconcileEndpointSlicesReplaceDeprecated(t *testing.T) {
slice1 := newEmptyEndpointSlice(1, namespace, endpointMeta, svc)
for i := 0; i < 80; i++ {
pod := newPod(i, namespace, true, 1)
slice1.Endpoints = append(slice1.Endpoints, podToEndpoint(pod, &corev1.Node{}, &corev1.Service{Spec: corev1.ServiceSpec{}}))
slice1.Endpoints = append(slice1.Endpoints, podToEndpoint(pod, &corev1.Node{}, &corev1.Service{Spec: corev1.ServiceSpec{}}, discovery.AddressTypeIPv4))
pods = append(pods, pod)
}
existingSlices = append(existingSlices, slice1)
@ -732,7 +897,7 @@ func TestReconcileEndpointSlicesReplaceDeprecated(t *testing.T) {
slice2 := newEmptyEndpointSlice(2, namespace, endpointMeta, svc)
for i := 100; i < 150; i++ {
pod := newPod(i, namespace, true, 1)
slice2.Endpoints = append(slice2.Endpoints, podToEndpoint(pod, &corev1.Node{}, &corev1.Service{Spec: corev1.ServiceSpec{}}))
slice2.Endpoints = append(slice2.Endpoints, podToEndpoint(pod, &corev1.Node{}, &corev1.Service{Spec: corev1.ServiceSpec{}}, discovery.AddressTypeIPv4))
pods = append(pods, pod)
}
existingSlices = append(existingSlices, slice2)
@ -791,7 +956,7 @@ func TestReconcileEndpointSlicesRecreation(t *testing.T) {
slice := newEmptyEndpointSlice(1, namespace, endpointMeta, svc)
pod := newPod(1, namespace, true, 1)
slice.Endpoints = append(slice.Endpoints, podToEndpoint(pod, &corev1.Node{}, &corev1.Service{Spec: corev1.ServiceSpec{}}))
slice.Endpoints = append(slice.Endpoints, podToEndpoint(pod, &corev1.Node{}, &corev1.Service{Spec: corev1.ServiceSpec{}}, discovery.AddressTypeIPv4))
if !tc.ownedByService {
slice.OwnerReferences[0].UID = "different"
@ -848,7 +1013,8 @@ func TestReconcileEndpointSlicesNamedPorts(t *testing.T) {
TargetPort: portNameIntStr,
Protocol: corev1.ProtocolTCP,
}},
Selector: map[string]string{"foo": "bar"},
Selector: map[string]string{"foo": "bar"},
IPFamilies: []corev1.IPFamily{corev1.IPv4Protocol},
},
}
@ -1221,6 +1387,11 @@ func expectUnorderedSlicesWithTopLevelAttrs(t *testing.T, endpointSlices []disco
func expectActions(t *testing.T, actions []k8stesting.Action, num int, verb, resource string) {
t.Helper()
// if actions are less the below logic will panic
if num > len(actions) {
t.Fatalf("len of actions %v is unexpected. Expected to be at least %v", len(actions), num+1)
}
for i := 0; i < num; i++ {
relativePos := len(actions) - i - 1
assert.Equal(t, verb, actions[relativePos].GetVerb(), "Expected action -%d verb to be %s", i, verb)

73
pkg/controller/endpointslice/utils.go
View File

@ -37,8 +37,8 @@ import (
utilnet "k8s.io/utils/net"
)
// podToEndpoint returns an Endpoint object generated from pod, node, and service.
func podToEndpoint(pod *corev1.Pod, node *corev1.Node, service *corev1.Service) discovery.Endpoint {
// podToEndpoint returns an Endpoint object generated from a Pod, a Node, and a Service for a particular addressType.
func podToEndpoint(pod *corev1.Pod, node *corev1.Node, service *corev1.Service, addressType discovery.AddressType) discovery.Endpoint {
// Build out topology information. This is currently limited to hostname,
// zone, and region, but this will be expanded in the future.
topology := map[string]string{}
@ -62,7 +62,7 @@ func podToEndpoint(pod *corev1.Pod, node *corev1.Node, service *corev1.Service)
ready := service.Spec.PublishNotReadyAddresses || podutil.IsPodReady(pod)
ep := discovery.Endpoint{
Addresses: getEndpointAddresses(pod.Status, service),
Addresses: getEndpointAddresses(pod.Status, service, addressType),
Conditions: discovery.EndpointConditions{
Ready: &ready,
},
@ -117,12 +117,16 @@ func getEndpointPorts(service *corev1.Service, pod *corev1.Pod) []discovery.Endp
}
// getEndpointAddresses returns a list of addresses generated from a pod status.
func getEndpointAddresses(podStatus corev1.PodStatus, service *corev1.Service) []string {
func getEndpointAddresses(podStatus corev1.PodStatus, service *corev1.Service, addressType discovery.AddressType) []string {
addresses := []string{}
for _, podIP := range podStatus.PodIPs {
isIPv6PodIP := utilnet.IsIPv6String(podIP.IP)
if isIPv6PodIP == endpointutil.IsIPv6Service(service) {
if isIPv6PodIP && addressType == discovery.AddressTypeIPv6 {
addresses = append(addresses, podIP.IP)
}
if !isIPv6PodIP && addressType == discovery.AddressTypeIPv4 {
addresses = append(addresses, podIP.IP)
}
}
@ -346,3 +350,62 @@ func (sl endpointSliceEndpointLen) Swap(i, j int) { sl[i], sl[j] = sl[j], sl[i]
func (sl endpointSliceEndpointLen) Less(i, j int) bool {
return len(sl[i].Endpoints) > len(sl[j].Endpoints)
}
// returns a map of address types used by a service
func getAddressTypesForService(service *corev1.Service) map[discovery.AddressType]struct{} {
serviceSupportedAddresses := make(map[discovery.AddressType]struct{})
// TODO: (khenidak) when address types are removed in favor of
// v1.IPFamily this will need to be removed, and work directly with
// v1.IPFamily types
// IMPORTANT: we assume that IP of (discovery.AddressType enum) is never in use
// as it gets deprecated
for _, family := range service.Spec.IPFamilies {
if family == corev1.IPv4Protocol {
serviceSupportedAddresses[discovery.AddressTypeIPv4] = struct{}{}
}
if family == corev1.IPv6Protocol {
serviceSupportedAddresses[discovery.AddressTypeIPv6] = struct{}{}
}
}
if len(serviceSupportedAddresses) > 0 {
return serviceSupportedAddresses // we have found families for this service
}
// TODO (khenidak) remove when (1) dual stack becomes
// enabled by default (2) v1.19 falls off supported versions
// Why do we need this:
// a cluster being upgraded to the new apis
// will have service.spec.IPFamilies: nil
// if the controller manager connected to old api
// server. This will have the nasty side effect of
// removing all slices already created for this service.
// this will disable all routing to service vip (ClusterIP)
// this ensures that this does not happen. Same for headless services
// we assume it is dual stack, until they get defaulted by *new* api-server
// this ensures that traffic is not disrupted until then. But *may*
// include undesired families for headless services until then.
if len(service.Spec.ClusterIP) > 0 && service.Spec.ClusterIP != corev1.ClusterIPNone { // headfull
addrType := discovery.AddressTypeIPv4
if utilnet.IsIPv6String(service.Spec.ClusterIP) {
addrType = discovery.AddressTypeIPv6
}
serviceSupportedAddresses[addrType] = struct{}{}
klog.V(2).Infof("couldn't find ipfamilies for headless service: %v/%v. This could happen if controller manager is connected to an old apiserver that does not support ip families yet. EndpointSlices for this Service will use %s as the IP Family based on familyOf(ClusterIP:%v).", service.Namespace, service.Name, addrType, service.Spec.ClusterIP)
return serviceSupportedAddresses
}
// headless
// for now we assume two families. This should have minimal side effect
// if the service is headless with no selector, then this will remain the case
// if the service is headless with selector then chances are pods are still using single family
// since kubelet will need to restart in order to start patching pod status with multiple ips
serviceSupportedAddresses[discovery.AddressTypeIPv4] = struct{}{}
serviceSupportedAddresses[discovery.AddressTypeIPv6] = struct{}{}
klog.V(2).Infof("couldn't find ipfamilies for headless service: %v/%v likely because controller manager is likely connected to an old apiserver that does not support ip families yet. The service endpoint slice will use dual stack families until api-server default it correctly", service.Namespace, service.Name)
return serviceSupportedAddresses
}

137
pkg/controller/endpointslice/utils_test.go
View File

@ -385,7 +385,7 @@ func TestPodToEndpoint(t *testing.T) {
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
endpoint := podToEndpoint(testCase.pod, testCase.node, testCase.svc)
endpoint := podToEndpoint(testCase.pod, testCase.node, testCase.svc, discovery.AddressTypeIPv4)
if !reflect.DeepEqual(testCase.expectedEndpoint, endpoint) {
t.Errorf("Expected endpoint: %v, got: %v", testCase.expectedEndpoint, endpoint)
}
@ -889,7 +889,8 @@ func newServiceAndEndpointMeta(name, namespace string) (v1.Service, endpointMeta
Protocol: v1.ProtocolTCP,
Name: name,
}},
Selector: map[string]string{"foo": "bar"},
Selector: map[string]string{"foo": "bar"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
}
@ -918,3 +919,135 @@ func newEmptyEndpointSlice(n int, namespace string, endpointMeta endpointMeta, s
Endpoints: []discovery.Endpoint{},
}
}
func TestSupportedServiceAddressType(t *testing.T) {
testCases := []struct {
name string
service v1.Service
expectedAddressTypes []discovery.AddressType
}{
{
name: "v4 service with no ip families (cluster upgrade)",
expectedAddressTypes: []discovery.AddressType{discovery.AddressTypeIPv4},
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIP: "10.0.0.10",
IPFamilies: nil,
},
},
},
{
name: "v6 service with no ip families (cluster upgrade)",
expectedAddressTypes: []discovery.AddressType{discovery.AddressTypeIPv6},
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIP: "2000::1",
IPFamilies: nil,
},
},
},
{
name: "v4 service",
expectedAddressTypes: []discovery.AddressType{discovery.AddressTypeIPv4},
service: v1.Service{
Spec: v1.ServiceSpec{
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
},
{
name: "v6 services",
expectedAddressTypes: []discovery.AddressType{discovery.AddressTypeIPv6},
service: v1.Service{
Spec: v1.ServiceSpec{
IPFamilies: []v1.IPFamily{v1.IPv6Protocol},
},
},
},
{
name: "v4,v6 service",
expectedAddressTypes: []discovery.AddressType{discovery.AddressTypeIPv4, discovery.AddressTypeIPv6},
service: v1.Service{
Spec: v1.ServiceSpec{
IPFamilies: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
},
},
},
{
name: "v6,v4 service",
expectedAddressTypes: []discovery.AddressType{discovery.AddressTypeIPv6, discovery.AddressTypeIPv4},
service: v1.Service{
Spec: v1.ServiceSpec{
IPFamilies: []v1.IPFamily{v1.IPv6Protocol, v1.IPv4Protocol},
},
},
},
{
name: "headless with no selector and no families (old api-server)",
expectedAddressTypes: []discovery.AddressType{discovery.AddressTypeIPv6, discovery.AddressTypeIPv4},
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIP: v1.ClusterIPNone,
IPFamilies: nil,
},
},
},
{
name: "headless with selector and no families (old api-server)",
expectedAddressTypes: []discovery.AddressType{discovery.AddressTypeIPv6, discovery.AddressTypeIPv4},
service: v1.Service{
Spec: v1.ServiceSpec{
Selector: map[string]string{"foo": "bar"},
ClusterIP: v1.ClusterIPNone,
IPFamilies: nil,
},
},
},
{
name: "headless with no selector with families",
expectedAddressTypes: []discovery.AddressType{discovery.AddressTypeIPv4, discovery.AddressTypeIPv6},
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIP: v1.ClusterIPNone,
IPFamilies: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
},
},
},
{
name: "headless with selector with families",
expectedAddressTypes: []discovery.AddressType{discovery.AddressTypeIPv4, discovery.AddressTypeIPv6},
service: v1.Service{
Spec: v1.ServiceSpec{
Selector: map[string]string{"foo": "bar"},
ClusterIP: v1.ClusterIPNone,
IPFamilies: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
},
},
},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
addressTypes := getAddressTypesForService(&testCase.service)
if len(addressTypes) != len(testCase.expectedAddressTypes) {
t.Fatalf("expected count address types %v got %v", len(testCase.expectedAddressTypes), len(addressTypes))
}
// compare
for _, expectedAddressType := range testCase.expectedAddressTypes {
found := false
for key := range addressTypes {
if key == expectedAddressType {
found = true
break
}
}
if !found {
t.Fatalf("expected address type %v was not found in the result", expectedAddressType)
}
}
})
}
}

2
pkg/controller/util/endpoint/BUILD
View File

@ -10,7 +10,6 @@ go_library(
visibility = ["//visibility:public"],
deps = [
"//pkg/api/v1/pod:go_default_library",
"//pkg/apis/core/v1/helper:go_default_library",
"//pkg/controller:go_default_library",
"//pkg/util/hash:go_default_library",
"//staging/src/k8s.io/api/core/v1:go_default_library",
@ -20,7 +19,6 @@ go_library(
"//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library",
"//staging/src/k8s.io/client-go/listers/core/v1:go_default_library",
"//staging/src/k8s.io/client-go/tools/cache:go_default_library",
"//vendor/k8s.io/utils/net:go_default_library",
],
)

17
pkg/controller/util/endpoint/controller_utils.go
View File

@ -32,10 +32,8 @@ import (
v1listers "k8s.io/client-go/listers/core/v1"
"k8s.io/client-go/tools/cache"
podutil "k8s.io/kubernetes/pkg/api/v1/pod"
"k8s.io/kubernetes/pkg/apis/core/v1/helper"
"k8s.io/kubernetes/pkg/controller"
"k8s.io/kubernetes/pkg/util/hash"
utilnet "k8s.io/utils/net"
)
// ServiceSelectorCache is a cache of service selectors to avoid high CPU consumption caused by frequent calls to AsSelectorPreValidated (see #73527)
@ -277,18 +275,3 @@ func (sl portsInOrder) Less(i, j int) bool {
h2 := DeepHashObjectToString(sl[j])
return h1 < h2
}
// IsIPv6Service checks if svc should have IPv6 endpoints
func IsIPv6Service(svc *v1.Service) bool {
if helper.IsServiceIPSet(svc) {
return utilnet.IsIPv6String(svc.Spec.ClusterIP)
} else if svc.Spec.IPFamily != nil {
return *svc.Spec.IPFamily == v1.IPv6Protocol
} else {
// FIXME: for legacy headless Services with no IPFamily, the current
// thinking is that we should use the cluster default. Unfortunately
// the endpoint controller doesn't know the cluster default. For now,
// assume it's IPv4.
return false
}
}

2
pkg/controlplane/controller.go
View File

@ -292,6 +292,7 @@ func (c *Controller) CreateOrUpdateMasterServiceIfNeeded(serviceName string, ser
}
return nil
}
singleStack := corev1.IPFamilyPolicySingleStack
svc := &corev1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: serviceName,
@ -303,6 +304,7 @@ func (c *Controller) CreateOrUpdateMasterServiceIfNeeded(serviceName string, ser
// maintained by this code, not by the pod selector
Selector: nil,
ClusterIP: serviceIP.String(),
IPFamilyPolicy: &singleStack,
SessionAffinity: corev1.ServiceAffinityNone,
Type: serviceType,
},

2
pkg/controlplane/controller_test.go
View File

@ -592,6 +592,7 @@ func TestEmptySubsets(t *testing.T) {
}
func TestCreateOrUpdateMasterService(t *testing.T) {
singleStack := corev1.IPFamilyPolicySingleStack
ns := metav1.NamespaceDefault
om := func(name string) metav1.ObjectMeta {
return metav1.ObjectMeta{Namespace: ns, Name: name}
@ -619,6 +620,7 @@ func TestCreateOrUpdateMasterService(t *testing.T) {
},
Selector: nil,
ClusterIP: "1.2.3.4",
IPFamilyPolicy: &singleStack,
SessionAffinity: corev1.ServiceAffinityNone,
Type: corev1.ServiceTypeClusterIP,
},

19
pkg/kubelet/dockershim/network/kubenet/kubenet_linux.go
View File

@ -628,19 +628,14 @@ func (plugin *kubenetNetworkPlugin) getNetworkStatus(id kubecontainer.ContainerI
if !ok {
return nil
}
// sort making v4 first
// TODO: (khenidak) IPv6 beta stage.
// This - forced sort - could be avoided by checking which cidr that an IP belongs
// to, then placing the IP according to cidr index. But before doing that. Check how IP is collected
// across all of kubelet code (against cni and cri).
ips := make([]net.IP, 0)
if len(iplist) == 0 {
return nil
}
ips := make([]net.IP, 0, len(iplist))
for _, ip := range iplist {
isV6 := netutils.IsIPv6String(ip)
if !isV6 {
ips = append([]net.IP{net.ParseIP(ip)}, ips...)
} else {
ips = append(ips, net.ParseIP(ip))
}
ips = append(ips, net.ParseIP(ip))
}
return &network.PodNetworkStatus{

7
pkg/printers/internalversion/printers.go
View File

@ -1131,10 +1131,11 @@ func printService(obj *api.Service, options printers.GenerateOptions) ([]metav1.
Object: runtime.RawExtension{Object: obj},
}
svcType := obj.Spec.Type
internalIP := obj.Spec.ClusterIP
if len(internalIP) == 0 {
internalIP = "<none>"
internalIP := "<none>"
if len(obj.Spec.ClusterIPs) > 0 {
internalIP = obj.Spec.ClusterIPs[0]
}
externalIP := getServiceExternalIP(obj, options.Wide)
svcPorts := makePortString(obj.Spec.Ports)
if len(svcPorts) == 0 {

54
pkg/printers/internalversion/printers_test.go
View File

@ -1074,9 +1074,9 @@ func TestPrintServiceLoadBalancer(t *testing.T) {
service: api.Service{
ObjectMeta: metav1.ObjectMeta{Name: "service1"},
Spec: api.ServiceSpec{
ClusterIP: "1.2.3.4",
Type: "LoadBalancer",
Ports: []api.ServicePort{{Port: 80, Protocol: "TCP"}},
ClusterIPs: []string{"1.2.3.4"},
Type: "LoadBalancer",
Ports: []api.ServicePort{{Port: 80, Protocol: "TCP"}},
},
Status: api.ServiceStatus{
LoadBalancer: api.LoadBalancerStatus{
@ -1092,9 +1092,9 @@ func TestPrintServiceLoadBalancer(t *testing.T) {
service: api.Service{
ObjectMeta: metav1.ObjectMeta{Name: "service2"},
Spec: api.ServiceSpec{
ClusterIP: "1.3.4.5",
Type: "LoadBalancer",
Ports: []api.ServicePort{{Port: 80, Protocol: "TCP"}, {Port: 8090, Protocol: "UDP"}, {Port: 8000, Protocol: "TCP"}, {Port: 7777, Protocol: "SCTP"}},
ClusterIPs: []string{"1.3.4.5"},
Type: "LoadBalancer",
Ports: []api.ServicePort{{Port: 80, Protocol: "TCP"}, {Port: 8090, Protocol: "UDP"}, {Port: 8000, Protocol: "TCP"}, {Port: 7777, Protocol: "SCTP"}},
},
},
options: printers.GenerateOptions{},
@ -1106,9 +1106,9 @@ func TestPrintServiceLoadBalancer(t *testing.T) {
service: api.Service{
ObjectMeta: metav1.ObjectMeta{Name: "service3"},
Spec: api.ServiceSpec{
ClusterIP: "1.4.5.6",
Type: "LoadBalancer",
Ports: []api.ServicePort{{Port: 80, Protocol: "TCP"}, {Port: 8090, Protocol: "UDP"}, {Port: 8000, Protocol: "TCP"}},
ClusterIPs: []string{"1.4.5.6"},
Type: "LoadBalancer",
Ports: []api.ServicePort{{Port: 80, Protocol: "TCP"}, {Port: 8090, Protocol: "UDP"}, {Port: 8000, Protocol: "TCP"}},
},
Status: api.ServiceStatus{
LoadBalancer: api.LoadBalancerStatus{
@ -1124,9 +1124,9 @@ func TestPrintServiceLoadBalancer(t *testing.T) {
service: api.Service{
ObjectMeta: metav1.ObjectMeta{Name: "service4"},
Spec: api.ServiceSpec{
ClusterIP: "1.5.6.7",
Type: "LoadBalancer",
Ports: []api.ServicePort{{Port: 80, Protocol: "TCP"}, {Port: 8090, Protocol: "UDP"}, {Port: 8000, Protocol: "TCP"}},
ClusterIPs: []string{"1.5.6.7"},
Type: "LoadBalancer",
Ports: []api.ServicePort{{Port: 80, Protocol: "TCP"}, {Port: 8090, Protocol: "UDP"}, {Port: 8000, Protocol: "TCP"}},
},
Status: api.ServiceStatus{
LoadBalancer: api.LoadBalancerStatus{
@ -1142,10 +1142,10 @@ func TestPrintServiceLoadBalancer(t *testing.T) {
service: api.Service{
ObjectMeta: metav1.ObjectMeta{Name: "service4"},
Spec: api.ServiceSpec{
ClusterIP: "1.5.6.7",
Type: "LoadBalancer",
Ports: []api.ServicePort{{Port: 80, Protocol: "TCP"}, {Port: 8090, Protocol: "UDP"}, {Port: 8000, Protocol: "TCP"}},
Selector: map[string]string{"foo": "bar"},
ClusterIPs: []string{"1.5.6.7"},
Type: "LoadBalancer",
Ports: []api.ServicePort{{Port: 80, Protocol: "TCP"}, {Port: 8090, Protocol: "UDP"}, {Port: 8000, Protocol: "TCP"}},
Selector: map[string]string{"foo": "bar"},
},
Status: api.ServiceStatus{
LoadBalancer: api.LoadBalancerStatus{
@ -3083,8 +3083,8 @@ func TestPrintService(t *testing.T) {
Port: 2233,
},
},
ClusterIP: "10.9.8.7",
Selector: map[string]string{"foo": "bar"}, // Does NOT get printed.
ClusterIPs: []string{"10.9.8.7"},
Selector: map[string]string{"foo": "bar"}, // Does NOT get printed.
},
},
options: printers.GenerateOptions{},
@ -3103,8 +3103,8 @@ func TestPrintService(t *testing.T) {
Port: 2233,
},
},
ClusterIP: "10.9.8.7",
Selector: map[string]string{"foo": "bar"},
ClusterIPs: []string{"10.9.8.7"},
Selector: map[string]string{"foo": "bar"},
},
},
options: printers.GenerateOptions{Wide: true},
@ -3124,7 +3124,7 @@ func TestPrintService(t *testing.T) {
NodePort: 9999,
},
},
ClusterIP: "10.9.8.7",
ClusterIPs: []string{"10.9.8.7"},
},
},
options: printers.GenerateOptions{},
@ -3143,7 +3143,7 @@ func TestPrintService(t *testing.T) {
Port: 8888,
},
},
ClusterIP: "10.9.8.7",
ClusterIPs: []string{"10.9.8.7"},
},
},
options: printers.GenerateOptions{},
@ -3162,7 +3162,7 @@ func TestPrintService(t *testing.T) {
Port: 8888,
},
},
ClusterIP: "10.9.8.7",
ClusterIPs: []string{"10.9.8.7"},
ExternalIPs: singleExternalIP,
},
},
@ -3182,7 +3182,7 @@ func TestPrintService(t *testing.T) {
Port: 8888,
},
},
ClusterIP: "10.9.8.7",
ClusterIPs: []string{"10.9.8.7"},
ExternalIPs: singleExternalIP,
},
Status: api.ServiceStatus{
@ -3212,7 +3212,7 @@ func TestPrintService(t *testing.T) {
Port: 8888,
},
},
ClusterIP: "10.9.8.7",
ClusterIPs: []string{"10.9.8.7"},
ExternalIPs: mulExternalIP,
},
Status: api.ServiceStatus{
@ -3276,7 +3276,7 @@ func TestPrintServiceList(t *testing.T) {
Port: 2233,
},
},
ClusterIP: "10.9.8.7",
ClusterIPs: []string{"10.9.8.7"},
},
},
{
@ -3289,7 +3289,7 @@ func TestPrintServiceList(t *testing.T) {
Port: 5566,
},
},
ClusterIP: "1.2.3.4",
ClusterIPs: []string{"1.2.3.4"},
},
},
},

12
pkg/proxy/apis/config/validation/validation.go
View File

@ -75,10 +75,18 @@ func Validate(config *kubeproxyconfig.KubeProxyConfiguration) field.ErrorList {
}
allErrs = append(allErrs, validateHostPort(config.MetricsBindAddress, newPath.Child("MetricsBindAddress"))...)
dualStackEnabled := effectiveFeatures.Enabled(kubefeatures.IPv6DualStack)
endpointSliceEnabled := effectiveFeatures.Enabled(kubefeatures.EndpointSlice)
// dual stack has strong dependency on endpoint slice since
// endpoint slice controller is the only capabable of producing
// slices for *all* clusterIPs
if dualStackEnabled && !endpointSliceEnabled {
allErrs = append(allErrs, field.Invalid(newPath.Child("FeatureGates"), config.FeatureGates, "EndpointSlice feature flag must be turned on when turning on DualStack"))
}
if config.ClusterCIDR != "" {
cidrs := strings.Split(config.ClusterCIDR, ",")
dualStackEnabled := effectiveFeatures.Enabled(kubefeatures.IPv6DualStack)
switch {
// if DualStack only valid one cidr or two cidrs with one of each IP family
case dualStackEnabled && len(cidrs) > 2:

35
pkg/proxy/apis/config/validation/validation_test.go
View File

@ -124,7 +124,7 @@ func TestValidateKubeProxyConfiguration(t *testing.T) {
BindAddress: "10.10.12.11",
HealthzBindAddress: "0.0.0.0:12345",
MetricsBindAddress: "127.0.0.1:10249",
FeatureGates: map[string]bool{"IPv6DualStack": true},
FeatureGates: map[string]bool{"IPv6DualStack": true, "EndpointSlice": true},
ClusterCIDR: "192.168.59.0/24",
UDPIdleTimeout: metav1.Duration{Duration: 1 * time.Second},
ConfigSyncPeriod: metav1.Duration{Duration: 1 * time.Second},
@ -290,7 +290,7 @@ func TestValidateKubeProxyConfiguration(t *testing.T) {
HealthzBindAddress: "0.0.0.0:12345",
MetricsBindAddress: "127.0.0.1:10249",
// DualStack ClusterCIDR without feature flag enabled
FeatureGates: map[string]bool{"IPv6DualStack": false},
FeatureGates: map[string]bool{"IPv6DualStack": false, "EndpointSlice": false},
ClusterCIDR: "192.168.59.0/24,fd00:192:168::/64",
UDPIdleTimeout: metav1.Duration{Duration: 1 * time.Second},
ConfigSyncPeriod: metav1.Duration{Duration: 1 * time.Second},
@ -308,13 +308,38 @@ func TestValidateKubeProxyConfiguration(t *testing.T) {
},
msg: "only one CIDR allowed (e.g. 10.100.0.0/16 or fde4:8dba:82e1::/48)",
},
{
config: kubeproxyconfig.KubeProxyConfiguration{
BindAddress: "10.10.12.11",
HealthzBindAddress: "0.0.0.0:12345",
MetricsBindAddress: "127.0.0.1:10249",
// DualStack ClusterCIDR with feature flag enabled but EndpointSlice is not enabled
FeatureGates: map[string]bool{"IPv6DualStack": true, "EndpointSlice": false},
ClusterCIDR: "192.168.59.0/24,fd00:192:168::/64",
UDPIdleTimeout: metav1.Duration{Duration: 1 * time.Second},
ConfigSyncPeriod: metav1.Duration{Duration: 1 * time.Second},
IPTables: kubeproxyconfig.KubeProxyIPTablesConfiguration{
MasqueradeAll: true,
SyncPeriod: metav1.Duration{Duration: 5 * time.Second},
MinSyncPeriod: metav1.Duration{Duration: 2 * time.Second},
},
Conntrack: kubeproxyconfig.KubeProxyConntrackConfiguration{
MaxPerCore: pointer.Int32Ptr(1),
Min: pointer.Int32Ptr(1),
TCPEstablishedTimeout: &metav1.Duration{Duration: 5 * time.Second},
TCPCloseWaitTimeout: &metav1.Duration{Duration: 5 * time.Second},
},
},
msg: "EndpointSlice feature flag must be turned on",
},
{
config: kubeproxyconfig.KubeProxyConfiguration{
BindAddress: "10.10.12.11",
HealthzBindAddress: "0.0.0.0:12345",
MetricsBindAddress: "127.0.0.1:10249",
// DualStack with multiple CIDRs but only one IP family
FeatureGates: map[string]bool{"IPv6DualStack": true},
FeatureGates: map[string]bool{"IPv6DualStack": true, "EndpointSlice": true},
ClusterCIDR: "192.168.59.0/24,10.0.0.0/16",
UDPIdleTimeout: metav1.Duration{Duration: 1 * time.Second},
ConfigSyncPeriod: metav1.Duration{Duration: 1 * time.Second},
@ -338,7 +363,7 @@ func TestValidateKubeProxyConfiguration(t *testing.T) {
HealthzBindAddress: "0.0.0.0:12345",
MetricsBindAddress: "127.0.0.1:10249",
// DualStack with an invalid subnet
FeatureGates: map[string]bool{"IPv6DualStack": true},
FeatureGates: map[string]bool{"IPv6DualStack": true, "EndpointSlice": true},
ClusterCIDR: "192.168.59.0/24,fd00:192:168::/64,a.b.c.d/f",
UDPIdleTimeout: metav1.Duration{Duration: 1 * time.Second},
ConfigSyncPeriod: metav1.Duration{Duration: 1 * time.Second},
@ -361,7 +386,7 @@ func TestValidateKubeProxyConfiguration(t *testing.T) {
BindAddress: "10.10.12.11",
HealthzBindAddress: "0.0.0.0:12345",
MetricsBindAddress: "127.0.0.1:10249",
FeatureGates: map[string]bool{"IPv6DualStack": true},
FeatureGates: map[string]bool{"IPv6DualStack": true, "EndpointSlice": true},
ClusterCIDR: "192.168.59.0/24,fd00:192:168::/64,10.0.0.0/16",
UDPIdleTimeout: metav1.Duration{Duration: 1 * time.Second},
ConfigSyncPeriod: metav1.Duration{Duration: 1 * time.Second},

16
pkg/proxy/endpoints.go
View File

@ -112,27 +112,27 @@ type EndpointChangeTracker struct {
processEndpointsMapChange processEndpointsMapChangeFunc
// endpointSliceCache holds a simplified version of endpoint slices.
endpointSliceCache *EndpointSliceCache
// isIPv6Mode indicates if change tracker is under IPv6/IPv4 mode. Nil means not applicable.
isIPv6Mode *bool
recorder record.EventRecorder
// ipfamily identify the ip family on which the tracker is operating on
ipFamily v1.IPFamily
recorder record.EventRecorder
// Map from the Endpoints namespaced-name to the times of the triggers that caused the endpoints
// object to change. Used to calculate the network-programming-latency.
lastChangeTriggerTimes map[types.NamespacedName][]time.Time
}
// NewEndpointChangeTracker initializes an EndpointsChangeMap
func NewEndpointChangeTracker(hostname string, makeEndpointInfo makeEndpointFunc, isIPv6Mode *bool, recorder record.EventRecorder, endpointSlicesEnabled bool, processEndpointsMapChange processEndpointsMapChangeFunc) *EndpointChangeTracker {
func NewEndpointChangeTracker(hostname string, makeEndpointInfo makeEndpointFunc, ipFamily v1.IPFamily, recorder record.EventRecorder, endpointSlicesEnabled bool, processEndpointsMapChange processEndpointsMapChangeFunc) *EndpointChangeTracker {
ect := &EndpointChangeTracker{
hostname: hostname,
items: make(map[types.NamespacedName]*endpointsChange),
makeEndpointInfo: makeEndpointInfo,
isIPv6Mode: isIPv6Mode,
ipFamily: ipFamily,
recorder: recorder,
lastChangeTriggerTimes: make(map[types.NamespacedName][]time.Time),
processEndpointsMapChange: processEndpointsMapChange,
}
if endpointSlicesEnabled {
ect.endpointSliceCache = NewEndpointSliceCache(hostname, isIPv6Mode, recorder, makeEndpointInfo)
ect.endpointSliceCache = NewEndpointSliceCache(hostname, ipFamily, recorder, makeEndpointInfo)
}
return ect
}
@ -374,14 +374,16 @@ func (ect *EndpointChangeTracker) endpointsToEndpointsMap(endpoints *v1.Endpoint
klog.Warningf("ignoring invalid endpoint port %s with empty host", port.Name)
continue
}
// Filter out the incorrect IP version case.
// Any endpoint port that contains incorrect IP version will be ignored.
if ect.isIPv6Mode != nil && utilnet.IsIPv6String(addr.IP) != *ect.isIPv6Mode {
if (ect.ipFamily == v1.IPv6Protocol) != utilnet.IsIPv6String(addr.IP) {
// Emit event on the corresponding service which had a different
// IP version than the endpoint.
utilproxy.LogAndEmitIncorrectIPVersionEvent(ect.recorder, "endpoints", addr.IP, endpoints.Namespace, endpoints.Name, "")
continue
}
isLocal := addr.NodeName != nil && *addr.NodeName == ect.hostname
baseEndpointInfo := newBaseEndpointInfo(addr.IP, int(port.Port), isLocal, nil)
if ect.makeEndpointInfo != nil {

128
pkg/proxy/endpoints_test.go
View File

@ -135,24 +135,24 @@ func makeTestEndpoints(namespace, name string, eptFunc func(*v1.Endpoints)) *v1.
// This is a coarse test, but it offers some modicum of confidence as the code is evolved.
func TestEndpointsToEndpointsMap(t *testing.T) {
epTracker := NewEndpointChangeTracker("test-hostname", nil, nil, nil, false, nil)
trueVal := true
falseVal := false
testCases := []struct {
desc string
newEndpoints *v1.Endpoints
expected map[ServicePortName][]*BaseEndpointInfo
isIPv6Mode *bool
ipFamily v1.IPFamily
}{
{
desc: "nothing",
desc: "nothing",
ipFamily: v1.IPv4Protocol,
newEndpoints: makeTestEndpoints("ns1", "ep1", func(ept *v1.Endpoints) {}),
expected: map[ServicePortName][]*BaseEndpointInfo{},
},
{
desc: "no changes, unnamed port",
desc: "no changes, unnamed port",
ipFamily: v1.IPv4Protocol,
newEndpoints: makeTestEndpoints("ns1", "ep1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{
{
@ -174,7 +174,9 @@ func TestEndpointsToEndpointsMap(t *testing.T) {
},
},
{
desc: "no changes, named port",
desc: "no changes, named port",
ipFamily: v1.IPv4Protocol,
newEndpoints: makeTestEndpoints("ns1", "ep1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{
{
@ -196,7 +198,9 @@ func TestEndpointsToEndpointsMap(t *testing.T) {
},
},
{
desc: "new port",
desc: "new port",
ipFamily: v1.IPv4Protocol,
newEndpoints: makeTestEndpoints("ns1", "ep1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{
{
@ -217,12 +221,16 @@ func TestEndpointsToEndpointsMap(t *testing.T) {
},
},
{
desc: "remove port",
desc: "remove port",
ipFamily: v1.IPv4Protocol,
newEndpoints: makeTestEndpoints("ns1", "ep1", func(ept *v1.Endpoints) {}),
expected: map[ServicePortName][]*BaseEndpointInfo{},
},
{
desc: "new IP and port",
desc: "new IP and port",
ipFamily: v1.IPv4Protocol,
newEndpoints: makeTestEndpoints("ns1", "ep1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{
{
@ -255,7 +263,9 @@ func TestEndpointsToEndpointsMap(t *testing.T) {
},
},
{
desc: "remove IP and port",
desc: "remove IP and port",
ipFamily: v1.IPv4Protocol,
newEndpoints: makeTestEndpoints("ns1", "ep1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{
{
@ -277,7 +287,9 @@ func TestEndpointsToEndpointsMap(t *testing.T) {
},
},
{
desc: "rename port",
desc: "rename port",
ipFamily: v1.IPv4Protocol,
newEndpoints: makeTestEndpoints("ns1", "ep1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{
{
@ -299,7 +311,9 @@ func TestEndpointsToEndpointsMap(t *testing.T) {
},
},
{
desc: "renumber port",
desc: "renumber port",
ipFamily: v1.IPv4Protocol,
newEndpoints: makeTestEndpoints("ns1", "ep1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{
{
@ -321,7 +335,9 @@ func TestEndpointsToEndpointsMap(t *testing.T) {
},
},
{
desc: "should omit IPv6 address in IPv4 mode",
desc: "should omit IPv6 address in IPv4 mode",
ipFamily: v1.IPv4Protocol,
newEndpoints: makeTestEndpoints("ns1", "ep1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{
{
@ -350,10 +366,11 @@ func TestEndpointsToEndpointsMap(t *testing.T) {
{Endpoint: "1.1.1.1:22", IsLocal: false},
},
},
isIPv6Mode: &falseVal,
},
{
desc: "should omit IPv4 address in IPv6 mode",
desc: "should omit IPv4 address in IPv6 mode",
ipFamily: v1.IPv6Protocol,
newEndpoints: makeTestEndpoints("ns1", "ep1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{
{
@ -382,30 +399,33 @@ func TestEndpointsToEndpointsMap(t *testing.T) {
{Endpoint: "[2001:db8:85a3:0:0:8a2e:370:7334]:22", IsLocal: false},
},
},
isIPv6Mode: &trueVal,
},
}
for _, tc := range testCases {
epTracker.isIPv6Mode = tc.isIPv6Mode
// outputs
newEndpoints := epTracker.endpointsToEndpointsMap(tc.newEndpoints)
t.Run(tc.desc, func(t *testing.T) {
if len(newEndpoints) != len(tc.expected) {
t.Errorf("[%s] expected %d new, got %d: %v", tc.desc, len(tc.expected), len(newEndpoints), spew.Sdump(newEndpoints))
}
for x := range tc.expected {
if len(newEndpoints[x]) != len(tc.expected[x]) {
t.Errorf("[%s] expected %d endpoints for %v, got %d", tc.desc, len(tc.expected[x]), x, len(newEndpoints[x]))
} else {
for i := range newEndpoints[x] {
ep := newEndpoints[x][i].(*BaseEndpointInfo)
if !(reflect.DeepEqual(*ep, *(tc.expected[x][i]))) {
t.Errorf("[%s] expected new[%v][%d] to be %v, got %v", tc.desc, x, i, tc.expected[x][i], *ep)
epTracker := NewEndpointChangeTracker("test-hostname", nil, tc.ipFamily, nil, false, nil)
// outputs
newEndpoints := epTracker.endpointsToEndpointsMap(tc.newEndpoints)
if len(newEndpoints) != len(tc.expected) {
t.Fatalf("[%s] expected %d new, got %d: %v", tc.desc, len(tc.expected), len(newEndpoints), spew.Sdump(newEndpoints))
}
for x := range tc.expected {
if len(newEndpoints[x]) != len(tc.expected[x]) {
t.Fatalf("[%s] expected %d endpoints for %v, got %d", tc.desc, len(tc.expected[x]), x, len(newEndpoints[x]))
} else {
for i := range newEndpoints[x] {
ep := newEndpoints[x][i].(*BaseEndpointInfo)
if !(reflect.DeepEqual(*ep, *(tc.expected[x][i]))) {
t.Fatalf("[%s] expected new[%v][%d] to be %v, got %v", tc.desc, x, i, tc.expected[x][i], *ep)
}
}
}
}
}
})
}
}
@ -1249,7 +1269,7 @@ func TestUpdateEndpointsMap(t *testing.T) {
for tci, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fp := newFakeProxier()
fp := newFakeProxier(v1.IPv4Protocol)
fp.hostname = nodeName
// First check that after adding all previous versions of endpoints,
@ -1424,7 +1444,7 @@ func TestLastChangeTriggerTime(t *testing.T) {
}
for _, tc := range testCases {
fp := newFakeProxier()
fp := newFakeProxier(v1.IPv4Protocol)
tc.scenario(fp)
@ -1454,7 +1474,7 @@ func TestEndpointSliceUpdate(t *testing.T) {
// test starting from an empty state
"add a simple slice that doesn't already exist": {
startingSlices: []*discovery.EndpointSlice{},
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, v1.IPv4Protocol, nil, true, nil),
namespacedName: types.NamespacedName{Name: "svc1", Namespace: "ns1"},
paramEndpointSlice: generateEndpointSlice("svc1", "ns1", 1, 3, 999, []string{"host1", "host2"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
paramRemoveSlice: false,
@ -1477,7 +1497,7 @@ func TestEndpointSliceUpdate(t *testing.T) {
startingSlices: []*discovery.EndpointSlice{
generateEndpointSlice("svc1", "ns1", 1, 3, 999, []string{"host1", "host2"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
},
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, v1.IPv4Protocol, nil, true, nil),
namespacedName: types.NamespacedName{Name: "svc1", Namespace: "ns1"},
paramEndpointSlice: generateEndpointSlice("svc1", "ns1", 1, 3, 999, []string{"host1", "host2"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
paramRemoveSlice: false,
@ -1489,7 +1509,7 @@ func TestEndpointSliceUpdate(t *testing.T) {
startingSlices: []*discovery.EndpointSlice{
generateEndpointSlice("svc1", "ns1", 1, 3, 999, []string{"host1", "host2"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
},
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, v1.IPv4Protocol, nil, true, nil),
namespacedName: types.NamespacedName{Name: "svc1", Namespace: "ns1"},
paramEndpointSlice: fqdnSlice,
paramRemoveSlice: false,
@ -1502,7 +1522,7 @@ func TestEndpointSliceUpdate(t *testing.T) {
generateEndpointSlice("svc1", "ns1", 1, 3, 999, []string{"host1", "host2"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
generateEndpointSlice("svc1", "ns1", 2, 2, 999, []string{"host1", "host2"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
},
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, v1.IPv4Protocol, nil, true, nil),
namespacedName: types.NamespacedName{Name: "svc1", Namespace: "ns1"},
paramEndpointSlice: generateEndpointSlice("svc1", "ns1", 1, 5, 999, []string{"host1"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
paramRemoveSlice: false,
@ -1534,7 +1554,7 @@ func TestEndpointSliceUpdate(t *testing.T) {
generateEndpointSlice("svc1", "ns1", 1, 3, 999, []string{"host1", "host2"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
generateEndpointSlice("svc1", "ns1", 2, 2, 999, []string{"host1", "host2"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
},
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, v1.IPv4Protocol, nil, true, nil),
namespacedName: types.NamespacedName{Name: "svc1", Namespace: "ns1"},
paramEndpointSlice: generateEndpointSliceWithOffset("svc1", "ns1", 3, 1, 5, 999, []string{"host1"}, []*int32{utilpointer.Int32Ptr(80)}),
paramRemoveSlice: false,
@ -1564,7 +1584,7 @@ func TestEndpointSliceUpdate(t *testing.T) {
generateEndpointSlice("svc1", "ns1", 1, 3, 999, []string{"host1", "host2"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
generateEndpointSlice("svc1", "ns1", 2, 2, 999, []string{"host1", "host2"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
},
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, v1.IPv4Protocol, nil, true, nil),
namespacedName: types.NamespacedName{Name: "svc1", Namespace: "ns1"},
paramEndpointSlice: generateEndpointSlice("svc1", "ns1", 1, 5, 999, []string{"host1"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
paramRemoveSlice: true,
@ -1586,7 +1606,7 @@ func TestEndpointSliceUpdate(t *testing.T) {
generateEndpointSlice("svc1", "ns1", 1, 5, 999, []string{"host1"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
generateEndpointSlice("svc1", "ns1", 2, 2, 999, []string{"host1"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
},
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, v1.IPv4Protocol, nil, true, nil),
namespacedName: types.NamespacedName{Name: "svc1", Namespace: "ns1"},
paramEndpointSlice: generateEndpointSlice("svc1", "ns1", 3, 5, 999, []string{"host1"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
paramRemoveSlice: true,
@ -1598,7 +1618,7 @@ func TestEndpointSliceUpdate(t *testing.T) {
startingSlices: []*discovery.EndpointSlice{
generateEndpointSlice("svc1", "ns1", 1, 3, 999, []string{"host1", "host2"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
},
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, v1.IPv4Protocol, nil, true, nil),
namespacedName: types.NamespacedName{Name: "svc1", Namespace: "ns1"},
paramEndpointSlice: generateEndpointSlice("svc1", "ns1", 1, 3, 1, []string{"host1"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
paramRemoveSlice: false,
@ -1610,7 +1630,7 @@ func TestEndpointSliceUpdate(t *testing.T) {
startingSlices: []*discovery.EndpointSlice{
generateEndpointSlice("svc1", "ns1", 1, 2, 1, []string{"host1", "host2"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
},
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, v1.IPv4Protocol, nil, true, nil),
namespacedName: types.NamespacedName{Name: "svc1", Namespace: "ns1"},
paramEndpointSlice: generateEndpointSlice("svc1", "ns1", 1, 2, 999, []string{"host1"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
paramRemoveSlice: false,
@ -1632,7 +1652,7 @@ func TestEndpointSliceUpdate(t *testing.T) {
generateEndpointSlice("svc1", "ns1", 1, 3, 2, []string{"host1"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
generateEndpointSlice("svc1", "ns1", 2, 2, 2, []string{"host1"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
},
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("host1", nil, v1.IPv4Protocol, nil, true, nil),
namespacedName: types.NamespacedName{Name: "svc1", Namespace: "ns1"},
paramEndpointSlice: generateEndpointSlice("svc1", "ns1", 1, 3, 3, []string{"host1"}, []*int32{utilpointer.Int32Ptr(80), utilpointer.Int32Ptr(443)}),
paramRemoveSlice: false,
@ -1691,20 +1711,20 @@ func TestCheckoutChanges(t *testing.T) {
pendingSlices []*discovery.EndpointSlice
}{
"empty slices": {
endpointChangeTracker: NewEndpointChangeTracker("", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("", nil, v1.IPv4Protocol, nil, true, nil),
expectedChanges: []*endpointsChange{},
useEndpointSlices: true,
appliedSlices: []*discovery.EndpointSlice{},
pendingSlices: []*discovery.EndpointSlice{},
},
"without slices, empty items": {
endpointChangeTracker: NewEndpointChangeTracker("", nil, nil, nil, false, nil),
endpointChangeTracker: NewEndpointChangeTracker("", nil, v1.IPv4Protocol, nil, false, nil),
expectedChanges: []*endpointsChange{},
items: map[types.NamespacedName]*endpointsChange{},
useEndpointSlices: false,
},
"without slices, simple items": {
endpointChangeTracker: NewEndpointChangeTracker("", nil, nil, nil, false, nil),
endpointChangeTracker: NewEndpointChangeTracker("", nil, v1.IPv4Protocol, nil, false, nil),
expectedChanges: []*endpointsChange{{
previous: EndpointsMap{
svcPortName0: []Endpoint{newTestEp("10.0.1.1:80", ""), newTestEp("10.0.1.2:80", "")},
@ -1728,7 +1748,7 @@ func TestCheckoutChanges(t *testing.T) {
useEndpointSlices: false,
},
"adding initial slice": {
endpointChangeTracker: NewEndpointChangeTracker("", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("", nil, v1.IPv4Protocol, nil, true, nil),
expectedChanges: []*endpointsChange{{
previous: EndpointsMap{},
current: EndpointsMap{
@ -1742,7 +1762,7 @@ func TestCheckoutChanges(t *testing.T) {
},
},
"removing port in update": {
endpointChangeTracker: NewEndpointChangeTracker("", nil, nil, nil, true, nil),
endpointChangeTracker: NewEndpointChangeTracker("", nil, v1.IPv4Protocol, nil, true, nil),
expectedChanges: []*endpointsChange{{
previous: EndpointsMap{
svcPortName0: []Endpoint{newTestEp("10.0.1.1:80", "host1"), newTestEp("10.0.1.2:80", "host1")},
@ -1802,24 +1822,24 @@ func TestCheckoutChanges(t *testing.T) {
func compareEndpointsMapsStr(t *testing.T, newMap EndpointsMap, expected map[ServicePortName][]*BaseEndpointInfo) {
t.Helper()
if len(newMap) != len(expected) {
t.Errorf("expected %d results, got %d: %v", len(expected), len(newMap), newMap)
t.Fatalf("expected %d results, got %d: %v", len(expected), len(newMap), newMap)
}
endpointEqual := func(a, b *BaseEndpointInfo) bool {
return a.Endpoint == b.Endpoint && a.IsLocal == b.IsLocal
}
for x := range expected {
if len(newMap[x]) != len(expected[x]) {
t.Errorf("expected %d endpoints for %v, got %d", len(expected[x]), x, len(newMap[x]))
t.Logf("Endpoints %+v", newMap[x])
t.Fatalf("expected %d endpoints for %v, got %d", len(expected[x]), x, len(newMap[x]))
} else {
for i := range expected[x] {
newEp, ok := newMap[x][i].(*BaseEndpointInfo)
if !ok {
t.Errorf("Failed to cast endpointsInfo")
t.Fatalf("Failed to cast endpointsInfo")
continue
}
if !endpointEqual(newEp, expected[x][i]) {
t.Errorf("expected new[%v][%d] to be %v, got %v (IsLocal expected %v, got %v)", x, i, expected[x][i], newEp, expected[x][i].IsLocal, newEp.IsLocal)
t.Fatalf("expected new[%v][%d] to be %v, got %v (IsLocal expected %v, got %v)", x, i, expected[x][i], newEp, expected[x][i].IsLocal, newEp.IsLocal)
}
}
}

8
pkg/proxy/endpointslicecache.go
View File

@ -47,7 +47,7 @@ type EndpointSliceCache struct {
makeEndpointInfo makeEndpointFunc
hostname string
isIPv6Mode *bool
ipFamily v1.IPFamily
recorder record.EventRecorder
}
@ -84,14 +84,14 @@ type endpointInfo struct {
type spToEndpointMap map[ServicePortName]map[string]Endpoint
// NewEndpointSliceCache initializes an EndpointSliceCache.
func NewEndpointSliceCache(hostname string, isIPv6Mode *bool, recorder record.EventRecorder, makeEndpointInfo makeEndpointFunc) *EndpointSliceCache {
func NewEndpointSliceCache(hostname string, ipFamily v1.IPFamily, recorder record.EventRecorder, makeEndpointInfo makeEndpointFunc) *EndpointSliceCache {
if makeEndpointInfo == nil {
makeEndpointInfo = standardEndpointInfo
}
return &EndpointSliceCache{
trackerByServiceMap: map[types.NamespacedName]*endpointSliceTracker{},
hostname: hostname,
isIPv6Mode: isIPv6Mode,
ipFamily: ipFamily,
makeEndpointInfo: makeEndpointInfo,
recorder: recorder,
}
@ -248,7 +248,7 @@ func (cache *EndpointSliceCache) addEndpointsByIP(serviceNN types.NamespacedName
// Filter out the incorrect IP version case. Any endpoint port that
// contains incorrect IP version will be ignored.
if cache.isIPv6Mode != nil && utilnet.IsIPv6String(endpoint.Addresses[0]) != *cache.isIPv6Mode {
if (cache.ipFamily == v1.IPv6Protocol) != utilnet.IsIPv6String(endpoint.Addresses[0]) {
// Emit event on the corresponding service which had a different IP
// version than the endpoint.
utilproxy.LogAndEmitIncorrectIPVersionEvent(cache.recorder, "endpointslice", endpoint.Addresses[0], serviceNN.Namespace, serviceNN.Name, "")

18
pkg/proxy/endpointslicecache_test.go
View File

@ -151,7 +151,7 @@ func TestEndpointsMapFromESC(t *testing.T) {
for name, tc := range testCases {
t.Run(name, func(t *testing.T) {
esCache := NewEndpointSliceCache(tc.hostname, nil, nil, nil)
esCache := NewEndpointSliceCache(tc.hostname, v1.IPv4Protocol, nil, nil)
cmc := newCacheMutationCheck(tc.endpointSlices)
for _, endpointSlice := range tc.endpointSlices {
@ -189,7 +189,7 @@ func TestEndpointInfoByServicePort(t *testing.T) {
for name, tc := range testCases {
t.Run(name, func(t *testing.T) {
esCache := NewEndpointSliceCache(tc.hostname, nil, nil, nil)
esCache := NewEndpointSliceCache(tc.hostname, v1.IPv4Protocol, nil, nil)
for _, endpointSlice := range tc.endpointSlices {
esCache.updatePending(endpointSlice, false)
@ -225,7 +225,7 @@ func TestEsInfoChanged(t *testing.T) {
expectChanged bool
}{
"identical slices, ports only": {
cache: NewEndpointSliceCache("", nil, nil, nil),
cache: NewEndpointSliceCache("", v1.IPv4Protocol, nil, nil),
initialSlice: &discovery.EndpointSlice{
ObjectMeta: objMeta,
Ports: []discovery.EndpointPort{port80},
@ -237,7 +237,7 @@ func TestEsInfoChanged(t *testing.T) {
expectChanged: false,
},
"identical slices, ports out of order": {
cache: NewEndpointSliceCache("", nil, nil, nil),
cache: NewEndpointSliceCache("", v1.IPv4Protocol, nil, nil),
initialSlice: &discovery.EndpointSlice{
ObjectMeta: objMeta,
Ports: []discovery.EndpointPort{port443, port80},
@ -249,7 +249,7 @@ func TestEsInfoChanged(t *testing.T) {
expectChanged: false,
},
"port removed": {
cache: NewEndpointSliceCache("", nil, nil, nil),
cache: NewEndpointSliceCache("", v1.IPv4Protocol, nil, nil),
initialSlice: &discovery.EndpointSlice{
ObjectMeta: objMeta,
Ports: []discovery.EndpointPort{port443, port80},
@ -261,7 +261,7 @@ func TestEsInfoChanged(t *testing.T) {
expectChanged: true,
},
"port added": {
cache: NewEndpointSliceCache("", nil, nil, nil),
cache: NewEndpointSliceCache("", v1.IPv4Protocol, nil, nil),
initialSlice: &discovery.EndpointSlice{
ObjectMeta: objMeta,
Ports: []discovery.EndpointPort{port443},
@ -273,7 +273,7 @@ func TestEsInfoChanged(t *testing.T) {
expectChanged: true,
},
"identical with endpoints": {
cache: NewEndpointSliceCache("", nil, nil, nil),
cache: NewEndpointSliceCache("", v1.IPv4Protocol, nil, nil),
initialSlice: &discovery.EndpointSlice{
ObjectMeta: objMeta,
Ports: []discovery.EndpointPort{port443},
@ -287,7 +287,7 @@ func TestEsInfoChanged(t *testing.T) {
expectChanged: false,
},
"identical with endpoints out of order": {
cache: NewEndpointSliceCache("", nil, nil, nil),
cache: NewEndpointSliceCache("", v1.IPv4Protocol, nil, nil),
initialSlice: &discovery.EndpointSlice{
ObjectMeta: objMeta,
Ports: []discovery.EndpointPort{port443},
@ -301,7 +301,7 @@ func TestEsInfoChanged(t *testing.T) {
expectChanged: false,
},
"identical with endpoint added": {
cache: NewEndpointSliceCache("", nil, nil, nil),
cache: NewEndpointSliceCache("", v1.IPv4Protocol, nil, nil),
initialSlice: &discovery.EndpointSlice{
ObjectMeta: objMeta,
Ports: []discovery.EndpointPort{port443},

18
pkg/proxy/iptables/proxier.go
View File

@ -289,18 +289,23 @@ func NewProxier(ipt utiliptables.Interface,
serviceHealthServer := healthcheck.NewServiceHealthServer(hostname, recorder)
isIPv6 := ipt.IsIPv6()
ipFamily := v1.IPv4Protocol
if ipt.IsIPv6() {
ipFamily = v1.IPv6Protocol
}
var incorrectAddresses []string
nodePortAddresses, incorrectAddresses = utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, isIPv6)
nodePortAddresses, incorrectAddresses = utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, ipFamily)
if len(incorrectAddresses) > 0 {
klog.Warning("NodePortAddresses of wrong family; ", incorrectAddresses)
}
proxier := &Proxier{
portsMap: make(map[utilproxy.LocalPort]utilproxy.Closeable),
serviceMap: make(proxy.ServiceMap),
serviceChanges: proxy.NewServiceChangeTracker(newServiceInfo, &isIPv6, recorder, nil),
serviceChanges: proxy.NewServiceChangeTracker(newServiceInfo, ipFamily, recorder, nil),
endpointsMap: make(proxy.EndpointsMap),
endpointsChanges: proxy.NewEndpointChangeTracker(hostname, newEndpointInfo, &isIPv6, recorder, endpointSlicesEnabled, nil),
endpointsChanges: proxy.NewEndpointChangeTracker(hostname, newEndpointInfo, ipFamily, recorder, endpointSlicesEnabled, nil),
syncPeriod: syncPeriod,
iptables: ipt,
masqueradeAll: masqueradeAll,
@ -362,7 +367,7 @@ func NewDualStackProxier(
nodePortAddresses []string,
) (proxy.Provider, error) {
// Create an ipv4 instance of the single-stack proxier
nodePortAddresses4, nodePortAddresses6 := utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, false)
nodePortAddresses4, nodePortAddresses6 := utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, v1.IPv4Protocol)
ipv4Proxier, err := NewProxier(ipt[0], sysctl,
exec, syncPeriod, minSyncPeriod, masqueradeAll, masqueradeBit, localDetectors[0], hostname,
nodeIP[0], recorder, healthzServer, nodePortAddresses4)
@ -376,8 +381,7 @@ func NewDualStackProxier(
if err != nil {
return nil, fmt.Errorf("unable to create ipv6 proxier: %v", err)
}
return metaproxier.NewMetaProxier(ipv4Proxier, ipv6Proxier), nil // TODO move meta-proxier to mode-neutral package
return metaproxier.NewMetaProxier(ipv4Proxier, ipv6Proxier), nil
}
type iptablesJumpChain struct {

187
pkg/proxy/iptables/proxier_test.go
View File

@ -150,8 +150,7 @@ func TestGetChainLinesMultipleTables(t *testing.T) {
}
checkAllLines(t, utiliptables.TableNAT, []byte(iptablesSave), expected)
}
func TestDeleteEndpointConnections(t *testing.T) {
func TestDeleteEndpointConnectionsIPv4(t *testing.T) {
const (
UDP = v1.ProtocolUDP
TCP = v1.ProtocolTCP
@ -175,21 +174,24 @@ func TestDeleteEndpointConnections(t *testing.T) {
svcPort: 80,
protocol: UDP,
endpoint: "10.240.0.3:80",
}, {
},
{
description: "V4 TCP",
svcName: "v4-tcp",
svcIP: "10.96.2.2",
svcPort: 80,
protocol: TCP,
endpoint: "10.240.0.4:80",
}, {
},
{
description: "V4 SCTP",
svcName: "v4-sctp",
svcIP: "10.96.3.3",
svcPort: 80,
protocol: SCTP,
endpoint: "10.240.0.5:80",
}, {
},
{
description: "V4 UDP, nothing to delete, benign error",
svcName: "v4-udp-nothing-to-delete",
svcIP: "10.96.1.1",
@ -197,7 +199,8 @@ func TestDeleteEndpointConnections(t *testing.T) {
protocol: UDP,
endpoint: "10.240.0.3:80",
simulatedErr: conntrack.NoConnectionToDelete,
}, {
},
{
description: "V4 UDP, unexpected error, should be glogged",
svcName: "v4-udp-simulated-error",
svcIP: "10.96.1.1",
@ -205,27 +208,6 @@ func TestDeleteEndpointConnections(t *testing.T) {
protocol: UDP,
endpoint: "10.240.0.3:80",
simulatedErr: "simulated error",
}, {
description: "V6 UDP",
svcName: "v6-udp",
svcIP: "fd00:1234::20",
svcPort: 80,
protocol: UDP,
endpoint: "[2001:db8::2]:80",
}, {
description: "V6 TCP",
svcName: "v6-tcp",
svcIP: "fd00:1234::30",
svcPort: 80,
protocol: TCP,
endpoint: "[2001:db8::3]:80",
}, {
description: "V6 SCTP",
svcName: "v6-sctp",
svcIP: "fd00:1234::40",
svcPort: 80,
protocol: SCTP,
endpoint: "[2001:db8::4]:80",
},
}
@ -329,6 +311,149 @@ func TestDeleteEndpointConnections(t *testing.T) {
}
}
func TestDeleteEndpointConnectionsIPv6(t *testing.T) {
const (
UDP = v1.ProtocolUDP
TCP = v1.ProtocolTCP
SCTP = v1.ProtocolSCTP
)
testCases := []struct {
description string
svcName string
svcIP string
svcPort int32
protocol v1.Protocol
endpoint string // IP:port endpoint
epSvcPair proxy.ServiceEndpoint // Will be generated by test
simulatedErr string
}{
{
description: "V6 UDP",
svcName: "v6-udp",
svcIP: "fd00:1234::20",
svcPort: 80,
protocol: UDP,
endpoint: "[2001:db8::2]:80",
},
{
description: "V6 TCP",
svcName: "v6-tcp",
svcIP: "fd00:1234::30",
svcPort: 80,
protocol: TCP,
endpoint: "[2001:db8::3]:80",
},
{
description: "V6 SCTP",
svcName: "v6-sctp",
svcIP: "fd00:1234::40",
svcPort: 80,
protocol: SCTP,
endpoint: "[2001:db8::4]:80",
},
}
// Create a fake executor for the conntrack utility. This should only be
// invoked for UDP and SCTP connections, since no conntrack cleanup is needed for TCP
fcmd := fakeexec.FakeCmd{}
fexec := fakeexec.FakeExec{
LookPathFunc: func(cmd string) (string, error) { return cmd, nil },
}
execFunc := func(cmd string, args ...string) exec.Cmd {
return fakeexec.InitFakeCmd(&fcmd, cmd, args...)
}
for _, tc := range testCases {
if conntrack.IsClearConntrackNeeded(tc.protocol) {
var cmdOutput string
var simErr error
if tc.simulatedErr == "" {
cmdOutput = "1 flow entries have been deleted"
} else {
simErr = fmt.Errorf(tc.simulatedErr)
}
cmdFunc := func() ([]byte, []byte, error) { return []byte(cmdOutput), nil, simErr }
fcmd.CombinedOutputScript = append(fcmd.CombinedOutputScript, cmdFunc)
fexec.CommandScript = append(fexec.CommandScript, execFunc)
}
}
ipt := iptablestest.NewIPv6Fake()
fp := NewFakeProxier(ipt, false)
fp.exec = &fexec
for _, tc := range testCases {
makeServiceMap(fp,
makeTestService("ns1", tc.svcName, func(svc *v1.Service) {
svc.Spec.ClusterIP = tc.svcIP
svc.Spec.Ports = []v1.ServicePort{{
Name: "p80",
Port: tc.svcPort,
Protocol: tc.protocol,
}}
svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeLocal
}),
)
proxy.UpdateServiceMap(fp.serviceMap, fp.serviceChanges)
}
// Run the test cases
for _, tc := range testCases {
priorExecs := fexec.CommandCalls
priorGlogErrs := klog.Stats.Error.Lines()
svc := proxy.ServicePortName{
NamespacedName: types.NamespacedName{Namespace: "ns1", Name: tc.svcName},
Port: "p80",
Protocol: tc.protocol,
}
input := []proxy.ServiceEndpoint{
{
Endpoint: tc.endpoint,
ServicePortName: svc,
},
}
fp.deleteEndpointConnections(input)
// For UDP and SCTP connections, check the executed conntrack command
var expExecs int
if conntrack.IsClearConntrackNeeded(tc.protocol) {
isIPv6 := func(ip string) bool {
netIP := net.ParseIP(ip)
return netIP.To4() == nil
}
endpointIP := utilproxy.IPPart(tc.endpoint)
expectCommand := fmt.Sprintf("conntrack -D --orig-dst %s --dst-nat %s -p %s", tc.svcIP, endpointIP, strings.ToLower(string((tc.protocol))))
if isIPv6(endpointIP) {
expectCommand += " -f ipv6"
}
actualCommand := strings.Join(fcmd.CombinedOutputLog[fexec.CommandCalls-1], " ")
if actualCommand != expectCommand {
t.Errorf("%s: Expected command: %s, but executed %s", tc.description, expectCommand, actualCommand)
}
expExecs = 1
}
// Check the number of times conntrack was executed
execs := fexec.CommandCalls - priorExecs
if execs != expExecs {
t.Errorf("%s: Expected conntrack to be executed %d times, but got %d", tc.description, expExecs, execs)
}
// Check the number of new glog errors
var expGlogErrs int64
if tc.simulatedErr != "" && tc.simulatedErr != conntrack.NoConnectionToDelete {
expGlogErrs = 1
}
glogErrs := klog.Stats.Error.Lines() - priorGlogErrs
if glogErrs != expGlogErrs {
t.Errorf("%s: Expected %d glogged errors, but got %d", tc.description, expGlogErrs, glogErrs)
}
}
}
// fakePortOpener implements portOpener.
type fakePortOpener struct {
openPorts []*utilproxy.LocalPort
@ -346,13 +471,17 @@ const testHostname = "test-hostname"
func NewFakeProxier(ipt utiliptables.Interface, endpointSlicesEnabled bool) *Proxier {
// TODO: Call NewProxier after refactoring out the goroutine
// invocation into a Run() method.
ipfamily := v1.IPv4Protocol
if ipt.IsIPv6() {
ipfamily = v1.IPv6Protocol
}
detectLocal, _ := proxyutiliptables.NewDetectLocalByCIDR("10.0.0.0/24", ipt)
p := &Proxier{
exec: &fakeexec.FakeExec{},
serviceMap: make(proxy.ServiceMap),
serviceChanges: proxy.NewServiceChangeTracker(newServiceInfo, nil, nil, nil),
serviceChanges: proxy.NewServiceChangeTracker(newServiceInfo, ipfamily, nil, nil),
endpointsMap: make(proxy.EndpointsMap),
endpointsChanges: proxy.NewEndpointChangeTracker(testHostname, newEndpointInfo, nil, nil, endpointSlicesEnabled, nil),
endpointsChanges: proxy.NewEndpointChangeTracker(testHostname, newEndpointInfo, ipfamily, nil, endpointSlicesEnabled, nil),
iptables: ipt,
masqueradeMark: "0x4000",
localDetector: detectLocal,

1
pkg/proxy/ipvs/BUILD
View File

@ -37,6 +37,7 @@ go_test(
"//vendor/github.com/stretchr/testify/assert:go_default_library",
"//vendor/k8s.io/utils/exec:go_default_library",
"//vendor/k8s.io/utils/exec/testing:go_default_library",
"//vendor/k8s.io/utils/net:go_default_library",
"//vendor/k8s.io/utils/pointer:go_default_library",
],
)

35
pkg/proxy/ipvs/proxier.go
View File

@ -202,6 +202,8 @@ const sysctlArpAnnounce = "net/ipv4/conf/all/arp_announce"
// Proxier is an ipvs based proxy for connections between a localhost:lport
// and services that provide the actual backends.
type Proxier struct {
// the ipfamily on which this proxy is operating on.
ipFamily v1.IPFamily
// endpointsChanges and serviceChanges contains all changes to endpoints and
// services that happened since last syncProxyRules call. For a single object,
// changes are accumulated, i.e. previous is state from before all of them,
@ -432,9 +434,12 @@ func NewProxier(ipt utiliptables.Interface,
masqueradeValue := 1 << uint(masqueradeBit)
masqueradeMark := fmt.Sprintf("%#08x", masqueradeValue)
isIPv6 := utilnet.IsIPv6(nodeIP)
ipFamily := v1.IPv4Protocol
if ipt.IsIPv6() {
ipFamily = v1.IPv6Protocol
}
klog.V(2).Infof("nodeIP: %v, isIPv6: %v", nodeIP, isIPv6)
klog.V(2).Infof("nodeIP: %v, family: %v", nodeIP, ipFamily)
if len(scheduler) == 0 {
klog.Warningf("IPVS scheduler not specified, use %s by default", DefaultScheduler)
@ -446,16 +451,17 @@ func NewProxier(ipt utiliptables.Interface,
endpointSlicesEnabled := utilfeature.DefaultFeatureGate.Enabled(features.EndpointSliceProxying)
var incorrectAddresses []string
nodePortAddresses, incorrectAddresses = utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, isIPv6)
nodePortAddresses, incorrectAddresses = utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, ipFamily)
if len(incorrectAddresses) > 0 {
klog.Warning("NodePortAddresses of wrong family; ", incorrectAddresses)
}
proxier := &Proxier{
ipFamily: ipFamily,
portsMap: make(map[utilproxy.LocalPort]utilproxy.Closeable),
serviceMap: make(proxy.ServiceMap),
serviceChanges: proxy.NewServiceChangeTracker(newServiceInfo, &isIPv6, recorder, nil),
serviceChanges: proxy.NewServiceChangeTracker(newServiceInfo, ipFamily, recorder, nil),
endpointsMap: make(proxy.EndpointsMap),
endpointsChanges: proxy.NewEndpointChangeTracker(hostname, nil, &isIPv6, recorder, endpointSlicesEnabled, nil),
endpointsChanges: proxy.NewEndpointChangeTracker(hostname, nil, ipFamily, recorder, endpointSlicesEnabled, nil),
syncPeriod: syncPeriod,
minSyncPeriod: minSyncPeriod,
excludeCIDRs: parseExcludedCIDRs(excludeCIDRs),
@ -472,14 +478,14 @@ func NewProxier(ipt utiliptables.Interface,
healthzServer: healthzServer,
ipvs: ipvs,
ipvsScheduler: scheduler,
ipGetter: &realIPGetter{nl: NewNetLinkHandle(isIPv6)},
ipGetter: &realIPGetter{nl: NewNetLinkHandle(ipFamily == v1.IPv6Protocol)},
iptablesData: bytes.NewBuffer(nil),
filterChainsData: bytes.NewBuffer(nil),
natChains: bytes.NewBuffer(nil),
natRules: bytes.NewBuffer(nil),
filterChains: bytes.NewBuffer(nil),
filterRules: bytes.NewBuffer(nil),
netlinkHandle: NewNetLinkHandle(isIPv6),
netlinkHandle: NewNetLinkHandle(ipFamily == v1.IPv6Protocol),
ipset: ipset,
nodePortAddresses: nodePortAddresses,
networkInterfacer: utilproxy.RealNetwork{},
@ -488,7 +494,7 @@ func NewProxier(ipt utiliptables.Interface,
// initialize ipsetList with all sets we needed
proxier.ipsetList = make(map[string]*IPSet)
for _, is := range ipsetInfo {
proxier.ipsetList[is.name] = NewIPSet(ipset, is.name, is.setType, isIPv6, is.comment)
proxier.ipsetList[is.name] = NewIPSet(ipset, is.name, is.setType, (ipFamily == v1.IPv6Protocol), is.comment)
}
burstSyncs := 2
klog.V(2).Infof("ipvs(%s) sync params: minSyncPeriod=%v, syncPeriod=%v, burstSyncs=%d",
@ -526,7 +532,7 @@ func NewDualStackProxier(
safeIpset := newSafeIpset(ipset)
nodePortAddresses4, nodePortAddresses6 := utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, false)
nodePortAddresses4, nodePortAddresses6 := utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, v1.IPv4Protocol)
// Create an ipv4 instance of the single-stack proxier
ipv4Proxier, err := NewProxier(ipt[0], ipvs, safeIpset, sysctl,
@ -1149,6 +1155,17 @@ func (proxier *Proxier) syncProxyRules() {
}
}
// filter node IPs by proxier ipfamily
idx := 0
for _, nodeIP := range nodeIPs {
if (proxier.ipFamily == v1.IPv6Protocol) == utilnet.IsIPv6(nodeIP) {
nodeIPs[idx] = nodeIP
idx++
}
}
// reset slice to filtered entries
nodeIPs = nodeIPs[:idx]
// Build IPVS rules for each service.
for svcName, svc := range proxier.serviceMap {
svcInfo, ok := svc.(*serviceInfo)

711
pkg/proxy/ipvs/proxier_test.go
View File

@ -49,6 +49,8 @@ import (
"k8s.io/utils/exec"
fakeexec "k8s.io/utils/exec/testing"
utilpointer "k8s.io/utils/pointer"
utilnet "k8s.io/utils/net"
)
const testHostname = "test-hostname"
@ -102,7 +104,22 @@ func (fake *fakeIPSetVersioner) GetVersion() (string, error) {
return fake.version, fake.err
}
func NewFakeProxier(ipt utiliptables.Interface, ipvs utilipvs.Interface, ipset utilipset.Interface, nodeIPs []net.IP, excludeCIDRs []*net.IPNet, endpointSlicesEnabled bool) *Proxier {
func NewFakeProxier(ipt utiliptables.Interface, ipvs utilipvs.Interface, ipset utilipset.Interface, nodeIPs []net.IP, excludeCIDRs []*net.IPNet, endpointSlicesEnabled bool, ipFamily v1.IPFamily) *Proxier {
// unlike actual proxier, this fake proxier does not filter node IPs per family requested
// which can lead to false postives.
// filter node IPs by proxier ipfamily
idx := 0
for _, nodeIP := range nodeIPs {
if (ipFamily == v1.IPv6Protocol) == utilnet.IsIPv6(nodeIP) {
nodeIPs[idx] = nodeIP
idx++
}
}
// reset slice to filtered entries
nodeIPs = nodeIPs[:idx]
fcmd := fakeexec.FakeCmd{
CombinedOutputScript: []fakeexec.FakeAction{
func() ([]byte, []byte, error) { return []byte("dummy device have been created"), nil, nil },
@ -124,9 +141,9 @@ func NewFakeProxier(ipt utiliptables.Interface, ipvs utilipvs.Interface, ipset u
p := &Proxier{
exec: fexec,
serviceMap: make(proxy.ServiceMap),
serviceChanges: proxy.NewServiceChangeTracker(newServiceInfo, nil, nil, nil),
serviceChanges: proxy.NewServiceChangeTracker(newServiceInfo, ipFamily, nil, nil),
endpointsMap: make(proxy.EndpointsMap),
endpointsChanges: proxy.NewEndpointChangeTracker(testHostname, nil, nil, nil, endpointSlicesEnabled, nil),
endpointsChanges: proxy.NewEndpointChangeTracker(testHostname, nil, ipFamily, nil, endpointSlicesEnabled, nil),
excludeCIDRs: excludeCIDRs,
iptables: ipt,
ipvs: ipvs,
@ -150,6 +167,7 @@ func NewFakeProxier(ipt utiliptables.Interface, ipvs utilipvs.Interface, ipset u
nodePortAddresses: make([]string, 0),
networkInterfacer: proxyutiltest.NewFakeNetwork(),
gracefuldeleteManager: NewGracefulTerminationManager(ipvs),
ipFamily: ipFamily,
}
p.setInitialized(true)
p.syncRunner = async.NewBoundedFrequencyRunner("test-sync-runner", p.syncProxyRules, 0, time.Minute, 1)
@ -209,7 +227,7 @@ func TestCleanupLeftovers(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
svcIP := "10.20.30.41"
svcPort := 80
svcNodePort := 3001
@ -442,7 +460,7 @@ func TestGetNodeIPs(t *testing.T) {
}
}
func TestNodePort(t *testing.T) {
func TestNodePortIPv4(t *testing.T) {
tests := []struct {
name string
services []*v1.Service
@ -510,16 +528,6 @@ func TestNodePort(t *testing.T) {
Port: uint16(3001),
Scheduler: "rr",
},
{
IP: "2001:db8::1:1",
Port: 3001,
Protocol: "TCP",
}: {
Address: net.ParseIP("2001:db8::1:1"),
Protocol: "TCP",
Port: uint16(3001),
Scheduler: "rr",
},
},
Destinations: map[ipvstest.ServiceKey][]*utilipvs.RealServer{
{
@ -532,11 +540,6 @@ func TestNodePort(t *testing.T) {
Port: uint16(80),
Weight: 1,
},
{
Address: net.ParseIP("1002:ab8::2:10"),
Port: uint16(80),
Weight: 1,
},
},
{
IP: "100.101.102.103",
@ -548,27 +551,6 @@ func TestNodePort(t *testing.T) {
Port: uint16(80),
Weight: 1,
},
{
Address: net.ParseIP("1002:ab8::2:10"),
Port: uint16(80),
Weight: 1,
},
},
{
IP: "2001:db8::1:1",
Port: 3001,
Protocol: "TCP",
}: {
{
Address: net.ParseIP("10.180.0.1"),
Port: uint16(80),
Weight: 1,
},
{
Address: net.ParseIP("1002:ab8::2:10"),
Port: uint16(80),
Weight: 1,
},
},
},
},
@ -804,36 +786,6 @@ func TestNodePort(t *testing.T) {
Port: uint16(3001),
Scheduler: "rr",
},
{
IP: "2001:db8::1:1",
Port: 3001,
Protocol: "SCTP",
}: {
Address: net.ParseIP("2001:db8::1:1"),
Protocol: "SCTP",
Port: uint16(3001),
Scheduler: "rr",
},
{
IP: "2001:db8::1:2",
Port: 3001,
Protocol: "SCTP",
}: {
Address: net.ParseIP("2001:db8::1:2"),
Protocol: "SCTP",
Port: uint16(3001),
Scheduler: "rr",
},
{
IP: "2001:db8::1:3",
Port: 3001,
Protocol: "SCTP",
}: {
Address: net.ParseIP("2001:db8::1:3"),
Protocol: "SCTP",
Port: uint16(3001),
Scheduler: "rr",
},
},
Destinations: map[ipvstest.ServiceKey][]*utilipvs.RealServer{
{
@ -880,39 +832,6 @@ func TestNodePort(t *testing.T) {
Weight: 1,
},
},
{
IP: "2001:db8::1:1",
Port: 3001,
Protocol: "SCTP",
}: {
{
Address: net.ParseIP("10.180.0.1"),
Port: uint16(80),
Weight: 1,
},
},
{
IP: "2001:db8::1:2",
Port: 3001,
Protocol: "SCTP",
}: {
{
Address: net.ParseIP("10.180.0.1"),
Port: uint16(80),
Weight: 1,
},
},
{
IP: "2001:db8::1:3",
Port: 3001,
Protocol: "SCTP",
}: {
{
Address: net.ParseIP("10.180.0.1"),
Port: uint16(80),
Weight: 1,
},
},
},
},
expectedIPSets: netlinktest.ExpectedIPSet{
@ -935,24 +854,6 @@ func TestNodePort(t *testing.T) {
Protocol: strings.ToLower(string(v1.ProtocolSCTP)),
SetType: utilipset.HashIPPort,
},
{
IP: "2001:db8::1:1",
Port: 3001,
Protocol: strings.ToLower(string(v1.ProtocolSCTP)),
SetType: utilipset.HashIPPort,
},
{
IP: "2001:db8::1:2",
Port: 3001,
Protocol: strings.ToLower(string(v1.ProtocolSCTP)),
SetType: utilipset.HashIPPort,
},
{
IP: "2001:db8::1:3",
Port: 3001,
Protocol: strings.ToLower(string(v1.ProtocolSCTP)),
SetType: utilipset.HashIPPort,
},
},
},
},
@ -963,7 +864,7 @@ func TestNodePort(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, test.nodeIPs, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, test.nodeIPs, nil, false, v1.IPv4Protocol)
fp.nodePortAddresses = test.nodePortAddresses
makeServiceMap(fp, test.services...)
@ -972,8 +873,8 @@ func TestNodePort(t *testing.T) {
fp.syncProxyRules()
if !reflect.DeepEqual(ipvs, test.expectedIPVS) {
t.Logf("actual ipvs state: %v", ipvs)
t.Logf("expected ipvs state: %v", test.expectedIPVS)
t.Logf("actual ipvs state: %+v", ipvs)
t.Logf("expected ipvs state: %+v", test.expectedIPVS)
t.Errorf("unexpected IPVS state")
}
@ -988,7 +889,354 @@ func TestNodePort(t *testing.T) {
}
}
func TestClusterIP(t *testing.T) {
func TestNodePortIPv6(t *testing.T) {
tests := []struct {
name string
services []*v1.Service
endpoints []*v1.Endpoints
nodeIPs []net.IP
nodePortAddresses []string
expectedIPVS *ipvstest.FakeIPVS
expectedIPSets netlinktest.ExpectedIPSet
expectedIptablesChains netlinktest.ExpectedIptablesChain
}{
{
name: "1 service with node port, has 2 endpoints",
services: []*v1.Service{
makeTestService("ns1", "svc1", func(svc *v1.Service) {
svc.Spec.Type = "NodePort"
svc.Spec.ClusterIP = "2020::1"
svc.Spec.Ports = []v1.ServicePort{{
Name: "p80",
Port: int32(80),
Protocol: v1.ProtocolTCP,
NodePort: int32(3001),
}}
}),
},
endpoints: []*v1.Endpoints{
makeTestEndpoints("ns1", "svc1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{{
Addresses: []v1.EndpointAddress{{
IP: "10.180.0.1",
}, {
IP: "1002:ab8::2:10",
}},
Ports: []v1.EndpointPort{{
Name: "p80",
Port: int32(80),
Protocol: v1.ProtocolTCP,
}},
}}
}),
},
nodeIPs: []net.IP{
net.ParseIP("100.101.102.103"),
net.ParseIP("2001:db8::1:1"),
},
nodePortAddresses: []string{},
expectedIPVS: &ipvstest.FakeIPVS{
Services: map[ipvstest.ServiceKey]*utilipvs.VirtualServer{
{
IP: "2001:db8::1:1",
Port: 3001,
Protocol: "TCP",
}: {
Address: net.ParseIP("2001:db8::1:1"),
Protocol: "TCP",
Port: uint16(3001),
Scheduler: "rr",
},
{
IP: "2020::1",
Port: 80,
Protocol: "TCP",
}: {
Address: net.ParseIP("2020::1"),
Protocol: "TCP",
Port: uint16(80),
Scheduler: "rr",
},
},
Destinations: map[ipvstest.ServiceKey][]*utilipvs.RealServer{
{
IP: "2001:db8::1:1",
Port: 3001,
Protocol: "TCP",
}: {
{
Address: net.ParseIP("1002:ab8::2:10"),
Port: uint16(80),
Weight: 1,
},
},
{
IP: "2020::1",
Port: 80,
Protocol: "TCP",
}: {
{
Address: net.ParseIP("1002:ab8::2:10"),
Port: uint16(80),
Weight: 1,
},
},
},
},
},
{
name: "1 UDP service with node port, has endpoints (no action on IPv6 Proxier)",
services: []*v1.Service{
makeTestService("ns1", "svc1", func(svc *v1.Service) {
svc.Spec.Type = "NodePort"
svc.Spec.ClusterIP = "10.20.30.41"
svc.Spec.Ports = []v1.ServicePort{{
Name: "p80",
Port: int32(80),
Protocol: v1.ProtocolUDP,
NodePort: int32(3001),
}}
}),
},
endpoints: []*v1.Endpoints{
makeTestEndpoints("ns1", "svc1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{{
Addresses: []v1.EndpointAddress{{
IP: "10.180.0.1",
}},
Ports: []v1.EndpointPort{{
Name: "p80",
Port: int32(80),
Protocol: v1.ProtocolUDP,
}},
}}
}),
},
nodeIPs: []net.IP{
net.ParseIP("100.101.102.103"),
},
nodePortAddresses: []string{"0.0.0.0/0"},
/*since this is a node with only IPv4, proxier should not do anything */
expectedIPVS: &ipvstest.FakeIPVS{
Services: map[ipvstest.ServiceKey]*utilipvs.VirtualServer{},
Destinations: map[ipvstest.ServiceKey][]*utilipvs.RealServer{},
},
expectedIPSets: nil,
expectedIptablesChains: nil,
},
{
name: "service has node port but no endpoints",
services: []*v1.Service{
makeTestService("ns1", "svc1", func(svc *v1.Service) {
svc.Spec.Type = "NodePort"
svc.Spec.ClusterIP = "2020::1"
svc.Spec.Ports = []v1.ServicePort{{
Name: "p80",
Port: int32(80),
Protocol: v1.ProtocolTCP,
NodePort: int32(3001),
}}
}),
},
endpoints: []*v1.Endpoints{},
nodeIPs: []net.IP{
net.ParseIP("100.101.102.103"),
net.ParseIP("2001:db8::1:1"),
},
nodePortAddresses: []string{},
expectedIPVS: &ipvstest.FakeIPVS{
Services: map[ipvstest.ServiceKey]*utilipvs.VirtualServer{
{
IP: "2001:db8::1:1",
Port: 3001,
Protocol: "TCP",
}: {
Address: net.ParseIP("2001:db8::1:1"),
Protocol: "TCP",
Port: uint16(3001),
Scheduler: "rr",
},
{
IP: "2020::1",
Port: 80,
Protocol: "TCP",
}: {
Address: net.ParseIP("2020::1"),
Protocol: "TCP",
Port: uint16(80),
Scheduler: "rr",
},
},
Destinations: map[ipvstest.ServiceKey][]*utilipvs.RealServer{
{
IP: "2020::1",
Port: 80,
Protocol: "TCP",
}: {}, // no real servers corresponding to no endpoints
{
IP: "2001:db8::1:1",
Port: 3001,
Protocol: "TCP",
}: {}, // no real servers corresponding to no endpoints
},
},
},
{
name: "node port service with protocol sctp on a node with multiple nodeIPs",
services: []*v1.Service{
makeTestService("ns1", "svc1", func(svc *v1.Service) {
svc.Spec.Type = "NodePort"
svc.Spec.ClusterIP = "2020::1"
svc.Spec.Ports = []v1.ServicePort{{
Name: "p80",
Port: int32(80),
Protocol: v1.ProtocolSCTP,
NodePort: int32(3001),
}}
}),
},
endpoints: []*v1.Endpoints{
makeTestEndpoints("ns1", "svc1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{{
Addresses: []v1.EndpointAddress{{
IP: "2001::1",
}},
Ports: []v1.EndpointPort{{
Name: "p80",
Port: int32(80),
Protocol: v1.ProtocolSCTP,
}},
}}
}),
},
nodeIPs: []net.IP{
net.ParseIP("2001:db8::1:1"),
net.ParseIP("2001:db8::1:2"),
},
nodePortAddresses: []string{},
expectedIPVS: &ipvstest.FakeIPVS{
Services: map[ipvstest.ServiceKey]*utilipvs.VirtualServer{
{
IP: "2001:db8::1:1",
Port: 3001,
Protocol: "SCTP",
}: {
Address: net.ParseIP("2001:db8::1:1"),
Protocol: "SCTP",
Port: uint16(3001),
Scheduler: "rr",
},
{
IP: "2001:db8::1:2",
Port: 3001,
Protocol: "SCTP",
}: {
Address: net.ParseIP("2001:db8::1:2"),
Protocol: "SCTP",
Port: uint16(3001),
Scheduler: "rr",
},
{
IP: "2020::1",
Port: 80,
Protocol: "SCTP",
}: {
Address: net.ParseIP("2020::1"),
Protocol: "SCTP",
Port: uint16(80),
Scheduler: "rr",
},
},
Destinations: map[ipvstest.ServiceKey][]*utilipvs.RealServer{
{
IP: "2001:db8::1:1",
Port: 3001,
Protocol: "SCTP",
}: {
{
Address: net.ParseIP("2001::1"),
Port: uint16(80),
Weight: 1,
},
},
{
IP: "2001:db8::1:2",
Port: 3001,
Protocol: "SCTP",
}: {
{
Address: net.ParseIP("2001::1"),
Port: uint16(80),
Weight: 1,
},
},
{
IP: "2020::1",
Port: 80,
Protocol: "SCTP",
}: {
{
Address: net.ParseIP("2001::1"),
Port: uint16(80),
Weight: 1,
},
},
},
},
expectedIPSets: netlinktest.ExpectedIPSet{
kubeNodePortSetSCTP: {
{
IP: "2001:db8::1:1",
Port: 3001,
Protocol: strings.ToLower(string(v1.ProtocolSCTP)),
SetType: utilipset.HashIPPort,
},
{
IP: "2001:db8::1:2",
Port: 3001,
Protocol: strings.ToLower(string(v1.ProtocolSCTP)),
SetType: utilipset.HashIPPort,
},
},
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, test.nodeIPs, nil, false, v1.IPv6Protocol)
fp.nodePortAddresses = test.nodePortAddresses
makeServiceMap(fp, test.services...)
makeEndpointsMap(fp, test.endpoints...)
fp.syncProxyRules()
if !reflect.DeepEqual(ipvs, test.expectedIPVS) {
t.Logf("actual ipvs state: %+v", ipvs)
t.Logf("expected ipvs state: %+v", test.expectedIPVS)
t.Errorf("unexpected IPVS state")
}
if test.expectedIPSets != nil {
checkIPSet(t, fp, test.expectedIPSets)
}
if test.expectedIptablesChains != nil {
checkIptables(t, ipt, test.expectedIptablesChains)
}
})
}
}
func TestIPv4Proxier(t *testing.T) {
tests := []struct {
name string
services []*v1.Service
@ -1053,16 +1301,6 @@ func TestClusterIP(t *testing.T) {
Port: uint16(80),
Scheduler: "rr",
},
{
IP: "1002:ab8::2:1",
Port: 8080,
Protocol: "TCP",
}: {
Address: net.ParseIP("1002:ab8::2:1"),
Protocol: "TCP",
Port: uint16(8080),
Scheduler: "rr",
},
},
Destinations: map[ipvstest.ServiceKey][]*utilipvs.RealServer{
{
@ -1076,17 +1314,6 @@ func TestClusterIP(t *testing.T) {
Weight: 1,
},
},
{
IP: "1002:ab8::2:1",
Port: 8080,
Protocol: "TCP",
}: {
{
Address: net.ParseIP("1009:ab8::5:6"),
Port: uint16(8080),
Weight: 1,
},
},
},
},
},
@ -1132,7 +1359,146 @@ func TestClusterIP(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
makeServiceMap(fp, test.services...)
makeEndpointsMap(fp, test.endpoints...)
fp.syncProxyRules()
if !reflect.DeepEqual(ipvs, test.expectedIPVS) {
t.Logf("actual ipvs state: %v", ipvs)
t.Logf("expected ipvs state: %v", test.expectedIPVS)
t.Errorf("unexpected IPVS state")
}
})
}
}
func TestIPv6Proxier(t *testing.T) {
tests := []struct {
name string
services []*v1.Service
endpoints []*v1.Endpoints
expectedIPVS *ipvstest.FakeIPVS
}{
{
name: "2 services with Cluster IP, each with endpoints",
services: []*v1.Service{
makeTestService("ns1", "svc1", func(svc *v1.Service) {
svc.Spec.ClusterIP = "10.20.30.41"
svc.Spec.Ports = []v1.ServicePort{{
Name: "p80",
Port: int32(80),
Protocol: v1.ProtocolTCP,
}}
}),
makeTestService("ns2", "svc2", func(svc *v1.Service) {
svc.Spec.ClusterIP = "1002:ab8::2:1"
svc.Spec.Ports = []v1.ServicePort{{
Name: "p8080",
Port: int32(8080),
Protocol: v1.ProtocolTCP,
}}
}),
},
endpoints: []*v1.Endpoints{
makeTestEndpoints("ns1", "svc1", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{{
Addresses: []v1.EndpointAddress{{
IP: "10.180.0.1",
}},
Ports: []v1.EndpointPort{{
Name: "p80",
Port: int32(80),
Protocol: v1.ProtocolTCP,
}},
}}
}),
makeTestEndpoints("ns2", "svc2", func(ept *v1.Endpoints) {
ept.Subsets = []v1.EndpointSubset{{
Addresses: []v1.EndpointAddress{{
IP: "1009:ab8::5:6",
}},
Ports: []v1.EndpointPort{{
Name: "p8080",
Port: int32(8080),
Protocol: v1.ProtocolTCP,
}},
}}
}),
},
expectedIPVS: &ipvstest.FakeIPVS{
Services: map[ipvstest.ServiceKey]*utilipvs.VirtualServer{
{
IP: "1002:ab8::2:1",
Port: 8080,
Protocol: "TCP",
}: {
Address: net.ParseIP("1002:ab8::2:1"),
Protocol: "TCP",
Port: uint16(8080),
Scheduler: "rr",
},
},
Destinations: map[ipvstest.ServiceKey][]*utilipvs.RealServer{
{
IP: "1002:ab8::2:1",
Port: 8080,
Protocol: "TCP",
}: {
{
Address: net.ParseIP("1009:ab8::5:6"),
Port: uint16(8080),
Weight: 1,
},
},
},
},
},
{
name: "cluster IP service with no endpoints",
services: []*v1.Service{
makeTestService("ns1", "svc1", func(svc *v1.Service) {
svc.Spec.ClusterIP = "2001::1"
svc.Spec.Ports = []v1.ServicePort{{
Name: "p80",
Port: int32(80),
Protocol: v1.ProtocolTCP,
}}
}),
},
endpoints: []*v1.Endpoints{},
expectedIPVS: &ipvstest.FakeIPVS{
Services: map[ipvstest.ServiceKey]*utilipvs.VirtualServer{
{
IP: "2001::1",
Port: 80,
Protocol: "TCP",
}: {
Address: net.ParseIP("2001::1"),
Protocol: "TCP",
Port: uint16(80),
Scheduler: "rr",
},
},
Destinations: map[ipvstest.ServiceKey][]*utilipvs.RealServer{
{
IP: "2001::1",
Port: 80,
Protocol: "TCP",
}: {},
},
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv6Protocol)
makeServiceMap(fp, test.services...)
makeEndpointsMap(fp, test.endpoints...)
@ -1153,7 +1519,7 @@ func TestMasqueradeRule(t *testing.T) {
ipt := iptablestest.NewFake().SetHasRandomFully(testcase)
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
makeServiceMap(fp)
makeEndpointsMap(fp)
fp.syncProxyRules()
@ -1173,7 +1539,7 @@ func TestExternalIPsNoEndpoint(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
svcIP := "10.20.30.41"
svcPort := 80
svcExternalIPs := "50.60.70.81"
@ -1228,7 +1594,7 @@ func TestExternalIPs(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
svcIP := "10.20.30.41"
svcPort := 80
svcExternalIPs := sets.NewString("50.60.70.81", "2012::51", "127.0.0.1")
@ -1273,8 +1639,8 @@ func TestExternalIPs(t *testing.T) {
if err != nil {
t.Errorf("Failed to get ipvs services, err: %v", err)
}
if len(services) != 4 {
t.Errorf("Expect 4 ipvs services, got %d", len(services))
if len(services) != 3 { // ipvs filters out by ipfamily
t.Errorf("Expect 3 ipvs services, got %d", len(services))
}
found := false
for _, svc := range services {
@ -1298,7 +1664,7 @@ func TestOnlyLocalExternalIPs(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
svcIP := "10.20.30.41"
svcPort := 80
svcExternalIPs := sets.NewString("50.60.70.81", "2012::51", "127.0.0.1")
@ -1353,8 +1719,8 @@ func TestOnlyLocalExternalIPs(t *testing.T) {
if err != nil {
t.Errorf("Failed to get ipvs services, err: %v", err)
}
if len(services) != 4 {
t.Errorf("Expect 4 ipvs services, got %d", len(services))
if len(services) != 3 { // ipvs filters out by IPFamily
t.Errorf("Expect 3 ipvs services, got %d", len(services))
}
found := false
for _, svc := range services {
@ -1520,9 +1886,9 @@ func TestOnlyLocalNodePorts(t *testing.T) {
fp.syncProxyRules()
// Expect 3 services and 1 destination
// Expect 2 (matching ipvs IPFamily field) services and 1 destination
epVS := &netlinktest.ExpectedVirtualServer{
VSNum: 3, IP: nodeIP.String(), Port: uint16(svcNodePort), Protocol: string(v1.ProtocolTCP),
VSNum: 2, IP: nodeIP.String(), Port: uint16(svcNodePort), Protocol: string(v1.ProtocolTCP),
RS: []netlinktest.ExpectedRealServer{{
IP: epIP, Port: uint16(svcPort),
}}}
@ -1835,7 +2201,7 @@ func TestBuildServiceMapAddRemove(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
services := []*v1.Service{
makeTestService("somewhere-else", "cluster-ip", func(svc *v1.Service) {
@ -1945,7 +2311,7 @@ func TestBuildServiceMapServiceHeadless(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
makeServiceMap(fp,
makeTestService("somewhere-else", "headless", func(svc *v1.Service) {
@ -1984,7 +2350,7 @@ func TestBuildServiceMapServiceTypeExternalName(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
makeServiceMap(fp,
makeTestService("somewhere-else", "external-name", func(svc *v1.Service) {
@ -2012,7 +2378,7 @@ func TestBuildServiceMapServiceUpdate(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
servicev1 := makeTestService("somewhere", "some-service", func(svc *v1.Service) {
svc.Spec.Type = v1.ServiceTypeClusterIP
@ -2096,7 +2462,7 @@ func TestSessionAffinity(t *testing.T) {
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
nodeIP := net.ParseIP("100.101.102.103")
fp := NewFakeProxier(ipt, ipvs, ipset, []net.IP{nodeIP}, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, []net.IP{nodeIP}, nil, false, v1.IPv4Protocol)
svcIP := "10.20.30.41"
svcPort := 80
svcNodePort := 3001
@ -2991,7 +3357,7 @@ func Test_updateEndpointsMap(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
fp.hostname = nodeName
// First check that after adding all previous versions of endpoints,
@ -3264,7 +3630,7 @@ func Test_syncService(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
proxier := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
proxier := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
proxier.netlinkHandle.EnsureDummyDevice(DefaultDummyDevice)
if testCases[i].oldVirtualServer != nil {
@ -3294,7 +3660,7 @@ func buildFakeProxier() (*iptablestest.FakeIPTables, *Proxier) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
return ipt, NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
return ipt, NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
}
func hasJump(rules []iptablestest.Rule, destChain, ipSet string) bool {
@ -3354,6 +3720,7 @@ func checkIPSet(t *testing.T, fp *Proxier, ipSet netlinktest.ExpectedIPSet) {
// checkIPVS to check expected ipvs service and destination
func checkIPVS(t *testing.T, fp *Proxier, vs *netlinktest.ExpectedVirtualServer) {
t.Helper()
services, err := fp.ipvs.GetVirtualServers()
if err != nil {
t.Errorf("Failed to get ipvs services, err: %v", err)
@ -3380,7 +3747,7 @@ func TestCleanLegacyService(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, parseExcludedCIDRs([]string{"3.3.3.0/24", "4.4.4.0/24"}), false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, parseExcludedCIDRs([]string{"3.3.3.0/24", "4.4.4.0/24"}), false, v1.IPv4Protocol)
// All ipvs services that were processed in the latest sync loop.
activeServices := map[string]bool{"ipvs0": true, "ipvs1": true}
@ -3486,7 +3853,7 @@ func TestCleanLegacyServiceWithRealServers(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
// all deleted expect ipvs2
activeServices := map[string]bool{"ipvs2": true}
@ -3580,7 +3947,7 @@ func TestCleanLegacyRealServersExcludeCIDRs(t *testing.T) {
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
gtm := NewGracefulTerminationManager(ipvs)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, parseExcludedCIDRs([]string{"4.4.4.4/32"}), false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, parseExcludedCIDRs([]string{"4.4.4.4/32"}), false, v1.IPv4Protocol)
fp.gracefuldeleteManager = gtm
vs := &utilipvs.VirtualServer{
@ -3634,7 +4001,7 @@ func TestCleanLegacyService6(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, parseExcludedCIDRs([]string{"3000::/64", "4000::/64"}), false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, parseExcludedCIDRs([]string{"3000::/64", "4000::/64"}), false, v1.IPv4Protocol)
fp.nodeIP = net.ParseIP("::1")
// All ipvs services that were processed in the latest sync loop.
@ -3741,7 +4108,7 @@ func TestMultiPortServiceBindAddr(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, false, v1.IPv4Protocol)
service1 := makeTestService("ns1", "svc1", func(svc *v1.Service) {
svc.Spec.Type = v1.ServiceTypeClusterIP
@ -3846,7 +4213,7 @@ func TestEndpointSliceE2E(t *testing.T) {
ipt := iptablestest.NewFake()
ipvs := ipvstest.NewFake()
ipset := ipsettest.NewFake(testIPSetVersion)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, true)
fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, true, v1.IPv4Protocol)
fp.servicesSynced = true
fp.endpointsSynced = true
fp.endpointSlicesSynced = true

1
pkg/proxy/metaproxier/BUILD
View File

@ -9,7 +9,6 @@ go_library(
deps = [
"//pkg/proxy:go_default_library",
"//pkg/proxy/config:go_default_library",
"//pkg/proxy/util:go_default_library",
"//staging/src/k8s.io/api/core/v1:go_default_library",
"//staging/src/k8s.io/api/discovery/v1beta1:go_default_library",
"//vendor/k8s.io/klog/v2:go_default_library",

40
pkg/proxy/metaproxier/meta_proxier.go
View File

@ -20,18 +20,18 @@ import (
"fmt"
v1 "k8s.io/api/core/v1"
discovery "k8s.io/api/discovery/v1beta1"
"k8s.io/klog/v2"
"k8s.io/kubernetes/pkg/proxy"
"k8s.io/kubernetes/pkg/proxy/config"
utilproxy "k8s.io/kubernetes/pkg/proxy/util"
utilnet "k8s.io/utils/net"
discovery "k8s.io/api/discovery/v1beta1"
)
type metaProxier struct {
// actual, wrapped
ipv4Proxier proxy.Provider
// actual, wrapped
ipv6Proxier proxy.Provider
// TODO(imroc): implement node handler for meta proxier.
config.NoopNodeHandler
@ -63,41 +63,23 @@ func (proxier *metaProxier) SyncLoop() {
// OnServiceAdd is called whenever creation of new service object is observed.
func (proxier *metaProxier) OnServiceAdd(service *v1.Service) {
if utilproxy.ShouldSkipService(service) {
return
}
if utilnet.IsIPv6String(service.Spec.ClusterIP) {
proxier.ipv6Proxier.OnServiceAdd(service)
} else {
proxier.ipv4Proxier.OnServiceAdd(service)
}
proxier.ipv4Proxier.OnServiceAdd(service)
proxier.ipv6Proxier.OnServiceAdd(service)
}
// OnServiceUpdate is called whenever modification of an existing
// service object is observed.
func (proxier *metaProxier) OnServiceUpdate(oldService, service *v1.Service) {
if utilproxy.ShouldSkipService(service) {
return
}
// IPFamily is immutable, hence we only need to check on the new service
if utilnet.IsIPv6String(service.Spec.ClusterIP) {
proxier.ipv6Proxier.OnServiceUpdate(oldService, service)
} else {
proxier.ipv4Proxier.OnServiceUpdate(oldService, service)
}
proxier.ipv4Proxier.OnServiceUpdate(oldService, service)
proxier.ipv6Proxier.OnServiceUpdate(oldService, service)
}
// OnServiceDelete is called whenever deletion of an existing service
// object is observed.
func (proxier *metaProxier) OnServiceDelete(service *v1.Service) {
if utilproxy.ShouldSkipService(service) {
return
}
if utilnet.IsIPv6String(service.Spec.ClusterIP) {
proxier.ipv6Proxier.OnServiceDelete(service)
} else {
proxier.ipv4Proxier.OnServiceDelete(service)
}
proxier.ipv4Proxier.OnServiceDelete(service)
proxier.ipv6Proxier.OnServiceDelete(service)
}
// OnServiceSynced is called once all the initial event handlers were
@ -161,8 +143,6 @@ func (proxier *metaProxier) OnEndpointsSynced() {
proxier.ipv6Proxier.OnEndpointsSynced()
}
// TODO: (khenidak) implement EndpointSlice handling
// OnEndpointSliceAdd is called whenever creation of a new endpoint slice object
// is observed.
func (proxier *metaProxier) OnEndpointSliceAdd(endpointSlice *discovery.EndpointSlice) {

88
pkg/proxy/service.go
View File

@ -32,7 +32,6 @@ import (
apiservice "k8s.io/kubernetes/pkg/api/v1/service"
"k8s.io/kubernetes/pkg/proxy/metrics"
utilproxy "k8s.io/kubernetes/pkg/proxy/util"
utilnet "k8s.io/utils/net"
)
// BaseServiceInfo contains base information that defines a service.
@ -135,8 +134,10 @@ func (sct *ServiceChangeTracker) newBaseServiceInfo(port *v1.ServicePort, servic
// Kube-apiserver side guarantees SessionAffinityConfig won't be nil when session affinity type is ClientIP
stickyMaxAgeSeconds = int(*service.Spec.SessionAffinityConfig.ClientIP.TimeoutSeconds)
}
clusterIP := utilproxy.GetClusterIPByFamily(sct.ipFamily, service)
info := &BaseServiceInfo{
clusterIP: net.ParseIP(service.Spec.ClusterIP),
clusterIP: net.ParseIP(clusterIP),
port: int(port.Port),
protocol: port.Protocol,
nodePort: int(port.NodePort),
@ -150,41 +151,38 @@ func (sct *ServiceChangeTracker) newBaseServiceInfo(port *v1.ServicePort, servic
for i, sourceRange := range service.Spec.LoadBalancerSourceRanges {
loadBalancerSourceRanges[i] = strings.TrimSpace(sourceRange)
}
// filter external ips, source ranges and ingress ips
// prior to dual stack services, this was considered an error, but with dual stack
// services, this is actually expected. Hence we downgraded from reporting by events
// to just log lines with high verbosity
if sct.isIPv6Mode == nil {
info.externalIPs = make([]string, len(service.Spec.ExternalIPs))
info.loadBalancerSourceRanges = loadBalancerSourceRanges
copy(info.externalIPs, service.Spec.ExternalIPs)
// Deep-copy in case the service instance changes
info.loadBalancerStatus = *service.Status.LoadBalancer.DeepCopy()
} else {
// Filter out the incorrect IP version case.
// If ExternalIPs, LoadBalancerSourceRanges and LoadBalancerStatus Ingress on service contains incorrect IP versions,
// only filter out the incorrect ones.
var incorrectIPs []string
info.externalIPs, incorrectIPs = utilproxy.FilterIncorrectIPVersion(service.Spec.ExternalIPs, *sct.isIPv6Mode)
if len(incorrectIPs) > 0 {
utilproxy.LogAndEmitIncorrectIPVersionEvent(sct.recorder, "externalIPs", strings.Join(incorrectIPs, ","), service.Namespace, service.Name, service.UID)
}
info.loadBalancerSourceRanges, incorrectIPs = utilproxy.FilterIncorrectCIDRVersion(loadBalancerSourceRanges, *sct.isIPv6Mode)
if len(incorrectIPs) > 0 {
utilproxy.LogAndEmitIncorrectIPVersionEvent(sct.recorder, "loadBalancerSourceRanges", strings.Join(incorrectIPs, ","), service.Namespace, service.Name, service.UID)
}
// Obtain Load Balancer Ingress IPs
var ips []string
for _, ing := range service.Status.LoadBalancer.Ingress {
ips = append(ips, ing.IP)
}
if len(ips) > 0 {
correctIPs, incorrectIPs := utilproxy.FilterIncorrectIPVersion(ips, *sct.isIPv6Mode)
if len(incorrectIPs) > 0 {
utilproxy.LogAndEmitIncorrectIPVersionEvent(sct.recorder, "Load Balancer ingress IPs", strings.Join(incorrectIPs, ","), service.Namespace, service.Name, service.UID)
}
// Create the LoadBalancerStatus with the filtererd IPs
for _, ip := range correctIPs {
info.loadBalancerStatus.Ingress = append(info.loadBalancerStatus.Ingress, v1.LoadBalancerIngress{IP: ip})
var incorrectIPs []string
info.externalIPs, incorrectIPs = utilproxy.FilterIncorrectIPVersion(service.Spec.ExternalIPs, sct.ipFamily)
if len(incorrectIPs) > 0 {
klog.V(4).Infof("service change tracker(%v) ignored the following external IPs(%s) for service %v/%v as they don't match IPFamily", sct.ipFamily, strings.Join(incorrectIPs, ","), service.Namespace, service.Name)
}
}
info.loadBalancerSourceRanges, incorrectIPs = utilproxy.FilterIncorrectCIDRVersion(loadBalancerSourceRanges, sct.ipFamily)
if len(incorrectIPs) > 0 {
klog.V(4).Infof("service change tracker(%v) ignored the following load balancer source ranges(%s) for service %v/%v as they don't match IPFamily", sct.ipFamily, strings.Join(incorrectIPs, ","), service.Namespace, service.Name)
}
// Obtain Load Balancer Ingress IPs
var ips []string
for _, ing := range service.Status.LoadBalancer.Ingress {
ips = append(ips, ing.IP)
}
if len(ips) > 0 {
correctIPs, incorrectIPs := utilproxy.FilterIncorrectIPVersion(ips, sct.ipFamily)
if len(incorrectIPs) > 0 {
klog.V(4).Infof("service change tracker(%v) ignored the following load balancer(%s) ingress ips for service %v/%v as they don't match IPFamily", sct.ipFamily, strings.Join(incorrectIPs, ","), service.Namespace, service.Name)
}
// Create the LoadBalancerStatus with the filtered IPs
for _, ip := range correctIPs {
info.loadBalancerStatus.Ingress = append(info.loadBalancerStatus.Ingress, v1.LoadBalancerIngress{IP: ip})
}
}
@ -224,18 +222,18 @@ type ServiceChangeTracker struct {
// makeServiceInfo allows proxier to inject customized information when processing service.
makeServiceInfo makeServicePortFunc
processServiceMapChange processServiceMapChangeFunc
// isIPv6Mode indicates if change tracker is under IPv6/IPv4 mode. Nil means not applicable.
isIPv6Mode *bool
recorder record.EventRecorder
ipFamily v1.IPFamily
recorder record.EventRecorder
}
// NewServiceChangeTracker initializes a ServiceChangeTracker
func NewServiceChangeTracker(makeServiceInfo makeServicePortFunc, isIPv6Mode *bool, recorder record.EventRecorder, processServiceMapChange processServiceMapChangeFunc) *ServiceChangeTracker {
func NewServiceChangeTracker(makeServiceInfo makeServicePortFunc, ipFamily v1.IPFamily, recorder record.EventRecorder, processServiceMapChange processServiceMapChangeFunc) *ServiceChangeTracker {
return &ServiceChangeTracker{
items: make(map[types.NamespacedName]*serviceChange),
makeServiceInfo: makeServiceInfo,
isIPv6Mode: isIPv6Mode,
recorder: recorder,
ipFamily: ipFamily,
processServiceMapChange: processServiceMapChange,
}
}
@ -322,13 +320,9 @@ func (sct *ServiceChangeTracker) serviceToServiceMap(service *v1.Service) Servic
return nil
}
if len(service.Spec.ClusterIP) != 0 {
// Filter out the incorrect IP version case.
// If ClusterIP on service has incorrect IP version, service itself will be ignored.
if sct.isIPv6Mode != nil && utilnet.IsIPv6String(service.Spec.ClusterIP) != *sct.isIPv6Mode {
utilproxy.LogAndEmitIncorrectIPVersionEvent(sct.recorder, "clusterIP", service.Spec.ClusterIP, service.Namespace, service.Name, service.UID)
return nil
}
clusterIP := utilproxy.GetClusterIPByFamily(sct.ipFamily, service)
if clusterIP == "" {
return nil
}
serviceMap := make(ServiceMap)

115
pkg/proxy/service_test.go
View File

@ -85,10 +85,6 @@ func makeServicePortName(ns, name, port string, protocol v1.Protocol) ServicePor
}
func TestServiceToServiceMap(t *testing.T) {
svcTracker := NewServiceChangeTracker(nil, nil, nil, nil)
trueVal := true
falseVal := false
testClusterIPv4 := "10.0.0.1"
testExternalIPv4 := "8.8.8.8"
testSourceRangeIPv4 := "0.0.0.0/1"
@ -97,18 +93,22 @@ func TestServiceToServiceMap(t *testing.T) {
testSourceRangeIPv6 := "2001:db8::/32"
testCases := []struct {
desc string
service *v1.Service
expected map[ServicePortName]*BaseServiceInfo
isIPv6Mode *bool
desc string
service *v1.Service
expected map[ServicePortName]*BaseServiceInfo
ipFamily v1.IPFamily
}{
{
desc: "nothing",
ipFamily: v1.IPv4Protocol,
service: nil,
expected: map[ServicePortName]*BaseServiceInfo{},
},
{
desc: "headless service",
desc: "headless service",
ipFamily: v1.IPv4Protocol,
service: makeTestService("ns2", "headless", func(svc *v1.Service) {
svc.Spec.Type = v1.ServiceTypeClusterIP
svc.Spec.ClusterIP = v1.ClusterIPNone
@ -117,7 +117,9 @@ func TestServiceToServiceMap(t *testing.T) {
expected: map[ServicePortName]*BaseServiceInfo{},
},
{
desc: "headless sctp service",
desc: "headless sctp service",
ipFamily: v1.IPv4Protocol,
service: makeTestService("ns2", "headless", func(svc *v1.Service) {
svc.Spec.Type = v1.ServiceTypeClusterIP
svc.Spec.ClusterIP = v1.ClusterIPNone
@ -126,7 +128,9 @@ func TestServiceToServiceMap(t *testing.T) {
expected: map[ServicePortName]*BaseServiceInfo{},
},
{
desc: "headless service without port",
desc: "headless service without port",
ipFamily: v1.IPv4Protocol,
service: makeTestService("ns2", "headless-without-port", func(svc *v1.Service) {
svc.Spec.Type = v1.ServiceTypeClusterIP
svc.Spec.ClusterIP = v1.ClusterIPNone
@ -134,7 +138,9 @@ func TestServiceToServiceMap(t *testing.T) {
expected: map[ServicePortName]*BaseServiceInfo{},
},
{
desc: "cluster ip service",
desc: "cluster ip service",
ipFamily: v1.IPv4Protocol,
service: makeTestService("ns2", "cluster-ip", func(svc *v1.Service) {
svc.Spec.Type = v1.ServiceTypeClusterIP
svc.Spec.ClusterIP = "172.16.55.4"
@ -147,7 +153,9 @@ func TestServiceToServiceMap(t *testing.T) {
},
},
{
desc: "nodeport service",
desc: "nodeport service",
ipFamily: v1.IPv4Protocol,
service: makeTestService("ns2", "node-port", func(svc *v1.Service) {
svc.Spec.Type = v1.ServiceTypeNodePort
svc.Spec.ClusterIP = "172.16.55.10"
@ -160,7 +168,9 @@ func TestServiceToServiceMap(t *testing.T) {
},
},
{
desc: "load balancer service",
desc: "load balancer service",
ipFamily: v1.IPv4Protocol,
service: makeTestService("ns1", "load-balancer", func(svc *v1.Service) {
svc.Spec.Type = v1.ServiceTypeLoadBalancer
svc.Spec.ClusterIP = "172.16.55.11"
@ -179,7 +189,9 @@ func TestServiceToServiceMap(t *testing.T) {
},
},
{
desc: "load balancer service with only local traffic policy",
desc: "load balancer service with only local traffic policy",
ipFamily: v1.IPv4Protocol,
service: makeTestService("ns1", "only-local-load-balancer", func(svc *v1.Service) {
svc.Spec.Type = v1.ServiceTypeLoadBalancer
svc.Spec.ClusterIP = "172.16.55.12"
@ -200,7 +212,9 @@ func TestServiceToServiceMap(t *testing.T) {
},
},
{
desc: "external name service",
desc: "external name service",
ipFamily: v1.IPv4Protocol,
service: makeTestService("ns2", "external-name", func(svc *v1.Service) {
svc.Spec.Type = v1.ServiceTypeExternalName
svc.Spec.ClusterIP = "172.16.55.4" // Should be ignored
@ -210,7 +224,9 @@ func TestServiceToServiceMap(t *testing.T) {
expected: map[ServicePortName]*BaseServiceInfo{},
},
{
desc: "service with ipv6 clusterIP under ipv4 mode, service should be filtered",
desc: "service with ipv6 clusterIP under ipv4 mode, service should be filtered",
ipFamily: v1.IPv4Protocol,
service: &v1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "invalidIPv6InIPV4Mode",
@ -235,10 +251,11 @@ func TestServiceToServiceMap(t *testing.T) {
},
},
},
isIPv6Mode: &falseVal,
},
{
desc: "service with ipv4 clusterIP under ipv6 mode, service should be filtered",
desc: "service with ipv4 clusterIP under ipv6 mode, service should be filtered",
ipFamily: v1.IPv6Protocol,
service: &v1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "invalidIPv4InIPV6Mode",
@ -263,10 +280,11 @@ func TestServiceToServiceMap(t *testing.T) {
},
},
},
isIPv6Mode: &trueVal,
},
{
desc: "service with ipv4 configurations under ipv4 mode",
desc: "service with ipv4 configurations under ipv4 mode",
ipFamily: v1.IPv4Protocol,
service: &v1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "validIPv4",
@ -300,10 +318,11 @@ func TestServiceToServiceMap(t *testing.T) {
info.loadBalancerStatus.Ingress = []v1.LoadBalancerIngress{{IP: testExternalIPv4}}
}),
},
isIPv6Mode: &falseVal,
},
{
desc: "service with ipv6 configurations under ipv6 mode",
desc: "service with ipv6 configurations under ipv6 mode",
ipFamily: v1.IPv6Protocol,
service: &v1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "validIPv6",
@ -337,10 +356,11 @@ func TestServiceToServiceMap(t *testing.T) {
info.loadBalancerStatus.Ingress = []v1.LoadBalancerIngress{{IP: testExternalIPv6}}
}),
},
isIPv6Mode: &trueVal,
},
{
desc: "service with both ipv4 and ipv6 configurations under ipv4 mode, ipv6 fields should be filtered",
desc: "service with both ipv4 and ipv6 configurations under ipv4 mode, ipv6 fields should be filtered",
ipFamily: v1.IPv4Protocol,
service: &v1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "filterIPv6InIPV4Mode",
@ -374,10 +394,11 @@ func TestServiceToServiceMap(t *testing.T) {
info.loadBalancerStatus.Ingress = []v1.LoadBalancerIngress{{IP: testExternalIPv4}}
}),
},
isIPv6Mode: &falseVal,
},
{
desc: "service with both ipv4 and ipv6 configurations under ipv6 mode, ipv4 fields should be filtered",
desc: "service with both ipv4 and ipv6 configurations under ipv6 mode, ipv4 fields should be filtered",
ipFamily: v1.IPv6Protocol,
service: &v1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "filterIPv4InIPV6Mode",
@ -411,7 +432,6 @@ func TestServiceToServiceMap(t *testing.T) {
info.loadBalancerStatus.Ingress = []v1.LoadBalancerIngress{{IP: testExternalIPv6}}
}),
},
isIPv6Mode: &trueVal,
},
{
desc: "service with extra space in LoadBalancerSourceRanges",
@ -437,21 +457,24 @@ func TestServiceToServiceMap(t *testing.T) {
info.loadBalancerSourceRanges = []string{"10.1.2.0/28"}
}),
},
isIPv6Mode: &falseVal,
},
}
for _, tc := range testCases {
t.Run(tc.desc, func(t *testing.T) {
svcTracker.isIPv6Mode = tc.isIPv6Mode
svcTracker := NewServiceChangeTracker(nil, tc.ipFamily, nil, nil)
// outputs
newServices := svcTracker.serviceToServiceMap(tc.service)
if len(newServices) != len(tc.expected) {
t.Errorf("expected %d new, got %d: %v", len(tc.expected), len(newServices), spew.Sdump(newServices))
t.Fatalf("expected %d new, got %d: %v", len(tc.expected), len(newServices), spew.Sdump(newServices))
}
for svcKey, expectedInfo := range tc.expected {
svcInfo, _ := newServices[svcKey].(*BaseServiceInfo)
svcInfo, exists := newServices[svcKey].(*BaseServiceInfo)
if !exists {
t.Fatalf("[%s] expected to find key %s", tc.desc, svcKey)
}
if !svcInfo.clusterIP.Equal(expectedInfo.clusterIP) ||
svcInfo.port != expectedInfo.port ||
svcInfo.protocol != expectedInfo.protocol ||
@ -459,7 +482,19 @@ func TestServiceToServiceMap(t *testing.T) {
!sets.NewString(svcInfo.externalIPs...).Equal(sets.NewString(expectedInfo.externalIPs...)) ||
!sets.NewString(svcInfo.loadBalancerSourceRanges...).Equal(sets.NewString(expectedInfo.loadBalancerSourceRanges...)) ||
!reflect.DeepEqual(svcInfo.loadBalancerStatus, expectedInfo.loadBalancerStatus) {
t.Errorf("expected new[%v]to be %v, got %v", svcKey, expectedInfo, *svcInfo)
t.Errorf("[%s] expected new[%v]to be %v, got %v", tc.desc, svcKey, expectedInfo, *svcInfo)
}
for svcKey, expectedInfo := range tc.expected {
svcInfo, _ := newServices[svcKey].(*BaseServiceInfo)
if !svcInfo.clusterIP.Equal(expectedInfo.clusterIP) ||
svcInfo.port != expectedInfo.port ||
svcInfo.protocol != expectedInfo.protocol ||
svcInfo.healthCheckNodePort != expectedInfo.healthCheckNodePort ||
!sets.NewString(svcInfo.externalIPs...).Equal(sets.NewString(expectedInfo.externalIPs...)) ||
!sets.NewString(svcInfo.loadBalancerSourceRanges...).Equal(sets.NewString(expectedInfo.loadBalancerSourceRanges...)) ||
!reflect.DeepEqual(svcInfo.loadBalancerStatus, expectedInfo.loadBalancerStatus) {
t.Errorf("expected new[%v]to be %v, got %v", svcKey, expectedInfo, *svcInfo)
}
}
}
})
@ -474,12 +509,12 @@ type FakeProxier struct {
hostname string
}
func newFakeProxier() *FakeProxier {
func newFakeProxier(ipFamily v1.IPFamily) *FakeProxier {
return &FakeProxier{
serviceMap: make(ServiceMap),
serviceChanges: NewServiceChangeTracker(nil, nil, nil, nil),
serviceChanges: NewServiceChangeTracker(nil, ipFamily, nil, nil),
endpointsMap: make(EndpointsMap),
endpointsChanges: NewEndpointChangeTracker(testHostname, nil, nil, nil, false, nil),
endpointsChanges: NewEndpointChangeTracker(testHostname, nil, ipFamily, nil, false, nil),
}
}
@ -502,7 +537,7 @@ func (fake *FakeProxier) deleteService(service *v1.Service) {
}
func TestUpdateServiceMapHeadless(t *testing.T) {
fp := newFakeProxier()
fp := newFakeProxier(v1.IPv4Protocol)
makeServiceMap(fp,
makeTestService("ns2", "headless", func(svc *v1.Service) {
@ -533,7 +568,7 @@ func TestUpdateServiceMapHeadless(t *testing.T) {
}
func TestUpdateServiceTypeExternalName(t *testing.T) {
fp := newFakeProxier()
fp := newFakeProxier(v1.IPv4Protocol)
makeServiceMap(fp,
makeTestService("ns2", "external-name", func(svc *v1.Service) {
@ -558,7 +593,7 @@ func TestUpdateServiceTypeExternalName(t *testing.T) {
}
func TestBuildServiceMapAddRemove(t *testing.T) {
fp := newFakeProxier()
fp := newFakeProxier(v1.IPv4Protocol)
services := []*v1.Service{
makeTestService("ns2", "cluster-ip", func(svc *v1.Service) {
@ -661,7 +696,7 @@ func TestBuildServiceMapAddRemove(t *testing.T) {
}
func TestBuildServiceMapServiceUpdate(t *testing.T) {
fp := newFakeProxier()
fp := newFakeProxier(v1.IPv4Protocol)
servicev1 := makeTestService("ns1", "svc1", func(svc *v1.Service) {
svc.Spec.Type = v1.ServiceTypeClusterIP

35
pkg/proxy/util/utils.go
View File

@ -256,13 +256,13 @@ func LogAndEmitIncorrectIPVersionEvent(recorder record.EventRecorder, fieldName,
}
// FilterIncorrectIPVersion filters out the incorrect IP version case from a slice of IP strings.
func FilterIncorrectIPVersion(ipStrings []string, isIPv6Mode bool) ([]string, []string) {
return filterWithCondition(ipStrings, isIPv6Mode, utilnet.IsIPv6String)
func FilterIncorrectIPVersion(ipStrings []string, ipfamily v1.IPFamily) ([]string, []string) {
return filterWithCondition(ipStrings, (ipfamily == v1.IPv6Protocol), utilnet.IsIPv6String)
}
// FilterIncorrectCIDRVersion filters out the incorrect IP version case from a slice of CIDR strings.
func FilterIncorrectCIDRVersion(ipStrings []string, isIPv6Mode bool) ([]string, []string) {
return filterWithCondition(ipStrings, isIPv6Mode, utilnet.IsIPv6CIDRString)
func FilterIncorrectCIDRVersion(ipStrings []string, ipfamily v1.IPFamily) ([]string, []string) {
return filterWithCondition(ipStrings, (ipfamily == v1.IPv6Protocol), utilnet.IsIPv6CIDRString)
}
func filterWithCondition(strs []string, expectedCondition bool, conditionFunc func(string) bool) ([]string, []string) {
@ -376,3 +376,30 @@ func NewFilteredDialContext(wrapped DialContext, resolv Resolver, opts *Filtered
return wrapped(ctx, network, address)
}
}
// GetClusterIPByFamily returns a service clusterip by family
func GetClusterIPByFamily(ipFamily v1.IPFamily, service *v1.Service) string {
// allowing skew
if len(service.Spec.IPFamilies) == 0 {
if len(service.Spec.ClusterIP) == 0 || service.Spec.ClusterIP == v1.ClusterIPNone {
return ""
}
IsIPv6Family := (ipFamily == v1.IPv6Protocol)
if IsIPv6Family == utilnet.IsIPv6String(service.Spec.ClusterIP) {
return service.Spec.ClusterIP
}
return ""
}
for idx, family := range service.Spec.IPFamilies {
if family == ipFamily {
if idx < len(service.Spec.ClusterIPs) {
return service.Spec.ClusterIPs[idx]
}
}
}
return ""
}

165
pkg/proxy/util/utils_test.go
View File

@ -649,7 +649,11 @@ func TestFilterIncorrectIPVersion(t *testing.T) {
for _, testcase := range testCases {
t.Run(testcase.desc, func(t *testing.T) {
correct, incorrect := FilterIncorrectIPVersion(testcase.ipString, testcase.wantIPv6)
ipFamily := v1.IPv4Protocol
if testcase.wantIPv6 {
ipFamily = v1.IPv6Protocol
}
correct, incorrect := FilterIncorrectIPVersion(testcase.ipString, ipFamily)
if !reflect.DeepEqual(testcase.expectCorrect, correct) {
t.Errorf("Test %v failed: expected %v, got %v", testcase.desc, testcase.expectCorrect, correct)
}
@ -659,3 +663,162 @@ func TestFilterIncorrectIPVersion(t *testing.T) {
})
}
}
func TestGetClusterIPByFamily(t *testing.T) {
testCases := []struct {
name string
service v1.Service
requestFamily v1.IPFamily
expectedResult string
}{
{
name: "old style service ipv4. want ipv4",
requestFamily: v1.IPv4Protocol,
expectedResult: "10.0.0.10",
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIP: "10.0.0.10",
},
},
},
{
name: "old style service ipv4. want ipv6",
requestFamily: v1.IPv6Protocol,
expectedResult: "",
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIP: "10.0.0.10",
},
},
},
{
name: "old style service ipv6. want ipv6",
requestFamily: v1.IPv6Protocol,
expectedResult: "2000::1",
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIP: "2000::1",
},
},
},
{
name: "old style service ipv6. want ipv4",
requestFamily: v1.IPv4Protocol,
expectedResult: "",
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIP: "2000::1",
},
},
},
{
name: "service single stack ipv4. want ipv4",
requestFamily: v1.IPv4Protocol,
expectedResult: "10.0.0.10",
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIPs: []string{"10.0.0.10"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
},
{
name: "service single stack ipv4. want ipv6",
requestFamily: v1.IPv6Protocol,
expectedResult: "",
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIPs: []string{"10.0.0.10"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
},
{
name: "service single stack ipv6. want ipv6",
requestFamily: v1.IPv6Protocol,
expectedResult: "2000::1",
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIPs: []string{"2000::1"},
IPFamilies: []v1.IPFamily{v1.IPv6Protocol},
},
},
},
{
name: "service single stack ipv6. want ipv4",
requestFamily: v1.IPv4Protocol,
expectedResult: "",
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIPs: []string{"2000::1"},
IPFamilies: []v1.IPFamily{v1.IPv6Protocol},
},
},
},
// dual stack
{
name: "service dual stack ipv4,6. want ipv4",
requestFamily: v1.IPv4Protocol,
expectedResult: "10.0.0.10",
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIPs: []string{"10.0.0.10", "2000::1"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
},
},
},
{
name: "service dual stack ipv4,6. want ipv6",
requestFamily: v1.IPv6Protocol,
expectedResult: "2000::1",
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIPs: []string{"10.0.0.10", "2000::1"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
},
},
},
{
name: "service dual stack ipv6,4. want ipv6",
requestFamily: v1.IPv6Protocol,
expectedResult: "2000::1",
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIPs: []string{"2000::1", "10.0.0.10"},
IPFamilies: []v1.IPFamily{v1.IPv6Protocol, v1.IPv4Protocol},
},
},
},
{
name: "service dual stack ipv6,4. want ipv4",
requestFamily: v1.IPv4Protocol,
expectedResult: "10.0.0.10",
service: v1.Service{
Spec: v1.ServiceSpec{
ClusterIPs: []string{"2000::1", "10.0.0.10"},
IPFamilies: []v1.IPFamily{v1.IPv6Protocol, v1.IPv4Protocol},
},
},
},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
ip := GetClusterIPByFamily(testCase.requestFamily, &testCase.service)
if ip != testCase.expectedResult {
t.Fatalf("expected ip:%v got %v", testCase.expectedResult, ip)
}
})
}
}

8
pkg/proxy/winkernel/proxier.go
View File

@ -604,8 +604,12 @@ func NewProxier(
isIPv6Mode: isIPv6,
}
serviceChanges := proxy.NewServiceChangeTracker(proxier.newServiceInfo, &isIPv6, recorder, proxier.serviceMapChange)
endPointChangeTracker := proxy.NewEndpointChangeTracker(hostname, proxier.newEndpointInfo, &isIPv6, recorder, endpointSlicesEnabled, proxier.endpointsMapChange)
ipFamily := v1.IPv4Protocol
if isIPv6 {
ipFamily = v1.IPv6Protocol
}
serviceChanges := proxy.NewServiceChangeTracker(proxier.newServiceInfo, ipFamily, recorder, proxier.serviceMapChange)
endPointChangeTracker := proxy.NewEndpointChangeTracker(hostname, proxier.newEndpointInfo, ipFamily, recorder, endpointSlicesEnabled, proxier.endpointsMapChange)
proxier.endpointsChanges = endPointChangeTracker
proxier.serviceChanges = serviceChanges

5
pkg/proxy/winkernel/proxier_test.go
View File

@ -124,9 +124,8 @@ func NewFakeProxier(syncPeriod time.Duration, minSyncPeriod time.Duration, clust
endPointsRefCount: make(endPointsReferenceCountMap),
}
isIPv6 := false
serviceChanges := proxy.NewServiceChangeTracker(proxier.newServiceInfo, &isIPv6, nil, proxier.serviceMapChange)
endpointChangeTracker := proxy.NewEndpointChangeTracker(hostname, proxier.newEndpointInfo, &isIPv6, nil, endpointSliceEnabled, proxier.endpointsMapChange)
serviceChanges := proxy.NewServiceChangeTracker(proxier.newServiceInfo, v1.IPv4Protocol, nil, proxier.serviceMapChange)
endpointChangeTracker := proxy.NewEndpointChangeTracker(hostname, proxier.newEndpointInfo, v1.IPv4Protocol, nil, endpointSliceEnabled, proxier.endpointsMapChange)
proxier.endpointsChanges = endpointChangeTracker
proxier.serviceChanges = serviceChanges

2
pkg/registry/core/service/ipallocator/controller/BUILD
View File

@ -14,7 +14,6 @@ go_library(
"//pkg/api/legacyscheme:go_default_library",
"//pkg/apis/core:go_default_library",
"//pkg/apis/core/v1/helper:go_default_library",
"//pkg/features:go_default_library",
"//pkg/registry/core/rangeallocation:go_default_library",
"//pkg/registry/core/service/ipallocator:go_default_library",
"//staging/src/k8s.io/api/core/v1:go_default_library",
@ -22,7 +21,6 @@ go_library(
"//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/runtime:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library",
"//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library",
"//staging/src/k8s.io/client-go/kubernetes/typed/core/v1:go_default_library",
"//staging/src/k8s.io/client-go/tools/record:go_default_library",
"//staging/src/k8s.io/client-go/util/retry:go_default_library",

257
pkg/registry/core/service/ipallocator/controller/repair.go
View File

@ -36,9 +36,6 @@ import (
"k8s.io/kubernetes/pkg/registry/core/rangeallocation"
"k8s.io/kubernetes/pkg/registry/core/service/ipallocator"
netutil "k8s.io/utils/net"
utilfeature "k8s.io/apiserver/pkg/util/feature"
"k8s.io/kubernetes/pkg/features"
)
// Repair is a controller loop that periodically examines all service ClusterIP allocations
@ -60,13 +57,11 @@ type Repair struct {
interval time.Duration
serviceClient corev1client.ServicesGetter
network *net.IPNet
alloc rangeallocation.RangeRegistry
secondaryNetwork *net.IPNet
secondaryAlloc rangeallocation.RangeRegistry
networkByFamily map[v1.IPFamily]*net.IPNet // networks we operate on, by their family
allocatorByFamily map[v1.IPFamily]rangeallocation.RangeRegistry // allocators we use, by their family
leaks map[string]int // counter per leaked IP
recorder record.EventRecorder
leaksByFamily map[v1.IPFamily]map[string]int // counter per leaked IP per family
recorder record.EventRecorder
}
// How many times we need to detect a leak before we clean up. This is to
@ -80,17 +75,39 @@ func NewRepair(interval time.Duration, serviceClient corev1client.ServicesGetter
eventBroadcaster.StartRecordingToSink(&corev1client.EventSinkImpl{Interface: eventClient.Events("")})
recorder := eventBroadcaster.NewRecorder(legacyscheme.Scheme, v1.EventSource{Component: "ipallocator-repair-controller"})
// build *ByFamily struct members
networkByFamily := make(map[v1.IPFamily]*net.IPNet)
allocatorByFamily := make(map[v1.IPFamily]rangeallocation.RangeRegistry)
leaksByFamily := make(map[v1.IPFamily]map[string]int)
primary := v1.IPv4Protocol
secondary := v1.IPv6Protocol
if netutil.IsIPv6(network.IP) {
primary = v1.IPv6Protocol
}
networkByFamily[primary] = network
allocatorByFamily[primary] = alloc
leaksByFamily[primary] = make(map[string]int)
if secondaryNetwork != nil && secondaryNetwork.IP != nil {
if primary == v1.IPv6Protocol {
secondary = v1.IPv4Protocol
}
networkByFamily[secondary] = secondaryNetwork
allocatorByFamily[secondary] = secondaryAlloc
leaksByFamily[secondary] = make(map[string]int)
}
return &Repair{
interval: interval,
serviceClient: serviceClient,
network: network,
alloc: alloc,
secondaryNetwork: secondaryNetwork,
secondaryAlloc: secondaryAlloc,
networkByFamily: networkByFamily,
allocatorByFamily: allocatorByFamily,
leaks: map[string]int{},
recorder: recorder,
leaksByFamily: leaksByFamily,
recorder: recorder,
}
}
@ -108,29 +125,6 @@ func (c *Repair) RunOnce() error {
return retry.RetryOnConflict(retry.DefaultBackoff, c.runOnce)
}
// selectAllocForIP returns an allocator for an IP based weather it belongs to the primary or the secondary allocator
func (c *Repair) selectAllocForIP(ip net.IP, primary ipallocator.Interface, secondary ipallocator.Interface) ipallocator.Interface {
if !c.shouldWorkOnSecondary() {
return primary
}
cidr := secondary.CIDR()
if netutil.IsIPv6CIDR(&cidr) && netutil.IsIPv6(ip) {
return secondary
}
return primary
}
// shouldWorkOnSecondary returns true if the repairer should perform work for secondary network (dual stack)
func (c *Repair) shouldWorkOnSecondary() bool {
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
return false
}
return c.secondaryNetwork != nil && c.secondaryNetwork.IP != nil
}
// runOnce verifies the state of the cluster IP allocations and returns an error if an unrecoverable problem occurs.
func (c *Repair) runOnce() error {
// TODO: (per smarterclayton) if Get() or ListServices() is a weak consistency read,
@ -142,50 +136,55 @@ func (c *Repair) runOnce() error {
// If etcd server is not running we should wait for some time and fail only then. This is particularly
// important when we start apiserver and etcd at the same time.
var snapshot *api.RangeAllocation
var secondarySnapshot *api.RangeAllocation
snapshotByFamily := make(map[v1.IPFamily]*api.RangeAllocation)
storedByFamily := make(map[v1.IPFamily]ipallocator.Interface)
var stored, secondaryStored ipallocator.Interface
var err, secondaryErr error
err := wait.PollImmediate(time.Second, 10*time.Second, func() (bool, error) {
for family, allocator := range c.allocatorByFamily {
// get snapshot if it is not there
if _, ok := snapshotByFamily[family]; !ok {
snapshot, err := allocator.Get()
if err != nil {
return false, err
}
err = wait.PollImmediate(time.Second, 10*time.Second, func() (bool, error) {
var err error
snapshot, err = c.alloc.Get()
if err != nil {
return false, err
}
if c.shouldWorkOnSecondary() {
secondarySnapshot, err = c.secondaryAlloc.Get()
if err != nil {
return false, err
snapshotByFamily[family] = snapshot
}
}
return true, nil
})
if err != nil {
return fmt.Errorf("unable to refresh the service IP block: %v", err)
}
// If not yet initialized.
if snapshot.Range == "" {
snapshot.Range = c.network.String()
// ensure that ranges are assigned
for family, snapshot := range snapshotByFamily {
if snapshot.Range == "" {
snapshot.Range = c.networkByFamily[family].String()
}
}
if c.shouldWorkOnSecondary() && secondarySnapshot.Range == "" {
secondarySnapshot.Range = c.secondaryNetwork.String()
}
// Create an allocator because it is easy to use.
for family, snapshot := range snapshotByFamily {
stored, err := ipallocator.NewFromSnapshot(snapshot)
if err != nil {
return fmt.Errorf("unable to rebuild allocator from snapshots for family:%v with error:%v", family, err)
}
stored, err = ipallocator.NewFromSnapshot(snapshot)
if c.shouldWorkOnSecondary() {
secondaryStored, secondaryErr = ipallocator.NewFromSnapshot(secondarySnapshot)
storedByFamily[family] = stored
}
if err != nil || secondaryErr != nil {
return fmt.Errorf("unable to rebuild allocator from snapshots: %v", err)
}
rebuiltByFamily := make(map[v1.IPFamily]*ipallocator.Range)
for family, network := range c.networkByFamily {
rebuilt, err := ipallocator.NewCIDRRange(network)
if err != nil {
return fmt.Errorf("unable to create CIDR range for family %v: %v", family, err)
}
rebuiltByFamily[family] = rebuilt
}
// We explicitly send no resource version, since the resource version
// of 'snapshot' is from a different collection, it's not comparable to
// the service collection. The caching layer keeps per-collection RVs,
@ -196,18 +195,11 @@ func (c *Repair) runOnce() error {
return fmt.Errorf("unable to refresh the service IP block: %v", err)
}
var rebuilt, secondaryRebuilt *ipallocator.Range
rebuilt, err = ipallocator.NewCIDRRange(c.network)
if err != nil {
return fmt.Errorf("unable to create CIDR range: %v", err)
}
if c.shouldWorkOnSecondary() {
secondaryRebuilt, err = ipallocator.NewCIDRRange(c.secondaryNetwork)
}
if err != nil {
return fmt.Errorf("unable to create CIDR range: %v", err)
getFamilyByIP := func(ip net.IP) v1.IPFamily {
if netutil.IsIPv6(ip) {
return v1.IPv6Protocol
}
return v1.IPv4Protocol
}
// Check every Service's ClusterIP, and rebuild the state as we think it should be.
@ -216,64 +208,72 @@ func (c *Repair) runOnce() error {
// didn't need a cluster IP
continue
}
ip := net.ParseIP(svc.Spec.ClusterIP)
if ip == nil {
// cluster IP is corrupt
c.recorder.Eventf(&svc, v1.EventTypeWarning, "ClusterIPNotValid", "Cluster IP %s is not a valid IP; please recreate service", svc.Spec.ClusterIP)
runtime.HandleError(fmt.Errorf("the cluster IP %s for service %s/%s is not a valid IP; please recreate", svc.Spec.ClusterIP, svc.Name, svc.Namespace))
continue
}
// mark it as in-use
actualAlloc := c.selectAllocForIP(ip, rebuilt, secondaryRebuilt)
switch err := actualAlloc.Allocate(ip); err {
case nil:
actualStored := c.selectAllocForIP(ip, stored, secondaryStored)
if actualStored.Has(ip) {
// remove it from the old set, so we can find leaks
actualStored.Release(ip)
} else {
// cluster IP doesn't seem to be allocated
c.recorder.Eventf(&svc, v1.EventTypeWarning, "ClusterIPNotAllocated", "Cluster IP %s is not allocated; repairing", ip)
runtime.HandleError(fmt.Errorf("the cluster IP %s for service %s/%s is not allocated; repairing", ip, svc.Name, svc.Namespace))
for _, ip := range svc.Spec.ClusterIPs {
ip := net.ParseIP(ip)
if ip == nil {
// cluster IP is corrupt
c.recorder.Eventf(&svc, v1.EventTypeWarning, "ClusterIPNotValid", "Cluster IP %s is not a valid IP; please recreate service", ip)
runtime.HandleError(fmt.Errorf("the cluster IP %s for service %s/%s is not a valid IP; please recreate", ip, svc.Name, svc.Namespace))
continue
}
family := getFamilyByIP(ip)
if _, ok := rebuiltByFamily[family]; !ok {
// this service is using an IPFamily no longer configured on cluster
c.recorder.Eventf(&svc, v1.EventTypeWarning, "ClusterIPNotValid", "Cluster IP %s(%s) is of ip family that is no longer configured on cluster; please recreate service", ip, family)
runtime.HandleError(fmt.Errorf("the cluster IP %s(%s) for service %s/%s is of ip family that is no longer configured on cluster; please recreate", ip, family, svc.Name, svc.Namespace))
continue
}
// mark it as in-use
actualAlloc := rebuiltByFamily[family]
switch err := actualAlloc.Allocate(ip); err {
case nil:
actualStored := storedByFamily[family]
if actualStored.Has(ip) {
// remove it from the old set, so we can find leaks
actualStored.Release(ip)
} else {
// cluster IP doesn't seem to be allocated
c.recorder.Eventf(&svc, v1.EventTypeWarning, "ClusterIPNotAllocated", "Cluster IP [%v]:%s is not allocated; repairing", family, ip)
runtime.HandleError(fmt.Errorf("the cluster IP [%v]:%s for service %s/%s is not allocated; repairing", family, ip, svc.Name, svc.Namespace))
}
delete(c.leaksByFamily[family], ip.String()) // it is used, so it can't be leaked
case ipallocator.ErrAllocated:
// cluster IP is duplicate
c.recorder.Eventf(&svc, v1.EventTypeWarning, "ClusterIPAlreadyAllocated", "Cluster IP [%v]:%s was assigned to multiple services; please recreate service", family, ip)
runtime.HandleError(fmt.Errorf("the cluster IP [%v]:%s for service %s/%s was assigned to multiple services; please recreate", family, ip, svc.Name, svc.Namespace))
case err.(*ipallocator.ErrNotInRange):
// cluster IP is out of range
c.recorder.Eventf(&svc, v1.EventTypeWarning, "ClusterIPOutOfRange", "Cluster IP [%v]:%s is not within the service CIDR %s; please recreate service", family, ip, c.networkByFamily[family])
runtime.HandleError(fmt.Errorf("the cluster IP [%v]:%s for service %s/%s is not within the service CIDR %s; please recreate", family, ip, svc.Name, svc.Namespace, c.networkByFamily[family]))
case ipallocator.ErrFull:
// somehow we are out of IPs
cidr := actualAlloc.CIDR()
c.recorder.Eventf(&svc, v1.EventTypeWarning, "ServiceCIDRFull", "Service CIDR %v is full; you must widen the CIDR in order to create new services for Cluster IP [%v]:%s", cidr, family, ip)
return fmt.Errorf("the service CIDR %v is full; you must widen the CIDR in order to create new services for Cluster IP [%v]:%s", cidr, family, ip)
default:
c.recorder.Eventf(&svc, v1.EventTypeWarning, "UnknownError", "Unable to allocate cluster IP [%v]:%s due to an unknown error", family, ip)
return fmt.Errorf("unable to allocate cluster IP [%v]:%s for service %s/%s due to an unknown error, exiting: %v", family, ip, svc.Name, svc.Namespace, err)
}
delete(c.leaks, ip.String()) // it is used, so it can't be leaked
case ipallocator.ErrAllocated:
// cluster IP is duplicate
c.recorder.Eventf(&svc, v1.EventTypeWarning, "ClusterIPAlreadyAllocated", "Cluster IP %s was assigned to multiple services; please recreate service", ip)
runtime.HandleError(fmt.Errorf("the cluster IP %s for service %s/%s was assigned to multiple services; please recreate", ip, svc.Name, svc.Namespace))
case err.(*ipallocator.ErrNotInRange):
// cluster IP is out of range
c.recorder.Eventf(&svc, v1.EventTypeWarning, "ClusterIPOutOfRange", "Cluster IP %s is not within the service CIDR %s; please recreate service", ip, c.network)
runtime.HandleError(fmt.Errorf("the cluster IP %s for service %s/%s is not within the service CIDR %s; please recreate", ip, svc.Name, svc.Namespace, c.network))
case ipallocator.ErrFull:
// somehow we are out of IPs
cidr := actualAlloc.CIDR()
c.recorder.Eventf(&svc, v1.EventTypeWarning, "ServiceCIDRFull", "Service CIDR %v is full; you must widen the CIDR in order to create new services", cidr)
return fmt.Errorf("the service CIDR %v is full; you must widen the CIDR in order to create new services", cidr)
default:
c.recorder.Eventf(&svc, v1.EventTypeWarning, "UnknownError", "Unable to allocate cluster IP %s due to an unknown error", ip)
return fmt.Errorf("unable to allocate cluster IP %s for service %s/%s due to an unknown error, exiting: %v", ip, svc.Name, svc.Namespace, err)
}
}
c.checkLeaked(stored, rebuilt)
if c.shouldWorkOnSecondary() {
c.checkLeaked(secondaryStored, secondaryRebuilt)
// leak check
for family, leaks := range c.leaksByFamily {
c.checkLeaked(leaks, storedByFamily[family], rebuiltByFamily[family])
}
// save logic
// Blast the rebuilt state into storage.
err = c.saveSnapShot(rebuilt, c.alloc, snapshot)
if err != nil {
return err
}
if c.shouldWorkOnSecondary() {
err := c.saveSnapShot(secondaryRebuilt, c.secondaryAlloc, secondarySnapshot)
for family, rebuilt := range rebuiltByFamily {
err = c.saveSnapShot(rebuilt, c.allocatorByFamily[family], snapshotByFamily[family])
if err != nil {
return nil
return err
}
}
return nil
}
@ -291,10 +291,10 @@ func (c *Repair) saveSnapShot(rebuilt *ipallocator.Range, alloc rangeallocation.
return nil
}
func (c *Repair) checkLeaked(stored ipallocator.Interface, rebuilt *ipallocator.Range) {
func (c *Repair) checkLeaked(leaks map[string]int, stored ipallocator.Interface, rebuilt *ipallocator.Range) {
// Check for IPs that are left in the old set. They appear to have been leaked.
stored.ForEach(func(ip net.IP) {
count, found := c.leaks[ip.String()]
count, found := leaks[ip.String()]
switch {
case !found:
// flag it to be cleaned up after any races (hopefully) are gone
@ -303,7 +303,7 @@ func (c *Repair) checkLeaked(stored ipallocator.Interface, rebuilt *ipallocator.
fallthrough
case count > 0:
// pretend it is still in use until count expires
c.leaks[ip.String()] = count - 1
leaks[ip.String()] = count - 1
if err := rebuilt.Allocate(ip); err != nil {
runtime.HandleError(fmt.Errorf("the cluster IP %s may have leaked, but can not be allocated: %v", ip, err))
}
@ -312,5 +312,4 @@ func (c *Repair) checkLeaked(stored ipallocator.Interface, rebuilt *ipallocator.
runtime.HandleError(fmt.Errorf("the cluster IP %s appears to have leaked: cleaning up", ip))
}
})
}

211
pkg/registry/core/service/ipallocator/controller/repair_test.go
View File

@ -23,6 +23,7 @@ import (
"testing"
corev1 "k8s.io/api/core/v1"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes/fake"
api "k8s.io/kubernetes/pkg/apis/core"
@ -147,27 +148,49 @@ func TestRepairWithExisting(t *testing.T) {
fakeClient := fake.NewSimpleClientset(
&corev1.Service{
ObjectMeta: metav1.ObjectMeta{Namespace: "one", Name: "one"},
Spec: corev1.ServiceSpec{ClusterIP: "192.168.1.1"},
Spec: corev1.ServiceSpec{
ClusterIP: "192.168.1.1",
ClusterIPs: []string{"192.168.1.1"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
&corev1.Service{
ObjectMeta: metav1.ObjectMeta{Namespace: "two", Name: "two"},
Spec: corev1.ServiceSpec{ClusterIP: "192.168.1.100"},
Spec: corev1.ServiceSpec{
ClusterIP: "192.168.1.100",
ClusterIPs: []string{"192.168.1.100"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
&corev1.Service{ // outside CIDR, will be dropped
ObjectMeta: metav1.ObjectMeta{Namespace: "three", Name: "three"},
Spec: corev1.ServiceSpec{ClusterIP: "192.168.0.1"},
Spec: corev1.ServiceSpec{
ClusterIP: "192.168.0.1",
ClusterIPs: []string{"192.168.0.1"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
&corev1.Service{ // empty, ignored
ObjectMeta: metav1.ObjectMeta{Namespace: "four", Name: "four"},
Spec: corev1.ServiceSpec{ClusterIP: ""},
Spec: corev1.ServiceSpec{
ClusterIP: "",
ClusterIPs: []string{""},
},
},
&corev1.Service{ // duplicate, dropped
ObjectMeta: metav1.ObjectMeta{Namespace: "five", Name: "five"},
Spec: corev1.ServiceSpec{ClusterIP: "192.168.1.1"},
Spec: corev1.ServiceSpec{
ClusterIP: "192.168.1.1",
ClusterIPs: []string{"192.168.1.1"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
&corev1.Service{ // headless
ObjectMeta: metav1.ObjectMeta{Namespace: "six", Name: "six"},
Spec: corev1.ServiceSpec{ClusterIP: "None"},
Spec: corev1.ServiceSpec{
ClusterIP: "None",
ClusterIPs: []string{"None"},
},
},
)
@ -192,7 +215,7 @@ func TestRepairWithExisting(t *testing.T) {
t.Errorf("unexpected ipallocator state: %#v", after)
}
if free := after.Free(); free != 252 {
t.Errorf("unexpected ipallocator state: %d free", free)
t.Errorf("unexpected ipallocator state: %d free (expected 252)", free)
}
}
@ -229,44 +252,38 @@ func makeIPNet(cidr string) *net.IPNet {
}
func TestShouldWorkOnSecondary(t *testing.T) {
testCases := []struct {
name string
enableDualStack bool
expectedResult bool
primaryNet *net.IPNet
secondaryNet *net.IPNet
name string
expectedFamilies []v1.IPFamily
primaryNet *net.IPNet
secondaryNet *net.IPNet
}{
{
name: "not a dual stack, primary only",
enableDualStack: false,
expectedResult: false,
primaryNet: makeIPNet("10.0.0.0/16"),
secondaryNet: nil,
name: "primary only (v4)",
expectedFamilies: []v1.IPFamily{v1.IPv4Protocol},
primaryNet: makeIPNet("10.0.0.0/16"),
secondaryNet: nil,
},
{
name: "not a dual stack, primary and secondary provided",
enableDualStack: false,
expectedResult: false,
primaryNet: makeIPNet("10.0.0.0/16"),
secondaryNet: makeIPNet("2000::/120"),
name: "primary only (v6)",
expectedFamilies: []v1.IPFamily{v1.IPv6Protocol},
primaryNet: makeIPNet("2000::/120"),
secondaryNet: nil,
},
{
name: "dual stack, primary only",
enableDualStack: true,
expectedResult: false,
primaryNet: makeIPNet("10.0.0.0/16"),
secondaryNet: nil,
name: "primary and secondary provided (v4,v6)",
expectedFamilies: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
primaryNet: makeIPNet("10.0.0.0/16"),
secondaryNet: makeIPNet("2000::/120"),
},
{
name: "dual stack, primary and secondary",
enableDualStack: true,
expectedResult: true,
primaryNet: makeIPNet("10.0.0.0/16"),
secondaryNet: makeIPNet("2000::/120"),
name: "primary and secondary provided (v6,v4)",
expectedFamilies: []v1.IPFamily{v1.IPv6Protocol, v1.IPv4Protocol},
primaryNet: makeIPNet("2000::/120"),
secondaryNet: makeIPNet("10.0.0.0/16"),
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, tc.enableDualStack)()
fakeClient := makeFakeClientSet()
primaryRegistry := makeRangeRegistry(t, tc.primaryNet.String())
@ -277,8 +294,33 @@ func TestShouldWorkOnSecondary(t *testing.T) {
}
repair := NewRepair(0, fakeClient.CoreV1(), fakeClient.CoreV1(), tc.primaryNet, primaryRegistry, tc.secondaryNet, secondaryRegistry)
if repair.shouldWorkOnSecondary() != tc.expectedResult {
t.Errorf("shouldWorkOnSecondary should be %v and found %v", tc.expectedResult, repair.shouldWorkOnSecondary())
if len(repair.allocatorByFamily) != len(tc.expectedFamilies) {
t.Fatalf("expected to have allocator by family count:%v got %v", len(tc.expectedFamilies), len(repair.allocatorByFamily))
}
seen := make(map[v1.IPFamily]bool)
for _, family := range tc.expectedFamilies {
familySeen := true
if _, ok := repair.allocatorByFamily[family]; !ok {
familySeen = familySeen && ok
}
if _, ok := repair.networkByFamily[family]; !ok {
familySeen = familySeen && ok
}
if _, ok := repair.leaksByFamily[family]; !ok {
familySeen = familySeen && ok
}
seen[family] = familySeen
}
for family, seen := range seen {
if !seen {
t.Fatalf("expected repair look to have family %v, but it was not visible on either (or all) network, allocator, leaks", family)
}
}
})
}
@ -418,6 +460,11 @@ func TestRepairLeakDualStack(t *testing.T) {
}
func TestRepairWithExistingDualStack(t *testing.T) {
// because anything (other than allocator) depends
// on families assigned to service (not the value of IPFamilyPolicy)
// we can saftly create tests that has ipFamilyPolicy:nil
// this will work every where except alloc & validation
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, true)()
_, cidr, _ := net.ParseCIDR("192.168.1.0/24")
previous, err := ipallocator.NewCIDRRange(cidr)
@ -445,44 +492,94 @@ func TestRepairWithExistingDualStack(t *testing.T) {
fakeClient := fake.NewSimpleClientset(
&corev1.Service{
ObjectMeta: metav1.ObjectMeta{Namespace: "one", Name: "one"},
Spec: corev1.ServiceSpec{ClusterIP: "192.168.1.1"},
ObjectMeta: metav1.ObjectMeta{Namespace: "x1", Name: "one-v4-v6"},
Spec: corev1.ServiceSpec{
ClusterIP: "192.168.1.1",
ClusterIPs: []string{"192.168.1.1", "2000::1"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
},
},
&corev1.Service{
ObjectMeta: metav1.ObjectMeta{Namespace: "one", Name: "one-v6"},
Spec: corev1.ServiceSpec{ClusterIP: "2000::1"},
ObjectMeta: metav1.ObjectMeta{Namespace: "x2", Name: "one-v6-v4"},
Spec: corev1.ServiceSpec{
ClusterIP: "2000::1",
ClusterIPs: []string{"2000::1", "192.168.1.100"},
IPFamilies: []v1.IPFamily{v1.IPv6Protocol, v1.IPv4Protocol},
},
},
&corev1.Service{
ObjectMeta: metav1.ObjectMeta{Namespace: "two", Name: "two"},
Spec: corev1.ServiceSpec{ClusterIP: "192.168.1.100"},
ObjectMeta: metav1.ObjectMeta{Namespace: "x3", Name: "two-6"},
Spec: corev1.ServiceSpec{
ClusterIP: "2000::2",
ClusterIPs: []string{"2000::2"},
IPFamilies: []v1.IPFamily{v1.IPv6Protocol},
},
},
&corev1.Service{
ObjectMeta: metav1.ObjectMeta{Namespace: "two", Name: "two-6"},
Spec: corev1.ServiceSpec{ClusterIP: "2000::2"},
ObjectMeta: metav1.ObjectMeta{Namespace: "x4", Name: "two-4"},
Spec: corev1.ServiceSpec{
ClusterIP: "192.168.1.90",
ClusterIPs: []string{"192.168.1.90"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
// outside CIDR, will be dropped
&corev1.Service{
ObjectMeta: metav1.ObjectMeta{Namespace: "x5", Name: "out-v4"},
Spec: corev1.ServiceSpec{
ClusterIP: "192.168.0.1",
ClusterIPs: []string{"192.168.0.1"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
&corev1.Service{ // outside CIDR, will be dropped
ObjectMeta: metav1.ObjectMeta{Namespace: "three", Name: "three"},
Spec: corev1.ServiceSpec{ClusterIP: "192.168.0.1"},
ObjectMeta: metav1.ObjectMeta{Namespace: "x6", Name: "out-v6"},
Spec: corev1.ServiceSpec{
ClusterIP: "3000::1",
ClusterIPs: []string{"3000::1"},
IPFamilies: []v1.IPFamily{v1.IPv6Protocol},
},
},
&corev1.Service{ // outside CIDR, will be dropped
ObjectMeta: metav1.ObjectMeta{Namespace: "three", Name: "three-v6"},
Spec: corev1.ServiceSpec{ClusterIP: "3000::1"},
&corev1.Service{
ObjectMeta: metav1.ObjectMeta{Namespace: "x6", Name: "out-v4-v6"},
Spec: corev1.ServiceSpec{
ClusterIP: "192.168.0.1",
ClusterIPs: []string{"192.168.0.1", "3000::1"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
},
},
&corev1.Service{
ObjectMeta: metav1.ObjectMeta{Namespace: "x6", Name: "out-v6-v4"},
Spec: corev1.ServiceSpec{
ClusterIP: "3000::1",
ClusterIPs: []string{"3000::1", "192.168.0.1"},
IPFamilies: []v1.IPFamily{v1.IPv6Protocol, v1.IPv4Protocol},
},
},
&corev1.Service{ // empty, ignored
ObjectMeta: metav1.ObjectMeta{Namespace: "four", Name: "four"},
ObjectMeta: metav1.ObjectMeta{Namespace: "x7", Name: "out-empty"},
Spec: corev1.ServiceSpec{ClusterIP: ""},
},
&corev1.Service{ // duplicate, dropped
ObjectMeta: metav1.ObjectMeta{Namespace: "five", Name: "five"},
Spec: corev1.ServiceSpec{ClusterIP: "192.168.1.1"},
ObjectMeta: metav1.ObjectMeta{Namespace: "x8", Name: "duplicate"},
Spec: corev1.ServiceSpec{
ClusterIP: "192.168.1.1",
ClusterIPs: []string{"192.168.1.1"},
IPFamilies: []v1.IPFamily{v1.IPv4Protocol},
},
},
&corev1.Service{ // duplicate, dropped
ObjectMeta: metav1.ObjectMeta{Namespace: "five", Name: "five-v6"},
Spec: corev1.ServiceSpec{ClusterIP: "2000::2"},
ObjectMeta: metav1.ObjectMeta{Namespace: "x9", Name: "duplicate-v6"},
Spec: corev1.ServiceSpec{
ClusterIP: "2000::2",
ClusterIPs: []string{"2000::2"},
IPFamilies: []v1.IPFamily{v1.IPv6Protocol},
},
},
&corev1.Service{ // headless
ObjectMeta: metav1.ObjectMeta{Namespace: "six", Name: "six"},
ObjectMeta: metav1.ObjectMeta{Namespace: "x10", Name: "headless"},
Spec: corev1.ServiceSpec{ClusterIP: "None"},
},
)
@ -519,9 +616,10 @@ func TestRepairWithExistingDualStack(t *testing.T) {
if !after.Has(net.ParseIP("192.168.1.1")) || !after.Has(net.ParseIP("192.168.1.100")) {
t.Errorf("unexpected ipallocator state: %#v", after)
}
if free := after.Free(); free != 252 {
t.Errorf("unexpected ipallocator state: %d free (number of free ips is not 252)", free)
if free := after.Free(); free != 251 {
t.Errorf("unexpected ipallocator state: %d free (number of free ips is not 251)", free)
}
secondaryAfter, err := ipallocator.NewFromSnapshot(secondaryIPRegistry.updated)
if err != nil {
t.Fatal(err)
@ -532,5 +630,4 @@ func TestRepairWithExistingDualStack(t *testing.T) {
if free := secondaryAfter.Free(); free != 65533 {
t.Errorf("unexpected ipallocator state: %d free (number of free ips is not 65532)", free)
}
}

3
pkg/registry/core/service/storage/BUILD
View File

@ -29,7 +29,6 @@ go_test(
"//staging/src/k8s.io/apimachinery/pkg/fields:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/diff:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/net:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/watch:go_default_library",
@ -41,6 +40,7 @@ go_test(
"//staging/src/k8s.io/apiserver/pkg/util/dryrun:go_default_library",
"//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library",
"//staging/src/k8s.io/component-base/featuregate/testing:go_default_library",
"//vendor/k8s.io/utils/net:go_default_library",
],
)
@ -54,7 +54,6 @@ go_library(
deps = [
"//pkg/api/service:go_default_library",
"//pkg/apis/core:go_default_library",
"//pkg/apis/core/helper:go_default_library",
"//pkg/apis/core/validation:go_default_library",
"//pkg/features:go_default_library",
"//pkg/printers:go_default_library",

604
pkg/registry/core/service/storage/rest.go
View File

@ -40,7 +40,6 @@ import (
apiservice "k8s.io/kubernetes/pkg/api/service"
api "k8s.io/kubernetes/pkg/apis/core"
"k8s.io/kubernetes/pkg/apis/core/helper"
"k8s.io/kubernetes/pkg/apis/core/validation"
registry "k8s.io/kubernetes/pkg/registry/core/service"
"k8s.io/kubernetes/pkg/registry/core/service/ipallocator"
@ -53,14 +52,14 @@ import (
// REST adapts a service registry into apiserver's RESTStorage model.
type REST struct {
strategy rest.RESTCreateUpdateStrategy
services ServiceStorage
endpoints EndpointsStorage
serviceIPs ipallocator.Interface
secondaryServiceIPs ipallocator.Interface
serviceNodePorts portallocator.Interface
proxyTransport http.RoundTripper
pods rest.Getter
strategy rest.RESTCreateUpdateStrategy
services ServiceStorage
endpoints EndpointsStorage
serviceIPAllocatorsByFamily map[api.IPFamily]ipallocator.Interface
defaultServiceIPFamily api.IPFamily // --service-cluster-ip-range[0]
serviceNodePorts portallocator.Interface
proxyTransport http.RoundTripper
pods rest.Getter
}
// ServiceNodePort includes protocol and port number of a service NodePort.
@ -105,15 +104,41 @@ func NewREST(
strategy, _ := registry.StrategyForServiceCIDRs(serviceIPs.CIDR(), secondaryServiceIPs != nil)
byIPFamily := make(map[api.IPFamily]ipallocator.Interface)
// detect this cluster default Service IPFamily (ipfamily of --service-cluster-ip-range[0])
serviceIPFamily := api.IPv4Protocol
cidr := serviceIPs.CIDR()
if netutil.IsIPv6CIDR(&cidr) {
serviceIPFamily = api.IPv6Protocol
}
// add primary family
byIPFamily[serviceIPFamily] = serviceIPs
if secondaryServiceIPs != nil {
// process secondary family
secondaryServiceIPFamily := api.IPv6Protocol
// get family of secondary
if serviceIPFamily == api.IPv6Protocol {
secondaryServiceIPFamily = api.IPv4Protocol
}
// add it
byIPFamily[secondaryServiceIPFamily] = secondaryServiceIPs
}
klog.V(0).Infof("the default service ipfamily for this cluster is: %s", string(serviceIPFamily))
rest := &REST{
strategy: strategy,
services: services,
endpoints: endpoints,
serviceIPs: serviceIPs,
secondaryServiceIPs: secondaryServiceIPs,
serviceNodePorts: serviceNodePorts,
proxyTransport: proxyTransport,
pods: pods,
strategy: strategy,
services: services,
endpoints: endpoints,
serviceIPAllocatorsByFamily: byIPFamily,
serviceNodePorts: serviceNodePorts,
defaultServiceIPFamily: serviceIPFamily,
proxyTransport: proxyTransport,
pods: pods,
}
return rest, &registry.ProxyREST{Redirector: rest, ProxyTransport: proxyTransport}
@ -171,28 +196,35 @@ func (rs *REST) Export(ctx context.Context, name string, opts metav1.ExportOptio
func (rs *REST) Create(ctx context.Context, obj runtime.Object, createValidation rest.ValidateObjectFunc, options *metav1.CreateOptions) (runtime.Object, error) {
service := obj.(*api.Service)
// bag of clusterIPs allocated in the process of creation
// failed allocation will automatically trigger release
var toReleaseClusterIPs map[api.IPFamily]string
if err := rest.BeforeCreate(rs.strategy, ctx, obj); err != nil {
return nil, err
}
// TODO: this should probably move to strategy.PrepareForCreate()
releaseServiceIP := false
defer func() {
if releaseServiceIP {
if helper.IsServiceIPSet(service) {
allocator := rs.getAllocatorByClusterIP(service)
allocator.Release(net.ParseIP(service.Spec.ClusterIP))
}
released, err := rs.releaseClusterIPs(toReleaseClusterIPs)
if err != nil {
klog.Warningf("failed to release clusterIPs for failed new service:%v allocated:%v released:%v error:%v",
service.Name, toReleaseClusterIPs, released, err)
}
}()
// try set ip families (for missing ip families)
// we do it here, since we want this to be visible
// even when dryRun == true
if err := rs.tryDefaultValidateServiceClusterIPFields(service); err != nil {
return nil, err
}
var err error
if !dryrun.IsDryRun(options.DryRun) {
if service.Spec.Type != api.ServiceTypeExternalName {
allocator := rs.getAllocatorBySpec(service)
if releaseServiceIP, err = initClusterIP(service, allocator); err != nil {
return nil, err
}
toReleaseClusterIPs, err = rs.allocServiceClusterIPs(service)
if err != nil {
return nil, err
}
}
@ -227,7 +259,8 @@ func (rs *REST) Create(ctx context.Context, obj runtime.Object, createValidation
utilruntime.HandleError(fmt.Errorf("error(s) committing service node-ports changes: %v", el))
}
releaseServiceIP = false
// no clusterips to release
toReleaseClusterIPs = nil
}
return out, err
@ -241,6 +274,18 @@ func (rs *REST) Delete(ctx context.Context, id string, deleteValidation rest.Val
}
svc := obj.(*api.Service)
// (khenidak) double check that this is in fact the best place for this
// delete strategy handles graceful delete only. It expects strategy
// to implement Graceful-Delete related interface. Hence we are not doing
// the below there. instead we are doing it locally. Until strategy.BeforeDelete works without
// having to implement graceful delete management
// set ClusterIPs based on ClusterIP
// because we depend on ClusterIPs and data might be saved without ClusterIPs ..
if svc.Spec.ClusterIPs == nil && len(svc.Spec.ClusterIP) > 0 {
svc.Spec.ClusterIPs = []string{svc.Spec.ClusterIP}
}
// Only perform the cleanup if this is a non-dryrun deletion
if !dryrun.IsDryRun(options.DryRun) {
@ -268,10 +313,7 @@ func (rs *REST) Delete(ctx context.Context, id string, deleteValidation rest.Val
}
func (rs *REST) releaseAllocatedResources(svc *api.Service) {
if helper.IsServiceIPSet(svc) {
allocator := rs.getAllocatorByClusterIP(svc)
allocator.Release(net.ParseIP(svc.Spec.ClusterIP))
}
rs.releaseServiceClusterIPs(svc)
for _, nodePort := range collectServiceNodePorts(svc) {
err := rs.serviceNodePorts.Release(nodePort)
@ -372,7 +414,6 @@ func (rs *REST) Update(ctx context.Context, name string, objInfo rest.UpdatedObj
return nil, false, err
}
oldService := oldObj.(*api.Service)
obj, err := objInfo.UpdatedObject(ctx, oldService)
if err != nil {
return nil, false, err
@ -389,13 +430,24 @@ func (rs *REST) Update(ctx context.Context, name string, objInfo rest.UpdatedObj
return nil, false, err
}
// TODO: this should probably move to strategy.PrepareForCreate()
releaseServiceIP := false
var allocated map[api.IPFamily]string
var toReleaseIPs map[api.IPFamily]string
performRelease := false // when set, any clusterIP that should be released will be released
// cleanup
// on failure: Any allocated ip must be released back
// on failure: any ip that should be released, will *not* be released
// on success: any ip that should be released, will be released
defer func() {
if releaseServiceIP {
if helper.IsServiceIPSet(service) {
allocator := rs.getAllocatorByClusterIP(service)
allocator.Release(net.ParseIP(service.Spec.ClusterIP))
// release the allocated, this is expected to be cleared if the entire function ran to success
if allocated_released, err := rs.releaseClusterIPs(allocated); err != nil {
klog.V(4).Infof("service %v/%v failed to clean up after failed service update error:%v. Allocated/Released:%v/%v", service.Namespace, service.Name, err, allocated, allocated_released)
}
// performRelease is set when the enture function ran to success
if performRelease {
if toReleaseIPs_released, err := rs.releaseClusterIPs(toReleaseIPs); err != nil {
klog.V(4).Infof("service %v/%v failed to clean up after failed service update error:%v. ShouldRelease/Released:%v/%v", service.Namespace, service.Name, err, toReleaseIPs, toReleaseIPs_released)
}
}
}()
@ -403,22 +455,15 @@ func (rs *REST) Update(ctx context.Context, name string, objInfo rest.UpdatedObj
nodePortOp := portallocator.StartOperation(rs.serviceNodePorts, dryrun.IsDryRun(options.DryRun))
defer nodePortOp.Finish()
// try set ip families (for missing ip families)
if err := rs.tryDefaultValidateServiceClusterIPFields(service); err != nil {
return nil, false, err
}
if !dryrun.IsDryRun(options.DryRun) {
// Update service from ExternalName to non-ExternalName, should initialize ClusterIP.
// Since we don't support changing the ip family of a service we don't need to handle
// oldService.Spec.ServiceIPFamily != service.Spec.ServiceIPFamily
if oldService.Spec.Type == api.ServiceTypeExternalName && service.Spec.Type != api.ServiceTypeExternalName {
allocator := rs.getAllocatorBySpec(service)
if releaseServiceIP, err = initClusterIP(service, allocator); err != nil {
return nil, false, err
}
}
// Update service from non-ExternalName to ExternalName, should release ClusterIP if exists.
if oldService.Spec.Type != api.ServiceTypeExternalName && service.Spec.Type == api.ServiceTypeExternalName {
if helper.IsServiceIPSet(oldService) {
allocator := rs.getAllocatorByClusterIP(service)
allocator.Release(net.ParseIP(oldService.Spec.ClusterIP))
}
allocated, toReleaseIPs, err = rs.handleClusterIPsForUpdatedService(oldService, service)
if err != nil {
return nil, false, err
}
}
// Update service from NodePort or LoadBalancer to ExternalName or ClusterIP, should release NodePort if exists.
@ -455,9 +500,10 @@ func (rs *REST) Update(ctx context.Context, name string, objInfo rest.UpdatedObj
// problems should be fixed by an eventual reconciliation / restart
utilruntime.HandleError(fmt.Errorf("error(s) committing NodePorts changes: %v", el))
}
releaseServiceIP = false
}
// all good
allocated = nil // if something was allocated, keep it allocated
performRelease = true // if something that should be released then go ahead and release it
return out, created, err
}
@ -541,33 +587,397 @@ func (r *REST) ConvertToTable(ctx context.Context, object runtime.Object, tableO
return r.services.ConvertToTable(ctx, object, tableOptions)
}
// When allocating we always use BySpec, when releasing we always use ByClusterIP
func (r *REST) getAllocatorByClusterIP(service *api.Service) ipallocator.Interface {
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) || r.secondaryServiceIPs == nil {
return r.serviceIPs
}
func (rs *REST) allocClusterIPs(service *api.Service, toAlloc map[api.IPFamily]string) (map[api.IPFamily]string, error) {
allocated := make(map[api.IPFamily]string)
secondaryAllocatorCIDR := r.secondaryServiceIPs.CIDR()
if netutil.IsIPv6String(service.Spec.ClusterIP) == netutil.IsIPv6CIDR(&secondaryAllocatorCIDR) {
return r.secondaryServiceIPs
for family, ip := range toAlloc {
allocator := rs.serviceIPAllocatorsByFamily[family] // should always be there, as we pre validate
if ip == "" {
allocatedIP, err := allocator.AllocateNext()
if err != nil {
return allocated, errors.NewInternalError(fmt.Errorf("failed to allocate a serviceIP: %v", err))
}
allocated[family] = allocatedIP.String()
} else {
parsedIP := net.ParseIP(ip)
if err := allocator.Allocate(parsedIP); err != nil {
el := field.ErrorList{field.Invalid(field.NewPath("spec", "clusterIPs"), service.Spec.ClusterIPs, fmt.Sprintf("failed to allocated ip:%v with error:%v", ip, err))}
return allocated, errors.NewInvalid(api.Kind("Service"), service.Name, el)
}
allocated[family] = ip
}
}
return r.serviceIPs
return allocated, nil
}
func (r *REST) getAllocatorBySpec(service *api.Service) ipallocator.Interface {
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) ||
service.Spec.IPFamily == nil ||
r.secondaryServiceIPs == nil {
return r.serviceIPs
// releases clusterIPs per family
func (rs *REST) releaseClusterIPs(toRelease map[api.IPFamily]string) (map[api.IPFamily]string, error) {
if toRelease == nil {
return nil, nil
}
secondaryAllocatorCIDR := r.secondaryServiceIPs.CIDR()
if (*(service.Spec.IPFamily) == api.IPv6Protocol) == netutil.IsIPv6CIDR(&secondaryAllocatorCIDR) {
return r.secondaryServiceIPs
released := make(map[api.IPFamily]string)
for family, ip := range toRelease {
allocator, ok := rs.serviceIPAllocatorsByFamily[family]
if !ok {
// cluster was configured for dual stack, then single stack
klog.V(4).Infof("delete service. Not releasing ClusterIP:%v because IPFamily:%v is no longer configured on server", ip, family)
continue
}
parsedIP := net.ParseIP(ip)
if err := allocator.Release(parsedIP); err != nil {
return released, err
}
released[family] = ip
}
return r.serviceIPs
return released, nil
}
// standard allocator for dualstackgate==Off, hard wired dependency
// and ignores policy, families and clusterIPs
func (rs *REST) allocServiceClusterIP(service *api.Service) (map[api.IPFamily]string, error) {
toAlloc := make(map[api.IPFamily]string)
// get clusterIP.. empty string if user did not specify an ip
toAlloc[rs.defaultServiceIPFamily] = service.Spec.ClusterIP
// alloc
allocated, err := rs.allocClusterIPs(service, toAlloc)
// set
if err == nil {
service.Spec.ClusterIP = allocated[rs.defaultServiceIPFamily]
service.Spec.ClusterIPs = []string{allocated[rs.defaultServiceIPFamily]}
}
return allocated, err
}
// allocates ClusterIPs for a service
func (rs *REST) allocServiceClusterIPs(service *api.Service) (map[api.IPFamily]string, error) {
// external name don't get ClusterIPs
if service.Spec.Type == api.ServiceTypeExternalName {
return nil, nil
}
// headless don't get ClusterIPs
if len(service.Spec.ClusterIPs) > 0 && service.Spec.ClusterIPs[0] == api.ClusterIPNone {
return nil, nil
}
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
return rs.allocServiceClusterIP(service)
}
toAlloc := make(map[api.IPFamily]string)
// at this stage, the only fact we know is that service has correct ip families
// assigned to it. It may have partial assigned ClusterIPs (Upgrade to dual stack)
// may have no ips at all. The below loop is meant to fix this
// (we also know that this cluster has these families)
// if there is no slice to work with
if service.Spec.ClusterIPs == nil {
service.Spec.ClusterIPs = make([]string, 0, len(service.Spec.IPFamilies))
}
for i, ipFamily := range service.Spec.IPFamilies {
if i > (len(service.Spec.ClusterIPs) - 1) {
service.Spec.ClusterIPs = append(service.Spec.ClusterIPs, "" /* just a marker */)
}
toAlloc[ipFamily] = service.Spec.ClusterIPs[i]
}
// allocate
allocated, err := rs.allocClusterIPs(service, toAlloc)
// set if successful
if err == nil {
for family, ip := range allocated {
for i, check := range service.Spec.IPFamilies {
if family == check {
service.Spec.ClusterIPs[i] = ip
// while we technically don't need to do that testing rest does not
// go through conversion logic but goes through validation *sigh*.
// so we set ClusterIP here as well
// because the testing code expects valid (as they are output-ed from conversion)
// as it patches fields
if i == 0 {
service.Spec.ClusterIP = ip
}
}
}
}
}
return allocated, err
}
// handles type change/upgrade/downgrade change type for an update service
// this func does not perform actual release of clusterIPs. it returns
// a map[family]ip for the caller to release when everything else has
// executed successfully
func (rs *REST) handleClusterIPsForUpdatedService(oldService *api.Service, service *api.Service) (allocated map[api.IPFamily]string, toRelease map[api.IPFamily]string, err error) {
// use cases:
// A: service changing types from ExternalName TO ClusterIP types ==> allocate all new
// B: service changing types from ClusterIP types TO ExternalName ==> release all allocated
// C: Service upgrading to dual stack ==> partial allocation
// D: service downgrading from dual stack ==> partial release
// CASE A:
// Update service from ExternalName to non-ExternalName, should initialize ClusterIP.
if oldService.Spec.Type == api.ServiceTypeExternalName && service.Spec.Type != api.ServiceTypeExternalName {
allocated, err := rs.allocServiceClusterIPs(service)
return allocated, nil, err
}
// CASE B:
// Update service from non-ExternalName to ExternalName, should release ClusterIP if exists.
if oldService.Spec.Type != api.ServiceTypeExternalName && service.Spec.Type == api.ServiceTypeExternalName {
toRelease = make(map[api.IPFamily]string)
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
// for non dual stack enabled cluster we use clusterIPs
toRelease[rs.defaultServiceIPFamily] = oldService.Spec.ClusterIP
} else {
// dual stack is enabled, collect ClusterIPs by families
for i, family := range oldService.Spec.IPFamilies {
toRelease[family] = oldService.Spec.ClusterIPs[i]
}
}
return nil, toRelease, nil
}
// if headless service then we bail out early (no clusterIPs management needed)
if len(oldService.Spec.ClusterIPs) > 0 && oldService.Spec.ClusterIPs[0] == api.ClusterIPNone {
return nil, nil, nil
}
// upgrade and downgrade are specific to dualstack
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
return nil, nil, nil
}
upgraded := len(oldService.Spec.IPFamilies) == 1 && len(service.Spec.IPFamilies) == 2
downgraded := len(oldService.Spec.IPFamilies) == 2 && len(service.Spec.IPFamilies) == 1
// CASE C:
if upgraded {
toAllocate := make(map[api.IPFamily]string)
// if secondary ip was named, just get it. if not add a marker
if len(service.Spec.ClusterIPs) < 2 {
service.Spec.ClusterIPs = append(service.Spec.ClusterIPs, "" /* marker */)
}
toAllocate[service.Spec.IPFamilies[1]] = service.Spec.ClusterIPs[1]
// allocate
allocated, err := rs.allocClusterIPs(service, toAllocate)
// set if successful
if err == nil {
service.Spec.ClusterIPs[1] = allocated[service.Spec.IPFamilies[1]]
}
return allocated, nil, err
}
// CASE D:
if downgraded {
toRelease = make(map[api.IPFamily]string)
toRelease[oldService.Spec.IPFamilies[1]] = oldService.Spec.ClusterIPs[1]
// note: we don't release clusterIP, this is left to clean up in the action itself
return nil, toRelease, err
}
// it was not an upgrade nor downgrade
return nil, nil, nil
}
// for pre dual stack (gate == off). Hardwired to ClusterIP and ignores all new fields
func (rs *REST) releaseServiceClusterIP(service *api.Service) (released map[api.IPFamily]string, err error) {
toRelease := make(map[api.IPFamily]string)
// we need to do that to handle cases where allocator is no longer configured on
// cluster
if netutil.IsIPv6String(service.Spec.ClusterIP) {
toRelease[api.IPv6Protocol] = service.Spec.ClusterIP
} else {
toRelease[api.IPv4Protocol] = service.Spec.ClusterIP
}
return rs.releaseClusterIPs(toRelease)
}
// releases allocated ClusterIPs for service that is about to be deleted
func (rs *REST) releaseServiceClusterIPs(service *api.Service) (released map[api.IPFamily]string, err error) {
// external name don't get ClusterIPs
if service.Spec.Type == api.ServiceTypeExternalName {
return nil, nil
}
// headless don't get ClusterIPs
if len(service.Spec.ClusterIPs) > 0 && service.Spec.ClusterIPs[0] == api.ClusterIPNone {
return nil, nil
}
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
return rs.releaseServiceClusterIP(service)
}
toRelease := make(map[api.IPFamily]string)
for _, ip := range service.Spec.ClusterIPs {
if netutil.IsIPv6String(ip) {
toRelease[api.IPv6Protocol] = ip
} else {
toRelease[api.IPv4Protocol] = ip
}
}
return rs.releaseClusterIPs(toRelease)
}
// attempts to default service ip families according to cluster configuration
// while ensuring that provided families are configured on cluster.
func (rs *REST) tryDefaultValidateServiceClusterIPFields(service *api.Service) error {
// can not do anything here
if service.Spec.Type == api.ServiceTypeExternalName {
return nil
}
// gate off. We don't need to validate or default new fields
// we totally depend on existing validation in apis/validation
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
return nil
}
// two families or two IPs with SingleStack
if service.Spec.IPFamilyPolicy != nil {
el := make(field.ErrorList, 0)
if *(service.Spec.IPFamilyPolicy) == api.IPFamilyPolicySingleStack {
if len(service.Spec.ClusterIPs) == 2 {
el = append(el, field.Invalid(field.NewPath("spec", "ipFamilyPolicy"), service.Spec.IPFamilyPolicy, "must be RequireDualStack or PreferDualStack when multiple 'clusterIPs' are specified"))
}
if len(service.Spec.IPFamilies) == 2 {
el = append(el, field.Invalid(field.NewPath("spec", "ipFamilyPolicy"), service.Spec.IPFamilyPolicy, "must be RequireDualStack or PreferDualStack when multiple 'ipFamilies' are specified"))
}
}
if len(el) > 0 {
return errors.NewInvalid(api.Kind("Service"), service.Name, el)
}
}
// default families according to cluster IPs
for i, ip := range service.Spec.ClusterIPs {
if ip == api.ClusterIPNone {
break
}
// we have previously validated for ip correctness and if family exist it will match ip family
// so the following is safe to do
isIPv6 := netutil.IsIPv6String(ip)
// family is not there.
if i > len(service.Spec.IPFamilies)-1 {
if isIPv6 {
// first make sure that family(ip) is configured
if _, found := rs.serviceIPAllocatorsByFamily[api.IPv6Protocol]; !found {
el := field.ErrorList{field.Invalid(field.NewPath("spec", "clusterIPs").Index(i), service.Spec.ClusterIPs, "may not use IPv6 on a cluster which is not configured for it")}
return errors.NewInvalid(api.Kind("Service"), service.Name, el)
}
service.Spec.IPFamilies = append(service.Spec.IPFamilies, api.IPv6Protocol)
} else {
// first make sure that family(ip) is configured
if _, found := rs.serviceIPAllocatorsByFamily[api.IPv4Protocol]; !found {
el := field.ErrorList{field.Invalid(field.NewPath("spec", "clusterIPs").Index(i), service.Spec.ClusterIPs, "may not use IPv4 on a cluster which is not configured for it")}
return errors.NewInvalid(api.Kind("Service"), service.Name, el)
}
service.Spec.IPFamilies = append(service.Spec.IPFamilies, api.IPv4Protocol)
}
}
}
// default headless+selectorless
if len(service.Spec.ClusterIPs) > 0 && service.Spec.ClusterIPs[0] == api.ClusterIPNone && len(service.Spec.Selector) == 0 {
if service.Spec.IPFamilyPolicy == nil {
requireDualStack := api.IPFamilyPolicyRequireDualStack
service.Spec.IPFamilyPolicy = &requireDualStack
}
// if not set by user
if len(service.Spec.IPFamilies) == 0 {
service.Spec.IPFamilies = []api.IPFamily{rs.defaultServiceIPFamily}
}
// this follows headful services. With one exception on a single stack
// cluster the user is allowed to create headless services that has multi families
// the validation allows it
if len(service.Spec.IPFamilies) < 2 {
if *(service.Spec.IPFamilyPolicy) == api.IPFamilyPolicyRequireDualStack ||
(*(service.Spec.IPFamilyPolicy) == api.IPFamilyPolicyPreferDualStack && len(rs.serviceIPAllocatorsByFamily) == 2) {
// add the alt ipfamily
if service.Spec.IPFamilies[0] == api.IPv4Protocol {
service.Spec.IPFamilies = append(service.Spec.IPFamilies, api.IPv6Protocol)
} else {
service.Spec.IPFamilies = append(service.Spec.IPFamilies, api.IPv4Protocol)
}
}
}
// nothing more needed here
return nil
}
// ipfamily check
// the following applies on all type of services including headless w/ selector
el := make(field.ErrorList, 0)
// asking for dual stack on a non dual stack cluster
// should fail without assigning any family
if service.Spec.IPFamilyPolicy != nil && *(service.Spec.IPFamilyPolicy) == api.IPFamilyPolicyRequireDualStack && len(rs.serviceIPAllocatorsByFamily) < 2 {
el = append(el, field.Invalid(field.NewPath("spec", "ipFamilyPolicy"), service.Spec.IPFamilyPolicy, "Cluster is not configured for dual stack services"))
}
// if there is a family requested then it has to be configured on cluster
for i, ipFamily := range service.Spec.IPFamilies {
if _, found := rs.serviceIPAllocatorsByFamily[ipFamily]; !found {
el = append(el, field.Invalid(field.NewPath("spec", "ipFamilies").Index(i), service.Spec.ClusterIPs, fmt.Sprintf("ipfamily %v is not configured on cluster", ipFamily)))
}
}
// if we have validation errors return them and bail out
if len(el) > 0 {
return errors.NewInvalid(api.Kind("Service"), service.Name, el)
}
// default ipFamilyPolicy to SingleStack. if there are
// web hooks, they must have already ran by now
if service.Spec.IPFamilyPolicy == nil {
singleStack := api.IPFamilyPolicySingleStack
service.Spec.IPFamilyPolicy = &singleStack
}
// nil families, gets cluster default (if feature flag is not in effect, the strategy will take care of removing it)
if len(service.Spec.IPFamilies) == 0 {
service.Spec.IPFamilies = []api.IPFamily{rs.defaultServiceIPFamily}
}
// is this service looking for dual stack, and this cluster does have two families?
// if so, then append the missing family
if *(service.Spec.IPFamilyPolicy) != api.IPFamilyPolicySingleStack &&
len(service.Spec.IPFamilies) == 1 &&
len(rs.serviceIPAllocatorsByFamily) == 2 {
if service.Spec.IPFamilies[0] == api.IPv4Protocol {
service.Spec.IPFamilies = append(service.Spec.IPFamilies, api.IPv6Protocol)
}
if service.Spec.IPFamilies[0] == api.IPv6Protocol {
service.Spec.IPFamilies = append(service.Spec.IPFamilies, api.IPv4Protocol)
}
}
return nil
}
func isValidAddress(ctx context.Context, addr *api.EndpointAddress, pods rest.Getter) error {
@ -653,48 +1063,6 @@ func allocateHealthCheckNodePort(service *api.Service, nodePortOp *portallocator
return nil
}
// The return bool value indicates if a cluster IP is allocated successfully.
func initClusterIP(service *api.Service, allocator ipallocator.Interface) (bool, error) {
var allocatedIP net.IP
switch {
case service.Spec.ClusterIP == "":
// Allocate next available.
ip, err := allocator.AllocateNext()
if err != nil {
// TODO: what error should be returned here? It's not a
// field-level validation failure (the field is valid), and it's
// not really an internal error.
return false, errors.NewInternalError(fmt.Errorf("failed to allocate a serviceIP: %v", err))
}
allocatedIP = ip
service.Spec.ClusterIP = ip.String()
case service.Spec.ClusterIP != api.ClusterIPNone && service.Spec.ClusterIP != "":
// Try to respect the requested IP.
ip := net.ParseIP(service.Spec.ClusterIP)
if err := allocator.Allocate(ip); err != nil {
// TODO: when validation becomes versioned, this gets more complicated.
el := field.ErrorList{field.Invalid(field.NewPath("spec", "clusterIP"), service.Spec.ClusterIP, err.Error())}
return false, errors.NewInvalid(api.Kind("Service"), service.Name, el)
}
allocatedIP = ip
}
// assuming the object was valid prior to setting, always force the IPFamily
// to match the allocated IP at this point
if allocatedIP != nil {
if utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
ipFamily := api.IPv4Protocol
if netutil.IsIPv6(allocatedIP) {
ipFamily = api.IPv6Protocol
}
service.Spec.IPFamily = &ipFamily
}
}
return allocatedIP != nil, nil
}
func initNodePorts(service *api.Service, nodePortOp *portallocator.PortAllocationOperation) error {
svcPortToNodePort := map[int]int{}
for i := range service.Spec.Ports {

3176
pkg/registry/core/service/storage/rest_test.go
View File

File diff suppressed because it is too large Load Diff

137
pkg/registry/core/service/storage/storage.go
View File

@ -31,10 +31,17 @@ import (
printerstorage "k8s.io/kubernetes/pkg/printers/storage"
"k8s.io/kubernetes/pkg/registry/core/service"
registry "k8s.io/kubernetes/pkg/registry/core/service"
netutil "k8s.io/utils/net"
utilfeature "k8s.io/apiserver/pkg/util/feature"
"k8s.io/kubernetes/pkg/features"
)
type GenericREST struct {
*genericregistry.Store
primaryIPFamily *api.IPFamily
secondaryFamily *api.IPFamily
}
// NewREST returns a RESTStorage object that will work against services.
@ -61,7 +68,26 @@ func NewGenericREST(optsGetter generic.RESTOptionsGetter, serviceCIDR net.IPNet,
statusStore := *store
statusStore.UpdateStrategy = service.NewServiceStatusStrategy(strategy)
return &GenericREST{store}, &StatusREST{store: &statusStore}, nil
ipv4 := api.IPv4Protocol
ipv6 := api.IPv6Protocol
var primaryIPFamily *api.IPFamily = nil
var secondaryFamily *api.IPFamily = nil
if netutil.IsIPv6CIDR(&serviceCIDR) {
primaryIPFamily = &ipv6
if hasSecondary {
secondaryFamily = &ipv4
}
} else {
primaryIPFamily = &ipv4
if hasSecondary {
secondaryFamily = &ipv6
}
}
genericStore := &GenericREST{store, primaryIPFamily, secondaryFamily}
store.Decorator = genericStore.defaultServiceOnRead // default on read
return genericStore, &StatusREST{store: &statusStore}, nil
}
var (
@ -99,3 +125,112 @@ func (r *StatusREST) Update(ctx context.Context, name string, objInfo rest.Updat
// subresources should never allow create on update.
return r.store.Update(ctx, name, objInfo, createValidation, updateValidation, false, options)
}
// defaults fields that were not previously set on read. becomes an
// essential part of upgrading a service
func (r *GenericREST) defaultServiceOnRead(obj runtime.Object) error {
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
return nil
}
service, ok := obj.(*api.Service)
if ok {
return r.defaultAServiceOnRead(service)
}
serviceList, ok := obj.(*api.ServiceList)
if ok {
return r.defaultServiceList(serviceList)
}
// this was not an object we can default
return nil
}
// defaults a service list
func (r *GenericREST) defaultServiceList(serviceList *api.ServiceList) error {
if serviceList == nil {
return nil
}
for i := range serviceList.Items {
err := r.defaultAServiceOnRead(&serviceList.Items[i])
if err != nil {
return err
}
}
return nil
}
// defaults a single service
func (r *GenericREST) defaultAServiceOnRead(service *api.Service) error {
if service == nil {
return nil
}
if len(service.Spec.IPFamilies) > 0 {
return nil // already defaulted
}
// set clusterIPs based on ClusterIP
if len(service.Spec.ClusterIPs) == 0 {
if len(service.Spec.ClusterIP) > 0 {
service.Spec.ClusterIPs = []string{service.Spec.ClusterIP}
}
}
requireDualStack := api.IPFamilyPolicyRequireDualStack
singleStack := api.IPFamilyPolicySingleStack
preferDualStack := api.IPFamilyPolicyPreferDualStack
// headless services
if len(service.Spec.ClusterIPs) == 1 && service.Spec.ClusterIPs[0] == api.ClusterIPNone {
service.Spec.IPFamilies = []api.IPFamily{*r.primaryIPFamily}
// headless+selectorless
// headless+selectorless takes both families. Why?
// at this stage we don't know what kind of endpoints (specifically their IPFamilies) the
// user has assigned to this selectorless service. We assume it has dualstack and we default
// it to PreferDualStack on any cluster (single or dualstack configured).
if len(service.Spec.Selector) == 0 {
service.Spec.IPFamilyPolicy = &preferDualStack
if *r.primaryIPFamily == api.IPv4Protocol {
service.Spec.IPFamilies = append(service.Spec.IPFamilies, api.IPv6Protocol)
} else {
service.Spec.IPFamilies = append(service.Spec.IPFamilies, api.IPv4Protocol)
}
} else {
// headless w/ selector
// this service type follows cluster configuration. this service (selector based) uses a
// selector and will have to follow how the cluster is configured. If the cluster is
// configured to dual stack then the service defaults to PreferDualStack. Otherwise we
// default it to SingleStack.
if r.secondaryFamily != nil {
service.Spec.IPFamilies = append(service.Spec.IPFamilies, *r.secondaryFamily)
service.Spec.IPFamilyPolicy = &preferDualStack
} else {
service.Spec.IPFamilyPolicy = &singleStack
}
}
} else {
// headful
// make sure a slice exists to receive the families
service.Spec.IPFamilies = make([]api.IPFamily, len(service.Spec.ClusterIPs), len(service.Spec.ClusterIPs))
for idx, ip := range service.Spec.ClusterIPs {
if netutil.IsIPv6String(ip) {
service.Spec.IPFamilies[idx] = api.IPv6Protocol
} else {
service.Spec.IPFamilies[idx] = api.IPv4Protocol
}
if len(service.Spec.IPFamilies) == 1 {
service.Spec.IPFamilyPolicy = &singleStack
} else if len(service.Spec.IPFamilies) == 2 {
service.Spec.IPFamilyPolicy = &requireDualStack
}
}
}
return nil
}

292
pkg/registry/core/service/storage/storage_test.go
View File

@ -17,6 +17,7 @@ limitations under the License.
package storage
import (
"net"
"testing"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@ -29,6 +30,10 @@ import (
etcd3testing "k8s.io/apiserver/pkg/storage/etcd3/testing"
api "k8s.io/kubernetes/pkg/apis/core"
"k8s.io/kubernetes/pkg/registry/registrytest"
utilfeature "k8s.io/apiserver/pkg/util/feature"
featuregatetesting "k8s.io/component-base/featuregate/testing"
"k8s.io/kubernetes/pkg/features"
)
func newStorage(t *testing.T) (*GenericREST, *StatusREST, *etcd3testing.EtcdTestServer) {
@ -47,6 +52,8 @@ func newStorage(t *testing.T) (*GenericREST, *StatusREST, *etcd3testing.EtcdTest
}
func validService() *api.Service {
singleStack := api.IPFamilyPolicySingleStack
return &api.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "foo",
@ -54,7 +61,10 @@ func validService() *api.Service {
},
Spec: api.ServiceSpec{
Selector: map[string]string{"bar": "baz"},
ClusterIP: "None",
ClusterIP: api.ClusterIPNone,
ClusterIPs: []string{api.ClusterIPNone},
IPFamilyPolicy: &singleStack,
IPFamilies: []api.IPFamily{api.IPv4Protocol},
SessionAffinity: "None",
Type: api.ServiceTypeClusterIP,
Ports: []api.ServicePort{{
@ -84,7 +94,7 @@ func TestCreate(t *testing.T) {
&api.Service{
Spec: api.ServiceSpec{
Selector: map[string]string{"bar": "baz"},
ClusterIP: "invalid",
ClusterIPs: []string{"invalid"},
SessionAffinity: "None",
Type: api.ServiceTypeClusterIP,
Ports: []api.ServicePort{{
@ -110,7 +120,8 @@ func TestUpdate(t *testing.T) {
object := obj.(*api.Service)
object.Spec = api.ServiceSpec{
Selector: map[string]string{"bar": "baz2"},
ClusterIP: "None",
ClusterIP: api.ClusterIPNone,
ClusterIPs: []string{api.ClusterIPNone},
SessionAffinity: api.ServiceAffinityNone,
Type: api.ServiceTypeClusterIP,
Ports: []api.ServicePort{{
@ -187,3 +198,278 @@ func TestCategories(t *testing.T) {
expected := []string{"all"}
registrytest.AssertCategories(t, storage, expected)
}
func makeServiceList() (undefaulted, defaulted *api.ServiceList) {
undefaulted = &api.ServiceList{Items: []api.Service{}}
defaulted = &api.ServiceList{Items: []api.Service{}}
singleStack := api.IPFamilyPolicySingleStack
requireDualStack := api.IPFamilyPolicyRequireDualStack
var undefaultedSvc *api.Service
var defaultedSvc *api.Service
// (for headless) tests must set fields manually according to how the cluster configured
// headless w selector (subject to how the cluster is configured)
undefaultedSvc = &api.Service{
ObjectMeta: metav1.ObjectMeta{Name: "headless_with_selector", ResourceVersion: "1", Namespace: metav1.NamespaceDefault},
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
ClusterIPs: []string{api.ClusterIPNone},
Selector: map[string]string{"foo": "bar"},
},
}
defaultedSvc = undefaultedSvc.DeepCopy()
defaultedSvc.Spec.IPFamilyPolicy = nil // forcing tests to set them
defaultedSvc.Spec.IPFamilies = nil // forcing tests to them
undefaulted.Items = append(undefaulted.Items, *(undefaultedSvc))
defaulted.Items = append(defaulted.Items, *(defaultedSvc))
// headless w/o selector (always set to require and families according to cluster)
undefaultedSvc = &api.Service{
ObjectMeta: metav1.ObjectMeta{Name: "headless_no_selector", ResourceVersion: "1", Namespace: metav1.NamespaceDefault},
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
ClusterIPs: []string{api.ClusterIPNone},
Selector: nil,
},
}
defaultedSvc = undefaultedSvc.DeepCopy()
defaultedSvc.Spec.IPFamilyPolicy = nil // forcing tests to set them
defaultedSvc.Spec.IPFamilies = nil // forcing tests to them
undefaulted.Items = append(undefaulted.Items, *(undefaultedSvc))
defaulted.Items = append(defaulted.Items, *(defaultedSvc))
// single stack IPv4
undefaultedSvc = &api.Service{
ObjectMeta: metav1.ObjectMeta{Name: "ipv4", ResourceVersion: "1", Namespace: metav1.NamespaceDefault},
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
ClusterIP: "10.0.0.4",
},
}
defaultedSvc = undefaultedSvc.DeepCopy()
defaultedSvc.Spec.IPFamilyPolicy = &singleStack
defaultedSvc.Spec.IPFamilies = []api.IPFamily{api.IPv4Protocol}
undefaulted.Items = append(undefaulted.Items, *(undefaultedSvc))
defaulted.Items = append(defaulted.Items, *(defaultedSvc))
// single stack IPv6
undefaultedSvc = &api.Service{
ObjectMeta: metav1.ObjectMeta{Name: "ipv6", ResourceVersion: "1", Namespace: metav1.NamespaceDefault},
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
ClusterIP: "2000::1",
},
}
defaultedSvc = undefaultedSvc.DeepCopy()
defaultedSvc.Spec.IPFamilyPolicy = &singleStack
defaultedSvc.Spec.IPFamilies = []api.IPFamily{api.IPv6Protocol}
undefaulted.Items = append(undefaulted.Items, *(undefaultedSvc))
defaulted.Items = append(defaulted.Items, *(defaultedSvc))
// dualstack IPv4 IPv6
undefaultedSvc = &api.Service{
ObjectMeta: metav1.ObjectMeta{Name: "ipv4_ipv6", ResourceVersion: "1", Namespace: metav1.NamespaceDefault},
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
ClusterIP: "10.0.0.4",
ClusterIPs: []string{"10.0.0.4", "2000::1"},
},
}
defaultedSvc = undefaultedSvc.DeepCopy()
defaultedSvc.Spec.IPFamilyPolicy = &requireDualStack
defaultedSvc.Spec.IPFamilies = []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol}
undefaulted.Items = append(undefaulted.Items, *(undefaultedSvc))
defaulted.Items = append(defaulted.Items, *(defaultedSvc))
// dualstack IPv6 IPv4
undefaultedSvc = &api.Service{
ObjectMeta: metav1.ObjectMeta{Name: "ipv6_ipv4", ResourceVersion: "1", Namespace: metav1.NamespaceDefault},
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
ClusterIP: "2000::1",
ClusterIPs: []string{"2000::1", "10.0.0.4"},
},
}
defaultedSvc = undefaultedSvc.DeepCopy()
defaultedSvc.Spec.IPFamilyPolicy = &requireDualStack
defaultedSvc.Spec.IPFamilies = []api.IPFamily{api.IPv6Protocol, api.IPv4Protocol}
undefaulted.Items = append(undefaulted.Items, *(undefaultedSvc))
defaulted.Items = append(defaulted.Items, *(defaultedSvc))
// external name
undefaultedSvc = &api.Service{
ObjectMeta: metav1.ObjectMeta{Name: "external_name", ResourceVersion: "1", Namespace: metav1.NamespaceDefault},
Spec: api.ServiceSpec{
Type: api.ServiceTypeExternalName,
},
}
defaultedSvc = undefaultedSvc.DeepCopy()
defaultedSvc.Spec.IPFamilyPolicy = nil
defaultedSvc.Spec.IPFamilies = nil
undefaulted.Items = append(undefaulted.Items, *(undefaultedSvc))
defaulted.Items = append(defaulted.Items, *(defaultedSvc))
return undefaulted, defaulted
}
func TestServiceDefaulting(t *testing.T) {
makeStorage := func(t *testing.T, primaryCIDR string, isDualStack bool) (*GenericREST, *StatusREST, *etcd3testing.EtcdTestServer) {
etcdStorage, server := registrytest.NewEtcdStorage(t, "")
restOptions := generic.RESTOptions{
StorageConfig: etcdStorage,
Decorator: generic.UndecoratedStorage,
DeleteCollectionWorkers: 1,
ResourcePrefix: "services",
}
_, cidr, err := net.ParseCIDR(primaryCIDR)
if err != nil {
t.Fatalf("failed to parse CIDR %s", primaryCIDR)
}
serviceStorage, statusStorage, err := NewGenericREST(restOptions, *(cidr), isDualStack)
if err != nil {
t.Fatalf("unexpected error from REST storage: %v", err)
}
return serviceStorage, statusStorage, server
}
testCases := []struct {
name string
primaryCIDR string
PrimaryIPv6 bool
isDualStack bool
}{
{
name: "IPv4 single stack cluster",
primaryCIDR: "10.0.0.0/16",
PrimaryIPv6: false,
isDualStack: false,
},
{
name: "IPv6 single stack cluster",
primaryCIDR: "2000::/108",
PrimaryIPv6: true,
isDualStack: false,
},
{
name: "IPv4, IPv6 dual stack cluster",
primaryCIDR: "10.0.0.0/16",
PrimaryIPv6: false,
isDualStack: true,
},
{
name: "IPv6, IPv4 dual stack cluster",
primaryCIDR: "2000::/108",
PrimaryIPv6: true,
isDualStack: true,
},
}
singleStack := api.IPFamilyPolicySingleStack
preferDualStack := api.IPFamilyPolicyPreferDualStack
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
// this func only works with dual stack feature gate on.
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, true)()
storage, _, server := makeStorage(t, testCase.primaryCIDR, testCase.isDualStack)
defer server.Terminate(t)
defer storage.Store.DestroyFunc()
undefaultedServiceList, defaultedServiceList := makeServiceList()
// set the two special ones (0: w/ selector, 1: w/o selector)
// review default*OnRead(...)
// Single stack cluster:
// headless w/selector => singlestack
// headless w/o selector => preferDualStack
// dual stack cluster:
// headless w/selector => preferDualStack
// headless w/o selector => preferDualStack
// assume single stack
defaultedServiceList.Items[0].Spec.IPFamilyPolicy = &singleStack
// primary family
if testCase.PrimaryIPv6 {
// no selector, gets both families
defaultedServiceList.Items[1].Spec.IPFamilyPolicy = &preferDualStack
defaultedServiceList.Items[1].Spec.IPFamilies = []api.IPFamily{api.IPv6Protocol, api.IPv4Protocol}
//assume single stack for w/selector
defaultedServiceList.Items[0].Spec.IPFamilies = []api.IPFamily{api.IPv6Protocol}
// make dualstacked. if needed
if testCase.isDualStack {
defaultedServiceList.Items[0].Spec.IPFamilyPolicy = &preferDualStack
defaultedServiceList.Items[0].Spec.IPFamilies = append(defaultedServiceList.Items[0].Spec.IPFamilies, api.IPv4Protocol)
}
} else {
// no selector gets both families
defaultedServiceList.Items[1].Spec.IPFamilyPolicy = &preferDualStack
defaultedServiceList.Items[1].Spec.IPFamilies = []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol}
// assume single stack for w/selector
defaultedServiceList.Items[0].Spec.IPFamilies = []api.IPFamily{api.IPv4Protocol}
// make dualstacked. if needed
if testCase.isDualStack {
defaultedServiceList.Items[0].Spec.IPFamilyPolicy = &preferDualStack
defaultedServiceList.Items[0].Spec.IPFamilies = append(defaultedServiceList.Items[0].Spec.IPFamilies, api.IPv6Protocol)
}
}
// data is now ready for testing over various cluster configuration
compareSvc := func(out api.Service, expected api.Service) {
if expected.Spec.IPFamilyPolicy == nil && out.Spec.IPFamilyPolicy != nil {
t.Fatalf("service %+v expected IPFamilyPolicy to be nil", out)
}
if expected.Spec.IPFamilyPolicy != nil && out.Spec.IPFamilyPolicy == nil {
t.Fatalf("service %+v expected IPFamilyPolicy not to be nil", out)
}
if expected.Spec.IPFamilyPolicy != nil {
if *out.Spec.IPFamilyPolicy != *expected.Spec.IPFamilyPolicy {
t.Fatalf("service %+v expected IPFamilyPolicy %v got %v", out, *expected.Spec.IPFamilyPolicy, *out.Spec.IPFamilyPolicy)
}
}
if len(out.Spec.IPFamilies) != len(expected.Spec.IPFamilies) {
t.Fatalf("service %+v expected len(IPFamilies) == %v", out, len(expected.Spec.IPFamilies))
}
for i, ipfamily := range out.Spec.IPFamilies {
if expected.Spec.IPFamilies[i] != ipfamily {
t.Fatalf("service %+v expected ip families %+v", out, expected.Spec.IPFamilies)
}
}
}
copyUndefaultedList := undefaultedServiceList.DeepCopy()
// run for each service
for i, svc := range copyUndefaultedList.Items {
storage.defaultServiceOnRead(&svc)
compareSvc(svc, defaultedServiceList.Items[i])
}
copyUndefaultedList = undefaultedServiceList.DeepCopy()
// run as a servicr list
storage.defaultServiceOnRead(copyUndefaultedList)
for i, svc := range copyUndefaultedList.Items {
compareSvc(svc, defaultedServiceList.Items[i])
}
// if there are more tests needed then the last call need to work
// with copy of undefaulted list since
})
}
}

176
pkg/registry/core/service/strategy.go
View File

@ -95,12 +95,8 @@ func (strategy svcStrategy) PrepareForCreate(ctx context.Context, obj runtime.Ob
service := obj.(*api.Service)
service.Status = api.ServiceStatus{}
if utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) && service.Spec.IPFamily == nil {
family := strategy.ipFamilies[0]
service.Spec.IPFamily = &family
}
dropServiceDisabledFields(service, nil)
normalizeClusterIPs(nil, service)
strategy.dropServiceDisabledFields(service, nil)
}
// PrepareForUpdate sets contextual defaults and clears fields that are not allowed to be set by end users on update.
@ -109,23 +105,18 @@ func (strategy svcStrategy) PrepareForUpdate(ctx context.Context, obj, old runti
oldService := old.(*api.Service)
newService.Status = oldService.Status
if utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) && newService.Spec.IPFamily == nil {
if oldService.Spec.IPFamily != nil {
newService.Spec.IPFamily = oldService.Spec.IPFamily
} else {
family := strategy.ipFamilies[0]
newService.Spec.IPFamily = &family
}
}
dropServiceDisabledFields(newService, oldService)
normalizeClusterIPs(oldService, newService)
strategy.dropServiceDisabledFields(newService, oldService)
// if service was converted from ClusterIP => ExternalName
// then clear ClusterIPs, IPFamilyPolicy and IPFamilies
clearClusterIPRelatedFields(newService, oldService)
trimFieldsForDualStackDowngrade(newService, oldService)
}
// Validate validates a new service.
func (strategy svcStrategy) Validate(ctx context.Context, obj runtime.Object) field.ErrorList {
service := obj.(*api.Service)
allErrs := validation.ValidateServiceCreate(service)
allErrs = append(allErrs, validation.ValidateConditionalService(service, nil, strategy.ipFamilies)...)
return allErrs
}
@ -139,7 +130,6 @@ func (svcStrategy) AllowCreateOnUpdate() bool {
func (strategy svcStrategy) ValidateUpdate(ctx context.Context, obj, old runtime.Object) field.ErrorList {
allErrs := validation.ValidateServiceUpdate(obj.(*api.Service), old.(*api.Service))
allErrs = append(allErrs, validation.ValidateConditionalService(obj.(*api.Service), old.(*api.Service), strategy.ipFamilies)...)
return allErrs
}
@ -158,8 +148,10 @@ func (svcStrategy) Export(ctx context.Context, obj runtime.Object, exact bool) e
if exact {
return nil
}
if t.Spec.ClusterIP != api.ClusterIPNone {
//set ClusterIPs as nil - if ClusterIPs[0] != None
if len(t.Spec.ClusterIPs) > 0 && t.Spec.ClusterIPs[0] != api.ClusterIPNone {
t.Spec.ClusterIP = ""
t.Spec.ClusterIPs = nil
}
if t.Spec.Type == api.ServiceTypeNodePort {
for i := range t.Spec.Ports {
@ -174,10 +166,13 @@ func (svcStrategy) Export(ctx context.Context, obj runtime.Object, exact bool) e
// if !utilfeature.DefaultFeatureGate.Enabled(features.MyFeature) && !myFeatureInUse(oldSvc) {
// newSvc.Spec.MyFeature = nil
// }
func dropServiceDisabledFields(newSvc *api.Service, oldSvc *api.Service) {
// Drop IPFamily if DualStack is not enabled
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) && !serviceIPFamilyInUse(oldSvc) {
newSvc.Spec.IPFamily = nil
func (strategy svcStrategy) dropServiceDisabledFields(newSvc *api.Service, oldSvc *api.Service) {
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) && !strategy.serviceDualStackFieldsInUse(oldSvc) {
newSvc.Spec.IPFamilies = nil
newSvc.Spec.IPFamilyPolicy = nil
if len(newSvc.Spec.ClusterIPs) > 1 {
newSvc.Spec.ClusterIPs = newSvc.Spec.ClusterIPs[0:1]
}
}
// Drop TopologyKeys if ServiceTopology is not enabled
@ -187,14 +182,16 @@ func dropServiceDisabledFields(newSvc *api.Service, oldSvc *api.Service) {
}
// returns true if svc.Spec.ServiceIPFamily field is in use
func serviceIPFamilyInUse(svc *api.Service) bool {
func (strategy svcStrategy) serviceDualStackFieldsInUse(svc *api.Service) bool {
if svc == nil {
return false
}
if svc.Spec.IPFamily != nil {
return true
}
return false
ipFamilyPolicyInUse := svc.Spec.IPFamilyPolicy != nil
ipFamiliesInUse := len(svc.Spec.IPFamilies) > 0
ClusterIPsInUse := len(svc.Spec.ClusterIPs) > 1
return ipFamilyPolicyInUse || ipFamiliesInUse || ClusterIPsInUse
}
// returns true if svc.Spec.TopologyKeys field is in use
@ -226,3 +223,126 @@ func (serviceStatusStrategy) PrepareForUpdate(ctx context.Context, obj, old runt
func (serviceStatusStrategy) ValidateUpdate(ctx context.Context, obj, old runtime.Object) field.ErrorList {
return validation.ValidateServiceStatusUpdate(obj.(*api.Service), old.(*api.Service))
}
// normalizeClusterIPs adjust clusterIPs based on ClusterIP
func normalizeClusterIPs(oldSvc *api.Service, newSvc *api.Service) {
// In all cases here, we don't need to over-think the inputs. Validation
// will be called on the new object soon enough. All this needs to do is
// try to divine what user meant with these linked fields. The below
// is verbosely written for clarity.
// **** IMPORTANT *****
// as a governing rule. User must (either)
// -- Use singular only (old client)
// -- singular and plural fields (new clients)
if oldSvc == nil {
// This was a create operation.
// User specified singular and not plural (e.g. an old client), so init
// plural for them.
if len(newSvc.Spec.ClusterIP) > 0 && len(newSvc.Spec.ClusterIPs) == 0 {
newSvc.Spec.ClusterIPs = []string{newSvc.Spec.ClusterIP}
return
}
// we don't init singular based on plural because
// new client must use both fields
// Either both were not specified (will be allocated) or both were
// specified (will be validated).
return
}
// This was an update operation
// ClusterIPs were cleared by an old client which was trying to patch
// some field and didn't provide ClusterIPs
if len(oldSvc.Spec.ClusterIPs) > 0 && len(newSvc.Spec.ClusterIPs) == 0 {
// if ClusterIP is the same, then it is an old client trying to
// patch service and didn't provide ClusterIPs
if oldSvc.Spec.ClusterIP == newSvc.Spec.ClusterIP {
newSvc.Spec.ClusterIPs = oldSvc.Spec.ClusterIPs
}
}
// clusterIP is not the same
if oldSvc.Spec.ClusterIP != newSvc.Spec.ClusterIP {
// this is a client trying to clear it
if len(oldSvc.Spec.ClusterIP) > 0 && len(newSvc.Spec.ClusterIP) == 0 {
// if clusterIPs are the same, then clear on their behalf
if sameStringSlice(oldSvc.Spec.ClusterIPs, newSvc.Spec.ClusterIPs) {
newSvc.Spec.ClusterIPs = nil
}
// if they provided nil, then we are fine (handled by patching case above)
// if they changed it then validation will catch it
} else {
// ClusterIP has changed but not cleared *and* ClusterIPs are the same
// then we set ClusterIPs based on ClusterIP
if sameStringSlice(oldSvc.Spec.ClusterIPs, newSvc.Spec.ClusterIPs) {
newSvc.Spec.ClusterIPs = []string{newSvc.Spec.ClusterIP}
}
}
}
}
func sameStringSlice(a []string, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
// clearClusterIPRelatedFields ensures a backward compatible behavior when the user uses
// an older client to convert a service from ClusterIP to ExternalName. We do that by removing
// the newly introduced fields.
func clearClusterIPRelatedFields(newService, oldService *api.Service) {
if newService.Spec.Type == api.ServiceTypeExternalName && oldService.Spec.Type != api.ServiceTypeExternalName {
// IMPORTANT: this function is always called AFTER ClusterIPs normalization
// which clears ClusterIPs according to ClusterIP. The below checks for ClusterIP
clusterIPReset := len(newService.Spec.ClusterIP) == 0 && len(oldService.Spec.ClusterIP) > 0
if clusterIPReset {
// reset other fields
newService.Spec.ClusterIP = ""
newService.Spec.ClusterIPs = nil
newService.Spec.IPFamilies = nil
newService.Spec.IPFamilyPolicy = nil
}
}
}
// this func allows user to downgrade a service by just changing
// IPFamilyPolicy to SingleStack
func trimFieldsForDualStackDowngrade(newService, oldService *api.Service) {
if !utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) {
return
}
// not an update
if oldService == nil {
return
}
oldIsDualStack := oldService.Spec.IPFamilyPolicy != nil &&
(*oldService.Spec.IPFamilyPolicy == api.IPFamilyPolicyRequireDualStack ||
*oldService.Spec.IPFamilyPolicy == api.IPFamilyPolicyPreferDualStack)
newIsNotDualStack := newService.Spec.IPFamilyPolicy != nil && *newService.Spec.IPFamilyPolicy == api.IPFamilyPolicySingleStack
// if user want to downgrade then we auto remove secondary ip and family
if oldIsDualStack && newIsNotDualStack {
if len(newService.Spec.ClusterIPs) > 1 {
newService.Spec.ClusterIPs = newService.Spec.ClusterIPs[0:1]
}
if len(newService.Spec.IPFamilies) > 1 {
newService.Spec.IPFamilies = newService.Spec.IPFamilies[0:1]
}
}
}

903
pkg/registry/core/service/strategy_test.go
View File

@ -83,7 +83,7 @@ func TestExportService(t *testing.T) {
Namespace: "bar",
},
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.1",
ClusterIPs: []string{"10.0.0.1"},
},
Status: api.ServiceStatus{
LoadBalancer: api.LoadBalancerStatus{
@ -99,10 +99,38 @@ func TestExportService(t *testing.T) {
Namespace: "bar",
},
Spec: api.ServiceSpec{
ClusterIP: "",
ClusterIPs: nil,
},
},
},
{
objIn: &api.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "foo",
Namespace: "bar",
},
Spec: api.ServiceSpec{
ClusterIPs: []string{"10.0.0.1", "2001::1"},
},
Status: api.ServiceStatus{
LoadBalancer: api.LoadBalancerStatus{
Ingress: []api.LoadBalancerIngress{
{IP: "1.2.3.4"},
},
},
},
},
objOut: &api.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "foo",
Namespace: "bar",
},
Spec: api.ServiceSpec{
ClusterIPs: nil,
},
},
},
{
objIn: &api.Pod{},
expectErr: true,
@ -146,7 +174,6 @@ func TestCheckGeneratedNameError(t *testing.T) {
}
func makeValidService() api.Service {
defaultServiceIPFamily := api.IPv4Protocol
return api.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "valid",
@ -160,137 +187,16 @@ func makeValidService() api.Service {
SessionAffinity: "None",
Type: api.ServiceTypeClusterIP,
Ports: []api.ServicePort{{Name: "p", Protocol: "TCP", Port: 8675, TargetPort: intstr.FromInt(8675)}},
IPFamily: &defaultServiceIPFamily,
},
}
}
// TODO: This should be done on types that are not part of our API
func TestBeforeCreate(t *testing.T) {
withIP := func(family *api.IPFamily, ip string) *api.Service {
svc := makeValidService()
svc.Spec.IPFamily = family
svc.Spec.ClusterIP = ip
return &svc
}
ipv4 := api.IPv4Protocol
ipv6 := api.IPv6Protocol
testCases := []struct {
name string
cidr string
configureDualStack bool
enableDualStack bool
in *api.Service
expect *api.Service
expectErr bool
}{
{
name: "does not set ipfamily when dual stack gate is disabled",
cidr: "10.0.0.0/16",
in: withIP(nil, ""),
expect: withIP(nil, ""),
},
{
name: "clears ipfamily when dual stack gate is disabled",
cidr: "10.0.0.0/16",
in: withIP(&ipv4, ""),
expect: withIP(nil, ""),
},
{
name: "allows ipfamily to configured ipv4 value",
cidr: "10.0.0.0/16",
enableDualStack: true,
in: withIP(nil, ""),
expect: withIP(&ipv4, ""),
},
{
name: "allows ipfamily to configured ipv4 value when dual stack is in use",
cidr: "10.0.0.0/16",
enableDualStack: true,
configureDualStack: true,
in: withIP(nil, ""),
expect: withIP(&ipv4, ""),
},
{
name: "allows ipfamily to configured ipv6 value",
cidr: "fd00::/64",
enableDualStack: true,
in: withIP(nil, ""),
expect: withIP(&ipv6, ""),
},
{
name: "allows ipfamily to configured ipv6 value when dual stack is in use",
cidr: "fd00::/64",
enableDualStack: true,
configureDualStack: true,
in: withIP(nil, ""),
expect: withIP(&ipv6, ""),
},
{
name: "rejects ipv6 ipfamily when single-stack ipv4",
enableDualStack: true,
cidr: "10.0.0.0/16",
in: withIP(&ipv6, ""),
expectErr: true,
},
{
name: "rejects ipv4 ipfamily when single-stack ipv6",
enableDualStack: true,
cidr: "fd00::/64",
in: withIP(&ipv4, ""),
expectErr: true,
},
{
name: "rejects implicit ipv4 ipfamily when single-stack ipv6",
enableDualStack: true,
cidr: "fd00::/64",
in: withIP(nil, "10.0.1.0"),
expectErr: true,
},
{
name: "rejects implicit ipv6 ipfamily when single-stack ipv4",
enableDualStack: true,
cidr: "10.0.0.0/16",
in: withIP(nil, "fd00::1"),
expectErr: true,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, tc.enableDualStack)()
testStrategy, _ := newStrategy(tc.cidr, tc.configureDualStack)
ctx := genericapirequest.NewDefaultContext()
err := rest.BeforeCreate(testStrategy, ctx, runtime.Object(tc.in))
if tc.expectErr != (err != nil) {
t.Fatalf("unexpected error: %v", err)
}
if err != nil {
return
}
if tc.expect != nil && tc.in != nil {
tc.expect.ObjectMeta = tc.in.ObjectMeta
}
if !reflect.DeepEqual(tc.expect, tc.in) {
t.Fatalf("unexpected change: %s", diff.ObjectReflectDiff(tc.expect, tc.in))
}
})
}
}
func TestBeforeUpdate(t *testing.T) {
testCases := []struct {
name string
enableDualStack bool
defaultIPv6 bool
allowSecondary bool
tweakSvc func(oldSvc, newSvc *api.Service) // given basic valid services, each test case can customize them
expectErr bool
expectObj func(t *testing.T, svc *api.Service)
name string
tweakSvc func(oldSvc, newSvc *api.Service) // given basic valid services, each test case can customize them
expectErr bool
}{
{
name: "no change",
@ -323,24 +229,11 @@ func TestBeforeUpdate(t *testing.T) {
{
name: "change ClusterIP",
tweakSvc: func(oldSvc, newSvc *api.Service) {
oldSvc.Spec.ClusterIP = "1.2.3.4"
newSvc.Spec.ClusterIP = "4.3.2.1"
oldSvc.Spec.ClusterIPs = []string{"1.2.3.4"}
newSvc.Spec.ClusterIPs = []string{"4.3.2.1"}
},
expectErr: true,
},
{
name: "clear IP family is allowed (defaulted back by before update)",
enableDualStack: true,
tweakSvc: func(oldSvc, newSvc *api.Service) {
oldSvc.Spec.IPFamily = nil
},
expectErr: false,
expectObj: func(t *testing.T, svc *api.Service) {
if svc.Spec.IPFamily == nil {
t.Errorf("ipfamily was not defaulted")
}
},
},
{
name: "change selector",
tweakSvc: func(oldSvc, newSvc *api.Service) {
@ -351,32 +244,22 @@ func TestBeforeUpdate(t *testing.T) {
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, tc.enableDualStack)()
var cidr string
if tc.defaultIPv6 {
cidr = "ffd0::/64"
} else {
cidr = "172.30.0.0/16"
}
strategy, _ := newStrategy(cidr, tc.allowSecondary)
oldSvc := makeValidService()
newSvc := makeValidService()
tc.tweakSvc(&oldSvc, &newSvc)
ctx := genericapirequest.NewDefaultContext()
err := rest.BeforeUpdate(strategy, ctx, runtime.Object(&newSvc), runtime.Object(&oldSvc))
if tc.expectObj != nil {
tc.expectObj(t, &newSvc)
}
if tc.expectErr && err == nil {
t.Fatalf("unexpected non-error: %v", err)
}
if !tc.expectErr && err != nil {
t.Fatalf("unexpected error: %v", err)
}
})
strategy, _ := newStrategy("172.30.0.0/16", false)
oldSvc := makeValidService()
newSvc := makeValidService()
tc.tweakSvc(&oldSvc, &newSvc)
ctx := genericapirequest.NewDefaultContext()
err := rest.BeforeUpdate(strategy, ctx, runtime.Object(&oldSvc), runtime.Object(&newSvc))
if tc.expectErr && err == nil {
t.Errorf("unexpected non-error for %q", tc.name)
}
if !tc.expectErr && err != nil {
t.Errorf("unexpected error for %q: %v", tc.name, err)
}
}
}
func TestServiceStatusStrategy(t *testing.T) {
_, testStatusStrategy := newStrategy("10.0.0.0/16", false)
ctx := genericapirequest.NewDefaultContext()
@ -408,16 +291,20 @@ func TestServiceStatusStrategy(t *testing.T) {
}
}
func makeServiceWithIPFamily(ipFamily *api.IPFamily) *api.Service {
func makeServiceWithIPFamilies(ipfamilies []api.IPFamily, ipFamilyPolicy *api.IPFamilyPolicyType) *api.Service {
return &api.Service{
Spec: api.ServiceSpec{
IPFamily: ipFamily,
IPFamilies: ipfamilies,
IPFamilyPolicy: ipFamilyPolicy,
},
}
}
func TestDropDisabledField(t *testing.T) {
ipv4Service := api.IPv4Protocol
ipv6Service := api.IPv6Protocol
requireDualStack := api.IPFamilyPolicyRequireDualStack
preferDualStack := api.IPFamilyPolicyPreferDualStack
singleStack := api.IPFamilyPolicySingleStack
testCases := []struct {
name string
enableDualStack bool
@ -428,44 +315,60 @@ func TestDropDisabledField(t *testing.T) {
{
name: "not dual stack, field not used",
enableDualStack: false,
svc: makeServiceWithIPFamily(nil),
svc: makeServiceWithIPFamilies(nil, nil),
oldSvc: nil,
compareSvc: makeServiceWithIPFamily(nil),
},
{
name: "not dual stack, field used in new, not in old",
enableDualStack: false,
svc: makeServiceWithIPFamily(&ipv4Service),
oldSvc: nil,
compareSvc: makeServiceWithIPFamily(nil),
compareSvc: makeServiceWithIPFamilies(nil, nil),
},
{
name: "not dual stack, field used in old and new",
enableDualStack: false,
svc: makeServiceWithIPFamily(&ipv4Service),
oldSvc: makeServiceWithIPFamily(&ipv4Service),
compareSvc: makeServiceWithIPFamily(&ipv4Service),
svc: makeServiceWithIPFamilies([]api.IPFamily{api.IPv4Protocol}, nil),
oldSvc: makeServiceWithIPFamilies([]api.IPFamily{api.IPv4Protocol}, nil),
compareSvc: makeServiceWithIPFamilies([]api.IPFamily{api.IPv4Protocol}, nil),
},
{
name: "dualstack, field used",
enableDualStack: true,
svc: makeServiceWithIPFamily(&ipv6Service),
svc: makeServiceWithIPFamilies([]api.IPFamily{api.IPv6Protocol}, nil),
oldSvc: nil,
compareSvc: makeServiceWithIPFamily(&ipv6Service),
compareSvc: makeServiceWithIPFamilies([]api.IPFamily{api.IPv6Protocol}, nil),
},
/* preferDualStack field */
{
name: "not dual stack, fields is not use",
enableDualStack: false,
svc: makeServiceWithIPFamilies(nil, nil),
oldSvc: nil,
compareSvc: makeServiceWithIPFamilies(nil, nil),
},
{
name: "dualstack, field used, changed",
enableDualStack: true,
svc: makeServiceWithIPFamily(&ipv6Service),
oldSvc: makeServiceWithIPFamily(&ipv4Service),
compareSvc: makeServiceWithIPFamily(&ipv6Service),
name: "not dual stack, fields used in new, not in old",
enableDualStack: false,
svc: makeServiceWithIPFamilies(nil, &preferDualStack),
oldSvc: nil,
compareSvc: makeServiceWithIPFamilies(nil, nil),
},
{
name: "dualstack, field used, not changed",
name: "not dual stack, fields used in new, not in old",
enableDualStack: false,
svc: makeServiceWithIPFamilies(nil, &requireDualStack),
oldSvc: nil,
compareSvc: makeServiceWithIPFamilies(nil, nil),
},
{
name: "not dual stack, fields not used in old (single stack)",
enableDualStack: false,
svc: makeServiceWithIPFamilies(nil, nil),
oldSvc: makeServiceWithIPFamilies(nil, &singleStack),
compareSvc: makeServiceWithIPFamilies(nil, nil),
},
{
name: "dualstack, field used",
enableDualStack: true,
svc: makeServiceWithIPFamily(&ipv6Service),
oldSvc: makeServiceWithIPFamily(&ipv6Service),
compareSvc: makeServiceWithIPFamily(&ipv6Service),
svc: makeServiceWithIPFamilies(nil, &singleStack),
oldSvc: nil,
compareSvc: makeServiceWithIPFamilies(nil, &singleStack),
},
/* add more tests for other dropped fields as needed */
@ -474,7 +377,10 @@ func TestDropDisabledField(t *testing.T) {
func() {
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, tc.enableDualStack)()
old := tc.oldSvc.DeepCopy()
dropServiceDisabledFields(tc.svc, tc.oldSvc)
// to test against user using IPFamily not set on cluster
svcStrategy := svcStrategy{ipFamilies: []api.IPFamily{api.IPv4Protocol}}
svcStrategy.dropServiceDisabledFields(tc.svc, tc.oldSvc)
// old node should never be changed
if !reflect.DeepEqual(tc.oldSvc, old) {
@ -488,3 +394,612 @@ func TestDropDisabledField(t *testing.T) {
}
}
func TestNormalizeClusterIPs(t *testing.T) {
testCases := []struct {
name string
oldService *api.Service
newService *api.Service
expectedClusterIP string
expectedClusterIPs []string
}{
{
name: "new - only clusterip used",
oldService: nil,
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: nil,
},
},
expectedClusterIP: "10.0.0.10",
expectedClusterIPs: []string{"10.0.0.10"},
},
{
name: "new - only clusterips used",
oldService: nil,
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "",
ClusterIPs: []string{"10.0.0.10"},
},
},
expectedClusterIP: "", // this is a validation issue, and validation will catch it
expectedClusterIPs: []string{"10.0.0.10"},
},
{
name: "new - both used",
oldService: nil,
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10"},
},
},
expectedClusterIP: "10.0.0.10",
expectedClusterIPs: []string{"10.0.0.10"},
},
{
name: "update - no change",
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10"},
},
},
expectedClusterIP: "10.0.0.10",
expectedClusterIPs: []string{"10.0.0.10"},
},
{
name: "update - malformed change",
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.11",
ClusterIPs: []string{"10.0.0.11"},
},
},
expectedClusterIP: "10.0.0.11",
expectedClusterIPs: []string{"10.0.0.11"},
},
{
name: "update - malformed change on secondary ip",
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10", "2000::1"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.11",
ClusterIPs: []string{"10.0.0.11", "3000::1"},
},
},
expectedClusterIP: "10.0.0.11",
expectedClusterIPs: []string{"10.0.0.11", "3000::1"},
},
{
name: "update - upgrade",
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10", "2000::1"},
},
},
expectedClusterIP: "10.0.0.10",
expectedClusterIPs: []string{"10.0.0.10", "2000::1"},
},
{
name: "update - downgrade",
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10", "2000::1"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10"},
},
},
expectedClusterIP: "10.0.0.10",
expectedClusterIPs: []string{"10.0.0.10"},
},
{
name: "update - user cleared cluster IP",
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "",
ClusterIPs: []string{"10.0.0.10"},
},
},
expectedClusterIP: "",
expectedClusterIPs: nil,
},
{
name: "update - user cleared clusterIPs", // *MUST* REMAIN FOR OLD CLIENTS
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: nil,
},
},
expectedClusterIP: "10.0.0.10",
expectedClusterIPs: []string{"10.0.0.10"},
},
{
name: "update - user cleared both",
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "",
ClusterIPs: nil,
},
},
expectedClusterIP: "",
expectedClusterIPs: nil,
},
{
name: "update - user cleared ClusterIP but changed clusterIPs",
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "",
ClusterIPs: []string{"10.0.0.11"},
},
},
expectedClusterIP: "", /* validation catches this */
expectedClusterIPs: []string{"10.0.0.11"},
},
{
name: "update - user cleared ClusterIPs but changed ClusterIP",
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10", "2000::1"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.11",
ClusterIPs: nil,
},
},
expectedClusterIP: "10.0.0.11",
expectedClusterIPs: nil,
},
{
name: "update - user changed from None to ClusterIP",
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "None",
ClusterIPs: []string{"None"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"None"},
},
},
expectedClusterIP: "10.0.0.10",
expectedClusterIPs: []string{"10.0.0.10"},
},
{
name: "update - user changed from ClusterIP to None",
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "None",
ClusterIPs: []string{"10.0.0.10"},
},
},
expectedClusterIP: "None",
expectedClusterIPs: []string{"None"},
},
{
name: "update - user changed from ClusterIP to None and changed ClusterIPs in a dual stack (new client making a mistake)",
oldService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "10.0.0.10",
ClusterIPs: []string{"10.0.0.10", "2000::1"},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
ClusterIP: "None",
ClusterIPs: []string{"10.0.0.11", "2000::1"},
},
},
expectedClusterIP: "None",
expectedClusterIPs: []string{"10.0.0.11", "2000::1"},
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
normalizeClusterIPs(tc.oldService, tc.newService)
if tc.newService == nil {
t.Fatalf("unexpected new service to be nil")
}
if tc.newService.Spec.ClusterIP != tc.expectedClusterIP {
t.Fatalf("expected clusterIP [%v] got [%v]", tc.expectedClusterIP, tc.newService.Spec.ClusterIP)
}
if len(tc.newService.Spec.ClusterIPs) != len(tc.expectedClusterIPs) {
t.Fatalf("expected clusterIPs %v got %v", tc.expectedClusterIPs, tc.newService.Spec.ClusterIPs)
}
for idx, clusterIP := range tc.newService.Spec.ClusterIPs {
if clusterIP != tc.expectedClusterIPs[idx] {
t.Fatalf("expected clusterIP [%v] at index[%v] got [%v]", tc.expectedClusterIPs[idx], idx, tc.newService.Spec.ClusterIPs[idx])
}
}
})
}
}
func TestClearClusterIPRelatedFields(t *testing.T) {
//
// NOTE the data fed to this test assums that ClusterIPs normalization is
// already done check PrepareFor*(..) strategy
//
singleStack := api.IPFamilyPolicySingleStack
requireDualStack := api.IPFamilyPolicyRequireDualStack
testCases := []struct {
name string
oldService *api.Service
newService *api.Service
shouldClear bool
}{
{
name: "should clear, single stack converting to external name",
shouldClear: true,
oldService: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
IPFamilyPolicy: &singleStack,
ClusterIP: "10.0.0.4",
ClusterIPs: []string{"10.0.0.4"},
IPFamilies: []api.IPFamily{api.IPv4Protocol},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeExternalName,
IPFamilyPolicy: &singleStack,
ClusterIP: "",
ClusterIPs: nil,
IPFamilies: []api.IPFamily{api.IPv4Protocol},
},
},
},
{
name: "should clear, dual stack converting to external name(normalization removed all ips)",
shouldClear: true,
oldService: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
IPFamilyPolicy: &requireDualStack,
ClusterIP: "2000::1",
ClusterIPs: []string{"2000::1", "10.0.0.4"},
IPFamilies: []api.IPFamily{api.IPv6Protocol, api.IPv4Protocol},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeExternalName,
IPFamilyPolicy: &singleStack,
ClusterIP: "",
ClusterIPs: nil,
IPFamilies: []api.IPFamily{api.IPv4Protocol},
},
},
},
{
name: "should NOT clear, single stack converting to external name ClusterIPs was not cleared",
shouldClear: false,
oldService: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
IPFamilyPolicy: &singleStack,
ClusterIP: "2000::1",
ClusterIPs: []string{"2000::1"},
IPFamilies: []api.IPFamily{api.IPv6Protocol},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeExternalName,
IPFamilyPolicy: &singleStack,
ClusterIP: "2000::1",
ClusterIPs: []string{"2000::1"},
IPFamilies: []api.IPFamily{api.IPv4Protocol},
},
},
},
{
name: "should NOT clear, dualstack cleared primary and changed ClusterIPs",
shouldClear: true,
oldService: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
IPFamilyPolicy: &requireDualStack,
ClusterIP: "2000::1",
ClusterIPs: []string{"2000::1", "10.0.0.4"},
IPFamilies: []api.IPFamily{api.IPv6Protocol, api.IPv4Protocol},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeExternalName,
IPFamilyPolicy: &singleStack,
ClusterIP: "",
ClusterIPs: []string{"2000::1", "10.0.0.5"},
IPFamilies: []api.IPFamily{api.IPv4Protocol},
},
},
},
{
name: "should clear, dualstack user removed ClusterIPs",
shouldClear: true,
oldService: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
IPFamilyPolicy: &requireDualStack,
ClusterIP: "2000::1",
ClusterIPs: []string{"2000::1", "10.0.0.4"},
IPFamilies: []api.IPFamily{api.IPv6Protocol, api.IPv4Protocol},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeExternalName,
IPFamilyPolicy: &singleStack,
ClusterIP: "",
ClusterIPs: nil,
IPFamilies: []api.IPFamily{api.IPv4Protocol},
},
},
},
{
name: "should NOT clear, dualstack service changing selector",
shouldClear: false,
oldService: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
Selector: map[string]string{"foo": "bar"},
IPFamilyPolicy: &requireDualStack,
ClusterIP: "2000::1",
ClusterIPs: []string{"2000::1", "10.0.0.4"},
IPFamilies: []api.IPFamily{api.IPv6Protocol, api.IPv4Protocol},
},
},
newService: &api.Service{
Spec: api.ServiceSpec{
Type: api.ServiceTypeClusterIP,
Selector: map[string]string{"foo": "baz"},
IPFamilyPolicy: &singleStack,
ClusterIP: "2000::1",
ClusterIPs: []string{"2000::1", "10.0.0.4"},
IPFamilies: []api.IPFamily{api.IPv4Protocol},
},
},
},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
clearClusterIPRelatedFields(testCase.newService, testCase.oldService)
if testCase.shouldClear && len(testCase.newService.Spec.ClusterIPs) != 0 {
t.Fatalf("expected clusterIPs to be cleared")
}
if testCase.shouldClear && len(testCase.newService.Spec.IPFamilies) != 0 {
t.Fatalf("expected ipfamilies to be cleared")
}
if testCase.shouldClear && testCase.newService.Spec.IPFamilyPolicy != nil {
t.Fatalf("expected ipfamilypolicy to be cleared")
}
if !testCase.shouldClear && len(testCase.newService.Spec.ClusterIPs) == 0 {
t.Fatalf("expected clusterIPs NOT to be cleared")
}
if !testCase.shouldClear && len(testCase.newService.Spec.IPFamilies) == 0 {
t.Fatalf("expected ipfamilies NOT to be cleared")
}
if !testCase.shouldClear && testCase.newService.Spec.IPFamilyPolicy == nil {
t.Fatalf("expected ipfamilypolicy NOT to be cleared")
}
})
}
}
func TestTrimFieldsForDualStackDowngrade(t *testing.T) {
singleStack := api.IPFamilyPolicySingleStack
preferDualStack := api.IPFamilyPolicyPreferDualStack
requireDualStack := api.IPFamilyPolicyRequireDualStack
testCases := []struct {
name string
oldPolicy *api.IPFamilyPolicyType
oldClusterIPs []string
oldFamilies []api.IPFamily
newPolicy *api.IPFamilyPolicyType
expectedClusterIPs []string
expectedIPFamilies []api.IPFamily
}{
{
name: "no change single to single",
oldPolicy: &singleStack,
oldClusterIPs: []string{"10.10.10.10"},
oldFamilies: []api.IPFamily{api.IPv4Protocol},
newPolicy: &singleStack,
expectedClusterIPs: []string{"10.10.10.10"},
expectedIPFamilies: []api.IPFamily{api.IPv4Protocol},
},
{
name: "dualstack to dualstack (preferred)",
oldPolicy: &preferDualStack,
oldClusterIPs: []string{"10.10.10.10", "2000::1"},
oldFamilies: []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol},
newPolicy: &preferDualStack,
expectedClusterIPs: []string{"10.10.10.10", "2000::1"},
expectedIPFamilies: []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol},
},
{
name: "dualstack to dualstack (required)",
oldPolicy: &requireDualStack,
oldClusterIPs: []string{"10.10.10.10", "2000::1"},
oldFamilies: []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol},
newPolicy: &preferDualStack,
expectedClusterIPs: []string{"10.10.10.10", "2000::1"},
expectedIPFamilies: []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol},
},
{
name: "dualstack (preferred) to single",
oldPolicy: &preferDualStack,
oldClusterIPs: []string{"10.10.10.10", "2000::1"},
oldFamilies: []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol},
newPolicy: &singleStack,
expectedClusterIPs: []string{"10.10.10.10"},
expectedIPFamilies: []api.IPFamily{api.IPv4Protocol},
},
{
name: "dualstack (require) to single",
oldPolicy: &requireDualStack,
oldClusterIPs: []string{"2000::1", "10.10.10.10"},
oldFamilies: []api.IPFamily{api.IPv6Protocol, api.IPv4Protocol},
newPolicy: &singleStack,
expectedClusterIPs: []string{"2000::1"},
expectedIPFamilies: []api.IPFamily{api.IPv6Protocol},
},
}
// only when gate is on
defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, true)()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
oldService := &api.Service{
Spec: api.ServiceSpec{
IPFamilyPolicy: tc.oldPolicy,
ClusterIPs: tc.oldClusterIPs,
IPFamilies: tc.oldFamilies,
},
}
newService := oldService.DeepCopy()
newService.Spec.IPFamilyPolicy = tc.newPolicy
trimFieldsForDualStackDowngrade(newService, oldService)
if len(newService.Spec.ClusterIPs) != len(tc.expectedClusterIPs) {
t.Fatalf("unexpected clusterIPs. expected %v and got %v", tc.expectedClusterIPs, newService.Spec.ClusterIPs)
}
// compare clusterIPS
for i, expectedIP := range tc.expectedClusterIPs {
if expectedIP != newService.Spec.ClusterIPs[i] {
t.Fatalf("unexpected clusterIPs. expected %v and got %v", tc.expectedClusterIPs, newService.Spec.ClusterIPs)
}
}
// families
if len(newService.Spec.IPFamilies) != len(tc.expectedIPFamilies) {
t.Fatalf("unexpected ipfamilies. expected %v and got %v", tc.expectedIPFamilies, newService.Spec.IPFamilies)
}
// compare clusterIPS
for i, expectedIPFamily := range tc.expectedIPFamilies {
if expectedIPFamily != newService.Spec.IPFamilies[i] {
t.Fatalf("unexpected ipfamilies. expected %v and got %v", tc.expectedIPFamilies, newService.Spec.IPFamilies)
}
}
})
}
}

1918
staging/src/k8s.io/api/core/v1/generated.pb.go generated
View File

File diff suppressed because it is too large Load Diff

46
staging/src/k8s.io/api/core/v1/generated.proto
View File

@ -4802,6 +4802,17 @@ message ServiceSpec {
// +optional
optional string clusterIP = 3;
// ClusterIPs identifies all the ClusterIPs assigned to this
// service. ClusterIPs are assigned or reserved based on the values of
// service.spec.ipFamilies. A maximum of two entries (dual-stack IPs) are
// allowed in ClusterIPs. The IPFamily of each ClusterIP must match
// values provided in service.spec.ipFamilies. Clients using ClusterIPs must
// keep it in sync with ClusterIP (if provided) by having ClusterIP matching
// first element of ClusterIPs.
// +listType=atomic
// +optional
repeated string clusterIPs = 18;
// type determines how the Service is exposed. Defaults to ClusterIP. Valid
// options are ExternalName, ClusterIP, NodePort, and LoadBalancer.
// "ExternalName" maps to the specified externalName.
@ -4889,23 +4900,15 @@ message ServiceSpec {
// +optional
optional SessionAffinityConfig sessionAffinityConfig = 14;
// ipFamily specifies whether this Service has a preference for a particular IP family (e.g.
// IPv4 vs. IPv6) when the IPv6DualStack feature gate is enabled. In a dual-stack cluster,
// you can specify ipFamily when creating a ClusterIP Service to determine whether the
// controller will allocate an IPv4 or IPv6 IP for it, and you can specify ipFamily when
// creating a headless Service to determine whether it will have IPv4 or IPv6 Endpoints. In
// either case, if you do not specify an ipFamily explicitly, it will default to the
// cluster's primary IP family.
// This field is part of an alpha feature, and you should not make any assumptions about its
// semantics other than those described above. In particular, you should not assume that it
// can (or cannot) be changed after creation time; that it can only have the values "IPv4"
// and "IPv6"; or that its current value on a given Service correctly reflects the current
// state of that Service. (For ClusterIP Services, look at clusterIP to see if the Service
// is IPv4 or IPv6. For headless Services, look at the endpoints, which may be dual-stack in
// the future. For ExternalName Services, ipFamily has no meaning, but it may be set to an
// irrelevant value anyway.)
// IPFamilies identifies all the IPFamilies assigned for this Service. If a value
// was not provided for IPFamilies it will be defaulted based on the cluster
// configuration and the value of service.spec.ipFamilyPolicy. A maximum of two
// values (dual-stack IPFamilies) are allowed in IPFamilies. IPFamilies field is
// conditionally mutable: it allows for adding or removing a secondary IPFamily,
// but it does not allow changing the primary IPFamily of the service.
// +listType=atomic
// +optional
optional string ipFamily = 15;
repeated string ipFamilies = 19;
// topologyKeys is a preference-order list of topology keys which
// implementations of services should use to preferentially sort endpoints
@ -4921,6 +4924,17 @@ message ServiceSpec {
// If this is not specified or empty, no topology constraints will be applied.
// +optional
repeated string topologyKeys = 16;
// IPFamilyPolicy represents the dual-stack-ness requested or required by this
// Service. If there is no value provided, then this Service will be considered
// SingleStack (single IPFamily). Services can be SingleStack (single IPFamily),
// PreferDualStack (two dual-stack IPFamilies on dual-stack clusters or single
// IPFamily on single-stack clusters), or RequireDualStack (two dual-stack IPFamilies
// on dual-stack configured clusters, otherwise fail). The IPFamilies and ClusterIPs assigned
// to this service can be controlled by service.spec.ipFamilies and service.spec.clusterIPs
// respectively.
// +optional
optional string ipFamilyPolicy = 17;
}
// ServiceStatus represents the current status of a service.

78
staging/src/k8s.io/api/core/v1/types.go
View File

@ -3973,8 +3973,13 @@ type LoadBalancerIngress struct {
Hostname string `json:"hostname,omitempty" protobuf:"bytes,2,opt,name=hostname"`
}
const (
// MaxServiceTopologyKeys is the largest number of topology keys allowed on a service
MaxServiceTopologyKeys = 16
)
// IPFamily represents the IP Family (IPv4 or IPv6). This type is used
// to express the family of an IP expressed by a type (i.e. service.Spec.IPFamily)
// to express the family of an IP expressed by a type (e.g. service.spec.ipFamilies).
type IPFamily string
const (
@ -3982,8 +3987,29 @@ const (
IPv4Protocol IPFamily = "IPv4"
// IPv6Protocol indicates that this IP is IPv6 protocol
IPv6Protocol IPFamily = "IPv6"
// MaxServiceTopologyKeys is the largest number of topology keys allowed on a service
MaxServiceTopologyKeys = 16
)
// IPFamilyPolicyType represents the dual-stack-ness requested or required by a Service
type IPFamilyPolicyType string
const (
// IPFamilyPolicySingleStack indicates that this service is required to have a single IPFamily.
// The IPFamily assigned is based on the default IPFamily used by the cluster
// or as identified by service.spec.ipFamilies field
IPFamilyPolicySingleStack IPFamilyPolicyType = "SingleStack"
// IPFamilyPolicyPreferDualStack indicates that this service prefers dual-stack when
// the cluster is configured for dual-stack. If the cluster is not configured
// for dual-stack the service will be assigned a single IPFamily. If the IPFamily is not
// set in service.spec.ipFamilies then the service will be assigned the default IPFamily
// configured on the cluster
IPFamilyPolicyPreferDualStack IPFamilyPolicyType = "PreferDualStack"
// IPFamilyPolicyRequireDualStack indicates that this service requires dual-stack. Using
// IPFamilyPolicyRequireDualStack on a single stack cluster will result in validation errors. The
// IPFamilies (and their order) assigned to this service is based on service.spec.ipFamilies. If
// service.spec.ipFamilies was not provided then it will be assigned according to how they are
// configured on the cluster. If service.spec.ipFamilies has only one entry then the alternative
// IPFamily will be added by apiserver
IPFamilyPolicyRequireDualStack IPFamilyPolicyType = "RequireDualStack"
)
// ServiceSpec describes the attributes that a user creates on a service.
@ -4018,6 +4044,17 @@ type ServiceSpec struct {
// +optional
ClusterIP string `json:"clusterIP,omitempty" protobuf:"bytes,3,opt,name=clusterIP"`
// ClusterIPs identifies all the ClusterIPs assigned to this
// service. ClusterIPs are assigned or reserved based on the values of
// service.spec.ipFamilies. A maximum of two entries (dual-stack IPs) are
// allowed in ClusterIPs. The IPFamily of each ClusterIP must match
// values provided in service.spec.ipFamilies. Clients using ClusterIPs must
// keep it in sync with ClusterIP (if provided) by having ClusterIP matching
// first element of ClusterIPs.
// +listType=atomic
// +optional
ClusterIPs []string `json:"clusterIPs,omitempty" protobuf:"bytes,18,opt,name=clusterIPs"`
// type determines how the Service is exposed. Defaults to ClusterIP. Valid
// options are ExternalName, ClusterIP, NodePort, and LoadBalancer.
// "ExternalName" maps to the specified externalName.
@ -4105,23 +4142,15 @@ type ServiceSpec struct {
// +optional
SessionAffinityConfig *SessionAffinityConfig `json:"sessionAffinityConfig,omitempty" protobuf:"bytes,14,opt,name=sessionAffinityConfig"`
// ipFamily specifies whether this Service has a preference for a particular IP family (e.g.
// IPv4 vs. IPv6) when the IPv6DualStack feature gate is enabled. In a dual-stack cluster,
// you can specify ipFamily when creating a ClusterIP Service to determine whether the
// controller will allocate an IPv4 or IPv6 IP for it, and you can specify ipFamily when
// creating a headless Service to determine whether it will have IPv4 or IPv6 Endpoints. In
// either case, if you do not specify an ipFamily explicitly, it will default to the
// cluster's primary IP family.
// This field is part of an alpha feature, and you should not make any assumptions about its
// semantics other than those described above. In particular, you should not assume that it
// can (or cannot) be changed after creation time; that it can only have the values "IPv4"
// and "IPv6"; or that its current value on a given Service correctly reflects the current
// state of that Service. (For ClusterIP Services, look at clusterIP to see if the Service
// is IPv4 or IPv6. For headless Services, look at the endpoints, which may be dual-stack in
// the future. For ExternalName Services, ipFamily has no meaning, but it may be set to an
// irrelevant value anyway.)
// IPFamilies identifies all the IPFamilies assigned for this Service. If a value
// was not provided for IPFamilies it will be defaulted based on the cluster
// configuration and the value of service.spec.ipFamilyPolicy. A maximum of two
// values (dual-stack IPFamilies) are allowed in IPFamilies. IPFamilies field is
// conditionally mutable: it allows for adding or removing a secondary IPFamily,
// but it does not allow changing the primary IPFamily of the service.
// +listType=atomic
// +optional
IPFamily *IPFamily `json:"ipFamily,omitempty" protobuf:"bytes,15,opt,name=ipFamily,Configcasttype=IPFamily"`
IPFamilies []IPFamily `json:"ipFamilies,omitempty" protobuf:"bytes,19,opt,name=ipFamilies,casttype=IPFamily"`
// topologyKeys is a preference-order list of topology keys which
// implementations of services should use to preferentially sort endpoints
@ -4137,6 +4166,17 @@ type ServiceSpec struct {
// If this is not specified or empty, no topology constraints will be applied.
// +optional
TopologyKeys []string `json:"topologyKeys,omitempty" protobuf:"bytes,16,opt,name=topologyKeys"`
// IPFamilyPolicy represents the dual-stack-ness requested or required by this
// Service. If there is no value provided, then this Service will be considered
// SingleStack (single IPFamily). Services can be SingleStack (single IPFamily),
// PreferDualStack (two dual-stack IPFamilies on dual-stack clusters or single
// IPFamily on single-stack clusters), or RequireDualStack (two dual-stack IPFamilies
// on dual-stack configured clusters, otherwise fail). The IPFamilies and ClusterIPs assigned
// to this service can be controlled by service.spec.ipFamilies and service.spec.clusterIPs
// respectively.
// +optional
IPFamilyPolicy *IPFamilyPolicyType `json:"ipFamilyPolicy,omitempty" protobuf:"bytes,17,opt,name=ipFamilyPolicy,casttype=IPFamilyPolicyType"`
}
// ServicePort contains information on service's port.

4
staging/src/k8s.io/api/core/v1/types_swagger_doc_generated.go generated
View File

@ -2230,6 +2230,7 @@ var map_ServiceSpec = map[string]string{
"ports": "The list of ports that are exposed by this service. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies",
"selector": "Route service traffic to pods with label keys and values matching this selector. If empty or not present, the service is assumed to have an external process managing its endpoints, which Kubernetes will not modify. Only applies to types ClusterIP, NodePort, and LoadBalancer. Ignored if type is ExternalName. More info: https://kubernetes.io/docs/concepts/services-networking/service/",
"clusterIP": "clusterIP is the IP address of the service and is usually assigned randomly by the master. If an address is specified manually and is not in use by others, it will be allocated to the service; otherwise, creation of the service will fail. This field can not be changed through updates. Valid values are \"None\", empty string (\"\"), or a valid IP address. \"None\" can be specified for headless services when proxying is not required. Only applies to types ClusterIP, NodePort, and LoadBalancer. Ignored if type is ExternalName. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies",
"clusterIPs": "ClusterIPs identifies all the ClusterIPs assigned to this service. ClusterIPs are assigned or reserved based on the values of service.spec.ipFamilies. A maximum of two entries (dual-stack IPs) are allowed in ClusterIPs. The IPFamily of each ClusterIP must match values provided in service.spec.ipFamilies. Clients using ClusterIPs must keep it in sync with ClusterIP (if provided) by having ClusterIP matching first element of ClusterIPs.",
"type": "type determines how the Service is exposed. Defaults to ClusterIP. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer. \"ExternalName\" maps to the specified externalName. \"ClusterIP\" allocates a cluster-internal IP address for load-balancing to endpoints. Endpoints are determined by the selector or if that is not specified, by manual construction of an Endpoints object. If clusterIP is \"None\", no virtual IP is allocated and the endpoints are published as a set of endpoints rather than a stable IP. \"NodePort\" builds on ClusterIP and allocates a port on every node which routes to the clusterIP. \"LoadBalancer\" builds on NodePort and creates an external load-balancer (if supported in the current cloud) which routes to the clusterIP. More info: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types",
"externalIPs": "externalIPs is a list of IP addresses for which nodes in the cluster will also accept traffic for this service. These IPs are not managed by Kubernetes. The user is responsible for ensuring that traffic arrives at a node with this IP. A common example is external load-balancers that are not part of the Kubernetes system.",
"sessionAffinity": "Supports \"ClientIP\" and \"None\". Used to maintain session affinity. Enable client IP based session affinity. Must be ClientIP or None. Defaults to None. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies",
@ -2240,8 +2241,9 @@ var map_ServiceSpec = map[string]string{
"healthCheckNodePort": "healthCheckNodePort specifies the healthcheck nodePort for the service. If not specified, HealthCheckNodePort is created by the service api backend with the allocated nodePort. Will use user-specified nodePort value if specified by the client. Only effects when Type is set to LoadBalancer and ExternalTrafficPolicy is set to Local.",
"publishNotReadyAddresses": "publishNotReadyAddresses indicates that any agent which deals with endpoints for this Service should disregard any indications of ready/not-ready. The primary use case for setting this field is for a StatefulSet's Headless Service to propagate SRV DNS records for its Pods for the purpose of peer discovery. The Kubernetes controllers that generate Endpoints and EndpointSlice resources for Services interpret this to mean that all endpoints are considered \"ready\" even if the Pods themselves are not. Agents which consume only Kubernetes generated endpoints through the Endpoints or EndpointSlice resources can safely assume this behavior.",
"sessionAffinityConfig": "sessionAffinityConfig contains the configurations of session affinity.",
"ipFamily": "ipFamily specifies whether this Service has a preference for a particular IP family (e.g. IPv4 vs. IPv6) when the IPv6DualStack feature gate is enabled. In a dual-stack cluster, you can specify ipFamily when creating a ClusterIP Service to determine whether the controller will allocate an IPv4 or IPv6 IP for it, and you can specify ipFamily when creating a headless Service to determine whether it will have IPv4 or IPv6 Endpoints. In either case, if you do not specify an ipFamily explicitly, it will default to the cluster's primary IP family. This field is part of an alpha feature, and you should not make any assumptions about its semantics other than those described above. In particular, you should not assume that it can (or cannot) be changed after creation time; that it can only have the values \"IPv4\" and \"IPv6\"; or that its current value on a given Service correctly reflects the current state of that Service. (For ClusterIP Services, look at clusterIP to see if the Service is IPv4 or IPv6. For headless Services, look at the endpoints, which may be dual-stack in the future. For ExternalName Services, ipFamily has no meaning, but it may be set to an irrelevant value anyway.)",
"ipFamilies": "IPFamilies identifies all the IPFamilies assigned for this Service. If a value was not provided for IPFamilies it will be defaulted based on the cluster configuration and the value of service.spec.ipFamilyPolicy. A maximum of two values (dual-stack IPFamilies) are allowed in IPFamilies. IPFamilies field is conditionally mutable: it allows for adding or removing a secondary IPFamily, but it does not allow changing the primary IPFamily of the service.",
"topologyKeys": "topologyKeys is a preference-order list of topology keys which implementations of services should use to preferentially sort endpoints when accessing this Service, it can not be used at the same time as externalTrafficPolicy=Local. Topology keys must be valid label keys and at most 16 keys may be specified. Endpoints are chosen based on the first topology key with available backends. If this field is specified and all entries have no backends that match the topology of the client, the service has no backends for that client and connections should fail. The special value \"*\" may be used to mean \"any topology\". This catch-all value, if used, only makes sense as the last value in the list. If this is not specified or empty, no topology constraints will be applied.",
"ipFamilyPolicy": "IPFamilyPolicy represents the dual-stack-ness requested or required by this Service. If there is no value provided, then this Service will be considered SingleStack (single IPFamily). Services can be SingleStack (single IPFamily), PreferDualStack (two dual-stack IPFamilies on dual-stack clusters or single IPFamily on single-stack clusters), or RequireDualStack (two dual-stack IPFamilies on dual-stack configured clusters, otherwise fail). The IPFamilies and ClusterIPs assigned to this service can be controlled by service.spec.ipFamilies and service.spec.clusterIPs respectively.",
}
func (ServiceSpec) SwaggerDoc() map[string]string {

18
staging/src/k8s.io/api/core/v1/zz_generated.deepcopy.go generated
View File

@ -5270,6 +5270,11 @@ func (in *ServiceSpec) DeepCopyInto(out *ServiceSpec) {
(*out)[key] = val
}
}
if in.ClusterIPs != nil {
in, out := &in.ClusterIPs, &out.ClusterIPs
*out = make([]string, len(*in))
copy(*out, *in)
}
if in.ExternalIPs != nil {
in, out := &in.ExternalIPs, &out.ExternalIPs
*out = make([]string, len(*in))
@ -5285,16 +5290,21 @@ func (in *ServiceSpec) DeepCopyInto(out *ServiceSpec) {
*out = new(SessionAffinityConfig)
(*in).DeepCopyInto(*out)
}
if in.IPFamily != nil {
in, out := &in.IPFamily, &out.IPFamily
*out = new(IPFamily)
**out = **in
if in.IPFamilies != nil {
in, out := &in.IPFamilies, &out.IPFamilies
*out = make([]IPFamily, len(*in))
copy(*out, *in)
}
if in.TopologyKeys != nil {
in, out := &in.TopologyKeys, &out.TopologyKeys
*out = make([]string, len(*in))
copy(*out, *in)
}
if in.IPFamilyPolicy != nil {
in, out := &in.IPFamilyPolicy, &out.IPFamilyPolicy
*out = new(IPFamilyPolicyType)
**out = **in
}
return
}

36
staging/src/k8s.io/api/testdata/HEAD/core.v1.Service.json vendored
View File

@ -55,35 +55,41 @@
"22": "23"
},
"clusterIP": "24",
"type": ".蘯6ċV夸",
"externalIPs": [
"clusterIPs": [
"25"
],
"sessionAffinity": "ɑ",
"loadBalancerIP": "26",
"loadBalancerSourceRanges": [
"27"
"type": "鮽ort昍řČ扷5ƗǸƢ6/ʕVŚ(Ŀ",
"externalIPs": [
"26"
],
"externalName": "28",
"externalTrafficPolicy": "ʤ脽ěĂ凗蓏Ŋ蛊ĉy緅縕\u003eŽ",
"healthCheckNodePort": -1095807277,
"sessionAffinity": "甞谐颋DžS",
"loadBalancerIP": "27",
"loadBalancerSourceRanges": [
"28"
],
"externalName": "29",
"externalTrafficPolicy": "ƏS$+½H牗洝尿",
"healthCheckNodePort": -1965738697,
"publishNotReadyAddresses": true,
"sessionAffinityConfig": {
"clientIP": {
"timeoutSeconds": -1973740160
"timeoutSeconds": 2072604405
}
},
"ipFamily": "³-Ǐ忄*齧獚敆ȎțêɘIJ斬",
"ipFamilies": [
"x"
],
"topologyKeys": [
"29"
]
"30"
],
"ipFamilyPolicy": ";Ơ歿:狞夌碕ʂ"
},
"status": {
"loadBalancer": {
"ingress": [
{
"ip": "30",
"hostname": "31"
"ip": "31",
"hostname": "32"
}
]
}

BIN
staging/src/k8s.io/api/testdata/HEAD/core.v1.Service.pb vendored
View File

Binary file not shown.

30
staging/src/k8s.io/api/testdata/HEAD/core.v1.Service.yaml vendored
View File

@ -31,15 +31,19 @@ metadata:
uid: "7"
spec:
clusterIP: "24"
externalIPs:
clusterIPs:
- "25"
externalName: "28"
externalTrafficPolicy: ʤ脽ěĂ凗蓏Ŋ蛊ĉy緅縕>Ž
healthCheckNodePort: -1095807277
ipFamily: ³-Ǐ忄*齧獚敆ȎțêɘIJ斬
loadBalancerIP: "26"
externalIPs:
- "26"
externalName: "29"
externalTrafficPolicy: ƏS$+½H牗洝尿
healthCheckNodePort: -1965738697
ipFamilies:
- x
ipFamilyPolicy: ;Ơ歿:狞夌碕ʂ
loadBalancerIP: "27"
loadBalancerSourceRanges:
- "27"
- "28"
ports:
- appProtocol: "20"
name: "19"
@ -50,15 +54,15 @@ spec:
publishNotReadyAddresses: true
selector:
"22": "23"
sessionAffinity: ɑ
sessionAffinity: 甞谐颋DžS
sessionAffinityConfig:
clientIP:
timeoutSeconds: -1973740160
timeoutSeconds: 2072604405
topologyKeys:
- "29"
type: .蘯6ċV夸
- "30"
type: 鮽ort昍řČ扷5ƗǸƢ6/ʕVŚ(Ŀ
status:
loadBalancer:
ingress:
- hostname: "31"
ip: "30"
- hostname: "32"
ip: "31"

90
staging/src/k8s.io/api/testdata/v1.18.0/core.v1.Service.after_roundtrip.json vendored Normal file
View File

@ -0,0 +1,90 @@
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "2",
"generateName": "3",
"namespace": "4",
"selfLink": "5",
"uid": "7",
"resourceVersion": "11042405498087606203",
"generation": 8071137005907523419,
"creationTimestamp": null,
"deletionGracePeriodSeconds": -4955867275792137171,
"labels": {
"7": "8"
},
"annotations": {
"9": "10"
},
"ownerReferences": [
{
"apiVersion": "11",
"kind": "12",
"name": "13",
"uid": "Dz廔ȇ{sŊƏp",
"controller": false,
"blockOwnerDeletion": true
}
],
"finalizers": [
"14"
],
"clusterName": "15",
"managedFields": [
{
"manager": "16",
"operation": "鐊唊飙Ş-U圴÷a/ɔ}摁(湗Ć]",
"apiVersion": "17",
"fieldsType": "18"
}
]
},
"spec": {
"ports": [
{
"name": "19",
"protocol": "@Hr鯹)晿",
"appProtocol": "20",
"port": 202283346,
"targetPort": "21",
"nodePort": -474380055
}
],
"selector": {
"22": "23"
},
"clusterIP": "24",
"type": ".蘯6ċV夸",
"externalIPs": [
"25"
],
"sessionAffinity": "ɑ",
"loadBalancerIP": "26",
"loadBalancerSourceRanges": [
"27"
],
"externalName": "28",
"externalTrafficPolicy": "ʤ脽ěĂ凗蓏Ŋ蛊ĉy緅縕\u003eŽ",
"healthCheckNodePort": -1095807277,
"publishNotReadyAddresses": true,
"sessionAffinityConfig": {
"clientIP": {
"timeoutSeconds": -1973740160
}
},
"topologyKeys": [
"29"
]
},
"status": {
"loadBalancer": {
"ingress": [
{
"ip": "30",
"hostname": "31"
}
]
}
}
}

BIN
staging/src/k8s.io/api/testdata/v1.18.0/core.v1.Service.after_roundtrip.pb vendored Normal file
View File

Binary file not shown.

63
staging/src/k8s.io/api/testdata/v1.18.0/core.v1.Service.after_roundtrip.yaml vendored Normal file
View File

@ -0,0 +1,63 @@
apiVersion: v1
kind: Service
metadata:
annotations:
"9": "10"
clusterName: "15"
creationTimestamp: null
deletionGracePeriodSeconds: -4955867275792137171
finalizers:
- "14"
generateName: "3"
generation: 8071137005907523419
labels:
"7": "8"
managedFields:
- apiVersion: "17"
fieldsType: "18"
manager: "16"
operation: 鐊唊飙Ş-U圴÷a/ɔ}摁(湗Ć]
name: "2"
namespace: "4"
ownerReferences:
- apiVersion: "11"
blockOwnerDeletion: true
controller: false
kind: "12"
name: "13"
uid: Dz廔ȇ{sŊƏp
resourceVersion: "11042405498087606203"
selfLink: "5"
uid: "7"
spec:
clusterIP: "24"
externalIPs:
- "25"
externalName: "28"
externalTrafficPolicy: ʤ脽ěĂ凗蓏Ŋ蛊ĉy緅縕>Ž
healthCheckNodePort: -1095807277
loadBalancerIP: "26"
loadBalancerSourceRanges:
- "27"
ports:
- appProtocol: "20"
name: "19"
nodePort: -474380055
port: 202283346
protocol: '@Hr鯹)晿'
targetPort: "21"
publishNotReadyAddresses: true
selector:
"22": "23"
sessionAffinity: ɑ
sessionAffinityConfig:
clientIP:
timeoutSeconds: -1973740160
topologyKeys:
- "29"
type: .蘯6ċV夸
status:
loadBalancer:
ingress:
- hostname: "31"
ip: "30"

90
staging/src/k8s.io/api/testdata/v1.19.0/core.v1.Service.after_roundtrip.json vendored Normal file
View File

@ -0,0 +1,90 @@
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "2",
"generateName": "3",
"namespace": "4",
"selfLink": "5",
"uid": "7",
"resourceVersion": "11042405498087606203",
"generation": 8071137005907523419,
"creationTimestamp": null,
"deletionGracePeriodSeconds": -4955867275792137171,
"labels": {
"7": "8"
},
"annotations": {
"9": "10"
},
"ownerReferences": [
{
"apiVersion": "11",
"kind": "12",
"name": "13",
"uid": "Dz廔ȇ{sŊƏp",
"controller": false,
"blockOwnerDeletion": true
}
],
"finalizers": [
"14"
],
"clusterName": "15",
"managedFields": [
{
"manager": "16",
"operation": "鐊唊飙Ş-U圴÷a/ɔ}摁(湗Ć]",
"apiVersion": "17",
"fieldsType": "18"
}
]
},
"spec": {
"ports": [
{
"name": "19",
"protocol": "@Hr鯹)晿",
"appProtocol": "20",
"port": 202283346,
"targetPort": "21",
"nodePort": -474380055
}
],
"selector": {
"22": "23"
},
"clusterIP": "24",
"type": ".蘯6ċV夸",
"externalIPs": [
"25"
],
"sessionAffinity": "ɑ",
"loadBalancerIP": "26",
"loadBalancerSourceRanges": [
"27"
],
"externalName": "28",
"externalTrafficPolicy": "ʤ脽ěĂ凗蓏Ŋ蛊ĉy緅縕\u003eŽ",
"healthCheckNodePort": -1095807277,
"publishNotReadyAddresses": true,
"sessionAffinityConfig": {
"clientIP": {
"timeoutSeconds": -1973740160
}
},
"topologyKeys": [
"29"
]
},
"status": {
"loadBalancer": {
"ingress": [
{
"ip": "30",
"hostname": "31"
}
]
}
}
}

BIN
staging/src/k8s.io/api/testdata/v1.19.0/core.v1.Service.after_roundtrip.pb vendored Normal file
View File

Binary file not shown.

63
staging/src/k8s.io/api/testdata/v1.19.0/core.v1.Service.after_roundtrip.yaml vendored Normal file
View File

@ -0,0 +1,63 @@
apiVersion: v1
kind: Service
metadata:
annotations:
"9": "10"
clusterName: "15"
creationTimestamp: null
deletionGracePeriodSeconds: -4955867275792137171
finalizers:
- "14"
generateName: "3"
generation: 8071137005907523419
labels:
"7": "8"
managedFields:
- apiVersion: "17"
fieldsType: "18"
manager: "16"
operation: 鐊唊飙Ş-U圴÷a/ɔ}摁(湗Ć]
name: "2"
namespace: "4"
ownerReferences:
- apiVersion: "11"
blockOwnerDeletion: true
controller: false
kind: "12"
name: "13"
uid: Dz廔ȇ{sŊƏp
resourceVersion: "11042405498087606203"
selfLink: "5"
uid: "7"
spec:
clusterIP: "24"
externalIPs:
- "25"
externalName: "28"
externalTrafficPolicy: ʤ脽ěĂ凗蓏Ŋ蛊ĉy緅縕>Ž
healthCheckNodePort: -1095807277
loadBalancerIP: "26"
loadBalancerSourceRanges:
- "27"
ports:
- appProtocol: "20"
name: "19"
nodePort: -474380055
port: 202283346
protocol: '@Hr鯹)晿'
targetPort: "21"
publishNotReadyAddresses: true
selector:
"22": "23"
sessionAffinity: ɑ
sessionAffinityConfig:
clientIP:
timeoutSeconds: -1973740160
topologyKeys:
- "29"
type: .蘯6ċV夸
status:
loadBalancer:
ingress:
- hostname: "31"
ip: "30"

2
staging/src/k8s.io/apimachinery/pkg/util/validation/validation.go
View File

@ -347,7 +347,7 @@ func IsValidPortName(port string) []string {
// IsValidIP tests that the argument is a valid IP address.
func IsValidIP(value string) []string {
if net.ParseIP(value) == nil {
return []string{"must be a valid IP address, (e.g. 10.9.8.7)"}
return []string{"must be a valid IP address, (e.g. 10.9.8.7 or 2001:db8::ffff)"}
}
return nil
}

23
staging/src/k8s.io/kubectl/pkg/describe/describe.go
View File

@ -2721,10 +2721,27 @@ func describeService(service *corev1.Service, endpoints *corev1.Endpoints, event
printAnnotationsMultiline(w, "Annotations", service.Annotations)
w.Write(LEVEL_0, "Selector:\t%s\n", labels.FormatLabels(service.Spec.Selector))
w.Write(LEVEL_0, "Type:\t%s\n", service.Spec.Type)
w.Write(LEVEL_0, "IP:\t%s\n", service.Spec.ClusterIP)
if service.Spec.IPFamily != nil {
w.Write(LEVEL_0, "IPFamily:\t%s\n", *(service.Spec.IPFamily))
if service.Spec.IPFamilyPolicy != nil {
w.Write(LEVEL_0, "IP Family Policy:\t%s\n", *(service.Spec.IPFamilyPolicy))
}
if len(service.Spec.IPFamilies) > 0 {
ipfamiliesStrings := make([]string, 0, len(service.Spec.IPFamilies))
for _, family := range service.Spec.IPFamilies {
ipfamiliesStrings = append(ipfamiliesStrings, string(family))
}
w.Write(LEVEL_0, "IP Families:\t%s\n", strings.Join(ipfamiliesStrings, ","))
} else {
w.Write(LEVEL_0, "IP Families:\t%s\n", "<none>")
}
w.Write(LEVEL_0, "IP:\t%s\n", service.Spec.ClusterIP)
if len(service.Spec.ClusterIPs) > 0 {
w.Write(LEVEL_0, "IPs:\t%s\n", strings.Join(service.Spec.ClusterIPs, ","))
} else {
w.Write(LEVEL_0, "IPs:\t%s\n", "<none>")
}
if len(service.Spec.ExternalIPs) > 0 {

61
staging/src/k8s.io/kubectl/pkg/describe/describe_test.go
View File

@ -358,8 +358,7 @@ func getResourceList(cpu, memory string) corev1.ResourceList {
}
func TestDescribeService(t *testing.T) {
defaultServiceIPFamily := corev1.IPv4Protocol
singleStack := corev1.IPFamilyPolicySingleStack
testCases := []struct {
name string
service *corev1.Service
@ -373,8 +372,7 @@ func TestDescribeService(t *testing.T) {
Namespace: "foo",
},
Spec: corev1.ServiceSpec{
Type: corev1.ServiceTypeLoadBalancer,
IPFamily: &defaultServiceIPFamily,
Type: corev1.ServiceTypeLoadBalancer,
Ports: []corev1.ServicePort{{
Name: "port-tcp",
Port: 8080,
@ -384,6 +382,7 @@ func TestDescribeService(t *testing.T) {
}},
Selector: map[string]string{"blah": "heh"},
ClusterIP: "1.2.3.4",
IPFamilies: []corev1.IPFamily{corev1.IPv4Protocol},
LoadBalancerIP: "5.6.7.8",
SessionAffinity: "None",
ExternalTrafficPolicy: "Local",
@ -412,8 +411,7 @@ func TestDescribeService(t *testing.T) {
Namespace: "foo",
},
Spec: corev1.ServiceSpec{
Type: corev1.ServiceTypeLoadBalancer,
IPFamily: &defaultServiceIPFamily,
Type: corev1.ServiceTypeLoadBalancer,
Ports: []corev1.ServicePort{{
Name: "port-tcp",
Port: 8080,
@ -423,6 +421,7 @@ func TestDescribeService(t *testing.T) {
}},
Selector: map[string]string{"blah": "heh"},
ClusterIP: "1.2.3.4",
IPFamilies: []corev1.IPFamily{corev1.IPv4Protocol},
LoadBalancerIP: "5.6.7.8",
SessionAffinity: "None",
ExternalTrafficPolicy: "Local",
@ -451,8 +450,7 @@ func TestDescribeService(t *testing.T) {
Namespace: "foo",
},
Spec: corev1.ServiceSpec{
Type: corev1.ServiceTypeLoadBalancer,
IPFamily: &defaultServiceIPFamily,
Type: corev1.ServiceTypeLoadBalancer,
Ports: []corev1.ServicePort{{
Name: "port-tcp",
Port: 8080,
@ -462,6 +460,7 @@ func TestDescribeService(t *testing.T) {
}},
Selector: map[string]string{"blah": "heh"},
ClusterIP: "1.2.3.4",
IPFamilies: []corev1.IPFamily{corev1.IPv4Protocol},
LoadBalancerIP: "5.6.7.8",
SessionAffinity: "None",
ExternalTrafficPolicy: "Local",
@ -474,7 +473,51 @@ func TestDescribeService(t *testing.T) {
"Selector", "blah=heh",
"Type", "LoadBalancer",
"IP", "1.2.3.4",
"IPFamily", "IPv4",
"IP Families", "IPv4",
"Port", "port-tcp", "8080/TCP",
"TargetPort", "targetPort/TCP",
"NodePort", "port-tcp", "31111/TCP",
"Session Affinity", "None",
"External Traffic Policy", "Local",
"HealthCheck NodePort", "32222",
},
},
{
name: "test-ServiceIPFamilyPolicy+ClusterIPs",
service: &corev1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "bar",
Namespace: "foo",
},
Spec: corev1.ServiceSpec{
Type: corev1.ServiceTypeLoadBalancer,
Ports: []corev1.ServicePort{{
Name: "port-tcp",
Port: 8080,
Protocol: corev1.ProtocolTCP,
TargetPort: intstr.FromString("targetPort"),
NodePort: 31111,
}},
Selector: map[string]string{"blah": "heh"},
ClusterIP: "1.2.3.4",
IPFamilies: []corev1.IPFamily{corev1.IPv4Protocol},
IPFamilyPolicy: &singleStack,
ClusterIPs: []string{"1.2.3.4"},
LoadBalancerIP: "5.6.7.8",
SessionAffinity: "None",
ExternalTrafficPolicy: "Local",
HealthCheckNodePort: 32222,
},
},
expect: []string{
"Name", "bar",
"Namespace", "foo",
"Selector", "blah=heh",
"Type", "LoadBalancer",
"IP", "1.2.3.4",
"IP Families", "IPv4",
"IP Family Policy", "SingleStack",
"IPs", "1.2.3.4",
"Port", "port-tcp", "8080/TCP",
"TargetPort", "targetPort/TCP",
"NodePort", "port-tcp", "31111/TCP",

2
test/cmd/apply.sh
View File

@ -269,7 +269,7 @@ __EOF__
kube::test::get_object_assert 'services a' "{{${id_field:?}}}" 'a'
# change immutable field and apply service a
output_message=$(! kubectl apply -f hack/testdata/service-revision2.yaml 2>&1 "${kube_flags[@]:?}")
kube::test::if_has_string "${output_message}" 'field is immutable'
kube::test::if_has_string "${output_message}" 'may not change once set'
# apply --force to recreate resources for immutable fields
kubectl apply -f hack/testdata/service-revision2.yaml --force "${kube_flags[@]:?}"
# check immutable field exists

177
test/e2e/network/dual_stack.go
View File

@ -212,16 +212,11 @@ var _ = SIGDescribe("[Feature:IPv6DualStackAlphaFeature] [LinuxOnly]", func() {
assertNetworkConnectivity(f, *serverPods, *clientPods, "dualstack-test-client", "80")
})
ginkgo.It("should create service with cluster ip from primary service range [Feature:IPv6DualStackAlphaFeature:Phase2]", func() {
ginkgo.It("should create a single stack service with cluster ip from primary service range [Feature:IPv6DualStackAlphaFeature:Phase2]", func() {
serviceName := "defaultclusterip"
ns := f.Namespace.Name
jig := e2eservice.NewTestJig(cs, ns, serviceName)
defaultIPFamily := v1.IPv4Protocol
if framework.TestContext.ClusterIsIPv6() {
defaultIPFamily = v1.IPv6Protocol
}
t := NewServerTest(cs, ns, serviceName)
defer func() {
defer ginkgo.GinkgoRecover()
@ -230,8 +225,8 @@ var _ = SIGDescribe("[Feature:IPv6DualStackAlphaFeature] [LinuxOnly]", func() {
}
}()
ginkgo.By("creating service " + ns + "/" + serviceName + " with Service.Spec.IPFamily not set")
service := createService(t.ServiceName, t.Namespace, t.Labels, nil)
ginkgo.By("creating service " + ns + "/" + serviceName + " with Service.Spec.IPFamilies not set nil policy")
service := createService(t.ServiceName, t.Namespace, t.Labels, nil, nil)
jig.Labels = t.Labels
err := jig.CreateServicePods(2)
@ -241,8 +236,14 @@ var _ = SIGDescribe("[Feature:IPv6DualStackAlphaFeature] [LinuxOnly]", func() {
validateNumOfServicePorts(svc, 2)
expectedPolicy := v1.IPFamilyPolicySingleStack
expectedFamilies := []v1.IPFamily{v1.IPv4Protocol}
if framework.TestContext.ClusterIsIPv6() {
expectedFamilies = []v1.IPFamily{v1.IPv6Protocol}
}
// check the spec has been set to default ip family
validateServiceAndClusterIPFamily(svc, defaultIPFamily)
validateServiceAndClusterIPFamily(svc, expectedFamilies, &expectedPolicy)
// ensure endpoint belong to same ipfamily as service
if err := wait.PollImmediate(500*time.Millisecond, 10*time.Second, func() (bool, error) {
@ -250,7 +251,8 @@ var _ = SIGDescribe("[Feature:IPv6DualStackAlphaFeature] [LinuxOnly]", func() {
if err != nil {
return false, nil
}
validateEndpointsBelongToIPFamily(svc, endpoint, defaultIPFamily)
validateEndpointsBelongToIPFamily(svc, endpoint, expectedFamilies[0] /*endpoint controller works on primary ip*/)
return true, nil
}); err != nil {
framework.Failf("Get endpoints for service %s/%s failed (%s)", svc.Namespace, svc.Name, err)
@ -260,7 +262,6 @@ var _ = SIGDescribe("[Feature:IPv6DualStackAlphaFeature] [LinuxOnly]", func() {
ginkgo.It("should create service with ipv4 cluster ip [Feature:IPv6DualStackAlphaFeature:Phase2]", func() {
serviceName := "ipv4clusterip"
ns := f.Namespace.Name
ipv4 := v1.IPv4Protocol
jig := e2eservice.NewTestJig(cs, ns, serviceName)
@ -273,7 +274,11 @@ var _ = SIGDescribe("[Feature:IPv6DualStackAlphaFeature] [LinuxOnly]", func() {
}()
ginkgo.By("creating service " + ns + "/" + serviceName + " with Service.Spec.IPFamily IPv4" + ns)
service := createService(t.ServiceName, t.Namespace, t.Labels, &ipv4)
expectedPolicy := v1.IPFamilyPolicySingleStack
expectedFamilies := []v1.IPFamily{v1.IPv4Protocol}
service := createService(t.ServiceName, t.Namespace, t.Labels, nil, expectedFamilies)
jig.Labels = t.Labels
err := jig.CreateServicePods(2)
@ -284,7 +289,7 @@ var _ = SIGDescribe("[Feature:IPv6DualStackAlphaFeature] [LinuxOnly]", func() {
validateNumOfServicePorts(svc, 2)
// check the spec has been set to IPv4 and cluster ip belong to IPv4 family
validateServiceAndClusterIPFamily(svc, ipv4)
validateServiceAndClusterIPFamily(svc, expectedFamilies, &expectedPolicy)
// ensure endpoints belong to same ipfamily as service
if err := wait.PollImmediate(500*time.Millisecond, 10*time.Second, func() (bool, error) {
@ -292,7 +297,7 @@ var _ = SIGDescribe("[Feature:IPv6DualStackAlphaFeature] [LinuxOnly]", func() {
if err != nil {
return false, nil
}
validateEndpointsBelongToIPFamily(svc, endpoint, ipv4)
validateEndpointsBelongToIPFamily(svc, endpoint, expectedFamilies[0] /* endpoint controller operates on primary ip */)
return true, nil
}); err != nil {
framework.Failf("Get endpoints for service %s/%s failed (%s)", svc.Namespace, svc.Name, err)
@ -315,7 +320,10 @@ var _ = SIGDescribe("[Feature:IPv6DualStackAlphaFeature] [LinuxOnly]", func() {
}()
ginkgo.By("creating service " + ns + "/" + serviceName + " with Service.Spec.IPFamily IPv6" + ns)
service := createService(t.ServiceName, t.Namespace, t.Labels, &ipv6)
expectedPolicy := v1.IPFamilyPolicySingleStack
expectedFamilies := []v1.IPFamily{v1.IPv6Protocol}
service := createService(t.ServiceName, t.Namespace, t.Labels, nil, expectedFamilies)
jig.Labels = t.Labels
err := jig.CreateServicePods(2)
@ -326,7 +334,7 @@ var _ = SIGDescribe("[Feature:IPv6DualStackAlphaFeature] [LinuxOnly]", func() {
validateNumOfServicePorts(svc, 2)
// check the spec has been set to IPv6 and cluster ip belongs to IPv6 family
validateServiceAndClusterIPFamily(svc, ipv6)
validateServiceAndClusterIPFamily(svc, expectedFamilies, &expectedPolicy)
// ensure endpoints belong to same ipfamily as service
if err := wait.PollImmediate(500*time.Millisecond, 10*time.Second, func() (bool, error) {
@ -340,6 +348,98 @@ var _ = SIGDescribe("[Feature:IPv6DualStackAlphaFeature] [LinuxOnly]", func() {
framework.Failf("Get endpoints for service %s/%s failed (%s)", svc.Namespace, svc.Name, err)
}
})
ginkgo.It("should create service with ipv4,v6 cluster ip [Feature:IPv6DualStackAlphaFeature:Phase2]", func() {
serviceName := "ipv4ipv6clusterip"
ns := f.Namespace.Name
jig := e2eservice.NewTestJig(cs, ns, serviceName)
t := NewServerTest(cs, ns, serviceName)
defer func() {
defer ginkgo.GinkgoRecover()
if errs := t.Cleanup(); len(errs) != 0 {
framework.Failf("errors in cleanup: %v", errs)
}
}()
ginkgo.By("creating service " + ns + "/" + serviceName + " with Service.Spec.IPFamily IPv4, IPv6" + ns)
expectedPolicy := v1.IPFamilyPolicyRequireDualStack
expectedFamilies := []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol}
service := createService(t.ServiceName, t.Namespace, t.Labels, nil, expectedFamilies)
jig.Labels = t.Labels
err := jig.CreateServicePods(2)
framework.ExpectNoError(err)
svc, err := t.CreateService(service)
framework.ExpectNoError(err, "failed to create service: %s in namespace: %s", serviceName, ns)
validateNumOfServicePorts(svc, 2)
// check the spec has been set to IPv4 and cluster ip belong to IPv4 family
validateServiceAndClusterIPFamily(svc, expectedFamilies, &expectedPolicy)
// ensure endpoints belong to same ipfamily as service
if err := wait.PollImmediate(500*time.Millisecond, 10*time.Second, func() (bool, error) {
endpoint, err := cs.CoreV1().Endpoints(svc.Namespace).Get(context.TODO(), svc.Name, metav1.GetOptions{})
if err != nil {
return false, nil
}
validateEndpointsBelongToIPFamily(svc, endpoint, expectedFamilies[0] /* endpoint controller operates on primary ip */)
return true, nil
}); err != nil {
framework.Failf("Get endpoints for service %s/%s failed (%s)", svc.Namespace, svc.Name, err)
}
})
ginkgo.It("should create service with ipv6,v4 cluster ip [Feature:IPv6DualStackAlphaFeature:Phase2]", func() {
serviceName := "ipv6ipv4clusterip"
ns := f.Namespace.Name
jig := e2eservice.NewTestJig(cs, ns, serviceName)
t := NewServerTest(cs, ns, serviceName)
defer func() {
defer ginkgo.GinkgoRecover()
if errs := t.Cleanup(); len(errs) != 0 {
framework.Failf("errors in cleanup: %v", errs)
}
}()
ginkgo.By("creating service " + ns + "/" + serviceName + " with Service.Spec.IPFamily IPv4, IPv6" + ns)
expectedPolicy := v1.IPFamilyPolicyRequireDualStack
expectedFamilies := []v1.IPFamily{v1.IPv6Protocol, v1.IPv4Protocol}
service := createService(t.ServiceName, t.Namespace, t.Labels, nil, expectedFamilies)
jig.Labels = t.Labels
err := jig.CreateServicePods(2)
framework.ExpectNoError(err)
svc, err := t.CreateService(service)
framework.ExpectNoError(err, "failed to create service: %s in namespace: %s", serviceName, ns)
validateNumOfServicePorts(svc, 2)
// check the spec has been set to IPv4 and cluster ip belong to IPv4 family
validateServiceAndClusterIPFamily(svc, expectedFamilies, &expectedPolicy)
// ensure endpoints belong to same ipfamily as service
if err := wait.PollImmediate(500*time.Millisecond, 10*time.Second, func() (bool, error) {
endpoint, err := cs.CoreV1().Endpoints(svc.Namespace).Get(context.TODO(), svc.Name, metav1.GetOptions{})
if err != nil {
return false, nil
}
validateEndpointsBelongToIPFamily(svc, endpoint, expectedFamilies[0] /* endpoint controller operates on primary ip */)
return true, nil
}); err != nil {
framework.Failf("Get endpoints for service %s/%s failed (%s)", svc.Namespace, svc.Name, err)
}
})
// TODO (khenidak add slice validation logic, since endpoint controller only operates
// on primary ClusterIP
})
func validateNumOfServicePorts(svc *v1.Service, expectedNumOfPorts int) {
@ -348,17 +448,37 @@ func validateNumOfServicePorts(svc *v1.Service, expectedNumOfPorts int) {
}
}
func validateServiceAndClusterIPFamily(svc *v1.Service, expectedIPFamily v1.IPFamily) {
if svc.Spec.IPFamily == nil {
func validateServiceAndClusterIPFamily(svc *v1.Service, expectedIPFamilies []v1.IPFamily, expectedPolicy *v1.IPFamilyPolicyType) {
if len(svc.Spec.IPFamilies) != len(expectedIPFamilies) {
framework.Failf("service ip family nil for service %s/%s", svc.Namespace, svc.Name)
}
if *svc.Spec.IPFamily != expectedIPFamily {
framework.Failf("ip family mismatch for service: %s/%s, expected: %s, actual: %s", svc.Namespace, svc.Name, expectedIPFamily, *svc.Spec.IPFamily)
for idx, family := range expectedIPFamilies {
if svc.Spec.IPFamilies[idx] != family {
framework.Failf("service %s/%s expected family %v at index[%v] got %v", svc.Namespace, svc.Name, family, idx, svc.Spec.IPFamilies[idx])
}
}
isIPv6ClusterIP := netutils.IsIPv6String(svc.Spec.ClusterIP)
if (expectedIPFamily == v1.IPv4Protocol && isIPv6ClusterIP) || (expectedIPFamily == v1.IPv6Protocol && !isIPv6ClusterIP) {
framework.Failf("got unexpected service ip %s, should belong to %s ip family", svc.Spec.ClusterIP, expectedIPFamily)
// validate ip assigned is from the family
if len(svc.Spec.ClusterIPs) != len(svc.Spec.IPFamilies) {
framework.Failf("service %s/%s assigned ips [%+v] does not match families [%+v]", svc.Namespace, svc.Name, svc.Spec.ClusterIPs, svc.Spec.IPFamilies)
}
for idx, family := range svc.Spec.IPFamilies {
if (family == v1.IPv6Protocol) != netutils.IsIPv6String(svc.Spec.ClusterIPs[idx]) {
framework.Failf("service %s/%s assigned ips at [%v]:%v does not match family:%v", svc.Namespace, svc.Name, idx, svc.Spec.ClusterIPs[idx], family)
}
}
// validate policy
if expectedPolicy == nil && svc.Spec.IPFamilyPolicy != nil {
framework.Failf("service %s/%s expected nil for IPFamilyPolicy", svc.Namespace, svc.Name)
}
if expectedPolicy != nil && svc.Spec.IPFamilyPolicy == nil {
framework.Failf("service %s/%s expected value %v for IPFamilyPolicy", svc.Namespace, svc.Name, expectedPolicy)
}
if expectedPolicy != nil && *(svc.Spec.IPFamilyPolicy) != *(expectedPolicy) {
framework.Failf("service %s/%s expected value %v for IPFamilyPolicy", svc.Namespace, svc.Name, expectedPolicy)
}
}
@ -368,7 +488,7 @@ func validateEndpointsBelongToIPFamily(svc *v1.Service, endpoint *v1.Endpoints,
}
for _, ss := range endpoint.Subsets {
for _, e := range ss.Addresses {
if (expectedIPFamily == v1.IPv6Protocol && isIPv4(e.IP)) || (expectedIPFamily == v1.IPv4Protocol && netutils.IsIPv6String(e.IP)) {
if (expectedIPFamily == v1.IPv6Protocol) != netutils.IsIPv6String(e.IP) {
framework.Failf("service endpoint %s doesn't belong to %s ip family", e.IP, expectedIPFamily)
}
}
@ -427,16 +547,17 @@ func isIPv4CIDR(cidr string) bool {
}
// createService returns a service spec with defined arguments
func createService(name, ns string, labels map[string]string, ipFamily *v1.IPFamily) *v1.Service {
func createService(name, ns string, labels map[string]string, ipFamilyPolicy *v1.IPFamilyPolicyType, ipFamilies []v1.IPFamily) *v1.Service {
return &v1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: name,
Namespace: ns,
},
Spec: v1.ServiceSpec{
Selector: labels,
Type: v1.ServiceTypeNodePort,
IPFamily: ipFamily,
Selector: labels,
Type: v1.ServiceTypeNodePort,
IPFamilyPolicy: ipFamilyPolicy,
IPFamilies: ipFamilies,
Ports: []v1.ServicePort{
{
Name: "tcp-port",

1
test/integration/BUILD
View File

@ -50,6 +50,7 @@ filegroup(
"//test/integration/deployment:all-srcs",
"//test/integration/disruption:all-srcs",
"//test/integration/dryrun:all-srcs",
"//test/integration/dualstack:all-srcs",
"//test/integration/endpoints:all-srcs",
"//test/integration/endpointslice:all-srcs",
"//test/integration/etcd:all-srcs",

38
test/integration/dualstack/BUILD Normal file
View File

@ -0,0 +1,38 @@
load("@io_bazel_rules_go//go:def.bzl", "go_test")
go_test(
name = "go_default_test",
srcs = [
"dualstack_test.go",
"main_test.go",
],
tags = ["integration"],
deps = [
"//pkg/features:go_default_library",
"//staging/src/k8s.io/api/core/v1:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library",
"//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library",
"//staging/src/k8s.io/client-go/kubernetes:go_default_library",
"//staging/src/k8s.io/client-go/rest:go_default_library",
"//staging/src/k8s.io/component-base/featuregate/testing:go_default_library",
"//test/integration/framework:go_default_library",
"//vendor/k8s.io/utils/net:go_default_library",
],
)
filegroup(
name = "package-srcs",
srcs = glob(["**"]),
tags = ["automanaged"],
visibility = ["//visibility:private"],
)
filegroup(
name = "all-srcs",
srcs = [":package-srcs"],
tags = ["automanaged"],
visibility = ["//visibility:public"],
)

1353
test/integration/dualstack/dualstack_test.go Normal file
View File

File diff suppressed because it is too large Load Diff

27
test/integration/dualstack/main_test.go Normal file
View File

@ -0,0 +1,27 @@
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package dualstack
import (
"testing"
"k8s.io/kubernetes/test/integration/framework"
)
func TestMain(m *testing.M) {
framework.EtcdMain(m.Run)
}