Propose Windows operator updates

[jsturtevant]

Expected Behavior

I would like to be able to install the operator with and Installation CRD that enables me to create a cluster including Windows nodes with no manual steps.

Current Behavior

Following the documentation, I have additional steps after my cluster is running:

1. Configure IPAM affinity
2. Query my cluster to get the API server address (https://docs.tigera.io/calico/latest/getting-started/kubernetes/windows-calico/operator#operator-installation) and create a kubernetes-services-endpoint

Possible Solution

1. Provide a new field in the Installation CRD that tells the operator to make sure this is turned on.
2. The docs mentions that some clusters create a CM for kube-proxy (Kubeadm deployments for example), If this is present (maybe can make the CM name configurable in Installation CRD) then query and auto create (if not present) the kubernetes-services-endpoint. I think this is common enough set up (Kubeadm, maybe others?) to try for this type of automation and still provides ability to customize.

I did a small prototype of and tested with capz's deployment. Some things could be cleaned up and might not have everything in the right place on the Installation CRD.

release-v1.32...jsturtevant:operator:windows-updates
projectcalico/calico@release-v3.27...jsturtevant:calico:windows-updates

Example changes to helm (and Installation CRD):

installation:
  cni:
    type: Calico
    ipam:
      strictAffinity: true
      type: Calico
  calicoNetwork:
    bgp: Disabled
    mtu: 1350
    ipPools:
    - cidr: 192.168.0.0/16
      encapsulation: VXLAN
    windowsDataplane: HNS
  serviceCIDRs: 
    - 10.96.0.0/12 # must match cluster service CIDR (this is the default)
  registry: docker.io/
# Image and registry configuration for the tigera/operator pod.
tigeraOperator:
  image: tigera-operator
  registry: jsturtevant
  version: latest
calicoctl:
  image: mcr.microsoft.com/oss/calico/ctl

Steps to Reproduce (for bugs)

Follow the docs at https://docs.tigera.io/calico/latest/getting-started/kubernetes/windows-calico/operator

Context

I am updating CAPZ to support the new Calico 3.27 version. We install the operator via a helm chart and want to be able to deploy the helm and ensure the cluster starts with out needing additional manual steps (or even via scripts).

kubernetes-sigs/cluster-api-provider-azure#4435

Your Environment

• Operating System and version:
• Link to your project (optional):

[jsturtevant]

/cc @coutinhop

[tmjd]

I'm not saying we shouldn't do what is being suggested with some auto-detection but I think the 2 steps that you're suggesting are necessary after your cluster is running would be possible to setup before Calico is deployed.
Creating the ConfigMap ahead of time requires knowing the API server address. The IPAMConfiguration could be created with an init container on the operator pod.

I'm not sure about a field that is only for turning on auto-detection for Windows, I'd have to see a design with the new field's behavior to see if it makes sense for the operator's API.

[jsturtevant]

I'm not tied to any particular solution. It was more for myself to see what would be required for changes to automate without any additional interactions. I have little experience with this operator so I probably made choice that might not make sense.

I am not sure I follow about the API server needing to be known. At least in our set-up, the operator is running in the cluster and is already talking to the api server. Deploying the IPAMConfiguration is similar to the way the networkPolicy rules are deployed throughout the other renderers.

[jsturtevant]

Any further thoughts on these two additions? Are there other approaches to setting these values that would let us use the operator?

[tmjd]

@coutinhop perhaps you should weigh in here since you've been handling most of the windows changes.

[coutinhop]

@tmjd @jsturtevant I did have an initial discussion with @caseydavenport and captured that in a ticket, it's in my backlog and I'll probably pick it up next sprint...

[jsturtevant]

thanks for the update, I am happy to try out any changes

[desek]

We're facing the same issue with our Windows clusters. We'd like to be able to provide an Installation manifest that installs and configured everything end-to-end rather than having to glue the last mile with patching the IPAMConfig and creating the ConfigMap.

Would love to see this implemented!

[jsturtevant]

any progress on this?

[jsturtevant]

looks like kubernetes-services-endpoint might be partially added projectcalico/calico#8443

[jsturtevant]

looks like kubernetes-services-endpoint might be partially added projectcalico/calico#8443

This doesn't work well in the windows scenario on a fresh bring up because you run into projectcalico/calico#9536 since it tries to do dns lookups via coredns which isn't available yet.