Merge pull request #62113 from juju-solutions/bug/worker-upgrade

Automatic merge from submit-queue (batch tested with PRs 61705, 61609, 62103, 62113, 62115). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Remove GPU label during upgrade if needed

**What this PR does / why we need it**: During upgrade of juju charms we might fail to remove the GPU labels. With this PR we try to remove this label only when GPUs are available. In case the label removal fails we print a warning in hope the GPUs will continue to be available after the upgrade. 

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/537

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```

cluster/juju/layers/kubernetes-worker/reactive/kubernetes_worker.py

@@ -69,8 +69,14 @@ def upgrade_charm():
 
     # Remove gpu.enabled state so we can reconfigure gpu-related kubelet flags,
     # since they can differ between k8s versions
+    if is_state('kubernetes-worker.gpu.enabled'):
         remove_state('kubernetes-worker.gpu.enabled')
+        try:
             disable_gpu()
+        except ApplyNodeLabelFailed:
+            # Removing node label failed. Probably the master is unavailable.
+            # Proceed with the upgrade in hope GPUs will still be there.
+            hookenv.log('Failed to remove GPU labels. Proceed with upgrade.')
 
     remove_state('kubernetes-worker.cni-plugins.installed')
     remove_state('kubernetes-worker.config.created')