What can break in a managed upgrade
On a managed cluster, the provider upgrades the control plane. What can break is whatever the provider does not look at: manifests stored in Helm releases, your CRD versions, PodDisruptionBudgets that block draining, and webhooks that depend on the very nodes you are draining.
Each provider's detectors see recent calls to the API, not your inventory. That is why the work happens before you start the upgrade, and in a specific order.
In this article:
Waiting has an expiry date
The Kubernetes project maintains patch branches for the three most recent minor versions. As of 16/11/2026 those are 1.37, 1.36 and 1.35. Version 1.34 reached end of life on 27/10/2026 (kubernetes.io/releases). Each provider has its own calendar, and AKS already has a date in writing.
From 1 June 2027, AKS will automatically upgrade clusters that are still on an unsupported version once the 60-day grace period ends. According to its documentation, that upgrade does not follow the maintenance window, does not stop for deprecated API usage, and can override a PodDisruptionBudget that blocks the drain (AKS: stop upgrades on API breaking changes). If you arrive late, those protections do not apply.
GKE pauses the automatic upgrade to the next minor version when it detects deprecated API usage, and does not resume it until 30 days have passed without detection. It also warns that it does not guarantee either that it detects all usage or that the pause always happens. To block an upgrade reliably, it points to maintenance exclusions (GKE: deprecations).
The rules of each layer: Kubernetes sets the frame and each provider changes it
Kubernetes' version skew policy sets the frame:
kube-apiserver cannot skip minor versions, not even on a single-instance cluster.kubelet and kube-proxy can never be newer than the apiserver and can lag up to three minor versions behind (two if they are older than 1.25).kubectl is supported within ±1 minor version of the apiserver.Each provider adds its own rules, and that is where they differ:
| Behaviour | AKS | EKS | GKE |
|---|---|---|---|
| Deprecated API detection | Blocks the minor-version upgrade of the control plane (target 1.26 or later) if it saw usage in the last 12 h. Does not count get/list/watch | *Upgrade insights* with a rolling 30-day window. The block that required --force is temporarily reverted | Pauses the auto-upgrade and resumes it after 30 days without usage, with no guarantee |
| PDB that prevents draining a node | Waits for the *drain timeout* (30 min by default, from 5 min to 24 h) and stops the upgrade. A later PUT resumes it | On managed node groups: it fails with PodEvictionFailure after 15 min. With force it deletes the pods. (EKS Auto Mode respects PDBs.) | With the *surge* strategy (the default), it respects the PDB for up to 1 h and then force-evicts |
Sources: AKS API breaking changes, AKS rolling upgrades, EKS update cluster, EKS managed node update, GKE node pool upgrade strategies.
Look at the second row. The same PodDisruptionBudget produces three different outcomes: on AKS the upgrade stops with the pool half done, on EKS (managed node groups) the upgrade fails, and on GKE you only gain an hour. If your availability plan assumes that "the PDB protects it", on GKE, with *surge* upgrades, that does not hold beyond 60 minutes.
EKS adds two nuances. Its guide asks for managed nodes and Fargate nodes to be on the same minor version as the control plane before you upgrade it. The same page also describes that, with kubelet 1.25 or later, the cluster can move ahead by up to three versions without touching the kubelet. Plan with the stricter rule. EKS also lets you roll back to the previous minor version during the 7 days after the upgrade; after that, you have to create a new cluster and migrate (EKS update cluster).
Four places where a deprecated API hides (and detectors only look at one)
Provider detectors are useful, but they only see requests to the API. Your problems can be in four different places.
1. Live calls: controllers, pipelines and operators
These are the ones detectors see, along with the apiserver_requested_deprecated_apis metric. The metric has existed since 1.19 and includes the removed_release label, which tells you in which version each API will stop being served (Kubernetes blog, 2020).
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
# Formato ilustrativo de la salida:
# apiserver_requested_deprecated_apis{group="...",removed_release="1.xx",resource="...",subresource="",version="v1beta1"} 1There is a subtle limitation. AKS excludes reads (get/list/watch) from its detection, so a controller that only *watches* a deprecated API does not block the upgrade and will still stop working afterwards.
2. Manifests in Git that are not applied every day
A recovery environment, a CronJob that is re-applied once a quarter, or a chart nobody has touched in a year. No detector sees them until someone applies them.
grep -rnE '^apiVersion: (policy/v1beta1|batch/v1beta1|autoscaling/v2beta[12]|flowcontrol\.apiserver\.k8s\.io/v1beta[123])' ./manifiestosThe expression uses examples from the official migration guide. Adapt it to your version jump.
3. Manifests stored in Helm releases
Helm stores each release's manifest in the cluster. If it contains an API that has already been removed, the next helm upgrade fails with *"current release manifest contains removed kubernetes api(s) for this kubernetes version"*. There is no API call to detect, so no provider detector sees it. Helm recommends updating charts *before* updating the cluster. If you are already late, it suggests editing the stored manifest or using the mapkubeapis plugin (Helm: Kubernetes APIs).
helm list -A -o json | jq -r '.[] | "\(.namespace) \(.name)"' |
while read -r ns rel; do
helm get manifest "$rel" -n "$ns" | grep -E '^apiVersion:' | sort -u | sed "s|^|$ns/$rel |"
done4. CRD versions
The official guide covers Kubernetes' built-in APIs. The versions of your CRDs (cert-manager, database operators, Argo CD…) are decided by each project. To retire a version, the official procedure requires the new one to have storage: true and the old one to no longer appear in status.storedVersions before you remove it from spec.versions (CRD versioning).
kubectl get crd -o custom-columns='CRD:.metadata.name,SERVIDAS:.spec.versions[*].name,ALMACENADAS:.status.storedVersions'If a version that will disappear in the next operator release is still listed under ALMACENADAS (stored versions), order matters: migrate the objects first, then update the operator.
If you are moving between 1.33 and 1.37, the focus changes
On 16/11/2026, the most recent section of the API migration guide was v1.32. If you are moving between 1.33 and 1.37, the risk is no longer so much in the built-in APIs as in CRDs, add-ons and provider behaviour changes. One example: on AKS Standard, when you upgrade a node pool to 1.37 or later, LocalDNS can be enabled if the pool has no explicit profile, and if you use custom DNS you need to check that it answers on UDP and TCP port 53 (AKS rolling upgrades). You will not find that in the Kubernetes changelog.
The order of operations
1. Choose the target version and read the provider's notes for each jump.
2. Take inventory of the four sources and fix them in Git. Deploy those fixes on the cluster's current version, not on upgrade day. That way, if something fails, you know your change caused it and not the upgrade.
3. Check compatibility of the add-ons you manage (ingress, cert-manager, CSI drivers, autoscaler, policy engine) with the target version, using each project's matrix. On EKS, for example, Cluster Autoscaler must run the version that matches the cluster's minor version.
4. Review PDBs and webhooks (next sections).
5. Upgrade only the control plane and observe. AKS allows this with --control-plane-only, unless you have cluster auto-upgrade enabled (AKS control plane). The EKS control plane needs up to five free IPs in its subnets.
6. Upgrade node pools one at a time, starting with the non-production ones. Check quota and free IPs: AKS surge nodes consume quota and, with Azure CNI, subnet IPs. For production, AKS recommends a *max surge* of 33 %. On GKE, the defaults are maxSurge=1 and maxUnavailable=0.
7. Update the managed add-ons (on EKS, VPC CNI, CoreDNS and kube-proxy) and the clients (kubectl within ±1 version).
Draining nodes: three patterns that block the upgrade
minAvailable equal to the number of replicas, or maxUnavailable: 0. The documentation says it plainly: this demands *zero* voluntary evictions (configure PDB).CrashLoopBackOff protected by a PDB. With the default policy (IfHealthyBudget), running pods that are not healthy are only evicted if the application is not already degraded. The documentation itself warns that this can block draining. unhealthyPodEvictionPolicy: AlwaysAllow has been stable since 1.31. What you give up is that the pods of an already degraded application may never become healthy.PodEvictionFailure.Our criterion for a stateless service with three replicas (not a recommendation from the documentation):
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api
namespace: tienda
spec:
maxUnavailable: 1
unhealthyPodEvictionPolicy: AlwaysAllow
selector:
matchLabels:
app.kubernetes.io/name: apiCombine it with per-node topologySpreadConstraints in the Deployment. With whenUnsatisfiable: ScheduleAnyway the spread is a preference, not a guarantee; with DoNotSchedule it is guaranteed, at the price that a pod may stay Pending if there are not enough nodes. If all three replicas land on the same node, the PDB works as intended and the drain of that node still ends up waiting.
kubectl get pdb -A -o custom-columns='NS:.metadata.namespace,PDB:.metadata.name,PERMITIDOS:.status.disruptionsAllowed,POLITICA:.spec.unhealthyPodEvictionPolicy'Any row with PERMITIDOS (allowed disruptions) at 0 outside an incident is an upgrade that is going to stall.
Webhooks: they can block the pods that replace them
By default, admission webhooks use failurePolicy: Fail: if the webhook does not respond, the apiserver rejects the request (webhook good practices). During an upgrade, the webhook's own pods are drained too. If at that moment no replica is available and the webhook intercepts pod creation, it also rejects the pods that replace the evicted ones, including the webhook's own. The documentation describes this mutual lock and recommends not intercepting kube-system, excluding the webhook's own namespace, and running it with high availability.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: politicas-ejemplo
webhooks:
- name: validar.politicas.example.com
admissionReviewVersions: ["v1"]
sideEffects: None
failurePolicy: Fail
timeoutSeconds: 5
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values: ["kube-system", "politicas-system"]
rules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
clientConfig:
service:
name: politicas-webhook
namespace: politicas-system
path: /validate
port: 443kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
-o custom-columns='NOMBRE:.metadata.name,FALLO:.webhooks[*].failurePolicy,NS_SELECTOR:.webhooks[*].namespaceSelector'If a webhook has Fail, has no namespace selector and runs with a single replica, review it before the upgrade.
Whoever controls an admission webhook can also read, and if it is a mutating one modify, everything that enters the cluster. We cover that in Kubernetes security: why a “secure” cluster still lets the important things through.
Common mistakes and how to detect them
| Mistake | How to detect it |
|---|---|
| Treating the upgrade as safe because the portal shows no deprecated APIs | AKS does not count reads, EKS uses a 30-day window and GKE does not guarantee detection. Check all four sources |
| Migrating APIs and upgrading on the same day | Check the deployment history: API fixes should already be deployed, on the current version |
| A Helm release with a removed API stored in it | Walk through helm get manifest (command above) |
| A PDB that allows no evictions | PERMITIDOS column at 0 in kubectl get pdb -A |
| A degraded application that blocks the drain | Non-Ready pods covered by a PDB without AlwaysAllow |
"Unblocking" with force (EKS) or with the validation bypass (AKS) | On EKS it deletes pods without respecting the PDB. On AKS, calls to removed APIs will fail afterwards |
| Running out of quota or IPs for the extra nodes | Check provider quota and free IPs in the subnets before the window |
Forgetting managed add-ons and kubectl | Add-on versions compared with the provider's recommended ones after the upgrade |
Check commands (read-only)
kubectl version
kubectl get nodes -o custom-columns='NODO:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion'
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
kubectl get crd -o custom-columns='CRD:.metadata.name,ALMACENADAS:.status.storedVersions'
kubectl get pdb -A -o custom-columns='NS:.metadata.namespace,PDB:.metadata.name,PERMITIDOS:.status.disruptionsAllowed'
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o custom-columns='NOMBRE:.metadata.name,FALLO:.webhooks[*].failurePolicy'
# AKS
az aks get-upgrades --resource-group <grupo> --name <cluster> --output table
# EKS: consola → Cluster insights → Upgrade insights
# GKE: insights y recomendaciones de deprecación en la consolaIf a provider command does not return what you expect, check its documentation page first (linked below). Options change.
Where to start
1. Take inventory of the four sources (live calls, Git, Helm releases and CRDs) and note which API affects which team.
2. Run kubectl get pdb -A and fix every PDB that allows zero disruptions.
3. List the webhooks with failurePolicy: Fail and check that they exclude kube-system and their own namespace, and that they have more than one replica.
4. Set the date of your next upgrade before your version's end of support, especially on AKS because of the change in June 2027.
At Nubyron we work on Kubernetes upgrades on AKS, EKS and GKE: versions, deprecated APIs and agreed windows. If you would like another pair of eyes on your inventory, get in touch.
Sources (accessed on 16/11/2026)
Command outputs are illustrative and were not captured from a real cluster. Each provider's options and deadlines change: always check their documentation before acting.
Frequently asked questions
Why can a Kubernetes upgrade fail if the provider upgrades the control plane?
Because the provider upgrades the control plane but does not review your inventory. Manifests stored in Helm releases, CRD versions, PodDisruptionBudgets and webhooks can block or break the upgrade, and provider detectors only see recent calls to the API.
What happens if a PodDisruptionBudget prevents draining a node?
It depends on the provider. According to its documentation, AKS waits for the drain timeout (30 min by default) and stops the upgrade; EKS, on managed node groups, fails after 15 minutes; GKE, with the surge strategy, respects the PDB for up to 1 hour and then force-evicts. Check their documentation, because options change.
Do deprecated API detectors see everything that is going to break?
No. They see requests to the API, and the limits vary: AKS does not count reads, EKS uses a 30-day window and GKE does not guarantee detection. Manifests in Git, those stored in Helm releases and CRD versions need to be reviewed separately.
What changes on AKS from June 2027?
According to the AKS documentation, from 1 June 2027 it will automatically upgrade clusters on an unsupported version after a 60-day grace period. That upgrade does not follow the maintenance window, does not stop for deprecated API usage and can override a PodDisruptionBudget.
Want to check your inventory before the next upgrade?
We work on Kubernetes upgrades on AKS, EKS and GKE: versions, deprecated APIs and agreed windows. We can review your case with you and tell you where to start.
See Kubernetes upgrades