Airflow Not Able to Found the Kubernetive POD looks like API Pool Issue

[Amar1404]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.10.2

What happened?

Airflow version: 2.10.2

[2025-03-01, 00:33:13 IST] {taskinstance.py:3310} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 637, in execute_sync
istio_enabled = self.is_istio_enabled(self.pod)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 940, in is_istio_enabled
remote_pod = self.pod_manager.read_pod(pod)
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 336, in wrapped_f
return copy(f, *args, **kw)
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 475, in call
do = self.iter(retry_state=retry_state)
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 376, in iter
result = action(retry_state)
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 418, in exc_check
raise retry_exc.reraise()
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 185, in reraise
raise self.last_attempt.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 478, in call
result = fn(*args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 720, in read_pod
return self._client.read_namespaced_pod(pod.metadata.name, pod.metadata.namespace)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 23693, in read_namespaced_pod
return self.read_namespaced_pod_with_http_info(name, namespace, **kwargs) # noqa: E501
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 23780, in read_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 244, in GET
return self.request("GET", url,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 238, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '90b2298d-56c6-406b-95fb-661a4bd1942e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'b97d9c8b-7397-47a1-b30e-b99f7c2ae92e', 'X-Kubernetes-Pf-Prioritylevel-Uid': '48b7c182-bee3-4b45-b4dc-a0f80814e25c', 'Date': 'Fri, 28 Feb 2025 19:03:10 GMT', 'Content-Length': '294'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods "co-rl-v2-1-model-update-cb-opt-arm-15-cb-opt-arm-16-v2-r2980b03" not found","reason":"NotFound","details":{"name":"co-rl-v2-1-model-update-cb-opt-arm-15-cb-opt-arm-16-v2-r2980b03","kind":"pods"},"code":404}
During handling of the above exception, another exception occurred:

What you think should happen instead?

There is the issue in pooling the K8 Pod the pod got deleted before the airflow reach out to it check the status.

How to reproduce

Create a small K8 pod doing simple things. When the airflow reach out to k8 pod.

Operating System

linux

Versions of Apache Airflow Providers

No response

Deployment

Other 3rd-party Helm chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[RNHTTR]

Can you please provide more detail to help reproduce the issue? As is, this is not sufficient to be able to understand what's wrong and provide a fix.