Airflow Scheduler无法以kubernetes执行器启动


12

我正在使用 https://github.com/helm/charts/tree/master/stable/airflow掌舵图并puckle/docker-airflow使用安装的Kubernetes构建v1.10.8 图像并在掌舵图中使用该图像,但是我一直在获取

  File "/usr/local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1140, in initdb
    db.initdb(settings.RBAC)
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 332, in initdb
    dagbag = models.DagBag()
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 95, in __init__
    executor = get_default_executor()
  File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 48, in get_default_executor
    DEFAULT_EXECUTOR = _get_executor(executor_name)
  File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 87, in _get_executor
    return KubernetesExecutor()
  File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 702, in __init__
    self.kube_config = KubeConfig()
  File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 283, in __init__
    self.kube_client_request_args = json.loads(kube_client_request_args)
  File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

在我的调度程序中,正如各种消息来源所建议的那样,我尝试设置:

AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS: {"_request_timeout" : [60,60] }

以我的掌舵价值观 那也没有用的人有什么想法我想念什么?

这是我的价值观


airflow:
  image:
     repository: airflow-docker-local
     tag: 1.10.8
  executor: Kubernetes
  service:
    type: LoadBalancer
  config:
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: airflow-docker-local
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: 1.10.8
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_IMAGE_PULL_POLICY: Never

    AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME: airflow
    AIRFLOW__KUBERNETES__DAGS_VOLUME_CLAIM: airflow
    AIRFLOW__KUBERNETES__NAMESPACE: airflow
    AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS: {"_request_timeout" : [60,60] }

    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://postgres:airflow@airflow-postgresql:5432/airflow

persistence:
  enabled: true
  existingClaim: ''

workers:
  enabled: false

postgresql:
  enabled: true

redis:
  enabled: false

编辑:

在我添加之后,尝试在helm values.yaml中设置环境变量的各种尝试均不起作用(请注意双引号和单引号)

ENV AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS='{"_request_timeout" : [60,60] }'

Dockerfile的位置https : //github.com/puckel/docker-airflow/blob/1.10.9/Dockerfile#L19 之后,我的airflow-schedulerPod启动,但是随后我的调度程序Pod上一直出现以下错误。

Process KubernetesJobWatcher-9: Traceback (most recent call last): 
    File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 313, 
    in recv_into return self.connection.recv_into(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py", 
    line 1840, in recv_into self._raise_ssl_error(self._ssl, result) File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py", 
    line 1646, in _raise_ssl_error raise WantReadError() OpenSSL.SSL.WantReadError

同样的问题。我检查了docker-airflow:1.10.8,并取出改变L931的config/airflow.cfgkube_client_request_args =,并使用该图片。似乎工作
拉夫

同样的问题在这里:-(
LiorH

Answers:


3

对于helm值,模板使用一个循环,该循环将airflow.config地图置于双引号中"。这意味着"需要转义任何值,以使输出模板化YAML有效。

airflow:
  config:
    AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS: '{\"_request_timeout\":60}'

部署并运行(但是我还没有完成端到端测试)

根据此github问题,由于调度程序在60秒的连接超时后再次启动,因此python调度程序SSL超时可能不是问题。


是的,工作了,谢谢:-)
LiorH

没问题!您是否能够验证调度程序是否正常运行,只是每X秒重新连接一次?
马特

是的,调度程序可以工作。虽然
遇到
By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.