Airflow 2.1.2 daemon flag & configuration file bug
What is the current bug behavior?
-
When trying to start an Airflow Scheduler Instance with
meltano invoke airflow scheduler -Dand/ormeltano invoke airflow scheduler --daemon. However, despite producing a normal CLI output... you'll see a 'scheduler does not have a heartbeat' warning message. Currently the only was to remove this warning message and produce a 'heartbeat' is to runmeltano invoke airflow schedulerand leave the terminal window open. -
Currently we are trying to set the Airflow configurations to match the following:
a. executor = LocalExecutor
b. sql_alchemy_conn = postgresql
However these should be set in the airflow.cfg, under the path .meltano/run/airflow There are not the correct files... Currently it only host the files; airflow-scheduler.err, airflow-scheduler.log, airflow-scheduler.out, and webserver_config.py
What is the expected correct behavior?
-
airflow.cfgshould be located under the path and.meltano/run/airflow. However this file is not always present and seems to me removed and sporadically reproduced overwriting any previous airflow configurations applied using environmental variables.
Steps to reproduce
- Open a meltano project directory
- Run
meltano invoke airflow scheduler -Dormeltano invoke airflow scheduler --daemon - Open the Airflow UI
-
Meltano version (output with
meltano --version) -
1.102.0
-
Airflow Configurations (output with
airflow info) -
apache-airflow-providers-ftp | 2.0.0
-
apache-airflow-providers-http | 2.0.0
-
apache-airflow-providers-imap | 2.0.0
-
apache-airflow-providers-slack | 4.0.0
-
apache-airflow-providers-sqlite | 2.0.0
Relevant logs and/or screenshots
- Run
meltano invoke airflow scheduler -Dormeltano invoke airflow scheduler --daemon - Open the Airflow UI
-
Configure Airflow using the
meltano.ymlfilea.
meltano config airflow set core.executor LocalExecutorb.
meltano config airflow set core.sql_alchemy_conn postgresql://airflow:vegqwesdef@ec2-34-222-99-188.us-west-2.compute.amazonaws.com:5432/airflow
Possible fixes
Determine how airflow.cfg is created by the Meltano project.

