Windows support for Airflow

Description

Currently, the airflow project uses PEP-3143 style daemons to launch tasks (as implemented in https://pypi.org/project/python-daemon/), however this is targeted towards unix daemons. As a result, running airflow on windows requires multiple levels of abstraction each with their own problems. Would it be possible to use something like daemoniker (https://daemoniker.readthedocs.io/en/latest/) to launch tasks? What are the challenges and issues?

In machine learning workflows, with large datasets, it is a huge time-saver if the pipeline tasks can be run on the GPU. WSL 1 does not support GPU passthrough, docker through WSL 2 supports GPU passthrough only with the Insiders build, additionally it has issues with networking when connected to VPN (microsoft/WSL#5068).

Use case / motivation

Natively running airflow without WSL 1/2 or docker on Windows. This is helpful in cases where the company ecosystem is windows-based.

Possible implementation

The daemon module is only used to daemonize the scheduler and webserver. Here's a sample code that runs the scheduler (airflow origin/v1-10-stable) using daemoniker, comments are welcome:

# airflow/bin/cli.py
from daemoniker import Daemonizer

...

if args.daemon:
    with Daemonizer() as (is_setup, daemonizer):
        if is_setup:
            pid, stdout, stderr, log_file = setup_locations("scheduler",
                                                    args.pid,
                                                    args.stdout,
                                                    args.stderr,
                                                    args.log_file)
        
        _is_parent = daemonizer(
            pid,
            stdout_goto=stdout,
            stderr_goto=stderr
        )

    job.run()

Thanks for opening your first issue here! Be sure to follow the issue template!

have you encountered other problems with running Airflow on Windows? Windows support is highly anticipated by our users, but no one has dealt with this topic intensively yet. Personally, I use MacOS, but I support the idea of adding support for Windows.

Yes. Following the installation manual on the homepage pip install apache-airflow installs the airflow command, but it is not a windows executable and windows does not recognize the #! ..../python3.exe shebang.

@shachibista

@shachibista Have you tried installing the development version from source? I think this change should fix this problem.
https://github.com/apache/airflow/pull/7808/files#r396126977

I think it would be great if someone could invest in Windows support. I believe there are few things - not only the daemon model but also Local Executor uses fork mechanisms which won't be able on Windows, also there might be some problem if you want to use Celery Executor on Windows: https://www.distributedpython.com/2018/08/21/celery-4-windows/ There are few POSIX-compliant packages used as well with might not work on Windows. And automated testing might be a problem since we are using Docker. It looks like quite a big effort to invest..

@mik-laj

@mik-laj No, I haven't tried installing the development version from source. Is there a simple way to do it within windows?

I am afraid not. We know Airflow works in WSL2, but we also know it does not work on Windows. Unless you can convince someone to make it works for Windows, I am afraid it's not going to happen.

you can install the application from local sources by cloning the repository and then running the pip install -e . command

@mik-laj

@mik-laj Yes, the development version fixes the issue with the airflow command, at least. But, I cannot start the scheduler due to the aforementioned issues.

@potiuk Are you sure there are no fork-like mechanisms for windows? I would really like to get this working at least using Local/SequentialExecutor.

@mik-laj

@mik-laj Yes, the development version fixes the issue with the airflow command, at least. But, I cannot start the scheduler due to the aforementioned issues.

@potiuk Are you sure there are no fork-like mechanisms for windows? I would really like to get this working at least using Local/SequentialExecutor.

There are different mechanisms - here is the whole discussion about it: https://docs.python.org/3/library/subprocess.html#popen-constructor - but they work differently and Airflow relies on some of the properties of Popen and passing opened file handlers (for example to opened log files). I think there are also a number of other dependencies and possibly hard-coded UNIX path "/" across the code, also Windows is not POSIX-compliant, and I think there are many places where we rely on some tools or binaries which are part of POSIX standard.

I am not saying it's impossible, I just think it's quite an effort and unless you make all the tests pass on windows we can't even start thinking about it. You can start with forking Airflow and trying to make the test work on Windows. Github Actions support Windows runners, so this should be easy to enable.

We are heavily relying on Bash scripts for executing the tests and building Docker Images - and all our tests are run in Ubuntu docker image - however if you want to run it on Windows, it has to be done differently and likey not using Docker images - simply creating a virtualenv and installing everything.

Maybe you can find others who have time and would like to take a look at that together with you ? Simply start a discussion on our devlist and ask for help. I am afraid at this stage for the community, the fact that it works for WSL2 for Windows users is quite enough.

I know there were some changes implemented by @evgenyshulman from DataBand to make Airlfow work in a very limited way on Windows - so maybe rather than run a full set of tests on Windows, just getting a very simple support for Local Executor is possible ? Still Starting from a GitHub actions step installing Airflow on Windows is a good start, we cannot accept the code that is not tested, so being able to test it automatically is a prerequisite.

Happy to review any changes if you come up with tests running on Windows :).

Absolutely! I think that might be great thing to add to Airflow. Maybe you would like to open a PR about this (cc: me) with your changes and we can discuss how to approach it.

We also want probably to add some tests in the CI of ours to run on Windows. GitHub supports Windows runners as well so I am happy to work on incrementally adding more tests and run them in our CI.

Just split the changes needed maybe start with some small few lines part - I could then add the Windows CI tests around it on top. And we could add other PRs afterwards. Generally the smaller PR - the better :)

Any updates on this?

@pforai

If there is no update here, then there is no update. Everything here happens if somoene does it. Airflow is created by > 2100 users - more of them like you @pforai - users who contribute stuff if they need it.

Maybe you would like to take a lead on it and move it forward? We need users like you who apparenly have both the need and capabiliity (and in this case use Windows) who would like to move things forward and improve compatibility.

One incompatibility with Windows has to do with sqlite-db path absoluteness detection. Specifically, airflow currently requires absolute paths to start with / (i.e. sqlite:////...) otherwise they're determined to be invalid. On Windows, absolute paths start with a drive letter and not /.

Places where I saw checks like this:

airflow.settings.configure_orm
airflow.www.app

Yes. There are many more incompatibilities - like forking, and POSIX -only libraries used. but if someone would like to take on the task and implement and test all those, they are absolutely welcome to.

Another issue related to this "epic": #45172

Did quick test on current state. There are 2 Windows unfriendly libs (kerberos and ldap). Once removed manually from pyproject.yaml, I was able to uv sync successfully. Tried to run basic airflow info command with no luck.

PS F:\git\airflow> uv run airflow info
F:\git\airflow\airflow-core\src\airflow\__init__.py:45: RuntimeWarning: Airflow currently can be run on POSIX-compliant Operating Systems. For development, it is regularly tested on fairly modern Linux Distros and recent versions of macOS. On Windows you can run it via WSL2 (Windows Subsystem for Linux 2) or via Linux Containers. The work to add Windows support is tracked via https://github.com/apache/airflow/issues/10388, but it is not a high priority.
  warnings.warn(
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "F:\git\airflow\.venv\Scripts\airflow.exe\__main__.py", line 4, in <module>
  File "F:\git\airflow\airflow-core\src\airflow\__init__.py", line 62, in <module>
    from airflow import configuration, settings
  File "F:\git\airflow\airflow-core\src\airflow\configuration.py", line 2285, in <module>
    secrets_backend_list = initialize_secrets_backends()
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\git\airflow\airflow-core\src\airflow\configuration.py", line 2233, in initialize_secrets_backends
    secrets_backend_cls = import_string(class_name)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\git\airflow\airflow-core\src\airflow\utils\module_loading.py", line 41, in import_string
    module = import_module(module_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\josef\AppData\Local\Programs\Python\Python312\Lib\importlib\__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\git\airflow\airflow-core\src\airflow\secrets\metastore.py", line 27, in <module>
    from airflow.utils.session import NEW_SESSION, provide_session
  File "F:\git\airflow\airflow-core\src\airflow\utils\session.py", line 25, in <module>
    from airflow import settings
  File "F:\git\airflow\airflow-core\src\airflow\settings.py", line 39, in <module>
    from airflow._shared.timezones.timezone import local_timezone, parse_timezone, utc
ModuleNotFoundError: No module named 'airflow._shared.timezones'
PS F:\git\airflow>

@simi

@simi workarounds for the libs you mentioned:

ldap: https://github.com/cgohlke/python-ldap-build/releases
kerberos: https://pypi.org/project/winkerberos/

With symbolic links used now as part of our "code sharing" - it's further than ever to have "windows" support. I don't think, honestly we will ever want to go there. WSL2 support is there. For example IntelliJ/PyCharm has better and better support for WSL2 from locally run IDE https://blog.jetbrains.com/idea/2025/08/whats-fixed-intellij-idea-2025-2/ - and they are committed to do it eve more, I honestly don't see why we should ever want to make us bound the dev environment to Windows.

BTW if we're talking about developing using Airflow (as opposed to Airflow itself), limited development/debugging on Windows is possible using dag.test().

limited development/debugging on Windows is possible using dag.test().

Just a watchout. MAYBE. We neither check, nor verify that, so this is not guaranteed at all. It might work in some versions and might not work in others. It's behaviour under Windows is not specified and it will not block us from making any changes that might break it - simply because there is no point in the development lifecycle where it would be tested.

i have this wsl of rlinux installed on windows...
it doesnt open at all on windows home edition at all for airflow to run....

Navigation Menu

Repository navigation

Windows support for Airflow #10388

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Windows support for Airflow #10388

Description

Activity

boring-cyborg commented on Aug 18, 2020

mik-laj commented on Aug 18, 2020

shachibista commented on Aug 19, 2020

mik-laj commented on Aug 19, 2020

potiuk commented on Aug 19, 2020

shachibista commented on Aug 22, 2020

potiuk commented on Aug 22, 2020

mik-laj commented on Aug 22, 2020

shachibista commented on Aug 25, 2020

potiuk commented on Aug 25, 2020

potiuk commented on May 26, 2021

3 remaining items

potiuk commented on May 26, 2021

potiuk commented on May 26, 2021

pforai commented on Jul 11, 2022

potiuk commented on Jul 11, 2022

Dev-iL commented on May 26, 2024

potiuk commented on May 26, 2024

Dev-iL commented on May 29, 2025

simi commented on Sep 14, 2025

Dev-iL commented on Sep 15, 2025

potiuk commented on Sep 15, 2025

Dev-iL commented on Sep 16, 2025

potiuk commented on Sep 17, 2025

ravish80shankar-svg commented on Jul 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions