Home on Apache Airflow

Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design

Wed, 15 Apr 2026 00:00:00 +0000

A question like “How does AI tool usage vary across Airflow versions?” has a natural SQL shape: one cross-tabulation, one result. A question like “What does a typical Airflow deployment look like for practitioners who are actively using AI in their workflow?” does not. It requires querying executor type, deployment method, cloud provider, and Airflow version independently, each filtered to the same respondent group, then synthesizing the results into a coherent picture. No single query returns the answer. The answer emerges from the relationship between all of them.

Ask Your Survey Anything: Building AI Analysis Pipelines with Airflow 3

Wed, 15 Apr 2026 00:00:00 +0000

The 2025 Airflow Community Survey collected responses from nearly 6,000 practitioners across 168 questions. You can open a spreadsheet and filter, or write SQL by hand. But what if you could just ask a question and have Airflow figure out the query, run it, and bring the result back for your approval?

This post builds two pipelines that do exactly that, using the apache-airflow-providers-common-ai provider for Airflow 3.

The first pipeline is interactive: a human reviews the question before it reaches the LLM and approves the result before the DAG finishes. The second is scheduled: it downloads fresh survey data, validates the schema, runs the query unattended, and emails the result.

Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow

Tue, 14 Apr 2026 00:00:00 +0000

At Airflow Summit 2025, we previewed what native AI integration in Apache Airflow could look like. Today we’re shipping it.

apache-airflow-providers-common-ai 0.1.0 adds LLM and agent capabilities directly to Airflow. Not a wrapper around another framework, but a provider package that plugs into the orchestrator you already run. It’s built on Pydantic AI and supports 20+ model providers (OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, and more) through a single install.

pip install 'apache-airflow-providers-common-ai'

Requires Apache Airflow 3.0+.

Apache Airflow 3.2.0: Data-Aware Workflows at Scale

Tue, 07 Apr 2026 00:00:00 +0000

We’re proud to announce the release of Apache Airflow 3.2.0! Airflow 3.1 puts humans at the center of automated workflows. 3.2 brings that same precision to data: Asset partitioning for granular pipeline orchestration, multi-team deployments for enterprise scale, synchronous deadline alert callbacks, and continued progress toward full Task SDK separation.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/3.2.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.2.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.2.0/release_notes.html
🐳 Docker Image: docker pull apache/airflow:3.2.0
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.2.0

🗂️ Asset Partitioning (AIP-76): Only the Right Work Gets Triggered

Asset partitioning has been one of the most requested additions to data-aware scheduling. If you work with date-partitioned S3 paths, Hive table partitions, BigQuery partitions, or really any partitioned data store, you’ve dealt with this: An upstream task updates one partition, and every downstream Dag fires regardless of which slice actually changed. It’s wasteful, and for large deployments it creates real operational noise.

Introducing the Apache Airflow Registry

Thu, 19 Mar 2026 00:00:00 +0000

Today we’re launching the Apache Airflow Registry — a searchable catalog of every official Airflow provider and its modules, live at airflow.apache.org/registry/.

Need an S3 operator? A Snowflake hook? An OpenAI sensor? The Registry helps you find, compare, and configure the right components for your data pipelines — without digging through docs or PyPI pages.

By the Numbers


98	Official providers
1,602	Modules (operators, hooks, sensors, triggers, transfers, and more)
329M+	Monthly PyPI downloads across all providers
125+	Integrations with cloud platforms, databases, ML tools, and messaging services

Search Everything

Hit Cmd+K from any page and start typing. Results show up instantly, grouped by Providers and Modules, with type badges so you can tell a hook from an operator at a glance.

Airflow Survey 2025

Thu, 22 Jan 2026 00:00:00 +0000

The interactive report is hosted by Astronomer. The Apache Airflow community thanks Astronomer for running this survey, for sponsoring it and providing the report in this form, and for their effort in marketing, analysis, and preparing the graphics.

View raw data

Download survey responses (CSV)

Apache Airflow CTL aka airflowctl 0.1.0

Wed, 15 Oct 2025 00:00:00 +0000

We are thrilled to announce the first major release of airflowctl 0.1.0, the new secure, API-driven command-line interface (CLI) for Apache Airflow — built under AIP-81.

This release marks CLI to join the general posture on communicating through API. Airflow CLI joins the modern era of secure, auditable, and remote-first operations.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow-ctl/0.1.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow-ctl/stable/release_notes.html
🪶 Source Code: https://github.com/apache/airflow/tree/main/airflow-ctl

🎯 What is airflowctl?

airflowctl is a new command-line interface for Apache Airflow that interacts exclusively with the Airflow REST API. It provides a secure, auditable, and consistent way to manage Airflow deployments — without direct access to the metadata database.

Apache Airflow 3.1.0: Human-Centered Workflows

Thu, 25 Sep 2025 00:00:00 +0000

We are thrilled to announce the release of Apache Airflow 3.1.0, an update that puts humans at the center of data workflows. This release introduces powerful new capabilities for human decision-making in automated processes, comprehensive internationalization support, and significant developer experience improvements.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/3.1.0/
📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.1.0/
📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.1.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.0/release_notes.html
🪶 Sources: https://airflow.apache.org/docs/apache-airflow/3.1.0/installation/installing-from-sources.html
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.0

🤝 Human-in-the-Loop (HITL): When Automation Meets Human Judgment

This powerful capability bridges the gap between automated processes and human expertise, making Airflow invaluable for:

Apache Airflow® 3 is Generally Available!

Tue, 22 Apr 2025 00:00:00 +0000

We announced our intent to focus on Apache Airflow 3.0® as the next big milestone for the Airflow project at the Airflow Summit in September 2024. We are delighted to announce that Airflow 3.0 is now released!

A Major Release, Four Years in the Making

Airflow 3.0 is the biggest release in Airflow’s history—2.0 was released in 2020, and the last 4 years have seen incremental updates and releases every quarter with version 2.10 released in Q4 2024. With over 30 million monthly downloads (up over 30x since 2020) and 80,000 organizations (up from 25,000 in 2020) now using Airflow, we’ve seen an incredible growth in popularity since 2.0.

Airflow Survey 2024

Thu, 27 Feb 2025 00:00:00 +0000

View raw data

Download survey responses (CSV)

Apache Airflow 2.10.0 is here

Thu, 08 Aug 2024 00:00:00 +0000

I’m happy to announce that Apache Airflow 2.10.0 is now available, bringing an array of noteworthy enhancements and new features that will greatly serve our community.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.10.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.10.0/
🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.10.0/release_notes.html
🐳 Docker Image: “docker pull apache/airflow:2.10.0”
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.10.0

Airflow now collects Telemetry data by default

With the release of Airflow 2.10.0, we’ve introduced the collection of basic telemetry data, as outlined here. This data will play a crucial role in helping Airflow maintainers gain a deeper understanding of how Airflow is utilized across various deployments. The insights derived from this information are invaluable in guiding the prioritization of patches, minor releases, and security fixes. Moreover, this data will inform key decisions regarding the development roadmap, ensuring that Airflow continues to evolve in line with community needs.

Apache Airflow 2.9.0: Dataset and UI Improvements

Mon, 08 Apr 2024 00:00:00 +0000

I’m happy to announce that Apache Airflow 2.9.0 has been released! This time around we have new features for data-aware scheduling and a bunch of UI-related improvements.

Apache Airflow 2.9.0 contains over 550 commits, which include 38 new features, 70 improvements, 31 bug fixes, and 18 documentation changes.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.9.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.9.0/
🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.9.0/release_notes.html
🐳 Docker Image: “docker pull apache/airflow:2.9.0”
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.9.0

Airflow 2.9.0 is also the first release that supports Python 3.12. However, Pendulum 2 does not support Python 3.12, so you’ll need to use Pendulum 3 if you upgrade to Python 3.12.

Vulnerability in long deprecated OpenID authentication method in Flask AppBuilder

Mon, 26 Feb 2024 00:00:00 +0000

Vulnerability in long deprecated OpenID authentication method in Flask AppBuilder

Recently Islam Rzayev made us aware of a vulnerability in the long deprecated OpenID authentication method in Flask AppBuilder. This vulnerability allowed a malicious user to take over the identity of any Airflow UI user by forging a specially crafted request and implementing their own OpenID service. While this is an old, deprecated and almost not used authentication method, we still took the issue seriously.

Apache Airflow 2.8.0 is here

Fri, 15 Dec 2023 00:00:00 +0000

I am thrilled to announce the release of Apache Airflow 2.8.0, featuring a host of significant enhancements and new features that will greatly benefit our community.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.8.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.8.0/
🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.8.0/release_notes.html
🐳 Docker Image: “docker pull apache/airflow:2.8.0”
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.8.0

Airflow Object Storage (AIP-58)

This feature is experimental and subject to change.

Airflow now offers a generic abstraction layer over various object stores like S3, GCS, and Azure Blob Storage, enabling the use of different storage systems in DAGs without code modification.

Airflow Survey 2023

Thu, 21 Sep 2023 00:00:00 +0000

View Raw Data

Apache Airflow 2.7.0 is here

Fri, 18 Aug 2023 00:00:00 +0000

I’m happy to announce that Apache Airflow 2.7.0 has been released! Some notable features have been added that we are excited for the community to use.

Apache Airflow 2.7.0 contains over 500 commits, which include 40 new features, 49 improvements, 53 bug fixes, and 15 documentation changes.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.7.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.7.0/
🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.7.0/release_notes.html
🐳 Docker Image: “docker pull apache/airflow:2.7.0”
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.7.0

Airflow 2.7.0 is a release that focuses on security. The Airflow security team, working together with security researchers, identified a number of areas that required strengthening of security. This resulted in, among others things, an improved description of the Airflow security model, a better explanation of our security policy and the disabling of certain, potentially dangerous, features by default - like, for example, connection testing (#32052).

Introducing Setup and Teardown tasks

Fri, 18 Aug 2023 00:00:00 +0000

In data pipelines, commonly we need to create infrastructure resources, like a cluster or GPU nodes in an existing cluster, before doing the actual “work” and delete them after the work is done. Airflow 2.7 adds “setup” and “teardown” tasks to better support this type of pipeline. This blog post aims to highlight the key features so you know what’s possible. For full documentation on how to use setup and teardown tasks, see the setup and teardown docs.

what's new in Apache Airflow 2.6.0

Sun, 30 Apr 2023 00:00:00 +0000

I am excited to announce that Apache Airflow 2.6.0 has been released, bringing many minor features and improvements to the community.

Apache Airflow 2.6.0 contains over 500 commits, which include 42 new features, 58 improvements, 38 bug fixes, and 17 documentation changes.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.6.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.6.0/
🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.6.0/release_notes.html
🐳 Docker Image: “docker pull apache/airflow:2.6.0”
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.6.0

As the changelog is quite large, the following are some notable new features that shipped in this release.

Apache Airflow 2.5.0: Tick-Tock

Fri, 02 Dec 2022 00:00:00 +0000

Apache Airflow 2.5 has just been released, barely two and a half months after 2.4!

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.5.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.5.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.5.0/release_notes.html
🐳 Docker Image: docker pull apache/airflow:2.5.0
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.5.0

This quicker release cadence is a departure from our previous habit of releasing every five-to-seven months and was a deliberate effort to listen to you, our users, and get the changes and improvements into your workflows earlier.

Usability improvements to the Datasets UI

When we released Dataset aware scheduling in September we knew that the tools we gave to manage the Datasets were very much a Minimum Viable Product, and in the last two months the committers and contributors have been hard at work at making the UI much more usable when it comes to Datasets.

Apache Airflow 2.4.0: That Data Aware Release

Mon, 19 Sep 2022 00:00:00 +0000

Apache Airflow 2.4.0 contains over 650 “user-facing” commits (excluding commits to providers or chart) and over 870 total. That includes 46 new features, 39 improvements, 52 bug fixes, and several documentation changes.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html
🐳 Docker Image: docker pull apache/airflow:2.4.0
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0

Data-aware scheduling (AIP-48)

This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.

What does this mean, exactly? This is a great new feature that lets DAG authors create smaller, more self-contained DAGs, which chain together into a larger data-based workflow. If you are currently using ExternalTaskSensor or TriggerDagRunOperator you should take a look at datasets – in most cases you can replace them with something that will speed up the scheduling!

Airflow Survey 2022

Fri, 17 Jun 2022 00:00:00 +0000

Airflow User Survey 2022

This year’s survey has come and gone, and with it we’ve got a new batch of data for everyone! We collected 210 responses over two weeks. We continue to see growth in both contributions and downloads over the last two years, and expect that trend will continue through 2022.

The raw response data will be made available here soon, in the meantime, feel free to email john.thomas@astronomer.io for a copy.

Airflow Summit 2022

Mon, 16 May 2022 00:00:00 +0000

The biggest Airflow Event of the Year returns May 23–27! Airflow Summit 2022 will bring together the global community of Apache Airflow practitioners and data leaders.

What’s on the Agenda

During the free conference, you will hear about Apache Airflow best practices, trends in building data pipelines, data governance, Airflow and machine learning, and the future of Airflow. There will also be a series of presentations on non-code contributions driving the open-source project.

Apache Airflow 2.3.0 is here

Sat, 30 Apr 2022 00:00:00 +0000

Apache Airflow 2.3.0 contains over 700 commits since 2.2.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.3.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.3.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html
🐳 Docker Image: docker pull apache/airflow:2.3.0
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.3.0

As the changelog is quite large, the following are some notable new features that shipped in this release.

Dynamic Task Mapping(AIP-42)

There’s now first-class support for dynamic tasks in Airflow. What this means is that you can generate tasks dynamically at runtime. Much like using a for loop to create a list of tasks, here you can create the same tasks without having to know the exact number of tasks ahead of time.

What's new in Apache Airflow 2.2.0

Mon, 11 Oct 2021 00:00:00 +0000

I’m proud to announce that Apache Airflow 2.2.0 has been released. It contains over 600 commits since 2.1.4 and includes 30 new features, 84 improvements, 85 bug fixes, and many internal and doc changes.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.2.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.2.0/
🛠️ Changelog: https://airflow.apache.org/docs/apache-airflow/2.2.0/changelog.html
🐳 Docker Image: docker pull apache/airflow:2.2.0
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.2.0

As the changelog is quite large, the following are some notable new features that shipped in this release.

Custom Timetables (AIP-39)

Airflow has historically used cron expressions and timedeltas to represent when a DAG should run. This worked for a lot of use cases, but not all. For example, running daily on Monday-Friday, but not on weekends wasn’t possible.

Airflow Summit 2021

Sun, 21 Mar 2021 00:00:00 +0000

Airflow Summit 2021 is here!

The summit will be held online, July 8-16, 2021. Join us from all over the world to find out how Airflow is being used by leading companies, what is its roadmap and how you can participate in its development.

Useful information:

The official website: https://airflowsummit.org
Call for proposals is open until 12 April 2021. To submit your talk go to https://sessionize.com/airflow-summit-2021/
In case of any questions reach out to us via info@airflowsummit.org

Airflow Survey 2020

Tue, 09 Mar 2021 00:00:00 +0000

Apache Airflow Survey 2020

World of data processing tools is growing steadily. Apache Airflow seems to be already considered as crucial component of this complex ecosystem. We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.

It’s worth to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this is true, we will learn next year!

Apache Airflow 2.0 is here!

Thu, 17 Dec 2020 00:00:00 +0000

I am proud to announce that Apache Airflow 2.0.0 has been released.

The full changelog is about 3,000 lines long (already excluding everything backported to 1.10), so for now I’ll simply share some of the major features in 2.0.0 compared to 1.10.14:

A new way of writing dags: the TaskFlow API (AIP-31)

(Known in 2.0.0alphas as Functional DAGs.)

DAGs are now much much nicer to author especially when using PythonOperator. Dependencies are handled more clearly and XCom is nicer to use

Journey with Airflow as an Outreachy Intern

Sun, 30 Aug 2020 00:00:00 +0000

Outreachy is a program which organises three months paid internships with FOSS projects for people who are typically underrepresented in those projects.

Contribution Period

The first thing I had to do was choose a project under an organisation. After going through all the projects I chose “Extending the REST API of Apache Airflow”, because I had a good idea of what REST API(s) are, so I thought it would be easier to get started with the contributions. The next step was to set up Airflow’s dev environment which thanks to Breeze, was a breeze. Since I had never contributed to FOSS before so this part was overwhelming but there were plenty of issues labelled “good first issues” with detailed descriptions and some even had code snippets so luckily that nudged me in the right direction. These things about Airflow and the positive vibes from the community were the reasons why I chose to stick with Airflow as my Outreachy project.

Apache Airflow 1.10.12

Tue, 25 Aug 2020 00:00:00 +0000

Airflow 1.10.12 contains 113 commits since 1.10.11 and includes 5 new features, 23 improvements, 23 bug fixes, and several doc changes.

Details:

PyPI: https://pypi.org/project/apache-airflow/1.10.12/
Docs: https://airflow.apache.org/docs/1.10.12/
Changelog: http://airflow.apache.org/docs/1.10.12/changelog.html

Airflow 1.10.11 has breaking changes with respect to KubernetesExecutor & KubernetesPodOperator so I recommend users to directly upgrade to Airflow 1.10.12 instead.

Some of the noteworthy new features (user-facing) are:

Allow defining Custom XCom class

Until Airflow 1.10.11, the XCom data was only stored in Airflow Metadatabase. From Airflow 1.10.12, users would be able to define custom XCom classes. This will allow users to transfer larger data between tasks. An example here would be to store XCom in S3 or GCS Bucket if the size of data that needs to be stored is larger than XCom.MAX_XCOM_SIZE (48 KB).

Apache Airflow For Newcomers

Mon, 17 Aug 2020 00:00:00 +0000

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. A workflow is a sequence of tasks that processes a set of data. You can think of workflow as the path that describes how tasks go from being undone to done. Scheduling, on the other hand, is the process of planning, controlling, and optimizing when a particular task should be done.

Authoring Workflow in Apache Airflow.

Airflow makes it easy to author workflows using python scripts. A Directed Acyclic Graph (DAG) represents a workflow in Airflow. It is a collection of tasks in a way that shows each task’s relationships and dependencies. You can have as many DAGs as you want, and Airflow will execute them according to the task’s relationships and dependencies. If task B depends on the successful execution of another task A, it means Airflow will run task A and only run task B after task A. This dependency is very easy to express in Airflow. For example, the above scenario is expressed as

Implementing Stable API for Apache Airflow

Sun, 19 Jul 2020 00:00:00 +0000

My Outreachy internship is coming to its ends which is also the best time to look back and reflect on the progress so far.

The goal of my project is to Extend and Improve the Apache Airflow REST API. In this post, I will be sharing my progress so far.

We started a bit late implementing the REST API because it took time for the OpenAPI 3.0 specification we were to use for the project to be merged. Thanks to Kamil, who paved the way for us to start implementing the REST API endpoints. Below are the endpoints I implemented and the challenges I encountered, including how I overcame them.

Apache Airflow 1.10.10

Thu, 09 Apr 2020 00:00:00 +0000

Airflow 1.10.10 contains 199 commits since 1.10.9 and includes 11 new features, 43 improvements, 44 bug fixes, and several doc changes.

Details:

PyPI: https://pypi.org/project/apache-airflow/1.10.10/
Docs: https://airflow.apache.org/docs/1.10.10/
Changelog: http://airflow.apache.org/docs/1.10.10/changelog.html

Some of the noteworthy new features (user-facing) are:

Allow user to choose timezone to use in the RBAC UI

By default the Web UI will show times in UTC. It is possible to change the timezone shown by using the menu in the top right (click on the clock to activate it):

Apache Airflow 1.10.8 & 1.10.9

Sun, 23 Feb 2020 00:00:00 +0000

Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes.

We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8.

Details:

PyPI: https://pypi.org/project/apache-airflow/1.10.9/
Docs: https://airflow.apache.org/docs/1.10.9/
Changelog (1.10.8): http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07
Changelog (1.10.9): http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10

Some of the noteworthy new features (user-facing) are:

Add tags to DAGs and use it for filtering in the UI

In order to filter DAGs (e.g. by team), you can add tags in each dag. The filter is saved in a cookie and can be reset by the reset button.

Experience in Google Season of Docs 2019 with Apache Airflow

Fri, 20 Dec 2019 00:00:00 +0000

I came across Google Season of Docs (GSoD) almost by accident, thanks to my extensive HackerNews and Twitter addiction. I was familiar with the Google Summer of Code but not with this program. It turns out it was the inaugural phase. I read the details, and the process felt a lot like GSoC except that this was about documentation.

About Me

I have been writing tech articles on medium as well as my blog for the past 1.5 years. Blogging helps me test my understanding of the concepts as untangling the toughest of ideas in simple sentences requires a considerable time investment.

Airflow Survey 2019

Wed, 11 Dec 2019 00:00:00 +0000

Apache Airflow Survey 2019

Apache Airflow is growing faster than ever. Thus, receiving and adjusting to our users’ feedback is a must. We created survey and we got 308 responses. Let’s see who Airflow users are, how they play with it, and what they miss.

Overview of the user

What best describes your current occupation?

	No.	%
Data Engineer	194	62.99%
Developer	34	11.04%
Architect	23	7.47%
Data Scientist	19	6.17%
Data Analyst	13	4.22%
DevOps	13	4.22%
IT Administrator	2	0.65%
Machine Learning Engineer	2	0.65%
Manager	2	0.65%
Operations	2	0.65%
Chief Data Officer	1	0.32%
Engineering Manager	1	0.32%
Intern	1	0.32%
Product owner	1	0.32%
Quant	1	0.32%

In your day to day job, what do you use Airflow for?

New Airflow website

Wed, 11 Dec 2019 00:00:00 +0000

The brand new Airflow website has arrived! Those who have been following the process know that the journey to update the old Airflow website started at the beginning of the year. Thanks to sponsorship from the Cloud Composer team at Google that allowed us to collaborate with Polidea and with their design studio Utilo, and deliver an awesome website.

Documentation of open source projects is key to engaging new contributors in the maintenance, development, and adoption of software. We want the Apache Airflow community to have the best possible experience to contribute and use the project. We also took this opportunity to make the project more accessible, and in doing so, increase its reach.

ApacheCon Europe 2019 — Thoughts and Insights by Airflow Committers

Fri, 22 Nov 2019 00:00:00 +0000

Is it possible to create an organization that delivers tens of projects used by millions, nearly no one is paid for doing their job, and still, it has been fruitfully carrying on for more than 20 years? Apache Software Foundation proves it is possible. For the last two decades, ASF has been crafting a model called the Apache Way—a way of organizing and leading tech open source projects. Due to this approach, which is strongly based on the “community over code” motto, we can enjoy such awesome projects like Apache Spark, Flink, Beam, or Airflow (and many more).

Documenting using local development environment

Fri, 22 Nov 2019 00:00:00 +0000

Documenting local development environment of Apache Airflow

From Sept to November 2019 I have been participating in a wonderful initiative, Google Season of Docs.

I had a pleasure to contribute to the Apache Airflow open source project as a technical writer. My initial assignment was an extension to the GitHub-based Contribution guide.

From the very first days I have been pretty closely involved into inter-project communications via emails/slack and had regular 1:1s with my mentor, Jarek Potiuk.

It's a "Breeze" to develop Apache Airflow

Fri, 22 Nov 2019 00:00:00 +0000

The story behind the Airflow Breeze tool

Initially, we started contributing to this fantastic open-source project [Apache Airflow] with a team of three which then grew to five. When we kicked it off a year ago, I realized pretty soon where the biggest bottlenecks and areas for improvement in terms of productivity were. Even with the help of our client, who provided us with a “homegrown” development environment it took us literally days to set it up and learn some basics.