{"id":13521,"date":"2020-11-19T07:55:19","date_gmt":"2020-11-19T07:55:19","guid":{"rendered":"https:\/\/slack.engineering\/?p=13521"},"modified":"2020-11-19T23:00:11","modified_gmt":"2020-11-19T23:00:11","slug":"migrating-slack-airflow-to-python-3-without-disruption","status":"publish","type":"post","link":"https:\/\/slack.engineering\/migrating-slack-airflow-to-python-3-without-disruption\/","title":{"rendered":"Migrating Slack Airflow to Python 3 Without Disruption"},"content":{"rendered":"

Last year, we migrated Airflow from 1.8 to 1.10 at Slack (see here<\/a>) and we did a \u201cBig bang\u201d upgrade because of the constraints we had. This year, due to Python 2 reaching end of life, we again had a major migration of Airflow from Python 2 to 3 and we wanted to put our learnings from 2019 into practice. This post describes how we moved our Apache Airflow infrastructure, and hundreds of DAGs, from Python 2 to Python 3 in a more reliable \u201cRed-black\u201d deployment using Celery<\/a> queues, resulting in the migration being completely transparent to our users.<\/p>\n

Apache Airflow<\/a> is a tool for describing, executing, and monitoring workflows. At Slack, we use Airflow to schedule our data warehouse workflows, which includes product metrics, business metrics, and also other use-cases like search and offline indexing. We run hundreds of workflows and tens of thousands of airflow tasks every day.<\/p>\n

High-level steps for Airflow Python 3 migration<\/h2>\n

Here are the list of things we did for the migration:<\/p>\n

    \n
  1. Set up a virtual environment:<\/b> Created a working Python 3 virtual environment with Airflow and all the dependencies.<\/li>\n
  2. Set up Python 3 workers:<\/b> Launched Python 3 celery workers running the virtual environment from step 1, and make them pick tasks from a \u201cpython3\u201d queue on Redis.<\/li>\n
  3. Clean up phase 1:<\/b> Removed stale or unused DAGs to make the migration easier.<\/li>\n
  4. Fix Python 3 incompatibilities in DAGs<\/b>: Updated DAGs to be Python 2\/3 compatible but let them continue to run on Python 2 workers.<\/li>\n
  5. Move the DAGs to Python 3 workers: <\/b>Tested and switched DAGs from Python 2 to Python 3 workers. We started with a small set and kept increasing depending on success rate. If DAGs failed, we fixed forward or rolled back to Python 2 workers.<\/li>\n
  6. Migrate the Airflow services to Python 3:<\/b> Switched web server, scheduler, and flower to Python 3.<\/li>\n
  7. Clean up phase 2:<\/b> Cleaned up Python 2 references and virtual environments, and terminated Python 2 celery workers.<\/li>\n<\/ol>\n

    We did these steps in a dev environment first and then in prod. Let’s delve into each step in the following sections (except for the clean-up steps, which are self-explanatory).<\/p>\n

    Pro tip:<\/b> The above order of migration is important; migrating scheduler or webserver to Python 3 before fixing DAG incompatibilities will result in errors because the scheduler in Python 3 will be unable to parse the DAG.<\/p>\n

    Set up a virtual environment<\/h3>\n

    To set up a working python3 virtual environment, we did the following:<\/p>\n