Our automated pipelines directory contains code shared by our automated pipelines, including REST, GraphQL, Webhooks, CodeQL CLI, and GitHub Apps.

An automated pipeline consumes data from an external source that is used to create content for docs.github.com. An automated pipeline does not automate documentation that is created by our content writing team. For example, if a writer creates a structured data file like YAML or JSON that lives in the `docs-internal` repo, using that data to create a page does not create an automated pipeline.

Automated pages allow for manually created content to be prepended to the automated content, but do not allow for manually created content to be appended or interspersed within automated content. Manually created content (that is prepended to automated content) lives in the Markdown file associated with the automated page, along with the article's frontmatter metadata.

- [CodeQL CLI](../codeql-cli/README.md)

- [GitHub Apps](../github-apps/README.md)

- [GraphQL](../graphql/README.md)

- [REST](../rest/README.md)

- [Webhooks](../webhooks/README.md)

We currently have two patterns that we used to create automated pipelines:

- REST, Webhooks, GitHub Apps, and GraphQL pipelines consume external structured data and transform that data into a JSON file that is used to create content for a page on docs.github.com. Typically, data files are a 1:1 mapping to a specific page on docs.github.com.

- The CodeQL CLI pipeline takes an unstructured ReStructuredText file and transforms it directly into a Markdown file with frontmatter, that uses the same authoring format as the rest of the docs.

Each pipeline should be evaluated individually to determine the best architecture for simplicity, maintainability, and requirements.

For example:

- Is the content being displayed basic Markdown content? For example, does the content avoid using complex tables and interactive elements? If so, then writing the Markdown content directly and avoiding the need to create a structured data file that requires a React component may be the best approach. This was the case for the CodeQL CLI pipeline. One caveat to think about before writing Markdown directly is whether the content will need liquid versioning. The current pipeline that writes Markdown directly does not need to use liquid versioning. Liquid versioning which would increase the complexity quite a bit. All of the Markdown content in each article that is generated from the CodeQL CLI pipeline applies to all versions listed in the `versions` frontmatter property, simplifying the Markdown generation process.

- Is the page interactive like the REST and Webhooks pages? If so, then the data will likely need to be structured data. In that case, a new React component may be needed to display the data.

When creating a new pipeline, the source data that is being consumed may not have all of the necessary data needed to create the page. Oftentimes, source data does not contain descriptions and prose that our content writers have crafted to describe properties or concepts. In this case, it's common to need to scrape data from our docs and merge it into a new field in the structured data file that we intend to consume. When creating a new pipeline, you'll need to work with the team that owns the source data to create a plan for adding any additional properties and agreeing on a format that will work best for both teams.

- Create a new directory in the `src` directory with the name of the pipeline. For example, `src/codeql-cli`.

- Add a README.md file that describes the pipeline and how to use it. This should include any dependencies, how to run the pipeline, and any other information that is needed to use the pipeline. It's strongly recommended to include a diagram showing the overall flow of the pipeline.

- Each pipeline typically requires a workflow to allow scheduling or manually running the pipeline. The workflow should be placed in the `.github/workflows` directory and named `sync-<pipeline-name>.js`. Each workflow typically requires adding a manual run option and an input parameter to specify the source repo's branch to use.

- Each pipeline will need a `scripts` directory with (at minimum) a `scripts/sync.js` file to run the pipeline.

- If the pipeline will contain structured data, you will need to add a `src/<pipeline-name>/data` directory. The files inside the `data` directory are typically organized by version (e.g., `src/webhooks/data/fpt/*`).

- Pipelines typically have tests specific to the pipeline that are placed in the `src/<pipeline-name>/tests` directory. There is no need to add tests that render the page because all autogenerated pages are tested in `src/automated-pipelines/tests/rendering.js`.

- If the pipeline uses a Next.js page component (e.g., `pages/**/*.tsx`), ensure there is a test that fails if that page component is moved or deleted.

Slack: `#docs-engineering`

Repo: `github/docs-engineering`

If you have a question about automation pipelines, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with one of the automation pipelines, you can open an issue in the `github/docs-engineering` repository.

Our CodeQL CLI pipeline creates autogenerated documentation for docs.github.com from the source code used to build the [CodeQL CLI](https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/about-the-codeql-cli).

The pipeline is used to generate Markdown files that create article pages on the docs.github.com site.


A [workflow](.github/workflows/sync-codeql-cli.yml) is used to trigger the automation of the CodeQL CLI documentation. The workflow is manually triggered by a member of the GitHub Docs team approximately every two weeks to align to releases of the CodeQL CLI. The workflow takes an input parameter that specifies the branch to pull the source files from in the semmle-code repo. If the branch input is omitted, the workflow will default to the `main` branch.

The workflow runs the `src/codeql-cli/scripts/sync.js` script, which generates Markdown files under `content/code-security/codeql-cli/codeql-cli-manual`.

The workflow automatically creates a new pull request with the changes and the label `codeql-cli-pipeline`.

To run the CodeQL CLI pipeline locally:

1. Clone the `semmle-code` repository inside your local `docs-internal` repository.

2. [Install Pandoc](https://pandoc.org/installing.html). You can `brew install pandoc` on macOS.

3. Run `src/codeql-cli/scripts/sync.js`.

- `src/rest/lib/config.json` - A configuration file used to specify metadata about the REST pipeline.

- `src/rest/scripts` - The scripts and source code used run the CodeQL CLI pipeline.

- `src/rest/scripts/sync.js` - The entrypoint script that runs the CodeQL CLI pipeline.

The content writers can manually update parts of the autogenerated Markdown files in `content/code-security/codeql-cli/codeql-cli-manual`. When new Markdown files are added they will get all of the frontmatter properties defined in the `defaultFrontmatter` property in `src/codeql-cli/lib/config.js`.

When a new Markdown file is created, a writer can manually change any of the frontmatter. The pipeline will not overwrite the frontmatter on subsequent runs.

Writers can also add an introduction paragraph _above_ the following Markdown comment:

Slack: `#docs-engineering`

Repo: `github/docs-engineering`

If you have a question about the CodeQL CLI pipeline, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with the CodeQL CLI pipeline, you can open an issue in the `github/docs-engineering` repository.

Our GitHub Apps pipeline creates autogenerated documentation for docs.github.com from the OpenAPI stored in the open-source repository [`github/rest-api-description`](https://github.com/github/rest-api-description).

The pipeline is used to generate data that is used by the docs.github.com site when deployed locally, in preview environments, or in production.


A [workflow](.github/workflows/sync-openapi.yml) is used to trigger the automation of the GitHub Apps documentation. The workflow runs automatically on a schedule. The workflow that triggers the GitHub Apps pipeline also triggers other automation pipelines that use the OpenAPI as the source data:

- GitHub Apps

- REST

- Webhooks

The workflow automatically creates a pull request with the changes (for all three pipelines) and the label `github-openapi-bot`.

The workflow runs the `src/rest/scripts/update-files.js` script.

To run the GitHub Apps pipeline locally:

1. Clone the [`github/rest-api-description`](https://github.com/github/rest-api-description) repository inside your local `docs-internal` repository.

1. Run `src/rest/scripts/update-files.js -s rest-api-description -o github-apps`.

- `src/rest/data` - The automatically generated data files created by running this pipeline.

- `src/rest/lib` - The source code used in production for the automated documentation generated by the GitHub Apps pipeline and configuration files edited by content and engineering team members.

- `src/rest/lib/config.json` - A configuration file used to specify metadata about the GitHub Apps pipeline. This file contains the SHA of the commit in `github/rest-api-description` that was used to generate the data in `src/github-apps/data`. This value is automatically updated when the pipeline runs.

- `src/rest/scripts` - The scripts and source code used run the GitHub Apps pipeline, which updates the `src/github-apps/data` directory.

- `src/rest/tests` - The tests used to verify the GitHub Apps pipeline.

The `src/github-apps/lib/config.json` file can contain any metadata needed by the content and engineering teams to configure the webhooks pipeline. The file currently only contains the following property:

- `sha` - The SHA of the commit in `github/rest-api-description` that was used to generate the data in `src/github-apps/data`. This value is automatically updated when the pipeline runs.

The content writers can manually update parts of the autogenerated Markdown file `content/rest/overview/endpoints-available-for-github-apps.md`. All frontmatter properties except `versions` can be manually modified and will not be overwritten on subsequent runs of the pipeline. The pipeline does update the `versions` property on each run.

Writers can also add an introduction paragraph _above_ the following Markdown comment:

Slack: `#docs-engineering`

Repo: `github/docs-engineering`

If you have a question about the GitHub Apps pipeline, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with the GitHub Apps pipeline, you can open an issue in the `github/docs-engineering` repository.

Our REST pipeline creates autogenerated REST API documentation for docs.github.com/rest from the OpenAPI stored in the open-source repository [`github/rest-api-description`](https://github.com/github/rest-api-description).

The pipeline is used to generate data that is used by the docs.github.com site when deployed locally, in preview environments, or in production.


A [workflow](.github/workflows/sync-openapi.yml) is used to trigger the automation of the REST documentation. The workflow runs automatically on a schedule. The workflow that triggers the REST pipeline also triggers other automation pipelines that use the OpenAPI as the source data:

- GitHub Apps

- REST

- Webhooks

The workflow automatically creates a pull request with the changes (for all three pipelines) and the label `github-openapi-bot`. The workflow runs the `src/rest/scripts/update-files.js` script, which creates, deletes, or updates Markdown files in the `content/rest` directory.

You can manually run the workflow leaving the source branch input parameter with the default value of `main`. Running the workflow creates a new pull request in the `github/docs-internal` repo with the changes.

If there is a pull request that is still unmerged in `github/rest-api-description` that needs to be propagated to the docs.github.com site, manually run the workflow and provide the branch name of the pull request in the `github/rest-api-description` repo that you would like to get the changes from. This will create a new pull request in the `github/docs-internal` repo with the changes.

To run the REST pipeline locally:

1. Clone the [`github/rest-api-description`](https://github.com/github/rest-api-description) repository inside your local `docs-internal` repository.

1. Run `src/rest/scripts/update-files.js -s rest-api-description -o rest`. Note, by default `-o rest` is specified, so you can omit it.

- `src/rest/api` - The source code for the api endpoint used in production that redirects specific anchor links on a page under `docs.github.com/rest`.

- `src/rest/data` - The automatically generated data files created by running this pipeline.

- `src/rest/lib` - The source code used in production for the automated documentation generated by the REST pipeline and configuration files edited by content and engineering team members.

- `src/rest/lib/config.json` - A configuration file used to specify metadata about the REST pipeline.

- `src/rest/scripts` - The scripts and source code used run the REST pipeline, which updates the `src/rest/data` directory.

- `src/rest/scripts/update-files.js` - The entrypoint script that runs the REST pipeline.

- `src/rest/tests` - The tests used to verify the REST pipeline.

The `src/rest/lib/config.json` file contain metadata used by the content and engineering team to configure the REST pipeline. The file contains the following properties:

- `api-versions` - A list of the calendar-date API versions that are available for each version.

- `versionMapping` - A mapping of the API version name used in the OpenAPI schema in `github/rest-api-description` to the version short name used in the `github/docs-internal` repo.

- `frontmatterDefaults` - A list of default frontmatter values to append to each Markdown file in `content/rest`.

- `targetDirectory` - The directory in `content` where the Markdown files that correspond to this pipeline are located.

- `indexOrder` - The order of the children in the `index.md` file in the `targetDirectory`. The `startsWith` keyword lists the directories under `targetDirectory` that should be listed first. All other automatically created directories are appended to this list.

- `sha` - The SHA of the commit in `github/rest-api-description` that was used to generate the data in `src/rest/data` and Markdown files in `content/rest`. This value is automatically updated when the pipeline runs.

The content writers can manually update parts of the autogenerated Markdown files in `content/rest`. When new Markdown files are added they will get all of the frontmatter properties defined in the `defaultFrontmatter` property in `src/rest/lib/config.js`.

Writers can also add an introduction paragraph _above_ the following Markdown comment:

Slack: `#docs-engineering`

Repo: `github/docs-engineering`

If you have a question about the REST pipeline, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with the REST pipeline, you can open an issue in the `github/docs-engineering` repository.