# CodeQL CLI pipeline

Our CodeQL CLI pipeline creates autogenerated documentation for docs.github.com from the source code used to build the [CodeQL CLI](https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/about-the-codeql-cli).

The pipeline is used to generate Markdown files that create article pages on the docs.github.com site.

## How does it work

![A flow chart describing how the automation pipeline for CodeQL CLI generates documentation](./codeql-cli-pipeline-flowchart.png)

A [workflow](https://github.com/github/docs-internal/blob/main/.github/workflows/sync-codeql-cli.yml) is used to trigger the automation of the CodeQL CLI documentation. The workflow is manually triggered by a member of the GitHub Docs team approximately every two weeks to align to releases of the CodeQL CLI. The workflow takes an input parameter that specifies the branch to pull the source files from in the semmle-code repo. If the branch input is omitted, the workflow will default to the `main` branch.

The workflow runs the `src/codeql-cli/scripts/sync.ts` script, which generates Markdown files under `content/code-security/codeql-cli/codeql-cli-manual`.

The workflow automatically creates a new pull request with the changes and the label `codeql-cli-pipeline`.

## Local development

To run the CodeQL CLI pipeline locally:

1. Clone the `semmle-code` repository inside your local `docs-internal` repository.
2. [Install Pandoc](https://pandoc.org/installing.html). You can `brew install pandoc` on macOS.
3. Run `src/codeql-cli/scripts/sync.ts`.

## About this directory

- `src/codeql-cli/lib/config.json` - A configuration file used to specify metadata about the CodeQL CLI pipeline.
- `src/codeql-cli/scripts` - The scripts and source code used run the CodeQL CLI pipeline. 
  - `src/codeql-cli/scripts/sync.ts` - The entrypoint script that runs the CodeQL CLI pipeline.

## Content team

The content writers can manually update parts of the autogenerated Markdown files in `content/code-security/codeql-cli/codeql-cli-manual`. When new Markdown files are added they will get all of the frontmatter properties defined in the `defaultFrontmatter` property in `src/codeql-cli/lib/config.ts`. 

When a new Markdown file is created, a writer can manually change any of the frontmatter. The pipeline will not overwrite the frontmatter on subsequent runs. 

Writers can also add an introduction paragraph _above_ the following Markdown comment:

```markdown
<!-- Content after this section is automatically generated -->
```

## How to get help

Slack: `#docs-engineering`
Repo: `github/docs-engineering`

If you have a question about the CodeQL CLI pipeline, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with the CodeQL CLI pipeline, you can open an issue in the `github/docs-engineering` repository.

## Dependency Matrix

### Required dependencies

| Dependency | Purpose | Installation | Notes |
|------------|---------|--------------|-------|
| **Pandoc** | Convert ReStructuredText to Markdown | `brew install pandoc` (macOS) | Version 2.x+ required |
| **semmle-code repo** | Source ReStructuredText files | Clone at root of docs-internal | Private GitHub repo |
| **Node.js** | Run sync script | Already in project | Version specified in `.nvmrc` |

### Source data location
- **Repository**: `github/semmle-code` (private)
- **Branch**: Configurable (default: `main`)
- **Source files**: ReStructuredText documentation in semmle-code
- **Output location**: `content/code-security/codeql-cli/codeql-cli-manual/`

### Version compatibility
- CodeQL CLI releases approximately every 2 weeks
- Pipeline should be run to align with CLI releases
- Multiple CLI versions may be documented simultaneously

## Escalation Policy

### Ownership
- **Team**: Docs Engineering
- **Source team**: Code Security (CodeQL team)

### Escalation path
1. **Pipeline failures** → #docs-engineering Slack
2. **Source data issues** → #code-security-docs Slack
3. **Pandoc conversion issues** → Check Pandoc version and file format
4. **Repo access issues** → Request access to `github/semmle-code`

### On-call procedures
If the pipeline fails:
1. Check workflow logs for errors
2. Verify Pandoc is installed and working
3. Confirm access to `semmle-code` repo
4. Check for ReStructuredText format changes
5. Test conversion locally before re-running
6. Escalate to Code Security team if source data issue

### Common failure modes
- **Pandoc not found** - Install Pandoc locally or in CI
- **semmle-code not cloned** - Clone repo at docs-internal root
- **Conversion errors** - Check for unsupported RST syntax
- **Branch not found** - Verify branch name in workflow input

## Known Constraints

### Pipeline limitations
- **Manual trigger only** - No automated schedule (runs on-demand)
- **Pandoc dependency** - Requires external tool installation
- **Repo access** - Requires access to private `semmle-code` repo
- **Local testing** - Must clone semmle-code locally to test

### Content constraints
- **Frontmatter preservation** - Only first sync sets frontmatter
- **Introduction placement** - Must be above auto-generation comment
- **No inline editing** - Cannot edit autogenerated sections
- **Version handling** - Limited liquid versioning support

### Technical constraints
- **ReStructuredText format** - Source must be valid RST
- **Pandoc conversion** - Limited control over conversion output
- **File naming** - Determined by source file structure
- **Link rewriting** - May need manual adjustment after generation

### Process constraints
- **Release alignment** - Sync timing depends on CodeQL releases
- **PR review required** - Changes must be reviewed before merge
- **Content coordination** - Writers must coordinate intro additions
- **Multiple versions** - May need separate syncs for version branches

We are not expecting significant new work to this pipeline, but we will support incoming changes from semmle-code.