diff --git a/.claude/skills/audit-the-list/SKILL.md b/.claude/skills/audit-the-list/SKILL.md new file mode 100644 index 0000000000..254c57e05b --- /dev/null +++ b/.claude/skills/audit-the-list/SKILL.md @@ -0,0 +1,42 @@ +--- +name: audit-the-list +description: Audit README.md sections against the shortlist rules — re-verify every entry's verdict with live data, promote or demote challengers, restructure oversized use cases, prune, and evaluate proposed additions. Fire when the maintainer asks to audit, sweep, prune, re-check, or maintain sections, or asks whether an entry (or a proposed one) deserves its slot. +argument-hint: [all, specific sections, or none for a recommendation] +--- + +# Audit the list + +An Audit re-runs the admission test over live sections of README.md: every entry re-verified against current data, tiers reassessed, structure reconsidered, then the maintainer adjudicates and the result is committed. The rules live in CONTRIBUTING.md (Admission, Evidence, Entry Ordering) — this skill is the process that applies them, not a second copy of them. Entry changes land only on the maintainer's explicit go. + +## 1. Scope + +Resolve the scope from the arguments. Named sections mean exactly those, whether or not they were audited before (a re-run is how a past verdict gets rechecked). `all` is ambiguous — `AskUserQuestion` whether it means every section or only never-audited ones. Batch the work one Thematic Group per sitting. Done when the section list is settled. + +With no arguments, or to see which sections were never audited, run `uv run python .claude/skills/audit-the-list/scripts/audit_priority.py`. It dates each section's last audit from `audit:`/`prune: sweep` and `style: audit` commit subjects plus the Last Audited lines in `docs/audit-logs.md`, then ranks sections: entries added or removed since that audit first, most changes first, then unchanged sections by oldest audit, never-audited first. Print its ranking, then AskUserQuestion with the top four sections as options, the first marked Recommended, each description carrying its changed entries or its last-audit date. A renamed section loses its audit history and ranks as never audited; the script lists the orphaned subjects, so say so when that happens. + +## 2. Evidence + +Fetch live evidence for every entry in scope before judging anything (CLAUDE.md verification rule): + +- **Downloads/month**: `cd website && uv run python fetch_pypi_downloads_via_clickpy.py` — free keyless ClickPy sweep of the full README, sole writer of `data/pypi_downloads.tsv` (rewritten from scratch each run; header row, every row stamped with its `fetched_at` date). A cache whose `fetched_at` is within the last 7 days is current enough for verdicts — skip the sweep; older than that, re-run it (costs ~1s). Cross-checks print to stdout, take explicit names, and never touch the cache: `fetch_pypi_downloads_via_bigquery.py ...` (canonical source, maintainer's own GCP account, `--dry-run` first — the docstring carries the cost constraints; full-README sweeps exceed the free tier, keep name lists small), `fetch_pypi_downloads_via_pepy.py ...` (needs `PEPY_TECH_API_KEY` in repo-root `.env`, throttled to 5 requests/minute), or `https://pypistats.org/api/packages/{name}/recent` paced 8s or slower. pypistats excludes mirror/CI traffic; ClickPy, BigQuery, and pepy include it — never mix sources within one comparison. +- **Repo state**: archived flag, last push, created date, stars, description — `gh api repos/{owner}/{repo}`, GitLab API for GitLab-hosted projects. +- **PyPI metadata** (`https://pypi.org/pypi/{name}/json`) wherever a name might not be the canonical package — ownership collisions and wrong display names surface here. Wrong-package cache rows are common: the sweep looks up README display names, so when the display name differs from the canonical package, the row silently measures an unrelated squatter or a dead predecessor. When a famous entry shows a count that looks off (too small, or absurdly round), verify identity before citing it, and fetch the canonical package's count via pepy. +- **Renaming an entry can silently expose it to the sweep, and no automatic check catches that.** The sweep only queries names matching `PYPI_NAME_RE`, so a display name carrying a space or other non-PyPI shape is skipped outright. Rename it into a PyPI-shaped one and it starts being queried: `uv audit` became `uv-audit`, and PyPI `uv-audit` is an unrelated third-party tool by rocshers, not Astral's subcommand. After any rename, fetch `https://pypi.org/pypi/{new-name}/json` and confirm the package is the project the entry links to; when it is not, add a null entry to `website/data/pypi_name_overrides.json` with the reason. Only one case is guarded automatically: `test_bundled_entries_are_never_queried_on_pypi` requires a null override for every `(part of X)` entry. Two broader checks were measured against the full list and rejected — do not re-propose either. Requiring a package's PyPI metadata to link back to the entry's GitHub repo misses this case completely, since `uv-audit` declares no repo URL at all, as do 26 legitimate entries including numba, selenium and pyglet; it also flags 10 benign org moves and binding splits (gensim, msgspec, pendulum, instructor, duckdb). Flagging entries whose display name differs from their repo name returns 46 hits, every one legitimate, and `uv-build` sits among them with the shape identical to `uv-audit` despite being a real Astral package. + +Done when every entry in scope has downloads (or a stated no-signal reason), repo state, and a confirmed PyPI name. + +## 3. Verdicts + +Draft a verdict and reason for every entry, restructure before cap: decide Splits, mints, and re-homes first — cutting before restructuring destroys entries — then apply the cap tier by tier, including promotions and demotions between obvious choice and challenger. Ground every reason in the fetched evidence; label anything unverifiable as a judgment call. Where fresh evidence contradicts the standing verdict, say so in the reason instead of silently keeping the seed. A proposed addition under `docs/audit-logs.md` "Rejected Additions" seeds as reject from that line. + +## 4. Review and go + +Run the `preview-verdicts` skill: it generates the interactive review page and defines how the maintainer's feedback JSON comes back and gets processed. Their verdicts are final. Before touching README.md, surface what the feedback implies but does not decide — cap overflows, homeless entries after a Split, tier for a tierless flip — as named assumptions or questions, then get an explicit go. + +## 5. Execute + +One commit per section: body lists each removal with its reason and downloads figure, naming the figure's source and fetch date so a later audit can tell whether a trend compares like with like (ClickPy counts mirror traffic, pypistats does not); restructures, tier moves, and reorders ride the same commit. A restructure that renames or dissolves a category or subcategory adds `"old path": "new path"` to `website/data/redirects.json` in that commit, so the old URL keeps its search ranking; the build fails when a redirect target no longer exists, so a later rename also updates the entries pointing at the renamed path. Format-only outcomes (no removals) are a single style commit whose subject starts `style: audit
`. A section whose audit changes nothing gets a line under Last Audited in `docs/audit-logs.md` instead (`-
-- , no changes.`, replacing its earlier line), committed, since the ranking script reads an audit only from a commit or that line. `make test` before every commit. After the last add or rename, re-run the ClickPy sweep, since new names have no cache row and the build would count them without downloads, then `make build`. Generic commit helpers tend to split a section audit into structural and per-subcategory commits — if that happens, squash back to one commit per section. Done when the tree is clean, tests passed before each commit, and the build count reconciles with the adjudicated changes. + +## 6. Record + +A conclusion that outlives the sitting goes into the repo before the audit ends: admission or evidence rules into CONTRIBUTING.md, repo process and environment facts into CLAUDE.md and AGENTS.md (kept in sync). Done when nothing the next audit needs is stranded in conversation. diff --git a/.claude/skills/audit-the-list/scripts/audit_priority.py b/.claude/skills/audit-the-list/scripts/audit_priority.py new file mode 100644 index 0000000000..f67668364c --- /dev/null +++ b/.claude/skills/audit-the-list/scripts/audit_priority.py @@ -0,0 +1,128 @@ +"""Rank README.md sections for the next audit. + +A section's last audit is the newest of: a commit whose subject starts with +`audit: sweep`, `prune: sweep`, or `style: audit` and names the section or its +Thematic Group, and its line under docs/audit-logs.md "Last Audited". A commit +on a merged branch dates from that merge. Sections whose entries changed since +then come first, most changes first; the rest +follow by oldest audit, never-audited first. +""" + +import re +import subprocess +import sys +from functools import cache +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[4] +sys.path.insert(0, str(ROOT / "website")) + +from readme_parser import parse_readme + +# The shortlist reform; audits start after it, and never-audited sections diff against it. +REFORM = "553bf88" +AUDIT_SUBJECT_RE = re.compile(r"^(?:(?:audit|prune): sweep|style: audit) (.+?)(?: —|,| to the shortlist cap|$)") +REGISTER_LINE_RE = re.compile(r"^- (.+?) -- (\d{4}-\d{2}-\d{2})") + + +def git(*args: str) -> str: + return subprocess.run(["git", "-C", str(ROOT), *args], capture_output=True, text=True, check=True).stdout + + +@cache +def entries_at(rev: str) -> dict[str, dict[str, set[str]]]: + """Section -> {display name: {normalized name, normalized URL}}. + + An entry matches across revisions on either key, so a rename that keeps its link, or a link fix that keeps its name, is no change. + """ + sections = {} + for group in parse_readme(git("show", f"{rev}:README.md")): + for category in group["categories"]: + links = [*category["entries"], *(sub for entry in category["entries"] for sub in entry["also_see"])] + sections[category["name"]] = {link["name"]: {re.sub(r"[-_.]+", "-", link["name"]).lower(), link["url"].rstrip("/").lower()} for link in links} + return sections + + +def missing(entries: dict[str, set[str]], other: dict[str, set[str]]) -> list[str]: + """Display names in entries that match no entry in other.""" + keys = set().union(*other.values()) + return [name for name, own in entries.items() if not own & keys] + + +def resolve(text: str, groups: dict[str, list[str]]) -> list[str]: + """Sections named in text, longest name first so "Database Drivers" never counts as "Database".""" + found = [] + for name in sorted(groups, key=len, reverse=True): + if re.search(rf"(? None: + current = parse_readme(git("show", "HEAD:README.md")) + group_of = {c["name"]: g["name"] for g in current for c in g["categories"]} + # Section names map to themselves, Thematic Group names to all their sections. + names = {s: [s] for s in group_of} | {g["name"]: [c["name"] for c in g["categories"]] for g in current} + + last: dict[str, tuple[int, str, str]] = {} # section -> (timestamp, date, rev) + + def stamp(section: str, timestamp: int, date: str, rev: str) -> None: + if section not in last or timestamp > last[section][0]: + last[section] = (timestamp, date, rev) + + # Branch commit -> the merge that landed it, so an audit's follow-up `feat: add` commits count as audited. + landed = {commit: merge for merge in git("rev-list", "--first-parent", "--merges", f"{REFORM}..HEAD").split() for commit in git("rev-list", f"{merge}^1..{merge}").split()} + + unmatched = [] + for line in git("log", "--format=%H%x09%ct%x09%cs%x09%s", f"{REFORM}..HEAD").splitlines(): + rev, timestamp, date, subject = line.split("\t", 3) + rev = landed.get(rev, rev) + if match := AUDIT_SUBJECT_RE.match(subject): + sections = resolve(match.group(1), names) + for section in sections: + stamp(section, int(timestamp), date, rev) + if not sections: + unmatched.append(subject) + + register = (ROOT / "docs" / "audit-logs.md").read_text().splitlines() + start = register.index("## Last Audited") + for lineno, line in enumerate(register[start:], start + 1): + if match := REGISTER_LINE_RE.match(line): + blame = git("blame", "--porcelain", "-L", f"{lineno},{lineno}", "--", "docs/audit-logs.md").splitlines() + rev = blame[0].split()[0] + if set(rev) == {"0"}: # uncommitted line + rev = "HEAD" + timestamp = int(next(b.split()[1] for b in blame if b.startswith("committer-time "))) + for section in resolve(match.group(1), names): + stamp(section, timestamp, match.group(2), rev) + + head = entries_at("HEAD") + changed, unchanged = [], [] + for section in group_of: + _, date, rev = last.get(section, (0, "never", REFORM)) + before = entries_at(rev).get(section, {}) + added = missing(head[section], before) + removed = missing(before, head[section]) + row = (section, date, rev[:7] if section in last else "", added, removed) + (changed if added or removed else unchanged).append(row) + + # "never" sorts after digits, so map it to "" to put never-audited first. + changed.sort(key=lambda r: (-(len(r[3]) + len(r[4])), "" if r[1] == "never" else r[1])) + unchanged.sort(key=lambda r: "" if r[1] == "never" else r[1]) + + print("Entries added or removed since last audit:") + for section, date, rev, added, removed in changed: + diff = " ".join([*(f"+{n}" for n in sorted(added)), *(f"-{n}" for n in sorted(removed))]) + print(f" {section} [{group_of[section]}] last {date} {rev} {diff}") + print("\nNo entry changes, oldest audit first:") + for section, date, rev, _, _ in unchanged: + print(f" {section} [{group_of[section]}] last {date} {rev}") + if unmatched: + print("\nAudit subjects naming no current section or group (renamed or dissolved since):") + for subject in unmatched: + print(f" {subject}") + + +if __name__ == "__main__": + main() diff --git a/.claude/skills/preview-verdicts/SKILL.md b/.claude/skills/preview-verdicts/SKILL.md new file mode 100644 index 0000000000..fbae400339 --- /dev/null +++ b/.claude/skills/preview-verdicts/SKILL.md @@ -0,0 +1,30 @@ +--- +name: preview-verdicts +description: Generate the interactive keep/remove/add/reject verdict preview (HTML page with per-row feedback controls) whenever a prune sweep, batch entry edit, or restructure needs maintainer review before touching README.md — and process the feedback JSON the maintainer pastes back. +--- + +# Verdict preview + +Maintainer review happens through an interactive HTML page: one row per entry with your seeded verdict and reason, verdict buttons and a reason field for the maintainer, and a **Copy feedback** button that exports only changed or commented rows as JSON. Generate the page, wait for the pasted JSON, then apply it. Entry changes land in README.md only after the review — and only on an explicit go. + +## Generate the preview + +1. Build the `DATA` array. A group is `[section, subcategory, rows]`; a row is `[entry, url, downloads, kind, verdict, reason]`, plus an optional seventh `source` field. + - `source` says where a row came from when that is not obvious, such as the starred repo behind a scan candidate or an incumbent pulled in during the audit. The page shows a Source column only when a row has one. + - `subcategory` may carry a note after ` — ` (rendered muted): use it for proposed splits, re-homes, or anything the maintainer should weigh for the whole group. + - `downloads` is PyPI last-month as a comma-formatted string; use `—` when no signal exists (e.g. agent skill packs), `stdlib` for standard-library modules, `fetch failed` when the lookup failed. State the fetch date in the sub-header. + - `kind` sets the buttons independently of the seeded `verdict`: + - `listed`: `keep`, `remove` (taken out of README.md), or `park`. + - `candidate`: `add`, `reject` (not added, logged under "Rejected Additions"), or `park`. + - `structure`: `restructure` (take the change the reason or group note describes), `park`, or `reject` (logged under "Rejected Additions"). + - Seed `verdict` from the current adjudication or dry-run. `park` defers a decision for a later review; preserve the row's `kind` so its original choices remain available. + - Show a replacement as two rows: the old entry `listed` / `remove`, the new entry `candidate` / `add`. + - `reason` is plain language the maintainer reads cold — no invented shorthand. When fresh evidence contradicts the seeded verdict (a big download count on a remove or reject, a dead repo on a keep), say so in that row's reason instead of silently changing the seed. +2. Copy `template.html` (sibling of this file) and replace the placeholders: `__TITLE__` (page title), `__SUB__` (sub-header: scope, seed provenance, fetch date, and the standing instruction to flip/comment then Copy feedback), `__KEY__` (localStorage key), `__DATA__` (the array). `__KEY__` must be unique per review — slug plus date, e.g. `awesome-python-science-2026-09-01` — because saved state under a reused key bleeds a previous review's flips into rows with the same section and entry name. +3. Write the page to `local/awesome-python--preview.html` in the repo root (git-ignored; create the directory if needed), `open` it, and tell the maintainer the path and the return path: flip or comment rows (they highlight yellow), press **Copy feedback**, paste the JSON into the chat. Done when the page is open and the return path is stated. + +## Process the pasted feedback + +Each JSON row is `{section, subcategory, entry, kind, my_verdict, your_verdict, reason}`, with kinds and verdicts as defined above. The maintainer's verdict is final — apply it, never re-argue it. An empty reason means the verdict stands unexplained; that is enough. + +Before executing, surface anything the flips imply that the maintainer has not decided: a use case pushed past its cap, an entry left homeless by a proposed split, a request that is already satisfied (a no-op). Ask, then execute on their go. Done when every pasted row is either applied or surfaced back — none silently dropped. diff --git a/.claude/skills/preview-verdicts/template.html b/.claude/skills/preview-verdicts/template.html new file mode 100644 index 0000000000..d71b01f74d --- /dev/null +++ b/.claude/skills/preview-verdicts/template.html @@ -0,0 +1,171 @@ + + + + + +__TITLE__ + + + +
+

__TITLE__

+

__SUB__

+
+ +
+
+ + + Copies only rows you changed or commented, as JSON. +
+ + + diff --git a/.claude/skills/review-prs/SKILL.md b/.claude/skills/review-prs/SKILL.md new file mode 100644 index 0000000000..70bb8ba3fc --- /dev/null +++ b/.claude/skills/review-prs/SKILL.md @@ -0,0 +1,35 @@ +--- +name: review-prs +description: Triage open PRs — screen from the diff, delegate the admission judgment to audit-the-list, then merge or close on GitHub. Fire when the maintainer asks to review PRs, process the PR queue, or judge whether a specific PR should be merged. +argument-hint: [PR numbers] +--- + +# Review PRs + +A PR review turns the open queue into terminal states: merged, closed, or explicitly parked. The rules live in CONTRIBUTING.md (Quality Requirements, Admission, Review Process, Automatic Rejection) — this skill is the workflow that applies them, not a second copy. Every PR named in chat is a PR link, `[#](https://github.com/vinta/awesome-python/pull/)`, so the maintainer can open it from the terminal. + +## 1. Fetch + +Arguments name specific PRs; otherwise take the queue: `gh pr list --repo vinta/awesome-python --limit 10 --json number,title,author,url,body,files,mergeable,mergeStateStatus`. Fetch all diffs in parallel: `gh pr diff --repo vinta/awesome-python`. + +Sort the batch: entry additions continue; anything else (typo fixes, website changes, docs) is out of scope — report it under "needs human" and touch nothing, so it resurfaces every run until a human acts. Done when every PR is sorted and has its diff. + +## 2. Screen + +Apply the Automatic Rejection rules that the diff and PR metadata answer without judgment. The recently-closed-duplicate rule needs a lookup: `gh pr list --repo vinta/awesome-python --state closed --search "" --limit 10`; a project under `docs/audit-logs.md` "Rejected Additions" also counts as previously rejected unless the PR brings evidence newer than its line. A screened-out PR goes straight to step 4 as a close, with the rule it broke as its reason. + +Resolve each survivor's target use case against the current README — a diff's context lines show the base the PR was written on, which may have changed since. A merge conflict rides to the Merge arm, which absorbs it. Done when every surviving PR names its target use case. + +## 3. Judge + +Group survivors by target use case — PRs proposing entries for the same use case compete for the same slots, so they ride one invocation. Per group, invoke the `audit-the-list` skill with arguments in this shape: "Judge proposed entries for the
use case. Evidence and Verdicts steps only; report the verdicts back. No preview page, no README changes, no commits." The verdict for each PR is merge or close, grounded in the fetched evidence; when the use case is at cap, a merge verdict names the entry that leaves. An entry no current use case fits is a third outcome — a structure question: carry it to Act with evidence attached, and the maintainer decides there (mint the subcategory and merge, close, or leave open). Done when every survivor holds a verdict with a reason. + +## 4. Act + +- **Close**: draft each closing comment — it states the reason and links CONTRIBUTING.md. Per batch of up to 4 closing PRs: print each draft in chat, its PR link as a label above a fenced block holding only the comment bytes, then AskUserQuestion over that batch — arms: close with the draft as printed, close without a comment, keep open. The chat print is the review copy; an option description only points at it. Keep a checklist of which verdicts have been asked; answers often arrive as custom text, and that text is the decision — an edited comment replaces the draft. Then `gh pr close --repo vinta/awesome-python --comment ""`, or a plain close. +- **Merge**: a clean PR merges with `gh pr merge --repo vinta/awesome-python --merge`. A conflicted one merges locally: `git fetch origin pull//head`, `git merge FETCH_HEAD` with the standard `Merge pull request # from /` message, resolving the conflict by placing the entry correctly — GitHub still marks the PR merged and the contributor keeps credit. Either way, reconcile the section per CONTRIBUTING before pushing: remove the entry the verdict displaced, fix the new entry's display name and Entry Ordering position, `make test`, commit. An add-only diff that displaces is the normal case — the removal is this step's job, not the contributor's. +Done when every verdict has been adjudicated by the maintainer and its action executed, with every posted comment byte-identical to a printed draft or to the custom text the maintainer supplied. A kept-open PR resurfaces next run — that is its point. + +## 5. Report + +Summary table: PR link, verdict, action taken, plus the needs-human list. Done when every fetched PR ends in exactly one state — merged with its section reconciled, closed, or left open (kept open, needs-human, or structure question pending). diff --git a/.claude/skills/scan-github-stars/SKILL.md b/.claude/skills/scan-github-stars/SKILL.md new file mode 100644 index 0000000000..5b45b05943 --- /dev/null +++ b/.claude/skills/scan-github-stars/SKILL.md @@ -0,0 +1,35 @@ +--- +name: scan-github-stars +description: Scan the maintainer's GitHub stars for README.md addition candidates; incremental by default, `full` rejudges stars judged before. +argument-hint: [incremental | full] +--- + +# Scan GitHub stars + +A star scan turns the maintainer's GitHub stars into proposed additions. Every star the scan sees ends in one place: listed in README.md, screened out in `local/starred-scan.tsv`, rejected under `docs/audit-logs.md` "Rejected Additions", or parked for another review. Judgment belongs to `audit-the-list` and the rules to CONTRIBUTING.md; this skill only sources candidates and records outcomes. + +## 1. Scan + +Run `uv run python .claude/skills/scan-github-stars/scripts/scan_github_stars.py`, adding `--full` when the argument is `full`. With no ledger yet, a plain run already covers every star. Print the drop counts from stderr. Done when the candidate TSV is in hand. + +## 2. Screen + +Judge each candidate from its name, description, topics, and language against CONTRIBUTING Quality Requirement 1 (Serves Python Developers), and name the current use case it would compete in. The screen is loose: drop only what plainly fails, and pass anything uncertain on, since `audit-the-list` is the real judge. In a full scan, read the `prior` column as context and overturn a rejection only on new evidence. + +Append each screen-out to `local/starred-scan.tsv` as it is decided, one `repoyyyy-mm-ddreason` row (create the file with header `repo date reason`), so an interrupted run keeps its work. The file is append-only; the script reads the last row per repo. A survivor no current use case fits is a structure question: carry it to review with its evidence. Done when every candidate is a ledger row, holds a target use case, or is a structure question. + +## 3. Judge + +Group survivors by target use case. Per group, invoke `audit-the-list` with: "Judge proposed entries for the
use case. Evidence and Verdicts steps only; report the verdicts back. No preview page, no README changes, no commits." A reject whose reason names an unlisted project as the better pick turns that project into a candidate too: judge it in the same group and mark its source as found in the audit, not starred. Done when every survivor holds an add or reject verdict with a reason, and each add at cap names the entry it displaces. + +## 4. Review + +Run `preview-verdicts` over all groups: survivors with kind `candidate` and verdict `add` or `reject`, displaced entries with kind `listed` and verdict `remove`, and structure questions with kind `structure` and verdict `park` or `restructure`. Quote any `prior` verdict in the reason. Seed `park` on any row whose verdict rests on a judgment call, except an add where only the tier is in doubt; preserve its kind. Done when the maintainer's feedback is processed and they gave the go. + +## 5. Record + +- **Added**: follow `audit-the-list` step 5 for the README change, one commit per entry. When the entry links a different repo than the star (a Python binding or client of a starred engine), also append a ledger row `already listed as ()` for the star, since the script only recognizes a star whose own repo is linked. +- **Rejected**: a candidate with verdict `reject` gets one line under "Rejected Additions": `- owner/repo -- yyyy-mm-dd, /month (): .`, replacing its earlier line. This exact rejection takes precedence over a parked ledger row. +- **Parked**: leave listed entries in README.md. An unlisted repo with no prior decision needs no record. To reopen one screened out or rejected before, remove its exact "Rejected Additions" line and append `owner/repoyyyy-mm-ddparked` to the ledger. Its earlier screen-out rows stay as history. The marker also exempts this repo from an `owner/*` rejection; leave that organization line in place for its other repos. A later screen-out or exact rejection excludes the repo again. A new organization-wide rejection must explicitly resolve any parked exceptions it is intended to cover. + +Done when every candidate from step 1 is listed in README.md, a ledger row, a Rejected Additions line, or parked by the maintainer. diff --git a/.claude/skills/scan-github-stars/scripts/scan_github_stars.py b/.claude/skills/scan-github-stars/scripts/scan_github_stars.py new file mode 100644 index 0000000000..23e0994190 --- /dev/null +++ b/.claude/skills/scan-github-stars/scripts/scan_github_stars.py @@ -0,0 +1,121 @@ +"""List the maintainer's GitHub stars that could be README.md candidates. + +Drops stars that fail a check needing no judgment: already listed, own repo, +archived, no push in 12 months, or created under a month ago. These drops are +recomputed every run, so a repo that becomes active again resurfaces. An +incremental run also drops stars already judged: model screen-outs in +local/starred-scan.tsv and maintainer rejections under docs/audit-logs.md +"Rejected Additions". `--full` keeps them and shows the earlier verdict in the +`prior` column. Prints survivors as TSV on stdout, drop counts on stderr. +""" + +import argparse +import json +import re +import subprocess +import sys +from collections import Counter +from datetime import UTC, datetime, timedelta +from pathlib import Path +from typing import TypedDict + +ROOT = Path(__file__).resolve().parents[4] +sys.path.insert(0, str(ROOT / "website")) + +from readme_parser import parse_readme + +LEDGER = ROOT / "local" / "starred-scan.tsv" +REGISTER = ROOT / "docs" / "audit-logs.md" +GITHUB_REPO_RE = re.compile(r"^https://github\.com/([^/#?]+/[^/#?]+)") +REJECTED_LINE_RE = re.compile(r"^- (\S+) -- (.+)$") + + +class Owner(TypedDict): + login: str + + +class Star(TypedDict): + full_name: str + owner: Owner + archived: bool + pushed_at: str + created_at: str + language: str | None + stargazers_count: int + topics: list[str] + description: str | None + + +def gh(*args: str) -> str: + return subprocess.run(["gh", *args], capture_output=True, text=True, check=True).stdout + + +def listed_repos() -> set[str]: + repos = set() + for group in parse_readme((ROOT / "README.md").read_text()): + for category in group["categories"]: + for link in [*category["entries"], *(sub for entry in category["entries"] for sub in entry["also_see"])]: + if match := GITHUB_REPO_RE.match(link["url"]): + repos.add(match.group(1).removesuffix(".git").lower()) + return repos + + +def prior_verdicts() -> dict[str, str | None]: + """The last ledger row wins; `parked` reopens a repo. An exact registered rejection still wins.""" + prior: dict[str, str | None] = {} + if LEDGER.exists(): + for line in LEDGER.read_text().splitlines()[1:]: + repo, date, reason = line.split("\t", 2) + if reason == "parked": + prior[repo.lower()] = None + else: + prior[repo.lower()] = f"screened out {date}: {reason}" + register = REGISTER.read_text().splitlines() + start = register.index("## Rejected Additions") + for line in register[start + 1 :]: + if line.startswith("## "): + break + if match := REJECTED_LINE_RE.match(line): + prior[match.group(1).lower()] = f"rejected {match.group(2)}" + return prior + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--full", action="store_true", help="rejudge stars already screened out or rejected") + args = parser.parse_args() + + now = datetime.now(UTC) + login = gh("api", "user", "--jq", ".login").strip().lower() + listed = listed_repos() + prior = prior_verdicts() + + stars: list[Star] = [json.loads(line) for line in gh("api", "--paginate", "user/starred?per_page=100", "--jq", ".[]").splitlines()] + dropped = Counter() + print("repo\tlanguage\tstars\tpushed\ttopics\tdescription\tprior") + for star in stars: + repo = star["full_name"].lower() + verdict = prior[repo] if repo in prior else prior.get(f"{star['owner']['login'].lower()}/*") + if repo in listed: + dropped["already listed"] += 1 + elif star["owner"]["login"].lower() == login: + dropped["own repo"] += 1 + elif star["archived"]: + dropped["archived"] += 1 + elif datetime.fromisoformat(star["pushed_at"]) < now - timedelta(days=365): + dropped["no push in 12 months"] += 1 + elif datetime.fromisoformat(star["created_at"]) > now - timedelta(days=30): + dropped["created under a month ago"] += 1 + elif verdict and not args.full: + dropped["judged before"] += 1 + else: + description = " ".join((star["description"] or "").split()) + print("\t".join([star["full_name"], star["language"] or "-", str(star["stargazers_count"]), star["pushed_at"][:10], ",".join(star["topics"]), description, verdict or ""])) + + print(f"{len(stars)} stars; {len(stars) - dropped.total()} candidates", file=sys.stderr) + for reason, count in dropped.most_common(): + print(f" dropped {count}: {reason}", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 0000000000..7d0bb19f22 --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1,2 @@ +open_collective: awesome-python +custom: https://awesome-python.com/sponsorship/ diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 8feada4e8a..41890f4696 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,11 +1,26 @@ -## What is this Python project? +## Project -Describe features. +[project-name](url) -## What's the difference between this Python project and similar ones? +## Checklist -Enumerate comparisons. +- [ ] I read [CONTRIBUTING.md](https://github.com/vinta/awesome-python/blob/master/CONTRIBUTING.md) - awesome-python is a shortlist, not a catalog +- [ ] One project per PR +- [ ] PR title format: `Add project-name` +- [ ] Entry format: `- [pypi-name](https://github.com/owner/repo) - Description ending with period.` +- [ ] Display name is the PyPI package name +- [ ] Placed in an existing use case (new sections and subcategories are maintainer-only) +- [ ] Meets all Quality Requirements: active, stable, documented, at least 1 month old --- +## Which Tier -Anyone who agrees with this pull request could vote for it by adding a :+1: to it, and usually, the maintainer will merge it when votes reach **20**. +Pick one: + +- [ ] **Obvious choice** - a tool an experienced Python developer would name when asked "what do I use for this?" +- [ ] **Challenger** - not yet the obvious choice, but a credible successor to one. Give adoption-trajectory evidence, not popularity alone. + +Explain: + +## Displacement + +If the use case is at its cap, name the entry your project replaces and argue that yours does that entry's job better. One in, one out. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000000..fc8084bcb5 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,33 @@ +name: CI + +on: + push: + branches: [master] + pull_request: + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + + - name: Install uv + uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 + with: + enable-cache: true + + - name: Install dependencies + run: uv sync --group build --locked + + - name: Run tests + run: make test + + - name: Build website + run: make build diff --git a/.github/workflows/deploy-website.yml b/.github/workflows/deploy-website.yml new file mode 100644 index 0000000000..4f5f28c764 --- /dev/null +++ b/.github/workflows/deploy-website.yml @@ -0,0 +1,105 @@ +name: Deploy Website + +on: + push: + branches: [master] + schedule: + - cron: "0 0 * * *" + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: pages + cancel-in-progress: false + +jobs: + deploy: + if: github.repository == 'vinta/awesome-python' + runs-on: ubuntu-latest + environment: + name: github-pages + url: https://awesome-python.com/ + steps: + - uses: actions/checkout@v7 + with: + # build.py reads each page's last change from git history for the sitemap's lastmod + fetch-depth: 0 + + - name: Install uv + uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 + with: + enable-cache: true + + - name: Install dependencies + run: uv sync --group build --locked + + - name: Run tests + if: github.event_name == 'schedule' + run: make test + + - name: Get date + id: date + run: echo "today=$(date -u +%Y-%m-%d)" >> "$GITHUB_OUTPUT" + + - name: Restore star data cache + id: cache-stars + uses: actions/cache/restore@v6 + with: + path: website/data/github_stars.json + key: github-stars-${{ steps.date.outputs.today }} + restore-keys: github-stars- + + - name: Fetch GitHub stars + id: fetch-stars + continue-on-error: true + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: make fetch_github_stars + + - name: Save star data cache + if: steps.fetch-stars.outcome == 'success' + uses: actions/cache/save@v6 + with: + path: website/data/github_stars.json + key: github-stars-${{ steps.date.outputs.today }} + + - name: Verify star data exists + run: | + if [ ! -f website/data/github_stars.json ]; then + echo "::error::github_stars.json not found. No cache and fetch failed or was skipped." + exit 1 + fi + python -m json.tool website/data/github_stars.json > /dev/null + + - name: Restore PyPI download data cache + uses: actions/cache/restore@v6 + with: + path: website/data/pypi_downloads.tsv + key: pypi-downloads-${{ steps.date.outputs.today }} + restore-keys: pypi-downloads- + + - name: Fetch PyPI downloads + id: fetch-downloads + continue-on-error: true + run: make fetch_pypi_downloads + + - name: Save PyPI download data cache + if: steps.fetch-downloads.outcome == 'success' + uses: actions/cache/save@v6 + with: + path: website/data/pypi_downloads.tsv + key: pypi-downloads-${{ steps.date.outputs.today }} + + - name: Build website + run: make build + + - name: Upload artifact + uses: actions/upload-pages-artifact@v5 + with: + path: website/output/ + + - name: Deploy to GitHub Pages + uses: actions/deploy-pages@v5 diff --git a/.gitignore b/.gitignore index 4046e40f0f..0aa12a638f 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,21 @@ +# secrets +.env* + +# macOS .DS_Store +# python +.venv/ +__pycache__/ *.py[co] -docs/index.md -site/ +# website +website/output/ +website/data/* +!website/data/pypi_name_overrides.json +!website/data/redirects.json +!website/data/category_intros/ + +# agents +.playwright-cli/ +local/ diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index 44c97353bd..0000000000 --- a/.travis.yml +++ /dev/null @@ -1,10 +0,0 @@ -language: ruby - -rvm: - - 2.2 - -before_script: - - gem install awesome_bot - -script: - - awesome_bot README.md --allow-dupe --white-list pyparsing,graphviz.org diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000000..01e007a952 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,23 @@ +# AGENTS.md + +An opinionated guide to the best Python frameworks, libraries, and tools. + +[README.md](README.md) is the single source of content truth; `website/` renders it into the static site: [awesome-python.com](https://awesome-python.com/). + +## Entry Rules + +[CONTRIBUTING.md](CONTRIBUTING.md) holds the admission rules, quality requirements, rejection rules, entry format, and ordering. Apply it whenever adding or removing an entry — direct commits included, not only PR reviews. + +- Every keep/drop reason must be verified against current online data at decision time — download counts, repo activity and archived status, PyPI metadata, project docs. Judging tiers — obvious choice vs challenger — also requires WebSearch evidence (adoption trajectory, community sentiment), not download counts alone. Training-data recollections are not evidence; label anything unverifiable as a judgment call. +- Checks stay read-only: judge whether a listed project installs or works from its PyPI metadata (`requires_python`, classifiers, wheel tags) and issue tracker, never by installing or running it, since that runs untrusted code. +- A download count is not automatically independent demand. When one listed entry depends on another, check `requires_dist` on PyPI before citing the depended-on entry's count: mkdocs-material hard-depends on mkdocs, so mkdocs' figure exceeded mkdocs-material's by only about one percent and nearly all of it was mkdocs-material pulling it in. +- One entry per commit when adding or deleting entries. Exceptions: a prune sweep is one commit per section, its body listing each removal with its reason; format, wording, or categorization changes may be bundled. Cross-section re-homes ride the originating audit's commit (both sides of the move in one diff). +- Resources sections are not project entries: out of audit scope, and the website never parses them. +- Sponsor placement never influences which projects get listed — see [SPONSORSHIP.md](SPONSORSHIP.md). + +## Website + +- Model the layout on https://www.placestoread.xyz: the whole list on one page, dense rows, a row that expands inline with its content aligned to the Name column, sorting by column headers, full-text search, tags that filter, a divider under the header row instead of a strong top border, a solid dark footer, and minimal decoration. Keep that model (no card grid, modal details, or pagination) unless the maintainer asks, and keep green out of the palette. +- `.section-shell` (`--shell-max: 84rem`) is the only width cap: sections, tables, rows, CTAs, and paragraphs stay uncapped, since the maintainer reads on wide screens and prose-width advice like 65-75ch doesn't apply. Remove inner `max-width` rules you come across, and ask with a concrete reason before adding one. +- Pick font sizes one step larger than feels right (`--text-base` over `--text-sm`, 1.75rem over 1.5rem for a heading), and shrink an existing size only when the maintainer asks: they asked for bigger type 11 times across 8 sessions and never for smaller. +- When you style one link, tag, or label, give its peers the same style in the same change: hero, nav, footer, project, and sponsor links share hover styles, and tag variants build on `.tag`. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000000..22541c01fc --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,23 @@ +# CLAUDE.md + +An opinionated guide to the best Python frameworks, libraries, and tools. + +[README.md](README.md) is the single source of content truth; `website/` renders it into the static site: [awesome-python.com](https://awesome-python.com/). + +## Entry Rules + +[CONTRIBUTING.md](CONTRIBUTING.md) holds the admission rules, quality requirements, rejection rules, entry format, and ordering. Apply it whenever adding or removing an entry — direct commits included, not only PR reviews. + +- Every keep/drop reason must be verified against current online data at decision time — download counts, repo activity and archived status, PyPI metadata, project docs. Judging tiers — obvious choice vs challenger — also requires WebSearch evidence (adoption trajectory, community sentiment), not download counts alone. Training-data recollections are not evidence; label anything unverifiable as a judgment call. +- Checks stay read-only: judge whether a listed project installs or works from its PyPI metadata (`requires_python`, classifiers, wheel tags) and issue tracker, never by installing or running it, since that runs untrusted code. +- A download count is not automatically independent demand. When one listed entry depends on another, check `requires_dist` on PyPI before citing the depended-on entry's count: mkdocs-material hard-depends on mkdocs, so mkdocs' figure exceeded mkdocs-material's by only about one percent and nearly all of it was mkdocs-material pulling it in. +- One entry per commit when adding or deleting entries. Exceptions: a prune sweep is one commit per section, its body listing each removal with its reason; format, wording, or categorization changes may be bundled. Cross-section re-homes ride the originating audit's commit (both sides of the move in one diff). +- Resources sections are not project entries: out of audit scope, and the website never parses them. +- Sponsor placement never influences which projects get listed — see [SPONSORSHIP.md](SPONSORSHIP.md). + +## Website + +- Model the layout on https://www.placestoread.xyz: the whole list on one page, dense rows, a row that expands inline with its content aligned to the Name column, sorting by column headers, full-text search, tags that filter, a divider under the header row instead of a strong top border, a solid dark footer, and minimal decoration. Keep that model (no card grid, modal details, or pagination) unless the maintainer asks, and keep green out of the palette. +- `.section-shell` (`--shell-max: 84rem`) is the only width cap: sections, tables, rows, CTAs, and paragraphs stay uncapped, since the maintainer reads on wide screens and prose-width advice like 65-75ch doesn't apply. Remove inner `max-width` rules you come across, and ask with a concrete reason before adding one. +- Pick font sizes one step larger than feels right (`--text-base` over `--text-sm`, 1.75rem over 1.5rem for a heading), and shrink an existing size only when the maintainer asks: they asked for bigger type 11 times across 8 sessions and never for smaller. +- When you style one link, tag, or label, give its peers the same style in the same change: hero, nav, footer, project, and sponsor links share hover styles, and tag variants build on `.tag`. diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000000..f13d736d08 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,40 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity and +orientation. + +## Our Standards + +Examples of behavior that contributes to a positive environment: + +- Using welcoming and inclusive language +- Being respectful of differing viewpoints and experiences +- Gracefully accepting constructive criticism +- Focusing on what is best for the community +- Showing empathy towards other community members + +Examples of unacceptable behavior: + +- The use of sexualized language or imagery +- Trolling, insulting or derogatory comments +- Public or private harassment +- Publishing others' private information without permission +- Other conduct which could reasonably be considered inappropriate + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the project maintainers. All complaints will be reviewed and +investigated and will result in a response that is deemed necessary and +appropriate to the circumstances. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org), +version 2.0. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bcb24bfa63..e9fe3a82ba 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,17 +1,132 @@ # Contributing -Your contributions are always welcome! - -## Guidelines - -* Add one link per Pull Request. -* Add the link: `* [project-name](http://example.com/) - A short description ends with a period.` - * Keep descriptions concise. -* Add a section if needed. - * Add the section description. - * Add the section title to Table of Contents. -* Search previous suggestions before making a new one, as yours may be a duplicate. -* Don't mention `Python` in the description as it's implied. -* Check your spelling and grammar. -* Remove any trailing whitespace. -* Send a Pull Request with the reason why the library is awesome. +awesome-python is a shortlist, not a catalog. Each use case lists only its obvious choices, and most rejections mean "the use case is full", not "your project is bad". Read this whole page before opening a PR. + +## Quality Requirements + +All submissions must satisfy **ALL** of these: + +1. **Serves Python Developers**: Python developers use it in their Python work. Implementation language and packaging are irrelevant — uv and ty are written in Rust, and agent skill packs are markdown, yet all belong; a pure-Python project nobody uses in Python work does not. +2. **Active**: Commits within the last 12 months +3. **Stable**: Production-ready, not alpha/beta/experimental. A PyPI "Development Status" classifier alone does not decide this; judge by releases, documentation, and production use. +4. **Documented**: Clear README with examples and use cases +5. **Established**: Repository at least 1 month old + +## Admission + +A **use case** is one distinct job a reader needs done. Use cases are defined by the list's structure: each subcategory is a use case, and a section without subcategories is a single use case. Structure changes — new sections, new subcategories, splitting an oversized use case into finer ones — are made only by the maintainer; an entry PR can never create the subcategory it needs. + +Each use case lists at most: + +- **Up to 3 obvious choices** — tools an experienced Python developer would name when asked "what do I use for this?" +- **Up to 2 challengers** — tools that are not yet the obvious choice but are credible successors to one. Admission as a challenger requires adoption-trajectory evidence, not popularity alone. + +Hard maximum: 5 entries per use case. This is a qualitative bar first and a numeric backstop second — most use cases should carry fewer. + +**Overrides**: the maintainer may exceed any limit on this page — the caps, the activity requirement, the stability requirement — for a specific entry or use case by explicit decision. An override is case-by-case; it does not loosen these rules for submissions, and citing one in a PR carries no weight. + +**Displacement**: once a use case is at its cap, the only way in is to name the entry your project replaces and argue that yours does that entry's job better. One in, one out. + +**Dual-listing**: a tool may hold entries in multiple use cases, but only when it is an obvious choice in each. List the full entry in each home with identical lines; never a "see X above" note, since the website only renders list items. Description edits update every copy in the same commit. Each slot is audited on its own: dropping one home keeps the other, and dropping the tool entirely removes all copies in one commit. Dual-listing is a maintainer decision; a PR adding a second home for an existing entry is treated as a duplicate. + +**Standard library**: a standard-library module is listed only where the stdlib is itself the obvious choice for the use case (tomllib yes, unittest no). + +**Python versions**: missing support for the newest Python releases is not by itself grounds for removal while the entry still has a large user base; it can place the entry in the Second Tier instead. + +**Second Tier**: a challenger slot held by a demoted former obvious choice instead of a rising successor. It counts against the two challenger slots, and the adoption-trajectory bar gates only new admissions, not demotions. + +**Evidence**: admission is decided by maintainer editorial judgment, informed primarily by PyPI download counts rather than GitHub stars. Judgment overrides the signal's known failure modes (CI-inflated counts, model releases consumed as weights rather than pip installs, projects used outside pip such as SDK downloads (renpy) or deployed services (thumbor), large-but-specific audiences misread as "niche"). The maintainer's decision is final. + +Looking for an exhaustive catalog instead? Follow the awesome-\* lists linked under individual entries (for example awesome-python-testing) — they exist precisely so this list doesn't have to be one. + +## Entry Format Reference + +**Use GitHub repository URLs** whenever possible. [awesome-python.com](https://awesome-python.com/) shows stars and last-commit dates only for GitHub repos, and ranks entries without PyPI downloads by stars. + +### Naming Convention + +Use the **PyPI package name** as the display name so developers can copy it directly to `pip install`. Check the canonical name at `https://pypi.org/pypi/{package}/json`. If the project is not on PyPI, use the GitHub repository name instead. + +### Standard Entry + +```markdown +- [pypi-name](https://github.com/owner/repo) - Description ending with period. +``` + +### Standard Library Module + +```markdown +- [module](https://docs.python.org/3/library/module.html) - (Python standard library) Description. +``` + +### Bundled Feature + +A feature that ships inside a larger library, framework, or tool rather than as its own PyPI package. The `(part of X)` prefix is parsed by the website, which renders a "Bundled" badge in place of a download count, so write it exactly in that form and name the parent as developers say it. + +```markdown +- [name](url) - (part of Parent) Description ending with period. +``` + +### Fork of Another Project + +```markdown +- [new-name](https://github.com/owner/new-name) - Description ([original-name](original-url) fork). +``` + +### Entry with Related Awesome List + +```markdown +- [project](https://github.com/owner/project) - Description. + - [awesome-project](https://github.com/someone/awesome-project) +``` + +Only awesome-\* lists go here, placed by the maintainer. A sub-item holds no slot, does not count against the cap, and moves or goes away with its parent. A companion project either earns its own entry or is not listed. + +### Subcategory Format + +```markdown +- Subcategory Name + - [project](url) - Description. +``` + +### Entry Ordering + +Within a use case, the obvious choices are listed first, ordered by PyPI downloads per month from high to low; challengers follow, in the same order. Standard-library modules sort first in the use case — built-ins lead, before everything else — alphabetically when there are several. (This never conflicts with tier order: a standard-library module is only listed where it is itself the obvious choice, so it always belongs to the first tier.) Other entries without a download signal (agent skill packs, projects distributed outside PyPI) sort last within their tier, alphabetically. There is no marker in the entry text — position is the marker, so the last entries of a use case may be its challengers. + +## Changing the Structure + +Adding sections or subcategories is maintainer-only (see Admission). For maintainer reference: + +1. Add the section description in italics: `*Libraries for doing X.*` +2. Add the section under the appropriate thematic group (e.g., **AI & ML**, **Web**, **Data & Science**) +3. Add the section title to the Table of Contents under its group +4. Order entries per Entry Ordering above + +A subcategory never repeats its section's name: one named like its section competes with the section page in search results. Name the section's catch-all or main use case General instead. + +## Review Process + +PRs are reviewed by automated tools and maintainers: + +1. **Format Check**: Entry follows the correct format +2. **Category Check**: Placed in the appropriate use case +3. **Duplicate Check**: Not already listed or previously rejected +4. **Activity Check**: Project shows recent activity +5. **Admission Check**: Meets the Admission rules above, including Displacement when the use case is at its cap + +Search previous Pull Requests and Issues before submitting, as yours may be a duplicate. + +## Automatic Rejection + +PRs will be **closed** if: + +- Adding multiple projects in one PR +- The use case is at its cap and the PR makes no Displacement argument +- The PR creates a new section or subcategory and fills it (structure changes are maintainer-only) +- Coordinated multi-entry self-promotion: multiple related projects from the same organization or author, across one or several PRs +- Duplicate of existing entry or recently-closed PR +- Empty or placeholder PR descriptions +- Placed under an inappropriate category +- Project is archived or abandoned (no commits in 12+ months) +- No documentation or unclear use case +- Repository less than 1 month old diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000000..63b4934840 --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,226 @@ +--- +version: alpha +name: awesome-python.com +description: Warm editorial Python index. Light cream canvas, brown-red interactive accent, Cormorant Garamond plus Manrope, table-driven single-page reference. +--- + +# awesome-python.com DESIGN.md + +awesome-python.com is a searchable, filterable index of ~500 curated Python projects. It is a reference tool, not a landing page and not a GitHub README mirror. + +This file follows the [Google Stitch DESIGN.md format](https://stitch.withgoogle.com/docs/design-md/overview/). The source of truth for token values lives in `website/static/style.css`. Color tokens here are written in OKLCH because the project mandates OKLCH over hex, which is a deliberate divergence from the spec's hex-only token requirement. + +## Overview + +Three words: **opinionated, confident, dense**. + +Working Python developers (mid to senior) are the target reader. They write Python daily and arrive with a specific question in mind: "what's a good HTTP client these days", "is there still a maintained ORM for X", "what are people using for task queues now". Secondary readers: polyglot developers evaluating Python's ecosystem, and curious browsers. + +Jobs to be done: + +1. Find a library for a specific need fast (search + tag filter). +2. Compare candidates at a glance (downloads, stars, last commit, tags, one-line description). +3. Confirm a project is alive before clicking through. + +These users skim. They reward density and terse copy. They penalize marketing fluff. + +Voice: + +- Editorial. Every word earns its place. +- Confident, not combative. "This is the list" energy, not "check out these cool projects". +- No hype. The content is what's interesting. +- Calm authority. Closer to a well-edited technical reference (O'Reilly index, The Economist briefing, a good man page) than a blog or product site. + +Emotional goals: trust, efficiency, craft. The reader should feel the list was edited by someone with taste, find what they need in seconds, and notice the typographic care as a signal that the curation is careful too. + +Reference points (stay close to these): + +- **https://www.placestoread.xyz** is the primary visual model for the table, expand row, sorting, and footer. "Like placestoread" means dense single-page list, inline click-to-expand rows that indent under the Name column, sortable headers, minimum decoration. When in doubt about a table or row treatment, check placestoread first. +- Magazine reference pages (The Economist, FT Weekend, Monocle). +- Field-guide books. Curated, functional, hand-made. +- Library card catalogs. Dense tabular information, excellent typography, no decoration for decoration's sake. + +Anti-references (avoid strictly): + +- Generic dark developer-tool look. No cyan on near-black, neon gradients, VSCode-palette dashboards, terminal-green monospace branding. +- Other awesome-\* sites. No plain README dumps, bare lists of links, no voice. +- SaaS marketing pages. No big metric counters, testimonial cards, feature grids, pricing tiers, or "join 10,000+ developers" social proof bands. + +Design principles: + +1. **The list is the hero.** Hero, sponsor band, and CTA exist, but they must not compete with the table for attention. +2. **Density is a feature.** Prefer tables and tight rhythm over giant cards with one fact each. Mid-senior developers want to see more at once. +3. **Editorial typography over decoration.** Visual interest comes from the serif/sans pairing, type hierarchy, and whitespace. Not from gradients, shadows, badges, or icon boxes above headings. +4. **Warm, not cool.** Neutrals tint toward warm hues (roughly 55 to 80 in OKLCH). Pure grays and cool blues do not belong. +5. **One point of view.** No dark mode, no theme picker, no alternate palettes. Consistency signals curation. + +## Colors + +Warm editorial palette. Light theme only (`color-scheme: light`). OKLCH only. + +Each token below shows the OKLCH value (canonical, lives in `style.css`) followed by an approximate hex sRGB equivalent for spec linters and any tool that expects hex. + +Surfaces: + +- `--bg-page` `oklch(96.8% 0.018 80)` ≈ `#FBF3E7`. Cream/ivory canvas, the body floor. The body uses a vertical gradient between `--bg-page-top` `oklch(95.2% 0.018 78)` ≈ `#F7EFE3`, `--bg-page` at 24rem, and `--bg-page-end` `oklch(98.4% 0.01 80)` ≈ `#FCF8F0`, with a soft radial highlight in the top-left corner. +- `--bg-paper` `oklch(98.6% 0.01 80)` ≈ `#FEFAF3`. Warm white for the content shell. +- `--bg-paper-strong` `oklch(95.7% 0.016 76)` ≈ `#F7F0E5`. Tinted paper for sponsor band, CTA backgrounds, static decoration. +- `--hero-bg-start` `oklch(14% 0.03 32)` ≈ `#130503` through `--hero-bg-mid` `oklch(19% 0.035 35)` ≈ `#22120B` to `--hero-bg-end` `oklch(28% 0.05 42)` ≈ `#3D2014`. Dark earthy hero gradient. +- `--footer-bg` `oklch(16% 0.025 35)` ≈ `#170906`. Dark warm charcoal footer, part of the same system. + +Ink: + +- `--ink` `oklch(22% 0.02 55)` ≈ `#221812`. Body text. +- `--ink-soft` `oklch(38% 0.018 55)` ≈ `#4A4039`. Secondary copy. +- `--ink-muted` `oklch(52% 0.02 55)` ≈ `#72665E`. Meta rows, captions, static labels. +- `--line` / `--line-strong`. Hairlines and dividers. + +Accent (warm brown-red, reserved for interactive): + +- `--accent` `oklch(58% 0.16 45)` ≈ `#C4530F`. Primary accent. +- `--accent-deep` `oklch(44% 0.15 42)` ≈ `#922900`. Link text, hover. +- `--accent-soft` `oklch(92% 0.045 55)` ≈ `#FDDDC9`. Tinted background for filter tags. +- `--accent-underline` `oklch(58% 0.16 45 / 0.4)` ≈ `#C4530F66`. Subtle text-decoration-color. + +Rules: + +- Use OKLCH for any new color. Not HSL, not hex. +- Accent tokens (`--accent`, `--accent-deep`, `--accent-soft`) are reserved for interactive elements. Clickable filter tags (`.tag`) correctly use `--accent-soft` background with `--accent-deep` text. Interactive link states (`.col-name > a:hover`, `.sponsor-link:hover`, `.hero-action-primary`, `.back-to-top`, CTAs) use accent tokens. +- Non-interactive elements (inline code, `.source-badge`, static labels, decorative pills) must use ink tokens (`--ink`, `--ink-soft`, `--ink-muted`) on `--bg-paper-strong` or `--bg-paper`, never the accent. `.source-badge` uses `--ink-soft`; `.sponsorship-body code` uses `--ink`. Users should not mistake static decoration for something clickable. +- Same role uses the same token everywhere. No one-off inline `color: oklch(...)` buried in a rule. + +Aversions: + +- **No green.** The user rejected it when picking the palette. Warm brown-red, ivory, and dark earthy tones are the established system. Do not introduce green even for success states or ancillary accents. +- No cyan, no neon gradients, no pure grays, no cool blues. + +## Typography + +Pairing (do not swap): + +- **Display**: `Cormorant Garamond` (serif, 600 only). +- **Body**: `Manrope` (sans, 400 / 600 / 700 / 800). + +Scale: + +| Role | Token | Size | Family | Weight | Notes | +| ---------------- | ----------------- | ----------------------------- | ------------------ | --------- | ---------------------------------------------- | +| Hero headline | (literal `clamp`) | `clamp(4.5rem, 11vw, 8.5rem)` | Cormorant Garamond | 600 | Magazine-cover scale, single use on the hero | +| Body large | `--text-lg` | `1.125rem` | Manrope | 400 | Content-heavy passages | +| Body | `--text-base` | `1rem` (16px) | Manrope | 400 | Body floor, do not go smaller | +| Meta / secondary | `--text-sm` | `0.95rem` | Manrope | 400 / 600 | Meta rows, secondary copy | +| Caption / pill | `--text-xs` | `0.8rem` (12.8px) | Manrope | 600 / 700 | Smallest token, pills, badges, tags, footnotes | + +Hard-won sizing rules (do not relax): + +- **Body type floor is 16px.** Do not go smaller. +- **Absolute minimum font size is 12px (`0.75rem`) for ANY text**, including pills, badges, tags, captions, footnotes. Anything smaller hits Chrome's default minimum-font-size floor and renders inconsistently across browsers and user accessibility settings. Use `var(--text-xs)` (`0.8rem`) as the smallest token in code. +- **When in doubt, pick one step larger** than what generic scale references suggest. The user has corrected sizes upward 11+ times across 8 sessions. Footer, meta rows, expand content, labels, and headings all trend too small by default. **Never reduce an existing size unprompted.** +- Adjacent heading levels differ by at least 0.25rem of rendered size. +- Row numbers in the table: left-align, no leading zeros. Zero-padding was tried and rejected. +- **Never `text-transform`.** Write the casing in the markup. + +## Layout + +- **Single width cap: `--shell-max: 84rem` (~1344px) applied via `.section-shell`.** This is the ONLY width cap in the project. Widescreen monitors are the default viewing context. +- **Do NOT add `max-width`** to sections, cards, table cells, table rows, expanded rows, CTA backgrounds, sponsor descriptions, hero subcopy, paragraphs, or list items. The user has removed narrow inner caps repeatedly (`56ch`, `65-75ch`, etc.). Default is no inner cap. +- The "cap line length at ~65-75ch" rule does NOT apply here. Ignore it. Readability at wide widths is carried by vertical rhythm, leading, and the modular type scale instead. +- If a width cap is genuinely necessary for a specific element, ask first with a concrete reason before adding it. +- Shell padding: `--shell-pad: clamp(1.25rem, 3vw, 2.5rem)`. Symmetric gutters: logo left-gap equals logo right-gap, column paddings match across header and body. +- `gap` over child margins in flex and grid. +- Logical properties (`margin-inline`, `padding-block`) over physical (`margin-left`, `padding-top`). +- `rem` for spacing and type. `px` only for borders and shadows. +- CSS custom properties for all colors and repeated values. +- Sibling components (card lists, grid items) share identical spacing. +- Use flexbox or grid for layout. Avoid floats and absolute positioning except for genuine overlay cases (focus rings, sticky headers). +- Never `!important`. Fix specificity instead. + +## Elevation & Depth + +Depth comes from **tonal layers**, not heavy shadows. + +- The page is a quiet warm canvas (`--bg-page`). The content shell is slightly brighter paper (`--bg-paper`). The sponsor band, CTA backgrounds, and inline decorative blocks step up to `--bg-paper-strong`. +- The hero and the category guide band are the only places with real atmosphere: warm gradients on a dark earthy ground (`--hero-bg-start` → `--hero-bg-mid` → `--hero-bg-end`). Only the hero adds the subtle grid and slow sheen. The sheen and any other motion respect `prefers-reduced-motion`. +- The footer is a single tonal block in `--footer-bg`, no internal gradients. +- Two depth treatments are allowed and only these two. The search input combines a 1px inset highlight (`--search-inset`) with a soft warm drop shadow (`--search-shadow`, intensified by `--search-focus-shadow` on focus). The primary CTA button (`.hero-action-primary`) carries a warm drop shadow for press affordance. Both shadows are soft, warm-tinted, and tied to interactive elements. No new drop shadows on cards, panels, rows, or static decoration. +- No glassmorphism as default decoration. +- No bounce or elastic easing. Real objects decelerate smoothly. + +## Shapes + +The shape language is overwhelmingly **pill on small, zero radius on large**. + +- **Pills** (`border-radius: 999px`) for tags, search, sponsor logo chip, source badges, back-to-top, and primary CTA buttons. +- **`0.4rem`** is used in exactly one place: inline `` inside `.sponsorship-body`. Do not introduce a tokenized radius scale. The project does not need one. +- Containers use the page surface itself, not rounded panels. When a panel is needed, prefer pill on small chips and zero radius on large surfaces. +- **No `border-left` or `border-right` greater than 1px as a colored accent stripe** on cards, list items, callouts, or alerts. Use a different structure. + +## Components + +The component vocabulary is small and table-led. Source of truth: `website/static/style.css`. + +- **Table-driven index** (the hero of the page). Sticky header, sortable columns, click-to-expand rows that indent under the Name column. Modeled on placestoread.xyz. Not a card grid. +- **Group rows**. Section, subcategory, and group pages list rows in README order. Section pages add one H2 row per use case; group pages add one per section, linking to that section's page. A note replaces the sort arrow until someone sorts a column, which flattens the list and hides the group rows. +- **Filter tags** (`.tag`). `--accent-soft` background with `--accent-deep` text. Pill shape. Hover swaps to `--highlight` background with `--tag-hover-border` border and ink text. Active state uses the warm `--tag-active-start` → `--tag-active-end` gradient with hero-ink text. Tag variants (`tag-group`, `tag-source`) inherit the base `.tag` style today and differ only at narrow widths (`tag-group` hides under 960px). Add a new variant only when a real visual difference is needed. +- **Hero**. Magazine-cover headline, dark earthy ground, kicker and proof microcopy, primary CTA button using `--hero-btn-start` / `--hero-btn-end`. Subtle grid plus slow sheen. Respects `prefers-reduced-motion`. +- **Jump links**. One per use case in a category hero, plus one for the guide. Underlined like the intro's links, with a trailing ↓. Never tag-styled: tags filter the table or open another page, jump links only scroll. +- **Guide band**. Everything in a category intro after its "How to choose" list. It sits below the table on the hero's dark gradient, with its H2 above the text. +- **Sponsor band**. Sits in the README header on `--bg-paper-strong`. Editorial layout, not a logo wall. Sponsor links share the global accent treatment. +- **CTA**. Warm `--cta-bg`, full-bleed within shell. The button itself uses accent tokens. +- **Footer**. Dark warm charcoal, part of the same system. Footer links share the global hover and focus treatment. +- **Search**. Pill input with `--search-inset` interior and `--search-focus-ring` focus ring. Focus shadow uses `--search-focus-shadow`. +- **Source badge / inline code**. Static decoration on `--bg-paper-strong`. `.source-badge` uses `--ink-soft` text in pill shape; `.sponsorship-body code` uses `--ink` text with the lone `0.4rem` radius. Never the accent. + +Peer-consistency check (run before shipping any visual change): + +- Hover and focus states: if one link type gets a treatment, peer links (hero topbar, footer, project names, sponsor names, expand-meta) share it. +- Tag variants inherit the base `.tag` style. Differ only where a real difference is needed. +- Typography tiers: labels that play the same role share size, weight, and letter-spacing. +- Symmetric gutters: logo left-gap equals logo right-gap, column paddings match across header and body. +- Role-based color tokens: same role uses the same token everywhere. + +## Do's and Don'ts + +- **Do** keep the table the focal point. Hero, sponsor band, and CTA must not compete. +- **Do** use accent tokens only on interactive elements. +- **Do** prefer density over whitespace expansion. +- **Do** check peer elements before shipping a visual change. +- **Do** use OKLCH for every new color. +- **Don't** add inner `max-width` to anything. The shell handles width. +- **Don't** introduce green, cyan, neon, pure gray, or cool blue. +- **Don't** add a dark mode, theme picker, or alternate palette. +- **Don't** use gradient text (`background-clip: text` on gradients). Solid color only. +- **Don't** use `!important`. Fix specificity instead. +- **Don't** use `text-transform`. Write the casing in markup. +- **Don't** use a `border-left` or `border-right` greater than 1px as an accent stripe. +- **Don't** use bounce or elastic easing. +- **Don't** use glassmorphism as default decoration. +- **Don't** mimic generic dark developer-tool sites, other awesome-\* sites, or SaaS marketing pages. + +## Narrow-Screen Behavior + +The user actively tests `< 960px` and `< 680px`. Narrow screens must stay functional. + +- Do not drop features the user might want (sort affordance, filter chips, sticky header where reasonable). Hiding is a last resort and requires justification. +- Always run the `playwright-cli` skill at a narrow viewport after any layout change. + +## Iteration Guide + +Run this audit after generating or modifying a screen. Failure on any item means revise before moving on. + +1. **Width caps.** Inspect every section, card, paragraph, table cell, expanded row, CTA, sponsor description, hero subcopy. Only `.section-shell` (`--shell-max: 84rem`) may cap width. Anything else with a `max-width` is wrong. +2. **Accent reservation.** Grep the changed CSS for `--accent`, `--accent-deep`, `--accent-soft`. Each match must back an interactive element (link, button, focus ring, filter tag). Static decoration must use ink tokens (`--ink`, `--ink-soft`, `--ink-muted`) on `--bg-paper-strong` or `--bg-paper`. +3. **Shape language.** Containers are square or pill. Anything in the 4px-to-16px radius range is suspect. The lone `0.4rem` on `.sponsorship-body code` is the only allowed exception. +4. **Type sizes.** Confirm no rendered text falls below 12px. If a size feels small to a mid-senior reader on a 27-inch display, bump one step up. Never reduce an existing size. +5. **Peer consistency.** Compare against the closest peer element (sibling link type, sibling tag variant, sibling label). Hover, focus, color token, and gutter must match unless there is a stated reason to differ. +6. **Narrow viewport.** Run the `playwright-cli` skill at `< 960px` and `< 680px`. Sort affordance, filter chips, and sticky header must remain functional. + +## Known Gaps + +- **Color format diverges from the Stitch spec.** The official linter requires hex sRGB (`/^#([0-9a-fA-F]{3,8})$/`). The project mandates OKLCH in `style.css`. The Colors section above resolves this by showing both: OKLCH is canonical, hex is the linter-friendly approximation. +- **YAML frontmatter is minimal.** Only `version`, `name`, and `description` are encoded. The project has no JSON / Figma export pipeline that would consume token-level frontmatter, so the prose-led format is preferred for everything else. +- **No formal spacing or radius scale.** The codebase uses `clamp()` and ad-hoc rem values rather than a tokenized scale. Adding one would be invention, not documentation. + +## Verification + +After any frontend change, use the `playwright-cli` skill to visually verify in a real browser. Check layout, responsiveness, and interactive behavior. Do not claim a UI change works based on code alone. diff --git a/Makefile b/Makefile index 0c42fe4f1f..5a07ad9119 100644 --- a/Makefile +++ b/Makefile @@ -1,16 +1,32 @@ -BASEDIR=$(CURDIR) -DOCDIR=$(BASEDIR)/docs +-include .env +export install: - pip install mkdocs + uv sync --locked -link: - ln -sf $(BASEDIR)/README.md $(DOCDIR)/index.md +fetch_github_stars: + uv run python website/fetch_github_stars.py -preview: - $(MAKE) link - mkdocs serve +fetch_pypi_downloads: + uv run python website/fetch_pypi_downloads_via_clickpy.py -deploy: - $(MAKE) link - mkdocs gh-deploy --clean +test: + uv run pytest website/tests/ -v + +lint: + uv run ruff check . + +format: + uv run ruff format . + +typecheck: + uv run ty check website + +build: + uv run python website/build.py + +preview: build + uv run watchfiles \ + 'uv run python website/build.py' \ + README.md website/templates website/static website/data & \ + python -m http.server -b 127.0.0.1 -d website/output/ 8000 diff --git a/README.md b/README.md index cb8479754b..408f65f008 100644 --- a/README.md +++ b/README.md @@ -1,1282 +1,1243 @@ -# Awesome Python [![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome) - -A curated list of awesome Python frameworks, libraries, software and resources. - -Inspired by [awesome-php](https://github.com/ziadoz/awesome-php). - -- [Awesome Python](#awesome-python) - - [Admin Panels](#admin-panels) - - [Algorithms and Design Patterns](#algorithms-and-design-patterns) - - [Anti-spam](#anti-spam) - - [Asset Management](#asset-management) - - [Audio](#audio) - - [Authentication](#authentication) - - [Build Tools](#build-tools) - - [Caching](#caching) - - [ChatOps Tools](#chatops-tools) - - [CMS](#cms) - - [Code Analysis and Linter](#code-analysis-and-linter) - - [Command-line Tools](#command-line-tools) - - [Compatibility](#compatibility) - - [Computer Vision](#computer-vision) - - [Concurrency and Parallelism](#concurrency-and-parallelism) - - [Configuration](#configuration) - - [Cryptography](#cryptography) - - [Data Analysis](#data-analysis) - - [Data Validation](#data-validation) - - [Data Visualization](#data-visualization) - - [Database Drivers](#database-drivers) - - [Database](#database) - - [Date and Time](#date-and-time) - - [Debugging Tools](#debugging-tools) - - [Deep Learning](#deep-learning) - - [DevOps Tools](#devops-tools) - - [Distribution](#distribution) - - [Documentation](#documentation) - - [Downloader](#downloader) - - [E-commerce](#e-commerce) - - [Editor Plugins and IDEs](#editor-plugins-and-ides) - - [Email](#email) - - [Environment Management](#environment-management) - - [Files](#files) - - [Foreign Function Interface](#foreign-function-interface) - - [Forms](#forms) - - [Functional Programming](#functional-programming) - - [Game Development](#game-development) - - [Geolocation](#geolocation) - - [GUI](#gui) - - [Hardware](#hardware) - - [HTML Manipulation](#html-manipulation) - - [HTTP](#http) - - [Imagery](#imagery) - - [Implementations](#implementations) - - [Interactive Interpreter](#interactive-interpreter) - - [Internationalization](#internationalization) - - [Job Scheduler](#job-scheduler) - - [Logging](#logging) - - [Machine Learning](#machine-learning) - - [MapReduce](#mapreduce) - - [Miscellaneous](#miscellaneous) - - [Natural Language Processing](#natural-language-processing) - - [Network Virtualization](#network-virtualization) - - [Networking](#networking) - - [News Feed](#news-feed) - - [ORM](#orm) - - [Package Management](#package-management) - - [Package Repositories](#package-repositories) - - [Permissions](#permissions) - - [Processes](#processes) - - [Queue](#queue) - - [RESTful API](#restful-api) - - [RPC Servers](#rpc-servers) - - [Science](#science) - - [Search](#search) - - [Serialization](#serialization) - - [Serverless Frameworks](#serverless-frameworks) - - [Specific Formats Processing](#specific-formats-processing) - - [Static Site Generator](#static-site-generator) - - [Tagging](#tagging) - - [Template Engine](#template-engine) - - [Testing](#testing) - - [Text Processing](#text-processing) - - [Third-party APIs](#third-party-apis) - - [URL Manipulation](#url-manipulation) - - [Video](#video) - - [Web Content Extracting](#web-content-extracting) - - [Web Crawling](#web-crawling) - - [Web Frameworks](#web-frameworks) - - [WebSocket](#websocket) - - [WSGI Servers](#wsgi-servers) -- [Services](#services) - - [Code Quality](#code-quality) - - [Continuous Integration](#continuous-integration) -- [Resources](#resources) - - [Podcasts](#podcasts) - - [Twitter](#twitter) - - [Websites](#websites) - - [Weekly](#weekly) -- [Other Awesome Lists](#other-awesome-lists) -- [Contributing](#contributing) - -- - - - -## Admin Panels - -*Libraries for administrative interfaces.* - -* [Ajenti](https://github.com/Eugeny/ajenti) - The admin panel your servers deserve. -* [django-suit](http://djangosuit.com/) - Alternative Django Admin-Interface (free only for Non-commercial use). -* [django-xadmin](https://github.com/sshwsfc/xadmin) - Drop-in replacement of Django admin comes with lots of goodies. -* [flask-admin](https://github.com/flask-admin/flask-admin) - Simple and extensible administrative interface framework for Flask. -* [flower](https://github.com/mher/flower) - Real-time monitor and web admin for Celery. -* [Grappelli](http://grappelliproject.com) - A jazzy skin for the Django Admin-Interface. -* [Wooey](https://github.com/wooey/wooey) - A Django app which creates automatic web UIs for Python scripts. - -## Algorithms and Design Patterns - -*Python implementation of algorithms and design patterns.* - -* [algorithms](https://github.com/nryoung/algorithms) - A module of algorithms for Python. -* [PyPattyrn](https://github.com/tylerlaberge/PyPattyrn) - A simple yet effective library for implementing common design patterns. -* [python-patterns](https://github.com/faif/python-patterns) - A collection of design patterns in Python. -* [sortedcontainers](http://www.grantjenks.com/docs/sortedcontainers/) - Fast, pure-Python implementation of SortedList, SortedDict, and SortedSet types. - -## Anti-spam - -*Libraries for fighting spam.* - -* [django-simple-captcha](https://github.com/mbi/django-simple-captcha) - A simple and highly customizable Django app to add captcha images to any Django form. -* [django-simple-spam-blocker](https://github.com/moqada/django-simple-spam-blocker) - Simple spam blocker for Django. - -## Asset Management - -*Tools for managing, compressing and minifying website assets.* - -* [django-compressor](https://github.com/django-compressor/django-compressor) - Compresses linked and inline JavaScript or CSS into a single cached file. -* [django-pipeline](https://github.com/jazzband/django-pipeline) - An asset packaging library for Django. -* [django-storages](https://github.com/jschneier/django-storages) - A collection of custom storage back ends for Django. -* [fanstatic](http://www.fanstatic.org/en/latest/) - Packages, optimizes, and serves static file dependencies as Python packages. -* [fileconveyor](http://fileconveyor.org/) - A daemon to detect and sync files to CDNs, S3 and FTP. -* [flask-assets](https://github.com/miracle2k/flask-assets) - Helps you integrate webassets into your Flask app. -* [jinja-assets-compressor](https://github.com/jaysonsantos/jinja-assets-compressor) - A Jinja extension to compile and compress your assets. -* [webassets](https://github.com/miracle2k/webassets) - Bundles, optimizes, and manages unique cache-busting URLs for static resources. - -## Audio - -*Libraries for manipulating audio.* - -* [audiolazy](https://github.com/danilobellini/audiolazy) - Expressive Digital Signal Processing (DSP) package for Python. -* [audioread](https://github.com/beetbox/audioread) - Cross-library (GStreamer + Core Audio + MAD + FFmpeg) audio decoding. -* [beets](http://beets.io/) - A music library manager and [MusicBrainz](https://musicbrainz.org/) tagger. -* [dejavu](https://github.com/worldveil/dejavu) - Audio fingerprinting and recognition. -* [django-elastic-transcoder](https://github.com/StreetVoice/django-elastic-transcoder) - Django + [Amazon Elastic Transcoder](http://aws.amazon.com/elastictranscoder/). -* [eyeD3](http://eyed3.nicfit.net/) - A tool for working with audio files, specifically MP3 files containing ID3 metadata. -* [id3reader](http://nedbatchelder.com/code/modules/id3reader.py) - A Python module for reading MP3 meta data. -* [m3u8](https://github.com/globocom/m3u8) - A module for parsing m3u8 file. -* [mingus](http://bspaans.github.io/python-mingus/) - An advanced music theory and notation package with MIDI file and playback support. -* [mutagen](https://github.com/quodlibet/mutagen) - A Python module to handle audio metadata. -* [pydub](https://github.com/jiaaro/pydub) - Manipulate audio with a simple and easy high level interface. -* [pyechonest](https://github.com/echonest/pyechonest) - Python client for the [Echo Nest](http://developer.echonest.com/) API. -* [talkbox](http://scikits.appspot.com/talkbox) - A Python library for speech/signal processing. -* [TimeSide](https://github.com/Parisson/TimeSide) - Open web audio processing framework. -* [tinytag](https://github.com/devsnd/tinytag) - A library for reading music meta data of MP3, OGG, FLAC and Wave files. - -## Authentication - -*Libraries for implementing authentications schemes.* - -* OAuth - * [Authomatic](http://peterhudec.github.io/authomatic/) - Simple but powerful framework agnostic authentication/authorization client. - * [django-allauth](https://github.com/pennersr/django-allauth) - Authentication app for Django that "just works." - * [django-oauth-toolkit](https://github.com/evonove/django-oauth-toolkit) - OAuth 2 goodies for Django. - * [Flask-OAuthlib](https://github.com/lepture/flask-oauthlib) - OAuth 1.0/a, 2.0 implementation of client and provider for Flask. - * [OAuthLib](https://github.com/idan/oauthlib) - A generic and thorough implementation of the OAuth request-signing logic. - * [python-oauth2](https://github.com/joestump/python-oauth2) - A fully tested, abstract interface to creating OAuth clients and servers. - * [python-social-auth](https://github.com/omab/python-social-auth) - An easy-to-setup social authentication mechanism. - * [rauth](https://github.com/litl/rauth) - A Python library for OAuth 1.0/a, 2.0, and Ofly. - * [sanction](https://github.com/demianbrecht/sanction) - A dead simple OAuth2 client implementation. -* Others - * [jose](https://github.com/demonware/jose) - JavaScript Object Signing and Encryption draft implementation. - * [PyJWT](https://github.com/jpadilla/pyjwt) - Implementation of the JSON Web Token draft 01. - * [python-jws](https://github.com/brianloveswords/python-jws) - Implementation of JSON Web Signatures draft 02. - * [python-jwt](https://github.com/davedoesdev/python-jwt) - Module for generating and verifying JSON Web Tokens. - -## Build Tools - -*Compile software from source code.* - -* [BitBake](http://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html) - A make-like build tool for embedded Linux. -* [buildout](http://www.buildout.org/en/latest/) - A build system for creating, assembling and deploying applications from multiple parts. -* [PlatformIO](https://github.com/platformio/platformio) - A console tool to build code with different development platforms. -* [PyBuilder](https://github.com/pybuilder/pybuilder) - A continuous build tool written in pure Python. -* [SCons](http://www.scons.org/) - A software construction tool. - -## CMS - -*Content Management Systems.* - -* [django-cms](http://www.django-cms.org/en/) - An Open source enterprise CMS based on the Django. -* [djedi-cms](http://djedi-cms.org/) - A lightweight but yet powerful Django CMS with plugins, inline editing and performance in mind. -* [FeinCMS](http://www.feincms.org/) - One of the most advanced Content Management Systems built on Django. -* [Kotti](http://kotti.pylonsproject.org/) - A high-level, Pythonic web application framework built on Pyramid. -* [Mezzanine](http://mezzanine.jupo.org/) - A powerful, consistent, and flexible content management platform. -* [Opps](http://opps.github.io/opps/) - A Django-based CMS for magazines, newspapers websites and portals with high-traffic. -* [Plone](https://plone.org/) - A CMS built on top of the open source application server Zope. -* [Quokka](http://quokkaproject.org/) - Flexible, extensible, small CMS powered by Flask and MongoDB. -* [Wagtail](https://wagtail.io/) - A Django content management system. -* [Widgy](https://wid.gy/) - Last CMS framework, based on Django. - -## Caching - -*Libraries for caching data.* - -* [Beaker](https://github.com/bbangert/beaker) - A library for caching and sessions for use with web applications and stand-alone Python scripts and applications. -* [DiskCache](http://www.grantjenks.com/docs/diskcache/) - SQLite and file backed cache backend with faster lookups than memcached and redis. -* [django-cache-machine](https://github.com/django-cache-machine/django-cache-machine) - Automatic caching and invalidation for Django models. -* [django-cacheops](https://github.com/Suor/django-cacheops) - A slick ORM cache with automatic granular event-driven invalidation. -* [django-viewlet](https://github.com/5monkeys/django-viewlet) - Render template parts with extended cache control. -* [dogpile.cache](http://dogpilecache.readthedocs.io/) - dogpile.cache is next generation replacement for Beaker made by same authors. -* [HermesCache](https://pypi.python.org/pypi/HermesCache) - Python caching library with tag-based invalidation and dogpile effect prevention. -* [johnny-cache](https://github.com/jmoiron/johnny-cache) - A caching framework for django applications. -* [pylibmc](https://github.com/lericson/pylibmc) - A Python wrapper around the [libmemcached](http://libmemcached.org/libMemcached.html) interface. - -## ChatOps Tools - -*Libraries for chatbot development.* - -* [Errbot](http://errbot.io/) - The easiest and most popular chatbot to implement ChatOps. - -## Code Analysis and Linter - -*Libraries and tools for analysing, parsing and manipulation codebases.* - -* Code Analysis - * [coala](http://coala-analyzer.org/) - Language independent and easily extendable code analysis application. - * [code2flow](https://github.com/scottrogowski/code2flow) - Turn your Python and JavaScript code into DOT flowcharts. - * [pycallgraph](https://github.com/gak/pycallgraph) - A library that visualises the flow (call graph) of your Python application. - * [pysonar2](https://github.com/yinwang0/pysonar2) - A type inferencer and indexer for Python. -* Linter - * [Flake8](https://pypi.python.org/pypi/flake8) - The modular source code checker: pep8, pyflakes and co. - * [pylama](https://github.com/klen/pylama) - Code audit tool for Python and JavaScript. - * [Pylint](https://www.pylint.org/) - A Fully customizable source code analyzer. - -## Command-line Tools - -*Libraries for building command-line application.* - -* Command-line Application Development - * [asciimatics](https://github.com/peterbrittain/asciimatics) - Cross-platform, full-screen terminal package (i.e. mouse/keyboard input and coloured, positioned text output) complete with high-level API for complex animations and special effects. - * [cement](http://builtoncement.com/) - CLI Application Framework for Python. - * [click](http://click.pocoo.org/dev/) - A package for creating beautiful command line interfaces in a composable way. - * [cliff](http://docs.openstack.org/developer/cliff/) - A framework for creating command-line programs with multi-level commands. - * [clint](https://github.com/kennethreitz/clint) - Python Command-line Application Tools. - * [colorama](https://pypi.python.org/pypi/colorama) - Cross-platform colored terminal text. - * [docopt](http://docopt.org/) - Pythonic command line arguments parser. - * [Gooey](https://github.com/chriskiehl/Gooey) - Turn command line programs into a full GUI application with one line - * [python-prompt-toolkit](https://github.com/jonathanslenders/python-prompt-toolkit) - A Library for building powerful interactive command lines. -* Productivity Tools - * [aws-cli](https://github.com/aws/aws-cli) - A universal command-line interface for Amazon Web Services. - * [bashplotlib](https://github.com/glamp/bashplotlib) - Making basic plots in the terminal. - * [caniusepython3](https://github.com/brettcannon/caniusepython3) - Determine what projects are blocking you from porting to Python 3. - * [cookiecutter](https://github.com/audreyr/cookiecutter) - A command-line utility that creates projects from cookiecutters (project templates). - * [doitlive](https://github.com/sloria/doitlive) - A tool for live presentations in the terminal. - * [howdoi](https://github.com/gleitz/howdoi) - Instant coding answers via the command line. - * [httpie](https://github.com/jkbrzt/httpie) - A command line HTTP client, a user-friendly cURL replacement. - * [mycli](https://github.com/dbcli/mycli) - A Terminal Client for MySQL with AutoCompletion and Syntax Highlighting. - * [PathPicker](https://github.com/facebook/PathPicker) - Select files out of bash output. - * [percol](https://github.com/mooz/percol) - Adds flavor of interactive selection to the traditional pipe concept on UNIX. - * [pgcli](https://github.com/dbcli/pgcli) - Postgres CLI with autocompletion and syntax highlighting. - * [SAWS](https://github.com/donnemartin/saws) - A Supercharged AWS CLI. - * [thefuck](https://github.com/nvbn/thefuck) - Correcting your previous console command. - * [try](https://github.com/timofurrer/try) - A dead simple CLI to try out python packages - It's never been easier. - -## Compatibility - -*Libraries for migrating from Python 2 to 3.* - -* [Python-Future](http://python-future.org/index.html) - The missing compatibility layer between Python 2 and Python 3. -* [Python-Modernize](https://github.com/mitsuhiko/python-modernize) - Modernizes Python code for eventual Python 3 migration. -* [Six](https://pypi.python.org/pypi/six) - Python 2 and 3 compatibility utilities. - -## Computer Vision - -*Libraries for computer vision.* - -* [OpenCV](http://opencv.org/) - Open Source Computer Vision Library. -* [pyocr](https://github.com/jflesch/pyocr) - A wrapper for Tesseract and Cuneiform. -* [pytesseract](https://github.com/madmaze/pytesseract) - Another wrapper for [Google Tesseract OCR](https://github.com/tesseract-ocr). -* [SimpleCV](http://simplecv.org/) - An open source framework for building computer vision applications. - -## Concurrency and Parallelism - -*Libraries for concurrent and parallel execution.* - -* [eventlet](http://eventlet.net/) - Asynchronous framework with WSGI support. -* [gevent](http://www.gevent.org/) - A coroutine-based Python networking library that uses [greenlet](https://github.com/python-greenlet/greenlet). -* [multiprocessing](https://docs.python.org/2/library/multiprocessing.html) - (Python standard library) Process-based "threading" interface. -* [threading](https://docs.python.org/2/library/threading.html) - (Python standard library) Higher-level threading interface. -* [Tomorrow](https://github.com/madisonmay/Tomorrow) - Magic decorator syntax for asynchronous code. -* [uvloop](https://github.com/MagicStack/uvloop) - Ultra fast implementation of asyncio event loop on top of libuv. - -## Configuration - -*Libraries for storing and parsing configuration options.* - -* [config](https://www.red-dove.com/config-doc/) - Hierarchical config from the author of [logging](https://docs.python.org/2/library/logging.html). -* [ConfigObj](http://www.voidspace.org.uk/python/configobj.html) - INI file parser with validation. -* [ConfigParser](https://docs.python.org/2/library/configparser.html) - (Python standard library) INI file parser. -* [profig](http://profig.readthedocs.org/en/default/) - Config from multiple formats with value conversion. -* [python-decouple](https://github.com/henriquebastos/python-decouple) - Strict separation of settings from code. - -## Cryptography - -* [cryptography](https://cryptography.io/en/latest/) - A package designed to expose cryptographic primitives and recipes to Python developers. -* [hashids](https://github.com/davidaurelio/hashids-python) - Implementation of [hashids](http://hashids.org) in Python. -* [Paramiko](http://www.paramiko.org/) - A Python (2.6+, 3.3+) implementation of the SSHv2 protocol, providing both client and server functionality. -* [Passlib](https://pythonhosted.org/passlib/) - Secure password storage/hashing library, very high level. -* [PyNacl](https://github.com/pyca/pynacl) - Python binding to the Networking and Cryptography (NaCl) library. - -## Data Analysis - -*Libraries for data analyzing.* - -* [Blaze](https://github.com/blaze/blaze) - NumPy and Pandas interface to Big Data. -* [Open Mining](https://github.com/mining/mining) - Business Intelligence (BI) in Pandas interface. -* [Orange](http://orange.biolab.si/) - Data mining, data visualization, analysis and machine learning through visual programming or scripts. -* [Pandas](http://pandas.pydata.org/) - A library providing high-performance, easy-to-use data structures and data analysis tools. - -## Data Validation - -*Libraries for validating data. Used for forms in many cases.* - -* [Cerberus](https://github.com/nicolaiarocci/cerberus/) - A lightweight and extensible data validation library. -* [colander](http://docs.pylonsproject.org/projects/colander/en/latest/) - Validating and deserializing data obtained via XML, JSON, an HTML form post. -* [jsonschema](https://github.com/Julian/jsonschema) - An implementation of [JSON Schema](http://json-schema.org/) for Python. -* [schema](https://github.com/keleshev/schema) - A library for validating Python data structures. -* [Schematics](https://github.com/schematics/schematics) - Data Structure Validation. -* [valideer](https://github.com/podio/valideer) - Lightweight extensible data validation and adaptation library. -* [voluptuous](https://github.com/alecthomas/voluptuous) - A Python data validation library. - -## Data Visualization - -*Libraries for visualizing data. See: [awesome-javascript](https://github.com/sorrycc/awesome-javascript#data-visualization).* - -* [Altair](https://github.com/altair-viz/altair) - Declarative statistical visualization library for Python. -* [Bokeh](https://github.com/bokeh/bokeh) - Interactive Web Plotting for Python. -* [ggplot](https://github.com/yhat/ggplot) - Same API as ggplot2 for R. -* [Matplotlib](http://matplotlib.org/) - A Python 2D plotting library. -* [Pygal](http://www.pygal.org/en/latest/) - A Python SVG Charts Creator. -* [PyGraphviz](https://pypi.python.org/pypi/pygraphviz) - Python interface to [Graphviz](http://www.graphviz.org/). -* [PyQtGraph](http://www.pyqtgraph.org/) - Interactive and realtime 2D/3D/Image plotting and science/engineering widgets. -* [Seaborn](https://github.com/mwaskom/seaborn) - Statistical data visualization using Matplotlib. -* [VisPy](https://github.com/vispy/vispy) - High-performance scientific visualization based on OpenGL. - -## Database - -*Databases implemented in Python.* - -* [pickleDB](https://pythonhosted.org/pickleDB/) - A simple and lightweight key-value store for Python. -* [PipelineDB](https://www.pipelinedb.com/) - The Streaming SQL Database. -* [TinyDB](https://github.com/msiemens/tinydb) - A tiny, document-oriented database. -* [ZODB](http://www.zodb.org/en/latest/) - A native object database for Python. A key-value and object graph database. - - -## Database Drivers - -*Libraries for connecting and operating databases.* - -* MySQL - [awesome-mysql](http://shlomi-noach.github.io/awesome-mysql/) - * [mysql-python](https://sourceforge.net/projects/mysql-python/) - The MySQL database connector for Python. - * [mysqlclient](https://github.com/PyMySQL/mysqlclient-python) - mysql-python fork supporting Python 3. - * [oursql](https://pythonhosted.org/oursql/) - A better MySQL connector with support for native prepared statements and BLOBs. - * [PyMySQL](https://github.com/PyMySQL/PyMySQL) - Pure Python MySQL driver compatible to mysql-python. -* PostgreSQL - * [psycopg2](http://initd.org/psycopg/) - The most popular PostgreSQL adapter for Python. - * [queries](https://github.com/gmr/queries) - A wrapper of the psycopg2 library for interacting with PostgreSQL. - * [txpostgres](https://github.com/wulczer/txpostgres) - Twisted based asynchronous driver for PostgreSQL. -* Other Relational Databases - * [apsw](http://rogerbinns.github.io/apsw/) - Another Python SQLite wrapper. - * [dataset](https://github.com/pudo/dataset) - Store Python dicts in a database - works with SQLite, MySQL, and PostgreSQL. - * [pymssql](http://www.pymssql.org/en/latest/) - A simple database interface to Microsoft SQL Server. -* NoSQL Databases - * [cassandra-python-driver](https://github.com/datastax/python-driver) - Python driver for Cassandra. - * [HappyBase](https://github.com/wbolster/happybase) - A developer-friendly library for Apache HBase. - * [Plyvel](https://github.com/wbolster/plyvel) - A fast and feature-rich Python interface to LevelDB. - * [py2neo](http://py2neo.org/2.0/) - Python wrapper client for Neo4j's restful interface. - * [pycassa](https://github.com/pycassa/pycassa) - Python Thrift driver for Cassandra. - * [PyMongo](https://docs.mongodb.org/ecosystem/drivers/python/) - The official Python client for MongoDB. - * [redis-py](https://github.com/andymccurdy/redis-py) - The Redis Python Client. - * [telephus](https://github.com/driftx/Telephus) - Twisted based client for Cassandra. - * [txRedis](https://github.com/deldotdr/txRedis) - Twisted based client for Redis. - -## Date and Time - -*Libraries for working with dates and times.* - -* [arrow](https://github.com/crsmithdev/arrow) - Better dates & times for Python. -* [Chronyk](https://github.com/KoffeinFlummi/Chronyk) - A Python 3 library for parsing human-written times and dates. -* [dateutil](https://github.com/dateutil/dateutil) - Extensions to the standard Python [datetime](https://docs.python.org/2/library/datetime.html) module. -* [delorean](https://github.com/myusuf3/delorean/) - A library for clearing up the inconvenient truths that arise dealing with datetimes. -* [moment](https://github.com/zachwill/moment) - A Python library for dealing with dates/times. Inspired by [Moment.js](http://momentjs.com/). -* [Pendulum](https://github.com/sdispater/pendulum) - Python datetimes made easy. -* [PyTime](https://github.com/shinux/PyTime) - A easy-use Python module which aims to operate date/time/datetime by string. -* [pytz](https://launchpad.net/pytz) - World timezone definitions, modern and historical. Brings the [tz database](https://en.wikipedia.org/wiki/Tz_database) into Python. -* [when.py](https://github.com/dirn/When.py) - Providing user-friendly functions to help perform common date and time actions. - -## Debugging Tools - -*Libraries for debugging code.* - -* pdb-like Debugger - * [ipdb](https://pypi.python.org/pypi/ipdb) - IPython-enabled [pdb](https://docs.python.org/3/library/pdb.html). - * [pdb++](https://pypi.python.org/pypi/pdbpp/) - Another drop-in replacement for pdb. - * [pudb](https://pypi.python.org/pypi/pudb) - A full-screen, console-based Python debugger. - * [remote-pdb](https://github.com/ionelmc/python-remote-pdb) - Remote vanilla PDB (over TCP sockets). - * [wdb](https://github.com/Kozea/wdb) - An improbable web debugger through WebSockets. -* Profiler - * [line_profiler](https://github.com/rkern/line_profiler) - Line-by-line profiling. - * [memory_profiler](https://github.com/fabianp/memory_profiler) - Monitor Memory usage of Python code. - * [profiling](https://github.com/what-studio/profiling) - An interactive Python profiler. - * [vprof](https://github.com/nvdv/vprof) - Visual Python profiler. -* Others - * [django-debug-toolbar](https://github.com/django-debug-toolbar/django-debug-toolbar) - Display various debug information for Django. - * [django-devserver](https://github.com/dcramer/django-devserver) - A drop-in replacement for Django's runserver. - * [flask-debugtoolbar](https://github.com/mgood/flask-debugtoolbar) - A port of the django-debug-toolbar to flask. - * [hunter](https://github.com/ionelmc/python-hunter) - Hunter is a flexible code tracing toolkit. - * [lptrace](https://github.com/khamidou/lptrace) - [strace](http://man7.org/linux/man-pages/man1/strace.1.html) for Python programs. - * [manhole](https://github.com/ionelmc/python-manhole) - Debug service that will accept unix domain socket connections and present the stacktraces for all threads and an interactive prompt. - * [pyelftools](https://github.com/eliben/pyelftools) - Parsing and analyzing ELF files and DWARF debugging information. - * [pyringe](https://github.com/google/pyringe) - Debugger capable of attaching to and injecting code into Python processes. - -## Deep Learning - -*Frameworks for Neural Networks and Deep Learning. See: [awesome-deep-learning](https://github.com/ChristosChristofidis/awesome-deep-learning).* - -* [Caffe](https://github.com/BVLC/caffe) - A fast open framework for deep learning.. -* [Keras](https://github.com/fchollet/keras) - A high-level neural networks library and capable of running on top of either TensorFlow or Theano. -* [MXNet](https://github.com/dmlc/mxnet) - A deep learning framework designed for both efficiency and flexibility. -* [Neupy](http://neupy.com/pages/home.html) - Running and testing different Artificial Neural Networks algorithms. -* [Pytorch](http://pytorch.org/) - Tensors and Dynamic neural networks in Python with strong GPU acceleration. -* [TensorFlow](https://github.com/tensorflow/tensorflow) - The most popular Deep Learning framework created by Google. -* [Theano](https://github.com/Theano/Theano) - A library for fast numerical computation. - -## DevOps Tools - -*Software and libraries for DevOps.* - -* [Ansible](https://github.com/ansible/ansible) - A radically simple IT automation platform. -* [Cloud-Init](http://cloudinit.readthedocs.io/) - A multi-distribution package that handles early initialization of a cloud instance. -* [cuisine](https://github.com/sebastien/cuisine) - Chef-like functionality for Fabric. -* [Docker Compose](https://docs.docker.com/compose/) - Fast, isolated development environments using [Docker](https://www.docker.com/). -* [Fabric](http://www.fabfile.org/) - A simple, Pythonic tool for remote execution and deployment. -* [Fabtools](https://github.com/ronnix/fabtools) - Tools for writing awesome Fabric files. -* [honcho](https://github.com/nickstenning/honcho) - A Python clone of [Foreman](https://github.com/ddollar/foreman), for managing Procfile-based applications. -* [OpenStack](http://www.openstack.org/) - Open source software for building private and public clouds. -* [pexpect](https://github.com/pexpect/pexpect) - Controlling interactive programs in a pseudo-terminal like GNU expect. -* [psutil](https://github.com/giampaolo/psutil) - A cross-platform process and system utilities module. -* [SaltStack](https://github.com/saltstack/salt) - Infrastructure automation and management system. -* [supervisor](https://github.com/Supervisor/supervisor) - Supervisor process control system for UNIX. - -## Distribution - -*Libraries to create packaged executables for release distribution.* - -* [dh-virtualenv](https://github.com/spotify/dh-virtualenv) - Build and distribute a virtualenv as a Debian package. -* [Nuitka](http://nuitka.net/) - Compile scripts, modules, packages to an executable or extension module. -* [py2app](http://pythonhosted.org/py2app/) - Freezes Python scripts (Mac OS X). -* [py2exe](http://www.py2exe.org/) - Freezes Python scripts (Windows). -* [PyInstaller](https://github.com/pyinstaller/pyinstaller) - Converts Python programs into stand-alone executables (cross-platform). -* [pynsist](http://pynsist.readthedocs.io/) - A tool to build Windows installers, installers bundle Python itself. - -## Documentation - -*Libraries for generating project documentation.* - -* [Sphinx](http://www.sphinx-doc.org/en/latest/) - Python Documentation generator. - * [awesome-sphinxdoc](https://github.com/yoloseem/awesome-sphinxdoc) -* [MkDocs](http://www.mkdocs.org/) - Markdown friendly documentation generator. -* [pdoc](https://github.com/BurntSushi/pdoc) - Epydoc replacement to auto generate API documentation for Python libraries. -* [Pycco](https://github.com/pycco-docs/pycco) - The literate-programming-style documentation generator. - -## Downloader - -*Libraries for downloading.* - -* [s3cmd](https://github.com/s3tools/s3cmd) - A command line tool for managing Amazon S3 and CloudFront. -* [s4cmd](https://github.com/bloomreach/s4cmd) - Super S3 command line tool, good for higher performance. -* [you-get](https://www.soimort.org/you-get/) - A YouTube/Youku/Niconico video downloader written in Python 3. -* [youtube-dl](http://rg3.github.io/youtube-dl/) - A small command-line program to download videos from YouTube. - -## E-commerce - -*Frameworks and libraries for e-commerce and payments.* - -* [alipay](https://github.com/lxneng/alipay) - Unofficial Alipay API for Python. -* [Cartridge](https://github.com/stephenmcd/cartridge) - A shopping cart app built using the Mezzanine. -* [django-oscar](http://oscarcommerce.com/) - An open-source e-commerce framework for Django. -* [django-shop](https://github.com/awesto/django-shop) - A Django based shop system. -* [merchant](https://github.com/agiliq/merchant) - A Django app to accept payments from various payment processors. -* [money](https://github.com/carlospalol/money) - Money class with optional CLDR-backed locale-aware formatting and an extensible currency exchange solution. -* [python-currencies](https://github.com/Alir3z4/python-currencies) - Display money format and its filthy currencies. -* [shoop](https://www.shoop.io/en/) - An open source E-Commerce platform based on Django. - -## Editor Plugins and IDEs - -* Emacs - * [Elpy](https://github.com/jorgenschaefer/elpy) - Emacs Python Development Environment. -* Sublime Text - * [Anaconda](https://github.com/DamnWidget/anaconda) - Anaconda turns your Sublime Text 3 in a full featured Python development IDE. - * [SublimeJEDI](https://github.com/srusskih/SublimeJEDI) - A Sublime Text plugin to the awesome auto-complete library Jedi. -* Vim - * [Jedi-vim](https://github.com/davidhalter/jedi-vim) - Vim bindings for the Jedi auto-completion library for Python. - * [Python-mode](https://github.com/klen/python-mode) - An all in one plugin for turning Vim into a Python IDE. - * [YouCompleteMe](https://github.com/Valloric/YouCompleteMe) - Includes [Jedi](https://github.com/davidhalter/jedi)-based completion engine for Python. -* Visual Studio - * [PTVS](https://github.com/Microsoft/PTVS) - Python Tools for Visual Studio. -* Visual Studio Code - * [Python](https://github.com/DonJayamanne/pythonVSCode) - An extension with rich support for the Python language, with features including linting, IntelliSense, formatting, refactoring, debugging, unit tests, and jupyter support. - * [Magic Python](https://github.com/MagicStack/MagicPython) - Cutting edge Python syntax highlighter for Sublime Text, Atom, and Visual Studio Code. Used by GitHub to highlight your Python code! -* IDE - * [LiClipse](http://www.liclipse.com/) - Free polyglot IDE based on Eclipse. Uses PyDev for Python support. - * [PyCharm](https://www.jetbrains.com/pycharm/) - Commercial Python IDE by JetBrains. Has free community edition available. - * [Spyder](https://github.com/spyder-ide/spyder) - Open Source Python IDE. - -## Email - -*Libraries for sending and parsing email.* - -* [envelopes](http://tomekwojcik.github.io/envelopes/) - Mailing for human beings. -* [flanker](https://github.com/mailgun/flanker) - A email address and Mime parsing library. -* [imbox](https://github.com/martinrusev/imbox) - Python IMAP for Humans. -* [inbox.py](https://github.com/kennethreitz/inbox.py) - Python SMTP Server for Humans. -* [lamson](https://github.com/zedshaw/lamson) - Pythonic SMTP Application Server. -* [Marrow Mailer](https://github.com/marrow/mailer) - High-performance extensible mail delivery framework. -* [modoboa](https://github.com/tonioo/modoboa) - A mail hosting and management platform including a modern and simplified Web UI. -* [Nylas Sync Engine](https://github.com/nylas/sync-engine) - Providing a RESTful API on top of a powerful email sync platform. -* [yagmail](https://github.com/kootenpv/yagmail) - Yet another Gmail/SMTP client. - -## Environment Management - -*Libraries for Python version and environment management.* - -* [p](https://github.com/qw3rtman/p) - Dead simple interactive Python version management. -* [pyenv](https://github.com/yyuu/pyenv) - Simple Python version management. -* [venv](https://docs.python.org/3/library/venv.html) - (Python standard library in Python 3.3+) Creating lightweight virtual environments. -* [virtualenv](https://pypi.python.org/pypi/virtualenv) - A tool to create isolated Python environments. -* [virtualenvwrapper](https://pypi.python.org/pypi/virtualenvwrapper) - A set of extensions to virtualenv. - -## Files - -*Libraries for file manipulation and MIME type detection.* - -* [imghdr](https://docs.python.org/2/library/imghdr.html) - (Python standard library) Determine the type of an image. -* [mimetypes](https://docs.python.org/2/library/mimetypes.html) - (Python standard library) Map filenames to MIME types. -* [path.py](https://github.com/jaraco/path.py) - A module wrapper for [os.path](https://docs.python.org/2/library/os.path.html). -* [pathlib](https://pathlib.readthedocs.org/en/pep428/) - (Python standard library in Python 3.4+) An cross-platform, object-oriented path library. -* [python-magic](https://github.com/ahupp/python-magic) - A Python interface to the libmagic file type identification library. -* [Unipath](https://github.com/mikeorr/Unipath) - An object-oriented approach to file/directory operations. -* [watchdog](https://github.com/gorakhargosh/watchdog) - API and shell utilities to monitor file system events. +# [Awesome Python](https://awesome-python.com/) -## Foreign Function Interface - -*Libraries for providing foreign function interface.* - -* [cffi](https://pypi.python.org/pypi/cffi) - Foreign Function Interface for Python calling C code. -* [ctypes](https://docs.python.org/2/library/ctypes.html) - (Python standard library) Foreign Function Interface for Python calling C code. -* [PyCUDA](https://mathema.tician.de/software/pycuda/) - A Python wrapper for Nvidia's CUDA API. -* [SWIG](http://www.swig.org/Doc1.3/Python.html) - Simplified Wrapper and Interface Generator. +An opinionated guide to the best Python frameworks, libraries, and tools. -## Forms +**Visit the [website](https://awesome-python.com/) to search and filter projects more easily.** -*Libraries for working with forms.* - -* [Deform](https://github.com/Pylons/deform) - Python HTML form generation library influenced by the formish form generation library. -* [django-bootstrap3](https://github.com/dyve/django-bootstrap3) - Bootstrap 3 integration with Django. -* [django-crispy-forms](https://github.com/maraujop/django-crispy-forms) - A Django app which lets you create beautiful forms in a very elegant and DRY way. -* [django-remote-forms](https://github.com/WiserTogether/django-remote-forms) - A platform independent Django form serializer. -* [WTForms](https://github.com/wtforms/wtforms) - A flexible forms validation and rendering library. - -## Functional Programming - -*Functional Programming with Python.* - -* [CyToolz](https://github.com/pytoolz/cytoolz/) - Cython implementation of Toolz: High performance functional utilities. -* [fn.py](https://github.com/kachayev/fn.py) - Functional programming in Python: implementation of missing features to enjoy FP. -* [funcy](https://github.com/Suor/funcy) - A fancy and practical functional tools. -* [Toolz](https://github.com/pytoolz/toolz) - A collection of functional utilities for iterators, functions, and dictionaries. - -## GUI - -*Libraries for working with graphical user interface applications.* - -* [curses](https://docs.python.org/2/library/curses.html#module-curses) - Built-in wrapper for [ncurses](http://www.gnu.org/software/ncurses/) used to create terminal GUI applications. -* [enaml](https://github.com/nucleic/enaml) - Creating beautiful user-interfaces with Declaratic Syntax like QML. -* [Flexx](https://github.com/zoofIO/flexx) - Flexx is a pure Python toolkit for creating GUI's, that uses web technology for its rendering. -* [kivy](https://kivy.org/) - A library for creating NUI applications, running on Windows, Linux, Mac OS X, Android and iOS. -* [pyglet](https://bitbucket.org/pyglet/pyglet/wiki/Home) - A cross-platform windowing and multimedia library for Python. -* [PyGObject](https://wiki.gnome.org/Projects/PyGObject) - Python Bindings for GLib/GObject/GIO/GTK+ (GTK+3) -* [PyQt](https://riverbankcomputing.com/software/pyqt/intro) - Python bindings for the [Qt](http://www.qt.io/) cross-platform application and UI framework, with support for both Qt v4 and Qt v5 frameworks. -* [PySide](https://wiki.qt.io/PySide) - Python bindings for the [Qt](http://www.qt.io/) cross-platform application and UI framework, supporting the Qt v4 framework. -* [Tkinter](https://wiki.python.org/moin/TkInter) - Tkinter is Python's de-facto standard GUI package. -* [Toga](https://github.com/pybee/toga) - A Python native, OS native GUI toolkit. -* [urwid](http://urwid.org/) - A library for creating terminal GUI applications with strong support for widgets, events, rich colors, etc. -* [wxPython](http://wxpython.org/) - A blending of the wxWidgets C++ class library with the Python. - -## Game Development - -*Awesome game development libraries.* - -* [Cocos2d](http://cocos2d.org/) - cocos2d is a framework for building 2D games, demos, and other graphical/interactive applications. It is based on pyglet. -* [Panda3D](https://www.panda3d.org/) - 3D game engine developed by Disney and maintained by Carnegie Mellon's Entertainment Technology Center. Written in C++, completely wrapped in Python. -* [Pygame](http://www.pygame.org/news.html) - Pygame is a set of Python modules designed for writing games. -* [PyOgre](http://www.ogre3d.org/tikiwiki/PyOgre) - Python bindings for the Ogre 3D render engine, can be used for games, simulations, anything 3D. -* [PyOpenGL](http://pyopengl.sourceforge.net/) - Python ctypes bindings for OpenGL and it's related APIs. -* [PySDL2](http://pysdl2.readthedocs.io/) - A ctypes based wrapper for the SDL2 library. -* [RenPy](https://www.renpy.org/) - A Visual Novel engine. +## **Sponsors** -## Geolocation +- [iPulse AI](https://ipulseai.com) - Open Agentic Investment Research Platform. Inspect deep multi-agent market forecasts and stock picks. -*Libraries for geocoding addresses and working with latitudes and longitudes.* +> The **#10 most-starred repo on GitHub**. Put your product in front of Python developers. [Become a sponsor](SPONSORSHIP.md). -* [django-countries](https://github.com/SmileyChris/django-countries) - A Django app that provides country choices for use with forms, flag icons static files, and a country field for models. -* [GeoDjango](https://docs.djangoproject.com/en/dev/ref/contrib/gis/) - A world-class geographic web framework. -* [GeoIP](https://github.com/maxmind/geoip-api-python) - Python API for MaxMind GeoIP Legacy Database. -* [geojson](https://github.com/frewsxcv/python-geojson) - Python bindings and utilities for GeoJSON. -* [geopy](https://github.com/geopy/geopy) - Python Geocoding Toolbox. -* [pygeoip](https://github.com/appliedsec/pygeoip) - Pure Python GeoIP API. - -## HTML Manipulation - -*Libraries for working with HTML and XML.* +## Categories -* [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) - Providing Pythonic idioms for iterating, searching, and modifying HTML or XML. -* [bleach](https://github.com/mozilla/bleach) - A whitelist-based HTML sanitization and text linkification library. -* [cssutils](https://pypi.python.org/pypi/cssutils/) - A CSS library for Python. -* [html5lib](https://github.com/html5lib/html5lib-python) - A standards-compliant library for parsing and serializing HTML documents and fragments. -* [lxml](http://lxml.de/) - A very fast, easy-to-use and versatile library for handling HTML and XML. -* [MarkupSafe](https://github.com/pallets/markupsafe) - Implements a XML/HTML/XHTML Markup safe string for Python. -* [pyquery](https://github.com/gawel/pyquery) - A jQuery-like library for parsing HTML. -* [untangle](https://github.com/stchris/untangle) - Converts XML documents to Python objects for easy access. -* [WeasyPrint](http://weasyprint.org) - A visual rendering engine for HTML and CSS that can export to PDF. -* [xmldataset](https://xmldataset.readthedocs.io) - Simple XML Parsing. -* [xmltodict](https://github.com/martinblech/xmltodict) - Working with XML feel like you are working with JSON. +**AI & ML** -## HTTP +- [AI and Agents](#ai-and-agents) +- [Deep Learning](#deep-learning) +- [Machine Learning](#machine-learning) +- [Natural Language Processing](#natural-language-processing) +- [Computer Vision](#computer-vision) +- [Recommender Systems](#recommender-systems) -*Libraries for working with HTTP.* +**Web Development** -* [grequests](https://github.com/kennethreitz/grequests) - requests + gevent for asynchronous HTTP requests. -* [httplib2](https://github.com/jcgregorio/httplib2) - Comprehensive HTTP client library. -* [requests](http://docs.python-requests.org/en/latest/) - HTTP Requests for Humans™. -* [treq](https://github.com/twisted/treq) - Python requests like API built on top of Twisted's HTTP client. -* [urllib3](https://github.com/shazow/urllib3) - A HTTP library with thread-safe connection pooling, file post support, sanity friendly. - -## Hardware +- [Web Frameworks](#web-frameworks) +- [Web APIs](#web-apis) +- [Web Servers](#web-servers) +- [WebSocket](#websocket) +- [Template Engines](#template-engines) +- [Web Asset Management](#web-asset-management) +- [Authentication](#authentication) +- [Admin Panels](#admin-panels) +- [CMS](#cms) +- [ERP](#erp) +- [Static Site Generators](#static-site-generators) -*Libraries for programming with hardware.* +**HTTP & Scraping** -* [ino](http://inotool.org/) - Command line toolkit for working with [Arduino](https://www.arduino.cc/). -* [Pingo](http://www.pingo.io/) - Pingo provides a uniform API to program devices like the Raspberry Pi, pcDuino, Intel Galileo, etc. -* [Pyro](http://pyrorobotics.com/) - Python Robotics. -* [PyUserInput](https://github.com/SavinaRoja/PyUserInput) - A module for cross-platform control of the mouse and keyboard. -* [scapy](https://github.com/secdev/scapy) - A brilliant packet manipulation library. -* [wifi](https://github.com/rockymeza/wifi) - A Python library and command line tool for working with WiFi on Linux. - -## Imagery +- [HTTP Clients](#http-clients) +- [Web Scraping](#web-scraping) +- [Email](#email) -*Libraries for manipulating images.* +**Database & Storage** -* [hmap](https://github.com/rossgoodwin/hmap) - Image histogram remapping. -* [imgSeek](https://sourceforge.net/projects/imgseek/) - A project for searching a collection of images using visual similarity. -* [nude.py](https://github.com/hhatto/nude.py) - Nudity detection. -* [pagan](https://github.com/daboth/pagan) - Retro identicon (Avatar) generation based on input string and hash. -* [pillow](https://github.com/python-pillow/Pillow) - Pillow is the friendly [PIL](http://www.pythonware.com/products/pil/) fork. -* [pyBarcode](https://pythonhosted.org/pyBarcode/) - Create barcodes in Python without needing PIL. -* [pygram](https://github.com/ajkumar25/pygram) - Instagram-like image filters. -* [python-qrcode](https://github.com/lincolnloop/python-qrcode) - A pure Python QR Code generator. -* [Quads](https://github.com/fogleman/Quads) - Computer art based on quadtrees. -* [scikit-image](http://scikit-image.org/) - A Python library for (scientific) image processing. -* [thumbor](https://github.com/thumbor/thumbor) - A smart imaging service. It enables on-demand crop, re-sizing and flipping of images. -* [wand](https://github.com/dahlia/wand) - Python bindings for [MagickWand](http://www.imagemagick.org/script/magick-wand.php), C API for ImageMagick. - -## Implementations - -*Implementations of Python.* +- [ORM](#orm) +- [Database Drivers](#database-drivers) +- [Database](#database) +- [Caching](#caching) +- [Search](#search) +- [Serialization](#serialization) -* [CLPython](https://github.com/metawilm/cl-python) - Implementation of the Python programming language written in Common Lisp. -* [CPython](https://hg.python.org/cpython) - **Default, most widely used implementation of the Python programming language written in C.** Optimizing Static Compiler for Python. Uses type mixins to compile Python into C or C++ modules resulting in large performance gains. -* [Grumpy](http://grump.io) - More compiler than interpreter as more powerful CPython2.7 replacement (alpha). -* [IronPython](https://github.com/IronLanguages/ironpython3) - Implementation of the Python programming language written in C# targeting the .NET Framework and Mono. -* [Jython](https://hg.python.org/jython) - Implementation of Python programming language written in Java for the Java virtual machine (JVM). -* [MicroPython](https://github.com/micropython/micropython) - MicroPython - a lean and efficient Python programming language implementation for microcontrollers and constrained systems -* [Numba](http://numba.pydata.org/) - Python JIT complier to LLVM aimed at scientific Python. -* [PeachPy](https://github.com/Maratyszcza/PeachPy) - x86-64 assembler embedded in Python. Can be used as inline assembler for Python or as a stand-alone assembler for Windows, Linux, OS X, Native Client and Go. -* [Pyjion](https://github.com/Microsoft/Pyjion) - A JIT for Python based upon CoreCLR. -* [PyPy](https://bitbucket.org/pypy/pypy) - Implementation of the Python programming language written in RPython and translated into C. PyPy focuses on speed, efficiency and compatibility with the original CPython interpreter. The interpreter uses black magic to make Python very fast without having to add in additional type information. -* [PySec](https://github.com/ebranca/owasp-pysec) - Hardened version of python that makes it easier for security professionals and developers to write applications more resilient to attacks and manipulations. -* [Pyston](https://github.com/dropbox/pyston) - A Python implementation built using LLVM and modern JIT techniques with the goal of achieving good performance. -* [Stackless Python](https://bitbucket.org/stackless-dev/stackless/wiki/Home) - An enhanced version of the Python programming language which allows programmers to reap the benefits of thread-based programming without the performance and complexity problems associated with conventional threads. +**Data & Science** -## Interactive Interpreter +- [Data Analysis](#data-analysis) +- [Data Ingestion / ETL](#data-ingestion--etl) +- [Data Validation](#data-validation) +- [Data Visualization](#data-visualization) +- [Geolocation](#geolocation) +- [Science](#science) +- [Quantum Computing](#quantum-computing) + +**Developer Tools** + +- [Algorithms and Design Patterns](#algorithms-and-design-patterns) +- [Interactive Interpreter](#interactive-interpreter) +- [Code Analysis](#code-analysis) +- [Testing](#testing) +- [Debugging Tools](#debugging-tools) +- [Build Tools](#build-tools) +- [Documentation](#documentation) + +**DevOps** + +- [DevOps Tools](#devops-tools) +- [Distributed Computing](#distributed-computing) +- [Task Queues](#task-queues) +- [Messaging](#messaging) +- [Job Schedulers](#job-schedulers) +- [Logging](#logging) +- [Network Virtualization](#network-virtualization) + +**CLI & GUI** + +- [CLI Development](#cli-development) +- [CLI Tools](#cli-tools) +- [GUI Development](#gui-development) + +**Text & Documents** + +- [Text Processing](#text-processing) +- [HTML Manipulation](#html-manipulation) +- [File Format Processing](#file-format-processing) +- [File Manipulation](#file-manipulation) + +**Media** + +- [Image Processing](#image-processing) +- [Audio & Video Processing](#audio--video-processing) +- [Game Development](#game-development) + +**Python Language** + +- [Implementations](#implementations) +- [Built-in Classes Enhancement](#built-in-classes-enhancement) +- [Functional Programming](#functional-programming) +- [Asynchronous Programming](#asynchronous-programming) +- [Date and Time](#date-and-time) + +**Python Toolchain** + +- [Environment Management](#environment-management) +- [Package Management](#package-management) +- [Package Repositories](#package-repositories) +- [Distribution](#distribution) +- [Configuration Files](#configuration-files) + +**Security** + +- [Cryptography](#cryptography) +- [Penetration Testing](#penetration-testing) +- [Supply Chain Security](#supply-chain-security) +- [Web Security](#web-security) + +**Other** + +- [Hardware](#hardware) +- [Microsoft Windows](#microsoft-windows) +- [Miscellaneous](#miscellaneous) + +## Projects + +**AI & ML** + +### AI and Agents + +_Libraries for building AI applications, LLM integrations, and autonomous agents._ + +- Agent Skills + - [django-ai-plugins](https://github.com/vintasoftware/django-ai-plugins) - Django backend agent skills for Django, DRF, Celery, and Django-specific code review. + - [scientific-agent-skills](https://github.com/K-Dense-AI/scientific-agent-skills) - Agent skills for scientific research and analysis with Python libraries like scikit-learn, polars, and biopython. + - [sentry-skills](https://github.com/getsentry/skills) - Agent skills the Sentry team uses for code review, pull requests, and Django reviews. + - [trailofbits-skills](https://github.com/trailofbits/skills) - Security skills for vulnerability detection, auditing, and testing. + - [wshobson-agents](https://github.com/wshobson/agents) - Plugins, agents, and skills for AI coding tools, including Python skills for testing, async, packaging, and uv. +- Orchestration + - [langchain](https://github.com/langchain-ai/langchain) - A framework for building agents and LLM-powered applications. + - [langgraph](https://github.com/langchain-ai/langgraph) - Low-level orchestration framework for building stateful, long-running LLM agents. + - [pydantic-ai](https://github.com/pydantic/pydantic-ai) - A Python agent framework for building generative AI applications with structured schemas. + - [crewai](https://github.com/crewAIInc/crewAI) - A framework for orchestrating role-playing autonomous AI agents for collaborative task solving. +- Vendor Agent SDKs + - [claude-agent-sdk](https://github.com/anthropics/claude-agent-sdk-python) - Anthropic's Python SDK for building AI agents on Claude Code's harness — custom tools, in-process MCP servers, hooks. + - [openai-agents](https://github.com/openai/openai-agents-python) - OpenAI's framework for building and managing AI agents. + - [google-adk](https://github.com/google/adk-python) - Google's code-first toolkit for building, evaluating, and deploying AI agents. +- Model Context Protocol + - [mcp](https://github.com/modelcontextprotocol/python-sdk) - The official Python SDK for building Model Context Protocol servers and clients. + - [fastmcp](https://github.com/PrefectHQ/fastmcp) - A high-level, Pythonic framework for building MCP servers and clients. +- Personal Assistants + - [hermes-agent](https://github.com/NousResearch/hermes-agent) - An adaptive personal AI assistant that grows with you. + - [AstrBot](https://github.com/AstrBotDevs/AstrBot) - A multi-platform AI assistant that connects LLMs to chat apps like Telegram, Slack, and QQ, extensible with Python plugins. +- Prompt Optimization + - [dspy](https://github.com/stanfordnlp/dspy) - A framework for programming, not prompting, language models. +- Data Layer + - [instructor](https://github.com/567-labs/instructor) - A library for extracting structured data from LLMs, powered by Pydantic. + - [llama-index](https://github.com/run-llama/llama_index) - A toolkit for building RAG pipelines and agents over your data. + - [mem0](https://github.com/mem0ai/mem0) - An intelligent memory layer for AI agents enabling personalized interactions. + - [openviking](https://github.com/volcengine/OpenViking) - A context database for AI agents that unifies memory, resources, and skills. + - [semantica](https://github.com/semantica-agi/semantica) - A graph-native context and knowledge layer for AI agents with reasoning, provenance, and governance. +- Pre-trained Models + - [transformers](https://github.com/huggingface/transformers) - The model-definition framework for pretrained models in text, computer vision, audio, video, and multimodal tasks, for inference and training. + - [sentence-transformers](https://github.com/huggingface/sentence-transformers) - Compute embeddings, run retrieval, and rerank results with pretrained and fine-tuned models. +- Local LLM Inference + - [ollama](https://github.com/ollama/ollama-python) - Python client for Ollama, which downloads and runs open-weight LLMs locally. + - [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) - Python bindings for llama.cpp, for running GGUF models locally. + - [mlx-lm](https://github.com/ml-explore/mlx-lm) - Run and fine-tune large language models on Apple Silicon with MLX. +- LLM Serving + - [vllm](https://github.com/vllm-project/vllm) - A high-throughput and memory-efficient inference and serving engine for LLMs. + - [sglang](https://github.com/sgl-project/sglang) - A high-performance serving framework for large language models and multimodal models. +- LLM Gateways + - [litellm](https://github.com/BerriAI/litellm) - Call 100+ LLMs using OpenAI format. +- Image and Video Generation + - [diffusers](https://github.com/huggingface/diffusers) - A library that provides pre-trained diffusion models for generating and editing images, audio, and video. +- Fine-tuning + - [peft](https://github.com/huggingface/peft) - A library for parameter-efficient fine-tuning of large pretrained models. + - [trl](https://github.com/huggingface/trl) - A library for post-training transformer language models with SFT, DPO, GRPO, and other trainers. + - [unsloth](https://github.com/unslothai/unsloth) - Faster, lower-memory LLM fine-tuning, as a Python library or a desktop app. + - [axolotl](https://github.com/axolotl-ai-cloud/axolotl) - A framework for fine-tuning and post-training large language models. +- Speech + - [faster-whisper](https://github.com/SYSTRAN/faster-whisper) - A Whisper reimplementation on CTranslate2, up to 4 times faster than openai-whisper with less memory. + - [openai-whisper](https://github.com/openai/whisper) - A general-purpose automatic speech recognition model trained on 680k hours of multilingual and multitask supervised data. + - [gTTS](https://github.com/pndurette/gTTS) - Python library and CLI tool for converting text to speech using Google Translate TTS. + - [funasr](https://github.com/modelscope/FunASR) - Industrial-grade speech recognition toolkit with speaker diarization and emotion detection. + +### Deep Learning + +_Frameworks for Neural Networks and Deep Learning. Also see [awesome-deep-learning](https://github.com/ChristosChristofidis/awesome-deep-learning)._ + +- Frameworks + - [pytorch](https://github.com/pytorch/pytorch) - Tensors and Dynamic neural networks in Python with strong GPU acceleration. + - [pytorch-lightning](https://github.com/Lightning-AI/pytorch-lightning) - Deep learning framework to train, deploy, and ship AI products Lightning fast. + - [jax](https://github.com/jax-ml/jax) - A library for high-performance numerical computing with automatic differentiation and JIT compilation. + - [keras](https://github.com/keras-team/keras) - A high-level deep learning library with support for JAX, TensorFlow, and PyTorch backends. + - [tensorflow](https://github.com/tensorflow/tensorflow) - An end-to-end machine learning platform from Google. +- Reinforcement Learning + - [gymnasium](https://github.com/Farama-Foundation/Gymnasium) - A standard API for reinforcement learning environments with popular reference environments ([gym](https://github.com/openai/gym) successor). + - [stable-baselines3](https://github.com/DLR-RM/stable-baselines3) - PyTorch implementations of Stable Baselines (deep) reinforcement learning algorithms. + +### Machine Learning + +_Libraries for Machine Learning. Also see [awesome-machine-learning](https://github.com/josephmisiti/awesome-machine-learning#python)._ + +- General + - [scikit-learn](https://github.com/scikit-learn/scikit-learn) - The most popular Python library for Machine Learning with extensive documentation and community support. + - [imbalanced-learn](https://github.com/scikit-learn-contrib/imbalanced-learn) - Resampling techniques for datasets with imbalanced classes, compatible with scikit-learn. + - [pgmpy](https://github.com/pgmpy/pgmpy) - A Python library for causal and probabilistic reasoning with graphical models. + - [feature-engine](https://github.com/feature-engine/feature_engine) - sklearn compatible API with the widest toolset for feature engineering and selection. +- Gradient Boosting + - [xgboost](https://github.com/dmlc/xgboost) - A scalable, portable, and distributed gradient boosting library. + - [lightgbm](https://github.com/lightgbm-org/LightGBM) - A fast, distributed, high performance gradient boosting framework. + - [catboost](https://github.com/catboost/catboost) - A fast, scalable, high performance gradient boosting on decision trees library. +- Time Series Forecasting + - [prophet](https://github.com/facebook/prophet) - A tool for producing forecasts for time series with multiple seasonality and trend changes. + - [statsforecast](https://github.com/Nixtla/statsforecast) - Fast statistical forecasting models such as ARIMA, ETS, and Theta, compiled with numba. + - [sktime](https://github.com/sktime/sktime) - A unified scikit-learn-style framework for forecasting and other time-series learning tasks. + - [timesfm](https://github.com/google-research/timesfm) - A pretrained foundation model from Google Research for time-series forecasting, with non-commercial default weights. +- Hyperparameter Optimization + - [optuna](https://github.com/optuna/optuna) - A hyperparameter optimization framework with define-by-run search spaces and pruning of unpromising trials. + - [hyperopt](https://github.com/hyperopt/hyperopt) - Distributed asynchronous hyperparameter optimization with Tree-structured Parzen Estimators. + +### Natural Language Processing + +_Libraries for working with human languages._ + +- General + - [nltk](https://github.com/nltk/nltk) - A leading platform for building Python programs to work with human language data. + - [spacy](https://github.com/explosion/spaCy) - A library for industrial-strength natural language processing in Python and Cython. + - [gensim](https://github.com/piskvorky/gensim) - Topic Modeling for Humans. + - [stanza](https://github.com/stanfordnlp/stanza) - The Stanford NLP Group's official Python library, supporting 60+ languages. +- Chinese + - [jieba](https://github.com/fxsjy/jieba) - The most popular Chinese text segmentation library. + - [pypinyin](https://github.com/mozillazg/python-pinyin) - Convert Chinese hanzi (漢字) to pinyin (拼音). + - [OpenCC](https://github.com/BYVoid/OpenCC) - Conversion between Traditional and Simplified Chinese, with regional variants and phrase-level rules. + - [pangu.py](https://github.com/vinta/pangu.py) - Paranoid text spacing. + +### Computer Vision + +_Libraries for image and video analysis, object detection, and OCR._ + +- General + - [opencv-python](https://github.com/opencv/opencv-python) - Open Source Computer Vision Library. + - [ultralytics](https://github.com/ultralytics/ultralytics) - Ultralytics YOLO for object detection, segmentation, pose estimation, classification, and tracking. + - [kornia](https://github.com/kornia/kornia) - Open Source Differentiable Computer Vision Library for PyTorch. + - [fiftyone](https://github.com/voxel51/fiftyone) - The open-source tool for building high-quality datasets and computer vision models. +- OCR + - [pytesseract](https://github.com/madmaze/pytesseract) - A wrapper for the [Tesseract OCR](https://github.com/tesseract-ocr/tesseract) engine. + - [easyocr](https://github.com/JaidedAI/EasyOCR) - Ready-to-use OCR with 80+ languages supported. + - [paddleocr](https://github.com/PaddlePaddle/PaddleOCR) - Multilingual OCR and document parsing toolkit based on PaddlePaddle. + +### Recommender Systems + +_Libraries for building recommender systems._ + +- [annoy](https://github.com/spotify/annoy) - Approximate Nearest Neighbors in C++/Python optimized for memory usage. +- [implicit](https://github.com/benfred/implicit) - A fast Python implementation of collaborative filtering for implicit datasets. +- [scikit-surprise](https://github.com/NicolasHug/Surprise) - A scikit for building and analyzing recommender systems. + +**Web Development** + +### Web Frameworks + +_Traditional full stack web frameworks. Also see [Web APIs](#web-apis)._ + +- Synchronous + - [flask](https://github.com/pallets/flask) - A microframework for Python. + - [awesome-flask](https://github.com/humiaozuzu/awesome-flask) + - [django](https://github.com/django/django) - A high-level web framework that encourages rapid development and clean, pragmatic design. + - [awesome-django](https://github.com/wsvincent/awesome-django) + - [bottle](https://github.com/bottlepy/bottle) - A fast and simple micro-framework distributed as a single file with no dependencies. + - [pyramid](https://github.com/Pylons/pyramid) - A small, fast, down-to-earth, open source Python web framework. + - [fasthtml](https://github.com/AnswerDotAI/fasthtml) - The fastest way to create an HTML app. +- Asynchronous + - [starlette](https://github.com/Kludex/starlette) - A lightweight ASGI framework and toolkit for building high-performance async services. + - [tornado](https://github.com/tornadoweb/tornado) - A web framework and asynchronous networking library. + - [litestar](https://github.com/litestar-org/litestar) - Production-ready, capable and extensible ASGI Web framework. + - [reflex](https://github.com/reflex-dev/reflex) - A framework for building reactive, full-stack web applications entirely with Python. -*Interactive Python interpreters (REPL).* +### Web APIs -* [bpython](https://github.com/bpython/bpython) - A fancy interface to the Python interpreter. -* [Jupyter Notebook (IPython)](https://jupyter.org) - A rich toolkit to help you make the most out of using Python interactively. -* [ptpython](https://github.com/jonathanslenders/ptpython) - Advanced Python REPL built on top of the [python-prompt-toolkit](https://github.com/jonathanslenders/python-prompt-toolkit). +_Libraries for building RESTful, GraphQL, and RPC APIs._ -## Internationalization +- Django + - [django-rest-framework](https://github.com/encode/django-rest-framework) - A powerful and flexible toolkit to build web APIs. + - [django-ninja](https://github.com/vitalik/django-ninja) - Fast, Django REST framework based on type hints and Pydantic. + - [strawberry-django](https://github.com/strawberry-graphql/strawberry-django) - Strawberry GraphQL integration with Django. + - [django-modern-rest](https://github.com/wemake-services/django-modern-rest) - Modern REST with speed, types, async, `msgspec`, `pydantic` and other goodies. +- Flask + - [flask-restx](https://github.com/python-restx/flask-restx) - Fully featured framework for fast, easy and documented API development with Flask. + - [flask-smorest](https://github.com/marshmallow-code/flask-smorest) - A Flask/Marshmallow-based REST API framework with automatic OpenAPI documentation. + - [apiflask](https://github.com/apiflask/apiflask) - A lightweight Python web API framework based on Flask, supporting marshmallow schemas and Pydantic models. +- Framework Agnostic + - [fastapi](https://github.com/fastapi/fastapi) - A modern, fast, web framework for building APIs with standard Python type hints. + - [strawberry](https://github.com/strawberry-graphql/strawberry) - A GraphQL library that leverages Python type annotations for schema definition. + - [connexion](https://github.com/spec-first/connexion) - A spec-first framework that automatically handles requests based on your OpenAPI specification. +- RPC + - [grpcio](https://github.com/grpc/grpc) - HTTP/2-based RPC framework with Python bindings, built by Google. -*Libraries for working with i18n.* +### Web Servers -* [Babel](http://babel.pocoo.org/en/latest/) - An internationalization library for Python. -* [PyICU](https://github.com/ovalhub/pyicu) - A wrapper of International Components for Unicode C++ library ([ICU](http://site.icu-project.org/)). +_ASGI and WSGI compatible web servers._ -## Job Scheduler +- ASGI + - [uvicorn](https://github.com/Kludex/uvicorn) - A lightning-fast ASGI server implementation. + - [granian](https://github.com/emmett-framework/granian) - A Rust HTTP server for Python applications built on top of Hyper and Tokio, supporting WSGI/ASGI/RSGI. + - [hypercorn](https://github.com/pgjones/hypercorn) - An ASGI and WSGI Server based on Hyper libraries and inspired by Gunicorn. +- WSGI + - [gunicorn](https://github.com/benoitc/gunicorn) - A pre-fork WSGI server with a native ASGI worker, ported from Ruby's Unicorn project. + - [waitress](https://github.com/Pylons/waitress) - Multi-threaded, powers Pyramid. -*Libraries for scheduling jobs.* +### WebSocket -* [APScheduler](http://apscheduler.readthedocs.io/) - A light but powerful in-process task scheduler that lets you schedule functions. -* [django-schedule](https://github.com/thauber/django-schedule) - A calendaring app for Django. -* [doit](http://pydoit.org/) - A task runner and build tool. -* [gunnery](https://github.com/gunnery/gunnery) - Multipurpose task execution tool for distributed systems with web-based interface. -* [Joblib](http://pythonhosted.org/joblib/index.html) - A set of tools to provide lightweight pipelining in Python. -* [Plan](https://github.com/fengsp/plan) - Writing crontab file in Python like a charm. -* [schedule](https://github.com/dbader/schedule) - Python job scheduling for humans. -* [Spiff](https://github.com/knipknap/SpiffWorkflow) - A powerful workflow engine implemented in pure Python. -* [TaskFlow](http://docs.openstack.org/developer/taskflow/) - A Python library that helps to make task execution easy, consistent and reliable. +_Libraries for working with WebSocket._ -## Logging +- [websockets](https://github.com/python-websockets/websockets) - A library for building WebSocket servers and clients with a focus on correctness and simplicity. +- [channels](https://github.com/django/channels) - Brings WebSocket, long-poll HTTP, and other async support to Django. +- [flask-socketio](https://github.com/miguelgrinberg/Flask-SocketIO) - Socket.IO integration for Flask applications. +- [autobahn-python](https://github.com/crossbario/autobahn-python) - WebSocket & WAMP for Python on Twisted and [asyncio](https://docs.python.org/3/library/asyncio.html). -*Libraries for generating and working with logs.* +### Template Engines -* [Eliot](https://github.com/ClusterHQ/eliot) - Logging for complex & distributed systems. -* [logbook](http://logbook.readthedocs.io/en/stable/) - Logging replacement for Python. -* [logging](https://docs.python.org/2/library/logging.html) - (Python standard library) Logging facility for Python. -* [Raven](https://github.com/getsentry/raven-python) - The Python client for Sentry. -* [Sentry](https://pypi.python.org/pypi/sentry) - A realtime logging and aggregation server. +_Libraries for rendering text and HTML from templates._ -## Machine Learning +- [jinja](https://github.com/pallets/jinja) - A modern and designer friendly templating language. +- [mako](https://github.com/sqlalchemy/mako) - Hyperfast and lightweight templating for the Python platform. -*Libraries for Machine Learning. See: [awesome-machine-learning](https://github.com/josephmisiti/awesome-machine-learning#python).* +### Web Asset Management -* [gensim](https://github.com/piskvorky/gensim) - Topic Modelling for Humans. -* [LightFM](https://github.com/lyst/lightfm) - A Python implementation of a number of popular recommendation algorithms. -* [MLlib](http://spark.apache.org/mllib/) - [Apache Spark](http://spark.apache.org/)'s scalable Machine Learning library. -* [NuPIC](https://github.com/numenta/nupic) - Numenta Platform for Intelligent Computing. -* [Pylearn2](https://github.com/lisa-lab/pylearn2) - A Machine Learning library based on [Theano](https://github.com/Theano/Theano). -* [scikit-learn](http://scikit-learn.org/) - The most popular Python library for Machine Learning. -* [surprise](http://surpriselib.com) - A scikit for building and analyzing recommender systems. -* [vowpal_porpoise](https://github.com/josephreisinger/vowpal_porpoise) - A lightweight Python wrapper for [Vowpal Wabbit](https://github.com/JohnLangford/vowpal_wabbit/). -* [xgboost](https://github.com/dmlc/xgboost) - A scalable, portable, and distributed gradient boosting library. +_Tools for managing, storing, compressing and minifying website assets._ -## MapReduce +- [whitenoise](https://github.com/evansd/whitenoise) - Radically simplified static file serving for WSGI applications, with compression and caching headers. +- [django-storages](https://github.com/jschneier/django-storages) - A collection of custom storage back ends for Django. +- [django-compressor](https://github.com/django-compressor/django-compressor) - Compresses linked and inline JavaScript or CSS into a single cached file. -*Frameworks and libraries for MapReduce.* - -* [dpark](https://github.com/douban/dpark) - Python clone of Spark, a MapReduce alike framework in Python. -* [dumbo](https://github.com/klbostee/dumbo) - Python module that allows one to easily write and run Hadoop programs. -* [luigi](https://github.com/spotify/luigi) - A module that helps you build complex pipelines of batch jobs. -* [mrjob](https://github.com/Yelp/mrjob) - Run MapReduce jobs on Hadoop or Amazon Web Services. -* [PySpark](http://spark.apache.org/docs/latest/programming-guide.html) - The Spark Python API. -* [streamparse](https://github.com/Parsely/streamparse) - Run Python code against real-time streams of data. Integrates with [Apache Storm](http://storm.apache.org/). +### Authentication -## Microsoft Windows +_Libraries for implementing authentication schemes._ -*Python programming on Microsoft Windows.* - -* [Python(x,y)](http://python-xy.github.io/) - Scientific-applications-oriented Python Distribution based on Qt and Spyder. -* [pythonlibs](http://www.lfd.uci.edu/~gohlke/pythonlibs/) - Unofficial Windows binaries for Python extension packages. -* [PythonNet](https://github.com/pythonnet/pythonnet) - Python Integration with the .NET Common Language Runtime (CLR). -* [PyWin32](https://sourceforge.net/projects/pywin32/) - Python Extensions for Windows. -* [WinPython](https://winpython.github.io/) - Portable development environment for Windows 7/8. +- OAuth + - [oauthlib](https://github.com/oauthlib/oauthlib) - A generic and thorough implementation of the OAuth request-signing logic. + - [authlib](https://github.com/authlib/authlib) - A comprehensive library for building OAuth, OpenID Connect, and JWT/JWS/JWE/JWK/JWA. + - [django-allauth](https://github.com/pennersr/django-allauth) - Authentication app for Django that "just works." + - [django-oauth-toolkit](https://github.com/django-oauth/django-oauth-toolkit) - An OAuth 2.0 authorization server for Django. +- JWT + - [pyjwt](https://github.com/jpadilla/pyjwt) - JSON Web Token implementation in Python. +- Permissions + - [django-guardian](https://github.com/django-guardian/django-guardian) - Implementation of per-object permissions for Django. + - [django-rules](https://github.com/dfunckt/django-rules) - A tiny but powerful app providing object-level permissions to Django, without requiring a database. -## Miscellaneous +### Admin Panels -*Useful libraries or tools that don't fit in the categories above.* +_Libraries for administrative interfaces._ -* [blinker](https://github.com/jek/blinker) - A fast Python in-process signal/event dispatching system. -* [itsdangerous](https://github.com/pallets/itsdangerous) - Various helpers to pass trusted data to untrusted environments. -* [pluginbase](https://github.com/mitsuhiko/pluginbase) - A simple but flexible plugin system for Python. -* [Pychievements](https://github.com/PacketPerception/pychievements) - A framework for creating and tracking achievements. -* [Tryton](http://www.tryton.org/) - A general purpose business framework. +- [flask-admin](https://github.com/pallets-eco/flask-admin) - Simple and extensible administrative interface framework for Flask. +- [django-unfold](https://github.com/unfoldadmin/django-unfold) - A modern Django admin theme for building dashboards, internal tools, and business applications. +- [sqladmin](https://github.com/smithyhq/sqladmin) - An admin interface for SQLAlchemy models in FastAPI and Starlette. +- [django-grappelli](https://github.com/sehmaschine/django-grappelli) - A jazzy skin for the Django Admin-Interface. -## Natural Language Processing +### CMS -*Libraries for working with human languages.* +_Content Management Systems._ -* [Jieba](https://github.com/fxsjy/jieba) - Chinese text segmentation. -* [langid.py](https://github.com/saffsd/langid.py) - Stand-alone language identification system. -* [NLTK](http://www.nltk.org/) - A leading platform for building Python programs to work with human language data. -* [Pattern](http://www.clips.ua.ac.be/pattern) - A web mining module for the Python. -* [SnowNLP](https://github.com/isnowfy/snownlp) - A library for processing Chinese text. -* [spaCy](https://spacy.io/) - A library for industrial-strength natural language processing in Python and Cython. -* [TextBlob](https://github.com/sloria/TextBlob) - Providing a consistent API for diving into common NLP tasks. -* [TextGrocery](https://github.com/2shou/TextGrocery) - A simple, efficient short-text classification tool based on LibLinear and Jieba. +- [wagtail](https://github.com/wagtail/wagtail) - A Django content management system. +- [django-cms](https://github.com/django-cms/django-cms) - The easy-to-use and developer-friendly enterprise CMS powered by Django. -## Network Virtualization +### ERP -*Tools and libraries for Virtual Networking and SDN (Software Defined Networking).* +_Enterprise resource planning frameworks._ -* [Mininet](http://mininet.org/) - A popular network emulator and API written in Python. -* [POX](https://github.com/noxrepo/pox) - An open source development platform for Python-based Software Defined Networking (SDN) control applications, such as OpenFlow SDN controllers. -* [Pyretic](http://frenetic-lang.org/pyretic/) - A member of the Frenetic family of SDN programming languages that provides powerful abstractions over network switches or emulators. -* [SDX Platform](https://github.com/sdn-ixp/internet2award) - SDN based IXP implementation that leverages Mininet, POX and Pyretic. +- [odoo](https://github.com/odoo/odoo) - A suite of open source business apps: CRM, e-commerce, accounting, inventory, and thousands of community modules. -## Networking +### Static Site Generators -*Libraries for networking programming.* +_Static site generator is a software that takes some text + templates as input and produces HTML files on the output._ -* [asyncio](https://docs.python.org/3/library/asyncio.html) - (Python standard library) Asynchronous I/O, event loop, coroutines and tasks. -* [diesel](https://github.com/dieseldev/diesel) - Greenlet-based event I/O Framework for Python. -* [pulsar](https://github.com/quantmind/pulsar) - Event-driven concurrent framework for Python. -* [pyzmq](http://zeromq.github.io/pyzmq/) - A Python wrapper for the ZeroMQ message library. -* [Twisted](https://twistedmatrix.com/trac/) - An event-driven networking engine. -* [txZMQ](https://github.com/smira/txZMQ) - Twisted based wrapper for the ZeroMQ message library. +- [pelican](https://github.com/getpelican/pelican) - Static site generator that supports Markdown and reST syntax. +- [nikola](https://github.com/getnikola/nikola) - A static website and blog generator. -## News Feed +**HTTP & Scraping** -*Libraries for building user's activities.* +### HTTP Clients -* [django-activity-stream](https://github.com/justquick/django-activity-stream) - Generating generic activity streams from the actions on your site. -* [Stream-Framework](https://github.com/tschellenbach/Stream-Framework) - Building newsfeed and notification systems using Cassandra and Redis. +_Libraries for working with HTTP._ -## ORM +- General + - [requests](https://github.com/psf/requests) - HTTP Requests for Humans. + - [aiohttp](https://github.com/aio-libs/aiohttp) - Asynchronous HTTP client/server framework for asyncio and Python. + - [httpx2](https://github.com/pydantic/httpx2) - HTTP/1.1 and HTTP/2 client with sync and async APIs, maintained by Pydantic ([httpx](https://github.com/encode/httpx) fork). + - [urllib3](https://github.com/urllib3/urllib3) - An HTTP library with thread-safe connection pooling, file post, and more. + - [httpx](https://github.com/encode/httpx) - A next generation HTTP client for Python. +- URL Manipulation + - [yarl](https://github.com/aio-libs/yarl) - Yet another URL library. -*Libraries that implement Object-Relational Mapping or data mapping techniques.* +### Web Scraping -* Relational Databases - * [Django Models](https://docs.djangoproject.com/en/dev/topics/db/models/) - A part of Django. - * [SQLAlchemy](http://www.sqlalchemy.org/) - The Python SQL Toolkit and Object Relational Mapper. - * [awesome-sqlalchemy](https://github.com/dahlia/awesome-sqlalchemy) - * [Orator](https://orator-orm.com) - The Orator ORM provides a simple yet beautiful ActiveRecord implementation. - * [Peewee](https://github.com/coleifer/peewee) - A small, expressive ORM. - * [PonyORM](https://ponyorm.com/) - ORM that provides a generator-oriented interface to SQL. - * [pyDAL](https://github.com/web2py/pydal/) - A pure Python Database Abstraction Layer. - * [python-sql](https://pypi.python.org/pypi/python-sql) - Write SQL queries pythonically. -* NoSQL Databases - * [django-mongodb-engine](https://github.com/django-nonrel/mongodb-engine) - Django MongoDB Backend. - * [flywheel](https://github.com/mathcamp/flywheel) - Object mapper for Amazon DynamoDB. - * [hot-redis](https://github.com/stephenmcd/hot-redis) - Rich Python data types for Redis. - * [MongoEngine](http://mongoengine.org/) - A Python Object-Document-Mapper for working with MongoDB. - * [PynamoDB](https://github.com/jlafon/PynamoDB) - A Pythonic interface for [Amazon DynamoDB](https://aws.amazon.com/dynamodb/). - * [redisco](https://github.com/kiddouk/redisco) - A Python Library for Simple Models and Containers Persisted in Redis. -* Others - * [butterdb](https://github.com/Widdershin/butterdb) - A Python ORM for Google Drive Spreadsheets. - * [dataset](https://github.com/pudo/dataset) - A JSON-based database. - -## Package Management +_Libraries to automate web scraping and extract web content._ -*Libraries for package and dependency management.* +- Frameworks + - [browser-use](https://github.com/browser-use/browser-use) - Make websites accessible for AI agents with easy browser automation. + - [scrapy](https://github.com/scrapy/scrapy) - A fast high-level web crawling and scraping framework. + - [crawl4ai](https://github.com/unclecode/crawl4ai) - An open-source, LLM-friendly web crawler that provides lightning-fast, structured data extraction specifically designed for AI agents. + - [stagehand](https://github.com/browserbase/stagehand) - A fast and token-efficient browser automation SDK to extract data and perform self-healing actions on web pages. + - [scrapling](https://github.com/D4Vinci/Scrapling) - An adaptive scraping framework whose selectors relocate elements after page changes, with stealth fetchers and spiders. +- Content Extraction + - [markdownify](https://github.com/matthewwithanm/python-markdownify) - Convert HTML to Markdown, with customizable tag handling. + - [feedparser](https://github.com/kurtmckee/feedparser) - Universal feed parser. + - [trafilatura](https://github.com/adbar/trafilatura) - A tool for gathering text and metadata from the web, with built-in content filtering. -* [pip](https://pip.pypa.io/en/stable/) - The Python package and dependency manager. - * [Python Package Index](https://pypi.python.org/pypi) -* [conda](https://github.com/conda/conda/) - Cross-platform, Python-agnostic binary package manager. -* [Curdling](http://clarete.li/curdling/) - Curdling is a command line tool for managing Python packages. -* [pip-tools](https://github.com/nvie/pip-tools) - A set of tools to keep your pinned Python dependencies fresh. -* [wheel](http://pythonwheels.com/) - The new standard of Python distribution and are intended to replace eggs. - -## Package Repositories +### Email -*Local PyPI repository server and proxies.* +_Libraries for sending email._ -* [warehouse](https://github.com/pypa/warehouse) - Next generation Python Package Repository (PyPI). - * [Warehouse](https://pypi.org/) -* [bandersnatch](https://bitbucket.org/pypa/bandersnatch) - PyPI mirroring tool provided by Python Packaging Authority (PyPA). -* [devpi](http://doc.devpi.net/latest/) - PyPI server and packaging/testing/release tool. -* [localshop](https://github.com/mvantellingen/localshop) - Local PyPI server (custom packages and auto-mirroring of pypi). +- [aiosmtplib](https://github.com/cole/aiosmtplib) - An asyncio SMTP client. +- [django-anymail](https://github.com/anymail/django-anymail) - Django email backends and webhooks for transactional email services such as Amazon SES, Brevo, Mailgun, Postmark, and Resend. +- [yagmail](https://github.com/kootenpv/yagmail) - Yet another Gmail/SMTP client. -## Permissions +**Database & Storage** -*Libraries that allow or deny users access to data or functionality.* +### ORM -* [Carteblanche](https://github.com/neuman/python-carteblanche/) - Module to align code with thoughts of users and designers. Also magically handles navigation and permissions. -* [django-guardian](https://github.com/django-guardian/django-guardian) - Implementation of per object permissions for Django 1.2+ -* [django-rules](https://github.com/dfunckt/django-rules) - A tiny but powerful app providing object-level permissions to Django, without requiring a database. +_Libraries that implement Object-Relational Mapping or data mapping techniques._ -## Processes +- Relational Databases + - [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) - The Python SQL Toolkit and Object Relational Mapper. + - [awesome-sqlalchemy](https://github.com/dahlia/awesome-sqlalchemy) + - [django.db.models](https://github.com/django/django) - (part of Django) The Django [ORM](https://docs.djangoproject.com/en/stable/topics/db/models/). + - [peewee](https://github.com/coleifer/peewee) - A small, expressive ORM. + - [sqlmodel](https://github.com/fastapi/sqlmodel) - SQLModel is based on Python type annotations, and powered by Pydantic and SQLAlchemy. +- NoSQL Databases + - [pynamodb](https://github.com/pynamodb/PynamoDB) - A Pythonic interface for [Amazon DynamoDB](https://aws.amazon.com/dynamodb/). + - [mongoengine](https://github.com/MongoEngine/mongoengine) - A Python Object-Document-Mapper for working with MongoDB. + - [beanie](https://github.com/BeanieODM/beanie) - An asynchronous Python object-document mapper (ODM) for MongoDB. + - [django-mongodb-backend](https://github.com/mongodb/django-mongodb-backend) - Official MongoDB database backend for Django. -*Libraries for starting and communicating with OS processes.* +### Database Drivers -* [envoy](https://github.com/kennethreitz/envoy) - Python [subprocess](https://docs.python.org/2/library/subprocess.html) for Humans™. -* [sarge](http://sarge.readthedocs.io/) - Yet another wrapper for subprocess. -* [sh](https://github.com/amoffat/sh) - A full-fledged subprocess replacement for Python. +_Libraries for connecting and operating databases._ -## Queue - -*Libraries for working with event and task queues.* - -* [celery](http://www.celeryproject.org/) - An asynchronous task queue/job queue based on distributed message passing. -* [huey](https://github.com/coleifer/huey) - Little multi-threaded task queue. -* [mrq](https://github.com/pricingassistant/mrq) - Mr. Queue - A distributed worker task queue in Python using Redis & gevent. -* [rq](http://python-rq.org/) - Simple job queues for Python. -* [simpleq](https://github.com/rdegges/simpleq) - A simple, infinitely scalable, Amazon SQS based queue. - -## RESTful API - -*Libraries for developing RESTful APIs.* - -* Django - * [django-formapi](https://github.com/5monkeys/django-formapi) - Create JSON APIs with Django's form validation. - * [django-rest-framework](http://www.django-rest-framework.org/) - A powerful and flexible toolkit to build web APIs. - * [django-tastypie](http://tastypieapi.org/) - Creating delicious APIs for Django apps. -* Flask - * [eve](https://github.com/nicolaiarocci/eve) - REST API framework powered by Flask, MongoDB and good intentions. - * [flask-api-utils](https://github.com/marselester/flask-api-utils) - Taking care of API representation and authentication for Flask. - * [flask-api](http://www.flaskapi.org/) - Browsable Web APIs for Flask. - * [flask-restful](https://github.com/flask-restful/flask-restful) - Quickly building REST APIs for Flask. - * [flask-restless](https://github.com/jfinkels/flask-restless) - Generating RESTful APIs for database models defined with SQLAlchemy. -* Pyramid - * [cornice](https://github.com/mozilla-services/cornice) - A RESTful framework for Pyramid. -* Framework agnostic - * [falcon](http://falconframework.org/) - A high-performance framework for building cloud APIs and web app backends. - * [hug](https://github.com/timothycrosley/hug) - A Python3 framework for cleanly exposing APIs over HTTP and the Command Line with automatic documentation and validation. - * [restless](https://github.com/toastdriven/restless) - Framework agnostic REST framework based on lessons learned from Tastypie. - * [ripozo](https://github.com/vertical-knowledge/ripozo) - Quickly creating REST/HATEOAS/Hypermedia APIs. - * [sandman](https://github.com/jeffknupp/sandman) - Automated REST APIs for existing database-driven systems. - -## RPC Servers - -*RPC-compatible servers.* - -* [SimpleJSONRPCServer](https://github.com/joshmarshall/jsonrpclib/) - This library is an implementation of the JSON-RPC specification. -* [SimpleXMLRPCServer](https://docs.python.org/2/library/simplexmlrpcserver.html) - (Python standard library) Simple XML-RPC server implementation, single-threaded. -* [zeroRPC](https://github.com/0rpc/zerorpc-python) - zerorpc is a flexible RPC implementation based on [ZeroMQ](http://zeromq.org/) and [MessagePack](http://msgpack.org/). - -## Science - -*Libraries for scientific computing.* - -* [astropy](http://www.astropy.org/) - A community Python library for Astronomy. -* [bcbio-nextgen](https://github.com/chapmanb/bcbio-nextgen) - Providing best-practice pipelines for fully automated high throughput sequencing analysis. -* [bccb](https://github.com/chapmanb/bcbb) - Collection of useful code related to biological analysis. -* [Biopython](http://biopython.org/wiki/Main_Page) - Biopython is a set of freely available tools for biological computation. -* [cclib](http://cclib.github.io/) - A library for parsing and interpreting the results of computational chemistry packages. -* [NetworkX](https://networkx.github.io/) - A high-productivity software for complex networks. -* [NIPY](http://nipy.org) - A collection of neuroimaging toolkits. -* [NumPy](http://www.numpy.org/) - A fundamental package for scientific computing with Python. -* [Open Babel](http://openbabel.org/wiki/Main_Page) - A chemical toolbox designed to speak the many languages of chemical data. -* [PyDy](http://www.pydy.org/) - Short for Python Dynamics, used to assist with workflow in the modeling of dynamic motion. -* [PyMC](https://github.com/pymc-devs/pymc3) - Markov Chain Monte Carlo sampling toolkit. -* [RDKit](http://www.rdkit.org/) - Cheminformatics and Machine Learning Software. -* [SciPy](http://www.scipy.org/) - A Python-based ecosystem of open-source software for mathematics, science, and engineering. -* [statsmodels](https://github.com/statsmodels/statsmodels) - Statistical modeling and econometrics in Python. -* [SymPy](https://github.com/sympy/sympy) - A Python library for symbolic mathematics. -* [Zipline](https://github.com/quantopian/zipline) - A Pythonic algorithmic trading library. - -## Search - -*Libraries and software for indexing and performing search queries on data.* - -* [django-haystack](https://github.com/django-haystack/django-haystack) - Modular search for Django. -* [elasticsearch-dsl-py](https://github.com/elastic/elasticsearch-dsl-py) - The official high-level Python client for Elasticsearch. -* [elasticsearch-py](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html) - The official low-level Python client for [Elasticsearch](https://www.elastic.co/products/elasticsearch). -* [esengine](https://github.com/catholabs/esengine) - ElasticSearch ODM (Object Document Mapper) for Python. -* [solrpy](https://github.com/edsu/solrpy) - A Python client for [solr](http://lucene.apache.org/solr/). -* [Whoosh](http://whoosh.readthedocs.io/) - A fast, pure Python search engine library. - -## Serialization - -*Libraries for serializing complex data types* - -* [marshmallow](https://github.com/marshmallow-code/marshmallow) - marshmallow is an ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, to and from native Python datatypes. - -## Serverless Frameworks - -*Frameworks for developing serverless Python code.* - -* [apex](https://github.com/apex/apex) - Build, deploy, and manage [AWS Lambda](https://aws.amazon.com/lambda/) functions with ease. -* [python-lambda](https://github.com/nficano/python-lambda) - A toolkit for developing and deploying Python code in AWS Lambda. -* [Zappa](https://github.com/Miserlou/Zappa) - A tool for deploying WSGI applications on AWS Lambda and API Gateway. - -## Specific Formats Processing - -*Libraries for parsing and manipulating specific text formats.* - -* General - * [tablib](https://github.com/kennethreitz/tablib) - A module for Tabular Datasets in XLS, CSV, JSON, YAML. -* Office - * [Marmir](https://github.com/brianray/mm) - Takes Python data structures and turns them into spreadsheets. - * [openpyxl](https://openpyxl.readthedocs.io/) - A library for reading and writing Excel 2010 xlsx/xlsm/xltx/xltm files. - * [pyexcel](https://github.com/pyexcel/pyexcel) - Providing one API for reading, manipulating and writing csv, ods, xls, xlsx and xlsm files. - * [python-docx](https://github.com/python-openxml/python-docx) - Reads, queries and modifies Microsoft Word 2007/2008 docx files. - * [relatorio](http://relatorio.tryton.org/) - Templating OpenDocument files. - * [unoconv](https://github.com/dagwieers/unoconv) - Convert between any document format supported by LibreOffice/OpenOffice. - * [XlsxWriter](https://xlsxwriter.readthedocs.io) - A Python module for creating Excel .xlsx files. - * [xlwings](http://xlwings.org/) - A BSD-licensed library that makes it easy to call Python from Excel and vice versa. - * [xlwt](https://github.com/python-excel/xlwt) / [xlrd](https://github.com/python-excel/xlrd) - Writing and reading data and formatting information from Excel files. -* PDF - * [PDFMiner](https://github.com/euske/pdfminer) - A tool for extracting information from PDF documents. - * [PyPDF2](https://github.com/mstamy2/PyPDF2) - A library capable of splitting, merging and transforming PDF pages. - * [ReportLab](http://www.reportlab.com/opensource/) - Allowing Rapid creation of rich PDF documents. -* Markdown - * [Mistune](https://github.com/lepture/mistune) - Fastest and full featured pure Python parsers of Markdown. - * [Python-Markdown](https://github.com/waylan/Python-Markdown) - A Python implementation of John Gruber’s Markdown. -* YAML - * [PyYAML](http://pyyaml.org/) - YAML implementations for Python. -* CSV - * [csvkit](https://github.com/wireservice/csvkit) - Utilities for converting to and working with CSV. -* Archive - * [unp](https://github.com/mitsuhiko/unp) - A command line tool that can unpack archives easily. - -## Static Site Generator - -*Static site generator is a software that takes some text + templates as input and produces HTML files on the output.* - -* [Cactus](https://github.com/koenbok/Cactus/) - Static site generator for designers. -* [Hyde](http://hyde.github.io/) - Jinja2-based static web site generator. -* [Lektor](https://www.getlektor.com/) - An easy to use static CMS and blog engine. -* [Nikola](https://www.getnikola.com/) - A static website and blog generator. -* [Pelican](http://blog.getpelican.com/) - Uses Markdown or ReST for content and Jinja 2 for themes. Supports DVCS, Disqus. AGPL. -* [Tinkerer](http://tinkerer.me/) - Tinkerer is a blogging engine/.static website generator powered by Sphinx. - -## Tagging - -*Libraries for tagging items.* - -* [django-taggit](https://github.com/alex/django-taggit) - Simple tagging for Django. - -## Template Engine - -*Libraries and tools for templating and lexing.* - -* [Genshi](https://genshi.edgewall.org/) - Python templating toolkit for generation of web-aware output. -* [Jinja2](https://github.com/pallets/jinja) - A modern and designer friendly templating language. -* [Mako](http://www.makotemplates.org/) - Hyperfast and lightweight templating for the Python platform. - -## Testing - -*Libraries for testing codebases and generating test data.* - -* Testing Frameworks - * [hypothesis](https://github.com/HypothesisWorks/hypothesis-python) - Hypothesis is an advanced Quickcheck style property based testing library. - * [mamba](http://nestorsalceda.github.io/mamba/) - The definitive testing tool for Python. Born under the banner of BDD. - * [nose](https://github.com/nose-devs/nose) - A nicer unittest for Python. - * [nose2](https://github.com/nose-devs/nose2) - The successor to nose, based on unittest2. - * [pytest](http://pytest.org/latest/) - A mature full-featured Python testing tool. - * [Robot Framework](https://github.com/robotframework/robotframework) - A generic test automation framework. - * [unittest](https://docs.python.org/2/library/unittest.html) - (Python standard library) Unit testing framework. -* Test Runners - * [green](https://github.com/CleanCut/green) - A clean, colorful test runner. - * [tox](https://tox.readthedocs.io/) - Auto builds and tests distributions in multiple Python versions -* GUI / Web Testing - * [locust](https://github.com/locustio/locust) - Scalable user load testing tool written in Python. - * [PyAutoGUI](https://github.com/asweigart/pyautogui) - PyAutoGUI is a cross-platform GUI automation Python module for human beings. - * [Selenium](https://pypi.python.org/pypi/selenium) - Python bindings for [Selenium](http://www.seleniumhq.org/) WebDriver. - * [sixpack](https://github.com/seatgeek/sixpack) - A language-agnostic A/B Testing framework. - * [splinter](https://github.com/cobrateam/splinter) - Open source tool for testing web applications. -* Mock - * [doublex](https://pypi.python.org/pypi/doublex) - Powerful test doubles framework for Python. - * [freezegun](https://github.com/spulec/freezegun) - Travel through time by mocking the datetime module. - * [httmock](https://github.com/patrys/httmock) - A mocking library for requests for Python 2.6+ and 3.2+. - * [httpretty](https://github.com/gabrielfalcao/HTTPretty) - HTTP request mock tool for Python. - * [mock](https://docs.python.org/3/library/unittest.mock.html) - (Python standard library) A mocking and patching library. - * [responses](https://github.com/getsentry/responses) - A utility library for mocking out the requests Python library. - * [VCR.py](https://github.com/kevin1024/vcrpy) - Record and replay HTTP interactions on your tests. -* Object Factories - * [factory_boy](https://github.com/rbarrois/factory_boy) - A test fixtures replacement for Python. - * [mixer](https://github.com/klen/mixer) - Another fixtures replacement. Supported Django, Flask, SQLAlchemy, Peewee and etc. - * [model_mommy](https://github.com/vandersonmota/model_mommy) - Creating random fixtures for testing in Django. -* Code Coverage - * [coverage](https://pypi.python.org/pypi/coverage) - Code coverage measurement. -* Fake Data - * [church](https://github.com/lk-geimfari/church) - is a Python library that help you generate fake data. - * [fake2db](https://github.com/emirozer/fake2db) - Fake database generator. - * [faker](https://github.com/joke2k/faker) - A Python package that generates fake data. - * [radar](https://pypi.python.org/pypi/radar) - Generate random datetime / time. -* Error Handler - * [FuckIt.py](https://github.com/ajalt/fuckitpy) - FuckIt.py uses state-of-the-art technology to make sure your Python code runs whether it has any right to or not. - -## Text Processing - -*Libraries for parsing and manipulating plain texts.* - -* General - * [chardet](https://github.com/chardet/chardet) - Python 2/3 compatible character encoding detector. - * [difflib](https://docs.python.org/2/library/difflib.html) - (Python standard library) Helpers for computing deltas. - * [ftfy](https://github.com/LuminosoInsight/python-ftfy) - Makes Unicode text less broken and more consistent automagically. - * [fuzzywuzzy](https://github.com/seatgeek/fuzzywuzzy) - Fuzzy String Matching. - * [Levenshtein](https://github.com/ztane/python-Levenshtein/) - Fast computation of Levenshtein distance and string similarity. - * [pangu.py](https://github.com/vinta/pangu.py) - Spacing texts for CJK and alphanumerics. - * [pyfiglet](https://github.com/pwaller/pyfiglet) - An implementation of figlet written in Python. - * [shortuuid](https://github.com/stochastic-technologies/shortuuid) - A generator library for concise, unambiguous and URL-safe UUIDs. - * [unidecode](https://pypi.python.org/pypi/Unidecode) - ASCII transliterations of Unicode text. - * [uniout](https://github.com/moskytw/uniout) - Print readable chars instead of the escaped string. - * [xpinyin](https://github.com/lxneng/xpinyin) - A library to translate Chinese hanzi (漢字) to pinyin (拼音). -* Slugify - * [awesome-slugify](https://github.com/dimka665/awesome-slugify) - A Python slugify library that can preserve unicode. - * [python-slugify](https://github.com/un33k/python-slugify) - A Python slugify library that translates unicode to ASCII. - * [unicode-slugify](https://github.com/mozilla/unicode-slugify) - A slugifier that generates unicode slugs with Django as a dependency. -* Parser - * [phonenumbers](https://github.com/daviddrysdale/python-phonenumbers) - Parsing, formatting, storing and validating international phone numbers. - * [PLY](http://www.dabeaz.com/ply/) - Implementation of lex and yacc parsing tools for Python - * [Pygments](http://pygments.org/) - A generic syntax highlighter. - * [pyparsing](http://pyparsing.wikispaces.com/) - A general purpose framework for generating parsers. - * [python-nameparser](https://github.com/derek73/python-nameparser) - Parsing human names into their individual components. - * [python-user-agents](https://github.com/selwin/python-user-agents) - Browser user agent parser. - * [sqlparse](https://github.com/andialbrecht/sqlparse) - A non-validating SQL parser. - -## Third-party APIs - -*Libraries for accessing third party services APIs. See: [List of Python API Wrappers and Libraries](https://github.com/realpython/list-of-python-api-wrappers).* - -* [apache-libcloud](https://libcloud.apache.org/) - One Python library for all clouds. -* [boto3](https://github.com/boto/boto3) - Python interface to Amazon Web Services. -* [django-wordpress](https://github.com/sunlightlabs/django-wordpress/) - WordPress models and views for Django. -* [facebook-sdk](https://github.com/mobolic/facebook-sdk) - Facebook Platform Python SDK. -* [facepy](https://github.com/jgorset/facepy) - Facepy makes it really easy to interact with Facebook's Graph API -* [gmail](https://github.com/charlierguo/gmail) - A Pythonic interface for Gmail. -* [google-api-python-client](https://github.com/google/google-api-python-client) - Google APIs Client Library for Python. -* [gspread](https://github.com/burnash/gspread) - Google Spreadsheets Python API. -* [twython](https://github.com/ryanmcgrath/twython) - A Python wrapper for the Twitter API. - -## URL Manipulation - -*Libraries for parsing URLs.* - -* [furl](https://github.com/gruns/furl) - A small Python library that makes manipulating URLs simple. -* [purl](https://github.com/codeinthehole/purl) - A simple, immutable URL class with a clean API for interrogation and manipulation. -* [pyshorteners](https://github.com/ellisonleao/pyshorteners) - A pure Python URL shortening lib. -* [short_url](https://github.com/Alir3z4/python-short_url) - Python implementation for generating Tiny URL and bit.ly-like URLs. -* [webargs](https://github.com/sloria/webargs) - A friendly library for parsing HTTP request arguments, with built-in support for popular web frameworks, including Flask, Django, Bottle, Tornado, and Pyramid. - -## Video - -*Libraries for manipulating video and GIFs.* - -* [moviepy](http://zulko.github.io/moviepy/) - A module for script-based movie editing with many formats, including animated GIFs. -* [scikit-video](https://github.com/aizvorski/scikit-video) - Video processing routines for SciPy. - -## WSGI Servers - -*WSGI-compatible web servers.* - -* [bjoern](https://pypi.python.org/pypi/bjoern) - Asynchronous, very fast and written in C. -* [fapws3](http://www.fapws.org/) - Asynchronous (network side only), written in C. -* [gunicorn](https://pypi.python.org/pypi/gunicorn) - Pre-forked, partly written in C. -* [meinheld](https://pypi.python.org/pypi/meinheld) - Asynchronous, partly written in C. -* [netius](https://github.com/hivesolutions/netius) - Asynchronous, very fast. -* [paste](http://pythonpaste.org/) - Multi-threaded, stable, tried and tested. -* [rocket](https://pypi.python.org/pypi/rocket) - Multi-threaded. -* [uWSGI](https://uwsgi-docs.readthedocs.io/) - A project aims at developing a full stack for building hosting services, written in C. -* [waitress](https://waitress.readthedocs.io/) - Multi-threaded, powers Pyramid. -* [Werkzeug](http://werkzeug.pocoo.org/) - A WSGI utility library for Python that powers Flask and can easily be embedded into your own projects. - -## Web Content Extracting - -*Libraries for extracting web contents.* - -* [Haul](https://github.com/vinta/Haul) - An Extensible Image Crawler. -* [html2text](https://github.com/Alir3z4/html2text) - Convert HTML to Markdown-formatted text. -* [lassie](https://github.com/michaelhelmick/lassie) - Web Content Retrieval for Humans. -* [micawber](https://github.com/coleifer/micawber) - A small library for extracting rich content from URLs. -* [newspaper](https://github.com/codelucas/newspaper) - News extraction, article extraction and content curation in Python. -* [opengraph](https://github.com/erikriver/opengraph) - A Python module to parse the Open Graph Protocol -* [python-goose](https://github.com/grangier/python-goose) - HTML Content/Article Extractor. -* [python-readability](https://github.com/buriy/python-readability) - Fast Python port of arc90's readability tool. -* [sanitize](https://github.com/Alir3z4/python-sanitize) - Bringing sanity to world of messed-up data. -* [sumy](https://github.com/miso-belica/sumy) - A module for automatic summarization of text documents and HTML pages. -* [textract](https://github.com/deanmalmgren/textract) - Extract text from any document, Word, PowerPoint, PDFs, etc. - -## Web Crawling - -*Libraries for scraping websites.* - -* [cola](https://github.com/chineking/cola) - A distributed crawling framework. -* [Demiurge](https://github.com/matiasb/demiurge) - PyQuery-based scraping micro-framework. -* [feedparser](http://pythonhosted.org/feedparser/) - Universal feed parser. -* [Grab](http://grablib.org/) - Site scraping framework. -* [MechanicalSoup](https://github.com/hickford/MechanicalSoup) - A Python library for automating interaction with websites. -* [portia](https://github.com/scrapinghub/portia) - Visual scraping for Scrapy. -* [pyspider](https://github.com/binux/pyspider) - A powerful spider system. -* [RoboBrowser](https://github.com/jmcarp/robobrowser) - A simple, Pythonic library for browsing the web without a standalone web browser. -* [Scrapy](http://scrapy.org/) - A fast high-level screen scraping and web crawling framework. - -## Web Frameworks - -*Full stack web frameworks.* - -* [Bottle](http://bottlepy.org/docs/dev/index.html) - A fast, simple and lightweight WSGI micro web-framework. -* [CherryPy](http://www.cherrypy.org/) - A minimalist Python web framework, HTTP/1.1-compliant and WSGI thread-pooled. -* [Django](https://www.djangoproject.com/) - The most popular web framework in Python. - * [awesome-django](https://github.com/rosarior/awesome-django) -* [Flask](http://flask.pocoo.org/) - A microframework for Python. - * [awesome-flask](https://github.com/humiaozuzu/awesome-flask) -* [Pyramid](http://www.pylonsproject.org/) - A small, fast, down-to-earth, open source Python web framework. - * [awesome-pyramid](https://github.com/uralbash/awesome-pyramid) -* [Tornado](http://www.tornadoweb.org/en/latest/) - A Web framework and asynchronous networking library. -* [TurboGears](http://www.turbogears.org/) - A microframework that can scale up to a full stack solution. -* [Web2py](http://www.web2py.com/) - Full-stack enterprise framework for secure database-driven web-based applications. - * [GitHub Web2py](https://github.com/web2py) - -## WebSocket +- MySQL - [awesome-mysql](https://github.com/shlomi-noach/awesome-mysql) + - [pymysql](https://github.com/PyMySQL/PyMySQL) - A pure-Python MySQL and MariaDB client library, based on PEP 249. + - [mysqlclient](https://github.com/PyMySQL/mysqlclient) - MySQL and MariaDB connector ([MySQLdb1](https://github.com/farcepest/MySQLdb1) fork). +- PostgreSQL - [awesome-postgres](https://github.com/dhamaniasad/awesome-postgres) + - [psycopg](https://github.com/psycopg/psycopg) - A PostgreSQL adapter for Python, the successor to psycopg2. + - [asyncpg](https://github.com/MagicStack/asyncpg) - A fast PostgreSQL Database Client Library for Python/asyncio. + - [pgvector](https://github.com/pgvector/pgvector-python) - pgvector support for Python, adding vector types and similarity search to psycopg, asyncpg, SQLAlchemy, and Django. +- SQLite - [awesome-sqlite](https://github.com/planetopendata/awesome-sqlite) + - [sqlite3](https://docs.python.org/3/library/sqlite3.html) - (Python standard library) SQLite interface compliant with DB-API 2.0. + - [apsw](https://github.com/rogerbinns/apsw) - Another Python SQLite Wrapper, exposing the complete SQLite C API. + - [sqlite-utils](https://github.com/simonw/sqlite-utils) - Python CLI utility and library for manipulating SQLite databases. +- ClickHouse + - [clickhouse-connect](https://github.com/ClickHouse/clickhouse-connect) - The official ClickHouse client, with SQLAlchemy and Superset connectors. + - [clickhouse-driver](https://github.com/mymarilyn/clickhouse-driver) - Python driver with native interface for ClickHouse. +- Other Relational Databases + - [pyodbc](https://github.com/mkleehammer/pyodbc) - An ODBC bridge for connecting to SQL Server and any other ODBC-accessible database. + - [oracledb](https://github.com/oracle/python-oracledb) - The official Python driver for Oracle Database, successor to cx_Oracle. + - [mssql-python](https://github.com/microsoft/mssql-python) - Official Microsoft driver for SQL Server and Azure SQL, built on ODBC for high performance. +- NoSQL Databases + - [redis](https://github.com/redis/redis-py) - The Python client for Redis. + - [pymongo](https://github.com/mongodb/mongo-python-driver) - The official Python client for MongoDB. + - [cassandra-driver](https://github.com/apache/cassandra-python-driver) - The Python Driver for Apache Cassandra. + +### Database + +_In-process databases usable directly from Python._ + +- Analytical + - [duckdb](https://github.com/duckdb/duckdb) - An in-process SQL OLAP database management system; optimized for analytics and fast queries, similar to SQLite but for analytical workloads. + - [chdb](https://github.com/chdb-io/chdb) - In-process OLAP SQL engine with the full ClickHouse dialect, zero-copy pandas/Arrow interop, and federation to remote ClickHouse clusters via `remoteSecure()`. +- Vector + - [lancedb](https://github.com/lancedb/lancedb) - A developer-friendly embedded retrieval database for multimodal AI. + - [chromadb](https://github.com/chroma-core/chroma) - An open-source embedding database for building AI applications with embeddings and semantic search. + - [zvec](https://github.com/alibaba/zvec) - A lightweight, in-process vector database that embeds directly into applications. +- Key-Value & Document + - [tinydb](https://github.com/msiemens/tinydb) - A tiny, document-oriented database. + +### Caching + +_Libraries for caching data._ + +- General + - [cachetools](https://github.com/tkem/cachetools) - Extensible memoizing collections and decorators. + - [diskcache](https://github.com/grantjenks/python-diskcache) - SQLite and file backed cache backend, compatible with Django. + - [dogpile.cache](https://github.com/sqlalchemy/dogpile.cache) - A caching API for Redis, Memcached, and other backends, with coordinated value regeneration. +- HTTP Responses + - [hishel](https://github.com/karpetrosyan/hishel) - RFC 9111 compliant HTTP caching for clients like httpx and requests and servers like FastAPI, with sync and async support. +- Django + - [django-redis](https://github.com/jazzband/django-redis) - Full-featured Redis cache backend for Django. + - [django-cacheops](https://github.com/Suor/django-cacheops) - A slick ORM cache with automatic granular event-driven invalidation. + +### Search + +_Libraries and software for indexing and performing search queries on data._ + +- Full-Text Search + - [opensearch-py](https://github.com/opensearch-project/opensearch-py) - The official low-level Python client for [OpenSearch](https://opensearch.org/). + - [elasticsearch](https://github.com/elastic/elasticsearch-py) - The official low-level Python client for [Elasticsearch](https://www.elastic.co/elasticsearch). + - [meilisearch](https://github.com/meilisearch/meilisearch-python) - The official Python client for the [Meilisearch](https://www.meilisearch.com/) search engine. + - [django-haystack](https://github.com/django-haystack/django-haystack) - Modular search for Django. +- Vector Search + - [faiss-cpu](https://github.com/facebookresearch/faiss) - A library for efficient similarity search and clustering of dense vectors. + +### Serialization + +_Libraries for serializing complex data types._ + +- Data Formats + - [msgpack](https://github.com/msgpack/msgpack-python) - MessagePack serializer implementation for Python. + - [orjson](https://github.com/ijl/orjson) - Fast, correct JSON library. + - [marshmallow](https://github.com/marshmallow-code/marshmallow) - A lightweight library for converting complex objects to and from simple Python datatypes. + - [msgspec](https://github.com/msgspec/msgspec) - A fast serialization and validation library with built-in support for JSON, MessagePack, YAML, and TOML. +- Object Pickling + - [cloudpickle](https://github.com/cloudpipe/cloudpickle) - Extends pickle to functions, lambdas, and interactively defined classes, for shipping code to remote workers. + - [dill](https://github.com/uqfoundation/dill) - Extends pickle to serialize almost any Python object, including closures and whole interpreter sessions. + +**Data & Science** + +### Data Analysis + +_Libraries for data analysis._ + +- [pandas](https://github.com/pandas-dev/pandas) - A library providing high-performance, easy-to-use data structures and data analysis tools. +- [polars](https://github.com/pola-rs/polars) - A fast DataFrame library implemented in Rust with a Python API. +- [ibis-framework](https://github.com/ibis-project/ibis) - A portable Python dataframe library with a single API for 20+ backends. + +### Data Ingestion / ETL + +_Libraries for data extraction, transformation, and loading pipelines across multiple sources and destinations._ + +- General + - [awswrangler](https://github.com/aws/aws-sdk-pandas) - Pandas integration with AWS services like Athena, Glue, Redshift, S3, and DynamoDB. + - [dlt](https://github.com/dlt-hub/dlt) - A Python library for building data pipelines with automatic schema inference, incremental loading, and support for multiple sources and destinations. +- Financial Data + - [yfinance](https://github.com/ranaroussi/yfinance) - Easy Pythonic way to download market and financial data from Yahoo Finance. + - [akshare](https://github.com/akfamily/akshare) - A financial data interface library, with data provided for academic research only. + - [edgartools](https://github.com/dgunning/edgartools) - Library for downloading structured data from SEC EDGAR filings and XBRL financial statements. + - [openbb](https://github.com/openbq-org/OpenBB) - A financial data platform for analysts, quants and AI agents. + +### Data Validation + +_Libraries for validating data._ + +- [pydantic](https://github.com/pydantic/pydantic) - Data validation using Python type hints. +- [jsonschema](https://github.com/python-jsonschema/jsonschema) - An implementation of [JSON Schema](https://json-schema.org/) for Python. +- [great-expectations](https://github.com/fivetran/great_expectations) - A data quality framework for validating, documenting, and profiling data with declarative expectations. +- [pandera](https://github.com/unionai-oss/pandera) - A data validation library for dataframes, with support for pandas, polars, PySpark, and more. + +### Data Visualization + +_Libraries for visualizing data. Also see [awesome-javascript](https://github.com/sorrycc/awesome-javascript#data-visualization)._ + +- Plotting + - [matplotlib](https://github.com/matplotlib/matplotlib) - A comprehensive library for creating static, animated, and interactive visualizations. + - [plotly](https://github.com/plotly/plotly.py) - Interactive graphing library for Python. + - [seaborn](https://github.com/mwaskom/seaborn) - Statistical data visualization using Matplotlib. + - [altair](https://github.com/vega/altair) - Declarative statistical visualization library for Python. + - [bokeh](https://github.com/bokeh/bokeh) - Interactive Web Plotting for Python. +- Specialized + - [graphviz](https://github.com/xflr6/graphviz) - Simple Python interface for creating and rendering Graphviz graphs. + - [cartopy](https://github.com/SciTools/cartopy) - A cartographic Python library with matplotlib support. + - [graphify](https://github.com/Graphify-Labs/graphify) - Turn any folder of code, SQL schemas, docs, papers, images, or videos into a queryable knowledge graph. +- Dashboards and Apps + - [streamlit](https://github.com/streamlit/streamlit) - A framework which lets you build dashboards, generate reports, or create chat apps in minutes. + - [dash](https://github.com/plotly/dash) - A framework for building data apps and dashboards in pure Python, built on Plotly. + - [gradio](https://github.com/gradio-app/gradio) - Build and share machine learning apps, all in Python. + +### Geolocation + +_Libraries for geocoding addresses and working with latitudes and longitudes._ + +- [geopandas](https://github.com/geopandas/geopandas) - Python tools for geographic data (GeoSeries/GeoDataFrame) built on pandas. +- [geopy](https://github.com/geopy/geopy) - Python Geocoding Toolbox. +- [geojson](https://github.com/jazzband/geojson) - Python bindings and utilities for GeoJSON. +- [geodjango](https://github.com/django/django) - (part of Django) A world-class [geographic web framework](https://docs.djangoproject.com/en/stable/ref/contrib/gis/). + +### Science + +_Libraries for scientific computing. Also see [Python-for-Scientists](https://github.com/TomNicholas/Python-for-Scientists)._ + +- Core + - [numpy](https://github.com/numpy/numpy) - A fundamental package for scientific computing with Python. + - [scipy](https://github.com/scipy/scipy) - Fundamental algorithms for scientific computing in Python. + - [numba](https://github.com/numba/numba) - A NumPy-aware JIT compiler for Python, using LLVM. +- Symbolic Mathematics + - [sympy](https://github.com/sympy/sympy) - A Python library for symbolic mathematics. +- Statistics + - [statsmodels](https://github.com/statsmodels/statsmodels) - Statistical modeling and econometrics in Python. +- Biology and Chemistry + - [rdkit](https://github.com/rdkit/rdkit) - Cheminformatics and Machine Learning Software. + - [biopython](https://github.com/biopython/biopython) - Biopython is a set of freely available tools for biological computation. +- Physics and Engineering + - [pint](https://github.com/hgrecco/pint) - Operate and manipulate physical quantities with units and dimensional analysis. + - [astropy](https://github.com/astropy/astropy) - A community Python library for Astronomy. + - [obspy](https://github.com/obspy/obspy) - A Python toolbox for seismology. +- Simulation and Modeling + - [pymc](https://github.com/pymc-devs/pymc) - Probabilistic programming and Bayesian modeling in Python. + - [simpy](https://gitlab.com/team-simpy/simpy) - A process-based discrete-event simulation framework. + - [mesa](https://github.com/mesa/mesa) - An agent-based modeling framework for building, analyzing, and visualizing complex system simulations. +- Graphs and Networks + - [networkx](https://github.com/networkx/networkx) - A high-productivity software for complex networks. +- Computational Geometry + - [shapely](https://github.com/shapely/shapely) - Manipulation and analysis of geometric objects in the Cartesian plane. +- Other + - [colour-science](https://github.com/colour-science/colour) - Implementing a comprehensive number of colour theory transformations and algorithms. + - [manim](https://github.com/ManimCommunity/manim) - An animation engine for explanatory math videos. + +### Quantum Computing + +_Libraries for quantum computing._ + +- [qiskit](https://github.com/Qiskit/qiskit) - An IBM-backed quantum SDK for building, simulating, and running circuits on real quantum hardware. +- [qutip](https://github.com/qutip/qutip) - Quantum Toolbox in Python. +- [pennylane](https://github.com/PennyLaneAI/pennylane) - A cross-platform library for quantum computing, quantum machine learning, and quantum chemistry. +- [cirq](https://github.com/quantumlib/Cirq) - A Google-developed framework focused on hardware-aware quantum circuit design for NISQ devices. + +**Developer Tools** + +### Algorithms and Design Patterns + +_Python implementation of data structures, algorithms and design patterns. Also see [awesome-algorithms](https://github.com/tayllan/awesome-algorithms)._ + +- Algorithms + - [sortedcontainers](https://github.com/grantjenks/python-sortedcontainers) - Fast and pure-Python implementation of sorted collections. + - [algorithms](https://github.com/keon/algorithms) - Minimal examples of data structures and algorithms. + - [thealgorithms](https://github.com/TheAlgorithms/Python) - All Algorithms implemented in Python. +- Design Patterns + - [python-patterns](https://github.com/faif/python-patterns) - A collection of design patterns and idioms in Python. + - [python-statemachine](https://github.com/fgmacedo/python-statemachine) - Expressive statecharts and finite state machines with a declarative API, in sync and async codebases. + +### Interactive Interpreter + +_Interactive Python interpreters (REPL)._ + +- [ipython](https://github.com/ipython/ipython) - A powerful interactive Python shell, and the kernel behind Jupyter notebooks. +- [notebook](https://github.com/jupyter/notebook) - A web-based notebook environment for interactive computing. + - [awesome-jupyter](https://github.com/markusschanta/awesome-jupyter) +- [marimo](https://github.com/marimo-team/marimo) - A reactive notebook for Python, stored as pure Python and runnable as a script or app. +- [ptpython](https://github.com/prompt-toolkit/ptpython) - Advanced Python REPL built on top of the [python-prompt-toolkit](https://github.com/prompt-toolkit/python-prompt-toolkit). + +### Code Analysis + +_Tools of static analysis, linters and code quality checkers. Also see [awesome-static-analysis](https://github.com/analysis-tools-dev/static-analysis)._ + +- General + - [import-linter](https://github.com/seddonym/import-linter) - A linter that enforces architectural constraints on imports between Python modules. + - [vulture](https://github.com/jendrikseipp/vulture) - A tool for finding and analyzing dead Python code. + - [complexipy](https://github.com/rohaquinlop/complexipy) - Cognitive complexity analysis for Python code, written in Rust. + - [prospector](https://github.com/prospector-dev/prospector) - A tool to analyze Python code. +- Git Hooks + - [pre-commit](https://github.com/pre-commit/pre-commit) - A framework for managing and maintaining multi-language pre-commit hooks. +- Linters and Formatters + - [ruff](https://github.com/astral-sh/ruff) - An extremely fast Python linter and code formatter. + - [black](https://github.com/psf/black) - The uncompromising Python code formatter. + - [isort](https://github.com/PyCQA/isort) - A Python utility / library to sort imports. + - [pylint](https://github.com/pylint-dev/pylint) - A fully customizable source code analyzer. + - [flake8](https://github.com/PyCQA/flake8) - A wrapper around `pycodestyle`, `pyflakes` and McCabe. + - [awesome-flake8-extensions](https://github.com/DmytroLitvinov/awesome-flake8-extensions) + - [bandit](https://github.com/PyCQA/bandit) - A tool designed to find common security issues in Python code. +- Refactoring + - [rope](https://github.com/python-rope/rope) - Rope is a Python refactoring library. + - [ast-grep-cli](https://github.com/ast-grep/ast-grep) - A CLI tool for structural code search, linting, and rewriting based on syntax trees. +- Type Checkers - [awesome-python-typing](https://github.com/typeddjango/awesome-python-typing) + - [mypy](https://github.com/python/mypy) - A static type checker for Python. + - [ty](https://github.com/astral-sh/ty) - An extremely fast Python type checker and language server. + - [pyright](https://github.com/microsoft/pyright) - Full-featured static type checker for Python from Microsoft, the engine behind Pylance. + - [pyrefly](https://github.com/facebook/pyrefly) - A fast type checker and language server for Python. + +### Testing + +_Libraries for testing codebases and generating test data. Also see [awesome-python-testing](https://github.com/cleder/awesome-python-testing)._ + +- Frameworks + - [pytest](https://github.com/pytest-dev/pytest) - A mature full-featured Python testing tool. + - [awesome-pytest](https://github.com/augustogoulart/awesome-pytest) + - [hypothesis](https://github.com/HypothesisWorks/hypothesis) - Hypothesis is an advanced Quickcheck style property based testing library. + - [robotframework](https://github.com/robotframework/robotframework) - A generic test automation framework. +- Test Runners + - [tox](https://github.com/tox-dev/tox) - Auto builds and tests distributions in multiple Python versions. + - [nox](https://github.com/wntrblm/nox) - Flexible test automation for Python. +- Browser Automation + - [playwright-python](https://github.com/microsoft/playwright-python) - Python version of the Playwright testing and automation library. + - [selenium](https://github.com/SeleniumHQ/selenium) - Python bindings for [Selenium](https://selenium.dev/) [WebDriver](https://selenium.dev/documentation/webdriver/). + - [seleniumbase](https://github.com/seleniumbase/SeleniumBase) - Python framework for web automation & testing, with stealth options. +- Load Testing + - [locust](https://github.com/locustio/locust) - Scalable user load testing tool written in Python. +- API Testing + - [schemathesis](https://github.com/schemathesis/schemathesis) - A tool for automatic property-based testing of web APIs from OpenAPI or GraphQL schemas. +- Mock + - [unittest.mock](https://docs.python.org/3/library/unittest.mock.html) - (Python standard library) A mocking and patching library. + - [responses](https://github.com/getsentry/responses) - A utility library for mocking out the requests Python library. + - [vcrpy](https://github.com/kevin1024/vcrpy) - Record and replay HTTP interactions on your tests. + - [respx](https://github.com/lundberg/respx) - Mock HTTPX with awesome request patterns and response side effects. + - [time-machine](https://github.com/adamchainz/time-machine) - Travel through time in your tests by mocking the current time at the C level. +- Object Factories + - [factory-boy](https://github.com/FactoryBoy/factory_boy) - A test fixtures replacement for Python. + - [polyfactory](https://github.com/litestar-org/polyfactory) - A mock data generation library based on type hints (continuation of `pydantic-factories`). +- Code Coverage + - [coverage](https://github.com/coveragepy/coveragepy) - Code coverage measurement. +- Fake Data + - [faker](https://github.com/joke2k/faker) - A Python package that generates fake data. + - [mimesis](https://github.com/lk-geimfari/mimesis) - A Python library for generating fake but realistic data in multiple languages and locales. + +### Debugging Tools + +_Libraries for debugging code._ + +- pdb-like Debugger + - [ipdb](https://github.com/gotcha/ipdb) - IPython-enabled [pdb](https://docs.python.org/3/library/pdb.html). + - [pudb](https://github.com/inducer/pudb) - A full-screen, console-based Python debugger. +- Tracing + - [viztracer](https://github.com/gaogaotiantian/viztracer) - A low-overhead tool that traces and visualizes Python code execution. +- Profiler + - [py-spy](https://github.com/benfred/py-spy) - A sampling profiler for Python programs. Written in Rust. + - [memray](https://github.com/bloomberg/memray) - A memory profiler that tracks allocations in Python code, native extensions, and the interpreter itself. + - [pyinstrument](https://github.com/joerick/pyinstrument) - A statistical wall-clock profiler with low overhead and readable call-tree output. + - [scalene](https://github.com/plasma-umass/scalene) - A high-performance, high-precision CPU, GPU, and memory profiler for Python. +- Others + - [django-debug-toolbar](https://github.com/django-commons/django-debug-toolbar) - Display various debug information for Django. + - [django-silk](https://github.com/jazzband/django-silk) - Live profiling and inspection of HTTP requests and database queries for Django. + - [icecream](https://github.com/gruns/icecream) - Inspect variables, expressions, and program execution with a single, simple function call. + - [flask-debugtoolbar](https://github.com/pallets-eco/flask-debugtoolbar) - A port of the django-debug-toolbar to flask. + +### Build Tools + +_Task runners and software build tools. If you're looking for Python packaging/build tools, see [Package Management](#package-management)._ + +- [invoke](https://github.com/pyinvoke/invoke) - A tool for managing shell-oriented subprocesses and organizing executable Python code into CLI-invokable tasks. +- [poethepoet](https://github.com/nat-n/poethepoet) - A task runner that defines tasks in pyproject.toml and works with poetry or uv. +- [scons](https://github.com/SCons/scons) - A software construction tool. +- [doit](https://github.com/pydoit/doit) - A task runner and build tool. + +### Documentation + +_Libraries for generating project documentation._ + +- [sphinx](https://github.com/sphinx-doc/sphinx) - Python Documentation generator. + - [awesome-sphinxdoc](https://github.com/ygzgxyz/awesome-sphinxdoc) +- [mkdocs-material](https://github.com/squidfunk/mkdocs-material) - A documentation framework and Material Design theme built on MkDocs. +- [diagrams](https://github.com/mingrammer/diagrams) - Diagram as Code. +- [pdoc](https://github.com/mitmproxy/pdoc) - Auto-generates API documentation for Python projects. +- [zensical](https://github.com/zensical/zensical) - A modern static site generator for technical documentation. + +**DevOps** + +### DevOps Tools + +_Software and libraries for DevOps._ + +- Cloud Providers + - [boto3](https://github.com/boto/boto3) - Python interface to Amazon Web Services. + - [awscli](https://github.com/aws/aws-cli) - Universal Command Line Interface for Amazon Web Services; the PyPI package is v1, in maintenance mode, while v2 ships as AWS's bundled installer. + - [azure-sdk-for-python](https://github.com/Azure/azure-sdk-for-python) - Microsoft Azure SDK for Python, published as per-service packages. + - [google-cloud-python](https://github.com/googleapis/google-cloud-python) - Google Cloud client libraries for Python, published as per-service packages. +- Configuration Management + - [ansible](https://github.com/ansible/ansible) - A radically simple IT automation platform. + - [cloud-init](https://github.com/canonical/cloud-init) - A multi-distribution package that handles early initialization of a cloud instance. + - [pyinfra](https://github.com/pyinfra-dev/pyinfra) - Turns Python code into shell commands and runs them on your servers. + - [salt](https://github.com/saltstack/salt) - Infrastructure automation and management system. +- Deployment + - [fabric](https://github.com/fabric/fabric) - A simple, Pythonic tool for remote execution and deployment. + - [chalice](https://github.com/aws/chalice) - A Python serverless microframework for AWS. +- Monitoring and Processes + - [psutil](https://github.com/giampaolo/psutil) - A cross-platform process and system utilities module. + - [sentry-sdk](https://github.com/getsentry/sentry-python) - Sentry SDK for Python. + - [supervisor](https://github.com/Supervisor/supervisor) - Supervisor process control system for UNIX. + - [flower](https://github.com/mher/flower) - A real-time monitor and web admin for Celery task queues. + - [sh](https://github.com/amoffat/sh) - A full-fledged subprocess replacement for Python. +- Other + - [borgbackup](https://github.com/borgbackup/borg) - A deduplicating archiver with compression and encryption. + +### Distributed Computing + +_Frameworks and libraries for Distributed Computing._ + +- [pyspark](https://github.com/apache/spark) - [Apache Spark](https://spark.apache.org/) Python API. +- [dask](https://github.com/dask/dask) - A flexible parallel computing library for analytic computing. +- [ray](https://github.com/ray-project/ray) - A unified framework for scaling AI and Python applications. +- [joblib](https://github.com/joblib/joblib) - Parallel computing and disk-based caching for Python functions. +- [mpi4py](https://github.com/mpi4py/mpi4py) - Python bindings for MPI. + +### Task Queues + +_Libraries for working with task queues._ + +- [celery](https://github.com/celery/celery) - An asynchronous task queue/job queue based on distributed message passing. +- [rq](https://github.com/rq/rq) - Simple job queues for Python. +- [dramatiq](https://github.com/Bogdanp/dramatiq) - A fast and reliable background task processing library for Python 3. +- [huey](https://github.com/coleifer/huey) - A little task queue with multi-process, multi-thread, or greenlet workers. +- [taskiq](https://github.com/taskiq-python/taskiq) - Distributed task queue with native asyncio support and pluggable brokers. + +### Messaging + +_Libraries for working with message brokers and event streaming._ + +- [confluent-kafka](https://github.com/confluentinc/confluent-kafka-python) - Confluent's Python client for Apache Kafka, built on librdkafka. +- [pika](https://github.com/pika/pika) - Pure-Python RabbitMQ/AMQP 0-9-1 client library. +- [paho-mqtt](https://github.com/eclipse-paho/paho.mqtt.python) - The Eclipse Paho MQTT client for Python. +- [faststream](https://github.com/ag2ai/faststream) - A framework for building asynchronous services over Apache Kafka, RabbitMQ, NATS, MQTT and Redis. + +### Job Schedulers + +_Libraries for scheduling jobs._ + +- Task Scheduling + - [apscheduler](https://github.com/agronholm/apscheduler) - A light but powerful in-process task scheduler that lets you schedule functions. + - [schedule](https://github.com/dbader/schedule) - Python job scheduling for humans. +- Workflow Orchestration + - [temporalio](https://github.com/temporalio/sdk-python) - Temporal SDK for durable, fault-tolerant workflows that survive crashes and retries. + - [dagster](https://github.com/dagster-io/dagster) - An orchestration platform for the development, production, and observation of data assets. + - [apache-airflow](https://github.com/apache/airflow) - Airflow is a platform to programmatically author, schedule and monitor workflows. + - [prefect](https://github.com/PrefectHQ/prefect) - A modern workflow orchestration framework that makes it easy to build, schedule and monitor robust data pipelines. + +### Logging + +_Libraries for generating and working with logs._ + +- [logging](https://docs.python.org/3/library/logging.html) - (Python standard library) Logging facility for Python. +- [structlog](https://github.com/hynek/structlog) - Structured logging made easy. +- [loguru](https://github.com/Delgan/loguru) - Library which aims to bring enjoyable logging in Python. +- [logfire](https://github.com/pydantic/logfire) - The observability platform for Python, from the makers of Pydantic. + +### Network Virtualization + +_Tools and libraries for packet manipulation and network device automation._ + +- [scapy](https://github.com/secdev/scapy) - A brilliant packet manipulation library. +- [netmiko](https://github.com/ktbyers/netmiko) - Multi-vendor library to simplify CLI connections to network devices. +- [napalm](https://github.com/napalm-automation/napalm) - Cross-vendor API to manipulate network devices. + +**CLI & GUI** + +### CLI Development + +_Libraries for building command-line applications._ + +- General + - [argparse](https://docs.python.org/3/library/argparse.html) - (Python standard library) Command-line option and argument parsing. + - [click](https://github.com/pallets/click) - A package for creating beautiful command line interfaces in a composable way. + - [typer](https://github.com/fastapi/typer) - Modern CLI framework that uses Python type hints. Built on Click. + - [prompt_toolkit](https://github.com/prompt-toolkit/python-prompt-toolkit) - A library for building powerful interactive command lines. + - [cyclopts](https://github.com/BrianPugh/cyclopts) - Build CLIs from type-hinted functions, with docstring-based help. +- Terminal Rendering + - [rich](https://github.com/Textualize/rich) - Python library for rich text and beautiful formatting in the terminal. Also provides a great `RichHandler` log handler. + - [tqdm](https://github.com/tqdm/tqdm) - Fast, extensible progress bar for loops and CLI. + - [colorama](https://github.com/tartley/colorama) - Cross-platform colored terminal text. + - [alive-progress](https://github.com/rsalmei/alive-progress) - A new kind of Progress Bar, with real-time throughput, eta and very cool animations. +- TUI Frameworks + - [textual](https://github.com/Textualize/textual) - A framework for building interactive user interfaces that run in the terminal and the browser. + - [urwid](https://github.com/urwid/urwid) - A library for creating terminal GUI applications with strong support for widgets, events, rich colors, etc. + - [asciimatics](https://github.com/peterbrittain/asciimatics) - A package to create full-screen text UIs (from interactive forms to ASCII animations). + +### CLI Tools + +_Useful CLI-based tools._ + +- Database CLIs + - [pgcli](https://github.com/dbcli/pgcli) - PostgreSQL CLI with autocompletion and syntax highlighting. + - [mycli](https://github.com/dbcli/mycli) - MySQL CLI with autocompletion and syntax highlighting. + - [litecli](https://github.com/dbcli/litecli) - SQLite CLI with autocompletion and syntax highlighting. + - [iredis](https://github.com/laixintao/iredis) - Redis CLI with autocompletion and syntax highlighting. +- Downloaders + - [yt-dlp](https://github.com/yt-dlp/yt-dlp) - A command-line program to download videos from YouTube and other video sites, a fork of youtube-dl. +- HTTP Clients + - [httpie](https://github.com/httpie/cli) - A command line HTTP client, a user-friendly cURL replacement. +- Project Scaffolding + - [cookiecutter](https://github.com/cookiecutter/cookiecutter) - A command-line utility that creates projects from cookiecutters (project templates). + - [copier](https://github.com/copier-org/copier) - A library and command-line utility for rendering project templates. +- Shells + - [xonsh](https://github.com/xonsh/xonsh) - A Python-powered shell. Full-featured and cross-platform. +- Terminal Workflow + - [tmuxp](https://github.com/tmux-python/tmuxp) - A [tmux](https://github.com/tmux/tmux) session manager. + +### GUI Development + +_Libraries for working with graphical user interface applications._ + +- Desktop + - [PyGObject](https://github.com/GNOME/pygobject) - Python Bindings for GLib/GObject/GIO/GTK. + - [dearpygui](https://github.com/hoffstadt/DearPyGui) - A simple GPU-accelerated Python GUI framework. + - [Kivy](https://github.com/kivy/kivy) - An open-source framework for cross-platform GUI apps on desktop, mobile, and embedded platforms. + - [wxPython](https://github.com/wxWidgets/Phoenix) - A cross-platform GUI toolkit that wraps the wxWidgets C++ library. + - [toga](https://github.com/beeware/toga) - A Python native, OS native GUI toolkit. +- Qt + - [PySide6](https://github.com/pyside/pyside-setup) - Qt for Python offers the official Python bindings for [Qt](https://www.qt.io/), largely API-compatible with PyQt6 but with different licensing. + - [PyQt6](https://www.riverbankcomputing.com/static/Docs/PyQt6/) - Python bindings for the [Qt](https://www.qt.io/) cross-platform application and UI framework. +- Tkinter + - [tkinter](https://docs.python.org/3/library/tkinter.html) - (Python standard library) The standard Python interface to the Tcl/Tk GUI toolkit. + - [customtkinter](https://github.com/tomschimansky/customtkinter) - A modern and customizable Python UI-library based on Tkinter. +- Web-based + - [pywebview](https://github.com/r0x0r/pywebview) - A lightweight cross-platform native wrapper around a webview component. + - [nicegui](https://github.com/zauberzeug/nicegui) - An easy-to-use, Python-based UI framework, which shows up in your web browser. + - [flet](https://github.com/flet-dev/flet) - Cross-platform GUI framework for building modern apps in pure Python. + +**Text & Documents** + +### Text Processing + +_Libraries for parsing and manipulating plain texts._ + +- Encoding and Unicode + - [charset-normalizer](https://github.com/jawah/charset_normalizer) - Universal character encoding detector, and a dependency of requests. + - [chardet](https://github.com/chardet/chardet) - Python character encoding detector. + - [ftfy](https://github.com/rspeer/python-ftfy) - Makes Unicode text less broken and more consistent automagically. +- Fuzzy Matching + - [rapidfuzz](https://github.com/rapidfuzz/RapidFuzz) - Rapid fuzzy string matching using various string metrics, with a C++ core. +- General + - [difflib](https://docs.python.org/3/library/difflib.html) - (Python standard library) Helpers for computing deltas. + - [pyfiglet](https://github.com/pwaller/pyfiglet) - An implementation of figlet written in Python. +- Internationalization + - [babel](https://github.com/python-babel/babel) - An internationalization library for Python. +- Parser + - [pygments](https://github.com/pygments/pygments) - A generic syntax highlighter. + - [pyparsing](https://github.com/pyparsing/pyparsing) - A Python library for creating PEG parsers. + - [sqlparse](https://github.com/andialbrecht/sqlparse) - A non-validating SQL parser. + - [phonenumbers](https://github.com/daviddrysdale/python-phonenumbers) - Parsing, formatting, storing and validating international phone numbers. + - [parsy](https://github.com/python-parsy/parsy) - Easy, generic parser combinator library for creating parsers. +- Transliteration and Slugs + - [python-slugify](https://github.com/un33k/python-slugify) - A Python slugify library that translates unicode to ASCII. + - [unidecode](https://github.com/avian2/unidecode) - ASCII transliterations of Unicode text. +- Unique identifiers + - [shortuuid](https://github.com/skorokithakis/shortuuid) - A generator library for concise, unambiguous and URL-safe UUIDs. + +### HTML Manipulation + +_Libraries for working with HTML and XML._ + +- [lxml](https://github.com/lxml/lxml) - A very fast, easy-to-use and versatile library for handling HTML and XML. +- [beautifulsoup4](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) - Providing Pythonic idioms for iterating, searching, and modifying HTML or XML. +- [xmltodict](https://github.com/martinblech/xmltodict) - Working with XML feel like you are working with JSON. +- [markupsafe](https://github.com/pallets/markupsafe) - Safely adds untrusted strings to HTML/XML markup. +- [justhtml](https://github.com/EmilStenstrom/justhtml) - A pure Python HTML5 parser that sanitizes untrusted HTML by default. + +### File Format Processing + +_Libraries for parsing and manipulating specific file formats._ + +- General + - [pyelftools](https://github.com/eliben/pyelftools) - Parsing and analyzing ELF files and DWARF debugging information. + - [tablib](https://github.com/jazzband/tablib) - A module for Tabular Datasets in XLS, CSV, JSON, YAML. +- File Conversion + - [markitdown](https://github.com/microsoft/markitdown) - Python tool for converting files and office documents to Markdown. + - [pypandoc](https://github.com/JessicaTegner/pypandoc) - A thin wrapper for pandoc, the universal document converter. + - [docling](https://github.com/docling-project/docling) - Library for converting documents into structured data. +- Excel + - [openpyxl](https://openpyxl.readthedocs.io/en/stable/) - A library for reading and writing Excel 2010 xlsx/xlsm/xltx/xltm files. + - [xlsxwriter](https://github.com/jmcnamara/XlsxWriter) - A Python module for creating Excel .xlsx files. +- Word + - [python-docx](https://github.com/python-openxml/python-docx) - Creates, reads, and updates Microsoft Word (.docx) files. +- PowerPoint + - [python-pptx](https://github.com/scanny/python-pptx) - Python library for creating and updating PowerPoint (.pptx) files. +- PDF + - [pypdf](https://github.com/py-pdf/pypdf) - A library capable of splitting, merging, cropping, and transforming PDF pages. + - [pymupdf](https://github.com/pymupdf/PyMuPDF) - A fast library for extracting, rendering, and editing PDF and other document formats, built on MuPDF. + - [reportlab](https://docs.reportlab.com/) - An open-source library for generating PDFs and graphics. + - [pdfminer.six](https://github.com/pdfminer/pdfminer.six) - A community-maintained fork of PDFMiner for extracting information from PDF documents. +- HTML-to-PDF + - [weasyprint](https://github.com/Kozea/WeasyPrint) - A visual rendering engine for HTML and CSS that can export to PDF. +- Markdown + - [markdown-it-py](https://github.com/executablebooks/markdown-it-py) - Markdown parser with 100% CommonMark support, extensions, and syntax plugins. + - [markdown](https://github.com/Python-Markdown/markdown) - A Python implementation of John Gruber’s Markdown. + - [mistune](https://github.com/lepture/mistune) - A fast yet powerful Python Markdown parser with renderers and plugins. +- Data Formats + - [tomllib](https://docs.python.org/3/library/tomllib.html) - (Python standard library) Parse TOML files. + - [pyyaml](https://github.com/yaml/pyyaml) - A full-featured YAML framework for Python. + +### File Manipulation + +_Libraries for file manipulation._ + +- Paths and File Watching + - [pathlib](https://docs.python.org/3/library/pathlib.html) - (Python standard library) A cross-platform, object-oriented path library. + - [watchfiles](https://github.com/samuelcolvin/watchfiles) - Simple, modern and fast file watching and code reload in Python. + - [watchdog](https://github.com/gorakhargosh/watchdog) - API and shell utilities to monitor file system events. +- File Type Detection + - [mimetypes](https://docs.python.org/3/library/mimetypes.html) - (Python standard library) Map filenames to MIME types. + - [python-magic](https://github.com/ahupp/python-magic) - A Python interface to the libmagic file type identification library. + - [magika](https://github.com/google/magika) - Detects file content types with a deep-learning model. + +**Media** + +### Image Processing + +_Libraries for manipulating images._ + +- Barcodes and QR Codes + - [qrcode](https://github.com/lincolnloop/python-qrcode) - A pure Python QR Code generator. + - [python-barcode](https://github.com/WhyNotHugo/python-barcode) - Create barcodes in Python with no extra dependencies. +- General + - [pillow](https://github.com/python-pillow/Pillow) - Pillow is the friendly [PIL](https://pillow.readthedocs.io/en/stable/about.html) fork. + - [scikit-image](https://github.com/scikit-image/scikit-image) - A Python library for (scientific) image processing. + - [rembg](https://github.com/danielgatis/rembg) - A tool to remove image backgrounds. + - [wand](https://github.com/emcconville/wand) - Python bindings for [MagickWand](https://imagemagick.org/magick-wand/), C API for ImageMagick. + - [pyvips](https://github.com/libvips/pyvips) - A binding for libvips, a fast image processing library with low memory needs. +- Image Serving + - [thumbor](https://github.com/thumbor/thumbor) - A smart imaging service. It enables on-demand crop, re-sizing and flipping of images. + +### Audio & Video Processing + +_Libraries for manipulating audio, video, and their metadata._ + +- Audio + - [soundfile](https://github.com/bastibe/python-soundfile) - An audio library for reading and writing sound files, based on libsndfile, CFFI, and NumPy. + - [librosa](https://github.com/librosa/librosa) - Python library for audio and music analysis. + - [pydub](https://github.com/jiaaro/pydub) - Manipulate audio with a simple and easy high level interface. +- Video + - [av](https://github.com/PyAV-Org/PyAV) - Pythonic bindings for FFmpeg's libraries. + - [moviepy](https://github.com/Zulko/moviepy) - A module for script-based movie editing with many formats, including animated GIFs. +- Metadata + - [mutagen](https://github.com/quodlibet/mutagen) - A Python module to handle audio metadata. + - [tinytag](https://github.com/tinytag/tinytag) - A library for reading audio file metadata of MP3, MP4, WAV, OGG, FLAC, WMA, and AIFF files. + - [beets](https://github.com/beetbox/beets) - A music library manager and [MusicBrainz](https://musicbrainz.org/) tagger. + +### Game Development + +_Awesome game development libraries._ + +- 3D Engines + - [panda3d](https://github.com/panda3d/panda3d) - 3D game engine developed jointly by Disney and contributors from around the world. +- Game Frameworks + - [pyglet](https://github.com/pyglet/pyglet) - A cross-platform windowing and multimedia library for Python. + - [pygame-ce](https://github.com/pygame-community/pygame-ce) - An actively developed drop-in replacement with new features and performance improvements ([pygame](https://github.com/pygame/pygame) fork). + - [pygame](https://github.com/pygame/pygame) - Pygame is a set of Python modules designed for writing games. + - [arcade](https://github.com/pythonarcade/arcade) - An easy-to-use library for creating 2D arcade games. +- Visual Novels + - [renpy](https://github.com/renpy/renpy) - A Visual Novel engine. + +**Python Language** + +### Implementations + +_Implementations of Python._ + +- [cpython](https://github.com/python/cpython) - Default, most widely used implementation of the Python programming language written in C. +- [micropython](https://github.com/micropython/micropython) - A lean and efficient Python implementation for microcontrollers and constrained systems. +- [pypy](https://github.com/pypy/pypy) - A very fast and compliant implementation of the Python language. +- [Cython](https://github.com/cython/cython) - Optimizing Static Compiler for Python. +- [pyodide](https://github.com/pyodide/pyodide) - Python distribution for the browser and Node.js based on WebAssembly. + +### Built-in Classes Enhancement + +_Libraries for enhancing Python built-in classes._ + +- [attrs](https://github.com/python-attrs/attrs) - Replacement for `__init__`, `__eq__`, `__repr__`, etc. boilerplate in class definitions. +- [bidict](https://github.com/jab/bidict) - Efficient, Pythonic bidirectional map data structures and related functionality. +- [uuid-utils](https://github.com/aminalaee/uuid-utils) - A fast, Rust-backed drop-in replacement for Python's built-in `uuid` module. +- [python-box](https://github.com/cdgriffith/Box) - Python dictionaries with advanced dot notation access. + +### Functional Programming + +_Functional Programming with Python._ + +- [functools](https://docs.python.org/3/library/functools.html) - (Python standard library) Higher-order functions and operations on callable objects. +- [more-itertools](https://github.com/more-itertools/more-itertools) - More routines for operating on iterables, beyond `itertools`. +- [toolz](https://github.com/pytoolz/toolz) - A collection of functional utilities for iterators, functions, and dictionaries. Also available as [cytoolz](https://github.com/pytoolz/cytoolz) for Cython-accelerated performance. +- [funcy](https://github.com/Suor/funcy) - A fancy and practical functional tools. +- [returns](https://github.com/dry-python/returns) - A set of type-safe monads, transformers, and composition utilities. -*Libraries for working with WebSocket.* +### Asynchronous Programming -* [AutobahnPython](https://github.com/crossbario/autobahn-python) - WebSocket & WAMP for Python on Twisted and [asyncio](https://docs.python.org/3/library/asyncio.html). -* [Crossbar](https://github.com/crossbario/crossbar/) - Open-source Unified Application Router (Websocket & WAMP for Python on Autobahn). -* [django-socketio](https://github.com/stephenmcd/django-socketio) - WebSockets for Django. -* [WebSocket-for-Python](https://github.com/Lawouach/WebSocket-for-Python) - WebSocket client and server library for Python 2 and 3 as well as PyPy. +_Libraries for asynchronous, concurrent and parallel execution. Also see [awesome-asyncio](https://github.com/timofurrer/awesome-asyncio)._ -# Services +- Async I/O + - [asyncio](https://docs.python.org/3/library/asyncio.html) - (Python standard library) Asynchronous I/O, event loop, coroutines and tasks. + - [awesome-asyncio](https://github.com/timofurrer/awesome-asyncio) + - [anyio](https://github.com/agronholm/anyio) - A high-level async concurrency and networking framework that works on top of asyncio or trio. + - [uvloop](https://github.com/MagicStack/uvloop) - Ultra fast asyncio event loop. + - [trio](https://github.com/python-trio/trio) - A friendly library for async concurrency and I/O. + - [gevent](https://github.com/gevent/gevent) - A coroutine-based Python networking library that uses [greenlet](https://github.com/python-greenlet/greenlet). + - [Twisted](https://github.com/twisted/twisted) - An event-driven networking engine. +- Parallelism + - [concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html) - (Python standard library) A high-level interface for asynchronously executing callables. + - [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) - (Python standard library) Process-based parallelism. -Online tools and APIs to simplify development. - -## Continuous Integration +### Date and Time -*See: [awesome-CIandCD](https://github.com/ciandcd/awesome-ciandcd#online-build-system).* +_Libraries for working with dates and times._ -* [CircleCI](https://circleci.com/) - A CI service that can run very fast parallel testing. (GitHub only) -* [Travis CI](https://travis-ci.org) - A popular CI service for your open source and [private](https://travis-ci.com) projects. (GitHub only) -* [Vexor CI](https://vexor.io) - A continuous integration tool for private apps with pay-per-minute billing model. -* [Wercker](http://wercker.com/) - A Docker-based platform for building and deploying applications and microservices. +- [zoneinfo](https://docs.python.org/3/library/zoneinfo.html) - (Python standard library) IANA time zone support. Brings the [tz database](https://en.wikipedia.org/wiki/Tz_database) into Python. +- [python-dateutil](https://github.com/dateutil/dateutil) - Extensions to the standard Python [datetime](https://docs.python.org/3/library/datetime.html) module. +- [dateparser](https://github.com/scrapinghub/dateparser) - A Python parser for human-readable dates in over 200 language locales. +- [pendulum](https://github.com/python-pendulum/pendulum) - Python datetimes made easy. +- [whenever](https://github.com/ariebovenberg/whenever) - A modern datetime library, type-safe and DST-safe, in Rust or pure Python. -## Code Quality +**Python Toolchain** -* [Codacy](https://www.codacy.com/) - Automated Code Review to ship better code, faster. Free for Open Source. -* [Codecov](https://codecov.io/) - Code coverage dashboard. -* [Landscape](https://landscape.io/) - Hosted continuous Python code metrics. -* [QuantifiedCode](https://www.quantifiedcode.com/) - A data-driven, automated, continuous code review tool. +### Environment Management -# Resources +_Libraries for Python version and virtual environment management._ -Where to discover new Python libraries. +- [virtualenv](https://github.com/pypa/virtualenv) - A tool to create isolated Python environments. +- [uv](https://github.com/astral-sh/uv) - An extremely fast Python version, package and project manager, written in Rust. +- [pyenv](https://github.com/pyenv/pyenv) - Simple Python version management. +- [mise](https://github.com/jdx/mise) - A polyglot tool version manager that installs Python versions and reads .python-version files. -## Podcasts +### Package Management -* [Podcast.init](http://podcastinit.com/) -* [Talk Python To Me](https://talkpython.fm/) -* [Python Bytes](https://pythonbytes.fm) - -## Twitter - -* [@codetengu](https://twitter.com/codetengu) -* [@getpy](https://twitter.com/getpy) -* [@importpython](https://twitter.com/importpython) -* [@planetpython](https://twitter.com/planetpython) -* [@pycoders](https://twitter.com/pycoders) -* [@pypi](https://twitter.com/pypi) -* [@pythontrending](https://twitter.com/pythontrending) -* [@PythonWeekly](https://twitter.com/PythonWeekly) -* [@TalkPython](https://twitter.com/talkpython) +_Libraries for package and dependency management._ -## Websites +- Package Managers + - [pip](https://github.com/pypa/pip) - The package installer for Python. + - [uv](https://github.com/astral-sh/uv) - An extremely fast Python version, package and project manager, written in Rust. + - [poetry](https://github.com/python-poetry/poetry) - Python dependency management and packaging made easy. + - [hatch](https://github.com/pypa/hatch) - Modern, extensible Python project manager for environments, builds, and publishing. + - [pipx](https://github.com/pypa/pipx) - Install and Run Python Applications in Isolated Environments. Like `npx` in Node.js. + - [conda](https://github.com/conda/conda) - Cross-platform, Python-agnostic binary package manager. +- Build Backends + - [setuptools](https://github.com/pypa/setuptools) - The historical and still most widely used pyproject build backend. + - [hatchling](https://github.com/pypa/hatch) - Modern, extensible build backend from the hatch project. + - [poetry-core](https://github.com/python-poetry/poetry-core) - Poetry's PEP 517 build backend, usable without Poetry itself. + - [flit-core](https://github.com/pypa/flit) - Minimal, dependency-free build backend for pure-Python packages. + - [uv-build](https://github.com/astral-sh/uv) - uv's fast, minimal build backend for pure-Python projects. -* [/r/CoolGithubProjects](https://www.reddit.com/r/coolgithubprojects/) -* [/r/Python](https://www.reddit.com/r/python) -* [Awesome Python @LibHunt](http://python.libhunt.com) -* [Django Packages](https://www.djangopackages.com/) -* [Full Stack Python](https://www.fullstackpython.com/) -* [PyPI Ranking](http://pypi-ranking.info/alltime) -* [Python 3 Wall of Superpowers](http://python3wos.appspot.com/) -* [Python Hackers](http://pythonhackers.com/open-source/) -* [Python ZEEF](https://python.zeef.com/alan.richmond) -* [Python 开发社区](http://python.ctolib.com) -* [Trending Python repositories on GitHub today](https://github.com/trending?l=python) +### Package Repositories -## Weekly +_Local PyPI repository servers, proxies, and mirrors._ -* [CodeTengu Weekly](http://weekly.codetengu.com/) -* [Import Python Newsletter](http://importpython.com/newsletter/) -* [Pycoder's Weekly](http://pycoders.com/) -* [Python Weekly](http://www.pythonweekly.com/) -* [Python Bytes](https://pythonbytes.fm) +- [pypiserver](https://github.com/pypiserver/pypiserver) - A minimal PyPI server for uploading and installing packages with pip. +- [devpi](https://github.com/devpi/devpi) - PyPI server and packaging/testing/release tool. +- [bandersnatch](https://github.com/pypa/bandersnatch) - PyPI mirroring tool provided by Python Packaging Authority (PyPA). -# Other Awesome Lists +### Distribution -List of lists. +_Libraries to create packaged executables for release distribution._ -* Monty - * [awesome](https://github.com/sindresorhus/awesome) - * [lists](https://github.com/jnv/lists) -* Python - * [pycrumbs](https://github.com/kirang89/pycrumbs) - * [python-github-projects](https://github.com/checkcheckzz/python-github-projects) - * [python_reference](https://github.com/rasbt/python_reference) - * [pythonidae](https://github.com/svaksha/pythonidae) +- Executables + - [pyinstaller](https://github.com/pyinstaller/pyinstaller) - Converts Python programs into stand-alone executables (cross-platform). + - [Nuitka](https://github.com/Nuitka/Nuitka) - Compiles Python programs into high-performance standalone executables (cross-platform). + - [pex](https://github.com/pex-tool/pex) - A tool for building self-contained Python executable environments (PEP 441 zipapps). + - [cx-Freeze](https://github.com/marcelotduarte/cx_Freeze) - Converts Python scripts into standalone executables and installers for Windows, macOS, and Linux. +- Obfuscation + - [pyarmor](https://github.com/dashingsoft/pyarmor) - A tool used to obfuscate Python scripts, bind obfuscated scripts to fixed machine or expire obfuscated scripts. -# Contributing +### Configuration Files + +_Libraries for storing and parsing configuration options._ + +- [configparser](https://docs.python.org/3/library/configparser.html) - (Python standard library) INI file parser. +- [python-dotenv](https://github.com/theskumar/python-dotenv) - Reads key-value pairs from a `.env` file and sets them as environment variables. +- [pydantic-settings](https://github.com/pydantic/pydantic-settings) - Settings management using Pydantic models with validation, loading from environment variables and secrets files. +- [hydra-core](https://github.com/hydra-ecosystem/hydra) - Hydra is a framework for elegantly configuring complex applications. +- [dynaconf](https://github.com/dynaconf/dynaconf) - Dynaconf is a configuration manager with plugins for Django and Flask. + +**Security** + +### Cryptography + +_Libraries for cryptographic primitives and secure protocols._ + +- [cryptography](https://github.com/pyca/cryptography) - A package designed to expose cryptographic primitives and recipes to Python developers. +- [pynacl](https://github.com/pyca/pynacl) - Python binding to libsodium, a fork of the Networking and Cryptography (NaCl) library. +- [paramiko](https://github.com/paramiko/paramiko) - The leading native Python SSHv2 protocol library. +- [itsdangerous](https://github.com/pallets/itsdangerous) - Safely pass trusted data to untrusted environments and back. + +### Penetration Testing + +_Frameworks and tools for penetration testing._ + +- [mitmproxy](https://github.com/mitmproxy/mitmproxy) - An interactive TLS-capable intercepting HTTP proxy for penetration testers and software developers. +- [impacket](https://github.com/fortra/impacket) - A collection of Python classes for working with network protocols, widely used for Windows and Active Directory testing. +- [sqlmap](https://github.com/sqlmapproject/sqlmap) - Automatic SQL injection and database takeover tool. +- [pwntools](https://github.com/Gallopsled/pwntools) - A CTF framework and exploit development library. +- [sherlock-project](https://github.com/sherlock-project/sherlock) - Hunt down social media accounts by username across social networks. + +### Supply Chain Security + +_Tools for auditing dependencies against known vulnerabilities and scanning for leaked secrets._ + +- Dependency Audits + - [pip-audit](https://github.com/pypa/pip-audit) - Audits Python environments and dependency trees for known vulnerabilities, using the Python Packaging Advisory Database or OSV. + - [uv-audit](https://github.com/astral-sh/uv) - (part of uv) uv's [dependency vulnerability scanning](https://docs.astral.sh/uv/reference/cli/#uv-audit) backed by OSV. +- Secret Scanning + - [betterleaks](https://github.com/betterleaks/betterleaks) - Finds leaked secrets in git history, directories, and files, from the original author of gitleaks. + +### Web Security + +_Libraries for application-layer web security._ + +- [nh3](https://github.com/messense/nh3) - Python binding to the ammonia HTML sanitizer, a fast replacement for bleach. +- [secure](https://github.com/TypeError/secure) - HTTP security headers for Python web applications with ASGI and WSGI middleware. + +**Other** + +### Hardware + +_Libraries for programming with hardware._ + +- [pyserial](https://github.com/pyserial/pyserial) - Python serial port access library for Windows, macOS, Linux, and BSD. +- [pynput](https://github.com/moses-palmer/pynput) - A library to control and monitor input devices. +- [bleak](https://github.com/hbldh/bleak) - A cross platform Bluetooth Low Energy Client for Python using asyncio. +- [jumpstarter](https://github.com/jumpstarter-dev/jumpstarter) - A hardware-in-the-loop testing framework with a Python client library for automated testing on real and virtual hardware. + +### Microsoft Windows + +_Python programming on Microsoft Windows._ + +- [pywin32](https://github.com/mhammond/pywin32) - Python Extensions for Windows. +- [pythonnet](https://github.com/pythonnet/pythonnet) - Python Integration with the .NET Common Language Runtime (CLR). +- [pyenv-win](https://github.com/pyenv-win/pyenv-win) - A Python version manager for Windows ([rbenv-win](https://github.com/nak1114/rbenv-win) fork). +- [winpython](https://github.com/winpython/winpython) - Portable Python distribution for Windows. + +### Miscellaneous + +_Useful libraries or tools that don't fit in the categories above._ + +- [blinker](https://github.com/pallets-eco/blinker) - A fast Python in-process signal/event dispatching system. +- [boltons](https://github.com/mahmoud/boltons) - A set of pure-Python utilities. + +## Resources + +Where to discover learning resources or new Python libraries. + +### Newsletters + +- [Awesome Python Newsletter](https://python.libhunt.com/newsletter) +- [Pycoder's Weekly](https://pycoders.com/) +- [Python Tricks](https://realpython.com/python-tricks/) +- [Python Weekly](https://www.pythonweekly.com/) + +### Podcasts + +- [Django Chat](https://djangochat.com/) +- [PyPodcats](https://pypodcats.live) +- [Python Bytes](https://pythonbytes.fm) +- [Talk Python To Me](https://talkpython.fm/) +- [The Real Python Podcast](https://realpython.com/podcasts/rpp/) + +### Websites + +- [Python Developer Tooling Handbook](https://pydevtools.com/) - Comprehensive guide to modern Python developer tools covering package management, linting, type checking, testing, and more. + +## Contributing Your contributions are always welcome! Please take a look at the [contribution guidelines](https://github.com/vinta/awesome-python/blob/master/CONTRIBUTING.md) first. -I will keep some pull requests open if I'm not sure whether those libraries are awesome, you could [vote for them](https://github.com/vinta/awesome-python/pulls) by adding :+1: to them. Pull requests will be merged when their votes reach **20**. +--- + +If you have any question about this opinionated list, do not hesitate to contact [@vinta](https://x.com/vinta) on X (Twitter). diff --git a/SPONSORSHIP.md b/SPONSORSHIP.md new file mode 100644 index 0000000000..1e27bee5e9 --- /dev/null +++ b/SPONSORSHIP.md @@ -0,0 +1,62 @@ +# Sponsor awesome-python + +**The #10 most-starred repository on all of GitHub.** + +awesome-python is where Python developers go to discover tools. It ranks on the first page of Google for "best Python libraries," is referenced by ChatGPT and other LLMs when recommending Python tools, and is the list developers check when evaluating frameworks. + +Your sponsorship puts your product in front of developers at the exact moment they're choosing what to use. + +## Audience + +| Metric | Value | +| ------------ | ---------------------------------------------------------------------------------------------------- | +| Stars | ![Stars](https://img.shields.io/github/stars/vinta/awesome-python?style=for-the-badge) | +| Forks | ![Forks](https://img.shields.io/github/forks/vinta/awesome-python?style=for-the-badge) | +| Watchers | ![Watchers](https://img.shields.io/github/watchers/vinta/awesome-python?style=for-the-badge) | +| Contributors | ![Contributors](https://img.shields.io/github/contributors/vinta/awesome-python?style=for-the-badge) | + +**Who visits:** Professional Python developers evaluating libraries and tools for production use. Not beginners browsing tutorials. People making adoption decisions. + +**Where they come from:** Google Search, GitHub, Reddit, YouTube, ChatGPT/LLMs, Hacker News. + +## Sponsorship Tiers + +### Headline Sponsor - $500/month + +- Large logo and one-line description (max 120 characters) pinned at the very top of the README, above all project entries +- Logo link in the sponsor section of [awesome-python.com](https://awesome-python.com/) + +### Featured Sponsor - $200/month + +- Text entry (`[Name](URL) - Description.`, max 120 characters) pinned at the top of the README, directly below Headline sponsors +- Text link in the sponsor section of [awesome-python.com](https://awesome-python.com/) + +### Indie Dev Sponsor - $99/month or $249/quarter + +- Lower-cost tier for indie developers, solo founders, and one-person companies only +- Text entry (`[Name](URL) - Description.`, max 120 characters) in the README sponsor section, directly below Featured sponsors +- Text link in the sponsor section of [awesome-python.com](https://awesome-python.com/) + +Indie Dev Sponsor is available only to founder-led (one-person company) products. Funded startups, larger companies, and products with enterprise sales teams should use Featured or Headline Sponsor. + +## Get Started + +Email [sponsorship@awesome-python.com](mailto:sponsorship@awesome-python.com?subject=Awesome%20Python%20Sponsorship) with: + +- **Tier:** Headline Sponsor ($500/mo), Featured Sponsor ($200/mo), or Indie Dev Sponsor ($99/mo or $249/quarter) +- **Content:** Product name, URL, logo, and description (Headline tier) or `[Name](URL) - Description.` entry (Featured or Indie Dev tier) +- **Duration:** 1, 3, 6 months, or longer +- **Payment method:** card via [Open Collective](https://opencollective.com/awesome-python), US bank transfer (ACH/wire), or PayPal + +One upfront payment per term. Setup takes less than 24 hours. + +## Previously Sponsored By + +- [Warp](https://www.warp.dev/) - The terminal for modern developers. +- [pyr](https://pyrun.dev) - Zero-config Python project manager. + +## Editorial Independence + +Sponsorship is logo or link placement in the README header. It does not influence which projects are listed. Listings are curated on merit through the normal [contribution process](CONTRIBUTING.md). + +We reserve the right to request changes to sponsor text, logos, or links that are misleading, off-topic, or incompatible with the README formatting. diff --git a/docs/CNAME b/docs/CNAME deleted file mode 100644 index 0f6ced6634..0000000000 --- a/docs/CNAME +++ /dev/null @@ -1 +0,0 @@ -awesome-python.com \ No newline at end of file diff --git a/docs/adr/0001-shortlist-not-catalog.md b/docs/adr/0001-shortlist-not-catalog.md new file mode 100644 index 0000000000..3e22e1289f --- /dev/null +++ b/docs/adr/0001-shortlist-not-catalog.md @@ -0,0 +1,24 @@ +--- +status: accepted +--- + +# awesome-python is a shortlist of obvious choices, not a catalog + +By mid-2026 the list held 576 entries across 75 sections, with entry inflow up 24x year over year (96 additions in the trailing 12 months vs 4 the year before), concentrated in sections like AI and Agents (41 entries). The old acceptance model — three lanes (Industry Standard, Rising Star, Hidden Gem), only the first capped — admitted any project good enough in isolation, so categories grew without bound and stopped answering the reader's actual question: "what do I use for X?" We decided to reposition the list as a curated shortlist, per the Zen of Python: there should be one — and preferably only one — obvious way to do it. + +## The decision + +The list's scope test is "serves Python developers", replacing the old "primarily written in Python (>50%)" requirement: implementation language and packaging are irrelevant when Python developers use the thing in their Python work (uv and ty are Rust; agent skill packs are markdown), while a pure-Python project nobody uses in Python work does not belong. Each Use Case (a subcategory, or a flat section) lists at most its Obvious Choices — up to 3, plus up to 2 Challengers, hard maximum 5 (numbers provisional, to be reviewed after the prune). Admission is by maintainer editorial judgment, informed primarily by PyPI download counts rather than GitHub stars, and stated as final; judgment overrides the signal's known failure modes (CI/dependency-inflated counts, model releases consumed as weights rather than pip installs, large-but-specific audiences misread as "niche"). Once a Use Case is at cap, the only way in is Displacement: the PR names the entry it replaces and argues the newcomer does that job better. Use Cases are defined by the list's existing structure; an entry PR can never create the subcategory it needs. Standard-library entries hold a slot only where the stdlib module is itself the obvious choice. The existing stock gets the same test retroactively: a staged, worst-first prune (per-section sweep commits), with removed entries deleted outright — git history is the archive. Resources sections (Newsletters, Podcasts, Websites) are out of scope for now. + +## Considered options + +- Keep the three-lane model with caps on every lane: rejected because the lanes answer the wrong question ("is this good enough to enter?") once admission is comparative; rising-star momentum becomes evidence for a Challenger slot or a Displacement rather than an admission ticket of its own, and Hidden Gem is definitionally incompatible with "obvious". +- Rules-only, no retroactive prune: rejected because every rejection would face "but X is listed" precedent arguments, and readers would see no change. +- Archive removed entries in a separate file: rejected because it recreates the catalog one click away and dilutes the identity the change exists to restore. + +## Consequences + +- The list shrinks substantially: the maintainer-reviewed preview of the three largest sections kept 45 of 80 entries. Most future PRs will be rejected for fullness, not badness. +- awesome-python.com loses long-tail search traffic for the hundreds of niche tool names it will no longer carry. Accepted deliberately: reader trust over search surface. +- Fast-moving domains (e.g. AI and Agents) list current leaders by usage and absorb churn through Displacement; an oversized Use Case is either trimmed or Split into finer Use Cases by the maintainer. +- Linked awesome-* lists (e.g. awesome-python-testing) remain as the escape valve for readers who want exhaustive catalogs. diff --git a/docs/audit-logs.md b/docs/audit-logs.md new file mode 100644 index 0000000000..14c6713e80 --- /dev/null +++ b/docs/audit-logs.md @@ -0,0 +1,195 @@ +# Audit Log + +awesome-python is audited section by section. Every entry gets re-verified against live data, and every removal lands in a commit whose body carries the reason. Git history is the archive. This file is the at-a-glance register of maintainer decisions that a single commit can't show. + +## Overrides + +[CONTRIBUTING.md](../CONTRIBUTING.md) allows the maintainer to exceed any limit for a specific entry or use case. Each override is recorded here. + +### Naming Exceptions + +Display names follow the canonical PyPI package name. These entries keep a different name by maintainer decision: + +- autobahn-python -- `autobahn`. +- django-rest-framework -- `djangorestframework`. +- django-rules -- `rules`. +- fasthtml -- `python-fasthtml`. +- graphify -- `graphifyy`. +- jinja -- `Jinja2`. +- mem0 -- `mem0ai`. +- pangu.py -- `pangu`. +- playwright-python -- `playwright`. +- pytorch -- `torch`. +- strawberry -- `strawberry-graphql`. +- strawberry-django -- `strawberry-graphql-django`. +- thealgorithms -- not on PyPI; its GitHub repo name is `Python`. + +### Stability Exceptions + +Entries admitted despite the Stable quality requirement, by maintainer decision: + +- zensical -- admitted 2026-08-23 at version 0.0.57, PyPI classifier `Development Status :: 3 - Alpha`, with no 1.0 target announced. Admitted as a challenger displacing mkdocs because the Documentation section was carrying a dying upstream: the Material for MkDocs team's announcement of 2025-11-05 put Material into maintenance mode for twelve months and called MkDocs itself unmaintained since 2024-08 and a supply chain risk. Re-check when 1.0 lands. +- openviking -- admitted 2026-08-25 (PR #3302) at version 0.4.16, PyPI classifier `Development Status :: 3 - Alpha`, with a README that said "OpenViking is still in its early stages". Admitted as a Data Layer challenger because the adoption is real: 459,446 downloads/month on pypistats (mirror-excluded) on 2026-08-24, and releases every few days. Re-check when 1.0 lands. +- claude-agent-sdk -- kept 2026-10-01 in the AI and Agents audit at version 0.2.163, PyPI classifier `Development Status :: 3 - Alpha`. Kept as a Vendor Agent SDKs obvious choice because the adoption is real: 30,033,689 downloads/month on ClickPy on 2026-10-01, up from 1.2M in March 2026 on pypistats. Re-check when 1.0 lands. +- uv-audit -- kept 2026-08-16 as a Supply Chain Security entry, recorded 2026-10-02. uv itself prints "`uv audit` is experimental and may change without warning". Kept because it is part of uv, the obvious package manager. Re-check when uv drops the experimental notice. + +### Challenger Exceptions + +Entries admitted as challengers without adoption-trajectory evidence, by maintainer decision: + +- seleniumbase -- admitted 2026-08-16 (PR #3284) as a Browser Automation challenger. It wraps selenium, the successor trajectory in the use case belongs to playwright, and its distinct value is its stealth/UC mode. Admitted because that stealth value counts toward Browser Automation. +- semantica -- kept 2026-10-01 in the AI and Agents audit as a Data Layer challenger at 16,582 downloads/month on ClickPy, with no adoption-trajectory evidence found. Kept by maintainer decision. +- hyperopt -- admitted 2026-10-16 as a Hyperparameter Optimization challenger at 1,793,872 downloads/month on ClickPy. Its 0.3.0 release came after a 4.5-year gap and it has no adoption-trajectory evidence. Admitted by maintainer decision. +- magika -- admitted 2026-10-16 as a File Type Detection challenger at 16,388,380 downloads/month on ClickPy. markitdown requires it, so most of that count is transitive, and no adoption-trajectory evidence was found. Admitted by maintainer decision. + +### Cap Exceptions + +Use cases holding more than 5 entries, by maintainer decision: + +- Asynchronous Programming > Async I/O -- 6 entries since the 2026-08-16 split (19875cf), recorded 2026-10-02. +- Code Analysis > Linters and Formatters -- 6 entries since Security Linters merged in on 2026-08-16 (3290ce5), recorded 2026-10-02. +- Package Management > Package Managers -- 6 entries since the subcategory was minted on 2026-08-16 (1b51907), recorded 2026-10-02. + +### Mature-stable Keeps + +These entries sit past the 12-month activity requirement without an override. Each one is kept by editorial judgment: mature, stable, and no successor exists. + +- annoy -- kept 2026-10-02 in the AI & ML audit, last push 2025-10-29. Unlike the others it has a successor, Spotify's own Voyager (2023), but kept by maintainer decision on 911,913 downloads/month on pypistats. +- blinker -- recorded 2026-10-02 ahead of its activity-line crossing on 2026-11-19 (last push 2025-11-19, last release 1.9.0 on 2024-11-08). Flask requires it. +- diskcache -- kept 2026-10-02, last push 2024-08-10, last release 5.6.3 (2023-08-31). Kept despite an unpatched pickle deserialization advisory, CVE-2025-69872 (GHSA-w8v5-vhqr-4h9v), on 25,526,045 downloads/month on ClickPy. +- django-rules -- recorded 2026-10-02 ahead of its activity-line crossing on 2026-10-11 (last push 2025-10-11, last release 3.5 on 2024-09-02). +- ftfy +- gensim -- recorded 2026-10-15 ahead of its activity-line crossing on 2026-11-01 (last push 2025-11-01, last release 4.4.0 on 2025-10-18), on 2,904,015 downloads/month on ClickPy (2026-10-12). +- httpie -- kept 2026-08-16, recorded 2026-10-02. Last commit 2024-12-17, last release 3.2.4 on 2024-11-01. +- itsdangerous +- jieba +- jinja +- python-pptx -- kept 2026-10-02, last commit 2024-08-06, last release 1.0.2 on 2024-08-07. No successor for generating .pptx files. +- schedule -- kept 2026-08-16, recorded 2026-10-02. Last commit 2024-05-25. +- sortedcontainers + +## Rejected Additions + +Projects the maintainer decided not to list, with the evidence at decision time, keyed by GitHub `owner/repo` (`owner/*` for a whole organization). PR reviews and star scans check here before proposing a project again. Rejudge one only on new evidence. + +- BrianPugh/autoregistry -- 2026-08-23, 16,758 downloads/month: 77 stars. +- MarcBresson/schedium -- 2026-08-23, 381 downloads/month: repository created 2026-02-05. +- ets-labs/python-dependency-injector -- 2026-08-23, 9,086,069 downloads/month: the Dependency Injection use case was declined, since FastAPI's Depends and pytest fixtures absorb most of the need, leaving a leader with no challenger field. +- konradhalas/dacite -- 2026-08-23, 23,147,569 downloads/month: last push 2025-03-17, fails the activity rule. +- reagento/dishka -- 2026-08-23, 769,893 downloads/month: declined with the Dependency Injection use case. +- xberg-io/* -- 2026-08-16: coordinated multi-entry self-promotion; four planted entries were removed (xberg, liter-llm, tree-sitter-language-pack, html-to-markdown). Reject any project from this organization on sight. +- AbstractUmbra/audioop -- 2026-10-16, 7,714,737 downloads/month (ClickPy): gradio and discord.py require it on Python 3.13+, so the count is transitive. It is a backport of a removed stdlib module, not an audio tool readers choose. +- alteryx/featuretools -- 2026-10-16, 106,521 downloads/month (ClickPy): Fails Active: last commit 2024-11-13, last release 2024-05-14. +- AnswerDotAI/fastcore -- 2026-10-16, 24,139,424 downloads/month (ClickPy): ghapi requires it, and the jump from 11.5M to 50.9M is pull from that dependency. It is a fastai grab-bag, not a built-in class enhancement. +- anthropics/claude-code-security-review -- 2026-10-16, no PyPI download signal: A language-agnostic GitHub Action with no releases and no commits since 2026-02-11. Not Python work. +- apple-aiml-research/ml-stable-diffusion -- 2026-10-16, no PyPI download signal: Fails Active: the last human commit is 2025-07-03, and the 2026 commit is a bot. It is a Core ML/Swift path, not a Python generation library. +- arrobalytics/django-ledger -- 2026-10-16, 17,102 downloads/month (ClickPy): An accounting engine, not an ERP suite. Small demand, no Development Status classifier, and no production case studies found. +- asdf-vm/asdf -- 2026-10-16, no PyPI download signal: The tool users are leaving for mise, with the 0.16 Go rewrite cited as the trigger. PyPI "asdf" is an unrelated astronomy package. +- AUTOMATIC1111/stable-diffusion-webui -- 2026-10-16, no PyPI download signal: Fails Active: the last commit on the default branch is 2024-07-27, and its pushed_at date doesn't reflect code changes. +- binaryornot/binaryornot -- 2026-10-16, 13,794,027 downloads/month (ClickPy): cookiecutter requires it, which accounts for about half the count. It does a narrow job, and the use case is at cap. +- bisohns/search-engine-parser -- 2026-10-16, 22,138 downloads/month (ClickPy): Last release 2022-11, and it pins lxml<5. It scrapes search results pages, not page content. +- bookieio/breadability -- 2026-10-16, 106,712 downloads/month (ClickPy): sumy pulls in almost all of the count. Its first release since 2014 lists Python 3.6 classifiers. +- boson-ai/higgs-audio -- 2026-10-16, 416 downloads/month (ClickPy): Two releases, both on 2025-10-03, and the PyPI homepage is a placeholder. v3 moved to Hugging Face plus an API, so nothing pip-installable is maintained. +- caesar0301/treelib -- 2026-10-16, 1,532,895 downloads/month (ClickPy): anytree (6.0M) is the bigger tree library at 4x the downloads, so treelib is not the obvious choice. +- chibisov/drf-extensions -- 2026-10-16, 2,067,903 downloads/month (ClickPy): Last release declares Django<6.0, and it is a DRF add-on, not an API framework. +- cknd/stackprinter -- 2026-10-16, 334,597 downloads/month (ClickPy): Flat downloads, and one release after a two-year gap. No sign of a challenger trajectory. +- clips/pattern -- 2026-10-16, no PyPI download signal: The code ended in 2018, and the 2026 commit is a README edit. PyPI "pattern" is now an unrelated package. Previously removed 2026-03-21 (f52c85a) as unmaintained, and nothing has changed since. +- closeio/ciso8601 -- 2026-10-16, 6,373,074 downloads/month (ClickPy): The use case is full. Its own docs tell users on Python 3.11+ to use datetime.fromisoformat unless they need the speed, so it is a narrow speed-up, not a general datetime library. +- codelucas/newspaper -- 2026-10-16, 468,795 downloads/month (ClickPy): The 2026-03-21 removal (1d09426, unmaintained) still holds: last release was 2018-09, and the 2026 push came with no release. +- coleifer/unqlite-python -- 2026-10-16, 11,386 downloads/month (ClickPy): It binds an embedded database rather than being a server driver, and has negligible demand in either home. +- coleifer/walrus -- 2026-10-16, 182,584 downloads/month (ClickPy): A utility layer over redis-py, not a driver, at under 0.1% of redis's 204M downloads. +- crccheck/django-object-actions -- 2026-10-16, 2,366,676 downloads/month (ClickPy): An add-on for Django admin buttons, not an admin panel. The source of its download count is unknown. +- davidhalter/jedi -- 2026-10-16, 102,370,894 downloads/month (ClickPy): ipython requires it, and its count almost exactly matches ipython's 102.6M. Its job is editor and REPL autocompletion, which the list dropped along with the Editor Plugins section (c87fdac). +- DerwenAI/pytextrank -- 2026-10-16, 82,707 downloads/month (ClickPy): Fails Active: last commit 2024-05-21, last release 2024-02-21. +- django-haystack/celery-haystack -- 2026-10-16, 1,919 downloads/month (ClickPy): Last release 2015-12. +- django-parler/django-parler -- 2026-10-16, 160,493 downloads/month (ClickPy): django-modeltranslation (826,853) has 5x its downloads, so it is not the obvious choice for Django model translation. +- doableware/djongo -- 2026-10-16, 57,392 downloads/month (ClickPy): Its latest release (1.3.7, 2024-11) pins django<=3.1.12, so it can't be installed with any supported Django. django-mongodb-backend already does this job. +- Doist/bitmapist -- 2026-10-16, 7,385 downloads/month (ClickPy): A Redis bitmap analytics library, not a driver, with negligible demand. Its metadata caps Python below 3.14 and redis below 6.0. +- dpath-maintainers/dpath-python -- 2026-10-16, 8,884,938 downloads/month (ClickPy): checkov and airbyte-cdk pull it in, leaving about 3.5M independent, below the section's floor (python-box, 4.9M). Last release 2024-06. +- dpkp/kafka-python -- 2026-10-16, 17,148,353 downloads/month (ClickPy): It was removed as unmaintained on 2026-03-21 (e8d3d29), but it was still releasing then and has shipped 3.0.0 through 3.0.11 since (June to August 2026), so that reason no longer holds. It is held back by tier: confluent-kafka (41.5M) is the default for new projects, and kafka-python is the pure-Python fallback rather than a successor. +- dwolfhub/zxcvbn-python -- 2026-10-16, 4,673,093 downloads/month (ClickPy): Password-strength estimation is not cryptography, and there is no use case for passwords. It also fails Active: last commit on master was 2025-10-06. +- explorerhq/sql-explorer -- 2026-10-16, 213,897 downloads/month (ClickPy): A SQL reporting tool, not an admin panel, and below the smallest incumbent (django-grappelli, 281,247). +- frostming/marko -- 2026-10-16, 5,798,780 downloads/month (ClickPy): The obvious tier (markdown-it-py, markdown, mistune) is full. Its growth roughly tracks the general ClickPy inflation, and no source shows it replacing any of the three. +- ggml-org/whisper.cpp -- 2026-10-16, 734,136 downloads/month (ClickPy): Python uses it through the separate third-party absadiki/pywhispercpp (349 stars). It does faster-whisper's job (12.1M) with no sign of overtaking it. The download count is unexplained for 349 stars. +- gitleaks/gitleaks -- 2026-10-16, no PyPI download signal: Secret Scanning was minted with betterleaks alone. gitleaks is a language-agnostic Go binary with slowing activity (last release 2026-03-21). +- hhatto/autopep8 -- 2026-10-16, 7,132,827 downloads/month (ClickPy): The use case holds 6 entries under a cap exception, and none is weaker (flake8 39.6M, bandit 24.5M). Last release was 2025-01, and the 2026 commits are dependabot only. +- html5lib/html5lib-python -- 2026-10-16, 26,055,801 downloads/month (ClickPy): The 2026-03-21 removal (7f21136, no release since 2019) still holds: last release 1.1 in 2020-06. The count comes from 3,281 dependents rather than people picking it, and justhtml already covers pure-Python HTML5 parsing. +- huggingface/speech-to-speech -- 2026-10-16, 2,714 downloads/month (ClickPy): An end-to-end voice pipeline app, not an ASR/TTS library, and far below the smallest Speech entry (funasr, 297,003). +- jazzband/django-pipeline -- 2026-10-16, 222,640 downloads/month (ClickPy): The 2026-03-21 removal (6b59080) gave overlap with django-compressor as the reason, which still holds: django-compressor has 11x its downloads. +- jundot/omlx -- 2026-10-16, no PyPI download signal: A macOS menu-bar app (repo created 2026-02-13), not a library. mlx-lm already covers Apple Silicon. +- jupyter/docker-stacks -- 2026-10-16, no PyPI download signal: Container images that package notebook (already listed), not an interpreter. +- kaifcodec/user-scanner -- 2026-10-16, 16,772 downloads/month (ClickPy): The use case is at cap. sherlock-project does the same username OSINT job at 4x the downloads. Repo created 2025-10. +- keleshev/schema -- 2026-10-16, 19,658,077 downloads/month (ClickPy): Removed as unmaintained on 2026-03-21 (e8d3d29), and activity has not resumed: the last release was 0.7.8 in 2025-10, with only a push since. It is classified Alpha and is not a successor to pydantic or jsonschema. +- kellyjonbrazil/jc -- 2026-10-16, 626,452 downloads/month (ClickPy): Parser is at cap (floor: parsy, 3.96M). jc does a different job (CLI output to JSON) and has no entry to replace. +- lexiforest/curl_cffi -- 2026-10-16, 31,930,921 downloads/month (ClickPy): About 17.4M of this is pulled in by yfinance, leaving roughly 14M of its own demand. HTTP Clients General is at cap, and web sources frame it as a TLS-fingerprint impersonation tool for getting past anti-bot blocks, not as a successor to requests or httpx2. No use case covers impersonation. +- LPgenerator/django-db-mailer -- 2026-10-16, 1,143 downloads/month (ClickPy): Last release 2018-10. +- mahmoud/glom -- 2026-10-16, 23,112,427 downloads/month (ClickPy): semgrep requires it, leaving about 3.5M independent. It is a transformer for nested data rather than a built-in class enhancement. +- mailgun/flanker -- 2026-10-16, 36,887 downloads/month (ClickPy): The 2026-03-21 removal (7f21136, abandoned by Mailgun) still holds: last release 2019-12. It parses mail rather than sending it. +- matthewwithanm/django-imagekit -- 2026-10-16, 380,436 downloads/month (ClickPy): Django-specific and below General's floor. Django thumbnailing has no clear leader (sorl-thumbnail 485K, easy-thumbnails 469K). +- matthewwithanm/pilkit -- 2026-10-16, 380,588 downloads/month (ClickPy): Pulled in by django-imagekit (nearly identical counts). Last release 2023-09. +- maxfischer2781/asyncstdlib -- 2026-10-16, 4,288,321 downloads/month (ClickPy): Async I/O already holds 6 entries under a cap exception. This is a different job (async versions of builtins and itertools), and aiostream (3.8M) is an equal peer. +- MemPalace/mempalace -- 2026-10-16, 56,453 downloads/month (ClickPy): Data Layer is at cap. Agent memory is mem0's job (2.0M), so there is no one-in-one-out case. Its April 2026 launch coverage centred on disputed benchmarks (the 100% LongMemEval claim was pulled), it is classified Beta, and its popularity is driven by stars (59.6k stars vs 56k downloads). +- mewwts/addict -- 2026-10-16, 6,120,693 downloads/month (ClickPy): Fails Active: last commit 2021-01, last release 2020-11. python-box, already listed, does the same job. +- microsoft/bocpy -- 2026-10-16, 9,933 downloads/month (ClickPy): Created 2026-02, classified Beta, 201 stars. No adoption-trajectory evidence. +- mongodb/motor -- 2026-10-16, 9,226,799 downloads/month (ClickPy): Deprecated in favor of the PyMongo Async API (2025-05-14) and end-of-life since 2026-05-14. Its last release was 3.7.1 in 2025-05, and pymongo is listed. The earlier removal (d9bba73, 2026-01-16, "unmaintained") still holds. +- mpdavis/python-jose -- 2026-10-16, 32,386,854 downloads/month (ClickPy): The 2026-03-21 removal (f50bd8f, superseded) still holds: FastAPI's JWT tutorial now uses PyJWT, and several dependents migrated away after CVE-2024-33663. No release since 2025-05. +- myusuf3/delorean -- 2026-10-16, 107,732 downloads/month (ClickPy): Removed 2024-02-23 (8524f28). 2.0.0 (2026-08) is its first release since 2018, and that day's commits are docs and CI only. It is Beta, has little demand, and the use case is full. +- noripyt/django-cachalot -- 2026-10-16, 244,358 downloads/month (ClickPy): Does the same job as django-cacheops (listed), which has 8x its downloads. +- oobabooga/textgen -- 2026-10-16, no PyPI download signal: A desktop app, not a pip library. PyPI "textgen" is an unrelated package. +- opensanctions/countrynames -- 2026-10-16, 15,360 downloads/month (ClickPy): Narrow job. pycountry (32.2M, unlisted) is the obvious country-data library. +- osantana/prettyconf -- 2026-10-16, 29,957 downloads/month (ClickPy): The use case is full at 5 entries, and prettyconf has about 0.6% of the downloads of dynaconf, the smallest entry. +- pydata/numexpr -- 2026-10-16, 12,883,844 downloads/month (ClickPy): Healthy (2.14.2 released 2026-07), but Core already holds numpy, scipy, and numba with far more demand. numexpr is an older accelerator, not a successor to any of them. +- python-excel/xlrd -- 2026-10-16, 70,465,563 downloads/month (ClickPy): Removed 2026-01-16 (d9bba73) as archived, but the repo was never archived. Releases have not resumed since 2.0.2 (2025-06), its own tagline says to use openpyxl where possible, it reads only legacy .xls, and python-calamine (7.7M) is the recommended replacement. Most of the count is direct demand. +- rajatim/zhtw -- 2026-10-16, 719 downloads/month (ClickPy): 14 stars, created 2025-12. It covers a narrower part of OpenCC's job. +- rushter/MLAlgorithms -- 2026-10-16, no PyPI download signal: Educational code, not on PyPI. thealgorithms already covers this. +- s3tools/s3cmd -- 2026-10-16, 1,243,432 downloads/month (ClickPy): The 2026-03-21 removal (e8d3d29) still holds: last release 2023-12, and it crosses the 12-month activity line on 2026-10-22. awscli and boto3 cover the job. +- saresend/KSUID -- 2026-10-16, 74,743 downloads/month (ClickPy): Fails Active: last commit 2018-11, last release 2017. svix-ksuid (1.0M) is the bigger KSUID library. +- shibing624/text2vec -- 2026-10-16, 2,291 downloads/month (ClickPy): sentence-transformers (listed) covers this, and there is no independent demand. +- smartcrop/smartcrop.py -- 2026-10-16, 29,956 downloads/month (ClickPy): A niche content-aware cropping job at 30K, and General is at cap (floor: pyvips, 867,739). +- SmileyChris/easy-thumbnails -- 2026-10-16, 469,327 downloads/month (ClickPy): Django-specific and below General's floor, with no lead over sorl-thumbnail. Last release 2025-08. +- ssut/py-googletrans -- 2026-10-16, 1,941,718 downloads/month (ClickPy): Fails Active: last commit 2025-04. It is an unofficial Google Translate scraper (machine translation, not i18n). +- TeamHG-Memex/eli5 -- 2026-10-16, 84,428 downloads/month (ClickPy): Fails Active: the maintained fork eli5-org/eli5 last committed on 2025-04-20, and recent commits to the original repo only delete workflow files. +- TheTom/turboquant_plus -- 2026-10-16, no PyPI download signal: A research repo for KV-cache quantization. Every commit since 2026-06-12 is a README edit, and the latest says the work was merged upstream into mlx-swift-lm. Nothing pip-installable. +- thumbor/libthumbor -- 2026-10-16, 66,538 downloads/month (ClickPy): A companion to the listed thumbor. Last release 2022-03, with only dependabot commits since. +- THUwangcy/ReChorus -- 2026-10-16, no PyPI download signal: An academic research framework with 648 stars, not on PyPI. +- tyiannak/pyAudioAnalysis -- 2026-10-16, 13,577 downloads/month (ClickPy): Removed 2026-03-21 (e8d3d29) as unmaintained, and that still holds: last release 2022-02, with only README edits since. +- uploadcare/pillow-simd -- 2026-10-16, 66,174 downloads/month (ClickPy): A fork of the listed pillow with 0.01% of its downloads, and General is at cap. +- wroberts/pytimeparse -- 2026-10-16, 23,637,352 downloads/month (ClickPy): Its count matches agate's (20.8M), which dbt-core pulls in, so very little of it is people choosing pytimeparse. 1.1.9 is its first release since 2018 and is classified Beta. The use case is full. +- x-hw/amazing-qr -- 2026-10-16, 812 downloads/month (ClickPy): Its 10.8k stars don't show up as installs. It pins Pillow<12, while qrcode (28.7M) holds the use case. +- xhtml2pdf/xhtml2pdf -- 2026-10-16, 3,813,456 downloads/month (ClickPy): It was removed in 2016 (a936fd1) as "deprecated in favor of WeasyPrint", which is no longer true: it shipped 0.2.22–0.2.24 in 2026-10. It still fails the tier bar, since the web calls WeasyPrint the default (32.5M) and describes xhtml2pdf as having limited CSS. It is an older alternative, not a successor. +- Yelp/detect-secrets -- 2026-10-16, 9,698,855 downloads/month (ClickPy): Weak maintenance: last release 2024-05, last code commit 2025-01, and only a CI commit in 2026. Demand rose 6.6x with no explained cause. +- youssofal/MTPLX -- 2026-10-16, 7,276 downloads/month (ClickPy): Narrow: its own summary is "Native MTP speculative decoding for Qwen3-Next on Apple Silicon". Repo created 2026-05-02, and it hard-requires mlx-lm, which is already listed, so it is an add-on rather than a separate pick. +- ytdl-org/youtube-dl -- 2026-10-16, 164,180 downloads/month (ClickPy): The 2026-03-21 removal (f50bd8f, replaced by yt-dlp) still holds: no release since 2021.12.17, commits stopped 2025-11, and downloads are falling. + +## Last Audited + +An audit that changes nothing in a section leaves no commit, so its date is recorded here, one line per section, replaced on each re-audit. + +- Asynchronous Programming -- 2026-10-15, no changes. +- Build Tools -- 2026-10-15, no changes. +- CLI Tools -- 2026-10-15, no changes. +- Computer Vision -- 2026-10-15, no changes. +- Configuration Files -- 2026-10-15, no changes. +- Cryptography -- 2026-10-15, no changes. +- Data Analysis -- 2026-10-15, no changes. +- Data Validation -- 2026-10-15, no changes. +- Date and Time -- 2026-10-15, no changes. +- Email -- 2026-10-15, no changes. +- Environment Management -- 2026-10-15, no changes. +- File Manipulation -- 2026-10-15, no changes. +- Functional Programming -- 2026-10-15, no changes. +- Geolocation -- 2026-10-15, no changes. +- HTML Manipulation -- 2026-10-15, no changes. +- Image Processing -- 2026-10-15, no changes. +- Implementations -- 2026-10-15, no changes. +- Interactive Interpreter -- 2026-10-15, no changes. +- Machine Learning -- 2026-10-15, no changes. +- Microsoft Windows -- 2026-10-15, no changes. +- Miscellaneous -- 2026-10-15, no changes. +- Natural Language Processing -- 2026-10-15, no changes. +- Package Management -- 2026-10-15, no changes. +- Quantum Computing -- 2026-10-15, no changes. +- Recommender Systems -- 2026-10-15, no changes. +- Science -- 2026-10-15, no changes. +- Supply Chain Security -- 2026-10-15, no changes. +- Web Security -- 2026-10-15, no changes. diff --git a/docs/extra.css b/docs/extra.css deleted file mode 100644 index 2f58f224ab..0000000000 --- a/docs/extra.css +++ /dev/null @@ -1,37 +0,0 @@ -.bs-sidebar.well { - /* margin-bottom: 0; */ -} - -.bs-sidebar .nav > li > a:hover, -.bs-sidebar .nav > li > a:focus { - border: none; - border-left: 2px solid; -} - -.bs-sidebar .nav > .active > a, -.bs-sidebar .nav > .active:hover > a, -.bs-sidebar .nav > .active:focus > a { - border: none; - border-left: 2px solid; -} - -@media (min-width: 992px) { - .bs-sidebar.affix { - /* top: 60px; */ - /* bottom: 0px; */ - top: 80px; - bottom: 23px; - overflow: auto; - } -} - -@media (min-width: 1200px) { - .bs-sidebar.affix-bottom, - .bs-sidebar.affix { - width: 280px; - } - - .container > .col-md-9 { - padding-left: 40px; - } -} diff --git a/mkdocs.yml b/mkdocs.yml deleted file mode 100644 index 6d2dd3937e..0000000000 --- a/mkdocs.yml +++ /dev/null @@ -1,9 +0,0 @@ -site_name: Awesome Python -site_url: http://awesome-python.com -site_description: A curated list of awesome Python frameworks, libraries and software -site_author: Vinta Chen -repo_url: https://github.com/vinta/awesome-python -theme: flatly -google_analytics: ['UA-510626-7', 'auto'] -pages: - - "Life is short, you need Python.": "index.md" diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000000..acdaf82b73 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,54 @@ +[project] +name = "awesome-python" +version = "0.1.0" +description = "An opinionated guide to the best Python frameworks, libraries, and tools." +authors = [{ name = "Vinta Chen", email = "vinta.chen@gmail.com" }] +readme = "README.md" +license = "MIT" +requires-python = ">=3.14" +dependencies = [] + +[project.urls] +Homepage = "https://awesome-python.com/" +Repository = "https://github.com/vinta/awesome-python" + +[dependency-groups] +build = ["httpx>=0.28.1", "jinja2>=3.1.6", "markdown-it-py>=4.2.0"] +lint = ["ruff>=0.16.10", "ty>=0.0.85"] +test = ["pytest>=9.1.1"] +preview = ["watchfiles>=1.3.0"] +dev = [ + { include-group = "build" }, + { include-group = "lint" }, + { include-group = "test" }, + { include-group = "preview" }, +] + +[tool.uv] +exclude-newer = "3 days" +no-build = true + +[tool.uv.pip] +only-binary = [":all:"] + +[tool.ruff] +line-length = 200 +src = ["website"] + +[tool.ty.environment] +python-version = "3.14" +root = ["website"] + +[tool.ty.terminal] +error-on-warning = true + +[tool.ty.rules] +division-by-zero = "error" +possibly-missing-attribute = "error" +possibly-missing-import = "error" +possibly-unresolved-reference = "error" +unused-ignore-comment = "error" + +[tool.pytest.ini_options] +testpaths = ["website/tests"] +pythonpath = ["website"] diff --git a/sort.py b/sort.py deleted file mode 100644 index d646c3faf7..0000000000 --- a/sort.py +++ /dev/null @@ -1,80 +0,0 @@ -# coding: utf-8 - -""" - The approach taken is explained below. I decided to do it simply. - Initially I was considering parsing the data into some sort of - structure and then generating an appropriate README. I am still - considering doing it - but for now this should work. The only issue - I see is that it only sorts the entries at the lowest level, and that - the order of the top-level contents do not match the order of the actual - entries. - - This could be extended by having nested blocks, sorting them recursively - and flattening the end structure into a list of lines. Revision 2 maybe ^.^. -""" - -def sort_blocks(): - # First, we load the current README into memory - with open('README.md', 'r') as read_me_file: - read_me = read_me_file.read() - - # Separating the 'table of contents' from the contents (blocks) - table_of_contents = ''.join(read_me.split('- - -')[0]) - blocks = ''.join(read_me.split('- - -')[1]).split('\n# ') - for i in range(len(blocks)): - if i == 0: - blocks[i] = blocks[i] + '\n' - else: - blocks[i] = '# ' + blocks[i] + '\n' - - # Sorting the libraries - inner_blocks = sorted(blocks[0].split('##')) - for i in range(1 , len(inner_blocks)): - if inner_blocks[i][0] != '#': - inner_blocks[i] = '##' + inner_blocks[i] - inner_blocks=''.join(inner_blocks) - - # Replacing the non-sorted libraries by the sorted ones and gathering all at the final_README file - blocks[0] = inner_blocks - final_README = table_of_contents + '- - -' + ''.join(blocks) - - with open('README.md', 'w+') as sorted_file: - sorted_file.write(final_README) - -def main(): - # First, we load the current README into memory as an array of lines - with open('README.md', 'r') as read_me_file: - read_me = read_me_file.readlines() - - # Then we cluster the lines together as blocks - # Each block represents a collection of lines that should be sorted - # This was done by assuming only links ([...](...)) are meant to be sorted - # Clustering is done by indentation - blocks = [] - last_indent = None - for line in read_me: - s_line = line.lstrip() - indent = len(line) - len(s_line) - - if any([s_line.startswith(s) for s in ['* [', '- [']]): - if indent == last_indent: - blocks[-1].append(line) - else: - blocks.append([line]) - last_indent = indent - else: - blocks.append([line]) - last_indent = None - - with open('README.md', 'w+') as sorted_file: - # Then all of the blocks are sorted individually - blocks = [''.join(sorted(block, key=lambda s: s.lower())) for block in blocks] - # And the result is written back to README.md - sorted_file.write(''.join(blocks)) - - # Then we call the sorting method - sort_blocks() - - -if __name__ == "__main__": - main() diff --git a/uv.lock b/uv.lock new file mode 100644 index 0000000000..f88d48afba --- /dev/null +++ b/uv.lock @@ -0,0 +1,400 @@ +version = 1 +revision = 3 +requires-python = ">=3.14" + +[options] +exclude-newer = "0001-01-01T00:00:00Z" # This has no effect and is included for backwards compatibility when using relative exclude-newer values. +exclude-newer-span = "P3D" + +[[package]] +name = "anyio" +version = "4.15.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "typing-extensions", marker = "python_full_version < '3.15'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a9/d2/f4d173e22df740bc37b1db102b386ba719b66e95b0f0d751f556b387e6d2/anyio-4.15.1.tar.gz", hash = "sha256:9f28306018cbd6d329e64a36d58256edff76dd996fe423bc957326e578b82a94", size = 276966, upload-time = "2026-09-05T10:42:39.44Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/12/b8/4bd346e22b28902df4d651910f5242c28d84e4a5c2435ca5c3f797ed7e2e/anyio-4.15.1-py3-none-any.whl", hash = "sha256:6152fdbbf9a77fdec97731721bebf7c4c44f7c29b424b0065826173efc7ed101", size = 132079, upload-time = "2026-09-05T10:42:37.923Z" }, +] + +[[package]] +name = "awesome-python" +version = "0.1.0" +source = { virtual = "." } + +[package.dev-dependencies] +build = [ + { name = "httpx" }, + { name = "jinja2" }, + { name = "markdown-it-py" }, +] +dev = [ + { name = "httpx" }, + { name = "jinja2" }, + { name = "markdown-it-py" }, + { name = "pytest" }, + { name = "ruff" }, + { name = "ty" }, + { name = "watchfiles" }, +] +lint = [ + { name = "ruff" }, + { name = "ty" }, +] +preview = [ + { name = "watchfiles" }, +] +test = [ + { name = "pytest" }, +] + +[package.metadata] + +[package.metadata.requires-dev] +build = [ + { name = "httpx", specifier = ">=0.28.1" }, + { name = "jinja2", specifier = ">=3.1.6" }, + { name = "markdown-it-py", specifier = ">=4.2.0" }, +] +dev = [ + { name = "httpx", specifier = ">=0.28.1" }, + { name = "jinja2", specifier = ">=3.1.6" }, + { name = "markdown-it-py", specifier = ">=4.2.0" }, + { name = "pytest", specifier = ">=9.1.1" }, + { name = "ruff", specifier = ">=0.16.10" }, + { name = "ty", specifier = ">=0.0.85" }, + { name = "watchfiles", specifier = ">=1.3.0" }, +] +lint = [ + { name = "ruff", specifier = ">=0.16.10" }, + { name = "ty", specifier = ">=0.0.85" }, +] +preview = [{ name = "watchfiles", specifier = ">=1.3.0" }] +test = [{ name = "pytest", specifier = ">=9.1.1" }] + +[[package]] +name = "certifi" +version = "2026.7.22" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a3/c2/24167ea9858356b47a87a50d39908bfdb72ceeefe0041586e704e5376b3a/certifi-2026.7.22.tar.gz", hash = "sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55", size = 138112, upload-time = "2026-07-22T03:35:12.644Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/a7/71ac2cff56fec219ed242bb11b8efb69fcc4bec75db06fb7bfe35de520e6/certifi-2026.7.22-py3-none-any.whl", hash = "sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775", size = 136983, upload-time = "2026-07-22T03:35:11.276Z" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "httpcore" +version = "1.0.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, +] + +[[package]] +name = "httpx" +version = "0.28.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "certifi" }, + { name = "httpcore" }, + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, +] + +[[package]] +name = "idna" +version = "3.20" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f5/08/8eea9d4b8302028f3abb2c0813953f7aec26d33b7a8960ed760e65ff29fa/idna-3.20.tar.gz", hash = "sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44", size = 216463, upload-time = "2026-09-17T14:11:04.752Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/58/a2/bb081bab032533a855d44de1d56f8e8426114ff1ba5d1f07a438a0a654f8/idna-3.20-py3-none-any.whl", hash = "sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c", size = 69583, upload-time = "2026-09-17T14:11:03.168Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.3.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/e1/2069291243c926a2ff1cd706c7f3eeb9b62144bf60f77c9fb9ff2fb26bd3/iniconfig-2.3.1.tar.gz", hash = "sha256:67f4b9c50da0dedf52af349e7749a80a9057a5031199791b906c3bb3ae878960", size = 21209, upload-time = "2026-10-06T22:48:38.076Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/56/43/4ca9e49d27a1fcf6bece6f6aec0ea46bb9112489b93d4b688fb415457bdb/iniconfig-2.3.1-py3-none-any.whl", hash = "sha256:9121e2c1fdb355232495be3194c8dfe87ccc2d5dee45947b78e68f499790d7a7", size = 7552, upload-time = "2026-10-06T22:48:36.959Z" }, +] + +[[package]] +name = "jinja2" +version = "3.1.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markupsafe" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/df/bf/f7da0350254c0ed7c72f3e33cef02e048281fec7ecec5f032d4aac52226b/jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d", size = 245115, upload-time = "2025-03-05T20:05:02.478Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, +] + +[[package]] +name = "markdown-it-py" +version = "4.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mdurl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/ff/7841249c247aa650a76b9ee4bbaeae59370dc8bfd2f6c01f3630c35eb134/markdown_it_py-4.2.0.tar.gz", hash = "sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49", size = 82454, upload-time = "2026-05-07T12:08:28.36Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl", hash = "sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a", size = 91687, upload-time = "2026-05-07T12:08:27.182Z" }, +] + +[[package]] +name = "markupsafe" +version = "3.0.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/38/9b/e422a865e1d5d57d0e509b4e0bf1c1a70a7f6382c29a5aa428df994c8bc8/markupsafe-3.0.4.tar.gz", hash = "sha256:2e9ad7dd851bf45fab9f75cbff4cb493fee9979e8d8c7c9c3ee119022518edd6", size = 153777, upload-time = "2026-10-02T23:07:22.29Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5c/c3/a944f3b0df22bd129e96915b9f4e98d2eeca6516687d7618304a966c3c74/markupsafe-3.0.4-cp314-cp314-android_24_arm64_v8a.whl", hash = "sha256:4ed644d75aa94a2baf7ec3a96eaa160ea58c742eb9d27c6506053c5c40fc84ed", size = 12787, upload-time = "2026-10-02T23:05:30.971Z" }, + { url = "https://files.pythonhosted.org/packages/d4/d6/a44863f69d88b6c7e27889108f70d47aed259edf89d5df3c5fca1eac87d6/markupsafe-3.0.4-cp314-cp314-android_24_x86_64.whl", hash = "sha256:6d2a9efe686f9de00d0d1ea32a4a5a86d558a2277501bd78d964214eab625e59", size = 12525, upload-time = "2026-10-02T23:05:32.263Z" }, + { url = "https://files.pythonhosted.org/packages/17/8f/168ba80e532dd6a93f96f8f706f1ad41d7990b6e1aeedc1cc0d211a33497/markupsafe-3.0.4-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:8781a792a070cf2bd1b86d3aa943894115faaba6e88122a7bf32d62072742453", size = 11589, upload-time = "2026-10-02T23:05:33.251Z" }, + { url = "https://files.pythonhosted.org/packages/32/b3/aa2c95a574d3af39403a469b295886eb9b6d448da568cbebb5a2cbfdc2e5/markupsafe-3.0.4-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:971a3bbb75d97ae4e2e8f7d4834236f86f85f0c85e04ab2e191db1123b04f80b", size = 11801, upload-time = "2026-10-02T23:05:34.315Z" }, + { url = "https://files.pythonhosted.org/packages/60/d0/34b810107d83840e768bf485de795893ebbae35b26ab061b487adfa0a692/markupsafe-3.0.4-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:8909c2f1c6dd65e054ac4b573a91c8384d1492281e55d82d159d653f7a13adf6", size = 11447, upload-time = "2026-10-02T23:05:35.302Z" }, + { url = "https://files.pythonhosted.org/packages/6c/ab/2f8488f0f817a39fca068d2b17daf446bf5cdb3eae28c3720af534d873b4/markupsafe-3.0.4-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:4cf3468d5ec187ffffcaca8e61929a37448f215dafc1386a12c750a72fe53634", size = 11722, upload-time = "2026-10-02T23:05:36.363Z" }, + { url = "https://files.pythonhosted.org/packages/ad/40/e2d117b048d47282ade906fbfd92814cbee5647afc13fda88a3406039372/markupsafe-3.0.4-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:52704c5d36eb6dda8866493decd61111fff86244c9b1ad225ca01b9e91e5970f", size = 12061, upload-time = "2026-10-02T23:05:37.397Z" }, + { url = "https://files.pythonhosted.org/packages/9a/a8/73a81135e85ba66217f5af7facb03bbb386807e1a729ab64532e4c802652/markupsafe-3.0.4-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1caa2fa5a6184fb233153b35f654e6687bd555476f6170f29d8ee9be1a8b0af9", size = 24434, upload-time = "2026-10-02T23:05:38.407Z" }, + { url = "https://files.pythonhosted.org/packages/ac/ca/fa9216dd01efee2dfdacafe7df32b4d0170fbac694b0c258a193d6e53999/markupsafe-3.0.4-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:387d8cd30e69b3f0a72877b9ae717033396404e19095b17fe89753a981fda44f", size = 27183, upload-time = "2026-10-02T23:05:39.581Z" }, + { url = "https://files.pythonhosted.org/packages/fa/4e/a469509e538d37af51103b17b073126973f2b1cbf197ff32c7ddf025cfe5/markupsafe-3.0.4-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:051417f74bcaaefa316276e0ff723f541616ca51043d070da00249d9bddd3e3c", size = 24813, upload-time = "2026-10-02T23:05:40.671Z" }, + { url = "https://files.pythonhosted.org/packages/8f/db/d7282caf7ab03af44d5d6fdbaa019b35c7d7f1c90588b839c07cba640d6a/markupsafe-3.0.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a8e9f292fcda89b324f2f5c91d13f1424a153e40fc2756f38ee23b15835ff300", size = 23049, upload-time = "2026-10-02T23:05:41.864Z" }, + { url = "https://files.pythonhosted.org/packages/30/f3/b6a425206e6964efda6acee544d0eb01d1501784d0b8e2dcc74986f33b17/markupsafe-3.0.4-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:df1ae86ff54725a01fa1a0510b914ca53a161b7050be74f6204e24aded5971d0", size = 22088, upload-time = "2026-10-02T23:05:43.014Z" }, + { url = "https://files.pythonhosted.org/packages/ea/8a/84d3582fc1f0d5bd466cdf2eebf175e172158a6e70701aacec1de1b35430/markupsafe-3.0.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:8965520ac587c94a4ac48b729be3d8b8de00af39699b17585dfb599babe77977", size = 23846, upload-time = "2026-10-02T23:05:44.098Z" }, + { url = "https://files.pythonhosted.org/packages/1c/65/db101cce51b7ba4864ac491a9859d297dd1adf0e55b103fee9db9c47c527/markupsafe-3.0.4-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:340cbb1957ba99929cbf19a75626d36ba1ae21d1730b287d1cf7f824a20c4fc7", size = 22674, upload-time = "2026-10-02T23:05:45.23Z" }, + { url = "https://files.pythonhosted.org/packages/e0/49/ddee9813d71db0c7a5c9d97c832125e6758a0c844777f1cf076569bb0e22/markupsafe-3.0.4-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:3a93d9616ddecfb393727a0041a562cf0b15a244e20f2bd25efc7949be4c4f17", size = 24104, upload-time = "2026-10-02T23:05:46.398Z" }, + { url = "https://files.pythonhosted.org/packages/aa/0e/7d8518d726726870a2399d69fd30d0fa36c5e57a2132c336b58d7c491073/markupsafe-3.0.4-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:d2e56fd3b00222722abfb3f5f0759ddbae4b90811b5ad4343c64030ad1bde70c", size = 21637, upload-time = "2026-10-02T23:05:47.48Z" }, + { url = "https://files.pythonhosted.org/packages/b4/b0/b505e8a361ba557dbf3b3aa7331ea39b00d2022a26e925ff8463b9714bb3/markupsafe-3.0.4-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:0d9c47709875fdb321452056622e930c52afbc07a7d780762fbb8b4d91ce6fa4", size = 23081, upload-time = "2026-10-02T23:05:48.611Z" }, + { url = "https://files.pythonhosted.org/packages/1c/ea/9cc3cea873f980c75cbdb6f4277ce30ee955de38be0b3d02f14c108e0698/markupsafe-3.0.4-cp314-cp314-win32.whl", hash = "sha256:38fc55594dab834470b6733dead2ee9e3f657fb0608c769dcafa0ba5ab52f45c", size = 14272, upload-time = "2026-10-02T23:05:49.707Z" }, + { url = "https://files.pythonhosted.org/packages/80/f0/5792ff768a410f93ee3f84fc19345295ffc352d2c936b424cb37e514714c/markupsafe-3.0.4-cp314-cp314-win_amd64.whl", hash = "sha256:c1bc67752d5f21013cfe430df4062441714eab79f65a6a05e01505957e9c35fe", size = 14501, upload-time = "2026-10-02T23:05:50.788Z" }, + { url = "https://files.pythonhosted.org/packages/5f/cf/3d074a8edffcc6899355232ff2543ae8d929733239596423b7db79698bc9/markupsafe-3.0.4-cp314-cp314-win_arm64.whl", hash = "sha256:7e1636da3d8dfc220b6dd10264db5f2b165e4888c4518594898fbe381049af8a", size = 14364, upload-time = "2026-10-02T23:05:51.857Z" }, + { url = "https://files.pythonhosted.org/packages/d9/31/87ce42159aae2163cf3bbbd0c44bc87780510eecab1ea3859099aed95dcb/markupsafe-3.0.4-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:805c8b84534fa10891890f0e4be39f3a99e94615d93e8836bf9fa1fdca2feeb2", size = 11759, upload-time = "2026-10-02T23:05:52.951Z" }, + { url = "https://files.pythonhosted.org/packages/5f/53/b047207eeb7752e960aca3eb1df5fb7eefa7dd4c62ac49bb156456c8a702/markupsafe-3.0.4-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:fa95848c929b6a75f6848d3c9793e59db365ee436776e57db835cdbfa79ba977", size = 12096, upload-time = "2026-10-02T23:05:54.066Z" }, + { url = "https://files.pythonhosted.org/packages/ee/51/4326c88a13c7b755657d44b4bb986f8c3d9843ecba7e22d98661d87f9a57/markupsafe-3.0.4-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e916035e3e9930cbdfdd10abf48861340221857f45509565898e012263f7b289", size = 25699, upload-time = "2026-10-02T23:05:55.15Z" }, + { url = "https://files.pythonhosted.org/packages/f2/bb/990581b7474bfcf2cf34bed6ba5ea23bd87adb9d671213d68e88620e7a6b/markupsafe-3.0.4-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:b4d12837e0203bbace818ff4a7461afdcd78bcd782351cea148139180d7bcffe", size = 27804, upload-time = "2026-10-02T23:05:56.29Z" }, + { url = "https://files.pythonhosted.org/packages/6b/89/89491878c28e8291f5aa2fffe2c2d57230d10ae366d55dd810b840513d78/markupsafe-3.0.4-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5086f9975abb1ab531ee6afca1761e4b59a19b446f3f6522ed776963228cfe5a", size = 25912, upload-time = "2026-10-02T23:05:57.416Z" }, + { url = "https://files.pythonhosted.org/packages/30/77/680998b54efdea06fc114565cd739b6d059f826a0279219b218dfa750d29/markupsafe-3.0.4-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b4a635a0487774f841cb1fb62e907e7195cc95bc761e053184b8acc3ceb20733", size = 23623, upload-time = "2026-10-02T23:05:58.557Z" }, + { url = "https://files.pythonhosted.org/packages/ae/75/2709f5ac5de9467b40b10e2bb8f89cc63dfb74582e09aa734b1124a217de/markupsafe-3.0.4-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:cb96e6e088d6cf71c1ea977510948320234824cf226e32f6f6e044f7a9c82b34", size = 23301, upload-time = "2026-10-02T23:05:59.94Z" }, + { url = "https://files.pythonhosted.org/packages/a0/c8/39eadc6c5b14c9c7679bfb98f4d4c6a97863b5beb91839aca4d2d6e16e55/markupsafe-3.0.4-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8b5d563170ff8ba3181caa967c99a3c804d1dedb702c7cb93a6a7c32247da978", size = 24950, upload-time = "2026-10-02T23:06:01.289Z" }, + { url = "https://files.pythonhosted.org/packages/1a/5e/01037f8a43e8ccb0bffb4fbdc5212db05bf080fdd7286cd392332d58128a/markupsafe-3.0.4-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:396ec4e65cc889f69786b3b89478b471cee5a3bcf468b9d9bb03e1a30fb291fc", size = 23330, upload-time = "2026-10-02T23:06:02.441Z" }, + { url = "https://files.pythonhosted.org/packages/d4/f4/23e83ce0596bb0cbe670502d31df8f757bbd01a392aa486fa3b40d1ed399/markupsafe-3.0.4-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:15ba9e28640feef770374b116a6f019c21f52404aeabe516aa7f800587b98cfc", size = 25239, upload-time = "2026-10-02T23:06:03.579Z" }, + { url = "https://files.pythonhosted.org/packages/88/5b/3708897368073cc683d524750474f41a77d2986152c380dcc55b20fdf340/markupsafe-3.0.4-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:d920abdfa61279ba1a2ef9484aab07bf03331f8c08a10120fa332353d06e6932", size = 22767, upload-time = "2026-10-02T23:06:04.699Z" }, + { url = "https://files.pythonhosted.org/packages/c6/61/ebda1307864b409e6b3115757a3d4a09cca46cfb6cc65191b5de226b424b/markupsafe-3.0.4-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a9f54054101545a9a9cccefddf54316aa6e4491611fcbef9e91b3b6bebec04f6", size = 23668, upload-time = "2026-10-02T23:06:05.9Z" }, + { url = "https://files.pythonhosted.org/packages/09/15/98075cceac3b5ba0dbb8e4762a847be967d2befc349a2cf2d0ac77f62c9d/markupsafe-3.0.4-cp314-cp314t-win32.whl", hash = "sha256:12a606a492de952afcb43b59a14aaaaad120e708d3663dd0fdf2d738d427a691", size = 14301, upload-time = "2026-10-02T23:06:07.109Z" }, + { url = "https://files.pythonhosted.org/packages/0b/a3/768b560fcc4156685cb563d922b217810cfa7bc135773367f62f1f9d2078/markupsafe-3.0.4-cp314-cp314t-win_amd64.whl", hash = "sha256:a18f38cafc329bac5e3c2b96c765b4c96d3d103421ed22ab7988c1e3fce27464", size = 14526, upload-time = "2026-10-02T23:06:08.276Z" }, + { url = "https://files.pythonhosted.org/packages/93/63/da554b4c97a6b0ea3229ca7fe8cbfb620be81613d517f482e85958550537/markupsafe-3.0.4-cp314-cp314t-win_arm64.whl", hash = "sha256:eba154571c16e032112afac0dc2dfe9e63c2ceb7aedd07bb7eecf2ce26d4dd4c", size = 14421, upload-time = "2026-10-02T23:06:09.402Z" }, + { url = "https://files.pythonhosted.org/packages/a9/30/54d11c8ca027114898cab97421fb39e4ffd9ddf47cdbc44df2ec76722da9/markupsafe-3.0.4-cp315-cp315-android_24_arm64_v8a.whl", hash = "sha256:737c9c3981998eba27f11786f84fddcbabc74068b72a4a1f454ea02094b57b65", size = 12791, upload-time = "2026-10-02T23:06:10.485Z" }, + { url = "https://files.pythonhosted.org/packages/10/6d/97c913e253a14bd3cd0e15a5c56d13203b823fa7ee32498342896a072dc4/markupsafe-3.0.4-cp315-cp315-android_24_x86_64.whl", hash = "sha256:489505b03f692c3f376394e49194fa7a7f9e8558d6e293a7056a0032b0c38163", size = 12527, upload-time = "2026-10-02T23:06:11.834Z" }, + { url = "https://files.pythonhosted.org/packages/26/f9/b86d032042a4d597d9e1997f0e5f63a3eedaf11258e0a05760b0a0a826ea/markupsafe-3.0.4-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:077293e425f28ec737dbcad442a71752e28f8ae27cde3d68acd1fb212091cd92", size = 11602, upload-time = "2026-10-02T23:06:13.122Z" }, + { url = "https://files.pythonhosted.org/packages/f2/dc/73c14c1eedf0ac5fa3292ba43435e6c49d2c2050f33cebde541f8f4807f1/markupsafe-3.0.4-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:9348cbb300d224fe3b89793262cb093504d4ae927004468463f745188a193e4a", size = 11817, upload-time = "2026-10-02T23:06:14.227Z" }, + { url = "https://files.pythonhosted.org/packages/8f/69/2c2fcaa5fcee22d72c7819c0d536fd181c74a688e6143845419579cd2863/markupsafe-3.0.4-cp315-cp315-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:b807e598953730f82e4eae3bd30f6a122cf6b31c398c6b504c0e04c13c170429", size = 11446, upload-time = "2026-10-02T23:06:15.574Z" }, + { url = "https://files.pythonhosted.org/packages/88/54/9e5ec76c62e6e2834d5a93623018c943e8b3bb41d663e3fd4c03303b9b85/markupsafe-3.0.4-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:799c39bdf5e2f1292fedd3009f7b3c9e760f10b2420cb9638d56920840ff6db8", size = 11719, upload-time = "2026-10-02T23:06:16.701Z" }, + { url = "https://files.pythonhosted.org/packages/96/24/3ec292b44064c16229e064d770b2625bd8ea941aa61f44905a9fa44942c0/markupsafe-3.0.4-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:ae9dcb8fbe244cb82f8a6458b455b927a03685e383d9bacf1ea5ce180b96dc97", size = 12075, upload-time = "2026-10-02T23:06:17.855Z" }, + { url = "https://files.pythonhosted.org/packages/aa/85/b64fdb1f304848518742136983c24e96d967bfb59a0ea160e92736901ab0/markupsafe-3.0.4-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4bced6e2a6dba6a28f7dd3c6ce14df1b2dd495923f16ea484cad03decd463b2b", size = 24656, upload-time = "2026-10-02T23:06:18.963Z" }, + { url = "https://files.pythonhosted.org/packages/9c/18/23997d4c65b355da6390d61cd56e0ab3befd6ba8dda25cb40c602bd0fa6b/markupsafe-3.0.4-cp315-cp315-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3882fb412298575bae3b9c46868251f15cc69307359f87bb1b382e53d6e5a2c9", size = 27737, upload-time = "2026-10-02T23:06:20.117Z" }, + { url = "https://files.pythonhosted.org/packages/d4/36/35998dead3c6af88c38265a56e58100211f036234ab88eb2283fd4cbce44/markupsafe-3.0.4-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:04e7902ba80ee4bac1d50a549606527a1dcf0476cd81403db41099d3b60ec653", size = 25003, upload-time = "2026-10-02T23:06:21.284Z" }, + { url = "https://files.pythonhosted.org/packages/82/96/ef49135ce260db4ca4a12b119ed468449cd248db6b1468e2112b546d7a2e/markupsafe-3.0.4-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:925f929d6b59a8b3f8b8c6ac363cd0af7eecc81efb3071770b3c6717c450a369", size = 23111, upload-time = "2026-10-02T23:06:22.524Z" }, + { url = "https://files.pythonhosted.org/packages/50/7d/83126e338bd88c17a220668235368ad719fd4638e426739858cbb8508f77/markupsafe-3.0.4-cp315-cp315-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f68edfc67aabac33708941f26f22a7b8e9f81429bc0cf249fcf7d66b23af8d19", size = 22403, upload-time = "2026-10-02T23:06:23.785Z" }, + { url = "https://files.pythonhosted.org/packages/83/dd/daf7e420de23c8206c365204e7b85e1251d8e19d34196a56336f316e5ed2/markupsafe-3.0.4-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:e5c802729725bd07e2bc3ab7b76dc7e0bbfc53129d8f1eb1c002c24cf774717e", size = 24083, upload-time = "2026-10-02T23:06:25.037Z" }, + { url = "https://files.pythonhosted.org/packages/19/3c/11eecdc06bc44ad5570350085b572ebf049e8f9a38d1ece6d76640b739cd/markupsafe-3.0.4-cp315-cp315-musllinux_1_2_armv7l.whl", hash = "sha256:55ffd6ce583d97dc71dc92e930324c8c0d25aea7e3ade6ae54ef77cedb096811", size = 22931, upload-time = "2026-10-02T23:06:26.328Z" }, + { url = "https://files.pythonhosted.org/packages/0d/9e/ac0fd77f2a726e56ecc3ca0235d095feace1358d1b822406c2a2ef26a4dc/markupsafe-3.0.4-cp315-cp315-musllinux_1_2_ppc64le.whl", hash = "sha256:2cb3dd71fc6be918ad4264346a8ed69485f9b7ed7bf35495d8e22807cd6b8bea", size = 24299, upload-time = "2026-10-02T23:06:27.742Z" }, + { url = "https://files.pythonhosted.org/packages/d7/09/c6bd842ad58ff5b3bc76eeed7e9a42a6f11adc5d090ec697b72c9672731e/markupsafe-3.0.4-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:94f5407f7bc64fa6463906b896f9904beeeb7dd8dc116ee8e9056c8714ff9916", size = 21948, upload-time = "2026-10-02T23:06:29.274Z" }, + { url = "https://files.pythonhosted.org/packages/a3/46/82f586711fed61e86faa1ee1bc317d68cd45a10c8bdbe3f7d1fdf9026ad8/markupsafe-3.0.4-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:2dad610540cb2e6272855c178f08ae9a1c7ac258a7fb71660553a5f104b42741", size = 23150, upload-time = "2026-10-02T23:06:30.583Z" }, + { url = "https://files.pythonhosted.org/packages/19/2d/2dfdce99318abbfa26925195fbc17db188c46a1ec6457be121b6f9cfeb42/markupsafe-3.0.4-cp315-cp315-win32.whl", hash = "sha256:03470d1a8268e692ecf79ecd565593e59d44219377a7ead61f1f1b94c1f7ff6b", size = 14269, upload-time = "2026-10-02T23:06:31.949Z" }, + { url = "https://files.pythonhosted.org/packages/5b/ec/6000fd82e8791e58fcd0456ec20f098957e2b03d5ed02eb73241a577c0ba/markupsafe-3.0.4-cp315-cp315-win_amd64.whl", hash = "sha256:d882a373d8093c2941e01291b7ced96e9cbe4781da9a7751ca7e6c70385e5214", size = 14496, upload-time = "2026-10-02T23:06:33.258Z" }, + { url = "https://files.pythonhosted.org/packages/bc/66/e73bd5016421d5d6e2fb6de7dd609f9de020942ac8c626526bd8c6eeaf82/markupsafe-3.0.4-cp315-cp315-win_arm64.whl", hash = "sha256:353bd63081912ab8cfa6a0c7d185934cdf8426f04c618bba6bc4b394f2069b67", size = 14361, upload-time = "2026-10-02T23:06:34.539Z" }, + { url = "https://files.pythonhosted.org/packages/90/df/cb8c3dc98d313a951df2f8968f44e4cb5643df6d3cab749a530ce2f7d972/markupsafe-3.0.4-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:c61750fadcd119d0825bcb7d7d675dd264dcc89cc05292aab5be68ebdbb374ad", size = 11760, upload-time = "2026-10-02T23:06:35.807Z" }, + { url = "https://files.pythonhosted.org/packages/d6/bb/4af9b3ca0753d654ac75f9531d5bd741bb77ca6e696f36807c475ffc099a/markupsafe-3.0.4-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:1c0df495a977d10460a94941799c72d5b5ab03d3858d949b55b5a66c8f371c99", size = 12108, upload-time = "2026-10-02T23:06:37.089Z" }, + { url = "https://files.pythonhosted.org/packages/3f/d4/b56429313aee5fd59b079c3df5615299959e25e7113eb6d8caadbdd7d38a/markupsafe-3.0.4-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:02fa4acbc6a3fc5c693c34d4dd8c1130b7fe99cc915181b0ddd6f72aeb296002", size = 25762, upload-time = "2026-10-02T23:06:38.419Z" }, + { url = "https://files.pythonhosted.org/packages/65/f5/34c181e891aa4f7d59c918584672e0c5eb7fffe76c1387d1246008bf4081/markupsafe-3.0.4-cp315-cp315t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:05295589e619b9bed252a86b532b8e27350abc372d18ba89b59375325e91ec1e", size = 28400, upload-time = "2026-10-02T23:06:39.819Z" }, + { url = "https://files.pythonhosted.org/packages/ce/b5/ad14694fd0ac9a5ce30bc6498f2999378f418583dd1679cca5a1b512957e/markupsafe-3.0.4-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:be6cb0c799abb0e2ba3e618e6d28ddddf7e485f6c2ce938dfa237daf3905072c", size = 25946, upload-time = "2026-10-02T23:06:41.381Z" }, + { url = "https://files.pythonhosted.org/packages/d6/a8/26b606445387d0ceb1eb1f21840094b84e4e3c3c3983d80d10b89823b490/markupsafe-3.0.4-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:26e9867520db70d37f7fb421a7f0d8adb40171011fb84ce869afa1a83370dfa8", size = 23634, upload-time = "2026-10-02T23:06:42.748Z" }, + { url = "https://files.pythonhosted.org/packages/39/a2/b8814de672f1f0094d498bf646f2fec9d6356b503d28ef500b71c5095377/markupsafe-3.0.4-cp315-cp315t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f03460ff076f70ab595bb45a0205ccea1971443575b6920c52e755dec2b3fbfe", size = 23488, upload-time = "2026-10-02T23:06:44.176Z" }, + { url = "https://files.pythonhosted.org/packages/db/c7/287223376fb73335a3cc5d6eb22c6ab01358cf33945a9c39c06b9dac3f4b/markupsafe-3.0.4-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:436e3ffc6310d3c41878c601db29098102fe5d8a467c49da4a4125254e0980f2", size = 25037, upload-time = "2026-10-02T23:06:45.646Z" }, + { url = "https://files.pythonhosted.org/packages/f9/29/4df8355e313426d19e62ba33e0253c009ca12a0894ee77d67fa67255361c/markupsafe-3.0.4-cp315-cp315t-musllinux_1_2_armv7l.whl", hash = "sha256:4e2c4809c14559aa7ef426f27fb35afbb38104c349a903bf8f3600456764bb38", size = 23543, upload-time = "2026-10-02T23:06:47.264Z" }, + { url = "https://files.pythonhosted.org/packages/71/e5/8377731e8495668dcc768f645e717df18318c841edaf023a99395f6da9b4/markupsafe-3.0.4-cp315-cp315t-musllinux_1_2_ppc64le.whl", hash = "sha256:da2af0d7aebfc2074080d72efa6ab8317c62481ef1f896f65d9999c1c01f4494", size = 25282, upload-time = "2026-10-02T23:06:48.795Z" }, + { url = "https://files.pythonhosted.org/packages/ed/5f/373456e37ceb1478d657d6fe769cbe0a39f0a8dfc1548eeb19c471eefdd9/markupsafe-3.0.4-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:aa2c838cc024642cc04c6854232f32b43e5e22833dd11119c1766c7873b8370d", size = 22993, upload-time = "2026-10-02T23:06:50.31Z" }, + { url = "https://files.pythonhosted.org/packages/d7/93/2cbd5628435afb6f541bbaced4bce0c2edac4b09a142e6e928b8b0da9858/markupsafe-3.0.4-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:b91cc9d336957239ff200f30097e6fea2dc6d6fb3c81e853eaa09eac904fd894", size = 23712, upload-time = "2026-10-02T23:06:51.759Z" }, + { url = "https://files.pythonhosted.org/packages/81/99/157e10966b033b363aeda5263e82596ee232a0b1d082fdbf90aa417ff083/markupsafe-3.0.4-cp315-cp315t-win32.whl", hash = "sha256:e49fb0d1ce92cfa0cb198cc5b1b11cdf9d0638658e2a2db2687e39db7c87fc78", size = 14300, upload-time = "2026-10-02T23:06:53.241Z" }, + { url = "https://files.pythonhosted.org/packages/33/05/55884815414c9706a23deca150b72c25a62109e65b0b6ce232077802c719/markupsafe-3.0.4-cp315-cp315t-win_amd64.whl", hash = "sha256:4f6e0852a0283b1b1fd776eeb7b766a5f440b3e2bd31ab51af3b400585f3965c", size = 14522, upload-time = "2026-10-02T23:06:54.729Z" }, + { url = "https://files.pythonhosted.org/packages/92/f9/ecbde7149e95b8a0f18e16d5d747f7dc06049d5da2e4f77f6f5e4a1f46a8/markupsafe-3.0.4-cp315-cp315t-win_arm64.whl", hash = "sha256:39dbacefc411633db5b4378b066a9aca70a3d7e2922c9e578d825f844026eeba", size = 14417, upload-time = "2026-10-02T23:06:56.246Z" }, +] + +[[package]] +name = "mdurl" +version = "0.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" }, +] + +[[package]] +name = "packaging" +version = "26.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/fa/3944b40b07da9ce895c0e6303a5ab7d53da063554f534556b134a54d6093/packaging-26.3.tar.gz", hash = "sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79", size = 313412, upload-time = "2026-08-04T18:15:28.737Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/63/34/ba1c580383c9eada3711951fef0795c80b829a078d72188184bcab9dd527/packaging-26.3-py3-none-any.whl", hash = "sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", size = 129956, upload-time = "2026-08-04T18:15:27.159Z" }, +] + +[[package]] +name = "pluggy" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + +[[package]] +name = "pygments" +version = "2.21.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/49/2e/ced460408999b33da6b31b0021b0f37d329e202d4169aeb164493778f25b/pygments-2.21.0.tar.gz", hash = "sha256:610ca751c9bc2492b38eb9a38a7fbc93edbbb2d7182edaf34e66ae493dee5c8c", size = 5005329, upload-time = "2026-08-17T08:02:48.824Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/71/46/17f022dd3e953bf20a04a028a21ec746d942f8d2af30fa0f124fa0e6a684/pygments-2.21.0-py3-none-any.whl", hash = "sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9", size = 1250147, upload-time = "2026-08-17T08:02:44.912Z" }, +] + +[[package]] +name = "pytest" +version = "9.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e4/47/b9efed96c114afcfa3c9d3fe98a76a1d14c74a9e266d397cf6eb64be5e01/pytest-9.1.1.tar.gz", hash = "sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313", size = 1636369, upload-time = "2026-06-19T10:58:32.857Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/24/25/1de2678b631f5a49215c6c96fff41ba892b0a34df68d6d80292b1b48aa7f/pytest-9.1.1-py3-none-any.whl", hash = "sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c", size = 386536, upload-time = "2026-06-19T10:58:31.347Z" }, +] + +[[package]] +name = "ruff" +version = "0.16.10" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c4/49/23802c45f093eb14bde54b141d2b2f058edfa63a06db7beed047308cc08f/ruff-0.16.10.tar.gz", hash = "sha256:eff4728c4eaae93f0955cd264d24b2ab348e74bf59986ccf282ba6dc16b3b017", size = 4958724, upload-time = "2026-10-01T18:03:21.697Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2f/21/ebce22e1d90cdb2cd691026b9c6e9bec6499f0b396089481755b5d49efff/ruff-0.16.10-py3-none-linux_armv6l.whl", hash = "sha256:488b0fe3f3574210e5cf80d9f59b9e3ab17a127a8155de3f307b392589cfb511", size = 10095557, upload-time = "2026-10-01T18:02:36.072Z" }, + { url = "https://files.pythonhosted.org/packages/cb/98/a54de85876a8b2612bfa0d84c7b9abfb39c6a3354aee7800b09c1649b9e3/ruff-0.16.10-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:e748ff95c934c4e978783b8e687bc174e7bd84e8ad24e3243e1ecfcda5e0282d", size = 10340577, upload-time = "2026-10-01T18:02:39.258Z" }, + { url = "https://files.pythonhosted.org/packages/9f/16/1a5a4a2657effe4806110f2b907313802f1367fcdbb29e8122766e407fab/ruff-0.16.10-py3-none-macosx_11_0_arm64.whl", hash = "sha256:3031a4a2e8e7b8a46f70be45f198c35a11ece509a94b80334d8d397a33c67550", size = 9774282, upload-time = "2026-10-01T18:02:41.79Z" }, + { url = "https://files.pythonhosted.org/packages/6e/fb/470085af734da396e68cd80fb0a3e7c109a459716ae59588c0f6fab8a17d/ruff-0.16.10-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:494401c86df4c4c25f69b9419605d944467ee98c42fb6ad405ef4fa40b8fb67d", size = 9920895, upload-time = "2026-10-01T18:02:44.458Z" }, + { url = "https://files.pythonhosted.org/packages/57/de/f10cffe4f88a37ea6615ec460f01bf76f0bc1477737a35d9ee61a630a78b/ruff-0.16.10-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d203abc0ff2b773ee33d00ab8df0bb67046fbc7c07b119332f08b9b345cf8221", size = 9902707, upload-time = "2026-10-01T18:02:47.041Z" }, + { url = "https://files.pythonhosted.org/packages/ff/44/3fdcedf83ae60ef837dd239170e480dce606a9142cfa2db911afdf855fd9/ruff-0.16.10-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:bd83d1235a5258d318477bc5b576303974cbdef5df0c01a1bff14efcc23bd12a", size = 10619287, upload-time = "2026-10-01T18:02:49.485Z" }, + { url = "https://files.pythonhosted.org/packages/c1/62/02e76a5574002153618eafbb70e159468addc72a5d2c488e65cbb4639d2d/ruff-0.16.10-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:bc2610fb269fa56dd8a68669ae470fa6272902668c0fc2ebc3aa112b2633d5b8", size = 11339942, upload-time = "2026-10-01T18:02:52.008Z" }, + { url = "https://files.pythonhosted.org/packages/6b/c4/cde27d47ad8d4126c587e608c67c46e7feac11763f606d261a1bc8a489e6/ruff-0.16.10-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3e70175e29cc94c26ea296c80e470180b744b7419026898e58f520c6ab32578e", size = 10934316, upload-time = "2026-10-01T18:02:54.811Z" }, + { url = "https://files.pythonhosted.org/packages/e4/03/17234145f302a645a123e8c3bb2411ecf4669fbc350de3b0d803230a1729/ruff-0.16.10-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f33f43a864a8483eebd160e713336c8bab02c934feaff0a33cf5ccb41546d09a", size = 10387968, upload-time = "2026-10-01T18:02:57.497Z" }, + { url = "https://files.pythonhosted.org/packages/71/29/2493af60240ee7644b38a4b821f5f9c3b5a4fa3178770fe1d0c685217395/ruff-0.16.10-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:1dfc6f0088149fb6a362c1c446bcbb3fd2157b3852fe2fa68409276eab9ad9b3", size = 10537906, upload-time = "2026-10-01T18:03:00.006Z" }, + { url = "https://files.pythonhosted.org/packages/2e/9a/f56b28f3b143bb9e518c34fb89ab191b626bca8b70dbf8af0d2e2d473572/ruff-0.16.10-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:6553498afc35f580f036030795810b9e6bcea31604b0fd9e8d352795473042e3", size = 10012938, upload-time = "2026-10-01T18:03:03.006Z" }, + { url = "https://files.pythonhosted.org/packages/c2/c4/fca37362848ea4d4d80712df13632e645e7c7cfb1bdedf140699ac7a090b/ruff-0.16.10-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:a3b8471dea115d37f123882be852bed13403746d3a76c11de4a19ec5f9ff5a03", size = 9897945, upload-time = "2026-10-01T18:03:05.818Z" }, + { url = "https://files.pythonhosted.org/packages/7d/c7/e0bc57664d6e0c61fd22f620af260fe9662d7e1ddb320ea7f160e76ef165/ruff-0.16.10-py3-none-musllinux_1_2_i686.whl", hash = "sha256:92e59a70bcbd9d3a5483656da906ec28edfdacfce00afd99edb8b4e9d15644be", size = 10333134, upload-time = "2026-10-01T18:03:08.25Z" }, + { url = "https://files.pythonhosted.org/packages/21/aa/5c9f3b68737c0e4a33a1db5dfab44f7b786d96ca91a7233112ddab1e6dc2/ruff-0.16.10-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:7ae7375f803b5520dc9f546bed7e3a0acb70b91812e9bb4b19927de22f25b77d", size = 10741702, upload-time = "2026-10-01T18:03:10.775Z" }, + { url = "https://files.pythonhosted.org/packages/78/fa/0f9c2020dc316c53d983be011720b8157cc052b97e3f4dfa7db6d0f880a6/ruff-0.16.10-py3-none-win32.whl", hash = "sha256:2a12e01cb9156c10c466f63b46eaae5ecea28dfbd21b5836353ae498e7d1349a", size = 10139176, upload-time = "2026-10-01T18:03:13.267Z" }, + { url = "https://files.pythonhosted.org/packages/99/29/cfb0df9448d4d4ad48c2de029ada9ebd71baa6da983a6c77ee6c6cd0fe82/ruff-0.16.10-py3-none-win_amd64.whl", hash = "sha256:97f2015c92aa97105b0eab19eb5d224884399281cfc5da86a92db4ab5e7fb2ca", size = 10584734, upload-time = "2026-10-01T18:03:16.006Z" }, + { url = "https://files.pythonhosted.org/packages/fc/05/c16957eb287c3fc062e032619a25868d93d408a844b725bcf514f7a378ff/ruff-0.16.10-py3-none-win_arm64.whl", hash = "sha256:25a65fe998c4e6861ec079ada5826a2fc605e6cbccbe9dcd7fac1f54e791621b", size = 10366440, upload-time = "2026-10-01T18:03:19.04Z" }, +] + +[[package]] +name = "ty" +version = "0.0.85" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/66/8f/8de9c4ff90e7131a3c832ea074e123d551fed84c899d37a522f649ca7c6a/ty-0.0.85.tar.gz", hash = "sha256:ef442cbc2fd6dad02eb00dee4213c1e0f18e09bcbeec2cdce6aa802603c5c6bb", size = 7519853, upload-time = "2026-10-06T22:05:41.732Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1f/06/1716c1778088c5c8f96b248f0e0304e839ca69c096bde82b827bc00e4194/ty-0.0.85-py3-none-linux_armv6l.whl", hash = "sha256:012d71e0dcb51e74599adfb0e5c1416bdac3bf983115e42230545c782b06cfca", size = 14097338, upload-time = "2026-10-06T22:04:58.838Z" }, + { url = "https://files.pythonhosted.org/packages/4a/26/d6b2c7221096c3fa36a0afd4aa8134ffd450df63635ec3862a5a8cf6c9fd/ty-0.0.85-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:391d1993ea3b26509ddb3878dbe79e4cb92487c73109893ee2e51394846e0a69", size = 13684794, upload-time = "2026-10-06T22:05:02.204Z" }, + { url = "https://files.pythonhosted.org/packages/e7/7a/dd1e2e1c54551d61451c18eeb918db833fdbc0a48022fd2bffc93c755735/ty-0.0.85-py3-none-macosx_11_0_arm64.whl", hash = "sha256:90e1bb090348cb560c7ba6109f2c898735b0520fd3114cd316d8bd80439aabb7", size = 13318789, upload-time = "2026-10-06T22:05:04.61Z" }, + { url = "https://files.pythonhosted.org/packages/ff/9c/6c4fc7badf3e1e2e160d0b5c68632d85eb48dd0341422ac44fdd855d1eaf/ty-0.0.85-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:311c131f735bf76be8852305817d5fd10165d6289c76350f6188b0749a66a7f8", size = 13526827, upload-time = "2026-10-06T22:05:07.046Z" }, + { url = "https://files.pythonhosted.org/packages/ba/e5/f6f2d279ef810d89b688425a72150bbbd84a2d6bf54eb9a5ae172f13e3a5/ty-0.0.85-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a0971dc7b2f9e9cabdf654f53c8a40e563921227d05567b670f171cba2b3e385", size = 13823871, upload-time = "2026-10-06T22:05:09.731Z" }, + { url = "https://files.pythonhosted.org/packages/47/5d/1210c1f59e8c8c970807a2f13c180fa953ce58a6ddf28e90564db6f8890d/ty-0.0.85-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:5fada6d10172b4c88d44b1f9697c9f5f40d34f8fb40fe716da0366f094268c9b", size = 14727151, upload-time = "2026-10-06T22:05:12.299Z" }, + { url = "https://files.pythonhosted.org/packages/20/26/cdd96e45c33079174d5cd70649a5ae77758bc209f0685e8a4fecee9b7094/ty-0.0.85-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d06c9aa616672e9f9856ff7b539be5c3a4719702e5636e58b4196a8c850a884b", size = 15148511, upload-time = "2026-10-06T22:05:15.304Z" }, + { url = "https://files.pythonhosted.org/packages/f0/28/fd471f2edc0f38a70f0494f9949d1983591073555d254031542c0ed29454/ty-0.0.85-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2280d31c369ef61ed9a4c2399cd663340d85a5fd7965a8ad881203ea6b8b130e", size = 15090530, upload-time = "2026-10-06T22:05:17.874Z" }, + { url = "https://files.pythonhosted.org/packages/7b/4f/6ebfe4f14c399856fc80c448ab442521072f688dfde72b075d8a90d7cafa/ty-0.0.85-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:cc1fff0eab4e73fcc66df761a7a4759da04d3e9c2b82d57c628c50434e2ff0d7", size = 14322051, upload-time = "2026-10-06T22:05:20.366Z" }, + { url = "https://files.pythonhosted.org/packages/36/38/866054d20e12e72a76a37eaf60f1374f7ad4d0b973c145685782065d84cf/ty-0.0.85-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:453dcb85b5f2486ddc6ade57a50698e9a8d8e3aae34e50f0b20bc5e8c2a3bc2b", size = 14637631, upload-time = "2026-10-06T22:05:22.628Z" }, + { url = "https://files.pythonhosted.org/packages/08/16/700c730b4978de81bab82974c23db0229e58d852e10767e1f1f8b0e217db/ty-0.0.85-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:820eb594b1d30cd6a0776e59449ed6994b03c27af8da0c287ad351de193eb4a7", size = 13553358, upload-time = "2026-10-06T22:05:24.903Z" }, + { url = "https://files.pythonhosted.org/packages/2e/60/d2dda3a3443d55083429b1eccabed95cbf8769bc61c51831e0ece890780f/ty-0.0.85-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:4b76c6a2fc5974fc2c5f95156a34f44127be794f4b3dc9793aae5b6a4b774e83", size = 13837412, upload-time = "2026-10-06T22:05:27.163Z" }, + { url = "https://files.pythonhosted.org/packages/f2/54/05043adf78fa5c0440fe9b956505b24d2568da256b60e7479c691510bb92/ty-0.0.85-py3-none-musllinux_1_2_i686.whl", hash = "sha256:0e3a32de0731efce574367fdd680b7524c681909a92e6cf64bb40ad48ccbc110", size = 14167568, upload-time = "2026-10-06T22:05:29.519Z" }, + { url = "https://files.pythonhosted.org/packages/9d/31/0372621051cc807f23d2e9c55bab65818ed131852c05fbe4c65be365895e/ty-0.0.85-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:d622348c05a9fe99ad087dc93f174617a96f9fef23dd7a0acbdb0adaf8bd5ae6", size = 14489824, upload-time = "2026-10-06T22:05:32.089Z" }, + { url = "https://files.pythonhosted.org/packages/b5/64/0fca31db8c48dc72285f17b878e2c6a015279ab04a26e16f62abb384094f/ty-0.0.85-py3-none-win32.whl", hash = "sha256:0742d21a002481af8d2324b16ce5df210b5bb7673b0ea1096cfc02745c6b7280", size = 13410281, upload-time = "2026-10-06T22:05:34.418Z" }, + { url = "https://files.pythonhosted.org/packages/f6/a8/65d865185ecb18ee45b85cd99b34aedb3d8a4a9fe0438dab695e1390dbb7/ty-0.0.85-py3-none-win_amd64.whl", hash = "sha256:2636f8255b97c357dac1f9f50ca58f58a49642c8adb5c76643f483401a577014", size = 14112013, upload-time = "2026-10-06T22:05:36.946Z" }, + { url = "https://files.pythonhosted.org/packages/24/40/110b4f01af47fbb061f03e2f8e85cb30dd3d9ea614b2059dfaa058a044a5/ty-0.0.85-py3-none-win_arm64.whl", hash = "sha256:953574167a79d9dba7781238879955fa37fea405393710bfe53069ac82dff4d9", size = 13825189, upload-time = "2026-10-06T22:05:39.467Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" }, +] + +[[package]] +name = "watchfiles" +version = "1.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b3/68/e6aa0b77d217b31f8f486ec0cdfe5e00e6e38dc0be657e7d85819b9faf0a/watchfiles-1.3.0.tar.gz", hash = "sha256:99aee4a07847c06820765fd7b1b49ceac4f3f711ccb7d104655a33231de1c207", size = 108119, upload-time = "2026-09-21T09:08:55.664Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/68/fa/c0b840d5d8bafe640925408ce11095948c8945bb61be1e67d0ab629b872a/watchfiles-1.3.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:000b9688fc8133037a8b075c8ebf98f32844ff8964dda61e85c1db547dafc441", size = 399703, upload-time = "2026-09-21T09:08:13.518Z" }, + { url = "https://files.pythonhosted.org/packages/c7/8a/894799b485fe9473ad10422a0d9668e53fb78e3a2b5cc6061159572844ad/watchfiles-1.3.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:bbc1198edfdc90fda0600f825aa94150f428dfcbf8138746f55998e0e660d64c", size = 397647, upload-time = "2026-09-21T09:08:15.16Z" }, + { url = "https://files.pythonhosted.org/packages/5a/61/f277cd5ead05f5d37b98b41fbdd0c943ee0c04d49738f62e902cb291cf43/watchfiles-1.3.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4af9c464d410e8c44b58ddf1cbec7c3a02c1cb2c6c80df9c74659ba2fbda0ab5", size = 454994, upload-time = "2026-09-21T09:08:16.587Z" }, + { url = "https://files.pythonhosted.org/packages/46/66/ea4c01382975e53b1ac1b0a0ea1b353416b8127fb6e04f5c1c313119aab6/watchfiles-1.3.0-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:eb94e40b9a0db19636b5e3fed0a9803452ecc7381df13b385272abb779ee83ca", size = 460105, upload-time = "2026-09-21T09:08:17.961Z" }, + { url = "https://files.pythonhosted.org/packages/31/c3/1a501b597817096c9831c255f6270c124ce0e2faeb74124be24dcc93b86d/watchfiles-1.3.0-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7398930a2b76b9dc67a4e6b8bfc7baf30946892a86f417a3920c07bdd6febc30", size = 493761, upload-time = "2026-09-21T09:08:19.327Z" }, + { url = "https://files.pythonhosted.org/packages/5a/16/77352d05e152c7786a016e9f1decdb4507e8aa5bcf3343c72142a0413d14/watchfiles-1.3.0-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:025b108f2d5cc2799cb941f79380df3e8b22bfb589c218e801767fe78d598c95", size = 577500, upload-time = "2026-09-21T09:08:20.723Z" }, + { url = "https://files.pythonhosted.org/packages/95/fe/81001c55b24466fe9911135e50269fd3a29f43e446627a92adfab54553b6/watchfiles-1.3.0-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:be9ef3cd403d756a304a0a08a112163c63da7ec72c8c9d083c98818291ff8f29", size = 469209, upload-time = "2026-09-21T09:08:22.346Z" }, + { url = "https://files.pythonhosted.org/packages/8a/cf/e3ad894ed5909a7c44b0707e89bc25d19a3d78b3a7aec846119443b6ae56/watchfiles-1.3.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b5768b49e426fd5b550b012c866db347cdf15c398ef98dc557b6e6b72fa74cd1", size = 458353, upload-time = "2026-09-21T09:08:23.7Z" }, + { url = "https://files.pythonhosted.org/packages/7f/f5/2f106829e6e2bc91ad74614730f6df484202b9a4d20d927dbf2f08d37d44/watchfiles-1.3.0-cp310-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:d36c72fcdc08f143d87316a4c4c3b064e340da2c7383d39803950740ed418935", size = 468707, upload-time = "2026-09-21T09:08:25.167Z" }, + { url = "https://files.pythonhosted.org/packages/15/37/2741ab776b978669c1f3bba1e573f1a184f727ab8b4d3a6d9c61090800ed/watchfiles-1.3.0-cp310-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:f0c9865240e065a2247f4b5248528ae3d01e0e8ae071462c250c884dc16e3ee3", size = 632510, upload-time = "2026-09-21T09:08:26.532Z" }, + { url = "https://files.pythonhosted.org/packages/48/64/db4a4684275fe9b696ee5118c194705c1423b3d42181c3f8ee70194d5c73/watchfiles-1.3.0-cp310-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:380f513d26cc2e598266b88d66456e07d67ef4be8f0f435a1154d9c44b43d509", size = 661723, upload-time = "2026-09-21T09:08:28.004Z" }, + { url = "https://files.pythonhosted.org/packages/52/80/95ddb24f6a2f595c031c24432ab7b8aa7d6d1ada0c6d63a552166e06e8d1/watchfiles-1.3.0-cp310-abi3-win32.whl", hash = "sha256:509d9f74d2bec5c1f4cd868ddcfe0c9601f7ea53066dfcb19d8ede742c4de661", size = 277761, upload-time = "2026-09-21T09:08:29.335Z" }, + { url = "https://files.pythonhosted.org/packages/48/a1/1c4a1b3c8030cd03206232612be8a6e8799093490c220b0fcb935d02b75a/watchfiles-1.3.0-cp310-abi3-win_amd64.whl", hash = "sha256:1acabde19b67e673274e89a04d613b7fb1f122e2c4b8ec9a581e59c149264b61", size = 290643, upload-time = "2026-09-21T09:08:30.45Z" }, + { url = "https://files.pythonhosted.org/packages/82/e0/d2db577c632d26d08078c4390dd870b8555c4e558df25d22270f0166c71f/watchfiles-1.3.0-cp310-abi3-win_arm64.whl", hash = "sha256:d978cc1dd7ba44f5590d7de74f7a9c8be7262d9e469c2c0efb8bbfc2574321dc", size = 285510, upload-time = "2026-09-21T09:08:31.561Z" }, + { url = "https://files.pythonhosted.org/packages/4f/55/a122743607939429ab6a7272672f49303aa02bbd34c3f06676765a3861de/watchfiles-1.3.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:e0204f90677b7fb8d3a824279a71a01166bea9f6e53923b8056592d87b33b0b6", size = 395978, upload-time = "2026-09-21T09:08:32.747Z" }, + { url = "https://files.pythonhosted.org/packages/3e/9d/0ee1b0ea42acfe72afa2484f0222e2386dd09992e9034a81e63cc3cdab0a/watchfiles-1.3.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7c1125e99f84934a92996ae343501285617373acd66f16b59fd5654b27ad8d80", size = 387997, upload-time = "2026-09-21T09:08:33.953Z" }, + { url = "https://files.pythonhosted.org/packages/b7/f2/4319c6b683e8bd28465dfc53c750617a2a07f072eb2e3ab0182ccf812229/watchfiles-1.3.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a85e243c87109d9f48a2e0d10d6abdee8768cc114848b32281ec83d7f81d86ef", size = 448067, upload-time = "2026-09-21T09:08:35.104Z" }, + { url = "https://files.pythonhosted.org/packages/b7/71/9f37ac192a51a24de3a2d7fe9aac1e99b2b2668300674ff7a9f85d95beb1/watchfiles-1.3.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:dd538c71766c59c732e2c5cedc39323c99c11897b99f93eac773438e54af206a", size = 452881, upload-time = "2026-09-21T09:08:36.238Z" }, + { url = "https://files.pythonhosted.org/packages/34/07/1224667c32a4a96876ddd642da69733cbfe6a15897db0865aec5a882cebc/watchfiles-1.3.0-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:da213822ec9082b62cadf3008e07a8c690ec7a3c89f6dc5e55a8cf07b56a1b26", size = 484802, upload-time = "2026-09-21T09:08:37.717Z" }, + { url = "https://files.pythonhosted.org/packages/bd/4d/7f29cd87e1f826df86fc0418d0b8f370ca0311c29c618406dc492aad9951/watchfiles-1.3.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d4593392e87669836670f87d62fc754059eaeb0e6157af44060a46558cebe5a3", size = 567522, upload-time = "2026-09-21T09:08:38.94Z" }, + { url = "https://files.pythonhosted.org/packages/e3/ac/3aa1f84abd4dbbfbae99a817dd931f77b439261c911189cad4c30fdfd470/watchfiles-1.3.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:26b81bc515a0f03f66a69a6fea86eea54c8eca4173c46dd7222569f79dd2f977", size = 461997, upload-time = "2026-09-21T09:08:40.142Z" }, + { url = "https://files.pythonhosted.org/packages/82/1c/27c66764d5b0dbb70a36770a32eca217eba739c28b24e8583a035087a3d7/watchfiles-1.3.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ca39661934749df580d89f3dde266f2af7c6968f4c21c72ce2dc387d299e6aa7", size = 451856, upload-time = "2026-09-21T09:08:41.305Z" }, + { url = "https://files.pythonhosted.org/packages/1e/b3/a45827084d953c1d0dd3d75abf85dcac26a0f5d115ad2fe0b7e9e931573b/watchfiles-1.3.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:04c5500faa725d69a99b0f63750e80795e455bb951849e9c98a811b4324997c4", size = 459889, upload-time = "2026-09-21T09:08:42.58Z" }, + { url = "https://files.pythonhosted.org/packages/7a/3d/07740d23ce2bb23419bb24055c435a9abe5fe03efdaea1ba0720b831aab6/watchfiles-1.3.0-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:ec3cd4bb181a7b6329134204c2277c7144ab5f62cce9c5335d6f901ee7ce8d6a", size = 624482, upload-time = "2026-09-21T09:08:44.064Z" }, + { url = "https://files.pythonhosted.org/packages/91/46/9540f8123eaec86df989f26d7704cad0448d9fb7271763a4f8f9fe23617d/watchfiles-1.3.0-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:e5871d4a7f788a9f64fb06f5793389d29d171e5c32d37c3ebfd520b017cc7694", size = 653724, upload-time = "2026-09-21T09:08:45.257Z" }, + { url = "https://files.pythonhosted.org/packages/25/9d/b9deb52f1e4528ffe44338c5ebab95c32ed79203082da661de50288e130a/watchfiles-1.3.0-cp314-cp314t-win32.whl", hash = "sha256:81c989d7267bfb7b676cd32c83bec97914e396870983cbf493e29415037bff5e", size = 270863, upload-time = "2026-09-21T09:08:46.518Z" }, + { url = "https://files.pythonhosted.org/packages/59/59/8b1881afccbafda06248be4185e28034dd73c80c85e4da24f2cb0c1321eb/watchfiles-1.3.0-cp314-cp314t-win_amd64.whl", hash = "sha256:d754e049006e81a9be49dc78ba7802081cee426e968cbc63ee19e164fa2d0e39", size = 286879, upload-time = "2026-09-21T09:08:47.874Z" }, + { url = "https://files.pythonhosted.org/packages/46/dd/41c12ae9745d5d72f709333287679d71cbba68ad7678fed872207bba94f4/watchfiles-1.3.0-cp314-cp314t-win_arm64.whl", hash = "sha256:a5b631db08fd3032db57ec6a92d1777cd60f88803287e4e941a4858c53aab1e0", size = 276924, upload-time = "2026-09-21T09:08:48.976Z" }, +] diff --git a/website/build.py b/website/build.py new file mode 100755 index 0000000000..fde14149a5 --- /dev/null +++ b/website/build.py @@ -0,0 +1,1018 @@ +#!/usr/bin/env python3 +"""Build a single-page HTML site from README.md for the awesome-python website.""" + +import json +import re +import shutil +import subprocess +import xml.etree.ElementTree as ET +from collections import Counter +from collections.abc import Sequence +from datetime import UTC, datetime +from pathlib import Path +from typing import TypedDict + +from jinja2 import Environment, FileSystemLoader +from markdown_it import MarkdownIt +from markdown_it.tree import SyntaxTreeNode + +from fetch_pypi_downloads_via_clickpy import OVERRIDES_FILE, normalize +from readme_parser import AlsoSee, ParsedGroup, ParsedSection, parse_readme, parse_sponsors, render_inline_text, slugify + +GITHUB_REPO_URL_RE = re.compile(r"^https?://github\.com/([^/]+/[^/]+?)(?:\.git)?/?$") +MARKDOWN_LINK_RE = re.compile(r"\[([^\]]+)\]\(([^)\s]+)\)") +BULLET_LINE_RE = re.compile(r"^\s*-\s") +ANCHOR_LINK_ATTRS_RE = re.compile(r'href="(#[^"]*)" target="_blank" rel="noopener"') +SITE_URL = "https://awesome-python.com/" +SITEMAP_URL = f"{SITE_URL}sitemap.xml" +SITEMAP_NS = "http://www.sitemaps.org/schemas/sitemap/0.9" +PLURAL_NOUNS = {"Clients", "Drivers", "Engines", "Files", "Frameworks", "Generators", "Implementations", "Panels", "Queues", "Repositories", "Schedulers", "Servers", "Stdlib", "Tools"} + +BUILTIN_FILTER = "Stdlib" +BUILTIN_SLUG = "built-in" +BUILTIN_PATH = f"/categories/{BUILTIN_SLUG}/" +BUILTIN_PUBLIC_URL = f"{SITE_URL}categories/{BUILTIN_SLUG}/" + +SPONSORSHIP_PATH = "/sponsorship/" +SPONSORSHIP_PUBLIC_URL = f"{SITE_URL}sponsorship/" +SPONSORSHIP_DESCRIPTION = "Sponsorship for awesome-python: tiers, audience, and how to get your product in front of professional Python developers evaluating tools for production use." + +BUNDLED_PREFIX_RE = re.compile(r"^\(part of ") + +# Category intros name projects as their docs do, README entries by their PyPI name +PICK_ALIASES = { + "Airflow": "apache-airflow", + "Beautiful Soup": "beautifulsoup4", + "Cacheops": "django-cacheops", + "Chroma": "chromadb", + "Coverage.py": "coverage", + "Faiss": "faiss-cpu", + "gRPC": "grpcio", + "Grappelli": "django-grappelli", + "Hydra": "hydra-core", + "Ibis": "ibis-framework", + "Material for MkDocs": "mkdocs-material", + "OpenCV": "opencv-python", + "Playwright": "playwright-python", + "PyAV": "av", + "python-oracledb": "oracledb", + "Sherlock": "sherlock-project", + "Silk": "django-silk", + "Surprise": "scikit-surprise", + "Unfold": "django-unfold", + "Vega-Altair": "altair", +} + +SOURCE_TYPE_DOMAINS = { + "docs.python.org": BUILTIN_FILTER, + "gitlab.com": "GitLab", + "bitbucket.org": "Bitbucket", +} + + +class TemplateSubcategory(TypedDict): + name: str + value: str + slug: str + url: str + + +class TemplateEntry(TypedDict): + name: str + url: str + description: str + categories: list[str] + groups: list[str] + subcategories: list[TemplateSubcategory] + stars: int | None + downloads: int | None + owner: str | None + last_commit_at: str | None + source_type: str | None + bundled: bool + badge: str | None + also_see: list[AlsoSee] + + +class EntryGroup(TypedDict): + name: str # empty for a page with a single unnamed group + slug: str + url: str # links the group heading to its own page, empty when it has none + lead: str # the section intro's first paragraph as HTML, shown under the heading on group pages + entries: list[TemplateEntry] + + +class SyntheticCategory(TypedDict): + name: str + slug: str + description: str + description_html: str + + +TemplateCategory = ParsedSection | SyntheticCategory + + +def detect_source_type(url: str) -> str | None: + """Detect source type from URL domain. Returns None for GitHub URLs.""" + if GITHUB_REPO_URL_RE.match(url): + return None + for domain, source_type in SOURCE_TYPE_DOMAINS.items(): + if domain in url: + return source_type + if "github.com" not in url: + return "External" + return None + + +def extract_github_repo(url: str) -> str | None: + """Extract owner/repo from a GitHub repo URL. Returns None for non-GitHub URLs.""" + m = GITHUB_REPO_URL_RE.match(url) + return m.group(1) if m else None + + +def load_stars(path: Path) -> dict[str, dict]: + """Load star data from JSON. Returns empty dict if file doesn't exist or is corrupt.""" + if path.exists(): + try: + return json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + return {} + return {} + + +def load_pypi_badges() -> dict[str, str]: + """Load the badge each entry shows in place of a download count. + + Keyed by normalized README name. Only entries whose missing count needs + explaining beyond "Not on PyPI" carry one, e.g. the SDK monorepos that do + ship on PyPI, just as many per-service packages. + """ + raw = json.loads(OVERRIDES_FILE.read_text()) + return {name: entry["badge"] for name, entry in raw.items() if entry.get("badge")} + + +def load_downloads(path: Path) -> dict[str, int]: + """Load last-30-day download counts from the TSV cache, keyed by normalized README name. + + Columns: name, package, downloads, fetched_at. Skips the header and + NOT_FOUND rows. Returns empty dict if the file doesn't exist. + """ + if not path.exists(): + return {} + downloads: dict[str, int] = {} + for line in path.read_text(encoding="utf-8").splitlines()[1:]: + parts = line.split("\t") + if len(parts) >= 3 and parts[2].isdigit(): + downloads[parts[0]] = int(parts[2]) + return downloads + + +def sort_entries(entries: Sequence[TemplateEntry]) -> list[TemplateEntry]: + """Sort entries by downloads descending, then name ascending. + + Four tiers: entries with download counts first, then starred entries by + stars descending, then stdlib, then the rest. + """ + + def sort_key(entry: TemplateEntry) -> tuple[int, int, int, str]: + downloads = entry["downloads"] + stars = entry["stars"] + name = entry["name"].lower() + if downloads is not None: + return (0, -downloads, 0, name) + if stars is not None: + builtin = 1 if entry.get("source_type") == BUILTIN_FILTER else 0 + return (1, -stars, builtin, name) + if entry.get("source_type") == BUILTIN_FILTER: + return (2, 0, 0, name) + return (3, 0, 0, name) + + return sorted(entries, key=sort_key) + + +def build_robots_txt() -> str: + return f"User-agent: *\nContent-Signal: search=yes, ai-input=yes, ai-train=yes\nAllow: /\n\nSitemap: {SITEMAP_URL}\n" + + +WEBSITE_ID = f"{SITE_URL}#website" +ISPARTOF_WEBSITE = {"@type": "WebSite", "@id": WEBSITE_ID} + + +def _website_node() -> dict: + return { + "@type": "WebSite", + "@id": WEBSITE_ID, + "name": "Awesome Python", + "url": SITE_URL, + "inLanguage": "en", + "sameAs": "https://github.com/vinta/awesome-python", + } + + +def _item_list_payload(entries: Sequence[TemplateEntry]) -> dict: + return { + "@type": "ItemList", + "numberOfItems": len(entries), + "itemListElement": [ + { + "@type": "ListItem", + "position": i, + "name": entry["name"], + "url": entry["url"], + } + for i, entry in enumerate(entries, start=1) + ], + } + + +def build_homepage_json_ld(entries: Sequence[TemplateEntry], total_categories: int) -> dict: + description = ( + "An opinionated guide to the best Python frameworks, libraries, and tools. " + f"Explore {len(entries)} curated projects across {total_categories} categories, " + "from AI and agents to data science and web development." + ) + return { + "@context": "https://schema.org", + "@graph": [ + _website_node(), + { + "@type": "CollectionPage", + "@id": SITE_URL, + "name": "Awesome Python", + "url": SITE_URL, + "description": description, + "isPartOf": ISPARTOF_WEBSITE, + "inLanguage": "en", + "mainEntity": _item_list_payload(entries), + }, + ], + } + + +def category_meta_title(name: str, parent_name: str | None = None) -> str: + if parent_name: + # Lead with the section's title so the page matches searches like "python jwt library". + title = f"{category_meta_title(parent_name).removesuffix(' - Awesome Python')}: {name}" + # Google shows the site name above each result, so a long title drops it rather than get cut off. + return f"{title} - Awesome Python" if len(title) <= 43 else title + # Names ending in one of these nouns already say what the entries are. + noun = "" if name.rsplit(" ", 1)[-1] in PLURAL_NOUNS else " Libraries" + prefix = "" if name.startswith("Python ") else "Python " + title = f"{prefix}{name}{noun} - Awesome Python" + if len(title) <= 60: + return title + return f"{name} - Awesome Python" + + +def category_meta_description(name: str, entry_count: int, description: str) -> str: + count_sentence = f"Explore {entry_count} curated Python project{'s' if entry_count != 1 else ''} in {name}." + if description: + lead = description if description.endswith((".", "!", "?")) else f"{description}." + return f"{lead} {count_sentence}" + return f"{count_sentence} Part of the Awesome Python catalog." + + +def subcategory_meta_description(name: str, parent_name: str, entry_names: Sequence[str]) -> str: + names = entry_names[0] if len(entry_names) == 1 else f"{', '.join(entry_names[:-1])}{',' if len(entry_names) > 2 else ''} and {entry_names[-1]}" + return f"The {name} picks in Awesome Python's {parent_name} list: {names}." + + +def entry_name_key(name: str) -> str: + return re.sub(r"[^a-z0-9]", "", name.lower()) + + +def link_pick_names(pick: str, slugs_by_key: dict[str, str]) -> str: + """Link entry names in a "How to choose" pick to their rows, preferring the longest match. + + Names match ignoring case and punctuation, so the README's PyPI-style + "django-ninja" matches the intro's "Django Ninja". + """ + words = list(re.finditer(r"`[^`]*`|[^\s,]+", pick)) + parts, pos, i = [], 0, 0 + while i < len(words): + for j in range(min(len(words), i + 4), i, -1): + start, end = words[i].start(), words[j - 1].end() + span = pick[start:end] + slug = slugs_by_key.get(entry_name_key(span)) + # Code spans and dangling punctuation like "The Algorithms -" stay outside links + if slug and span[0].isalnum() and span[-1].isalnum() and "," not in span and "`" not in span: + parts.append(f"{pick[pos:start]}[{span}](#entry-{slug})") + pos, i = end, j + break + else: + i += 1 + return "".join(parts) + pick[pos:] + + +def link_intro_picks(text: str, entry_names: Sequence[str]) -> str: + """Link the picks in an intro's "How to choose:" list to their rows in the table below.""" + head, marker, rest = text.partition("\nHow to choose:\n\n") + if not marker: + return text + slugs_by_key = {entry_name_key(name): slugify(name) for name in entry_names} + for alias, name in PICK_ALIASES.items(): + if entry_name_key(name) in slugs_by_key: + slugs_by_key[entry_name_key(alias)] = slugs_by_key[entry_name_key(name)] + lines = rest.split("\n") + for i, line in enumerate(lines): + if not line.startswith("- "): + break + situation, _, pick = line.rpartition(": ") + lines[i] = f"{situation}: {link_pick_names(pick, slugs_by_key)}" + return head + marker + "\n".join(lines) + + +def load_category_intro(path: Path, entry_names: Sequence[str] = ()) -> tuple[str, str, str, str]: + """Render a category intro file to HTML, split at the end of its "How to choose:" list. + + Returns the part shown above the table, the guide shown below it, and the + first paragraph as plain text for the meta description and as HTML for + group pages. A file without the list keeps everything above the table. + Returns empty strings if the category has no intro file. + """ + if not path.exists(): + return "", "", "", "" + md = MarkdownIt("commonmark") + tokens = md.parse(link_intro_picks(path.read_text(encoding="utf-8"), entry_names)) + for token in tokens: + for child in token.children or []: + if child.type == "link_open" and not str(child.attrGet("href")).startswith("#"): + child.attrSet("target", "_blank") + child.attrSet("rel", "noopener") + lead = next(node for node in SyntaxTreeNode(tokens).children if node.type == "paragraph") + split_at = len(tokens) + for i, token in enumerate(tokens): + if token.type == "inline" and token.level == 1 and token.content == "How to choose:" and i + 2 < len(tokens) and tokens[i + 2].type == "bullet_list_open": + split_at = next(j for j in range(i + 3, len(tokens)) if tokens[j].type == "bullet_list_close" and tokens[j].level == 0) + 1 + break + render = md.renderer.render + return ( + render(tokens[:split_at], md.options, {}), + render(tokens[split_at:], md.options, {}), + render_inline_text(lead.children[0].children), + render(lead.to_tokens(), md.options, {}), + ) + + +def group_section_entries(section: ParsedSection, entries_by_key: dict[tuple[str, str], TemplateEntry]) -> list[EntryGroup]: + """Group a section's entries by use case (subcategory), both in README order.""" + groups: dict[str, EntryGroup] = {} + for parsed in section["entries"]: + name = parsed["subcategory"] + slug = slugify(name) if name else "" + group = groups.setdefault(name, EntryGroup(name=name, slug=slug, url=subcategory_path(section["slug"], slug) if name else "", lead="", entries=[])) + group["entries"].append(entries_by_key[(parsed["url"], parsed["name"])]) + return list(groups.values()) + + +def group_entries_by_section(sections: Sequence[ParsedSection], entries_by_key: dict[tuple[str, str], TemplateEntry], intros_dir: Path) -> list[EntryGroup]: + """Group a thematic group's entries by section, both in README order, listing each entry once.""" + placed: set[tuple[str, str]] = set() + groups: list[EntryGroup] = [] + for section in sections: + entries: list[TemplateEntry] = [] + for parsed in section["entries"]: + key = (parsed["url"], parsed["name"]) + if key not in placed: + placed.add(key) + entries.append(entries_by_key[key]) + groups.append(EntryGroup(name=section["name"], slug=section["slug"], url=category_path(section), lead=load_category_intro(intros_dir / f"{section['slug']}.md")[3], entries=entries)) + return groups + + +def build_breadcrumb_json_ld(items: Sequence[tuple[str, str]]) -> dict: + return { + "@type": "BreadcrumbList", + "itemListElement": [ + { + "@type": "ListItem", + "position": i, + "name": name, + "item": url, + } + for i, (name, url) in enumerate(items, start=1) + ], + } + + +def build_category_json_ld( + name: str, + url: str, + description: str, + entries: Sequence[TemplateEntry], + breadcrumbs: Sequence[tuple[str, str]], +) -> dict: + return { + "@context": "https://schema.org", + "@graph": [ + _website_node(), + { + "@type": "CollectionPage", + "@id": url, + "name": name, + "url": url, + "description": description, + "isPartOf": ISPARTOF_WEBSITE, + "inLanguage": "en", + "mainEntity": _item_list_payload(entries), + }, + build_breadcrumb_json_ld(breadcrumbs), + ], + } + + +def build_sponsorship_json_ld() -> dict: + return { + "@context": "https://schema.org", + "@graph": [ + _website_node(), + { + "@type": "WebPage", + "@id": SPONSORSHIP_PUBLIC_URL, + "name": "Sponsor Awesome Python", + "url": SPONSORSHIP_PUBLIC_URL, + "description": SPONSORSHIP_DESCRIPTION, + "isPartOf": ISPARTOF_WEBSITE, + "inLanguage": "en", + }, + build_breadcrumb_json_ld( + [ + ("Awesome Python", SITE_URL), + ("Sponsorship", SPONSORSHIP_PUBLIC_URL), + ] + ), + ], + } + + +def category_path(category: ParsedSection) -> str: + return f"/categories/{category['slug']}/" + + +def category_public_url(category: ParsedSection) -> str: + return f"{SITE_URL}categories/{category['slug']}/" + + +def group_path(group_slug: str) -> str: + return f"/categories/{group_slug}/" + + +def group_public_url(group_slug: str) -> str: + return f"{SITE_URL}categories/{group_slug}/" + + +def subcategory_path(category_slug: str, subcategory_slug: str) -> str: + return f"/categories/{category_slug}/{subcategory_slug}/" + + +def subcategory_public_url(category_slug: str, subcategory_slug: str) -> str: + return f"{SITE_URL}categories/{category_slug}/{subcategory_slug}/" + + +def synthetic_category(name: str, slug: str) -> SyntheticCategory: + return {"name": name, "slug": slug, "description": "", "description_html": ""} + + +def git_last_change_date(repo_root: Path, *log_args: str) -> str: + """Return the date of the last commit matching `git log` args, or "" without git history.""" + result = subprocess.run(["git", "log", "-1", "--format=%cs", "--no-patch", *log_args], cwd=repo_root, capture_output=True, text=True, check=False) + return result.stdout.strip() + + +def section_line_range(readme_text: str, name: str) -> str: + """Return a section's README lines as a `git log -L` range, from its heading to the line before the next heading.""" + lines = readme_text.split("\n") + start = next(i for i, line in enumerate(lines) if re.fullmatch(rf"#+ {re.escape(name)}", line)) + end = next((i for i in range(start + 1, len(lines)) if lines[i].startswith("#")), len(lines)) + return f"{start + 1},{end}:README.md" + + +def write_sitemap_xml(path: Path, urls: Sequence[tuple[str, str]]) -> None: + ET.register_namespace("", SITEMAP_NS) + urlset = ET.Element(f"{{{SITEMAP_NS}}}urlset") + for url, lastmod in urls: + url_el = ET.SubElement(urlset, f"{{{SITEMAP_NS}}}url") + loc_el = ET.SubElement(url_el, f"{{{SITEMAP_NS}}}loc") + loc_el.text = url + lastmod_el = ET.SubElement(url_el, f"{{{SITEMAP_NS}}}lastmod") + lastmod_el.text = lastmod + + tree = ET.ElementTree(urlset) + ET.indent(tree, space=" ") + tree.write(path, encoding="utf-8", xml_declaration=True) + with path.open("ab") as f: + f.write(b"\n") + + +def top_level_heading_text(line: str) -> str | None: + stripped = line.strip() + match = re.match(r"^(#{1,2})\s+(.+)$", stripped) + if match is None: + return None + return match.group(2).strip().strip("#").strip().strip("*").strip() + + +def extract_categories_body(markdown: str) -> str: + """Return content from `Categories` through `Projects`, excluding later sections.""" + lines = markdown.splitlines(keepends=True) + start_idx = None + end_idx = len(lines) + for i, line in enumerate(lines): + heading = top_level_heading_text(line) + if heading is None: + continue + if start_idx is None and heading.lower() == "categories": + start_idx = i + 1 + while start_idx < len(lines) and lines[start_idx].strip() == "": + start_idx += 1 + elif start_idx is not None and heading.lower() in ("resources", "contributing"): + end_idx = i + break + if start_idx is None: + return "" + return "".join(lines[start_idx:end_idx]).rstrip() + "\n" + + +def github_markdown_anchor(text: str) -> str: + anchor = text.strip().lower() + anchor = re.sub(r"[^\w\s-]", "", anchor) + anchor = re.sub(r"\s", "-", anchor) + return f"#{anchor}" + + +def link_llms_category_index_to_canonical_pages(markdown: str, categories: Sequence[ParsedSection]) -> str: + """Point the README-derived category index at canonical category pages.""" + category_urls = {} + for category in categories: + public_url = category_public_url(category) + category_urls[f"#{category['slug']}"] = public_url + category_urls[github_markdown_anchor(category["name"])] = public_url + lines = markdown.splitlines(keepends=True) + out: list[str] = [] + + def replace_link(match: re.Match[str]) -> str: + target = match.group(2) + url = category_urls.get(target) + if url is None: + return match.group(0) + return match.group(0).replace(f"({target})", f"({url})", 1) + + for line in lines: + out.append(MARKDOWN_LINK_RE.sub(replace_link, line)) + + return "".join(out) + + +def link_description_anchors_to_category_pages(categories: Sequence[ParsedSection]) -> None: + """Point README anchor links in section descriptions at category pages, which lack those anchors.""" + category_paths = {} + for category in categories: + category_paths[f"#{category['slug']}"] = category_path(category) + category_paths[github_markdown_anchor(category["name"])] = category_path(category) + + def replace_attrs(match: re.Match[str]) -> str: + path = category_paths.get(match.group(1)) + return f'href="{path}"' if path else match.group(0) + + for category in categories: + category["description_html"] = ANCHOR_LINK_ATTRS_RE.sub(replace_attrs, category["description_html"]) + + +def build_llms_txt( + template_text: str, + *, + readme_text: str, + subtitle: str, + build_date: str, + stars_data: dict[str, dict], + downloads_data: dict[str, int], + categories: Sequence[ParsedSection], + total_entries: int, +) -> str: + """Render the llms.txt entry point with the curated category shortlist.""" + categories_md = annotate_entries_with_stats( + link_llms_category_index_to_canonical_pages( + extract_categories_body(readme_text).rstrip(), + categories, + ), + stars_data, + downloads_data, + ) + text_env = Environment(autoescape=False, trim_blocks=True, lstrip_blocks=True) + rendered = text_env.from_string(template_text).render( + subtitle=subtitle, + build_date=build_date, + site_url=SITE_URL, + github_repo_url="https://github.com/vinta/awesome-python", + contributing_url="https://github.com/vinta/awesome-python/blob/master/CONTRIBUTING.md", + sponsorship_url=SPONSORSHIP_PUBLIC_URL, + sitemap_url=SITEMAP_URL, + categories_md=categories_md, + total_entries=total_entries, + total_categories=len(categories), + ) + return rendered.rstrip() + "\n" + + +def annotate_entries_with_stats( + markdown: str, + stars_data: dict[str, dict], + downloads_data: dict[str, int], +) -> str: + """Append download and star counts to bullet entry lines. + + Downloads are looked up by the first link's display name (the same key the + TSV cache uses); stars by the first GitHub link with known star data. + """ + lines = markdown.splitlines(keepends=True) + out: list[str] = [] + for line in lines: + if not BULLET_LINE_RE.match(line): + out.append(line) + continue + links = MARKDOWN_LINK_RE.findall(line) + parts: list[str] = [] + if links: + name, url = links[0] + # Category-index bullets link into the site itself, and stdlib entries + # would hit same-named PyPI backports (e.g. logging) — no counts for either. + if not url.startswith((SITE_URL, "#")) and detect_source_type(url) != BUILTIN_FILTER: + downloads = downloads_data.get(normalize(name)) + if downloads is not None: + parts.append(f"PyPI downloads/month: {downloads}") + for _, url in links: + repo_key = extract_github_repo(url) + if not repo_key: + continue + entry = stars_data.get(repo_key) + if entry and "stars" in entry: + parts.append(f"GitHub stars: {entry['stars']}") + break + if not parts: + out.append(line) + continue + stripped = line.rstrip("\n") + ending = line[len(stripped) :] + out.append(f"{stripped} ({', '.join(parts)}){ending}") + return "".join(out) + + +def remove_sponsors_section(markdown: str) -> str: + lines = markdown.splitlines(keepends=True) + start_idx = None + for i, line in enumerate(lines): + heading = top_level_heading_text(line) + if heading and heading.lower() == "sponsors": + start_idx = i + break + + if start_idx is None: + return markdown + + end_idx = len(lines) + for i, line in enumerate(lines[start_idx + 1 :], start=start_idx + 1): + if top_level_heading_text(line): + end_idx = i + break + + return "".join(lines[:start_idx] + lines[end_idx:]) + + +def extract_entries( + categories: list[ParsedSection], + groups: list[ParsedGroup], +) -> list[TemplateEntry]: + """Flatten categories into individual library entries for table display. + + Entries appearing in multiple categories are merged into a single entry + with lists of categories and groups. + """ + cat_to_group = {cat["name"]: group["name"] for group in groups for cat in group["categories"]} + + seen: dict[tuple[str, str], TemplateEntry] = {} # (url, name) -> entry + entries: list[TemplateEntry] = [] + for cat in categories: + group_name = cat_to_group.get(cat["name"], "Other") + for entry in cat["entries"]: + key = (entry["url"], entry["name"]) + existing = seen.get(key) + if existing is None: + existing = TemplateEntry( + name=entry["name"], + url=entry["url"], + description=entry["description"], + categories=[], + groups=[], + subcategories=[], + stars=None, + downloads=None, + owner=None, + last_commit_at=None, + source_type=detect_source_type(entry["url"]), + bundled=bool(BUNDLED_PREFIX_RE.match(entry["description"])), + badge=None, + also_see=entry["also_see"], + ) + seen[key] = existing + entries.append(existing) + if cat["name"] not in existing["categories"]: + existing["categories"].append(cat["name"]) + if group_name not in existing["groups"]: + existing["groups"].append(group_name) + subcat = entry["subcategory"] + if subcat: + scoped = f"{cat['name']} > {subcat}" + if not any(s["value"] == scoped for s in existing["subcategories"]): + sub_slug = slugify(subcat) + existing["subcategories"].append( + TemplateSubcategory( + name=subcat, + value=scoped, + slug=sub_slug, + url=f"/categories/{cat['slug']}/{sub_slug}/", + ) + ) + return entries + + +def build(repo_root: Path) -> None: + """Main build: parse README, render single-page HTML via Jinja2 templates.""" + website = repo_root / "website" + readme_text = (repo_root / "README.md").read_text(encoding="utf-8") + + subtitle = "" + for line in readme_text.split("\n"): + stripped = line.strip() + if stripped and not stripped.startswith("#"): + subtitle = stripped + break + + parsed_groups = parse_readme(readme_text) + sponsors = parse_sponsors(readme_text) + + categories = [cat for g in parsed_groups for cat in g["categories"]] + cat_slugs = [cat["slug"] for cat in categories] + group_slugs = [g["slug"] for g in parsed_groups] + all_top_level_slugs = cat_slugs + group_slugs + [BUILTIN_SLUG] + duplicates = {s for s, n in Counter(all_top_level_slugs).items() if n > 1} + if duplicates: + raise ValueError(f"slug collision in /categories/ namespace: {sorted(duplicates)}. Rename a category or group so their slugs differ.") + link_description_anchors_to_category_pages(categories) + total_entries = sum(c["entry_count"] for c in categories) + entries = extract_entries(categories, parsed_groups) + build_date = datetime.now(UTC) + + stars_data = load_stars(website / "data" / "github_stars.json") + downloads_data = load_downloads(website / "data" / "pypi_downloads.tsv") + pypi_badges = load_pypi_badges() + + repo_self = stars_data.get("vinta/awesome-python", {}) + repo_stars = None + if "stars" in repo_self: + stars_val = repo_self["stars"] + repo_stars = f"{stars_val // 1000}k" if stars_val >= 1000 else str(stars_val) + + for entry in entries: + repo_key = extract_github_repo(entry["url"]) + if not repo_key and entry.get("source_type") == BUILTIN_FILTER: + repo_key = "python/cpython" + if repo_key and repo_key in stars_data: + sd = stars_data[repo_key] + entry["stars"] = sd["stars"] + entry["owner"] = sd["owner"] + entry["last_commit_at"] = sd.get("last_commit_at", "") + # Stdlib entries would hit same-named PyPI backports (e.g. asyncio), not the stdlib. + if entry.get("source_type") != BUILTIN_FILTER: + entry["downloads"] = downloads_data.get(normalize(entry["name"])) + entry["badge"] = pypi_badges.get(normalize(entry["name"])) + + entries = sort_entries(entries) + category_urls = {cat["name"]: category_path(cat) for cat in categories} + + filter_urls: dict[str, str] = dict(category_urls) + for group in parsed_groups: + filter_urls[group["name"]] = group_path(group["slug"]) + builtin_entries = [e for e in entries if e.get("source_type") == BUILTIN_FILTER] + + env = Environment( + loader=FileSystemLoader(website / "templates"), + autoescape=True, + trim_blocks=True, + lstrip_blocks=True, + ) + env.filters["slugify"] = slugify + site_dir = website / "output" + if site_dir.exists(): + shutil.rmtree(site_dir) + site_dir.mkdir(parents=True) + + homepage_json_ld = json.dumps( + build_homepage_json_ld(entries, len(categories)), + ensure_ascii=False, + ).replace(" None: + page_dir.mkdir(parents=True, exist_ok=True) + parent_name = parent_category["name"] if parent_category else None + category_title = category_meta_title(category["name"], parent_name) + page_entries = [entry for group in entry_groups for entry in group["entries"]] if entry_groups else entries + intro_html, guide_html, intro_lead, _ = load_category_intro( + website / "data" / "category_intros" / f"{current_path.removeprefix('/categories/').strip('/')}.md", [e["name"] for e in page_entries] + ) + if intro_lead: + category_description = intro_lead + elif parent_name: + category_description = subcategory_meta_description(category["name"], parent_name, [e["name"] for e in entries]) + else: + category_description = category_meta_description(category["name"], len(entries), category["description"]) + breadcrumbs = [("Awesome Python", SITE_URL)] + if parent_category: + breadcrumbs.append((parent_category["name"], category_public_url(parent_category))) + breadcrumbs.append((category["name"], category_url)) + category_json_ld = json.dumps( + build_category_json_ld(category_title.removesuffix(" - Awesome Python"), category_url, category_description, page_entries, breadcrumbs), + ensure_ascii=False, + ).replace(" (cat_slug, sub_slug, sub_name) + cat_slug_by_url_prefix = {f"/categories/{c['slug']}/": c["slug"] for c in categories} + cat_by_slug = {c["slug"]: c for c in categories} + for entry in entries: + for sub in entry.get("subcategories", []): + value = sub["value"] + subcat_to_entries.setdefault(value, []).append(entry) + if value not in subcat_meta: + for prefix, cat_slug in cat_slug_by_url_prefix.items(): + if sub["url"].startswith(prefix): + subcat_meta[value] = (cat_slug, sub["slug"], sub["name"]) + break + + for value, (cat_slug, sub_slug, sub_name) in subcat_meta.items(): + render_category( + synthetic_category(sub_name, sub_slug), + category_url=subcategory_public_url(cat_slug, sub_slug), + entries=subcat_to_entries[value], + current_path=subcategory_path(cat_slug, sub_slug), + page_dir=categories_dir / cat_slug / sub_slug, + parent_category=cat_by_slug[cat_slug], + entry_groups=[EntryGroup(name="", slug="", url="", lead="", entries=group["entries"]) for group in section_groups[cat_by_slug[cat_slug]["name"]] if group["name"] == sub_name], + ) + + redirects_file = website / "data" / "redirects.json" + redirects = json.loads(redirects_file.read_text(encoding="utf-8")) if redirects_file.exists() else {} + for old_path, new_path in redirects.items(): + tpl_redirect = env.get_template("redirect.html") + stub = site_dir / old_path.strip("/") / "index.html" + if stub.exists(): + raise ValueError(f"redirect source {old_path} is a live page; remove it from redirects.json") + if not (site_dir / new_path.strip("/") / "index.html").exists(): + raise ValueError(f"redirect target {new_path} for {old_path} does not exist") + stub.parent.mkdir(parents=True, exist_ok=True) + stub.write_text(tpl_redirect.render(target_url=SITE_URL + new_path.lstrip("/")), encoding="utf-8") + + static_src = website / "static" + static_dst = site_dir / "static" + if static_src.exists(): + shutil.copytree(static_src, static_dst, dirs_exist_ok=True) + + llms_template = (website / "templates" / "llms.txt").read_text(encoding="utf-8") + llms_txt = build_llms_txt( + llms_template, + readme_text=readme_text, + subtitle=subtitle, + build_date=build_date.strftime("%Y/%m/%d"), + stars_data=stars_data, + downloads_data=downloads_data, + categories=categories, + total_entries=total_entries, + ) + (site_dir / "robots.txt").write_text(build_robots_txt(), encoding="utf-8") + # Daily star and download refreshes are not significant changes, so lastmod follows commits to the page's own content. + build_day = build_date.date().isoformat() + readme_date = git_last_change_date(repo_root, "--", "README.md") or build_day + section_dates = { + c["slug"]: max( + git_last_change_date(repo_root, "-L", section_line_range(readme_text, c["name"])), + git_last_change_date(repo_root, "--", f"website/data/category_intros/{c['slug']}.md"), + ) + or build_day + for c in categories + } + sitemap_urls = [(SITE_URL, readme_date)] + sitemap_urls.extend((category_public_url(c), section_dates[c["slug"]]) for c in categories) + sitemap_urls.extend((group_public_url(g["slug"]), max(section_dates[c["slug"]] for c in g["categories"])) for g in parsed_groups) + if builtin_entries: + sitemap_urls.append((BUILTIN_PUBLIC_URL, readme_date)) + for cat_slug, sub_slug, _ in sorted(subcat_meta.values()): + sitemap_urls.append((subcategory_public_url(cat_slug, sub_slug), section_dates[cat_slug])) + sitemap_urls.append((SPONSORSHIP_PUBLIC_URL, git_last_change_date(repo_root, "--", "SPONSORSHIP.md") or build_day)) + write_sitemap_xml(site_dir / "sitemap.xml", sitemap_urls) + (site_dir / "llms.txt").write_text(llms_txt, encoding="utf-8") + + print(f"Built site with {len(parsed_groups)} groups, {len(categories)} categories") + print(f"Total entries: {total_entries}") + print(f"Output: {site_dir}") + + +if __name__ == "__main__": + build(Path(__file__).parent.parent) diff --git a/website/data/category_intros/admin-panels.md b/website/data/category_intros/admin-panels.md new file mode 100644 index 0000000000..66ff686fad --- /dev/null +++ b/website/data/category_intros/admin-panels.md @@ -0,0 +1,18 @@ +Django saves you from building a Python admin panel: its admin comes built in. Flask-Admin and SQLAdmin add one to Flask and FastAPI apps. + +How to choose: + +- A Flask app: Flask-Admin +- SQLAlchemy models in FastAPI or Starlette: SQLAdmin +- A Django admin grown into dashboards and internal tools: Unfold +- A new look for the Django admin, on your stock ModelAdmin classes: Grappelli + +Flask-Admin [gives each model a `ModelView`](https://flask-admin.readthedocs.io/en/latest/introduction/#adding-model-views): one `admin.add_view(ModelView(User, db.session))` line gets you list, create, and edit pages for it. To fit a model, [subclass `ModelView`](https://flask-admin.readthedocs.io/en/latest/introduction/#customizing-built-in-views) and set class attributes like `can_delete` or `column_list`. Expect fewer defaults than in Django's admin: [it's up to you to tell Flask-Admin](https://flask-admin.readthedocs.io/en/latest/advanced/#design-philosophy) what to display and how. Besides SQLAlchemy, it has [backends](https://flask-admin.readthedocs.io/en/latest/advanced/#using-different-database-backends) for Peewee, pymongo, and MongoEngine. + +SQLAdmin brings Flask-Admin's approach to SQLAlchemy models on FastAPI or Starlette: Flask-Admin [inspired most of its features and configuration](https://smithyhq.github.io/sqladmin/#related-projects-and-inspirations). Create an `Admin` from your app and engine, then [write one `ModelView` subclass per model](https://smithyhq.github.io/sqladmin/#quickstart), declared with `model=User`, and register it with `admin.add_view`. + +Unfold is a theme for Django's own admin, [made for building dashboards, internal tools, and business applications](https://github.com/unfoldadmin/django-unfold). Put `"unfold"` [before `django.contrib.admin`](https://unfoldadmin.com/docs/installation/quickstart/) in `INSTALLED_APPS`, then make your admin classes inherit from `unfold.admin.ModelAdmin`: the stock `ModelAdmin` leaves forms unstyled. + +Grappelli is [a jazzy skin for the Django admin](https://django-grappelli.readthedocs.io/en/latest/) that also adds collapsibles, sortable inlines, and related and autocomplete lookups. Put `'grappelli'` [before `django.contrib.admin`](https://django-grappelli.readthedocs.io/en/latest/quickstart.html) in `INSTALLED_APPS`, and include `grappelli.urls`, which the lookups need. You switch features on with [attributes on your `admin.ModelAdmin` classes](https://django-grappelli.readthedocs.io/en/latest/customization.html#autocomplete-lookups), like `autocomplete_lookup_fields`. + +By default, Django's admin [lets in only users with `is_staff` set](https://docs.djangoproject.com/en/stable/ref/contrib/admin/#overview). Flask-Admin and SQLAdmin leave access control to you: in Flask-Admin, [override `is_accessible`](https://flask-admin.readthedocs.io/en/latest/introduction/#rolling-your-own) on your views, and SQLAdmin [enforces no authentication](https://smithyhq.github.io/sqladmin/authentication/) until you pass it an `AuthenticationBackend`. Django's docs limit its admin to [an internal management tool](https://docs.djangoproject.com/en/stable/ref/contrib/admin/), and once you need a process-centric interface, they say to write your own views. diff --git a/website/data/category_intros/ai-and-agents.md b/website/data/category_intros/ai-and-agents.md new file mode 100644 index 0000000000..d614a7055e --- /dev/null +++ b/website/data/category_intros/ai-and-agents.md @@ -0,0 +1,58 @@ +Build a typed agent in Pydantic AI. Long-running, stateful agents suit LangGraph. Apps tied to one model vendor can use that vendor's Python AI agent library. + +How to choose: + +- Typed agents: Pydantic AI, or LangGraph for long-running, stateful ones +- Agents on Claude, OpenAI, or Gemini models: Claude Agent SDK, OpenAI Agents SDK, or Google ADK +- Skills for your coding agent: Django AI Skills, Sentry Skills, or Trail of Bits Skills +- MCP servers and clients: MCP Python SDK, or FastMCP to compose and proxy servers +- A personal assistant in your chat apps: Hermes Agent, or AstrBot for QQ, Feishu, and DingTalk +- Prompts tuned against a metric: DSPy +- Agent data: Instructor for structured output, LlamaIndex for RAG, Mem0 for memory +- Open models from the Hugging Face Hub: Transformers +- Running open models locally: Ollama, llama-cpp-python in your own process, MLX LM on Apple silicon +- Serving open models: vLLM or SGLang +- One OpenAI-style API for many model providers: LiteLLM +- Image and video generation: Diffusers +- Fine-tuning: TRL with PEFT adapters, Unsloth for less GPU memory, Axolotl from YAML +- Speech: faster-whisper, FunASR for Chinese, gTTS for text to speech + +Pydantic AI comes from the Pydantic team and is [typed end to end](https://pydantic.dev/docs/ai/overview/). Outputs, tools, and dependencies all carry types, so your type checker knows what an agent returns. Switching models takes one string. [Give an agent an output type and tools](https://github.com/pydantic/pydantic-ai), and every run comes back validated. + +LangGraph is a [low-level orchestration framework](https://docs.langchain.com/oss/python/langgraph/overview) for long-running, stateful agents, and it can mix hand-coded steps with LLM-driven ones in one graph. For your first agent, its docs suggest LangChain's agents instead: [a minimal harness](https://docs.langchain.com/oss/python/langchain/overview) built on LangGraph, which you extend through middleware. + +The Claude Agent SDK gives you [the tools and agent loop behind Claude Code](https://code.claude.com/docs/en/agent-sdk/overview), so it fits agents that read files, run commands, and edit code. Call `query()` for a one-off task, and switch to [`ClaudeSDKClient`](https://code.claude.com/docs/en/agent-sdk/python#choosing-between-query-and-claudesdkclient) when the next step depends on Claude's reply, as in a chat. + +The OpenAI Agents SDK keeps to [a small set of primitives](https://openai.github.io/openai-agents-python/): use it when you want the runtime to handle turns, tool calls, handoffs, and guardrails for you. Despite the name, it [isn't limited to OpenAI models](https://github.com/openai/openai-agents-python). + +Google ADK is [optimized for Gemini but model-agnostic](https://github.com/google/adk-python). Start a project with [`adk create`](https://adk.dev/get-started/python/). + +All three skill collections install from a Claude Code plugin marketplace. Django AI Skills and Sentry Skills follow the open [Agent Skills](https://agentskills.io/) format, so other coding agents can load them too. Sentry Skills is [written for Sentry's own engineers](https://github.com/getsentry/skills), so some of its skills follow Sentry's internal standards. + +CrewAI splits an app into Flows, which manage state and control execution, and Crews, teams of agents that work on one task together. For a production app, [start with a Flow](https://docs.crewai.com/en/introduction), and hand a Crew only the steps that need autonomy. + +The MCP Python SDK reads each tool's schema [from its type hints and docstring](https://py.sdk.modelcontextprotocol.io/get-started/first-steps/), so you write no JSON Schema or request handlers. FastMCP wrote the high-level API that the SDK [took in](https://gofastmcp.com/getting-started/welcome). The standalone project is a fuller framework: it can [mount several servers into one](https://gofastmcp.com/servers/composition) and [proxy another MCP server](https://gofastmcp.com/servers/providers/proxy) through your own. + +Hermes Agent talks to you on Telegram, Discord, Slack, WhatsApp, and Signal [from one gateway process](https://github.com/NousResearch/hermes-agent), and writes itself new skills as it works. Since it runs commands for whoever it lets in, allow only your own account on its gateway with an [allowlist](https://hermes-agent.nousresearch.com/docs/user-guide/security#user-authorization-gateway), and leave approval prompts on for dangerous commands. AstrBot covers chat apps such as QQ, Feishu, and DingTalk. It's AGPL, while Hermes Agent is MIT. + +DSPy has you write [structured signatures, not prompts](https://dspy.ai/current/), and its optimizers tune the prompts for you. Before they can, [define a metric and score a baseline](https://dspy.ai/current/getting-started/metrics/) to beat. + +Instructor [turns an LLM's reply into a validated Pydantic model](https://github.com/567-labs/instructor), and when validation fails, it retries with the error message. Its docs draw the line at [extraction versus agents](https://python.useinstructor.com/): Instructor for the first, Pydantic AI for the second. + +LlamaIndex builds [agents over your own data](https://developers.llamaindex.ai/python/framework/). Its high-level API goes from documents to answers in a few lines, and its lower-level API lets you replace any part. + +Mem0 gives an agent [memory of each user](https://github.com/mem0ai/mem0): search their memories before the model answers, then add the new exchange. Run it [as a library in your app](https://docs.mem0.ai/open-source/overview), or as a self-hosted server for a team. + +Transformers is the [model definition that most training frameworks and inference engines share](https://huggingface.co/docs/transformers/index), Axolotl, Unsloth, vLLM, and SGLang among them. Load a model from the Hugging Face Hub with [`from_pretrained()`](https://huggingface.co/docs/transformers/quicktour), then run it through `Pipeline` for inference or `Trainer` for training. For large production deployments, its docs [send you to vLLM or SGLang](https://huggingface.co/docs/transformers/serve-cli/serving). + +The Ollama Python library talks to Ollama, which [runs models on your computer](https://docs.ollama.com/api/introduction). [Start Ollama and pull a model](https://github.com/ollama/ollama-python) with `ollama pull`, then pass the model's name to `chat()`. llama-cpp-python gives you [Python bindings for llama.cpp](https://github.com/abetlen/llama-cpp-python), so the model runs inside your program. Load a GGUF file with the `Llama` class, or fetch one from the Hugging Face Hub with `Llama.from_pretrained()`. Its completions come back in the OpenAI format. On a Mac, MLX LM [runs and fine-tunes models on Apple silicon](https://github.com/ml-explore/mlx-lm). + +vLLM runs [batch inference offline or an OpenAI-compatible server](https://docs.vllm.ai/en/latest/getting_started/quickstart/), so apps written for the OpenAI API work unchanged. SGLang is [optimized for agentic workloads, RL rollouts, and large-scale serving](https://github.com/sgl-project/sglang). + +LiteLLM [returns every provider's responses in the OpenAI format](https://docs.litellm.ai/docs/) and maps their errors to OpenAI's exception types, so one code path covers them all. Use its Python SDK inside your app, or run its proxy as a self-hosted gateway for your team. + +Diffusers centers on the [`DiffusionPipeline`](https://huggingface.co/docs/diffusers/index): a few lines generate an image, video, or audio clip. + +Start fine-tuning with LoRA or QLoRA rather than training every weight, as both [PEFT](https://huggingface.co/docs/peft/methods/overview) and [Unsloth](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide) advise. TRL trains with methods from SFT to DPO and GRPO, and every TRL trainer takes a [`peft_config`](https://huggingface.co/docs/trl/peft_integration) for PEFT adapters. Axolotl keeps the whole pipeline, from dataset preprocessing to inference, in [one YAML file](https://github.com/axolotl-ai-cloud/axolotl). + +faster-whisper reimplements Whisper on CTranslate2 for [the same accuracy in less time and memory](https://github.com/SYSTRAN/faster-whisper). For Chinese audio, FunASR's `AutoModel` [chains speech recognition with voice activity detection, punctuation, and speaker models](https://github.com/modelscope/FunASR) in one call. gTTS speaks through [Google Translate's undocumented speech feature](https://github.com/pndurette/gTTS), so upstream changes can break it without notice. diff --git a/website/data/category_intros/algorithms-and-design-patterns.md b/website/data/category_intros/algorithms-and-design-patterns.md new file mode 100644 index 0000000000..ff8234b1f4 --- /dev/null +++ b/website/data/category_intros/algorithms-and-design-patterns.md @@ -0,0 +1,21 @@ +If a Python algorithms library has to run in production, Sorted Containers gives you sorted lists, dicts, and sets. To learn, read The Algorithms - Python. + +How to choose: + +- Lists, dicts, and sets that stay sorted as you add and remove: Sorted Containers +- Small, typed implementations you can install and read: algorithms +- Algorithm code to study, browsed by topic: The Algorithms - Python +- How classic design patterns look in Python, and which ones to skip: python-patterns +- Workflows as states and transitions, up to full statecharts: python-statemachine + +Python's standard library [is great until you need a sorted collections type](https://grantjenks.com/docs/sortedcontainers/), and Sorted Containers fills that gap in pure Python, as fast as C extensions. SortedList keeps its values in order as you `add()` them, SortedDict is [a dict that keeps a sorted list of its keys](https://grantjenks.com/docs/sortedcontainers/introduction.html#sorted-dict), and SortedSet does the same for a set. + +algorithms collects [minimal, clean, and well-documented implementations](https://github.com/keon/algorithms) of data structures and algorithms. Each file stands alone, with docstrings, type hints, and complexity notes, and its README says they're designed to be read and learned from. Install it with `pip install algorithms`, then [import an implementation by topic](https://github.com/keon/algorithms#quick-start), like a sort or a graph search. + +The Algorithms - Python implements algorithms [for education](https://github.com/TheAlgorithms/Python), and its README says they may be less efficient than the standard library's own: read them to learn, not to ship. Start from [its directory](https://github.com/TheAlgorithms/Python/blob/master/DIRECTORY.md), which sorts them by topic. Its contributing guide asks each algorithm to [carry type hints and doctests](https://github.com/TheAlgorithms/Python/blob/master/CONTRIBUTING.md#what-is-an-algorithm) that test both valid and erroneous input, so the doctests show you how it behaves. + +python-patterns is [a collection of design patterns and idioms](https://github.com/faif/python-patterns) in Python, one file per pattern, grouped as creational, structural, behavioral, and more. Its README asks you to pay more attention to why you choose a pattern than to how you implement it. It also lists the patterns [not recommended in Python](https://github.com/faif/python-patterns#-anti-patterns): skip a Singleton class, since modules are already singletons, and prefer composition and delegation over deep inheritance. + +python-statemachine builds [flat state machines or full statecharts](https://python-statemachine.readthedocs.io/en/latest/) with compound states, parallel regions, and history, in sync and async code. Declare states as class attributes and transitions with `state.to(target)`, and the [events are the names you assign them to](https://python-statemachine.readthedocs.io/en/latest/tutorial.html#your-first-state-machine). + +Before you hand-write python-patterns' State pattern for a real workflow, read python-statemachine's guide [for coming from the State pattern](https://python-statemachine.readthedocs.io/en/latest/how-to/coming_from_state_pattern.html). The whole workflow sits in one class body, and unreachable states fail when the class is defined, not at runtime. For more places to learn and practice algorithms, see [awesome-algorithms](https://github.com/tayllan/awesome-algorithms). diff --git a/website/data/category_intros/asynchronous-programming.md b/website/data/category_intros/asynchronous-programming.md new file mode 100644 index 0000000000..f71a36b997 --- /dev/null +++ b/website/data/category_intros/asynchronous-programming.md @@ -0,0 +1,30 @@ +I/O-bound or CPU-bound? I/O-bound code wants a Python async library, and asyncio comes built in. CPU-bound work goes to a concurrent.futures process pool. + +How to choose: + +- I/O-bound code written with async/await: asyncio +- CPU-bound work, or blocking calls, in a pool of processes or threads: concurrent.futures +- Trio-style task groups and cancel scopes on asyncio, or a library that runs on both asyncio and Trio: AnyIO +- A faster, drop-in event loop for an asyncio app on Linux or macOS: uvloop +- Structured concurrency from the ground up, with its own libraries: Trio +- Existing synchronous code, made concurrent without async/await: gevent +- Network servers and clients with protocols built in, like SSH, mail, and DNS: Twisted +- Processes you manage yourself, talking through queues and pipes: multiprocessing + +asyncio is the standard library's way to [write concurrent code with async/await](https://docs.python.org/3/library/asyncio.html), and many async web servers, database drivers, and task queues build on it. Start your program with `asyncio.run()`, [called once as the main entry point](https://docs.python.org/3/library/asyncio-runner.html#asyncio.run). App code should [rarely need the event loop object](https://docs.python.org/3/library/asyncio-eventloop.html) itself. Run related tasks in an `asyncio.TaskGroup`: when one task fails, it [cancels the rest, which `gather()` doesn't](https://docs.python.org/3/library/asyncio-task.html#asyncio.gather). CPU-heavy code holds up every task on the loop, so [run it in another process](https://docs.python.org/3/library/asyncio-dev.html#asyncio-handle-blocking): hand it to a `ProcessPoolExecutor` with `loop.run_in_executor()`. + +concurrent.futures runs callables on threads or processes behind [the same interface](https://docs.python.org/3/library/concurrent.futures.html). `ThreadPoolExecutor` is [for overlapping I/O](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor). For CPU-bound work on a multi-core machine, the threading docs [advise processes](https://docs.python.org/3/library/threading.html) instead, and `ProcessPoolExecutor` runs them for you. It [takes only picklable functions and arguments](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor), though. Use either executor [in a `with` block](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor.shutdown), which shuts it down and waits for its work to finish. + +AnyIO brings [Trio-like structured concurrency to asyncio](https://anyio.readthedocs.io/en/stable/). Code written against its API runs unmodified on asyncio or Trio, so a library built on it doesn't choose for its users. Its docs see [strong merits in its APIs for applications too](https://anyio.readthedocs.io/en/stable/why.html), starting with cancel scopes for [more predictable cancellation](https://anyio.readthedocs.io/en/stable/why.html#design-problems-with-cancellation). Start with `anyio.run(main)`, which [runs on asyncio unless you pass `backend="trio"`](https://anyio.readthedocs.io/en/stable/basics.html#running-async-programs). Spawn tasks in a [task group](https://anyio.readthedocs.io/en/stable/tasks.html): when one child task raises, the rest are cancelled. + +uvloop is [a drop-in replacement for asyncio's event loop](https://github.com/MagicStack/uvloop), built on libuv. Its README prefers `uvloop.run(main())`, which configures `asyncio.run()` to use uvloop, so the rest of your asyncio code stays the same. uvloop runs on Linux and macOS. + +Trio has [an obsessive focus on usability and correctness](https://trio.readthedocs.io/en/stable/). Child tasks run in a nursery, opened with `async with trio.open_nursery()`, and [Trio never discards their exceptions](https://trio.readthedocs.io/en/stable/tutorial.html#okay-let-s-see-something-cool-already). Functions [take no timeout arguments](https://trio.readthedocs.io/en/stable/reference-core.html#blocking-and-non-blocking-methods): you wrap the code in a cancel scope like `trio.move_on_after()`. Trio runs its own event loop, so asyncio functions [don't work inside `trio.run()`](https://trio.readthedocs.io/en/stable/tutorial.html#task-switching-illustrated). Check [the Trio library list](https://trio.readthedocs.io/en/stable/awesome-trio-libraries.html) for what you need first. + +gevent uses greenlets to give you [a synchronous API on top of an event loop](https://www.gevent.org/intro.html). Its monkey patching swaps the standard library's blocking sockets for cooperative ones, so code that knows nothing about gevent runs concurrently. Most programs should [patch everything with `monkey.patch_all()`](https://www.gevent.org/intro.html#beyond-sockets), and the main module should do it [before any other imports](https://www.gevent.org/api/gevent.monkey.html). + +Twisted is [an event-based framework for internet applications](https://github.com/twisted/twisted) that ships clients and servers for HTTP, SSH, IMAP, POP3, SMTP, DNS, IRC, and XMPP. Write new Twisted code as `async def` coroutines, which its docs [prefer over `inlineCallbacks`](https://docs.twisted.org/en/stable/core/howto/defer-intro.html#inline-callbacks-using-yield), and start one with [`Deferred.fromCoroutine()`](https://docs.twisted.org/en/stable/core/howto/defer-intro.html#coroutines-with-async-await). + +multiprocessing runs work in subprocesses, so it can [use every processor on a machine](https://docs.python.org/3/library/multiprocessing.html#introduction). Its own docs point to `ProcessPoolExecutor` as the higher-level interface for pooled tasks. Reach for multiprocessing when you need what it adds, like killing a running process or passing data through queues and pipes. Its guidelines say to [avoid shared state](https://docs.python.org/3/library/multiprocessing.html#all-start-methods) and keep the data moving between processes small. + +On an event loop, a blocking call holds up every other task, so push it to a worker thread: asyncio has [`asyncio.to_thread()`](https://docs.python.org/3/library/asyncio-task.html#asyncio.to_thread), AnyIO [`to_thread.run_sync()`](https://anyio.readthedocs.io/en/stable/threads.html), and Trio [`trio.to_thread.run_sync()`](https://trio.readthedocs.io/en/stable/reference-core.html#threads-if-you-must). When multiprocessing or `ProcessPoolExecutor` runs your code in child processes, define its functions in a module. Guard your entry point with `if __name__ == '__main__':` too, so each new process can [import your main module safely](https://docs.python.org/3/library/multiprocessing.html#multiprocessing-safe-main-import). diff --git a/website/data/category_intros/audio-video-processing.md b/website/data/category_intros/audio-video-processing.md new file mode 100644 index 0000000000..feac35ff08 --- /dev/null +++ b/website/data/category_intros/audio-video-processing.md @@ -0,0 +1,28 @@ +To track tempo and beats with a Python audio processing library, load music into librosa. MoviePy cuts and joins video in code. Mutagen reads and writes tags. + +How to choose: + +- Tempo, beats, and other music analysis: librosa +- Scripted video edits, titles, and compositing: MoviePy +- Reading and writing tags: Mutagen +- Reading and writing sound files as NumPy arrays: soundfile +- Slicing, fading, and joining audio clips: pydub +- Frame- and packet-level access to FFmpeg: PyAV +- Reading tags only, without GPL code: tinytag +- Organizing a music collection with MusicBrainz data: beets + +librosa is a Python library for [audio and music signal processing](https://librosa.org/doc/latest/index.html), with the building blocks for music information retrieval: spectrograms, chroma, onsets, tempo, and beats. Load each file with [`librosa.load()`](https://librosa.org/doc/latest/auto_tutorials/01-intro/01-load.html), which resamples it to a standard rate and mixes it down to mono, the defaults the rest of librosa is designed around. + +soundfile reads and writes [any format libsndfile supports](https://python-soundfile.readthedocs.io/en/latest/#read-write-functions), like WAV, FLAC, and OGG: `sf.read()` returns the samples as a NumPy array along with the sample rate, and `sf.write()` saves them. librosa reads files through soundfile, and its docs [recommend using soundfile directly](https://librosa.org/doc/latest/ioformats.html) to write audio, or when you need more control than loading a whole recording into memory. + +MoviePy is for [automating video edits](https://zulko.github.io/moviepy/getting_started/quick_presentation.html#do-i-need-moviepy): cutting scenes, adding titles and subtitles, or composing many videos into one. A [typical script](https://zulko.github.io/moviepy/getting_started/quick_presentation.html#example-code) loads videos as clips, modifies them, puts them together in a `CompositeVideoClip`, and writes the result with `write_videofile()`. + +PyAV binds FFmpeg's libraries for [direct and precise access](https://pyav.basswood.io/docs/stable/) to your media's containers, streams, packets, codecs, and frames. Open a file with `av.open()` and loop over `container.decode(video=0)` to get frames, which [convert to NumPy arrays](https://pyav.basswood.io/docs/stable/cookbook/numpy.html) with `to_ndarray()` and back with `av.VideoFrame.from_ndarray()`. + +Mutagen reads and writes tags in many audio formats through [one API](https://mutagen.readthedocs.io/en/latest/#why-mutagen) that's roughly the same across them. `mutagen.File()` [guesses the file's type](https://mutagen.readthedocs.io/en/latest/user/gettingstarted.html), you set tags like dict keys, and `save()` writes them. Mutagen is GPL. + +tinytag, which is MIT and pure Python, only reads: `TinyTag.get(path)` returns the [title, artist, album, duration, and more](https://github.com/tinytag/tinytag) as attributes. + +beets runs from the command line: it [catalogs your music collection](https://beets.io/) and improves its metadata from MusicBrainz as it goes. Point [`beet import`](https://beets.readthedocs.io/en/stable/guides/tagger.html) at a folder of albums, and it tags the files as it adds them to your library. The docs [recommend this autotagged import](https://beets.readthedocs.io/en/stable/guides/main.html#importing-your-music): it asks you questions along the way, but gets every song's tags right from the start. + +MoviePy and PyAV both run on FFmpeg. When the `ffmpeg` command [does the job](https://pyav.basswood.io/docs/stable/) on its own, call it directly: MoviePy's docs say that's [faster and more memory-efficient](https://zulko.github.io/moviepy/getting_started/quick_presentation.html#do-i-need-moviepy). diff --git a/website/data/category_intros/authentication.md b/website/data/category_intros/authentication.md new file mode 100644 index 0000000000..a02aef1aaf --- /dev/null +++ b/website/data/category_intros/authentication.md @@ -0,0 +1,27 @@ +Letting users log in with Google takes one Python authentication library: django-allauth on Django, which does password sign-up too, or Authlib elsewhere. + +How to choose: + +- Sign-up, password login, and Google or GitHub login on a Django site: django-allauth +- Google or GitHub login, OAuth API calls, or an OAuth server outside Django: Authlib +- Adding OAuth to a web framework or HTTP library you maintain: oauthlib +- Your own OAuth 2.0 server on Django, or tokens for a Django REST framework API: django-oauth-toolkit +- Signing and verifying JSON Web Tokens: PyJWT +- Permissions you grant per object to users and groups: django-guardian +- Permissions that follow from the object's data, with no extra tables: django-rules + +django-allauth exists to handle [local and social accounts in one app](https://docs.allauth.org/en/latest/introduction/index.html): a user can sign up with a password or log in with Google. Follow the [quickstart](https://docs.allauth.org/en/latest/installation/quickstart.html): add its authentication backend, apps, and middleware, then include `allauth.urls` under `accounts/`. Its login, logout, and password views can replace the ones in `django.contrib.auth.urls`. Give each OAuth provider its client ID and secret in a [`SocialApp` or the `SOCIALACCOUNT_PROVIDERS` setting](https://docs.allauth.org/en/latest/installation/quickstart.html#post-installation). For a single-page or mobile app, add [`allauth.headless`](https://docs.allauth.org/en/latest/headless/introduction.html). + +Authlib covers both sides of OAuth 2.0 and OpenID Connect. As a client, it comes in [two kinds](https://docs.authlib.org/en/latest/oauth2/client/index.html): HTTP clients for scripts and service-to-service calls, and web clients that log users in through Flask, Django, Starlette, or FastAPI. For login, [create an `OAuth` registry and `register()` each provider](https://docs.authlib.org/en/latest/oauth2/client/web/index.html). For an OpenID Connect provider like Google, pass its discovery URL as [`server_metadata_url`](https://docs.authlib.org/en/latest/oauth2/client/web/index.html#parsing-id-token), and Authlib reads the other endpoints from it. To [run your own OAuth 2.0 or OpenID Connect provider](https://docs.authlib.org/en/latest/oauth2/authorization-server/index.html), it has server integrations for Flask and Django. + +PyJWT [encodes and decodes JSON Web Tokens](https://pyjwt.readthedocs.io/en/latest/). Treat every token as untrusted input: [hard-code the `algorithms` you pass to `jwt.decode()`](https://pyjwt.readthedocs.io/en/latest/algorithms.html#specifying-an-algorithm), never read them from the token's own header, and don't mix HS and RS algorithms. For tokens from an OpenID Connect provider, point [`PyJWKClient` at its JWKS endpoint](https://pyjwt.readthedocs.io/en/latest/usage.html#retrieve-rsa-signing-keys-from-a-jwks-endpoint) instead of hard-coding its public keys. + +oauthlib implements OAuth's logic [without assuming a web framework or HTTP request object](https://github.com/oauthlib/oauthlib), so a framework or library maintainer can write a thin layer on top and get OAuth support. Its FAQ says [most people use it indirectly](https://oauthlib.readthedocs.io/en/latest/faq.html#how-do-i-use-oauthlib-with-google-twitter-and-other-providers): django-oauth-toolkit and django-allauth both build on it. To build a provider on it, most of the work is [a `RequestValidator`](https://oauthlib.readthedocs.io/en/latest/oauth2/server.html#implement-a-validator) that maps its checks to your storage. + +django-oauth-toolkit is an [OAuth 2.0 authorization server for Django](https://django-oauth-toolkit.readthedocs.io/en/latest/): it issues and manages tokens from your existing project, and can also protect a Django or Django REST framework API. Django REST framework's docs [recommend it for OAuth 2.0](https://www.django-rest-framework.org/api-guide/authentication/#django-oauth-toolkit). [Install it](https://django-oauth-toolkit.readthedocs.io/en/latest/install.html) by adding `oauth2_provider` to `INSTALLED_APPS`, including its URLs under `o/`, and migrating. For an API, [set `OAuth2Authentication`](https://django-oauth-toolkit.readthedocs.io/en/latest/rest-framework/getting_started.html) as Django REST framework's authentication class and guard views with `TokenHasScope`. + +Django has [a foundation for object permissions but no implementation](https://docs.djangoproject.com/en/stable/topics/auth/customizing/#handling-object-permissions), and django-guardian fills it with an [extra authentication backend](https://django-guardian.readthedocs.io/en/latest/configuration/). Grant a permission on one object with `assign_perm()`, then check it with Django's own [`user.has_perm(perm, obj)`](https://django-guardian.readthedocs.io/en/latest/userguide/checks/). Django REST framework's `DjangoObjectPermissions` [names it as an example backend](https://www.django-rest-framework.org/api-guide/permissions/#djangoobjectpermissions). + +django-rules gives Django object permissions [without a database](https://github.com/dfunckt/django-rules). A permission is a rule built from predicates, plain functions like `is_book_author(user, book)` that you combine with `|`, `&`, and `~`. Add its [backend to `AUTHENTICATION_BACKENDS`](https://github.com/dfunckt/django-rules#configuring-django), register each permission with `rules.add_perm()`, and check it with `user.has_perm()`. Keep predicates and rules in their own modules, and let [`AutodiscoverRulesConfig` import each app's `rules.py`](https://github.com/dfunckt/django-rules#best-practices) at startup. + +On either side of OAuth, use PKCE. django-oauth-toolkit's security guide maps [the OAuth security recommendations of RFC 9700](https://django-oauth-toolkit.readthedocs.io/en/latest/security.html) to its settings: PKCE is one, and the implicit and password grants must not be used. Authlib's client [turns PKCE on with `code_challenge_method`](https://docs.authlib.org/en/latest/oauth2/client/web/index.html#oauth-2-0-code-challenge). Also, don't pick JWTs for sessions just because they sound stateless. django-allauth's docs point out that [if a token must stop working at logout, you need state to revoke it](https://docs.allauth.org/en/latest/headless/token-strategies/jwt-tokens.html). JWTs help most when each service checks tokens without asking a central server. diff --git a/website/data/category_intros/build-tools.md b/website/data/category_intros/build-tools.md new file mode 100644 index 0000000000..0e50a00bf0 --- /dev/null +++ b/website/data/category_intros/build-tools.md @@ -0,0 +1,16 @@ +Tasks like running your tests go in pyproject.toml for Poe the Poet to run. C and C++ projects that replace Make with a Python build tool pick SCons. + +How to choose: + +- Project tasks in pyproject.toml, run in your uv or Poetry environment: Poe the Poet +- Compiling C or C++ in place of Make: SCons +- Tasks written as Python functions that run shell commands: Invoke +- Steps that skip when their input files haven't changed: doit + +Poe the Poet reads tasks from a [`[tool.poe.tasks]` table](https://poethepoet.natn.io/tasks/index.html) in pyproject.toml. A string like `test = "pytest"` is a whole task, run as a command without a shell. In a uv or Poetry project, poe [runs tasks in the project's virtualenv](https://poethepoet.natn.io/#quick-start), so you don't type `uv run` or `poetry run` in front. + +Invoke tasks are Python functions in a [`tasks.py` file](https://docs.pyinvoke.org/en/latest/getting-started.html#defining-and-running-task-functions), decorated with `@task` and taking a context as their first argument. Inside, `c.run()` runs a shell command. A function's arguments [become CLI flags](https://docs.pyinvoke.org/en/latest/getting-started.html#task-parameters), and a task can name [pre-tasks](https://docs.pyinvoke.org/en/latest/getting-started.html#declaring-pre-tasks) to always run before it. + +doit reads tasks from a [`dodo.py` file](https://pydoit.org/tutorial-1.html), where each `task_` function returns a dict of actions, file dependencies, and targets instead of running anything. It saves a [signature of each `file_dep`](https://pydoit.org/tasks.html#file-dep) and skips a task whose inputs haven't changed. A task that [declares no input](https://pydoit.org/dependencies.html#doit-up-to-date-definition) never counts as up to date, so list the files each task reads in `file_dep`. + +SCons reads your build from an [`SConstruct` file, which is a Python script](https://scons.org/doc/production/HTML/scons-user.html#sect-sconstruct-python): `Program('hello.c')` there is enough for `scons` to compile a program. Calling a builder like `Program()` [declares what you want built](https://scons.org/doc/production/HTML/scons-user.html#sect-order-independent) instead of building it on that line, and SCons works out when to run each step. diff --git a/website/data/category_intros/built-in-classes-enhancement.md b/website/data/category_intros/built-in-classes-enhancement.md new file mode 100644 index 0000000000..dfe74eed0d --- /dev/null +++ b/website/data/category_intros/built-in-classes-enhancement.md @@ -0,0 +1,16 @@ +Skip hand-written dunder methods, and when you compare Python dataclass alternatives, pick attrs for its validators and converters. Box gives dicts dot access. + +How to choose: + +- Classes with generated dunder methods, validators, and converters: attrs +- Dicts read with dot notation, nested ones included: python-box +- Looking up a key by its value, with both directions in sync: bidict +- A faster drop-in for the `uuid` module: uuid-utils + +attrs [writes the dunder methods](https://www.attrs.org/en/latest/) that implement object protocols, so you don't have to. For new code, its docs [recommend the modern API](https://www.attrs.org/en/latest/names.html#tl-dr): `@define` on the class and `field()` for an attribute, or `@frozen` for immutable instances, with slots on by default. Type annotations stay optional. The standard library's dataclasses [gave up features for simplicity](https://www.attrs.org/en/latest/why.html#data-classes), like validators and converters, so a class that needs them goes to attrs. It [isn't a full serialization library](https://www.attrs.org/en/latest/overview.html#what-attrs-is-not), though: to serialize and validate your attrs classes, its docs point you to its sibling project, cattrs. + +python-box installs Box, a [`dict` subclass](https://github.com/cdgriffith/Box) whose keys you can also read as attributes, even ones like `"imdb stars"`. Nested dicts and lists become Box and BoxList objects, so dot access works all the way down: `movie_box.Robin_Hood_Men_in_Tights.imdb_stars`. It [reads and writes JSON, YAML, TOML, and msgpack](https://github.com/cdgriffith/Box/wiki/Converters) with methods like `from_json()` and `to_json()`, and `to_dict()` gives you plain dicts back. Keep Box for data whose keys change: attrs' docs say a dict with a fixed and known set of keys [is an object, not a hash](https://www.attrs.org/en/latest/why.html#dicts), and belongs in a class. + +bidict gives you [bidirectional mappings](https://bidict.readthedocs.io/intro.html) that work like dicts: look up a value by its key as usual, or a key by its value through `.inverse`, which stays in sync as you update the mapping. A single dict holding both directions mixes keys with values. Modeling the mapping correctly takes two one-way mappings kept in sync, [which is what bidict does](https://bidict.readthedocs.io/intro.html#why-can-t-i-just-use-a-dict) under the hood. + +uuid-utils is a [fast, drop-in replacement for Python's `uuid` module](https://aminalaee.github.io/uuid-utils/latest/), powered by Rust. Import it under the module's name, `import uuid_utils as uuid`, and call `uuid.uuid4()` or `uuid.uuid7()` as usual. Django and some other frameworks require the standard library's own `uuid.UUID` instances. For those, [import `uuid_utils.compat`](https://aminalaee.github.io/uuid-utils/latest/#compatibility-with-python-uuid) instead, which returns them and still outperforms the standard library. diff --git a/website/data/category_intros/caching.md b/website/data/category_intros/caching.md new file mode 100644 index 0000000000..cca8b803fa --- /dev/null +++ b/website/data/category_intros/caching.md @@ -0,0 +1,23 @@ +Expensive function calls get cached in memory by one cachetools decorator. DiskCache stores cached data on local disk, with no server to run. + +How to choose: + +- Memoizing a function in one process, with a size limit or an expiry time: cachetools +- A persistent cache on local disk that several processes share: DiskCache +- A Django cache backend on local disk: DiskCache +- A shared Redis or Memcached cache outside Django, with coordinated value regeneration: dogpile.cache +- A Django cache and session store on Redis: django-redis +- Caching HTTP responses in HTTPX or Requests, or setting cache headers in FastAPI: Hishel +- Django ORM queries cached in Redis and invalidated when the data changes: Cacheops + +cachetools offers [variants of the standard library's `@lru_cache` decorator](https://cachetools.readthedocs.io/en/latest/). Its cache classes have a maximum size and drop items by different rules, like least recently used or a per-item time-to-live. Wrap a function in `@cached` with one of them: the README's `@cached(cache=TTLCache(maxsize=1024, ttl=600))` [keeps results for ten minutes](https://github.com/tkem/cachetools). To keep the `lru_cache` API instead, use the [`cachetools.func` decorators](https://cachetools.readthedocs.io/en/latest/#module-cachetools.func). + +DiskCache keeps keys and values in [a SQLite database and a directory of files](https://grantjenks.com/docs/diskcache/), with no server process to run. Point a `Cache` at a directory, and [other threads and processes that open the same directory](https://grantjenks.com/docs/diskcache/tutorial.html#tutorial-cache) share its entries. Its `memoize()` decorator takes [arguments like `lru_cache`'s](https://grantjenks.com/docs/diskcache/tutorial.html#tutorial-fanoutcache), plus an expiry time. In Django, [set `diskcache.DjangoCache` as the `BACKEND`](https://grantjenks.com/docs/diskcache/tutorial.html#tutorial-djangocache) and a directory as the `LOCATION`. + +dogpile.cache gives you [one caching API over Redis, Memcached, and other backends](https://dogpilecache.sqlalchemy.org/en/latest/usage.html), without depending on a web framework. Create a region with `make_region()`, configure its backend, and call `get_or_create()` with a key and a function that computes the value. For shared Redis or Memcached caches, enable [`distributed_lock`](https://dogpilecache.sqlalchemy.org/en/latest/api.html) so one worker regenerates an expired value while others use the previous one. + +django-redis is a [full-featured Redis cache and session backend](https://github.com/jazzband/django-redis#introduction) for Django. [Set `django_redis.cache.RedisCache` as the `BACKEND`](https://github.com/jazzband/django-redis#configure-as-cache-backend) and a redis-py URL like `redis://127.0.0.1:6379/1` as the `LOCATION`. Sessions then need no extra backend: [point `SESSION_ENGINE` at Django's cache sessions](https://github.com/jazzband/django-redis#configure-as-session-backend), and they live in Redis too. Its clients, parsers, and serializers [are pluggable](https://github.com/jazzband/django-redis#why-use-django-redis). + +Hishel caches HTTP responses [by RFC 9111](https://hishel.com/overview.html), the rules browsers follow. It works in sync and async code, and comes with a backend that stores responses in SQLite. In HTTPX, [use `SyncCacheClient` or `AsyncCacheClient`](https://hishel.com/httpx.html#cache-clients), drop-in replacements for `Client` and `AsyncClient`. In Requests, [mount its `CacheAdapter`](https://hishel.com/requests.html#quick-start) on a `Session`. In FastAPI, [the `cache()` dependency](https://hishel.com/fastapi.html) sends Cache-Control headers to browsers and CDNs, and Hishel's ASGI middleware adds a cache on your server. + +Cacheops caches Django ORM queries in Redis and [invalidates them on `save()`, `delete()`, and m2m changes](https://github.com/Suor/django-cacheops#invalidation), only where a change could affect the result. Add `cacheops` to `INSTALLED_APPS`, then set `CACHEOPS_REDIS` and [a `CACHEOPS` profile per app](https://github.com/Suor/django-cacheops#setup) like `'app_name.*'`. Profiles with `ops` cache those queries automatically. To cache by hand, [call `.cache()` on a queryset, or decorate a function with `@cached_as`](https://github.com/Suor/django-cacheops#usage), and it still invalidates with the model. diff --git a/website/data/category_intros/cli-development.md b/website/data/category_intros/cli-development.md new file mode 100644 index 0000000000..9c2009bcf6 --- /dev/null +++ b/website/data/category_intros/cli-development.md @@ -0,0 +1,38 @@ +Python's own argparse covers basic command-line apps. Beyond that, pick Click or Typer as your Python CLI library, and style the output with Rich. + +How to choose: + +- Basic command-line app with no dependencies: argparse +- Nested commands, or subcommands loaded lazily: Click +- Options declared once as type-hinted function parameters: Typer +- Interactive prompts and REPLs: prompt_toolkit +- CLI from existing code without writing a parser: Python Fire +- Progress bar for a loop: tqdm, or alive-progress for animated bars +- Colored text, tables, and logs: Rich, or colorama for ANSI colors on Windows +- Full-screen app in the terminal or a browser: Textual +- Terminal widgets on an event loop you already run: Urwid +- Full-screen forms and ASCII animations: asciimatics + +argparse is in the standard library. Python's docs call it [the recommended choice](https://docs.python.org/3/library/optparse.html#choosing-an-argument-parser) when you have no more specific needs, since it gives you the most out of the box for the least code. It [writes the help and usage messages](https://docs.python.org/3/library/argparse.html) and reports invalid arguments for you. For commands written as decorated functions, the same docs point to Click, and for a CLI that works with static type checking, to Typer. + +Click builds a CLI from [commands declared with decorators](https://click.palletsprojects.com/en/stable/quickstart/). You can nest them to any depth, and Click can [load subcommands lazily](https://click.palletsprojects.com/en/stable/) at runtime. It also [reads option values from environment variables](https://click.palletsprojects.com/en/stable/why/) and comes with helpers for ANSI colors, terminal size, and launching editors. + +Typer builds a CLI from your function signatures: you [declare each argument and option once](https://typer.tiangolo.com/), as a type-hinted function parameter. Write those hints with `Annotated`, which its tutorial [recommends wherever possible](https://typer.tiangolo.com/tutorial/arguments/optional/). Its docs say you get [the best results by pairing it with Rich](https://typer.tiangolo.com/tutorial/printing/): Typer structures the commands and options, and Rich displays the output. + +prompt_toolkit is for interactive input. It can [replace GNU readline](https://python-prompt-toolkit.readthedocs.io/en/stable/) or build full-screen apps. For a REPL, use a [`PromptSession`](https://python-prompt-toolkit.readthedocs.io/en/stable/pages/asking_for_input.html), which keeps the history for the whole session. + +Python Fire turns any Python object into a CLI: [call `fire.Fire()`](https://github.com/google/python-fire/blob/master/docs/guide.md) at the end of your program. Its README pitches it for [developing and debugging your code](https://github.com/google/python-fire), exploring existing code, and turning other people's code into a CLI. It takes each argument's type from the value you pass, not from the function signature. + +tqdm adds a progress bar to any loop: [wrap the iterable in `tqdm()`](https://github.com/tqdm/tqdm). Import it from `tqdm.auto`, which picks the console bar or the Jupyter widget for you. alive-progress wraps the loop in an [`alive_bar` context manager](https://github.com/rsalmei/alive-progress) instead, with a spinner that speeds up and slows down with your throughput. + +Rich writes colored text, tables, Markdown, and syntax-highlighted code to the terminal. Start with its [drop-in `print`](https://rich.readthedocs.io/en/stable/introduction.html), then create [one `Console` at module level](https://rich.readthedocs.io/en/stable/console.html) for the rest of your app. For colored logs, send the logging module's output through [`RichHandler`](https://rich.readthedocs.io/en/stable/logging.html). + +colorama makes ANSI escape codes work on Windows and does nothing on other platforms. If that's all you need, call [`just_fix_windows_console()`](https://github.com/tartley/colorama). + +Textual apps run in the terminal, and the `textual serve` command from textual-dev [serves them in a browser](https://textual.textualize.io/guide/devtools/). Build one by [subclassing `App`](https://textual.textualize.io/guide/app/), and style its widgets with [CSS](https://textual.textualize.io/guide/CSS/). + +Urwid is a [console widget construction set](https://urwid.org/manual/overview.html) rather than a finished UI library. It runs on [your choice of event loop](https://github.com/urwid/urwid): asyncio, or another one you already use. It's [licensed under the LGPL](https://github.com/urwid/urwid/blob/master/COPYING), while Textual and asciimatics use permissive licenses. + +asciimatics does [full-screen text UIs, from interactive forms to ASCII animations](https://github.com/peterbrittain/asciimatics): create a Screen, build a Scene from Effect objects, and let the Screen play it. + +Ship your CLI as an installable package with a `[project.scripts]` entry point, not a file users run with `python`. [Click recommends it](https://click.palletsprojects.com/en/stable/entry-points/), and shell completion in Click and [Typer](https://typer.tiangolo.com/tutorial/package/) works through that entry point. Test your commands in-process: Click and Typer both provide a [`CliRunner`](https://click.palletsprojects.com/en/stable/testing/), and Textual's [`run_test()`](https://textual.textualize.io/guide/testing/) drives your app as if you were using the keyboard and mouse. diff --git a/website/data/category_intros/cli-tools.md b/website/data/category_intros/cli-tools.md new file mode 100644 index 0000000000..e630e5b2e9 --- /dev/null +++ b/website/data/category_intros/cli-tools.md @@ -0,0 +1,26 @@ +Swap cURL for HTTPie, and your database's own client for a Python CLI tool with autocompletion: pgcli, mycli, litecli, or IRedis. + +How to choose: + +- Calling HTTP APIs from the terminal: HTTPie +- A database shell with autocompletion: pgcli for PostgreSQL, mycli for MySQL, litecli for SQLite, IRedis for Redis +- Downloading video or audio from YouTube and other sites: yt-dlp +- A new project from a template: Cookiecutter, or Copier to pull in later template changes +- A shell you script in Python: xonsh +- A tmux session with its windows and panes, from one YAML file: tmuxp + +HTTPie is a command-line HTTP client for [testing, debugging, and generally interacting with APIs](https://httpie.io/docs/cli) and HTTP servers. It formats and colors the output. A request reads like `http PUT pie.dev/put X-API-Token:123 name=John`: `Header:Value` items set headers, and `field=value` items fill the body, which HTTPie [sends as JSON by default](https://httpie.io/docs/cli/json). + +pgcli, mycli, litecli, and IRedis are terminal clients with autocompletion and syntax highlighting, one per database. Give [pgcli](https://github.com/dbcli/pgcli) a database name or a `postgresql://` URI, and [litecli](https://github.com/dbcli/litecli) the path to a SQLite file. mycli also [works with MariaDB](https://github.com/dbcli/mycli), Percona, TiDB, and Apache Doris. IRedis behaves like Redis's own client in most cases, but it's [safer on production servers](https://github.com/laixintao/iredis): it stops you from accidentally running dangerous commands like `KEYS *`. + +yt-dlp downloads audio and video from [thousands of sites](https://github.com/yt-dlp/yt-dlp). Install ffmpeg too: yt-dlp [needs it to merge separate video and audio files](https://github.com/yt-dlp/yt-dlp#dependencies). + +Cookiecutter creates projects from templates, and its templates [work for any language](https://github.com/cookiecutter/cookiecutter), not only Python. [Run `cookiecutter gh:audreyfeldroy/cookiecutter-pypackage`](https://cookiecutter.readthedocs.io/en/stable/usage.html), answer its prompts, and you get a new project from that GitHub template. + +Copier also generates projects from templates, but it calls itself [a code lifecycle management tool](https://copier.readthedocs.io/en/stable/comparisons/). When the template changes, `copier update` brings the changes into projects you already generated. [It works best](https://copier.readthedocs.io/en/stable/updating/) when both are in Git: the template tagged, your project clean, with the `.copier-answers.yml` file that records your answers. Templates can run code: Cookiecutter's [pre- and post-generate scripts](https://github.com/cookiecutter/cookiecutter), Copier's tasks. [Generate projects only from templates you trust](https://copier.readthedocs.io/en/stable/generating/). + +xonsh is a shell whose language is [a superset of Python](https://github.com/xonsh/xonsh), with shell commands built in. It isn't POSIX-compatible, so don't make it your login shell with `chsh`. Its docs [recommend a xonsh profile in your terminal emulator](https://xon.sh/install.html#before-installing) instead. + +tmuxp launches a whole tmux session [from one YAML or JSON file](https://tmuxp.git-pull.com/quickstart/): its windows, its panes, and the commands in them. Save the file as `.tmuxp.yaml` in a project, and [`tmuxp load path/to/project/`](https://github.com/tmux-python/tmuxp) builds the session. + +Several of these tools' docs have you [install them as CLI tools](https://github.com/cookiecutter/cookiecutter), and add them to a project only to use them from Python. Looking for a library to build your own Python CLI tool? That's [CLI Development](/categories/cli-development/). diff --git a/website/data/category_intros/cms.md b/website/data/category_intros/cms.md new file mode 100644 index 0000000000..fa32524f2b --- /dev/null +++ b/website/data/category_intros/cms.md @@ -0,0 +1,12 @@ +Both Python CMS picks run on Django: Wagtail for page types your developers define, like articles, and django CMS for editors building pages on the live site. + +How to choose: + +- Content types your developers define, like articles and events: Wagtail +- Editors composing pages from reusable components on the live site: django CMS + +Wagtail is [not an instant website in a box](https://docs.wagtail.org/en/stable/getting_started/the_zen_of_wagtail.html#wagtail-is-not-an-instant-website-in-a-box): expect to write code. Start a project with [`wagtail start`](https://docs.wagtail.org/en/stable/getting_started/quick_install.html). Each page type is [a Django model that inherits from `Page`](https://docs.wagtail.org/en/stable/topics/pages.html), so give each kind of content its own type with its own fields. An event page with a date and a location [can show up in a calendar](https://docs.wagtail.org/en/stable/getting_started/the_zen_of_wagtail.html#a-cms-should-get-information-out-of-an-editor-s-head-and-into-a-database-as-efficiently-and-directly-as-possible), and a styled heading on a generic page can't. Use [StreamField](https://docs.wagtail.org/en/stable/topics/streamfield.html) for pages without a fixed structure, like blog posts, and [snippets](https://docs.wagtail.org/en/stable/topics/snippets/index.html) for content that doesn't need its own page, like headers and footers. For a headless site, use its [built-in API](https://docs.wagtail.org/en/stable/advanced_topics/api/index.html). + +In django CMS, [editors compose pages from plugins](https://docs.django-cms.org/en/stable/explanation/philosophy.html#three-disciplines-three-surfaces) in a toolbar on the live site, and developers write standard Django. Each template [declares its placeholders](https://docs.django-cms.org/en/stable/tutorials/02-templates-placeholders.html) with `{% placeholder %}`, and `CMS_TEMPLATES` lists the templates editors can pick from. For a region that is the same on every page, like a footer, use [`{% static_alias %}`](https://docs.django-cms.org/en/stable/tutorials/02-templates-placeholders.html#a-reusable-region-with-static-alias), so the content is stored once. For content from your own app, ask [where it lives](https://docs.django-cms.org/en/stable/explanation/composition.html). If it fits in a placeholder on a page, write a plugin. If it has its own list view, detail view, and URL, mount it as an app with an apphook. The core [publishes as you edit](https://docs.django-cms.org/en/stable/explanation/publishing.html), which is rarely enough for a production site with editors, so add a versioning package. + +Both are built on Django. Wagtail [deploys like a Django site](https://docs.wagtail.org/en/stable/deployment/index.html). You can add either one to an existing Django project: Wagtail [integrates into one](https://docs.wagtail.org/en/stable/getting_started/integrating_into_django.html), and django CMS [doesn't make you rebuild around it](https://docs.django-cms.org/en/stable/explanation/philosophy.html#implications-for-projects). Neither fits a very small or static site. For a two-page brochure, django CMS calls itself [overkill](https://docs.django-cms.org/en/stable/explanation/philosophy.html#when-django-cms-may-not-be-the-right-fit), and a static site generator is lighter. diff --git a/website/data/category_intros/code-analysis.md b/website/data/category_intros/code-analysis.md new file mode 100644 index 0000000000..14864750e1 --- /dev/null +++ b/website/data/category_intros/code-analysis.md @@ -0,0 +1,51 @@ +Lint and format with Ruff. It doesn't check types, so add a type checker, and let pre-commit run each Python code analysis tool on every commit. + +How to choose: + +- Linting, formatting, and import sorting in one tool: Ruff +- Running every check before each commit: pre-commit +- Rules on which of your modules may import which: Import Linter +- Unused code: Vulture +- Functions too complex for people to read: complexipy +- Several analyzers under one command, with ready-made profiles: Prospector +- Black's own formatting, with imports sorted to match: Black and isort +- Checks Ruff can't run: Pylint for deeper inference, Flake8 for your own plugins +- Security issues: Bandit +- Refactoring: Rope from your editor or a script, ast-grep for pattern rewrites +- Type checking Django or Pydantic code: mypy with its plugins, Pyrefly without +- Adding types to a partly typed codebase: ty +- Type checking in VS Code: Pyright, which Pylance runs on + +Ruff [can replace Flake8, Black, and isort](https://docs.astral.sh/ruff/), plus dozens of Flake8 plugins. Its docs suggest you [start with a small rule set](https://docs.astral.sh/ruff/linter/#rule-selection), add one group at a time, and list them explicitly in `select`. Ruff is [a linter, not a type checker](https://docs.astral.sh/ruff/faq/#how-does-ruff-compare-to-ty-mypy-pyright-or-pyre), so run a type checker alongside it. + +pre-commit installs and runs the hooks you list in `.pre-commit-config.yaml`, [written in any language](https://pre-commit.com/). Run `pre-commit install` [first thing after you clone](https://pre-commit.com/#usage) a repo that uses it, so the hooks run on every `git commit`, and add `pre-commit run --all-files` [as a CI step](https://pre-commit.com/#usage-in-continuous-integration). + +Import Linter checks [contracts on the imports between your modules](https://import-linter.readthedocs.io/en/stable/): `lint-imports` fails when code breaks one. A [layers contract](https://import-linter.readthedocs.io/en/stable/contract_types/layers/) lets higher layers import lower ones but not the other way around, and it counts chains of imports through other modules too. + +Vulture finds unused code, and [running it on your tests too](https://github.com/jendrikseipp/vulture) shows you untested code. + +complexipy measures [cognitive complexity](https://complexipy.com/): how hard code is for people to understand. Unlike cyclomatic complexity, it counts nesting. Its docs recommend [running it alongside Ruff](https://complexipy.com/comparison-with-ruff/#use-both-recommended), whose complexity rule measures something else. + +Prospector runs Pylint and other analyzers together, and aims to be [useful out of the box](https://prospector.landscape.io/en/stable/), with default profiles so you don't have to sort out which warnings matter. When your project uses Django, Flask, or Celery, it [adapts its checks](https://prospector.landscape.io/en/stable/#adapting-to-dependencies) to that framework. + +With Black, you [give up control over hand-formatting](https://black.readthedocs.io/en/stable/), and in return, code looks the same in every project and diffs stay small. Its style options are [deliberately limited](https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html). If you sort imports with isort, set [the black profile](https://isort.readthedocs.io/en/latest/configuration/black_compatibility.html) so the two agree. + +Pylint [infers the actual values](https://pylint.readthedocs.io/en/latest/#what-differentiates-pylint) in your code instead of trusting its type hints. That makes it slower than other linters, but it finds more in code that isn't fully typed, and it's why Ruff [isn't a drop-in replacement](https://docs.astral.sh/ruff/faq/#how-does-ruffs-linter-compare-to-pylint) for Pylint. Because of that speed, run it [in CI or a pre-push hook](https://pylint.readthedocs.io/en/latest/user_guide/installation/pre-commit-integration.html) rather than on every commit. + +Flake8 [wraps pyflakes, pycodestyle, and McCabe](https://github.com/PyCQA/flake8) in one command, and [its plugins](https://flake8.pycqa.org/en/latest/user/using-plugins.html) are much of its appeal. Keep it when you depend on a plugin of your own, since Ruff [doesn't support custom lint rules](https://docs.astral.sh/ruff/faq/#how-does-ruffs-linter-compare-to-flake8). + +Bandit [finds common security issues](https://bandit.readthedocs.io/en/latest/) in Python code. Scan a whole tree with [`bandit -r`](https://bandit.readthedocs.io/en/latest/start.html). + +Rope is a refactoring library that [many IDEs and editors use](https://rope.readthedocs.io/en/latest/overview.html) to rename, extract, and move code, and scripts can run the same refactorings through [its library API](https://rope.readthedocs.io/en/latest/library.html). + +ast-grep matches code by its syntax tree, with [patterns you write like ordinary code](https://ast-grep.github.io/guide/quick-start.html#pattern). Start with a one-off rewrite on the command line: [`ast-grep run --pattern ... --rewrite ...`](https://ast-grep.github.io/guide/rewrite-code.html#using-ast-grep-run-p-pat-rewrite) shows you a diff of the changes. When the same code smell keeps coming back, its docs suggest you [write a YAML rule](https://ast-grep.github.io/guide/introduction.html#progressiveness) and run all your rules with `ast-grep scan`. + +mypy is [designed for gradual typing](https://mypy.readthedocs.io/en/stable/): by default, it [skips functions without annotations](https://mypy.readthedocs.io/en/stable/getting_started.html#dynamic-vs-static-typing), so you can add type hints slowly. Check in one config file, [run mypy in CI](https://mypy.readthedocs.io/en/stable/existing_code.html#run-mypy-consistently-and-prevent-regressions) so new type errors don't land, and aim for `--strict`. Its [plugins](https://mypy.readthedocs.io/en/stable/extending_mypy.html#extending-mypy-using-plugins) teach it libraries the standard type system can't describe. + +Pyrefly is a type checker and language server with [built-in support for Pydantic and Django](https://github.com/facebook/pyrefly), so those work without plugins. It reads an existing mypy or Pyright config, so you can [try it alongside your current checker](https://pyrefly.org/en/docs/compare/#try-it-alongside-your-current-checker) before you switch. + +ty is a type checker and language server [designed for adoption](https://docs.astral.sh/ty/) on partly typed code. Unlike mypy, it [checks the bodies of unannotated functions](https://docs.astral.sh/ty/coming-from-mypy-or-pyright/), and it leaves [missing-annotation checks](https://docs.astral.sh/ty/reference/typing-faq/#why-doesnt-ty-warn-about-missing-type-annotations) to Ruff's `ANN` rules. + +Pyright [checks all code by default](https://github.com/microsoft/pyright/blob/main/docs/mypy-comparison.md#type-checking-unannotated-code), annotated or not, which its language server features rely on. In VS Code, its docs [recommend Pylance](https://github.com/microsoft/pyright/blob/main/docs/installation.md), which includes Pyright. Run it [in CI to keep the project type clean](https://github.com/microsoft/pyright/blob/main/docs/getting-started.md), then turn on `# pyright: strict` file by file. + +Pylint, Prospector, Import Linter, and ty all need to see your project's dependencies, so run them [from the project's own environment](https://pylint.readthedocs.io/en/latest/user_guide/installation/pre-commit-integration.html), not a separate one. For more Python code analysis tools, see [awesome-static-analysis](https://github.com/analysis-tools-dev/static-analysis). diff --git a/website/data/category_intros/computer-vision.md b/website/data/category_intros/computer-vision.md new file mode 100644 index 0000000000..1fa53d777b --- /dev/null +++ b/website/data/category_intros/computer-vision.md @@ -0,0 +1,27 @@ +OpenCV processes images and video. Training your own detector needs more than a Python computer vision library scoped to inference, so reach for Ultralytics. + +How to choose: + +- Reading, editing, and analyzing images and video: OpenCV +- Detecting, segmenting, or tracking objects with a trained model: Ultralytics +- Image operations inside a PyTorch model or training loop: Kornia +- Browsing a dataset, finding label mistakes, and seeing where a model fails: FiftyOne +- Printed text in clean scans: pytesseract +- Text in photos or degraded scans: EasyOCR +- Tables and page layouts, turned into Markdown or JSON: PaddleOCR + +OpenCV's Python bindings [convert every OpenCV array to and from a NumPy array](https://github.com/opencv/opencv/blob/5.x/doc/py_tutorials/py_setup/py_intro/py_intro.markdown), so the images you load work with any other library that uses NumPy. + +Ultralytics runs YOLO models for detection, segmentation, pose estimation, and tracking from [one Python package and CLI](https://docs.ultralytics.com/). Load a pretrained model and [call `.train()` on your own dataset](https://docs.ultralytics.com/guides/finetuning-guide/): the weights carry over with no extra code. Ultralytics is AGPL, so [using it without open-sourcing your project](https://www.ultralytics.com/license) takes its Enterprise License. + +Kornia is [a differentiable computer vision library like OpenCV, with strong GPU support](https://kornia.readthedocs.io/en/latest/get-started/introduction.html), built on PyTorch tensors. Its FAQ states the split: Kornia [provides operators to train networks, while OpenCV's scope is inference](https://kornia.readthedocs.io/en/latest/community/faqs.html). + +FiftyOne is for building better datasets. [Load your images and labels as a Dataset](https://docs.voxel51.com/user_guide/basics.html) to search and sort samples and spot label mistakes. Then add your model's predictions and [evaluate them sample by sample](https://docs.voxel51.com/user_guide/evaluation/index.html) in the FiftyOne App to see where it goes wrong. + +pytesseract is [a wrapper for Google's Tesseract engine](https://github.com/madmaze/pytesseract), which you install on its own. `image_to_string()` returns the text, and `image_to_data()` adds boxes and confidences. + +EasyOCR reads [many languages and scripts](https://github.com/JaidedAI/EasyOCR) with deep learning models on PyTorch. [Create one `Reader` and reuse it](https://www.jaided.ai/easyocr/tutorial/): it loads the models into memory once, then reads as many images as you give it. + +PaddleOCR does two jobs: [general OCR](https://www.paddleocr.ai/latest/en/version3.x/pipeline_usage/OCR.html) reads text, and document parsing [turns PDFs and document images into Markdown and JSON](https://www.paddleocr.ai/latest/en/index.html) that keep the original structure. + +Load and crop images with OpenCV, then pass the NumPy arrays on: [EasyOCR](https://www.jaided.ai/easyocr/tutorial/), [PaddleOCR](https://www.paddleocr.ai/latest/en/version3.x/pipeline_usage/OCR.html), and [Ultralytics](https://docs.ultralytics.com/modes/predict/) all take them as input. diff --git a/website/data/category_intros/configuration-files.md b/website/data/category_intros/configuration-files.md new file mode 100644 index 0000000000..e6e2921a43 --- /dev/null +++ b/website/data/category_intros/configuration-files.md @@ -0,0 +1,21 @@ +Settings that change between deployments belong in the environment, whatever Python configuration library you use. pydantic-settings reads them as typed fields. + +How to choose: + +- Typed, validated settings from environment variables and `.env` files: pydantic-settings +- An INI file your users edit, with nothing to install: configparser +- A `.env` file loaded into the environment during development: python-dotenv +- Research code that composes its config and overrides it from the command line: Hydra +- Layered settings per environment, or settings for Django and Flask: Dynaconf + +pydantic-settings loads [a settings class from environment variables or secrets files](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/). Subclass `BaseSettings` and declare each setting as a type-hinted field. [Any field you don't pass to the initializer gets its value from the environment](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#usage), or its default when the variable isn't set. To read a `.env` file too, [set `env_file` in `model_config`](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#dotenv-env-support), and its values get validated like the rest. + +configparser comes with Python. It reads [a basic configuration language structured like Windows INI files](https://docs.python.org/3/library/configparser.html), so end users can customize your program by editing a file. Read values with mapping access, `config['section']['option']`, which the docs [prefer for new projects](https://docs.python.org/3/library/configparser.html#legacy-api-examples) over the legacy get and set methods. Values always come back as strings, so convert them with [`getint()`, `getfloat()`, and `getboolean()`](https://docs.python.org/3/library/configparser.html#supported-datatypes). + +python-dotenv is for apps that [take their configuration from environment variables](https://saurabh-kumar.com/python-dotenv/#getting-started). In development, setting each one yourself isn't practical, so call `load_dotenv()` before the rest of your code. It reads a `.env` file when there is one and adds its values to `os.environ`, so your code reads them with `os.getenv()` as if they came from the real environment. + +Hydra is a framework for [research and other complex applications](https://hydra.cc/docs/1.3/intro/). It builds a hierarchical configuration by composition, and you override it from config files and the command line. Decorate your entry point with [`@hydra.main()`](https://hydra.cc/docs/1.3/intro/#basic-example) and point it at a YAML config. Then change a value with an argument like `db.user=root`. For options you switch between, like two databases, [create a config group](https://hydra.cc/docs/1.3/intro/#composition-example) with one file per option, and pick one in the config's `defaults` list. With [`--multirun`](https://hydra.cc/docs/1.3/intro/#multirun), one command runs your function once for each configuration you list. + +Dynaconf reads settings from [files in several formats, environment variables, and Vault or Redis](https://www.dynaconf.com/#features). Start with [`dynaconf init -f toml`](https://www.dynaconf.com/#using-python-only), which creates `config.py`, `settings.toml`, and `.secrets.toml`. Then import `settings` from `config` in your code. TOML is its default and most recommended format. To give development and production their own sections in one file, [set `environments=True`](https://www.dynaconf.com/#layered-environments-on-files). In a Django app, `django.conf.settings` [becomes a Dynaconf settings object](https://www.dynaconf.com/#using-django), and in Flask, `app.config` does. + +Keep settings that change between deployments out of your code. The [twelve-factor app stores config in environment variables](https://12factor.net/config), and python-dotenv and Dynaconf both cite it. python-dotenv, pydantic-settings, and Dynaconf let the real environment win over files. `load_dotenv()` [doesn't override variables already set](https://saurabh-kumar.com/python-dotenv/#getting-started), pydantic-settings [ranks environment variables above `.env` and secrets files](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#field-value-priority), and Dynaconf [prioritizes environment variables over files](https://www.dynaconf.com/envvars/). Keep secrets out of Git, too: add `.env` to your `.gitignore`, and `dynaconf init` adds `.secrets.*` there for you. diff --git a/website/data/category_intros/cryptography.md b/website/data/category_intros/cryptography.md new file mode 100644 index 0000000000..21f803b392 --- /dev/null +++ b/website/data/category_intros/cryptography.md @@ -0,0 +1,19 @@ +Don't touch a Python cryptography library's raw ciphers: encrypt with cryptography's Fernet or PyNaCl. Paramiko runs SSH, and ItsDangerous signs tokens. + +How to choose: + +- Encrypting data, X.509 certificates, and other standard formats: cryptography +- Encryption and signatures between your own apps, with algorithms picked for you: PyNaCl +- Hashing passwords for storage: PyNaCl +- SSH clients and servers, and SFTP: Paramiko +- Signed links, tokens, and cookies that users may read but not change: ItsDangerous + +cryptography has 2 layers: recipes that need few decisions, and low-level primitives it calls the "hazmat" layer. Its docs [recommend the recipes layer whenever possible](https://cryptography.io/en/stable/#layout), and hazmat only when necessary. For data you encrypt with a key, the recipe is [Fernet](https://cryptography.io/en/stable/fernet/): a message encrypted with it can't be read or changed without the key. To encrypt with a password, run it through a key derivation function first. The docs [recommend Argon2id](https://cryptography.io/en/stable/fernet/#using-passwords-with-fernet), and you keep the salt to derive the same key again. When Fernet doesn't fit, the docs point you to [authenticated encryption](https://cryptography.io/en/stable/hazmat/primitives/symmetric-encryption/) before any raw cipher, since encryption alone keeps data secret but doesn't stop tampering. The recipes also cover [X.509 certificates](https://cryptography.io/en/stable/x509/tutorial/), from signing requests to self-signed certs. + +PyNaCl is a binding to libsodium, a fork of NaCl. cryptography's FAQ explains the split: cryptography is [general purpose and interoperable with existing systems](https://cryptography.io/en/stable/faq/#how-does-cryptography-compare-to-nacl-networking-and-cryptography-library), while NaCl gives you a set of hand-selected algorithms. If you prefer NaCl's design, the FAQ recommends PyNaCl. With a shared key, encrypt through `SecretBox` or `Aead` and let PyNaCl [generate a random nonce](https://pynacl.readthedocs.io/en/latest/secret/#nacl.secret.Aead.encrypt) for each message, which its docs strongly recommend. Between 2 parties, `Box` [authenticates both sides](https://pynacl.readthedocs.io/en/latest/public/#nacl-public-box), and `SealedBox` sends a message only the recipient can decrypt, without proving who sent it. To store passwords, `nacl.pwhash.str()` [hashes with argon2id by default](https://pynacl.readthedocs.io/en/latest/password_hashing/#password-storage-and-verification), and `nacl.pwhash.verify()` checks a password against the hash. + +Paramiko implements SSHv2 in pure Python, as both client and server. For common client jobs like running remote commands or transferring files, its homepage [recommends a higher-level library built on it](https://www.paramiko.org/). Use Paramiko directly for low-level primitives or to run an SSH server in Python. Its client API [starts with `SSHClient`](https://docs.paramiko.org/en/latest/), and checking the server's host key is your job: call `load_system_host_keys()` before `connect()`. A host key it can't find is [rejected by default](https://docs.paramiko.org/en/latest/api/client.html#paramiko.client.SSHClient.set_missing_host_key_policy) with an `SSHException`. `open_sftp()` opens an SFTP session on the same connection. + +ItsDangerous signs data, so you can send it somewhere untrusted and get it back: the receiver [can see the data but can't modify it](https://itsdangerous.palletsprojects.com/en/stable/) without your key. Flask's default sessions are [cookies signed with it](https://flask.palletsprojects.com/en/stable/api/#flask.sessions.SecureCookieSessionInterface). Signing hides nothing, so when the data must stay secret, encrypt it with Fernet instead. Its docs say you'll [typically want a serializer, not a signer](https://itsdangerous.palletsprojects.com/en/stable/concepts/#serializer-vs-signer). `URLSafeSerializer` turns data into a URL-safe string, and `URLSafeTimedSerializer` rejects tokens older than the `max_age` you pass to `loads()`. Give each use its own [salt](https://itsdangerous.palletsprojects.com/en/stable/concepts/#the-salt), so an activation link's signature won't pass as an upgrade link's. + +Whichever pick you use, keep secret keys out of your code. ItsDangerous's docs say the key [shouldn't be saved in source code or committed to version control](https://itsdangerous.palletsprojects.com/en/stable/concepts/#the-secret-key), and suggest reading it from an environment variable. When you generate a key or token yourself, use `os.urandom()`, [never the `random` module](https://cryptography.io/en/stable/random-numbers/), which isn't cryptographically secure. Plan for rotation, too: cryptography's [`MultiFernet`](https://cryptography.io/en/stable/fernet/#cryptography.fernet.MultiFernet) and ItsDangerous both [take a list of keys](https://itsdangerous.palletsprojects.com/en/stable/concepts/#key-rotation), so old tokens keep working after you add a new key. diff --git a/website/data/category_intros/data-analysis.md b/website/data/category_intros/data-analysis.md new file mode 100644 index 0000000000..5d64080263 --- /dev/null +++ b/website/data/category_intros/data-analysis.md @@ -0,0 +1,17 @@ +Once your data nears the size of your RAM, switch Python data analysis libraries from pandas to Polars. Once it lives in a database, Ibis runs your code there. + +How to choose: + +- Data that fits in memory: pandas +- Handing DataFrames to libraries that expect pandas: pandas +- Large data on one machine, even bigger than your RAM: Polars +- Data already in a database or warehouse: Ibis +- Code you prototype locally, then run on a warehouse or cluster: Ibis + +pandas aims to be [the fundamental high-level building block](https://pandas.pydata.org/docs/getting_started/overview.html) for practical data analysis in Python. Its DataFrame fits time series and tables with mixed column types, like an SQL table or a spreadsheet. It's built on NumPy to work with the rest of the scientific Python stack. In production code, the docs [recommend `.loc` and `.iloc` over `[]`](https://pandas.pydata.org/docs/user_guide/indexing.html) to select data. Set values in [one `.loc` call](https://pandas.pydata.org/docs/user_guide/copy_on_write.html#chained-assignment), not chained indexing like `df["foo"][mask] = 100`. pandas keeps everything in memory, so a dataset that takes a sizable share of your RAM [gets unwieldy](https://pandas.pydata.org/docs/user_guide/scale.html), and the docs point you to other libraries. + +Polars is [built for multithreaded computing on a single machine](https://docs.pola.rs/user-guide/misc/comparison/#pandas), and its stricter API leads to fewer schema bugs. It has no index: rows are known by their position, so [no index state can change what a query means](https://docs.pola.rs/user-guide/migration/pandas/#polars-does-not-have-a-multi-indexindex). Make the lazy API [your default](https://docs.pola.rs/user-guide/migration/pandas/#be-lazy): start from a function like `scan_csv()` or call `.lazy()`, then `.collect()` the result. That lets the optimizer [filter rows and pick columns while reading the data](https://docs.pola.rs/user-guide/concepts/lazy-api/). Stay eager for exploratory work, when [you don't know yet what your query will look like](https://docs.pola.rs/user-guide/concepts/lazy-api/#when-to-use-which). Write expressions inside `select`, `with_columns`, `filter`, and `group_by`, since Polars code that [looks like pandas code](https://docs.pola.rs/user-guide/migration/pandas/#key-syntax-differences) likely runs slower than it should. For data that doesn't fit in memory, run the query on the [streaming engine](https://docs.pola.rs/user-guide/concepts/streaming/), which works through it in batches. + +Ibis [compiles one Python dataframe API](https://ibis-project.org/why#how-does-ibis-work) into each backend's native language, mostly SQL, so the database or engine does the work. pandas' ecosystem page lists it for [bridging local Python and remote databases](https://pandas.pydata.org/community/ecosystem.html#ibis). Start on the default DuckDB backend, as the tutorial [recommends](https://ibis-project.org/tutorials/basics#install-ibis), then [change the connection string](https://ibis-project.org/why#scaling-up-and-out) to run the same code on PySpark, BigQuery, or Trino. Expressions are lazy: nothing runs until you call a method like `to_pandas()`, and only then does Ibis [send the compiled query](https://ibis-project.org/tutorials/coming-from/pandas) to the backend. + +You can mix all three. Polars converts a DataFrame [with `to_pandas()`](https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.to_pandas.html), Ibis returns results [as pandas or Polars DataFrames](https://ibis-project.org/reference/expression-tables#ibis.expr.types.relations.Table.to_pandas), and pandas [can use PyArrow](https://pandas.pydata.org/docs/user_guide/pyarrow.html) to trade data with Arrow-based libraries like Polars. Do the heavy work in Polars or Ibis, and hand a pandas DataFrame to the libraries that expect one. diff --git a/website/data/category_intros/data-ingestion-etl.md b/website/data/category_intros/data-ingestion-etl.md new file mode 100644 index 0000000000..2d488f5369 --- /dev/null +++ b/website/data/category_intros/data-ingestion-etl.md @@ -0,0 +1,24 @@ +dlt loads data from APIs, databases, and files into your warehouse. You pip install the Python ETL library into your own code, with no platform to run. + +How to choose: + +- Data from APIs, databases, or files, loaded into a warehouse or DuckDB: dlt +- pandas DataFrames in and out of S3, Athena, Redshift, and other AWS services: awswrangler +- Stock prices, financials, and options from Yahoo Finance, for personal research: yfinance +- Chinese stocks, futures, and funds, for academic research: AKShare +- US company filings and financial statements from SEC EDGAR: EdgarTools +- Many data providers behind one Python API, also served over REST and MCP: OpenBB + +dlt [loads data from messy sources into well-structured datasets](https://dlthub.com/docs/intro): it infers the schema and data types, normalizes the data, and handles nested structures. Start a project with [`dlt init rest_api duckdb`](https://dlthub.com/docs/tutorial/rest-api), which sets up a pipeline script with a REST API source and a DuckDB destination. Build and test on DuckDB, then [switch out the destination](https://dlthub.com/docs/reference/explainers/how-dlt-works) when you deploy. Beyond the built-in sources, any Python iterable can feed it, since [a resource is just a generator](https://github.com/dlt-hub/dlt). dlt [runs anywhere Python runs](https://dlthub.com/docs/walkthroughs/deploy-a-pipeline), so schedule it with the orchestrator you already have. + +AWS SDK for pandas [extends pandas to AWS](https://aws-sdk-pandas.readthedocs.io/en/stable/about.html): import it as `awswrangler`, and it connects DataFrames to S3, Athena, Glue, Redshift, and DynamoDB. Its [quick start](https://github.com/aws/aws-sdk-pandas) stores a DataFrame on S3 as a Parquet dataset with `wr.s3.to_parquet()`, passing a database and table name. Then `wr.athena.read_sql_query()` queries that table with SQL. + +yfinance offers [a Pythonic way to fetch financial and market data](https://github.com/ranaroussi/yfinance) from Yahoo Finance. Its [quick start](https://ranaroussi.github.io/yfinance/) uses `Ticker` for one symbol's prices, financial statements, and options, and `download` for prices across several symbols at once. + +AKShare [aims to simplify fetching financial data](https://github.com/akfamily/akshare), with one function call per dataset, like `ak.stock_zh_a_hist()` for the daily history of a Chinese stock. Its README credits mostly Chinese exchanges and finance sites as its sources. Its [docs are in Chinese](https://akshare.akfamily.xyz/introduction.html), and each data interface comes with an example you can copy and paste. + +EdgarTools [turns any SEC filing into a typed Python object](https://github.com/dgunning/edgartools), so a 10-K's revenue is one line instead of an afternoon of XBRL parsing. EDGAR requires an email with every request, so [set your identity](https://edgartools.readthedocs.io/en/latest/configuration/) with `set_identity()` before anything else. Everything starts with a `Company` or a `Filing`. `Company("AAPL").get_financials().income_statement()` returns a standardized income statement, and `.obj()` turns a filing into an object built for its form, with its data as pandas DataFrames. + +OpenBB is a ["connect once, consume everywhere" layer](https://docs.openbb.co/odp) over financial data sources. [Each provider package](https://docs.openbb.co/odp/python) connects one source and adds its own namespace to the `obb` client. The same commands run from Python, as a REST API, and as MCP tools. A call returns an OBBject, and [`to_dataframe()`](https://docs.openbb.co/odp/python/quickstart/user) turns it into a pandas DataFrame. + +Before you build on financial data, check whose terms you're under. yfinance [isn't affiliated with Yahoo](https://ranaroussi.github.io/yfinance/) and is meant for research and education, and Yahoo's API is for personal use only. AKShare's data is [for academic research](https://github.com/akfamily/akshare#statement) only. OpenBB [hosts no data itself](https://docs.openbb.co/odp/python/extensions/providers): each provider sets its own coverage and terms of use. diff --git a/website/data/category_intros/data-validation.md b/website/data/category_intros/data-validation.md new file mode 100644 index 0000000000..5302718c57 --- /dev/null +++ b/website/data/category_intros/data-validation.md @@ -0,0 +1,18 @@ +Pydantic checks API input against your type hints. Python data validation extends to dataframes in Pandera, where each column gets a type hint. + +How to choose: + +- API payloads and other records, typed with hints: Pydantic +- Dataframes in pandas, Polars, PySpark, and more: Pandera +- Data checked against a JSON Schema document: jsonschema +- Data quality checks on databases and files, with alerts and reports: Great Expectations + +Pydantic [takes its schema from Python type hints](https://pydantic.dev/docs/validation/latest/get-started/why/#type-hints), so your type checker and IDE read the same code it validates. Subclass `BaseModel` and annotate its fields, then pass untrusted data to the model: if no `ValidationError` is raised, [every field matches its declared type](https://pydantic.dev/docs/validation/latest/concepts/models/). Your models can also [generate JSON Schema](https://pydantic.dev/docs/validation/latest/get-started/why/#json-schema) for self-documenting APIs and other tools. + +Pandera [validates pandas, Polars, PySpark, and other dataframes against one schema](https://pandera.readthedocs.io/en/latest/). Write that schema as a `DataFrameModel`, [much the way you'd define a Pydantic model](https://pandera.readthedocs.io/en/latest/dataframe_models.html), with a typed field per column. Then decorate your pipeline functions with `check_types()`, which [validates both inputs and outputs](https://pandera.readthedocs.io/en/latest/decorators.html#check-inputs-and-outputs) from their type annotations. + +jsonschema [implements the JSON Schema specification](https://python-jsonschema.readthedocs.io/en/latest/), so the schema is a JSON document rather than Python code. That suits a schema that has to work outside Python too, since JSON Schema [establishes a common language for data exchange](https://json-schema.org/overview/what-is-jsonschema) across systems. Declare `$schema` in each schema to [identify which version it's written for](https://python-jsonschema.readthedocs.io/en/latest/referencing/). + +Great Expectations is [a framework for describing data with expressive tests](https://docs.greatexpectations.io/docs/core/introduction/gx_overview/#gx-core-components-and-workflows) and validating data against them, from databases to files in cloud storage. Every script [starts by creating a Data Context](https://docs.greatexpectations.io/docs/core/set_up_a_gx_environment/create_a_data_context/). Group your production Expectations into [Expectation Suites](https://docs.greatexpectations.io/docs/core/define_expectations/organize_expectation_suites/), and run those through [a Checkpoint](https://docs.greatexpectations.io/docs/core/introduction/gx_overview/#run-validations), which can send email or Slack alerts and write the results to Data Docs, a human-readable report. + +Records and tables need different validators, but the picks work together. A Pandera `DataFrameModel` [works as a field on a Pydantic model](https://pandera.readthedocs.io/en/latest/pydantic_integration.html), with Pydantic checking the built-in types next to it. Pandera's maintainer suggests [Pandera for in-memory dataframes and Great Expectations for data on disk](https://github.com/unionai-oss/pandera/discussions/598). Pandera needs zero configuration, while Great Expectations takes some setup up front. diff --git a/website/data/category_intros/data-visualization.md b/website/data/category_intros/data-visualization.md new file mode 100644 index 0000000000..263d992d4f --- /dev/null +++ b/website/data/category_intros/data-visualization.md @@ -0,0 +1,39 @@ +Charts for print come from Matplotlib, and interactive web ones from Plotly. Put either Python data visualization library in a Streamlit script for a dashboard. + +How to choose: + +- Publication figures you control down to the last detail: Matplotlib +- Interactive charts with little code: Plotly +- A dashboard or data app from a plain Python script: Streamlit +- Statistical charts straight from a pandas DataFrame: seaborn +- Charts you declare by mapping columns to x, y, and color: Vega-Altair +- Interactive browser plots with full control, or Python code behind them: Bokeh +- Diagrams of nodes and edges, drawn by Graphviz: graphviz +- Maps of large areas, in the projection you choose: Cartopy +- A knowledge graph of your codebase for your AI coding assistant: graphify +- Dashboards built from callbacks and custom layouts: Dash +- A demo UI for an ML model or any Python function: Gradio + +Matplotlib [makes easy things easy and hard things possible](https://matplotlib.org/), and it's built for publication-quality plots. Start with `fig, ax = plt.subplots()` and draw on the `Axes` it returns. The docs [suggest this object-oriented style](https://matplotlib.org/stable/users/explain/quick_start.html#coding-styles) for complicated plots and for code you'll reuse. The pyplot style, like `plt.plot()`, is handy for quick interactive work. + +Plotly draws interactive charts that show up in Jupyter or [run inside Dash apps](https://plotly.com/python/getting-started/). Start with Plotly Express, the [recommended starting point](https://plotly.com/python/plotly-express/) for most common figures. Every Plotly Express function returns a graph objects `Figure`, so you can still fine-tune it with `update_layout()` and `add_trace()`. + +Streamlit turns a Python script into a web app: add a few `st.` calls, then run it with `streamlit run`. Whenever something on the screen must update, Streamlit [reruns your entire script from top to bottom](https://docs.streamlit.io/get-started/fundamentals/main-concepts#data-flow). So wrap slow work in a cached function: [`st.cache_data`](https://docs.streamlit.io/develop/concepts/architecture/caching) for data like a loaded DataFrame or a query result, and `st.cache_resource` for ML models and database connections. + +seaborn builds on Matplotlib and [integrates closely with pandas](https://seaborn.pydata.org/tutorial/introduction.html): you name DataFrame columns, and it draws the statistical chart. Its docs [recommend figure-level functions](https://seaborn.pydata.org/tutorial/function_overview.html#relative-merits-of-figure-level-functions) like `displot()` for most plots. + +Vega-Altair is declarative: you [declare links between data columns and encoding channels](https://altair-viz.github.io/getting_started/overview.html) like the x-axis, y-axis, and color, and it handles the rest. + +Bokeh [creates JavaScript-powered visualizations without writing any JavaScript](https://docs.bokeh.org/en/latest/index.html). Build plots with `figure()` from `bokeh.plotting`, its primary interface. Add the Bokeh server when your plots need to [respond to browser events with Python code](https://docs.bokeh.org/en/latest/docs/user_guide/server/server_introduction.html), like widgets that rerun a query. + +graphviz writes graphs in the DOT language, and [renders them with Graphviz](https://graphviz.readthedocs.io/en/latest/), which you install separately. Build a `Graph` or `Digraph`, add nodes and edges, then call [`render()`](https://graphviz.readthedocs.io/en/latest/manual.html) to save the DOT source and the rendered file. + +Cartopy draws maps on Matplotlib, and its docs say it's [especially useful for large areas](https://cartopy.readthedocs.io/latest/), where flat-map assumptions break down at the poles and the dateline. [Give the axes a map projection](https://cartopy.readthedocs.io/latest/matplotlib/intro.html), like `plt.axes(projection=ccrs.PlateCarree())`, then add coastlines and your data. Your data's coordinate system is separate from that projection, so [always pass `transform=`](https://cartopy.readthedocs.io/latest/tutorials/understanding_transform.html) when you plot. + +graphify maps your project into a knowledge graph that your AI coding assistant can [query instead of grepping](https://github.com/Graphify-Labs/graphify) through files. Code is parsed locally, with no LLM, and you can click through the graph in a browser. + +Dash [ties UI elements like dropdowns, sliders, and graphs](https://github.com/plotly/dash) to your Python code, and draws charts with Plotly through `dcc.Graph`. You add interactivity with [callbacks](https://dash.plotly.com/basic-callbacks): functions Dash calls whenever an input changes, to update another component. + +Gradio [builds a demo or web app](https://www.gradio.app/guides/quickstart) for an ML model, an API, or any Python function, with no JavaScript or CSS. Start with `gr.Interface`, which takes your function plus its inputs and outputs. Use `gr.ChatInterface` for a chatbot, and `gr.Blocks` when you need your own layout and data flow. + +Pick the chart library and the way you share it separately. Plotly, Vega-Altair, and Bokeh all save interactive charts as standalone HTML files: [`write_html()`](https://plotly.com/python/interactive-html-export/), [`chart.save("chart.html")`](https://altair-viz.github.io/user_guide/saving_charts.html), and [`output_file()`](https://docs.bokeh.org/en/latest/docs/user_guide/intro.html). Streamlit has [chart elements](https://docs.streamlit.io/develop/api-reference/charts) for Matplotlib, Vega-Altair, Plotly, and Graphviz, and Gradio's [`gr.Plot`](https://www.gradio.app/docs/gradio/plot) takes Matplotlib, Plotly, Vega-Altair, and Bokeh figures. diff --git a/website/data/category_intros/database-drivers.md b/website/data/category_intros/database-drivers.md new file mode 100644 index 0000000000..380fa92b12 --- /dev/null +++ b/website/data/category_intros/database-drivers.md @@ -0,0 +1,41 @@ +Each database calls for a Python database driver built for it. Reach PostgreSQL through Psycopg, and MySQL through mysqlclient. sqlite3 comes with Python. + +How to choose: + +- PostgreSQL, from sync or async code: Psycopg +- MySQL or MariaDB, Django projects included: mysqlclient +- SQLite with nothing to install, or with its full C API: sqlite3 or APSW +- MySQL in pure Python: PyMySQL +- An asyncio-only PostgreSQL service: asyncpg +- Vector similarity search in PostgreSQL: pgvector +- Loading JSON or CSV into SQLite, from Python or the shell: sqlite-utils +- ClickHouse, even behind proxies and load balancers: clickhouse-connect +- ClickHouse over its native TCP protocol: clickhouse-driver +- Any database with an ODBC driver: pyodbc +- Oracle Database: python-oracledb +- SQL Server or Azure SQL, with no driver manager to set up: mssql-python +- Redis, MongoDB, or Apache Cassandra: redis-py, PyMongo, or cassandra-driver + +Psycopg is the PostgreSQL driver [Django recommends](https://docs.djangoproject.com/en/stable/ref/databases/#postgresql-notes). One design serves sync and async code: [`AsyncConnection` and `AsyncCursor`](https://www.psycopg.org/psycopg3/docs/advanced/async.html#asynchronous-operations) mirror the sync classes, so you mostly add `await`. Open each connection in a [`with` block](https://www.psycopg.org/psycopg3/docs/basic/usage.html#connection-context). It commits when the block ends, rolls back on an exception, and closes the connection either way. + +asyncpg is built for asyncio only, and [skips DB-API on purpose](https://magicstack.github.io/asyncpg/current/faq.html) to fit PostgreSQL better. In a server, [create a pool](https://magicstack.github.io/asyncpg/current/usage.html#connection-pools) with `asyncpg.create_pool()` at startup, then take a connection per request with `async with pool.acquire()`. + +pgvector adds the vector types and distance operators of the pgvector Postgres extension to Psycopg, asyncpg, SQLAlchemy, and Django. Enable the extension with `CREATE EXTENSION IF NOT EXISTS vector`, then [register the types](https://github.com/pgvector/pgvector-python#psycopg-3) on each connection with `register_vector()`, or in your pool's connection hook. To find nearest neighbors, order by distance, with `<->` for L2. + +Django calls mysqlclient [the recommended choice](https://docs.djangoproject.com/en/stable/ref/databases/#mysql-db-api-drivers) for MySQL, and ships its own adapter for it. It's a native driver built against MySQL's client library. Where you can't install that library and its headers, use PyMySQL, which SQLAlchemy calls [a pure Python port](https://docs.sqlalchemy.org/en/latest/dialects/mysql.html#mysql-python-compatibility) of MySQLdb that targets full compatibility. The licenses differ too: mysqlclient is [GPL](https://github.com/PyMySQL/mysqlclient/blob/main/LICENSE), and PyMySQL is MIT. + +sqlite3 comes with Python, and SQLite needs no server. Use the connection as a [context manager](https://docs.python.org/3/library/sqlite3.html#sqlite3-connection-context-manager): it commits when the block succeeds and rolls back when it raises. + +APSW wraps SQLite's complete C API. Unlike sqlite3, it [makes no effort to hide](https://rogerbinns.github.io/apsw/pysqlite.html) how SQLite differs from other databases. Its docs suggest it when you want SQLite's own features, and sqlite3 when you want DB-API and an easy switch to another database. At startup, call [`apsw.bestpractice.apply(apsw.bestpractice.recommended)`](https://rogerbinns.github.io/apsw/bestpractice.html) before opening any connection. It configures SQLite for best performance and catches common mistakes. + +sqlite-utils sits on top of sqlite3 to create and fill databases, not to be a full ORM. Pass [`insert_all()`](https://sqlite-utils.datasette.io/en/latest/python-api.html#bulk-inserts) a list of dicts, and it creates the table and columns for you. Its command-line tool pipes JSON or CSV straight into a new database file. + +clickhouse-connect is ClickHouse's official client and [talks HTTP](https://clickhouse.com/docs/integrations/language-clients/python), so it works through load balancers, proxies, and enterprise network controls. clickhouse-driver uses the [native TCP protocol](https://clickhouse-driver.readthedocs.io/en/latest/) instead, which its docs say is more configurable and sends compact binary data. + +pyodbc reaches any database through that database's ODBC driver, plus one [driver manager](https://github.com/mkleehammer/pyodbc/wiki/Drivers-and-Driver-Managers) for all of them. Windows has one built in; on macOS and Unix, you install one first. For SQL Server and Azure SQL, Microsoft's own mssql-python [connects without an external driver manager](https://learn.microsoft.com/en-us/sql/connect/python/mssql-python/python-sql-driver-mssql-python). + +python-oracledb is Oracle's official driver. In its default [Thin mode](https://python-oracledb.readthedocs.io/en/latest/user_guide/initialization.html), it connects straight to Oracle Database without Oracle Client libraries. + +redis-py is [the Python client for Redis](https://redis.io/docs/latest/develop/clients/redis-py/), and MongoDB's docs call PyMongo [the recommended way](https://www.mongodb.com/docs/languages/python/pymongo-driver/current/) to work with MongoDB from Python. cassandra-driver talks to Apache Cassandra over Cassandra's binary protocol, with sync and async APIs. + +With any SQL database, pass values as query parameters instead of formatting them into the SQL string. The sqlite3 docs say this [avoids SQL injection](https://docs.python.org/3/library/sqlite3.html#sqlite3-placeholders). For more per database, see [awesome-mysql](https://github.com/shlomi-noach/awesome-mysql), [awesome-postgres](https://github.com/dhamaniasad/awesome-postgres), and [awesome-sqlite](https://github.com/planetopendata/awesome-sqlite). diff --git a/website/data/category_intros/database.md b/website/data/category_intros/database.md new file mode 100644 index 0000000000..408271eb1f --- /dev/null +++ b/website/data/category_intros/database.md @@ -0,0 +1,22 @@ +Inside your process, a Python database library runs without a server. DuckDB handles analytical SQL there; to search vectors, start with Chroma. + +How to choose: + +- Analytical SQL over Parquet, CSV, or DataFrames: DuckDB +- Vector search in a prototype or notebook: Chroma +- ClickHouse's SQL dialect, or pandas code that runs on ClickHouse: chDB +- Vectors stored with their raw images and text, on disk: LanceDB +- Vector search embedded in an app or an edge device: Zvec +- A small JSON document store for one script: TinyDB + +Start DuckDB with `duckdb.sql()`, which runs on an [in-memory database kept inside the module](https://duckdb.org/docs/current/clients/python/overview#basic-api-usage). It queries Parquet, CSV, and JSON files by name, and [pandas DataFrames, Polars DataFrames, and Arrow tables](https://duckdb.org/docs/current/clients/python/overview#dataframes) by variable name, with no import step. When the data should stay, [pass a file name to `duckdb.connect()`](https://duckdb.org/docs/current/clients/python/overview#persistent-storage). + +chDB runs ClickHouse inside your process, so you get its SQL dialect without a ClickHouse server. If you come from pandas, [start with the DataStore API](https://clickhouse.com/docs/chdb#for-pandas-users): change the import, and your pandas code works unchanged. For SQL, use the [connection-based API](https://clickhouse.com/docs/chdb/install/python#connection-based-api), and query DataFrames with the [`Python()` table engine](https://clickhouse.com/docs/chdb/install/python#python-table-engine-recommended). + +Chroma's [in-memory client](https://docs.trychroma.com/docs/run-chroma/clients) is the quick way to try it in a notebook, and `PersistentClient` keeps the data on disk. Add text documents, and Chroma [embeds and indexes them for you](https://docs.trychroma.com/docs/overview/getting-started), so you can query with text too. + +LanceDB starts as an [embedded library pointed at a local directory](https://docs.lancedb.com/quickstart). Its indexes live on disk, and a table [stores the raw data, metadata, and embeddings together](https://docs.lancedb.com/faq/faq-oss), which suits images and other multimodal data. Its [embedding function registry](https://docs.lancedb.com/embedding) generates vectors as you insert rows, and Python code can then query with text. + +Zvec runs entirely in-process, with no server or daemon, and its docs pitch it [from prototypes to embedded apps and edge devices](https://zvec.org/en/docs/db/). Its [quickstart](https://zvec.org/en/docs/db/quickstart/) defines a schema of scalar and vector fields first, then combines vector search with filters on those scalar fields. + +TinyDB is pure Python with no dependencies, and stores Python dicts in a JSON file: [open it with `TinyDB('db.json')` and search with `Query()`](https://tinydb.readthedocs.io/en/latest/getting-started.html). Its own docs say to [pick something else](https://tinydb.readthedocs.io/en/latest/intro.html#why-not-use-tinydb) when you need access from several processes or threads, indexes, or ACID guarantees. diff --git a/website/data/category_intros/date-and-time.md b/website/data/category_intros/date-and-time.md new file mode 100644 index 0000000000..fcaf5a8c92 --- /dev/null +++ b/website/data/category_intros/date-and-time.md @@ -0,0 +1,21 @@ +Handle time zones with the built-in zoneinfo. Parsing date strings and adding months is where a Python date library earns its install: python-dateutil. + +How to choose: + +- Time zones by IANA name, like Europe/Paris: zoneinfo +- Parsing date strings, adding months, and recurring dates: python-dateutil +- Dates as people write them, like "3 days ago", in many languages: dateparser +- An easier datetime API whose objects are still datetimes: pendulum +- Exact and local times as separate types, with DST-safe math: whenever + +zoneinfo brings the IANA time zone database to Python, and the datetime docs say [its usage is recommended](https://docs.python.org/3/library/datetime.html#tzinfo-objects). Attach a ZoneInfo to a datetime [through the constructor, `replace()`, or `astimezone()`](https://docs.python.org/3/library/zoneinfo.html#using-zoneinfo). Some systems, Windows among them, have no IANA database, so if your code runs across platforms, [declare a dependency on tzdata](https://docs.python.org/3/library/zoneinfo.html#data-sources). + +python-dateutil adds [extensions to the standard datetime module](https://dateutil.readthedocs.io/en/stable/); install it as `python-dateutil` and import it as `dateutil`. Its parser [reads most known formats](https://dateutil.readthedocs.io/en/stable/parser.html) and returns a datetime even for an ambiguous date. For input like 01/05/09, set `dayfirst` or `yearfirst` to match your data. For calendar math, `relativedelta` takes [plural arguments that add and singular ones that replace](https://dateutil.readthedocs.io/en/stable/relativedelta.html): `months=+1` moves a month ahead, and `day=1` jumps to the first. `rrule` builds [recurring dates from iCalendar rules](https://dateutil.readthedocs.io/en/stable/rrule.html). + +dateparser reads relative dates like "two weeks ago" and absolute ones in more than 200 language locales. Its docs say it [stands out](https://dateparser.readthedocs.io/en/latest/#common-use-cases) for scraped pages, logs, and other data from mixed sources, and for letting users type dates in their own words. Call `dateparser.parse()`, and [pass `languages` when you know them](https://dateparser.readthedocs.io/en/latest/#how-to-use), so it skips language detection. When you parse many dates from one source, [use `DateDataParser`](https://dateparser.readthedocs.io/en/latest/usage.html), which remembers the languages it has found. + +pendulum's classes are [drop-in replacements for the native ones](https://pendulum.eustace.io/docs/#introduction), since they inherit from datetime. Every instance is time zone aware and in UTC by default. Its docs call aware datetimes [the preferred and recommended way](https://pendulum.eustace.io/docs/#instantiation) to use it. For tests, install `pendulum[test]` and [travel in time](https://pendulum.eustace.io/docs/#testing). + +whenever puts exact time and local time in [separate types](https://whenever.readthedocs.io/en/latest/guide/choosing-a-type.html): an instant when only the moment matters, a zoned datetime when the local time matters too. Mixing up naive and aware [becomes a type error](https://whenever.readthedocs.io/en/latest/), and DST is handled in all arithmetic. A standard datetime [does no time zone adjustment](https://docs.python.org/3/library/datetime.html#datetime-objects) when you add a timedelta to it. In production, [turn whenever's DST warnings into errors](https://whenever.readthedocs.io/en/latest/faq.html#why-warnings-instead-of-errors) with Python's standard warnings filter. + +Decide whether you extend datetime or replace it. zoneinfo, python-dateutil, and dateparser all use standard datetime objects, so they work together. pendulum's objects are datetimes too, but code that checks the exact type, like sqlite3 and some database drivers, [needs an adapter registered](https://pendulum.eustace.io/docs/#limitations). whenever [doesn't subclass datetime at all](https://whenever.readthedocs.io/en/latest/faq.html#why-no-drop-in-replacement-for-datetime), so [convert to and from standard datetimes](https://whenever.readthedocs.io/en/latest/guide/stdlib-convert.html) where other code needs one. diff --git a/website/data/category_intros/debugging-tools.md b/website/data/category_intros/debugging-tools.md new file mode 100644 index 0000000000..47bbd0ca17 --- /dev/null +++ b/website/data/category_intros/debugging-tools.md @@ -0,0 +1,39 @@ +Stepping through code in ipdb gets you pdb with IPython's tab completion. Python debugging tools cover slow code too: py-spy profiles a running Python program. + +How to choose: + +- pdb with tab completion and syntax highlighting: ipdb +- Finding where a running program spends its time, without changing it: py-spy +- A full-screen debugger in your terminal: PuDB +- A timeline of every function call: VizTracer +- Memory use and leaks, native extensions included: Memray +- A readable call tree of where your script spends wall-clock time: pyinstrument +- CPU, GPU, and memory use, line by line: Scalene +- SQL queries and request data on Django pages: Django Debug Toolbar +- Every Django request and its SQL queries, recorded to inspect later: Silk +- Print debugging that labels each value: IceCream +- The same toolbar on Flask pages: Flask-DebugToolbar + +ipdb gives you IPython's debugger, with [tab completion, syntax highlighting, and better tracebacks](https://github.com/gotcha/ipdb), behind the same interface as pdb. Drop into it with `import ipdb; ipdb.set_trace()`, or call `ipdb.pm()` for post-mortem debugging after an exception. + +py-spy is a sampling profiler that shows what a program spends its time on [without restarting it or modifying its code](https://github.com/benfred/py-spy). It runs outside the profiled process, which makes it safe to use on production code. `py-spy record` writes a [flame graph](https://github.com/benfred/py-spy#record) of a process ID or a command you hand it, and `py-spy dump` prints [the call stack where a program is hung](https://github.com/benfred/py-spy#dump). + +PuDB keeps the source, the stack, breakpoints, and variables [all visible at once](https://documen.tician.de/pudb/) in one full-screen terminal view. Start it from your code with `from pudb import set_trace; set_trace()`, or [run a whole script](https://documen.tician.de/pudb/starting.html#starting-the-debugger) with `python -m pudb my-script.py`. + +VizTracer records every function's entry and exit time across a whole run, which [helps catch sporadic performance issues](https://viztracer.readthedocs.io/en/latest/). Run your script [with `viztracer my_script.py`](https://viztracer.readthedocs.io/en/latest/basic_usage.html#command-line) instead of `python`, then open the `result.json` it writes with `vizviewer`. + +Memray tracks allocations [in Python code, native extension modules, and the interpreter itself](https://bloomberg.github.io/memray/overview.html), to find memory leaks and the code behind high memory use. Profiling [takes two steps](https://bloomberg.github.io/memray/getting_started.html): `memray run` saves the allocations to a file, then `memray flamegraph` builds a report from it. + +pyinstrument measures wall-clock time, so time spent [downloading data, reading files, and talking to databases](https://pyinstrument.readthedocs.io/en/latest/how-it-works.html#wall-clock-time-not-cpu-time) counts too. Type [`pyinstrument script.py`](https://pyinstrument.readthedocs.io/en/latest/guide.html#profile-a-python-script) instead of `python script.py`, or wrap a block in `with pyinstrument.profile():`. + +Scalene profiles CPU, GPU, and memory [line by line and per function](https://github.com/plasma-umass/scalene#fast-and-accurate). It [separates time in Python from time in native code](https://github.com/plasma-umass/scalene#cpu-profiling), so you can focus on code you can change. Run [`scalene run your_prog.py`](https://github.com/plasma-umass/scalene#using-scalene), then `scalene view` opens the profile in your browser. + +Django Debug Toolbar adds [panels](https://django-debug-toolbar.readthedocs.io/en/latest/panels.html) to your pages for the current request's SQL queries, templates, cache calls, and more. [Its setup](https://django-debug-toolbar.readthedocs.io/en/latest/installation.html) adds the app, its URLs, and its middleware, and shows the toolbar only to IP addresses in `INTERNAL_IPS`. + +Silk [intercepts and stores HTTP requests and database queries](https://github.com/jazzband/django-silk), then shows them in its own UI: each request's time, query count, headers, and bodies. [Its setup](https://github.com/jazzband/django-silk#installation) adds its middleware and app to your settings, includes `silk.urls` in your URLconf, and runs `migrate`. The UI then lives at `/silk/`. + +IceCream's `ic()` [prints each expression along with its value](https://github.com/gruns/icecream). With no arguments, it prints [the file, line, and function it runs in](https://github.com/gruns/icecream#inspect-execution). Since `ic()` [returns its arguments](https://github.com/gruns/icecream#return-value), you can wrap an expression in existing code without rewriting the line. + +Flask-DebugToolbar is [a port of Django Debug Toolbar](https://github.com/pallets-eco/flask-debugtoolbar) to Flask. Wrap your app in `DebugToolbarExtension(app)`, and the toolbar [shows up in HTML responses while debug mode is on](https://flask-debugtoolbar.readthedocs.io/en/latest/). + +Both debuggers also open from the built-in `breakpoint()`: set the [`PYTHONBREAKPOINT` environment variable](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONBREAKPOINT) to `ipdb.set_trace` or `pudb.set_trace`. Among profilers, sampling ones like py-spy, pyinstrument, and Scalene [cost much less overhead than tracing ones](https://pyinstrument.readthedocs.io/en/latest/how-it-works.html#statistical-profiling-not-tracing), while a tracer like VizTracer records every call and [gives you more information](https://github.com/gaogaotiantian/viztracer#performance). diff --git a/website/data/category_intros/deep-learning.md b/website/data/category_intros/deep-learning.md new file mode 100644 index 0000000000..9637db9f87 --- /dev/null +++ b/website/data/category_intros/deep-learning.md @@ -0,0 +1,23 @@ +PyTorch suits new models, since it debugs like plain Python. Gymnasium environments pass arrays to whichever Python deep learning framework trains your agent. + +How to choose: + +- Building and training neural networks: PyTorch +- RL environments, built-in or your own: Gymnasium +- A standard training loop scaled across GPUs: PyTorch Lightning +- NumPy-style code compiled for GPUs and TPUs: JAX +- One model API across JAX, TensorFlow, and PyTorch: Keras +- An existing TensorFlow codebase: TensorFlow +- Ready-made RL algorithms to train an agent: Stable Baselines3 + +PyTorch runs each line of code as it's reached, so [a debugger and stack traces read the way you'd expect](https://github.com/pytorch/pytorch#imperative-experiences). Its [quickstart](https://docs.pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html) shows the pieces of a project: a `Dataset` holds your samples and labels, and a `DataLoader` wraps it in an iterable. A model subclasses `nn.Module`, defines its layers in `__init__`, and passes data through them in `forward`. To keep a trained model, [save its `state_dict`](https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html) rather than pickling the whole model, which ties the file to your exact classes and directory layout. + +PyTorch Lightning [organizes PyTorch code](https://github.com/Lightning-AI/pytorch-lightning#why-pytorch-lightning) so you write the model logic and it handles the engineering. A `LightningModule` is still an `nn.Module`, with your training step and `configure_optimizers` added, and the `Trainer` runs the loop around it. Moving to more GPUs, to TPUs, or to 16-bit precision is [a Trainer argument](https://github.com/Lightning-AI/pytorch-lightning#advanced-features), with no change to your model code. + +JAX pairs a NumPy-like API with [composable transformations](https://github.com/jax-ml/jax#transformations): `jax.grad` takes gradients, `jax.jit` compiles, and `jax.vmap` vectorizes. They're designed for [functionally pure functions](https://docs.jax.dev/en/latest/notebooks/Common_Gotchas_in_JAX.html#pure-functions), so pass every input as an argument and return every result. The core holds little that's specific to deep learning. [For neural networks](https://docs.jaxstack.ai/en/latest/), add the JAX AI Stack: Flax for models, Optax for optimizers, and Orbax for checkpoints. + +Gymnasium, a fork of OpenAI's Gym, is [a standard API for reinforcement learning](https://github.com/Farama-Foundation/Gymnasium): one interface between learning algorithms and environments, plus reference environments that follow it. Its [basic loop](https://gymnasium.farama.org/introduction/basic_usage/) comes down to `make()`, `Env.reset()`, `Env.step()`, and `Env.render()`, and wrappers change an environment without touching its code. For your own problem, [subclass `gymnasium.Env`](https://gymnasium.farama.org/introduction/create_custom_env/) with observation and action spaces, and register it so `gymnasium.make()` builds it like a built-in one. + +Stable Baselines3 is a set of [reliable implementations of RL algorithms in PyTorch](https://github.com/DLR-RM/stable-baselines3), and it assumes you already know some RL. It follows [a scikit-learn-like syntax](https://stable-baselines3.readthedocs.io/en/master/guide/quickstart.html): create a model such as `A2C("MlpPolicy", env)`, call `learn()`, then `predict()`. It trains on any environment that [follows the Gymnasium interface](https://stable-baselines3.readthedocs.io/en/master/guide/custom_env.html). Its [RL tips](https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html) pick the algorithm by action space, since DQN handles only discrete actions and SAC only continuous ones. + +For more frameworks, papers, and courses, see [awesome-deep-learning](https://github.com/ChristosChristofidis/awesome-deep-learning). diff --git a/website/data/category_intros/devops-tools.md b/website/data/category_intros/devops-tools.md new file mode 100644 index 0000000000..bcb51f7b28 --- /dev/null +++ b/website/data/category_intros/devops-tools.md @@ -0,0 +1,50 @@ +Server setup goes in YAML playbooks that Ansible runs, unless you'd rather write Python with pyinfra. Neither Python DevOps tool needs an agent: SSH is enough. + +How to choose: + +- Server setup over SSH: Ansible in YAML, pyinfra in Python +- Cloud APIs: Boto3, Azure SDK for Python, Google Cloud client libraries, and the AWS CLI from your shell +- First-boot setup of a new cloud instance: cloud-init +- Large fleets, with an agent on every host: Salt +- Shell commands on remote servers, run from Python: Fabric +- Serverless apps on AWS Lambda: Chalice +- CPU, memory, disk, and process stats: psutil +- Errors from your app, sent to Sentry: Sentry SDK +- Processes kept running and restarted after a crash: Supervisor +- Celery workers and tasks in a browser: Flower +- Shell programs called as Python functions: sh +- Deduplicated, encrypted backups: BorgBackup + +Ansible is [agentless](https://github.com/ansible/ansible): it uses the SSH daemon your servers already run. You install it on one control node and [declare the desired state](https://docs.ansible.com/projects/ansible/latest/getting_started/introduction.html) of each machine in a playbook. When a machine already matches, Ansible changes nothing, however many times the playbook runs. + +pyinfra calls itself ["ansible but Python instead of YAML"](https://github.com/pyinfra-dev/pyinfra). Its getting-started guide puts your hosts in an `inventory.py` and your operations in a `deploy.py`, then runs `pyinfra inventory.py deploy.py`. The target hosts [need nothing but an SSH daemon](https://docs.pyinfra.com/en/latest/getting-started.html). Most operations describe an end state: pyinfra [checks the host first](https://docs.pyinfra.com/en/latest/using-operations.html) and runs commands only if reality doesn't match. + +Boto3 is the AWS SDK for Python. Its [clients](https://docs.aws.amazon.com/boto3/latest/guide/clients.html) map close to 1:1 onto AWS service APIs and support every service operation. Resources are a higher-level, object-oriented layer on top. + +The AWS CLI gives you [direct access to AWS's public APIs](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) from your terminal, to explore a service or write shell scripts. + +The Azure SDK for Python is a set of per-service packages, so you install only what you use. [Management libraries](https://learn.microsoft.com/en-us/azure/developer/python/sdk/azure-sdk-overview), named `azure-mgmt-*`, create and configure Azure resources, while client libraries work with services already provisioned. + +The Google Cloud client libraries are [the recommended way](https://docs.cloud.google.com/python/docs/reference) to call Google Cloud APIs from code. + +cloud-init handles a new instance's first boot. It [takes an initial configuration you supply](https://docs.cloud-init.io/en/latest/explanation/introduction.html) and applies it when the instance is created, from the hostname and network to user accounts and scripts. [Most clouds and Linux distributions ship it](https://github.com/canonical/cloud-init), so there's nothing to install. Write that configuration as [cloud-config](https://docs.cloud-init.io/en/latest/explanation/format/cloud-config.html), YAML that describes the state you want. + +Salt has [a master managing minions](https://docs.saltproject.io/salt/user-guide/en/latest/topics/overview.html): an agent runs on each host and listens on an event bus. One command can reach thousands of systems in seconds. + +Fabric runs shell commands on remote servers over SSH and [hands you back Python objects](https://www.fabfile.org/). A [`Connection`](https://docs.fabfile.org/en/latest/getting-started.html#run-commands-via-connections-and-run) is the core of its API: `run()` a command on it, and `put()` or `get()` files. For deploys and sysadmin jobs, write `@task` functions in a `fabfile.py` and [run them with `fab -H web1,web2`](https://docs.fabfile.org/en/latest/getting-started.html#addendum-the-fab-command-line-tool). + +Chalice deploys Python apps to AWS Lambda. You write handlers with [a decorator-based API](https://github.com/aws/chalice) for API Gateway, S3, SQS, and other AWS services. Then `chalice deploy` provisions what your app needs, including its IAM policy. + +psutil reads process and system information, like CPU, memory, disks, and network, across platforms. It covers [what tools like `ps`, `top`, and `netstat` show](https://github.com/giampaolo/psutil). + +The Sentry SDK reports your app's errors to Sentry. [Call `sentry_sdk.init()`](https://docs.sentry.io/platforms/python/#initialize-the-sentry-sdk) with your DSN as early as possible, in your app's entry point. Its [integrations](https://docs.sentry.io/platforms/python/integrations/) for Django, Flask, Celery, and many other libraries turn on by themselves. + +Supervisor runs your programs as its subprocesses on UNIX-like systems and [can restart them after a crash](https://supervisord.org/introduction.html). Add a [`[program:x]` section](https://supervisord.org/running.html#adding-a-program) per program to its INI-style config, start `supervisord`, and control processes with `supervisorctl`. + +Flower is Celery's [recommended monitor](https://docs.celeryq.dev/en/stable/userguide/monitoring.html): a web app that shows workers and tasks in real time, and can restart worker pools or shut workers down. Start it with your app's configuration, [`celery -A tasks.app flower`](https://flower.readthedocs.io/en/latest/install.html). + +sh lets you [call any program as if it were a function](https://sh.readthedocs.io/en/latest/): `from sh import git`, then `git("status", "--short")`. It runs the real binary on your `$PATH`, and a non-zero exit code raises an exception like `sh.ErrorReturnCode_2`. + +BorgBackup is a [deduplicating backup program](https://borgbackup.readthedocs.io/en/stable/). It stores only changes, so it suits daily backups, and its authenticated encryption suits backup targets you don't fully trust. Initialize an encrypted repository, create an archive per backup, and prune old ones on a schedule. The repository is useless without its key, so the quick start says to [back up the key](https://borgbackup.readthedocs.io/en/stable/quickstart.html#repository-encryption) and test your backups. + +Keep secrets out of your code and plain files, whichever tool you pick. Let each cloud's SDK find credentials: Boto3 [searches a chain of locations](https://docs.aws.amazon.com/boto3/latest/guide/credentials.html#configuring-credentials), Azure recommends [token-based authentication through Microsoft Entra ID](https://learn.microsoft.com/en-us/azure/developer/python/sdk/authentication/overview#recommended-approach-for-app-authentication), and Google's client libraries pick up [Application Default Credentials](https://docs.cloud.google.com/docs/authentication/application-default-credentials). In config management, encrypt secrets with [Ansible Vault](https://docs.ansible.com/projects/ansible/latest/tips_tricks/ansible_tips_tricks.html#keep-vaulted-variables-safely-visible) or keep them in [Salt's pillar](https://docs.saltproject.io/en/latest/topics/best_practices.html#storing-secure-data). diff --git a/website/data/category_intros/distributed-computing.md b/website/data/category_intros/distributed-computing.md new file mode 100644 index 0000000000..10bbf4f83d --- /dev/null +++ b/website/data/category_intros/distributed-computing.md @@ -0,0 +1,22 @@ +From a laptop to a cluster, Python distributed computing comes down to the workload: joblib runs loops, Dask scales pandas, Ray scales ML, and PySpark runs SQL. + +How to choose: + +- Parallel for loops on one machine: joblib +- Scaling pandas, NumPy, or scikit-learn code: Dask +- Training, tuning, and serving ML models, or GPU jobs: Ray +- Your own Python code with stateful workers on a cluster: Ray Core +- ETL and SQL on structured data, or a JVM shop: PySpark +- MPI programs on HPC clusters and supercomputers: mpi4py + +joblib is a [package for parallel computing and disk-based caching](https://joblib.readthedocs.io/en/stable/) that leaves your code as unmodified as possible. You [write a parallel for loop](https://joblib.readthedocs.io/en/stable/user_guide/parallel.html#common-usage) as a generator expression: `Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10))`. By default, joblib runs the calls in separate worker processes. When one machine isn't enough, [switch its backend](https://joblib.readthedocs.io/en/stable/user_guide/parallel.html#setting-up-joblib-s-backend-with-parallel-config), and the same loop runs on a Dask, Ray, or Spark cluster. scikit-learn's [`n_jobs` runs on joblib](https://scikit-learn.org/stable/computing/parallelism.html) too, so it follows the backend you pick. + +Dask [scales pandas, scikit-learn, and NumPy workflows](https://docs.dask.org/en/stable/why.html) with minimal rewriting. A Dask DataFrame is [a collection of pandas dataframes](https://docs.dask.org/en/stable/) on different computers, and Dask Arrays parallelize NumPy. Dask [runs without any setup](https://docs.dask.org/en/stable/deploying.html#local-machine) on your laptop. Its `LocalCluster` follows the same interface as every other Dask cluster manager, so you swap it out when you're ready to scale up. + +Ray is a [unified framework for scaling AI and Python applications](https://docs.ray.io/en/latest/ray-overview/index.html). Its libraries each distribute one ML task: Ray Data, Train, Tune, Serve, and RLlib. Ray Core runs your own code: [decorate a function with `@ray.remote`](https://docs.ray.io/en/latest/ray-core/walkthrough.html), call it with `.remote()`, and fetch the result with `ray.get()`. For workers that keep state between calls, decorate a class the same way to get an actor. Ray runs on one machine with `ray.init()`, and on several nodes once you [deploy a Ray cluster](https://docs.ray.io/en/latest/cluster/getting-started.html). Ray Data [suits GPU workloads for deep learning inference](https://docs.ray.io/en/latest/data/comparisons.html#how-does-ray-data-compare-to-other-solutions-for-offline-inference) better than Spark does, but unlike Spark, it has no SQL interface. + +PySpark is the Python API for Apache Spark, for large-scale data processing. Dask's own comparison suggests Spark [when you prefer SQL, run mostly JVM infrastructure, or want an all-in-one solution](https://docs.dask.org/en/stable/spark.html#reasons-you-might-choose-spark). PySpark's docs [recommend DataFrames over RDDs](https://spark.apache.org/docs/latest/api/python/index.html), so Spark builds the most efficient query for you. Write it in [SQL or the DataFrame API](https://spark.apache.org/docs/latest/api/python/user_guide/sql.html), whichever you think in, and switch between the two as you go. Installing PySpark with pip is [for local use or as a client](https://spark.apache.org/docs/latest/api/python/getting_started/install.html) that connects to a cluster, not for setting up the cluster itself. The `pyspark` package also needs Java. + +mpi4py provides [Python bindings for MPI](https://mpi4py.readthedocs.io/en/stable/), the Message Passing Interface, so your Python code runs across workstations, clusters, and supercomputers. Lowercase methods like `comm.send` pass any picklable Python object, and uppercase ones like `comm.Send` [pass NumPy arrays the fast way](https://mpi4py.readthedocs.io/en/stable/tutorial.html). Run your script with `mpiexec -n 4 python -m mpi4py script.py`, so an unhandled exception [aborts the whole MPI run instead of deadlocking](https://mpi4py.readthedocs.io/en/stable/mpi4py.run.html#exceptions-and-deadlocks). mpi4py runs on an MPI implementation like MPICH or Open MPI, and in production its docs [recommend a custom-built or system-provided one](https://mpi4py.readthedocs.io/en/stable/install.html). + +Start on one machine: parallelism [brings extra complexity and overhead](https://docs.dask.org/en/stable/best-practices.html#start-small), and often you don't need it. When you do scale out, collect results at the end, since Dask's `compute()` and Ray's `ray.get()` both block until the work finishes. Call [`compute()` once](https://docs.dask.org/en/stable/best-practices.html#avoid-calling-compute-repeatedly) rather than in a loop, and [`ray.get()` as late as possible](https://docs.ray.io/en/latest/ray-core/tips-for-first-time.html#tip-1-delay-ray-get). diff --git a/website/data/category_intros/distribution.md b/website/data/category_intros/distribution.md new file mode 100644 index 0000000000..fbf1b7f893 --- /dev/null +++ b/website/data/category_intros/distribution.md @@ -0,0 +1,21 @@ +Users without Python installed can still run your app once PyInstaller takes it from Python script to exe. Pyarmor obfuscates the code you ship. + +How to choose: + +- An executable with the least setup: PyInstaller +- Obfuscated scripts, or ones that expire or run on one machine only: Pyarmor +- Your code compiled to C rather than bundled as bytecode: Nuitka +- One file to copy to your servers and laptops: pex +- A Windows installer or macOS disk image along with the executable: cx_Freeze + +PyInstaller [reads your script, finds every module and library it needs](https://pyinstaller.org/en/stable/operating-mode.html), and collects them with the Python interpreter into one folder, or optionally into one executable file. For most programs that takes one command, `pyinstaller myscript.py`, and your users don't need Python installed at all. It [bundles major packages like NumPy and the Qt bindings out of the box](https://github.com/pyinstaller/pyinstaller). The bundle holds compiled `.pyc` files instead of your source code, but those [could in principle be decompiled](https://pyinstaller.org/en/stable/operating-mode.html#hiding-the-source-code) to reveal your logic. + +Pyarmor is a command-line tool for [obfuscating Python scripts, binding them to specific machines, and setting expiration dates](https://pyarmor.readthedocs.io/en/latest/tutorial/getting-started.html#what-s-pyarmor). The obfuscated scripts stay standard `.py` files: `pyarmor gen foo.py` writes them to `dist/` with a runtime package, and you [copy all of `dist/`](https://pyarmor.readthedocs.io/en/latest/tutorial/getting-started.html#distributing-the-obfuscated-script) to the target. For an executable, Pyarmor [packs the obfuscated scripts with PyInstaller](https://pyarmor.readthedocs.io/en/latest/tutorial/obfuscation.html#packing-obfuscated-scripts). Pyarmor is commercial software: its free version [covers scripts that won't make you much money](https://pyarmor.readthedocs.io/en/latest/licenses.html), and anything else needs a paid license. + +Nuitka [translates your Python modules into a C program](https://github.com/Nuitka/Nuitka) that runs on `libpython` the way CPython does, so building takes [a C compiler](https://nuitka.net/user-documentation/user-manual.html#c-compiler). Its homepage pitches that compile step for [protecting your source code](https://nuitka.net/) and for speed. Run it as [`python -m nuitka`](https://nuitka.net/user-documentation/user-manual.html#recommended-way), so you know which interpreter compiles your program. Its default mode still needs Python on the target. For users without Python, build in standalone or onefile mode. + +pex builds `.pex` files, [self-contained executable Python virtual environments](https://docs.pex-tool.org/whatispex.html) in a single zip file, so deploying your app comes down to `scp`. Pass it your requirements as you would to pip, [`pex -r requirements.txt -o my_application.pex`](https://docs.pex-tool.org/buildingpex.html#specifying-requirements), and give it an entry point to run, or the file starts an interpreter. A plain `.pex` runs on the target's own Python; [build it as a scie](https://docs.pex-tool.org/scie.html) to get a native executable with the interpreter inside. + +cx_Freeze [normally produces a folder](https://cx-freeze.readthedocs.io/en/stable/overview.html) with your executable and the shared libraries it needs, and it can also wrap that in a simple Windows installer or a macOS disk image. Keep its options in a [`[tool.cxfreeze]` table in `pyproject.toml`](https://cx-freeze.readthedocs.io/en/stable/setup_script.html), then run `cxfreeze build`. + +Get the folder build working before you switch to a single file, since both [PyInstaller](https://pyinstaller.org/en/stable/operating-mode.html#bundling-to-one-file) and [Nuitka](https://nuitka.net/user-documentation/use-cases.html#standalone-program-distribution) say problems are much easier to debug that way. Then build on each operating system you ship for, as [PyInstaller](https://pyinstaller.org/en/stable/usage.html#supporting-multiple-operating-systems) and [cx_Freeze](https://cx-freeze.readthedocs.io/en/stable/faq.html#freezing-for-other-platforms) both tell you to. diff --git a/website/data/category_intros/documentation.md b/website/data/category_intros/documentation.md new file mode 100644 index 0000000000..c0fad33565 --- /dev/null +++ b/website/data/category_intros/documentation.md @@ -0,0 +1,21 @@ +Write only docstrings, and pdoc is all the Python documentation generator you need. Write guides too, and build the site with Sphinx or Material for MkDocs. + +How to choose: + +- API docs straight from docstrings, with no configuration: pdoc +- Handwritten docs plus an API reference, as HTML, PDF, and more: Sphinx +- A searchable Markdown site, with no HTML, CSS, or JavaScript to learn: Material for MkDocs +- Architecture diagrams as code, kept in version control: Diagrams +- A Material for MkDocs site, new or existing, on its team's own generator: Zensical + +pdoc [aims to do one thing and do it well](https://pdoc.dev/docs/pdoc.html#what-is-pdoc): API documentation that follows your module hierarchy, with no configuration. Docstrings are Markdown, and it understands Google and numpydoc styles too. Run `pdoc ./demo.py` or `pdoc my_module_name` for a [preview in your browser that reloads](https://pdoc.dev/docs/pdoc.html#quickstart) when you edit the code, then `pdoc ./demo.py -o ./docs` to export the HTML. Its output is self-contained HTML, and for substantially more complex documentation needs, [pdoc's docs recommend Sphinx](https://pdoc.dev/docs/pdoc.html#limitations). + +Sphinx [focuses on handwritten documentation](https://www.sphinx-doc.org/en/stable/usage/quickstart.html) and turns one set of source files into HTML, a PDF via LaTeX, man pages, and more. Its default markup is reStructuredText, and it can [read Markdown through MyST-Parser](https://www.sphinx-doc.org/en/stable/usage/markdown.html). Run `sphinx-quickstart` to set up a source directory with a `conf.py`, then list your pages in the root document's toctree. `make html` builds the site, and `make latexpdf` the PDF. To document your code, [autodoc](https://www.sphinx-doc.org/en/stable/usage/extensions/autodoc.html) pulls in its docstrings, which you mix with your handwritten pages. + +Material for MkDocs is [a documentation framework on top of MkDocs](https://squidfunk.github.io/mkdocs-material/getting-started/), and `pip install mkdocs-material` installs MkDocs with it. You write Markdown and get a searchable static site, with [no HTML, CSS, or JavaScript to know](https://squidfunk.github.io/mkdocs-material/). Run `mkdocs new .`, then [set `site_name`, `site_url`, and `theme: name: material`](https://squidfunk.github.io/mkdocs-material/creating-your-site/#minimal-configuration) in `mkdocs.yml`, and `mkdocs serve` previews the site as you write. For reference docs from docstrings, try [mkdocstrings](https://squidfunk.github.io/mkdocs-material/alternatives/#sphinx) before switching to Sphinx: Material's alternatives page says it builds on MkDocs and adds Sphinx-like functionality. + +Zensical is [a static site generator from the creators of Material for MkDocs](https://zensical.org/docs/get-started/), with the same batteries-included approach. After `pip install zensical`, [`zensical new .`](https://zensical.org/docs/create-your-site/) creates a `docs/` folder, a `zensical.toml` config, and a GitHub Actions workflow. `zensical serve` previews as you write, and `zensical build` writes the static site. It also [builds existing MkDocs projects without changes](https://zensical.org/docs/compatibility/mkdocs/) from their `mkdocs.yml`, and its classic theme variant keeps the Material for MkDocs look. + +Diagrams [draws cloud system architecture in Python code](https://diagrams.mingrammer.com/), so you can track changes to a diagram in version control. It renders with Graphviz, so [install Graphviz first](https://diagrams.mingrammer.com/docs/getting-started/installation), then `pip install diagrams`. Describe the system in a `with Diagram("Web Service", show=False):` block and chain nodes with `>>`. Running `python diagram.py` saves it as a PNG in your working directory. + +Install [Sphinx](https://www.sphinx-doc.org/en/stable/usage/installation.html), [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/getting-started/#with-pip), or [Zensical](https://zensical.org/docs/get-started/#install-with-pip) into your project's virtual environment, as each one's docs recommend. The documentation generators here all write static HTML, so publish it from CI. All four document a GitHub Actions workflow that deploys to GitHub Pages: [Sphinx](https://www.sphinx-doc.org/en/stable/tutorial/deploying.html#publishing-your-html-documentation), [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/publishing-your-site/), [Zensical](https://zensical.org/docs/publish-your-site/), and [pdoc](https://pdoc.dev/docs/pdoc.html#deploying-to-github-pages). diff --git a/website/data/category_intros/email.md b/website/data/category_intros/email.md new file mode 100644 index 0000000000..f5140ea2ba --- /dev/null +++ b/website/data/category_intros/email.md @@ -0,0 +1,14 @@ +Sending from Django, you learn no new Python email library API: django-anymail runs send_mail(). Async code sends mail through aiosmtplib. + +How to choose: + +- A Django app sending through Amazon SES, Mailgun, Postmark, or another email service: django-anymail +- A Django app that tracks delivery or receives email through that service: django-anymail +- Async code talking to an SMTP server: aiosmtplib +- A script sending from Gmail, or any SMTP account, with attachments: yagmail + +django-anymail extends Django's `django.core.mail` with features email services add, through [one API that doesn't lock your code to one service](https://anymail.dev/en/latest/). [Change Django's email backend to Anymail's backend for your service](https://anymail.dev/en/latest/installation/), and [`send_mail()`, `EmailMessage`, and even `mail_admins()`](https://anymail.dev/en/latest/sending/django_email/) send through it unchanged. Send from a background task, not the view: Anymail calls that a [best practice for sending email from Django](https://anymail.dev/en/latest/tips/transient_errors/), since your views respond faster. To track delivery or receive email, connect the service's webhooks to [Anymail's Django signals](https://anymail.dev/en/latest/sending/tracking/), and [protect them with https and a shared secret](https://anymail.dev/en/latest/tips/securing_webhooks/). + +aiosmtplib is an [async version of smtplib](https://aiosmtplib.readthedocs.io/en/latest/), with similar APIs. Build each message as the standard library's `EmailMessage`, which makes headers easier to work with. Then pass it to the [`send()` coroutine](https://aiosmtplib.readthedocs.io/en/latest/usage.html), which the docs recommend for most use cases. + +yagmail aims to make sending email as simple as possible, with Gmail as its default server. Its [contents argument is guessed](https://yagmail.readthedocs.io/en/latest/usage.html#magical-contents) item by item: a path to a local file becomes an attachment, valid HTML becomes HTML, and anything else is text. For Gmail, sign in with an [app password or OAuth2](https://github.com/kootenpv/yagmail). diff --git a/website/data/category_intros/environment-management.md b/website/data/category_intros/environment-management.md new file mode 100644 index 0000000000..94a6ae1635 --- /dev/null +++ b/website/data/category_intros/environment-management.md @@ -0,0 +1,18 @@ +Since uv both downloads Python and creates virtual environments, one Python environment manager does the jobs that pyenv and virtualenv split between them. + +How to choose: + +- Python versions and virtual environments from one tool: uv +- A Python API for creating environments, or plugins: virtualenv +- Switching the `python` command between versions, and nothing more: pyenv +- One version manager for Python and other languages' tools: mise + +uv [manages Python versions](https://docs.astral.sh/uv/) as well as packages. When a command needs a Python you don't have, uv [downloads it for you](https://docs.astral.sh/uv/guides/install-python/), so you don't need Python installed to get started. A project's virtual environment lives in a `.venv` folder [inside the project, where editors can find it](https://docs.astral.sh/uv/concepts/projects/layout/#the-project-environment); keep it out of version control. Outside a project, `uv venv --python ` [creates an environment](https://docs.astral.sh/uv/pip/environments/) with that version, and downloads it if needed. + +virtualenv [creates isolated Python environments](https://virtualenv.pypa.io/en/latest/), and a subset of it ships with Python as the venv module. Its docs place it [between venv and uv](https://virtualenv.pypa.io/en/latest/explanation.html#virtualenv-vs-venv-vs-uv): faster and more featureful than venv, and still pure Python. Pick it when you need plugins, or a Python API to create environments. It uses the Python it runs under unless you [pass `-p`](https://virtualenv.pypa.io/en/latest/how-to/usage.html#select-a-python-version) with another installed version. As a command-line tool, it [belongs in an isolated environment](https://virtualenv.pypa.io/en/latest/how-to/install.html), not your system Python. + +pyenv does one job: it [lets you switch between multiple versions of Python](https://github.com/pyenv/pyenv), and leaves virtual environments [to you or its pyenv-virtualenv plugin](https://github.com/pyenv/pyenv#in-contrast-with-pythonbrew-and-pythonz-pyenv-does-not). Install it with the [automatic installer](https://github.com/pyenv/pyenv#1-automatic-installer-recommended) or, on macOS, Homebrew, then [set up your shell](https://github.com/pyenv/pyenv#b-set-up-your-shell-environment-for-pyenv) for it. [Choose the version](https://github.com/pyenv/pyenv#switch-between-python-versions) with `pyenv shell` for the session, `pyenv local` for a directory, or `pyenv global` for your user account. [Most versions are built from source](https://github.com/pyenv/pyenv#install-additional-python-versions) as you install them, so install Python's build dependencies first. pyenv [doesn't work on Windows](https://github.com/pyenv/pyenv#windows) outside the Windows Subsystem for Linux, so on Windows, use uv, which [supports Windows](https://docs.astral.sh/uv/). + +mise [installs tools such as Node.js, Python, and Go](https://mise.jdx.dev/dev-tools/) and selects their versions per project, so one tool covers Python and the rest of your stack. In a project, `mise use python@` [writes that version to a `mise.toml` file](https://mise.jdx.dev/lang/python.html#quick-start). [Commit the file](https://mise.jdx.dev/getting-started.html#share-the-project), and teammates and CI get the same tools with `mise install`. To also create and activate the project's virtual environment, add [`_.python.venv`](https://mise.jdx.dev/lang/python.html#automatic-virtualenv-activation) to the file's `[env]` section. [Activate mise in your shell](https://mise.jdx.dev/getting-started.html#activate-mise), and entering the project puts its Python on your `PATH`. + +Whichever tools you combine, name each project's Python in a `.python-version` file. `pyenv local` [writes that file](https://github.com/pyenv/pyenv#understanding-python-version-selection), and so does `uv python pin`, whose docs [recommend a plain version number](https://docs.astral.sh/uv/concepts/python-versions/#python-version-files) there so other tools can read it. mise reads the file too, once you [enable it for Python](https://mise.jdx.dev/lang/python.html#version-files). virtualenv [uses the version pyenv selected](https://virtualenv.pypa.io/en/latest/how-to/usage.html#using-version-managers-pyenv-mise-asdf) with no extra configuration. Don't install packages into the Python that came with your operating system, which [often manages its packages itself](https://docs.astral.sh/uv/pip/environments/); a virtual environment keeps yours apart. diff --git a/website/data/category_intros/erp.md b/website/data/category_intros/erp.md new file mode 100644 index 0000000000..390c2921ce --- /dev/null +++ b/website/data/category_intros/erp.md @@ -0,0 +1,19 @@ +Out of the box, Odoo's CRM, eCommerce, and accounting apps run alone or combine into a full Python ERP. Anything they lack, you add as a module. + +How to choose: + +- Trying Odoo, or customizing it without code: Odoo Online +- Writing your own modules: a source install +- Hosting your own modules in the cloud: Odoo.sh +- Free and open source: Odoo Community +- More features, with support and upgrades: Odoo Enterprise + +Everything in Odoo starts and ends with modules, and [the main user-facing ones are flagged as Apps](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/01_architecture.html#odoo-modules). Odoo Enterprise is [extra modules installed on top of Community](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/01_architecture.html#odoo-editions). [Community is free and open source under the LGPL](https://www.odoo.com/documentation/latest/administration.html#editions). Enterprise is shared source, and its license [ties running it to an Enterprise subscription](https://www.odoo.com/documentation/latest/legal/licenses.html). + +[Odoo Online](https://www.odoo.com/documentation/latest/administration/odoo_online.html) runs in your browser with nothing to install, and handles customizations that need no code. Your own modules need Odoo.sh or your own server. Odoo.sh is Odoo's official cloud platform, and it [builds your modules from a GitHub repository](https://www.odoo.com/documentation/latest/administration/odoo_sh/create_module.html). + +To develop modules, Odoo's developer docs prefer [a source install](https://www.odoo.com/documentation/latest/developer/tutorials/setup_guide.html), which runs Odoo straight from its code. Odoo's logic is [written in Python, and it stores data only in PostgreSQL](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/01_architecture.html#multitier-application). Keep your modules in a directory of their own, and [start the server with `odoo-bin`](https://www.odoo.com/documentation/latest/administration/on_premise/source.html#running-odoo), adding that directory to `--addons-path`. Then work through [Server framework 101](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101.html), which builds one module chapter by chapter. + +A module can [add new business logic or change what's already there](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/01_architecture.html#odoo-modules). To change a standard model, extend it from your own module: [model inheritance](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/12_inheritance.html#model-inheritance) adds fields and overrides methods on a model another module defines. Screens work the same way: [view inheritance](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/12_inheritance.html#view-inheritance) applies your extension views on top of the originals instead of overwriting them. + +Before you write a module, [check whether Odoo already covers the case](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/02_newapp.html). A database with custom modules [can't be upgraded until they're ready for the new version](https://www.odoo.com/documentation/latest/administration/upgrade.html), so [cut what duplicates the standard modules](https://www.odoo.com/documentation/latest/developer/howtos/upgrade_custom_db.html#step-1-stop-the-developments). diff --git a/website/data/category_intros/file-format-processing.md b/website/data/category_intros/file-format-processing.md new file mode 100644 index 0000000000..659160f2c2 --- /dev/null +++ b/website/data/category_intros/file-format-processing.md @@ -0,0 +1,57 @@ +PDFs you merge, split, or encrypt need only pypdf. Which Python file format library you need beyond that depends on whether you read a file or create it. + +How to choose: + +- Merging, splitting, cropping, or encrypting PDFs: pypdf +- ELF binaries and DWARF debug info: pyelftools +- A table you import or export as CSV, JSON, YAML, or Excel: Tablib +- Documents turned into Markdown for an LLM: MarkItDown, or Docling for complex layouts and scans +- Documents converted between Markdown, Word, LaTeX, and HTML: pypandoc +- Excel: openpyxl to edit a workbook, XlsxWriter to create one with charts and formatting +- Word documents: python-docx +- PowerPoint decks: python-pptx +- Text and images out of PDFs, or pages rendered as images: PyMuPDF, or pdfminer.six for each character's position and font +- PDFs drawn from Python code: ReportLab +- PDFs from HTML and CSS: WeasyPrint +- Markdown to HTML: markdown-it-py for CommonMark, Python-Markdown for its extensions, Mistune for speed +- Config files: tomllib for TOML, PyYAML for YAML + +pypdf is a [pure-Python PDF library](https://pypdf.readthedocs.io/en/stable/) for splitting, merging, cropping, and transforming pages, and it adds passwords too. To merge files, [`append()` each one to a `PdfWriter`](https://pypdf.readthedocs.io/en/stable/user/merging-pdfs.html) and write it out. With [no C dependency](https://pypdf.readthedocs.io/en/stable/meta/comparisons.html), it's easy to install, but it doesn't generate new PDFs. It reads text with `page.extract_text()`, while a scanned page needs [OCR software](https://pypdf.readthedocs.io/en/stable/user/extract-text.html). + +pyelftools [parses ELF files and DWARF debug info](https://github.com/eliben/pyelftools) in pure Python. Pass an open file to [`ELFFile`](https://github.com/eliben/pyelftools/blob/main/doc/user-guide.md), its main entry point, and stay on the high-level API: its guide says to use the low-level structs at your own peril. + +Tablib is a [format-agnostic tabular dataset library](https://tablib.readthedocs.io/en/stable/). `Dataset().load(fh)` [detects the format of the file](https://tablib.readthedocs.io/en/stable/tutorial.html) you load, and `export()` writes the same data as CSV, JSON, or another format. + +MarkItDown converts PDF, Word, Excel, PowerPoint, HTML, and more into Markdown [for LLMs and text analysis](https://github.com/microsoft/markitdown). Its README says it may not be the best option for high-fidelity conversions that people read. Don't pass untrusted input straight to it, and when you only read local files, [call `convert_local()` instead of `convert()`](https://github.com/microsoft/markitdown#security-considerations), which also fetches URLs. + +Docling is for documents where layout matters: it [understands page layout, reading order, and tables, and runs OCR](https://docling-project.github.io/docling/) on scanned PDFs. It converts each file into a `DoclingDocument`, which you export to Markdown. + +pypandoc wraps pandoc, which [converts between Markdown, Word, LaTeX, HTML, and many other formats](https://pandoc.org/) in both directions. It needs pandoc on your machine, and the [`pypandoc_binary` package](https://github.com/JessicaTegner/pypandoc#installation) includes it. Pass a path to `convert_file()`, or a string and its `format` to `convert_text()`. For output formats like DOCX and EPUB, [pass an `outputfile`](https://github.com/JessicaTegner/pypandoc#usage), since that's the only way to convert to them. + +openpyxl reads and writes Excel workbooks, so it's the one to use when the file already exists. Install defusedxml when you open spreadsheets from other people, since openpyxl [doesn't guard against XML attacks](https://openpyxl.readthedocs.io/en/stable/#security) by default. + +XlsxWriter [can't read or modify existing files](https://xlsxwriter.readthedocs.io/introduction.html). In return, it supports more Excel features than the alternatives, and its files are close to what Excel itself writes. + +python-docx works on `.docx` files, not the older `.doc`. [Start from a document that holds your styles](https://python-docx.readthedocs.io/en/latest/user/documents.html), headers, and footers, and open it with `Document('template.docx')`: python-docx can only use a style that [the starting document defines](https://python-docx.readthedocs.io/en/latest/user/styles-understanding.html). + +python-pptx [creates and updates PowerPoint files](https://python-pptx.readthedocs.io/en/latest/user/intro.html). A deck's look comes from its theme, slide master, and layouts, so [open a template deck](https://python-pptx.readthedocs.io/en/latest/user/presentations.html) with `Presentation('template.pptx')`. Add slides from its layouts and fill their [placeholders](https://python-pptx.readthedocs.io/en/latest/user/placeholders-understanding.html), which carry the template's formatting. + +PyMuPDF is a [high-performance library](https://pymupdf.readthedocs.io/en/latest/) for extracting, converting, and editing PDFs and other documents, built on MuPDF, and it also renders pages to images. It's [available under the AGPL or a commercial license](https://pymupdf.readthedocs.io/en/latest/about.html#license-and-copyright). For reading text and editing pages without that condition, use pypdf or pdfminer.six. + +ReportLab [creates PDFs directly from Python](https://docs.reportlab.com/reportlab/userguide/ch1_intro/), like reports generated on a web server, with charts and tables. Lay out documents with Platypus: [create a document from a `DocTemplate` class and pass a list of flowables](https://docs.reportlab.com/reportlab/userguide/ch5_platypus/), such as paragraphs and tables, to its `build()` method. + +pdfminer.six [extracts text from a PDF's source](https://github.com/pdfminer/pdfminer.six), with the exact location, font, and color of each piece, but it can't write PDFs. The simplest start is [`extract_text()`](https://pdfminersix.readthedocs.io/en/latest/tutorial/highlevel.html). + +WeasyPrint turns HTML and CSS into PDFs, like [reports, invoices, and tickets](https://doc.courtbouillon.org/weasyprint/stable/), with a layout engine built for pagination instead of a browser engine. It runs [no JavaScript](https://doc.courtbouillon.org/weasyprint/stable/going_further.html). Call `HTML(...).write_pdf()`. If users supply the HTML or CSS, its docs say you'll need [extra configuration](https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#security) against high memory use, endless renderings, and local file leaks. + +markdown-it-py [follows the CommonMark spec](https://markdown-it-py.readthedocs.io/en/latest/), and plugins add syntax or replace existing rules. + +Python-Markdown implements John Gruber's original Markdown. It's [not a CommonMark implementation](https://python-markdown.github.io/), and its docs say to look elsewhere if you want one. Pick it for its extension API and bundled [extensions](https://python-markdown.github.io/extensions/), like tables, fenced code blocks, and a table of contents. + +Mistune is a [fast parser with renderers and plugins](https://mistune.lepture.com/en/latest/), compatible with "sane CommonMark rules". + +tomllib comes with Python and parses TOML, but doesn't write it. With untrusted input, [limit its size](https://docs.python.org/3/library/tomllib.html), since a malicious TOML string can use a lot of CPU and memory. + +PyYAML parses and emits YAML. Load it with `yaml.safe_load()`, since [`yaml.load` can call any Python function](https://pyyaml.org/wiki/PyYAMLDocumentation) when the data comes from an untrusted source. + +Markdown your users write can carry HTML, so give each parser its safe setting. With markdown-it-py, use the [`js-default` preset](https://markdown-it-py.readthedocs.io/en/latest/security.html), which its docs strongly recommend for user content in web apps. Python-Markdown [doesn't sanitize its output](https://python-markdown.github.io/sanitization/), so run a sanitizer over the HTML. With Mistune, build your parser with [`mistune.create_markdown()`](https://mistune.lepture.com/en/latest/guide.html), which escapes HTML, instead of `mistune.html()`, which doesn't. diff --git a/website/data/category_intros/file-manipulation.md b/website/data/category_intros/file-manipulation.md new file mode 100644 index 0000000000..eb4cf41949 --- /dev/null +++ b/website/data/category_intros/file-manipulation.md @@ -0,0 +1,22 @@ +Without installing anything, two Python file manipulation libraries cover paths and file types by name: pathlib and mimetypes. python-magic reads the bytes. + +How to choose: + +- Building and joining paths, and listing directories: pathlib +- A MIME type from a file name or URL: mimetypes +- A file's type from its content, like the Unix `file` command: python-magic +- Rerunning or reloading code when files change: watchfiles +- Your own handler for each created, modified, or moved file: watchdog +- Telling text file types apart, like source code from config files: Magika + +If you aren't sure which pathlib class you need, the docs say [`Path` is most likely it](https://docs.python.org/3/library/pathlib.html): it makes a concrete path for the platform your code runs on. [Join paths with the `/` operator](https://docs.python.org/3/library/pathlib.html#basic-use), and list a directory with `iterdir()` or `glob()`. + +mimetypes [maps a file name's extension to a MIME type](https://docs.python.org/3/library/mimetypes.html), and a MIME type back to extensions. A type guess returns 2 values: a type for the Content-Type header, and an encoding like gzip for the Content-Encoding header. The type is [`None` for a missing or unknown suffix](https://docs.python.org/3/library/mimetypes.html#mimetypes.guess_type). + +python-magic wraps libmagic, which [identifies file types by checking their headers](https://github.com/ahupp/python-magic), the way the Unix `file` command does. It's a thin wrapper, so install libmagic too, with `apt-get install libmagic1` or `brew install libmagic`. Call `magic.from_file(path, mime=True)` for a MIME type, or `magic.from_buffer()` on bytes you already have. + +watchfiles is built for [file watching and code reload](https://watchfiles.helpmanual.io/), and its Rust core groups changes into batches instead of firing once per file. `watch()` is a generator that [yields sets of changes](https://watchfiles.helpmanual.io/api/watch/#watchfiles.watch), and `awatch()` is the async version. To restart code when files change, use [`run_process()`](https://watchfiles.helpmanual.io/api/run_process/#watchfiles.run_process) with a function or a command. From a shell, the [`watchfiles` CLI](https://watchfiles.helpmanual.io/cli/) does the same: `watchfiles --filter python 'pytest --lf' src tests`. + +watchdog gives you [an API and a shell tool](https://python-watchdog.readthedocs.io/en/latest/) for monitoring directories. Its [quickstart](https://python-watchdog.readthedocs.io/en/latest/quickstart.html) has you subclass `FileSystemEventHandler`, override methods like `on_created()` and `on_modified()`, schedule the handler on an `Observer`, and start that thread. An observer skips subdirectories unless you pass `recursive=True` to `schedule()`. + +Magika runs [a compact deep-learning model](https://securityresearch.google/magika/core-concepts/how-magika-works/) on a few chunks of a file, without loading large files entirely into memory. On its own test set, it [outperforms existing approaches, especially on textual content types](https://github.com/google/magika#highlights). Create a `Magika()` instance, then [call `identify_path()` on a file or `identify_bytes()` on bytes you already have](https://securityresearch.google/magika/cli-and-bindings/python/#instantiating-magika). Check `result.ok` before you read `result.output.label` or `result.output.mime_type`. diff --git a/website/data/category_intros/functional-programming.md b/website/data/category_intros/functional-programming.md new file mode 100644 index 0000000000..6374a00937 --- /dev/null +++ b/website/data/category_intros/functional-programming.md @@ -0,0 +1,21 @@ +Beyond functools, install more-itertools, as the itertools docs suggest. toolz is a fuller Python functional programming library, and returns adds typed errors. + +How to choose: + +- Partial application, decorators, and caching: functools +- More iterator tools, the itertools recipes included: more-itertools +- Composing functions into pipelines, currying, and dict helpers: toolz, or cytoolz for speed +- Everyday helpers for collections, decorators, retries, and debugging: funcy +- Errors and missing values as typed containers checked by mypy: returns + +functools is the standard library's module [for higher-order functions](https://docs.python.org/3/library/functools.html), functions that act on or return other functions. Python's Functional Programming HOWTO calls `partial()` [the most useful tool in the module](https://docs.python.org/3/howto/functional.html#the-functools-module): it fills in some of a function's arguments and gives you a new function. When you write a decorator, wrap its inner function with [`wraps`](https://docs.python.org/3/library/functools.html#functools.wraps), so the decorated function keeps its name and docstring. The same HOWTO finds many uses of `reduce()` [clearer as a `for` loop](https://docs.python.org/3/howto/functional.html#small-functions-and-the-lambda-expression). + +The itertools docs point you to more-itertools for [their recipes and many more](https://docs.python.org/3/library/itertools.html#itertools-recipes). It collects [building blocks beyond itertools](https://more-itertools.readthedocs.io/en/stable/), for grouping, windowing, lookahead, and more. The itertools recipes sit in its top-level package, so `from more_itertools import flatten` works. + +toolz [extends itertools and functools](https://toolz.readthedocs.io/en/latest/) with functions that are composable, pure, and lazy, and its API [follows Clojure's standard library](https://toolz.readthedocs.io/en/latest/heritage.html). Each function takes and returns only iterables, dictionaries, and functions, so they [compose to solve your own problems](https://toolz.readthedocs.io/en/latest/composition.html). [`pipe`](https://toolz.readthedocs.io/en/latest/api.html#toolz.functoolz.pipe) runs a value through a sequence of functions, like pipes in Unix. Stick with `partial` at first, and once it shows up several times in your code, [switch to the `toolz.curried` namespace](https://toolz.readthedocs.io/en/latest/curry.html#curry). toolz is a general-purpose library, and for data analytics its docs say [a library built for it](https://toolz.readthedocs.io/en/latest/streaming-analytics.html#disclaimer) may serve you better. [cytoolz](https://github.com/pytoolz/cytoolz) implements the same API in Cython, as a drop-in replacement when you need more speed. + +funcy is a collection of functional tools [focused on practicality](https://github.com/Suor/funcy), inspired by Clojure and underscore. Next to sequence tools, it has [collection functions that keep the type](https://funcy.readthedocs.io/en/stable/overview.html) of a dict or set. It also has control flow helpers, like `@retry` and `silent`, and debugging helpers, like `tap` and `log_calls`. Many of its functions take a regex, a mapping, or a set [where you'd pass a function](https://funcy.readthedocs.io/en/stable/extended_fns.html#extended-function-semantics). + +returns puts results in typed containers: [`Maybe` for None and `Result` for exceptions](https://returns.readthedocs.io/en/latest/pages/quickstart.html#why), plus `IO` for impure code and `Future` for async code. Its docs [really recommend mypy](https://returns.readthedocs.io/en/latest/pages/quickstart.html#typechecking-and-other-integrations), and typing [only works correctly with its mypy plugin](https://returns.readthedocs.io/en/latest/pages/result.html). So it fits projects that check types with mypy. Turn functions that raise into ones that return a `Result` with [`@safe`](https://returns.readthedocs.io/en/latest/pages/result.html#safe), and chain the steps with [`flow`](https://returns.readthedocs.io/en/latest/pages/pipeline.html#flow), which its docs call the recommended way to write code with returns. + +Mix functional style with the rest of your code: Python's HOWTO says functional-style programs usually [give a functional-appearing interface](https://docs.python.org/3/howto/functional.html) and use non-functional features inside. For a plain map or filter, toolz's own docs call comprehensions [more Pythonic](https://toolz.readthedocs.io/en/latest/streaming-analytics.html). Learn a core set of functions; toolz says [about a dozen covers most tasks](https://toolz.readthedocs.io/en/latest/control.html), and the right word only helps when your readers know it too. diff --git a/website/data/category_intros/game-development.md b/website/data/category_intros/game-development.md new file mode 100644 index 0000000000..5638cc3e2b --- /dev/null +++ b/website/data/category_intros/game-development.md @@ -0,0 +1,20 @@ +2D games need a game loop: pygame-ce has you write it; Arcade runs it. Ren'Py is built for visual novels, as Panda3D is for 3D in Python game development. + +How to choose: + +- A game loop you write and control yourself: pygame-ce +- A game loop, physics, and screen switching built in: Arcade +- 3D games, visualizations, and simulations: Panda3D +- Visual novels and other writing-heavy games: Ren'Py +- Windowing, OpenGL graphics, and media with no dependencies: pyglet +- Code already written for pygame: pygame-ce + +pygame-ce installs as `pygame-ce` and still imports as `pygame`. It [gives you full control of program execution](https://pyga.me/docs/): you write the loop that handles events, draws the frame, and flips the display. pygame-ce is LGPL, which [still allows closed-source and commercial games](https://github.com/pygame-community/pygame-ce); only changes to pygame-ce itself must be released under a compatible license. + +Arcade is built on pyglet, and it's [ideal for beginning programmers](https://github.com/pythonarcade/arcade) or anyone who wants 2D games without learning a complex framework. Give each screen its own `arcade.View` subclass that overrides `on_draw()` and `on_update()`, then [hand control to `arcade.run()`](https://api.arcade.academy/en/stable/tutorials/views/index.html). Draw sprites through a `SpriteList`, [the only way to draw a sprite](https://api.arcade.academy/en/stable/programming_guide/performance_tips.html), which batches them for you. For movement and collisions, start with the [built-in physics engines](https://api.arcade.academy/en/stable/api_docs/api/physics_engines.html): a simple one for top-down games, and a platformer one with gravity and moving platforms. + +Panda3D is a C++ engine scripted in Python, and [its manual says you must be a skilled programmer](https://docs.panda3d.org/latest/python/introduction/index) to use it. Subclass `ShowBase`, which opens the 3D window, and [call `run()` once, as the last line](https://docs.panda3d.org/latest/python/introduction/tutorial/starting-panda3d). A model shows up only once it's [attached to the scene graph](https://docs.panda3d.org/latest/python/programming/scene-graph/index), a tree rooted at `render`. + +To use Ren'Py, you [download it and work in its launcher](https://www.renpy.org/doc/html/quickstart.html), which creates, edits, and runs your projects. Write the story in its script language, and reach for [its screen language and Python](https://www.renpy.org/why.html) when you need custom interfaces or game logic. + +pyglet is a windowing and multimedia library [with no external dependencies](https://pyglet.readthedocs.io/en/latest/), for games and other visually rich apps. Attach handlers like `on_draw()` with `@window.event` and call `pyglet.app.run()`, since [writing your own event loop is generally not necessary](https://pyglet.readthedocs.io/en/latest/programming_guide/quickstart.html). Draw many sprites in [a `Batch`, which the docs strongly recommend](https://pyglet.readthedocs.io/en/latest/programming_guide/image.html). diff --git a/website/data/category_intros/geolocation.md b/website/data/category_intros/geolocation.md new file mode 100644 index 0000000000..3c5757c3ae --- /dev/null +++ b/website/data/category_intros/geolocation.md @@ -0,0 +1,20 @@ +Addresses become coordinates with geopy, whose Python geolocation API covers many geocoding services. GeoPandas analyzes map data, and GeoDjango serves it. + +How to choose: + +- Addresses to coordinates, and coordinates back to addresses: geopy +- The distance between two points: geopy +- Geographic data in tables and files, like shapefiles and GeoJSON: GeoPandas +- Web apps that store and query geographic data: GeoDjango +- A visitor's country or city from their IP address, in a Django project: GeoDjango +- Building, encoding, and validating GeoJSON objects by hand: geojson + +geopy is [a client for geocoding services, not a service itself](https://geopy.readthedocs.io/en/stable/#geopy-is-not-a-service), and each service has its own terms of use, quotas, and pricing. Every geocoder has a `geocode()` method that turns an address into a location, and most also have `reverse()` for the other way around. For OpenStreetMap's Nominatim, [set a `user_agent` that names your app](https://geopy.readthedocs.io/en/stable/#geopy.geocoders.Nominatim) and follow [its usage policy](https://operations.osmfoundation.org/policies/nominatim/). The policy bans heavy use, auto-complete, and systematic queries. It also asks you to cache results and show attribution, and it discourages bulk geocoding. To geocode a DataFrame, [wrap the call in `RateLimiter`](https://geopy.readthedocs.io/en/stable/#usage-with-pandas), which adds delays between requests and retries failed ones. Check first that your service allows bulk requests at all. For distances, `geopy.distance.distance` [computes the geodesic distance](https://geopy.readthedocs.io/en/stable/#module-geopy.distance) between two points. + +GeoPandas adds geometry columns to pandas, so you can [do in Python what would otherwise take a spatial database such as PostGIS](https://geopandas.org/en/stable/). Load a shapefile, GeoJSON, or GeoPackage with `read_file()`, which [detects the file type and returns a GeoDataFrame](https://geopandas.org/en/stable/getting_started/introduction.html), and write it back with `to_file()`. For a table of latitudes and longitudes, [build the points with `points_from_xy()`](https://geopandas.org/en/stable/gallery/create_geopandas_from_pandas.html) and set `crs="EPSG:4326"`. To geocode a column of addresses, GeoPandas [calls geopy for you](https://geopandas.org/en/stable/docs/user_guide/geocoding.html), so the geocoding service's terms still apply. + +GeoDjango is [a contrib module that ships with Django](https://docs.djangoproject.com/en/stable/ref/contrib/gis/tutorial/). It adds model fields for geometries, spatial queries to the ORM, and geometry editing to the admin. Its tutorial assumes you already know Django. Run it on PostGIS, which its docs recommend as [the most mature and feature-rich open source spatial database](https://docs.djangoproject.com/en/stable/ref/contrib/gis/install/#spatial-database). To load a shapefile, [`ogrinspect` writes the model and a `LayerMapping` imports the rows](https://docs.djangoproject.com/en/stable/ref/contrib/gis/tutorial/#importing-spatial-data). For IP geolocation, [GeoIP2](https://docs.djangoproject.com/en/stable/ref/contrib/gis/geoip2/) looks up a country or city in a MaxMind or DB-IP database file you download. + +geojson has [a class for every object in the GeoJSON spec](https://github.com/jazzband/geojson), and `geojson.dumps()` and `geojson.loads()` [wrap the standard json functions](https://github.com/jazzband/geojson#geojson-encodingdecoding) to encode and decode them. Check an object with [its `is_valid` property and `errors()` method](https://github.com/jazzband/geojson#validation). To encode your own classes the same way, [give them a `__geo_interface__`](https://github.com/jazzband/geojson#custom-classes), which GeoPandas' `from_features()` [also accepts](https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.from_features.html). + +Latitude and longitude are angles, so measure distance and area in a projected coordinate system, in meters or feet. In GeoPandas, you [always need one](https://geopandas.org/en/stable/getting_started/introduction.html#Projections) for those operations, so reproject with `to_crs()` first. GeoDjango calls choosing a geometry field's SRID [an important decision](https://docs.djangoproject.com/en/stable/ref/contrib/gis/model-api/#selecting-an-srid), since projected systems ease distance calculations. geopy's `distance` works on an ellipsoidal model of the earth, so it takes latitudes and longitudes as they are. diff --git a/website/data/category_intros/gui-development.md b/website/data/category_intros/gui-development.md new file mode 100644 index 0000000000..fcf1dc830e --- /dev/null +++ b/website/data/category_intros/gui-development.md @@ -0,0 +1,40 @@ +Desktop apps with room to grow start on Qt, through PySide6. A small tool needs no third-party Python GUI library, since tkinter is in the standard library. + +How to choose: + +- A desktop app that will grow, even a closed-source one: PySide6 +- A small tool, built with the standard library: tkinter +- GNOME apps, or anything else on GTK: PyGObject +- Fast, GPU-drawn tools and live plots: Dear PyGui +- Touch UIs that also run on Android and iOS: Kivy +- Native widgets on Windows, macOS, and Linux: wxPython +- Native widgets on desktop and mobile: Toga +- A Qt app under the GPL, or with a commercial license: PyQt6 +- A modern look and dark mode for tkinter apps: CustomTkinter +- Your own HTML, CSS, and JavaScript in a desktop window: pywebview +- A pure Python UI in the browser or a desktop window: NiceGUI +- One Python codebase for desktop, mobile, and web: Flet + +PySide6 is [the official Python binding for Qt](https://doc.qt.io/qtforpython-6/), under the LGPL, which [lets your app's code stay closed](https://www.qt.io/licensing/open-source-lgpl-obligations) as long as you meet its terms. Build the UI with Qt Widgets for an app that [looks native](https://doc.qt.io/qtforpython-6/faq/whatisqt.html#widgets), or with QML, which was [first made for mobile apps](https://doc.qt.io/qtforpython-6/faq/whatisqt.html#qml) and runs on desktop too. To lay out forms visually, draw them in Qt Widgets Designer, and pyside6-uic [turns the .ui files into Python](https://doc.qt.io/qtforpython-6/tutorials/basictutorial/uifiles.html). + +PyQt6 binds the same Qt, and PySide6 [aims to be API compatible with PyQt](https://doc.qt.io/qtforpython-6/considerations.html#api-changes), with some exceptions. The license is what splits them: PyQt6 is [GPL or commercial, never LGPL](https://www.riverbankcomputing.com/static/Docs/PyQt6/introduction.html#license), so with the GPL version your own code must use a compatible license. + +tkinter is part of the standard library. Use the themed `tkinter.ttk` widgets, which [adapt to each platform's native theme](https://docs.python.org/3/library/tkinter.ttk.html). Most tutorials online still teach the old API, so [the docs point you to TkDocs](https://docs.python.org/3/library/tkinter.html), which teaches the modern one. + +CustomTkinter widgets [work like normal Tkinter widgets](https://github.com/TomSchimansky/CustomTkinter) and mix with them, and they follow the system's light or dark mode. + +PyGObject is [the way to go](https://pygobject.gnome.org/) for a GNOME app or any GTK app. Start with the GNOME Developer Documentation's [beginner lessons](https://developer.gnome.org/documentation/tutorials/beginners.html). In code, subclass `Gtk.Application` and build your window in `do_activate()`, as [the getting-started example](https://pygobject.gnome.org/getting_started.html) does. + +Dear PyGui is built on Dear ImGui and [draws in immediate mode on your GPU](https://github.com/hoffstadt/DearPyGui), for [scientific, engineering, and data science apps](https://dearpygui.readthedocs.io/en/latest/) that need fast, interactive interfaces. Every main script [runs the same steps](https://dearpygui.readthedocs.io/en/latest/tutorials/first-steps.html#first-run): create the context and the viewport, set up, show the viewport, start, then destroy the context. + +Kivy was built [from scratch for multi-touch](https://kivy.org/doc/stable/philosophy.html), and the same code runs on Windows, macOS, Linux, Android, and iOS. Subclass `App` and [return your root widget from `build()`](https://kivy.org/doc/stable/guide/basic.html#create-an-application). As the UI grows, move it into the KV language, which [separates your app's logic from its interface](https://kivy.org/doc/stable/guide/lang.html#concept-behind-the-language). + +wxPython wraps the wxWidgets C++ library and, in most cases, [uses each platform's native widgets](https://wxpython.org/pages/overview/#what-is-wxpython). Its [Hello World](https://wxpython.org/pages/overview/#hello-world) subclasses `wx.Frame`, puts the widgets on a `wx.Panel`, and lays them out with a sizer. + +Toga [uses native system widgets, not themes](https://toga.beeware.org/en/latest/about/philosophy/#native-widgets-not-themes), on macOS, Windows, Linux, Android, and iOS. It's part of BeeWare, whose [tutorial](https://tutorial.beeware.org/) builds an app with Toga. + +pywebview [shows your HTML, CSS, and JavaScript in a native window](https://pywebview.flowrl.com/). It doesn't bundle a GUI toolkit or web renderer, so a frozen app [stays small](https://pywebview.flowrl.com/guide/). Point the window at a web server you already run, or call Python from JavaScript through its [JS API bridge](https://pywebview.flowrl.com/guide/architecture.html#js-api-with-internal-http-server) and serve static files from the built-in HTTP server. + +NiceGUI keeps [all UI logic in Python](https://github.com/zauberzeug/nicegui#architecture) on a FastAPI backend and handles the web details for you. The UI shows up in your browser, or in a native desktop window. + +Flet makes [web, desktop, and mobile apps](https://flet.dev/docs/) without HTML, CSS, or JavaScript, and [builds them with Flutter](https://flet.dev/docs/publish/). `flet run` [opens your app](https://github.com/flet-dev/flet) in a desktop window, and `flet run --web` in the browser. As the app gets more interactive, its docs [favor the declarative style](https://flet.dev/docs/cookbook/declarative-vs-imperative/), where the UI is derived from your app's state. diff --git a/website/data/category_intros/hardware.md b/website/data/category_intros/hardware.md new file mode 100644 index 0000000000..6b1ac3915d --- /dev/null +++ b/website/data/category_intros/hardware.md @@ -0,0 +1,19 @@ +Serial ports open with pySerial. Past those, pick a Python hardware library by the device you talk to, like Bleak for Bluetooth LE sensors. + +How to choose: + +- Serial ports on Windows, macOS, Linux, or BSD: pySerial +- Bluetooth Low Energy devices, like sensors: Bleak +- Controlling or monitoring the keyboard and mouse: pynput +- Global hotkeys: pynput +- Automated tests on real or virtual hardware, shared across a team or CI: Jumpstarter + +pySerial is imported as `serial`, and `serial.Serial` opens a port as a context manager. [Set a timeout when you open it](https://pyserial.readthedocs.io/en/latest/shortintro.html#readline), or `readline()` can block forever when no newline arrives. The docs also suggest [`serial_for_url()`](https://pyserial.readthedocs.io/en/latest/pyserial_api.html#serial.serial_for_url) over creating `Serial` directly, so the same code takes a local port, a remote one, or a `loop://` loopback for your tests. + +pynput sends input through a `Controller` and watches it through a `Listener`, for the keyboard and the mouse alike. A listener is a thread that runs your callbacks. On some platforms, notably Windows, a slow callback risks freezing input for every process, so [keep callbacks short](https://pynput.readthedocs.io/en/latest/keyboard.html#the-keyboard-listener-thread): hand events to a queue, and let another thread work through it. + +Bleak is async, so [call `asyncio.run()` once](https://bleak.readthedocs.io/en/latest/troubleshooting.html#calling-asyncio-run-more-than-once) and do all your Bluetooth work in one async main function. Scan with `BleakScanner` and connect with `BleakClient`, each [in an `async with` block](https://bleak.readthedocs.io/en/latest/api/client.html#connecting-and-disconnecting), so the device disconnects when the block exits. + +Jumpstarter puts your device behind an exporter, which manages its interfaces, and your tests reach it through a client. [Local mode](https://jumpstarter.dev/main/introduction/#local-mode) needs no Kubernetes or other infrastructure: run `jmp shell --exporter `, then pytest inside it. When teams share hardware, especially in CI, distributed mode adds a Kubernetes-based controller that leases devices out. Write tests as subclasses of [`JumpstarterTest`](https://jumpstarter.dev/main/getting-started/guides/examples/testing.html), whose `client` fixture handles the connection. + +Jumpstarter builds on the other picks: its serial driver runs on pySerial and takes the port [in pySerial's format](https://jumpstarter.dev/main/reference/package-apis/drivers/pyserial.html), and its Bluetooth LE driver runs on Bleak. diff --git a/website/data/category_intros/html-manipulation.md b/website/data/category_intros/html-manipulation.md new file mode 100644 index 0000000000..8481977768 --- /dev/null +++ b/website/data/category_intros/html-manipulation.md @@ -0,0 +1,21 @@ +Scraped HTML, however broken, parses in Beautiful Soup on lxml's parser. JustHTML packs a sanitizer into a Python HTML library and runs it by default. + +How to choose: + +- Pulling data out of scraped pages: Beautiful Soup, on lxml's parser +- Speed, XPath, or XML documents: lxml +- Sanitizing HTML your users submit: JustHTML +- XML you'd rather handle like JSON: xmltodict +- Escaping text you put into HTML: MarkupSafe + +Beautiful Soup is for [pulling data out of HTML and XML files](https://www.crummy.com/software/BeautifulSoup/bs4/doc/). It sits on a parser you pick and gives you one way to navigate, search, and change the tree. Its docs [recommend lxml as that parser](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser) for speed. Name the parser in the constructor, as in `BeautifulSoup(markup, "lxml")`. [Different parsers build different trees](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#differences-between-parsers) from the same broken page, and your script should parse it the same way on every machine. The docs also say when to skip Beautiful Soup: [work directly atop lxml](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#improving-performance) when computer time costs more than programmer time, and [parse with lxml](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#css-selectors-through-the-css-property) when CSS selectors are all you need. + +lxml is [a Pythonic binding for libxml2 and libxslt](https://lxml.de/): the speed and XML features of those C libraries, with an API mostly compatible with ElementTree's. For HTML, use [lxml.html](https://lxml.de/lxmlhtml.html), which adds HTML-specific methods to lxml's elements, and every element takes `.xpath()`. On broken pages, lxml's docs say you often need only [Beautiful Soup's encoding detection](https://lxml.de/lxmlhtml.html#really-broken-pages). Leave the rest to lxml's own parser, which is several times faster. XPath has the same injection problem as SQL. When a value comes from outside, [pass it as an XPath variable](https://lxml.de/FAQ.html#how-do-i-use-lxml-safely-as-a-web-service-endpoint) instead of formatting it into the expression. + +JustHTML [parses HTML like a browser](https://github.com/EmilStenstrom/justhtml), including broken markup. `JustHTML(html)` [sanitizes by default](https://emilstenstrom.github.io/justhtml/sanitization.html) against a strict allowlist. The output is safe in a page body, but [not automatically safe inside a ` + + {% block extra_head %}{% endblock %} + + + + + {% block header %}{% endblock %} +
+ + {% block content %}{% endblock %} +
+ + + + + + diff --git a/website/templates/category.html b/website/templates/category.html new file mode 100644 index 0000000000..4563dc4fea --- /dev/null +++ b/website/templates/category.html @@ -0,0 +1,359 @@ +{% extends "base.html" %} +{% block title %}{{ category_title }}{% endblock %} +{% block description %}{{ category_description }}{% endblock %} +{% block canonical_url %}{{ category_url }}{% endblock %} +{% block alternate_links %}{% endblock %} +{% block extra_head %} + +{% endblock %} +{% block header %} +
+ + + +
+ + +
+ {% if parent_category %} +

+ {{ parent_category.name }} +

+ {% endif %} +

{{ category.name }}

+ {% if category.description_html %} +

{{ category.description_html | safe }}

+ {% endif %} + {% if intro_html %} +
{{ intro_html | safe }}
+ {% endif %} + {% set named_groups = entry_groups | selectattr("name") | list %} + {% if (named_groups and not group_categories) or guide_html %} + + {% endif %} +
+ + {% if group_categories %} + + {% endif %} +
+
+{% endblock %} +{% block content %} +{% macro entry_rows(entry, index) %} + {# On section and subcategory pages the group heading already names the row's use case #} + {% set grouped_page = entry_groups and not group_categories %} + {% set section_path = "/categories/" ~ parent_category.slug ~ "/" if parent_category else current_path %} + + {{ index }} + + {{ entry.name }} + {% if entry.subcategories %}{{ entry.subcategories[0].name }}{% + else %}{{ category.name }}{% endif %} + + + {% if entry.downloads is not none %}{{ + "{:,}".format(entry.downloads) }}{% elif entry.source_type == + 'Stdlib' %}Stdlib{% elif + entry.bundled %}Bundled{% elif + entry.badge %}{{ entry.badge }}{% + else %}Not on PyPI{% endif %} + + + {% if entry.stars is not none %}{{ "{:,}".format(entry.stars) }}{% + elif entry.source_type %}{{ entry.source_type }}{% else %}—{% endif %} + + + {% if entry.last_commit_at %}{% else %}—{% endif %} + + + {% for subcat in entry.subcategories %} + + {{ subcat.name }} + + {% endfor %} + {% for cat in entry.categories %} + {{ cat }} + {% endfor %} + {% if entry.groups %} + {% set group_url = filter_urls[entry.groups[0]] %} + + {{ entry.groups[0] }} + + {% endif %} + {% if entry.source_type == 'Stdlib' %} + + Stdlib + + {% endif %} + + → + + {% if entry.description %} + + + +
{{ entry.description | safe }}
+ + + {% endif %} + + + +
+ {% if entry.also_see %} +
+ Also see: {% for see in entry.also_see %}{{ see.name }}{% if not loop.last %}, {% endif %}{% endfor %} +
+ {% endif %} +
+ {% if entry.owner %}{{ entry.owner }}/{% endif %}{{ entry.url | replace("https://", "") }} + {% if entry.last_commit_at %}/{% endif %} {% if entry.downloads is not none %}/{{ + "{:,}".format(entry.downloads) }} downloads/month{% endif %} +
+
+ + + +{% endmacro %} +
+
+ {% if entry_groups %} + {% set named_groups = entry_groups | selectattr("name") | list %} +

+ {% if group_categories %}Listed by section, in editorial order.{% elif named_groups %}Listed in editorial order, grouped by use case.{% else %}Listed in editorial order.{% endif %} + Click a column to re-sort the whole list. +

+ {% else %} +
+

{{ entries | length }} {{ category.name }} projects

+
+ {% endif %} +

+ Press / to search. Tap a tag to filter. Click any row for + details. +

+
+ +
+

Search and filter

+
+ + + + + +
+
+ +

Results

+
+ + + + + + + + + + + + + + {% if entry_groups %} + {% set row = namespace(index=0) %} + {% for group in entry_groups %} + {% if group.name %} + + + + {% endif %} + {% for entry in group.entries %} + {% set row.index = row.index + 1 %} + {{ entry_rows(entry, row.index) }} + {% endfor %} + {% endfor %} + {% else %} + {% for entry in entries %} + {{ entry_rows(entry, loop.index) }} + {% endfor %} + {% endif %} + +
Row number + + + + + + + + Tags + +
+

+ {% if group.url %}{{ group.name }}{% else %}{{ group.name }}{% endif %} + {{ group.entries | length }} project{{ "s" if group.entries | length != 1 }} +

+ {% if group.lead %} +
{{ group.lead | safe }}
+ {% endif %} +
+
+ + +
+ +{% if guide_html %} +
+
+

{{ category.name }} guide

+
{{ guide_html | safe }}
+
+
+{% endif %} + +
+
+ +

Know a project that belongs here?

+

Tell us what it does and why it stands out.

+ +
+
+{% endblock %} diff --git a/website/templates/index.html b/website/templates/index.html new file mode 100644 index 0000000000..4cb99d4c5a --- /dev/null +++ b/website/templates/index.html @@ -0,0 +1,331 @@ +{% extends "base.html" %} +{% block extra_head %} + +{% endblock %} +{% block header %} +
+ + + +
+ + +
+
+

The definitive list that answers "I want to do X in Python, which tool should I use?"

+

Awesome Python

+

{{ subtitle }}

+

+ {% if repo_stars %}{{ repo_stars }}+ stars on GitHub{% endif %} {% if repo_stars and build_date %}/{% endif %} {% if build_date %}Updated {{ build_date }}{% endif %} {% if repo_stars or + build_date %}/{% endif %} Maintained by + @vinta + and + @JinyangWang27 +

+ +
+
+ + +
+
+{% endblock %} +{% block content %} +{% if sponsors %} + +{% endif %} + +
+
+
+

Search every project in one place

+
+

+ Press / to search. Tap a tag to filter. Click any row for + details. +

+
+ +
+

Search and filter

+
+ + + + + +
+
+ +

Results

+
+ + + + + + + + + + + + + + {% for entry in entries %} + + + + + + + + + + {% if entry.description %} + + + + + {% endif %} + + + + + + {% endfor %} + +
Row number + + + + + + + + Tags + +
+
+ {% if entry.description %} +
{{ entry.description | safe }}
+ {% endif %} {% if entry.also_see %} +
+ Also see: {% for see in entry.also_see %}{{ see.name }}{% if not loop.last %}, {% endif %}{% endfor %} +
+ {% endif %} +
+ {% if entry.owner %}{{ entry.owner }}/{% endif %}{{ entry.url | replace("https://", "") }} + {% if entry.last_commit_at %}/{% endif %} {% if entry.downloads is not none %}/{{ + "{:,}".format(entry.downloads) }} downloads/month{% endif %} +
+
+
+
+ + +
+ +
+
+ +

Know a project that belongs here?

+

Tell us what it does and why it stands out.

+ +
+
+{% endblock %} diff --git a/website/templates/llms.txt b/website/templates/llms.txt new file mode 100644 index 0000000000..c29782d8c3 --- /dev/null +++ b/website/templates/llms.txt @@ -0,0 +1,19 @@ +# Awesome Python + +{{ subtitle }} + +{{ total_entries }} projects across {{ total_categories }} {% if total_categories == 1 %}category{% else %}categories{% endif %}, updated on {{ build_date }}. + +Scan the category index, then jump to the matching section for direct project links and short descriptions. Entries end with `PyPI downloads/month: N` (last 30 days, mirrors and CI included) and `GitHub stars: N` notes in parentheses where known; treat both as popularity context, not a quality guarantee. Use the homepage for project context outside the shortlist. + +## Primary Links + +- Homepage: {{ site_url }} +- GitHub repository: {{ github_repo_url }} +- Contributing guide: {{ contributing_url }} +- Sponsorship: {{ sponsorship_url }} +- Sitemap: {{ sitemap_url }} + +## Categories + +{{ categories_md }} diff --git a/website/templates/redirect.html b/website/templates/redirect.html new file mode 100644 index 0000000000..d62c587d6f --- /dev/null +++ b/website/templates/redirect.html @@ -0,0 +1,10 @@ + + + +Redirecting… + + + +

Redirecting…

+Click here if you are not redirected. + diff --git a/website/templates/sponsorship.html b/website/templates/sponsorship.html new file mode 100644 index 0000000000..34de3517a3 --- /dev/null +++ b/website/templates/sponsorship.html @@ -0,0 +1,297 @@ +{% extends "base.html" %} +{% block title %}Sponsor Awesome Python{% endblock %} +{% block description %}{{ sponsorship_description }}{% endblock %} +{% block canonical_url %}https://awesome-python.com/sponsorship/{% endblock %} +{% block alternate_links %}{% endblock %} +{% block extra_head %} + +{% endblock %} +{% block header %} +
+ + + +
+ + +
+

Sponsorship

+

Sponsor Awesome Python

+

+ The #10 most-starred repository on GitHub, and the list Python + developers check when choosing what to use. Your sponsorship puts your + product in front of them at the moment of decision. +

+ + {% if hero_stats %} +

+ {% for stat in hero_stats %}{{ stat }}{% if not loop.last %} + / {% endif %}{% endfor %} +

+ {% endif %} + + +
+
+
+{% endblock %} {% block content %} +
+
+
+ +
+
+

+ Professional Python developers evaluating libraries and tools for + production use. Not beginners browsing tutorials. People making adoption + decisions. +

+
+
+
Who visits
+
+ Mid to senior Python developers arriving with a specific question: a + maintained ORM, a fast HTTP client, a task queue worth running in + production. +
+
+
+
Where they come from
+
+ Google Search, GitHub, Reddit, YouTube, ChatGPT and other LLMs, + Hacker News. +
+
+
+
Why it works
+
+ Ranks on the first page of Google for "best Python libraries". + ChatGPT and other LLMs cite it when recommending Python tools. + Developers send it to each other. +
+
+
+
+
+
+ +
+
+
+ +

+ One upfront payment per term. Setup takes less than 24 hours. +

+
+
+
    +
  1. +

    Headline Sponsor

    +

    + $500 + / month +

    +

    + Logo pinned at the top of the README. Logo on the website. +

    +
      +
    • + Large logo and one-line description (max 120 characters) pinned at + the very top of the README, above all project entries. +
    • +
    • Logo link in the sponsor section of awesome-python.com.
    • +
    + Email about Headline tier +
  2. +
  3. +

    Featured Sponsor

    +

    + $200 + / month +

    +

    + Text link pinned at the top of the README. Text link on the website. +

    +
      +
    • + Text entry ([Name](URL) - Description., max 120 + characters) pinned at the top of the README, directly below + Headline sponsors. +
    • +
    • Text link in the sponsor section of awesome-python.com.
    • +
    + Email about Featured tier +
  4. +
  5. +

    Indie Dev Sponsor

    +

    + $99 + / month or $249 / quarter +

    +

    + Lower-cost text link for eligible indie developers and solo + founders. +

    +
      +
    • + Text entry ([Name](URL) - Description., max 120 + characters) in the README sponsor section, directly below Featured + sponsors. +
    • +
    • Text link in the sponsor section of awesome-python.com.
    • +
    • + Available only to founder-led, one-person-company products. Funded + startups, agencies, recruiters, larger companies, and products + with enterprise sales teams should use Featured or Headline. +
    • +
    + Email about Indie Dev tier +
  6. +
+
+
+
+ +
+
+
+ +
+
+
    +
  • + Warp + The terminal for modern developers. +
  • +
  • + pyr + Zero-config Python project manager. +
  • +
+
+
+
+ +
+
+
+ +
+
+

+ Email + sponsorship@awesome-python.com + with the four items below. +

+
+
+
Tier
+
+ Headline Sponsor ($500/mo), Featured Sponsor ($200/mo), or Indie Dev + Sponsor ($99/mo or $249/quarter). +
+
+
+
Content
+
+ Product name, URL, logo, and description (Headline tier), or + [Name](URL) - Description. entry (Featured or Indie Dev + tier). +
+
+
+
Duration
+
1, 3, 6 months, or longer.
+
+
+
Payment method
+
+ Card via + Open Collective, + US bank transfer (ACH/wire), or PayPal. +
+
+
+ +
+
+
+ +
+
+
+ +
+
+

+ Sponsorship is logo and link placement in the README header. It does not + influence which projects are listed. We curate listings on merit through + the normal + contribution process. +

+

+ We reserve the right to request changes to sponsor text, logos, or links + that are misleading, off-topic, or incompatible with the README + formatting. +

+
+
+
+{% endblock %} diff --git a/website/tests/test_build.py b/website/tests/test_build.py new file mode 100644 index 0000000000..6b32f756e9 --- /dev/null +++ b/website/tests/test_build.py @@ -0,0 +1,1618 @@ +"""Tests for the build module.""" + +import json +import os +import re +import shutil +import subprocess +import textwrap +import xml.etree.ElementTree as ET +from datetime import UTC, date, datetime +from html.parser import HTMLParser +from pathlib import Path + +import pytest + +from build import ( + TemplateEntry, + annotate_entries_with_stats, + build, + category_meta_title, + detect_source_type, + extract_entries, + extract_github_repo, + load_category_intro, + load_downloads, + load_pypi_badges, + load_stars, + sort_entries, + subcategory_path, +) +from readme_parser import parse_readme, slugify + + +class HeadMetadataParser(HTMLParser): + def __init__(self): + super().__init__() + self.title_count = 0 + self.title = "" + self.meta_by_name = {} + self.meta_by_property = {} + self.links_by_rel = {} + self._in_title = False + + def handle_starttag(self, tag, attrs): + attrs = dict(attrs) + if tag == "title": + self.title_count += 1 + self._in_title = True + elif tag == "meta": + if "name" in attrs: + self.meta_by_name[attrs["name"]] = attrs.get("content", "") + if "property" in attrs: + self.meta_by_property[attrs["property"]] = attrs.get("content", "") + elif tag == "link" and attrs.get("rel"): + for rel in attrs["rel"].split(): + self.links_by_rel[rel] = attrs.get("href", "") + + def handle_endtag(self, tag): + if tag == "title": + self._in_title = False + + def handle_data(self, data): + if self._in_title: + self.title += data + + +# --------------------------------------------------------------------------- +# slugify +# --------------------------------------------------------------------------- + + +class TestSlugify: + def test_simple(self): + assert slugify("Admin Panels") == "admin-panels" + + def test_uppercase_acronym(self): + assert slugify("RESTful API") == "restful-api" + + def test_hyphenated_input(self): + assert slugify("Command-line Tools") == "command-line-tools" + + def test_extra_spaces(self): + assert slugify(" Date and Time ") == "date-and-time" + + +class TestSubcategoryPath: + def test_builds_path(self): + assert subcategory_path("web-frameworks", "synchronous") == "/categories/web-frameworks/synchronous/" + + +# --------------------------------------------------------------------------- +# build (integration) +# --------------------------------------------------------------------------- + + +class TestBuild: + @pytest.fixture(autouse=True) + def _make_sponsorship_md(self, tmp_path): + (tmp_path / "SPONSORSHIP.md").write_text("# Sponsorship\n", encoding="utf-8") + + def _make_repo(self, tmp_path, readme): + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + tpl_dir = tmp_path / "website" / "templates" + tpl_dir.mkdir(parents=True) + (tpl_dir / "base.html").write_text( + "{% block title %}{% endblock %}" + "" + "{% block content %}{% endblock %}", + encoding="utf-8", + ) + (tpl_dir / "index.html").write_text( + '{% extends "base.html" %}{% block content %}' + "{% for entry in entries %}" + '
' + "{{ entry.name }}" + "{{ entry.categories | join(', ') }}" + "{{ entry.groups | join(', ') }}" + "
" + "{% endfor %}" + "{% endblock %}", + encoding="utf-8", + ) + (tpl_dir / "category.html").write_text( + '{% extends "base.html" %}{% block content %}

{{ category.name }}

{% for entry in entries %}{{ entry.name }}{% endfor %}{% endblock %}', + encoding="utf-8", + ) + (tpl_dir / "sponsorship.html").write_text( + '{% extends "base.html" %}{% block content %}

Sponsor

{% endblock %}', + encoding="utf-8", + ) + (tpl_dir / "llms.txt").write_text( + "# Awesome Python\n\nHomepage: {{ site_url }}\n\n## Categories\n\n{{ categories_md }}\n", + encoding="utf-8", + ) + + def _copy_real_templates(self, tmp_path): + real_tpl = Path(__file__).parent / ".." / "templates" + tpl_dir = tmp_path / "website" / "templates" + shutil.copytree(real_tpl, tpl_dir) + + def test_build_creates_homepage_and_category_pages(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ### Widgets + + _Widget libraries. Also see [awesome-widgets](https://example.com/widgets)._ + + - [w1](https://example.com) - A widget. + + ### Gadgets + + _Gadget tools._ + + - [g1](https://example.com) - A gadget. + + ## Resources + + Info. + + ### Newsletters + + - [NL](https://example.com) + + ## Contributing + + Help! + """) + self._make_repo(tmp_path, readme) + build(tmp_path) + + site = tmp_path / "website" / "output" + assert (site / "index.html").exists() + assert (site / "categories" / "widgets" / "index.html").exists() + assert (site / "categories" / "gadgets" / "index.html").exists() + + def test_build_creates_root_discovery_files(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ### Widgets + + - Sync + + - [w1](https://example.com) - A widget. + + ## Contributing + + Help! + """) + self._make_repo(tmp_path, readme) + start_date = datetime.now(UTC).date() + build(tmp_path) + end_date = datetime.now(UTC).date() + + site = tmp_path / "website" / "output" + robots = (site / "robots.txt").read_text(encoding="utf-8") + assert robots == ("User-agent: *\nContent-Signal: search=yes, ai-input=yes, ai-train=yes\nAllow: /\n\nSitemap: https://awesome-python.com/sitemap.xml\n") + + sitemap = ET.parse(site / "sitemap.xml") + root = sitemap.getroot() + ns = {"sitemap": "http://www.sitemaps.org/schemas/sitemap/0.9"} + locs = [loc.text or "" for loc in root.findall("sitemap:url/sitemap:loc", ns)] + lastmods = [lastmod.text or "" for lastmod in root.findall("sitemap:url/sitemap:lastmod", ns)] + + assert root.tag == "{http://www.sitemaps.org/schemas/sitemap/0.9}urlset" + assert locs == [ + "https://awesome-python.com/", + "https://awesome-python.com/categories/widgets/", + "https://awesome-python.com/categories/tools/", + "https://awesome-python.com/categories/widgets/sync/", + "https://awesome-python.com/sponsorship/", + ] + assert len(lastmods) == len(locs) + # Outside a git repository every page falls back to the build date + assert all(start_date <= date.fromisoformat(lastmod) <= end_date for lastmod in lastmods) + assert all(loc.startswith("https://awesome-python.com/") for loc in locs) + assert all("?" not in loc for loc in locs) + + def test_sitemap_lastmod_follows_git_history_of_each_page(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ### Widgets + + - Sync + + - [w1](https://example.com/w1) - A widget. + + ### Gadgets + + - [g1](https://example.com/g1) - A gadget. + + ## Contributing + + Help! + """) + self._make_repo(tmp_path, readme) + + def commit(day, message): + env = {**os.environ, "GIT_AUTHOR_DATE": f"{day}T12:00:00Z", "GIT_COMMITTER_DATE": f"{day}T12:00:00Z"} + subprocess.run(["git", "add", "-A"], cwd=tmp_path, check=True) + subprocess.run(["git", "-c", "user.name=t", "-c", "user.email=t@t", "commit", "-qm", message], cwd=tmp_path, env=env, check=True) + + subprocess.run(["git", "init", "-q"], cwd=tmp_path, check=True) + commit("2024-01-01", "initial") + (tmp_path / "README.md").write_text(readme.replace("A gadget.", "A better gadget."), encoding="utf-8") + commit("2024-02-01", "edit gadgets") + intros_dir = tmp_path / "website" / "data" / "category_intros" + intros_dir.mkdir(parents=True) + (intros_dir / "widgets.md").write_text("Use w1.\n", encoding="utf-8") + commit("2024-03-01", "add widgets intro") + build(tmp_path) + + sitemap = (tmp_path / "website" / "output" / "sitemap.xml").read_text(encoding="utf-8") + lastmod_by_loc = dict(re.findall(r"https://awesome-python\.com(\S*?)\s*(\S+)", sitemap)) + assert lastmod_by_loc == { + "/": "2024-02-01", + "/categories/widgets/": "2024-03-01", + "/categories/gadgets/": "2024-02-01", + "/categories/tools/": "2024-03-01", + "/categories/widgets/sync/": "2024-03-01", + "/sponsorship/": "2024-01-01", + } + + def test_build_creates_category_pages_with_metadata_and_links(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ## Widgets + + _Widget libraries. Also see [awesome-widgets](https://example.com/widgets)._ + + - [w1](https://example.com/w1) - A widget. + - [w2](https://github.com/owner/w2) - A starred widget. + + ## Gadgets + + _Gadget tools._ + + - [g1](https://example.com/g1) - A gadget. + + # Contributing + + Help! + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + + data_dir = tmp_path / "website" / "data" + data_dir.mkdir(parents=True) + stars = { + "owner/w2": { + "stars": 42, + "owner": "owner", + "last_commit_at": "2026-01-01T00:00:00+00:00", + "fetched_at": "2026-01-01T00:00:00+00:00", + }, + } + (data_dir / "github_stars.json").write_text(json.dumps(stars), encoding="utf-8") + + build(tmp_path) + + site = tmp_path / "website" / "output" + index_html = (site / "index.html").read_text(encoding="utf-8") + category_html = (site / "categories" / "widgets" / "index.html").read_text(encoding="utf-8") + parser = HeadMetadataParser() + parser.feed(category_html) + + assert 'href="/categories/widgets/"' in index_html + assert parser.title.strip() == "Python Widgets Libraries - Awesome Python" + assert parser.meta_by_name["description"] == "Widget libraries. Also see awesome-widgets. Explore 2 curated Python projects in Widgets." + assert parser.links_by_rel["canonical"] == "https://awesome-python.com/categories/widgets/" + assert parser.meta_by_property["og:url"] == "https://awesome-python.com/categories/widgets/" + assert '' not in category_html + assert 'Sponsorship' in category_html + assert "

Widgets

" in category_html + assert 'Widget libraries. Also see awesome-widgets.' in category_html + assert 'href="https://example.com/w1"' in category_html + assert "A widget." in category_html + assert 'href="https://github.com/owner/w2"' in category_html + assert '' in category_html + assert "42" in category_html + assert "2026-01-01T00:00:00+00:00" in category_html + + def test_build_links_description_anchors_to_category_pages(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ## Audio & Video + + _Media tools._ + + - [a1](https://example.com/a1) - A media tool. + + ## Widgets + + _Widget libraries. Also see [Audio & Video](#audio--video) and [Gadgets](#gadgets)._ + + - [w1](https://example.com/w1) - A widget. + + # Contributing + + Help! + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + + build(tmp_path) + + category_html = (tmp_path / "website" / "output" / "categories" / "widgets" / "index.html").read_text(encoding="utf-8") + assert 'Also see Audio & Video' in category_html + assert 'Gadgets' in category_html + + def test_build_creates_llms_text_alternate_without_sponsors(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## **Sponsors** + + - **[Sponsor](https://sponsor.example.com)**: Sponsored tool. + + > Become a sponsor: [Sponsor us](SPONSORSHIP.md). + + ## Categories + + **Tools** + + - [Widgets](#widgets) + + ## Projects + + **Tools** + + ### Widgets + + - [w1](https://example.com) - A widget. + - [w2](https://github.com/owner/w2) - A starred widget. + + ## Contributing + + Help! + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + + data_dir = tmp_path / "website" / "data" + data_dir.mkdir(parents=True) + stars = { + "owner/w2": {"stars": 42, "owner": "owner", "fetched_at": "2026-01-01T00:00:00+00:00"}, + } + (data_dir / "github_stars.json").write_text(json.dumps(stars), encoding="utf-8") + (data_dir / "pypi_downloads.tsv").write_text( + "name\tpackage\tdownloads\tfetched_at\nw1\tw1\t777\t2026-08-16\n", + encoding="utf-8", + ) + + build(tmp_path) + + site = tmp_path / "website" / "output" + index_html = (site / "index.html").read_text(encoding="utf-8") + llms_txt = (site / "llms.txt").read_text(encoding="utf-8") + + assert '' in index_html + + assert llms_txt.startswith("# Awesome Python\n\nIntro.\n") + assert "2 projects across 1 category, updated on " in llms_txt + assert "Scan the category index" in llms_txt + assert "Homepage: https://awesome-python.com/" in llms_txt + assert "Markdown homepage" not in llms_txt + assert "https://awesome-python.com/index.md" not in llms_txt + assert "GitHub repository: https://github.com/vinta/awesome-python" in llms_txt + assert "Contributing guide: https://github.com/vinta/awesome-python/blob/master/CONTRIBUTING.md" in llms_txt + assert "Sponsorship: https://awesome-python.com/sponsorship/" in llms_txt + assert "Sitemap: https://awesome-python.com/sitemap.xml" in llms_txt + assert "## Categories" in llms_txt + assert "**Tools**" in llms_txt + assert "- [Widgets](https://awesome-python.com/categories/widgets/)" in llms_txt + assert "- [Widgets](#widgets)" not in llms_txt + assert "### Widgets" in llms_txt + assert "- [w1](https://example.com) - A widget. (PyPI downloads/month: 777)" in llms_txt + assert "- [w2](https://github.com/owner/w2) - A starred widget. (GitHub stars: 42)" in llms_txt + assert llms_txt != readme + assert "# Contributing" not in llms_txt + + def test_build_cleans_stale_output(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + ## Only + + - [x](https://x.com) - X. + + # Contributing + + Done. + """) + self._make_repo(tmp_path, readme) + + stale = tmp_path / "website" / "output" / "categories" / "stale" + stale.mkdir(parents=True) + (stale / "index.html").write_text("old", encoding="utf-8") + + build(tmp_path) + + assert not (tmp_path / "website" / "output" / "categories" / "stale").exists() + + def test_build_with_stars_sorts_by_stars(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + ## Stuff + + - [low-stars](https://github.com/org/low) - Low. + - [high-stars](https://github.com/org/high) - High. + - [no-stars](https://example.com/none) - None. + + # Contributing + + Done. + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + + # Copy real templates + real_tpl = Path(__file__).parent / ".." / "templates" + tpl_dir = tmp_path / "website" / "templates" + shutil.copytree(real_tpl, tpl_dir) + + # Create mock star data + data_dir = tmp_path / "website" / "data" + data_dir.mkdir(parents=True) + stars = { + "org/high": {"stars": 5000, "owner": "org", "fetched_at": "2026-01-01T00:00:00+00:00"}, + "org/low": {"stars": 100, "owner": "org", "fetched_at": "2026-01-01T00:00:00+00:00"}, + } + (data_dir / "github_stars.json").write_text(json.dumps(stars), encoding="utf-8") + + build(tmp_path) + + html = (tmp_path / "website" / "output" / "index.html").read_text(encoding="utf-8") + # Star-sorted: high-stars (5000) before low-stars (100) before no-stars (None) + assert html.index("high-stars") < html.index("low-stars") + assert html.index("low-stars") < html.index("no-stars") + # Formatted star counts + assert "5,000" in html + assert "100" in html + # Expand content present + assert "expand-content" in html + + def test_build_with_downloads_renders_column(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + ## Stuff + + - [My-Lib](https://github.com/org/mylib) - On PyPI. + - [no-pypi](https://example.com/none) - Not on PyPI. + - [asyncio](https://docs.python.org/3/library/asyncio.html) - Stdlib. + - [my-lib.thing](https://github.com/org/mylib) - (part of My-Lib) A bundled feature. + + # Contributing + + Done. + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + + data_dir = tmp_path / "website" / "data" + data_dir.mkdir(parents=True) + # Keyed by normalized README display name, like fetch_pypi_downloads_via_clickpy.py writes it + (data_dir / "pypi_downloads.tsv").write_text( + "name\tpackage\tdownloads\tfetched_at\nasyncio\tasyncio\t26305454\t2026-08-16\nmy-lib\tmy-lib\t1234567\t2026-08-16\n", + encoding="utf-8", + ) + + build(tmp_path) + + html = (tmp_path / "website" / "output" / "index.html").read_text(encoding="utf-8") + assert "1,234,567" in html + # Stdlib entries never show PyPI counts: the asyncio row is the backport package + assert "26,305,454" not in html + # Default sort: entries with download counts come first + assert html.index("My-Lib") < html.index("no-pypi") + # Each no-download entry gets the badge matching why it has no count + assert html.count('Stdlib') == 1 + assert html.count('Bundled') == 1 + assert html.count('Not on PyPI') == 1 + + def test_build_fails_when_group_and_category_slug_collide(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Widgets** + + ## Widgets + + - [w1](https://example.com) - W. + + # Contributing + + Done. + """) + self._make_repo(tmp_path, readme) + with pytest.raises(ValueError, match="slug collision"): + build(tmp_path) + + def test_index_contains_aligned_homepage_metadata(self, tmp_path): + readme = (Path(__file__).parents[2] / "README.md").read_text(encoding="utf-8") + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + + build(tmp_path) + + parsed_groups = parse_readme(readme) + categories = [cat for group in parsed_groups for cat in group["categories"]] + entries = extract_entries(categories, parsed_groups) + html = (tmp_path / "website" / "output" / "index.html").read_text(encoding="utf-8") + parser = HeadMetadataParser() + parser.feed(html) + + expected_title = "Awesome Python" + expected_description = f"An opinionated guide to the best Python frameworks, libraries, and tools. Explore {len(entries)} curated projects across {len(categories)} categories, from AI and agents to data science and web development." + expected_url = "https://awesome-python.com/" + expected_image = "https://awesome-python.com/static/og-image.png" + + assert parser.title_count == 1 + assert parser.title.strip() == expected_title + assert parser.meta_by_name["description"] == expected_description + assert parser.links_by_rel["canonical"] == expected_url + assert parser.meta_by_property["og:type"] == "website" + assert parser.meta_by_property["og:title"] == expected_title + assert parser.meta_by_property["og:description"] == expected_description + assert parser.meta_by_property["og:image"] == expected_image + assert parser.meta_by_property["og:url"] == expected_url + assert parser.meta_by_name["twitter:card"] == "summary_large_image" + assert parser.meta_by_name["twitter:title"] == expected_title + assert parser.meta_by_name["twitter:description"] == expected_description + assert parser.meta_by_name["twitter:image"] == expected_image + assert "\n Sponsorship' in html + assert 'id="hero-category-heading">Browse by category' in html + assert 'class="hero-category-link" href="/categories/ai-and-agents/"' in html + + def test_index_contains_homepage_json_ld(self, tmp_path): + readme = (Path(__file__).parents[2] / "README.md").read_text(encoding="utf-8") + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + + build(tmp_path) + + parsed_groups = parse_readme(readme) + categories = [cat for group in parsed_groups for cat in group["categories"]] + entries = extract_entries(categories, parsed_groups) + html = (tmp_path / "website" / "output" / "index.html").read_text(encoding="utf-8") + + marker = '", start) + block = html[start:end] + assert "" not in block + data = json.loads(block) + + assert data["@context"] == "https://schema.org" + graph = {node["@type"]: node for node in data["@graph"]} + assert set(graph) == {"WebSite", "CollectionPage"} + assert graph["WebSite"]["url"] == "https://awesome-python.com/" + assert graph["WebSite"]["name"] == "Awesome Python" + assert graph["WebSite"]["@id"] == "https://awesome-python.com/#website" + + collection = graph["CollectionPage"] + assert collection["@id"] == "https://awesome-python.com/" + assert collection["url"] == "https://awesome-python.com/" + assert collection["isPartOf"] == {"@type": "WebSite", "@id": graph["WebSite"]["@id"]} + expected_description = f"An opinionated guide to the best Python frameworks, libraries, and tools. Explore {len(entries)} curated projects across {len(categories)} categories, from AI and agents to data science and web development." + assert collection["description"] == expected_description + + item_list = collection["mainEntity"] + assert item_list["@type"] == "ItemList" + assert item_list["numberOfItems"] == len(entries) + assert len(item_list["itemListElement"]) == len(entries) + + positions = [item["position"] for item in item_list["itemListElement"]] + assert positions == list(range(1, len(entries) + 1)) + assert all(item["@type"] == "ListItem" for item in item_list["itemListElement"]) + assert all(item["url"].startswith(("http://", "https://")) for item in item_list["itemListElement"]) + + rendered_names = {item["name"] for item in item_list["itemListElement"]} + rendered_urls = {item["url"] for item in item_list["itemListElement"]} + assert rendered_names == {e["name"] for e in entries} + assert rendered_urls == {e["url"] for e in entries} + + def test_category_page_contains_json_ld(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ## Widgets + + _Widget libraries._ + + - [w1](https://example.com/w1) - A widget. + - [w2](https://github.com/owner/w2) - A starred widget. + + # Contributing + + Help! + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + build(tmp_path) + + category_html = (tmp_path / "website" / "output" / "categories" / "widgets" / "index.html").read_text(encoding="utf-8") + marker = '", start) + block = category_html[start:end] + assert "" not in block + data = json.loads(block) + + assert data["@context"] == "https://schema.org" + graph = {node["@type"]: node for node in data["@graph"]} + assert set(graph) == {"WebSite", "CollectionPage", "BreadcrumbList"} + assert graph["WebSite"]["@id"] == "https://awesome-python.com/#website" + collection = graph["CollectionPage"] + assert collection["name"] == "Python Widgets Libraries" + assert collection["@id"] == "https://awesome-python.com/categories/widgets/" + assert collection["url"] == "https://awesome-python.com/categories/widgets/" + assert collection["description"] == "Widget libraries. Explore 2 curated Python projects in Widgets." + assert collection["isPartOf"] == {"@type": "WebSite", "@id": "https://awesome-python.com/#website"} + + item_list = collection["mainEntity"] + assert item_list["@type"] == "ItemList" + assert item_list["numberOfItems"] == 2 + names = {item["name"] for item in item_list["itemListElement"]} + urls = {item["url"] for item in item_list["itemListElement"]} + assert names == {"w1", "w2"} + assert urls == {"https://example.com/w1", "https://github.com/owner/w2"} + positions = sorted(item["position"] for item in item_list["itemListElement"]) + assert positions == [1, 2] + + breadcrumbs = graph["BreadcrumbList"]["itemListElement"] + assert breadcrumbs == [ + {"@type": "ListItem", "position": 1, "name": "Awesome Python", "item": "https://awesome-python.com/"}, + {"@type": "ListItem", "position": 2, "name": "Widgets", "item": "https://awesome-python.com/categories/widgets/"}, + ] + + def test_category_json_ld_follows_page_order(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ## Widgets + + _Widget libraries._ + + - [zeta](https://example.com/zeta) - Listed first. + - [alpha](https://example.com/alpha) - Listed second. + + # Contributing + + Help! + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + build(tmp_path) + + category_html = (tmp_path / "website" / "output" / "categories" / "widgets" / "index.html").read_text(encoding="utf-8") + marker = '", start)]) + collection = next(node for node in data["@graph"] if node["@type"] == "CollectionPage") + items = collection["mainEntity"]["itemListElement"] + assert [(item["position"], item["name"]) for item in items] == [(1, "zeta"), (2, "alpha")] + + def test_group_page_falls_back_to_default_description_in_json_ld(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **AI & ML** + + ## Deep Learning + + - [dl1](https://example.com/dl1) - DL. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + group_html = (tmp_path / "website" / "output" / "categories" / "ai-ml" / "index.html").read_text(encoding="utf-8") + marker = '", start) + data = json.loads(group_html[start:end]) + + graph = {node["@type"]: node for node in data["@graph"]} + collection = graph["CollectionPage"] + assert collection["name"] == "Python AI & ML Libraries" + assert collection["@id"] == "https://awesome-python.com/categories/ai-ml/" + assert collection["url"] == "https://awesome-python.com/categories/ai-ml/" + assert collection["description"] == "Explore 1 curated Python project in AI & ML. Part of the Awesome Python catalog." + + def test_category_title_skips_libraries_after_plural_noun(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Web Development** + + ## Web Frameworks + + - [wf1](https://example.com/wf1) - WF. + + ## Web APIs + + - [api1](https://example.com/api1) - API. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + categories_dir = tmp_path / "website" / "output" / "categories" + frameworks_html = (categories_dir / "web-frameworks" / "index.html").read_text(encoding="utf-8") + apis_html = (categories_dir / "web-apis" / "index.html").read_text(encoding="utf-8") + parser = HeadMetadataParser() + parser.feed(frameworks_html) + marker = '", start) + graph = {node["@type"]: node for node in json.loads(frameworks_html[start:end])["@graph"]} + + assert parser.title.strip() == "Python Web Frameworks - Awesome Python" + assert parser.meta_by_property["og:title"] == "Python Web Frameworks - Awesome Python" + assert graph["CollectionPage"]["name"] == "Python Web Frameworks" + assert "Python Web APIs Libraries - Awesome Python" in apis_html + + def test_category_title_does_not_repeat_leading_python(self): + assert category_meta_title("Python Language") == "Python Language Libraries - Awesome Python" + + def test_build_creates_subcategory_pages(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Web** + + ## Web Frameworks + + - Synchronous + + - [django](https://example.com/django) - Sync framework. + + - Asynchronous + + - [fastapi](https://example.com/fastapi) - Async framework. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + site = tmp_path / "website" / "output" + sync = (site / "categories" / "web-frameworks" / "synchronous" / "index.html").read_text(encoding="utf-8") + async_ = (site / "categories" / "web-frameworks" / "asynchronous" / "index.html").read_text(encoding="utf-8") + + assert "django" in sync + assert "fastapi" not in sync + assert "fastapi" in async_ + assert "django" not in async_ + + def test_subcategory_page_shows_breadcrumb(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Web** + + ## Web Frameworks + + - Synchronous + + - [django](https://example.com/django) - Sync. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + site = tmp_path / "website" / "output" + sync = (site / "categories" / "web-frameworks" / "synchronous" / "index.html").read_text(encoding="utf-8") + assert 'href="/categories/web-frameworks/"' in sync + assert "Web Frameworks" in sync + assert "

Synchronous

" in sync + assert "category-breadcrumb" in sync + + parser = HeadMetadataParser() + parser.feed(sync) + assert parser.title.strip() == "Python Web Frameworks: Synchronous - Awesome Python" + assert parser.meta_by_name["description"] == "The Synchronous picks in Awesome Python's Web Frameworks list: django." + + marker = '", start) + graph = {node["@type"]: node for node in json.loads(sync[start:end])["@graph"]} + assert graph["CollectionPage"]["name"] == "Python Web Frameworks: Synchronous" + assert graph["BreadcrumbList"]["itemListElement"] == [ + {"@type": "ListItem", "position": 1, "name": "Awesome Python", "item": "https://awesome-python.com/"}, + { + "@type": "ListItem", + "position": 2, + "name": "Web Frameworks", + "item": "https://awesome-python.com/categories/web-frameworks/", + }, + { + "@type": "ListItem", + "position": 3, + "name": "Synchronous", + "item": "https://awesome-python.com/categories/web-frameworks/synchronous/", + }, + ] + + parent = (site / "categories" / "web-frameworks" / "index.html").read_text(encoding="utf-8") + assert "category-breadcrumb" not in parent + + def test_sponsorship_page_contains_json_ld(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Tools** + + ## Widgets + + - [w1](https://example.com/w1) - A widget. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + site = tmp_path / "website" / "output" + html = (site / "sponsorship" / "index.html").read_text(encoding="utf-8") + parser = HeadMetadataParser() + parser.feed(html) + + assert parser.title.strip() == "Sponsor Awesome Python" + assert parser.meta_by_name["description"] == ( + "Sponsorship for awesome-python: tiers, audience, and how to get your product in front of professional Python developers evaluating tools for production use." + ) + assert parser.links_by_rel["canonical"] == "https://awesome-python.com/sponsorship/" + assert 'Sponsorship' in html + + marker = '", start) + graph = {node["@type"]: node for node in json.loads(html[start:end])["@graph"]} + + assert set(graph) == {"WebSite", "WebPage", "BreadcrumbList"} + assert graph["WebPage"]["@id"] == "https://awesome-python.com/sponsorship/" + assert graph["WebPage"]["url"] == "https://awesome-python.com/sponsorship/" + assert graph["BreadcrumbList"]["itemListElement"] == [ + {"@type": "ListItem", "position": 1, "name": "Awesome Python", "item": "https://awesome-python.com/"}, + {"@type": "ListItem", "position": 2, "name": "Sponsorship", "item": "https://awesome-python.com/sponsorship/"}, + ] + + def test_build_creates_group_pages(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **AI & ML** + + ## Deep Learning + + - [dl1](https://example.com/dl1) - DL. + + ## Machine Learning + + - [ml1](https://example.com/ml1) - ML. + + **Web Development** + + ## Web Frameworks + + - [wf1](https://example.com/wf1) - WF. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + site = tmp_path / "website" / "output" + ai_ml = (site / "categories" / "ai-ml" / "index.html").read_text(encoding="utf-8") + web_dev = (site / "categories" / "web-development" / "index.html").read_text(encoding="utf-8") + + assert "dl1" in ai_ml + assert "ml1" in ai_ml + assert "wf1" not in ai_ml + assert "wf1" in web_dev + assert "dl1" not in web_dev + + def test_tags_link_to_pages_and_subcategory_anchors(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **AI & ML** + + ## Deep Learning + + - Vision + + - [v1](https://example.com/v1) - Vision lib. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + site = tmp_path / "website" / "output" + index_html = (site / "index.html").read_text(encoding="utf-8") + + category_html = (site / "categories" / "deep-learning" / "index.html").read_text(encoding="utf-8") + + for html in (index_html, category_html): + assert 'href="/categories/deep-learning/#vision"' in html + assert 'href="/categories/ai-ml/"' in html + assert "data-url=" not in html + assert 'href="/categories/deep-learning/"' in index_html + assert 'id="vision"' in category_html + + _REDIRECT_README = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ### Widgets + + - [w1](https://example.com/w1) - A widget. + + ## Contributing + + Help! + """) + + def _write_redirects(self, tmp_path, redirects): + data_dir = tmp_path / "website" / "data" + data_dir.mkdir(parents=True) + (data_dir / "redirects.json").write_text(json.dumps(redirects), encoding="utf-8") + + def test_build_writes_redirect_stub_outside_sitemap(self, tmp_path): + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(self._REDIRECT_README, encoding="utf-8") + self._write_redirects(tmp_path, {"/categories/old-widgets/": "/categories/widgets/"}) + build(tmp_path) + + site = tmp_path / "website" / "output" + stub = (site / "categories" / "old-widgets" / "index.html").read_text(encoding="utf-8") + assert '' in stub + assert '' in stub + assert "noindex" not in stub + assert "old-widgets" not in (site / "sitemap.xml").read_text(encoding="utf-8") + + def test_build_renders_category_intro_and_uses_lead_as_meta_description(self, tmp_path): + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(self._REDIRECT_README, encoding="utf-8") + intros_dir = tmp_path / "website" / "data" / "category_intros" + intros_dir.mkdir(parents=True) + (intros_dir / "widgets.md").write_text("Use `w1` for most apps.\n\nSee [the docs](https://example.com/docs).\n\nHow to choose:\n\n- Small apps: w1\n", encoding="utf-8") + build(tmp_path) + + category_html = (tmp_path / "website" / "output" / "categories" / "widgets" / "index.html").read_text(encoding="utf-8") + parser = HeadMetadataParser() + parser.feed(category_html) + assert parser.meta_by_name["description"] == "Use w1 for most apps." + assert '

Use w1 for most apps.

' in category_html + assert '
  • Small apps: w1
  • ' in category_html + assert 'the docs' in category_html + + def test_build_renders_category_guide_below_table(self, tmp_path): + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(self._REDIRECT_README, encoding="utf-8") + intros_dir = tmp_path / "website" / "data" / "category_intros" + intros_dir.mkdir(parents=True) + (intros_dir / "widgets.md").write_text("Use w1.\n\nHow to choose:\n\n- Small apps: w1\n\nSet up w1 once per process.\n", encoding="utf-8") + build(tmp_path) + + category_html = (tmp_path / "website" / "output" / "categories" / "widgets" / "index.html").read_text(encoding="utf-8") + intro_html = category_html.split('
    ', 1)[1].split("
    ", 1)[0] + assert '
  • Small apps: w1
  • ' in intro_html + assert "Set up w1" not in intro_html + guide_html = category_html.split('
    ', 1)[1] + assert "

    Widgets guide

    " in guide_html + assert "

    Set up w1 once per process.

    " in guide_html + assert category_html.index('id="guide"') > category_html.index("
    ") + assert 'Widgets guide' in category_html + + def test_section_page_groups_rows_by_use_case_in_readme_order(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Tools** + + ### Widgets + + - Small + - [w2](https://example.com/w2) - Second. + - [w1](https://example.com/w1) - First. + - Large + - [w3](https://example.com/w3) - Third. + - [sqlite3](https://docs.python.org/3/library/sqlite3.html) - Stdlib. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + site = tmp_path / "website" / "output" / "categories" + html = (site / "widgets" / "index.html").read_text(encoding="utf-8") + assert 'data-default-sort="editorial"' in html + positions = [html.index(marker) for marker in ('

    ', ">w2w1', ">w3Small' in html + assert 'Small' in html + assert '' in html + assert '' in html + + subcategory_html = (site / "widgets" / "small" / "index.html").read_text(encoding="utf-8") + assert 'data-default-sort="editorial"' in subcategory_html + assert "group-row" not in subcategory_html + assert subcategory_html.index(">w2w1Machine Learning') + dl_heading = html.index('Deep Learning') + assert ml_heading < html.index(">ml1dl1ml1

    Start with ml1.

    ') + assert html.index('Machine Learning') < lead < html.index(">ml1 TemplateEntry: + return TemplateEntry( + name=name, + url="", + description="", + categories=[], + groups=[], + subcategories=[], + stars=stars, + downloads=None, + owner=None, + last_commit_at=None, + source_type=source_type, + bundled=False, + badge=None, + also_see=[], + ) + + +class TestSortEntries: + def test_sorts_by_stars_descending(self): + entries = [ + _template_entry("a", 100), + _template_entry("b", 500), + _template_entry("c", 200), + ] + result = sort_entries(entries) + assert [e["name"] for e in result] == ["b", "c", "a"] + + def test_equal_stars_sorted_alphabetically(self): + entries = [ + _template_entry("beta", 100), + _template_entry("alpha", 100), + ] + result = sort_entries(entries) + assert [e["name"] for e in result] == ["alpha", "beta"] + + def test_no_stars_go_to_bottom(self): + entries = [ + _template_entry("no-stars", None), + _template_entry("has-stars", 50), + ] + result = sort_entries(entries) + assert [e["name"] for e in result] == ["has-stars", "no-stars"] + + def test_no_stars_sorted_alphabetically(self): + entries = [ + _template_entry("zebra", None), + _template_entry("apple", None), + ] + result = sort_entries(entries) + assert [e["name"] for e in result] == ["apple", "zebra"] + + def test_builtin_between_starred_and_unstarred(self): + entries = [ + _template_entry("builtin", None, "Stdlib"), + _template_entry("starred", 100), + _template_entry("unstarred", None), + ] + result = sort_entries(entries) + assert [e["name"] for e in result] == ["starred", "builtin", "unstarred"] + + +# --------------------------------------------------------------------------- +# detect_source_type +# --------------------------------------------------------------------------- + + +class TestDetectSourceType: + def test_github_repo_returns_none(self): + assert detect_source_type("https://github.com/psf/requests") is None + + def test_stdlib_url(self): + assert detect_source_type("https://docs.python.org/3/library/asyncio.html") == "Stdlib" + + def test_gitlab_url(self): + assert detect_source_type("https://gitlab.com/org/repo") == "GitLab" + + def test_bitbucket_url(self): + assert detect_source_type("https://bitbucket.org/org/repo") == "Bitbucket" + + def test_non_github_external(self): + assert detect_source_type("https://example.com/tool") == "External" + + def test_github_non_repo_returns_none(self): + assert detect_source_type("https://github.com/org/repo/wiki") is None + + +# --------------------------------------------------------------------------- +# extract_entries +# --------------------------------------------------------------------------- + + +class TestExtractEntries: + def test_basic_extraction(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Tools** + + ## Widgets + + - [widget](https://example.com) - A widget. + + # Contributing + + Done. + """) + groups = parse_readme(readme) + categories = [c for g in groups for c in g["categories"]] + entries = extract_entries(categories, groups) + assert len(entries) == 1 + assert entries[0]["name"] == "widget" + assert entries[0]["categories"] == ["Widgets"] + assert entries[0]["groups"] == ["Tools"] + + def test_duplicate_entry_merged(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Tools** + + ## Alpha + + - [shared](https://example.com/shared) - Shared lib. + + ## Beta + + - [shared](https://example.com/shared) - Shared lib. + + # Contributing + + Done. + """) + groups = parse_readme(readme) + categories = [c for g in groups for c in g["categories"]] + entries = extract_entries(categories, groups) + shared = [e for e in entries if e["name"] == "shared"] + assert len(shared) == 1 + assert sorted(shared[0]["categories"]) == ["Alpha", "Beta"] + + def test_source_type_detected(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + ## Stdlib + + - [asyncio](https://docs.python.org/3/library/asyncio.html) - Async I/O. + + # Contributing + + Done. + """) + groups = parse_readme(readme) + categories = [c for g in groups for c in g["categories"]] + entries = extract_entries(categories, groups) + assert entries[0]["source_type"] == "Stdlib" + + def test_subcategory_includes_slug_and_url(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Tools** + + ## Web Frameworks + + - Synchronous + + - [django](https://example.com/django) - A framework. + + # Contributing + + Done. + """) + groups = parse_readme(readme) + categories = [c for g in groups for c in g["categories"]] + entries = extract_entries(categories, groups) + assert entries[0]["subcategories"] == [ + { + "name": "Synchronous", + "value": "Web Frameworks > Synchronous", + "slug": "synchronous", + "url": "/categories/web-frameworks/synchronous/", + }, + ] + + +# --------------------------------------------------------------------------- +# annotate_entries_with_stats +# --------------------------------------------------------------------------- + + +class TestAnnotateEntriesWithStats: + def test_appends_star_count_to_bullet(self): + markdown = "- [foo](https://github.com/owner/foo) - A foo.\n" + stars = {"owner/foo": {"stars": 123, "owner": "owner"}} + assert annotate_entries_with_stats(markdown, stars, {}) == ("- [foo](https://github.com/owner/foo) - A foo. (GitHub stars: 123)\n") + + def test_appends_downloads_by_display_name(self): + markdown = "- [Foo.py](https://example.com) - A foo.\n" + assert annotate_entries_with_stats(markdown, {}, {"foo-py": 777}) == ("- [Foo.py](https://example.com) - A foo. (PyPI downloads/month: 777)\n") + + def test_appends_downloads_and_stars_together(self): + markdown = "- [foo](https://github.com/owner/foo) - A foo.\n" + stars = {"owner/foo": {"stars": 123, "owner": "owner"}} + assert annotate_entries_with_stats(markdown, stars, {"foo": 777}) == ("- [foo](https://github.com/owner/foo) - A foo. (PyPI downloads/month: 777, GitHub stars: 123)\n") + + def test_uses_first_github_link(self): + markdown = "- [foo](https://github.com/owner/foo) - A foo. Also [bar](https://github.com/owner/bar).\n" + stars = { + "owner/foo": {"stars": 10, "owner": "owner"}, + "owner/bar": {"stars": 99, "owner": "owner"}, + } + assert annotate_entries_with_stats(markdown, stars, {}) == ("- [foo](https://github.com/owner/foo) - A foo. Also [bar](https://github.com/owner/bar). (GitHub stars: 10)\n") + + def test_skips_entries_without_data(self): + markdown = "- [foo](https://github.com/owner/foo) - A foo.\n" + assert annotate_entries_with_stats(markdown, {}, {}) == markdown + + def test_skips_non_github_links_for_stars(self): + markdown = "- [foo](https://example.com) - A foo.\n" + stars = {"owner/foo": {"stars": 1, "owner": "owner"}} + assert annotate_entries_with_stats(markdown, stars, {}) == markdown + + def test_skips_non_bullet_lines(self): + markdown = "See [foo](https://github.com/owner/foo) for details.\n" + stars = {"owner/foo": {"stars": 1, "owner": "owner"}} + assert annotate_entries_with_stats(markdown, stars, {"foo": 5}) == markdown + + def test_handles_indented_bullets(self): + markdown = " - [foo](https://github.com/owner/foo)\n" + stars = {"owner/foo": {"stars": 7, "owner": "owner"}} + assert annotate_entries_with_stats(markdown, stars, {}) == (" - [foo](https://github.com/owner/foo) (GitHub stars: 7)\n") + + def test_preserves_lines_without_trailing_newline(self): + markdown = "- [foo](https://github.com/owner/foo) - A foo." + stars = {"owner/foo": {"stars": 5, "owner": "owner"}} + assert annotate_entries_with_stats(markdown, stars, {}) == ("- [foo](https://github.com/owner/foo) - A foo. (GitHub stars: 5)") + + +class TestLoadDownloads: + def test_parses_tsv_and_skips_not_found(self, tmp_path): + tsv = tmp_path / "pypi_downloads.tsv" + tsv.write_text( + "name\tpackage\tdownloads\tfetched_at\naiohttp\taiohttp\t649105404\t2026-08-16\npytorch\ttorch\t50000000\t2026-08-16\ndead-pkg\t-\tNOT_FOUND\t2026-08-16\n", + encoding="utf-8", + ) + assert load_downloads(tsv) == {"aiohttp": 649105404, "pytorch": 50000000} + + def test_missing_file_returns_empty(self, tmp_path): + assert load_downloads(tmp_path / "nope.tsv") == {} + + +# --------------------------------------------------------------------------- +# load_category_intro +# --------------------------------------------------------------------------- + + +class TestLoadCategoryIntro: + def test_splits_after_how_to_choose_list(self, tmp_path): + path = tmp_path / "widgets.md" + path.write_text("Use `w1` for most apps.\n\nHow to choose:\n\n- Small apps: w1\n- Big apps: w2\n\nConfigure w1 once.\n\nPin w2.\n", encoding="utf-8") + intro_html, guide_html, lead, lead_html = load_category_intro(path) + assert intro_html == "

    Use w1 for most apps.

    \n

    How to choose:

    \n
      \n
    • Small apps: w1
    • \n
    • Big apps: w2
    • \n
    \n" + assert guide_html == "

    Configure w1 once.

    \n

    Pin w2.

    \n" + assert lead == "Use w1 for most apps." + assert lead_html == "

    Use w1 for most apps.

    \n" + + def test_keeps_everything_above_table_without_how_to_choose_list(self, tmp_path): + path = tmp_path / "widgets.md" + path.write_text("Use w1.\n\n- Small apps: w1\n\nConfigure w1 once.\n", encoding="utf-8") + intro_html, guide_html, _, _ = load_category_intro(path) + assert "Configure w1 once." in intro_html + assert guide_html == "" + + def test_returns_empty_strings_without_intro_file(self, tmp_path): + assert load_category_intro(tmp_path / "missing.md") == ("", "", "", "") + + def test_links_picks_to_entry_rows(self, tmp_path): + path = tmp_path / "widgets.md" + path.write_text( + "Use w1.\n\nHow to choose:\n\n- Small apps: Django Ninja, or `uv tool` on uv\n- Big apps: Django\n- Old apps: The Algorithms - Python\n\nSee [w1](https://w1.example).\n", encoding="utf-8" + ) + intro_html, guide_html, _, _ = load_category_intro(path, ["django-ninja", "Django", "uv", "TheAlgorithms"]) + assert '
  • Small apps: Django Ninja, or uv tool on uv
  • ' in intro_html + assert '
  • Big apps: Django
  • ' in intro_html + assert '
  • Old apps: The Algorithms - Python
  • ' in intro_html + assert 'w1' in guide_html + + def test_links_pick_aliases_to_entry_rows(self, tmp_path): + path = tmp_path / "widgets.md" + path.write_text("Use w1.\n\nHow to choose:\n\n- Vision: OpenCV\n", encoding="utf-8") + intro_html, _, _, _ = load_category_intro(path, ["opencv-python"]) + assert '
  • Vision: OpenCV
  • ' in intro_html diff --git a/website/tests/test_fetch_github_stars.py b/website/tests/test_fetch_github_stars.py new file mode 100644 index 0000000000..7b694e51aa --- /dev/null +++ b/website/tests/test_fetch_github_stars.py @@ -0,0 +1,267 @@ +"""Tests for fetch_github_stars module.""" + +import json +from datetime import UTC + +from fetch_github_stars import ( + build_graphql_query, + extract_github_repos, + parse_graphql_response, + save_cache, +) + + +class TestExtractGithubRepos: + def test_extracts_owner_repo_from_github_url(self): + readme = "* [requests](https://github.com/psf/requests) - HTTP lib." + result = extract_github_repos(readme) + assert result == {"psf/requests"} + + def test_multiple_repos(self): + readme = "* [requests](https://github.com/psf/requests) - HTTP.\n* [flask](https://github.com/pallets/flask) - Micro." + result = extract_github_repos(readme) + assert result == {"psf/requests", "pallets/flask"} + + def test_deduplicates(self): + readme = "* [a](https://github.com/org/repo) - A.\n* [b](https://github.com/org/repo) - B." + result = extract_github_repos(readme) + assert result == {"org/repo"} + + def test_strips_fragment(self): + readme = "* [lib](https://github.com/org/repo#section) - Lib." + result = extract_github_repos(readme) + assert result == {"org/repo"} + + +class TestSaveCache: + def test_creates_directory_and_writes_json(self, tmp_path, monkeypatch): + data_dir = tmp_path / "data" + cache_file = data_dir / "stars.json" + monkeypatch.setattr("fetch_github_stars.DATA_DIR", data_dir) + monkeypatch.setattr("fetch_github_stars.CACHE_FILE", cache_file) + save_cache({"a/b": {"stars": 1}}) + assert cache_file.exists() + assert json.loads(cache_file.read_text(encoding="utf-8")) == {"a/b": {"stars": 1}} + + +class TestBuildGraphqlQuery: + def test_single_repo(self): + query = build_graphql_query(["psf/requests"]) + assert "repository" in query + assert 'owner: "psf"' in query + assert 'name: "requests"' in query + assert "stargazerCount" in query + + def test_multiple_repos_use_aliases(self): + query = build_graphql_query(["psf/requests", "pallets/flask"]) + assert "repo_0:" in query + assert "repo_1:" in query + + def test_empty_list(self): + query = build_graphql_query([]) + assert query == "" + + def test_skips_repos_with_quotes_in_name(self): + query = build_graphql_query(['org/"bad"']) + assert query == "" + + def test_skips_only_bad_repos(self): + query = build_graphql_query(["good/repo", 'bad/"repo"']) + assert "good" in query + assert "bad" not in query + + def test_skips_graphql_injection_in_owner(self): + query = build_graphql_query(['org"){evil}/repo']) + assert query == "" + + def test_skips_graphql_injection_in_name(self): + query = build_graphql_query(['org/repo"){evil}']) + assert query == "" + + def test_skips_owner_starting_with_hyphen(self): + query = build_graphql_query(["-bad/repo"]) + assert query == "" + + def test_skips_owner_starting_with_dot(self): + query = build_graphql_query([".bad/repo"]) + assert query == "" + + def test_skips_repo_starting_with_dot(self): + query = build_graphql_query(["org/.hidden"]) + assert query == "" + + def test_allows_repo_with_dots_and_underscores(self): + query = build_graphql_query(["org/my_repo.py"]) + assert 'name: "my_repo.py"' in query + + def test_allows_hyphenated_owner(self): + query = build_graphql_query(["my-org/repo"]) + assert 'owner: "my-org"' in query + + def test_skips_owner_with_underscore(self): + query = build_graphql_query(["bad_owner/repo"]) + assert query == "" + + +class TestParseGraphqlResponse: + def test_parses_star_count_and_owner(self): + data = { + "repo_0": { + "stargazerCount": 52467, + "owner": {"login": "psf"}, + } + } + repos = ["psf/requests"] + result = parse_graphql_response(data, repos) + assert result["psf/requests"]["stars"] == 52467 + assert result["psf/requests"]["owner"] == "psf" + + def test_skips_null_repos(self): + data = {"repo_0": None} + repos = ["deleted/repo"] + result = parse_graphql_response(data, repos) + assert result == {} + + def test_handles_missing_owner(self): + data = {"repo_0": {"stargazerCount": 100}} + repos = ["org/repo"] + result = parse_graphql_response(data, repos) + assert result["org/repo"]["owner"] == "" + + def test_multiple_repos(self): + data = { + "repo_0": {"stargazerCount": 100, "owner": {"login": "a"}}, + "repo_1": {"stargazerCount": 200, "owner": {"login": "b"}}, + } + repos = ["a/x", "b/y"] + result = parse_graphql_response(data, repos) + assert len(result) == 2 + assert result["a/x"]["stars"] == 100 + assert result["b/y"]["stars"] == 200 + + def test_extracts_last_commit_at(self): + data = { + "repo_0": { + "stargazerCount": 100, + "owner": {"login": "org"}, + "defaultBranchRef": {"target": {"committedDate": "2025-06-01T00:00:00Z"}}, + } + } + repos = ["org/repo"] + result = parse_graphql_response(data, repos) + assert result["org/repo"]["last_commit_at"] == "2025-06-01T00:00:00Z" + + def test_missing_default_branch_ref(self): + data = {"repo_0": {"stargazerCount": 50, "owner": {"login": "org"}}} + repos = ["org/repo"] + result = parse_graphql_response(data, repos) + assert result["org/repo"]["last_commit_at"] == "" + + +class TestMainSkipsFreshCache: + """Verify that main() skips fetching when all cache entries are fresh.""" + + def test_skips_fetch_when_cache_is_fresh(self, tmp_path, monkeypatch, capsys): + from datetime import datetime, timedelta + + from fetch_github_stars import main + + # Set up a minimal README with one repo + readme = tmp_path / "README.md" + readme.write_text("* [req](https://github.com/psf/requests) - HTTP.\n") + monkeypatch.setattr("fetch_github_stars.README_PATH", readme) + + # Pre-populate cache with a fresh entry (1 hour ago) + data_dir = tmp_path / "data" + data_dir.mkdir() + cache_file = data_dir / "github_stars.json" + now = datetime.now(UTC) + fresh_cache = { + "psf/requests": { + "stars": 52000, + "owner": "psf", + "last_commit_at": "2025-01-01T00:00:00+00:00", + "fetched_at": (now - timedelta(hours=1)).isoformat(), + }, + "vinta/awesome-python": { + "stars": 230000, + "owner": "vinta", + "last_commit_at": "2025-01-01T00:00:00+00:00", + "fetched_at": (now - timedelta(hours=1)).isoformat(), + }, + } + cache_file.write_text(json.dumps(fresh_cache), encoding="utf-8") + monkeypatch.setattr("fetch_github_stars.CACHE_FILE", cache_file) + monkeypatch.setattr("fetch_github_stars.DATA_DIR", data_dir) + monkeypatch.setenv("GITHUB_TOKEN", "fake-token") + + main() + + output = capsys.readouterr().out + assert "0 repos to fetch" in output + assert "Cache is up to date" in output + + def test_fetches_when_cache_is_stale(self, tmp_path, monkeypatch, capsys): + from datetime import datetime, timedelta + from unittest.mock import MagicMock + + from fetch_github_stars import main + + # Set up a minimal README with one repo + readme = tmp_path / "README.md" + readme.write_text("* [req](https://github.com/psf/requests) - HTTP.\n") + monkeypatch.setattr("fetch_github_stars.README_PATH", readme) + + # Pre-populate cache with a stale entry (24 hours ago) + data_dir = tmp_path / "data" + data_dir.mkdir() + cache_file = data_dir / "github_stars.json" + now = datetime.now(UTC) + stale_cache = { + "psf/requests": { + "stars": 52000, + "owner": "psf", + "last_commit_at": "2025-01-01T00:00:00+00:00", + "fetched_at": (now - timedelta(hours=24)).isoformat(), + }, + "vinta/awesome-python": { + "stars": 230000, + "owner": "vinta", + "last_commit_at": "2025-01-01T00:00:00+00:00", + "fetched_at": (now - timedelta(hours=24)).isoformat(), + }, + } + cache_file.write_text(json.dumps(stale_cache), encoding="utf-8") + monkeypatch.setattr("fetch_github_stars.CACHE_FILE", cache_file) + monkeypatch.setattr("fetch_github_stars.DATA_DIR", data_dir) + monkeypatch.setenv("GITHUB_TOKEN", "fake-token") + + # Mock httpx.Client to avoid real API calls + mock_response = MagicMock() + mock_response.json.return_value = { + "data": { + "repo_0": { + "stargazerCount": 53000, + "owner": {"login": "psf"}, + "defaultBranchRef": {"target": {"committedDate": "2025-06-01T00:00:00Z"}}, + }, + "repo_1": { + "stargazerCount": 231000, + "owner": {"login": "vinta"}, + "defaultBranchRef": {"target": {"committedDate": "2025-06-01T00:00:00Z"}}, + }, + } + } + mock_response.raise_for_status = MagicMock() + mock_client = MagicMock() + mock_client.__enter__ = MagicMock(return_value=mock_client) + mock_client.__exit__ = MagicMock(return_value=False) + mock_client.post.return_value = mock_response + monkeypatch.setattr("fetch_github_stars.httpx.Client", lambda **kwargs: mock_client) + + main() + + output = capsys.readouterr().out + assert "2 repos to fetch" in output + assert "Done. Fetched 2 repos" in output + mock_client.post.assert_called_once() diff --git a/website/tests/test_preview_verdicts.py b/website/tests/test_preview_verdicts.py new file mode 100644 index 0000000000..08c5674da9 --- /dev/null +++ b/website/tests/test_preview_verdicts.py @@ -0,0 +1,108 @@ +import json +import re +import shutil +import subprocess +from pathlib import Path + +import pytest + +TEMPLATE = Path(__file__).resolve().parents[2] / ".claude/skills/preview-verdicts/template.html" +NODE = shutil.which("node") + +RUNNER = r""" +const fs = require("node:fs"); +const vm = require("node:vm"); +const { script, actions } = JSON.parse(fs.readFileSync(0, "utf8")); +const storage = {}; +let elements, context, copied; +function reload() { + elements = {}; + context = vm.createContext({ + document: { getElementById: id => elements[id] ||= {}, title: "Review" }, + localStorage: { getItem: key => storage[key], setItem: (key, value) => storage[key] = value }, + navigator: { clipboard: { writeText: text => { copied = text; return Promise.resolve(); } } } + }); + vm.runInContext(script, context); +} +function snapshot() { + return { + html: elements.app.innerHTML, + counts: elements.counts.innerHTML, + feedback: context.collect(), + rowClass: elements["r-Section__entry"]?.className, + copied + }; +} +reload(); +const snapshots = [snapshot()]; +for (const [action, ...args] of actions) { + if (action === "reload") reload(); + else context[action](...args); + snapshots.push(snapshot()); +} +process.stdout.write(JSON.stringify(snapshots)); +""" + + +def preview(kind, verdict, actions=(), source=None): + if NODE is None: + pytest.skip("Preview script tests require Node.js") + row = ["entry", "https://github.com/owner/entry", "1,000", kind, verdict, "Review reason"] + if source is not None: + row.append(source) + data = [["Section", "Use case — review note", [row]]] + script = TEMPLATE.read_text().split("", 1)[0].replace("__DATA__", json.dumps(data)) + result = subprocess.run([NODE, "-e", RUNNER], input=json.dumps({"script": script, "actions": actions}), capture_output=True, text=True, check=True) + return json.loads(result.stdout) + + +@pytest.mark.parametrize( + ("kind", "accept", "reject", "buttons"), + [ + ("listed", "keep", "remove", ["Keep", "Remove", "Park"]), + ("candidate", "add", "reject", ["Add", "Reject", "Park"]), + ("structure", "restructure", "reject", ["Restructure", "Park", "Reject"]), + ], +) +def test_parked_rows_keep_their_choices_and_feedback_after_reload(kind, accept, reject, buttons): + initial, accepted, reloaded, copied, rejected, parked = preview( + kind, + "park", + [("setV", "Section::entry", accept), ("reload",), ("copyFeedback",), ("setV", "Section::entry", reject), ("setV", "Section::entry", "park")], + ) + for snapshot in (initial, accepted, reloaded, copied, rejected, parked): + assert re.findall(r"]*>([^<]+)", snapshot["html"]) == buttons + assert initial["feedback"] == parked["feedback"] == [] + expected = [{"section": "Section", "subcategory": "Use case", "entry": "entry", "kind": kind, "my_verdict": "park", "your_verdict": accept, "reason": ""}] + assert accepted["feedback"] == reloaded["feedback"] == expected + assert json.loads(copied["copied"]) == expected + assert rejected["feedback"][0]["your_verdict"] == reject + assert f'