DataOps is the combined DataTalks.Club operations portal, merging DTC Operations process docs and docs app and DataTasks task execution. It has no built-in assistant: assistants run externally and work through the portal's API.

The project is in the consolidation stage. Source systems have been merged into a single TypeScript backend; see docs/TARGET_ARCHITECTURE.md. The single backend/ package serves the frontend, docs content API, search, and work APIs from one TypeScript Lambda. The Python docs/SOP backend has been retired; the remaining content-validation tooling lives in tools/content_tools/.

Domain vocabulary: a Template is a reusable definition, a Card is one instantiated unit of work created from a template, and a Task is a checklist item contained by a card.

Technology direction:

• Backend: TypeScript, single backend/ package (supersedes the earlier "long-term backend: Python" direction; see docs/MERGE_PLAN.md)
• Search: zerosearch-node (BM25-lite, zero-dependency; supersedes minsearch)
• Frontend shell: the existing dtc-operations vanilla JavaScript frontend
• Execution state: DynamoDB
• Process docs: GitHub markdown
• Package management: npm for TypeScript, uv for Python tooling (content validation)

No backwards compatibility:

• This project has no external consumers and no released API to protect. When something is renamed, restructured, or replaced, change it everywhere in one step and delete the old shape.
• Do not add compatibility shims, aliases, legacy fallbacks, duplicate routes, deprecated fields kept "just in case", dual-read/dual-write paths, or version branches that keep old behavior alive alongside new behavior.
• Migrate persisted data as part of the change rather than teaching the code to read both the old and the new format.
• If a change cannot be made atomically, say so and propose the sequencing instead of leaving a permanent compatibility layer behind.

Commits and worktrees:

• Land work on main as focused commits: one logical change per commit, each verified before it lands. Do not bundle unrelated changes to "get them in".
• The checkout is shared with other sessions. Never sweep another session's uncommitted work into your commit. Stage only your own hunks; when a file contains both your edits and someone else's, commit hunk-by-hunk and leave their work untouched in the working tree.
• Verify isolated work in an isolated worktree under .tmp/worktrees/, then delete the worktree when the work has landed (git worktree remove). Stale worktrees on old bases rot: their uncommitted diffs silently diverge from main and cannot land cleanly.
• While working, commit regularly: each logical unit gets its own focused commit as it lands, not one big commit at the end.
• When the task is done: run the tests and fix any failures, merge with main and push, then delete your branch and remove your worktree so no stale branches or worktrees are left behind.

Operational knowledge boundary:

• DataTalksClub/dataops stays public and owns product/runtime code, CI/CD, tests, schemas, sanitized fixtures, and public-safe planning docs.
• Operational documents and knowledge belong in a separate private repository. The planned repo name is DataTalksClub/dataops-knowledge.
• Do not add raw SOPs, workflow templates, assistant prompts/process instructions, screenshots, private links, credentials-adjacent setup notes, contact details, sponsor or finance context, or generated operational artifacts to this public repo.
• Existing content/ material in this repo is transitional migration debt. Treat it as public-sensitive until it is audited and moved behind the private knowledge boundary.

Current planning docs:

• docs/MERGE_PLAN.md
• PORTAL_ANALYSIS.md
• PROJECT_PLAN.md

AWS infrastructure source:

• Infrastructure lives in ../aws-infra/, not in this repo. DataOps-specific infrastructure is in ../aws-infra/sandbox/dataops/. Edit it there and commit it there; do not add or copy infrastructure templates into this repo.
• That directory owns the GitHub Actions OIDC deploy role (template.github-actions.yaml), the CloudFront domain (template.domain.yaml), the runtime secrets (template.runtime-secrets.yaml), and the knowledge backups bucket (template.knowledge-backups.yaml).
• The one exception is infra/template.full.yaml, the application stack. Its tiny infra/sam-build/ CodeUri delegates to the repository packaging script, so the template has to sit next to the code it packages. Moving it requires changing how the Lambda artifact is produced, not just moving a file.
• Keeping a second copy of an aws-infra template here is what caused the two to drift apart. If a template needs changing, change it in aws-infra.
• aws-infra does not currently deploy itself through CI/CD. If that template changes, committing/pushing it is not enough; a credentialed AWS operator must apply the dataops-github-actions CloudFormation stack.
• The DataOps app itself deploys through this repo's GitHub Actions CI/CD using OIDC after main is pushed. Do not replace that with a normal manual app deploy.

The application never creates infrastructure:

• Tables, buckets, queues and secrets are declared in infrastructure. Nothing in backend/src creates or mutates them, and no environment variable enables it. If a resource is missing the code fails loudly and names it, rather than quietly creating one that nothing manages or backs up.
• Local development and tests may stand up throwaway local resources, but that belongs in a setup script, not in code that ships to the Lambda.
• DynamoDB allows only one global secondary index created or deleted per stack update, so an index rename cannot ride a normal deploy. Do index changes as an offline operation and let the template declare the end state; do not build migration machinery into the deploy pipeline, where it outlives the migration and rots. While data is disposable, replace the table instead. See docs/dynamodb-index-changes.md.
• A migration that has run, and a guard for a migration that has finished, are dead code. Delete them.

Initial source systems:

• ../dtc-operations
• ../datatasks
• ../podcast-assistant

Do not modify those source repos while working in dataops unless the task explicitly asks for it.