Skip to content

Various SEO fixes#6606

Merged
Karakatiza666 merged 2 commits into
mainfrom
worktree-docs-sitemap
Jul 13, 2026
Merged

Various SEO fixes#6606
Karakatiza666 merged 2 commits into
mainfrom
worktree-docs-sitemap

Conversation

@Karakatiza666

Copy link
Copy Markdown
Contributor
  • Make docs links consistent in terms of file extention and trailing slashes
  • Add robots.txt to pipeline-manager to avoid crawling on try.feldera.com
  • Add redirects for moved blogs
  • Remove a testing endpoint from OpenAPI spec
  • Fix Link Check error of trying to reach Slack invite URL

@Karakatiza666
Karakatiza666 requested a review from gz July 9, 2026 13:08

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Test coverage for the new /robots.txt route is exactly right, and the openapi.json shrinkage is a legit "drop an unstable test-only endpoint from the public spec" (route stays registered for the harness), not a breaking API change.

Comment thread .github/workflows/docs-linkcheck.yml Outdated
- name: Check links on docs.feldera.com
run: |
uv run --locked linkchecker https://docs.feldera.com --check-extern --no-warnings --ignore-url "https?://localhost|https://crates.io|https://www.linkedin.com|https://ieeexplore.ieee.org|https://x.com|https?://127.0.0.1" --no-robots
uv run --locked linkchecker https://docs.feldera.com --check-extern --no-warnings --ignore-url "https?://localhost|https://crates.io|https://www.linkedin.com|https://ieeexplore.ieee.org|https://x.com|https?://127.0.0.1|https://join.slack.com|https://felderacommunity.slack.com" --no-robots --user-agent "Mozilla/5.0 (compatible; feldera-linkcheck)"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont change the user agent IMO it leads to more failures

## Authentication

When `auth.enabled` is `true`, the API server requires OIDC authentication. See the guides for [AWS Cognito](./authentication/aws-cognito), [Okta](./authentication/okta-sso), or [generic OIDC providers](./authentication/generic-oidc).
When `auth.enabled` is `true`, the API server requires OIDC authentication. See the guides for [AWS Cognito](./authentication/aws-cognito.md), [Okta](./authentication/okta-sso.md), or [generic OIDC providers](./authentication/generic-oidc.md).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this .md solve?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without an extension, the reference gets resolved by Docusaurus incorrectly at build time and renders an invalid link

…e Search Console

  Docs (docs.feldera.com):
  - Set `trailingSlash: true` to match the host, which serves canonical
    pages with a trailing slash - this avoids unnecessary redirects for
    links, canonical tags and sitemap entries, generated by Docusaurus.
  - Sitemap: emit `lastmod` (the only freshness signal search engines
    honor) and drop utility routes and the raw OpenAPI spec from it.
  - Add `static/robots.txt` (allow all + sitemap pointer).
  - Fix internal doc links to use explicit `.md`/`.mdx` targets so they
    resolve to the canonical URL instead of a redirect hop.
  - De-slash blog/paper links (what-is-feldera, literature/papers x2,
    fraud_detection) to drop the 308 hop to the canonical no-slash URL.
  - Add three client-redirects for real 404s (eks/ingress, k3d, kafka/
    debezium) so old indexed URLs land on live pages.
  - Mark the sandbox "Run in sandbox" link `nofollow`: it deep-links to
    `try.feldera.com/create?code=...`, and following it floods Search
    Console with duplicate/redirect entries for un-indexable app URLs.
  - Remove the unused markdown-page.md example stub.

  Pipeline-manager:
  - Serve `GET /robots.txt` with a blanket `Disallow: /`. The web console
    is a client-side SPA, so per-page robots hints never reach crawlers; a
    root disallow is the only reliable way to keep app URLs (the sandbox's
    `/create?...` deep-links) out of search indexes. Covers every
    deployment, which is the desired default. Regression test included.
  - Exclude the test-only `POST /pipelines/{name}/testing` endpoint from
    the public OpenAPI spec (dropped from `paths()`, route kept for the
    test harness); regenerate openapi.json. Stops docs from publishing —
    and Google from indexing — a URL that was never a supported API.

Signed-off-by: Karakatiza666 <bulakh.96@gmail.com>
…nly work for browsers

Signed-off-by: Karakatiza666 <bulakh.96@gmail.com>
@Karakatiza666
Karakatiza666 force-pushed the worktree-docs-sitemap branch from f95bdfa to 293f712 Compare July 13, 2026 17:51
@Karakatiza666
Karakatiza666 enabled auto-merge July 13, 2026 17:51
@Karakatiza666
Karakatiza666 added this pull request to the merge queue Jul 13, 2026
Merged via the queue into main with commit dbdb924 Jul 13, 2026
1 check passed
@Karakatiza666
Karakatiza666 deleted the worktree-docs-sitemap branch July 13, 2026 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants