Various SEO fixes#6606
Merged
Merged
Conversation
Karakatiza666
commented
Jul 9, 2026
Contributor
- Make docs links consistent in terms of file extention and trailing slashes
- Add robots.txt to pipeline-manager to avoid crawling on try.feldera.com
- Add redirects for moved blogs
- Remove a testing endpoint from OpenAPI spec
- Fix Link Check error of trying to reach Slack invite URL
mythical-fred
approved these changes
Jul 9, 2026
mythical-fred
left a comment
There was a problem hiding this comment.
LGTM. Test coverage for the new /robots.txt route is exactly right, and the openapi.json shrinkage is a legit "drop an unstable test-only endpoint from the public spec" (route stays registered for the harness), not a breaking API change.
gz
approved these changes
Jul 9, 2026
| - name: Check links on docs.feldera.com | ||
| run: | | ||
| uv run --locked linkchecker https://docs.feldera.com --check-extern --no-warnings --ignore-url "https?://localhost|https://crates.io|https://www.linkedin.com|https://ieeexplore.ieee.org|https://x.com|https?://127.0.0.1" --no-robots | ||
| uv run --locked linkchecker https://docs.feldera.com --check-extern --no-warnings --ignore-url "https?://localhost|https://crates.io|https://www.linkedin.com|https://ieeexplore.ieee.org|https://x.com|https?://127.0.0.1|https://join.slack.com|https://felderacommunity.slack.com" --no-robots --user-agent "Mozilla/5.0 (compatible; feldera-linkcheck)" |
Contributor
There was a problem hiding this comment.
dont change the user agent IMO it leads to more failures
| ## Authentication | ||
|
|
||
| When `auth.enabled` is `true`, the API server requires OIDC authentication. See the guides for [AWS Cognito](./authentication/aws-cognito), [Okta](./authentication/okta-sso), or [generic OIDC providers](./authentication/generic-oidc). | ||
| When `auth.enabled` is `true`, the API server requires OIDC authentication. See the guides for [AWS Cognito](./authentication/aws-cognito.md), [Okta](./authentication/okta-sso.md), or [generic OIDC providers](./authentication/generic-oidc.md). |
Contributor
Author
There was a problem hiding this comment.
Without an extension, the reference gets resolved by Docusaurus incorrectly at build time and renders an invalid link
…e Search Console
Docs (docs.feldera.com):
- Set `trailingSlash: true` to match the host, which serves canonical
pages with a trailing slash - this avoids unnecessary redirects for
links, canonical tags and sitemap entries, generated by Docusaurus.
- Sitemap: emit `lastmod` (the only freshness signal search engines
honor) and drop utility routes and the raw OpenAPI spec from it.
- Add `static/robots.txt` (allow all + sitemap pointer).
- Fix internal doc links to use explicit `.md`/`.mdx` targets so they
resolve to the canonical URL instead of a redirect hop.
- De-slash blog/paper links (what-is-feldera, literature/papers x2,
fraud_detection) to drop the 308 hop to the canonical no-slash URL.
- Add three client-redirects for real 404s (eks/ingress, k3d, kafka/
debezium) so old indexed URLs land on live pages.
- Mark the sandbox "Run in sandbox" link `nofollow`: it deep-links to
`try.feldera.com/create?code=...`, and following it floods Search
Console with duplicate/redirect entries for un-indexable app URLs.
- Remove the unused markdown-page.md example stub.
Pipeline-manager:
- Serve `GET /robots.txt` with a blanket `Disallow: /`. The web console
is a client-side SPA, so per-page robots hints never reach crawlers; a
root disallow is the only reliable way to keep app URLs (the sandbox's
`/create?...` deep-links) out of search indexes. Covers every
deployment, which is the desired default. Regression test included.
- Exclude the test-only `POST /pipelines/{name}/testing` endpoint from
the public OpenAPI spec (dropped from `paths()`, route kept for the
test harness); regenerate openapi.json. Stops docs from publishing —
and Google from indexing — a URL that was never a supported API.
Signed-off-by: Karakatiza666 <bulakh.96@gmail.com>
…nly work for browsers Signed-off-by: Karakatiza666 <bulakh.96@gmail.com>
Karakatiza666
force-pushed
the
worktree-docs-sitemap
branch
from
July 13, 2026 17:51
f95bdfa to
293f712
Compare
Karakatiza666
enabled auto-merge
July 13, 2026 17:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.