feat(gateway): DNS credential verification and cert attempt observability - #1062
Draft
Leechael wants to merge 2 commits into
Draft
feat(gateway): DNS credential verification and cert attempt observability#1062Leechael wants to merge 2 commits into
Leechael wants to merge 2 commits into
Conversation
Read-only check of a DNS credential against the provider API: reports the
Cloudflare token status (active/expired/disabled/invalid), the granted
permission groups, whether issuance requirements (Zone Read, DNS Write)
are covered, and zone resolution for every domain referencing the
credential. Today an invalid or under-privileged token only surfaces as a
log line during the next renewal attempt.
certbot gains verify_cloudflare_token() (GET /user/tokens/verify followed
by GET /user/tokens/{id}) and resolve_cloudflare_zone(); the mock CF API
learns the matching token endpoints, with MOCK_TOKEN_STATUS and
MOCK_TOKEN_PERMISSIONS env knobs for failure testing.
Certificate issuance and renewal run as a background task whose failures only reach the logs, so an expiring certificate is the first visible signal of a broken renewal. DistributedCertBot now keeps a node-local, in-memory record per domain: last attempt time and node, last success, consecutive failures, and the (16 KiB head+tail truncated) error chain of the last failed attempt. Both the periodic/manual renewal path (try_renew) and startup issuance (init_domain) record their outcome; nodes that skip because a peer holds the renew lock do not write. The record is exposed on the existing ZtDomainCertStatus, so GetZtDomain and ListZtDomains show it without a new RPC. In-memory storage targets the current single-node deployments; multi-node visibility can later move the record into WaveKV without changing the response shape.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two admin-facing observability additions for the gateway's certificate lifecycle:
VerifyDnsCredentialadmin RPC — read-only check of a DNS credential against the provider API. Reports the Cloudflare token status (active/expired/disabled/invalid), the granted permission groups, whether issuance requirements (Zone Read, DNS Write) are covered, and zone resolution for every domain referencing the credential (directly or via the default-credential fallback). Today an invalid or under-privileged token only surfaces as a log line during the next renewal attempt.DistributedCertBotnow keeps a node-local, in-memory record per domain (last attempt time and node, last success, consecutive failures, truncated error chain of the last failure). Exposed by extendingZtDomainCertStatus, so existingGetZtDomain/ListZtDomainsreturn it without a new RPC.try_renewandinit_domainboth record outcomes; nodes that skip because a peer holds the renew lock do not write.Design notes
GET /user/tokens/verifyyields the token id, thenGET /user/tokens/{id}returns the granted permission groups. No DNS records are created or removed.Test plan
cargo test -p dstack-gateway -p certbot— 161 passed, including new unit tests for permission matching, error truncation, and attempt-status trackingtools/mock-cf-dns-api(extended with/user/tokens/verifyand/user/tokens/{id}, plusMOCK_TOKEN_STATUS/MOCK_TOKEN_PERMISSIONSknobs): token verify, permission group parsing, zone resolution, and unknown-zone failure all verifiedcargo clippyclean,prek run --all-filesclean