Skip to content

desc-alias-and-otlp-push-exporter - #795

Merged
jeffreyaven merged 4 commits into
mainfrom
feat/desc-alias-and-otlp-push
Sep 26, 2026
Merged

jeffreyaven merged 4 commits into
mainfrom
feat/desc-alias-and-otlp-push

Conversation

@jeffreyaven

@jeffreyaven jeffreyaven commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

Closes #773. Implements phase 2 of #755; the phase 3 backend recipes are website docs and are listed under follow-ups.

DESC as an alias for DESCRIBE (#773)

  • The parser (stackql-parser v0.0.16-alpha02) already lists DESC as an explain synonym, so DESC <table> and DESC EXTENDED <table> parse and execute as DESCRIBE on the CLI, shell and pg wire paths. No engine change was needed there.
  • The MCP query classifier (pkg/mcp_server/policy.ClassifyQuery) switches on the literal first token and did not know DESC, so run_select_query with DESC ... was classed unknown: refused on read_only servers and gated on safe ones. DESC is now classed read-only alongside DESCRIBE.
  • A leading DESC cannot clash with the ORDER BY ... DESC direction keyword, which never starts a statement; the grammar already distinguishes the two.
  • Not covered: DESC METHOD .... The DESCRIBE METHOD rule in stackql-parser (sql.y, explain_statement) names DESCRIBE only; extending it needs a parser release and a go.mod bump, so it is a follow-up in that repo.

--output otel OTLP/HTTP push exporter (#755 phase 2)

Configured through a new --otel.config flag (a JSON / YAML string, like the other raw-config flags), parsed once at startup into an OTelConfig DTO whose optional exporter member is a sink.OTLPHTTPConfig (json/yaml-tagged, alongside sink.FileConfig). No environment variables are read; this replaced the OTEL_EXPORTER_OTLP_* design after review.

{ "exporter": { "endpoint": "http://collector:4318/v1/logs", "headers": { "authorization": "<token>" }, "timeout_ms": 10000, "batch_size": 512 } }

exporter.endpoint is required to enable the exporter and is used verbatim; timeout_ms and batch_size default to 10000 and 512. A malformed value, or an exporter without an endpoint, fails startup with invalid --otel.config: ....

Behaviour:

  • Records still stream to stdout exactly as before. When an endpoint is configured they are also collected per statement and POSTed as OTLP/JSON LogsData (one resourceLogs / scopeLogs envelope, N logRecords), chunked by the batch size. The last chunk ships after the completion record, so the process does not exit before the push completes.
  • Retry: 429, 5xx and transport errors are retried up to 4 attempts with a 0.5s doubling backoff, or the server's Retry-After (delay-seconds) when sent. Client errors are not retried. A failed export is written to stderr after the rows have streamed, because the response handler discards writer errors.
  • Only OTLP/JSON over HTTP is emitted (Content-Type: application/json); OTEL_EXPORTER_OTLP_PROTOCOL is not consulted.

Design choices:

  • The exporter lives in pkg/sink as otlpHTTPSink behind NewOTLPHTTPSink(OTLPHTTPConfig) (Sink, error); the DTO is exported only because it is deserialised configuration, the sink itself stays unexported. output.InitOTelConfig mirrors intrinsic.Init for the --preview flag. The MCP audit otel format can reuse the same sink and DTO later.
  • sink.NewMultiSink is the fan-out (stdout + exporter); OTelWriter now closes its sink at the end of Write / WriteError to flush the batch.
  • Test assets: the OTLP flask mock (test/python/stackql_test_tooling/flask/otlp/app.py, port 1200) captures requests for assertion and has a keyed flaky route. Only the port and base URL were added to stackql_context.py (config, mirroring the retry mock); queries are inline in the scenarios. The /requests fetch writes curl's stdout to a file: robot's Run Process captures stdout through a pipe it reads only after exit, and the 11 KB response deadlocks against the 4 KB pipe buffer on Windows (Linux and macOS have 64 KB), which is what hung the Windows job on the first run.
  • The mock is wired for both mock launch paths: the native flask launcher in web_service_keywords.py and docker-compose-testing.yml, which the Linux runners use for the mocks whenever docker is present (regardless of EXECUTION_PLATFORM). The push scenarios skip on EXECUTION_PLATFORM=docker, like the --env.file scenario, because the exec container neither receives the exporter environment nor reaches the host mock.

Test plan

  • Unit: pkg/sink (batching and envelope merge, retry/backoff/Retry-After, no retry on 4xx, max attempts, env precedence, multi sink), internal/stackql/output (push through the writer via env, chunking, failure on stderr), pkg/mcp_server/policy (desc, DESC EXTENDED).
  • Robot, stackql_mocked_from_cmd_line.robot: Desc Extended Is Accepted As An Alias For Describe Extended (exact DESCRIBE EXTENDED output for DESC EXTENDED, plus plain desc), OTel Output Pushes Statement Batch To OTLP Endpoint, OTel Output Chunks Pushed Batches By Configured Batch Size, OTel Output Retries OTLP Push On Server Error. All pass locally with the cross-compiled binary.
  • Robot, mcp.robot: MCP HTTP Mode Read Only Allows Desc Alias (DESC EXTENDED google.storage.buckets via run_select_query on the 9920 read_only server). Verified by hand against a read_only server: main refuses, this branch returns rows.
  • golangci-lint v2.5.0: 0 issues.

Follow-ups

🤖 Generated with Claude Code

jeffreyaven and others added 3 commits September 26, 2026 04:24
- MCP: class DESC as read-only like DESCRIBE (#773); the parser already
  accepts DESC and DESC EXTENDED on every other path
- --output otel: OTLP/HTTP push exporter behind the standard
  OTEL_EXPORTER_OTLP_* variables, batched per statement, chunked,
  retried with backoff (#755 phase 2)
- robot: DESC alias scenarios (CLI and MCP read_only), OTLP mock plus
  push, chunk and retry scenarios

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Linux runners start the mocks via docker compose whenever docker is
present, regardless of EXECUTION_PLATFORM, so the OTLP mock needs a
compose service as well as the native launcher; the arm64 job failed
the three push scenarios with connection refused on 127.0.0.1:1200.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Robot's Run Process captures stdout through a pipe it reads only after
the child exits; the 11 KB /requests response exceeds the 4 KB pipe
buffer on Windows and curl blocked forever, hanging the Windows job.
Route stdout to a file and cap the mock calls with --max-time.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Comment thread pkg/sink/otlp_http.go Outdated
// OTEL_EXPORTER_OTLP_* environment variables; a logs-specific variable
// takes precedence over its generic counterpart.
const (
otlpLogsEndpointEnv = "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be. None of this functionality should be driven by env vars. Configuration for stackql belongs in idiomatic DTO structures. There are countless existing ones to folllow. For a groupling like this, they probably go in a dedicated struct which is an optional member of some higher level DTO

Review: exporter settings belong in a DTO, not environment variables.
The exporter is now sink.OTLPHTTPConfig (json/yaml tagged), the optional
exporter member of output.OTelConfig, parsed once from the new
--otel.config flag; no OTEL_EXPORTER_OTLP_* variables are read and a
malformed value fails startup.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jeffreyaven
jeffreyaven merged commit 15c83e7 into main Sep 26, 2026
30 checks passed
@jeffreyaven
jeffreyaven deleted the feat/desc-alias-and-otlp-push branch September 26, 2026 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Accept DESC as an alias for DESCRIBE

2 participants