Json field names runtime support#1286
Conversation
jnthntatum
left a comment
There was a problem hiding this comment.
LGTM -- just suggest adding a test case.
This PR contains the following updates: | Package | Type | Update | Change | OpenSSF | |---|---|---|---|---| | [github.com/google/cel-go](https://github.com/google/cel-go) | require | minor | `v0.27.0` → `v0.28.0` | [](https://securityscorecards.dev/viewer/?uri=github.com/google/cel-go) | --- >⚠️ **Warning** > > Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/23) for more information. --- ### Release Notes <details> <summary>google/cel-go (github.com/google/cel-go)</summary> ### [`v0.28.0`](https://github.com/google/cel-go/releases/tag/v0.28.0) [Compare Source](google/cel-go@v0.27.0...v0.28.0) #### High-Level Changes - **Enhanced JSON Interoperability:** New support for JSON names across the checker, AST, and runtime allows for more seamless data handling when working with JSON-native structures. - **Improved Developer Tooling:** Integration is now smoother thanks to new utilities for converting Go errors into `cel.Issues` and more descriptive, context-aware error messages. - **Greater Environment Flexibility:** You can now redeclare variables as constants and export parse limit options, providing finer control over how CEL environments are configured and constrained. - **Native Struct Improvements:** Support for mixing CEL and native values within native structs simplifies the handling of complex, hybrid data types. *** #### 🚀 Features - Add helper method to check whether a function has a singleton binding in [#​1266](google/cel-go#1266) - Helper utility for converting a Go error into `cel.Issues` in [#​1267](google/cel-go#1267) - Policy API improvements in [#​1268](google/cel-go#1268) - CEL Test usability requirements in [#​1269](google/cel-go#1269) - Better context-related error messages in [#​1271](google/cel-go#1271) - Sort `env.Config` values where reasonable in [#​1273](google/cel-go#1273) - Support redeclaring variables as constants in `NewEnv` in [#​1275](google/cel-go#1275) - Add support for exporting parse limit options in [#​1277](google/cel-go#1277) - Support mixing CEL values and native values in native structs in [#​1270](google/cel-go#1270) - Add checker, AST, and type-provider support for JSON names in [#​1283](google/cel-go#1283) - JSON field names runtime support in [#​1286](google/cel-go#1286) - Optionally include reachable fieldpaths in prompt in [#​1285](google/cel-go#1285) - REPL -- cel-spec pb2 and json name support [#​1294](google/cel-go#1294) #### 🐞 Bug Fixes - Fix support for config-based type references in [#​1265](google/cel-go#1265) - Check arg kinds in `optional.or` and `.orValue` impl in [#​1276](google/cel-go#1276) - Bazel fixes for import in [#​1278](google/cel-go#1278) - Support zero-value literals in presence test inlining [#​1280](google/cel-go#1280) - Cache concatList.Size() to prevent O(N^2) evaluation time [#​1291](google/cel-go#1291) - Preserve runtime error node IDs from Resolve [#​1290](google/cel-go#1290) - Default enable identifier escaping with backticks [#​1295](google/cel-go#1295) - Cap format string precision to prevent memory exhaustion [#​1292](google/cel-go#1292) #### 🛠️ Maintenance & Internal - **chore:** Migrate gsutil usage to gcloud storage in [#​1274](google/cel-go#1274) - Lint fixes for exported function/type comments in [#​1279](google/cel-go#1279) - Lint fixes for import in [#​1287](google/cel-go#1287) *** **Full Changelog**: [https://github.com/google/cel-go/compare/v0.27.0...v0.28.0-alpha](https://github.com/google/cel-go/compare/v0.27.0...v0.28.0) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At 12:00 AM through 04:59 AM and 10:00 PM through 11:59 PM, Monday through Friday (`* 0-4,22-23 * * 1-5`) - Only on Sunday and Saturday (`* * * * 0,6`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMDQuNSIsInVwZGF0ZWRJblZlciI6IjQzLjEwNC41IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJLaW5kL0RlcGVuZGVuY2llcyJdfQ==--> Reviewed-on: https://altlinux.space/stapler/stplr/pulls/402 Co-authored-by: Renovate Bot <stapler-helper-bot@noreply.altlinux.space> Co-committed-by: Renovate Bot <stapler-helper-bot@noreply.altlinux.space>
|
@TristonianJones We've been needing this feature, and upon checking again, I was very happy to see it just so happened to have been implemented. However, unfortunately this seems like a hard switch, or am I missing something? We have stored CEL expressions in our system, and making it either-or means we'd have to migrate all expressions at the same time as rolling out enabling this feature. Is there any way opting into JSON name support can be made smoother, e.g. by allowing a mixed mode where either JSON name or field name is supported? |
|
@Airblader Old expressions should continue to work as-is since the JSON name support is only used at runtime if the expression was type-checked with JSON names enabled. Your existing stored and compiled expressions should continue to work. The test case below, reproduced from {
name: "json with proto fallback",
expr: `dyn(msg).single_int32 == dyn(msg).singleInt32`,
jsonFieldNames: true,
},If you have expressions that don't work with json names, you could always extend the existing environment via |
|
@TristonianJones Thanks, I appreciate the reply.
We basically do something akin to Kubernetes where we have CRDs that define CEL expressions for validation. We have existing CRDs using snake_case (= field names in the proto schema), but we'd like to move towards using JSON names as we use those for some other things, too. When I simply added in Interestingly, using Edit: I suppose the important detail implicitly mentioned here is that we have a custom registry. |
|
I am running this a bit through Claude, and it suggests a gap in the implementation. By no means do I want to mindlessly repeat that evaluation, I trust you folks more than my half-informed Claude conversation, but perhaps you can help me clear up this misunderstanding somehow? |
|
Sure. Missing context is that this was ported from cel-java -- that implementation made the decision to only resolve JSON names at check time to avoid ambiguous expressions like My comment that Claude picked up on was just a request to document the behavior, I think it was correct to behave the same as the Java implementation here. @l46kok fyi |
|
While it's technically possible to make the type-checker permissive to allow resolution of both styles, we made an explicit design choice to require an all-or-nothing flag. Mixing naming conventions in a single expression causes both ambiguity as @jnthntatum mentioned, and degrades readability over time (e.g., Another issue is collisions: expression: With the current implementation, the type-checker unambiguously resolves to field 1 or 2 depending on the json field name option. If the type-checker were to allow resolution of both styles, we would either have to make such resolution an error state (undesirable from UX perspective), or define a strict, convoluted fallback priority. A fallback priority is dangerous for a policy language: if a developer explicitly types the native name of field 2 (previous_status), a JSON-first fallback would silently intercept and evaluate field 1 instead. We'd much rather force a loud compile-time error than allow silent logical shadowing. Hopefully that provides some clarity. If you really wanted to allow both, type-checking the expression twice with and without the option set is a workaround. |
|
I appreciate your guys' time to reply to me. This all makes a ton of sense, and we definitely also want to get there (meaning using only JSON names), but we need a migration path that allows either field names or JSON names in a given expression, though not being able to mix both in the same expression is OK. In our specific case, collisions isn't something we are worried about, though of course a fair point. Our schemas today use Is using |
|
My advice @Airblader, would be to try the following: env, err := cel.NewEnv(...options w/o JSON names...)
jsonEnv, err := env.Extend(cel.JSONFieldNames(true))
ast, iss := jsonEnv.Compile(expr)
if iss.Err() != nil {
pbAst, pbIss := env.Compile(expr)
if pbIss.Err() == nil {
ast = pbAst
}
}You end up potentially compiling twice, but it also gives you the option to signal to your users (or internal processes) that an update to JSON names is available should they wish to translate to the new format (or have you translate for them) |
|
Got it, again, really appreciate the discussion and support. Thanks all! |
Support for JSON or proto-based field name accesses when the
JSONFieldNames(true)feature is enabled.