Skip to content

fix(bigquery-jdbc): normalize timestamp string representation and improve temporal coercions - #14037

Open
keshavdandeva wants to merge 1 commit into
mainfrom
jdbc/timestamp-normalization
Open

fix(bigquery-jdbc): normalize timestamp string representation and improve temporal coercions#14037
keshavdandeva wants to merge 1 commit into
mainfrom
jdbc/timestamp-normalization

Conversation

@keshavdandeva

Copy link
Copy Markdown
Contributor

b/544843125

This PR standardizes TIMESTAMP string representation across both Arrow and JSON result sets and introduces a unified mathematical epoch parser for temporal coercions.

Problem Statement

  1. Engine Inconsistency in getString(): Previously, getString() on the JSON REST path returned raw numeric epoch decimal strings (e.g. "1408452095.22" or scientific notation "1.6905474E9"), diverging from the Arrow engine which returns formatted timestamp strings.
  2. Timezone Discrepancies: Legacy timestamp string formatting was sensitive to JVM-default timezones rather than adhering to UTC.
  3. Lossy Nanosecond Precision on JSON Path: FieldValueToTimestamp coerced timestamps through getTimestampValue(), truncating precision to microseconds and causing a single-second offset for pre-1970 negative epoch timestamps due to un-floored division.

Changes Made

  • TIMESTAMP String Normalization:

    • Updated BigQueryJsonResultSet.getString() to consistently format TIMESTAMP columns into standard UTC timestamp strings ("yyyy-MM-dd HH:mm:ss.ffffff"), matching the Arrow engine while safely delegating RECORD and REPEATED fields.
  • Unified Mathematical Temporal Parser:

    • Introduced BigQueryTemporalUtility.parseEpochDecimalToInstant() using BigDecimal mathematical flooring (RoundingMode.DOWN). This losslessly handles standard epoch decimals, scientific notation (e.g. "1.6905474E9"), whole integer epochs, and pre-1970 negative epoch decimals without sign-inversion or off-by-one errors.
    • Added formatTimestampStringFromMicroseconds() using primitive integer arithmetic (Math.floorDiv/Math.floorMod) for high-throughput streaming on Arrow paths.
  • Improved Temporal Coercion:

    • Updated FieldValueToTimestamp in BigQueryTypeCoercionUtility to delegate directly to parseEpochDecimalToInstant(), preserving up to nanosecond precision for getTimestamp() and getObject().
  • Resource Management:

    • Added try-with-resources in ArrowUtilities.serializeVectorSchemaRoot() to properly close ArrowRecordBatch and prevent buffer leaks in tests.

Testing

  • BigQueryTemporalUtilityTest: Added tests for standard epoch decimals, scientific notation, pre-1970 negative timestamps ("-1.5", "-0.123456"), and deterministic truncation (RoundingMode.DOWN).
  • FieldValueTypeBigQueryCoercionUtilityTest: Added tests for nanosecond precision preservation, scientific notation, pre-1970 negative epochs, and sub-second truncation.
  • BigQueryJsonResultSetTest: Verified that getString() returns standardized UTC formatted timestamp strings for scalar timestamps and handles RECORD/REPEATED fields.
  • BigQueryJsonArrayOfPrimitivesTest: Updated test fixture to exact 6-digit timestamp.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances the BigQuery JDBC driver's handling of TIMESTAMP values, specifically improving the parsing and formatting of numeric epoch decimal strings and supporting sub-microsecond precision. The review feedback highlights three key issues: a bug in BigQueryTypeCoercionUtility where negative epoch decimal strings incorrectly trigger the ISO/SQL format branch due to a leading hyphen; a bug in BigQueryTemporalUtility.boxTimestamp where Instant.parse fails due to a missing timezone offset and the fallback fails on trailing timezone suffixes; and a potential NumberFormatException in BigQueryJsonResultSet if the timestamp value is already formatted as an ISO/SQL string.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant