Add DateTime wrapper type for DATETIME round-trip support#6
Open
junyahondarubrik wants to merge 1 commit intomasterfrom
Open
Add DateTime wrapper type for DATETIME round-trip support#6junyahondarubrik wants to merge 1 commit intomasterfrom
junyahondarubrik wants to merge 1 commit intomasterfrom
Conversation
The BigQuery Go client maps time.Time to TIMESTAMP when used as a query parameter. This causes a type mismatch when DATETIME values read from BigQuery are passed back as parameters in subsequent queries (e.g., MERGE partition filtering). Introduce a DateTime wrapper type that: - Embeds time.Time for transparent consumer compatibility - Implements driver.Valuer returning time.Time for sql.Scan - Gets intercepted in buildParameter to convert back to civil.DateTime, ensuring BigQuery maps it to DATETIME JIRA: SPARK-717719
junyahondarubrik
force-pushed
the
datetime-wrapper-type
branch
from
4347aa0 to
ee2a275
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DateTimewrapper type indriver/columns.gothat embedstime.Timeto preserve BigQuery DATETIME (timezone-naive) semanticsConvertValuenow returnsDateTimeinstead of rawtime.Timefor DATETIME columnsbuildParameterandbuildParameterFromNamedValueinstatement.gointerceptDateTimeand convert it back tocivil.DateTime, ensuring BigQuery maps the parameter to DATETIME (not TIMESTAMP)DateTimeimplementsdriver.Valuerreturningtime.Time, sodatabase/sql.Scantransparently assigns it to*time.Timetargets — no consumer changes neededProblem
The BigQuery Go client always maps
time.TimetoTIMESTAMPwhen used as a query parameter. This causes a type mismatch when DATETIME values read from BigQuery are used in subsequent parameterized queries (e.g., MERGE statements with partition filtering).Test Plan
driver/columns_test.go: tests forDateTime.Value(),ConvertValuereturningDateTimefor DATETIME columns, DATE and TIME column handling, compile-timedriver.Valuercheckdriver/statement_test.go: tests forbuildParameterconvertingDateTimetocivil.DateTime, named/unnamed parameter paths, accumulation behaviorpolaris/src/rubrik/vendor/github.com/scaledata/bigquery/driver/) and verified that the downstream consumer (OLAP delta table merge korg job) builds and passes all tests with the newDateTimetypeAll 20 driver subtests pass. sdmain consumer tests pass via
bazel test //rubrik/turing/taskchains/olap-delta-table-merge/merge-tables:go_default_test.JIRA Issues
SPARK-717719