Repository navigation
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Ford544
left a comment
There was a problem hiding this comment.
I think we could really use a e2e integration test for this (e.g. in cli/index_run_e2e_test.ts or a new adjacent file) in addition to the unit tests, to make sure everything behaves as expected against real BigQuery.
| "When defining an incremental Lakehouse table, 'defaultDataset' must be defined in workflow_settings.yaml.", | ||
| ); | ||
| } | ||
| const scratchSchema = this.project?.schemaSuffix |
There was a problem hiding this comment.
This schema may not exist (for example if a new project only has Lakehouse targets). Consider registering the default schema for creation if there is an incremental Lakehouse target with onSchemaChange != IGNORE; or preceding the emitted CREATE OR REPLACE TABLE with CREATE SCHEMA IF NOT EXISTS
There was a problem hiding this comment.
+1. Additionally, this dataset has to be in the same location as the Lakehouse catalog, otherwise the CREATE TABLE ... AS SELECT reading from the catalog won't run. If it's going to be created automatically, it's worth using defaultLocation.
There was a problem hiding this comment.
Done in fbb8d2d — prepended CREATE SCHEMA IF NOT EXISTS (with OPTIONS(location=...) when defaultLocation is configured) before the BEGIN ... END block.
|
|
||
| public async setMetadata(action: dataform.IExecutionAction): Promise<void> { | ||
| const { target, actionDescriptor } = action; | ||
| if (isLakehouseTarget(target)) { |
There was a problem hiding this comment.
Would it be possible to set metadata for Lakehouse tables by running SQL directly instead of skipping it outright (like with delete)?
There was a problem hiding this comment.
+1 to setting descriptions via SQL inside setMetadata() in this PR so table and top-level column descriptions work consistently in dataform run (after task success, skipped on --dry-run).
Note that BigQuery does not allow mixing SET OPTIONS and ALTER COLUMN in a single ALTER TABLE statement, so it needs up to two statements (with properly escaped string literals, e.g. via CompilationSql.sqlString, and forwarding the run's execution location if set):
ALTER TABLE `project.catalog.namespace.table` SET OPTIONS(description = '...');
ALTER TABLE `project.catalog.namespace.table`
ALTER COLUMN IF EXISTS `col_a` SET OPTIONS(description = '...'),
ALTER COLUMN IF EXISTS `col_b` SET OPTIONS(description = '...');(ALTER COLUMN only supports top-level columns, i.e. path.length === 1.)
There was a problem hiding this comment.
Done in 0c41020 — BigQueryDbAdapter.setMetadata() now applies table and top-level column (path.length === 1) descriptions via separate ALTER TABLE ... SET OPTIONS and ALTER TABLE ... ALTER COLUMN IF EXISTS ... SET OPTIONS SQL statements (forwarding execution options / location), and the duplicate !isLakehouseTarget check in Runner.executeAction() has been removed.
|
On a somewhat-related note, if the defaultDatabase is not defined in either the target or workflow_settings.yaml, it will result in a three-part identifier (catalog.namespace.table) that BigQuery will interpret as project.schema.table. This should probably be caught at compile step ( in /core) so it's not directly relevant for this PR, but might be a good idea for a follow-up. |
| : this.project.defaultDatabase | ||
| : undefined; | ||
| return { | ||
| database: tableTarget.database || defaultDatabase, |
There was a problem hiding this comment.
The scratch dataset is defaultDataset, which lives in defaultProject, but the project here is taken from the Lakehouse table first. If the catalog is in a different project, this produces <catalogProject>.<defaultDataset>.<tmp>, which doesn't exist. I think the precedence should be flipped:
| database: tableTarget.database || defaultDatabase, | |
| database: defaultDatabase || tableTarget.database, |
A test with two different projects would be useful.
There was a problem hiding this comment.
+1 to using defaultDatabase here. Rather than falling back to tableTarget.database when defaultProject is unset, let's throw a clear error if this.project?.defaultDatabase is missing (just like we do above for defaultDataset), so scratch tables never accidentally get created in the catalog's project.
There was a problem hiding this comment.
Done in df5b2b0 — resolveScratchTarget() now always places the scratch table in defaultDatabase (with databaseSuffix), throws if defaultProject is unset, and includes unit tests for cross-project catalogs and missing defaultProject.
| const tempTableName = `${table.target.name}_df_temp_${uniqueId}_empty`; | ||
| const emptyTempTableTarget = this.resolveScratchTarget(table.target, tempTableName); | ||
| const emptyTempTableName = this.resolveTarget(emptyTempTableTarget); | ||
| const existingColumns = tableMetadata?.fields?.map((f) => f.name) || []; |
There was a problem hiding this comment.
|| [] conflates "the table has no columns" with "metadata had no schema". This PR itself guards metadata.schema?.fields in BigQueryDbAdapter.table(), so the schema may be missing; in that case every column looks added and FAIL always aborts the run. Better to throw explicitly when tableMetadata?.fields is missing.
Also, this list is a snapshot from build time, whereas the standard path reads INFORMATION_SCHEMA at execution time, so schema changes made by incrementalPreOps aren't visible here. Worth documenting in a comment.
There was a problem hiding this comment.
+1 — one small protobufjs gotcha when implementing this: dataform.TableMetadata.create() in BigQueryDbAdapter.table() initializes omitted repeated fields to [], so tableMetadata?.fields won't be undefined. Checking !tableMetadata?.fields?.length (since an existing table always has at least one column) will catch a missing schema reliably.
There was a problem hiding this comment.
Done in cdc1865 — we now check !tableMetadata?.fields?.length and throw an explicit error when schema fields are missing, and added a comment noting that existingColumns is a build-time snapshot (incrementalPreOps schema changes are not visible).
| if ( | ||
| hasTableMetadata(action.actionDescriptor) && | ||
| // BigQuery tables.patch rejects 4-part Lakehouse Iceberg tables. | ||
| !isLakehouseTarget(action.target) && |
There was a problem hiding this comment.
The same check exists in BigQueryDbAdapter.setMetadata() (bigquery.ts:416); I'd keep only one, in the adapter. Regardless of whether the SQL approach suggested above works out: as long as descriptions are skipped, core should at least warn at compile time that description / column descriptions aren't supported for Lakehouse targets (as it already does for labels).
There was a problem hiding this comment.
+1 on keeping the check in one place (BigQueryDbAdapter.setMetadata()). On @dataform/core: core is shared with Google Cloud Dataform, where table and column descriptions on Lakehouse tables are supported (and core only has compileError, no warnings), so core should not reject descriptions on Lakehouse targets. Setting them via ALTER TABLE SQL in BigQueryDbAdapter.setMetadata() avoids any discrepancy altogether.
There was a problem hiding this comment.
Done in 0c41020 — BigQueryDbAdapter.setMetadata() now applies table and top-level column (path.length === 1) descriptions via separate ALTER TABLE ... SET OPTIONS and ALTER TABLE ... ALTER COLUMN IF EXISTS ... SET OPTIONS SQL statements (forwarding execution options / location), and the duplicate !isLakehouseTarget check in Runner.executeAction() has been removed.
|
|
||
| public async deleteTable(target: dataform.ITarget): Promise<void> { | ||
| if (isLakehouseTarget(target)) { | ||
| const database = target.database || this.bigQueryCredentials.projectId; |
There was a problem hiding this comment.
Falling back to credentials.projectId is inconsistent with CompilationSql.resolveTarget() / Runner.targetDatabase(), which fall back to projectConfig.defaultDatabase. When defaultProject != the credentials project, the DROP hits a different table than the one written to. Related to @Ford544's general comment about a missing defaultDatabase: if core starts requiring a project for Lakehouse targets, this can simply throw instead of guessing.
There was a problem hiding this comment.
Done in f158431 — deleteTable() now throws an explicit error if target.database is missing instead of falling back to credentials.projectId.
| task: dataform.IExecutionTask, | ||
| ): IMissingSchema | null { | ||
| if (!action.target?.schema || !writesToTarget(action)) { | ||
| if (!action.target?.schema || isLakehouseTarget(action.target) || !writesToTarget(action)) { |
There was a problem hiding this comment.
nit: redundant, findMissingSchemas() already filters out catalog schemas. If it stays, use the same helper as there (isCatalogSchema).
There was a problem hiding this comment.
Done in 5d4c6e4 — removed the redundant isLakehouseTarget check from missingSchemaToCreate().
| * normalized 4-part Lakehouse target (`project.catalog.namespace.table`). | ||
| */ | ||
| export function isCatalogSchema(schema?: string | null): boolean { | ||
| return Boolean(schema && schema.includes(".")); |
There was a problem hiding this comment.
nit: the same schema.includes(".") rule exists in core (Session.finalizeSchema, core/session.ts:671). Better to have a single helper exported from core.
There was a problem hiding this comment.
Done in e9c10c3 — moved isCatalogSchema() and isLakehouseTarget() to core/targets.ts and reused isCatalogSchema() in Session.finalizeSchema().
| "When defining an incremental Lakehouse table, 'defaultDataset' must be defined in workflow_settings.yaml.", | ||
| ); | ||
| } | ||
| const scratchSchema = this.project?.schemaSuffix |
There was a problem hiding this comment.
+1. Additionally, this dataset has to be in the same location as the Lakehouse catalog, otherwise the CREATE TABLE ... AS SELECT reading from the catalog won't run. If it's going to be created automatically, it's worth using defaultLocation.
SergikL
left a comment
There was a problem hiding this comment.
Thanks for putting this together! Overall direction looks solid. +1 to the existing comments on the scratch dataset (project, existence, location) and verifying against a live Lakehouse catalog.
A couple of additional items that aren't in the current diff yet and would be good to address directly in this PR:
- Lineage (
cli/api/lineage/payload_builder.ts): When--emit-lineage(orlineage.enabledinworkflow_settings.yaml) is enabled,LineagePayloadBuildercurrently formats every target as a 3-part BigQuery dataset name (<project>.<schema>.<name>). For a 4-part Lakehouse target (project.catalog.namespace.table), Data Lineage parses that as tablenamespace.tablein projectproject.catalog. Could we skip Lakehouse targets ininputs, and omitinputs/outputswhen the action's output target itself is a Lakehouse target (plus a unit test inpayload_builder_test.tsand a brief note indocs/lineage.md)? - CLI documentation (
cli/README.md): Could we add a short note/section covering Lakehouse targets indataform run— specifically that the catalog/namespace must already exist, that incremental tables withonSchemaChangerequire bothdefaultProjectanddefaultDatasetin the same location as the catalog (with table create/drop permissions ondefaultDataset), and how table/column descriptions are applied?
|
|
||
| const baseTableType = this.baseTableType(table.enumType); | ||
| if (tableMetadata && tableMetadata.type !== baseTableType) { | ||
| if (!isLakehouseTarget(table.target) && tableMetadata && tableMetadata.type !== baseTableType) { |
There was a problem hiding this comment.
Right below at lines 148–150: for a Lakehouse incremental table, whenever tables.get reports no existing metadata (tableMetadata is null), !shouldWriteIncrementally(...) falls through to createOrReplace(table) — even when protected: true. If tables.get ever returns 404 for a Lakehouse table that actually exists in the catalog, CREATE OR REPLACE TABLE would silently overwrite the table and its history.
Could we emit plain CREATE TABLE (without OR REPLACE) for Lakehouse incremental tables unless runConfig.fullRefresh && !table.protected? That way, if the table already exists, BigQuery fails safely with Already Exists instead of replacing it.
There was a problem hiding this comment.
Done in c5badcd — incremental Lakehouse tables now emit plain CREATE TABLE unless runConfig.fullRefresh && !table.protected.
| ) { | ||
| const uniqueId = this.uniqueIdGenerator(); | ||
| const tempTableName = `${table.target.name}_df_temp_${uniqueId}_empty`; | ||
| const emptyTempTableTarget = this.resolveScratchTarget(table.target, tempTableName); |
There was a problem hiding this comment.
If a CLI run is cancelled or times out between CREATE OR REPLACE TABLE of the empty scratch table and DROP TABLE IF EXISTS, <table>_df_temp_<id>_empty will remain in defaultDataset indefinitely. Could we add a 1-day expiration option on the Lakehouse scratch table so orphaned scratch tables clean themselves up automatically?
OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 1 DAY))There was a problem hiding this comment.
Done in 1e2c0d4 — added OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)) to the Lakehouse scratch table DDL.
| !!action.target && | ||
| !!action.target.schema && | ||
| // Lakehouse catalog.namespace targets are managed externally via BigLake REST APIs | ||
| // and cannot be listed via datasets.list or created via CREATE SCHEMA. |
There was a problem hiding this comment.
nit on the comment: datasets.list actually can return Lakehouse namespaces (with id: "catalog.namespace" and type BIGLAKE_ICEBERG), though skipping here is still right because CREATE SCHEMA cannot create a Lakehouse namespace. Maybe: // Lakehouse catalog.namespace targets cannot be created via CREATE SCHEMA; they must exist before the run.
…form/core Move isCatalogSchema() and isLakehouseTarget() from cli/api/utils/lakehouse.ts to core/targets.ts and reuse isCatalogSchema() in Session.finalizeSchema(). Addresses #2445 (comment)
…e incremental tables Avoid overwriting an existing Lakehouse table if tables.get ever reports no metadata; only emit OR REPLACE when fullRefresh is set and the table is not protected. Addresses #2445 (comment)
… on schema change Check !tableMetadata?.fields?.length (accounting for protobufjs empty repeated field initialization) instead of defaulting to [], and document that existingColumns is a build-time snapshot. Addresses: - #2445 (comment) - #2445 (comment)
…re defaultDatabase Always create the temporary schema-inspection table in defaultDatabase (with databaseSuffix) instead of the catalog's project, and throw when defaultProject is unset. Addresses: - #2445 (comment) - #2445 (comment)
…house onSchemaChange Prepend CREATE SCHEMA IF NOT EXISTS (using project defaultLocation when configured) before the Lakehouse schema-change block so projects with only Lakehouse targets have the scratch dataset created in the catalog's region. Addresses: - #2445 (comment) - #2445 (comment)
…scratch tables Ensure orphaned scratch tables in defaultDataset are automatically cleaned up if a run times out or is cancelled before DROP TABLE IF EXISTS executes. Addresses #2445 (comment)
…r Lakehouse targets Throw explicitly when target.database is missing instead of falling back to credentials.projectId, avoiding dropping a table in the wrong project when defaultProject differs from credentials. Addresses #2445 (comment)
…BLE SQL - Execute separate ALTER TABLE SET OPTIONS and ALTER COLUMN IF EXISTS statements in BigQueryDbAdapter.setMetadata() for 4-part Lakehouse targets, forwarding execution location. - Keep Lakehouse metadata handling solely in BigQueryDbAdapter instead of duplicating the check in Runner.executeAction(). Addresses: - #2445 (comment) - #2445 (comment) - #2445 (comment) - #2445 (comment)
…unner - Remove redundant isLakehouseTarget() guard in missingSchemaToCreate() since findMissingSchemas() already excludes catalog schemas. - Clarify that datasets.list can list BIGLAKE_ICEBERG namespaces, while CREATE SCHEMA cannot create them. Addresses: - #2445 (comment) - #2445 (comment)
… and outputs Data Lineage parses bigquery namespace dataset names as 3-part <project>.<dataset>.<table> identifiers. Filter out 4-part Lakehouse targets from inputs and omit inputs/outputs when the action target itself is a Lakehouse target. Addresses #2445 (review)
…E.md Document catalog/namespace pre-existence, defaultProject/defaultDataset location and permission requirements for onSchemaChange, and SQL description updates. Addresses #2445 (review)
|
Thank you for the review @SergikL! Addressed both additional items as well:
|
| const baseDefaultSchema = this.project?.defaultSchema; | ||
| if (!baseDefaultSchema) { | ||
| throw new Error( | ||
| "When defining an incremental Lakehouse table, 'defaultDataset' must be defined in workflow_settings.yaml.", |
There was a problem hiding this comment.
nit: unlike the error message above, this one does not clarify that this affects onSchemaChange.
| setMetadata( | ||
| action: dataform.IExecutionAction, | ||
| options?: { | ||
| bigquery?: { |
There was a problem hiding this comment.
nit: I think we can reuse the IBigQueryExecutionOptions interface here instead of typing out all the properties.
Summary
Adds
@dataform/cliexecution support (dataform run/dfapi.build/dfapi.run) for 4-part BigQuery Lakehouse Catalog (BigLake Iceberg REST Catalog) table identifiers (project.catalog.namespace.table), building on the@dataform/corecompilation and target normalization introduced in #2384.Problem
In #2384,
@dataform/corenormalizes 4-part Lakehouse targets intodataform.Targetby packingcatalogandnamespaceintotarget.schema("${catalog}.${namespace}"), which interpolates as`project.catalog.namespace.table`in GoogleSQL. However, executing these targets in@dataform/clifailed across several execution stages:Runner):Runner.findMissingSchemas()listed standard BigQuery datasets viadatasets.listand treatedcatalog.namespaceas a missing dataset, wrapping tasks inCREATE SCHEMA IF NOT EXISTS \project.catalog.namespace`. Lakehouse namespaces are managed externally via BigLake REST APIs and rejectCREATE SCHEMA` DDL.DROP VIEWandonSchemaChange(ExecutionSql):publishTasks()could emitDROP VIEW IF EXISTSon type mismatch, whereas views are unsupported in Lakehouse namespaces.onSchemaChange(FAIL,EXTEND,SYNCHRONIZE),buildIncrementalSchemaChangeTasks()attempted to create a stored procedure (df_osc_<id>) insidecatalog.namespaceand query 5-partproject.catalog.namespace.INFORMATION_SCHEMA.COLUMNS, neither of which BigQuery supports for Lakehouse namespaces.BigQueryDbAdapter&Runner):tables.getresponses may omitlastModifiedTime, causingLong.fromString(metadata.lastModifiedTime)inBigQueryDbAdapter.table()to throw duringstate()resolution, and federated 404 responses may reportcode: 404without a legacyerrorsarray.BigQueryDbAdapter.setMetadata()(tables.patch) is rejected by BigQuery on 4-part Lakehouse Iceberg tables ("Connection is required for BigQuery tables for Apache Iceberg").BigQueryDbAdapter.deleteTable()used the RESTtables.deleteendpoint instead of 4-partDROP TABLE IF EXISTSSQL.--actionsand--output-actionsoption descriptions only mentioned 3-partdatabase.schema.nametargets.Changes
cli/api/utils/lakehouse.ts: AddisCatalogSchema()andisLakehouseTarget()helpers to identify normalized 4-part Lakehouse targets (schema.includes(".")).cli/api/commands/run.ts:catalog.namespaceschemas fromRunner.findMissingSchemas()andRunner.missingSchemaToCreate()sodatasets.listis skipped when all targets in a project are Lakehouse targets and tasks are never wrapped withCREATE SCHEMA IF NOT EXISTS.dbadapter.setMetadata()inRunner.executeAction()for 4-part Lakehouse targets.cli/api/dbadapters/execution_sql.ts:dropIfExists(DROP VIEW IF EXISTS) on 4-part targets inpublishTasks().buildLakehouseIncrementalSchemaChangeTasks()andresolveScratchTarget()for incremental Lakehouse tables withonSchemaChange(FAIL,EXTEND,SYNCHRONIZE): execute aBEGIN ... EXCEPTION WHEN ERROR ... ENDscript block without stored procedures, populatedataform_columnsdirectly fromtableMetadata.fields, inspect the new query schema via a short-lived emptyLIMIT 0table indefaultSchema(respectingschemaSuffixanddatabaseSuffix), and clean up the scratch table in both normal and error paths.cli/api/dbadapters/bigquery.ts:lastUpdatedMillistoLong.ZEROwhenmetadata.lastModifiedTimeis absent and guardmetadata.schema?.fieldsintable().e?.code === 404as not-found ingetMetadata()(matchingtables()).setMetadata()to return early for 4-part Lakehouse targets.DROP TABLE IF EXISTSviathis.execute()indeleteTable()for Lakehouse targets (falling back tocredentials.projectIdwhentarget.databaseis unset).cli/common_options.ts,cli/commands/compile_options.ts,cli/README.md: Document 4-partdatabase.catalog.namespace.nametargets in--actionsand--output-actionsand regenerate the CLI reference tables.Testing
cli/api/dbadapters/bigquery_test.ts: Unit tests fortable()(dotteddatasetId, missinglastModifiedTime, 404 handling),setMetadata()skippingtables.patch, anddeleteTable()executing 4-partDROP TABLE IF EXISTS.cli/api/execution_sql_test.ts: Unit tests for 4-parttableDDL (skippingDROP VIEW),incrementalfirst run /fullRefresh,INSERT/MERGE/INSERT_OVERWRITEDML,onSchemaChange(FAIL,EXTENDwithschemaSuffix,SYNCHRONIZE), andoperations.cli/api/commands/run_test.ts: Unit tests verifyingRunnerskipsschemas()andCREATE SCHEMAwrappers (in both normal and--dry-runmodes), handles mixed 3-part and 4-part graphs, resolves warehouse state viabuild(), and skipssetMetadata()on 4-part targets.cli/api/commands/prune_test.ts: Unit tests verifying--actionsfiltering for 4-partdatabase.catalog.namespace.nametargets, bare action names, and upstream/downstream dependency inclusion (includeDependencies,includeDependents)../scripts/lintand./scripts/run_testspass.