Skip to content

CLI: support 4-part Lakehouse Catalog targets in dataform run - #2445

Draft
Pawbag123 wants to merge 21 commits into
mainfrom
fix-cli-lakehouse-4part-targets
Draft

Pawbag123 wants to merge 21 commits into
mainfrom
fix-cli-lakehouse-4part-targets

Conversation

@Pawbag123

Copy link
Copy Markdown

Summary

Adds @dataform/cli execution support (dataform run / dfapi.build / dfapi.run) for 4-part BigQuery Lakehouse Catalog (BigLake Iceberg REST Catalog) table identifiers (project.catalog.namespace.table), building on the @dataform/core compilation and target normalization introduced in #2384.

Problem

In #2384, @dataform/core normalizes 4-part Lakehouse targets into dataform.Target by packing catalog and namespace into target.schema ("${catalog}.${namespace}"), which interpolates as `project.catalog.namespace.table` in GoogleSQL. However, executing these targets in @dataform/cli failed across several execution stages:

  1. Lazy dataset creation (Runner): Runner.findMissingSchemas() listed standard BigQuery datasets via datasets.list and treated catalog.namespace as a missing dataset, wrapping tasks in CREATE SCHEMA IF NOT EXISTS \project.catalog.namespace`. Lakehouse namespaces are managed externally via BigLake REST APIs and reject CREATE SCHEMA` DDL.
  2. Incompatible-type DROP VIEW and onSchemaChange (ExecutionSql):
    • publishTasks() could emit DROP VIEW IF EXISTS on type mismatch, whereas views are unsupported in Lakehouse namespaces.
    • For incremental tables with onSchemaChange (FAIL, EXTEND, SYNCHRONIZE), buildIncrementalSchemaChangeTasks() attempted to create a stored procedure (df_osc_<id>) inside catalog.namespace and query 5-part project.catalog.namespace.INFORMATION_SCHEMA.COLUMNS, neither of which BigQuery supports for Lakehouse namespaces.
  3. Warehouse state, metadata patching, and table deletion (BigQueryDbAdapter & Runner):
    • Unified Iceberg / Lakehouse tables.get responses may omit lastModifiedTime, causing Long.fromString(metadata.lastModifiedTime) in BigQueryDbAdapter.table() to throw during state() resolution, and federated 404 responses may report code: 404 without a legacy errors array.
    • Post-execution BigQueryDbAdapter.setMetadata() (tables.patch) is rejected by BigQuery on 4-part Lakehouse Iceberg tables ("Connection is required for BigQuery tables for Apache Iceberg").
    • BigQueryDbAdapter.deleteTable() used the REST tables.delete endpoint instead of 4-part DROP TABLE IF EXISTS SQL.
  4. CLI documentation: --actions and --output-actions option descriptions only mentioned 3-part database.schema.name targets.

Changes

  • cli/api/utils/lakehouse.ts: Add isCatalogSchema() and isLakehouseTarget() helpers to identify normalized 4-part Lakehouse targets (schema.includes(".")).
  • cli/api/commands/run.ts:
    • Exclude compound catalog.namespace schemas from Runner.findMissingSchemas() and Runner.missingSchemaToCreate() so datasets.list is skipped when all targets in a project are Lakehouse targets and tasks are never wrapped with CREATE SCHEMA IF NOT EXISTS.
    • Skip post-job dbadapter.setMetadata() in Runner.executeAction() for 4-part Lakehouse targets.
  • cli/api/dbadapters/execution_sql.ts:
    • Skip incompatible-type dropIfExists (DROP VIEW IF EXISTS) on 4-part targets in publishTasks().
    • Add buildLakehouseIncrementalSchemaChangeTasks() and resolveScratchTarget() for incremental Lakehouse tables with onSchemaChange (FAIL, EXTEND, SYNCHRONIZE): execute a BEGIN ... EXCEPTION WHEN ERROR ... END script block without stored procedures, populate dataform_columns directly from tableMetadata.fields, inspect the new query schema via a short-lived empty LIMIT 0 table in defaultSchema (respecting schemaSuffix and databaseSuffix), and clean up the scratch table in both normal and error paths.
  • cli/api/dbadapters/bigquery.ts:
    • Default lastUpdatedMillis to Long.ZERO when metadata.lastModifiedTime is absent and guard metadata.schema?.fields in table().
    • Treat e?.code === 404 as not-found in getMetadata() (matching tables()).
    • Guard setMetadata() to return early for 4-part Lakehouse targets.
    • Execute 4-part DROP TABLE IF EXISTS via this.execute() in deleteTable() for Lakehouse targets (falling back to credentials.projectId when target.database is unset).
  • cli/common_options.ts, cli/commands/compile_options.ts, cli/README.md: Document 4-part database.catalog.namespace.name targets in --actions and --output-actions and regenerate the CLI reference tables.

Testing

  • cli/api/dbadapters/bigquery_test.ts: Unit tests for table() (dotted datasetId, missing lastModifiedTime, 404 handling), setMetadata() skipping tables.patch, and deleteTable() executing 4-part DROP TABLE IF EXISTS.
  • cli/api/execution_sql_test.ts: Unit tests for 4-part table DDL (skipping DROP VIEW), incremental first run / fullRefresh, INSERT / MERGE / INSERT_OVERWRITE DML, onSchemaChange (FAIL, EXTEND with schemaSuffix, SYNCHRONIZE), and operations.
  • cli/api/commands/run_test.ts: Unit tests verifying Runner skips schemas() and CREATE SCHEMA wrappers (in both normal and --dry-run modes), handles mixed 3-part and 4-part graphs, resolves warehouse state via build(), and skips setMetadata() on 4-part targets.
  • cli/api/commands/prune_test.ts: Unit tests verifying --actions filtering for 4-part database.catalog.namespace.name targets, bare action names, and upstream/downstream dependency inclusion (includeDependencies, includeDependents).
  • Verified ./scripts/lint and ./scripts/run_tests pass.

@google-cla

google-cla Bot commented Nov 13, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@Ford544 Ford544 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could really use a e2e integration test for this (e.g. in cli/index_run_e2e_test.ts or a new adjacent file) in addition to the unit tests, to make sure everything behaves as expected against real BigQuery.

"When defining an incremental Lakehouse table, 'defaultDataset' must be defined in workflow_settings.yaml.",
);
}
const scratchSchema = this.project?.schemaSuffix

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This schema may not exist (for example if a new project only has Lakehouse targets). Consider registering the default schema for creation if there is an incremental Lakehouse target with onSchemaChange != IGNORE; or preceding the emitted CREATE OR REPLACE TABLE with CREATE SCHEMA IF NOT EXISTS

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Additionally, this dataset has to be in the same location as the Lakehouse catalog, otherwise the CREATE TABLE ... AS SELECT reading from the catalog won't run. If it's going to be created automatically, it's worth using defaultLocation.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in fbb8d2d — prepended CREATE SCHEMA IF NOT EXISTS (with OPTIONS(location=...) when defaultLocation is configured) before the BEGIN ... END block.


public async setMetadata(action: dataform.IExecutionAction): Promise<void> {
const { target, actionDescriptor } = action;
if (isLakehouseTarget(target)) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to set metadata for Lakehouse tables by running SQL directly instead of skipping it outright (like with delete)?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to setting descriptions via SQL inside setMetadata() in this PR so table and top-level column descriptions work consistently in dataform run (after task success, skipped on --dry-run).

Note that BigQuery does not allow mixing SET OPTIONS and ALTER COLUMN in a single ALTER TABLE statement, so it needs up to two statements (with properly escaped string literals, e.g. via CompilationSql.sqlString, and forwarding the run's execution location if set):

ALTER TABLE `project.catalog.namespace.table` SET OPTIONS(description = '...');
ALTER TABLE `project.catalog.namespace.table`
  ALTER COLUMN IF EXISTS `col_a` SET OPTIONS(description = '...'),
  ALTER COLUMN IF EXISTS `col_b` SET OPTIONS(description = '...');

(ALTER COLUMN only supports top-level columns, i.e. path.length === 1.)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0c41020 — BigQueryDbAdapter.setMetadata() now applies table and top-level column (path.length === 1) descriptions via separate ALTER TABLE ... SET OPTIONS and ALTER TABLE ... ALTER COLUMN IF EXISTS ... SET OPTIONS SQL statements (forwarding execution options / location), and the duplicate !isLakehouseTarget check in Runner.executeAction() has been removed.

@Ford544

Ford544 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

On a somewhat-related note, if the defaultDatabase is not defined in either the target or workflow_settings.yaml, it will result in a three-part identifier (catalog.namespace.table) that BigQuery will interpret as project.schema.table.

This should probably be caught at compile step ( in /core) so it's not directly relevant for this PR, but might be a good idea for a follow-up.

Comment thread cli/api/dbadapters/execution_sql.ts Outdated
: this.project.defaultDatabase
: undefined;
return {
database: tableTarget.database || defaultDatabase,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scratch dataset is defaultDataset, which lives in defaultProject, but the project here is taken from the Lakehouse table first. If the catalog is in a different project, this produces <catalogProject>.<defaultDataset>.<tmp>, which doesn't exist. I think the precedence should be flipped:

Suggested change
database: tableTarget.database || defaultDatabase,
database: defaultDatabase || tableTarget.database,

A test with two different projects would be useful.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to using defaultDatabase here. Rather than falling back to tableTarget.database when defaultProject is unset, let's throw a clear error if this.project?.defaultDatabase is missing (just like we do above for defaultDataset), so scratch tables never accidentally get created in the catalog's project.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in df5b2b0 — resolveScratchTarget() now always places the scratch table in defaultDatabase (with databaseSuffix), throws if defaultProject is unset, and includes unit tests for cross-project catalogs and missing defaultProject.

Comment thread cli/api/dbadapters/execution_sql.ts Outdated
const tempTableName = `${table.target.name}_df_temp_${uniqueId}_empty`;
const emptyTempTableTarget = this.resolveScratchTarget(table.target, tempTableName);
const emptyTempTableName = this.resolveTarget(emptyTempTableTarget);
const existingColumns = tableMetadata?.fields?.map((f) => f.name) || [];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

|| [] conflates "the table has no columns" with "metadata had no schema". This PR itself guards metadata.schema?.fields in BigQueryDbAdapter.table(), so the schema may be missing; in that case every column looks added and FAIL always aborts the run. Better to throw explicitly when tableMetadata?.fields is missing.

Also, this list is a snapshot from build time, whereas the standard path reads INFORMATION_SCHEMA at execution time, so schema changes made by incrementalPreOps aren't visible here. Worth documenting in a comment.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 — one small protobufjs gotcha when implementing this: dataform.TableMetadata.create() in BigQueryDbAdapter.table() initializes omitted repeated fields to [], so tableMetadata?.fields won't be undefined. Checking !tableMetadata?.fields?.length (since an existing table always has at least one column) will catch a missing schema reliably.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in cdc1865 — we now check !tableMetadata?.fields?.length and throw an explicit error when schema fields are missing, and added a comment noting that existingColumns is a build-time snapshot (incrementalPreOps schema changes are not visible).

Comment thread cli/api/commands/run.ts Outdated
if (
hasTableMetadata(action.actionDescriptor) &&
// BigQuery tables.patch rejects 4-part Lakehouse Iceberg tables.
!isLakehouseTarget(action.target) &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same check exists in BigQueryDbAdapter.setMetadata() (bigquery.ts:416); I'd keep only one, in the adapter. Regardless of whether the SQL approach suggested above works out: as long as descriptions are skipped, core should at least warn at compile time that description / column descriptions aren't supported for Lakehouse targets (as it already does for labels).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on keeping the check in one place (BigQueryDbAdapter.setMetadata()). On @dataform/core: core is shared with Google Cloud Dataform, where table and column descriptions on Lakehouse tables are supported (and core only has compileError, no warnings), so core should not reject descriptions on Lakehouse targets. Setting them via ALTER TABLE SQL in BigQueryDbAdapter.setMetadata() avoids any discrepancy altogether.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0c41020 — BigQueryDbAdapter.setMetadata() now applies table and top-level column (path.length === 1) descriptions via separate ALTER TABLE ... SET OPTIONS and ALTER TABLE ... ALTER COLUMN IF EXISTS ... SET OPTIONS SQL statements (forwarding execution options / location), and the duplicate !isLakehouseTarget check in Runner.executeAction() has been removed.

Comment thread cli/api/dbadapters/bigquery.ts Outdated

public async deleteTable(target: dataform.ITarget): Promise<void> {
if (isLakehouseTarget(target)) {
const database = target.database || this.bigQueryCredentials.projectId;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Falling back to credentials.projectId is inconsistent with CompilationSql.resolveTarget() / Runner.targetDatabase(), which fall back to projectConfig.defaultDatabase. When defaultProject != the credentials project, the DROP hits a different table than the one written to. Related to @Ford544's general comment about a missing defaultDatabase: if core starts requiring a project for Lakehouse targets, this can simply throw instead of guessing.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f158431 — deleteTable() now throws an explicit error if target.database is missing instead of falling back to credentials.projectId.

Comment thread cli/api/commands/run.ts Outdated
task: dataform.IExecutionTask,
): IMissingSchema | null {
if (!action.target?.schema || !writesToTarget(action)) {
if (!action.target?.schema || isLakehouseTarget(action.target) || !writesToTarget(action)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: redundant, findMissingSchemas() already filters out catalog schemas. If it stays, use the same helper as there (isCatalogSchema).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5d4c6e4 — removed the redundant isLakehouseTarget check from missingSchemaToCreate().

Comment thread cli/api/utils/lakehouse.ts Outdated
* normalized 4-part Lakehouse target (`project.catalog.namespace.table`).
*/
export function isCatalogSchema(schema?: string | null): boolean {
return Boolean(schema && schema.includes("."));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the same schema.includes(".") rule exists in core (Session.finalizeSchema, core/session.ts:671). Better to have a single helper exported from core.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e9c10c3 — moved isCatalogSchema() and isLakehouseTarget() to core/targets.ts and reused isCatalogSchema() in Session.finalizeSchema().

"When defining an incremental Lakehouse table, 'defaultDataset' must be defined in workflow_settings.yaml.",
);
}
const scratchSchema = this.project?.schemaSuffix

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Additionally, this dataset has to be in the same location as the Lakehouse catalog, otherwise the CREATE TABLE ... AS SELECT reading from the catalog won't run. If it's going to be created automatically, it's worth using defaultLocation.

@SergikL SergikL left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together! Overall direction looks solid. +1 to the existing comments on the scratch dataset (project, existence, location) and verifying against a live Lakehouse catalog.

A couple of additional items that aren't in the current diff yet and would be good to address directly in this PR:

  1. Lineage (cli/api/lineage/payload_builder.ts): When --emit-lineage (or lineage.enabled in workflow_settings.yaml) is enabled, LineagePayloadBuilder currently formats every target as a 3-part BigQuery dataset name (<project>.<schema>.<name>). For a 4-part Lakehouse target (project.catalog.namespace.table), Data Lineage parses that as table namespace.table in project project.catalog. Could we skip Lakehouse targets in inputs, and omit inputs/outputs when the action's output target itself is a Lakehouse target (plus a unit test in payload_builder_test.ts and a brief note in docs/lineage.md)?
  2. CLI documentation (cli/README.md): Could we add a short note/section covering Lakehouse targets in dataform run — specifically that the catalog/namespace must already exist, that incremental tables with onSchemaChange require both defaultProject and defaultDataset in the same location as the catalog (with table create/drop permissions on defaultDataset), and how table/column descriptions are applied?


const baseTableType = this.baseTableType(table.enumType);
if (tableMetadata && tableMetadata.type !== baseTableType) {
if (!isLakehouseTarget(table.target) && tableMetadata && tableMetadata.type !== baseTableType) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right below at lines 148–150: for a Lakehouse incremental table, whenever tables.get reports no existing metadata (tableMetadata is null), !shouldWriteIncrementally(...) falls through to createOrReplace(table) — even when protected: true. If tables.get ever returns 404 for a Lakehouse table that actually exists in the catalog, CREATE OR REPLACE TABLE would silently overwrite the table and its history.

Could we emit plain CREATE TABLE (without OR REPLACE) for Lakehouse incremental tables unless runConfig.fullRefresh && !table.protected? That way, if the table already exists, BigQuery fails safely with Already Exists instead of replacing it.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in c5badcd — incremental Lakehouse tables now emit plain CREATE TABLE unless runConfig.fullRefresh && !table.protected.

Comment thread cli/api/dbadapters/execution_sql.ts Outdated
) {
const uniqueId = this.uniqueIdGenerator();
const tempTableName = `${table.target.name}_df_temp_${uniqueId}_empty`;
const emptyTempTableTarget = this.resolveScratchTarget(table.target, tempTableName);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a CLI run is cancelled or times out between CREATE OR REPLACE TABLE of the empty scratch table and DROP TABLE IF EXISTS, <table>_df_temp_<id>_empty will remain in defaultDataset indefinitely. Could we add a 1-day expiration option on the Lakehouse scratch table so orphaned scratch tables clean themselves up automatically?

OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 1 DAY))

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 1e2c0d4 — added OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)) to the Lakehouse scratch table DDL.

Comment thread cli/api/commands/run.ts Outdated
!!action.target &&
!!action.target.schema &&
// Lakehouse catalog.namespace targets are managed externally via BigLake REST APIs
// and cannot be listed via datasets.list or created via CREATE SCHEMA.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit on the comment: datasets.list actually can return Lakehouse namespaces (with id: "catalog.namespace" and type BIGLAKE_ICEBERG), though skipping here is still right because CREATE SCHEMA cannot create a Lakehouse namespace. Maybe: // Lakehouse catalog.namespace targets cannot be created via CREATE SCHEMA; they must exist before the run.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5d4c6e4 — updated the comment as suggested.

…form/core

Move isCatalogSchema() and isLakehouseTarget() from cli/api/utils/lakehouse.ts to core/targets.ts and reuse isCatalogSchema() in Session.finalizeSchema().
Addresses #2445 (comment)
…e incremental tables

Avoid overwriting an existing Lakehouse table if tables.get ever reports no metadata; only emit OR REPLACE when fullRefresh is set and the table is not protected.
Addresses #2445 (comment)
… on schema change

Check !tableMetadata?.fields?.length (accounting for protobufjs empty repeated field initialization) instead of defaulting to [], and document that existingColumns is a build-time snapshot.

Addresses:
- #2445 (comment)
- #2445 (comment)
…re defaultDatabase

Always create the temporary schema-inspection table in defaultDatabase (with databaseSuffix) instead of the catalog's project, and throw when defaultProject is unset.
Addresses:
- #2445 (comment)
- #2445 (comment)
…house onSchemaChange

Prepend CREATE SCHEMA IF NOT EXISTS (using project defaultLocation when configured) before the Lakehouse schema-change block so projects with only Lakehouse targets have the scratch dataset created in the catalog's region.
Addresses:
- #2445 (comment)
- #2445 (comment)
…scratch tables

Ensure orphaned scratch tables in defaultDataset are automatically cleaned up if a run times out or is cancelled before DROP TABLE IF EXISTS executes.
Addresses #2445 (comment)
…r Lakehouse targets

Throw explicitly when target.database is missing instead of falling back to credentials.projectId, avoiding dropping a table in the wrong project when defaultProject differs from credentials.
Addresses #2445 (comment)
…BLE SQL

- Execute separate ALTER TABLE SET OPTIONS and ALTER COLUMN IF EXISTS statements in BigQueryDbAdapter.setMetadata() for 4-part Lakehouse targets, forwarding execution location.
- Keep Lakehouse metadata handling solely in BigQueryDbAdapter instead of duplicating the check in Runner.executeAction().

Addresses:
- #2445 (comment)
- #2445 (comment)
- #2445 (comment)
- #2445 (comment)
…unner

- Remove redundant isLakehouseTarget() guard in missingSchemaToCreate() since findMissingSchemas() already excludes catalog schemas.
- Clarify that datasets.list can list BIGLAKE_ICEBERG namespaces, while CREATE SCHEMA cannot create them.
Addresses:
- #2445 (comment)
- #2445 (comment)
… and outputs

Data Lineage parses bigquery namespace dataset names as 3-part <project>.<dataset>.<table> identifiers. Filter out 4-part Lakehouse targets from inputs and omit inputs/outputs when the action target itself is a Lakehouse target.
Addresses #2445 (review)
…E.md

Document catalog/namespace pre-existence, defaultProject/defaultDataset location and permission requirements for onSchemaChange, and SQL description updates.

Addresses #2445 (review)
@Pawbag123

Copy link
Copy Markdown
Author

Thank you for the review @SergikL! Addressed both additional items as well:

  1. Lineage (cli/api/lineage/payload_builder.ts, payload_builder_test.ts, docs/lineage.md): Done in cc8dbce — Lakehouse targets are filtered out from inputs, and inputs/outputs are omitted when the action's own target is a Lakehouse target.
  2. CLI documentation (cli/README.md): Done in b171e5a — added a note under dataform run covering catalog/namespace pre-existence, defaultProject + defaultDataset location/permission requirements for onSchemaChange, and SQL-based table/column descriptions.

const baseDefaultSchema = this.project?.defaultSchema;
if (!baseDefaultSchema) {
throw new Error(
"When defining an incremental Lakehouse table, 'defaultDataset' must be defined in workflow_settings.yaml.",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: unlike the error message above, this one does not clarify that this affects onSchemaChange.

setMetadata(
action: dataform.IExecutionAction,
options?: {
bigquery?: {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think we can reuse the IBigQueryExecutionOptions interface here instead of typing out all the properties.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants