Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
acb79f2
feat(cli): add helper utilities to identify 4-part Lakehouse targets
Pawbag123 Nov 13, 2026
6c90224
feat(cli): support 4-part Lakehouse targets in BigQueryDbAdapter
Pawbag123 Nov 13, 2026
349350f
feat(cli): support 4-part Lakehouse targets and onSchemaChange in Exe…
Pawbag123 Nov 13, 2026
3d99e05
feat(cli): skip schema creation and metadata patching for Lakehouse t…
Pawbag123 Nov 13, 2026
94b7390
docs(cli): document 4-part Lakehouse targets in --actions and --outpu…
Pawbag123 Nov 13, 2026
1d69c91
test(cli): add BigQueryDbAdapter unit tests for 4-part Lakehouse targets
Pawbag123 Nov 13, 2026
690a6d7
test(cli): add ExecutionSql unit tests for 4-part Lakehouse targets
Pawbag123 Nov 13, 2026
fbfd75a
test(cli): add Runner and build unit tests for 4-part Lakehouse targets
Pawbag123 Nov 13, 2026
a98387c
test(cli): verify action pruning for 4-part Lakehouse targets
Pawbag123 Nov 13, 2026
2cd4bd0
style(cli): fix trailing whitespace in execution_sql.ts
Pawbag123 Nov 13, 2026
e9c10c3
refactor(core,cli): export shared Lakehouse target helpers from @data…
Pawbag123 Nov 17, 2026
c5badcd
fix(cli): emit plain CREATE TABLE on first run and protected Lakehous…
Pawbag123 Nov 17, 2026
cdc1865
fix(cli): throw when Lakehouse tableMetadata is missing schema fields…
Pawbag123 Nov 17, 2026
df5b2b0
fix(cli): resolve Lakehouse scratch table in defaultProject and requi…
Pawbag123 Nov 17, 2026
fbb8d2d
fix(cli): ensure scratch dataset exists with defaultLocation for Lake…
Pawbag123 Nov 17, 2026
1e2c0d4
fix(cli): set 1-day expiration_timestamp on Lakehouse onSchemaChange …
Pawbag123 Nov 17, 2026
f158431
fix(cli): require target.database in BigQueryDbAdapter.deleteTable fo…
Pawbag123 Nov 17, 2026
0c41020
feat(cli): apply Lakehouse table and column descriptions via ALTER TA…
Pawbag123 Nov 17, 2026
5d4c6e4
chore(cli): remove redundant Lakehouse check and clarify comment in R…
Pawbag123 Nov 17, 2026
cc8dbce
fix(cli): skip 4-part Lakehouse targets in OpenLineage dataset inputs…
Pawbag123 Nov 17, 2026
b171e5a
docs(cli): document 4-part Lakehouse target requirements in cli/READM…
Pawbag123 Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ Compiles the project and prints a summary (or JSON / Graphviz DOT representation
| `--quiet` | boolean | `false` | Less verbose compilation output. Mutually exclusive with --verbose. |
| `--watch` | boolean | `false` | Watches the project directory and recompiles on changes. |
| `--timeout` | string | `'60s'` | Duration to allow project compilation to complete, e.g. '1s', '10m'. |
| `--output-actions` | array | — | Filters the printed output to these actions (comma-separated or repeatable). Each entry is an action name or a fully-qualified database.schema.name target. The whole project is still compiled. |
| `--output-actions` | array | — | Filters the printed output to these actions (comma-separated or repeatable). Each entry is an action name or a fully-qualified database.schema.name (or 4-part database.catalog.namespace.name) target. The whole project is still compiled. |
| `--output-tags` | array | — | Filters the printed output to actions with any of these tags (comma-separated or repeatable). |
| `--output-include-deps` | boolean | — | Also includes upstream dependencies of the selected actions in the output. Requires --output-actions or --output-tags. |
| `--output-include-dependents` | boolean | — | Also includes downstream dependents of the selected actions in the output. Requires --output-actions or --output-tags. |
Expand Down Expand Up @@ -250,7 +250,7 @@ Compiles and executes the project against BigQuery (or validates it with `--dry-
| `--dry-run` | boolean | — | Validates the run SQL with BigQuery without applying any changes to the warehouse. |
| `--run-tests` | boolean | — | Requires the project's unit tests to pass before executing; aborts the run if any test fails. |
| `--action-retry-limit` | number | `0` | Retries idempotent actions up to this many times on failure. |
| `--actions` | array | — | Actions to run (comma-separated or repeatable). Each entry is an action name or a fully-qualified database.schema.name target. |
| `--actions` | array | — | Actions to run (comma-separated or repeatable). Each entry is an action name or a fully-qualified database.schema.name (or 4-part database.catalog.namespace.name) target. |
| `--tags` | array | — | Tags to filter the actions to run (comma-separated or repeatable). |
| `--include-deps` | boolean | — | Also runs upstream dependencies of the selected actions. Requires --actions or --tags. |
| `--include-dependents` | boolean | — | Also runs downstream dependents of the selected actions. Requires --actions or --tags. |
Expand All @@ -274,6 +274,10 @@ Compiles and executes the project against BigQuery (or validates it with `--dry-
- `--timeout` bounds compilation only, not execution. If it's set without `--execution-timeout`, `run` prints a warning reminding you of that.
- `--job-labels` override `defaultJobLabels`, action `jobLabels` and table `labels` that use the same key.
- `--emit-lineage` requires the Data Lineage API (`datalineage.googleapis.com`) to be enabled and the `roles/datalineage.producer` role; for configuration and troubleshooting, see the [Data Lineage guide](../docs/lineage.md).
- **4-part Lakehouse Catalog targets (`project.catalog.namespace.table`):**
- The target catalog and namespace must already exist before running `dataform run` (Lakehouse namespaces are managed via BigLake APIs and are not created via `CREATE SCHEMA`).
- Incremental Lakehouse tables with `onSchemaChange` (`FAIL`, `EXTEND`, or `SYNCHRONIZE`) inspect the incoming query schema using a short-lived empty table in `defaultDataset`. Both `defaultProject` and `defaultDataset` must be configured in `workflow_settings.yaml` (or via CLI overrides) in the same location as the Lakehouse catalog (`defaultLocation`), and the execution identity needs permissions to create the dataset (if missing) and create/drop tables in `defaultDataset`.
- Table descriptions and top-level column descriptions on Lakehouse tables are applied after action execution via `ALTER TABLE` SQL statements (nested `STRUCT` field descriptions are skipped because BigQuery `ALTER COLUMN` only supports top-level columns).

Exit codes: `0` on success, `1` if compilation fails, a BigQuery job would get more than 64 labels (checked before any action or unit test runs), unit tests fail (with `--run-tests`), the run fails, times out, or is cancelled (Ctrl+C).

Expand Down
2 changes: 2 additions & 0 deletions cli/api/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -58,13 +58,15 @@ ts_test_suite(
"commands/compile_copy_filter_test.ts",
"commands/jit/rpc_test.ts",
"commands/prune_test.ts",
"commands/run_test.ts",
"dbadapters/bigquery_test.ts",
"execution_sql_test.ts",
] + glob(["lineage/*_test.ts"]),
data = glob(["goldens/**"]),
deps = [
":api",
"//cli/api/lineage/testing",
"//cli/api/utils",
"//core",
"//protos:ts",
"//testing",
Expand Down
74 changes: 74 additions & 0 deletions cli/api/commands/prune_test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -167,4 +167,78 @@ suite("prune", () => {
}),
);
});

const LAKEHOUSE_TARGET: dataform.ITarget = {
database: "p",
schema: "cat.ns",
name: "orders",
};
const LAKEHOUSE_ACTION: dataform.ITable = {
type: "table",
enumType: dataform.TableType.TABLE,
target: LAKEHOUSE_TARGET,
canonicalTarget: LAKEHOUSE_TARGET,
query: "select 1 as order_id",
dependencyTargets: [],
};
const DOWNSTREAM_OF_LAKEHOUSE_TARGET: dataform.ITarget = {
database: "p",
schema: "d",
name: "order_summary",
};
const DOWNSTREAM_OF_LAKEHOUSE_ACTION: dataform.ITable = {
type: "table",
enumType: dataform.TableType.TABLE,
target: DOWNSTREAM_OF_LAKEHOUSE_TARGET,
canonicalTarget: DOWNSTREAM_OF_LAKEHOUSE_TARGET,
query: "select * from `p.cat.ns.orders`",
dependencyTargets: [LAKEHOUSE_TARGET],
};

test("--actions matches 4-part Lakehouse target by database.catalog.namespace.name", () => {
const compiledGraph: dataform.ICompiledGraph = {
tables: [LAKEHOUSE_ACTION, DOWNSTREAM_OF_LAKEHOUSE_ACTION],
operations: [],
assertions: [],
propertyGraphs: [],
targets: [LAKEHOUSE_TARGET, DOWNSTREAM_OF_LAKEHOUSE_TARGET],
};
expect(asPlainObject(prune(compiledGraph, { actions: ["p.cat.ns.orders"] }))).deep.equals(
asPlainObject({
tables: [LAKEHOUSE_ACTION],
operations: [],
assertions: [],
propertyGraphs: [],
targets: [LAKEHOUSE_TARGET],
}),
);
});

test("--actions matches 4-part Lakehouse target by bare name and resolves includeDependents", () => {
const compiledGraph: dataform.ICompiledGraph = {
tables: [LAKEHOUSE_ACTION, DOWNSTREAM_OF_LAKEHOUSE_ACTION],
operations: [],
assertions: [],
propertyGraphs: [],
targets: [LAKEHOUSE_TARGET, DOWNSTREAM_OF_LAKEHOUSE_TARGET],
};
expect(
asPlainObject(prune(compiledGraph, { actions: ["orders"], includeDependents: true })),
).deep.equals(asPlainObject(compiledGraph));
});

test("--actions with includeDependencies pulls in upstream 4-part Lakehouse table", () => {
const compiledGraph: dataform.ICompiledGraph = {
tables: [LAKEHOUSE_ACTION, DOWNSTREAM_OF_LAKEHOUSE_ACTION],
operations: [],
assertions: [],
propertyGraphs: [],
targets: [LAKEHOUSE_TARGET, DOWNSTREAM_OF_LAKEHOUSE_TARGET],
};
expect(
asPlainObject(
prune(compiledGraph, { actions: ["p.d.order_summary"], includeDependencies: true }),
),
).deep.equals(asPlainObject(compiledGraph));
});
});
8 changes: 6 additions & 2 deletions cli/api/commands/run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ import { Timer } from "df/cli/api/utils/timer";
import { Flags } from "df/common/flags";
import { retry } from "df/common/promises";
import { deepClone, equals } from "df/common/protos";
import { targetAsReadableString, targetStringifier } from "df/core/targets";
import { isCatalogSchema, targetAsReadableString, targetStringifier } from "df/core/targets";
import { version } from "df/core/version";
import { dataform } from "df/protos/ts";

Expand Down Expand Up @@ -358,6 +358,8 @@ export class Runner {
(action.tasks.length > 0 || !!action.jitCode) &&
!!action.target &&
!!action.target.schema &&
// Lakehouse catalog.namespace targets cannot be created via CREATE SCHEMA; they must exist before the run.
!isCatalogSchema(action.target.schema) &&
writesToTarget(action),
)
.forEach(({ target }) => {
Expand Down Expand Up @@ -620,7 +622,9 @@ export class Runner {
action.type === "table"
) {
try {
await this.dbadapter.setMetadata(action);
await this.dbadapter.setMetadata(action, {
bigquery: this.getBigQueryExecutionOptions(action),
});
} catch (e) {
// TODO: Setting the metadata is not a task itself, so we have nowhere to surface this error cleanly.
// For now, we can attach the error to the last task in the action so it gets
Expand Down
206 changes: 206 additions & 0 deletions cli/api/commands/run_test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
import { expect } from "chai";
import Long from "long";
import { anyString, anything, instance, mock, verify, when } from "ts-mockito";

import { build } from "df/cli/api/commands/build";
import { Runner } from "df/cli/api/commands/run";
import { BigQueryDbAdapter } from "df/cli/api/dbadapters/bigquery";
import { wrapWithCreateSchemaIfNotExists } from "df/cli/api/utils/query_transforms";
import { dataform } from "df/protos/ts";
import { suite, test } from "df/testing";

suite("Runner with 4-part Lakehouse targets", () => {
const projectConfig: dataform.IProjectConfig = {
warehouse: "bigquery",
defaultDatabase: "my-gcp-project",
defaultSchema: "default_dataset",
};

test("skips schemas() lookup and CREATE SCHEMA wrapper when all actions are 4-part Lakehouse targets", async () => {
const mockedDbAdapter = mock(BigQueryDbAdapter);
const executedStatements: string[] = [];
when(mockedDbAdapter.execute(anyString(), anything())).thenCall(async (statement: string) => {
executedStatements.push(statement);
return { rows: [], metadata: {} };
});

const lakehouseStatement =
"create or replace table `my-gcp-project.ecommerce_catalog.sales_dw.customer_orders` as select 1 as id";
const opStatement =
"insert into `my-gcp-project.ecommerce_catalog.sales_dw.audit_log` values (1)";
const graph = dataform.ExecutionGraph.create({
projectConfig,
runConfig: {},
warehouseState: {},
actions: [
{
type: "table",
tableType: "table",
target: {
database: "my-gcp-project",
schema: "ecommerce_catalog.sales_dw",
name: "customer_orders",
},
tasks: [{ type: "statement", statement: lakehouseStatement }],
dependencyTargets: [],
},
{
type: "operation",
hasOutput: true,
target: {
database: "my-gcp-project",
schema: "ecommerce_catalog.sales_dw",
name: "audit_log",
},
tasks: [{ type: "statement", statement: opStatement }],
dependencyTargets: [],
},
],
});

const realRunResult = await new Runner(instance(mockedDbAdapter), graph).execute().result();
expect(realRunResult.status).equals(dataform.RunResult.ExecutionStatus.SUCCESSFUL);

const dryRunResult = await new Runner(instance(mockedDbAdapter), graph, {
bigquery: { dryRun: true },
})
.execute()
.result();
expect(dryRunResult.status).equals(dataform.RunResult.ExecutionStatus.SUCCESSFUL);

verify(mockedDbAdapter.schemas(anything())).never();
expect(executedStatements).deep.equals([
lakehouseStatement,
opStatement,
lakehouseStatement,
opStatement,
]);
});

test("wraps missing 3-part dataset with CREATE SCHEMA while leaving 4-part Lakehouse target unwrapped", async () => {
const mockedDbAdapter = mock(BigQueryDbAdapter);
const executedStatements: string[] = [];
when(mockedDbAdapter.schemas("my-gcp-project")).thenResolve([]);
when(mockedDbAdapter.execute(anyString(), anything())).thenCall(async (statement: string) => {
executedStatements.push(statement);
return { rows: [], metadata: {} };
});

const standardStatement =
"create or replace table `my-gcp-project.new_dataset.std_table` as select 1";
const lakehouseStatement =
"create or replace table `my-gcp-project.my_cat.my_ns.lh_table` as select 1";
const graph = dataform.ExecutionGraph.create({
projectConfig,
runConfig: {},
warehouseState: {},
actions: [
{
type: "table",
tableType: "table",
target: { database: "my-gcp-project", schema: "new_dataset", name: "std_table" },
tasks: [{ type: "statement", statement: standardStatement }],
dependencyTargets: [],
},
{
type: "table",
tableType: "table",
target: { database: "my-gcp-project", schema: "my_cat.my_ns", name: "lh_table" },
tasks: [{ type: "statement", statement: lakehouseStatement }],
dependencyTargets: [],
},
],
});

const result = await new Runner(instance(mockedDbAdapter), graph).execute().result();

expect(result.status).equals(dataform.RunResult.ExecutionStatus.SUCCESSFUL);
verify(mockedDbAdapter.schemas("my-gcp-project")).once();
expect(executedStatements).to.have.members([
wrapWithCreateSchemaIfNotExists("my-gcp-project", "new_dataset", standardStatement),
lakehouseStatement,
]);
});

test("build() fetches 4-part target warehouse state via dbadapter.table() and builds incremental DML", async () => {
const mockedDbAdapter = mock(BigQueryDbAdapter);
const lakehouseTarget: dataform.ITarget = {
database: "my-gcp-project",
schema: "ecommerce_catalog.sales_dw",
name: "customer_orders",
};
when(mockedDbAdapter.table(anything())).thenResolve(
dataform.TableMetadata.create({
type: dataform.TableMetadata.Type.TABLE,
target: lakehouseTarget,
fields: [{ name: "order_id", primitive: dataform.Field.Primitive.INTEGER }],
lastUpdatedMillis: Long.ZERO,
}),
);

const compiledGraph = dataform.CompiledGraph.create({
projectConfig,
dataformCoreVersion: "3.1.5",
tables: [
{
type: "incremental",
enumType: dataform.TableType.INCREMENTAL,
target: lakehouseTarget,
query: "SELECT 1 AS order_id",
incrementalQuery: "SELECT 2 AS order_id",
uniqueKey: ["order_id"],
},
],
});

const executionGraph = await build(
compiledGraph,
{ fullRefresh: false },
instance(mockedDbAdapter),
);

expect(executionGraph.actions.length).equals(1);
expect(executionGraph.actions[0].tasks[0].statement).to.include(
"merge `my-gcp-project.ecommerce_catalog.sales_dw.customer_orders` DATAFORM_DEST",
);
});

test("delegates setMetadata() with execution options on 4-part Lakehouse tables with descriptions", async () => {
const mockedDbAdapter = mock(BigQueryDbAdapter);
when(mockedDbAdapter.execute(anyString(), anything())).thenResolve({ rows: [], metadata: {} });
when(mockedDbAdapter.setMetadata(anything(), anything())).thenResolve();

const graph = dataform.ExecutionGraph.create({
projectConfig,
runConfig: {},
warehouseState: {},
actions: [
{
type: "table",
tableType: "table",
target: {
database: "my-gcp-project",
schema: "ecommerce_catalog.sales_dw",
name: "customer_orders",
},
actionDescriptor: {
description: "Lakehouse orders table",
columns: [{ path: ["order_id"], description: "Order ID" }],
},
tasks: [
{
type: "statement",
statement:
"create or replace table `my-gcp-project.ecommerce_catalog.sales_dw.customer_orders` as select 1 as order_id",
},
],
dependencyTargets: [],
},
],
});

const result = await new Runner(instance(mockedDbAdapter), graph).execute().result();
expect(result.status).equals(dataform.RunResult.ExecutionStatus.SUCCESSFUL);
verify(mockedDbAdapter.setMetadata(anything(), anything())).once();
});
});
Loading
Loading