@dataform/core is the open-source SQLX compiler, JavaScript workflow SDK, and Protocol Buffer schema package for Dataform. It compiles SQLX (.sqlx), JavaScript (.js), action configuration (actions.yaml), BigQuery Property Graph (graph.yaml), and Jupyter Notebook (.ipynb) definitions into a dependency-ordered execution graph (dataform.CompiledGraph) for BigQuery.
For command-line execution (dataform init, compile, test, run, format), see the Dataform CLI Reference (@dataform/cli) and the root repository overview.
- Installation and Version Pinning
- Project Structure
- Project Configuration (
workflow_settings.yaml) - SQLX Syntax Reference
- Action Types Reference
- JavaScript API (
SessionGlobals) - Package Exports and Bundled
.protoFiles - Documentation and Links
In both Dataform in Google Cloud and @dataform/cli, specify dataformCoreVersion in your project's workflow_settings.yaml:
defaultProject: my-gcp-project
defaultLocation: US
defaultDataset: dataform
defaultAssertionDataset: dataform_assertions
dataformCoreVersion: <version>When dataformCoreVersion is set in workflow_settings.yaml, @dataform/cli automatically installs that version of @dataform/core in an isolated temporary directory at compile time (see Stateless installs).
If you manage @dataform/core and external JavaScript packages with package.json (omitting dataformCoreVersion from workflow_settings.yaml), or if you consume @dataform/core programmatically:
npm i @dataform/coreA standard Dataform project has the following layout:
my-dataform-project/
├── workflow_settings.yaml # Project-level settings and defaults
├── definitions/ # Action definitions (.sqlx, .js, actions.yaml, graph.yaml, .sql, .ipynb)
│ ├── sources.sqlx
│ ├── staging/
│ │ └── stg_orders.sqlx
│ └── marts/
│ └── fct_orders.sqlx
└── includes/ # Shared JavaScript modules and constants
└── constants.js
workflow_settings.yaml: Configures default project/dataset/location, custom variables (vars), target prefixes/suffixes, and runtime options.definitions/: Every.sqlxand.jsfile underdefinitions/(and any subdirectory) is evaluated to register actions in the compilation session.definitions/**/actions.yamlfiles register YAML-configured actions, anddefinitions/**/graph.yamldefines a BigQuery Property Graph.includes/: Top-levelincludes/*.jsfiles are automatically attached to the global scope by their file basename (for example,includes/constants.jsis available asconstantsinside any.sqlxor.jsfile indefinitions/). Nested files underincludes/can be imported via standardrequire("includes/...").
Note on Project Execution & Security: Dataform compiles SQLX and JavaScript project code using Node.js's native
node:vmcontext. While module resolution is scoped to the project directory,node:vmis not a security sandbox against untrusted code. Only compile and run Dataform projects from trusted sources.
workflow_settings.yaml is validated against the dataform.WorkflowSettings schema.
| Field | Type | Default | Description |
|---|---|---|---|
dataformCoreVersion |
string |
— | @dataform/core version to compile against (for example, 3.1.5). |
defaultProject |
string |
— | Default Google Cloud project ID (database) for compiled actions. |
defaultDataset |
string |
"dataform" |
Default BigQuery dataset (schema) for tables, views, and operations. |
defaultLocation |
string |
— | Default BigQuery dataset location (for example, US, EU, us-central1). |
defaultAssertionDataset |
string |
"dataform_assertions" |
Default BigQuery dataset (schema) for assertion views. |
vars |
map<string, string> |
{} |
Custom string key-value variables accessible via dataform.projectConfig.vars.<key>. |
projectSuffix |
string |
— | Suffix appended to target Google Cloud project IDs (<project>_<projectSuffix>). |
datasetSuffix |
string |
— | Suffix appended to target dataset names (<dataset>_<datasetSuffix>). |
namePrefix |
string |
— | Prefix prepended to target action names (<namePrefix>_<name>). |
builtinAssertionNamePrefix |
string |
— | Prefix prepended to auto-generated built-in assertion names. |
disableAssertions |
boolean |
false |
Marks all manual and built-in assertions as disabled: true in the compiled graph. |
defaultReservation |
string |
— | Default BigQuery reservation for query execution (projects/<p>/locations/<l>/reservations/<r> or none). |
defaultJobLabels |
map<string, string> |
{} |
Default labels applied to BigQuery jobs executed for actions in the project. |
preserveGovernanceControls |
boolean |
false |
Preserves existing BigQuery column governance controls (such as Data Policies) when recreating tables. |
lineage |
object |
— | Knowledge Catalog Data Lineage settings (lineage: { enabled: true }). See Data Lineage guide. |
defaultIcebergConfig |
object |
— | Project-level defaults for BigQuery Apache Iceberg tables (bucketName, tableFolderRoot, tableFolderSubpath, connection). |
defaultLakehouseConfig |
object |
— | Project-level defaults for 4-part Lakehouse targets (catalog, namespace). |
defaultNotebookRuntimeOptions |
object |
— | Default runtime options for Notebook actions (outputBucket, runtimeTemplateName, repositorySnapshotDestination). |
includeTestsInCompiledGraph |
boolean |
false |
Includes unit test actions (type: "test") in CompiledGraph.tests. |
extension |
object |
— | External compilation extension package configuration (name, compilationMode). |
A .sqlx file combines a config {} metadata block, optional JavaScript (js {}) and hook blocks, and a SQL body with ${...} JavaScript template interpolation.
| Block | Supported Action Types | Description |
|---|---|---|
config { ... } |
All .sqlx types |
Declares the action type (defaults to "operations" if omitted), target overrides, dependencies, assertions, and BigQuery options. |
js { ... } |
All .sqlx types |
Local JavaScript variables and helper functions scoped to the current .sqlx file. |
pre_operations { ... } |
table, incremental, view |
SQL statements executed before the main table/view query runs (multiple statements can be terminated with ; or split into separate compiled array entries with ---). |
post_operations { ... } |
table, incremental, view |
SQL statements executed after the main table/view query completes (multiple statements can be terminated with ; or split into separate compiled array entries with ---). |
incremental_where { ... } |
incremental |
Optional SQL predicate that sets Table.where to filter rows during incremental updates (SELECT * FROM (<query>) AS subquery WHERE <predicate>). Not supported with compilation_mode: "jit". |
input "<target>" { ... } |
test |
Mock SQL query replacing a referenced upstream dependency (ref("<target>")) during a unit test. |
--- |
operations, pre_operations, post_operations |
Statement separator for multi-statement SQL blocks. |
Each action maps to an action target (database.schema.name) in BigQuery. By default, name is the action's filename (without extension), schema is defaultDataset (or defaultAssertionDataset for assertions), and database is defaultProject from workflow_settings.yaml.
Inside the SQL body and operation blocks of a .sqlx file (or via the ctx argument in JavaScript API callbacks), the following context methods are available:
| Function | Signature | Description |
|---|---|---|
ref |
ref(name) / ref(schema, name) / ref(database, schema, name) / ref({ database?, schema?, name, includeDependentAssertions? }) |
Resolves an action target to its backtick-quoted SQL identifier and registers it as an upstream dependency. |
resolve |
resolve(name) / resolve(schema, name) / resolve(database, schema, name) / resolve(targetObj) |
Resolves an action target to its backtick-quoted SQL identifier without adding it as a dependency. |
self |
self() |
Returns the fully qualified, backtick-quoted SQL identifier of the current action's target. |
name |
name() |
Returns the finalized table/action name of the current action (including namePrefix if set). |
schema |
schema() |
Returns the finalized dataset (schema) name of the current action (including datasetSuffix if set). |
database |
database() |
Returns the finalized Google Cloud project (database) of the current action (including projectSuffix if set). |
when |
when(condition, trueCase, falseCase = "") |
Returns trueCase when condition is truthy and falseCase otherwise (table, incremental, view, operations, assertion, test, dataPreparation). |
incremental |
incremental() |
Returns true when compiling the incremental update query of an incremental table, or false when compiling its initial/full-refresh query (table, incremental, view). |
Full field definitions for all action configurations are documented in the Dataform Configs Reference.
Creates or replaces a BigQuery table (CREATE OR REPLACE TABLE). See TableConfig.
-- definitions/marts/orders_by_day.sqlx
config {
type: "table",
schema: "marts",
description: "Daily aggregated order metrics",
columns: {
order_date: "UTC date of the order",
total_orders: "Count of completed orders"
},
bigquery: {
partitionBy: "order_date",
clusterBy: ["country_code"],
labels: {
domain: "commerce"
}
},
assertions: {
uniqueKey: ["order_date", "country_code"],
nonNull: ["order_date"]
},
tags: ["daily"]
}
SELECT
DATE(order_timestamp) AS order_date,
country_code,
COUNT(*) AS total_orders
FROM ${ref("stg_orders")}
GROUP BY 1, 2Key table config properties:
- Target & metadata:
name,schema(ordataset),database(orproject),description,columns,tags,disabled,dependencies,dependOnDependencyAssertions,hermetic,metadata(overview,extraProperties). - BigQuery options:
partitionBy,partitionExpirationDays,requirePartitionFilter,clusterBy,labels,additionalOptions, andiceberg(all accepted at top level or nested underbigquery: {}; forview,bigquery: {}acceptspartitionBy,clusterBy,labels, andadditionalOptions), plus top-levelpreserveGovernanceControls. - Execution & assertions:
assertions(uniqueKey,uniqueKeys,nonNull,rowConditions),preOperations,postOperations,reservation,jobLabels,lakehouse,compilation_mode("aot"or"jit").
Builds a table from scratch on first run (or --full-refresh), and inserts or merges new rows on subsequent runs when the target table already exists. See IncrementalTableConfig.
-- definitions/marts/events_incremental.sqlx
config {
type: "incremental",
uniqueKey: ["event_id"],
bigquery: {
partitionBy: "DATE(event_timestamp)",
updatePartitionFilter: "event_timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)"
},
protected: false
}
SELECT
event_id,
event_timestamp,
user_id
FROM ${ref("raw_events")}
${when(incremental(), `WHERE event_timestamp > (SELECT MAX(event_timestamp) FROM ${self()})`)}Incremental-specific config properties (in addition to table properties):
uniqueKey: Column name(s) used toMERGErows on incremental updates; if omitted, new rows are appended (INSERT).incrementalStrategy:"merge"(requiresuniqueKey) or"insert_overwrite"(requirespartitionBy) in SQLX and JavaScript (or"INCREMENTAL_STRATEGY_MERGE"/"INCREMENTAL_STRATEGY_INSERT_OVERWRITE"inactions.yaml).updatePartitionFilter/incrementalPredicates: Mutually exclusive SQL filter(s) restricting target partitions scanned during incremental updates (setting both is a compile error; preferincrementalPredicates).onSchemaChange:"IGNORE"(default),"FAIL","EXTEND", or"SYNCHRONIZE".protected(defaulttrue): Prevents the table from being dropped and rebuilt from scratch on--full-refresh. Setprotected: falseto allow full refreshes.
Creates a virtual view (CREATE OR REPLACE VIEW) or a BigQuery materialized view (materialized: true). See ViewConfig.
-- definitions/staging/stg_orders.sqlx
config {
type: "view",
materialized: true,
bigquery: {
partitionBy: "DATE(order_timestamp)",
clusterBy: ["customer_id"]
}
}
SELECT
order_id,
customer_id,
order_timestamp
FROM ${ref("raw_orders")}Executes arbitrary SQL statements (DDL, DML, stored procedures, GRANT/REVOKE). Multiple statements are separated by ---. See OperationConfig.
-- definitions/ops/cleanup_temp_tables.sqlx
config {
type: "operations",
hasOutput: false,
dependencies: ["orders_by_day"]
}
DELETE FROM `my-gcp-project.staging.scratch_events`
WHERE created_at < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)When hasOutput: true is set, the operation registers a referenceable target at database.schema.name that downstream actions can reference with ${ref("<name>")} or ${self()}.
An assertion is a data quality query that fails if it returns one or more rows. Assertions materialize as views in defaultAssertionDataset (dataform_assertions by default). See AssertionConfig.
-- definitions/assertions/assert_positive_order_amount.sqlx
config {
type: "assertion",
tags: ["critical"]
}
SELECT order_id, amount
FROM ${ref("stg_orders")}
WHERE amount <= 0You can also configure built-in assertions directly inside a table, incremental, or view config block:
config {
type: "table",
assertions: {
uniqueKey: ["order_id"],
nonNull: ["order_id", "customer_id"],
rowConditions: [
"amount >= 0",
"order_timestamp <= CURRENT_TIMESTAMP()"
]
}
}Declares an external BigQuery table or view (not managed by Dataform) so it can be referenced via ${ref("<name>")}, documented, and shown in the dependency graph. See DeclarationConfig.
-- definitions/sources/raw_orders.sqlx
config {
type: "declaration",
database: "my-raw-data-project",
schema: "stripe",
name: "raw_orders",
description: "Raw Stripe orders landing table"
}Validates the SQL logic of a target dataset (table, view, or incremental action) by replacing its upstream ref() inputs with static mock queries (input "<ref_name>" { ... }) and comparing the output against an expected query. Executed via dataform test or dataform run --run-tests.
-- definitions/tests/orders_by_day_test.sqlx
config {
type: "test",
dataset: "orders_by_day"
}
input "stg_orders" {
SELECT TIMESTAMP("2026-01-01 10:00:00 UTC") AS order_timestamp, "PL" AS country_code
UNION ALL
SELECT TIMESTAMP("2026-01-01 12:00:00 UTC") AS order_timestamp, "PL" AS country_code
}
SELECT DATE("2026-01-01") AS order_date, "PL" AS country_code, 2 AS total_ordersConfigures a Jupyter Notebook (.ipynb) action for execution in Google Cloud Dataform (via Vertex AI Colab Enterprise). See NotebookConfig.
# definitions/actions.yaml
actions:
- notebook:
name: train_churn_model
filename: notebooks/train_churn_model.ipynb
dependencyTargets:
- name: orders_by_dayConfigures a BigQuery Data Preparation action from a .dp.yaml / .yaml definition (via actions.yaml) or a .dp.sqlx / .sqlx file (type: "dataPreparation"). See DataPreparationConfig.
Defines a BigQuery Property Graph (CREATE OR REPLACE PROPERTY GRAPH) from a definitions/**/graph.yaml file (at most one graph.yaml per project) following the Google Semantic Ontology schema (version: 0.2.0.dev0/google). See the BigQuery Property Graphs guide for full syntax and examples.
Plain .sql and .ipynb files can be registered as Dataform actions via definitions/**/actions.yaml files validated against dataform.ActionConfigs:
# definitions/actions.yaml
actions:
- declaration:
name: raw_users
project: my-raw-project
dataset: raw
- view:
name: stg_users
filename: staging/stg_users.sql
dependencyTargets:
- name: raw_users
- table:
name: dim_users
filename: marts/dim_users.sql
dependencyTargets:
- name: stg_users
partitionBy: created_dateInside definitions/**/*.js files (and js {} blocks in .sqlx files), Dataform attaches the active Session instance to global.dataform and binds its action-builder methods as global functions:
| Global Function / Object | Signature | Description |
|---|---|---|
publish |
publish(name, queryOrConfig?) |
Creates a table, view (default if type is omitted), or incremental action. Returns a Table, View, or IncrementalTable builder (.query(ctx => ...)). |
operate |
operate(name, queriesOrConfig?) |
Creates an operations action. Returns an Operation builder (.queries(ctx => ...)). |
assert |
assert(name, queryOrConfig?) |
Creates an assertion action. Returns an Assertion builder (.query(ctx => ...)). |
declare |
declare(config) |
Declares an external data source ({ database?, schema?, name, description?, columns? }). |
test |
test(name) |
Creates a unit test action (.dataset(target).input(refName, sql).expect(sql)). |
notebook |
notebook(config) |
Creates a Notebook action ({ name?, filename, location?, project?, dependencyTargets?, tags?, disabled?, description? }). |
getContents |
getContents(relativeMdPath) |
Reads a .md file inside the project directory (resolved relative to the caller file) and returns its contents as a string for description fields. |
jitData |
jitData(key, value) |
Registers structured data in CompiledGraph.jitData for Just-in-Time (JiT) compilation workflows. |
dataform.projectConfig |
dataform.ProjectConfig |
Read-only access to the resolved project configuration, including custom variables (dataform.projectConfig.vars). |
Example (definitions/generate_staging_views.js):
const sourceTables = ["orders", "customers", "products"];
sourceTables.forEach((tableName) => {
declare({
schema: "raw",
name: `raw_${tableName}`,
});
publish(`stg_${tableName}`, {
type: "view",
schema: "staging",
description: getContents(`./docs/stg_${tableName}.md`),
}).query((ctx) => `SELECT * FROM ${ctx.ref("raw", `raw_${tableName}`)}`);
});When imported as an npm library (import * as dataformCore from "@dataform/core"), @dataform/core exports the following symbols (with TypeScript typings in bundle.d.ts):
version(string): The installed@dataform/coresemantic version string (for example,"3.1.5").session(Session): The singleton compilationSessioninstance.compiler((code: string, path: string) => string): Transpiles.sqlx,.yaml/.yml,.sql,.ipynb, and.mdsource strings into executable JavaScript modules.main((coreExecutionRequest: Uint8Array | string) => Uint8Array | string): Entry point invoked by the compilation VM sandbox; accepts an encodeddataform.CoreExecutionRequestproto and returns an encodeddataform.CoreExecutionResponseproto containingCompiledGraph.jitCompiler(IJitCompiler): Just-in-Time (JiT) action compiler entry point.supportedFeatures(dataform.SupportedFeatures[]): Feature capability flags supported by this@dataform/coreversion (such asARRAY_BUFFER_IPC).IDataformExtension/IJitCompiler: TypeScript interfaces for extensions and JiT compilation.
The published @dataform/core package includes the canonical .proto definitions at the package root:
core.proto—CompiledGraph,Table,Operation,Assertion,Declaration,Test,Notebook,DataPreparation,PropertyGraph,Target,ProjectConfig,CoreExecutionRequest,CoreExecutionResponse.configs.proto—WorkflowSettings,ActionConfigs,ActionConfig(TableConfig,ViewConfig,IncrementalTableConfig,AssertionConfig,OperationConfig,DeclarationConfig,NotebookConfig,DataPreparationConfig).property_graph_config.proto—PropertyGraphConfig(SemanticModel,Entity,Relationship,Deployment).db_adapter.proto&jit.proto— Database adapter and Just-in-Time (JiT) compilation RPC messages.extension.proto—Extensionconfiguration schema.
- Dataform Documentation Home
- Dataform Core Reference (Google Cloud Docs)
- Dataform Configs Reference (
configs.proto) - Dataform JavaScript
SessionReference - Dataform CLI Reference (
@dataform/cli) - BigQuery Property Graphs Guide (
graph.yaml) - Knowledge Catalog Data Lineage Guide
- Stateless Installs Guide (
.dataformignore) - GitHub Repository & Issue Tracker