<- Back to Home · CLI Reference · Configs Reference
Dataform supports defining BigQuery Property Graphs using YAML configuration files (definitions/**/graph.yaml) aligned with the Google Semantic Ontology specification (version: 0.2.0.dev0/google). During compilation, Dataform resolves backing table dependencies, validates the semantic model, and emits a CREATE OR REPLACE PROPERTY GRAPH DDL action that can be referenced by downstream SQLX actions via ${ref("<graph_name>")}.
Note: BigQuery Property Graphs and the
version: 0.2.0.dev0/googlegraph.yamlschema are in Preview and subject to change.
Note: The Dataform compiler enforces the following file discovery and cardinality rules:
- File path (
graph.yaml): Property graph configurations must be placed insidedefinitions/in a file namedgraph.yaml(for example,definitions/graph.yamlordefinitions/graphs/graph.yaml).- One graph per project: At most one
graph.yamlfile and one entry insemantic_modelare allowed per Dataform project.
- Key casing: Both
snake_case(recommended, such assemantic_model,primary_key,from_columns,to_columns,ai_context,custom_extensions,is_time,job_labels,depend_on_dependency_assertions,include_dependent_assertions) andcamelCasekeys are accepted (keys insidejob_labels,ai_context.custom, and mapping-valuedcustom_extensions[].dataare preserved verbatim).
Assume the following workflow_settings.yaml:
# workflow_settings.yaml
defaultProject: my-gcp-project
defaultDataset: dataform
defaultLocation: US
dataformCoreVersion: 3.1.4-- definitions/authors.sqlx
config { type: "table" }
SELECT
1 AS id,
"Alice" AS name,
DATE "2020-01-15" AS joined_date,
STRUCT("PL" AS country) AS address-- definitions/books.sqlx
config { type: "table" }
SELECT 10 AS id, "Graph Systems" AS title, 320 AS pages-- definitions/wrote.sqlx
config { type: "table" }
SELECT 1 AS author_id, 10 AS book_idversion: 0.2.0.dev0/google
tags: ["graph"]
depend_on_dependency_assertions: true
job_labels:
team: "analytics"
semantic_model:
- name: "LibraryGraph"
description: "Library authorship semantic model"
deployments:
- name: "default"
target: "bigquery.googleapis.com/projects/my-gcp-project/datasets/analytics/propertyGraphs/LibraryGraph"
entities:
- name: "Author"
source:
ref: "authors"
primary_key: "id"
description: "Library author"
ai_context:
instructions: "Query by author id or country"
synonyms: ["writer", "creator"]
examples: ["Alice"]
fields:
- name: "id"
datatype: "INT64"
- name: "name"
datatype: "STRING"
label: "Author Name"
- name: "upper_name"
datatype: "STRING"
expression: "UPPER(name)"
- name: "joined_date"
dimension:
is_time: true
- name: "country"
expression: "address.country"
- name: "Book"
source:
ref: "books"
primary_key: "id"
fields:
- "id"
- "title"
- "pages"
- name: "Authorship"
source:
ref: "wrote"
primary_key: ["author_id", "book_id"]
relationships:
- name: "Wrote"
from: "Authorship"
to: "Author"
from_columns: ["author_id"]
to_columns: ["id"]
description: "Links an authorship record to its author"
- name: "WroteBook"
from: "Authorship"
to: "Book"
from_columns: ["book_id"]
to_columns: ["id"]
metrics:
- name: "book_count"
entity: "Book"
datatype: "INT64"
expression: "COUNT(id)"
description: "Total books"
- name: "total_pages"
entity: "Book"
datatype: "INT64"
expression: "SUM(pages)"
label: "Total Pages"
description: "Sum of pages across books"
- name: "avg_pages"
entity: "Book"
datatype: "FLOAT64"
expression: "SAFE_DIVIDE(AGG(total_pages), NULLIF(AGG(book_count), 0))"
description: "Average pages per book"When executed (dataform run), Dataform compiles the model above into the following BigQuery DDL statement:
CREATE OR REPLACE PROPERTY GRAPH `my-gcp-project.analytics.LibraryGraph`
NODE TABLES (
`my-gcp-project.dataform.authors` AS Author KEY (id) DEFAULT LABEL OPTIONS(description="Library author", synonyms=["writer", "creator"], extensions=[("instructions", "Query by author id or country"), ("examples", "[\"Alice\"]")]) PROPERTIES (id, name OPTIONS(description="Author Name"), UPPER(name) AS upper_name, joined_date OPTIONS(description="Time dimension."), address.country AS country),
`my-gcp-project.dataform.books` AS Book KEY (id) PROPERTIES (id, title, pages, MEASURE(COUNT(id)) AS book_count OPTIONS(description="Total books"), MEASURE(SUM(pages)) AS total_pages OPTIONS(description="Total Pages\n\nSum of pages across books"), MEASURE(SAFE_DIVIDE(AGG(total_pages), NULLIF(AGG(book_count), 0))) AS avg_pages OPTIONS(description="Average pages per book")),
`my-gcp-project.dataform.wrote` AS Authorship KEY (author_id, book_id) PROPERTIES ALL COLUMNS
)
EDGE TABLES (
`my-gcp-project.dataform.wrote` AS Wrote KEY (author_id, book_id) SOURCE KEY (author_id, book_id) REFERENCES Authorship (author_id, book_id) DESTINATION KEY (author_id) REFERENCES Author (id) DEFAULT LABEL OPTIONS(description="Links an authorship record to its author") NO PROPERTIES,
`my-gcp-project.dataform.wrote` AS WroteBook KEY (author_id, book_id) SOURCE KEY (author_id, book_id) REFERENCES Authorship (author_id, book_id) DESTINATION KEY (book_id) REFERENCES Book (id) NO PROPERTIES
)For full protocol message tables, see property_graph_config.proto in the Configs Reference.
Top-Level Settings (PropertyGraphConfig)
| Field | Required | Description |
|---|---|---|
version |
Yes | Must be "0.2.0.dev0/google". |
semantic_model |
Yes | List of semantic models (must contain 1 model). |
disabled |
No | If true, skips executing the property graph action while keeping it referenceable via ref(). |
depend_on_dependency_assertions |
No | If true, adds assertions attached to referenced dependencies as dependencies of this graph action. |
tags |
No | User-defined action tags for selective execution (dataform run --tags ...). |
job_labels |
No | Key-value pairs of BigQuery job labels applied to the graph creation job (merged with defaultJobLabels from workflow_settings.yaml, with keys defined here taking precedence). |
Model-level description and ai_context (instructions and examples) on semantic_model[0] populate Dataform's action metadata (PropertyGraph.description) only and are not emitted in the BigQuery DDL because CREATE PROPERTY GRAPH does not support graph-level OPTIONS.
- With
deployments: Provide a single-element list whosetargetis a GCP resource URI without a leading//(where<graph>must matchsemantic_model[0].name):Leadingdeployments: - name: "default" target: "bigquery.googleapis.com/projects/my-project/datasets/my_dataset/propertyGraphs/MyGraph"
//ontargetURIs is rejected. Dataform session overrides (databaseSuffix,schemaSuffix,namePrefix) still apply when resolving the final action target. - Without
deployments: The property graph defaults todefaultProject(defaultDatabase) anddefaultDataset(defaultSchema) fromworkflow_settings.yaml, usingsemantic_model[0].nameas the graph name.
Each entry under entities (or its alias datasets) compiles to a NODE TABLES entry and requires name, source, and primary_key.
| Format | Example | Registers Dataform DAG Dependency? |
|---|---|---|
Dataform ref shorthand |
source: { ref: "authors" } |
Yes |
Qualified Dataform ref |
source: { ref: { name: "authors", schema: "raw", database: "my-proj", include_dependent_assertions: true } } |
Yes |
bigquery: table shorthand |
source: "bigquery:my-proj.raw.authors" |
No (external table) |
| Bare 3-part table identifier | source: "my-proj.raw.authors" |
No (external table) |
GCP table resource URI (no leading //) |
source: "bigquery.googleapis.com/projects/my-proj/datasets/raw/tables/authors" |
No (external table) |
- Omitted
fields: Whenfieldsis omitted (and nometricstarget the entity), Dataform emitsPROPERTIES ALL COLUMNS. - Explicit
fields: Whenfieldsis provided, Dataform emitsPROPERTIES (...)containing only the listed fields (and anymetricstargeting the entity):- Bare string shorthand:
- "id"is equivalent to{ name: "id" }and emits the column identifieridas-is. - Derived SQL expression: Set
expressionto any valid BigQuery scalar expression (for example,UPPER(name)oraddress.country). Expressions are passed through as-is into<expression> AS <name>. - Informational
datatype: Accepted by the parser (for example,datatype: "INT64") for documentation and schema tooling; has no effect on the generated BigQuery DDL. - Temporal dimension (
dimension): Setdimension: { is_time: true }to fold"Time dimension."into the field'sOPTIONS(description=...).
- Bare string shorthand:
Every relationship is an N:1 foreign-key edge from from to to, backed by the from entity's table and primary_key:
- Required fields:
name,from,to,from_columns,to_columns. from_columnsvalidation:from_columnsmust be physical column names on thefromentity's backing table.to_columnsvalidation:to_columnsmust match the target (to) entity'sprimary_keyin both column count and column names (case-insensitively).- Compiled DDL: Each relationship compiles to an
EDGE TABLESentry withSOURCE KEY (<from.primary_key>) REFERENCES <from> (<from.primary_key>) DESTINATION KEY (<from_columns>) REFERENCES <to> (<to_columns>)andNO PROPERTIES. - Modeling M:N bridge tables: Because relationships do not accept a separate
sourceorprimary_key, model a many-to-many join table (such aswrote) as an intermediate entity (Authorshipwithprimary_key: ["author_id", "book_id"]) connected by two N:1 relationships (Wrote->AuthorandWroteBook->Book), as shown in the End-to-End Example.
Top-level metrics entries compile to BigQuery Graph Measures (MEASURE(<expression>) AS <name>) inside the target entity's PROPERTIES (...) list:
- Aggregate and derived expressions: Dataform validates that
expressionis a non-empty string and passes it through as-is intoMEASURE(<expression>) AS <name>. Supported BigQuery measure expressions include:- Standard and conditional aggregates (
SUM(...),AVG(...),COUNT(...),MIN(...),MAX(...),COUNT(DISTINCT ...),CASE WHEN ..., scalar arithmetic). - Derived sibling measures (
AGG(...)): Reference other metrics defined on the same entity viaAGG(<sibling_metric>)(for example,SAFE_DIVIDE(AGG(total_pages), NULLIF(AGG(book_count), 0))orAGG(gross_revenue) - AGG(refunds)). COUNT(*)andCOUNT(<literal>)restriction: BigQuery requires every measure expression to aggregate at least one graph property and rejects bothCOUNT(*)and literal-only counts such asCOUNT(1). Dataform rejectsCOUNT(*)andCOUNT(<literal>)at compile time: count an explicit property instead (for example,COUNT(id)).
- Standard and conditional aggregates (
- Informational
datatype: Accepted onmetrics(for example,datatype: "FLOAT64"ordatatype: "Float") for documentation and schema tooling; omitted from the generated BigQuery DDL, matchingfields. - Target entity requirements:
- When one or more
metricstarget an entity, that entity must declare an explicitfieldslist (because BigQuery does not allowMEASURE(...)alongsidePROPERTIES ALL COLUMNS). - Any column or derived property referenced inside a metric
expressionmust be declared in the target entity'sfieldslist.
- When one or more
On entities and relationships, metadata is emitted as DEFAULT LABEL OPTIONS(...) before PROPERTIES (...), PROPERTIES ALL COLUMNS, or NO PROPERTIES. On fields and metrics, metadata is emitted as OPTIONS(...) immediately after the property definition.
OPTIONS(description=...): Dataform joins the following present non-empty sections with"\n\n"in order:label(on fields and metrics)"Time dimension."(on fields whendimension.is_timeistrue)description
OPTIONS(synonyms=[...]): Populated fromai_context.synonyms.OPTIONS(extensions=[...]): Populated fromai_contextas(key, value)string pairs in order:("instructions", "<trimmed_instructions>")whenai_context.instructions(or a string shorthandai_context: "...") is non-empty.("examples", "<json_array>")whenai_context.examplesis non-empty (serialized viaJSON.stringify, for example("examples", "[\"Alice\"]")).
- Knowledge Catalog custom metadata (omitted from BigQuery DDL):
ai_context.custom: Accepts an optional mapping of custom key-value pairs (string,number,boolean,null, array, or nested object) on the model, entities, fields, relationships, and metrics. Nestedsnake_casekeys are preserved verbatim, and standardai_contextkeys (instructions,synonyms,examples) must be written directly underai_contextrather than insidecustom.custom_extensions: Accepts a list of{ vendor_name: "<vendor>", data: "<string_or_object>" }blocks on the model, entities, fields, relationships, and metrics for third-party vendor metadata. Entries withvendor_namematching"GOOGLE"(in any casing) are rejected in0.2.0.dev0/googledocuments because Google extensions are declared as native YAML keys.
Reference a property graph in any downstream SQLX table, view, or assertion using ${ref("<graph_name>")} inside BigQuery's GRAPH_TABLE operator (which expects a backticked identifier such as `project.dataset.LibraryGraph`):
-- definitions/author_books.sqlx
config {
type: "view"
}
SELECT
author_name,
book_title,
pages
FROM GRAPH_TABLE(
${ref("LibraryGraph")}
MATCH (a:Author)<-[:Wrote]-(:Authorship)-[:WroteBook]->(b:Book)
COLUMNS (
a.name AS author_name,
b.title AS book_title,
b.pages AS pages
)
)BigQuery's GRAPH_EXPAND table-valued function evaluates graph measures via AGG(...) and requires the property graph name as a STRING literal ("project.dataset.LibraryGraph") rather than a backticked identifier. Strip the surrounding backticks returned by ref() with .slice(1, -1) so Dataform still registers the DAG dependency:
-- definitions/author_book_metrics.sqlx
config {
type: "table"
}
SELECT
Author_country AS country,
AGG(Book_book_count) AS book_count,
AGG(Book_total_pages) AS total_pages,
AGG(Book_avg_pages) AS avg_pages
FROM GRAPH_EXPAND(
"${ref('LibraryGraph').slice(1, -1)}",
start_node => "Authorship",
flatten_results => TRUE
)
GROUP BY country