Skip to content

feat(UP-13203): gcp-databricks-audit-logs module, GCS delivery to Pub/Sub, new or existing delivery - #6

Open
itayhamou wants to merge 2 commits into
mainfrom
feat/UP-13203-gcp-databricks-audit-logs
Open

itayhamou wants to merge 2 commits into
mainfrom
feat/UP-13203-gcp-databricks-audit-logs

Conversation

@itayhamou

Copy link
Copy Markdown

What

New module modules/gcp-databricks-audit-logs, with two examples (examples/gcp-databricks-audit-logs, examples/gcp-databricks-audit-logs-existing) and plan-time tests.

Databricks delivers an account's audit logs as JSON Lines files into a Cloud Storage bucket in the customer's project. A storage notification publishes to a Pub/Sub topic on every write, and Upwind pulls the subscription and reads the files. Upwind reads as the management service account of the Upwind GCP onboarding, through the workload identity federation that onboarding already set up: the onboarding module is applied with enable_dataevents = true on a scope that includes the project (upwindsecurity/terraform-gcp-onboarding#70). This module creates no identity, pool or grant for Upwind.

Databricks log delivery ──writes JSON Lines──▶ GCS bucket ──OBJECT_FINALIZE──▶ Pub/Sub topic ─▶ subscription

Two modes, switched by one input:

New delivery (default) Existing delivery (existing_bucket_name)
Bucket Created, dedicated: multi-region Standard, uniform access, public access prevented, lifecycle rule after log_retention_days The customer's, which must be in project_id and use uniform bucket-level access
Databricks Its log-delivery service account gets roles/storage.admin on the bucket, as Databricks documents Nothing: the delivery already writes there
Notification Created, scoped to delivery_path_prefix Added next to the bucket's own
Destroy Removes everything the module made (bucket_force_destroy opt-in) Removes only the module's notification; the bucket and the delivery stay

The Databricks log delivery configuration itself is not created here: the Databricks Terraform provider supports it on AWS only. Upwind creates it through the Databricks account API once the integration is saved in the console.

Usage

module "gcp_databricks_audit_logs" {
  source = "upwindsecurity/cloudlogs/google//modules/gcp-databricks-audit-logs"

  project_id      = "my-gcp-project"
  resource_suffix = "<suffix shown in the Upwind console>"
  # Existing delivery:
  # existing_bucket_name = "<the delivery's bucket>"
  # delivery_path_prefix = "<the delivery's prefix>"
}

Conventions

  • No provider block in the module; every resource sets project explicitly.
  • _iam_member only, no authoritative bindings: roles/storage.admin on the bucket for Databricks' log-delivery service account (new delivery only) and roles/pubsub.publisher on the topic for the project's GCS service agent. Additive, so an existing bucket keeps its other members.
  • Plan-time guards: the plan fails when existing_bucket_name is in another project than project_id (its GCS service agent publishes the notifications).
  • google_project_service has disable_on_destroy = false; the subscription never expires.
  • Provider pins match gcp-audit-logs: google >= 6.23.0, < 8.0.0, required_version >= 1.11.0.

Testing

  • terraform test in modules/gcp-databricks-audit-logs: 10 passed (mocked google and random, no cloud calls)
  • terraform validate on the module and both examples
  • terraform fmt -check -recursive
  • READMEs generated with terraform-docs (make docs)

🤖 Generated with Claude Code

…/Sub, new or existing delivery

Add modules/gcp-databricks-audit-logs: makes a Databricks account on GCP
deliver its audit logs into a Cloud Storage bucket the customer owns and
notifies a Pub/Sub topic on every write. Upwind reads the bucket and the
subscription as the management service account of the Upwind GCP
onboarding (enable_dataevents = true), so the module creates no identity,
pool or grant for Upwind.

Two modes: a new delivery (the module creates the bucket and grants
Databricks' log-delivery service account on it) or an existing one
(existing_bucket_name: the bucket is used as is, with a notification added
next to its own). Examples for both, and plan-time tests with mocked
providers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Versioning off, no customer-managed key, no access logging and Storage Admin
for Databricks' log-delivery service account on its own bucket are each
deliberate; the comments say why and suppress the corresponding checks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Jira ticket verified

Key Status Summary
UP-13203 open · hidden · hidden — hidden: public repository

Linked automatically in the ticket's development panel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants