Repository navigation
feat(UP-13203): gcp-databricks-audit-logs module, GCS delivery to Pub/Sub, new or existing delivery - #6
Open
itayhamou wants to merge 2 commits into
Open
feat(UP-13203): gcp-databricks-audit-logs module, GCS delivery to Pub/Sub, new or existing delivery#6itayhamou wants to merge 2 commits into
itayhamou wants to merge 2 commits into
Conversation
…/Sub, new or existing delivery Add modules/gcp-databricks-audit-logs: makes a Databricks account on GCP deliver its audit logs into a Cloud Storage bucket the customer owns and notifies a Pub/Sub topic on every write. Upwind reads the bucket and the subscription as the management service account of the Upwind GCP onboarding (enable_dataevents = true), so the module creates no identity, pool or grant for Upwind. Two modes: a new delivery (the module creates the bucket and grants Databricks' log-delivery service account on it) or an existing one (existing_bucket_name: the bucket is used as is, with a notification added next to its own). Examples for both, and plan-time tests with mocked providers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Versioning off, no customer-managed key, no access logging and Storage Admin for Databricks' log-delivery service account on its own bucket are each deliberate; the comments say why and suppress the corresponding checks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Jira ticket verified
Linked automatically in the ticket's development panel. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New module
modules/gcp-databricks-audit-logs, with two examples (examples/gcp-databricks-audit-logs,examples/gcp-databricks-audit-logs-existing) and plan-time tests.Databricks delivers an account's audit logs as JSON Lines files into a Cloud Storage bucket in the customer's project. A storage notification publishes to a Pub/Sub topic on every write, and Upwind pulls the subscription and reads the files. Upwind reads as the management service account of the Upwind GCP onboarding, through the workload identity federation that onboarding already set up: the onboarding module is applied with
enable_dataevents = trueon a scope that includes the project (upwindsecurity/terraform-gcp-onboarding#70). This module creates no identity, pool or grant for Upwind.Two modes, switched by one input:
existing_bucket_name)log_retention_daysproject_idand use uniform bucket-level accessroles/storage.adminon the bucket, as Databricks documentsdelivery_path_prefixbucket_force_destroyopt-in)The Databricks log delivery configuration itself is not created here: the Databricks Terraform provider supports it on AWS only. Upwind creates it through the Databricks account API once the integration is saved in the console.
Usage
Conventions
projectexplicitly._iam_memberonly, no authoritative bindings:roles/storage.adminon the bucket for Databricks' log-delivery service account (new delivery only) androles/pubsub.publisheron the topic for the project's GCS service agent. Additive, so an existing bucket keeps its other members.existing_bucket_nameis in another project thanproject_id(its GCS service agent publishes the notifications).google_project_servicehasdisable_on_destroy = false; the subscription never expires.gcp-audit-logs:google >= 6.23.0, < 8.0.0,required_version >= 1.11.0.Testing
terraform testinmodules/gcp-databricks-audit-logs: 10 passed (mockedgoogleandrandom, no cloud calls)terraform validateon the module and both examplesterraform fmt -check -recursivemake docs)🤖 Generated with Claude Code