Data & AI Reference

Changelog | Contributing

This repository contains different Data & AI related resources like applications, examples, libraries, tools and more!

Contents

• Examples

  • Data & AI
    • Orchestration
    • AI Agents
    • Geospatial Analysis
    • RAG
    • Evaluation
    • Data Engineering
  • DevOps
    • Containers, Orchestration and Serverless
      • Containers and Compositions (Docker, Docker Compose, Buildpacks and more)
      • Development and Cloud Environments
    • Infrastructure as Code
      • Serverless Framework, SAM and CloudFormation
      • Terraform
    • A/B Testing

• Apps and Boilerplates

• Guides, Tutorials and Best Practices

• Contributing

• Contributors

Examples

Data & AI

Orchestration

Name	Description	Keywords
Dagster AI Orchestration Framework - Case Study & Architecture Evaluation	Comprehensive architectural evaluation of POC-Dagster data orchestration project, examining suitability for evolution into a 4-layer AI Agent platform. Includes gap analysis (18% completeness), production-ready code templates for all 5 layers (Frontend/Orchestration/Runtime/Data/Validation), and decision framework for archival vs fresh start.	Dagster, Data Orchestration, LangGraph, AI Agents, Architecture Evaluation, 4-Layer Stack, Vercel AI SDK, Okta, Cedar, MCP, Embeddings, FastAPI, Next.js, TypeScript, Python3, Prompt Evals, Decision Framework, Reference Architecture

AI Agents

Name	Description	Keywords
Dagster AI Orchestration Framework - Case Study & Architecture Evaluation	Comprehensive architectural evaluation of POC-Dagster data orchestration project, examining suitability for evolution into a 4-layer AI Agent platform. Includes gap analysis (18% completeness), production-ready code templates for all 5 layers (Frontend/Orchestration/Runtime/Data/Validation), and decision framework for archival vs fresh start.	Dagster, Data Orchestration, LangGraph, AI Agents, Architecture Evaluation, 4-Layer Stack, Vercel AI SDK, Okta, Cedar, MCP, Embeddings, FastAPI, Next.js, TypeScript, Python3, Prompt Evals, Decision Framework, Reference Architecture
LangGraph SLS FastAPI RAG	A Proof of Concept (POC) for a Retrieval Augmented Generation (RAG) system using LangGraph, deployed with the Serverless Framework on AWS Lambda. It integrates FastAPI for API development and DynamoDB for state management.	Python3, FastAPI, LangGraph, Serverless Framework, AWS Lambda, DynamoDB, Docker, RAG, Retrieval Augmented Generation

Geospatial Analysis

Name	Description	Keywords
Geospatial Python Urban Analysis with PostGIS	A comprehensive geospatial data analysis project for urban environments, featuring analysis of pedestrian zones, transportation networks, census data, and geographic boundaries. Built with PostgreSQL/PostGIS, Docker, and GeoPandas for efficient spatial queries, ETL pipelines, and geospatial machine learning models.	Python3, GeoPandas, PostGIS, PostgreSQL, Docker, Geospatial Analysis, Urban Planning, ETL, Machine Learning, Jupyter Notebook, Folium, Matplotlib, Scikit-learn, Rasterio, GIS

RAG

Name	Description	Keywords
LangGraph SLS FastAPI RAG	A Proof of Concept (POC) for a Retrieval Augmented Generation (RAG) system using LangGraph, deployed with the Serverless Framework on AWS Lambda. It integrates FastAPI for API development and DynamoDB for state management.	Python3, FastAPI, LangGraph, Serverless Framework, AWS Lambda, DynamoDB, Docker, RAG, Retrieval Augmented Generation

Evaluation

Name	Description	Keywords
AWS CloudWatch Evidently	An in-depth analysis and proof of concept demonstrating how to integrate AWS CloudWatch Evidently with a Node.js application for A/B testing and feature flag management.	AWS CloudWatch, CloudWatch Evidently, A/B Testing, Feature Flags, Node.js

Data Engineering

Name	Description	Keywords
Airflow and Spark environment using Docker and Docker Compose	Dockerfile and compose.yml to run Airflow locally with initialization scripts.	Docker, Docker Compose, Airflow, Spark
Amazon MSK Streaming (Terraform Module)	Opinionated Terraform wrapper for Amazon MSK (Kafka) with enhanced security, DNS integration and logging.	Streaming, Kafka, Terraform, Security
Analytics Platform with Document Store (Terraform Example)	Analytics platform combining data lake architecture with a document database for real-time analytics.	Analytics, Data Lake, Terraform, DocumentDB
AWS Data Lake Encryption (Terraform Module)	Terraform module with KMS keys and IAM policies for data lake encryption and service isolation.	Data Lake, KMS, Terraform, Security
AWS Data Lake Infrastructure (Terraform Module)	Terraform module for foundational S3 data lake infrastructure with medallion zones and Apache Iceberg support.	Data Lake, S3, Terraform, Medallion, Iceberg
AWS Glue	DevContainer for AWS Glue development. Uses docker-compose to run VSCode attached to a container with all the necessary tools to develop AWS Glue jobs such as AWS Glue Libs, Spark, Jupyter Notebook, AWS CLI among other tools.	Docker, Docker Compose, DevContainers, VSCode DevContainer, GitHub Codespaces, AWS Glue, Spark, Jupyter Notebook, AWS CLI
AWS Glue Code Registry (Terraform Module)	Terraform module for a Glue Code Registry centralizing code libraries, JARs and dependencies for Glue jobs.	AWS Glue, Terraform, Dependencies
AWS Glue Data Lake Catalog (Terraform Module)	Terraform module for AWS Glue databases backing medallion data lakes with flexible layer separation.	AWS Glue, Data Catalog, Terraform, Medallion
AWS Glue Jobs (Terraform Module)	Terraform module for AWS Glue jobs with IAM roles, S3 script bucket and CloudWatch log groups.	AWS Glue, ETL, Terraform, IAM
AWS Glue using Docker and Docker Compose	Dockerfile and compose.yml for AWS Glue development with AWS Glue Libs, Spark, Jupyter Notebook, AWS CLI among other tools.	Docker, Docker Compose, AWS Glue, Spark, Jupyter Notebook, AWS CLI
AWS Glue with Python Shell and PySpark Jobs	Serverless Framework example to deploy an AWS Glue job using Python Shell and PySpark.	Serverless Framework, AWS Glue, Python Shell, PySpark
AWS Glue Workflows (Terraform Module)	Terraform module for AWS Glue workflows and triggers orchestrating existing Glue jobs.	AWS Glue, Orchestration, Terraform, ETL
Basic AWS Glue ETL example app	A simplified yet practical example of an AWS Glue application. It uses the Serverless Framework for infrastructure deployment and supports local development with DevContainers or Docker Compose.	AWS Glue, ETL, Python3, PySpark, Serverless Framework, DevContainers, Docker Compose, Jupyter Notebook
Data Processing Pipeline (Terraform Example)	Batch and stream data processing pipeline on AWS managed services, deployed with Terraform.	Data Engineering, Streaming, Terraform, Batch
Data Quality Framework Checklist	Trustworthy data platforms checklist: contracts, orchestration reliability, validation, lineage, observability, governance, security, and scalability scoring.	Data Quality, Lineage, Observability, Governance, Best Practices
Multi-Account Data Platform (Terraform Example)	Self-contained multi-account data lake example using provider aliases; path toward production multi-account architecture.	Data Lake, Multi-Account, Terraform
Secure Multi-Environment Data Platform (Terraform Example)	Secure multi-environment data platform with strict isolation, advanced security controls and compliance features.	Data Platform, Security, Terraform, Compliance

DevOps

Containers, Orchestration and Serverless

Containers and Compositions (Docker, Docker Compose, Buildpacks and more)

Name	Description	Keywords
Airflow and Spark environment using Docker and Docker Compose	Dockerfile and compose.yml to run Airflow locally with initialization scripts.	Docker, Docker Compose, Airflow, Spark
AWS Glue using Docker and Docker Compose	Dockerfile and compose.yml for AWS Glue development with AWS Glue Libs, Spark, Jupyter Notebook, AWS CLI among other tools.	Docker, Docker Compose, AWS Glue, Spark, Jupyter Notebook, AWS CLI
Geospatial Python Urban Analysis with PostGIS	A comprehensive geospatial data analysis project for urban environments, featuring analysis of pedestrian zones, transportation networks, census data, and geographic boundaries. Built with PostgreSQL/PostGIS, Docker, and GeoPandas for efficient spatial queries, ETL pipelines, and geospatial machine learning models.	Python3, GeoPandas, PostGIS, PostgreSQL, Docker, Geospatial Analysis, Urban Planning, ETL, Machine Learning, Jupyter Notebook, Folium, Matplotlib, Scikit-learn, Rasterio, GIS

Development and Cloud Environments

Name	Description	Keywords
AWS Glue	DevContainer for AWS Glue development. Uses docker-compose to run VSCode attached to a container with all the necessary tools to develop AWS Glue jobs such as AWS Glue Libs, Spark, Jupyter Notebook, AWS CLI among other tools.	Docker, Docker Compose, DevContainers, VSCode DevContainer, GitHub Codespaces, AWS Glue, Spark, Jupyter Notebook, AWS CLI

Infrastructure as Code

Serverless Framework, SAM and CloudFormation

Name	Description	Keywords
AWS Glue with Python Shell and PySpark Jobs	Serverless Framework example to deploy an AWS Glue job using Python Shell and PySpark.	Serverless Framework, AWS Glue, Python Shell, PySpark
LangGraph SLS FastAPI RAG	A Proof of Concept (POC) for a Retrieval Augmented Generation (RAG) system using LangGraph, deployed with the Serverless Framework on AWS Lambda. It integrates FastAPI for API development and DynamoDB for state management.	Python3, FastAPI, LangGraph, Serverless Framework, AWS Lambda, DynamoDB, Docker, RAG, Retrieval Augmented Generation

Terraform

Name	Description	Keywords
Amazon MSK Streaming (Terraform Module)	Opinionated Terraform wrapper for Amazon MSK (Kafka) with enhanced security, DNS integration and logging.	Streaming, Kafka, Terraform, Security
Analytics Platform with Document Store (Terraform Example)	Analytics platform combining data lake architecture with a document database for real-time analytics.	Analytics, Data Lake, Terraform, DocumentDB
AWS Data Lake Encryption (Terraform Module)	Terraform module with KMS keys and IAM policies for data lake encryption and service isolation.	Data Lake, KMS, Terraform, Security
AWS Data Lake Infrastructure (Terraform Module)	Terraform module for foundational S3 data lake infrastructure with medallion zones and Apache Iceberg support.	Data Lake, S3, Terraform, Medallion, Iceberg
AWS Glue Code Registry (Terraform Module)	Terraform module for a Glue Code Registry centralizing code libraries, JARs and dependencies for Glue jobs.	AWS Glue, Terraform, Dependencies
AWS Glue Data Lake Catalog (Terraform Module)	Terraform module for AWS Glue databases backing medallion data lakes with flexible layer separation.	AWS Glue, Data Catalog, Terraform, Medallion
AWS Glue Jobs (Terraform Module)	Terraform module for AWS Glue jobs with IAM roles, S3 script bucket and CloudWatch log groups.	AWS Glue, ETL, Terraform, IAM
AWS Glue Workflows (Terraform Module)	Terraform module for AWS Glue workflows and triggers orchestrating existing Glue jobs.	AWS Glue, Orchestration, Terraform, ETL
Data Processing Pipeline (Terraform Example)	Batch and stream data processing pipeline on AWS managed services, deployed with Terraform.	Data Engineering, Streaming, Terraform, Batch
Multi-Account Data Platform (Terraform Example)	Self-contained multi-account data lake example using provider aliases; path toward production multi-account architecture.	Data Lake, Multi-Account, Terraform
Secure Multi-Environment Data Platform (Terraform Example)	Secure multi-environment data platform with strict isolation, advanced security controls and compliance features.	Data Platform, Security, Terraform, Compliance

A/B Testing

Name	Description	Keywords
AWS CloudWatch Evidently	An in-depth analysis and proof of concept demonstrating how to integrate AWS CloudWatch Evidently with a Node.js application for A/B testing and feature flag management.	AWS CloudWatch, CloudWatch Evidently, A/B Testing, Feature Flags, Node.js

Apps and Boilerplates

Name	Description	Keywords
Basic AWS Glue ETL example app	A simplified yet practical example of an AWS Glue application. It uses the Serverless Framework for infrastructure deployment and supports local development with DevContainers or Docker Compose.	AWS Glue, ETL, Python3, PySpark, Serverless Framework, DevContainers, Docker Compose, Jupyter Notebook
Geospatial Python Urban Analysis with PostGIS	A comprehensive geospatial data analysis project for urban environments, featuring analysis of pedestrian zones, transportation networks, census data, and geographic boundaries. Built with PostgreSQL/PostGIS, Docker, and GeoPandas for efficient spatial queries, ETL pipelines, and geospatial machine learning models.	Python3, GeoPandas, PostGIS, PostgreSQL, Docker, Geospatial Analysis, Urban Planning, ETL, Machine Learning, Jupyter Notebook, Folium, Matplotlib, Scikit-learn, Rasterio, GIS
LangGraph SLS FastAPI RAG	A Proof of Concept (POC) for a Retrieval Augmented Generation (RAG) system using LangGraph, deployed with the Serverless Framework on AWS Lambda. It integrates FastAPI for API development and DynamoDB for state management.	Python3, FastAPI, LangGraph, Serverless Framework, AWS Lambda, DynamoDB, Docker, RAG, Retrieval Augmented Generation

Guides, Tutorials and Best Practices

Name	Description	Keywords
Data Quality Framework Checklist	Trustworthy data platforms checklist: contracts, orchestration reliability, validation, lineage, observability, governance, security, and scalability scoring.	Data Quality, Lineage, Observability, Governance, Best Practices

Contributing

• Contributions make the open source community such an amazing place to learn, inspire, and create.
• Any contributions you make are truly appreciated.
• Check out our contribution guidelines for more information.

Contributors

Made with contributors-img.