Managed ETL Pipelines across clouds
Serverless data integration - Glue, Data Factory, Dataflow - with jobs that run as a scoped identity and pipelines that alert when they fail.
4 verified modules - all static-validated and publish-checked; live-test pending.
Compare by provider
| Provider | Module | Verification |
|---|---|---|
| AWS | Glue with Encrypted Job Bookmarks | static-validated |
| Azure | A Data Factory that Moves Data Privately and Keeps Pipelines in Git | static-validated |
| Google Cloud | A Dataflow Job on Private Workers that Drains on Destroy | static-validated |
| Oracle Cloud | A Data Integration Workspace on Your VCN so Pipelines Reach Private Databases | static-validated |
How to choose
Compare the programming model (a visual pipeline, Spark, Beam), what a job runs as and whether it can reach the internet, and how failures surface: a pipeline that fails silently at 03:00 is the default on every one of these until an alert is wired.
When not to use
A pipeline copies data and with it every access problem the data had. Encrypt the intermediate storage, scope the job identity to its sources and sinks, and expect the service account to become the most privileged identity in the project.
Other solutions
Managed Relational Database25Secrets & Key Management23DNS & Traffic Management23Messaging & Pub/Sub22Load Balancer21Virtual Machines21Virtual Private Cloud (VPC)19Cloud Firewall17Managed Kubernetes16Object Storage16Block Storage16Golden Images14Container Registry12Identity & Access12In-Memory Cache12Shared File Storage11Managed NoSQL Database11Serverless Functions10CDN & Edge Delivery10WAF & Edge Security10NAT Gateway10Serverless Containers9Monitoring & Observability9Managed Search9Static Site Hosting9Managed TLS Certificates9Network Peering9Audit Log Retention9Cloud Data Warehouse9API Gateway8Site-to-Site VPN8Dedicated Interconnect8Backup Service8Managed CI/CD8Transit Hub8Managed Spark8Network Flow Logs8Security Posture & Findings7Database Migration & CDC7Managed Network Firewall7Private Instance Access7Organisation Guardrails7Private Service Endpoints7Private Certificate Authority6Managed ML Platform6Transactional Email6Managed App Platform6Customer Identity6Uptime Checks6Landing Zone6Budgets & Cost Alerts5Patch Management5Workflow Orchestration4HPC Parallel File Systems4Managed BI4Data Transfer Service3Managed Active Directory3Zero Trust Application Access3Sensitive Data Discovery3