AWS

For most data platforms, the services that carry the weight are S3 (storage), RDS/Aurora (operational databases), one analytics engine (Redshift, Snowflake, or Databricks on AWS), IAM (access control), and CloudWatch (monitoring). Everything else is additive. Kinesis for streaming, Lambda for event-driven processing, Glue for cataloging, MWAA for orchestration. But the core is simpler than most AWS architecture diagrams suggest.

For a government data platform, we built an event-driven architecture that processed XML job posting data from S3, transformed it through serverless compute, and stored it in Aurora PostgreSQL. The platform delivered $500,000 in cost savings and enabled hundreds of researchers to access labor market data through a self-service API. The architecture was deliberately simple: S3, Aurora, and a Django API.

For a biotech manufacturer, the workload required real-time streaming from 60+ IoT machines at 5ms intervals. That workload justified Kinesis for the streaming layer and Databricks for processing. The key decision was separating the streaming layer from the analytical layer so each could scale independently.

The most common pattern we see is architectural accumulation. A team starts with a simple pipeline, adds Glue because someone found a tutorial, adds Step Functions for orchestration, adds Lambda for a small transformation, and ends up with a data platform spread across six services that nobody can diagram from memory. Each service was a reasonable local decision. The sum is a system that is expensive to run and difficult to change.

The second pattern is defaulting to managed services when simpler options exist. EMR for a workload that a single dbt run on Snowflake could handle. SageMaker for a model that scikit-learn on a Lambda function could serve. These choices add cost and complexity without proportional benefit.

AWS billing is detailed enough that teams benefit from auditing where their money goes. Data transfer charges, provisioned IOPS on RDS, unused Elastic IPs, and oversized EC2 instances running development workloads are common optimization opportunities that become clear once someone reviews the bill.

AWS has the broadest service catalog and the largest market share. If your engineering team already knows AWS and your infrastructure runs there, building your data platform on AWS avoids the cross-cloud complexity that a multi-cloud strategy introduces.

Where each cloud fits best: GCP leans into a tightly integrated native analytics stack (BigQuery and Looker). Azure leans into Microsoft enterprise infrastructure (Entra ID, Microsoft 365, Dynamics). If your organization's primary workload is analytics and you are starting fresh, GCP's serverless data stack is a natural fit. If you are a Microsoft shop, Azure aligns with your existing infrastructure. If your workloads are diverse and your team knows AWS, its breadth and depth make it the safest default.