Data: where it lives and how it arrives
The picture: before any ML happens, data must (a) arrive from somewhere, (b) be stored somewhere, (c) be in an efficient format, and (d) be processable. Four questions, four toolsets.
Where data lives: the storage family
| Member | What it is | Trigger phrase |
|---|---|---|
| S3 | the infinite cloud warehouse (files or "objects"), THE default answer | almost everything; "data lake" |
| EBS | one machine's personal hard drive | "attach a volume to an instance" |
| EFS | the shared NFS network folder. Many machines mount it at once, it supports file locking, and it's mountable from on-premises too | "NFS," "shared," "hybrid," "file locking" |
| FSx for Lustre | the GPU racetrack: an ultra-fast file system that can LINK to an S3 bucket (lazy-loads, no copying) | "millions of files," "distributed training performance" |
| Glacier | the cheap deep-freeze archive (via S3 lifecycle policies) | "archive after 30 days" |
EBS is one machine's disk. EFS is Everyone's Folder (Shared, and it speaks NFS). Lustre means luster and speed. S3 is the default.
A quick note before we go on: the guide is stuffed with little interactive drills like the two below. Later chapters add a loss-curve simulator, an endpoint picker, a security memory palace, and scored practice quizzes. Click around, that's how the patterns stick.
Click a trigger phrase to see which service it points to.
File formats
Parquet is columnar (reads only the columns a query needs) and compressed, so it's the answer for "analytics on S3 or data lake" (pair with gzip when asked to compress). CSV/JSON are simple text. Avro is streaming with evolving schemas. RecordIO-Protobuf is SageMaker's own format, and note that Glue cannot write it; converting at scale needs Spark on EMR (a famous exception). "AWS Glue Parquet" is a made-up format, so any option naming it is instantly wrong. Athena queries get fast when S3 data is partitioned by date prefix, because it scans only the days asked for.
How data arrives: streaming vs batch
The picture: batch is mail once a day (a file lands on schedule). Streaming is a phone call (data flows continuously). The conveyor-belt trio: Kinesis/MSK is the belt (ingest only; MSK when the question says "Kafka"); Flink is the machine that works ON the belt (computes on data as it flows, and supports SQL and notebooks); Firehose is the truck that takes items off the belt and delivers them to S3 (buffers ~60s; a zero-buffering mode exists for sub-second delivery; can convert to Parquet).
Firehose's arrow points STREAM to S3, never S3 to anywhere. If data is "already stored in S3," every Firehose option is dead, because ingestion of parked data is batch ETL, which means Glue. "Process or aggregate data WHILE it streams" points only to Flink; Athena, Glue, and QuickSight all require data to land first.
Choose what the scenario says and watch the right service surface.
Processing tools: who does the heavy lifting?
| Tool | One line | Trigger |
|---|---|---|
| Glue | serverless batch ETL machine (Spark under the hood). Accessories: Crawlers (schema into the Data Catalog), Data Quality (rules that gate a pipeline), Sensitive Data Detection (find AND mask PII), FindMatches (ML fuzzy-duplicate detection), Job Bookmarks (only new data) | "batch ETL," "terabytes," "scheduled job," data already in S3 |
| EMR | rented Spark or Hadoop clusters for massive or special jobs | "hundreds of nodes," "Spark," RecordIO conversion, repartitioning huge files |
| Lambda | tiny event-driven functions with a 15-minute limit | small quick transforms; the 15-min limit kills "hours-long" |
| Athena | serverless SQL queries on files in S3. It queries, and NEVER transforms or ingests | "query the data" (a classic wrong-answer for ETL) |
EMR is a construction crew. The primary node is the foreman (coordinates, runs nothing), core nodes are workers WITH toolsheds (they process AND store the data), and task nodes are day laborers (muscle only, store nothing). Spot instances are cheap workers who can be repossessed, so only ever put the day laborers (task nodes) on Spot: repossessing a core node loses data, and repossessing the foreman kills the site.