MapReduce, Hadoop and YARN, HDFS, Pig and Hive.
When data is too large for one machine, it must be stored and processed across a cluster of many computers. This chapter (the first half of the frameworks part of syllabus Unit V) covers the MapReduce programming model, Hadoop and its distributed file system HDFS, and the higher-level tools Pig and Hive. Chapter 12 covers HBase, MapR, sharding, NoSQL databases and Amazon S3. Each topic has a worked numerical example.
11.1 MapReduce
MapReduce is a programming model, introduced by Google in 2004, for processing very large datasets in parallel on a cluster of commodity machines. The programmer writes only two functions, and the framework handles distribution, scheduling, communication and failures. The map function takes one input record and emits zero or more intermediate (key, value) pairs. The framework then shuffles and sorts these pairs so that all values with the same key are brought together at one machine. The reduce function takes a key and the list of all its values and produces the final output for that key. In detail, a job runs in stages: the input is divided into splits (typically one HDFS block each) and one map task runs on each split, preferably on the machine that already stores it (data locality); an optional combiner performs a local reduce on each mapper's output to cut network traffic; a partitioner decides which reducer receives each key, usually by hash(key) mod R for R reducers; the shuffle and sort phase copies and merges the pairs; and the reduce tasks write the results to HDFS. If a machine fails, the framework simply re-runs its tasks elsewhere. MapReduce suits batch jobs such as counting, indexing, log analysis, joins and ETL, but it is slow for iterative algorithms and interactive queries, because every stage writes to disk — the weakness that Apache Spark's in-memory model addresses.
Each split is mapped and combined locally; the shuffle (shaded) brings every key's values together, and each reducer writes its own output file. Which reducer receives a key is decided by hash(key) mod 2.
Worked example 1 – word count. Two input splits: split 1 = "big data is big", split 2 = "data is new".
- Map, split 1: (big, 1), (data, 1), (is, 1), (big, 1). Map, split 2: (data, 1), (is, 1), (new, 1). Seven pairs in total.
- Combine (local reduce on each mapper): split 1 becomes (big, 2), (data, 1), (is, 1); split 2 stays (data, 1), (is, 1), (new, 1). Pairs sent across the network fall from 7 to 6; on real text with many repeated words the saving is far larger.
- Shuffle and sort: big → [2]; data → [1, 1]; is → [1, 1]; new → [1].
- Reduce (sum the list): big 2, data 2, is 2, new 1.
- Partitioning with R = 2 reducers: each key goes to reducer hash(key) mod 2, so all values of a key always meet at the same reducer.
Worked example 2 – a combiner must be used with care. We want the average temperature. Mapper 1 sees 20 and 30; mapper 2 sees 40.
- Wrong: each mapper sends its average — 25 and 40 — and the reducer averages them: (25 + 40) / 2 = 32.5.
- True average: (20 + 30 + 40) / 3 = 90 / 3 = 30.
- Right: mappers send (sum, count) pairs — (50, 2) and (40, 1) — and the reducer computes (50 + 40) / (2 + 1) = 30. A combiner is safe only for operations that are associative and commutative, such as sum, count, max and min.
11.2 Hadoop
Apache Hadoop is an open-source framework for the distributed storage and processing of very large datasets on clusters of commodity hardware. It was created by Doug Cutting and Mike Cafarella, inspired by Google's papers on the Google File System and MapReduce, and it is now an Apache Software Foundation project. Hadoop has four core modules. HDFS (Hadoop Distributed File System) stores data across the cluster (section 11.3). YARN (Yet Another Resource Negotiator) manages the cluster's resources: a ResourceManager allocates CPU and memory across the cluster, a NodeManager on each machine runs and monitors containers, and an ApplicationMaster manages each job. MapReduce is the original processing engine, running on YARN. Hadoop Common provides shared libraries and utilities. Around this core has grown a large ecosystem: Pig and Hive for high-level data processing, HBase for real-time NoSQL access, Sqoop for transferring data to and from relational databases, Flume and Kafka for ingesting streams, Oozie for scheduling workflows, ZooKeeper for coordination, and Spark for fast in-memory processing. Hadoop's key properties are horizontal scalability (add machines to add capacity), fault tolerance (data replication and automatic re-running of failed tasks), data locality (moving computation to the data rather than data to the computation), and low cost, because it runs on ordinary servers.
Worked example – capacity planning for a Hadoop cluster. A cluster has 20 DataNodes, each with 12 TB of disk, and HDFS uses replication factor 3.
- Raw capacity = 20 × 12 TB = 240 TB.
- Usable capacity after replication = 240 / 3 = 80 TB.
- About 25% is reserved for intermediate MapReduce output and temporary files, leaving 80 × 0.75 = 60 TB for data.
- If the company adds 2 TB of new data per month, the cluster is full in 60 / 2 = 30 months; adding 5 more nodes adds 5 × 12 / 3 × 0.75 = 15 TB, or another 7.5 months. Capacity grows simply by adding machines.
11.3 Hadoop Distributed File System (HDFS)
HDFS is the storage layer of Hadoop, designed to store very large files reliably across many machines and to deliver high throughput to batch processing jobs. It follows a master–slave architecture. The NameNode (master) holds the metadata — the directory tree, file permissions, and the mapping of each file to its blocks and of each block to the DataNodes that hold it — and keeps it in memory for speed. The DataNodes (slaves) store the actual data blocks on their local disks, serve read and write requests, and send the NameNode a heartbeat every 3 seconds and periodic block reports; a DataNode that stops sending heartbeats is declared dead, and its blocks are re-replicated elsewhere. A Secondary NameNode periodically merges the NameNode's edit log into its file-system image (a checkpoint); for high availability, production clusters run an active and a standby NameNode. Files are split into large blocks — 128 MB by default — and each block is replicated (3 copies by default). With rack awareness, HDFS places the first replica on the writer's node, the second on a node in a different rack, and the third on another node in that second rack, so data survives the loss of an entire rack. HDFS follows a write-once, read-many model: files can be appended to but not modified in place. To write, the client asks the NameNode for target DataNodes and streams each block through a pipeline of the three DataNodes; to read, the client gets the block locations from the NameNode and reads directly from the nearest DataNode. HDFS is optimised for large files and sequential access; it is not suited to millions of tiny files or to low-latency random access.
The client asks the NameNode where to write, then streams block B1 to D1, which passes it along the pipeline to D4 and D5 on the second rack (the shaded nodes). Every block has three replicas, spread so that no single rack holds all of them.
Worked example – storing a 500 MB file. Block size 128 MB, replication factor 3, two racks.
- Number of blocks = ⌈500 / 128⌉ = 4: three full blocks of 128 MB (384 MB) and a last block of 500 − 384 = 116 MB. The last block uses only 116 MB of disk, not 128 MB.
- Block replicas stored = 4 × 3 = 12, using 500 × 3 = 1,500 MB of disk.
- Placement of block 1 written from node D1 on rack 1: replica 1 on D1, replica 2 on a node in rack 2 (say D4), replica 3 on another node in rack 2 (D5).
- If D1 fails, block 1 is still on D4 and D5; the NameNode notices the missing heartbeats after the timeout and copies block 1 to a third node to restore 3 replicas.
- The NameNode holds only the metadata — about 150 bytes per file and per block in memory — so this file costs it roughly (1 + 4) × 150 = 750 bytes.
11.4 Pig
Apache Pig is a high-level platform for analysing large datasets on Hadoop, originally developed at Yahoo. Programs are written in Pig Latin, a data-flow language in which each statement transforms a relation and names the result, so a script reads as a step-by-step pipeline. Pig compiles the script into a series of MapReduce (or Tez or Spark) jobs, which saves the programmer from writing Java MapReduce code: a task that needs hundreds of lines of Java may need ten lines of Pig Latin. Pig's data model has four types: an atom (a single value), a tuple (an ordered set of fields, like a row), a bag (a collection of tuples, possibly with duplicates) and a map (key–value pairs). Its main operators are LOAD, FILTER, FOREACH … GENERATE (transform each tuple), GROUP, JOIN, ORDER BY, DISTINCT, LIMIT and STORE (or DUMP to the screen). Pig runs in local mode for testing on one machine or in MapReduce mode on a cluster, interactively through the Grunt shell or as a script. It is well suited to ETL pipelines and to semi-structured data, because it does not require a schema in advance (schema on read), and its behaviour can be extended with user-defined functions (UDFs).
sales = LOAD 'sales.csv' USING PigStorage(',') AS (region:chararray, amount:int);
big = FILTER sales BY amount > 40;
byreg = GROUP big BY region;
totals = FOREACH byreg GENERATE group AS region, SUM(big.amount) AS total;
sorted = ORDER totals BY total DESC;
DUMP sorted;
Worked example – tracing the script. sales.csv contains (N, 100), (S, 50), (N, 200), (E, 70), (S, 30).
- FILTER amount > 40 removes (S, 30), leaving (N, 100), (S, 50), (N, 200), (E, 70).
- GROUP BY region produces bags: (N, {(N,100), (N,200)}), (S, {(S,50)}), (E, {(E,70)}).
- FOREACH … SUM gives (N, 300), (S, 50), (E, 70).
- ORDER BY total DESC outputs (N, 300), (E, 70), (S, 50). Behind the scenes, Pig turned the GROUP into the shuffle of a MapReduce job.
11.5 Hive
Apache Hive is a data-warehouse system built on Hadoop, originally developed at Facebook, that lets users query large datasets with HiveQL, a SQL-like language. Hive translates each query into MapReduce, Tez or Spark jobs, so analysts who know SQL can work with big data without writing programs. Table definitions — names, columns, types and file locations — are stored in the metastore, a relational database, while the data itself stays in HDFS (or S3) as files. Hive applies the schema when data is read rather than when it is loaded (schema on read). Managed tables are fully controlled by Hive, so dropping the table deletes the data; external tables only point to data, which survives a drop. Partitioning divides a table into sub-directories by the value of a column such as date or country, so a query that filters on that column reads only the relevant partitions (partition pruning). Bucketing further divides data into a fixed number of files by the hash of a column, which speeds up joins and sampling. Hive is designed for batch analytical queries over very large data, not for row-level updates or low-latency transactions; for interactive speed it is often run on Tez or Spark or replaced by engines such as Impala, Presto or Spark SQL.
CREATE TABLE sales (customer_id INT, product STRING, amount DOUBLE)
PARTITIONED BY (sale_date STRING)
CLUSTERED BY (customer_id) INTO 32 BUCKETS
STORED AS ORC;
SELECT product, SUM(amount) AS revenue
FROM sales
WHERE sale_date = '2026-03-15'
GROUP BY product
ORDER BY revenue DESC
LIMIT 10;
Worked example – partition pruning and bucketing.
- The sales table holds one year of data, 3.65 TB, partitioned by sale_date into 365 partitions of about 3.65 TB / 365 = 10 GB each.
- The query above filters on one date, so Hive reads only that partition: 10 GB instead of 3,650 GB, i.e. 10 / 3,650 = 0.27% of the data — about 365 times less work.
- With 32 buckets on customer_id (for an integer the hash is the value itself), customer 1045 is stored in bucket 1045 mod 32 = 21. A query or join for that customer needs to read only bucket 21 of the partition.
| Basis | Pig | Hive |
|---|---|---|
| Language | Pig Latin (procedural data flow) | HiveQL (declarative, SQL-like) |
| Typical users | Programmers, data engineers | Analysts who know SQL |
| Best for | ETL pipelines, semi-structured data | Reporting and ad-hoc analytical queries |
| Schema | Optional | Defined in the metastore |
| Origin | Yahoo |
Independent study notes written in my own words, for learning purposes. Not affiliated with any university or course provider. R, RStudio, Hadoop, Amazon S3, Tableau, Power BI and other product names are trademarks of their respective owners.