Hadoop Interview Questions and Answers

Hadoop interview questions have changed since Hadoop was the default platform. Fewer teams run new MapReduce jobs, but HDFS, YARN and the MapReduce model still sit underneath many clusters and cloud services, and interviewers use them to test whether you understand how distributed storage and processing work. The answers below come from a real Hadoop 3.4.1 node, a NameNode and a DataNode in one Docker container, loaded with 2,075,259 one-minute electricity readings from a house near Paris (UCI: household power consumption). The file is 127 MB, just under one default block.

import os
import subprocess

# A single-node HDFS (NameNode + DataNode) and MapReduce in Apache's Hadoop 3.4.1 image
SCRIPT = r"""
set -e
cat > $HADOOP_CONF_DIR/core-site.xml <<'X'
<configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property></configuration>
X
cat > $HADOOP_CONF_DIR/hdfs-site.xml <<'X'
<configuration><property><name>dfs.replication</name><value>1</value></property></configuration>
X
hdfs namenode -format -force > /dev/null 2>&1
hdfs --daemon start namenode && hdfs --daemon start datanode
until hdfs dfsadmin -report 2>/dev/null | grep -q "Live datanodes (1)"; do sleep 2; done
hdfs dfsadmin -safemode wait > /dev/null 2>&1
q() { "$@" 2>/dev/null; }

echo "H1 blocks: the same 127 MB file at the default block size and at 32 MB"
q hdfs dfs -put /data/household_power_consumption.txt /power_default.txt
q hdfs dfs -D dfs.blocksize=33554432 -put /data/household_power_consumption.txt /power_32mb.txt
for f in /power_default.txt /power_32mb.txt; do
  echo "   $f: $(q hdfs fsck $f -files -blocks | grep -o 'Total blocks (validated):[[:space:]]*[0-9]*' | tr -s ' ')"
done

echo "H2 replication: ask for 3 copies on a cluster with 1 DataNode"
q hdfs dfs -setrep 3 /power_default.txt > /dev/null
echo "   $(q hdfs fsck /power_default.txt | grep -o 'Under-replicated blocks:[[:space:]]*[0-9]*' | tr -s ' ')"

echo "H3 small files: one month as 31 daily files vs one file"
q hdfs dfs -mkdir -p /daily
for d in $(seq -w 1 31); do grep "^$((10#$d))/1/2008;" /data/household_power_consumption.txt > /tmp/d$d.txt; done
q hdfs dfs -put /tmp/d*.txt /daily/
cat /tmp/d*.txt > /tmp/month.txt && q hdfs dfs -put /tmp/month.txt /month.txt
echo "   January 2008: $(du -m /tmp/month.txt | cut -f1) MB in total, about $(( $(du -k /tmp/month.txt | cut -f1) / 31 )) KB per daily file, block size 128 MB"
echo "   31 daily files: $(q hdfs fsck /daily | grep -o 'Total blocks (validated):[[:space:]]*[0-9]*' | tr -s ' ') | one file: $(q hdfs fsck /month.txt | grep -o 'Total blocks (validated):[[:space:]]*[0-9]*' | tr -s ' ')"

echo "H4 MapReduce (Hadoop Streaming): readings per year, map -> shuffle and sort -> reduce"
cat > /tmp/map.sh <<'X'
#!/bin/bash
# emit "year<TAB>1" for every reading; the header line has no slash in its first field
awk -F';' '$1 ~ /\// {split($1, d, "/"); print d[3] "\t1"}'
X
cat > /tmp/reduce.sh <<'X'
#!/bin/bash
# input arrives sorted by key, so each year's lines are together
awk -F'\t' '$1 != y {if (y != "") print y "\t" n; y = $1; n = 0} {n += $2} END {print y "\t" n}'
X
chmod +x /tmp/map.sh /tmp/reduce.sh
hadoop jar /opt/hadoop/share/hadoop/tools/lib/hadoop-streaming-3.4.1.jar \
  -D mapreduce.framework.name=local -D mapreduce.job.reduces=1 \
  -input /power_default.txt -output /per_year \
  -mapper /tmp/map.sh -reducer /tmp/reduce.sh > /tmp/job.log 2>&1 || { tail -30 /tmp/job.log; exit 1; }
q hdfs dfs -cat /per_year/part-00000 | sed 's/^/   /'
grep -E "Map input records|Map output records|Reduce input groups" /tmp/job.log | sed 's/^[[:space:]]*/   counter: /' | awk '!seen[$0]++'
"""

here = os.path.abspath(".").replace("\\", "/")
out = subprocess.run(["docker", "run", "--rm", "-v", f"{here}:/data", "apache/hadoop:3.4.1", "bash", "-c", SCRIPT],
                     capture_output=True, text=True, encoding="utf-8", errors="replace")
print(out.stdout.strip())

H1 blocks: the same 127 MB file at the default block size and at 32 MB
   /power_default.txt: Total blocks (validated):	1
   /power_32mb.txt: Total blocks (validated):	4
H2 replication: ask for 3 copies on a cluster with 1 DataNode
   Under-replicated blocks:	1
H3 small files: one month as 31 daily files vs one file
   January 2008: 3 MB in total, about 90 KB per daily file, block size 128 MB
   31 daily files: Total blocks (validated):	31 | one file: Total blocks (validated):	1
H4 MapReduce (Hadoop Streaming): readings per year, map -> shuffle and sort -> reduce
   2006	21996
   2007	525600
   2008	527040
   2009	525600
   2010	475023
   counter: Map input records=2075260
   counter: Map output records=2075259
   counter: Reduce input groups=5

H1. What is an HDFS block, and why does its size matter?

HDFS splits every file into blocks and stores each block on DataNodes. The default block size is 134,217,728 bytes, 128 MB (hdfs-default.xml). The 127 MB file fit in one block. Uploaded again with a 32 MB block size, the same file became four blocks.

Why it matters: for a splittable file, MapReduce and Spark normally create one input split, and so one task, per block. One block means one map task reads all 127 MB while other cores sit idle; four blocks allow four tasks in parallel. Very small blocks go the other way, with more tasks and more metadata than the work justifies. The block size is set per file when it is written, which is why -D dfs.blocksize on the upload was enough here.

Follow-up you may get: does a 1 KB file use a whole 128 MB block on disk? No. The block size is an upper limit; a small file's block only takes the space of its data. The cost of small files is elsewhere, as H3 shows.

H2. How does replication work, and what happens when it cannot be met?

Each block is copied to several DataNodes, three by default. With three replicas, the documented placement puts one on the writer's node, if the writer runs on a DataNode, and the other two on different nodes of one remote rack (HDFS architecture). Losing a whole rack therefore never loses every copy.

The node here had one DataNode, so the file was written with replication 1. Asking for three copies with setrep 3 was accepted, but HDFS could not place them, and fsck reported one under-replicated block. Nothing failed: reads still worked, because the minimum replication for a usable block is 1 by default. On a real cluster the NameNode keeps track of such blocks and copies them as soon as nodes are available. The same report appears when a DataNode dies: DataNodes send a heartbeat every 3 seconds by default, and the NameNode marks nodes without recent heartbeats as dead and re-replicates their blocks elsewhere.

What to say: replication is for durability and for data locality, since more copies give the scheduler more nodes where a task can read its block locally. In an interview, mention the cost too: three copies triple the storage.

H3. What is the small files problem?

The NameNode holds the whole namespace, every file, directory and block, in memory (HDFS architecture). January 2008 from the same data is 3 MB. Stored as 31 daily files of about 90 KB each, it became 31 files and 31 blocks for the NameNode to track; stored as one file, one file and one block. Multiply that by years of hourly files from thousands of sources and the NameNode's memory, not disk space, becomes the limit. Processing suffers too, because each small file usually becomes its own task, and starting a task can take longer than reading 90 KB.

Fixes to name: compact small files into larger ones on a schedule; write columnar formats such as Parquet or ORC in large files (see Parquet vs ORC vs Avro); batch streaming writes rather than writing one file per event; and, for old data that must stay as many files, Hadoop archives (HAR). The same problem exists in object-store data lakes, where many small files slow down every query that lists and opens them.

H4. Walk through a MapReduce job.

The job counted readings per year. In the map phase, each mapper read its block line by line and wrote a key and a value, here the year and a 1. Hadoop Streaming lets any executable act as mapper or reducer, reading lines on stdin and writing them to stdout, with the text before the first tab as the key (Hadoop Streaming). Then the framework shuffled and sorted, so all the lines for one year arrived at the same reducer, together. The reducer only had to add up a run of identical keys.

The counters are worth reading in an interview, because they explain the result. Map input records were 2,075,260, every line of the file; map output records were 2,075,259, because the header line was skipped; and reduce input groups were 5, one per year. The yearly totals check out against the calendar: 525,600 readings in 2007 and 2009 is 365 days of 1,440 minutes, and 527,040 in 2008 is a leap year. The 21,996 readings in 2006 cover the data's start on 16 December, and 2010 stops in November.

A trap worth mentioning: these are rows, not measurements. The dataset marks missing readings with "?", and those rows were counted too. Counting valid values would need a mapper that checks the measurement field, which is exactly the kind of detail interviewers like to probe.

A combiner, a local reduce on each mapper's output, would have shrunk the shuffle from two million lines to a handful per mapper. It works here because addition gives the same result however the inputs are grouped.

H5. What does YARN do?

YARN shares a cluster's resources between applications. The ResourceManager decides which application gets resources, a NodeManager on each machine runs and monitors containers, and every application has its own ApplicationMaster, which asks the ResourceManager for containers and works with the NodeManagers to run its tasks (Apache Hadoop YARN). MapReduce is one such application; Spark on YARN is another.

To be clear about the run above: the job used MapReduce's local mode, which runs map and reduce tasks inside one JVM without YARN. HDFS was real; the scheduling was not. On a cluster, the same command with mapreduce.framework.name=yarn would hand the job to the ResourceManager.

H6. What happens if the NameNode fails?

Before Hadoop 2.0.0, the NameNode was a single point of failure, and the cluster was unavailable until it came back. With high availability, one NameNode is Active and another is Standby; the Active one writes each namespace change to a majority of at least three JournalNodes, and the Standby reads them to stay current and take over quickly (HDFS high availability).

The classic trap is the Secondary NameNode. It is not a backup NameNode: it merges the edit log into the namespace image so restarts are faster. In an HA cluster the Standby does that checkpointing, and the documentation says running a Secondary NameNode there would be an error.

Other questions to expect

• What changed in Hadoop 3? Erasure coding is the one to know. Three-way replication costs 200% extra storage; erasure coding costs no more than 50%, at the price of CPU for encoding and more network traffic, so it suits warm and cold data (HDFS erasure coding).
• What is data locality? Running a task on a node that already holds its block, so the data does not cross the network.
• What is speculative execution? Starting a second copy of a slow task on another node and taking whichever finishes first.
• Why did Spark replace MapReduce for most work? MapReduce writes intermediate results to disk between jobs; Spark keeps them in memory and plans whole pipelines at once. The history is in Spark vs Hadoop.

How to answer Hadoop questions

• Explain the mechanism, then its cost: blocks and parallelism, replicas and storage, files and NameNode memory.
• Connect it to today's tools. Spark, cloud data lakes and managed services such as Amazon EMR still inherit these trade-offs.
• Use real numbers where you can, as the counters above do.

Spark questions are covered in Apache Spark interview questions, the shell commands used here in Linux interview questions, and the theory behind replication in distributed systems basics.

Hadoop, Spark and distributed processing are part of the Data Engineering course.

Sources

• HDFS Architecture (Apache Hadoop documentation) — The Apache Software Foundation
• hdfs-default.xml (Apache Hadoop 3.4.1) — The Apache Software Foundation
• Hadoop Streaming (Apache Hadoop documentation) — The Apache Software Foundation
• Apache Hadoop YARN (Apache Hadoop documentation) — The Apache Software Foundation
• HDFS High Availability Using the Quorum Journal Manager (Apache Hadoop documentation) — The Apache Software Foundation
• HDFS Erasure Coding (Apache Hadoop documentation) — The Apache Software Foundation
• Individual Household Electric Power Consumption (UCI Machine Learning Repository) — UCI Machine Learning Repository