Linux Interview Questions for Data Engineers

Linux interview questions for data engineers are less about system administration than about two practical skills: sizing up and slicing data files from the command line, and writing shell steps that fail loudly instead of silently passing bad data on. The answers below come from running real commands on real files. The main one is the UCI household power dataset, about two million one-minute readings with "?" marking missing values (UCI: household power consumption); the other is the NYC taxi zone lookup table.

The commands run through GNU bash from a short Python driver, so the output can be captured and repeated. They were run with Git for Windows' GNU tools; on Linux the same commands give the same results.

import shutil
import subprocess

BASH = shutil.which("bash")                                     # the bash on PATH: Linux, macOS or Git Bash


def sh(title, script):
    out = subprocess.run([BASH, "-c", script], capture_output=True, text=True)
    print(f"$ {title}")
    for line in (out.stdout + out.stderr).splitlines():
        if line.strip():
            print("   " + line.strip())


POWER = "household_power_consumption.txt"                       # UCI household power, ';'-separated, '?' = missing

sh("L1 size up a big file without opening it", f"""
wc -l < {POWER}; du -h {POWER} | cut -f1; head -2 {POWER} | cut -c1-60""")

sh("L2 count rows with missing readings", f"grep -c '?' {POWER}")

sh("L3 average power per year with awk, skipping missing values", f"""
awk -F';' 'NR > 1 && $3 != "?" {{ split($1, d, "/"); sum[d[3]] += $3; n[d[3]]++ }}
           END {{ for (y in sum) printf "%s %.3f kW over %d readings\\n", y, sum[y] / n[y], n[y] }}' {POWER} | sort""")

sh("L4 the most common values in a column: sort | uniq -c | sort -rn", """
tail -n +2 taxi_zone_lookup.csv | cut -d, -f2 | tr -d '"' | sort | uniq -c | sort -rn | head -4""")

sh("L5 a failing step hidden inside a pipeline", """
grep -c 'Manhattan' no_such_file.csv 2>/dev/null | wc -l; echo "exit status without pipefail: $?"
set -o pipefail
grep -c 'Manhattan' no_such_file.csv 2>/dev/null | wc -l; echo "exit status with pipefail: $?" """)

sh("L6 split a file and process the parts in parallel with xargs -P", f"""
rm -rf parts && mkdir parts && tail -n +2 {POWER} > body.txt     # split -n needs a real file, not a pipe
split -n l/4 body.txt parts/chunk_
ls parts | xargs -P 4 -I {{}} sh -c 'echo "{{}} $(grep -vc "?" parts/{{}})"' | sort
echo "total $(cat parts/* | grep -vc '?')"; rm -rf parts body.txt""")

sh("L7 find files by age and size", f"""
touch -d '10 days ago' old_export.csv && touch new_export.csv
find . -maxdepth 1 -name '*.csv' -mtime +7; find . -maxdepth 1 -size +100M -name '*.txt'
rm old_export.csv new_export.csv""")

sh("L8 environment variables reach child processes only when exported", """
RUN_DATE=2024-01-15; bash -c 'echo "child sees: [$RUN_DATE]"'
export RUN_DATE; bash -c 'echo "after export: [$RUN_DATE]"'""")

$ L1 size up a big file without opening it
   2075260
   127M
   Date;Time;Global_active_power;Global_reactive_power;Voltage;
   16/12/2006;17:24:00;4.216;0.418;234.840;18.400;0.000;1.000;1
$ L2 count rows with missing readings
   25979
$ L3 average power per year with awk, skipping missing values
   2006 1.901 kW over 21992 readings
   2007 1.117 kW over 521669 readings
   2008 1.072 kW over 526905 readings
   2009 1.079 kW over 521320 readings
   2010 1.061 kW over 457394 readings
$ L4 the most common values in a column: sort | uniq -c | sort -rn
   69 Queens
   69 Manhattan
   61 Brooklyn
   43 Bronx
$ L5 a failing step hidden inside a pipeline
   0
   exit status without pipefail: 0
   0
   exit status with pipefail: 2
$ L6 split a file and process the parts in parallel with xargs -P
   chunk_aa 513521
   chunk_ab 515668
   chunk_ac 513551
   chunk_ad 506540
   total 2049280
$ L7 find files by age and size
   ./old_export.csv
   ./household_power_consumption.txt
$ L8 environment variables reach child processes only when exported
   child sees: []
   after export: [2024-01-15]

L1. How do you size up a large file without opening it?

wc -l counts lines, du -h gives the size on disk, and head shows the first lines. The power file has 2,075,260 lines (one header plus 2,075,259 readings), takes 127 MB, and is semicolon-separated. None of these commands loads the file into memory, which is the point: an editor might choke on a multi-gigabyte export, while head returns instantly.

L2. How many rows have missing values?

grep -c '?' counts matching lines without printing them: 25,979 rows contain a question mark. The interview point is to check what a missing value looks like before counting. Here it is a literal "?", elsewhere an empty field, "NA" or "null", and each needs a different pattern.

L3. Can you aggregate in the shell?

Yes, with awk. Splitting each line on ;, skipping the header and missing values, and keeping a running sum and count per year in associative arrays gave yearly averages: about 1.06 to 1.12 kW for each full year, and 1.901 for the two weeks of 2006, which are December, a heating month. awk streams the file line by line, so memory stays flat however large the file is. For anything more complex, move to Python or SQL; for a quick check during an incident, awk is often fastest.

L4. How do you find the most common values in a column?

The classic pipeline: cut the column, sort, uniq -c to count adjacent duplicates, then sort -rn to rank. Queens and Manhattan have 69 taxi zones each, Brooklyn 61 and the Bronx 43. uniq only collapses adjacent lines, which is why the first sort is required; forgetting it is a common interview slip.

L5. Why did my pipeline report success when a step failed?

Because a pipeline's exit status is the last command's status by default. grep on a missing file failed, but wc -l after it succeeded, so the pipeline returned 0 and printed a count of 0, a wrong answer that looks valid. With set -o pipefail, the pipeline returns the status of the last command that failed, here 2 (bash manual). Data pipeline scripts usually start with set -euo pipefail for exactly this reason: stop on errors, on unset variables, and on failures anywhere in a pipe.

L6. How do you process a big file in parallel?

Split it, then fan out. split -n l/4 cut the readings into four files without breaking any line (split), and xargs -P 4 counted the valid readings in all four at once (xargs). The four counts added up to 2,049,280, the same total as the whole file, which is the check worth doing every time you split work. A detail found along the way: split -n needs a real file, because it must know the size in advance, so it could not read from a pipe.

L7. How do you find old or large files?

find with -mtime and -size. A file touched ten days ago matched -mtime +7, a new one did not (find), and -size +100M picked out the power file. Combined with -delete or -exec, this is how temporary exports get cleaned up. Run it without the delete first, because a wrong pattern removes the wrong files.

L8. Why can't my script see a variable I just set?

Because a shell variable stays in the current shell until it is exported. A child process saw RUN_DATE as empty until export RUN_DATE, after which it saw 2024-01-15 (bash manual). Schedulers and job runners pass settings this way, so a missing export shows up as a job that runs with an empty date, which is one more reason to use set -u.

How to answer Linux questions

• Look before you load: wc, head and du first.
• Make failures loud with set -euo pipefail.
• Check totals after splitting or filtering.
• Dry-run anything destructive.

Shell steps usually sit inside larger pipelines; see building your first data pipeline and, for scheduling them, Airflow interview questions. For the formats that replace large text files, see Parquet vs ORC vs Avro, and for the full round, data engineering interview questions and answers.

Linux and shell skills for pipelines are part of the Data Engineering course.

Sources

• bash(1) Linux manual page — man7.org
• split(1) Linux manual page — man7.org
• xargs(1) Linux manual page — man7.org
• find(1) Linux manual page — man7.org
• Individual Household Electric Power Consumption (UCI Machine Learning Repository) — UCI Machine Learning Repository