Pipes, variables and scripts

Engineering Toolkit, lesson 2

Join small commands into one, and save the steps as a script.

Each shell command does one small job. This lesson joins them up: send output to a file, feed one command into the next, hand settings to a program, and save the steps as a script that stops when something fails. You need the shell basics from "Files and folders in the shell".

By the end, you can:

Send a program's results and its errors to different places.
Chain small commands with pipes to answer a question about a log.
Hand a setting to a program through its environment.
Save commands as a script that stops when a step fails.

Part 1. Results and errors go separate ways

Send a program's results and its errors to different places.

Every program has two places to write: one for its results, one for warnings and errors. On screen the two look the same, but the shell can send them to different places. This small script writes to both.

report.py

import sys

print("rows: 12")
print("warning: 2 rows have no rating", file=sys.stderr)
print("mean rating: 3.9")

Splitting results from warnings

$ python3 report.py
rows: 12
warning: 2 rows have no rating
mean rating: 3.9
$ python3 report.py > out.txt
warning: 2 rows have no rating
$ cat out.txt
rows: 12
mean rating: 3.9
$ python3 report.py > out.txt 2> errors.txt
$ cat errors.txt
warning: 2 rows have no rating

Example 1.1

Where each line goes

report.py prints two results and one warning. Where does each line end up in each of the three runs?

Run it plainly3 lines on screen
Both outputs go to the screen, in the order they were written.
Redirect the results> out.txt → 1 line on screen
> points the results at out.txt. The warning leaves by the other way, which still leads to the screen.
Redirect the warning too2> errors.txt → 0 lines on screen
2> points the second output at a file of its own.

AnswerThe results in out.txt, the warning in errors.txt, nothing on screen.

The two outputs are numbered streams. Results go to stream 1, standard output, and > is short for 1>. Warnings and errors go to stream 2, standard error, so a file of results stays clean. Python's print writes to stream 1, and Python's own error messages, tracebacks included, go to stream 2.

Figure 1report.py writes to two streams. > sends stream 1 to out.txt, and stream 2 still reaches the screen until 2> sends it somewhere too.

Standard output and standard error

The two streams a program writes to: results to standard output (stream 1), warnings and errors to standard error (stream 2). > sends stream 1 to a file, replacing what was in it, >> adds to the end instead, and 2> sends stream 2.

python3 report.py > out.txt 2> errors.txt

2>&1 means "send stream 2 wherever stream 1 goes", and order matters: write > train.log 2>&1 to keep both in one file. The other way round, errors go where stream 1 pointed before, the screen. And > empties its file before the command starts, so sort names.txt > names.txt leaves names.txt empty.

Your turn

You run python3 train.py > log.txt, and the script crashes. Where does the traceback end up?

Python writes tracebacks to standard error, and > only redirects standard output. So log.txt holds everything printed before the crash, and the error shows on screen. Add 2>&1 to the end of the command to keep both in the file.

Show the answer

B. Python writes tracebacks to standard error, and > only redirects standard output. So log.txt holds everything printed before the crash, and the error shows on screen. Add 2>&1 to the end of the command to keep both in the file.

Part 2. Pipes join commands

Chain small commands with pipes to answer a question about a log.

The | character joins two commands: the first one's output becomes the second one's input, with no file in between. Here's a simplified web server log. Each line is one request: the address, the time, the method, the path, the status code and the size of the reply in bytes.

Counting failed requests by path

$ head -n 3 access.log
203.0.113.7 09:12:01 GET /api/runs 200 512
198.51.100.23 09:12:04 POST /api/runs 201 128
203.0.113.7 09:12:09 GET /api/runs/7 500 64
$ grep ' 500 ' access.log | wc -l
5
$ grep ' 500 ' access.log | cut -d' ' -f4 | sort | uniq -c | sort -rn
      3 /api/runs/7
      2 /api/login

Example 2.1

Which paths fail most?

Status 500 means the server failed. Which paths in access.log failed, and how often each?

Keep the failuresgrep ' 500 ' → 5 of 14 lines
grep keeps the lines that contain " 500 ". The spaces stop it matching a size such as 1500.
Keep the pathcut -d' ' -f4
cut splits each line at the spaces (-d' ') and keeps the fourth field (-f4).
Put copies side by sidesort
Sorting puts identical paths next to each other.
Count each rununiq -c → 2 and 3
uniq -c merges each run of identical lines into one line, with its count in front.
Biggest firstsort -rn
-n sorts by number and -r reverses, so the worst path is on top.

Answer/api/runs/7 failed 3 times and /api/login twice.

Each command does one small job and knows nothing about the others. The shell connects them and runs them all at once, each reading what the one before writes. Swap one stage and you have a new tool: cut -d' ' -f5 instead of -f4 counts status codes.

Figure 2The five failures flowing through the pipe: the paths, sorted so copies touch, counted, then ranked. On a Mac, uniq -c pads its counts with fewer spaces than on Linux.

Pipe

The | between two commands. The first command's standard output becomes the second one's input. Standard error doesn't go down the pipe: it still goes to the screen.

sort | uniq -c | sort -rn counts lines and ranks them

Because errors skip the pipe, a mistake can hide in plain sight. If access.log were missing, grep ' 500 ' access.log | wc -l would show grep's complaint on screen and then print 0, a count that looks like good news.

Your turn

Without the first sort, cut -d' ' -f4 access.log | uniq -c lists /api/runs on several different lines, each with a small count. Why?

uniq compares each line with the one just before it. Sorting first puts every copy of a path side by side, so each path ends up on one line with one total. That's why sort | uniq -c | sort -rn comes as a set: it's how the shell counts things.

Show the answer

D. uniq compares each line with the one just before it. Sorting first puts every copy of a path side by side, so each path ends up on one line with one total. That's why sort | uniq -c | sort -rn comes as a set: it's how the shell counts things.

Part 3. Settings a program can see

Hand a setting to a program through its environment.

Every command you run starts a process, a running program. As it starts, it gets a copy of a list of named settings from the shell that started it. Python reads that list as os.environ. A variable you set at the prompt joins the list only when you export it.

Set, export, read

$ echo $HOME
/home/ana
$ DATA_DIR=/home/ana/data
$ python3 -c 'import os; print(os.environ.get("DATA_DIR"))'
None
$ export DATA_DIR
$ python3 -c 'import os; print(os.environ.get("DATA_DIR"))'
/home/ana/data
$ LOG_LEVEL=debug python3 -c 'import os; print(os.environ["LOG_LEVEL"])'
debug

Example 3.1

Three ways to set a variable

Each python3 -c line prints a variable from its environment. When does Python see the value?

Set a shell variableDATA_DIR=/home/ana/data → None
The shell has the variable, but it isn't exported, so the new process gets no copy.
Export itexport DATA_DIR → /home/ana/data
Now every command started from this shell gets a copy.
Set it for one commandLOG_LEVEL=debug python3 … → debug
An assignment in front of a command puts the variable in that command's environment only.

AnswerOnly after export, or when it's set in front of the command itself.

A process gets a copy, not a link. Change a variable in the shell after a program has started and the program never sees it. Change one inside a program and the shell never sees it either.

Figure 3When python3 starts, it gets a copy of the shell's exported variables. DATA_DIR isn't exported yet, so Python's copy has no DATA_DIR.

Environment variable

A named setting, like HOME or DATA_DIR, that each process gets a copy of when it starts. export adds a shell variable to the list that gets copied, and Python reads the list through os.environ.

export DATA_DIR, then os.environ.get("DATA_DIR")

One variable decides what runs when you type a command's name. PATH lists folders, separated by :. The shell searches them in order and runs the first program it finds with that name. command -v python3 shows which one wins, and "command not found" means none of them has it.

Your turn

You type DATA_DIR=/data at the prompt, then run python3 load.py, which reads os.environ.get("DATA_DIR"). What does it get?

Without export, the variable belongs to the shell alone, and a new process only gets the exported ones. export DATA_DIR=/data passes it to every command after it. DATA_DIR=/data python3 load.py, on one line, passes it to that one command only.

Show the answer

B. Without export, the variable belongs to the shell alone, and a new process only gets the exported ones. export DATA_DIR=/data passes it to every command after it. DATA_DIR=/data python3 load.py, on one line, passes it to that one command only.

Part 4. A script that stops at the first error

Save commands as a script that stops when a step fails.

Once you've typed the same commands twice, put them in a file. A shell script is those commands, one per line, under a first line that says which program runs the file. This one summarises the web log from part 2.

report.sh

#!/usr/bin/env bash
# Summarise a web server log: ./report.sh access.log
set -euo pipefail

log="$1"
total=$(wc -l < "$log")
errors=$(grep -c ' 500 ' "$log" || true)
echo "requests: $total"
echo "server errors: $errors"

Running it, and making it fail

$ chmod +x report.sh
$ ./report.sh access.log
requests: 14
server errors: 5
$ ./report.sh missing.log
./report.sh: line 6: missing.log: No such file or directory
$ echo $?
1
$ ./report.sh
./report.sh: line 5: $1: unbound variable
$ ./report.sh access.log && echo "report done"
requests: 14
server errors: 5
report done

Example 4.1

Three runs of report.sh

What does each run of report.sh do, and why does it stop where it does?

Make it runnablechmod +x report.sh
Marks the file as a program. Its first line, #!/usr/bin/env bash, says bash runs it.
Run it on the logrequests: 14, server errors: 5
$1 is the first word after the script's name, and $(...) swaps in a command's output. A Mac's wc pads the 14 with spaces.
Give it a missing fileecho $? → 1
The shell can't open missing.log for wc, so line 6 fails with code 1. set -e ends the script there, with that code.
Forget the file name$1: unbound variable
set -u stops on a variable that was never set, instead of quietly using an empty string.

AnswerIt reports on a real log, and anything else stops it at the first problem, with a code that isn't 0.

Every program ends by handing back a number: 0 if it worked, anything else if it didn't. $? holds the last one. && runs the next command only after a 0, as in the session's last line, and || only after anything else. set -e makes a script stop at the first command that doesn't return 0.

Exit code

The number a program returns when it ends: 0 for success, anything else for failure. The shell keeps the last one in $?, and &&, || and set -e act on it.

set -euo pipefail: stop on a failure, an unset variable or a failed pipe stage

Without pipefail, a pipe's exit code is its last stage's alone, so the missing-log pipe from part 2 returns 0, because wc worked. And grep returns 1 when it finds nothing, which set -e would treat as a failure. || true on line 7 says no server errors is a normal result.

Your turn

A script with no set line runs python3 clean.py and then python3 train.py. clean.py crashes halfway through. What happens next?

By default bash carries on after a failure, and the script's own exit code is just that of its last command, so it can even report success. That's how a model ends up trained on half-cleaned data. set -e makes the script stop at the first command that fails.

Show the answer

C. By default bash carries on after a failure, and the script's own exit code is just that of its last command, so it can even report success. That's how a model ends up trained on half-cleaned data. set -e makes the script stop at the first command that fails.

Where you'll meet it

In interviews

Asked which value turns up most often in a log or a CSV column, you can answer in one line: pick the column with cut, then sort | uniq -c | sort -rn | head -n 3. Say why the first sort is there: uniq only merges lines that sit next to each other.

On the job

Services read their settings from environment variables: a database address, a log level, an API key. The same code then runs on a laptop and on a server with different values, and keys stay out of the code. The Twelve-Factor App, a guide to building web apps, recommends exactly this.

The same pipeline, counting status codes

$ cut -d' ' -f5 access.log | sort | uniq -c | sort -rn
      7 200
      5 500
      1 404
      1 201

Field 5 is the status code, so the same four stages now count every code in the log: 7 requests got 200 and 5 got 500.

Check yourself

Four questions from across the lesson. Try each one before you look back.

names.txt holds three names. You run sort names.txt > names.txt. What's in names.txt afterwards?

The shell sets up > before sort starts, and > empties the file first. So sort reads an empty file and writes nothing, and the command still returns 0. Write to a new file, then rename it with mv.

Show the answer

D. The shell sets up > before sort starts, and > empties the file first. So sort reads an empty file and writes nothing, and the command still returns 0. Write to a new file, then rename it with mv.

Your PATH is /usr/local/bin:/usr/bin:/bin, and both /usr/local/bin and /usr/bin hold a program called python3. You type python3. Which one runs?

The shell searches PATH's folders from left to right and runs the first match. command -v python3 would print /usr/local/bin/python3. Activating a Python virtual environment works the same way: it puts the environment's bin folder at the front of PATH.

Show the answer

D. The shell searches PATH's folders from left to right and runs the first match. command -v python3 would print /usr/local/bin/python3. Activating a Python virtual environment works the same way: it puts the environment's bin folder at the front of PATH.

A script has set -e but not pipefail. One line is grep ERROR app.log | wc -l, and app.log doesn't exist. What happens?

Without pipefail, a pipe's exit code is its last stage's, and wc worked, so the code is 0 and set -e sees no failure. With set -o pipefail, grep's code 2 becomes the pipe's, and the script stops.

Show the answer

B. Without pipefail, a pipe's exit code is its last stage's, and wc worked, so the code is 0 and set -e sees no failure. With set -o pipefail, grep's code 2 becomes the pipe's, and the script stops.

A file holds three lines, but the last one has no newline after it. What does wc -l report?

wc -l counts newline characters, and only two lines end with one. Python's len(open(f).readlines()) would say 3, so the two can disagree by one.

Show the answer

B. wc -l counts newline characters, and only two lines end with one. Python's len(open(f).readlines()) would say 3, so the two can disagree by one.

What to remember

1Results and errors go separate ways
Results go to standard output and errors to standard error: > out.txt catches only the results, 2> errors.txt the errors, and > log 2>&1 both.
2Pipes join commands
A pipe feeds one command's output into the next; sort | uniq -c | sort -rn counts and ranks lines, and needs the first sort because uniq only merges neighbours.
3Settings a program can see
A process gets a copy of the exported variables when it starts, so os.environ.get("DATA_DIR") is None until you export DATA_DIR.
4A script that stops at the first error
Exit code 0 means success, and set -euo pipefail stops a script at the first failure, unset variable or failed pipe stage.

Recap and going further

The lesson on one page

Two output streamsResults go to standard output (1)
Errors go to standard error (2)
Redirection> replaces a file, >> adds to it
2> catches errors, > log 2>&1 keeps both
> empties its file before the command starts
Pipes| feeds one command into the next
sort | uniq -c | sort -rn counts things

Small commands, joined up

The environment

Each process gets a copy at start
Only exported variables are passed on
PATH decides which program runs

Scripts

#!/usr/bin/env bash and chmod +x
set -euo pipefail stops on errors

Exit codes

0 means success
$?, && and || read it

Try this

Answer a question about a real log with one pipeline

Take any log or CSV you have on your machine: a web server log, a training log, an export from a spreadsheet. Pick one question about it, such as "which value appears most in column 3?", and answer it with cut, sort, uniq -c and sort -rn. Then put the pipeline in a script that takes the file name as $1, with set -euo pipefail at the top.

You need: A terminal and any text file with one record per line.

Look for

What changes when you take the first sort out.
Whether your file's columns are split by spaces, commas or tabs (cut -d',' for commas, cut -f2 for tabs).
What your script does when you forget the file name, with and without set -u.
The exit code (echo $?) after a run that worked and one that didn't.

The words, in plain language

Standard output and standard error: The two streams a program writes to: results to standard output (stream 1), warnings and errors to standard error (stream 2). > sends stream 1 to a file, replacing what was in it, >> adds to the end instead, and 2> sends stream 2.
Pipe: The | between two commands. The first command's standard output becomes the second one's input. Standard error doesn't go down the pipe: it still goes to the screen.
Environment variable: A named setting, like HOME or DATA_DIR, that each process gets a copy of when it starts. export adds a shell variable to the list that gets copied, and Python reads the list through os.environ.
Exit code: The number a program returns when it ends: 0 for success, anything else for failure. The shell keeps the last one in $?, and &&, || and set -e act on it.

Watch

Command-line EnvironmentThe Missing Semester of Your CS Education (MIT, 2026), on YouTubeArguments, streams, environment variables, exit codes and signals: the ideas under this lesson, typed live.

02 · Pipes, variables and scripts

Where you'll meet it

Check yourself

What to remember

Recap and going further

Answer a question about a real log with one pipeline