Pipe Commands

A pipeline connects one command’s output directly to another command’s input. In Unix-like shells, it is created with th

A pipeline connects one command’s output directly to another command’s input. In Unix-like shells, it is created with the pipe symbol (|), allowing small, focused utilities to form a larger data-processing workflow:

command1 | command2 | command3

This design reflects a central Unix principle: each program should perform a well-defined task, while the shell coordinates those programs into a complete solution. Pipelines are widely used for log analysis, text filtering, process management, and automation.
Most processes have three standard streams:

  • Standard input (stdin), usually attached to file descriptor 0
  • Standard output (stdout), attached to file descriptor 1
  • Standard error (stderr), attached to file descriptor 2

By default, a pipeline connects the standard output of one command to the standard input of the next:

echo "hello pipe" | tr 'a-z' 'A-Z'

The result is:

HELLO PIPE

Pipeline stages generally run concurrently. Data moves between them as it becomes available, so intermediate results do not need to be stored in temporary files. This makes pipelines efficient for processing large or continuous streams of information.

Common Pipeline Patterns

Pipelines are especially effective when filtering, sorting, counting, and presenting data in sequence.

Count .log files in the current directory:

ls -l | grep '\.log$' | wc -l

Find processes associated with Nginx without matching the grep command itself:

ps aux | grep '[n]ginx'

Identify the ten IP addresses with the most requests in an access log:

awk '{print $1}' access.log \
  | sort \
  | uniq -c \
  | sort -nr \
  | head -10

Follow a log file and display only error messages:

tail -f app.log | grep --line-buffered ERROR

Send potentially lengthy output to a pager:

dmesg | less

Each command in these examples has a narrow responsibility. Together, they create a readable processing pipeline.

Pipelines and Redirection

Pipelines and redirection both control where output goes, but they serve different purposes:

  • > redirects standard output to a file.
  • 2> redirects standard error.
  • 2>&1 sends standard error to the same destination as standard output.
  • | sends standard output to another command.

Standard error does not automatically enter a pipeline. To include it, redirect it before the pipe operator:

command 2>&1 | grep error

Bash also provides the |& shorthand:

command |& grep error

To display output while also saving it, use tee:

command | tee output.log | grep error

The tee command copies standard input to a file while continuing to write it to standard output. This is useful when a script needs both a live view and a persistent record.

Moving Stream Data into Arguments

Pipelines transport standard input, but many programs expect filenames or other values as command-line arguments. The xargs utility bridges this gap by converting input into arguments:

find . -name '*.tmp' -print0 | xargs -0 rm -f

Here, find emits null-terminated filenames, and xargs -0 reads them using the same delimiter. This approach safely handles names containing spaces, quotation marks, and other special characters.

A simpler version is often sufficient for ordinary input:

cat hosts.txt | xargs ping -c 1

Detecting Pipeline Failures

By default, the exit status of a pipeline comes from its final command. For example:

false | true
echo $?

The result is 0 because true, the last command, succeeded. This behavior can hide an earlier failure.

In Bash, enable pipefail to make the pipeline return a nonzero status if any stage fails:

set -o pipefail

false | true
echo $?

Bash also exposes the status of every stage through the PIPESTATUS array:

command1 | command2 | command3
echo "${PIPESTATUS[@]}"

The array contains exit codes in pipeline order. It is particularly useful in scripts that need to determine exactly which stage failed.

Streaming and Buffering

Although pipelines are stream-oriented, some programs buffer output when writing to a pipe rather than a terminal. This can delay live updates or make a log monitor appear unresponsive.

On systems that provide stdbuf, line buffering can be requested explicitly:

stdbuf -oL command | grep --line-buffered pattern

Many programs also offer their own buffering options. For example, GNU grep supports --line-buffered, while some logging tools provide immediate or unbuffered output modes.

Pipelines in PowerShell

PowerShell also uses |, but its pipeline usually carries structured objects rather than raw text:

Get-Process |
  Where-Object CPU -gt 100 |
  Sort-Object CPU -Descending |
  Select-Object -First 5

This command retrieves process objects, keeps those whose CPU property exceeds 100, sorts them in descending order, and selects the first five. Because properties remain available throughout the pipeline, PowerShell can filter and transform objects without parsing formatted text.

Practical Guidelines

  • Keep each stage focused. A command with one clear responsibility makes the pipeline easier to understand and debug.

  • Format long pipelines across multiple lines. A trailing backslash improves readability:

    cat access.log \
  | awk '{print $1}' \
  | sort \
  | uniq -c \
  | sort -nr \
  | head

  • Avoid unnecessary cat commands. If the next program can read a file directly, prefer:

    grep 'ERROR' app.log

    over:

    cat app.log | grep 'ERROR'

  • Enable pipefail in scripts. It prevents a successful final stage from concealing an earlier error.

  • Remember that sort may use substantial memory. Most pipeline stages stream data, but commands that perform global sorting often need to collect all or part of the input first.

  • Check exit statuses carefully. Commands such as head may close a pipe early, causing upstream programs to receive SIGPIPE. This can be normal in interactive use but should be considered in strict scripts.

  • Use null delimiters for untrusted filenames. The find -print0 and xargs -0 combination avoids many filename-parsing problems.

Mastering pipelines requires more than memorizing syntax. Understanding standard streams, process concurrency, exit statuses, and buffering enables reliable shell workflows for text processing, operational analysis, and automated tasks.