Lab 7: Pipes and filters in UNIX
In this lab, you will get to know the commands less and grep, and experiment with input and output redirection. Since Skill Demo 2 takes place next week (Week 8), we will then host a Practice Skill Demo on PrairieLearn.
Lab 7 learning objectives
- Use
lessandgrepto view and search through text files - Recognize and use pipe and redirection operators
Table of contents
Icebreaker
Can you guess what part of campus each of these photos was taken? Feel free to work with your groupmates! Google Maps is allowed!
Please write the answers of yourself and one of your group members in your lab report. No check-off is needed!
Searching and Filtering Program Output
Clone our starter code into your account on ieng6. In this lab, we’ll walk you through some helpful UNIX commands that are commonly used to save and filter program output. There are multiple ways to achieve the same result, and we’ll demonstrate advantages and disadvantages of each.
In the starter code, you are given a problem.c source code file. This program creates a large 2D array, fills it with numbers, and prints them out. Fortunately, whoever wrote this was thoughtful enough to write code to check for an out-of-bounds error. Unfortunately, whoever wrote this was not thoughtful enough to debug the program. Fortunately, you don’t need to debug this program; we’re going to use this program as an example of how the following UNIX commands can be used to parse a lot of program output easily.
Compile and run problem to see exactly what “a lot of program output” means.
$ make problem
$ ./problem
Every so often there’s an error message, but it’s separated by all the expected program output. For programs with more errors and more output, you could imagine how we wouldn’t want to keep scrolling up the terminal to read. There are multiple ways in which we can filter out everything but the error messages.
Say Less
In previous PAs, we’ve already used the indirection (<) and redirection (>) operators. These are used to feed input into a program, or redirect the output of a program into a file. Let’s first save the output of the program to a text file:
$ ./problem > problem.txt
Redirection overwrites any content of the destination file with the output. There also exists the append (>>) operator, which works similarly to redirection, but appends the output to the end of the destination file instead of overwriting.
$ ./problem >> problem.txt
You can use cat to view the contents of the file, but this is no better than letting the program output directly to the terminal. Instead, we can say less to start a program to display a portion of the file and navigate freely through it. It’s like cat, but shows less at a time.
$ less problem.txt
The one subcommand you need to know is to type h while in this program to display a list of subcommands. Some especially helpful subcommands include q to quit and f/b to move forward/backward one window. You may also notice that you can use j and k to move forward or backward by one line, just like in Vim. The subcommand we want to use to filter out all lines that do not have an error is to type “&” followed by the word “error”, then press Enter. less will insert a slash (“/”) in between to indicate that what follows is a pattern to match. This method of filtering is convenient if you don’t want to clutter the terminal interface, but requires that the output exists in a file already.
Once you have filtered the lines in problem.txt such that only the “error” lines remain, interpret the error and explain it to a tutor or TA. Then, put a screenshot of the filtered lines into your lab report.
Search with grep
The grep command is most commonly used to search for lines of a file that match some string. grep takes in two arguments: the “pattern” to match, and a file to read from.
$ grep error problem.txt
This command directly outputs to the terminal, which can be acceptable if you expect the output to be short, and helpful if you don’t want to run the search as a subcommand within another program, as with less.
Try practicing filtering file contents with message.txt, using either less or grep. This file contains the output of another command, which created a large array, assigned random capital character values, and printed it out. For some strange reason, the output shows that some characters were written way out of bounds at column index 42, but it’s hard to tell what’s happened when each line is separated by all the other output. Use less or grep to search for all lines that contain the string “42”. Are you surprised by what you find?
Flow Like Water
Piping
Both of these methods require you to create a file which contains the output of a file. This is generally pretty easy to do, but can be undesirable if you have a lot of output that you don’t want to be saved into a really large file (or are indecisive about naming files). The solution to circumvent this is to use pipes.
Within industry, piping is a system of pipes used to convey fluids (liquids and gases) from one location to another. Within UNIX, the pipe operator (|) is used to directly feed output from one program as input into another program. We can pipe directly from the problem program into either less or grep to accomplish the same result as running these commands on the problem.txt file.
$ ./problem | less
$ ./problem | grep error
We can compound indirection, redirection, and pipe operators to create helpful commands. Understanding how these operators work in conjunction with each other can be understood as an analogy to actual pipes, rendered below as cool animations.
By default, print statements in programs output directly to the terminal interface. So this command:
$ ./problem
looks like this:
By using the redirection operator, we install an elbow pipe to redirect the flow of output into a file. The output no longer flows into the terminal.
$ ./problem > problem.txt
By using the pipe operator, we install a coupling between the output and input of two programs. Without any other pipes at the end, the output of the second program flow into the terminal.
$ ./problem | grep error
By using both piping and redirection, we can chain multiple commands and file writing actions. In this command, we run the problem program, filter its output with grep, and output the filtered lines into a file. This file now contains the error messages from problem, and can serve as an error log. Does this output to the terminal as well?
$ ./problem | grep error > errors.txt
Having some tee and drinking it, too
Redirection lets us write to a file. Not having redirection lets us directly see the output in the terminal. Why don’t we have both?
By using the tee command, we can simultaneously write output to a file and and continue flowing data rightward, usually to the terminal. tee is a command, not an operator like redirection and piping, so we must pipe into tee first. tee takes a filename as an argument, which it outputs to.
$ ./problem | tee problem.txt
The tee command shares its name with the tee pipe in plumbing. Ironically, tee is not rendered as a tee pipe in this analogy. Did we really research plumbing terminology for this lab writeup? Yes.
We can continue to construct longer pipe systems to feed output through multiple commands. This command runs problem, filters its output with grep, then outputs to both a file and the terminal.
$ ./problem | grep error | tee errors.txt
Output Streams
In the examples above, we use redirection and piping to manipulate the standard output (stdout) of a program. By default, printf() outputs to this stdout stream, which we illustrate above. What these diagrams do not show (and will not show, please don’t make me try drawing this) is that there actually exists another output stream for standard error (stderr).
To use stderr, we use an alternative to printf(), fprintf(), and specify that it should output to the stderr stream. In problem.c, this line is given to you and commented out. Uncomment this line, and comment the line above (the original print statement). Recompile and run the program.
$ make problem
$ ./problem
The output looks the same as before. The default destination for both stdout and stderr streams is the terminal, but this can be redirected. Try redirecting output to problem.txt as we did before. What’s different this time?
$ ./problem > problem.txt
By default, redirection only operates on stdout, leaving stderr to continue to output to the terminal. We modify the redirection to operate only on stderr, leaving stdout in the terminal:
$ ./problem 2> errors.txt
UNIX uses 1> implicitly as the default redirection operator when we just use >. We can read this as “redirect to the stream with file descriptor 1”, where the stream with file descriptor 1 is stdout.
We can also use two redirections in conjunction with each other to redirect different streams to different files in one command.
$ ./problem > problem.txt 2> errors.txt
First, comment out the
fprintfstatement and uncomment theprintfstatement, so that all output redirects to stdout. Write a command that outputs all lines that involve the first column of the 2D array into both the terminal display and a file calledout0.txt. Then, ask a tutor or TA to check your work. Put a screenshot of your command into your lab report, and submit it on Gradescope.
Hint
The square brackets are special characters in grep, so to search for
[and], you must use the escape character. For example, if you wanted to search for"[5]", you would use grep like so:grep "\[5\]"
Next steps: Practice Skill Demo
To prepare you for Skill Demo 2 next week, we are conducting another practice Skill Demo. This time, you’ll complete it on PrairieLearn.
If you would like to leave early, please ask a tutor or TA to verify that you have completed the lab, submitted your lab report, and gotten full points on the practice Skill Demo.