Bash While Read Line by Line

When it comes to text processing or filtering logs, it becomes important to read the contents line by line. Using the bash while loop you can read the contents one line at a time and use it as per our needs. We can read from a file, a command and from a variable. We can even read lines into multiple variables.

Prerequisites

• A text editor

• Access to a terminal

Syntax - bash while read line

We can generalize how to read line by line using the while loop as follows:

while IFS='some field separator' read -r var1 var2 var3 …
do
    ….
    ….
done < Y

Where,

• var1, var2, var3 … are the variables to store the data of every line. For each line these variables are separated by the specified IFS. We don’t require the IFS and the multiple variables for reading the entire line in one variable.

• Y represents the method to feed the data to the loop

Examples

Let's go through use cases with some while read line examples.

Assume we have 2 sample files

1. sample1.txt with the following contents:

Roger Federer SUI
Rafael Nadal ESP
Novak Djokovic SER
Martina Hingis SUI
Steffi Graf GER

2. sample2.txt with the following contents:

USA;Washington D.C.;Dollar
UAE; Abu Dhabi;Dirham
South Africa;Pretoria;Rand
Japan;Tokyo;Yen

1. Read line from file

In Bash, reading lines from a file is pretty easy. We can do it using the while loop. We just have to make use of the built-in read command to read one line at a time of the specified file.

Example:

#!/bin/bashwhile read -r linedo echo "$line" #printing the line; perform any other operation on line variable count=`echo $line | wc -c` #counting the number of letters on each line echo "Characters: $count”done < sample1.txt

For each iteration of the while loop, the read command reads that particular line of the file and assigns it to the bash shell variable $line. The loop runs till the number of lines in the file. After reading the last line of the file, the while loop stops. The -r flag of the read command prevents the interpretation of any backslash-escaped characters while reading.

2. Read line from command

Just like we read each line from a file, we can read the entire multiline output of the command one line at a time. We can achieve this using process substitution, a very useful Bash extension. Process substitution delivers the output of a process into the stdin of another process.

Example

#!/bin/bashwhile read -r linedo echo line is "$line" #printing the line; perform any other operation on line variabledone < <(ls -l)

Process substitution allows us to turn the output of the command (ls-l) into a temporary file. In every iteration, the line variable is getting assigned that particular line of the output. This temporary file vanishes after exiting the loop.

3. Read line from variable

We can also read line by line from a variable storing output in multiple lines. We will use the same logic we have used earlier.

Example:

#!/bin/bashvar=`ls -l` #var holds the output of ls -lwhile read -r linedo echo line is "$line" done <<< $var #command substitution while read -r linedo echo line from here is "$line" done < <(echo "$var") #process substitution

In this example we are storing the output of ls -l command in a variable using the backticks (`) technique. In the first case we are doing the command substitution $() and making use of the here-string <<<. For the nth run of the while loop, the line number n will be stored in the variable line.

Another way to read line by line from a variable is to make use of the process substitution which we have seen in the previous example. For that, we are using the variable along with the echo command and feeding it to the loop.

4. Reading from pipe

Another interesting way to read line by line is to make use of the shell pipe. This technique can work both with multiline files, variables as well as commands. Pipe instructs the shell to use the output of the command on the left as the input.

Example:

#!/bin/bash x=`ls -l`echo "$x" | #pipe on variablewhile read -r linedo echo line is "$line" done ls -l | #pipe on commandwhile read -r linedo echo line is "$line" donecat sample1.txt | #pipe on filewhile read -r linedo echo "$line" done

From this example, we can see how we can make use of the shell pipe to feed the file, variable and the command to read the contents line by line. The read command will read from standard input by default, so we just pipe into it.

5. Reading lines into multiple variables

Sometimes we have to read comma separated value (CSV) files where the number of columns are fixed. We can read each entry of the csv in a variable and process it accordingly.

Example

#!/bin/bashwhile read -r a b cdo echo "First entry is $a" echo "Second entry is $b" echo "Third entry is $c"done < sample1.txt while IFS=';' read -r a b cdo echo "First entry is $a" echo "Second entry is $b" echo "Third entry is $c"done < sample2.txt

In this first case, each line of the entry is split into 3 variables. For each line $a will store the first entry, $b will hold the second and $c the third.

However, the entries may not always be separated by a space just like sample2.txt. To overcome that, we have added the internal field separator (IFS). This variable indicates how the entries are separated, which in our case is a semicolon (;).

Conclusion

• We can use the Bash while loop to read from a file, a command and from a variable.

• Process and command substitution come in handy while reading line by line.

• While reading lines into multiple variables, IFS can be used to split the entries and store them in the corresponding variables.