Bash Loop Through Lines in a File

The most efficient way to process any file, one line at a time, is to develop a bash script and give the file as input to the script. This can be done by reading a file line by line by iterating the lines over a loop. Bash treats each line as an element in a list. Understanding how to process the contents of a large data file one line at a time saves us from constantly opening them. There are many ways of reading each line using Bash.

Prerequisites

• A text editor (to create a text file to be read)

• Access to a terminal

The first thing needed is to create a text file with the name sample.txt with the following contents:

1. Using a while loop with read command

while loop when used with the read utility will read each line from the file and in each step store the content of the line in the variable.

Here’s the syntax:

while read -r line
do
    echo "$line"
done < sample.txt

To create the script (script1.sh), open the terminal in your favorite editor, put the following contents and save it:

while read -r linedo echo "$line" #printing the line; perform any other operation on line variabledone < sample.txt

For each iteration of the while loop, the read command reads that particular line of the file and assigns it to the bash shell variable $line. The loop runs till the number of lines in the file. After reading the last line of the file, the while loop stops. The -r flag of the read command prevents the interpretation of any backslash-escaped characters while reading.

2. Using a while loop with file descriptor

A file descriptor is an unsigned integer used by a Linux process to uniquely identify an open file. A minimum of one file descriptor exists for every open file. The first three file descriptors, by default, are STDIN (0), STDOUT (1), and STDERR (2).

Here’s the syntax:

while read -r -u5 line
do
    echo "$line"
done 5< sample.txt

Example:

while read -r -u5 linedo echo "$line" #printing the line; perform any other operation on line variabledone 5< sample.txt

For each iteration of the while loop, the read command reads input from a file descriptor specified by the -u argument and the file descriptor number. The input file's contents are sent to the specified file descriptor. It is always advised to use a number between 4 and 9 to avoid conflict with the internal shell file descriptors.

3. Using a for loop with cat command

To display the contents of a file in individual lines, one can iterate over the for loop and make use of the Linux cat command. The for loop enables printing of the lines from the cat command output until the end of the file.

Here’s the syntax:

for line in $(cat sample.txt)
do
    echo "$line"
done

Example:

#!/bin/bashfor line in $(cat sample.txt)do echo "$line" #printing the line; perform any other operation on line variabledone

For each iteration, the for loop iterates over every line of the cat command output and stores it in the variable $line. It then prints the value stored in the variable using the echo command until it reaches the end of the file. The loop will stop after the last line has been processed.

4. Using a while loop with input redirection

This method makes use of a here string to supply the contents of the file to the read command. Here String is used for input redirection from a string, a file or a variable. It is constructed using the <<< operator to redirect a string into a command.

Here’s the syntax of here string:

COMMAND <<< $VAR

Example:

#!/bin/bashwhile read -r linedo echo "$line" #printing the line; perform any other operation on line variabledone <<< $(cat sample.txt )

The here string supplies the output generated from the cat command as an input to the read command. In the loop this is read one line at a time and stored in the bash variable $line.

5. Using a while loop with IFS (Internal Field Separator)

To display the contents of a file in individual lines, one can iterate over the while loop with the Internal Field Separator. The IFS= argument is chosen as an empty string. This helps to preserve the whitespaces.

Here’s the syntax:

while IFS= read -r line
do
    echo "$line"
done < <(cat sample.txt)

Example:

#!/bin/bashwhile IFS= read -r linedo echo "$line" #printing the line; perform any other operation on line variabledone < <(cat sample.txt)

This technique is also known as process or command substitution which runs a bash command and stores its output in a variable or passes it to another command. Here, the standard output of the process is available as a file which in turn is fed into the standard input of another process. The input file is supplied and each line is printed separately.

Conclusion

• There are multiple ways of reading a file one line at a time using loops in bash.

• Reading a file line by line becomes important when it is huge and certain keywords need to be searched in real-time.

• Reading an entire file in one go can slow down the process as compared to reading it line by line because nothing is processed until the read is complete.