10 useful tricks to use Sed command for file Manipulation

Commands and Tools to get network statistics                                                  Netstat Command

If you are working or using unix operating system, then you must be proficient in sed language. Sed command in unix is very helpful for manipulation of file. While writing this article, I am considering this you are aware about the sed’s little basics. Sed is a great utility which solves complex tasks by few lines of code.

Sed stands for Stream Editor. It is very powerful command to file manipulatation, filter lines while reading file and transform text. Sed works both the way i.e. Sed can read file and do operations on it or Sed can take inputs from pipe.  If you are new learner then you will face difficulty to understand the sed command but once you understood then It will be easier to use as you need.

We are going to use below file in all 10 examples.

root@hello:~/test# cat salary.txt 1001,Ajay,Manager,25000 101,Satish,Founder,30000 302,Atul,CEO,26000 434,Raj,Senior Manager, 26000 1231,Kalyan,Human Resource,20000

The above salary database contains the columns as

•           Employee ID
•        Name
•          Designation
•          Salary

Basic Syntax of sed command:

sed [options] {sed-commands} {input-file} sed [options] -e {sed-command-1} -e {sed-command-2}{input-file}

Sed reads one line at a time from input file and executes sed - command on that particular line. If we want to execute multiple commands in a single line using several –e arguments, you can split them into multiple lines using a backslash as below:

sed -n \ -e '/^mysql/ p' \ -e '/^other/ p' \ /etc/groups

10 special tricks of Sed as follows :

1.       Print the lines of input file after specific interval i.e. print 1^st, 3^rd, 5^th line of file. Operator   ~ used in an address range. Its special meaning is to skip lines between commands. Below command you can use to print the odd numbered lines or to print the even numbered lines.

root@hello:~/test# sed -n '1~2 p' salary.txt 1001,Ajay,Manager,25000 302,Atul,CEO,26000 1231,Kalyan,Human Resource,20000

2.       Print the lines matching with keyword.

root@hello:~/test# sed -n '/Atul/p' salary.txt 302,Atul,CEO,26000

3.       Print the lines matching with keyword and 2 lines immediately after that. i.e. If matched keyword found then print 3 lines including matched keyword line and 2 lines immediately after that line.

root@hello:~/test# sed -n '/Atul/, +2p' salary.txt 302,Atul,CEO,26000 434,Raj,Senior Manager, 26000 1231,Kalyan,Human Resource,20000

You can use above command for deleting lines in output by using “d” instead of “p” option. i.e. if you want to delete lines starting from 1^st match of “Atul” keyword till 4^th line of file then you can use it as

sed '/Atul/,4 d' salary.txt

4.       Write the content of salary.txt file to new_salary.txt file. i.e. simple copy the salary.txt file into new_salary.txt file.

root@hello:~/test# sed 'w new_salary.txt' salary.txt 1001,Ajay,Manager,25000 101,Satish,Founder,30000 302,Atul,CEO,26000 434,Raj,Senior Manager, 26000 1231,Kalyan,Human Resource,20000 root@hello:~/test# root@hello:~/test# cat new_salary.txt 1001,Ajay,Manager,25000 101,Satish,Founder,30000 302,Atul,CEO,26000 434,Raj,Senior Manager, 26000 1231,Kalyan,Human Resource,20000 root@hello:~/test#

5.       Write the lines matching pattern “Atul” to pattern_found.txt file. Below command can be used for write only odd numbered line (1~2) or write only even numbered lines (2~2).

root@ctier:~/test# sed -n '/Atul/ w pattern_found.txt' salary.txt root@ctier:~/test# cat pattern_found.txt 302,Atul,CEO,26000 root@ctier:~/test#

 You can all above commands by replacing “p” or “d” by “w” option.

Want to know more about : How to run Windows softwares on Linux

6.       Mostly, sed used to substitute pattern by another pattern.

sed '[address-range|pattern-range] s/originalstring/replacement-string/[substitute-flags]' inputfile

Replace all occurrences of Atul with Mohan:

root@ctier:~/test# sed 's/Atul/Mohan/' salary.txt 1001,Ajay,Manager,25000 101,Satish,Founder,30000 302,Mohan,CEO,26000 434,Raj,Senior Manager, 26000 1231,Kalyan,Human Resource,20000 root@ctier:~/test#

Note: We are not changing the file. Sed only changes the display output not a file. If you want to ignore the case the use “I” option at end pattern. i.e. ‘s/Atul/Mohan/i’

Replace Satish with Pravin only on lines that contain thekeyword 'Founder':

root@ctier:~/test# sed '/Founder/s/Satish/Pravin/' salary.txt 1001,Ajay,Manager,25000 101,Pravin,Founder,30000 302,Atul,CEO,26000 434,Raj,Senior Manager, 26000 1231,Kalyan,Human Resource,20000 root@ctier:~/test#

7.       Replace the 1st occurrence of lower case “a” with upper case “A” in a line

root@ctier:~/test# sed 's/a/A/' salary.txt 1001,AjAy,Manager,25000 101,SAtish,Founder,30000 302,Atul,CEO,26000 434,RAj,Senior Manager, 26000 1231,KAlyan,Human Resource,20000 root@ctier:~/test#

                Same command we can use for replace all occurrences of “a” in a line by upper case “A” with below command:

               

root@ctier:~/test# sed 's/a/A/g' salary.txt 1001,AjAy,MAnAger,25000 101,SAtish,Founder,30000 302,Atul,CEO,26000 434,RAj,Senior MAnAger, 26000 1231,KAlyAn,HumAn Resource,20000 root@ctier:~/test#

If you want to replace the second occurrence of “a” with “A” in each line then use following syntax: sed 's/a/A/2' salary.txt

Sed flags in short:

g-            global

i-                    Ignore case

p-            Print

w-           Write

You can use this flags as per your need in sed command.

8.       Suppress the meaning of special characters in sed while mataching pattern. Use backslash to remove special meaning.

[root@rhel Documents]# cat test.text Hello How are you ? Hello /usr/local/bin directory used to store the commands script [root@rhel Documents]# [root@rhel Documents]# sed 's/\/usr\/local\/bin/\/usr\/bin/g' test.text Hello How are you ? Hello /usr/bin directory used to store the commands script [root@rhel Documents]#

9.       Get matched pattern in “&”. Lets see the use of “&” operator.

Enclose the employee id in [EMP ID] and show output.

[root@rhel Documents]# sed 's/^[0-9][0-9][0-9]/[&]/' salary.txt [100]1,Ajay,Manager,25000 [101],Satish,Founder,30000 [302],Atul,CEO,26000 [434],Raj,Senior Manager, 26000 [123]1,Kalyan,Human Resource,20000 [root@rhel Documents]# [root@rhel Documents]# sed 's/^[0-9][0-9][0-9][0-9]/[&]/' salary.txt [1001],Ajay,Manager,25000 101,Satish,Founder,30000 302,Atul,CEO,26000 434,Raj,Senior Manager, 26000 [1231],Kalyan,Human Resource,20000 [root@rhel Documents]# sed -e  's/^[0-9][0-9][0-9]/[&]/' -e 's/^[0-9][0-9][0-9][0-9]/[&]/' salary.txt [100]1,Ajay,Manager,25000 [101],Satish,Founder,30000 [302],Atul,CEO,26000 [434],Raj,Senior Manager, 26000 [123]1,Kalyan,Human Resource,20000 [root@rhel Documents]#

10.       Use execute flag (e). Run commands inside the files by sed command.

Suppose, we add the ls –l command in file.txt file at starting of each line. Then we want to run these lines then how to run :

[root@rhel Documents]# cat > file.txt /etc/passwd /etc/group [root@rhel Documents]# root@rhel Documents]# sed -i 's/^/ls -l /' file.txt [root@rhel Documents]# [root@rhel Documents]# cat file.txt ls -l /etc/passwd ls -l /etc/group [root@rhel Documents]# sed 's/^/ /e' file.txt -rw-r--r--. 1 root root 1527 Mar 20 18:18 /etc/passwd -rw-r--r--. 1 root root 786 Mar 20 18:18 /etc/group [root@rhel Documents]#

Sed has many more uses. This is the basic sed commands use. I have given here most of the sed Command in Unix with examples. If you need more details you can post on my facebook page.

Jinfo Command                                         Jstat Command           File System in Linux