The stream editor (sed) is a verypopularnon-interactive stream editor. Normally, whenever we edit files using the vi editor, we need to open the file using thevicommand, then we interact with the file to see the content of the file on screen, then edit it, and then save the file. Usingsed, we can type commands on the command line andsedwill make the changes to the text file.sedis a non-destructive editor.sedmakes the changes to the file and displays the content on screen. If we want to save the changed file, then we need to redirect the output ofsedto the file.
The procedure to installsedis shown here.
For Ubuntu or any Debian-based distributions, enter the following command:
$ apt-get install sedFor Red Hat or any rpm-based distribution enter the following command:
$ yum install sedTo check the version of sed, enter the following command:
$ sed -VOtherwise, enter this command:
$ sed --versionGNU sed version 3.02Understanding sed
Whenever you use sed commands on a text file, sed reads the first line of the file and stores it in a temporary buffer called pattern space. sed processes this pattern space buffer as per commands given by the user. Then, it prints the output on screen. This line from the pattern space is then removed and the next line of the file is loaded in the pattern space. In this way, it processes all the lines one by one. This line-by-line processing is continued till the last line of the file. As the sed commands are processed in the temporary buffer or pattern space, the original line is not modified. Therefore, we say sed is a non-destructive buffer:
Understanding regular expression usage in sed
While using sed, regular expressions are enclosed in forward slashes, as grep and sed use regular expressions and meta-characters for searching patterns in the file. An example of this would be the following:
sed -n '/Regular_Expression/p' filenamesed -n '/Mango/p' filenameThis will print lines matching the Mango pattern:
sed -n 's/RE/replacement string/' filenamesed -n 's/Mango/Apple/p' filenameThis will find the line containing theMangopattern and then theMangopattern will be replaced by theAppletext. This modified line will be shown on screen and the original file will be unchanged.
The following is a summary of various meta-characters and their usage insed:
Meta-character
Function
^
This is the beginning-of-line anchor
$
This is the end-of-line anchor
.
This matches one character, but not the newline character
*
This matches zero or more characters
[ ]
This matches one character in the set
[^ ]
This matches one character not in the set
(..)
This saves matched characters
This saves the search string so it can be remembered in the replacement string
<
This is the beginning-of-word anchor
>
This is the end-of-word anchor
x{m}
This is the repetition of the characterx:mtimes
x{m,}
This means at leastmtimes
x{m,n}
This means betweenmandntimes
Addressing in sed
We can specify which line or numberoflines the pattern search and commands are to be applied on while using thesedcommands. If line numbers are not specified, then the pattern search and commands will be applied to all lines of the input file.
The line numbers on which commands are to beappliedare called theaddress. The address can be a single line number or range of lines in which the starting number of the line and the ending number of the range will be separated by commas. Ranges can be composed of numbers, regular expressions, or a combination of both.
Thesedcommands specify actions such as printing, removing, replacing, and so on.
The syntax is as follows:
sed 'command' filename(s)Here is an example:
$ cat myfile | sed '1,3d'You could also use the following:
sed '1,3d' myfileThis will delete lines 1 to 3:
sed -n '/[Aa]pple/p' item.listIf theAppleorapplepattern is found in theitem.listfile, then those lines will be printed on screen and the originalmyfilefile will be unchanged.
To negate the command, the exclamation character (!) can be used.
Here’s an example:
sed '/Apple/d' item.listThis tellssedto delete all the lines containing theApplepattern.
Consider the following example:
sed '/Apple/!d' item.listThis will delete all the lines except the line containing the Apple pattern.
How to modify a file with sed
sed is a non-destructive editor. This means the output of sed is displayed on screen but the original file is unchanged. If we want to modify the file, then we can redirect the output of the sed command to the file. Deleting lines is illustrated in the following examples:
$ sed '1,3d' datafile > tempfile$ mv tempfile newfileIn this example, we have deleted lines 1 to 3 and stored the output in tempfile. Then, we have to rename tempfile to newfile
Printing – the p command
By default, the action of thesedcommand is toprintthe pattern space, such as every line that is copied into the buffer, and then print the result of processing it. Therefore, thesedoutput will consist of all lines along with the processed line by sed. If we do not want the default pattern space line to be printed, then we need to give the-noption. Therefore, we should use the-noption and thepcommand together to see the result of thesedprocessed output.
Here is an example:
$ cat country.txtThe output is as follows:
Output:
Country Capital ISD CodeUSA Washington 1China Beijing 86Japan Tokyo 81India Delhi 91$ sed '/USA/p' country.txtThe output is as follows:
Output:
Country Capital ISD CodeUSA Washington 1USA Washington 1China Beijing 86Japan Tokyo 81India Delhi 91All the lines from the file are printed by default and the lines with the USA pattern are also printed:
$ sed -n '/USA/p' country.txtThe output is as follows:
Output:
USA Washington 1As we have given the -n option, sed has suppressed default printing of all lines from the country file but has printed the line that contains the text pattern USA
Deleting – the d command
The d command is used to delete lines. After sed copies a line from a file and puts it into a pattern buffer, it processes commands on that line, and, finally, displays the contents of the pattern buffer on screen. When the d command is issued, the line currently in the pattern buffer is removed and not displayed, as follows:
$ cat country.txtCountry Capital ISD CodeUSA Washington 1China Beijing 86Japan Tokyo 81India Delhi 91$ sed '3d' country.txtThe output is as follows:
Output:
Country Capital ISD CodeUSA Washington 1Japan Tokyo 81India Delhi 91Here is the explanation.
The output will contain all the lines except the third line. The third line is deleted by the following command:
$ sed '3,$d' country.txtThe output is as follows:
Output:
Country Capital ISD CodeUSA Washington 1This will delete the third line to the last line. The dollar sign in the address indicates the last line. The comma is called a range operator:
$ sed '$d' country.txtThe output is as follows:
Output:
Country Capital ISD CodeUSA Washington 1China Beijing 86Japan Tokyo 81Here is the explanation.
This deletes the last line. All lines except lines will be displayed.
Here is an example:
$ sed '/Japan/d' country.txtThe output is as follows:
Output:
Country Capital ISD CodeUSA Washington 1China Beijing 86India Delhi 91The line containing the Japan pattern is deleted. All other lines are printed:
$ sed '/Japan/!d' country.txtThe output is as follows:
Japan Tokyo 81This has deleted all the lines that do not containJapan.
Let’s see a few more examples with thedeletecommand.
This will delete line 4 and the next five lines:
$ sed '4,+5d'This will keep lines 1 to 5 and delete all the other lines:
$ sed '1,5!d'This will delete lines 1, 4, 7, and so on:
$ sed '1~3d'Starting from 1, every third line step increments. The number that follows the tilde is what is called the step increment. The step increment indicates the following:
$ sed '2~2d'This will delete every other line, starting with line 2
Substitution – the s command
If we want to substitute some text with new text, then we can use commands. After the forward slash, the regular expression is enclosed and then the text to be substituted is placed. If the g option is used, then substitution will happen globally, meaning that it will be applied to the full document. Otherwise, only the first instance will be substituted:
$ cat shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15Cashew 1 10 10$ sed 's/Cashew/Almonds/g' shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_Cost Apple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15Almonds 1 10 10Thescommand has replacedCashewwithAlmonds. Thegflag at the end indicates that the substitution is to be applied globally. Otherwise, it will be applied to the first pattern match only.
The following substitution command will replace two-digit numbers at the end of the line with.5appended to them:
$ sed 's/[0-9][0-9]$/&.5/' shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15.5Cashew 1 10 10.5The ampersand in the search pattern represents the exact pattern found. This will be replaced by the exact pattern with .5 appended to it.
Range of selected lines the comma
To use sed effectively, we should be clear about how to define range. Range is typically two addresses in a file as follows:
- Range with numbers:
'6d': range of line 6
'3,6d': range from line 3 to 6 - Range with pattern:
'/pattern1/,/pattern2/ - This will specify the range of all the lines between
pattern1andpattern2. We can even specify the range with a combination of both, that is,'/pattern/,6'. This will specify the range of lines between the pattern and line6.
As mentioned, we can specify the range as numbers, pattern, or a combination of both, as shown here.
$ cat country.txtCountry Capital ISD CodeUSA Washington 1China Beijing 86Japan Tokyo 81India Delhi 91$ sed -n '/USA/,/Japan/p' country.txtThe output is as follows:
Output:
USA Washington 1China Beijing 86Japan Tokyo 81In this example, all the lines between addresses starting with USA and until the pattern Japan will be printed on screen, as shown here.
$ sed -n '2,/India/p' country.txtThe output is as follows:
Output:
USA Washington 1China Beijing 86Japan Tokyo 81India Delhi 91In this example, line 2 to the pattern India, are printed on screen as shown here.
$ sed '/Apple/,/Papaya/s/$/** Out of Stock **/' shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6** Out of Stock **Orange 2 .8 1.6** Out of Stock **Papaya 2 1.5 3** Out of Stock **Chicken 3 5 15Cashew 1 10 10In this example, for all the lines between the Apple and Papaya patterns, the end of line will be replaced by the ** Out of Stock ** string.
Multiple edits – the e command
If we need to perform multiple editing with the same command, then we can use the -e command. Each edit command should be separated by the -e command. sed will apply each editing command separated by -e on the pattern space before loading the next line in the pattern space:
$ cat shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15Cashew 1 10 10This is an example:
sed -e '5d' -e 's/Cashew/Almonds/' shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Almonds 1 10 10Initially, the command for deleting the fifth line is called, then, the next substitution command to replace Cashew with Almonds is processed.
Reading from files – the r command
If we need to insert text fromanotherfile into a file, processed bysed, then we can use thercommand. We can insert text from another file to the specified location:
Here is an example:
$ cat new.txtThe output will be:
Output:
********************************* Apples are out of stock*********************************$ sed '/Apple/r new.txt' shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6********************************* Apples are out of stock*********************************Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15Cashew 1 10 10This command has added the content of the new.txt file after the line containing the Apple pattern.
Writing to files – the w command
Thesedcommand for writing isw. Using this command, we canwritelines from one file to another file.
Here is an example:
$ cat new.txt The output is as follows:
Output:
new is a empty file$ sed -n '/Chicken/w new.txt' shopping.txt$ cat new.txt Chicken 3 5 15After the w command, we specify the file to which we will perform the write operation. In this example, the line containing the Chicken pattern is written to the new.txt file.
Appending – the a command
Theacommand is used forappending. When theappendcommand is used, it appendsthetext after the line in the pattern space in which the pattern is matched. The backslash should be placed immediately after theacommand. On the next line, the text to be appended is to be placed.
Here is an example:
$ cat shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15Cashew 1 10 10$ sed '/Orange/a**** Buy one get one free offer on this item ! ****' shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6**** Buy one get one free offer on this item ! ****Papaya 2 1.5 3Chicken 3 5 15Cashew 1 10 10The new text **** Buy one get one free offer on this item ! **** is appended after the line containing the Orange pattern.
Inserting – the i command
Theicommand is used forinsertingtext abovethecurrent pattern space line. When we use the append command, new text is inserted after the current line, which is in the pattern buffer. In this similar-to-append command, the backslash is inserted after theicommand.
Here is an example:
$ cat shopping.txtProduct Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15Cashew 1 10 10 $ sed '/Apple/i New Prices will apply from Next month ! ' shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_Cost New Prices will apply from Next month ! Apple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15Cashew 1 10 10In this example, the new text, New Prices will be applied from next month! is inserted before the line containing the Apple pattern. Please check the i command and the backslash following it.
Changing – the c command
The c command is the change command. It allows sed to modify or change existing text with new text. The old text is overwritten with the new:
$ cat shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15Cashew 1 10 10Here is an example:
$ sed '/Papaya/c Papaya is out of stock today !' shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6 Papaya is out of stock today !Chicken 3 5 15Cashew 1 10 10In this example, the line containing the expression Papaya is changed to the new line, Papaya is out of stock today!
Transform – the y command
Thetransformcommand issimilarto the Linuxtrcommand. The charactersaretranslated according to the character sequence given. For example,y/ABC/abc/will convert lowercaseabcinto uppercaseABC.
Here is an example:
$ cat shopping.txtThe output will be:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15Cashew 1 10 10$ sed '2,4y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRS
TUVWXYZ/' shopping.txtThe output will be:
Output:
Product Quantity Unit_Price Total_CostAPPLE 2 3 6ORANGE 2 .8 1.6PAPAYA 2 1.5 3Chicken 3 5 15Cashew 1 10 10In this example, for lines 2, 3, and 4, all the lowercase letters are converted to uppercase letters.
Quit – the q command
The q command is used for quitting the sed processing without proceeding to the next line:
$ cat shopping.txtThe output will be as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15Cashew 1 10 10Here is an example:
$ sed '3q' shopping.txtThe output will be as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6In this example, after printing the first to third lines, sed quits further processing.
Holding and getting – the h and g commands
We have already seen thatsedhas a pattern buffer.sedhas onemoretype ofbuffercalled aholding buffer. With thehcommand, we can informsedto store the pattern buffer in theholdingbuffer. Whenever we need the line that is stored in the pattern buffer, we can get it with thegcommand, that is, get the buffer.
Here is an example:
$ sed -e '/Product/h' -e '$g' shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15Cashew 1 10 10Product Quantity Unit_Price Total_CostIn this example, the line containing the Product pattern is stored in the holding buffer by the h command. Then, the next editing command asks sed to get the line from the holding buffer when the last line of the file is reached. It then appends the line from the holding buffer after the last line of the file.
Holding and exchanging – the h and x commands
This is anexchangecommand. By using this command, we can exchange the holdingbufferwith the current line in the pattern buffer.
Here is an example:
$ sed -e '/Apple/h' -e '/Cashew/x' shopping.txtThe output is as follows:
Output:
Product Quantity Unit_Price Total_CostApple 2 3 6Orange 2 .8 1.6Papaya 2 1.5 3Chicken 3 5 15Apple 2 3 6In this example, the line with the Apple pattern is stored in the holding buffer. When the pattern with Cashew is found, that line will be exchanged with the holding buffer.
sed scripting
Thesedscript file contains a list ofsedcommands in a file. To informsedaboutourscript file, we should use the-foption before the script filename. If thesedcommands are not separated by a new line, then every command should be separated by a colon “:“. We should make sure that there aren’t any trailing whitespaces after any of the commands in thesedscript file; otherwise,sedwill give an error.sedtakes each line in the pattern buffer and then it will process all commands on that line. After this line is processed, the next line will be loaded in the pattern buffer. For the continuation of anysedcommand that cannot be fitted on one line, we need to add one backslash at the end of the line to inform it of the continuation.
Here is an example:
$ cat shopping1.txtThe output is as follows:
Output:
Product Quantity Unit_PriceApple 200 3Orange 200 .8Papaya 100 1.5Chicken 65 5Cashew 50 10April, third week $ cat stock The output is as follows:
Output:
# This is my first sed script by : 1iStock status report/Orange/aFresh Oranges are not available in this season. Fresh Oranges will be available from next month/Chicken/c**********************************************************We will not be stocking this item for next few weeks.**********************************************************$dEnter the following command:
$ sed -f stock shopping1.txtThe output is as follows:
Output:
Stock status reportProduct Quantity Unit_PriceApple 200 3Orange 200 .8Fresh Oranges are not available in this season. Fresh Oranges will be available from next monthPapaya 100 1.5**********************************************************We will not be stocking this item for next few weeks.**********************************************************Cashew 50 10In this script, the following processing has taken place:
- The comment line starts with the pound (
#) sign. - The command
1iinformssedto insert the next text before line number1. - The command
/Orange/ainformssedto append the next text after the line containing theOrangepattern. - The command
/Chicken/cinformssedto replace the line containing theChickenpattern by the next line. - The last command,
$d, tellssedto delete the last line of the input file.
