#How to Modify Text Using Regular Expressions With the sed Stream Editor – CloudSavvy IT

Every computer user in the world sooner or later will modify a text string – either manually or automatically. Be introduced to the sed stream editor, which enables you to make bulk text modifications to strings and files.

Regular Expressions At The Command Line

The sed stream editor, a tool available by default on many Linux distributions, enables you to parse and transform text in an easy and straightforward manner, whether such text is inside files, or just plain strings at the command line. You can also parse directory listings, process listing output, and most other type of information which can flow in or out of your terminal session.

To verify if sed is available on your Linux distribution, type sed --version at the command line:

Do not worry if your version is slightly older then the one shown here. It will almost definitely fine for the examples we discuss here. If the tool is not available on your distribution, you can install it using sudo apt install sed for Ubuntu/Debian based operating systems, or sudo yum install sed for RedHat/RPM based operating systems.

sed applies small text transformation using what is called a RegEx, or Regular Expression. A regular expression is a form and method of expressing more complex text-based search, replace and modify operations in a still human-readable format. Regular expressions are complicated, and may be hard to read for beginners. You can read our guide, How Do You Actually Use Regex, for more information on how to read and write Regex.

RELATED: How Do You Actually Use Regex?

For the purpose of this article, we will stick with easy to understand and read regular expressions. For example, the following is a simple replacement:

As you can see, we echo’d a small piece of text, and subsequently transformed it using the sed tool. We employed the use of the ‘s’ (substitute) sed command and indicated what we wanted to change. You may read this in another way as s|from|to| where ‘|’ is the separator between sections/fields, ‘s’ is the command and ‘from’ and ‘to’ are the two options passed to the ‘s’ command.

We thus transformed ‘Hello Cloud Savvy IT fans!’ to ‘Hello cloudsavvyit.com fans!’ by only changing a part of the text using the sed stream editor. Let’s take a slightly more complex example.

Here we first create a file with 3 lines using the echo command with redirection to a file (‘>’ creates a new file, overwriting a file with the same name if it exists, and ‘>>’ appends to a file without otherwise modifying it). Next we displayed the file contents using the cat command.

We then transformed the contents of the file with the sed tool directly. To do so, we specified the ‘-i’ option to sed which edits file in-place. The actual textual transformation instruction is the same as before. And finally we specify the filename at the end of the command. We then check the file contents and notice how our input has changed.

As you can start to see, or already know from past experience, there is a lot of power available in the sed tool.

Let us spice things up a little by expanding on our regular expression and method of writing the sed text transformation instruction:

In this example, we substituted the letter ‘h’ for ‘H_’ and we did so in a global (note the ‘g’ qualifier near the end of the first instruction passed to sed) and case insensitive manner (note the ‘i’ qualifier also near the end of the first instruction). The ‘h’ is matched even though the file contains ‘H’ due to the ‘i’ qualifier.

We then started a new instruction (instructions are terminated/ended by ‘;’ alike to many other coding languages) in which we deleted ‘.com’ by basically search for it and replacing it with nothing.

Note also how we have used different field/command option separators. In the first example we used ‘|’ which is overall an easy-to-use, and fairly uncommon field separator. Many people use ‘/’ though this may lead to issues when you are parsing path names which often contain ‘/’.

Being able to use different field/command option separators enables you to easily use another separator if your text string does actually or will likely contain a specific character.

Using sed With Files

Let’s explore another sed based example where we will inject text into a file:

In this example, we created a file with 5 lines and subsequently used a slightly complex sed instruction to inject a ‘—‘ separator after every second line. While the syntax mostly explains itself, it is interesting to note that the ‘r’ (read/append text from a file) command specifies the inject filename directly and while still inside the main instruction, without any separator.

You can also start to see that, on top of learning a regular expression syntax, it is beneficial to learn sed syntax in depth. It is almost like learning two new foreign languages, but the benefit is that sed syntax can grow with your need for additional complexity.

Let’s explore a example where we replace a specific text area with something else:

In this example, we search for the ‘3’ text, and for that line we will substitute ‘C’ to ‘Z’. You see how versatile sed can be.

Finally, we can look at parsing other types of input:

As you can see, we created 3 files, and requested a directory listing using the ls command. We then took that output as input to sed (using the Linux pipe command, ‘|’) and transformed part of our directory listing to an underscore.

The sed stream editor is a versatile tool which can help you parse and transform almost any text directly from the terminal command line, whether the input comes from a simple echo statement, other Linux tools, or from a flat text file. Enjoy!