In centos 7, what command is to to extract fields from a file line (record)?

There are many tools that we can use at the Linux command line aimed directly at extracting data from files and searching that data. We can start by seeing how the modular nature of the UNIX and Linux command line shells allow for bespoke applications using pipes.

Piping

As mentioned in Linux Essentials objective 2.4: we have two types of pipes, un-named and named pipes. Mainly, we see un-named pipes but named pipes are commonly used between processes on your PC, allowing one application to talk to another. Making use of an unnamed pipe we use the vertical bar between two commands as shown below.

$ ls -l | wc -l

This is a command pipeline in its very simplest form, just two commands used in extracting data from files. The output of ls is sent to the input the command wc. In this case, the pipeline that we have built will count the number of lines of output from ls, or simply the number of entries in the current directory. Yes, we have extracted data, the number if files in a directory.

We call this an un-named pipe as it is created on the fly without the existence of a pipe file. This is very convenient for us on the command line but not so convenient for applications to be able to communicate.

In this situation the caped-crusader appears in the form of named pipes to save the day. The application will often create special files of the pipe type that can be used to link to commands together. The pipe file will never store data itself, but marshals  data, [controls the data movement], from one application to another. We can create our own named pipes using the command mkfifo [/usr/bin/mkfifo].

$ which mkfifo
/usr/bin/mkfifo

These special files are known as named pipes as they are represented by files of type PIPE in the file system and as such have a name. You can search for these file types on any Linux system using the find command. This is a very powerful tool and I would encourage you to practice with the command. The output from your system may differ:

$ find / -type p 2>/dev/null
/run/dmeventd-client
/run/dmeventd-server
/run/systemd/inhibit/14.ref
/run/systemd/inhibit/13.ref
/run/systemd/sessions/211.ref
/run/systemd/initctl/fifo

Redirection

Unlike piping, redirection takes the output from a command and sends that output to a text file. Alternatively a command may redirect a text file to its input. Using the file’s content as as input.

In the previous find command, we redirected any error output, denoted by the number 2, to the special file /dev/null. In this way we do not list all of the errors that are produced when we cannot access a file due to limited permissions.

Each command has three channels that can be used for redirection:

• Standard Input : Channel 0
• Standard Error : Channel 1
• Error Output : Channel 2

We only need to to use the channel number when redirecting error output; the symbol < indicates standard input when used without a number and > represents standard output without a number in use. As such:

• cat < file1 : file1 is read into standard input for the command cat
• ls /etc > file1 : the standard output of ls is sent to file1, errors are shown on the screen and not redirected
• ls /etc 2> file1 : Standard output is shown to the screen but errors are written to file1

We can use the >> symbols to append to files are create the files if they do not exist. When using the single greater than symbol > we can create the file and overwrite the file if it exists. If you are concerned about overwriting existing files in error you may set the shell option noclobber. When set, new files can be created but if the file exists normal operation will not permit you to overwrite the existing file. Using >| allows the file to be over-written. The noclobber option may usually be set in a login script or from the command line.

$ set -o noclobber

The -o option sets the option, or turns the option ON.

$ set +o noclobber

The option +o turns the option OFF. To view the current setting you can use the the command

$ set -o

The above command will show all settings and their current state. Taking what we have learned about piping; we now know that it is possible to pipe the output of the set [shell built-in command] command to grep[/bin/grep] which can search then for the particular option we wish to view.

$ set -o | grep noclobber
noclobber off

Currently the option is disabled on my system as we can see from the above output.

Settings that we make at the command line with the command set are transient. They are for that shell and that shell only. On new sheel will not have the same setting and it will be lost on logout. The make settings permanent then add the commands in your personal login script: .bashrc in your home directory. The file /home//.bashrc runs each time the bash shell is executed for the specific user.

Searching and Extracting Data from Files using Grep and Regular Expressions

The command grep  becomes a simple tool that we can make use of both practically in every day Linux usage as well as here in the course to help demonstrate regular expressions. To test regular expressions fully we may want to use egrep [/bin/egrep] or more simply grep -E to allow for extended regular expression matches.

In the previous graphic we can see that we search for the text string noclobber in the output of the set command. We literally search for the string noclobber. We may think we are looking for the word noclobber but computers think differently to us. Consider the following text file test.txt: It is shown in the following graphic using the command cat .

$ cat test.txt 
no color
color
colour
coloured
colored

To create the file use your favourite text editor such as nano or vim, or simple copy this code to the command line of your system. Practicing some redirection!

$ cat > test.txt /dev/null
/run/dmeventd-client
/run/dmeventd-server
/run/systemd/inhibit/14.ref
/run/systemd/inhibit/13.ref
/run/systemd/sessions/211.ref
/run/systemd/initctl/fifo

1

When reading log files it is common to follow the end of the log. This uses the -f option to tail and will continue to display the current last 10 lines of the log. Use control + c to stop following the file.

$ find / -type p 2>/dev/null
/run/dmeventd-client
/run/dmeventd-server
/run/systemd/inhibit/14.ref
/run/systemd/inhibit/13.ref
/run/systemd/sessions/211.ref
/run/systemd/initctl/fifo

2

Note: The log file used here is on Ubuntu on other systems the more general log file is /var/log/messages

Similarly we can use cat [/bin/cat] and tac [/usr/bin/tac], cat list or concatenates the file from top to bottom and tac from bottom to top. If you focus is on the bottom of the file use cat, you will be left at the bottom of the file. If your focus is on the top use tac as you will be left at the top of the file.

Counting Words, Lines or Characters Using wc

Using the command wc we are at the heart of extracting and searching data from files at the Linux command line. The command wc stands for word count but it can count much more that that. WE will step through some examples.

$ find / -type p 2>/dev/null
/run/dmeventd-client
/run/dmeventd-server
/run/systemd/inhibit/14.ref
/run/systemd/inhibit/13.ref
/run/systemd/sessions/211.ref
/run/systemd/initctl/fifo

3

The first counts the lines, words and characters. The second line counts just lines, then just words and then just characters. The output from ls -l is shown below note that it shows both the line count and the file name.

$ find / -type p 2>/dev/null
/run/dmeventd-client
/run/dmeventd-server
/run/systemd/inhibit/14.ref
/run/systemd/inhibit/13.ref
/run/systemd/sessions/211.ref
/run/systemd/initctl/fifo

4

Searching and Extracting Data Using Fields and the Command Cut

The command cut [/usr/bin/cut] can be useful where viewing every filed in a file is not required. We may only want to see certain fields. Even with the output of a command we can pipe the output to cut. Suppose we only need the line count from wc not the file name:

What command in the Windows 10 CLI will display the contents of a text file on the standard output device?

What tab in the Command Prompt properties box allows you to select the screen buffer and window sizes of the Command Prompt?

How to save command line output in text file?

What command is used to display information about the device?