02-shell文本处理三剑客之sed

sed 是流编辑器,但是它不会修改源文件。

sed (流文本编辑器)
用法:sed OPTIONS… [SCRIPT] [INPUTFILE…]
-r: 使用扩展的正则表达式;
-n, –quiet, –silent: 不输出模式空间的内容;
-i:直接编辑原文件;
-e: -e ” -e ” -e ”
‘CMD1;CMD2;CMD3’
-f /path/to/sed_script:从指定的文件读取处理脚本

1 模式空间是什么?
模式空间其实就是内存中一个空间,归sed所有s。ed读取流文本文件的时候,每次读取一行,然后载入内存中,这个内存就成为模式空间。那么我们编辑的内容是属于模式空间中的内容,而不是源文件内容。

[root@hadoop1 shelltest]# sed '1,10d' /etc/fstab 
UUID=04777095-5e99-46d5-87cf-1bc6665635ee swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@hadoop1 shelltest]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Tue Jan 26 09:32:28 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 /                       ext4    defaults        1 1
UUID=a4623924-f98a-47c9-a9e7-38c06e0fc7f6 /boot                   ext4    defaults        1 2
UUID=04777095-5e99-46d5-87cf-1bc6665635ee swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@hadoop1 shelltest]# 

sed ‘1,10d’ /etc/fstab 这一句语句的执行逻辑是,sed把内容读到模式空间中,判断是不是1——10的内容,是的话执行删除命令d,所以删除了1——10行内容,11行以后的内容就显示出来了。
sed的工作流程可以总结为:
第一步:提取一行内容到模式空间
第二步:针对模式空间中的内容执行编辑命令。
第三步:将模式空间中的内容进行输出。
第四步:清空模式空间。

2 地址定界:
#(行号): 指定行
$: 最后一行
/regexp/:任何能够被regexp匹配到的行
\%regexp%: 任何能够被regexp匹配到的行;换用%为边界符号;
/regexp/I
\%regexp%I:匹配时忽略字符大小写;
addr1, addr2: 指定范围内的所有行:
1·,/regexp/
#(行号),#(行号)
/regexp/,/regexp/
#(行号),+#
first~step:指定起始位置,以及步长 1~3

[root@hadoop1 shelltest]# sed '$d' /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Tue Jan 26 09:32:28 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 /                       ext4    defaults        1 1
UUID=a4623924-f98a-47c9-a9e7-38c06e0fc7f6 /boot                   ext4    defaults        1 2
UUID=04777095-5e99-46d5-87cf-1bc6665635ee swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
[root@hadoop1 shelltest]# 

以上功能是删除了最后一行。

[root@hadoop1 shelltest]# sed '/dev/d' /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Tue Jan 26 09:32:28 2016
#
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 /                       ext4    defaults        1 1
UUID=a4623924-f98a-47c9-a9e7-38c06e0fc7f6 /boot                   ext4    defaults        1 2
UUID=04777095-5e99-46d5-87cf-1bc6665635ee swap                    swap    defaults        0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@hadoop1 shelltest]# sed '\%/dev/mapper%d' /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Tue Jan 26 09:32:28 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 /                       ext4    defaults        1 1
UUID=a4623924-f98a-47c9-a9e7-38c06e0fc7f6 /boot                   ext4    defaults        1 2
UUID=04777095-5e99-46d5-87cf-1bc6665635ee swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@hadoop1 shelltest]# sed '1+2d' /etc/fstab 
sed: -e expression #1, char 2: unknown command: `+'
[root@hadoop1 shelltest]# sed '1,+2d' /etc/fstab 
# Created by anaconda on Tue Jan 26 09:32:28 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 /                       ext4    defaults        1 1
UUID=a4623924-f98a-47c9-a9e7-38c06e0fc7f6 /boot                   ext4    defaults        1 2
UUID=04777095-5e99-46d5-87cf-1bc6665635ee swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@hadoop1 shelltest]# sed '1~2d' /etc/fstab 
#
# Created by anaconda on Tue Jan 26 09:32:28 2016
# Accessible filesystems, by reference, are maintained under '/dev/disk'
#
UUID=a4623924-f98a-47c9-a9e7-38c06e0fc7f6 /boot                   ext4    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
sysfs                   /sys                    sysfs   defaults        0 0
[root@hadoop1 shelltest]#
基本正则表达式元字符:
    字符:.  []  [^]
    次数:*, \+, \?, \{i,j\}
    分组:\(\)
    锚定:^, $, \<, \>
    或者:|
    后向引用:\1, \2, ...

编辑命令:
    注意:命令之前可以使用!表示取反操作;

    =: 显示行号
        # sed -n '\%/dev/mapper%=' /etc/fstab

    d: 删除匹配到的行;
    p: 打印模式空间中的内容;
        通常与-n选项一同使用;
    a \text: append,
        a \text1\ntext2\ntext3: 附加多行文本
    i \text:insert, 
        i \text1\ntext2: 插入多行文本
    c \text: change,把匹配到的行替换为给定的文本
    s///: 替换命令
        g: 全局替换
        i: 不区分大小写
[root@hadoop1 shelltest]# sed -n '\%/dev/%=' /etc/fstab 
6
12
13
[root@hadoop1 shelltest]# sed -n '5,7p' /etc/fstab
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
[root@hadoop1 shelltest]# 

追加:

[root@hadoop1 shelltest]# sed '1a \xielaoshi' /etc/fstab 

xielaoshi
#
# /etc/fstab
# Created by anaconda on Tue Jan 26 09:32:28 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 /                       ext4    defaults        1 1
UUID=a4623924-f98a-47c9-a9e7-38c06e0fc7f6 /boot                   ext4    defaults        1 2
UUID=04777095-5e99-46d5-87cf-1bc6665635ee swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@hadoop1 shelltest]# 

还可以同时执行多条命令:

[root@hadoop1 shelltest]# sed '1d;2p;3=' /etc/fstab 
#
#
3
# /etc/fstab
# Created by anaconda on Tue Jan 26 09:32:28 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 /                       ext4    defaults        1 1
UUID=a4623924-f98a-47c9-a9e7-38c06e0fc7f6 /boot                   ext4    defaults        1 2
UUID=04777095-5e99-46d5-87cf-1bc6665635ee swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@hadoop1 shelltest]# 

替换:

[root@hadoop1 shelltest]# sed '3c \bbbb' /etc/fstab 

#
bbbb
# Created by anaconda on Tue Jan 26 09:32:28 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 /                       ext4    defaults        1 1
UUID=a4623924-f98a-47c9-a9e7-38c06e0fc7f6 /boot                   ext4    defaults        1 2
UUID=04777095-5e99-46d5-87cf-1bc6665635ee swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@hadoop1 shelltest]#
[root@hadoop1 shelltest]# sed 's/dev/xielaoshi/' /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Tue Jan 26 09:32:28 2016
#
# Accessible filesystems, by reference, are maintained under '/xielaoshi/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 /                       ext4    defaults        1 1
UUID=a4623924-f98a-47c9-a9e7-38c06e0fc7f6 /boot                   ext4    defaults        1 2
UUID=04777095-5e99-46d5-87cf-1bc6665635ee swap                    swap    defaults        0 0
tmpfs                   /xielaoshi/shm                tmpfs   defaults        0 0
xielaoshipts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@hadoop1 shelltest]#

全局替换:

[root@hadoop1 shelltest]# sed 's/dev/xielaoshi/g' /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Tue Jan 26 09:32:28 2016
#
# Accessible filesystems, by reference, are maintained under '/xielaoshi/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 /                       ext4    defaults        1 1
UUID=a4623924-f98a-47c9-a9e7-38c06e0fc7f6 /boot                   ext4    defaults        1 2
UUID=04777095-5e99-46d5-87cf-1bc6665635ee swap                    swap    defaults        0 0
tmpfs                   /xielaoshi/shm                tmpfs   defaults        0 0
xielaoshipts                  /xielaoshi/pts                xielaoshipts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@hadoop1 shelltest]#

3 练习
练习1:删除/boot/grub/grub.conf文件中所以空白开头的行行首的空白;

[root@hadoop1 shelltest]# cat /boot/grub/grub.conf 
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/sda2
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-431.el6.x86_64)
    root (hd0,0)
    kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
    initrd /initramfs-2.6.32-431.el6.x86_64.img
[root@hadoop1 shelltest]# sed 's/^[[:space:]]\+//g' /boot/grub/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/sda2
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-431.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-431.el6.x86_64.img
[root@hadoop1 shelltest]# 

练习2:删除/etc/fstab文件中所有以#开头,后跟至少一个空白字符的行的行首的#和空白字符;

[root@hadoop1 shelltest]# sed 's/^#[[:space:]]\+//g' /etc/fstab 

#
/etc/fstab
Created by anaconda on Tue Jan 26 09:32:28 2016
#
Accessible filesystems, by reference, are maintained under '/dev/disk'
See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b8a0a61e-c5e1-40e7-9a4f-e9a9096565a9 /                       ext4    defaults        1 1
UUID=a4623924-f98a-47c9-a9e7-38c06e0fc7f6 /boot                   ext4    defaults        1 2
UUID=04777095-5e99-46d5-87cf-1bc6665635ee swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@hadoop1 shelltest]# 

练习3:echo一个文件路径给sed命令,取出其基名;取出其目录名;

[root@hadoop1 shelltest]# echo "/etc/sysconfig/" | sed 's@[^/]\{1,\}/\?$@@g'
/etc/
[root@hadoop1 shelltest]# 

练习4:替换/etc/inittab文件“id:数字:initdefault:”一行中的那个数字为6;

[root@hadoop1 shelltest]# cat /etc/inittab 
# inittab is only used by upstart for the default runlevel.
#
# ADDING OTHER CONFIGURATION HERE WILL HAVE NO EFFECT ON YOUR SYSTEM.
#
# System initialization is started by /etc/init/rcS.conf
#
# Individual runlevels are started by /etc/init/rc.conf
#
# Ctrl-Alt-Delete is handled by /etc/init/control-alt-delete.conf
#
# Terminal gettys are handled by /etc/init/tty.conf and /etc/init/serial.conf,
# with configuration in /etc/sysconfig/init.
#
# For information on how to write upstart event handlers, or how
# upstart works, see init(5), init(8), and initctl(8).
#
# Default runlevel. The runlevels used are:
#   0 - halt (Do NOT set initdefault to this)
#   1 - Single user mode
#   2 - Multiuser, without NFS (The same as 3, if you do not have networking)
#   3 - Full multiuser mode
#   4 - unused
#   5 - X11
#   6 - reboot (Do NOT set initdefault to this)
# 
id:5:initdefault:
[root@hadoop1 shelltest]# sed 's@id:[0-9]:initdefault:@id:6:initdefault@' /etc/inittab 
# inittab is only used by upstart for the default runlevel.
#
# ADDING OTHER CONFIGURATION HERE WILL HAVE NO EFFECT ON YOUR SYSTEM.
#
# System initialization is started by /etc/init/rcS.conf
#
# Individual runlevels are started by /etc/init/rc.conf
#
# Ctrl-Alt-Delete is handled by /etc/init/control-alt-delete.conf
#
# Terminal gettys are handled by /etc/init/tty.conf and /etc/init/serial.conf,
# with configuration in /etc/sysconfig/init.
#
# For information on how to write upstart event handlers, or how
# upstart works, see init(5), init(8), and initctl(8).
#
# Default runlevel. The runlevels used are:
#   0 - halt (Do NOT set initdefault to this)
#   1 - Single user mode
#   2 - Multiuser, without NFS (The same as 3, if you do not have networking)
#   3 - Full multiuser mode
#   4 - unused
#   5 - X11
#   6 - reboot (Do NOT set initdefault to this)
# 
id:6:initdefault
[root@hadoop1 shelltest]# 

4 高级编辑命令:
h: 用模式空间中的内容覆盖保持空间中的内容;
H:把模式空间中的内容追求至保持空间;
g: get, 从保持空间取回数据至模式空间,覆盖模式空间中原有的内容;
G:GET, 从保持空间取回数据至模式空间,追加于模式空间中现有的内容后面;
x: 把模式空间和保持空间中的内容进行交换;
n: 读取匹配到的行下一行至模式空间;
N: 追加读匹配到的行下一行至模式空间;
d: 删除模式空间中的内容;
D:删除多行模式空间中首行;
sed ‘1!G;h;$!d’ FILE

[root@hadoop1 shelltest]# sed '1!G;h;$!d' /etc/issue

Kernel \r on an \m
CentOS release 6.5 (Final)
[root@hadoop1 shelltest]# cat /etc/issue
CentOS release 6.5 (Final)
Kernel \r on an \m

[root@hadoop1 shelltest]# 
    显示文件的偶数行:sed -n 'n;p' FILE;     sed -n '2~2p' FILE
        1
        2
        3
        4

    显示文件的奇数行:sed -n 'p;n' FILE;     sed -n '1~2;p' FILE

练习:用sed将文件的奇数行与偶数行合并为一行,中间用“&&”隔开;

[root@hadoop1 shelltest]# sed '$!N;s@\n@\&\&@g' /etc/issue
CentOS release 6.5 (Final)&&Kernel \r on an \m

[root@hadoop1 shelltest]# 

5 课后练习:说明以下sed命令的意义:
(1) sed ‘ !N; !D’ /etc/issue

[root@hadoop1 shelltest]# sed '$!N;$!D' /etc/issue
Kernel \r on an \m

[root@hadoop1 shelltest]# cat /etc/issue
CentOS release 6.5 (Final)
Kernel \r on an \m

[root@hadoop1 shelltest]#

(2) sed ‘$!d’ /etc/issue
删除非尾行。
(3) sed ‘G’ /etc/issue

[root@hadoop1 shelltest]# sed 'G' /etc/issue
CentOS release 6.5 (Final)

Kernel \r on an \m



[root@hadoop1 shelltest]# 

因为保持空间里面都是空的,所以追加到模式空间后面全是空行。
(4) sed ‘/^$/d;G’ /etc/issue

[root@hadoop1 shelltest]# sed '/^$/d;G' /etc/issue
CentOS release 6.5 (Final)

Kernel \r on an \m

[root@hadoop1 shelltest]# 

(5) sed ‘n;d’ /etc/issue

[root@hadoop1 shelltest]# sed 'n;d' /etc/issue
CentOS release 6.5 (Final)

[root@hadoop1 shelltest]# 

(6) sed -n ‘1!G;h;$p’ /etc/issue

[root@hadoop1 shelltest]# cat /etc/issue
CentOS release 6.5 (Final)
Kernel \r on an \m

[root@hadoop1 shelltest]# 

6 参考资料
模式空间和保持空间的理解:
什么是模式空间和保持空间

O(∩∩)O嗯!O(∩∩)O嗯!O(∩_∩)O嗯!今天就写到这里,休息一下。如果你看到此文,想进一步学习或者和我沟通,加我微信公众号:名字:五十年后 。
02-shell文本处理三剑客之sed_第1张图片

你可能感兴趣的:(shell)