grep sed awk命令学习过滤文件内容

grep

grep [options] pattern [files]

Search one or more files for lines that match a regular expression pattern. Exit status is 0 if any lines match, 1 if none match, and 2 for errors. See also egrep and fgrep.

Options
-c, –count
Print only a count of matched lines. With -v or –revert-match option, count nonmatching lines.
-i, –ignore-case
Ignore uppercase and lowercase distinctions.
-l, –files-with-matches
List the names of files with matches but not individual matched lines; scanning per file stops on the first match.
-n, –line-number
Print lines and their line numbers.
-q, –quiet, –silent
Suppress normal output in favor of quiet mode; scanning stops on the first match.
-v, –invert-match
Print all lines that don’t match pattern.
-E, -extended-regexp
Act like egrep, recognizing extended regular expressions such as (UN|POS)IX to find UNIX and POSIX.
-F, –fixed-strings
Act like fgrep, recognizing only fixed strings instead of regular expressions. Useful when searching for characters that grep normally recognizes as metacharacters.

Examples
[root@localhost-ziyi home]# grep -c /bin/ /etc/passwd
6
[root@localhost-ziyi home]# grep ‘^#include’ /usr/include/zlib.h
#include “zconf.h”
[root@localhost-ziyi home]# grep -l ‘^#include’ /usr/include/zlib.h
/usr/include/zlib.h
[root@localhost-ziyi home]# grep -n ‘^#include’ /usr/include/zlib.h
34:#include “zconf.h”
[root@localhost-ziyi home]#grep –E “3306|1521” /etc/services

---

• sed
  此文甚好，向高人学习了
  http://www.cnblogs.com/emanlee/archive/2013/09/07/3307642.html

  Stream editor(流编辑器，一次处理一行内容) for filtering and transforming text. A stream editor is used to perform basic text transformations on an input stream(a file or input from a pipeline).
  -e : script,
  add the script to the commands to be executed
  -d :
  add the contents of script-file to the commands to be executed

  • 删除以空行开头的人
    #sed ‘/^$/d’ test.txt
  • p 打印
    [root@localhost-ziyi /]# sed /oldboy/p /data/test.txt p:print
    test
    liyao
    oldboy
    Oldboy
  • -n取消默认输出
    [root@localhost-ziyi /]# sed -n /oldboy/p /data/test.txt n:仅 打印被过滤的, 取消默认输出
    Oldboy
  • 打印多行
    sed -n ‘20,30p’ ett.txt n:取消默认输出
  • -i :in place
    sed –i s#SELINUX=enforcing#SELINUX=disabled#g /etc/selinux/config
    -r: 忽略正则, use extended regular expressions in the script.
    sed -rn ‘s#^.dr:(.) Bc.*$#\1#gp’
    基本sed编程举例:
    使用p(rint)显示行: sed -n ‘2p’ temp.txt 只显示第2行,使用选项n
    打印范围: sed -n ‘1,3p’ temp.txt 打印第1行到第3行
    打印模式: sed -n ‘/movie/’p temp.txt 打印含movie的行
    使用模式和行号查询: sed -n ‘3,/movie/’p temp.txt 只在第3行查找movie并打印
    显示整个文件: sed -n ‘1,$’p temp.txt $为最后一行
    任意字符: sed -n ‘/.*ing/’p temp.txt 注意是.*ing,而不是*ing
    打印行号: sed -e ‘/music/=’ temp.txt
    附加文本:(创建sed脚本文件)chmod u+x script.sed,运行时./script.sed temp.txt
    #!/bin/sed -f
    /name1/ a/ #a/表示此处换行添加文本
    HERE ADD NEW LINE. #添加的文本内容
    插入文本: /name1/ a/ 改成 4 i/ 4表示行号,i插入
    修改文本: /name1/ a/ 改成 /name1/ c/ 将修改整行,c修改
    删除文本: sed ‘1d’ temp.txt 或者 sed ‘1,4d’ temp.txt
    替换文本: sed ‘s/source/OKSTR/’ temp.txt 将source替换成OKSTR
    sed ‘s//$//g’ temp.txt 将文本中所有的$符号全部删除
    sed ‘s/source/OKSTR/w temp2.txt’ temp.txt 将替换后的记录写入文件temp2.txt
    替换修改字符串: sed ‘s/source/”ADD BEFORE” &/p’ temp.txt
    结果将在source字符串前面加上”ADD BEFORE”,这里的&表示找到的source字符并保存
    sed结果写入到文件: sed ‘1,2 w temp2.txt’ temp.txt
    sed ‘/name/ w temp2.txt’ temp.txt
    从文件中读文本: sed ‘/name/r temp2.txt’ temp.txt
    在每列最后加文本: sed ‘s/[0-9]*/& Pass/g’ temp.txt
    从shell向sed传值: echo $NAME | sed “s/go/$REP/g” 注意需要使用双引号

---

1. awk
   
   1. 带[ ]和不带的区别
      http://www.cnblogs.com/emanlee/p/3327576.html
      [root@localhost-ziyi /]# awk /[^lzc]/ /data/test.txt
      test
      liyao
      [root@localhost-ziyi /]# awk /^lzc/ /data/test.txt
      lzc
   2. awk 用法：awk ’ pattern {action} ’
   3. 变量名的含义

变量名	含义	变量名	含义
FILENAME	当前输入的文件名	FS	输入域的分隔符，默认为空格
NF	每行域的个数	NR	每行的行号

有用的例子
[root@localhost-ziyi home]# cat test.txt
10 11 12 13 14 15 16
20 21 22 23 24 25 26
30 31 32 33 34 35 36
40 41 42 43 44 45 46
[root@localhost-ziyi home]# awk ‘/11/’ test.txt
10 11 12 13 14 15 16
[root@localhost-ziyi home]# awk ‘/11/,/23/’ test.txt
10 11 12 13 14 15 16
20 21 22 23 24 25 26
[root@localhost-ziyi home]# awk ‘/^[10 20]/’ test.txt
10 11 12 13 14 15 16
20 21 22 23 24 25 26
[root@localhost-ziyi home]# awk ‘/^(10|20)/’ test.txt
10 11 12 13 14 15 16
20 21 22 23 24 25 26
[root@localhost-ziyi home]# awk ‘$2==11’ test.txt
10 11 12 13 14 15 16
[root@localhost-ziyi home]# awk ‘$2==”11”’ test.txt
10 11 12 13 14 15 16
[root@localhost-ziyi home]# awk {‘print NR,NF,$1,$NF’} test.txt
1 7 10 16
2 7 20 26
3 7 30 36
4 7 40 46
[root@localhost-ziyi home]# awk ‘/31/ {print $1 $2}’ test.txt
3031
[root@localhost-ziyi home]# cat test.txt**|**awk ‘{print $2}’ :awk可通过管道获得输入
11
21
31
41
[root@localhost-ziyi home]# awk -F “5” ‘{print $1}’ test.txt ：指定行的分隔符. 也可指定多个：-F ‘[ :\t|]’代表空格，冒号，tab和|
10 11 12 13 14 1
20 21 22 23 24 2
30 31 32 33 34 3
40 41 42 43 44 4
[root@localhost-ziyi home]# awk -F “5” ‘{print $2}’ test.txt
16
26
36
46
[root@localhost-ziyi home]# awk ‘BEGIN {FS=”5”} {print $1}’ test.txt ：设定新的分隔符，而且‘’里面可以有多个{}。BEGIN 表示在处理任意行之前进行的操作。
10 11 12 13 14 1
20 21 22 23 24 2
30 31 32 33 34 3
40 41 42 43 44 4
[root@localhost-ziyi home]# awk ‘/33/ {count++;} END {print “tom was founds “count” times “}’ test.txt
tom was founds 1 times
在awk中调用系统变量必须用单引号，如果是双引号，则表示字符串
Flag=abcd
awk ‘{print ‘$Flag’}’ 结果为abcd
awk ‘{print “$Flag”}’ 结果为$Flag