归档—监控ORACLE数据库告警日志

ORACLE的告警日志里面包含许多有用的信息，尤其是一些ORACLE的ORA错误信息，所以有必要及时归档、监控数据库告警日志的ORA错误，及时提醒数据库管理员DBA处理这些错误信息，那么我们首先来看看告警日志的内容片断：

Thread 1 advanced to log sequence 37749 (LGWR switch)
  Current log# 6 seq# 37749 mem# 0: /u01/oradata/SCM2/redo06.log
Thu Jun 27 15:02:30 2013
Thread 1 advanced to log sequence 37750 (LGWR switch)
  Current log# 2 seq# 37750 mem# 0: /u01/oradata/SCM2/redo02.log
Thu Jun 27 15:13:43 2013
Thread 1 advanced to log sequence 37751 (LGWR switch)
  Current log# 3 seq# 37751 mem# 0: /u01/oradata/SCM2/redo03.log
Thu Jun 27 15:25:30 2013
Thread 1 advanced to log sequence 37752 (LGWR switch)
  Current log# 4 seq# 37752 mem# 0: /u01/oradata/SCM2/redo04.log
Thu Jun 27 15:32:20 2013
ORA-00060: Deadlock detected. More info in file /u01/app/oracle/admin/SCM2/bdump/scm2_s001_14052.trc.
Thu Jun 27 15:35:05 2013
Thread 1 advanced to log sequence 37753 (LGWR switch)
  Current log# 5 seq# 37753 mem# 0: /u01/oradata/SCM2/redo05.log
Thu Jun 27 15:43:11 2013
Thread 1 advanced to log sequence 37754 (LGWR switch)
  Current log# 1 seq# 37754 mem# 0: /u01/oradata/SCM2/redo01.log
Thu Jun 27 15:49:58 2013
Thread 1 advanced to log sequence 37755 (LGWR switch)
  Current log# 6 seq# 37755 mem# 0: /u01/oradata/SCM2/redo06.log
Thu Jun 27 16:01:25 2013
Thread 1 advanced to log sequence 37756 (LGWR switch)
  Current log# 2 seq# 37756 mem# 0: /u01/oradata/SCM2/redo02.log
Thu Jun 27 16:12:14 2013
Thread 1 advanced to log sequence 37757 (LGWR switch)
  Current log# 3 seq# 37757 mem# 0: /u01/oradata/SCM2/redo03.log
Thu Jun 27 16:24:10 2013
Thread 1 advanced to log sequence 37758 (LGWR switch)

View Code

 

归档告警日志文件

告警日志文件如果不加管理的话，那么文件会持续增长，有时候文件会变得非常大，不利于读写。一般建议将告警日志按天归档，归档文件保留三个月（视情况而定），下面来看看将告警日志文件归档的两个Shell脚本：

alert_log_archive.sh version 1

1. #*************************************************************************
2. #  FileName     :alert_log_archive.sh
3. #*************************************************************************
4. #  Author       :Kerry
5. #  CreateDate   :2013-07-02
6. #  blogs　　     :www.cnblogs.com/kerrycode
7. #  Description  :this script is made the alert log archived every day
8. #*************************************************************************
9.  
10. #! /bin/bash
11.  
12. date=`date +%Y%m%d`
13.  
14. alert_log_path="$ORACLE_BASE/admin/$ORACLE_SID/bdump"
15.  
16. alert_log_file="alert_$ORACLE_SID.log"
17.  
18. alert_arc_file="alert_$ORACLE_SID.log""."${date}
19.  
20. cd ${alert_log_path};
21.  
22.  
23. if [ ! -e "${alert_log_file}" ]; then
24.         echo "the alert log didn't exits, please check file path is correct!";
25.         exit;
26. fi
27.  
28.  
29. if [ -e ${alert_arc_file} ];then
30.  
31.         echo "the alert log file have been archived!"
32.  
33. else
34.  
35.         cat ${alert_log_file} >> ${alert_arc_file}
36.  
37.         cat /dev/null > ${alert_log_file}
38.  
39. fi

其实脚本1和脚本差别不大，仅仅是mv与cat >>的区别

alert_log_archive.sh version 2

1.  
2. #*************************************************************************
3. #  FileName     :alert_log_archive.sh
4. #*************************************************************************
5. #  Author       :Kerry
6. #  CreateDate   :2013-07-02
7. #  blogs　　     :www.cnblogs.com/kerrycode
8. #  Description  :this script is made the alert log archived every day
9. #*************************************************************************
10.  
11. #! /bin/bash
12.  
13. date=`date +%Y%m%d`
14.  
15. alert_log_path="$ORACLE_BASE/admin/$ORACLE_SID/bdump"
16.  
17. alert_log_file="alert_$ORACLE_SID.log"
18.  
19. alert_arc_file="alert_$ORACLE_SID.log""."${date}
20.  
21. cd ${alert_log_path};
22.  
23.  
24. if [ ! -e "${alert_log_file}" ]; then
25.         echo "the alert log didn't exits, please check file path is correct!";
26.         exit;
27. fi
28.  
29.  
30. if [ -e ${alert_arc_file} ];then
31.  
32.         echo "the alert log file have been archived!"
33.  
34. else
35.  
36.         mv ${alert_log_file}  ${alert_arc_file}
37.  
38.         cat /dev/null > ${alert_log_file}
39.  
40. fi

然后在crontab定时任务里面加上下面语句，每天23点59对告警日志进行归档。

[oracle@DB-Server scripts]$ crontab -l

# the alert log archived every day                    Add by kerry 2013-07-02

59 23 * * * /home/oracle/scripts/alert_log_archive.sh >/dev/null 2>$1

细心的朋友可能已经发现上面的脚本、配置错误了，我在部署测试的过程中，是指定二十分钟执行一次，但是等了四十分钟，发现定时任务一次都没有执行，手工执行上面脚本是完全没有问题的，最后仔细的检查一遍，居然发现悲剧的发现时自己一时粗心将&符号写成了$,真是很二的一个错误

59 23 * * * /home/oracle/scripts/alert_log_archive.sh >/dev/null 2>$1

59 23 * * * /home/oracle/scripts/alert_log_archive.sh >/dev/null 2>&1

 

接下来测试发现脚本执行有问题，在crontab 里执行该shell脚本时，获取不到ORACLE的环境变量，这是因为crontab环境变量问题，Crontab的环境默认情况下并不包含系统中当前用户的环境。所以，你需要在shell脚本中添加必要的环境变量的设置，修改的脚本如下：

alert_log_archive.sh V1

1. #*************************************************************************
2. #  FileName     :alert_log_archive.sh
3. #*************************************************************************
4. #  Author       :Kerry
5. #  CreateDate   :2013-07-02
6. #  blogs　　     :www.cnblogs.com/kerrycode
7. #  Description  :this script is made the alert log archived every day
8. #*************************************************************************
9.  
10. #! /bin/bash
11.  
12. # these solved the oracle variable problem.
13. export ORACLE_SID=gps
14. export ORACLE_BASE=/u01/app/oracle
15.  
16. date=`date +%Y%m%d`
17.  
18. alert_log_path="$ORACLE_BASE/admin/$ORACLE_SID/bdump"
19.  
20. alert_log_file="alert_$ORACLE_SID.log"
21.  
22. alert_arc_file="alert_$ORACLE_SID.log""."${date}
23.  
24. cd ${alert_log_path};
25.  
26.  
27. if [ ! -e "${alert_log_file}" ]; then
28.         echo "the alert log didn't exits, please check file path is correct!";
29.         exit;
30. fi
31.  
32.  
33. if [ -e ${alert_arc_file} ];then
34.  
35.         echo "the alert log file have been archived!"
36.  
37. else
38.  
39.         cat ${alert_log_file} >> ${alert_arc_file}
40.  
41.         cat /dev/null > ${alert_log_file}
42.  
43. fi

 

alert_log_archive.sh V2

1. #*************************************************************************
2. #  FileName     :alert_log_archive.sh
3. #*************************************************************************
4. #  Author       :Kerry
5. #  CreateDate   :2013-07-0
6. #  blogs　　     :www.cnblogs.com/kerrycode
7. #  Description  :this script is made the alert log archived every day
8. #*************************************************************************
9.  
10. #! /bin/bash
11.  
12. # these solved the oracle variable problem.
13. export ORACLE_SID=gps
14. export ORACLE_BASE=/u01/app/oracle
15.  
16. date=`date +%Y%m%d`
17.  
18. alert_log_path="$ORACLE_BASE/admin/$ORACLE_SID/bdump"
19.  
20. alert_log_file="alert_$ORACLE_SID.log"
21.  
22. alert_arc_file="alert_$ORACLE_SID.log""."${date}
23.  
24. cd ${alert_log_path};
25.  
26.  
27. if [ ! -e "${alert_log_file}" ]; then
28.         echo "the alert log didn't exits, please check file path is correct!";
29.         exit;
30. fi
31.  
32.  
33. if [ -e ${alert_arc_file} ];then
34.  
35.         echo "the alert log file have been archived!"
36.  
37. else
38.  
39.         mv ${alert_log_file}  ${alert_arc_file}
40.  
41.        cat /dev/null > ${alert_log_file}
42.  
43. fi

 

监控告警日志文件

接下来看看如何监控告警日志文件的ORA错误，这里是采用Perl结合Shell的方式，因为Shell获取错误的时间、行数等不如Perl操作字符串方便。

monitoring_alert_log.pl

1. #**********************************************************************************
2. #       FileName         :monitoring_alert_log.pl
3. #**********************************************************************************
4. #       Author           :Kerry
5. #       CreateDate       :2013-07-01
6. #       blogs　　         :www.cnblogs.com/kerrycode
7. #       Description      :check the alert log and find out the ora error
8. #**********************************************************************************
9. #    Modified Date    Modified User     Version   Modified Reason
10. #    2013-07-02         Kerry          V01.0.1    add comment for this script
11. #***********************************************************************************
12.  
13.  
14. #! /usr/bin/perl
15.  
16.   use strict;
17.    
18. my($argv) = @ARGV;
19.  
20. if ( @ARGV != 1)
21. {
22.  
23.   print '
24.       Parameter error:  you must assined the alert log file as a input parameter or the number of prarameter is not right.
25.  
26. ';
27.  
28.   exit
29. }
30.  
31.   if( ! -e $argv )
32. {  
33.   print '  
34.   Usage: monitoring_alert_log.pl  
35.                                                     
36.   $ cat alert_[sid].log | monitoring_alert_log.pl
37.   $ tail -f alert_[sid].log | monitoring_alert_log.pl  
38.   $ monitoring_alert_log.pl alert_[sid].log  
39.    
40.    ';
41.      exit;  
42. }  
43. my $err_regex = '^(\w+ \w+ \d{2} \d{2}:\d{2}:\d{2} \d{4})|(ORA-\d+:.+)$';  
44. my $date = "";  
45. my $line_counter = 0;  
46. while ( <> )  
47. {  
48.      $line_counter++;  
49.      if( m/$err_regex/oi )  
50.      {  
51.          if ($1)  
52.          {  
53.              $date = $1;  
54.              next;  
55.          }  
56.          print "$line_counter | $date | $2 \n" if ($2);  
57.      }  
58. }

 

monitoring_alert_log.sh

1. #**********************************************************************************
2. #    FileName     :            monitoring_alert_log.sh
3. #**********************************************************************************
4. #    Author       :            Kerry    
5. #    CreateDate   :            2013-07-01
6. #    blogs　　     :            www.cnblogs.com/kerrycode
7. #    Description:            check the alert log and find out the ora error
8. #**********************************************************************************
9. #    Modified Date    Modified User  Version             Modified Reason
10. #   2013-07-02          Kerry        V01.0.1             add comment and modified script
11. #***********************************************************************************    
12.  
13. #!/bin/bash
14.  
15. # these solved the oracle variable problem.
16. export ORACLE_SID=gsp
17. export ORACLE_BASE=/u01/app/oracle
18.  
19. logfile="/home/oracle/scripts/alter_err_log.txt"
20. pl_monitoring_alert="/home/oracle/scripts/monitoring_alert_log.pl"
21. pl_sendmail="/home/oracle/scripts/sendmail.pl"
22. alert_logfile="$ORACLE_BASE/admin/$ORACLE_SID/bdump/alert_$ORACLE_SID.log"
23.  
24.  
25.  
26. #delete the old alter error log file
27.  
28.   rm -f${logfile}
29.   rm -f${pl_sendmail}
30.  
31. #run the perl and check if exists the ora error
32.  
33.   perl ${pl_monitoring_alert} ${alert_logfile}> ${logfile}
34.  
35. #if have no error in alert log then exit the program
36.  
37. if [[ -e "${logfile}"  &&  ! -s "${logfile}"  ]]; then
38.   exit;
39. fi
40.  
41. date_today=`date +%Y_%m_%d`
42. subject="Monitoring the Oracle Alert logs and find ora errors"
43. content="Dear All,
44.  
45.    The Instance ${ORACLE_SID}\' alert log occured the ora errors ,please see the detail in attachment and take action for it. many thanks!
46.  
47.  
48. Oracle Alert Services
49. "
50.  
51. echo "#!/usr/bin/perl" >> ${pl_sendmail}
52. echo "use Mail::Sender;" >> ${pl_sendmail}
53. echo "\$sender = new Mail::Sender {smtp => '10.xxx.xxx.xxx', from => '[email protected]'}; ">> ${pl_sendmail}
54. echo "\$sender->MailFile({to => '[email protected]',">> ${pl_sendmail}
55. echo "cc=>'[email protected]'," >> ${pl_sendmail}
56. echo "subject => '$subject',">> ${pl_sendmail}
57. echo "msg => '$content',">> ${pl_sendmail}
58. echo "file => '$logfile'});">> ${pl_sendmail}
59.  
60. perl ${pl_sendmail}

*/20 6-21 * * * /home/oracle/scripts/monitoring_alert_log.sh  >/dev/null 2>&1

问题/优化脚本:Crontab 定时任务配置每二十分钟执行一次，结果，又有麻烦事情来了，假如8点发生了ORA错误，之后到下午6点都没有发生ORA错误，上面的脚本会每隔二十分钟发送一次邮件，重复发送，感觉比较烦人，而我需要的是：只有当新的ORA错误出现，才给DBA发送邮件，否则就不要发送，其次，感觉二十分钟的时间段太长了，如果出现了严重错误，二十分钟后才去处理，就显得时延比较滞后，但是如果你频率短的话， 基于第一个bug，你回收到N多邮件，那么我们继续改写，优化下面脚本吧

 

Code Snippet

1. #****************************************************************************************************
2. #       FileName         :monitoring_alert_log.sh
3. #****************************************************************************************************
4. #       Author           :Kerry
5. #       CreateDate       :2013-07-01
6. #       Description      :check the alert log and find out the ora error
7. #****************************************************************************************************
8. #       Modified Date  Modified User     Version      Modified Reason
9. #       2013-07-02       Kerry        V01.0.1      add comment and modified script
10. #       2013-07-02       Kerry        V01.0.2      Solved the email repated send problems, only
11. #                                                   the new ora error occured then send the email.
12. #****************************************************************************************************
13.  
14. #!/bin/bash
15.  
16. # these solved the oracle variable problem.
17. export ORACLE_SID=gsp
18. export ORACLE_BASE=/u01/app/oracle
19.  
20. new_log_file="/home/oracle/scripts/new_err_log.txt"
21. old_log_file="/home/oracle/scripts/old_err_log.txt"
22. pl_monitoring_alert="/home/oracle/scripts/monitoring_alert_log.pl"
23. pl_sendmail="/home/oracle/scripts/sendmail.pl"
24. alert_logfile="$ORACLE_BASE/admin/$ORACLE_SID/bdump/alert_${ORACLE_SID}.log"
25.  
26.  
27.  
28. #delete the old alter error log file
29.  
30.   #rm -f${new_log_file}
31.  
32.   rm -f${old_log_file}
33.  
34.   mv ${new_log_file}${old_log_file}
35.  
36.   rm -f${pl_sendmail}
37.  
38. #run the perl and check if exists the ora error
39.  
40.  
41.   perl ${pl_monitoring_alert} ${alert_logfile}> ${new_log_file}
42.  
43. #if have no error in alert log then exit the program
44.  
45. if [[ -e "${new_log_file}"  &&  ! -s "${new_log_file}"  ]]; then
46.   exit;
47. fi
48.  
49. new_err_num=`cat ${new_log_file} | wc -l`
50.  
51. old_err_num=`cat ${old_log_file} | wc -l`
52.  
53.  
54. if [ ${new_err_num} -le ${old_err_num} ]; then
55.  
56.    exit
57. fi
58.  
59. date_today=`date +%Y_%m_%d`
60. subject="xxx (192.168.xxx.xxx) Monitoring the Oracle Alert logs and find ora errors"
61. content="Dear All,
62.  
63.    The Instance ${ORACLE_SID}\' alert log occured the ora errors ,please see the detail in attachment and take action for it. many thanks!
64.  
65.  
66. Oracle Alert Services
67. "
68.  
69. echo "#!/usr/bin/perl" >> ${pl_sendmail}
70. echo "use Mail::Sender;" >> ${pl_sendmail}
71. echo "\$sender = new Mail::Sender {smtp => '10.xxx.xxx.xxx', from => '[email protected]'}; ">> ${pl_sendmail}
72. echo "\$sender->MailFile({to => '[email protected]',">> ${pl_sendmail}
73. echo "cc=>'[email protected]'," >> ${pl_sendmail}
74. echo "subject => '$subject',">> ${pl_sendmail}
75. echo "msg => '$content',">> ${pl_sendmail}
76. echo "file => '${new_log_file}'});">> ${pl_sendmail}
77.  
78. perl ${pl_sendmail}

但是我在部署过程中，由于环境问题（多台ORACLE服务器，不同的操作系统、不同的环境），发送邮件的部分出现改动，又有下面两个小版本的改动

Code Snippet

1. #**********************************************************************************
2. #    FileName       :            monitoring_alert_log.sh
3. #**********************************************************************************
4. #    Author         :            Kerry    
5. #    CreateDate     :            2013-07-01
6. #    Description:            check the alert log and find out the ora error
7. #***********************************************************************************
8. #    Modified Date    Modified User   Version                 Modified Reason
9. #    2013-07-02       Kerry            V01.0.1            add comment and modified script
10. #   2013-07-02        Kerry            V01.0.2     Solved the email repated send problems, only
11. #                                                  the new ora error occured then send the email
12. #***********************************************************************************    
13.  
14. #!/bin/bash
15.  
16. new_log_file="/home/oracle/scripts/new_err_log.txt"
17. old_log_file="/home/oracle/scripts/old_err_log.txt"
18. pl_monitoring_alert="/home/oracle/scripts/monitoring_alert_log.pl"
19. email_content="/home/oracle/scripts/sendmail.txt"
20. alert_logfile="$ORACLE_BASE/admin/$ORACLE_SID/bdump/alert_$ORACLE_SID.log"
21.  
22.  
23.  
24. #delete the old alter error log file
25.  
26. rm -f${old_log_file}
27.  
28. mv ${new_log_file} ${old_log_file}
29.  
30.   rm -f${pl_sendmail}
31.  
32. #run the perl and check if exists the ora error
33.  
34.   perl ${pl_monitoring_alert} ${alert_logfile}> ${new_log_file}
35.  
36. #if have no error in alert log then exit the program
37.  
38. if [[ -e "${new_log_file}"  &&  ! -s "${new_log_file}"  ]]; then
39.   exit;
40. fi
41.  
42.  
43.  
44. new_err_num=`cat ${new_log_file} | wc -l`
45.  
46. old_err_num=`cat ${old_log_file} | wc -l`
47.  
48.  
49. if [ ${new_err_num} -le ${old_err_num} ]; then
50.  
51.    exit
52. fi
53.  
54. date_today=`date +%Y_%m_%d`
55. subject="Monitoring the Oracle Alert logs and find ora errors"
56. content="Dear All,
57.  
58.    The Instance ${ORACLE_SID}\' alert log occured the ora errors ,please see the detail in attachment and take action for it. many thanks!
59.  
60.      The Error is blow :
61. "
62.  
63.  
64. echo 'Content-Type: text/html' > ${email_content}
65. echo 'To: [email protected]' >> ${email_content}
66. echo ${subject} >> ${email_content}
67. echo '

    ' >> ${email_content}

68. echo ${content} >> ${email_content}
69.  
70.   cat ${new_log_file} >>${email_content} 2>&1
71.  
72. echo 'Oracle Alert Services' >> ${email_content}
73.  
75. /usr/sbin/sendmail -t -f ${subject} < ${email_content}
76. rm -f ${email_content}

 

Code Snippet

1. #**********************************************************************************
2. #    FileName     :            monitoring_alert_log.sh
3. #**********************************************************************************
4. #    Author         :            Kerry    
5. #    CreateDate :2013-07-01
6. #    Description:            check the alert log and find out the ora error
7. #***********************************************************************************
8. #    Modified Date    Modified User   Version                 Modified Reason
9. #   2013-07-02        Kerry            V01.0.1            add comment and modified script
10. #   2013-07-02        Kerry            V01.0.2            Solved the email repated send problems, only
11. #                                                         the new ora error occured then send the email
12. #***********************************************************************************    
13.  
14. #!/bin/bash
15.  
16. new_log_file="/home/oracle/scripts/new_err_log.txt"
17. old_log_file="/home/oracle/scripts/old_err_log.txt"
18. pl_monitoring_alert="/home/oracle/scripts/monitoring_alert_log.pl"
19. email_content="/home/oracle/scripts/sendmail.pl"
20. alert_logfile="$ORACLE_BASE/admin/$ORACLE_SID/bdump/alert_$ORACLE_SID.log"
21. reportname="alert_log_err.txt"
22.  
23.  
24. #delete the old alter error log file
25.  
26.   rm -f${old_log_file}
27.  
28. mv ${new_log_file} ${old_log_file}
29.  
30.   rm -f${pl_sendmail}
31.  
32. #run the perl and check if exists the ora error
33.  
34.   perl ${pl_monitoring_alert} ${alert_logfile}> ${new_log_file}
35.  
36. #if have no error in alert log then exit the program
37.  
38. if [[ -e "${new_log_file}"  &&  ! -s "${new_log_file}"  ]]; then
39.   exit;
40. fi
41.  
42. date_today=`date +%Y_%m_%d`
43. subject="Monitoring the Oracle Alert logs and find ora errors"
44. content="Dear All,
45.  
46.    The Instance ${ORACLE_SID}\' alert log occured the ora errors ,please see the detail in attachment and take action for it. many thanks!
47.  
48.  
49. Oracle Alert Services
50. "
51.  
52. ( ${content} ; uuencode ${new_log_file} ${reportname} ) | /bin/mail -s ${subject} [email protected] [email protected]
53.  
54.  
55. /bin/mail