如何记录搜索引擎蜘蛛爬行痕迹

1、根目录建立一个robots.php文件,将如下代码加入

function get_naps_bot()  
{  
$useragent = strtolower($_SERVER['HTTP_USER_AGENT']);  
if (strpos($useragent, 'googlebot') !== false){  
return 'Google';  
}  
if (strpos($useragent, 'baiduspider') !== false){  
return 'Baidu';  
}  
if (strpos($useragent, 'msnbot') !== false){  
return 'Bing';  
}  
if (strpos($useragent, 'slurp') !== false){  
return 'Yahoo';  
}  
if (strpos($useragent, 'sosospider') !== false){  
return 'Soso';  
}  
if (strpos($useragent, 'sogou spider') !== false){  
return 'Sogou';  
}  
if (strpos($useragent, 'yodaobot') !== false){  
return 'Yodao';  
}  
return false;  
}  
function nowtime(){  
$date=date("Y-m-d.G:i:s");  
return $date;  
}  
$searchbot = get_naps_bot();  
if ($searchbot) {  
$tlc_thispage = addslashes($_SERVER['HTTP_USER_AGENT']);  
$url=$_SERVER['HTTP_REFERER'];  
$file="www.phpfensi.com.txt";  
$time=nowtime();  
$data=fopen($file,"a");  
fwrite($data,"Time:$time robot:$searchbot URL:$tlc_thispagen");  
fclose($data);  
}

将其上传于你的主题目录内.

 

2.在Footer.php或header.php的适当位置添加以下代码调用robots.php,查看源代码打印帮助:

 

程序原理:通过对蜘蛛标识符(如Baiduspider、Googlebot)的判断,记录蜘蛛爬行时间,并生成日志文件robotslogs.txt于根目录.

 

转载于:https://www.cnblogs.com/yesterdaypromise/p/4021552.html