Cefsharp抓取拼多多每日关键词

        最近业余时间都在学习Cefsharp实现本地客户端，发现Cefsharp可以很方便的爬取一些网站的信息，于是就一些爬取过程做一下记录。

 拼多多商品搜索请求过程分析

拼多多商品搜索的链接是:拼多多，我们在CefSharp发起请求时加上log打印相关的请求信息,经过分析log，发现这条有关键字的信息是在一次MimeType为json的HTTP资源请求。代码段和日志如下

protected override IResponseFilter GetResourceResponseFilter(IWebBrowser chromiumWebBrowser, IBrowser browser, IFrame frame, IRequest request, IResponse response)
        {
            logger.Debug(" request_url=" + request.Url + ";request_id=" + request.Identifier + ";response_MimeType=" + response.MimeType + ";response_charset=" + response.Charset + ";response_status=" + response.StatusText);

            return base.GetResourceResponseFilter(chromiumWebBrowser, browser, frame, request, response);
        }

2022-07-09 09:46:18.6335 DEBUG 20076-12 Chrome.MyChrome.CefHandlers.MyResourceRequestHandler.GetResourceResponseFilter  request_url=https://mobile.yangkeduo.com/proxy/api/search_hotquery?pdduid=0&plat=h5&source=index;request_id=759816;response_MimeType=application/json;response_charset=utf-8;response_status=
             

使用我上传的资源下载器也可以加载到对应的json文件.使用CefSharp结合vue3实现简单URL资源下载器-C#文档类资源-CSDN下载 

 商品关键词解析

        获取关键词对应的http请求后，我们就可以在C#里面对本次的资源请求进行截取分析，关键步骤如下。

重写CefSharp.Handler.RequestHandler类的GetResourceRequestHandler方法返回自定义的资源处理类

 protected override IResourceRequestHandler GetResourceRequestHandler(IWebBrowser chromiumWebBrowser, IBrowser browser, IFrame frame, IRequest request, bool isNavigation, bool isDownload, string requestInitiator, ref bool disableDefaultHandling)
        {
            chrome.Logger.Debug("request_url=" + request.Url + ";request_id=" + request.Identifier + ";TransitionType=" + request.TransitionType + ";ReferrerUrl=" + request.ReferrerUrl + ";Method=" + request.Method + ";IsReadOnly=" + request.IsReadOnly + ";isNavigation=" + isNavigation + ";isDownload=" + isDownload + ";requestInitiator=" + requestInitiator);
            return new MyResourceRequestHandler(webRequest);
        }

重写CefSharp.Handler.ResourceRequestHandler类的GetResourceResponseFilter方法，将自定义的Stream传入。

        Stream DataStream = new MemoryStream();   
protected override IResponseFilter GetResourceResponseFilter(IWebBrowser chromiumWebBrowser, IBrowser browser, IFrame frame, IRequest request, IResponse response)
        {
            logger.Debug(" request_url=" + request.Url + ";request_id=" + request.Identifier + ";response_MimeType=" + response.MimeType + ";response_charset=" + response.Charset + ";response_status=" + response.StatusText);
            return new CefSharp.ResponseFilter.StreamResponseFilter(DataStream);
        }

 重写CefSharp.Handler.ResourceRequestHandler类的OnResourceLoadComplete方法对Stream存储的数据进行解析处理

protected override void OnResourceLoadComplete(IWebBrowser chromiumWebBrowser, IBrowser browser, IFrame frame, IRequest request, IResponse response, UrlRequestStatus status, long receivedContentLength)
        {
            logger.Debug("request_url=" + request.Url + ";request_id=" + request.Identifier + ";request_status=" + status + ";recv_length=" + receivedContentLength);
             var ms = DataStream as MemoryStream;
            string Response2String = "";
                var bytes = ms.ToArray();
                if (Charset.IndexOf("utf-8", System.StringComparison.OrdinalIgnoreCase) >= 0)
                {
                    Response2String = System.Text.Encoding.UTF8.GetString(bytes);
                }
                else if (Charset.IndexOf("gbk", System.StringComparison.OrdinalIgnoreCase) >= 0)
                {
                    Response2String = System.Text.Encoding.GetEncoding("GB2312").GetString(bytes);
                }
                else
                {
                    Response2String = System.Text.Encoding.UTF8.GetString(bytes);
                    Logger.Error("unknow_charset Charset=" + Charset);
                }

                  JObject jsonObj = JObject.Parse(Response2String);
                    Logger.Debug("parse_json_success json_str=" + jsonObj["items"].ToString());
        }

json处理我选用的是Newtonsoft.Json.Linq的JObject进行动态解析。