Heritrix 3.1.0 源码解析（二十六）
上文分析了Heritrix3.1.0系统对HttpClient组件的请求处理类的封装，本文接下来分析Heritrix3.1.0系统是怎样封装请求证书的
Heritrix3.1.0系统的package org.archive.modules.credential里面的相关类都是与请求证书有关的
先来了解一下CredentialStore类，该类用Map类型存储了应用的所有证书(Credential)，外部只要调用这个类就可以获取证书
该类重要方法如下
KeyedProperties kp = new KeyedProperties(); public KeyedProperties getKeyedProperties() { return kp; } /** * Credentials used by heritrix authenticating. See * http://crawler.archive.org/proposals/auth/ for background. * * @see http://crawler.archive.org/proposals/auth/ */ { setCredentials(new HashMap<String, Credential>()); } @SuppressWarnings("unchecked") public Map<String,Credential> getCredentials() { return (Map<String,Credential>) kp.get("credentials"); } public void setCredentials(Map<String,Credential> map) { kp.put("credentials",map); } /** * List of possible credential types as a List. * * This types are inner classes of this credential type so they cannot * be created without their being associated with a credential list. */ private static final List<Class<?>> credentialTypes; // Initialize the credentialType data member. static { // Array of all known credential types. Class<?> [] tmp = {HtmlFormCredential.class, HttpAuthenticationCredential.class}; credentialTypes = Collections.unmodifiableList(Arrays.asList(tmp)); } /** * Constructor. */ public CredentialStore() { } /** * @return Unmodifable list of credential types. */ public static List<Class<?>> getCredentialTypes() { return CredentialStore.credentialTypes; } /** * @param context Pass a ProcessorURI. Used to set * context. * @return An iterator or null. */ public Collection<Credential> getAll() { Map<String,Credential> map = getCredentials(); return map.values(); } /** * @param context Used to set context. * @param name Name to give the manufactured credential. Should be unique * else the add of the credential to the list of credentials will fail. * @return Returns <code>name</code>'d credential. * @throws AttributeNotFoundException * @throws MBeanException * @throws ReflectionException */ public Credential get(/*StateProvider*/Object context, String name) { return getCredentials().get(name); } /** * Return set made up of all credentials of the passed * <code>type</code>. * * @param context Used to set context. * @param type Type of the list to return. Type is some superclass of * credentials. * @param rootUri RootUri to match. May be null. In this case we return * all. Currently we expect the CrawlServer name to equate to root Uri. * Its not. Currently it doesn't distingush between servers of same name * but different ports (e.g. http and https). * @return Unmodifable sublist of all elements of passed type. */ public Set<Credential> subset(CrawlURI context, Class<?> type, String rootUri) { Set<Credential> result = null; for (Credential c: getAll()) { if (!type.isInstance(c)) { continue; } if (rootUri != null) { String cd = c.getDomain(); if (cd == null) { continue; } if (!rootUri.equalsIgnoreCase(cd)) { continue; } } if (result == null) { result = new HashSet<Credential>(); } result.add(c); } return result; }
上面方法分别提供了获取所有证书（Map类型），根据名称（Map的key键）获取证书和获取所有证书类型
（注意到最后的subset方法，好像没有用到CrawlURI context参数，方法返回的只能是指定域并且指定证书类型的证书集合）
从它的静态代码块可以看到，系统提供了两种类型的证书类型，分别是HtmlFormCredential.class, HttpAuthenticationCredential.class，前者用于form认证，后者用于Basic/Digest HTTP认证
两种证书类型继承自抽象类Credential，先看一下该抽象类的方法
/** *域名 * The root domain this credential goes against: E.g. www.archive.org */ String domain = ""; /** * @param context Context to use when searching for credential domain. * @return The domain/root URI this credential is to go against. * @throws AttributeNotFoundException If attribute not found. */ public String getDomain() { return this.domain; } public void setDomain(String domain) { this.domain = domain; } /** *为CrawlURI curi对象添加当前证书 * Attach this credentials avatar to the passed <code>curi</code> . * * Override if credential knows internally what it wants to attach as * payload. Otherwise, if payload is external, use the below * {@link #attach(CrawlURI, String)}. * * @param curi CrawlURI to load with credentials. */ public void attach(CrawlURI curi) { curi.getCredentials().add(this); } /** *为CrawlURI curi对象移除当前证书 * Detach this credential from passed curi. * * @param curi * @return True if we detached a Credential reference. */ public boolean detach(CrawlURI curi) { return curi.getCredentials().remove(this); } /** *为CrawlURI curi对象移除所有证书 * Detach all credentials of this type from passed curi. * * @param curi * @return True if we detached references. */ public boolean detachAll(CrawlURI curi) { boolean result = false; Iterator<Credential> iter = curi.getCredentials().iterator(); while (iter.hasNext()) { Credential cred = iter.next(); if (cred.getClass() == this.getClass()) { iter.remove(); result = true; } } return result; } /** *判断CrawlURI curi对象是否需要当前证书认证 * @param curi CrawlURI to look at. * @return True if this credential IS a prerequisite for passed * CrawlURI. */ public abstract boolean isPrerequisite(CrawlURI curi); /** *判断CrawlURI curi对象是否存在认证URI * @param curi CrawlURI to look at. * @return True if this credential HAS a prerequisite for passed CrawlURI. */ public abstract boolean hasPrerequisite(CrawlURI curi); /** *获取CrawlURI curi对象的认证URI * Return the authentication URI, either absolute or relative, that serves * as prerequisite the passed <code>curi</code>. * * @param curi CrawlURI to look at. * @return Prerequisite URI for the passed curi. */ public abstract String getPrerequisite(CrawlURI curi); /** *获取CrawlURI curi对象的认证URI * @param context Context to use when searching for credential domain. * @return Key that is unique to this credential type. * @throws AttributeNotFoundException */ public abstract String getKey(); /** *判断CrawlURI curi对象是否每次都要认证 * @return True if this credential is of the type that needs to be offered * on each visit to the server (e.g. Rfc2617 is such a type). */ public abstract boolean isEveryTime(); /** *为HttpMethod method添加认证参数 * @param curi CrawlURI to as for context. * @param http Instance of httpclient. * @param method Method to populate. * @return True if added a credentials. */ public abstract boolean populate(CrawlURI curi, HttpClient http, HttpMethod method); /** *是否post认证 * @param curi CrawlURI to look at. * @return True if this credential is to be posted. Return false if the * credential is to be GET'd or if POST'd or GET'd are not pretinent to this * credential type. */ public abstract boolean isPost(); /** * 判断CrawlURI curi对象的CrawlServer类中的名称与当前认证对象的域名是否一致（用于排除不需要当前认证的CrawlURI curi对象） * Test passed curi matches this credentials rootUri. * @param controller * @param curi CrawlURI to test. * @return True if domain for credential matches that of the passed curi. */ public boolean rootUriMatch(ServerCache cache, CrawlURI curi) { String cd = getDomain(); CrawlServer serv = cache.getServerFor(curi.getUURI()); String serverName = serv.getName(); // String serverName = controller.getServerCache().getServerFor(curi). // getName(); logger.fine("RootURI: Comparing " + serverName + " " + cd); return cd != null && serverName != null && serverName.equalsIgnoreCase(cd); }
上述方法的功能是为CrawlURI curi对象添加当前证书、移除当前证书、为HttpMethod method对象添加证书参数、判断CrawlURI curi对象的域名与当前证书的域名是否一致等
HtmlFormCredential对象继承自上述证书类Credential，为CrawlURI curi对象提供form认证，相关方法实现如下
/** * Full URI of page that contains the HTML login form we're to apply these * credentials too: E.g. http://www.archive.org */ String loginUri = ""; public String getLoginUri() { return this.loginUri; } public void setLoginUri(String loginUri) { this.loginUri = loginUri; } /** * Form items. */ Map<String,String> formItems = new HashMap<String,String>(); public Map<String,String> getFormItems() { return this.formItems; } public void setFormItems(Map<String,String> formItems) { this.formItems = formItems; } enum Method { GET, POST } /** * GET or POST. */ Method httpMethod = Method.POST; public Method getHttpMethod() { return this.httpMethod; } public void setHttpMethod(Method method) { this.httpMethod = method; } /** * Constructor. */ public HtmlFormCredential() { } public boolean isPrerequisite(final CrawlURI curi) { boolean result = false; String curiStr = curi.getUURI().toString(); String loginUri = getPrerequisite(curi); if (loginUri != null) { try {
//登录url UURI uuri = UURIFactory.getInstance(curi.getUURI(), loginUri); if (uuri != null && curiStr != null && uuri.toString().equals(curiStr)) { result = true; if (!curi.isPrerequisite()) { curi.setPrerequisite(true); logger.fine(curi + " is prereq."); } } } catch (URIException e) { logger.severe("Failed to uuri: " + curi + ", " + e.getMessage()); } } return result; } public boolean hasPrerequisite(CrawlURI curi) { return getPrerequisite(curi) != null; } public String getPrerequisite(CrawlURI curi) { return getLoginUri(); } public String getKey() { return getLoginUri(); } public boolean isEveryTime() { // This authentication is one time only. return false; } public boolean populate(CrawlURI curi, HttpClient http, HttpMethod method) { // http is not used boolean result = false; Map<String,String> formItems = getFormItems(); if (formItems == null || formItems.size() <= 0) { try { logger.severe("No form items for " + method.getURI()); } catch (URIException e) { logger.severe("No form items and exception getting uri: " + e.getMessage()); } return result; } NameValuePair[] data = new NameValuePair[formItems.size()]; int index = 0; String key = null; for (Iterator<String> i = formItems.keySet().iterator(); i.hasNext();) { key = i.next(); data[index++] = new NameValuePair(key, (String)formItems.get(key)); } if (method instanceof PostMethod) { ((PostMethod)method).setRequestBody(data); result = true; } else if (method instanceof GetMethod) { // Append these values to the query string. // Get current query string, then add data, then get it again // only this time its our data only... then append. HttpMethodBase hmb = (HttpMethodBase)method; String currentQuery = hmb.getQueryString(); hmb.setQueryString(data); String newQuery = hmb.getQueryString(); hmb.setQueryString( ((StringUtils.isNotEmpty(currentQuery)) ? currentQuery + "&" : "") + newQuery); result = true; } else { logger.severe("Unknown method type: " + method); } return result; } public boolean isPost() { return Method.POST.equals(getHttpMethod()); }
上述方法的功能 我在它的接口方法里面已经注释了，这里不再重复
另外HttpAuthenticationCredential证书类提供了Basic/Digest HTTP认证功能，源码我就不具体分析了，可以参照HtmlFormCredential类的认证功能对比不难理解了
在Heritrix3.1.0官方的参考文档里面提供了两种认证方式在配置文件crawler-beans.cxml中的示例（官方的示例里面关键词有误）
<bean id="credentialStore" class="org.archive.modules.credential.CredentialStore"> <property name="credentials"> <map> <entry key="formCredential" value-ref="formCredential" /> </map> </property> </bean> <bean id="credential" class="org.archive.modules.credential.HtmlFormCredential"> <property name="domain" value="example.com" /> <property name="login-uri" value="http://example.com/login"/> <property name="form-items"> <map> <entry key="login" value="mylogin"/> <entry key="password" value="mypassword"/> <entry key="submit" value="submit"/> </map> </property> </bean> <bean id="credential" class="org.archive.modules.credential.HttpAuthenticationCredential"> <property name="domain"><value>domain</value></property> <property name="realm"><value>myrealm</value></property> <property name="login"><value>mylogin</value></property> <property name="password"><value>mypassword</value></property> </bean>
---------------------------------------------------------------------------
本系列Heritrix 3.1.0 源码解析系本人原创
转载请注明出处 博客园 刺猬的温驯
本文链接 http://www.cnblogs.com/chenying99/archive/2013/04/28/3049042.html