Heritrix 3.1.0 源码解析（三十）

作为CrawlURI uri对象在处理器链中的生命周期，本人认为逻辑上应该从FrontierPreparer处理器开始，再经过后续的处理器（其实具体CrawlURI uri对象的生命周期，是在它的父级CrawlURI uri对象的抽取处理器处理时已经初具雏形，父级CrawlURI uri对象与它的子级CrawlURI uri对象的生命周期是交错的，关于处理器的流程我在前面已经描述过）

经过FrontierPreparer处理器处理的CrawlURI uri对象下一步才是进入BdbFrontier对象的Schedule方法添加到BdbWorkQueue工作队列

该处理器主要是为CrawlURI uri对象初始化配置，包括调度等级、格式化URL链接、生成classkey、设置holderCost、设置优先级策略，为BdbFrontier对象对其调度做准备

本人在Heritrix 3.1.0 源码解析（二十）解析CandidateChain candidateChain处理器链相关联的处理器时已经提到FrontierPreparer处理器，此文并没有分析该处理器的作用，现在回顾一下

首先是设置CrawlURI curi对象的调度等级，是根据当前CrawlURI curi对象的pathFromSeed属性（从seed到当前CrawlURI curi的Hop值，不同链接类型有不同的代码）

/**

     * Calculate the coarse, original 'schedulingDirective' prioritization

     * for the given CrawlURI

     * 

     * @param curi

     * @return

     */

    protected int getSchedulingDirective(CrawlURI curi) {

        if(StringUtils.isNotEmpty(curi.getPathFromSeed())) {

            char lastHop = curi.getPathFromSeed().charAt(curi.getPathFromSeed().length()-1);

            if(lastHop == 'R') {

                // refer

                return getPreferenceDepthHops() >= 0 ? HIGH : MEDIUM;

            } 

        }

        if (getPreferenceDepthHops() == 0) {

            return HIGH;

            // this implies seed redirects are treated as path

            // length 1, which I belive is standard.

            // curi.getPathFromSeed() can never be null here, because

            // we're processing a link extracted from curi

        } else if (getPreferenceDepthHops() > 0 && 

            curi.getPathFromSeed().length() + 1 <= getPreferenceDepthHops()) {

            return HIGH;

        } else {

            // optionally preferencing embeds up to MEDIUM

            int prefHops = getPreferenceEmbedHops(); 

            if (prefHops > 0) {

                int embedHops = curi.getTransHops();

                if (embedHops > 0 && embedHops <= prefHops

                        && curi.getSchedulingDirective() == SchedulingConstants.NORMAL) {

                    // number of embed hops falls within the preferenced range, and

                    // uri is not already MEDIUM -- so promote it

                    return MEDIUM;

                }

            }

            // Everything else stays as previously assigned

            // (probably NORMAL, at least for now)

            return curi.getSchedulingDirective();

        }

    }

UriCanonicalizationPolicy，姑且称为URL格式化策略类，该类为抽象类，提供格式化URL的抽象方法，由具体子类实现

/**

 * URI Canonicalizatioon Policy

 * 

 * @contributor stack

 * @contributor gojomo

 */

public abstract class UriCanonicalizationPolicy {

    public abstract String canonicalize(String uri);

}

RulesCanonicalizationPolicy类继承自抽象类UriCanonicalizationPolicy，实现格式化URL方法

/**

 * URI Canonicalizatioon Policy

 * 

 * @contributor stack

 * @contributor gojomo

 */

public class RulesCanonicalizationPolicy 

    extends UriCanonicalizationPolicy

    implements HasKeyedProperties {

    private static Logger logger =

        Logger.getLogger(RulesCanonicalizationPolicy.class.getName());

    

    protected KeyedProperties kp = new KeyedProperties();

    public KeyedProperties getKeyedProperties() {

        return kp;

    }

    

    {

        setRules(getDefaultRules());

    }

    @SuppressWarnings("unchecked")

    public List<CanonicalizationRule> getRules() {

        return (List<CanonicalizationRule>) kp.get("rules");

    }

    public void setRules(List<CanonicalizationRule> rules) {

        kp.put("rules", rules);

    }

    

    /**

     * Run the passed uuri through the list of rules.

     * @param context Url to canonicalize.

     * @param rules Iterator of canonicalization rules to apply (Get one

     * of these on the url-canonicalizer-rules element in order files or

     * create a list externally).  Rules must implement the Rule interface.

     * @return Canonicalized URL.

     */

    public String canonicalize(String before) {

        String canonical = before;

        if (logger.isLoggable(Level.FINER)) {

            logger.finer("Canonicalizing: "+before);

        }

        for (CanonicalizationRule rule : getRules()) {

            if(rule.getEnabled()) {

                canonical = rule.canonicalize(canonical);

            }

            if (logger.isLoggable(Level.FINER)) {

                logger.finer(

                    "Rule " + rule.getClass().getName() + " "

                    + (rule.getEnabled()

                            ? canonical :" (disabled)"));

            }

        }

        return canonical;

    }

    

    /**

     * A reasonable set of default rules to use, if no others are

     * provided by operator configuration.

     */

    public static List<CanonicalizationRule> getDefaultRules() {

        List<CanonicalizationRule> rules = new ArrayList<CanonicalizationRule>(6);

        rules.add(new LowercaseRule());

        rules.add(new StripUserinfoRule());

        rules.add(new StripWWWNRule());

        rules.add(new StripSessionIDs());

        rules.add(new StripSessionCFIDs());

        rules.add(new FixupQueryString());

        return rules;

    }

}

格式化URL方法里面迭代调用CanonicalizationRule类型集合里面的成员对象的String canonicalize(String url)方法

CanonicalizationRule是接口，接口声明了String canonicalize(String url)方法，实现该接口的有上面静态方法List<CanonicalizationRule> getDefaultRules()中添加的类，这种处理方式有点类似composite模式与Iterator模式的结合，不过枝节点与叶节点并没有实现共同的接口类型

QueueAssignmentPolicy类为生成URL对象的Classkey策略，该类同样为抽象类，提供生成Classkey的方法（工作队列的标识也就是根据这个生成的Classkey）

默认的生成URL对象的Classkey策略为SurtAuthorityQueueAssignmentPolicy实现类，是根据URL对象的域名生成字符串，因此相同域名的站点里面的URL对象也就只有这一个Classkey标识，也就是只有一个工作队列

我们可以扩展Classkey生成策略，比较经典的是利用ELFHash算法为CrawlURI curi对象分配Key值 ，我这里做一个示例，新建MyQueueAssignmentPolicy类，继承自抽象类QueueAssignmentPolicy，相关源码如下：

/**

     * 

     */

    private static final long serialVersionUID = 1L;



    @Override

    public String getClassKey(CrawlURI cauri) 

    {

        // TODO Auto-generated method stub

        String uri = cauri.getURI().toString();         

        long hash = ELFHash(uri);//利用ELFHash算法为uri分配Key值         

        String a = Long.toString(hash % 50);//取模50，对应50个线程         

        return a;

    }

    public long ELFHash(String str)      

    {         

        long hash = 0;         

        long x   = 0;         

        for(int i = 0; i < str.length(); i++)         

        {            

            hash = (hash << 4) + str.charAt(i);//将字符中的每个元素依次按前四位与上            

            if((x = hash & 0xF0000000L) != 0)//个元素的低四位想与           

            {               

                hash ^= (x >> 24);//长整的高四位大于零，折回再与长整后四位异或              

                hash &= ~x;            

            }         

        }         

        return (hash & 0x7FFFFFFF);      

    }

然后我们在配置文件crawler-beans.cxml里面将FrontierPreparer处理器Bean的queueAssignmentPolicy属性设置成我们扩展的MyQueueAssignmentPolicy类的Bean就可以了

UriPrecedencePolicy类为CrawlURI curi对象优先级策略，该类同样为抽象类，提供设置CrawlURI curi对象的优先级的抽象方法

abstract public class UriPrecedencePolicy implements Serializable {



    /**

     * Add a precedence value to the supplied CrawlURI, which is being 

     * scheduled onto a frontier queue for the first time. 

     * @param curi CrawlURI to assign a precedence value

     */

    abstract public void uriScheduled(CrawlURI curi);



}

默认为CostUriPrecedencePolicy类，根据CrawlURI curi对象的持有成本设置其优先级

/**

 * UriPrecedencePolicy which sets a URI's precedence to its 'cost' -- which

 * simulates the in-queue sorting order in Heritrix 1.x, where cost 

 * contributed the same bits to the queue-insert-key that precedence now does.

 */

public class CostUriPrecedencePolicy extends UriPrecedencePolicy {

    private static final long serialVersionUID = -8164425278358540710L;



    /* (non-Javadoc)

     * @see org.archive.crawler.frontier.precedence.UriPrecedencePolicy#uriScheduled(org.archive.crawler.datamodel.CrawlURI)

     */

    @Override

    public void uriScheduled(CrawlURI curi) {

        curi.setPrecedence(curi.getHolderCost()); 

    }

}

FrontierPreparer处理器Bean的相关策略在crawler-beans.cxml配置文件中的配置如下

 <!-- 

   OPTIONAL BEANS

    Uncomment and expand as needed, or if non-default alternate 

    implementations are preferred.

  -->

  

 <!-- CANONICALIZATION POLICY -->

 <bean id="canonicalizationPolicy" 

   class="org.archive.modules.canonicalize.RulesCanonicalizationPolicy">

   <property name="rules">

    <list>

     <bean class="org.archive.modules.canonicalize.LowercaseRule" />

     <bean class="org.archive.modules.canonicalize.StripUserinfoRule" />

     <bean class="org.archive.modules.canonicalize.StripWWWNRule" />

     <bean class="org.archive.modules.canonicalize.StripSessionIDs" />

     <bean class="org.archive.modules.canonicalize.StripSessionCFIDs" />

     <bean class="org.archive.modules.canonicalize.FixupQueryString" />

    </list>

  </property>

 </bean> 



 <!-- QUEUE ASSIGNMENT POLICY -->

 <bean id="queueAssignmentPolicy" 

   class="org.archive.crawler.frontier.SurtAuthorityQueueAssignmentPolicy">

  <property name="forceQueueAssignment" value="" />

  <property name="deferToPrevious" value="true" />

  <property name="parallelQueues" value="1" />

 </bean>

 

 <!-- URI PRECEDENCE POLICY -->

 <bean id="uriPrecedencePolicy" 

   class="org.archive.crawler.frontier.precedence.CostUriPrecedencePolicy">

 </bean>

 

 <!-- COST ASSIGNMENT POLICY -->

 <bean id="costAssignmentPolicy" 

   class="org.archive.crawler.frontier.UnitCostAssignmentPolicy">

 </bean>

---------------------------------------------------------------------------

本系列Heritrix 3.1.0 源码解析系本人原创

转载请注明出处 博客园 刺猬的温驯

本文链接 http://www.cnblogs.com/chenying99/archive/2013/04/29/3050992.html