Heritrix 3.1.0 源码解析（十七）

我们接下来分析与与BdbFrontier对象void finished(CrawlURI cURI)方法相关的方法 

/**

     * Note that the previously emitted CrawlURI has completed

     * its processing (for now).

     *

     * The CrawlURI may be scheduled to retry, if appropriate,

     * and other related URIs may become eligible for release

     * via the next next() call, as a result of finished().

     *

     * TODO: make as many decisions about what happens to the CrawlURI

     * (success, failure, retry) and queue (retire, snooze, ready) as 

     * possible elsewhere, such as in DispositionProcessor. Then, break

     * this into simple branches or focused methods for each case. 

     *  

     * @see org.archive.crawler.framework.Frontier#finished(org.archive.modules.CrawlURI)

     */

    protected void processFinish(CrawlURI curi) {

//        assert Thread.currentThread() == managerThread;        

        long now = System.currentTimeMillis();

        //尝试次数

        curi.incrementFetchAttempts();

        logNonfatalErrors(curi);

        

        WorkQueue wq = (WorkQueue) curi.getHolder();

        // always refresh budgeting values from current curi

        // (whose overlay settings should be active here)

        wq.setSessionBudget(getBalanceReplenishAmount());

        wq.setTotalBudget(getQueueTotalBudget());

        

        assert (wq.peek(this) == curi) : "unexpected peek " + wq;



        int holderCost = curi.getHolderCost();

        //是否需要重新处理

        if (needsReenqueuing(curi)) {

            // codes/errors which don't consume the URI, leaving it atop queue

            if(curi.getFetchStatus()!=S_DEFERRED) {

                wq.expend(holderCost); // all retries but DEFERRED cost

            }

            //延时时间

            long delay_ms = retryDelayFor(curi) * 1000;

            curi.processingCleanup(); // lose state that shouldn't burden retry

            wq.unpeek(curi);

            //更新到WorkQueue wq

            wq.update(this, curi); // rewrite any changes

            //重新归队

            handleQueue(wq,curi.includesRetireDirective(),now,delay_ms);

            appCtx.publishEvent(new CrawlURIDispositionEvent(this,curi,DEFERRED_FOR_RETRY));

            doJournalReenqueued(curi);

            wq.makeDirty();

            return; // no further dequeueing, logging, rescheduling to occur

        }



        // Curi will definitely be disposed of without retry, so remove from queue

        //从WorkQueue wq中移除该CrawlURI curi对象

        wq.dequeue(this,curi);

        decrementQueuedCount(1);

        largestQueues.update(wq.getClassKey(), wq.getCount());

        log(curi);

        

        if (curi.isSuccess()) {

            // codes deemed 'success' 

            incrementSucceededFetchCount();

            totalProcessedBytes.addAndGet(curi.getRecordedSize());

            appCtx.publishEvent(new CrawlURIDispositionEvent(this,curi,SUCCEEDED));

            doJournalFinishedSuccess(curi);

           

        } else if (isDisregarded(curi)) {

            // codes meaning 'undo' (even though URI was enqueued, 

            // we now want to disregard it from normal success/failure tallies)

            // (eg robots-excluded, operator-changed-scope, etc)

            incrementDisregardedUriCount();

            appCtx.publishEvent(new CrawlURIDispositionEvent(this,curi,DISREGARDED));

            holderCost = 0; // no charge for disregarded URIs

            // TODO: consider reinstating forget-URI capability, so URI could be

            // re-enqueued if discovered again

            doJournalDisregarded(curi);

            

        } else {

            // codes meaning 'failure'

            incrementFailedFetchCount();

            appCtx.publishEvent(new CrawlURIDispositionEvent(this,curi,FAILED));

            // if exception, also send to crawlErrors

            if (curi.getFetchStatus() == S_RUNTIME_EXCEPTION) {

                Object[] array = { curi };

                loggerModule.getRuntimeErrors().log(Level.WARNING, curi.getUURI()

                        .toString(), array);

            }        

            // charge queue any extra error penalty

            wq.noteError(getErrorPenaltyAmount());

            doJournalFinishedFailure(curi);

            

        }



        wq.expend(holderCost); // successes & failures charge cost to queue

        //延时时间

        long delay_ms = curi.getPolitenessDelay();

        //long delay_ms = 0;

        //重新归队

        handleQueue(wq,curi.includesRetireDirective(),now,delay_ms);

        wq.makeDirty();

        

        if(curi.getRescheduleTime()>0) {

            // marked up for forced-revisit at a set time

            curi.processingCleanup();

            curi.resetForRescheduling(); 

            futureUris.put(curi.getRescheduleTime(),curi);

            futureUriCount.incrementAndGet(); 

        } else {

            curi.stripToMinimal();

            curi.processingCleanup();

        }

    }

首先判断CrawlURI curi对象是否需要重新放入队列，方法如下

/**

     * Checks if a recently processed CrawlURI that did not finish successfully

     * needs to be reenqueued (and thus possibly, processed again after some 

     * time elapses)

     * 

     * @param curi

     *            The CrawlURI to check

     * @return True if we need to retry.

     */

    protected boolean needsReenqueuing(CrawlURI curi) {

        //是否超过最大的尝试次数，默认为30次

        if (overMaxRetries(curi)) {

            return false;

        }

        //根据状态判断

        switch (curi.getFetchStatus()) {

        case HttpStatus.SC_UNAUTHORIZED:

            // We can get here though usually a positive status code is

            // a success. We get here if there is rfc2617 credential data

            // loaded and we're supposed to go around again. See if any

            // rfc2617 credential present and if there, assume it got

            // loaded in FetchHTTP on expectation that we're to go around

            // again. If no rfc2617 loaded, we should not be here.

            boolean loaded = curi.hasRfc2617Credential();

            if (!loaded && logger.isLoggable(Level.FINE)) {

                logger.fine("Have 401 but no creds loaded " + curi);

            }

            return loaded;

        case S_DEFERRED:

        case S_CONNECT_FAILED:

        case S_CONNECT_LOST:

        case S_DOMAIN_UNRESOLVABLE:

            // these are all worth a retry

            // TODO: consider if any others (S_TIMEOUT in some cases?) deserve

            // retry

            return true;

        case S_UNATTEMPTED:

            if(curi.includesRetireDirective()) {

                return true;

            } // otherwise, fall-through: no status is an error without queue-directive

        default:

            return false;

        }

    }

 long retryDelayFor(CrawlURI curi)方法为设置WorkQueue wq延时时间 

/**

     * Return a suitable value to wait before retrying the given URI.

     * 

     * @param curi

     *            CrawlURI to be retried

     * @return millisecond delay before retry

     */

    protected long retryDelayFor(CrawlURI curi) {

        int status = curi.getFetchStatus();

        return (status == S_CONNECT_FAILED || status == S_CONNECT_LOST ||

                status == S_DOMAIN_UNRESOLVABLE)? getRetryDelaySeconds() : 0;

                // no delay for most

    }

getRetryDelaySeconds()的值默认为900秒（15分）

后面为将CrawlURI curi对象更新到WorkQueue wq，最后 重置WorkQueue wq的队列归属（放入不再激活的队列或休眠队列或reenqueueQueue(wq)进一步处理）

/**

     * 重置WorkQueue wq的队列归属

     * Send an active queue to its next state, based on the supplied 

     * parameters.

     * 

     * @param wq

     * @param forceRetire

     * @param now

     * @param delay_ms

     */

    protected void handleQueue(WorkQueue wq, boolean forceRetire, long now, long delay_ms) {

        

        inProcessQueues.remove(wq);

        if(forceRetire) {

            retireQueue(wq);

        } else if (delay_ms > 0) {

            snoozeQueue(wq, now, delay_ms);

        } else {

            //Enqueue the given queue to either readyClassQueues or inactiveQueues,as appropriate

            reenqueueQueue(wq);

        }

    }

接下来看后面的方法wq.dequeue(this,curi)为将CrawlURI curi对象从WorkQueue wq中移除

最后重置WorkQueue wq的队列归属

 long delay_ms = curi.getPolitenessDelay(); 

 handleQueue(wq,curi.includesRetireDirective(),now,delay_ms);

handleQueue方法在上面部分

---------------------------------------------------------------------------

本系列Heritrix 3.1.0 源码解析系本人原创

转载请注明出处 博客园 刺猬的温驯

本文链接 http://www.cnblogs.com/chenying99/archive/2013/04/21/3033520.html