nicokani
nicokani

scrapy采集dmoz网站Home目录下的信息

一、实验背景

此次实验要求我们爬取DMOZ下的Home目录(http://www.dmoztools.net/Home/)的所有子目录.Home子目录下图所示。

Home子目录

二、实验目标

我们需要爬取Home目录下的所有的网站信息，爬取时主要爬取以下内容：
①爬取site时的当前路径（category_path）
②目录的目录名（cat_name）、链接即内链（cat_url）
③site的标题（site_title）、网址（site_url）、简介（site_desc）

三、网站分析

dmoz的网页结构总的来说是一棵树，其每一节点的子节点较多。
接下来先分析需要得到的Xpath：
1.<-----categories(目录)----->

subcategories——目录的div块

这里有两个目录的div块，尽管它们的id属性值不同，但是他们的class属性值是相同的，因此我们的目录部分的Xpath就写成： 

'//div[@class="cat-list results leaf-nodes"]/div[@class="cat-item"]'
目录部分


进一步细化到目录的链接（cat_url）和名称（cat_name）：

cat_url 的Xpath = '//div[@class="cat-list results leaf-nodes"]/div[@class="cat-item"]/a/@href'
cat_name 的Xpath = '//div[@class="cat-list results leaf-nodes"]/div[@class="cat-item"]/a/div/text()

2.<-----sites(网站)----->

sites(网站)部分

site的标题（site_title）、网址（site_url）、简介（site_desc）的Xpath分析出来如下： 

site_title的Xpath = '//div[@class="site-item "]/div[@class="title-and-desc"]/a/div[@class="site-title"]/text()'
site_url的Xpath = '/div[@class="site-item "]/div[@class="title-and-desc"]/a/@href'
site_desc的Xpath = '/div[@class="site-item "]/div[@class="title-and-desc"]/div[@class="site-descr "]/text()'

三、创建编辑项目——home

1.新建项目
scrapy startproject home
2.在Pycharm中导入项目进行编辑
导入项目
3.编辑items.py文件
import scrapy
class HomeItem(scrapy.Item):
    category_path = scrapy.Field()
    categories = scrapy.Field()
    cat_url = scrapy.Field()
    cat_name = scrapy.Field()
    sites = scrapy.Field()
    site_url =  scrapy.Field()
    site_desc = scrapy.Field()
    site_title = scrapy.Field()
    pass
4.在spider文件夹下创建homeSpider.py文件，并编辑
from home.items import *
from scrapy.spiders import CrawlSpider, Rule    #CrawlSpider与Spider类的最大不同是多了一个rules参数，通过rule可以定义提取动作。
from scrapy.linkextractors import LinkExtractor

class HomeSpider(CrawlSpider):
    name = "home"
    start_urls = ['http://www.dmoztools.net/Home/']
    rules = (
        Rule(LinkExtractor(allow=(r'http://www.dmoztools.net/Home/.*'),
                           restrict_xpaths=('//div[@class="cat-list results leaf-nodes"]/div[@class="cat-item"]')),
             callback="parse_item", follow=True),
    )
 # 利用rule的Rule类定义爬取规则；LinkExtractor，用于定义需要提取的链接；callback是当LinkExtractor获取到链接时参数所指定的值作为回调函数； follow指定了根据该规则从response提取的链接是否需要跟进。当callback为None,默认值为true。
 #其中allow后面是提取匹配 'https://curlie.org/Home/.*'；使用restrict_xpaths参数限制，将对url进行选择性抓取,抓取规定位置符合allow的url
   
 def parse_item(self, response):
        item = HomeItem()
        item['category_path'] = response.url.lstrip('http://www.dmoztools.net')   
 #python 的lstrip()方法，用于截掉字符串左边的空格或指定字符,在这里表示只取'http://www.dmoztools.net'右边的字符,用于提取爬取site时的当前路径（category_path）

        item['categories'] = []
        for cat in response.xpath('//div[@class="cat-list results leaf-nodes"]/div[@class="cat-item"]'):
            child_cat = {}
            child_cat["cat_url"] = cat.xpath('a/@href').extract()
            child_cat['cat_name'] = cat.xpath('a/div/text()').extract()
            child_cat['cat_name'] = [item.replace("\r\n", "") for item in child_cat['cat_name']]
            child_cat['cat_name'] = [item.replace(" ", "") for item in child_cat['cat_name']]
            item['categories'].append(child_cat)    
 #python list append()方法用于在列表末尾添加新的对象 

        item['sites'] = []
        for site in response.xpath('//div[@class="site-item "]/div[@class="title-and-desc"]'):
            result_site = {}
            result_site['site_title'] = site.xpath('a/div[@class="site-title"]/text()').extract_first().strip()
            result_site['site_url'] = site.xpath('a/@href').extract_first().strip()
            result_site['site_desc'] = site.xpath('div[@class="site-descr "]/text()').extract_first().strip()
            item['sites'].append(result_site)
        yield item
5.编辑begin.py文件

在home和home.cfg的同级目录下新建一个begin.py文件,然后进行编辑：

from scrapy import cmdline
cmdline.execute('scrapy crawl home -o home.jl'.split())

如果是在云服务器或者通过命令行启动爬虫的就不需要做这一步，只需要在对应的项目目录下输入以下内容即可，并且可以省去后面的一个步骤

scrapy crawl home -o home.jl
6.运行前配置Pycharm
①

②设置路径

③点击运行

四、爬取结果

给出部分吧，数据挺多的

{"category_path": "Home/Apartment_Living/", "categories": [{"cat_url": ["/Business/Real_Estate/Residential/Rentals/"], "cat_name": ["", "ApartmentLocators", ""]}, {"cat_url": ["/Home/Apartment_Living/Roommates/"], "cat_name": ["", "Roommates", ""]}, {"cat_url": ["/Society/Issues/Housing/Tenant_Rights/"], "cat_name": ["", "TenantRights", ""]}], "sites": [{"site_title": "Apartment Living", "site_url": "http://ths.gardenweb.com/forums/apt/", "site_desc": "A discussion forum for those living in apartments, condominiums and co-ops. Topics range from roommate problems, maintenance issues, leases to decorating."}, {"site_title": "Apartment Therapy", "site_url": "http://www.apartmenttherapy.com/", "site_desc": "Offers articles and advice on apartment living."}, {"site_title": "Good for Apartment Life", "site_url": "http://www.dogbreedinfo.com/apartment.htm", "site_desc": "Provides a list of dog breeds suited to apartment living. Each breed is linked to a detailed description."}, {"site_title": "Rental Decorating Digest", "site_url": "http://www.rentaldecorating.com/", "site_desc": "Tips and tricks for decorating for the renter."}, {"site_title": "TheDollarStretcher.com: How to Furnish a Studio Apartment", "site_url": "http://www.stretcher.com/stories/970929b.cfm", "site_desc": "Studio apartment solutions for \"hiding and otherwise disguising belongings that needed to be stored in a small space.\""}]}
{"category_path": "Home/Weblogs/", "categories": [{"cat_url": ["/Home/Family/Adoption/Weblogs/"], "cat_name": ["", "Adoption", ""]}, {"cat_url": ["/Home/Gardening/Bonsai_and_Suiseki/Bonsai/Weblogs/"], "cat_name": ["", "Bonsai", ""]}, {"cat_url": ["/Home/Cooking/Weblogs/"], "cat_name": ["", "Cooking", ""]}, {"cat_url": ["/Home/Consumer_Information/Electronics/Weblogs/"], "cat_name": ["", "Electronics", ""]}, {"cat_url": ["/Home/Consumer_Information/Food_and_Drink/Weblogs/"], "cat_name": ["", "FoodandDrink", ""]}, {"cat_url": ["/Home/Personal_Finance/Money_Management/Weblogs/"], "cat_name": ["", "MoneyManagement", ""]}, {"cat_url": ["/Home/Family/Parenting/Mothers/Weblogs/"], "cat_name": ["", "Mothers", ""]}, {"cat_url": ["/Home/Family/Parenting/Fathers/Stay_at_Home_Fathers/Weblogs/"], "cat_name": ["", "StayatHomeFathers", ""]}], "sites": []}
{"category_path": "Home/Software/", "categories": [{"cat_url": ["/Home/Family/Software/"], "cat_name": ["", "Family", ""]}, {"cat_url": ["/Home/Gardening/Software/"], "cat_name": ["", "Gardening", ""]}, {"cat_url": ["/Society/Genealogy/Software/"], "cat_name": ["", "Genealogy", ""]}, {"cat_url": ["/Computers/Home_Automation/Software/"], "cat_name": ["", "HomeAutomation", ""]}, {"cat_url": ["/Home/Personal_Finance/Software/"], "cat_name": ["", "PersonalFinance", ""]}, {"cat_url": ["/Home/Cooking/Recipe_Management/"], "cat_name": ["", "RecipeManagement", ""]}], "sites": [{"site_title": "Agilaire", "site_url": "http://www.agilairecorp.com/", "site_desc": "Environmental data management solutions. Products and support."}, {"site_title": "Chief Architect Inc: Home Designer Software", "site_url": "http://www.homedesignersoftware.com/", "site_desc": "Software package for home remodelling, interior design, decks and landscaping creation. Company info, products list, shop and support."}, {"site_title": "Chore Wars", "site_url": "http://www.chorewars.com/", "site_desc": "Browser-based system loosely based on D&D that allows household members to claim experience points for doing tasks.  Monsters and treasure may be optionally defined with each chore."}, {"site_title": "Kopy Kake", "site_url": "http://www.kopykake.com/", "site_desc": "Cake decorating software for professional bakers and hobbyists."}, {"site_title": "Let's Clean Up", "site_url": "http://www.lets-clean-up.com/", "site_desc": "Software to help the family and small businesses organize cleaning chores and maintenance activities."}, {"site_title": "Punch Software", "site_url": "http://www.punchsoftware.com/", "site_desc": "Offers 3D home design suite for professional home planning with real model technology. Plan a dream house with this architecture design and 3D landscape software."}]}
{"category_path": "Home/News_and_Media/", "categories": [{"cat_url": ["/Home/News_and_Media/Radio_Programs/"], "cat_name": ["", "RadioPrograms", ""]}, {"cat_url": ["/Home/News_and_Media/Television/"], "cat_name": ["", "Television", ""]}, {"cat_url": ["/Home/Homemaking/Frugality/Publications/"], "cat_name": ["", "BudgetLiving", ""]}, {"cat_url": ["/Home/Consumer_Information/News_and_Media/"], "cat_name": ["", "ConsumerInformation", ""]}, {"cat_url": ["/Home/Cooking/Magazines_and_E-zines/"], "cat_name": ["", "Cooking", ""]}, {"cat_url": ["/Home/Family/Publications/"], "cat_name": ["", "Families", ""]}, {"cat_url": ["/Home/Gardening/News_and_Media/"], "cat_name": ["", "Gardens", ""]}, {"cat_url": ["/Home/Home_Improvement/News_and_Media/"], "cat_name": ["", "HomeImprovement", ""]}, {"cat_url": ["/Home/Family/Parenting/Magazines_and_E-zines/"], "cat_name": ["", "Parenting", ""]}, {"cat_url": ["/Recreation/Pets/News_and_Media/"], "cat_name": ["", "Pets", ""]}, {"cat_url": ["/Business/Real_Estate/News_and_Media/"], "cat_name": ["", "RealEstate", ""]}], "sites": [{"site_title": "Better Homes & Gardens", "site_url": "http://www.bhg.com/", "site_desc": "Ideas and improvement projects for your home and garden plus recipes and entertaining ideas."}, {"site_title": "Better Homes and Gardens Australia", "site_url": "http://www.bhg.com.au/", "site_desc": "Offers items from both the weekly television show and monthly magazine."}, {"site_title": "Carolina Home and Garden", "site_url": "http://www.carolinahg.com/", "site_desc": "Features articles on the arts, gardens, local people, interior decor. Also includes calendar and resources."}, {"site_title": "Coastal Living Magazine", "site_url": "http://www.coastalliving.com/", "site_desc": "Features articles on homes, decorating, travel, food and living in coastal communities."}, {"site_title": "Country Living Magazine", "site_url": "http://www.countryliving.com/", "site_desc": "Features include home decorating, recipes and antiques and collectibles."}, {"site_title": "Homes and Gardens Magazine", "site_url": "http://www.housetohome.co.uk/homesandgardens", "site_desc": "Features beautiful houses and gardens, decorating ideas, designers and decorators, unusual shops. Includes subscription information, and a range of articles online."}, {"site_title": "House to Home UK", "site_url": "http://www.housetohome.co.uk/", "site_desc": "Provides a look inside British residences, ideas for rooms, videos and guest bloggers."}, {"site_title": "Howdini", "site_url": "http://www.howdini.com/", "site_desc": "An e-zine with video tips on how to do a variety of household tasks such as cooking or caring for a new baby. Includes 'life hacks'."}, {"site_title": "Martha Stewart Living", "site_url": "http://www.marthastewart.com/", "site_desc": "Official site, with links to personal information, Martha's Scrapbook, television highlights, radio guide, virtual studio, recipes, live chat."}, {"site_title": "Mother Earth Living", "site_url": "http://www.motherearthliving.com/", "site_desc": "Offers today's health-conscious, environmentally concerned homeowners information needed to practice earth-inspired living."}, {"site_title": "Mother Earth News", "site_url": "http://www.motherearthnews.com/", "site_desc": "Features and articles covering sustainable, self-reliant living. Topics include building, gardening, homesteading, do-it-yourself, kitchen, energy and health."}, {"site_title": "Pioneer Thinking", "site_url": "http://www.pioneerthinking.com/", "site_desc": "Features articles on gardening, cooking, crafts, personal finance, and beauty."}, {"site_title": "Real Simple", "site_url": "http://www.realsimple.com/", "site_desc": "Magazine about simplifying your life. Includes home solutions, meals, special features."}, {"site_title": "SFGate.com: Home and Garden", "site_url": "http://www.sfgate.com/homeandgarden/", "site_desc": "Daily features and articles from the San Francisco Chronicle covering decorating, entertaining, gardening, and family life."}, {"site_title": "Southern Living", "site_url": "http://www.southernliving.com/", "site_desc": "Features about fine interiors, gardens, design, antiques, travel, events, and the arts."}, {"site_title": "Style at Home", "site_url": "http://www.styleathome.com/", "site_desc": "Covers interior decoration and design tips, decorating on a budget, recipes and buying guides."}, {"site_title": "Sunset Magazine and Books", "site_url": "http://www.sunset.com/", "site_desc": "News and feature articles on Western living."}, {"site_title": "The Washington Post: Home & Garden", "site_url": "http://www.washingtonpost.com/wp-dyn/home/", "site_desc": "Daily features and articles covering decorating, home improvement, gardening, pets and family life."}]}
{"category_path": "Home/Urban_Living/", "categories": [{"cat_url": ["/Home/Apartment_Living/"], "cat_name": ["", "ApartmentLiving", ""]}, {"cat_url": ["/Science/Earth_Sciences/Atmospheric_Sciences/Climatology/Urban/"], "cat_name": ["", "Climate", ""]}, {"cat_url": ["/Arts/Photography/Photographers/Urban/Fine_Art/"], "cat_name": ["", "PhotographicExhibits", ""]}, {"cat_url": ["/Society/Subcultures/"], "cat_name": ["", "Subcultures", ""]}], "sites": [{"site_title": "City Noise", "site_url": "http://www.citynoise.org/", "site_desc": "A public photoblog where people with a love for the urban form, modern world, or a general appreciation of their environment gather to post stories, narratives and often upload photos of their favourite cities, hometowns, travels, or current locations."}, {"site_title": "CityCulture", "site_url": "http://cityculture.org/", "site_desc": "World city reviews and a free test which suggests cities and regions around the world that fit your personality."}, {"site_title": "Flickr: Urban Negative", "site_url": "http://www.flickr.com/groups/urban_negative/", "site_desc": "Photos of the negative object, material and people in urban lifestyles."}, {"site_title": "Pedestrian", "site_url": "http://www.turbulence.org/Works/pedestrian/pedestrian2.html", "site_desc": "An artistic work about urban life."}, {"site_title": "Self Sufficientish", "site_url": "http://www.selfsufficientish.com/", "site_desc": "Information on growing plants, wild food recipes, and alternatives to lead a low impact urban life."}, {"site_title": "Urbansome", "site_url": "http://urbansome.com/", "site_desc": "City travel information, tips, deals, news, photography and lots of entertainment brought to you by urban enthusiasts."}]}
{"category_path": "Home/Rural_Living/", "categories": [{"cat_url": ["/Science/Social_Sciences/Economics/Agricultural_and_Rural_Economics/"], "cat_name": ["", "AgriculturalandResourceEconomics", ""]}, {"cat_url": ["/Home/Cooking/"], "cat_name": ["", "Cooking", ""]}, {"cat_url": ["/Society/People/Cowboys/"], "cat_name": ["", "Cowboys", ""]}, {"cat_url": ["/Reference/Education/K_through_12/Rural_Issues/"], "cat_name": ["", "Education", ""]}, {"cat_url": ["/Business/Agriculture_and_Forestry/Farm_Real_Estate/"], "cat_name": ["", "FarmRealEstate", ""]}, {"cat_url": ["/Health/Public_Health_and_Safety/Rural_Health/"], "cat_name": ["", "Health", ""]}, {"cat_url": ["/Home/Rural_Living/Hobby_Farms/"], "cat_name": ["", "HobbyFarms", ""]}, {"cat_url": ["/Reference/Education/K_through_12/Home_Schooling/"], "cat_name": ["", "HomeSchooling", ""]}, {"cat_url": ["/Home/Rural_Living/Homesteading/"], "cat_name": ["", "Homesteadi­ng", ""]}, {"cat_url": ["/Science/Environment/Water_Resources/Wastewater/Household_Wastewater_Management/"], "cat_name": ["", "HouseholdWastewaterManagement", ""]}, {"cat_url": ["/Society/Lifestyle_Choices/Intentional_Communities/"], "cat_name": ["", "IntentionalCommunities", ""]}, {"cat_url": ["/Home/Rural_Living/Personal_Pages/"], "cat_name": ["", "PersonalPages", ""]}, {"cat_url": ["/Science/Technology/Energy/Renewable/"], "cat_name": ["", "RenewableEnergy", ""]}, {"cat_url": ["/Science/Social_Sciences/Sociology/Rural_Sociology/"], "cat_name": ["", "RuralSociology", ""]}, {"cat_url": ["/Science/Agriculture/Sustainable_Agriculture/"], "cat_name": ["", "SustainableAgriculture", ""]}, {"cat_url": ["/Business/Construction_and_Maintenance/Building_Types/Sustainable_Architecture/"], "cat_name": ["", "SustainableArchitecture", ""]}, {"cat_url": ["/Society/Lifestyle_Choices/Voluntary_Simplicity/"], "cat_name": ["", "VoluntarySimplicity", ""]}], "sites": [{"site_title": "Countryside Magazine", "site_url": "http://countrysidenetwork.com/daily/", "site_desc": "Selected articles from the printed magazine for readers seeking voluntary simplicity and greater self-reliance with emphasis on home food production. Gardening, cooking, food preservation, and livestock. Has an active forum."}, {"site_title": "DTN: The Progressive Farmer", "site_url": "https://www.dtnpf.com/agriculture/web/ag/home", "site_desc": "Forums and news for farmers and people involved in rural issues."}, {"site_title": "Internet Hay Exchange", "site_url": "http://www.hayexchange.com/", "site_desc": "Free International hay listing service for the US and Canada."}, {"site_title": "Kountry Life", "site_url": "http://www.kountrylife.com/index.htm", "site_desc": "An interactive country and rural living site with discussion forums, photo gallery, articles, how-to information, humor, sounds, and recipes."}, {"site_title": "Renewing the Countryside", "site_url": "http://www.renewingthecountryside.org/", "site_desc": "Aims to strengthen rural areas by highlighting the initiatives and projects of rural communities, farmers, artists, entrepreneurs, educators, and activists."}, {"site_title": "Rural Living Canada", "site_url": "http://rurallivingcanada.4t.com/", "site_desc": "A concise directory of Canadian non-urban lifestyle information, news and websites."}, {"site_title": "Soil and Health Library", "site_url": "http://www.soilandhealth.org/", "site_desc": "A free how-to and encouragement resource for self-supporters with little cash, for non-domineering environmentalists, and folks frustrated with urbanity."}, {"site_title": "The Urban Rancher", "site_url": "http://theurbanrancher.tamu.edu/", "site_desc": "The Texas A&M University site dedicated to improving rural living with information on natural resources, rural life, and the urban-rural interface."}]}
{"category_path": "Home/Personal_Organization/", "categories": [{"cat_url": ["/Home/Personal_Organization/Consultants/"], "cat_name": ["", "Consultant­s", ""]}, {"cat_url": ["/Business/Business_Services/Office_Services/Secretarial_Services_and_Virtual_Assistants/"], "cat_name": ["", "VirtualAssistants", ""]}, {"cat_url": ["/Health/Services/Health_Records_Services/"], "cat_name": ["", "HealthRecordsServices", ""]}, {"cat_url": ["/Society/Crime/Theft/Identity_Theft/"], "cat_name": ["", "IdentityTheft", ""]}, {"cat_url": ["/Shopping/Home_and_Garden/Furniture/Storage/"], "cat_name": ["", "StorageFurnitureShopping", ""]}, {"cat_url": ["/Business/Management/Education_and_Training/Time_Management/"], "cat_name": ["", "TimeManagement", ""]}, {"cat_url": ["/Reference/Knowledge_Management/Information_Overload/"], "cat_name": ["", "InformationOverload", ""]}, {"cat_url": ["/Computers/Software/Operating_Systems/Microsoft_Windows/Software/Shareware/Home_and_Hobby/Financial%2C_Insurance_and_Home_Inventory/"], "cat_name": ["", "InventorySoftware", ""]}, {"cat_url": ["/Computers/Mobile_Computing/"], "cat_name": ["", "MobileComputing", ""]}, {"cat_url": ["/Computers/Software/Freeware/Personal_Information_Managers/"], "cat_name": ["", "PersonalInformationManagersFreeware", ""]}, {"cat_url": ["/Computers/Software/Operating_Systems/Microsoft_Windows/Software/Shareware/Personal_Information_Managers/"], "cat_name": ["", "PersonalInformationManagersShareware", ""]}, {"cat_url": ["/Computers/Internet/On_the_Web/Web_Applications/Personal_Information_Managers/"], "cat_name": ["", "PersonalInformationManagersWebApplications", ""]}], "sites": [{"site_title": "43 Folders", "site_url": "http://www.43folders.com/", "site_desc": "Aids to living including (but not limited to) productivity and time management tips, Mac OS X programs and technologies, ideas about modest ways to improve a person's life and reduce stress, and cool or helpful shortcuts that makes life a bit easier."}, {"site_title": "52 Projects", "site_url": "http://www.52projects.com/", "site_desc": "Motivating people to work on whatever projects they have long wanted or needed to do."}, {"site_title": "ABCs of Life Skills", "site_url": "http://lifeskills.endlex.com/", "site_desc": "Articles and references about organizing a successful life. Divided into general knowledge, money, work, family, health, and communication."}, {"site_title": "Checklists.com", "site_url": "http://www.checklists.com/atoz.html", "site_desc": "Free checklists on a large number of activities."}, {"site_title": "Clutterbug Network", "site_url": "http://www.clutterbug.net/", "site_desc": "Organizer directory, free newsletter."}, {"site_title": "Creative Homemaking - Organize", "site_url": "http://www.creativehomemaking.com/organize_1.shtml", "site_desc": "Offers tips for home, family, clutter control, holidays, cooking and time management. Includes a newsletter and check lists."}, {"site_title": "Flylady.net", "site_url": "http://www.flylady.net/", "site_desc": "Offers a system for organizing and managing a home, based on the concept of daily routines and a focus on small, time- and space-limited tasks. Provides resources, tips and newsletter."}, {"site_title": "Get Organized Now", "site_url": "http://www.getorganizednow.com/", "site_desc": "Offers tools, ideas and articles. Features monthly checklists, a discussion forum, e-courses and a newsletter."}, {"site_title": "I Need More Time", "site_url": "http://ineedmoretime.com/", "site_desc": "Free organizing tips, ideas and articles.  Sells additional tips as an ebook."}, {"site_title": "Lifehack.org", "site_url": "http://www.lifehack.org/", "site_desc": "Pointers on productivity, getting things done and lifehacks."}, {"site_title": "List Organizer", "site_url": "http://listorganizer.com/", "site_desc": "Offers planning lists with time management instructions for home, personal, travel, budgets, children, pets. Includes a newsletter."}, {"site_title": "Messies Anonymous", "site_url": "http://www.messies.com/", "site_desc": "Dedicated to bringing harmony in the home through understanding and aiding the messie mindset. Provides resources, FAQ and newsletter."}, {"site_title": "National Association of Professional Organizers (NAPO)", "site_url": "http://www.napo.net/", "site_desc": "Non-profit educational association whose members include organizing consultants, speakers, trainers, authors, and manufacturers of organizing products. Includes membership information, events, chapters, FAQ and a newsletter."}, {"site_title": "OrganizeTips", "site_url": "http://www.organizetips.com/", "site_desc": "Tips on organizing daily life.  Free planners, organizers, free software for home, office, wedding, moving, pregnancy, holiday and budget."}, {"site_title": "Printable Checklists", "site_url": "http://www.printablechecklists.com/", "site_desc": "Printable charts and checklists on topics such as parenting, children and special occasions. Includes a newsletter."}, {"site_title": "Professional Organizer Academy", "site_url": "http://professionalorganizeracademy.com/", "site_desc": "Online training academy for professional organizers."}, {"site_title": "Professional Organizers Web Ring", "site_url": "http://www.organizerswebring.com/", "site_desc": "Works to promote the field and to provide information on products and services. Includes events and FAQ."}]}
{"category_path": "Home/Personal_Finance/", "categories": [{"cat_url": ["/Reference/Education/Colleges_and_Universities/Financial_Aid/"], "cat_name": ["", "CollegeFinancialAid", ""]}, {"cat_url": ["/Home/Personal_Finance/Money_Management/Debt_and_Bankruptcy/"], "cat_name": ["", "DebtandBankruptcy", ""]}, {"cat_url": ["/Society/Issues/Violence_and_Abuse/Elder/Financial_Abuse/"], "cat_name": ["", "ElderFraudIssues", ""]}, {"cat_url": ["/Society/Law/Legal_Information/Estate_Planning_and_Administration/"], "cat_name": ["", "EstatePlanning", ""]}, {"cat_url": ["/Society/Crime/Theft/Identity_Theft/"], "cat_name": ["", "IdentifyTheftIssues", ""]}, {"cat_url": ["/Home/Personal_Finance/Insurance/"], "cat_name": ["", "Insurance", ""]}, {"cat_url": ["/Home/Personal_Finance/Investing/"], "cat_name": ["", "Investing", ""]}, {"cat_url": ["/Home/Personal_Finance/Money_Management/Loans/"], "cat_name": ["", "Loans", ""]}, {"cat_url": ["/Home/Personal_Finance/Money_Management/"], "cat_name": ["", "MoneyManagement", ""]}, {"cat_url": ["/Home/Personal_Finance/Money_Management/Loans/Home/"], "cat_name": ["", "Mortgages", ""]}, {"cat_url": ["/Home/Personal_Finance/Philanthropy/"], "cat_name": ["", "Philanthro­py", ""]}, {"cat_url": ["/Home/Personal_Finance/Retirement/"], "cat_name": ["", "Retirement", ""]}, {"cat_url": ["/Home/Personal_Finance/Software/"], "cat_name": ["", "Software", ""]}, {"cat_url": ["/Home/Personal_Finance/Tax_Preparation/"], "cat_name": ["", "TaxPreparatio­n", ""]}, {"cat_url": ["/Home/Personal_Finance/Unclaimed_Money/"], "cat_name": ["", "UnclaimedMoney", ""]}], "sites": [{"site_title": "20 Something Finance", "site_url": "http://20somethingfinance.com/", "site_desc": "Articles focused on helping young people manage their money."}, {"site_title": "AARP - Money and Work", "site_url": "http://www.aarp.org/money/", "site_desc": "Discussion of money matters in considerable depth, especially those related to people who have retired or are planning to in the near future."}, {"site_title": "About.com: Financial Planning", "site_url": "http://financialplan.about.com/", "site_desc": "Information on personal financial planning, including budgeting, savings, investing, retirement, insurance, and taxes."}, {"site_title": "American Savings Education Council", "site_url": "http://www.asec.org/", "site_desc": "A coalition of government and industry institutions to educate people on all aspects of personal finance and wealth development, including credit management, college savings, home purchase, and retirement planning."}, {"site_title": "Bankrate.com", "site_url": "http://www.bankrate.com/", "site_desc": "An online publication that provides consumers with  financial data, research and editorial information on       non-investment financial products."}, {"site_title": "CCH Financial Planning Toolkit", "site_url": "http://www.finance.cch.com/", "site_desc": "Information to manage one's personal finances, including investments, insurance, risk and asset management strategies, and tax, retirement and estate planning."}, {"site_title": "CNBC", "site_url": "http://www.cnbc.com/", "site_desc": "Headline news, articles, reports, stocks and quotes, message boards, and a stock ticker."}, {"site_title": "CNN/Money", "site_url": "http://money.cnn.com/", "site_desc": "Combines practical personal finance advice, calculators and investing tips with business news, stock quotes, and financial market coverage from the editors of CNN and Money Magazine."}, {"site_title": "ConsumerReports.org: Money", "site_url": "http://www.consumerreports.org/cro/money/index.htm", "site_desc": "Information about adjustable rate mortgage, investment tools, and personal finance tips."}, {"site_title": "Federal Reserve Board: Consumer Information", "site_url": "http://www.federalreserve.gov/consumers.htm", "site_desc": "Centralized home for articles giving advice and warnings about financial topics, products, and scams."}, {"site_title": "Forbes.com - Personal Finance", "site_url": "http://www.forbes.com/finance/", "site_desc": "Financial information previously published in the print version of Forbes."}, {"site_title": "I Retire Early", "site_url": "http://www.iretireearly.com/", "site_desc": "How to save money, advance your career and manage your finances."}, {"site_title": "Inflation Calculator", "site_url": "http://www.westegg.com/inflation/", "site_desc": "Adjusts a given amount of money for inflation, according to the Consumer Price Index."}, {"site_title": "Institute of Consumer Financial Education", "site_url": "http://www.financial-education-icfe.org/", "site_desc": "Offers financial education for all age groups with a special section devoted to teaching children about money."}, {"site_title": "International Foundation for Retirement Education (InFRE)", "site_url": "http://www.infre.org/", "site_desc": "A not-for-profit educational foundation dedicated to empowering working Americans with the motivation and capability to save and plan for a successful retirement."}, {"site_title": "Joe Taxpayer", "site_url": "http://www.joetaxpayer.com/", "site_desc": "Blog covering nearly all of the personal finance topics, with an emphasis on taxes."}, {"site_title": "The JumpStart Coalition for Personal Financial Literacy", "site_url": "http://www.jumpstart.org/", "site_desc": "Purpose is to evaluate the financial literacy of young adults; develop, disseminate, and encourage the use of guidelines for grades K-12; and promote the teaching of personal finance."}, {"site_title": "Kiplinger Online", "site_url": "http://www.kiplinger.com/", "site_desc": "Investing, personal finance, calculators and financial advice."}, {"site_title": "Marketplace", "site_url": "http://www.marketplace.org/", "site_desc": "Public radio business and economic news and commentary."}, {"site_title": "MarketWatch.com: Personal Finance", "site_url": "http://www.marketwatch.com/personal-finance", "site_desc": "Tips and stories for managing your personal finances."}, {"site_title": "Money Instructor", "site_url": "http://www.moneyinstructor.com/", "site_desc": "Tools and information to help   teach money and money management, business, the economy, and investing."}, {"site_title": "Money Talks News", "site_url": "http://www.moneytalksnews.com/", "site_desc": "Tips and advice to help you spend less and save more."}, {"site_title": "The Money Ways", "site_url": "http://www.themoneyways.com/", "site_desc": "Advice on money management, saving money, and budgeting."}, {"site_title": "The Motley Fool", "site_url": "http://www.fool.com/", "site_desc": "Investing information and an enjoyably useful site. Updated hourly."}, {"site_title": "MsMoney.com", "site_url": "http://www.msmoney.com/", "site_desc": "Resource for women to learn about financial planning, personal finance and investing."}, {"site_title": "Mymoney.gov - Financial Literacy Education Commission", "site_url": "http://www.mymoney.gov/", "site_desc": "Starting point for information intended by the US government to help improve the financial literacy and education of persons in the United States."}, {"site_title": "The New York Times - Your Money", "site_url": "http://www.nytimes.com/pages/business/yourmoney/", "site_desc": "Articles and features on investing, pensions and retirement plans, mortgage rates, mutual funds, the stock market, bonds and notes.  Also has company research, earnings reports and market insight."}, {"site_title": "Wealth Informatics", "site_url": "http://www.wealthinformatics.com/", "site_desc": "Blog discussing everyday finance topics."}, {"site_title": "Yahoo! Finance", "site_url": "http://finance.yahoo.com/", "site_desc": "Personal finance, investing tips and news."}, {"site_title": "Your Money Page", "site_url": "http://www.yourmoneypage.com/", "site_desc": "Online calculators for financial planning and personal finance."}]}
{"category_path": "Home/Moving_and_Relocating/", "categories": [{"cat_url": ["/Home/Consumer_Information/Home_and_Family/Moving_and_Relocating/"], "cat_name": ["", "ConsumerInformation", ""]}, {"cat_url": ["/Business/Business_Services/Corporate_Relocation/"], "cat_name": ["", "CorporateRelocation", ""]}, {"cat_url": ["/Home/Moving_and_Relocating/Moving/"], "cat_name": ["", "Moving", ""]}, {"cat_url": ["/Home/Moving_and_Relocating/Publications/"], "cat_name": ["", "Publicatio­ns", ""]}, {"cat_url": ["/Business/Real_Estate/Agents_and_Agencies/"], "cat_name": ["", "RealEstateAgencies", ""]}, {"cat_url": ["/Business/Real_Estate/By_Region/"], "cat_name": ["", "RealEstatebyCountry", ""]}, {"cat_url": ["/Business/Real_Estate/Residential/Rentals/"], "cat_name": ["", "ApartmentsandRentals", ""]}, {"cat_url": ["/Society/Gay%2C_Lesbian%2C_and_Bisexual/Home_and_Living/Moving_and_Relocating/"], "cat_name": ["", "Gay,Lesbian,andBisexual", ""]}, {"cat_url": ["/Home/Moving_and_Relocating/International_Relocation/"], "cat_name": ["", "Internatio­nalRelocation", ""]}, {"cat_url": ["/Home/Moving_and_Relocating/Military_Relocation/"], "cat_name": ["", "MilitaryRelocation", ""]}, {"cat_url": ["/Business/Real_Estate/Residential/Rentals/Students/"], "cat_name": ["", "StudentHousing", ""]}], "sites": []}
{"category_path": "Home/Homeowners/", "categories": [{"cat_url": ["/Home/Home_Improvement/Decorating/"], "cat_name": ["", "Decorating", ""]}, {"cat_url": ["/Home/Home_Improvement/Design_and_Construction/"], "cat_name": ["", "DesignandConstruction", ""]}, {"cat_url": ["/Home/Do-It-Yourself/"], "cat_name": ["", "Do-It-Yourself", ""]}, {"cat_url": ["/Home/Home_Improvement/Energy_Efficiency/"], "cat_name": ["", "EnergyEfficiency", ""]}, {"cat_url": ["/Society/Religion_and_Spirituality/Taoism/Feng_Shui/"], "cat_name": ["", "FengShui", ""]}, {"cat_url": ["/Home/Home_Improvement/Automation/"], "cat_name": ["", "HomeAutomation", ""]}, {"cat_url": ["/Home/Homeowners/Home_Buyers/"], "cat_name": ["", "HomeBuyers", ""]}, {"cat_url": ["/Home/Home_Improvement/"], "cat_name": ["", "HomeImprovement", ""]}, {"cat_url": ["/Home/Homeowners/Homeowner_Associations/"], "cat_name": ["", "HomeownerAssociatio­ns", ""]}, {"cat_url": ["/Business/Real_Estate/Residential/Cooperatives/"], "cat_name": ["", "HousingCooperatives", ""]}, {"cat_url": ["/Science/Environment/Air_Quality/Indoor_Air_Quality/"], "cat_name": ["", "IndoorAirQuality", ""]}, {"cat_url": ["/Home/Homeowners/Pest_Control/"], "cat_name": ["", "PestControl", ""]}, {"cat_url": ["/Shopping/Home_and_Garden/Outdoor_Structures/Playsets/"], "cat_name": ["", "Playhouses", ""]}, {"cat_url": ["/Home/Home_Improvement/Restoration/"], "cat_name": ["", "Restoration", ""]}, {"cat_url": ["/Home/Homeowners/Treehouses/"], "cat_name": ["", "Treehouses", ""]}], "sites": [{"site_title": "The Condominium Bluebook (Condominium Laws)", "site_url": "http://www.condobook.com/", "site_desc": "The complete guide to the operations of condominiums, planned developments and other common interest developments in California."}, {"site_title": "EPA.gov - Refrigerant-22 Phaseout", "site_url": "http://www.epa.gov/ozone/title6/phaseout/22phaseout.html", "site_desc": "EPA site with homeowner information for consideration when purchasing or repairing a residential system or heat pump."}, {"site_title": "Home Owners Information Center", "site_url": "http://www.ourfamilyplace.com/homeowner/", "site_desc": "Guide to remodeling, refinancing, household budgets and getting the most enjoyment from your home."}, {"site_title": "Homebuilding Pitfalls", "site_url": "http://www.homebuildingpitfalls.com/", "site_desc": "Aims to help consumers save time, money and stress by providing advice on how to properly building their own home"}, {"site_title": "Popular Mechanics: Home Improvement", "site_url": "http://www.popularmechanics.com/home_journal/home_improvement/", "site_desc": "Project information for home and garden. Furniture making,  gardening, home improvement, tools, homeowner's clinic, how it works section. Illustrated."}, {"site_title": "WSJ.com's Real Estate Journal", "site_url": "http://www.realestatejournal.com/", "site_desc": "A guide to buying, selling and maintaining a home."}]}
{"category_path": "Home/Homemaking/", "categories": [{"cat_url": ["/Home/Homemaking/Cleaning_and_Stains/"], "cat_name": ["", "CleaningandStains", ""]}, {"cat_url": ["/Home/Home_Improvement/Decorating/"], "cat_name": ["", "Decorating", ""]}, {"cat_url": ["/Home/Homemaking/Frugality/"], "cat_name": ["", "Frugality", ""]}, {"cat_url": ["/Health/Alternative/Non-Toxic_Living/"], "cat_name": ["", "Non-ToxicLiving", ""]}, {"cat_url": ["/Home/Family/Parenting/"], "cat_name": ["", "Parenting", ""]}, {"cat_url": ["/Home/Personal_Organization/"], "cat_name": ["", "PersonalOrganization", ""]}, {"cat_url": ["/Home/Homemaking/Christian/"], "cat_name": ["", "Christian", ""]}, {"cat_url": ["/Home/Homemaking/Celebrity_Homemakers/"], "cat_name": ["", "CelebrityHomemakers", ""]}, {"cat_url": ["/Home/Homemaking/News_and_Media/"], "cat_name": ["", "NewsandMedia", ""]}], "sites": [{"site_title": "About.com - Housekeeping", "site_url": "http://housekeeping.about.com/", "site_desc": "Offers cleaning articles, how-to's, and product reviews. Includes a newsletter and message board."}, {"site_title": "All About Home by Service Master", "site_url": "http://www.allabouthome.com/", "site_desc": "Offers advice on topics including seasonal issues and disaster preparedness. Features a virtual tour and measurement calculators."}, {"site_title": "AOL Living", "site_url": "http://living.aol.com/", "site_desc": "Features homemaking help, organizational tips, recipes, beauty advice, and decorating ideas."}, {"site_title": "Barefoot Lass's Hints & Tips", "site_url": "http://members.tripod.com/~Barefoot_Lass/", "site_desc": "Offers information on topics such as removing crayon marks from walls, finding the best hangover cure and alternative uses for cola. Includes awards and information about trigminal neuralgia."}, {"site_title": "Berkeley Parents Network - Advice About Household Management", "site_url": "http://parents.berkeley.edu/advice/household/index.html", "site_desc": "Offers advice on topics such as health and safety, household organization, cleaning and laundry. Includes subscription information."}, {"site_title": "Bob Allison's Ask Your Neighbor - Helpful Household Hints", "site_url": "http://www.askyourneighbor.com/hhints.htm", "site_desc": "Features tips shared on Bob Allison's Ask your Neighbor radio program. Includes instructions for making cleaning products."}, {"site_title": "Creative Homemaking", "site_url": "http://www.creativehomemaking.com/", "site_desc": "Offers organization, decorating, crafts, frugal living and parenting hints. Includes holiday ideas, recipes and a newsletter."}, {"site_title": "DontForgetTheMilk.com", "site_url": "http://www.dontforgetthemilk.com/", "site_desc": "Creates shopping lists sorted by store and price. Features a message board. Requires free registration."}, {"site_title": "eHow: Organize Your Closet", "site_url": "http://www.ehow.com/closet-organizing/", "site_desc": "Full-length article covers the process of organizing the items in a closet."}, {"site_title": "The F.U.N. Place - Families United on the Net", "site_url": "http://www.thefunplace.com/", "site_desc": "Offers home tips, recipes, crafts and parenting articles. Includes forums, chat and a newsletter."}, {"site_title": "Forums for the Chaotic Home", "site_url": "http://ths.gardenweb.com/forums/", "site_desc": "Offers discussion forums on topics such as the house, cooking, crafts and hobbies and the family. Includes information on meetings."}, {"site_title": "Hints and Things", "site_url": "http://www.hintsandthings.com/", "site_desc": "Offers advice that used to be passed down from generation to generation. Features competitions and a newsletter."}, {"site_title": "Hints from Heloise", "site_url": "http://www.heloise.com/", "site_desc": "Offers tips for the home, garden and travel."}, {"site_title": "Home Made Simple", "site_url": "http://www.homemadesimple.com/", "site_desc": "Includes features on home decorating, gardening, and organizing, with ideas to simplify, organize, beautify and inspire life."}, {"site_title": "Homemaking School for Children", "site_url": "http://theparentsite.com/parenting/homemakingschool.asp", "site_desc": "Discusses how to teach children lessons in house cleaning and responsibility. By Monica Resinger."}, {"site_title": "Household Hints by Myra L. Fitch", "site_url": "http://lonestar.texas.net/~fitch/hints/hints.html#stainlesssteel", "site_desc": "Offers advice on topics such as getting whites white and removing hard water marks on polished marble. Includes reactions from readers."}, {"site_title": "Joey Green's WackyUses.com", "site_url": "http://www.wackyuses.com/", "site_desc": "Offers little-known uses for well-known products. Includes histories and facts behind the products."}, {"site_title": "John's House", "site_url": "http://johnshouse.itgo.com/", "site_desc": "Offers advice on topics such as food storage, removing pet hair from clothing and preventing dust build-up on television screens. Includes author's profile."}, {"site_title": "The New Homemaker", "site_url": "http://www.thenewhomemaker.com/", "site_desc": "Offers advice and resources on topics including parenting, thriftiness, kitchen, family health, crafts, decorating, and organization. Features news and chat."}, {"site_title": "Old Fashioned Living", "site_url": "http://oldfashionedliving.com/", "site_desc": "Presents old-fashioned traditions for the modern family. Features a newsletter and discussion forum."}, {"site_title": "Organized Home", "site_url": "http://organizedhome.com/", "site_desc": "Offers articles on uncluttering the house, cutting mealtime chaos, streamlining storage and finding more time. Includes a newsletter."}, {"site_title": "Robbie's Kitchen - Household Tips & Tricks", "site_url": "http://www.robbiehaf.com/RobbiesKitchen/RobbiesHints.html", "site_desc": "Offers tips for cleaning, cooking, laundry, and home remedies. Features a message board."}, {"site_title": "Seeking Sources", "site_url": "http://www.seekingsources.com/", "site_desc": "Offers articles about cooking, home finance and the holidays. Topics include gifts from the kitchen, how to take a financial inventory and how to choose the right cookware."}, {"site_title": "Uses for Vinegar", "site_url": "http://www.angelfire.com/cantina/homemaking/vinegar.html", "site_desc": "Offers suggestions on topics including cleaning tools, getting rid of an upset stomach and laundry care."}]}
{"category_path": "Home/Home_Improvement/", "categories": [{"cat_url": ["/Home/Home_Improvement/Appliances/"], "cat_name": ["", "Appliances", ""]}, {"cat_url": ["/Home/Home_Improvement/Bathrooms/"], "cat_name": ["", "Bathrooms", ""]}, {"cat_url": ["/Home/Home_Improvement/Exterior/"], "cat_name": ["", "Exterior", ""]}, {"cat_url": ["/Home/Home_Improvement/Floors/"], "cat_name": ["", "Floors", ""]}, {"cat_url": ["/Home/Home_Improvement/Furniture/"], "cat_name": ["", "Furniture", ""]}, {"cat_url": ["/Home/Home_Improvement/Kitchens/"], "cat_name": ["", "Kitchens", ""]}, {"cat_url": ["/Home/Home_Improvement/Storage/"], "cat_name": ["", "Storage", ""]}, {"cat_url": ["/Home/Home_Improvement/Walls/"], "cat_name": ["", "Walls", ""]}, {"cat_url": ["/Home/Home_Improvement/Windows_and_Doors/"], "cat_name": ["", "WindowsandDoors", ""]}, {"cat_url": ["/Home/Home_Improvement/Automation/"], "cat_name": ["", "Automation", ""]}, {"cat_url": ["/Home/Home_Improvement/Climate_Control/"], "cat_name": ["", "ClimateControl", ""]}, {"cat_url": ["/Home/Home_Improvement/Decorating/"], "cat_name": ["", "Decorating", ""]}, {"cat_url": ["/Home/Home_Improvement/Electrical/"], "cat_name": ["", "Electrical", ""]}, {"cat_url": ["/Home/Home_Improvement/Energy_Efficiency/"], "cat_name": ["", "EnergyEfficiency", ""]}, {"cat_url": ["/Home/Home_Improvement/Lighting/"], "cat_name": ["", "Lighting", ""]}, {"cat_url": ["/Home/Home_Improvement/Painting/"], "cat_name": ["", "Painting", ""]}, {"cat_url": ["/Home/Home_Improvement/Plumbing/"], "cat_name": ["", "Plumbing", ""]}, {"cat_url": ["/Home/Home_Improvement/Restoration/"], "cat_name": ["", "Restoratio­n", ""]}, {"cat_url": ["/Home/Home_Improvement/Safety_and_Security/"], "cat_name": ["", "SafetyandSecurity", ""]}, {"cat_url": ["/Home/Home_Improvement/Welding_and_Soldering/"], "cat_name": ["", "WeldingandSoldering", ""]}, {"cat_url": ["/Home/Home_Improvement/Chats_and_Forums/"], "cat_name": ["", "ChatsandForums", ""]}, {"cat_url": ["/Business/Construction_and_Maintenance/Commercial_Contractors/"], "cat_name": ["", "CommercialContractors", ""]}, {"cat_url": ["/Home/Home_Improvement/Design_and_Construction/"], "cat_name": ["", "DesignandConstructi­on", ""]}, {"cat_url": ["/Home/Home_Improvement/Glossaries/"], "cat_name": ["", "Glossaries", ""]}, {"cat_url": ["/Home/Home_Improvement/News_and_Media/"], "cat_name": ["", "NewsandMedia", ""]}, {"cat_url": ["/Home/Home_Improvement/Tools_and_Equipment/"], "cat_name": ["", "ToolsandEquipment", ""]}], "sites": [{"site_title": "411 Home Repair", "site_url": "http://www.411homerepair.com/", "site_desc": "Collection of short articles offering tips for home repair, gardens, tools, and appliances."}, {"site_title": "Acme How To", "site_url": "http://www.acmehowto.com/", "site_desc": "Articles on tasks and repairs around the home and garden."}, {"site_title": "Around the House", "site_url": "http://www.thefunplace.com/house/home/", "site_desc": "Articles on pool maintenance, water softeners, lead exposure, doors and windows, and heating safely."}, {"site_title": "Ask The Builder", "site_url": "http://www.askthebuilder.com/", "site_desc": "Information about home building and remodeling."}, {"site_title": "AskToolTalk.com", "site_url": "http://www.asktooltalk.com/", "site_desc": "Home improvement experts feature articles, product reviews, links to manufacturers, and online shopping."}, {"site_title": "Beaver House Addition", "site_url": "http://adam_sb.tripod.com/beaveraddition/", "site_desc": "Homeowner's journal of large, residential building addition project spanning several years. Includes photographs, descriptions, and stories."}, {"site_title": "BobVila.com", "site_url": "http://www.bobvila.com/", "site_desc": "Home improvement projects, featured products, tip library, bulletin board, designer tools, and information about television programs hosted by Bob Vila."}, {"site_title": "Construction Resource", "site_url": "http://www.construction-resource.com/", "site_desc": "Employment listings, forums, how to articles, and calculators."}, {"site_title": "Consumer Information Center: Housing", "site_url": "http://publications.usa.gov/USAPubs.php?CatID=8", "site_desc": "Collection of home maintenance articles ranging from two to thirty-six pages available for download or purchase. Most include detailed information and illustrations. From the United States Federal Consumer Information Center."}, {"site_title": "Dave's Shop Talk: Building Confidence", "site_url": "http://daveosborne.com/dave/index.php", "site_desc": "Features articles and plans for the do it yourself person on home renovations and home repair, from a professional carpenter, renovator and contractor."}, {"site_title": "DIY Doctor", "site_url": "http://www.diydoctor.org.uk/projects.htm", "site_desc": "Collection of articles and photographs covering a range of home projects."}, {"site_title": "Diy Fix It", "site_url": "http://www.diyfixit.co.uk/", "site_desc": "Collection of tips and advice covering building and repairs, plumbing, tiling, electrical, wallpapering, painting, and decorating."}, {"site_title": "DIY Not", "site_url": "http://www.diynot.com/", "site_desc": "Information including an encyclopedia, regular articles and a discussion forum."}, {"site_title": "DIY Repair and Home Improvement Forums", "site_url": "http://www.diychatroom.com/", "site_desc": "A community of homeowners and contractors sharing knowledge on painting, construction, electrical, plumbing, carpentry, flooring, and landscaping."}, {"site_title": "DIYData.com", "site_url": "http://www.diydata.com/", "site_desc": "Collection of short articles on topics including plumbing, painting, tool usage, and other repair projects around the home."}, {"site_title": "DIYonline.com", "site_url": "http://www.diyonline.com/", "site_desc": "Articles and step-by-step tutorials with photographs for a wide variety of home improvement projects, glossary, cost calculators (United States-based), and design tools."}, {"site_title": "HammerZone.com", "site_url": "http://www.hammerzone.com/", "site_desc": "Photographs and step-by-step instructions for electrical, plumbing, kitchen, bath, windows and doors, exterior, flooring, and carpentry projects."}, {"site_title": "Handymanwire", "site_url": "http://www.handymanwire.com/", "site_desc": "Collection of articles, tips, FAQs, and forums for a variety of home improvement projects."}, {"site_title": "Helpwithdiy.com", "site_url": "http://www.helpwithdiy.com/", "site_desc": "Illustrated tutorials on topics including plumbing, painting, tiling, and decorating."}, {"site_title": "Home Improvement", "site_url": "http://www.home-improvement-home-improvement.com/", "site_desc": "Do-it-yourself articles cover interior and exterior, decks and patios, and decorating."}, {"site_title": "Home Repair", "site_url": "http://homerepair.about.com/index.htm", "site_desc": "Information on do-it-yourself projects or major renovations. Offers time and money saving techniques, diagrams, and links to other home repair sites."}, {"site_title": "Home Repair Stuff", "site_url": "http://www.factsfacts.com/MyHomeRepair/", "site_desc": "Frequently asked questions from the alt.home.repair newsgroup, with plumbing, carpentry, and tool tips."}, {"site_title": "Home Repairs", "site_url": "http://www.repair-home.com/", "site_desc": "Home repair tips, discussion forum and contractor search utility."}, {"site_title": "Home Tips", "site_url": "http://www.hometips.com/", "site_desc": "Home repair and improvement advice and tips including \"How Your House Works\", a manual by Don Vandervort, buying guides, ideas for home and shop."}, {"site_title": "HomeDoctor.net", "site_url": "http://homedoctor.net/", "site_desc": "Short articles covering appliance repair, pest control, electrical repairs, energy savings, and heating and cooling in the home. Also includes discussion forums."}, {"site_title": "HomeImprove.com", "site_url": "http://homeimprove.com/", "site_desc": "Collection of short articles on dozens of home improvement and repair topics."}, {"site_title": "Hometime", "site_url": "http://www.hometime.com/Howto/howto.htm", "site_desc": "An assortment of how-to guides, with manufacturer and safety information."}, {"site_title": "How Not to Build an Addition", "site_url": "http://www.homehumor.com/", "site_desc": "All aspects of home improvement from getting a loan to finish carpentry.  Humor and practical advice."}, {"site_title": "Jackie Craven: The Fix", "site_url": "http://jackiecraven.com/fixit/thefix.htm", "site_desc": "Article archive with answers on a variety of subjects including kitchen and bath, paint and wallpaper, home design, and pests."}, {"site_title": "Jerry Built", "site_url": "http://jerrybuilt.com/", "site_desc": "Woodworking and home-improvement site offering the opportunity to participate in the construction of an actual online project, by joining the \"team\" and submitting ideas, plans, and criticisms."}, {"site_title": "Kenovations", "site_url": "http://www.kenovations.net/", "site_desc": "A homeowner's step-by-step guide to renovating a 1968 Cape Cod home."}, {"site_title": "Mobile Home Doctor", "site_url": "http://www.mobilehomedoctor.com/", "site_desc": "Short articles offering tips for repair and improvement specific to mobile homes. Includes a tutorial on construction of mobile homes."}, {"site_title": "Mobile Home Repair", "site_url": "http://www.mobilehomerepair.com/", "site_desc": "Advice; hardboard siding lawsuit update."}, {"site_title": "Move, Inc. Home Improvement", "site_url": "http://www.realtor.com/advice/home-improvement/", "site_desc": "Offers a variety of how-to guides and do-it-yourself related calculators."}, {"site_title": "National Association of the Remodeling Industry", "site_url": "http://www.nari.org/", "site_desc": "Tips on planning, what to look for when hiring a pro."}, {"site_title": "National Kitchen and Bath Association", "site_url": "http://www.nkba.org/", "site_desc": "Information for consumers and trade professionals. Consumers can locate design professionals, research design strategies, and request a free remodeling planning kit."}, {"site_title": "The Natural Handyman", "site_url": "http://www.naturalhandyman.com/", "site_desc": "Home repair help, humor, and encouragement through a collection of articles and a newsletter."}, {"site_title": "Practical DIY", "site_url": "http://www.practicaldiy.com/", "site_desc": "UK-specific information. Small series of Do-It-Yourself home repair articles."}, {"site_title": "Ten Square Metres", "site_url": "http://www.tensquaremetres.com/", "site_desc": "A photographic diary of a DIY project to extend a house by ten square metres."}, {"site_title": "This to That", "site_url": "http://www.thistothat.com/", "site_desc": "Advice about how to glue things to other              things. They are given with humor and good details."}, {"site_title": "Thumb and Hammer", "site_url": "http://www.thumbandhammer.com/", "site_desc": "A do-it-yourselfer's photo reviews of his past projects."}, {"site_title": "You Repair", "site_url": "http://www.yourepair.com/", "site_desc": "Helping people fix things around the house or understanding what a contractor will do to solve your troubles. Managed by Robert Marencin."}]}

不得不说这个网站真的太庞大了，我在编写这篇文章时，我的小蜘蛛都没有爬完home目录下的内容，但是我们组员之前已经自己编写并运行了爬取home目录的项目（据他说用了一个半小时左右，但是其他组的同学有4个小时才爬完他们的目录），结果显示是13280多条数据：

组员的运行成果

补上我的运行结果：
本次实验的运行结果

为什么组员爬了1万多条而我的只有2378条呢，这看起来很奇怪，但这是因为我把一个目录下的所有site作为对象放进了sites{}里面,所以整体的数量条数就变少了。

五、操作过程中出现的主要问题

1.代码出现的问题

child_cat['cat_name'] = cat.xpath('a/div/text()').extract().replace("\r\n", "").replace(" ", "")

以上是获取目录名称最初的代码，由于获取的结果含有\n,\r以及空格等标签，本来打算直接在后面使用replace代替，但是运行报错：

AttributeError: 'list' object has no attribute 'replace' when trying to remove character

后面发现是因为 xpath方法返回一个列表而不是字符串，替换只适用于字符串即replace是不能直接作用于列表的，因此需要迭代项目来完成一些替换工作。（参考：https://stackoverflow.com/questions/36642782/attributeerror-list-object-has-no-attribute-replace-when-trying-to-remove-c）
因而后面的代码便改成了：

 child_cat['cat_name'] = cat.xpath('a/div/text()').extract()
            child_cat['cat_name'] = [item.replace("\r\n", "") for item in child_cat['cat_name']]
            child_cat['cat_name'] = [item.replace(" ", "") for item in child_cat['cat_name']]

不过还存在一点问题，就是没有替换掉 ' 以及 ，
2.爬取过程中被禁止访问的问题
由于之前做过豆瓣的爬取，因此知道需要将爬虫伪装成浏览器访问——在settings.py中添加：

USER_AGENT = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"

但编写代码的过程中总会有很多出错的地方，编辑过程中出现了一个死循环，我还没有发现，当我运行时发现爬虫一直不断地在爬取同一个目录下的site里面的东西，没有跳出来，由于爬取频率太高，被这个网站发现了，于是IP就被禁止了，当然访问时就出现403错误。解禁过后，就立马去解决这个问题，在settings.py中找到：

#DOWNLOAD_DELAY = 3

将#去掉，爬取时设置3秒的时间间隔，这样爬取的频率就不会太高。

六、总结

在爬取一个网站之前我认为需要将整个网站的结构给分析透彻，清楚每一个链接会链到什么页面，每一个页面的div块哪里一样，这样才能便于分析，制定爬取计划。总之，我觉得这是我这几周以来做过的最难的爬虫项目了，也认识到了自己对python和scrapy其实还有好多好多不懂得地方，好多方法和包都不会用。在进行项目的过程中还需要和组员、同学多多交流讨论，互相学习。这个网站，真的不知道该怎么去说，结构化程很高，但是一层一层的深入下去，要爬取到数据还是需要好好的分析，制定对爬取的路线。这是一个练习爬虫的比较好的网站，但是数据量也有点太庞大了。

参考：
https://blog.csdn.net/u012150179/article/details/34913315
https://www.jianshu.com/p/83c73071d3cb
https://www.jianshu.com/p/83c73071d3cb
一个学习python基本知识的网站http://www.runoob.com/python/python-tutorial.html
十分重要的友情链接（对完成本次实验帮助很大）https://www.jianshu.com/p/d6b6feb0a504
我们小组关于dmoz/Home目录下的数据的采集另一种采集方法https://www.jianshu.com/p/51419fec3915

你可能感兴趣的:(scrapy采集dmoz网站Home目录下的信息)

  • 斤斤计较的婚姻到底有多难？ 白心之岂必有为
    很多人私聊我会问到在哪个人群当中斤斤计较的人最多？我都会回答他，一般婚姻出现问题的斤斤计较的人士会非常多，以我多年经验，在婚姻落的一塌糊涂的人当中，斤斤计较的人数占比在20～30%以上，也就是说10个婚姻出现问题的斤斤计较的人有2-3个有多不减。在婚姻出问题当中，有大量的心理不平衡的、尖酸刻薄的怨妇。在婚姻中仅斤斤计较有两种类型：第一种是物质上的，另一种是精神上的。在物质与精神上抠门已经严重的影响
  • 情绪觉察日记第37天 露露_e800
    今天是家庭关系规划师的第二阶最后一天，慧萍老师帮我做了个案，帮我处理了埋在心底好多年的一份恐惧，并给了我深深的力量！这几天出来学习，爸妈过来婆家帮我带小孩，妈妈出于爱帮我收拾东西，并跟我先生和婆婆产生矛盾，妈妈觉得他们没有照顾好我…。今晚回家见到妈妈，我很欣赏她并赞扬她，妈妈说今晚要跟我睡我说好，当我们俩躺在床上准备睡觉的时候，我握着妈妈的手对她说:妈妈这几天辛苦你了，你看你多利害把我们的家收拾得
  • 芦花鞋一四 许叶晗
    又是在一个寒冷的夏日里，青铜和葵花决定今天一起去卖芦花鞋，奶奶亲手给他们做了一碗热乎乎的粥对他们说:“就靠你们两挣生活费了这碗粥赶紧趁热喝了吧！”于是青铜和葵花喝完了奶奶给她们做的粥，就准备去镇上卖卢花鞋，这回青铜和葵花穿着新的芦花鞋来到了镇上。青铜这回看到了很多人都在卖，用手势表达对葵花说:“这回有好多人在抢我们生意呢！我们必须得吆喝起来。”葵花点了点头。可是谁知他们也大声的叫，卖芦花喽！卖芦花
  • QQ群采集助手，精准引流必备神器 2401_87347160 其他经验分享
    功能概述微信群查找与筛选工具是一款专为微信用户设计的辅助工具，它通过关键词搜索功能，帮助用户快速找到相关的微信群，并提供筛选是否需要验证的群组的功能。主要功能关键词搜索：用户可以输入关键词，工具将自动查找包含该关键词的微信群。筛选功能：工具提供筛选机制，用户可以选择是否只显示需要验证或不需要验证的群组。精准引流：通过上述功能，用户可以更精准地找到目标群组，进行有效的引流操作。3.设备需求该工具可以
  • 关于沟通这件事，项目经理不需要每次都面对面进行 流程大师兄
    很多项目经理都会遇到这样的问题，项目中由于事情太多，根本没有足够的时间去召开会议，那在这种情况下如何去有效地管理项目中的利益相关者？当然，不建议电子邮件也不需要开会的话，建议可以采取下面几种方式来形成有效的沟通，这几种方式可以帮助你努力的通过各种办法来保持和各方面的联系。项目经理首先要问自己几个问题，项目中哪些利益相关者是必须要进行沟通的？可以列出项目中所有的利益相关者清单，同时也整理出项目中哪些
  • 机器学习与深度学习间关系与区别 ℒℴѵℯ心·动ꦿ໊ོ꫞ 人工智能学习深度学习python
    一、机器学习概述定义机器学习（MachineLearning,ML）是一种通过数据驱动的方法，利用统计学和计算算法来训练模型，使计算机能够从数据中学习并自动进行预测或决策。机器学习通过分析大量数据样本，识别其中的模式和规律，从而对新的数据进行判断。其核心在于通过训练过程，让模型不断优化和提升其预测准确性。主要类型1.监督学习（SupervisedLearning）监督学习是指在训练数据集中包含输入
  • 铭刻于星（四十二） 随风至
    69夜晚，绍敏同学做完功课后，看了眼房外，没听到动静才敢从书包的夹层里拿出那个心形纸团。折痕压得很深，都有些旧了，想来是已经写好很久了。绍敏同学慢慢地、轻轻地捏开折叠处，待到全部拆开后，又反复抚平纸张，然后仔细地一字字默看。只是开头的三个字是第一次看到，让她心漏跳了几拍。“亲爱的绍敏：从四年级的时候，我就喜欢你了，但是我一直不敢说，怕影响你学习。六年级的时候听说有人跟你表白，你接受了，我很难过，但
  • 底层逆袭到底有多难，不甘平凡的你准备好了吗？让吴起给你说说 造命者说
    底层逆袭到底有多难，不甘平凡的你准备好了吗？让吴起给你说说我叫吴起，生于公元前440年的战国初期，正是群雄并起、天下纷争不断的时候。后人说我是军事家、政治家、改革家，是兵家代表人物。评价我一生历仕鲁、魏、楚三国，通晓兵家、法家、儒家三家思想，在内政军事上都有极高的成就。周安王二十一年（公元前381年），因变法得罪守旧贵族，被人乱箭射死。我出生在卫国一个“家累万金”的富有家庭，从年轻时候起就不甘平凡
  • 2020-01-25 晴岚85
    郑海燕坚持分享590天2020.1.24在生活中只存在两个问题。一个问题是：你知道想要达成的目标是什么，但却不知道如何才能达成；另一个问题是：你不知道你的目标是什么。前一个是行动的问题，后一个是结果的问题。通过制定具体的下一步行动，可以解决不知道如何开始行动的问题。而通过去想象结果，对结果做预估，可以解决找不着目标的问题。对于所有吸引我们注意力，想要完成的任务，你可以先想象一下，预期的结果究竟是什
  • 随笔 | 仙一般的灵气 海思沧海
    仙岛今天，我看了你全部，似乎已经进入你的世界我不知道，这是否是梦幻，还是你仙一般的灵气吸引了我也许每一个人都要有一份属于自己的追求，这样才能够符合人生的梦想，生活才能够充满着阳光与快乐我不知道，我为什么会这样的感叹，是在感叹自己的人生，还是感叹自己一直没有孜孜不倦的追求只感觉虚度了光阴，每天活在自己的梦中，活在一个不真实的世界是在逃避自己，还是在逃避周围的一切有时候我嘲笑自己，嘲笑自己如此的虚无，
  • 想家 爆米花机
    也许不同于大家对家乡的思念，我对家乡甚至是疯狂的不舍。还未踏出车站就感觉到幸福，我享受这里的夕阳、这里的浓烈柴火味、这里每一口家常菜。我是宅女，我贪恋家的安逸。刚刚踏出大学校门，初出茅庐，无法适应每年只能国庆和春节回家。我焦虑、失眠、无端发脾气，是无法适应工作的节奏，是无法接受我将一步步离开家乡的事实。我不想承认自己胸无大志，选择再次踏上征程。图片发自App
  • 【iOS】MVC设计模式 Magnetic_h iosmvc设计模式objective-c学习ui
    MVC前言如何设计一个程序的结构，这是一门专门的学问，叫做"架构模式"（architecturalpattern），属于编程的方法论。MVC模式就是架构模式的一种。它是Apple官方推荐的App开发架构，也是一般开发者最先遇到、最经典的架构。MVC各层controller层Controller/ViewController/VC（控制器）负责协调Model和View，处理大部分逻辑它将数据从Mod
  • OC语言多界面传值五大方式 Magnetic_h iosui学习objective-c开发语言
    前言在完成暑假仿写项目时，遇到了许多需要用到多界面传值的地方，这篇博客来总结一下比较常用的五种多界面传值的方式。属性传值属性传值一般用前一个界面向后一个界面传值，简单地说就是通过访问后一个视图控制器的属性来为它赋值，通过这个属性来做到从前一个界面向后一个界面传值。首先在后一个界面中定义属性@interfaceBViewController:UIViewController@propertyNSSt
  • 一百九十四章. 自相矛盾 巨木擎天
    唉！就这么一夜，林子感觉就像过了很多天似的，先是回了阳间家里，遇到了那么多不可思议的事情儿。特别是小伙伴们，第二次与自己见面时，僵硬的表情和恐怖的气氛，让自己如坐针毡，打从心眼里难受！还有东子，他现在还好吗？有没有被人欺负？护城河里的小鱼小虾们，还都在吗？水不会真的干枯了吧？那对相亲相爱漂亮的太平鸟儿，还好吧！春天了，到了做窝、下蛋、喂养小鸟宝宝的时候了，希望它们都能够平安啊！虽然没有看见家人，也
  • UI学习——cell的复用和自定义cell Magnetic_h ui学习
    目录cell的复用手动（非注册）自动（注册）自定义cellcell的复用在iOS开发中，单元格复用是一种提高表格（UITableView）和集合视图（UICollectionView）滚动性能的技术。当一个UITableViewCell或UICollectionViewCell首次需要显示时，如果没有可复用的单元格，则视图会创建一个新的单元格。一旦这个单元格滚动出屏幕，它就不会被销毁。相反，它被添
  • element实现动态路由+面包屑 软件技术NINI vue案例vue.js前端
    el-breadcrumb是ElementUI组件库中的一个面包屑导航组件，它用于显示当前页面的路径，帮助用户快速理解和导航到应用的各个部分。在Vue.js项目中，如果你已经安装了ElementUI，就可以很方便地使用el-breadcrumb组件。以下是一个基本的使用示例：安装ElementUI（如果你还没有安装的话）:你可以通过npm或yarn来安装ElementUI。bash复制代码npmi
  • 10月|愿你的青春不负梦想-读书笔记-01 Tracy的小书斋
    本书的作者是俞敏洪，大家都很熟悉他了吧。俞敏洪老师是我行业的领头羊吧，也是我事业上的偶像。本日摘录他书中第一章中的金句：『一个人如果什么目标都没有，就会浑浑噩噩，感觉生命中缺少能量。能给我们能量的，是对未来的期待。第一件事，我始终为了进步而努力。与其追寻全世界的骏马，不如种植丰美的草原，到时骏马自然会来。第二件事，我始终有阶段性的目标。什么东西能给我能量？答案是对未来的期待。』读到这里的时候，我便
  • C语言宏函数 南林yan C语言c语言
    一、什么是宏函数？通过宏定义的函数是宏函数。如下，编译器在预处理阶段会将Add(x,y)替换为((x)*(y))#defineAdd(x,y)((x)*(y))#defineAdd(x,y)((x)*(y))intmain(){inta=10;intb=20;intd=10;intc=Add(a+d,b)*2;cout<
  • 地推话术，如何应对地推过程中家长的拒绝 校师学
    相信校长们在做地推的时候经常遇到这种情况：市场专员反馈家长不接单，咨询师反馈难以邀约这些家长上门，校区地推疲软，招生难。为什么？仅从地推层面分析，一方面因为家长受到的信息轰炸越来越多，对信息越来越“免疫”；而另一方面地推人员的专业能力和营销话术没有提高，无法应对家长的拒绝，对有意向的家长也不知如何跟进，眼睁睁看着家长走远；对于家长的疑问，更不知道如何有技巧地回答，机会白白流失。由于回答没技巧和专业
  • 谢谢你们，爱你们！ 鹿游儿
    昨天家人去泡温泉，二个孩子也带着去，出发前一晚，匆匆下班，赶回家和孩子一起收拾。饭后，我拿出笔和本子（上次去澳门时做手帐的本子）写下了1\2\3\4\5\6\7\8\9,让后让小壹去思考，带什么出发去旅游呢？她在对应的数字旁边画上了，泳衣、泳圈、肖恩、内衣内裤、tapuy、拖鞋……画完后，就让她自己对着这个本子，将要带的，一一带上，没想到这次带的书还是这本《便便工厂》(晚上姑婆发照片过来，妹妹累得
  • C语言如何定义宏函数？ 小九格物 c语言
    在C语言中，宏函数是通过预处理器定义的，它在编译之前替换代码中的宏调用。宏函数可以模拟函数的行为，但它们不是真正的函数，因为它们在编译时不会进行类型检查，也不会分配存储空间。宏函数的定义通常使用#define指令，后面跟着宏的名称和参数列表，以及宏展开后的代码。宏函数的定义方式：1.基本宏函数：这是最简单的宏函数形式，它直接定义一个表达式。#defineSQUARE(x)((x)*(x))2.带参
  • 微服务下功能权限与数据权限的设计与实现 nbsaas-boot 微服务java架构
    在微服务架构下，系统的功能权限和数据权限控制显得尤为重要。随着系统规模的扩大和微服务数量的增加，如何保证不同用户和服务之间的访问权限准确、细粒度地控制，成为设计安全策略的关键。本文将讨论如何在微服务体系中设计和实现功能权限与数据权限控制。1.功能权限与数据权限的定义功能权限：指用户或系统角色对特定功能的访问权限。通常是某个用户角色能否执行某个操作，比如查看订单、创建订单、修改用户资料等。数据权限：
  • 理解Gunicorn：Python WSGI服务器的基石 范范0825 ipythonlinux运维
    理解Gunicorn：PythonWSGI服务器的基石介绍Gunicorn，全称GreenUnicorn，是一个为PythonWSGI（WebServerGatewayInterface）应用设计的高效、轻量级HTTP服务器。作为PythonWeb应用部署的常用工具，Gunicorn以其高性能和易用性著称。本文将介绍Gunicorn的基本概念、安装和配置，帮助初学者快速上手。1.什么是Gunico
  • 小丽成长记（四十三） 玲玲54321
    小丽发现，即使她好不容易调整好自己的心态下一秒总会有不确定的伤脑筋的事出现，一个接一个的问题，人生就没有停下的时候，小问题不断出现。不过她今天看的书，她接受了人生就是不确定的，厉害的人就是不断创造确定性，在Ta的领域比别人多的确定性就能让自己脱颖而出，显示价值从而获得的比别人多的利益。正是这样的原因，因为从前修炼自己太少，使得她现在在人生道路上打怪起来困难重重，她似乎永远摆脱不了那种无力感，有种习
  • 学点心理知识，呵护孩子健康 静候花开_7090
    昨天听了华中师范大学教育管理学系副教授张玲老师的《哪里才是学生心理健康的最后庇护所，超越教育与技术的思考》的讲座。今天又重新学习了一遍，收获匪浅。张玲博士也注意到了当今社会上的孩子由于心理问题导致的自残、自杀及伤害他人等恶性事件。她向我们普及了一个重要的命题，她说心理健康的一些基本命题，我们与我们通常的一些教育命题是不同的，她还举了几个例子，让我们明白我们原来以为的健康并非心理学上的健康。比如如果
  • 2021年12月19日，春蕾教育集团团建活动感受——黄晓丹 黄错错加油
    感受:1.从陌生到熟悉的过程。游戏环节让我们在轻松的氛围中得到了锻炼，也增长了不少知识。2.游戏过程中，我们贡献的是个人力量，展现的是团队的力量。它磨合的往往不止是工作的熟悉，更是观念上契合度的贴近。3.这和工作是一样的道理。在各自的岗位上，每个人摆正自己的位置、各司其职充分发挥才能，并团结一致劲往一处使，才能实现最大的成功。新知:1.团队精神需要不断地创新。过去，人们把创新看作是冒风险，现在人们
  • Cell Insight | 单细胞测序技术又一新发现，可用于HIV-1和Mtb共感染个体诊断 尐尐呅
    结核病是艾滋病合并其他疾病中导致患者死亡的主要原因。其中结核病由结核分枝杆菌（Mycobacteriumtuberculosis,Mtb）感染引起，获得性免疫缺陷综合症（艾滋病）由人免疫缺陷病毒（Humanimmunodeficiencyvirustype1,HIV-1）感染引起。国家感染性疾病临床医学研究中心/深圳市第三人民医院张国良团队携手深圳华大生命科学研究院吴靓团队，共同研究得出单细胞测序
  • c++ 的iostream 和 c++的stdio的区别和联系 黄卷青灯77 c++算法开发语言iostreamstdio
    在C++中，iostream和C语言的stdio.h都是用于处理输入输出的库，但它们在设计、用法和功能上有许多不同。以下是两者的区别和联系：区别1.编程风格iostream（C++风格）：C++标准库中的输入输出流类库，支持面向对象的输入输出操作。典型用法是cin（输入）和cout（输出），使用>操作符来处理数据。更加类型安全，支持用户自定义类型的输入输出。#includeintmain(){in
  • 瑶池防线 谜影梦蝶
    冥华虽然逃过了影梦的军队，但他是一个忠臣，他选择上报战况。败给影梦后成逃兵，高层亡尔还活着，七重天失守......随便一条，即可处死冥华。冥华自然是知道以仙界高层的习性此信一发自己必死无疑，但他还选择上报实情，因为责任。同样此信送到仙宫后，知道此事的人，大多数人都认定冥华要完了，所以上到仙界高层，下到扫大街的，包括冥华自己，全都准备好迎接冥华之死。如果仙界现在还属于两方之争的话，冥华必死无疑。然而
  • 爬山后遗症 璃绛
    爬山，攀登，一步一步走向制高点，是一种挑战。成功抵达是一种无法言语的快乐，在山顶吹吹风，看看风景，这是从未有过的体验。然而，爬山一时爽，下山腿打颤，颠簸的路，一路向下走，腿部力量不够，走起来抖到不行，停不下来了！第二天必定腿疼，浑身酸痛，坐立难安！
  • JVM StackMapTable 属性的作用及理解 lijingyao8206 jvm字节码Class文件StackMapTable
            在Java 6版本之后JVM引入了栈图(Stack Map Table)概念。为了提高验证过程的效率，在字节码规范中添加了Stack Map Table属性，以下简称栈图，其方法的code属性中存储了局部变量和操作数的类型验证以及字节码的偏移量。也就是一个method需要且仅对应一个Stack Map Table。在Java 7版
  • 回调函数调用方法 百合不是茶 java
    最近在看大神写的代码时,.发现其中使用了很多的回调 ,以前只是在学习的时候经常用到 ,现在写个笔记 记录一下   代码很简单:           MainDemo  :调用方法  得到方法的返回结果        
  • [时间机器]制造时间机器需要一些材料 comsci 制造
          根据我的计算和推测,要完全实现制造一台时间机器,需要某些我们这个世界不存在的物质     和材料...       甚至可以这样说,这种材料和物质,我们在反应堆中也无法获得......      
  • 开口埋怨不如闭口做事 邓集海 邓集海 做人 做事 工作
    “开口埋怨，不如闭口做事。”不是名人名言，而是一个普通父亲对儿子的训导。但是，因为这句训导，这位普通父亲却造就了一个名人儿子。这位普通父亲造就的名人儿子，叫张明正。 　　 　　张明正出身贫寒，读书时成绩差，常挨老师批评。高中毕业，张明正连普通大学的分数线都没上。高考成绩出来后，平时开口怨这怨那的张明正，不从自身找原因，而是不停地埋怨自己家庭条件不好、埋怨父母没有给他创造良好的学习环境。 　　 　　
  • jQuery插件开发全解析，类级别与对象级别开发 IT独行者 jquery开发插件　函数
    jQuery插件的开发包括两种： 一种是类级别的插件开发，即给 jQuery添加新的全局函数，相当于给 jQuery类本身添加方法。 jQuery的全局函数就是属于 jQuery命名空间的函数，另一种是对象级别的插件开发，即给 jQuery对象添加方法。下面就两种函数的开发做详细的说明。   1 、类级别的插件开发 类级别的插件开发最直接的理解就是给jQuer
  • Rome解析Rss 413277409 Rome解析Rss
    import java.net.URL;  import java.util.List;    import org.junit.Test;    import com.sun.syndication.feed.synd.SyndCategory;  import com.sun.syndication.feed.synd.S
  • RSA加密解密 无量 加密解密rsa
    RSA加密解密代码 代码有待整理 package com.tongbanjie.commons.util; import java.security.Key; import java.security.KeyFactory; import java.security.KeyPair; import java.security.KeyPairGenerat
  • linux 软件安装遇到的问题 aichenglong linux遇到的问题ftp
    1 ftp配置中遇到的问题    500 OOPS: cannot change directory   出现该问题的原因:是SELinux安装机制的问题.只要disable SELinux就可以了   修改方法:1 修改/etc/selinux/config 中SELINUX=disabled    2 source /etc
  • 面试心得 alafqq 面试
    最近面试了好几家公司。记录下； 支付宝，面试我的人胖胖的，看着人挺好的；博彦外包的职位，面试失败； 阿里金融，面试官人也挺和善，只不过我让他吐血了。。。 由于印象比较深，记录下； 1，自我介绍 2，说下八种基本类型；（算上string。楼主才答了3种，哈哈，string其实不是基本类型，是引用类型） 3，什么是包装类，包装类的优点； 4，平时看过什么书？NND，什么书都没看过。。照样
  • java的多态性探讨 百合不是茶 java
    java的多态性是指main方法在调用属性的时候类可以对这一属性做出反应的情况 //package 1; class A{ public void test(){ System.out.println("A"); } } class D extends A{ public void test(){ S
  • 网络编程基础篇之JavaScript-学习笔记 bijian1013 JavaScript
    1.documentWrite <html> <head> <script language="JavaScript"> document.write("这是电脑网络学校"); document.close(); </script> </h
  • 探索JUnit4扩展：深入Rule bijian1013 JUnitRule单元测试
            本文将进一步探究Rule的应用，展示如何使用Rule来替代@BeforeClass，@AfterClass，@Before和@After的功能。         在上一篇中提到，可以使用Rule替代现有的大部分Runner扩展，而且也不提倡对Runner中的withBefores()，withAfte
  • [CSS]CSS浮动十五条规则 bit1129 css
    这些浮动规则，主要是参考CSS权威指南关于浮动规则的总结，然后添加一些简单的例子以验证和理解这些规则。   1. 所有的页面元素都可以浮动 2. 一个元素浮动后，会成为块级元素，比如<span>,a, strong等都会变成块级元素 3.一个元素左浮动，会向最近的块级父元素的左上角移动，直到浮动元素的左外边界碰到块级父元素的左内边界；如果这个块级父元素已经有浮动元素停靠了
  • 【Kafka六】Kafka Producer和Consumer多Broker、多Partition场景 bit1129 partition
    0.Kafka服务器配置 3个broker 1个topic，6个partition，副本因子是2 2个consumer，每个consumer三个线程并发读取   1. Producer package kafka.examples.multibrokers.producers; import java.util.Properties; import java.util.
  • zabbix_agentd.conf配置文件详解 ronin47 zabbix 配置文件
    Aliaskey的别名，例如 Alias=ttlsa.userid:vfs.file.regexp[/etc/passwd,^ttlsa:.:([0-9]+),,,,\1]， 或者ttlsa的用户ID。你可以使用key：vfs.file.regexp[/etc/passwd,^ttlsa:.: ([0-9]+),,,,\1]，也可以使用ttlsa.userid。备注: 别名不能重复，但是可以有多个
  • java--19.用矩阵求Fibonacci数列的第N项 bylijinnan fibonacci
    参考了网上的思路，写了个Java版的： public class Fibonacci { final static int[] A={1,1,1,0}; public static void main(String[] args) { int n=7; for(int i=0;i<=n;i++){ int f=fibonac
  • Netty源码学习-LengthFieldBasedFrameDecoder bylijinnan javanetty
    先看看LengthFieldBasedFrameDecoder的官方API http://docs.jboss.org/netty/3.1/api/org/jboss/netty/handler/codec/frame/LengthFieldBasedFrameDecoder.html API举例说明了LengthFieldBasedFrameDecoder的解析机制，如下： 实
  • AES加密解密 chicony 加密解密
    AES加解密算法，使用Base64做转码以及辅助加密： package com.wintv.common; import javax.crypto.Cipher; import javax.crypto.spec.IvParameterSpec; import javax.crypto.spec.SecretKeySpec; import sun.misc.BASE64Decod
  • 文件编码格式转换 ctrain 编码格式
    package com.test; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream;
  • mysql 在linux客户端插入数据中文乱码 daizj mysql中文乱码
    1、查看系统客户端，数据库，连接层的编码  查看方法： http://daizj.iteye.com/blog/2174993 进入mysql，通过如下命令查看数据库编码方式： mysql>  show variables like 'character_set_%'; +--------------------------+------
  • 好代码是廉价的代码 dcj3sjt126com 程序员读书
      长久以来我一直主张：好代码是廉价的代码。 当我跟做开发的同事说出这话时，他们的第一反应是一种惊愕，然后是将近一个星期的嘲笑，把它当作一个笑话来讲。 当他们走近看我的表情、知道我是认真的时，才收敛一点。 当最初的惊愕消退后，他们会用一些这样的话来反驳： “好代码不廉价，好代码是采用经过数十年计算机科学研究和积累得出的最佳实践设计模式和方法论建立起来的精心制作的程序代码。” 我只
  • Android网络请求库——android-async-http dcj3sjt126com android
    在iOS开发中有大名鼎鼎的ASIHttpRequest库，用来处理网络请求操作，今天要介绍的是一个在Android上同样强大的网络请求库android-async-http，目前非常火的应用Instagram和Pinterest的Android版就是用的这个网络请求库。这个网络请求库是基于Apache HttpClient库之上的一个异步网络请求处理库，网络处理均基于Android的非UI线程，通
  • ORACLE 复习笔记之SQL语句的优化 eksliang SQL优化Oracle sql语句优化SQL语句的优化
    转载请出自出处：http://eksliang.iteye.com/blog/2097999   SQL语句的优化总结如下   sql语句的优化可以按照如下六个步骤进行： 合理使用索引 避免或者简化排序 消除对大表的扫描 避免复杂的通配符匹配 调整子查询的性能 EXISTS和IN运算符 下面我就按照上面这六个步骤分别进行总结：
  • 浅析：Android 嵌套滑动机制（NestedScrolling） gg163 android移动开发滑动机制嵌套
    谷歌在发布安卓 Lollipop版本之后，为了更好的用户体验，Google为Android的滑动机制提供了NestedScrolling特性 NestedScrolling的特性可以体现在哪里呢？<!--[if !supportLineBreakNewLine]--><!--[endif]--> 比如你使用了Toolbar，下面一个ScrollView，向上滚
  • 使用hovertree菜单作为后台导航 hvt JavaScriptjquery.nethovertreeasp.net
      hovertree是一个jquery菜单插件，官方网址：http://keleyi.com/jq/hovertree/ ，可以登录该网址体验效果。 0.1.3版本：http://keleyi.com/jq/hovertree/demo/demo.0.1.3.htm hovertree插件包含文件： http://keleyi.com/jq/hovertree/css
  • SVG 教程 （二）矩形 天梯梦 svg
    SVG <rect> SVG Shapes SVG有一些预定义的形状元素，可被开发者使用和操作： 矩形 <rect> 圆形 <circle> 椭圆 <ellipse> 线 <line> 折线 <polyline> 多边形 <polygon> 路径 <path>
  • 一个简单的队列 luyulong java数据结构队列
    public class MyQueue { private long[] arr; private int front; private int end; // 有效数据的大小 private int elements; public MyQueue() { arr = new long[10]; elements = 0; front
  • 基础数据结构和算法九：Binary Search Tree sunwinner Algorithm
      A binary search tree (BST) is a binary tree where each node has a Comparable key (and an associated value) and satisfies the restriction that the key in any node is larger than the keys in all
  • 项目出现的一些问题和体会 Steven-Walker DAOWebservlet
         第一篇博客不知道要写点什么，就先来点近阶段的感悟吧。     这几天学了servlet和数据库等知识，就参照老方的视频写了一个简单的增删改查的，完成了最简单的一些功能，使用了三层架构。 dao层完成的是对数据库具体的功能实现，service层调用了dao层的实现方法，具体对servlet提供支持。  &
  • 高手问答：Java老A带你全面提升Java单兵作战能力！ ITeye管理员 java
    本期特邀《Java特种兵》作者：谢宇，CSDN论坛ID: xieyuooo 针对JAVA问题给予大家解答，欢迎网友积极提问，与专家一起讨论! 作者简介： 淘宝网资深Java工程师，CSDN超人气博主，人称“胖哥”。 CSDN博客地址： http://blog.csdn.net/xieyuooo 作者在进入大学前是一个不折不扣的计算机白痴，曾经被人笑话过不懂鼠标是什么，
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他