如何使用selenium-grid进行多任务测试/分布式爬虫

1.修改本地文件$ /usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py 183行
old:
parameters = {"capabilities": w3c_caps,
"desiredCapabilities": capabilities}
new:
parameters = {"desiredCapabilities": capabilities}

2.http://blog.csdn.net/henni_719/article/details/51206982
这个链接里的例子可以用

from threading import Thread
from selenium import webdriver
import time

list = {'http://127.0.0.1:4444/wd/hub': 'chrome',
        'http://127.0.0.1:5555/wd/hub': 'chrome'
        }


def test_baidu(host, browser):
    print 'start:%s' % time.ctime()
    print host, browser
    driver = webdriver.Remote(command_executor=host,
                              desired_capabilities={'platform': 'ANY',
                                                    'browserName': browser,
                                                    'version': '',
                                                    'javascriptEnabled': True
                                                    })


    driver.get('http://www.baidu.com')
    time.sleep(20)
    driver.find_element_by_id("kw").send_keys(browser)
    driver.find_element_by_id("su").click()
    time.sleep(2)
    driver.quit()

threads = []
files = range(len(list))
for host, browser in list.items():
    t = Thread(target=test_baidu, args=(host, browser))
    threads.append(t)
if __name__ == '__main__':
    for i in files:
        threads[i].start()
    for i in files:
        threads[i].join()
print 'end:%s' % time.ctime()

3.先开hub
java -jar /Users/appleapple/Desktop/yanset/glassdoor/selenium-server-standalone-3.4.0.jar -role hub
然后绑node
java -jar /Users/appleapple/Desktop/yanset/glassdoor/selenium-server-standalone-3.4.0.jar -role node -hub http://localhost:4444/grid/register

4.remote如何应用chromeoptions
https://stackoverflow.com/questions/13227346/set-chrome-options-with-remote-driver

5.headless 出问题而一般的chrome或者firefox没问题的解决方法
https://github.com/angular/protractor/issues/585
主要是因为windowsize，可以找到配置文件设置，也可以再报错的位置前手动set_window_size