Python APP Scraping

ConnectionError, openssl

If you encounter this issue:

ConnectionError: ('Connection aborted.', error(54, 'Connection reset by peer'))

it is possible that you are using python2, old python version that does not support openssl sni. 

To fix this problem, check out website: 

This process wasn't totally straightforward so I thought I'd post a new answer to make it easy to follow for others.

Following this thread, I needed to install these libraries get SNI to work with Python 2:

pyOpenSSL

ndg-httpsclient

pyasn1

However, pyOpenSSL may cause problems when installed withpip install pyOpenSSL. I actually had to remove my existing openssl, since pyOpenSSL version 0.14 didn't seem to work:

pip uninstall pyOpenSSL

The following command installed all necessary dependencies:

pip install pyOpenSSL==0.13 ndg-httpsclient pyasn1

This should get requests to now work with SNI on python 2.

Keep reading for the issues with pyOpenSSL ver. 0.14...

When installing ver. 0.14 I get the following error:

Command /usr/local/opt/python/bin/python2.7 -c "import setuptools, tokenize;__file__='/private/var/folders/04/3f_y5fw166v03k7b51j1tsl80000gn/T/pip_build_alex/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/04/3f_y5fw166v03k7b51j1tsl80000gn/T/pip-7QR71B-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /private/var/folders/04/3f_y5fw166v03k7b51j1tsl80000gn/T/pip_build_alex/cryptography

Storing debug log for failure in /Users/alex/.pip/pip.log

and pyOpenSSL installs as ver. 0.14 incompletely:

$ pip show pyOpenSSL

---

Name: pyOpenSSL

Version: 0.14

Location: /usr/local/lib/python2.7/site-packages

Requires: cryptography, six

as can be seen from therequests.get()attempt:

import requests

response = requests.get("http://datamuster.marketdatasuite.com")

(...lots of errors...)

raise ConnectionError(err, request=request)

requests.exceptions.ConnectionError: ('Connection aborted.', error(54, 'Connection reset by peer'))

The following commands revert to pyOpenSSL ver. 0.13 and correct the issue

pip uninstall pyOpenSSL

pip install pyOpenSSL==0.13

and then in python:

import requests

requests.get("http://datamuster.marketdatasuite.com")