28万个开源项目之番外篇

一、工具

1. 数据抓取

最初是打算使用openhub.net的Open API的，他们有不错的API，还在Github上放了一个开源项目。只可惜，他们的API，最多申请5个API Key，每个Key明天的访问请求数量，不能超过1000次。当时我还不知道，其实openhub的数据只有28万多，还以为满打满算，至少得60多天才能全部抓完，顿时心就凉了。

后来有朋友介绍了一个很棒的直接抓取HTML页面，然后做DOM分析的工具，名叫noodle。

接下来，只要抓取： https://www.openhub.net/p?ref=homepage&q=&page={num}
就能够拿到所有项目的概要数据了。

当然，后续的331个项目的明细数据，还是得通过OpenHub的API来抓取。

2. 数据分析

完全是土法上马：sqlite3+numbers+csv+ruby，反正各种手法，什么称手用什么。

3. 数据展示

原本是打算在numbers里想想办法的，后来发现实在太弱。Excel也差不多，只能到网上搜索一些信息图制作的工具，后来找到了几个不错的在线工具，经过一番比较，最后决定用infogr.am来完成。的确非常不错。

二、释疑：项目大小与创建时间的关系

我与@云风 在微博上有一小段讨论，起因还是我之前的一些分析的观点：

• 是否使用Github，越是新的项目越愿意用；越是大的项目越没法用。
• 是否使用Github来管理项目的issue，越是新的项目越愿意用；越是大的项目越没法用。

这个结论，其实在用词上，是有些讲究的：按理说，新与老相对，小与大相对；愿意与不愿意相对，能用与没法用相对，我的两个结论，对仗都不公整。其实，确实故意为之。

于是，云风与我的对话如下：
云风：项目规模和项目历史本身有相关性吧。代码规模越大的项目历史很可能越久。
我：项目的规模，主要还是与项目本身的特性有关。原本就复杂的项目，才可能越长越大。原本就是小项目，也未必就会稳定的逐年增长。
云风：这只能说明小项目可以历史久，不能说明大项目可以历史短啊。很少有新项目一开始就很大啊。代码也是一行行写出来的啊。
我：那就是成长速度不同了。比如OpenStack，一开始就不小。
云风：一开始就不小只能说闭源开发过一段时间，或从别的地方搬迁过来的吧。你能想象不被版本管理工具管理的情况下，首次提交 10 万行以上的代码？看这个 link 提交日志写的 initial fork out of nova。

后来，我也没有再继续这个讨论，但是却一直在思考这个问题：「项目的大小，与项目的创建时间，究竟有大少相关性？」

后来，我将两个数据，做了一个分析：Log(第一次提交代码，至今的天数)/Log(代码行数)，大概得到如下一个图：

经过强大的Excel的计算，两个数据的相关系数，大约是0.203的样子，也就是说：大致上有较弱的正相关。

三、开源

目前，我已经将这个分析的相关数据，放在Github上开源了。简单介绍一下：

data.sqlite3.zip 是28万基础数据
projects.sqlite3 是331个项目的详细数据
projects.csv 是我用来做数据分析的大表格

四、名单

331一个开源项目，名单如下：

Name	Homepage
Metasploit Framework	http://www.metasploit.com/framework/
NetBSD	http://www.netbsd.org
GNU C Library	http://www.gnu.org/software/libc/
cURL	http://curl.haxx.se/
Python programming language	https://www.python.org
Linux Kernel	http://kernel.org/
GNU Emacs	http://www.gnu.org/software/emacs
gnulib	http://savannah.gnu.org/projects/gnulib/
GNU Core Utilities	http://savannah.gnu.org/projects/coreutils/
GNU Compiler Collection	http://gcc.gnu.org/
Wine	http://www.winehq.org
Debian	http://www.debian.org/
GNU Octave	http://www.octave.org
Visualization Toolkit	http://www.vtk.org
pf	http://www.benzedrine.cx/pf.html
GDB	http://www.gnu.org/software/gdb/
GNU binutils	http://www.gnu.org/software/binutils/
GHC	http://haskell.org/ghc/
Zope	http://zope2.zope.org
FreeBSD	https://github.com/trueos/trueos
Perl	http://www.perl.org/
GNU LilyPond Music Typesetter	http://lilypond.org/
Gnus	http://gnus.org/
ikiwiki	https://github.com/schmonz/ikiwiki
Samba	http://www.samba.org
PHP	http://php.net
FreeBSD Ports	http://www.freebsd.org/ports/
pkgsrc: The NetBSD Packages Collection	http://www.pkgsrc.org/
Mesa	http://www.mesa3d.org/
Squid Cache	http://www.squid-cache.org/
KDElibs (KDE)	http://www.kde.org/
gedit	http://www.gnome.org/projects/gedit/
Evolution	http://www.gnome.org/projects/evolution/
Kontact	http://kontact.org/
KDE PIM	http://pim.kde.org
Advanced Linux Sound Architecture (ALSA)	http://www.alsa-project.org/
Wireshark	http://www.wireshark.org
OpenSSL	http://www.openssl.org/
GIMP	http://www.gimp.org/
NetBeans IDE	http://www.netbeans.org
Koha Library Automation Package	http://www.koha-community.org
openSUSE Linux	http://www.opensuse.org/
Doxygen	http://doxygen.org/
libcurl	http://curl.haxx.se/libcurl
GStreamer	http://github.com/zaheerm/gst-plugins-good
GNOME	http://www.gnome.org/
Insight Toolkit	http://www.itk.org
zsh	http://zsh.sourceforge.net/
Nautilus	https://wiki.gnome.org/Apps/Nautilus
X.Org	http://www.x.org/wiki/
Mozilla Core	http://www.ahrcloud.com
MariaDB	http://mariadb.org/
CMake	http://www.cmake.org
LibreOffice	http://www.libreoffice.org
ALT Linux	http://www.altlinux.org
ParaView	http://www.paraview.org
GTK+	http://www.gtk.org/
Poedit	http://www.poedit.net/
Bugzilla	http://www.bugzilla.org/
Enlightenment (window manager)	http://www.enlightenment.org
FFmpeg	http://www.ffmpeg.org/
GLib	http://library.gnome.org/devel/glib/
PEAR	http://pear.php.net/
Ruby	http://www.ruby-lang.org/
GnuCash	http://www.gnucash.org/
phpMyAdmin	http://www.phpmyadmin.net/
Mono	http://www.mono-project.com
SWIG	http://www.swig.org
SWT (Standard Widget Toolkit)	http://www.eclipse.org/swt/
Checkstyle	http://checkstyle.sourceforge.net
Eclipse Java Development Tools (JDT)	http://www.eclipse.org/jdt/
Eclipse Platform Project	http://www.eclipse.org/eclipse/platform-ui/
Natural Language Toolkit (NLTK)	http://www.nltk.org
Ekiga	http://ekiga.org/
Boost C++ Libraries	http://www.boost.org
Kate (KDE)	http://kate-editor.org
Devhelp	http://live.gnome.org/devhelp
Arch Linux Packages	http://www.archlinux.org
SPIP	http://www.spip.net
GNOME Terminal	https://help.gnome.org/users/gnome-terminal/stable/
ScummVM	http://www.scummvm.org/
Anjuta DevStudio	http://anjuta.org
BlueZ	http://www.bluez.org/
Eye of GNOME	http://www.gnome.org/projects/eog
Tor	http://www.torproject.org/
Fedora Packages	http://fedoraproject.org
Haiku	http://www.haiku-os.org
Stellarium	http://stellarium.org/
Totem	http://projects.gnome.org/totem/
Rhythmbox	http://www.gnome.org/projects/rhythmbox/
Gentoo Linux	http://www.gentoo.org/
CDT (Eclipse)	http://www.eclipse.org/cdt/
JRuby	http://www.jruby.org
eZ Publish	http://share.ez.no
VLC media player	http://videolan.org/
Equinox	http://www.eclipse.org/equinox/
Epiphany	http://www.gnome.org/projects/epiphany/
Thunderbird	http://mozilla.org/thunderbird/
GeoTools	http://geotools.org
PyPy	http://pypy.org
KDE	http://www.kde.org
apt - Advanced Package Tool	https://wiki.debian.org/Apt
Moodle	http://git.moodle.org/gw?p=moodle.git
Calligra Suite	http://www.calligra.org
QGIS	http://qgis.org/
Mozilla Firefox	http://www.firefox.com/
coreboot	http://www.coreboot.org/Welcome_to_coreboot
Tiki Wiki CMS Groupware	http://tiki.org
Apache Maven 2	http://github.com/apache/maven-archetype
Plone	http://plone.org
Superior Lisp Interaction Mode for Emacs	http://common-lisp.net/project/slime/
Kodi	http://kodi.tv
MythTV	http://www.mythtv.org
systemd	http://www.freedesktop.org/wiki/Software/systemd
GeoServer	http://www.geoserver.org
Groovy	http://groovy.codehaus.org/
Blender	http://www.blender.org/
MySQL	http://www.mysql.com/
iproute2	http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2
MonoDevelop	http://www.monodevelop.com
Hibernate	http://www.hibernate.org/subprojects/ogm
NetworkManager	http://www.gnome.org/projects/NetworkManager/
NLog - Advanced .NET Logging	http://nlog-project.org/
GParted	http://gparted.org/
Seahorse	http://www.gnome.org/projects/seahorse/
Glade User Interface Designer	http://glade.gnome.org/
Jenkins	http://jenkins-ci.org/
IntelliJ IDEA Community Edition	http://www.jetbrains.org
Ruby on Rails	http://rubyonrails.org
BusyBox	http://busybox.net/
Evince	http://projects.gnome.org/evince/
DokuWiki	http://www.dokuwiki.org/
Linux NTFS file system support	http://www.linux-ntfs.org/
KVM	http://kvm.qumranet.com/kvmwiki
Battle for Wesnoth	http://wesnoth.org/
Git	http://git-scm.com/
SPIP-Zone	http://zone.spip.org/trac/spip-zone/
Mercurial	http://mercurial.selenic.com/
Hibernate Entity Manager	http://entitymanager.hibernate.org/
Racket	http://racket-lang.org/
RubyGems	http://rubygems.org
SQLAlchemy	http://www.sqlalchemy.org/
cabal	http://haskell.org/cabal/
U-Boot	http://www.denx.de/wiki/U-Boot/WebHome
WebKit	http://webkit.org
OpenEmbedded	http://openembedded.org
Yocto Project	http://www.yoctoproject.org
matplotlib	http://matplotlib.org/
Symfony	http://www.symfony.com/
Meld	http://meldmerge.org/
Haxe	http://haxe.org/
FreeSWITCH	http://www.freeswitch.org/
Geany	http://geany.org/
collectd	http://collectd.org/
Gramps	http://gramps-project.org
phpBB Forum Software	http://www.phpbb.com/
HAProxy	http://www.haproxy.org/
fail2ban	http://www.fail2ban.org/wiki/index.php/Main_Page
NumPy	http://numpy.scipy.org
Scala	http://www.scala-lang.org/
dpkg	http://wiki.debian.org/Teams/Dpkg/
Nette Framework	http://nette.org
Inkscape	http://www.inkscape.org
Phing	http://www.phing.info/
jBPM	http://jbpm.org
JBoss Drools	http://www.jboss.org/drools
Bitbake	http://developer.berlios.de/projects/bitbake/
Zotero	http://www.zotero.org/
Lutece	http://www.lutece.paris.fr
OTRS	http://www.otrs.com/
Sage: Open Source Mathematics Software	http://sagemath.org
Rockbox	http://rockbox.org
Liferay Portal	http://liferay.com
TYPO3 CMS	http://typo3.org
Vala	http://live.gnome.org/Vala
pylint	http://pylint.org
The LLVM Compiler Infrastructure	http://llvm.org/
libvirt	http://libvirt.org
TinyMCE	http://tinymce.moxiecode.com
Django	http://www.djangoproject.com/
PHPUnit	http://www.phpunit.de/
OpenStreetMap	http://www.openstreetmap.org/
SymPy	http://sympy.org
Xen Project (Hypervisor)	http://www.xenproject.org
Eclipse Mylyn	http://www.eclipse.org/mylyn/
PHP_CodeSniffer	http://pear.php.net/package/PHP_CodeSniffer
Sakai LMS (core)	http://www.sakaiproject.org/
Spring Framework	http://github.com/SpringSource/spring-framework
Joomla!	http://www.joomla.org/
Marble	http://edu.kde.org/marble/
LXDE	http://lxde.org
Pygments	http://pygments.org/
OpenLayers	http://openlayers.org/
The MacPorts Project	http://www.macports.org/
calibre	http://calibre-ebook.com/
Grails	http://grails.org
Alfresco Content Management	http://www.alfresco.com
util-linux	https://github.com/karelzak/util-linux
jQuery	http://jquery.com/
Vaadin	http://vaadin.com/
Cython	http://www.cython.org/
Dojo Toolkit	http://dojotoolkit.org/
MediaWiki	https://www.mediawiki.org/wiki/MediaWiki
Second Life Viewer	http://www.secondlife.com/
Munin	http://munin-monitoring.org/
Odoo	https://www.odoo.com
Mozilla Calendar	http://www.mozilla.org/projects/calendar/
KDevelop	http://kdevelop.org/
ZNC	http://znc.in
Werkzeug	http://werkzeug.pocoo.org/
cppcheck	http://cppcheck.sourceforge.net/
Wicket Stuff	http://wicketstuff.org
Drush	http://drupal.org/project/drush
Sphinx documentation builder	http://sphinx-doc.org/
Piwik	http://piwik.org
JDownloader	http://www.jdownloader.org
SeaMonkey	http://www.seamonkey-project.org/
Empathy	http://live.gnome.org/Empathy
SilverStripe	http://www.silverstripe.org
PulseAudio	http://pulseaudio.org
LLVM/Clang C family frontend	http://clang.llvm.org/
Pylons	http://pylonsproject.org
MongoDB	http://www.mongodb.org/
Mockito	https://github.com/mockito/mockito
Doctrine	http://www.doctrine-project.org
Pacman	http://www.archlinux.org/pacman/
MAME - Multiple Arcade Machine Emulator	http://mamedev.org/
Rubinius	http://rubini.us/
Apache Camel	http://camel.apache.org/
OpenJDK	http://openjdk.java.net/
Buildbot	http://buildbot.net/trac
MPD	http://sourceforge.net/projects/musicpd
Tracker	http://projects.gnome.org/tracker/
org-mode	http://orgmode.org
Sass	http://sass-lang.com/
WPA/WPA2/IEEE 802.1X Supplicant	http://hostap.epitest.fi/wpa_supplicant/
Go programming language	http://golang.org/
Apache CouchDB	http://couchdb.apache.org/
Qt 4	http://qt-project.org/
Apache CXF	http://cxf.apache.org/
CakePHP	http://cakephp.org
CKeditor WYSIWYG editor	http://ckeditor.com/
SciPy	http://www.scipy.org
gitg	http://trac.novowork.com/gitg/
Banshee	http://banshee-project.org
OGRE	http://www.ogre3d.org
Chromium (Google Chrome)	http://code.google.com/chromium/
Gradle	http://www.gradle.org/
Netty Project	http://netty.io/
Sinatra	http://www.sinatrarb.com
Chef	http://www.opscode.com/chef
Gerrit Code Review	http://code.google.com/p/gerrit
GNOME Shell	http://live.gnome.org/GnomeShell
Git Extensions	http://code.google.com/p/gitextensions
Qt Creator	http://qt-project.org/
Kohana v3	http://kohanaframework.org/
Android	http://www.android.com
JUnit	http://www.junit.org
PCSX2	http://pcsx2.net/
Shotwell	https://wiki.gnome.org/Apps/Shotwell
Redis	http://redis.io/
Cassandra	http://cassandra.apache.org/
PhoneGap	http://phonegap.com/
Trinity Core	http://www.trinitycore.org
Icinga	http://www.icinga.org
CyanogenMod	http://www.cyanogenmod.com/
Rygel	http://live.gnome.org/Rygel
QEMU	http://www.qemu.org/
Trinity Core2	http://www.trinitycore.org
Pitivi	http://github.com/jhoolmans
Openfire	http://www.igniterealtime.org/projects/openfire/
Apache Hadoop	http://hadoop.apache.org/core/
akka	http://akka.io
JGit	http://www.eclipse.org/jgit/
Homebrew	https://github.com/Homebrew/homebrew-apache
Oh My Zsh	http://github.com/robbyrussell/oh-my-zsh
ehcache	http://www.ehcache.org/
EGit	http://www.eclipse.org/egit/
node.js (NodeJs)	http://nodejs.org
Thunar	http://www.xfce.org
Selenium	http://seleniumhq.org/
Arquillian	http://jboss.org/arquillian
Erlang	http://www.erlang.org
YUI	http://yuilibrary.com/
Gunicorn	http://gunicorn.org
CoffeeScript	http://www.coffeescript.org/
Clementine Music Player	https://github.com/clementine-player/Clementine
scikit learn	http://scikit-learn.org
Processing	http://processing.org/
Vagrant	http://vagrantup.com/
Qt 5	http://www.qt-project.org/
Yii PHP Framework	http://www.yiiframework.com
Zend Framework	http://framework.zend.com/
Apache Spark	http://spark.apache.org
Flask	http://flask.pocoo.org/
OsmAnd	http://www.osmand.net
ownCloud	http://ownCloud.org
Open Computer Vision Library (OpenCV)	http://opencv.org/
phpDocumentor	http://www.phpdoc.org
IPython	http://ipython.org/
RSpec	http://rspec.info/
OpenStack	http://www.openstack.org/
OpenStack Nova	https://launchpad.net/nova
Apache CloudStack	https://github.com/apache/incubator-cloudstack
AngularJS	http://angularjs.org/
GWT (formerly Google Web Toolkit)	https://github.com/google-web-toolkit/gwt
Facter	http://puppetlabs.com/puppet/related-projects/facter/
salt	http://saltstack.org
jMonkey Engine	http://jmonkeyengine.org
Puppet	http://puppetlabs.com/puppet/
Play! framework	http://www.playframework.org/
Elasticsearch	http://www.elasticsearch.com
Bootstrap (Twitter)	http://twitter.github.com/bootstrap/
Apache OpenOffice	http://www.openoffice.org/
GlassFish	https://glassfish.dev.java.net/
Propel	http://propelorm.org
JabRef	http://jabref.sourceforge.net
CodeIgniter	http://www.codeigniter.com/
GNOME Boxes	http://live.gnome.org/Boxes
GitLab	https://www.gitlab.com/gitlab-ce/
TiddlyWiki	http://www.tiddlywiki.org
Fish shell	https://github.com/fish-shell/fish-shell
Ansible	http://ansible.com
Simple Machines Forum	http://www.simplemachines.org/
FontForge	http://www.fontforge.org
libgdx	http://libgdx.badlogicgames.com
py-pandas	http://pandas.sourceforge.net/
javascript	https://github.com/airbnb/javascript
EasyTAG	https://wiki.gnome.org/Apps/EasyTAG
docker	http://docker.io
Capistrano	http://capistranorb.com/