Benchmark suite

[immerrr]

This should close #184.

[kmike]

It could be better to use Twisted (or some other async framework) because SimpleHTTPServer is single-threaded, and Splash can download multiple resources in parallel. I wonder if it is important for tests, maybe not.

With Twisted we can also simulate conditions like non-responding servers, delayed responses, etc.

[kmike]

I think to fix tests it could be enough to add benchmarks here: https://github.com/scrapinghub/splash/blob/master/conftest.py

[kmike]

It is better to allow to set Splash server address from the command line. This way it'll be easier to compare performance of different versions of Splash, or performance of dockerized Splash (maybe several dockerized Splash'es which use different base operating systems and/or qt/... versions).

We should really allow to run the same benchmark against different Splash versions. If the benchmark can only be executed against current Splash version we can't just checkout to older Splash version and run a benchmark again - checking out old Splash could also change the benchmark version, so we'll run a different benchmark.

[immerrr]

@kmike is there anything else that prevents merging this?

[kmike]

I can't get it work :)

• splash.benchmark and splash.tests imports are undefined when you execute code from benchmarks folder because these packages are not distributed with Splash.
• if splash.benchmark and splash.tests are added to packages in setup.py download_sites.py raises an error while trying to start proxy server (Splash starts fine with python -m splash.server):

[-] Starting factory <splash.proxy_server.SplashProxyServerFactory instance at 0x114109710>
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/kmike/envs/splash/lib/python2.7/site-packages/splash-1.4-py2.7.egg/splash/tests/mockserver.py", line 725, in <module>
    run(opts.http_port, opts.https_port, opts.proxy_port, not opts.quiet)
  File "/Users/kmike/envs/splash/lib/python2.7/site-packages/splash-1.4-py2.7.egg/splash/tests/mockserver.py", line 698, in run
    sslport = reactor.listenSSL(sslport_num, factory, ssl_factory())
  File "/Users/kmike/envs/splash/lib/python2.7/site-packages/splash-1.4-py2.7.egg/splash/tests/mockserver.py", line 673, in ssl_factory
    return ssl.DefaultOpenSSLContextFactory(pem, pem)
  File "/Users/kmike/envs/splash/lib/python2.7/site-packages/twisted/internet/ssl.py", line 104, in __init__
    self.cacheContext()
  File "/Users/kmike/envs/splash/lib/python2.7/site-packages/twisted/internet/ssl.py", line 113, in cacheContext
    ctx.use_certificate_file(self.certificateFileName)
OpenSSL.SSL.Error: [('system library', 'fopen', 'No such file or directory'), ('BIO routines', 'FILE_CTRL', 'system lib'), ('SSL routines', 'SSL_CTX_use_certificate_file', 'system lib')]

• I'm not sure how I did it, but download_sites.py was working before I installed Splash, BUT it put nothing to splash/benchmark/sites/localhost_8806/ folder, so benchmark found zero websites.

I think we shouldn't distribute splash.benchmark with other Splash code (it shouldn't be installed by installing Splash). What does it take to move benchmarks folder to the root?

[immerrr]

it shouldn't be installed by installing Splash

if not, I'm not sure about organizing the code. The necessary subset of SplashServer functionality can probably be copy-pasted. But for the rest, there are three separate runnable scripts there: the downloader, the file server & the benchmark runner. they all use code from the file server module, so at least that module must be importable.

[immerrr]

it worked ok for me in pip install -e . mode.