Skip to content

Latest commit

 

History

History
88 lines (70 loc) · 3.23 KB

README.md

File metadata and controls

88 lines (70 loc) · 3.23 KB
Raw

Scalability Analysis Tools

Build Status codecov PyPI version

sca_tools analyzes software scalability benchmarking experiments, specifically measurements of throughput as a function of applied load or concurrency. It is a Python library and set of command line utilities and that will help you:

  • Model and quantify throughput bottlenecks in your application
  • Capacity plan
  • Compare performance benchmarks for regressions

We rely on Neil Gunther's Universal Scalability Law as a model and lmfit to perform model fitting. sca_tools differs from existing implementations (see Related Work) in a few ways:

  • Emphasis on robust parameter estimation
  • Support for propagation of parameter uncertanties when computing derived quantities, such as latency, queue size, etc.
  • Support for experimental measurement uncertainty
  • Written using Scientific Python libraries

This is alpha software: use it at your own risk (e.g., don't use it to make business decisions). It's very much a work in progress, but currently includes:

  • Fitting routines to the USL and basic reporting around goodness-of-fit for USL's convention and coherence parameters.
  • Graph outputs for best fit model, best fit model confidence intervals, fit data, residuals, etc.
  • Command line tools to produce these models from CSV data, as well as manipulate and aggregate CSV data from computer experiments.

Usage

The fixtures/ directory contains the SPEC SDM91 load-througput benchmark ported from Stefan Möding's R implementation of USL.

> python sca_tools/sca_fit.py --model_type usl fixtures/specsdm91.csv

----- Summary -----

[[Model]]
    Model(_usl_func)
[[Fit Statistics]]
    # function evals   = 41
    # data points      = 7
    # variables        = 3
    chi-square         = 27453.720
    reduced chi-square = 6863.430
    Akaike info crit   = 63.920
    Bayesian info crit = 63.758
[[Variables]]
    lambda_:   89.9954927 +/- 14.21296 (15.79%) (init= 1000)
    sigma_:    0.02772863 +/- 0.009121 (32.90%) (init= 0.1)
    kappa:     0.00010437 +/- 1.99e-05 (19.04%) (init= 0.001)
[[Correlations]] (unreported correlations are <  0.100)
    C(lambda_, sigma_)           =  0.964
    C(sigma_, kappa)             = -0.467
    C(lambda_, kappa)            = -0.243

Graphically:

Throughput model

Related Work

Citations

  • Neil J. Gunther. Guerrilla Capacity Planning: A Tactical Approach to Planning for Highly Scalable Applications and Services. Springer, Heidelberg, Germany, 1st edition, 2007.
  • Baron Schwartz. Practical Scalability Analysis with the Universal Scalability Law. VividCortex, November 2015.

License

Copyright © 2017 Bhaskar Mookerji

Distributed under the Apache License 2.0