sca_tools analyzes software scalability benchmarking experiments,
specifically measurements of throughput as a function of applied load
or concurrency. It is a Python library and set of command line
utilities and that will help you:
- Model and quantify throughput bottlenecks in your application
- Capacity plan
- Compare performance benchmarks for regressions
We rely on Neil Gunther's
Universal Scalability Law
as a model and lmfit to perform
model fitting. sca_tools differs from existing implementations (see
Related Work) in a few ways:
- Emphasis on robust parameter estimation
- Support for propagation of parameter uncertanties when computing derived quantities, such as latency, queue size, etc.
- Support for experimental measurement uncertainty
- Written using Scientific Python libraries
This is alpha software: use it at your own risk (e.g., don't use it to make business decisions). It's very much a work in progress, but currently includes:
- Fitting routines to the USL and basic reporting around goodness-of-fit for USL's convention and coherence parameters.
- Graph outputs for best fit model, best fit model confidence intervals, fit data, residuals, etc.
- Command line tools to produce these models from CSV data, as well as manipulate and aggregate CSV data from computer experiments.
The fixtures/ directory contains the SPEC SDM91 load-througput
benchmark ported from Stefan Möding's
R implementation of USL.
> python sca_tools/sca_fit.py --model_type usl fixtures/specsdm91.csv
----- Summary -----
[[Model]]
Model(_usl_func)
[[Fit Statistics]]
# function evals = 41
# data points = 7
# variables = 3
chi-square = 27453.720
reduced chi-square = 6863.430
Akaike info crit = 63.920
Bayesian info crit = 63.758
[[Variables]]
lambda_: 89.9954927 +/- 14.21296 (15.79%) (init= 1000)
sigma_: 0.02772863 +/- 0.009121 (32.90%) (init= 0.1)
kappa: 0.00010437 +/- 1.99e-05 (19.04%) (init= 0.001)
[[Correlations]] (unreported correlations are < 0.100)
C(lambda_, sigma_) = 0.964
C(sigma_, kappa) = -0.467
C(lambda_, kappa) = -0.243Graphically:
- Neil J. Gunther. Guerrilla Capacity Planning: A Tactical Approach to Planning for Highly Scalable Applications and Services. Springer, Heidelberg, Germany, 1st edition, 2007.
- Baron Schwartz. Practical Scalability Analysis with the Universal Scalability Law. VividCortex, November 2015.
Copyright © 2017 Bhaskar Mookerji
Distributed under the Apache License 2.0