Load/Stress Tests #6
Comments
|
Tried using benchfella to benchmark the cache. I was wondering why this 500,000 does not change on a laptop 2 cores or a desktop 4+ cores. https://github.com/fire/database_cache_benchmark/blob/master/bench/database_bench.exs#L13 |
|
This is using cockroachdb, but it should be compatible with postgresql. |
|
@fire thanks for the bench! I had added a simple bench to Nebulex using the same library benchfella, but only for local and distributed adapter, without Ecto, so it is an interesting and complementary bench test to Nebulex. On other hand, I don't know much about benchfella, at least not in terms of its implementation, so I'm not sure if benchfella is suitable to do load tests, spawning a bunch of processes to hit the cache and then be able to measure not only latency but also throughput (how many writes and reads per second is able to handle). For example, I did some load tests for shards using basho_bench, and that is precisely what I want to be able to do – check out this blog post, the performance tests section. |
|
Are you familiar with using basho_bench to test ecto (with postgresql / cockroachdb) driver? |
|
I'm familiar with basho_bench, but the tricky part is to integrate Elixir and Ecto within the project, since basho_bench doesn't works with rebar3 (AFAIK), then you cannot use the rebar3_elixir_compile plugin, and that makes it more difficult. However, I think it would be easy to add rebar3, and I'd also remove unnecessary dependencies, try to prune the project a bit, anyways, we should try to do it. |
|
There's two parts to my benchmark. A)The ecto schema / ecto caching and B) the phoenix json api. Only the ecto schema part is used in the benchfella benchmark. So I can delete the phoenix framework, but wish keep it because one of the goals is to either use thrift (elixir-thrift) or phoenix to provide access to the ecto database / caching. |
|
Absolutely, that's fine, you should keep everything that make sense to you. When I said "try to prune the project a bit" I was referring to basho_bench, because it brings with a lot of dependencies, which are not necessary to do the test. What I'd like to do is try to use basho_bench against Nebulex, I'll try to make it soon. |
|
Are you using a fork of basho_bench? I could look at it after work hours. |
|
Have you looked at basho bench lately? There are compatibility fixes I believe to erlang 20 or at least there's a riak working group now. |
|
@fire no, but I will, it is a very good news, I'll take a look at it and try to do this load testing soon. Thanks :) !! |
|
If you had problems, this is the one that has erlang 19 compatibility. https://github.com/mrallen1/basho_bench/blob/mra-rebar3/README.org#quick-start |
|
Ping! |
|
Hi @fire, my plan is start work on this issue this week (actually is the next one in my list), and of course I'll start checking out the link you sent me, as soon as I have some result I'll let you know. BTW, have you tried it out already? |
|
Was able to build it in Elixir and erlang 20, but didn't find time understanding how the drivers worked. That's where I stopped. |
|
Ok, basically you have to provide two things to
-module(basho_bench_nebulex).
-export([
new/1,
run/4
]).
-define(nbx_cache, 'Elixir.NbxCache').
new(_State) ->
{ok, #{}}.
run(get, KeyGen, _ValueGen, State) ->
Key = KeyGen(),
_ = ?nbx_cache:get(Key),
{ok, State};
run(put, KeyGen, ValueGen, State) ->
?nbx_cache:set(KeyGen(), ValueGen()),
{ok, State}.In this driver we are testing two operations,
{mode, max}.
{duration, 5}.
{concurrent, 8}.
{driver, basho_bench_driver_shards}.
{operations, [{put, 1}, {get, 1}]}.
{key_generator, {int_to_bin_bigendian, {uniform_int, 1000}}}.
{value_generator, {fixed_bin, 100000}}.See that Now, the tricky part is, we have to create an elixir app for the Nebulex cache ( I was able to compile |
|
When I do a iex --werl -S mix run inside of https://github.com/fire/database_cache_benchmark I'm able to access the basho_bench functions. Inside of the elixir shell I have the ability to use both ecto and basho_bench. I'm not sure what do though. |
Seems to work, but it crashes on |
|
Added https://github.com/fire/database_cache_benchmark/blob/master/src/basho_bench_driver_shards.erl but still failing: |
|
Loading https://github.com/fire/database_cache_benchmark: Will insert and fetch random uuids. How do you use key_gen and value_gen in |
|
https://github.com/fire/database_cache_benchmark/blob/master/lib/basho_bench/basho_bench_nebulex_driver.ex is the nebulex driver written in Elixir. |
|
fire/database_cache_benchmark@eca59d2 Has my last work on trying to get uuid_v4 generation. Reminder: |
|
Error log: |
|
Hi @fire, well actually you did a good progress, you took other approach (which I like it), instead of add |
|
@fire good news, I was able to run the basho_bench tests with your repo (database_cache_benchmark), I found some issues:
Replace
It would be something like: defmodule BashoBench.Nebulex.Driver do
def new(_state) do
{:ok, {}}
end
def run(:put, key_gen, value_gen, state) do
changeset = Hello.World.changeset(%Hello.World{}, %{id: key_gen.(), random_number: value_gen.()})
case Hello.CacheableRepo.insert(changeset) do
{:ok, _schema} -> {:ok, state}
{:error, reason} -> {:error, reason, state}
end
end
def run(:get, key_gen, value_gen, state) do
_ = Hello.CacheableRepo.get(Hello.World, key_gen.())
{:ok, state}
end
endTry that and let me know if it works for you, but I was able to run load tests against Nebulex finally, thanks to your help :) !!! I got with a humble machine (Mac OSX Sierra, Dual Core, 4 GB) about ~150K ops/seg, which is great!! Well, try to run your test and let me know how it goes! |
|
Nebulex ran on my machine but I need to generate uuids. |
|
Instructions: |
|
I modified the schema so it works with integer keys, but I wanted uuid results. |
|
Results on Windows and my desktop. Do not use this for scientific reference. {operations, [{put, 1}, {get, 1}]}. Currently trying: {operations, [{put, 2}, {get, 8}]}. |
{mode, max}.
{duration, 5}.
{concurrent, 8}.
{driver, 'Elixir.BashoBench.Nebulex.Driver'}.
{operations, [{put, 2}, {get, 8}]}.
{key_generator, {pareto_int, 1000}}.
{value_generator, {fixed_bin, 100000}}.Results on Windows and my desktop. Do not use this for scientific reference.
|
|
Previous results are wrong, the puts were failing. |
|
Edit: Note that these are using the on_conflict database functionality. |
|
These were my results, but only testing the cache, no Ecto (no DB access), I wanted to get some metrics from the Nebulex cache itself first. Keep in mind that you are testing Ecto with Nebulex, which means that for reads you fetch the data drom the DB sometimes, and for writes you always hit both, the cache and the DB. But well, with your tests we have metrics from only Results (Mac OSX 10.11.4, 2.2 GHz Intel Core i7, RAM 16 GB):
|
|
I recommend we pick a few databases with standard hardware try out nebulex_ecto. |
|
@fire 100% agreed, that would be great. Meanwhile, I'll create a separated project for these load tests (only for |
|
I've created a new project to run bench/load tests against |
|
Interesting results trying to copy ycsb's Hint. It mostly gets cached. |
|
@fire based on you suggestion (pick a few databases with standard hardware try out |
|
That's a good suggestion. You can close this issue. |
Make load/stress tests for Nebulex, including the local generational cache and distributed cache either. This can be done in a separated repo (e.g.:
nebulex_bench) and documenting results in a blog post maybe.The text was updated successfully, but these errors were encountered: