adding nosqlbench test scripts

[jeffreyscarpenter]

No description provided.

[ivansenic]

Have some questions and some improvement suggestions..

[ivansenic]

can we add response assertions for everything? We pretty much know that everything must be 200, that schema change commands respond with "ok": 1 (or smth like that), that insert docs would return inserted id, etc.. Also we should confirm if possible that there are no errors in the returned body.

[jeffreyscarpenter]

No, we should not do this. The response assertions are used to determine when the script should abort. There is a very limited case where we'd want to do this - if the setup (schema creation) fails.

[ivansenic]

I don't think this is true, assertions do not stop the script. The reason why I think that's important, is to be certain we are getting time results for correctly executed use-cases and not for error cases. Cause often error cases would actually be faster. Think about missing coordinator, this test would run really fast.

[tatu-at-datastax]

One possible problem is if there is no check that reads should target documents actually written: if not, we'd get 404 (correctly) when trying to access non-existing documents. So return value depends on true existence of document(s). I agree it would be very useful to have sanity checks so that we do not accidentally "test" performance of broken tests (nothing found, ever, which can be fast).

[jshook]

I have some ideas on this, but I don't have any really prescriptive answers so far. I do think that the answers come more easily if you know exactly what you are testing for. i.e, correctness, performance, etc. But in this case, we generally want to verify that schema creation works all the time but we want to allow for some "empty reads" in performance tests. If you say instead that you are doing a correctness test, then you move more towards wanting to qualify each and every result.

While it is fairly easy to construct bindings which can be used for correctness assertions the second time you do it, the first time is often a learning exercise. But, it is quite doable. [You can use specific strategies for building bindings] so that you know which operations should return non-empty results and which ones should, or even how many results each operation should return.

So, my main question is, what exactly are you testing for in this case?

[ivansenic]

I am not saying specifically for the schema change responses, I am saying lets add it everywhere. But question is actually good, what do we want from these tests?

[jeffreyscarpenter]

As per Shooky's example, we do want to allow empty reads. These are performance tests, not correctness tests.

By default the tests do exit on error condition, but we can override this with the setting errors=count to get statistics on errors which we can then analyze.

According to the JSON API spec this service is going to return 200 for basically everything except 500 server error. So identifying error conditions per call will require more work. We'll probably need to do something like

ok-status: "200"
ok-body: "some regex"

IMO that's beyond the scope of this initial PR. I'd just like to get these checked in initially and begin running them regularly, and then we can improve the tests as we go.

[ivansenic]

400 is not ok, fix everywhere

[jeffreyscarpenter]

No

[jeffreyscarpenter]

The service never returns a 4xx anyway, according to current spec. It's 200 or 500.

[ivansenic]

this is currently not supported, do we want it merged?

[jeffreyscarpenter]

The JSON API did not generate an error on a test run. But I can comment out this step for now.

[ivansenic]

• what the hack, how is this not reporting an error? @maheshrajamani @tatu-at-datastax
• in addition, should this anyway be $in?

[ivansenic]

I am approving to unblock, as I will not be here tomorrow. I resolved all the conversations that were addressed. Few unresolved are still there, please resolve before merging.