Allow to use significantly more clients #944
Conversation
danielmitterdorfer
added
enhancement
|
@elasticmachine run tests |
1 similar comment
|
@elasticmachine run tests |
There was a problem hiding this comment.
This is a surprisingly smooth addition leveraging the actor system to allow
us increase the amount of clients!
Unrelated to this specific PR, but since #935 where we introduced the MmapSource it appears that with ~4000 clients (on a single machine) there reported memory usage (from top and pmap) is ~1.1TB, so it's likely we need to look at this in more detail in a follow up.
| @@ -431,12 +434,19 @@ def prepare_benchmark(self, t): | |||
| self.prepare_telemetry(es_clients) | |||
| self.target.on_cluster_details_retrieved(self.retrieve_cluster_info(es_clients)) | |||
| for host in self.config.opts("driver", "load_driver_hosts"): | |||
| host_config = { | |||
| # for simplicity we assume that all benchmark machines have the same specs | |||
| self.config, self.client_id, task, schedule, self.sampler, self.cancel, self.complete, self.abort_on_error) | ||
| self.logger.info("Worker[%d] is executing tasks at index [%d].", self.worker_id, self.current_task_index) | ||
| # allow to buffer more events than by default as we expect to have way more clients. | ||
| self.sampler = Sampler(start_timestamp=time.perf_counter(), buffer_size=65536) |
There was a problem hiding this comment.
Does it make sense to declare static values like 65535 via a module variable?
| self.client_id, self.sub_task, self.schedule, es, self.sampler, self.cancel, self.complete, self.abort_on_error) | ||
| final_executor = AsyncProfiler(async_executor) if self.profiling_enabled else async_executor | ||
|
|
||
| aws = [] |
There was a problem hiding this comment.
nit: it's not easy to understand what this variable holds judging by its name (plus the name inadvertently conjures thoughts about cloud)
|
@elasticmachine test this please |
With this commit we provide a URL object when issuing a request instead of a string representation to avoid expensive string parsing in aiohttp. In our tests this has reduced the client side overhead by about one millisecond which is important when benchmarking queries which finish within single-digit milliseconds. Relates elastic#944
With this commit we provide a URL object when issuing a request instead of a string representation to avoid expensive string parsing in aiohttp. In our tests this has reduced the client side overhead by about one millisecond which is important when benchmarking queries which finish within single-digit milliseconds. Relates #944
With this commit we properly determine the most recent sample per client by looping over all clients instead of extracting only one sample. The reason this was working before is that each processes only simulated one client but with elastic#944 each worker can simulate multiple clients so we need to check all samples to determine the most recent one.
With this commit we properly determine the most recent sample per client by looping over all clients instead of extracting only one sample. The reason this was working before is that each processes only simulated one client but with #944 each worker can simulate multiple clients so we need to check all samples to determine the most recent one.
#963 to addresses this by sharing the memory-mapped file within each process. This means we memory-map a data file only once per worker (process) instead of once per client. |
With this commit we change how clients are assigned to worker processes in the
load generator. While historically Rally has assigned one worker (process) to
one client, with the changes done in #935 we can assign multiple clients to a
single worker process now and run the clients in an asyncio event loop. This
allows us to simulate many more clients than what is possible the process-based
approach which is very heavy-weight.
By default Rally will only create as many workers as there are CPUs available on
the system. This choice can be overridden in Rally's config file with the
configuration setting
available.coresin the sectionsystemto allocate moreor less workers. If multiple load generator machines note that we assume that
all of them have the same hardware specifications and only take the coordinating
machine's CPU count into account.