Limiting IOPS, possible?

[nsoubelet]

Hello guys,
It seems like I am the first one writing something in this discussions section, so I hope it is welcomed to you.

I would like to ask for your help with the following question: is it currently possible to limit IOPS from this lib (NFSServerV41)?
I know I could do that directly in the underlying VirtualFileSystem implementation, but I do not want to reinvent it if it was already implemented.

Many thanks!

[kofemann]

Hi @nsoubelet, Congratulations for openning the very first discussion topic for nfs4j project!

We don't have any kind of limits implemented. I general, can you elaborate what you expect by those limits? DoS attack protection from single use? Or IP? Or specific operation? Or only IO? Should be this a hard limit, or fare-share? Any of mentioned cases can be solved on different levels: RPC, NFS, VFS.

Thanks a lot for quite exciting subjec. Welcome to discussions!

[nsoubelet]

Hello @kofemann, many thanks for such a nice welcome!

Regarding the question, let me elaborate a bit.

I originally said IOPs, but thinking carefully about it what I really need to achieve is a combination of things, that is, IOPs and Throughput.
The idea is to start the NFS server indicating the maximum operations per unit of time (seconds) and bytes "operated" per unit of time (MB/s). I did something really naive which is registering operations and bytes in LongAccumulator's and then rejecting operations directly in the VFS.
It works, impacting a bit on the overall performance though.

However, and this is the difficult part, I don't really want to reject operations when those accumulators reach the threshold (before they are reset) but slow operations down.
I am aware that most NFS servers do not support those options, and that it is commonly implemented by rate limiting in the networking layer.

I presume there is not such a thing in nfs4j (please confirm) however, if you had to implement it, where do you place that logic? Maybe rate limiting in the RPC?

Many thanks!

[kofemann]

Hi @nsoubelet,

I think those are two different limits that have to be addressed. Thus I don't think you can address it in one go. Moreover, a per-client limiting mechanism will be a nice addition to protect regular clients from a greedy one.

To address general IOPS limits the simplest way will be to handle it at the RPC level. A stupid implementation can be as following:

import java.util.concurrent.ThreadPoolExecutor;
import com.google.common.util.concurrent.RateLimiter;

public class RateLimitingExecutor extends ThreadPoolExecutor {

    private final RateLimiter limiter;

    public RateLimitingExecutor(double rps, ...) {
        super(...);
        limiter = RateLimiter.create(rps);
    }

    @Override
    protected void beforeExecute(Thread t, Runnable r) {
        super.beforeExecute(t, r);
        limiter.acquire();
    }
}

and configure RPC later to use worker thread pool with given rate limit:

   OncRpcSvc nfsSvc =
        new OncRpcSvcBuilder()
            .withPort(2049)
            .withTCP()
            .withAutoPublish()
            .withWorkerThreadExecutionService( new RateLimitingExecutor(5000.0, ...))
            .build();

A more sophisticated solution should use adaptive thread pool that will dynamically adjust limits based on response time. I had a look at https://github.com/Netflix/concurrency-limits, but had no time to play with it.

I am pretty sure that both solutions can be applied to IO bandwidth in the file system itself. The concurrency-limits has a concept of partitions, which can be used by nfs server (in NFSServerV4#NFSPROC4_COMPOUND_4) to limit number of concurrent reads and writes. This of course can be achieved in an VFS implementation as well.

Does it makes sense?

[nsoubelet]

Hello @kofemann

Many thanks for answering.

That makes a lot of sense, good approach. I will investigate further how Netflix's library can be implemented in VFS to limit Throughput so I touch the core as few as possible.

If you are interested in the topic, once I am able to effectively solve this, I can share the approach.

[kofemann]

I am very interested in your results! So please keep up informed.