Benchmarking Numbers on Android

[nitish24p]

WHAT

I wanted some clarification on the benchmarking numbers that have been posted here https://github.com/Tencent/MMKV/wiki/android_benchmark

Not sure if this is the right place to discuss this, but wanted to know if im doing something wrong. Im trying to store JSON objects around (900Kb) as 4 different keys and read them in a micro benchmark test. Now from seeing the code the initial MMAP size is 4096 for the device im running so had a few questions which are listed below.

Test Cases

I have 4 payloads each of 900Kb which when i set up the test i write to mmkv module and do a sync method to flush to disc, then run a test for reading these 4 responses then a test with 20 iterations to read these numbers. What i observed was decode string of mmkv module ends up taking a lot of time (Makes sense since Size is large). Attached code below

TESTCASE

    @Test()
    public void readLargeLeOneResponse(){
        final BenchmarkState state = mBenchmarkRule.getState();
        String homeResponse = "";
        while (state.keepRunning()) {
            homeResponse = mmkv.getString(BenchmarkMockData.LE_RESPONSE_KEY_HOME);

        }
        Log.d("BENCHMARK RESPONSE", beautyResponse);
    }

SETUP

    public static void setUpMMKVData(Context base) {
         MMKVModule instance = createInstance(base);
         LE_RESPONSE = BenchmarkMockData.getLeResponse(base);

         instance
                 .setString("foo", "bar")
                 .setString(BenchmarkMockData.LE_RESPONSE_KEY_HOME, LE_RESPONSE)

         // Flushes and writes to disk
         instance.kv.sync();
    }

Now the numbers i got for these are as below

MMKV: 4231997ns
SharedPrefs: 3868ns

Wanted to know if in general MMKV is a better option for smaller key value stores rather than such large data sets. On seeing the code base the mmap size is 4Kb on the device i run so its way more than what is being written.

Sorry if this wasnt the right place to open this but didnt know where this couldve been asked. Thought it was better than a github issue.

[nitish24p]

Also isnt the payload size an issue for this case where the decode / encode tends to be an issue in this case. Or are you saying by not calling sync at all, numbers should be better. If i dont call sync Can there be a case if data sanctity is missed? Also for the benchmark tests mentioned in general what was the size of the payload which you considered.

I ran a benchmark on small data points / primitive values and the results are still more than the benchmarks provided in the benchmarking doc.

[lingol]

Ignore it, I have mistaken the setup function for the benchmark loop.

[lingol]

Can you run profile and check which method costs the most time?

[nitish24p]

Sure.. I ran a StackSampling Trace for the above test case attaching screenshot below

Stack Sampling

Most of the time seems be contributed by **art_jni_trampoline** not sure what this does. Which inturn contributes to a large decodeString function

I can increase the stack sample rate and size if that would help. Currently its at the default provided by android studio

[lingol]

Most of the work is done inside MMKV's JNI code, your profiling should include JNI stacks to make it meaningful.

[nitish24p]

Sure this is the Cxx code sample trace that i could get

Highest contributor seems to be the art::JNI::NewStringUTF method

Do tell me if you want a deeper stack trace will increase sampling depth

[nitish24p]

@lingol
This is how im using the library am i doing anything overly wrong. I think the transformation of Java object to a Cxx string and the JNI interface seems to be a bottle neck. DO let me know

    private static void setUpMMkvData() {
        MMKVModule.setStringData(BenchmarkingStorage.FOO, BenchmarkingStorage.BAR);
        MMKVModule.setStringData(KEY, LARGE_RESPONSE); // 900Kb
    }

In MMKV Module

public class MMKVModule {
   static MMKVModule mmkvInstance = null;
    public MMKVModule(Context ctx) {
        MMKV.initialize(ctx);
        this.kv = MMKV.defaultMMKV();
        Log.d("MMKV SIZE", String.valueOf(this.kv.getValueActualSize("bytes")));

    }

     public static MMKVModule getInstance(Context ctx) {
        if(MMKVModule.mmkvInstance == null)
        {
            MMKVModule.mmkvInstance = new MMKVModule(ctx);
        }
        return MMKVModule.mmkvInstance;
    }

      public static void setStringData(String key, String val) {
        MMKVModule instance = getInstance();
        instance.setString(key, val);
    }

    public static void getStringData(String key) {
        MMKVModule instance = getInstance();
        instance.getString(key);
    }

}

TEST CASES

    private static void readTestsMMKV() {
        Log.d("\n BENCHMARKING: STARTING MMKV PREFS READ", "STARTING \n");
        long smallValueStartTime = System.nanoTime();
        MMKVModule.getStringData(FOO);
        Log.d("BENCHMARKING: EXECUTION TIME SMALL VALUE", String.valueOf(System.nanoTime() - smallValueStartTime));

        long largeValueStartTime = System.nanoTime();
        MMKVModule.getStringData(LE_RESPONSE_KEY);
        Log.d("BENCHMARKING: EXECUTION TIME LARGE VALUE ", String.valueOf(System.nanoTime() - largeValueStartTime));
        Log.d("BENCHMARKING: ENDING MMKV END", "ENDING \n");

    }

I followed your advice of removing any form of sync / async calls to flush data to disc, but seem to be seeing the same bottlenecks. Can you please tell me if im doing something entirely wrong. Or are these values expected

[nitish24p]

@lingol Is this reasoning correct, where majority of the bottleneck is happening at the JNI layer and the conversion from java string to Cxx string.