[Request] Realworld examples for benchmarking

[okuramasafumi]

We seek speed and benchmarking helps a lot. We currently have only one variation of benchmark, but we'd like to run benchmark against multiple cases so that Alba will be fast for a variety of scenarios.

If you know (preferably open source) examples of combinations of database structure and API, please let us know in this discussion so that we can implement it as a benchmark!

[estepnv]

Hey there, I'm fast_serializer maintainer. I wondered who's solution is faster 😈
So I ran collection serialization benchmark. single resource does not count IMO.

It shows that latest panko is a winner, in both speed and also it allocates twice less memory for your benchmarks.
Arguments ? 😁

What about advanced features like includes, conditions and inheritance?

Turn off GC while benchmarking. Also the pg and DB dependency is useless. We are talking about object to hash translations, we don't care about the DB. so I would rewrite benchmark to instantiate plain old ruby objects.

Here are my results

Warming up --------------------------------------
                alba     2.000  i/100ms
     fast_serializer     2.000  i/100ms
         alba_inline     2.000  i/100ms
                 ams     1.000  i/100ms
         blueprinter     2.000  i/100ms
            jbuilder     2.000  i/100ms
         jserializer     2.000  i/100ms
             jsonapi     2.000  i/100ms
 jsonapi_same_format     2.000  i/100ms
               panko     2.000  i/100ms
           primalize     1.000  i/100ms
               rails     1.000  i/100ms
       representable     1.000  i/100ms
          simple_ams     1.000  i/100ms
Calculating -------------------------------------
                alba     24.002  (±25.0%) i/s -    112.000  in   5.160636s
     fast_serializer     20.873  (±14.4%) i/s -    104.000  in   5.122118s
         alba_inline     23.873  (±20.9%) i/s -    114.000  in   5.041079s
                 ams     14.728  (±20.4%) i/s -     72.000  in   5.049764s
         blueprinter     16.065  (±24.9%) i/s -     76.000  in   5.167510s
            jbuilder     19.386  (±20.6%) i/s -     94.000  in   5.089092s
         jserializer     20.408  (±19.6%) i/s -    100.000  in   5.051080s
             jsonapi     19.754  (±30.4%) i/s -     90.000  in   5.328454s
 jsonapi_same_format     20.278  (±19.7%) i/s -     96.000  in   5.140373s
               panko     31.963  (±15.6%) i/s -    156.000  in   5.035554s
           primalize     20.330  (±24.6%) i/s -     94.000  in   5.056357s
               rails     14.129  (±14.2%) i/s -     69.000  in   5.028093s
       representable     12.012  (±16.6%) i/s -     59.000  in   5.084435s
          simple_ams     13.028  (±23.0%) i/s -     62.000  in   5.013094s

Comparison:
               panko:       32.0 i/s
                alba:       24.0 i/s - same-ish: difference falls within error
         alba_inline:       23.9 i/s - same-ish: difference falls within error
     fast_serializer:       20.9 i/s - 1.53x  (± 0.00) slower
         jserializer:       20.4 i/s - 1.57x  (± 0.00) slower
           primalize:       20.3 i/s - 1.57x  (± 0.00) slower
 jsonapi_same_format:       20.3 i/s - 1.58x  (± 0.00) slower
             jsonapi:       19.8 i/s - 1.62x  (± 0.00) slower
            jbuilder:       19.4 i/s - 1.65x  (± 0.00) slower
         blueprinter:       16.1 i/s - 1.99x  (± 0.00) slower
                 ams:       14.7 i/s - 2.17x  (± 0.00) slower
               rails:       14.1 i/s - 2.26x  (± 0.00) slower
          simple_ams:       13.0 i/s - 2.45x  (± 0.00) slower
       representable:       12.0 i/s - 2.66x  (± 0.00) slower

Calculating -------------------------------------
                alba     2.155M memsize (    42.400k retained)
                        36.425k objects (   600.000  retained)
                        50.000  strings (     0.000  retained)
         alba_inline     2.162M memsize (    42.400k retained)
                        36.508k objects (   600.000  retained)
                        50.000  strings (     0.000  retained)
     fast_serializer     2.208M memsize (    42.400k retained)
                        34.605k objects (   600.000  retained)
                        50.000  strings (     0.000  retained)
                 ams     2.493M memsize (    42.400k retained)
                        48.044k objects (   600.000  retained)
                        50.000  strings (     0.000  retained)
         blueprinter     2.657M memsize (    42.400k retained)
                        42.804k objects (   600.000  retained)
                        50.000  strings (     0.000  retained)
            jbuilder     2.843M memsize (    42.400k retained)
                        41.709k objects (   600.000  retained)
                        50.000  strings (     0.000  retained)
         jserializer     1.888M memsize (    59.200k retained)
                        30.808k objects (   700.000  retained)
                        50.000  strings (     0.000  retained)
             jsonapi     3.349M memsize (    42.400k retained)
                        53.114k objects (   600.000  retained)
                        50.000  strings (     0.000  retained)
 jsonapi_same_format     3.201M memsize (    42.400k retained)
                        51.013k objects (   600.000  retained)
                        50.000  strings (     0.000  retained)
               panko     1.016M memsize (     0.000  retained)
                        18.030k objects (     0.000  retained)
                         5.000  strings (     0.000  retained)
           primalize     2.264M memsize (    42.400k retained)
                        40.714k objects (   600.000  retained)
                        50.000  strings (     0.000  retained)
               rails     3.374M memsize (    42.400k retained)
                        54.703k objects (   600.000  retained)
                        50.000  strings (     0.000  retained)
       representable     3.938M memsize (    42.400k retained)
                        51.724k objects (   600.000  retained)
                        50.000  strings (     0.000  retained)
          simple_ams     8.272M memsize (    42.400k retained)
                        89.215k objects (   600.000  retained)
                        50.000  strings (     0.000  retained)

Comparison:
               panko:    1016178 allocated
         jserializer:    1888241 allocated - 1.86x more
                alba:    2154913 allocated - 2.12x more
         alba_inline:    2162057 allocated - 2.13x more
     fast_serializer:    2208328 allocated - 2.17x more
           primalize:    2263699 allocated - 2.23x more
                 ams:    2492704 allocated - 2.45x more
         blueprinter:    2657473 allocated - 2.62x more
            jbuilder:    2842693 allocated - 2.80x more
 jsonapi_same_format:    3201249 allocated - 3.15x more
             jsonapi:    3348718 allocated - 3.30x more
               rails:    3374097 allocated - 3.32x more
       representable:    3937918 allocated - 3.88x more
          simple_ams:    8272177 allocated - 8.14x more

[okuramasafumi]

Hi!

It shows that latest panko is a winner, in both speed and also it allocates twice less memory for your benchmarks.
Arguments ? 😁

Panko works with C extension, which doesn't work with JRuby. Also, panko requires oj while Alba has no runtime dependencies. These two things make panko faster. But yes, panko is really efficient.

What about advanced features like includes, conditions and inheritance?

I'd like to make benchmarks more complete, but it requires a lot of work to check if all other serializers implement the feature and how to use it. Contributions are more than welcome!

Turn off GC while benchmarking. Also the pg and DB dependency is useless. We are talking about object to hash translations, we don't care about the DB. so I would rewrite benchmark to instantiate plain old ruby objects.

If you've done this improvements already, could you submit a Pull Request? I'm going to merge it.