hash/crc32.update is a major CPU hog

[majek]

When profiling my application, I found that crc32 uses 50% of my CPU resources:

    1165  49.1%  49.1%     1165  49.1% hash/crc32.update

This is being called by real_decoder.go:pop in.check:

• https://github.com/Shopify/sarama/blob/master/real_decoder.go#L224
• https://github.com/Shopify/sarama/blob/master/crc32_field.go#L27

While I generally do appreciate sarama verifying checksums by default, I'd like to have a toggle to turn it off. I'd much rather crash once in a while than spend half of my CPU on tedious work.

My app reads about 130k reasonably small requests per second.

[drdee]

hey @majek
Thanks for filing this issue! I am wondering, instead of adding a config parameter (which I doubt many people will use), maybe we should port this C++ crc32_fast algorithm to Go: http://create.stephan-brumme.com/crc32/. That blog post offers some neat ways to significantly improve performance.

@eapache what do you think?

[majek]

Yes, making crc32 faster could work as well. Relevant: https://issues.apache.org/jira/browse/KAFKA-1449

[joestein]

Those are some interesting stats. Providing a dynamic option for hashing is an interesting feature too.

For KAFKA-1449 it might be worth building a patch, testing it and then posting results to the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals process.

btw <- LZ4 compression is available now in 0.8.2.0 https://issues.apache.org/jira/browse/KAFKA-1456 & https://issues.apache.org/jira/browse/KAFKA-1493 if you want to give it a try.

[eapache]

I would be happy to get faster doing CRC32s - when I've benchmarked the producer in the past it showed up as a significant cost there too.

It should be done as a separate package though, that code shouldn't live in Sarama proper.

[drdee]

@eapache I can take a crack at this later this week.

[wvanbergen]

Instead of disabling CRC checks, we could use this faster implementation: https://github.com/klauspost/crc32