meshopt_encodeVertexBuffer not making very compressible data.

[Ono-Sendai]

Hi,
Am testing out meshopt_encodeVertexBuffer, hoping to get some good compression wins. However it's not working very well in some cases. In fact, if my code and measurements are correct, it's performing much worse than my very simple vertex attribute de-interleaving filtering, in the case where vertex positions coordinates are float32s. For uint16 coordinates it works much better though.
For the float32 case:

meshopt vert compressed_size: 150207 B
standard vertex compressed_size: 130799 B

So my simple attribute deinterleaving results in a much smaller compressed vertex size. Compression method used is zstd, compression level 20. (similar result different for compression level 9 etc.).

You can see my de-interleaving and compression code here: https://github.com/glaretechnologies/glare-core/blob/5b34793a7a53bf8b58b2f503a18a62d49e197988/graphics/BatchedMesh.cpp#L858

Vertex format is
float vert_pos[3];
uint32 packed_normal;
half uv[2];
uint32 mat_index;

This is a little disappointing, was hoping for some wins even with float vertex positions.
meshopt_encodeIndexBuffer on the other hand works fantastically.

I can supply you with the vertex data if you want to test.

cheers,
Nick

[Ono-Sendai]

Actually, upon further testing, it seems meshopt_encodeVertexBuffer performs poorly for uint16 vertex coordinates as well:

meshopt vert compressed_size: 110177 B

standard vertex compressed_size: 95158 B

[zeux]

This is a little disappointing, was hoping for some wins even with float vertex positions.

It's not really designed for floating point data without the use of filters (meshopt_encodeFilterExp for example); for poorly compressible data like this zstd might find more opportunities to reuse individual float values for some meshes, but it certainly depends. For integer data, you should also make sure that the vertex order is correct per documentation.

One good way to see what the reasonable compressed size would be to use gltfpack (with -c / -cc) as a reference, perhaps with -vp 16 to match quantization sizes; although since glTF doesn't support zstd you would need to compress the .bin file separately. You can also check floating point performance with -vpf.

Either way I don't think it's a bug so I'm going to convert this to a discussion.

[Ono-Sendai]

" For integer data, you should also make sure that the vertex order is correct per documentation."
I don't see this in the documentation, what is the correct vertex order?

[zeux]

I think actually if index compression is working well for you then it's already probably fine, but nevertheless, readme states:

Note that vertex encoding assumes that vertex buffer was optimized for vertex fetch, and that vertices are quantized

The assumption is that the mesh goes through the regular vertex cache -> vertex fetch optimization pipeline and as such the locality of reference between neighboring vertices is maximized, which improves the change that the delta compression works well. With a bad order, delta compression may result in less compressible data vs not doing it at all - this is one of the things I am considering extending the format for, so that for poorly compressible deltas the encoder can omit delta compression for parts of the stream in hopes that Zstd/et al may do better on the result.

[zeux]

... this should be replicated in header comments for meshoptimizer.h, I'll fix this.

[zeux]

And yeah if you can share the file I could double check what the behavior is on some builtin code. FWIW I'm planning to look into improving compression at some point, but that requires expanding the existing compression scheme to have a little more freedom to encode data in certain cases and it's not clear if significant wins are possible without sacrificing decoding speed - which is to say, within the current encoding scheme the compression rates are what they are, because the encoder has relatively little latitude and relies on external processing code to remove noise and optimize order.

[Ono-Sendai]

I'll try out your float filters. In the mean time, some more data:

vertex_data.size(): 225296 B (raw size)
meshopt res_encoded_vert_buf_size: 132095 B (result of meshopt meshopt_encodeVertexBuffer)
meshopt vert compressed_size: 111051 B (after zstd compression)

unfiltered_compressed_size: 101825 B (zstd applied directly to original vertex data)

So meshopt_encodeVertexBuffer is actually making the data significantly less compressible, relative to not using it.

[Ono-Sendai]

No offense hopefully, but I think there's something a bit wrong going on here. Regardless of vertex ordering or float filtering, I don't think an encoding scheme should negatively effect the achievable compression ratio this much.

[Ono-Sendai]

will supply data so u can test with it shortly

[Ono-Sendai]

Here you go, this is the raw uncompressed and unfiltered vertex data:
https://www.dropbox.com/scl/fi/vlrgeb8azim05molkb9xi/combined_vert_data_uncompressed.bin?rlkey=t1l5stlfnlrlht9tfxmd09s74&st=sk2h6whm&dl=0

Vertex format is
uint16 pos[3];
uint16 mat_index;
uint32 packed_normal;
half uv[2];

[zeux]

Thanks! Could you also share an obj or gltf file corresponding to this (with source geometry)?

[Ono-Sendai]

yeah will do

[Ono-Sendai]

haha actually i can't send you a GLTF file for this exact data, since GLTF only supports float format for position coords.

Nevertheless, here is pretty much the same data as a glb, using float position coords: https://www.dropbox.com/scl/fi/fld6jutall1eaqd9daxy8/chunk_128_4_-1.glb?rlkey=7buzj0e178wrx87bbeexptk7b&st=2izxm4r7&dl=0

[Ono-Sendai]

can also send u the index buffer if you want it.

[zeux]

Yeah sorry I just meant the source GLTF geometry with floating point data, so this is good. I'll reupload these as attachments so that these aren't lost.

[zeux]

chunk-data.zip

~/meshoptimizer (master) $ ./codectest 16 <~/Downloads/combined_vert_data_uncompressed.bin | lz4 -1 | wc -c
126846
~/meshoptimizer (master) $ cat ~/Downloads/combined_vert_data_uncompressed.bin | lz4 -1 | wc -c
155393

~/meshoptimizer (master) $ ./codectest 16 <~/Downloads/combined_vert_data_uncompressed.bin | lz4 -9 | wc -c
120976
~/meshoptimizer (master) $ cat ~/Downloads/combined_vert_data_uncompressed.bin | lz4 -9 | wc -c
139687

~/meshoptimizer (master) $ ./codectest 16 <~/Downloads/combined_vert_data_uncompressed.bin | gzip | wc -c
112674
~/meshoptimizer (master) $ cat ~/Downloads/combined_vert_data_uncompressed.bin | gzip | wc -c
113614

~/meshoptimizer (master) $ ./codectest 16 <~/Downloads/combined_vert_data_uncompressed.bin | zstd | wc -c
116091
~/meshoptimizer (master) $ cat ~/Downloads/combined_vert_data_uncompressed.bin | zstd | wc -c
120799

~/meshoptimizer (master) $ ./codectest 16 <~/Downloads/combined_vert_data_uncompressed.bin | zstd -9 | wc -c
115160
~/meshoptimizer (master) $ cat ~/Downloads/combined_vert_data_uncompressed.bin | zstd -9 | wc -c
119470

~/meshoptimizer (master) $ ./codectest 16 <~/Downloads/combined_vert_data_uncompressed.bin | zstd -19 | wc -c
111052
~/meshoptimizer (master) $ cat ~/Downloads/combined_vert_data_uncompressed.bin | zstd -19 | wc -c
101826

What I'm seeing by just running this data through a simple test binary is that zstd does result in better compression ratio on level 19, but is a little worse at 9 or default 6. Same goes for gzip. lz4 sees significant wins. All of this is more or less aligned with my expectations so far (except the zstd -19 results!)

Three notes on this (to address the "is something wrong?" comment):

1. In general, I agree - it would be nice for vertex codec to result in stronger compression by itself, with or without a zstd-like layer. It is impossible to guarantee that the size in a long pipeline doesn't expand without modeling the behavior of the final compressor precisely; for example from an unrelated domain, if you know the source data is ASCII, you can compress it by packing the 8-bit source data into 7-bit bitstream (which is guaranteed 10% savings!), but if you try to compress the result with zstd et al you will see a larger size. That said, there is an opportunity for improvement.

2. In particular, there are some limitations of the encoding scheme - for a given source input, there's relatively few valid output streams with the current (v0) encoding format. This means that when the source data is not perfectly aligned with the expectations of the encoding scheme, there is some efficiency left on the table. There are a couple known cases when this may happen: a) if deltas don't help the data, using them may break some existing patterns in the data; b) the deltas are computed with expectation of low bit aligned data; ideally the encoder would be able to rotate the bytes to adjust for the data patterns. There may be some others that I don't recall; the first step to making v1 encoding would be to analyze the problematic areas.

3. The decoder has a certain design point (reasonable compression on its own, data can still be compressed by a general purpose compressor, decompression is very fast - 5 GB/s on my system, input data has an arbitrary encoding). So there are some limitations on the transforms that it can/will do, and as such it's designed to be used in a layered scheme - it's unrealistic to expect it to compress an arbitrary sequence of bytes :) Moreover, the encoder can't really model what the subsequent compressor may or may not be able to do - one failure point I've seen in the past is that if a source mesh has a bunch of repetitive data in different places, zstd might be able to compress the attribute streams very well through just LZ repetitions, and deltas and chunking used in the encoder may break this pattern apart. Issues like this may also be present and it's always going to be the case that some encodings or some data patterns will cause a really strong compressor like zstd to perform better, at some cost (eg decompression performance).

The reason why I say "this is not a bug" is because of pt2 - I'm not aware of any decisions that the encoder can be making dramatically better right now. This is not to say that there are no improvements possible here! But that's a project to design the v1 format and implement an encoder for it, not a project to fix some sort of issue in the encoder.

[Ono-Sendai]

Yeah I'm not saying it's a bug necessarily, just that it's not doing great.
To reiterate, just my very simple filtering approach of de-interleaving the vertex attributes results in this: (after zstd level 20)

standard vertex compressed_size: 95903 B

Which is way better than the meshopt result of 111052 B.

[Ono-Sendai]

It's possible my testing with zstd level 20 is a bit unrealistic. level 20 has a very large backreference window IIRC. I'll have to check my code to see if I use it in production already.

[Ono-Sendai]

with uint16 vert positions, compression_level = 9
meshopt vert compressed_size: 113080 B
standard vertex compressed_size: 106174 B

[zeux]

Here's some results from gltfpack, this is not quite apples to apples because the normal & texture coordinate storage is different. But just noting that within the type of data that encoder expects right now, it's doing pretty well.

$ ./gltfpack -i ~/Downloads/chunk_128_4_-1.glb -o chunk.gltf -vp 16 -kv && ls -la chunk.bin && cat chunk.bin | zstd -19 | wc -c
-rw-rw-r-- 1 zeux zeux 261168 Sep 27 10:17 chunk.bin
107450

$ ./gltfpack -i ~/Downloads/chunk_128_4_-1.glb -o chunk.gltf -c -vp 16 -kv && ls -la chunk.bin && cat chunk.bin | zstd -19 | wc -c
-rw-rw-r-- 1 zeux zeux 103972 Sep 27 10:17 chunk.bin
80581

$ ./gltfpack -i ~/Downloads/chunk_128_4_-1.glb -o chunk.gltf -cc -vp 16 -kv && ls -la chunk.bin && cat chunk.bin | zstd -19 | wc -c
-rw-rw-r-- 1 zeux zeux 96632 Sep 27 10:17 chunk.bin
73435

-c just applies encoder on top of the same data, so it saves 20% extra data compared to not doing it. The data is stored as a stream per component (which doesn't matter for meshopt until filters are applied, but does matter for zstd). -cc mostly applies filters to normal data here.

One thing I will note is likely a significant problem here is a 10-bit encoded normal in your data. It is spread across the byte boundaries. This means it's basically incompressible other than LZ references zstd is doing, and deltas probably hurt this even more. I'd assume this is a major source of inefficiency here, and ability to at least disable delta compression on this stream would help.

[zeux]

(oh, and chunk.bin here is vertex & index data combined as it's just regular glTF storage)

[Ono-Sendai]

Ok, interesting. I'm not sure what all those command line flags do.
So basically if the data is filtered and arranged in just the way it expects, meshopt_encodeVertexBuffer works well ? :)

[Ono-Sendai]

(I'll build gltfpack myself and test it at some point.)

[zeux]

One final note, if you are looking to try to get better results without changing your data, I would probably recommend trying to just split the packed normal stream and not encode it. I would assume on your data you are seeing a lot of good LZ matches from zstd for normals specifically because a lot of normals are probably axis aligned. Because meshopt encoder can't see any reasonable structure in the normal data due to bit packing, it will likely just make those matches worse by applying delta encoding, which will hurt Zstd LZ matches, plus byte interleaving will also make LZ matches worse.

[Ono-Sendai]

yeah will try not packing normals.

[Ono-Sendai]

Ok got some good results using filtering:
Using zstd compression level 19 everywhere

BatchedMeshTests::test(), vec3 position
=====================================================
attr_data.size(): 158508        [raw size in bytes of attribute data]
compressed_sz: 77285         [zstd applied directly without filtering and encoding first]
meshopt encoded size: 93006
meshopt encoded, compressed size: 80379

With filtering, 14 bits mantissa:
-------------------------
attr_data.size(): 158508
compressed_sz: 77285
meshopt encoded size: 33634
meshopt encoded, compressed size: 26642

with filtering, 16 bits mantissa
------------------------------
attr_data.size(): 158508
compressed_sz: 77285
meshopt encoded size: 42547
meshopt encoded, compressed size: 34658

uv0 
==========
with filtering, 8 bits mantissa
---------------------------------
attr_data.size(): 105672
compressed_sz: 32763
meshopt encoded size: 8543
meshopt encoded, compressed size: 4252

Matindex
=================
attr_data.size(): 52836
compressed_sz: 981
meshopt encoded size: 1885
meshopt encoded, compressed size: 569

Packed normals
=============
attr_data.size(): 52836
compressed_sz: 23533
meshopt encoded size: 36524
meshopt encoded, compressed size: 31730


Unpacked normals, with oct filtering
=====================================
ttr_data.size(): 52836
compressed_sz: 23533
meshopt encoded size: 26222
meshopt encoded, compressed size: 20260