OdinLZ4Extension

The fastest and most efficient binary compression solution for Odin

Navigation

• OdinLZ4Extension
  • Navigation
  • About
    • What's under the hood
  • Performance tests
    • Tests
  • Roadmap
  • Compatibility
  • Dependencies
  • How to use
    • Preparation
    • Serialization
      • Base Serialization
      • Lazy Serialization
      • Serialization with Unity object references
      • Lazy Serialization with Unity object references
    • Deserialization
      • Base Deserialization
      • Lazy Deserialization
      • Deserialization with Unity object references
      • Lazy Deserialization with Unity object references
    • Compression levels
    • Examples
  • Installation
    • Install via OpenUPM
    • Install via Git URL
  • Easy update of existing code
  • How to install System libraries for LZ4
  • How to help the project
  • Contact

About

Odin - Inspector and Serializer is one of the most popular development tools for Unity, its power and functionality is incredible, one of the main functions is de/serialization of data, it is extremely important function for most who use "Odin".

If you use cloud solutions in your projects and store some data in a serialized form, then imagine how much data space you can save with fast and efficient compression.

The best available algorithm was chosen as the compression engine, this is LZ4, it is extremely fast and efficient.

Combining these two engineering wonders we get extremely fast de/serialization and more free space for data.

So, Odin + LZ4 = PROFIT 💪

What's under the hood

The best realization of the LZ4 algorithm. This solution is provided by Milos Krajewski under the MIT license.

Original LZ4 has been written by Yann Collet and original 'C' sources can be found here

More info here

If you find this project useful, star it, I will be grateful.

Performance tests

Serialization object:

[Serializable]
public class Eexperimental
{
  public DirectionalLight DirectionalLight;
  public SpotLight SpotLight;
  public PointLight PointLight;
}

Configs:

• Binary format
• FAST level of compression

Tests

Roadmap

Status	Milestone
🚀	Multithread de/compressing and de/serializing
✅	De/Serialization performance tests with and without compression
📌	Cut out the main parts of the LZ4 engine and compile a lightweight .dll

Compatibility

Backend	.Net	Odin	Compatible
IL2CPP	4.x	3.x	✅

Dependencies

• Odin
• System libraries
  • System.Memory
  • System.Buffers
  • System.Runtime.CompilerServices.Unsafe

The OdinLZ4Extension does not contain any source materials of the Sirenix, you can get Odin - Inspector and Serializer on the AssetStore

At the moment OdinLZ4Extension only supports Odin - Inspector and Serializer
If you are only using the Odin Serializer then this extension will not be successfully imported into the Unity project. The reason for this is different dlls and namespaces. If you are using only Odin Serializer and would like to have this extension, write me and I will add this point to the roadmap.

How to use

Preparation

1. Add the OdinLZ4Extension to your project (how to do it)

2. Include LZ4 necessary libraries (how to do it)

3. Plugin namespaces

using K4os.Compression.LZ4;
using OdinLZ4Extension;

Done, use de/serialization + LZ4 in one line. [Examples]

Serialization

Several methods are provided for serialization with subsequent compression.

Base Serialization

byte[] SerializeValue<T>(T value, DataFormat format, SerializationContext ctx = null, OdinLZ4Level level = OdinLZ4Level.FAST)

---

Lazy Serialization

Fast serialization in binary format.

byte[] LazySerialization<T>(T value, OdinLZ4Level level = OdinLZ4Level.FAST)

---

Serialization with Unity object references

byte[] SerializeValue<T>(T value, DataFormat format, out List<UnityEngine.Object> unityObjects, SerializationContext ctx = null, OdinLZ4Level level = OdinLZ4Level.FAST)

---

Lazy Serialization with Unity object references

byte[] LazySerialization<T>(T value, out List<UnityEngine.Object> unityObjects, OdinLZ4Level level = OdinLZ4Level.FAST)

Deserialization

Several convenient methods are provided for deserializing compressed objects.

Base Deserialization

TResult DeserializeValue<TResult>(byte[] bytes, DataFormat format, DeserializationContext ctx = null)

---

Lazy Deserialization

TResult LazyDeserialization<TResult>(byte[] bytes)

---

Deserialization with Unity object references

Decompresses and deserializes a value of a given type from the given byte array in the given format, using the given list of Unity objects for external index reference resolution

TResult DeserializeValue<TResult>(byte[] bytes, DataFormat format, List<UnityEngine.Object> referencedUnityObjects, DeserializationContext ctx = null)

---

Lazy Deserialization with Unity object references

Fast data deserialization from binary format, using the given list of Unity objects for external index reference resolution

TResult LazyDeserialization<TResult>(byte[] bytes, List<UnityEngine.Object> referencedUnityObjects)

---

Compression levels

When you compress serialized data, FAST compression is applied by default. You can choose the compression level for your needs.
Available compression levels:

• FAST (maximum compression speed)
• OPTIMAL (sweet spot, fast and efficient compression)
• MAX (slower than others but most efficient)

Code example:

OdinLZ4API.SerializeValue(SERIALIZABLE_VALUE, DataFormat.Binary, level: OdinLZ4Level.OPTIMAL);
OdinLZ4API.LazySerialization(SERIALIZABLE_VALUE, OdinLZ4Level.MAX);

Examples

Code examples:

[Serializable] public class SimpleData { public int a; public string someData; }
[Serializable] public class RefersData { public GameObject player1, player2; public Vector3 somePos; }

[SerializeField] private SimpleData simpleData = new SimpleData() { a = 100, someData = "It's work!" };
[SerializeField] private RefersData refersData;

SimpleData deserializedSimpleData;
RefersData deserializedRefersData;

byte[] smplSerialization, refersSerialization;
List<UnityEngine.Object> refs;

// --- Serialization ---

// Base serialization
smplSerialization = OdinLZ4API.SerializeValue(simpleData, DataFormat.Binary, level:OdinLZ4Level.MAX);
// Lazy Serialization
smplSerialization = OdinLZ4API.LazySerialization(simpleData);

// Serialization with Unity object references
refersSerialization = OdinLZ4API.SerializeValue(refersData, DataFormat.Binary, out refs);
// Lazy Serialization with Unity object references
refersSerialization = OdinLZ4API.LazySerialization(refersData, out refs);

// --- Deserialization ---

// Base deserialization
deserializedSimpleData = OdinLZ4API.DeserializeValue<SimpleData>(smplSerialization, DataFormat.Binary);
// Lazy deserialization
deserializedSimpleData = OdinLZ4API.LazyDeserialization<SimpleData>(smplSerialization);

// Deserialization using the given list of Unity objects for external index reference resolution
deserializedRefersData = OdinLZ4API.DeserializeValue<RefersData>(refersSerialization, DataFormat.Binary, refs);
// Lazy deserialization using the given list of Unity objects for external index reference resolution
deserializedRefersData = OdinLZ4API.LazyDeserialization<RefersData>(refersSerialization, refs);

Installation

Install via OpenUPM

The package is available on the openupm registry. It's recommended to install it via openupm-cli.

openupm add com.inc8877.odinlz4extension

Install via Git URL

Open Packages/manifest.json with your favorite text editor. Add the following line to the dependencies block.

{
  "dependencies": {
    "com.inc8877.odinlz4extension": "https://github.com/inc8877/OdinLZ4Extension.git",
   }
}

Easy update of existing code

You can easily upgrade data de/serialization that is already implemented in your project to de/serialization with de/compression. To do this, it is enough to plugin the necessary namespaces and simply replace SerializationUtility class to OdinLZ4API.

Code example:

// Necessary namespaces
using K4os.Compression.LZ4;
using OdinLZ4Extension;

// Existing code
SerializationUtility.SerializeValue(SERIALIZABLE_VALUE, DataFormat.Binary);
SerializationUtility.DeserializeValue<SimpleData>(BYTES_ARRAY, DataFormat.Binary);

// Upgraded code
OdinLZ4API.SerializeValue(SERIALIZABLE_VALUE, DataFormat.Binary);
OdinLZ4API.DeserializeValue<SimpleData>(BYTES_ARRAY, DataFormat.Binary);

Below is a list of methods in the SerializationUtility class that can be replaced by methods with compression:

// --- Serialization ---

byte[] SerializeValue<T>(T value, DataFormat format, SerializationContext context = null);

byte[] SerializeValue<T>(T value, DataFormat format, out List<UnityEngine.Object> unityObjects, SerializationContext context = null);

// --- Deserialization ---

T DeserializeValue<T>(byte[] bytes, DataFormat format, DeserializationContext context = null);

T DeserializeValue<T>(byte[] bytes, DataFormat format, List<UnityEngine.Object> referencedUnityObjects, DeserializationContext context = null);

How to install System libraries for LZ4

Unity does not provide the necessary libraries for the LZ4, so you need to install them manually. The libraries can be downloaded via NuGet, but since the Unity does not work well with it, you need to get the .dll from the downloaded packages and put them into the Assets folder.

If you don't want to mess with it, you can get the DLLs in a .zip file located in the OdinLZ4Extension root folder (path: Packages/OdinLZ4Extension/RelatedMaterials/SystemDLLs.zip). Just unzip and place them in the project. (A good location for .dll in the project would be the Assets/Plugins/System folder)

How to help the project

At the moment the project has just begun its life and requires a lot of work, if you are reading this means you may have liked this project and you can help by doing the following:

• do performance tests (compare data serialization with and without compression and make a graph)
• expand functionality:
  • serialization with compressing via multithreading
  • a pool of serialization tasks with callbacks

Contact

inc8877@gmail.com