Property Migration (Font Branch Diff Preview)

.github/workflows/ci.yml

@@ -17,7 +17,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        rust_version: [stable, "1.59.0"]
+        rust_version: [stable, "1.63.0"]
 
     steps:
     - uses: actions/checkout@v1

README.md

@@ -64,6 +64,7 @@ Roblox Lua implementation of DOM APIs, allowing Instance reflection from inside
 | BinaryString            | `Terrain.MaterialColors`        | ✔ | ➖ | ✔ | ✔ |
 | Bool                    | `Part.Anchored`                 | ✔ | ✔ | ✔ | ✔ |
 | BrickColor              | `Part.BrickColor`               | ✔ | ✔ | ✔ | ✔ |
+| Bytecode                | N/A                             | ❌ | ⛔ | ❌ | ❌ |
 | CFrame                  | `Camera.CFrame`                 | ✔ | ✔ | ✔ | ✔ |
 | Color3                  | `Lighting.Ambient`              | ✔ | ✔ | ✔ | ✔ |
 | Color3uint8             | `Part.BrickColor`               | ✔ | ✔ | ✔ | ✔ |
@@ -110,7 +111,7 @@ This project has unveiled a handful of interesting bugs and quirks in Roblox!
 - `ColorSequence`'s XML serialization contains an extra value per keypoint that was intended to be used as an envelope value, but was never implemented.
 
 ## Minimum Rust Version
-rbx-dom supports Rust 1.59.0 and newer. Updating the minimum supported Rust version will only be done when necessary, but may happen as part of minor version bumps.
+rbx-dom supports Rust 1.63.0 and newer. Updating the minimum supported Rust version will only be done when necessary, but may happen as part of minor version bumps.
 
 ## License
 rbx-dom is available under the MIT license. See [LICENSE.txt](LICENSE.txt) for details.

docs/binary.md

@@ -47,6 +47,7 @@ This document is based on:
 	- [PhysicalProperties](#physicalproperties)
 	- [Color3uint8](#color3uint8)
 	- [Int64](#int64)
+	- [Bytecode](#bytecode)
 	- [SharedString](#sharedstring)
 	- [OptionalCoordinateFrame](#optionalcoordinateframe)
 	- [UniqueId](#uniqueid)
@@ -93,20 +94,30 @@ Every file starts with a 32 byte header.
 ## Chunks
 Every chunk starts with a 16 byte header followed by the chunk's data.
 
-| Field Name          | Format  | Value                                             |
-|:--------------------|:--------|:--------------------------------------------------|
-| Chunk Name          | 4 bytes | The chunk's name, like `META` or `INST`           |
-| Compressed Length   | `u32`   | Length of the chunk in bytes, if it is compressed |
-| Uncompressed Length | `u32`   | Length of the chunk's data after decompression    |
-| Reserved            | 4 bytes | Always `0`                                        |
+| Field Name          | Format   | Value                                             |
+|:--------------------|:---------|:--------------------------------------------------|
+| Chunk Name          | 4 bytes  | The chunk's name, like `META` or `INST`           |
+| Compressed Length   | `u32`    | Length of the chunk in bytes, if it is compressed |
+| Uncompressed Length | `u32`    | Length of the chunk's data after decompression    |
+| Reserved            | 4 bytes  | Always `0`                                        |
+| Chunk Data          | Variable | The data contained in the chunk                   |
 
 If **Chunk Name** is less than four bytes, the remainder is filled with zeros.
 
 If **Compressed Length** is zero, **Chunk Data** contains **Uncompressed Length** bytes of data for the chunk.
 
-If **Compressed Length** is nonzero, **Chunk Data** contains an LZ4 compressed block. It is **Compressed Length** bytes long and will expand to **Uncompressed Length** bytes when decompressed.
+If **Compressed Length** is nonzero, **Chunk Data** will either contain an [LZ4][LZ4] or [ZSTD][ZSTD] compressed block. This compressed body is **Compressed Length** bytes long and will expand to **Uncompressed Length** bytes when decompressed.
 
-When the **chunk data** is compressed, it is done so using the [LZ4](https://github.com/lz4/lz4) compression algorithm.
+Which of the compression algorithms is used is indicated by the first several bytes of the compressed chunk body.
+- If the first 4 bytes of the block are the literal sequence `28 b5 2f fd`, the block is compressed using the ZSTD algorithm.
+- Otherwise, the block is compressed using the LZ4 algorithm.
+
+When a chunk is compressed using ZSTD, there is also a ZSTD frame present following the magic number that must be read by a decompressor. When it is compressed using LZ4, there is no frame and the compressed data begins immediately after the header.
+
+The data contained in **Chunk Data** varies in formatting based on the value of **Chunk Name**. Chunks used by Roblox are documented below:
+
+[ZSTD]: https://github.com/facebook/zstd/
+[LZ4]: https://github.com/lz4/lz4
 
 ### `META` Chunk
 The `META` chunk has this layout:
@@ -614,6 +625,19 @@ When an array of `Int64` values is present, the bytes of the integers are subjec
 
 `SharedString` values are stored as an [Interleaved Array](#byte-interleaving) of `u32` values that represent indices in the [`SSTR`](#sstr-chunk) string array.
 
+### Bytecode
+**Type ID `0x1d`**
+
+`Bytecode` values are stored identically to [`String`](#string) properties but contain precompiled [Luau][Luau] bytecode instructions rather than string data.
+
+This data type is disregarded by Roblox Studio and is only loaded by Roblox clients when cryptographically signed. Given that by design it is impossible to generate these signatures, the signing method is not documented in this spec file. For posterity however, the chunk used by Roblox is named `SIGN`. It is disregarded when loaded by Roblox Studio.
+
+It is highly recommended that implementations do not modify or interpret `Bytecode` data they encounter and instead simply read and write the data as-is.
+
+Implementors should be aware that running unsigned `Bytecode` is **incredibly unsafe** and should not be done unless the source can be validated somehow. Doing so is the equivalent to giving the author of the `Bytecode` unrestricted access to the system it is ran on.
+
+[Luau]: https://github.com/Roblox/luau
+
 ### OptionalCoordinateFrame
 **Type ID `0x1e`**
 

docs/patching-database.md

@@ -1,5 +1,5 @@
 # How to Fix a New Property Added by Roblox
-When Roblox introduces new properties, usually tools like Rojo can use them without any additional changes. Sometimes, though, properties are added with different names, multiple serialized forms, or aren't listed at all in the reflection dump that Roblox gives us.
+When Roblox introduces new properties, usually tools like Rojo can use them without any additional changes. Sometimes, though, properties are added with different names, multiple serialized forms, need to be migrated to a new property, or aren't listed at all in the reflection dump that Roblox gives us.
 
 This document describes some common scenarios, and what work needs to happen to fix them.
 
@@ -91,6 +91,32 @@ Change:
       AliasFor: MaxPlayers
 ```
 
+## Roblox added a new property, but it's a migration from an existing property, and the existing property no longer loads
+Sometimes Roblox migrates an existing property whose type is too constrained to a new property with a more flexible type.
+
+Without special handling, this can cause problems for binary files because when old and new models are mixed together, the binary serializer must add the new property to the old models. Without special instruction, it'll just add the default value. This can result in weird behavior like old text UI all having the Arial font, because the default value of a new property took priority.
+
+To fix this, we need to write a migration (in Rust) and apply it is as a patch (using database patch files).
+
+First, add your migration to the PropertyMigration enum in [`rbx_reflection/src/migrations`][migrations]. The migration should be named after the properties it's migrating. For example, migrating from `Font` to `FontFace` would be named `FontToFontFace`.
+
+Next, add code to convert from the old property's type to the new property's type. This code should be a new match arm in the `perform_migration` function in [`rbx_reflection/src/migrations`][migrations].
+
+Finally, add a patch in the [patches](patches) folder. This patch should change the old property's serialization type to `Migrate`, specifying the new property name and the migration name.
+
+For example, the patch for fonts looks like:
+```yaml
+Change:
+  TextLabel:
+    Font: # Property we're migrating *from*
+      Serialization:
+        Type: Migrate
+        Property: FontFace # Property we're migrating *to*
+        Migration: FontToFontFace # Migration we're using
+```
+
+If this property is present on multiple classes, you may need to specify the Serialization change for multiple properties on multiple classes. For example, the `Font` property is present on `TextLabel`, `TextButton`, `TextBox` without being derived from a superclass, so the real patch is approximately 3 times as long since it needs to be applied to each class.
+
 ## Roblox added a new property, but modifying it from Lua requires a special API
 Sometimes a property is added that cannot be assigned directly from Lua.
 
@@ -148,4 +174,5 @@ These pull requests outline how we implemented support for Attributes in rbx-dom
 
 [rbx-dom]: https://github.com/rojo-rbx/rbx-dom
 [patches]: https://github.com/rojo-rbx/rbx-dom/tree/master/patches
-[custom-properties]: https://github.com/rojo-rbx/rbx-dom/blob/master/rbx_dom_lua/src/customProperties.lua
+[custom-properties]: https://github.com/rojo-rbx/rbx-dom/blob/master/rbx_dom_lua/src/customProperties.lua
+[migrations]: https://github.com/rojo-rbx/rbx-dom/blob/master/rbx_reflection/src/migration.rs

generate_reflection/src/property_patches.rs

@@ -9,7 +9,8 @@ use std::path::Path;
 
 use anyhow::{anyhow, bail, Context};
 use rbx_reflection::{
-    DataType, PropertyDescriptor, PropertyKind, ReflectionDatabase, Scriptability,
+    DataType, PropertyDescriptor, PropertyKind, PropertyMigration, ReflectionDatabase,
+    Scriptability,
 };
 use serde::Deserialize;
 
@@ -84,6 +85,11 @@ pub enum PropertySerialization {
         #[serde(rename = "As")]
         serializes_as: String,
     },
+    #[serde(rename_all = "PascalCase")]
+    Migrate {
+        property: String,
+        migration: PropertyMigration,
+    },
 }
 
 impl From<PropertySerialization> for rbx_reflection::PropertySerialization<'_> {
@@ -96,6 +102,13 @@ impl From<PropertySerialization> for rbx_reflection::PropertySerialization<'_> {
             PropertySerialization::SerializesAs { serializes_as } => {
                 rbx_reflection::PropertySerialization::SerializesAs(Cow::Owned(serializes_as))
             }
+            PropertySerialization::Migrate {
+                property,
+                migration,
+            } => rbx_reflection::PropertySerialization::Migrate {
+                property: Cow::Owned(property),
+                migration,
+            },
         }
     }
 }

patches/model.yml

@@ -0,0 +1,15 @@
+Add:
+  Model:
+    ScaleFactor:
+      AliasFor: Scale
+      DataType:
+        Value: Float32
+      Scriptability: None
+
+Change:
+  Model:
+    Scale:
+      Serialization:
+        Type: SerializesAs
+        As: ScaleFactor
+      Scriptability: Custom

patches/screengui.yml

@@ -0,0 +1,7 @@
+Change:
+  ScreenGui:
+    IgnoreGuiInset:
+      Serialization:
+        Type: Migrate
+        Property: ScreenInsets
+        Migration: IgnoreGuiInsetToScreenInsets

patches/text-gui.yml

@@ -0,0 +1,19 @@
+Change:
+  TextLabel:
+    Font:
+      Serialization:
+        Type: Migrate
+        Property: FontFace
+        Migration: FontToFontFace
+  TextButton:
+    Font:
+      Serialization:
+        Type: Migrate
+        Property: FontFace
+        Migration: FontToFontFace
+  TextBox:
+    Font:
+      Serialization:
+        Type: Migrate
+        Property: FontFace
+        Migration: FontToFontFace

rbx_binary/CHANGELOG.md

@@ -2,8 +2,10 @@
 
 ## Unreleased
 * Added support for `Font` values. ([#248])
+* Fixed the nondeterministic output of SSTR chunk when multiple shared strings are present. ([#254])
 
 [#248]: https://github.com/rojo-rbx/rbx-dom/pull/248
+[#254]: https://github.com/rojo-rbx/rbx-dom/pull/254
 
 ## 0.6.6 (2022-06-29)
 * Fixed unserialized properties getting deserialized, like `BasePart.MaterialVariant`. ([#230])

rbx_binary/src/core.rs

@@ -434,7 +434,9 @@ fn find_serialized_from_canonical<'a>(
     match serialization {
         // This property serializes as-is. This is the happiest path: both the
         // canonical and serialized descriptors are the same!
-        PropertySerialization::Serializes => Some(canonical),
+        PropertySerialization::Serializes | PropertySerialization::Migrate { .. } => {
+            Some(canonical)
+        }
 
         // This property serializes under an alias. That property should have a
         // corresponding property descriptor within the same class descriptor.