- Codegen-less wrapper for FlatBuffers with get\set by name feature
- Status: under development
- Uses FlatBuffer to read\write values from\to byte array converting to the required type described in the Scheme.
- No codegen, no compilers, no (de)serialization. Just fields description and get\set by name.
- In contrast to FlatBuffers tracks if the field was unset or initially not set
- Supported types
int32, int64, float32, float64, bool, string, byte
- nested objects
- arrays
- Empty strings, nested objects and arrays are not stored (
Get()
returns nil) - Strings could be set as both
string
and[]byte
, string arrays - as both[]string
and[][]byte
.Get()
,ToJSON()
andToJSONMap()
returns string or array of strings - Scheme versioning
- Any data written with Scheme of any version will be correctly read using Scheme of any other version
- Written in old Scheme, read in New Scheme -> nil result on new field read, field considered as unset
- Written in new Scheme, read in old Scheme -> no errors
- Any data written with Scheme of any version will be correctly read using Scheme of any other version
- Data could be loaded from JSON (using gojay) or from
map[string]interface{}
- Only 2 cases of scheme modification are allowed: field rename and append fields to the end. This is necessary to have ability to read byte buffers in Scheme of any version
- Written in New -> read in Old -> write in Old -> New fields are lost (todo)
go get github.com/untillpro/dynobuffers
- Describe Scheme
- By yaml. Field types:
int32
int64
float32
float64
bool
string
byte
var schemeStr = ` name: string price: float32 quantity: int32 Id: int64 # first letter is capital -> field is mandatory ` scheme, err := dynobuffers.YamlToScheme(schemeStr) if err != nil { panic(err) }
- Manually
scheme := dynobuffers.NewScheme() scheme.AddField("name", dynobuffers.FieldTypeString, false) scheme.AddField("price", dynobuffers.FieldTypeFloat, false) scheme.AddField("quantity", dynobuffers.FieldTypeInt, false) scheme.AddField("id", dynobuffers.FieldTypeLong, true)
- By yaml. Field types:
- Create empty Dyno Buffer using Scheme
b := dynobuffers.NewBuffer(scheme)
- panics if nil provided
- Set\modify fields according to the Scheme
b.Set("price", float32(0.123)) b.Set("name", nil) // unset field
- To bytes array
bytes, err := b.ToBytes() if err != nil { panic(err) }
- To JSON key-value
Note: arrays of byte are encoded to base64 strings
jsonStr := b.ToJSON()
- To map key-value (JSON-compatible)
jsonMap := b.ToJSONMap()
- Read Buffer from bytes using Scheme
b = dynobuffers.ReadBuffer(bytes, scheme)
- panics if nil Scheme provided
- Work with Buffer
value, ok := b.GetFloat32("price") // read typed. !ok -> field is unset or no such field in the scheme. Works faster and takes less memory allocations than Get() b.Get("price") // read untyped. nil -> field is unset or no such field in the scheme b.Set("price", nil) // set to nil means unset bytes = b.ToBytes()
- Load data from JSON key-value and to bytes array
bytes, nilled, err := b.ApplyJSONAndToBytes([]byte(`{"name": "str", "price": 0.123, "fld": null}`)) if err != nil { panic(err) }
nilled
will contain list of field names whose values were effectively set to nil, i.e. fields names whose values were provided asnull
or as an empty object, array or string- note: nils are not stored in
bytes
- note: nils are not stored in
- value is nil and field is mandatory -> error
- value type and field type are incompatible (e.g. string provided for numeric field) -> error
- float value is provided for an integer field -> no error, integer part is considered only. E.g. 0.123 value in JSON is met -> integer field value is 0
- no such field in the scheme -> error
- array element value is nil -> error (not supported)
- values for byte arrays are expected to be base64 strings
- Load data from
map[string]interface{}
m := map[string]interface{} { "name": "str", "price": 0.123, "fld": nil, } if err := b.ApplyMap(m); err != nil { panic(err) } bytes, err := b.ToBytes() if err != nil { panic(err) }
- value type and field type differs but value fits into field (e.g. float64(255) fits into float, double, int, long, byte; float64(256) does not fit into byte etc) -> ok
- the rest is the same as for
ApplyJSONAndToBytes()
- Check if a field exists in the scheme and is set to non-nil
b.HasValue("name")
- Return
Buffer
to pool to prevent additional memory allocationsb.Release() // b itself, all objects created manually and used in b.Set(), all objects got using `b.Get()` are released also. // nor these objects neither result of `b.ToBytes()` must not be used from now on
- Iterate over fields which has value
b.IterateFields(nil, func(name string, value interface{}) bool { return true // continue iterating on true, stop on false })
- Iterate over specified fields only. Will iterate over each specified name if according field has a value. Unknown field name -> no iteration
b.IterateFields([]string{"name", "price", "unknown"}, func(name string, value interface{}) bool { return true // continue iterating on true, stop on false })
- See dynobuffers_test.go for usage examples
- Declare scheme
- by yaml
var schemeStr := ` name: string Nested: nes1: int32 Nes2: int32 Id: int64 ` schemeRoot := dynobuffers.YamlToScheme(schemeStr)
- manually
schemeNested := dynobuffers.NewScheme() schemeNested.AddField("nes1", dynobuffers.FieldTypeInt, false) schemeNested.AddField("nes2", dynobuffers.FieldTypeInt, true) schemeRoot := dynobuffers.NewScheme() schemeRoot.AddField("name", dynobuffers.FieldTypeString, false) schemeRoot.AddNested("nested", schemeNested, true) schemeRoot.AddField("id", dynobuffers.FieldTypeLong, true)
- by yaml
- Create Buffer, fill and to bytes
bRoot := dynobuffers.NewBuffer(schemeRoot) bNested := dynobuffers.NewBuffer(schemeNested) bNested.Set("nes1", 1) bNested.Set("nes2", 2) bRoot.Set("name", "str") bRoot.Set("nested", bNested) bytes, err := bRoot.ToBytes() if err != nil { panic(err) }
- Read from bytes, modify and to bytes again
bRoot = dynobuffers.ReadBuffer(bytes, schemeRoot) bRoot.Set("name", "str modified") bNested := bRoot.Get("nested").(*dynobuffers.Buffer) bNested.Set("nes2", 3) bytes, err := bRoot.ToBytes() if err != nil { panic(err) } bRoot = dynobuffers.ReadBuffer(bytes, scheme) // note: bNested is obsolete here. Need to obtain it again from bRoot
- Empty nested objects are not stored
- Unmodified nested objects are copied field by field on
ToBytes()
- See dynobuffers_test.go for usage examples
- Declare scheme
- by yaml. Append
..
to field name to make it arrayvar schemeStr = ` name: string Nested..: nes1: int32 Nes2: int32 Ids..: int64 ` schemeRoot := dynobuffers.YamlToScheme(schemeStr)
- manually
schemeNested := dynobuffers.NewScheme() schemeNested.AddField("nes1", dynobuffers.FieldTypeInt, false) schemeNested.AddField("nes2", dynobuffers.FieldTypeInt, true) schemeRoot := dynobuffers.NewScheme() schemeRoot.AddField("name", dynobuffers.FieldTypeString, false) schemeRoot.AddNestedArray("nested", schemeNested, true) schemeRoot.AddArray("ids", dynobuffers.FieldTypeLong, true)
- by yaml. Append
- Create Buffer, fill and to bytes
bRoot := dynobuffers.NewBuffer(schemeRoot) buffersNested := make([]*Buffer, 2) bNested := dynobuffers.NewBuffer(schemeNested) bNested.Set("nes1", 1) bNested.Set("nes2", 2) buffersNested = append(buffersNested, bNested) bNested = dynobuffers.NewBuffer(schemeNested) bNested.Set("nes1", 3) bNested.Set("nes2", 4) buffersNested = append(buffersNested, bNested) ids := []int64{5,6} bRoot.Set("name", "str") bRoot.Set("nested", buffersNested) bRoot.Set("ids", ids) bytes, err := bRoot.ToBytes() if err != nil { panic(err) }
- Read array
- By iterator
bRoot = dynobuffers.ReadBuffer(bytes, schemeRoot) int64Arr := assert.Equal(t, int64(5), bRoot.GetInt64Array("ids")) for i := 0; i < int64Arr.Len(); i++ { assertEqual(t, ids[i], int64Arr.At(i)) }
- Read filled array of non-objects
bRoot = dynobuffers.ReadBuffer(bytes, schemeRoot) arr := b.Get("ids").([]int64)
- Read array of objects using iterator
bRoot = dynobuffers.ReadBuffer(bytes, schemeRoot) arr := bRoot.Get("nested").(*dynobuffers.ObjectArray) // ObjectArray is iterator over nested entities for arr.Next() { // arr.Buffer is switched on each arr.Next() assert.Equal(t, int32(1), arr.Buffer.Get("nes1")) } // note: not need to release `arr`. It will be released on `b.Release()`
- By iterator
- Modify array and to bytes
- Set
bRoot = dynobuffers.ReadBuffer(bytes, schemeRoot) ids := []int64{5,6} bRoot.Set("ids", ids) arr := bRoot.Get("nested").(*dynobuffers.ObjectArray) arr.Next() arr.Buffer.Set("nes1", -1) bRoot.Set("nested", arr) bytes, err := bRoot.ToBytes() if err != nil { panic(err) } bRoot = dynobuffers.ReadBuffer(bytes, scheme) // note: `arr` is obsolete here. Need to obtain the array again from bRoot
- Append
bRoot = dynobuffers.ReadBuffer(bytes, schemeRoot) // if wasn't set then equivalent to bRoot.Set() bRoot.Append("ids", []int32{7, 8}) // 7 and 8 will be appended buffersNested := []*Buffer{} bNested = dynobuffers.NewBuffer(schemeNested) bNested.Set("nes1", 5) bNested.Set("nes2", 6) buffersNested = append(buffersNested, bNested) bRoot.Append("nested", buffersNested) // bNested will be appended bytes, err := bRoot.ToBytes() if err != nil { panic(err) }
- Set
- Null\nil array element is met on
ApplyJSONAndToBytes()
,Set()
,Append()
orApplyMap()
-> error, not supported - Arrays are appended (set if there is nothing to append to) if met on
ApplyJSONAndToBytes()
andApplyMap()
- Byte arrays are decoded to JSON as base64 strings
- Byte array value could be set from either byte array and base64-encoded string
- Empty array -> no array,
Get()
will return nil,HasValue()
will return false Append()
orSet()
nil or epmty array means unset the array- See dynobuffers_test.go for usage examples
- For now there are 2 same methods:
ApplyMapBuffer()
andApplyJSONAndToBytes()
. Need to get rid of one of them. - For now
ToBytes()
result must not be stored ifRelease()
is used because on nextToBytes()
the stored previousToBytes()
result will be damaged. SeeTestPreviousResultDamageOnReuse()
. The better soultion is to makeToBytes()
return not[]byte
but aninterface {Bytes() []byte; Release()}
.- use bytebufferpool on
flatbuffers.Builder.Bytes
?
- use bytebufferpool on
ToJSON()
: use bytebufferpool?- Array of nested entities modification is not supported
- cd benchmarks
- go test -bench=ReadAllArticleFields -benchmem
goos: windows
goarch: amd64
pkg: github.com/untillpro/dynobuffers/benchmarks
Benchmark_ReadAllArticleFields_Avro-8 71046 23394 ns/op 11257 B/op 149 allocs/op
Benchmark_ReadAllArticleFields_Dyno_Untyped-8 195801 6437 ns/op 808 B/op 84 allocs/op
Benchmark_ReadAllArticleFields_Dyno_Typed-8 313376 3776 ns/op 0 B/op 0 allocs/op
Benchmark_ReadAllArticleFields_Flat-8 1000000 1132 ns/op 0 B/op 0 allocs/op
Benchmark_ReadAllArticleFields_Json-8 14431 87331 ns/op 14145 B/op 603 allocs/op
- cd benchmarks
- go test -bench=ReadFewArticleFields -benchmem
goos: windows
goarch: amd64
pkg: github.com/untillpro/dynobuffers/benchmarks
Benchmark_ReadFewArticleFields_Avro-8 98437 19311 ns/op 11257 B/op 149 allocs/op
Benchmark_ReadFewArticleFields_Dyno_Typed-8 18520500 62.2 ns/op 0 B/op 0 allocs/op
Benchmark_ReadFewArticleFields_Flat-8 60032416 19.8 ns/op 0 B/op 0 allocs/op
Benchmark_ReadFewArticleFields_Json-8 15333 83824 ns/op 11073 B/op 603 allocs/op
- cd benchmarks
- go test -bench=ReadSimple -benchmem
goos: windows
goarch: amd64
pkg: github.com/untillpro/dynobuffers/benchmarks
Benchmark_ReadSimple_Avro-8 2038466 1193 ns/op 744 B/op 10 allocs/op
Benchmark_ReadSimple_Dyno_Typed-8 20017480 55.2 ns/op 0 B/op 0 allocs/op
Benchmark_ReadSimple_Flat-8 59981404 20.2 ns/op 0 B/op 0 allocs/op
Benchmark_ReadSimple_Flat_String-8 29275360 40.7 ns/op 0 B/op 0 allocs/op
Benchmark_ReadSimple_Json-8 545769 2423 ns/op 776 B/op 21 allocs/op
NOTE: DynoBuffers allocs caused by string types