Iterable ordered map alternative with improved performance

[hanjm]

Summary

fix #1151, introduce a v2 interface for type assert.

Checklist

Delete items not relevant to your PR:

• Unit and integration tests covering the common scenarios were added

[timmy21]

maybe use below interface?

type MapIter interface {
	Next() bool
	Key() any
	Value() any
}

type OrderedMapV2 interface {
	Put(key any, value any)
	Range() MapIter
}

so we can use more efficient way to store map, for example: two slices

type Pairs struct {
	Keys []any
	Values []any
}

[hanjm]

maybe use below interface?

type MapIter interface {
	Next() bool
	Key() any
	Value() any
}

type OrderedMapV2 interface {
	Put(key any, value any)
	Range() MapIter
}

so we can use more efficient way to store map, for example: two slices

type Pairs struct {
	Keys []any
	Values []any
}

Looks great.

$ CLICKHOUSE_USE_DOCKER=false CLICKHOUSE_PASSWORD=tps go test -gcflags="-l" . -bench OrderedMap -run OrderedMap
using random seed 1702120904860719000 for native tests
goos: darwin
goarch: arm64
pkg: github.com/ClickHouse/clickhouse-go/v2/tests
BenchmarkOrderedMapUseChanGo-10                   584689              2023 ns/op             120 B/op          2 allocs/op
BenchmarkOrderedMapKeysUseChanNoGo-10            2493792               502.1 ns/op           256 B/op          2 allocs/op
BenchmarkOrderedMapKeysUseSlice-10              11242239               123.8 ns/op             0 B/op          0 allocs/op
BenchmarkOrderedMapKeysUseIter-10               61061322                19.83 ns/op           16 B/op          1 allocs/op
PASS
ok      github.com/ClickHouse/clickhouse-go/v2/tests    7.274s

[timmy21]

you can also benchmark reflect map

func BenchmarkReflectMapIter(b *testing.B) {
	m := make(map[int]int)
	for i := 0; i < 10; i++ {
		m[i] = i
	}
	b.ResetTimer()
	b.ReportAllocs()
	for i := 0; i < b.N; i++ {
		value := reflect.Indirect(reflect.ValueOf(m))
		iter := value.MapRange()
		for iter.Next() {
			_ = iter.Key().Interface()
			_ = iter.Value().Interface()
		}
	}
}

[hanjm]

add benchmark reflect map.

$ CLICKHOUSE_USE_DOCKER=false CLICKHOUSE_PASSWORD=tps go test -gcflags="-l" . -bench OrderedMap -run OrderedMap
using random seed 1702126424636270000 for native tests
goos: darwin
goarch: arm64
pkg: github.com/ClickHouse/clickhouse-go/v2/tests
BenchmarkOrderedMapUseChanGo-10                   600756              1970 ns/op             120 B/op          2 allocs/op
BenchmarkOrderedMapKeysUseChanNoGo-10            2467634               483.4 ns/op           256 B/op          2 allocs/op
BenchmarkOrderedMapKeysUseSlice-10              11058926               110.1 ns/op             0 B/op          0 allocs/op
BenchmarkOrderedMapKeysUseIter-10               20183755                57.78 ns/op           16 B/op          1 allocs/op
BenchmarkOrderedMapReflectMapIter-10             1732650               697.0 ns/op           448 B/op         21 allocs/op
PASS
ok      github.com/ClickHouse/clickhouse-go/v2/tests    8.459s

[jkaflik]

@hanjm thanks for your contribution! Before I jump into review, could you elaborate on this implementation please?

[timmy21]

@hanjm thanks for your contribution! Before I jump into review, could you elaborate on this implementation please?

The problem I ran into was that when writing to map columns, a lot of time was wasted in the reflect operation

[jkaflik]

@timmy21 I understand the reasoning, but I want to understand what is your proposal to solve this.

[timmy21]

@timmy21 I understand the reasoning, but I want to understand what is your proposal to solve this.

By defining a map interface, this allows user to store map using different data structure, so that avoiding the reflect overhead and improve current OrderedMap interface performance. for example use Pairs:

type Pairs struct {
	Keys []any
	Values []any
}

[jkaflik]

Overall it looks good. Only minor naming convention changes.

Would be great if we provide additional example.

[hanjm]

Overall it looks good. Only minor naming convention changes.

Would be great if we provide additional example.

@jkaflik, Thanks, done.