[BUG] fatal error: concurrent map read and map write

[skyerus]

Observed behavior

While repeatedly running the integration tests I ran into the following race condition which crashed flagd.

fatal error: concurrent map read and map write

goroutine 1065 [running]:
github.com/open-feature/flagd/pkg/eval.(*JSONEvaluator).evaluateVariant(0x1400007f160, {0x14001aaa060, 0x14}, {0x140019c6060, 0xb}, 0x14001aca240)
	/Users/sgill/web/openfeature/flagd/pkg/eval/json_evaluator.go:269 +0x5bc
github.com/open-feature/flagd/pkg/eval.resolve[...]({0x14001aaa060, 0x140019c6060?}, {0x140019c6060?, 0x1?}, 0x0?, 0x1?, 0x0?)
	/Users/sgill/web/openfeature/flagd/pkg/eval/json_evaluator.go:93 +0x48
github.com/open-feature/flagd/pkg/eval.(*JSONEvaluator).ResolveStringValue(0x1400007f160, {0x14001aaa060, 0x14}, {0x140019c6060, 0xb}, 0x0?)
	/Users/sgill/web/openfeature/flagd/pkg/eval/json_evaluator.go:175 +0xf4
github.com/open-feature/flagd/pkg/service.(*ConnectService).ResolveString(0x14000000360, {0x10497db9c?, 0x106b17880?}, 0x14002228000)
	/Users/sgill/web/openfeature/flagd/pkg/service/connect_service.go:291 +0x3cc
github.com/bufbuild/connect-go.NewUnaryHandler[...].func1({0x105f0b2c0, 0x14002228000})
	/Users/sgill/go/pkg/mod/github.com/bufbuild/connect-go@v1.5.0/handler.go:51 +0x14c
github.com/bufbuild/connect-go.NewUnaryHandler[...].func2({0x10e8256e0, 0x14000394040})
	/Users/sgill/go/pkg/mod/github.com/bufbuild/connect-go@v1.5.0/handler.go:75 +0x1ac
github.com/bufbuild/connect-go.(*Handler).ServeHTTP(0x1400070e7d0, {0x105efd058, 0x14000394020}, 0x140003c2100)
	/Users/sgill/go/pkg/mod/github.com/bufbuild/connect-go@v1.5.0/handler.go:229 +0x4a8
net/http.(*ServeMux).ServeHTTP(0x14001681668?, {0x105efd058, 0x14000394020}, 0x140003c2100)
	/usr/local/go/src/net/http/server.go:2487 +0x140
net/http.(*ServeMux).ServeHTTP(0x3252b890016816f8?, {0x105efd058, 0x14000394020}, 0x140003c2100)
	/usr/local/go/src/net/http/server.go:2487 +0x140
github.com/open-feature/flagd/pkg/service.Handler.func1.1()
	/Users/sgill/web/openfeature/flagd/pkg/service/connect_metrics.go:227 +0x38
github.com/open-feature/flagd/pkg/service.Middleware.Measure({{{0x105efd028, 0x14000533338}, {0x0, 0x0}, 0x0, 0x0, 0x0}}, {0x0?, 0x10593efa9?}, {0x105f0a580, ...}, ...)
	/Users/sgill/web/openfeature/flagd/pkg/service/connect_metrics.go:211 +0xfc
github.com/open-feature/flagd/pkg/service.Handler.func1({0x105f09538?, 0x140003f40e0}, 0x140003c2100)
	/Users/sgill/web/openfeature/flagd/pkg/service/connect_metrics.go:226 +0x164
net/http.HandlerFunc.ServeHTTP(0x14000142000?, {0x105f09538?, 0x140003f40e0?}, 0x140003c2100?)
	/usr/local/go/src/net/http/server.go:2109 +0x38
github.com/rs/cors.(*Cors).Handler.func1({0x105f09538, 0x140003f40e0}, 0x140003c2100)
	/Users/sgill/go/pkg/mod/github.com/rs/cors@v1.8.3/cors.go:236 +0x1e4
net/http.HandlerFunc.ServeHTTP(0x106b17880?, {0x105f09538?, 0x140003f40e0?}, 0x0?)
	/usr/local/go/src/net/http/server.go:2109 +0x38
golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP({{0x105ef6260?, 0x14000708f40?}, 0x1400070ebe0?}, {0x105f09538, 0x140003f40e0}, 0x140003c2100)
	/Users/sgill/go/pkg/mod/golang.org/x/net@v0.5.0/http2/h2c/h2c.go:125 +0x41c
net/http.serverHandler.ServeHTTP({0x105efb628?}, {0x105f09538, 0x140003f40e0}, 0x140003c2100)
	/usr/local/go/src/net/http/server.go:2947 +0x2c4
net/http.(*conn).serve(0x14000338820, {0x105f0a318, 0x140001f4ea0})
	/usr/local/go/src/net/http/server.go:1991 +0x560
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3102 +0x444

Expected Behavior

No such crash.

Steps to reproduce

Run the integration tests, occurs when the flag change notification causes a flag state mutation at the same time as a flag is being retrieved.

[toddbaert]

ohh, nice one.