BUG - Duplicate trace ids & span by different hosts in distributed system

[fularac]

We've rolled out our use in production! With that, we've noticed that there are duplicate trace ids and trace parent id combinations. Our logs show that this is happening with incoming requests that do not have a traceparent header and they are on separate hosts. Our current configuration creates launches 20 hosts with 36 workers all around the same time. This is happening for .003 % of requests over a 4 hour period, but still concerning.

I suspect the math.randomseed seed is not sufficiently random to prevent the same seed from being used in different hosts.

opentelemetry-lua/lib/opentelemetry/util.lua

Line 4 in f72ebd3

math.randomseed(ngx.time() + ngx.worker.pid())

Problems I see with using ngx.time() + ngx.worker.pid():

• ngx.time() is returning seconds https://github.com/openresty/lua-nginx-module?tab=readme-ov-file#ngxtime

  Returns the elapsed seconds from the epoch for the current time stamp from the Nginx cached time (no syscall involved unlike Lua's date library).

• The PID across our hosts will be fairly similar values as they all launch from the same state.
• Launching multiple hosts with x many workers creates a x many second window where the random generator can be seeded the same between host. In my case with 36 workers per host, there's a 36 second window

Some improvements/fixes that would help with this issue:

• Use higher resolution timestamp e.g. ngx.now (Something with nano seconds?)
• Leverage host level random generator as the seed
• Expose ability to set math.randomseed for utility module

[yangxikun]

Thanks! I will fix this bug by random_seed() in skywalking-nginx-lua.

[fularac]

@yangxikun can we cut a version for this change?

[yangxikun]

🎉 https://luarocks.org/modules/yangxikun/opentelemetry-lua/0.2-6