Instrumented net/http client sends Traceparent header without SAMPLED flag

[eugeneRover]

Describe the bug

Instrumented net/http client sends Traceparent header without SAMPLED flag

Environment

• OS: Debian Linux 6.1.0-22-amd64 x86_64
• Go Version: 1.22.5
• Version: v0.14.0-alpha

To Reproduce

Steps to reproduce the behavior:

1. Create simple HTTP service (test_service) that listens on 8080 and calls another one HTTP service (ping-service):

package main

import (
	"io"
	"log"
	"net/http"
)

const PING_SERVICE_URL = "http://127.0.0.1:8081/ping"

func main() {
	http.HandleFunc("GET /index", func(resp http.ResponseWriter, req *http.Request) {

		newReq, _ := http.NewRequestWithContext(req.Context(), "GET", PING_SERVICE_URL, nil)
		newResp, _ := http.DefaultClient.Do(newReq)
		b, _ := io.ReadAll(newResp.Body)

		resp.Write([]byte("Remote answer: "))
		resp.Write(b)
	})

	log.Fatal(http.ListenAndServe(":8080", nil))
}

2. Create that ping-service, download this and launch it: ./server.py 8081
3. Launch instrumentation:

OTEL_GO_AUTO_TARGET_EXE=/service/test_service \
OTEL_SERVICE_NAME=test_service \
OTEL_PROPAGATORS=tracecontext,baggage \
/otel-go-instrumentation

Its log shows everything allright:

{"level":"info","ts":1723011542.0669825,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:86","msg":"building OpenTelemetry Go instrumentation ...","globalImpl":false}
{"level":"info","ts":1723011544.0729582,"logger":"Instrumentation.Analyzer","caller":"process/discover.go:67","msg":"found process","pid":1929962}
{"level":"info","ts":1723011544.0747728,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:73","msg":"Detaching from process","pid":1929962}
{"level":"info","ts":1723011544.074794,"logger":"Instrumentation","caller":"app/instrumentation.go:144","msg":"target process analysis completed","pid":1929962,"go_version":"1.22.5","dependencies":{"std":"1.22.5"},"total_functions_found":3}
{"level":"info","ts":1723011544.0748572,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:119","msg":"starting instrumentation..."}
{"level":"info","ts":1723011544.0749462,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:197","msg":"loading probe","name":"net/http/server"}
{"level":"info","ts":1723011544.3144674,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:197","msg":"loading probe","name":"net/http/client"}
{"level":"info","ts":1723011544.326916,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:115","msg":"instrumentation loaded successfully"}

4. Trigger request to test_service :

curl  http://127.0.0.1:8080/index

5. See the log of ping-service:

INFO:root:GET request,
Path: /ping
Headers:
Host: 127.0.0.1:8081
User-Agent: Go-http-client/1.1
Traceparent: 00-5d90f17caf1774ed829ea564b0143cc8-a27b927333936f8a-00
Accept-Encoding: gzip

Expected behavior

Expected Traceparent header with SAMPLED flag set, like this :

Traceparent: 00-5d90f17caf1774ed829ea564b0143cc8-a27b927333936f8a-01

Additional context

Traceparent with no SAMPLED flag set prevents ping-service from tracing (assuming ping-service is also instrumented).

Links

@RonFed
https://github.com/open-telemetry/opentelemetry-go-instrumentation/releases/tag/v0.14.0-alpha
#953
#868

[RonFed]

@eugeneRover Thank you for reporting this.
I've reproduced this in the v0.14.0-alpha tag. I also tested this in our latest main and it is fixed now, so this bug won't occur in the next release.

As you correctly linked, the sampling support is under development and refactoring, so the v0.14-0 release was published during this refactoring., and before #949 which solved this problem.

The reason this was not caught in the tests is that our e2e test has coverage for HTTP clients as root span and not as a child.
@MrAlias @damemi

[RonFed]

@eugeneRover Closing this for now as fixed in the v0.15.0-alpha release. If the problem still happens in the new release feel free to re-open.