[chore] [exporterhelper] Integrate capacity limiting into the communication channel #9232

dmitryax · 2024-01-06T03:44:48Z

Integrate capacity limiting into internal channels used by both memory and persistent queues. Otherwise, with the independent capacity limiter, it's hard to ensure that queue size is always accurate going forward.

Benchmarks before:

goos: darwin
goarch: arm64
Benchmark_QueueUsage_1000_requests-10      	    3252	    325010 ns/op	  246059 B/op	      10 allocs/op
Benchmark_QueueUsage_100000_requests-10    	      39	  29811116 ns/op	24002870 B/op	      10 allocs/op
Benchmark_QueueUsage_10000_items-10        	    3404	    349753 ns/op	  246052 B/op	      10 allocs/op
Benchmark_QueueUsage_1M_items-10           	      40	  29415583 ns/op	24002858 B/op	      10 allocs/op
BenchmarkPersistentQueue_TraceSpans
BenchmarkPersistentQueue_TraceSpans/#traces:_1_#spansPerTrace:_1-10         	  338180	      3836 ns/op	    2851 B/op	      78 allocs/op
BenchmarkPersistentQueue_TraceSpans/#traces:_1_#spansPerTrace:_10-10        	   81369	     15822 ns/op	   14598 B/op	     289 allocs/op
BenchmarkPersistentQueue_TraceSpans/#traces:_10_#spansPerTrace:_10-10       	   13066	     90155 ns/op	  130087 B/op	    2417 allocs/op

Benchmarks after:

Benchmark_QueueUsage_1000_requests-10      	    4210	    278175 ns/op	  246055 B/op	      10 allocs/op
Benchmark_QueueUsage_100000_requests-10    	      42	  25835945 ns/op	24002968 B/op	      10 allocs/op
Benchmark_QueueUsage_10000_items-10        	    4376	    279571 ns/op	  246056 B/op	      10 allocs/op
Benchmark_QueueUsage_1M_items-10           	      42	  26483907 ns/op	24002995 B/op	      10 allocs/op
BenchmarkPersistentQueue_TraceSpans
BenchmarkPersistentQueue_TraceSpans/#traces:_1_#spansPerTrace:_1-10         	  328268	      4251 ns/op	    2854 B/op	      78 allocs/op
BenchmarkPersistentQueue_TraceSpans/#traces:_1_#spansPerTrace:_10-10        	  101683	     12238 ns/op	   14582 B/op	     289 allocs/op
BenchmarkPersistentQueue_TraceSpans/#traces:_10_#spansPerTrace:_10-10       	   13382	     86464 ns/op	  130154 B/op	    2417 allocs/op

codecov · 2024-01-06T03:48:01Z

Codecov Report

Attention: Patch coverage is 88.18182% with 13 lines in your changes are missing coverage. Please review.

Project coverage is 91.86%. Comparing base (67d3718) to head (999718d).

Files	Patch %	Lines
exporter/internal/queue/persistent_queue.go	77.96%	10 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9232      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.02%     
==========================================
  Files         360      361       +1     
  Lines       16717    16722       +5     
==========================================
+ Hits        15359    15361       +2     
- Misses       1020     1024       +4     
+ Partials      338      337       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

exporter/exporterhelper/internal/persistent_queue.go

exporter/exporterhelper/internal/queue.go

exporter/exporterhelper/internal/sized_elements_channel.go

exporter/internal/queue/sized_elements_channel.go

github-actions · 2024-03-09T03:16:09Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2024-03-26T03:15:20Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2024-04-11T03:15:29Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2024-05-01T03:16:21Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

exporter/internal/queue/bounded_memory_queue.go

exporter/internal/queue/sized_elements_channel.go

bogdandrutu · 2024-05-02T15:22:26Z

exporter/internal/queue/sized_elements_channel.go

+
+// withPreloadElements puts the elements into the queue with the given size. It's used by the persistent queue to
+// initialize the queue with the elements recovered from the disk.
+func withPreloadElements[T any](els []T, totalSize int64) sizedElementsChannelOption[T] {


To simplify this, probably next PR, we need to have 2 sizes for persistent:

in memory size which is fixed and smaller than the storage;

storage size which is how much to store in the storage;

sounds good 👍

exporter/internal/queue/sized_elements_channel.go

bogdandrutu · 2024-05-02T15:30:33Z

exporter/internal/queue/sized_elements_channel.go

+// syncSize updates the used size to 0 if the queue is empty.
+// The caller must ensure that this call is not called concurrently with enqueue.
+// It's used by the persistent queue to ensure the used value correctly reflects the reality which may not be always
+// the case in case if the queue size is restored from the disk after a crash.


Can you explain why? Because we don't calculate the size correctly?

Because we don't flush the current queue size on the disk on every read/write. Should I mention that in the comment?

bogdandrutu · 2024-05-02T15:32:40Z

exporter/internal/queue/sized_elements_channel.go

+// enqueue puts the element into the queue with the given sized if there is enough capacity.
+// Returns an error if the queue is full. The callback is called before the element is committed to the queue.
+// If the callback returns an error, the element is not put into the queue and the error is returned.
+func (vcq *sizedElementsChannel[T]) enqueue(el T, size int64, callback func() error) error {


Add a comment that size MUST be positive. We can even consider to change it to uint64? That my make the -size ugly to use with atomics, but I think it is ok.

Added the comment. The atomic size field used to be in uint64, and then you asked to change it to int64 to simplify the subtraction. Do you think we should accept uint64 and convert it to int64?

exporter/internal/queue/sized_elements_channel.go

exporter/internal/queue/persistent_queue.go

bogdandrutu · 2024-05-02T15:42:18Z

exporter/internal/queue/sized_elements_channel.go

+		opt(sech)
+	}
+	if sech.ch == nil {
+		sech.ch = make(chan T, capacity)


This will be a problem if we switch to use bytes, since it will be very large and lots of memory.

Right. We can think of a solution outside of this PR?

bogdandrutu

Ship it :)

Integrate capacity limiting into internal channels used by both memory and persistent queues. Otherwise, with the independent capacity limiter, it's hard to ensure that queue size is always accurate going forward.

…cation channel (open-telemetry#9232) Integrate capacity limiting into internal channels used by both memory and persistent queues. Otherwise, with the independent capacity limiter, it's hard to ensure that queue size is always accurate going forward. Benchmarks before: ``` goos: darwin goarch: arm64 Benchmark_QueueUsage_1000_requests-10 3252 325010 ns/op 246059 B/op 10 allocs/op Benchmark_QueueUsage_100000_requests-10 39 29811116 ns/op 24002870 B/op 10 allocs/op Benchmark_QueueUsage_10000_items-10 3404 349753 ns/op 246052 B/op 10 allocs/op Benchmark_QueueUsage_1M_items-10 40 29415583 ns/op 24002858 B/op 10 allocs/op BenchmarkPersistentQueue_TraceSpans BenchmarkPersistentQueue_TraceSpans/#traces:_1_#spansPerTrace:_1-10 338180 3836 ns/op 2851 B/op 78 allocs/op BenchmarkPersistentQueue_TraceSpans/#traces:_1_#spansPerTrace:_10-10 81369 15822 ns/op 14598 B/op 289 allocs/op BenchmarkPersistentQueue_TraceSpans/#traces:_10_#spansPerTrace:_10-10 13066 90155 ns/op 130087 B/op 2417 allocs/op ``` Benchmarks after: ``` Benchmark_QueueUsage_1000_requests-10 4210 278175 ns/op 246055 B/op 10 allocs/op Benchmark_QueueUsage_100000_requests-10 42 25835945 ns/op 24002968 B/op 10 allocs/op Benchmark_QueueUsage_10000_items-10 4376 279571 ns/op 246056 B/op 10 allocs/op Benchmark_QueueUsage_1M_items-10 42 26483907 ns/op 24002995 B/op 10 allocs/op BenchmarkPersistentQueue_TraceSpans BenchmarkPersistentQueue_TraceSpans/#traces:_1_#spansPerTrace:_1-10 328268 4251 ns/op 2854 B/op 78 allocs/op BenchmarkPersistentQueue_TraceSpans/#traces:_1_#spansPerTrace:_10-10 101683 12238 ns/op 14582 B/op 289 allocs/op BenchmarkPersistentQueue_TraceSpans/#traces:_10_#spansPerTrace:_10-10 13382 86464 ns/op 130154 B/op 2417 allocs/op ```

dmitryax requested a review from a team as a code owner January 6, 2024 03:44

dmitryax requested a review from Aneurysm9 January 6, 2024 03:44

dmitryax changed the title ~~[chore] [exporterhelper] Integrate capacity limiting into a helper queue~~ [chore] [exporterhelper] Integrate capacity limiting into the communication channel Jan 6, 2024

bogdandrutu reviewed Jan 6, 2024

View reviewed changes

exporter/exporterhelper/internal/persistent_queue.go Outdated Show resolved Hide resolved

dmitryax force-pushed the itemized-queue-with-limiter branch 3 times, most recently from e7abbb5 to f994b06 Compare January 8, 2024 19:10

bogdandrutu reviewed Jan 29, 2024

View reviewed changes

exporter/exporterhelper/internal/queue.go Outdated Show resolved Hide resolved

dmitryax force-pushed the itemized-queue-with-limiter branch from e9e6ea6 to 6b47a4d Compare January 30, 2024 00:30

sfc-gh-bdrutu reviewed Jan 30, 2024

View reviewed changes

dmitryax force-pushed the itemized-queue-with-limiter branch 2 times, most recently from e57ef73 to 92d5cca Compare January 31, 2024 19:28

bogdandrutu reviewed Feb 3, 2024

View reviewed changes

exporter/exporterhelper/internal/sized_elements_channel.go Outdated Show resolved Hide resolved

bogdandrutu reviewed Feb 3, 2024

View reviewed changes

dmitryax mentioned this pull request Feb 5, 2024

[exporterhelper] Persistent queue can block collector start-up #9451

Closed

dmitryax force-pushed the itemized-queue-with-limiter branch 7 times, most recently from 65d4ef3 to f45b7e6 Compare February 10, 2024 06:43

dmitryax force-pushed the itemized-queue-with-limiter branch 4 times, most recently from 86baed4 to 8827d63 Compare February 12, 2024 21:33

sfc-gh-bdrutu reviewed Feb 12, 2024

View reviewed changes

exporter/internal/queue/sized_elements_channel.go Outdated Show resolved Hide resolved

dmitryax force-pushed the itemized-queue-with-limiter branch from 8827d63 to 418f5ba Compare February 13, 2024 19:23

dmitryax force-pushed the itemized-queue-with-limiter branch from 418f5ba to 70d723d Compare February 22, 2024 22:34

dmitryax force-pushed the itemized-queue-with-limiter branch 3 times, most recently from 308ee38 to cd3b7c6 Compare February 23, 2024 17:08

github-actions bot added the Stale label Mar 9, 2024

dmitryax removed the Stale label Mar 9, 2024

github-actions bot added the Stale label Mar 26, 2024

dmitryax removed the Stale label Mar 26, 2024

github-actions bot added the Stale label Apr 11, 2024

dmitryax removed the Stale label Apr 11, 2024

github-actions bot added the Stale label May 1, 2024

dmitryax removed the Stale label May 1, 2024

dmitryax force-pushed the itemized-queue-with-limiter branch from cd3b7c6 to 04e3a1f Compare May 2, 2024 04:20

bogdandrutu approved these changes May 2, 2024

View reviewed changes

bogdandrutu reviewed May 2, 2024

View reviewed changes

dmitryax force-pushed the itemized-queue-with-limiter branch 2 times, most recently from d9d6892 to 4afa371 Compare May 3, 2024 18:42

bogdandrutu approved these changes May 3, 2024

View reviewed changes

dmitryax force-pushed the itemized-queue-with-limiter branch from 4afa371 to aae0856 Compare May 3, 2024 22:34

[chore] [exporterhelper] Integrate capacity limiting into a helper queue

999718d

Integrate capacity limiting into internal channels used by both memory and persistent queues. Otherwise, with the independent capacity limiter, it's hard to ensure that queue size is always accurate going forward.

dmitryax force-pushed the itemized-queue-with-limiter branch from aae0856 to 999718d Compare May 4, 2024 05:07

dmitryax merged commit b7b7e51 into open-telemetry:main May 4, 2024
47 of 48 checks passed

github-actions bot added this to the next release milestone May 4, 2024

dmitryax deleted the itemized-queue-with-limiter branch May 4, 2024 17:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[chore] [exporterhelper] Integrate capacity limiting into the communication channel #9232

[chore] [exporterhelper] Integrate capacity limiting into the communication channel #9232

dmitryax commented Jan 6, 2024 •

edited

Loading

codecov bot commented Jan 6, 2024 •

edited

Loading

github-actions bot commented Mar 9, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Apr 11, 2024

github-actions bot commented May 1, 2024

bogdandrutu May 2, 2024

dmitryax May 3, 2024

bogdandrutu May 2, 2024

dmitryax May 3, 2024

bogdandrutu May 2, 2024

dmitryax May 3, 2024

bogdandrutu May 2, 2024

dmitryax May 3, 2024 •

edited

Loading

bogdandrutu left a comment

[chore] [exporterhelper] Integrate capacity limiting into the communication channel #9232

[chore] [exporterhelper] Integrate capacity limiting into the communication channel #9232

Conversation

dmitryax commented Jan 6, 2024 • edited Loading

codecov bot commented Jan 6, 2024 • edited Loading

Codecov Report

github-actions bot commented Mar 9, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Apr 11, 2024

github-actions bot commented May 1, 2024

bogdandrutu May 2, 2024

Choose a reason for hiding this comment

dmitryax May 3, 2024

Choose a reason for hiding this comment

bogdandrutu May 2, 2024

Choose a reason for hiding this comment

dmitryax May 3, 2024

Choose a reason for hiding this comment

bogdandrutu May 2, 2024

Choose a reason for hiding this comment

dmitryax May 3, 2024

Choose a reason for hiding this comment

bogdandrutu May 2, 2024

Choose a reason for hiding this comment

dmitryax May 3, 2024 • edited Loading

Choose a reason for hiding this comment

bogdandrutu left a comment

Choose a reason for hiding this comment

dmitryax commented Jan 6, 2024 •

edited

Loading

codecov bot commented Jan 6, 2024 •

edited

Loading

dmitryax May 3, 2024 •

edited

Loading