Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[release-1.10] dont reset maxsize in jl_array_to_string #55689

Merged
merged 1 commit into from
Sep 12, 2024

Conversation

d-netto
Copy link
Member

@d-netto d-netto commented Sep 4, 2024

Let's change jl_array_to_string so that we make the consequences of calling it in a thread-unsafe way less disastrous (i.e. let's avoid corrupting GC internal metrics).

Strictly speaking, this is not a bug-fix because calling jl_array_to_string concurrently from two threads is UB in any case.

To see how a race here may lead to negative live_bytes, consider this MWE from @NHDaly:

  • 1.10:
julia> GC.gc(true); Base.gc_live_bytes()
1842370

julia> g_vecs = Any[
                  UInt8['a' for _ in 1:1000000000]
                  for _ in 1:10
                  ];

julia> GC.gc(true); Base.gc_live_bytes()
10001774906

julia> Threads.@threads for _ in 1:1000
                  for v in g_vecs
                      String(v)
                  end
              end

julia> GC.gc(true); Base.gc_live_bytes()
-1997600207
  • This patch:
julia> GC.gc(true); Base.gc_live_bytes()
1862440

julia> g_vecs = Any[
                  UInt8['a' for _ in 1:1000000000]
                  for _ in 1:10
                  ];

julia> GC.gc(true); Base.gc_live_bytes()
10001796440

julia> Threads.@threads for _ in 1:1000
                  for v in g_vecs
                      String(v)
                  end
              end

julia> GC.gc(true); Base.gc_live_bytes()
10002390952

@d-netto d-netto added the GC Garbage collector label Sep 4, 2024
@oscardssmith
Copy link
Member

The issue with this is that it then becomes UB to call jl_array_to_string and then push! to the vector

@d-netto
Copy link
Member Author

d-netto commented Sep 4, 2024

The issue with this is that it then becomes UB to call jl_array_to_string and then push! to the vector

Not sure if I see why?

The return jl_pchar_to_string((const char*)jl_array_data(a), len); statement at the end of the function is allocating a new buffer for the String either way.

@d-netto
Copy link
Member Author

d-netto commented Sep 11, 2024

Bump.

@NHDaly
Copy link
Member

NHDaly commented Sep 11, 2024

Yeah, @oscardssmith I don't think you have that quite right.
This code is pretty tricky, and took us quite some time to wrap our heads around it. It is surely lacking in comments. (We would fix that, but the whole thing is deleted after the memory PR.)

Is there any chance that you misread things? Can you give it a closer read?

As diogo says: in the branch this PR modifies, the data is not being shared, it's being copied. The only reason this branch resets the values in the array is to be consistent with the branch where it is shared. So after this change, that branch will logically clear the array, in all user-visible ways, but it will allow the array to keep accounting for its memory, so that the memory accounting is correct.

Copy link
Member

@NHDaly NHDaly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM. I'm pretty sure this is the right fix, and it resolved the issues from our end.

@oscardssmith if you could give it another look that would be helpful, but i have medium-high confidence in this approach and I'm comfortable approving+merging.

@KristofferC KristofferC changed the base branch from release-1.10 to backports-release-1.10 September 12, 2024 08:56
@KristofferC KristofferC merged commit c5ab7c9 into backports-release-1.10 Sep 12, 2024
5 of 7 checks passed
@KristofferC KristofferC deleted the dcn-dont-reset-maxsize branch September 12, 2024 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GC Garbage collector
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants