pfree toasted Arrays on drop #1571
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For tables with many rows, or that include enough large arrays, being unable to pfree the entire array can cause needless memory pressure. As pgrx used to not bind objects to the lifetime of backing arrays, the assigned lifetime of any 'obj could exceed `&'arr Array<'_, T>`. This rendered any Drop implementation for Array or any of its parts unsound. The references, expecting to be valid for the underlying 'mcx or even longer, would yield clobbered memory when actually examined. This has finally become untrue with the introduction of UnboxDatum. It has an `unsafe trait` obligation that the implementation not allow lifetimes to exceed the Datum's lifetime except if the data is always inherently 'static or Copy. This allows deallocating detoasted items, as Arrays now only yield an object of an appropriate lifetime. It is possible for us to get this wrong, but only by implementing UnboxDatum in an incorrect, unsound way, so the fix would be to that code instead.
|
|
|
I’m unsure what you could have done to use up all the connection slots? Hanging processes or something from prior test runs? Very unlikely. |
|
It happens every now and then and usually clears on a rerun or clean-then-rerun. I think there's a problem in the test framework where it doesn't anticipate "CPU go brrr". |
NotGyro
approved these changes
Feb 21, 2024
|
I forget offhand what the default connection limit is. Part of me thinks it’s 100 but the other part thinks it’s 10. We should look that up and adjust our postmaster options accordingly |
|
It’s 100* https://www.postgresql.org/docs/current/runtime-config-connection.html *unless your kernel is dumb. |
|
Rust wouldn’t be generating a test binary that fires off more threads than you have CPUs, would it? you got 128 threads?! |
No.
I have no idea how many threads it generates! I have only about 16 logical threads locally. |
workingjubilee
added a commit
that referenced
this pull request
Mar 1, 2024
Welcome to pgrx 0.12.0-alpha.1! Say the magic words with me! ```shell cargo install cargo-pgrx --locked --version 0.12.0-alpha.1 ``` # Breaking Changes ## No more dlopen! Perhaps the most exciting change this round is @usamoi's contribution in #1468 which means that we no longer perform a `dlopen` in order to generate the schema. The cost, such as it is, is that your pgrx extensions now require a `src/bin/pgrx_embed.rs`, which will be used to generate the schema. This has much less cross-platform issues and will enable supporting things like `cargo binstall` down the line. It may be a bit touchy on first-time setup for transitioning older repos. If necessary, you may have to directly add a `src/bin/pgrx_embed.rs` and add the following code (which should be the only code in the file, though you can add comments if you like?): ```rust ::pgrx::pgrx_embed!(); ``` Your Cargo.toml will also want to update its crate-type key for the library: ```toml [lib] crate-type = ["cdylib", "lib"] ``` ## Library Code - pgrx-pg-sys will now use `ManuallyDropUnion` thanks to @NotGyro in #1547 - VARHDRSZ `const`s are no longer `fn`, thanks to @workingjubilee in #1584 - We no longer have `Interval::is_finite` since #1594 - We translate more `*_tree_walker` functions to the same signature their `*_impl` version in Postgres 16 has: #1596 - Thanks to @eeeebbbbrrrr in #1591 we no longer have the `pg_sql_graph_magic!()` macro, which should help with more things in the future! # What's New We have quite a lot of useful additions to our API: - `SpiClient::prepare_mut` was added thanks to @XeniaLu in #1275 - @usamoi also contributed bindings subscripting code in #1562 - For `#[pg_test]`, you have been able to use `#[should_panic(expected = "string")]` to anticipate a panic that contains that string in that test. For various reasons, `#[pg_test(error = "string")]` is much the same. Now, you can also use `#[pg_test(expected = "string")]`, in the hopes that is easier to stumble across, as of #1570 ## `Result<composite_type!("..."), E>` support - In #1560 @NotGyro contributed support for using `Result<composite_type!("Name"), E>`, as a case that had not been handled before. ## Significantly expanded docs Thanks to @rjuju, @NotGyro, and @workingjubilee, we now have significantly expanded docs for cargo-pgrx and pgrx in general. Some of these are in the API docs on https://docs.rs or the READMEs, but there's also a guide, now! It's not currently published, but is available as an [mdbook](https://github.com/rust-lang/mdBook) in the repo. Some diagnostic information that is also arguably documentation, like comments and the suggestion to `cargo install`, have also been improved, thanks to @workingjubilee in - #1579 - #1573 ## `#[pg_cast]` An experimental macro for a `CREATE CAST` was contributed by @xwkuang5 in #1445! ## Legal Stuff Thanks to @the-kenny in #1490 and @workingjubilee in #1504, it was brought to our attention that some dependencies had unusual legal requirements. So we fixed this with CI! We now check our code included into pgrx-using binaries is MIT/Apache 2.0 licensed, as is common across crates.io, using `cargo deny`!. The build tools will have more flexible legal requirements (partly due to the use of Mozilla Public License code in rustls). # Internal Changes Many internal cleanups were done thanks to - @workingjubilee in too many PRs to count! - @thomcc found a needless condition in #1501 - @nyurik in too many PRs to count! In particular: - we now actually `pfree` our `Array`s we detoasted as-of #1571 - creating a `RawArray` is now low-overhead due to #1587 ## Soundness Fixes We had a number of soundness issues uncovered or have added more tests to catch them. - Bounds-checking debug assertions for array access by @NotGyro in #1514 - Fix unsound `&` and `&mut` in `fcinfo.rs` by @workingjubilee in #1595 ## Less Deps Part of the cleanup by @workingjubilee was reducing the number of deps we compile: * cargo-pgrx: reduce trivial dep usages in #1499 * Update 2 syn in #1557 Hopefully it will reduce compile time and disk usage! ## New Contributors * @the-kenny made their first contribution in #1490 * @xwkuang5 made their first contribution in #1445 * @rjuju made their first contribution in #1516 * @nyurik made their first contribution in #1533 * @NotGyro made their first contribution in #1514 * @XeniaLu made their first contribution in #1275 **Full Changelog**: v0.12.0-alpha.0...v0.12.0-alpha.1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For tables with many rows, or that include enough large arrays, being unable to pfree the entire array can cause needless memory pressure. As pgrx used to not bind objects to the lifetime of backing arrays, the assigned lifetime of any 'obj could exceed
&'arr Array<'_, T>. This rendered any Drop implementation for Array or any of its parts unsound. The references, expecting to be valid for the underlying 'mcx or even longer, would yield clobbered memory when actually examined.This has finally become untrue with the introduction of UnboxDatum. It has an
unsafe traitobligation that the implementation not allow lifetimes to exceed the Datum's lifetime except if the data is always inherently 'static or Copy. This allows deallocating detoasted items, as Arrays now only yield an object of an appropriate lifetime. It is possible for us to get this wrong, but only by implementing UnboxDatum in an incorrect, unsound way, so the fix would be to that code instead.