timescaledb_toolkit: init at 1.14.0

[typetetris]

1. provide the function buildPgxExtension to build postgresql extensions with cargo-pgx
2. provide timescaledb_toolkit extension to postgresql build via buildPgxExtension
3. provide nixos tests making sure resulting timescaledb_toolkit extension can be loaded and used via postgresql nixos module

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • nixos test at least for postgresql_14
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[typetetris]

@winterqt @Hoverbear Maybe you want to take a look again.

[SuperSandro2000]

@marsam can you please take a look at this?

Also please squash the review commits to follow the contributing guide.

[typetetris]

Cross compilation still fails:

at 19:30:46 ❯ nix-build . --arg crossSystem '{config="armv7l-unknown-linux-gnueabihf"; }' -A postgresqlPackages.timescaledb_toolkit
this derivation will be built:
  /nix/store/4g9k3gd471m8159z2r598h61zpmlampn-timescaledb_toolkit-armv7l-unknown-linux-gnueabihf-1.11.0.drv
building '/nix/store/4g9k3gd471m8159z2r598h61zpmlampn-timescaledb_toolkit-armv7l-unknown-linux-gnueabihf-1.11.0.drv'...
unpacking sources
unpacking source archive /nix/store/n2dy6l5in80y08124w638fwqzznbrrwg-source
source root is source
Executing cargoSetupPostUnpackHook
unpacking source archive /nix/store/yfqjw70f43d50q6s5cshahwksc3zl75n-timescaledb_toolkit-1.11.0-vendor.tar.gz
Finished cargoSetupPostUnpackHook
patching sources
Executing cargoSetupPostPatchHook
Validating consistency between /build/source/Cargo.lock and /build/timescaledb_toolkit-1.11.0-vendor.tar.gz/Cargo.lock
Finished cargoSetupPostPatchHook
updateAutotoolsGnuConfigScriptsPhase
configuring
building
Executing cargo-pgx buildPhase
/nix/store/l43k9kqpwdr2a85d278www19k4s6609g-stdenv-linux/setup: line 1403: /nix/store/cdkf0n2g9zf7vknbjp4j4di503dkq3y4-cargo-pgx-armv7l-unknown-linux-gnueabihf-0.4.5/bin/cargo-pgx: cannot execute binary file: Exec format error
error: builder for '/nix/store/4g9k3gd471m8159z2r598h61zpmlampn-timescaledb_toolkit-armv7l-unknown-linux-gnueabihf-1.11.0.drv' failed with exit code 126;
       last 10 log lines:
       > Finished cargoSetupPostUnpackHook
       > patching sources
       > Executing cargoSetupPostPatchHook
       > Validating consistency between /build/source/Cargo.lock and /build/timescaledb_toolkit-1.11.0-vendor.tar.gz/Cargo.lock
       > Finished cargoSetupPostPatchHook
       > updateAutotoolsGnuConfigScriptsPhase
       > configuring
       > building
       > Executing cargo-pgx buildPhase
       > /nix/store/l43k9kqpwdr2a85d278www19k4s6609g-stdenv-linux/setup: line 1403: /nix/store/cdkf0n2g9zf7vknbjp4j4di503dkq3y4-cargo-pgx-armv7l-unknown-linux-gnueabihf-0.4.5/bin/cargo-pgx: cannot execute binary file: Exec format error
       For full logs, run 'nix log /nix/store/4g9k3gd471m8159z2r598h61zpmlampn-timescaledb_toolkit-armv7l-unknown-linux-gnueabihf-1.11.0.drv'.

it somehow uses cargo-pgx compiled for the host platform, despite it being listed in nativeBuildInputs and should be used in the version compiled for the build platform, or not? I have to admit, I don't understand that very well.

[typetetris]

Here are some evil hacks with which cargo-pgx/postgresql run at build time, but it now fails at a different place:

Code: typetetris@fa14c5e

Failure:

   Compiling pgx-utils v0.4.5                                                                                                                                 
   Compiling nalgebra v0.27.1                                                                                                                                 
   Compiling pgx-pg-sys v0.4.5                                                                                                                                
   Compiling pgx-macros v0.4.5                                                                                                                                
error: failed to run custom build command for `pgx-pg-sys v0.4.5`                                                                                             
                                                                                                                                                              
Caused by:                                                                                                                                                    
  process didn't exit successfully: `/build/source/target/release/build/pgx-pg-sys-525c36ceb182b0cd/build-script-build` (exit status: 101)                    
  --- stdout                                                                                                                                                  
  cargo:rerun-if-env-changed=PGX_PG_SYS_GENERATE_BINDINGS_FOR_RELEASE                                                                                         
  cargo:rerun-if-changed=/build/tmp.HYSAByhqtc/config.toml                                                                                                    
  cargo:rerun-if-changed=include/pg10.h                                                                                                                       
  cargo:rerun-if-changed=include/pg11.h                                        
  cargo:rerun-if-changed=include/pg12.h                                                                                                                       
  cargo:rerun-if-changed=include/pg13.h                                                                                                                       
  cargo:rerun-if-changed=include/pg14.h                                                                                                                       
  cargo:rerun-if-changed=cshim/pgx-cshim.c                                                                                                                    
  cargo:rerun-if-changed=cshim/Makefile                                                                                                                       
                                                                                                                                                              
  --- stderr                                                                                                                                                  
  manifest_dir=/build/timescaledb_toolkit-1.11.0-vendor.tar.gz/pgx-pg-sys                                                                                     
  shim_src=/build/timescaledb_toolkit-1.11.0-vendor.tar.gz/pgx-pg-sys/cshim                                                                                   
  shim_dst=/build/source/target/release/build/pgx-pg-sys-2320b93506f5120f/out/cshim                                                                           
  Generating bindings for pg14                                                                                                                                
  /nix/store/p6knl0hcfah9myg5xzn5kxd1kjp3fiy5-glibc-armv7l-unknown-linux-gnueabihf-2.35-163-dev/include/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-soft.h' file not found                                                                                                                                                    
  /nix/store/p6knl0hcfah9myg5xzn5kxd1kjp3fiy5-glibc-armv7l-unknown-linux-gnueabihf-2.35-163-dev/include/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-soft.h' file not found, err: true                                                                                                                                         
  thread 'main' panicked at 'Unable to generate bindings for pg14: ()', /build/timescaledb_toolkit-1.11.0-vendor.tar.gz/pgx-pg-sys/build.rs:530:13            
  stack backtrace:                                                                                                                                            
     0:     0x555555e3898c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hf2ea00d5f3b53d9f                             
     1:     0x555555e6a3de - core::fmt::write::ha076a7e65c6b1874                                                                                              
     2:     0x555555e4a671 - std::io::Write::write_fmt::h9630bb3b0fe886f5                                                                                                                                                                                                                                                   
     3:     0x555555e52a45 - std::panicking::default_hook::{{closure}}::h64163197f96398c1                                                                     
     4:     0x555555e526a2 - std::panicking::default_hook::h92625161492e6b48       

Again I am at a loss.

[typetetris]

Also please squash the review commits to follow the contributing guide.

Partially done. Kept two different commits for cargo-pgx changes and adding timescaledb_toolkit. I would have done two pull requests, but the second one would bring the first tests for the first one, which seems awkward.

[SuperSandro2000]

Doing this in one PR is fine. The nixos test should also get its own commit.

I think the changes look good from someone that has never worked with postgres plugins.

[typetetris]

The nixos test should also get its own commit.

Done.

I also fixed the comment on buildPgxExtension which claimed postgresql was the only argument in addition to all the arguments for buildRustPackage.

[typetetris]

@SuperSandro2000
@marsam

Updated to 1.13.1 so cargo-pgx 0.5.4 isn't needed any more.

This is open now for quite some time.
What do I need to change, so it can be merge?
Can it be merged at all?

[marsam]

Thanks a lot for working on this and sorry for the delay.

Unfortunately, I've been awful busy with life and work. I'm hoping to review this PR the next week.

[anpin]

thank you @typetetris for your effort! I was able to wrap promscale extension with the help of your buildPgxExtension and would open PR once this one is merged

[anpin]

hi everyone, wondering if there are any blockers here preventing the merge? Thanks.

[typetetris]

Rebased on master.

[marsam]

Sorry for the delay. LGTM overall, thanks a lot for working on this!

[marsam]

Can we move the pg_ctl stop to the postCheck phase? I'm thinking of starting the server in preBuild and shutting it down in postCheck.

[typetetris]

The server is only started for the createuser command. Afterwards it is not started again by the build script. I would have to check, whether it is started by cargo pgx.

[marsam]

I think postgresql should be in buildInputs

[typetetris]

Ok.

[typetetris]

Hmm. Maybe timescaledb-toolkit isn't easily cross compiled. cargo pgx wants to call pg_config so we need the postgresql derivation having the timescaledb-toolkits build platform as host platform.

But on the other hand, it builds a library, which hopefully isn't linked against anything from that postgresql package, but just uses the header files from that package and declares some symbols to be resolved at runtime, so it should work against postgresql compiled for a different platform, if cross compiled.

So timescaledb-toolkit shouldn't use anything from that postgresql derivation at runtime.

That is just my understanding.

I will test that, if I can. (Don't you need to by a mac to do darwin builds?)

[marsam]

LGTM. Thank a lot for working on this!
If there is no other reviews, I'll merge it this weekend.