Split Prolog libs into its own crate

[aarroyoc]

One of the ideas in the split thread talked about splitting Prolog libraries into a different crate. Since it's an easy one, here it is. The benefits are that changes only in Prolog libraries do not require running the whole build again, since they only trigger rebuild in the scryer-prolog-libs crate. But the linker stage is still needed, and still a big chunk of the compilation time.

[triska]

A very unfortunate consequence though: It breaks all existing links on the web that point directly to libraries.

We already had several such changes. Is the speedup really worth breaking the links yet again? Is it possible to do this without changing the library paths?

[aarroyoc]

I think there's no problem moving back the libraries to their original place, but it will be a little bit messy, and not sure if the change is worth then. I don't mind closing the PR if you think it's not worth doing.

[triska]

Personally, I do not consider this change worth it: If only a Prolog library changes, there is no need to recompile the entire system to test changes: One can simply consult the library using the existing executable. Recent changes to library(dif) are an example of this.

In other common cases, libraries are often changed in tandem with Rust code, as in the cryptographic and HTTP libraries. In such cases, the changes should be kept together, as they are now, and then must cause a complete recompilation also as is now already the cause.

On a general note, if there are such severe performance issues with the compilation of Rust code, I would also prefer that we first file this as issues with Rust itself instead of trying to "manually" work around the source of the issue like this. Maybe there are ways to improve the compilation and linking speed of Scryer in the Rust compiler and linker, and in this way also benefit many other Rust projects?

[infogulch]

Just to comment on this

Maybe there are ways to improve the compilation and linking speed of Scryer in the Rust compiler and linker, and in this way also benefit many other Rust projects?

Compile time performance has been a focus of the rust project for basically its entire existence. It has consistently been a top item in the state of rust survey, and there has been an entire working group focused on rust compiler performance since 2018. In 2023 they've reduced compile times by 13% with many small changes, and have recently introduced a feature on nightly that adds parallelization of the last available part of the compile pipeline which promises big improvements in some cases. For scryer prolog I found it improves by 17% for a fresh compile and 8% for an incremental rebuild, which is nice for sure but only applies to the nightly compiler with an experimental feature enabled. (See below for details.)

cargo version -v

cargo 1.75.0-nightly (b4d18d4bd 2023-10-31)
release: 1.75.0-nightly
commit-hash: b4d18d4bd3db6d872892f6c87c51a02999b80802
commit-date: 2023-10-31
host: x86_64-unknown-linux-gnu
libgit2: 1.7.1 (sys:0.18.1 vendored)
libcurl: 8.4.0-DEV (sys:0.4.68+curl-8.4.0 vendored ssl:OpenSSL/1.1.1u)
ssl: OpenSSL 1.1.1u  30 May 2023
os: Ubuntu 22.04 (jammy) [64-bit]

• No parallel: /usr/bin/time -v cargo build --all-targets --release
  • Clean build: 21m 12s
    User time (seconds): 4695.99
    System time (seconds): 170.04
    Percent of CPU this job got: 382%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 21:13.52
  • Incremental build: 16m 04s
    User time (seconds): 3431.63
    System time (seconds): 34.04
    Percent of CPU this job got: 359%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 16:04.75
• Parallel: RUSTFLAGS="-Z threads=8" /usr/bin/time -v cargo build --all-targets --release
  • Clean build: 17m 34s
    User time (seconds): 4279.87
    System time (seconds): 114.80
    Percent of CPU this job got: 416%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 17:35.14 (-17%)
  • Incremental build:
    User time (seconds): 3028.67
    System time (seconds): 32.29
    Percent of CPU this job got: 345%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 14:45.89 (-8%)

My point is that it's very unlikely that anyone will just drive by and pluck some low hanging fruit that dramatically improves compile times for all rust projects.

Hence the reason why I posted #2139: if we want to improve compile times for scryer-prolog the best bet is shortening path dependencies for the rust code in this project, and the best way to do that is breaking it up into multiple crates or refactoring to have less code in general. This is a well known and usually an effective strategy for reducing compile times. If that's not possible or not worth it for the project that's fine but directing attempts to improve compile performance for scryer-prolog to rustc instead is not a viable strategy.

[triska]

Thank you a lot for these additional details! I greatly appreciate that you are looking into these issues!

directing attempts to improve compile performance for scryer-prolog to rustc instead is not a viable strategy.

To be more explicit about what I meant above: Maybe there is something that is specific to Scryer Prolog that currently causes long compilation times and can be improved in the Rust compiler, but has so for not been addressed or even analyzed because the particular case does not arise to this extent in other projects and has therefore not been brought to the attention of Rust developers.

Similar issues arose in other projects: Around 2008 or so, there was a file system issue discussed at length in the Ubuntu forums. What I remember about it is that the issue was eagerly debated in the Ubuntu forums and Ubuntu bugtracker, and many potential and fruitless remedies were discussed. The actual file system developers were informed about the issue about one year later and immediately fixed the issue once it was brought to their attention. I think it was this issue, though maybe a different one.

Raising specific attention to the issues we face with Scryer Prolog may still lead to useful improvements in the Rust compiler, even if many other projects also face performance issues, for similar or unrelated reasons.

If all Rust projects work around performance issues in the compiler, then what motivation is left to improve the compiler?

[infogulch]

I agree it's worth investigating and reporting if we find something uniquely bad about how rust compiles scryer-prolog.

Unfortunately as far as I can tell our compile times are rather typical for organically grown projects that haven't payed particular attention to how the code is organized in order to optimize for compilation performance. That's not to dismiss the possibility of finding upstream improvements, but is just realistic about the probability of us happening upon big gains in a project where performance is not only a major focus for many years, but is also a value shared by the developers and community at large.

Trust that rust compiler performance is not neglected but for the motivation this project would bring.