Tracking issue: Rust library interface overhaul

[bakaq]

I think that if we are going to do things like #2475 (we did, it's merged), that is a breaking change in the Rust library interface, then we should use this opportunity to actually improve a lot of things and make the API more friendly, consistent and resilient to future changes. I going to use this as a tracking issue for that endeavor (I will keep this updated and link relevant issues and PRs). Basically everything here will be a breaking change for the scryer_prolog Rust library (that means landing on version 0.10.0 or later). Feedback (like on if this is even a good idea) and suggestions are very welcome!

• Visibility of items. Currently almost everything is public, which is a semver nightmare for a library. The main file for the binary only really needs scryer_prolog::{atom, machine::{Machine, INTERRUPT, config::{MachineConfig, StreamConfig}}} to be public (and we could probably put all of that in a Machine::run_binary() method and have just that be public), and the library use case only really needs scryer_prolog::machine::{parsed_results::*, lib_machine::*} additionally. We probably don't need all the current public methods on Machine to be public also. Everything else should be at least pub(crate), if not stricter.
  • Be conservative with visibility #2523
• Future proof API. MachineConfig and StreamConfig (or analogous types) will probably have to be public, but to enable changing their internals in the future (which to me seems very likely) without breaking API, MachineConfig should have private fields and StreamConfig should be #[non_exhaustive]. Should Value (or analogous) also be non-exhaustive/opaque (for future Unum support for example)?
• Names and terminology. I think it would be a really good idea to bikeshed on the terminology used in the API. For example, currently Prolog terms are represented by the Value type in parsed_results.rs, but I think a better name would be Term or PrologTerm. QueryState is an iterator of (Results of) QueryResolutionLine, and I don't know exactly what a good name would be here (PrologAnswer? What is the diferent between answer, solution, variable substitution, etc? Is there even a consensus on meaning here?).
  • Answers terminology #2497
• Ecosystem traits. Implement/derive things like Debug, Eq, serde::Serialize (probably behind a feature), etc, on everything that makes sense.
• Flexible signatures. Stuff like making functions take Into<String> instead of String, or IntoIterator<T> instead of Vec<T>, when it makes sense.
• Serialization. Decide on a serialization for things like Value and QueryResolution that ideally can be extended in the future to support things like residual goals. I made an initial proposal here.
  • Value and QueryResolution serialization with serde #2493
• Documentation. There is currently no item and module documentation for the Rust API. After reducing the number of public items, I think we should make decent documentation and even #![deny(missing_docs)].

[triska]

Thank you a lot for working on this! I can contribute a few points regarding terminology:

An answer is something we get in response to a query. In Scryer Prolog, an answer always has the form of a goal that is declaratively equivalent to the query.

The ISO standard defines a goal as:

3.81 goal: A predication which is to be executed (see body, query, and 7.7.3).

And also:

5.3 Prolog goal

A conforming Prolog goal is one whose execution is
defined by the constructs specified in this part of ISO/IEC
13211, and the implementation defined and implementation
specific features supported by the processor.

A strictly conforming Prolog goal is one whose execution
is defined by the constructs specified in this part of
ISO/IEC 13211, and the implementation defined features
supported by the processor.

This is very broad. For example, (f ; g ; h) is a goal. At the same time, each of f, g and h is also a goal!

Therefore, when we see for example:

?- length(Ls, L).
   Ls = [], L = 0
;  Ls = [_A], L = 1
;  Ls = [_A,_B], L = 2
;  Ls = [_A,_B,_C], L = 3
;  ... .

then, in addition to the entire answer, each of the disjuncts is also called an answer! We currently lack the terminology to properly distinguish these cases!

A solution is an answer that grounds all variables that appear in the query. In the above example, Ls = [], L = 0 is a solution, whereas Ls = [_A], L = 1 is an answer but not a solution. A single answer may describe infinitely many concrete solutions.

In the above example, all answers are answer substitutions, i.e., answers of the form Var1 = T1, Var2 = T2, .... A single conjunct of an answer substitution is also called a binding, such as L = 3.

An answer can also be an exception, reported (as all cases above) as a goal; a goal that yields that exception.

An answer may also include residual goals. Since all answers (including answer substitutions) are goals, one could all classify them as "residual goals", but we normally use "residual goals" only to mean pending constraints that are not trivial in the sense that: they are not answer substitutions. An example of a residual goal is dif(X, f(b)). Residual goals may occur by themselves, or together with answer substitutions. An answer that contains residual goals is also called a conditional solution, because it only implies solutions if the residual constraints can be satisfied.

The answers true and false are not called "residual goals" even though they are also goals.

[bakaq]

Answered in #2497 because I don't want to flood this issue too much.

[jjtolton]

Great discussion. I am sensing a bit of deadlock here, as we have several interlocking issues with unclear dependencies with rapidly changing scope/requirements.

An incomplete list of PRs concerning this:

• add deserialize impls for #2493 #2505
• Value and QueryResolution serialization with serde #2493
• ISSUE-2464: exposing scryer prolog functionality in libscryer_prolog.so for client library consumption #2465

Upstream dependencies, an incomplete list:

• Relax version constraint on reqwest dependency  #2491
• WIP: Add jit_compile/1, very basic JIT2 #2424 (maybe?)

Related issues and discussions:

• Tracking issue: Rust library interface overhaul #2490
• Efficient alternate representations for lists of characters (strings). #24 (maybe related to Value and QueryResolution serialization with serde #2493 ?)
• Partial string leak #1714 (maybe related to Value and QueryResolution serialization with serde #2493 ?)
• Splitting library, shared_library and wasm library to different crates. #2512

Within each of these, there are decisions an interests representing various user groups, an incomplete list:

• the Scryer core development team
• Scryer Prolog library developers
• Scryer Prolog application developers
• Scryer rust shared library (crate?) users
• Scryer server shared library (.so, .dll, .dylib) client library authors
• Scryer server shared library (.so, .dll, .dylib) application developers
• Scryer WASM shared libray (.wasm, .wat) client library authors
• Scryer WASM shared library (.wasm, .wat) application developers

I make it sound likes it's hundreds of people but right now it's like 5-7 people 😆 That being said, there are a lot of different points of view to consider, and a lot of moving parts, and right now it seems that the only tools we have for this are a good working memory and consistent communication.

We also have a number of outstanding conceptual decisions to make:

• The Great Renaming™ 😆
• JSON API (set of functions)
• JSON spec (internal tagging, adjacent tagging, primitive reification, some combination thereof... )
• JSON parser implementation
• C-Api (ABI?)
• C-Values
• C shared library implementation
• code organization (files, modules, libraries, crates...)
• How to document all of this in a sane way for users

I believe we are getting to the point where at least #2493 and #2465 are requiring significant rework and facing inefficiencies as a result of changing requirements. Which is ok, because a lot of this was a discovery process.

However, I'm thinking it might be worth asking @mthom if we can open a project board for this or I can volunteer to setup an Asana tracker (or other tool) to help track this stuff.

I would volunteer to write a path search / scheduling algorithm to find the quickest way to do this, but I'm still figuring it out!

Edit: posted this in the wrong place, intially

[jjtolton]

Alright so I know I'm feeling somewhat deadlocked on the shared library work and I am having trouble keeping track of the moving parts... is anyone waiting on me to do something?

I feel like I am waiting on some decisions regarding structure,
terminology, APi, etc.

[bakaq]

is anyone waiting on me to do something?

No, in fact the shared library would be the end consumer of all this (together with the Wasm interface). It may be better to wait for the Rust side to become a bit more stable before going forward with those, because if not then we will have a lot of conflicts. I will try to have a draft PR for the visibility issues up later today to get this going again (I feel like visibility is basically a blocker for everything else here).