Document modeling and retrieval strategies

README.md

@@ -45,6 +45,41 @@ All of this complexity is dealt with under the hood, but it is important to be a
 
 The way that models are stored in documents can be configured with relations, and there are some methods to get document information. `getDocumentUrl()` returns the document url inferred from the resource id, whilst `getSourceDocumentUrl()` returns the document url where the resource is actually stored. Most of the time, both should be the same document, but there is nothing in Solid that prevents doing otherwise.
 
+### Data modeling and retrieval strategies
+
+Given the mental model we just introduced, the easiest way to work with Soukai is with a single document per model within a container. But there are some other patterns you may encounter working with Solid data in the wild. We'll discuss some.
+
+Let's say we have a collection of movies. Ideally, each movie will stored within a single document, and so the collection of documents within a container will correspond to the collection of movies. This is the typical way to retrieve a list of `Movie` models from a container following this pattern:
+
+```js
+const movies = await Movie.from('https://example.org/movies/').all();
+```
+
+This approach is Soukai's bread and butter, but it has some limitations.
+
+For example, if you have a collection of movies in your POD, you may want to share movie lists publicly. But the way Solid works at the moment, it's not possible to selectively change the visibility of documents listed in a container. You either publish the entire container, or you don't. One solution to this problem would be to simply use different containers for each list. For example classified by genre: `/movies/comedy/`, `/movies/horror/`, etc. But that approach also has some drawbacks. It would complicate the retrieval of models using the `all()` method, you'd need to call it for each container. It also requires changing the url of a model if it has moved between lists. And finally, it doesn't support having the same movie in more than one list (unless you duplicate it).
+
+One solution to this problem is to create another model that points to a list of movies. For example, using the [schema:ItemList](https://schema.org/ItemList) class you can hold a list of items using the `schema:itemListElement` property that each points to a movie with `schema:item`. Using [relations](#relations), you could model these relationships and retrieve the movies as such:
+
+```js
+const horrorMovies = await MoviesList.find('https://example.org/movies/lists/horror#it');
+const movies = horrorMovies.items.map(async item => {
+    item.loadRelationIfUnloaded('movie');
+
+    return item.movie;
+})
+```
+
+These two techniques should be enough to model most common use-cases in your apps. However, you won't always read data that has been created by your app (or Soukai, for that matter). Because of that, there is a final tool you can use to read models from a single document rather than from containers, and that is using the `$in` filter:
+
+```js
+const movies = await Movie.all({
+    $in: ['https://example.org/movies/horror'],
+});
+```
+
+This is a special behaviour in `SolidEngine`. Using [the `$in` filter](https://soukai.js.org/guide/using-models.html#using-filters) with any other engine will still retrieve one single model per id, given that all the models will always be stored in the same collection. But this was implemented in order to handle the fact that models can be spread throughout containers and documents in Solid.
+
 ## Defining Solid Models
 
 All standard [model definition](https://soukai.js.org/guide/defining-models.html) rules apply, with some extra things to keep in mind.

src/tests/fixtures/solid-crud/movies-document.ttl

@@ -0,0 +1,27 @@
+@prefix : <#>.
+@prefix schema: <https://schema.org/>.
+@prefix terms: <http://purl.org/dc/terms/>.
+@prefix XML: <http://www.w3.org/2001/XMLSchema#>.
+
+:spirited-away
+    a schema:Movie;
+    terms:created "2021-01-30T11:47:00Z"^^XML:dateTime;
+    terms:modified "2021-01-30T11:47:00Z"^^XML:dateTime;
+    schema:datePublished "2001-07-20T00:00:00Z"^^XML:dateTime;
+    schema:name "Spirited Away";
+    schema:contentRating "PG".
+
+:spirited-away-watched
+    a schema:WatchAction;
+    terms:created "2020-12-10T19:20:57Z"^^XML:dateTime;
+    schema:endTime "2020-12-10T19:20:57Z"^^XML:dateTime;
+    schema:object :spirited-away;
+    schema:startTime "2020-12-10T19:20:57Z"^^XML:dateTime.
+
+:the-lord-of-the-rings
+    a schema:Movie;
+    terms:created "2021-01-30T11:47:24Z"^^XML:dateTime;
+    terms:modified "2021-01-30T11:47:24Z"^^XML:dateTime;
+    schema:datePublished "2001-12-18T00:00:00Z"^^XML:dateTime;
+    schema:name "The Lord of the Rings: The Fellowship of the Ring";
+    schema:contentRating "PG-13".

src/tests/solid-crud.test.ts

@@ -162,7 +162,7 @@ describe('Solid CRUD', () => {
         `);
     });
 
-    it('Reads models', async () => {
+    it('Reads single models', async () => {
         // Arrange
         StubFetcher.addFetchResponse(fixture('spirited-away.ttl'), {
             'WAC-Allow': 'public="read"',
@@ -205,9 +205,9 @@ describe('Solid CRUD', () => {
         expect(alice?.name).toEqual('Alice');
     });
 
-    it('Reads many models', async () => {
+    it('Reads many models from containers', async () => {
         // Arrange
-        StubFetcher.addFetchResponse(fixture('movies.ttl'));
+        StubFetcher.addFetchResponse(fixture('movies-container.ttl'));
         StubFetcher.addFetchResponse(fixture('the-lord-of-the-rings.ttl'));
         StubFetcher.addFetchResponse(fixture('spirited-away.ttl'));
         StubFetcher.addFetchResponse(fixture('the-tale-of-princess-kaguya.ttl'));
@@ -231,6 +231,32 @@ describe('Solid CRUD', () => {
         expect(spiritedAway.actions).toHaveLength(1);
     });
 
+    it('Reads many models from documents', async () => {
+        // Arrange
+        const documentUrl = fakeDocumentUrl();
+
+        StubFetcher.addFetchResponse(fixture('movies-document.ttl'));
+
+        // Act
+        const movies = await Movie.all({ $in: [documentUrl] });
+
+        // Assert
+        expect(movies).toHaveLength(2);
+        const spiritedAway = movies.find(movie => movie.url.endsWith('#spirited-away')) as Movie;
+        const theLordOfTheRings = movies.find(movie => movie.url.endsWith('#the-lord-of-the-rings')) as Movie;
+
+        expect(spiritedAway).not.toBeUndefined();
+        expect(spiritedAway.title).toEqual('Spirited Away');
+        expect(spiritedAway.actions).toHaveLength(1);
+
+        expect(theLordOfTheRings).not.toBeUndefined();
+        expect(theLordOfTheRings.title).toEqual('The Lord of the Rings: The Fellowship of the Ring');
+        expect(theLordOfTheRings.actions).toHaveLength(0);
+
+        expect(fetch).toHaveBeenCalledTimes(1);
+        expect(fetch.mock.calls[1]?.[0]).toEqual(documentUrl);
+    });
+
     it('Deletes models', async () => {
         // Arrange
         const containerUrl = urlResolveDirectory(faker.internet.url());

src/tests/solid-interop.test.ts

@@ -1,17 +1,24 @@
 import { bootModels, setEngine } from 'soukai';
+import { faker } from '@noeldemartin/faker';
 
 import Movie from '@/testing/lib/stubs/Movie';
 import StubFetcher from '@/testing/lib/stubs/StubFetcher';
 import WatchAction from '@/testing/lib/stubs/WatchAction';
+import { loadFixture } from '@/testing/utils';
 
-import { SolidEngine } from '@/engines';
+import SolidContainer from '@/models/SolidContainer';
+import SolidDocument from '@/models/SolidDocument';
+import { SolidEngine } from '@/engines/SolidEngine';
+import { fakeDocumentUrl } from '@noeldemartin/solid-utils';
 
 class MovieWithTimestamps extends Movie {
 
     public static timestamps = true;
 
 }
 
+const fixture = (name: string) => loadFixture(`solid-interop/${name}`);
+
 describe('Solid Interoperability', () => {
 
     let fetch: jest.Mock<Promise<Response>, [RequestInfo, RequestInit?]>;
@@ -101,4 +108,95 @@ describe('Solid Interoperability', () => {
         expect(movies[0]?.usesRdfAliases()).toBe(true);
     });
 
+    it('Reads instances from the type index', async () => {
+        // Arrange
+        const podUrl = faker.internet.url();
+        const typeIndexUrl = fakeDocumentUrl({ baseUrl: podUrl });
+        const movieUrl = `${podUrl}/movies/spirited-away`;
+
+        StubFetcher.addFetchResponse(fixture('type-index.ttl'));
+
+        // Act
+        const movieDocuments = await SolidDocument.fromTypeIndex(typeIndexUrl, Movie);
+
+        // Assert
+        expect(movieDocuments).toHaveLength(1);
+        expect(movieDocuments[0]?.url).toEqual(movieUrl);
+    });
+
+    it('Reads containers from the type index', async () => {
+        // Arrange
+        const podUrl = faker.internet.url();
+        const typeIndexUrl = fakeDocumentUrl({ baseUrl: podUrl });
+        const moviesContainerUrl = `${podUrl}/movies`;
+
+        StubFetcher.addFetchResponse(fixture('type-index.ttl'));
+
+        // Act
+        const movieContainers = await SolidContainer.fromTypeIndex(typeIndexUrl, Movie);
+
+        // Assert
+        expect(movieContainers).toHaveLength(1);
+        expect(movieContainers[0]?.url).toEqual(moviesContainerUrl);
+    });
+
+    it('Registers instances in the type index', async () => {
+        // Arrange
+        const podUrl = faker.internet.url();
+        const typeIndexUrl = fakeDocumentUrl({ baseUrl: podUrl });
+        const movieDocumentUrl = `${podUrl}/movies/midsommar`;
+        const movieUrl = `${movieDocumentUrl}#it`;
+
+        StubFetcher.addFetchResponse(fixture('type-index.ttl'));
+        StubFetcher.addFetchResponse();
+
+        // Act
+        const movie = new Movie({ url: movieUrl });
+
+        await movie.registerInTypeIndex(typeIndexUrl);
+
+        // Assert
+        expect(fetch).toHaveBeenCalledTimes(2);
+
+        expect(fetch.mock.calls[1]?.[1]?.body).toEqualSparql(`
+            INSERT DATA {
+                @prefix solid: <http://www.w3.org/ns/solid/terms#> .
+                @prefix schema: <https://schema.org/>.
+
+                <#[[.*]]> a solid:TypeRegistration;
+                    solid:forClass schema:Movie;
+                    solid:instance <${movieDocumentUrl}>.
+            }
+        `);
+    });
+
+    it('Registers containers in the type index', async () => {
+        // Arrange
+        const podUrl = faker.internet.url();
+        const typeIndexUrl = fakeDocumentUrl({ baseUrl: podUrl });
+        const moviesContainerUrl = `${podUrl}/great-movies`;
+
+        StubFetcher.addFetchResponse(fixture('type-index.ttl'));
+        StubFetcher.addFetchResponse();
+
+        // Act
+        const movies = new SolidContainer({ url: moviesContainerUrl });
+
+        await movies.register(typeIndexUrl, Movie);
+
+        // Assert
+        expect(fetch).toHaveBeenCalledTimes(2);
+
+        expect(fetch.mock.calls[1]?.[1]?.body).toEqualSparql(`
+            INSERT DATA {
+                @prefix solid: <http://www.w3.org/ns/solid/terms#> .
+                @prefix schema: <https://schema.org/>.
+
+                <#[[.*]]> a solid:TypeRegistration;
+                    solid:forClass schema:Movie;
+                    solid:instanceContainer <${moviesContainerUrl}>.
+            }
+        `);
+    });
+
 });