From a56ba8a15ee27124936e6a7fb4b86cc01afb5998 Mon Sep 17 00:00:00 2001 From: Laurent Le Meur Date: Mon, 31 May 2021 16:02:04 +0200 Subject: [PATCH 1/8] Addition of the Synchronized Narration Module --- modules/sync-narration.md | 155 ++++++++++++++++++++++++++++++++++++++ profiles/epub.md | 1 + 2 files changed, 156 insertions(+) create mode 100644 modules/sync-narration.md diff --git a/modules/sync-narration.md b/modules/sync-narration.md new file mode 100644 index 0000000..f236ca3 --- /dev/null +++ b/modules/sync-narration.md @@ -0,0 +1,155 @@ +# Synchronized Narration Module + +**Editors:** + +* Laurent Le Meur ([EDRLab](https://www.edrlab.org) + +## Introduction + +The Synchronized Narration Module defines how the content of a given textual resource can be synchronized with the content of an alternative audio resource. + +While the main use of this Module is an audio narration synchonized with text highlights in a textual publication, it may also be used "in reverse mode" for enhancing an audiobook experience with textual subtitles. + +## Status of this document + +This is a draft document from the Readium community. It is currently implemented in Readium Desktop. + +An useful evolution would be to extend this structure with a proper support for still and moving images. A textual publication could then be enhanced with the apparition of small illustrations or sign-language videos explaining words or utterances, upon manual selection of those textual segments. This feature would be very useful for cognitive disabled people, but also for children or people learning a new language. Such feature also has the advantage of being 100% descriptive (no javascript involved in the publication). + + +## Structure of a Synchronized Narration document + +The Synchronized Narration object is defined as: + +| Key | Definition | Format | Required? | +| ---- | -----------| -------| ----------| +| `textRef` | The absolute of relative URL of the textual resource which participates to the synchronized narration | URL | At the top level only | +| `audioRef` | The absolute of relative URL of the audio resource which participates to the synchronized narration | URL | At the top level only | +| `role` | Semantic information relative to the role of the object in the publication | "aside", "footnote" | No | +| `narration` | A recursive array of synchronization objects | Array of Synchronization Item or Synchronized Narration | Yes | + +The Synchronization Item object is defined as: + +| Key | Definition | Format | Required? | +| ---- | -----------| -------| ----------| +| `text` | The fragment identifier of the textual fragment which participates to the synchronized narration | URI Fragment Identifier | Yes | +| `audio` | The fragment identifier of the audio fragment which participates to the synchronized narration | Media Fragment URI | Yes | + +**Notes:** + +- The recursive structure offers a way to selectively filter out content based on semantics ("skippability" in the DAISY world) and to jump out of complex structures and back into the reading flow ("escapability" in the DAISY world). +- While the structure allows `textRef`and `audioRef` to be set at every level of a recursive tree of Synchronized Narration objects, these URL are usually defined only at the top level of the document, where they are mandatory. Nevertheless, this structure also allows the creation of a unique Synchronized Narration document structured as an array of Synchronized Narration objects (one per publication resource). + +## Example + +```json +{ + "textRef": "/text/chapter1.html", + "audioRef": "/audio/chapter1.mp3", + "narration": [ + { + "text": "#id1", + "audio": "#t=0.0,1.2" + }, + { + "text": "#id2", + "audio": "#t=1.2,3.4" + }, + { + "role": "footnote-ref", + "text": "#id3", + "audio": "#t=3.4,5.6" + }, + { + "role": "aside", + "narration": [ + { + "text": "#id4", + "audio": "#t=5.6,7.8" + }, + { + "text": "#id5", + "audio": "#t=7.8,9.1" + } + ] + }, + { + "text": "#id6", + "audio": "#t=9.1,10.2" + } + ] +} +``` + +## Mime Type and file extension + +This specification introduces a dedicated media type value to identify a Synchronized Narration document: `application/vnd.syncnarr+json`. + +When serialized as a file, Synchronized Narration document must have the following extension: `.sync`. + +## Declaring a Synchronized Narration document in a Manifest + +There is usually one Synchronized Narration document per resource in the reading order. + +Each Synchronized Narration document used in a publication must be declared in the Readium Webpub Manifest as an `alternate` resource with the proper media type. + +**Example** + +```json +{ + "...": "...", + "readingOrder": [{ + "href": "/text/c001.html", + "type": "text/html", + "title": "Chapter 1", + "alternate": { + "href": "/sync/c001.sync", + "type": "application/vnd.syncnarr+json" + } + }, + { + "href": "/text/c002.html", + "type": "text/html", + "title": "Chapter 2", + "alternate": { + "href": "/sync/c002.sync", + "type": "application/vnd.syncnarr+json" + } + } + ], + "resources": [ + { + "href": "/audio/c001.mp3", + "type": "audio/mp3" + }, + { + "href": "/audio/c002.mp3", + "type": "audio/mp3" + } + ] +} +``` + +## Processing a Synchronized Narration object + +** non-normative ** + +If a Synchronized Narration object is detected in a Manifest, the reading system should present to the user a way to activate / deactivate the synchronized narration. + +At the time a Readium Navigator is processing a Synchronized Narration object, a reading system should check that the `textRef` and `audioRef` properties reference existing resources in the manifest and that their media type is correct. + +In case of error, the Synchronized Narration object should be skipped. + +## References + +### Normative References + +- [URI fragment identfier](https://www.ietf.org/rfc/rfc3986.txt), IETF, 2005. +- [Media Fragments URI 1.0](https://www.w3.org/TR/media-frags/), W3C Recommendation, 2012. + +### Non normative Reference + +- [W3C Synchronized Narration](https://w3c.github.io/sync-media-pub/archived/synchronized-narration.html), W3C, archived, 2019. + +This specification was discussed in 2019 by the W3C Synchronized Media for Publications Community Group - which included EDRLab - and was used as an inspiration for our document; but it was then archived by the W3C community group and replaced by an XML based proposal which does not fit the needs of the Readium Architecture. + diff --git a/profiles/epub.md b/profiles/epub.md index 2cbf6a6..fe4946b 100644 --- a/profiles/epub.md +++ b/profiles/epub.md @@ -22,6 +22,7 @@ This profile relies on: * the definition of additional [collection roles](#3-collection-roles), * the definition of additional [Link properties](#4-link-properties), * the use of the [encryption module](../modules/encryption.md). +* the use of the [Synchronized Narration module](../modules/sync-narration.md). ## 1. Declaring conformance with the EPUB Profile From 2a1734e0c21fd65049b01df0bfd0c5cd6e056638 Mon Sep 17 00:00:00 2001 From: Laurent Le Meur Date: Mon, 31 May 2021 16:29:13 +0200 Subject: [PATCH 2/8] Simplification of the structure. --- modules/sync-narration.md | 40 ++++++++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 15 deletions(-) diff --git a/modules/sync-narration.md b/modules/sync-narration.md index f236ca3..db9844c 100644 --- a/modules/sync-narration.md +++ b/modules/sync-narration.md @@ -25,8 +25,16 @@ The Synchronized Narration object is defined as: | ---- | -----------| -------| ----------| | `textRef` | The absolute of relative URL of the textual resource which participates to the synchronized narration | URL | At the top level only | | `audioRef` | The absolute of relative URL of the audio resource which participates to the synchronized narration | URL | At the top level only | -| `role` | Semantic information relative to the role of the object in the publication | "aside", "footnote" | No | -| `narration` | A recursive array of synchronization objects | Array of Synchronization Item or Synchronized Narration | Yes | +| `narration` | A recursive array of synchronization objects | Array of Synchronization Item or Sub Narration | Yes | + + +The Sub Narration object is defined as: + +| Key | Definition | Format | Required? | +| ---- | -----------| -------| ----------| +| `role` | Semantic information relative to the role of the object in the publication | [EPUB Structural Semantics](#references) | No | +| `narration` | A recursive array of synchronization objects | Array of Synchronization Item or Sub Narration | Yes | + The Synchronization Item object is defined as: @@ -35,10 +43,9 @@ The Synchronization Item object is defined as: | `text` | The fragment identifier of the textual fragment which participates to the synchronized narration | URI Fragment Identifier | Yes | | `audio` | The fragment identifier of the audio fragment which participates to the synchronized narration | Media Fragment URI | Yes | -**Notes:** +**Note:** -- The recursive structure offers a way to selectively filter out content based on semantics ("skippability" in the DAISY world) and to jump out of complex structures and back into the reading flow ("escapability" in the DAISY world). -- While the structure allows `textRef`and `audioRef` to be set at every level of a recursive tree of Synchronized Narration objects, these URL are usually defined only at the top level of the document, where they are mandatory. Nevertheless, this structure also allows the creation of a unique Synchronized Narration document structured as an array of Synchronized Narration objects (one per publication resource). +The recursive structure offers a way to selectively filter out content based on semantics ("skippability" in the DAISY world) and to jump out of complex structures and back into the reading flow ("escapability" in the DAISY world). ## Example @@ -56,7 +63,7 @@ The Synchronization Item object is defined as: "audio": "#t=1.2,3.4" }, { - "role": "footnote-ref", + "role": "footnote", "text": "#id3", "audio": "#t=3.4,5.6" }, @@ -85,7 +92,7 @@ The Synchronization Item object is defined as: This specification introduces a dedicated media type value to identify a Synchronized Narration document: `application/vnd.syncnarr+json`. -When serialized as a file, Synchronized Narration document must have the following extension: `.sync`. +When saved as a file, a Synchronized Narration document must have the following extension: `.sync`. ## Declaring a Synchronized Narration document in a Manifest @@ -102,19 +109,21 @@ Each Synchronized Narration document used in a publication m "href": "/text/c001.html", "type": "text/html", "title": "Chapter 1", - "alternate": { - "href": "/sync/c001.sync", - "type": "application/vnd.syncnarr+json" - } + "alternate": [{ + "href": "/sync/c001.sync", + "type": "application/vnd.syncnarr+json" + } + ] }, { "href": "/text/c002.html", "type": "text/html", "title": "Chapter 2", - "alternate": { - "href": "/sync/c002.sync", - "type": "application/vnd.syncnarr+json" - } + "alternate": [{ + "href": "/sync/c002.sync", + "type": "application/vnd.syncnarr+json" + } + ] } ], "resources": [ @@ -146,6 +155,7 @@ In case of error, the Synchronized Narration object should be skipped. - [URI fragment identfier](https://www.ietf.org/rfc/rfc3986.txt), IETF, 2005. - [Media Fragments URI 1.0](https://www.w3.org/TR/media-frags/), W3C Recommendation, 2012. +- [EPUB 3 Structural Semantics Vocabulary](https://idpf.github.io/epub-vocabs/structure/), IDPF, 2019 ### Non normative Reference From 340b5ccde542755f6b2ee864a18e9f81ececea26 Mon Sep 17 00:00:00 2001 From: Laurent Le Meur Date: Mon, 31 May 2021 16:36:41 +0200 Subject: [PATCH 3/8] Add a note about the interest of structural semantics --- modules/sync-narration.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/modules/sync-narration.md b/modules/sync-narration.md index db9844c..4a358c0 100644 --- a/modules/sync-narration.md +++ b/modules/sync-narration.md @@ -43,9 +43,11 @@ The Synchronization Item object is defined as: | `text` | The fragment identifier of the textual fragment which participates to the synchronized narration | URI Fragment Identifier | Yes | | `audio` | The fragment identifier of the audio fragment which participates to the synchronized narration | Media Fragment URI | Yes | -**Note:** +**Notes:** -The recursive structure offers a way to selectively filter out content based on semantics ("skippability" in the DAISY world) and to jump out of complex structures and back into the reading flow ("escapability" in the DAISY world). +Structural semantics offer a way to selectively filter out content based on semantics ("skippability" in the DAISY world) + +The recursive structure offers a way to jump out of complex structures and back into the reading flow ("escapability" in the DAISY world). ## Example From 0c304f24ff8eb33948448855d65ccd0794c82c30 Mon Sep 17 00:00:00 2001 From: Laurent Le Meur Date: Mon, 31 May 2021 18:52:07 +0200 Subject: [PATCH 4/8] Add a note aoubt extension to image/video --- modules/sync-narration.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/modules/sync-narration.md b/modules/sync-narration.md index 4a358c0..900fcd7 100644 --- a/modules/sync-narration.md +++ b/modules/sync-narration.md @@ -49,6 +49,8 @@ Structural semantics offer a way to selectively filter out content based on sema The recursive structure offers a way to jump out of complex structures and back into the reading flow ("escapability" in the DAISY world). +In case we decide to extend the structure to image and video, using `image` and `video` would be consistent with the [latest work of the W3C CG](https://w3c.github.io/sync-media-pub/sync-media.html). + ## Example ```json From 0d3bb129884dce8acecbc91e5d6c15a0841de0e5 Mon Sep 17 00:00:00 2001 From: Laurent Le Meur Date: Tue, 1 Jun 2021 16:44:55 +0200 Subject: [PATCH 5/8] update after study of Thorium implem. The use of textRef and audioRef is totally open to discussion. This is not how Thorium implements it. --- modules/sync-narration.md | 61 +++++++++++++++++++++++++++++---------- 1 file changed, 46 insertions(+), 15 deletions(-) diff --git a/modules/sync-narration.md b/modules/sync-narration.md index 900fcd7..93da6dd 100644 --- a/modules/sync-narration.md +++ b/modules/sync-narration.md @@ -96,36 +96,40 @@ In case we decide to extend the structure to image and video, using `image` and This specification introduces a dedicated media type value to identify a Synchronized Narration document: `application/vnd.syncnarr+json`. -When saved as a file, a Synchronized Narration document must have the following extension: `.sync`. +When saved as a file, a Synchronized Narration document should have the following extension: `.json`. -## Declaring a Synchronized Narration document in a Manifest - -There is usually one Synchronized Narration document per resource in the reading order. +## Declaring Synchronized Narration documents in a Manifest Each Synchronized Narration document used in a publication must be declared in the Readium Webpub Manifest as an `alternate` resource with the proper media type. +The duration of the entire audio narration may also be declared in the alternate item. + +It is not an error if only some items in the reading order have a Synchronized Narration document attached. + **Example** ```json { "...": "...", "readingOrder": [{ - "href": "/text/c001.html", - "type": "text/html", + "href": "OPS/c001.xhtml", + "type": "application/xhtml+xml", "title": "Chapter 1", "alternate": [{ - "href": "/sync/c001.sync", - "type": "application/vnd.syncnarr+json" + "href": "sync/c001.json", + "type": "application/vnd.syncnarr+json", + "duration": 850.5 } ] }, { - "href": "/text/c002.html", - "type": "text/html", + "href": "OPS/c002.xhtml", + "type": "application/xhtml+xml", "title": "Chapter 2", "alternate": [{ - "href": "/sync/c002.sync", - "type": "application/vnd.syncnarr+json" + "href": "sync/c002.json", + "type": "application/vnd.syncnarr+json", + "duration": 1001 } ] } @@ -143,15 +147,42 @@ Each Synchronized Narration document used in a publication m } ``` +## Declaring the CSS associated with media overlays + +The manifest may contain metadata indicating how textual content should be highlighted. + +A `media-overlays` property may be added to the set of metadata included in the manifest. This object is defined as: + +| Key | Definition | Format | Required? | +| ---- | -----------| -------| ----------| +| `active-class` | The author-defined CSS class name to apply to the selected textual segment. | string | No | +| `playback-active-class` | The author-defined CSS class name to apply to the selected textual segment when playback is active. | string | No | + +**Example** + +```json +{ + "@context": "https://readium.org/webpub-manifest/context.jsonld", + "metadata": { + "@type": "http://schema.org/Book", + "duration": 1403.5, + "media-overlay": { + "active-class": "-epub-media-overlay-active" + }, + ... + } +} +``` + ## Processing a Synchronized Narration object ** non-normative ** -If a Synchronized Narration object is detected in a Manifest, the reading system should present to the user a way to activate / deactivate the synchronized narration. +If one or more Synchronized Narration objects are detected in a Manifest, the reading system should present to the user a way to activate / deactivate the synchronized narration. -At the time a Readium Navigator is processing a Synchronized Narration object, a reading system should check that the `textRef` and `audioRef` properties reference existing resources in the manifest and that their media type is correct. +At the time it is processing a Synchronized Narration object, the reading system should check that the `textRef` and `audioRef` properties reference existing resources in the manifest and that their media type is correct. -In case of error, the Synchronized Narration object should be skipped. +In case of error in the processing of a Synchronized Narration object, the object should be skipped. ## References From 7e3f4f280568f56793e8e6697ea9a123ef59c7a5 Mon Sep 17 00:00:00 2001 From: Laurent Le Meur Date: Tue, 1 Jun 2021 21:44:13 +0200 Subject: [PATCH 6/8] Typo corrections + new subtitle for style --- modules/sync-narration.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/sync-narration.md b/modules/sync-narration.md index 93da6dd..19e43c6 100644 --- a/modules/sync-narration.md +++ b/modules/sync-narration.md @@ -2,7 +2,7 @@ **Editors:** -* Laurent Le Meur ([EDRLab](https://www.edrlab.org) +* Laurent Le Meur ([EDRLab](https://www.edrlab.org)) ## Introduction @@ -147,7 +147,7 @@ It is not an error if only some items in the reading order have a Synchronized N } ``` -## Declaring the CSS associated with media overlays +## Associating Style Information The manifest may contain metadata indicating how textual content should be highlighted. @@ -169,14 +169,14 @@ A `media-overlays` property may be added to the set "media-overlay": { "active-class": "-epub-media-overlay-active" }, - ... + "...":"..." } } ``` ## Processing a Synchronized Narration object -** non-normative ** +**Non-normative** If one or more Synchronized Narration objects are detected in a Manifest, the reading system should present to the user a way to activate / deactivate the synchronized narration. From 56022db578e947d4a1fcd92c82928cddae8887df Mon Sep 17 00:00:00 2001 From: "L. Le Meur" Date: Fri, 4 Jun 2021 15:56:40 +0200 Subject: [PATCH 7/8] Update modules/sync-narration.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit typo correction Co-authored-by: Mickaël Menu --- modules/sync-narration.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/modules/sync-narration.md b/modules/sync-narration.md index 19e43c6..9e7b2c3 100644 --- a/modules/sync-narration.md +++ b/modules/sync-narration.md @@ -14,7 +14,7 @@ While the main use of this Module is an audio narration synchonized with text hi This is a draft document from the Readium community. It is currently implemented in Readium Desktop. -An useful evolution would be to extend this structure with a proper support for still and moving images. A textual publication could then be enhanced with the apparition of small illustrations or sign-language videos explaining words or utterances, upon manual selection of those textual segments. This feature would be very useful for cognitive disabled people, but also for children or people learning a new language. Such feature also has the advantage of being 100% descriptive (no javascript involved in the publication). +A useful evolution would be to extend this structure with a proper support for still and moving images. A textual publication could then be enhanced with the apparition of small illustrations or sign-language videos explaining words or utterances, upon manual selection of those textual segments. This feature would be very useful for cognitive disabled people, but also for children or people learning a new language. Such feature also has the advantage of being 100% descriptive (no javascript involved in the publication). ## Structure of a Synchronized Narration document @@ -197,4 +197,3 @@ In case of error in the processing of a Synchronized Narration object, the objec - [W3C Synchronized Narration](https://w3c.github.io/sync-media-pub/archived/synchronized-narration.html), W3C, archived, 2019. This specification was discussed in 2019 by the W3C Synchronized Media for Publications Community Group - which included EDRLab - and was used as an inspiration for our document; but it was then archived by the W3C community group and replaced by an XML based proposal which does not fit the needs of the Readium Architecture. - From d988f9cffd070397865c4370107b47c73f10c801 Mon Sep 17 00:00:00 2001 From: "L. Le Meur" Date: Fri, 4 Jun 2021 15:57:01 +0200 Subject: [PATCH 8/8] Update modules/sync-narration.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit typo correction Co-authored-by: Mickaël Menu --- modules/sync-narration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/sync-narration.md b/modules/sync-narration.md index 9e7b2c3..5ad88ba 100644 --- a/modules/sync-narration.md +++ b/modules/sync-narration.md @@ -23,8 +23,8 @@ The Synchronized Narration object is defined as: | Key | Definition | Format | Required? | | ---- | -----------| -------| ----------| -| `textRef` | The absolute of relative URL of the textual resource which participates to the synchronized narration | URL | At the top level only | -| `audioRef` | The absolute of relative URL of the audio resource which participates to the synchronized narration | URL | At the top level only | +| `textRef` | The absolute or relative URL of the textual resource which participates to the synchronized narration | URL | At the top level only | +| `audioRef` | The absolute or relative URL of the audio resource which participates to the synchronized narration | URL | At the top level only | | `narration` | A recursive array of synchronization objects | Array of Synchronization Item or Sub Narration | Yes |