[HIGH] New UIs for better bot response #4846

compulim · 2023-09-02T04:30:00Z

Resolves #4840. Resolves #4841. Resolves #4842.

Changelog Entry

Added

Resolves #4840. Added feedback buttons in activity status, by @compulim, in PR #4846
Resolves #4841. Added link definitions UI in Markdown, by @compulim, in PR #4846
Resolves #4842. Added provenance in activity status, by @compulim, in PR #4846

Description

Adding 3 new UIs:

Link definitions: Markdown supports link references in footnote. This UI will render these references similar to how they appear in plain text
- We also support citation. Citation should have URL starts with cite:. The citation text should be stored in activity.entities as Claim thing
Feedback buttons in activity status
- VoteAction in activity.entities with actionOption of "upvote" and "downvote" will appear as thumbs up/down button in activity status
- When clicked, they will send event activity with entities containing the chose VoteAction
Provenance in activity status
- ReplyAction in activity.entities with provider of Project will appear in the activity status

Design

Based on Schema.org

To encourage integration across multiple chat services, this work is heavily based on Schema.org.

CSS variables

We are slowly copying some styleOptions into CSS variables. This will help when we implement dark/light theme as CSS variables can be on top of media query selector.

In <Component.Composer>, we are inserting a top-level <div style="display: contents;"> which will include all CSS variables to use under this layer. We are using display: contents to minimize impact on layout.

Markdown-it "betterLink" plug-in

We converged a lot of our custom plug-in into a betterLink plug-in. The design is not fully flush. Thus, this component remains internal for now.

When we work on customization story of Markdown, we may open this plug-in, or provide it in another way. So our web devs can enjoy customization without worrying breaking accessibility and citation dialog.

Link definitions (a.k.a. reference-style links)

Details of Markdown link definitions (a.k.a. reference-style links) can be found in this article.

Markdown:

Sure, you should override the default proxy settings[1][2], when your proxy server requires authentication[3].

[1]: https://support.microsoft.com/en-us/windows/use-a-proxy-server-in-windows-03096c53-0554-4ffe-b6ab-8b1deee8dae1
[2]: https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/configure-proxy-server-settings "Configure proxy server settings - Windows Server"
[3]: cite:1 "Introduction Configuring proxy settings is a fundamental aspect..."

Before:

After:

For readability, links without a name (This is a link[1].) or links with same name and ID (This is a link[1][1].) will be rendered as This is a link[1]. rather than This is a link1. This is done through CSS ::before and ::after. Thus, in the future, it will be easy for us if we decided to style it as a superscripted badge.

Citation in link definitions

When clicking on a link starts with cite:, it will show a modal dialog.

Sample activity

{
  "entities": [
    {
      "@context": "https://schema.org",
      "@type": "Claim",
      "type": "https://schema.org/Claim",

      "@id": "cite:XXX",
      "text": "Hello, World!"
    }
  ]
}

activity.entities.find(entity => entity['@id'] === 'cite:XXX' && entity.type === 'https://schema.org/Claim).text

Design

We are using HTMLDialogElement.showModal() instead of a non-native modal for these reasons:

Non-native modal may conflict with keyboard help screen
Native modal can display out of the boundary of Web Chat, providing a wider canvas for long text content
Native modal has better focus trap (able to TAB into address bar while trapped)
Native modal supports ESCAPE key and navigational keys (UP/DOWN/PAGE UP/PAGE DOWN/HOME/END)

We added a new <ModalDialogComposer> with useShowModal() and useClose() hooks. We did not fully flush our design yet. Thus, this component remains internal for now.

Feedback buttons

Feedback buttons allows end-user to send feedback to the bot and potentially improve further conversation.

Sample activity

{
  "entities": [
    {
      "@context": "https://schema.org",
      "@type": "VoteAction",
      "type": "https://schema.org/VoteAction",

      "actionOption": "upvote"
    }
  ]
}

activity.entities.find(entity => entity.type === 'https://schema.org/VoteAction')

We currently only works with actionOption of "upvote" and "downvote"

Design

When clicked, Web Chat will send an event activity, excerpt:

{
  "entities": [
    {
       "@context": "https://schema.org",
       "@type": "VoteAction",
       "type": "https://schema.org/VoteAction",

       "actionOption": "downvote"
    }
  ],
  "name": "webchat:activity-status/feedback",
  "type": "event"
}

Provenance

The provenance helps the bot to identify where the origin of specific response come from.

Sample activity

{
  "entities": [
    {
      "@context": "https://schema.org",
      "@type": "ReplyAction",
      "type": "https://schema.org/ReplyAction",

      "provider": {
        "@context": "https://schema.org",
        "@type": "Project",

        "name": "Surfaced by Azure OpenAI",
        "url": "https://www.microsoft.com/en-us/ai/responsible-ai"
      }
    }
  ]
}

activity.entities.find(entity => entity.type === 'https://schema.org/ReplyAction').provider

Currently, the provider thing is expected to be Project with name and optional url field.

What is missing

The customization story on Markdown and activity status is weakened after this pull request. We will visit this area again immediately after this pull request is merged.

Markdown: we introduced citation to our Markdown-It engine. If the engine is replaced (by passing a different renderMarkdown function), it will lose citation
Activity status: we introduced feedback buttons and provenance as a slotted component (delimited by pipe). However, web devs are not able to add new thing as a slot or remove/replace some slots. They will need to nest another slotted container on top

What is breaking

There is a minor cosmetic issue with activity status. Previously, we did not add margin-top (or padding-top) correctly on activity status. Thus, web devs using our useCreateActivityStatusRenderer will see padding being added correctly.

We may move to gap in short future. Thus, the padding maybe removed again.

Specific Changes

-

I have added tests and executed them locally
I have updated CHANGELOG.md
I have updated documentation

Review Checklist

This section is for contributors to review your work.

Accessibility reviewed (tab order, content readability, alt text, color contrast)
~~Browser and platform compatibilities reviewed~~
CSS styles reviewed (minimal rules, no z-index)
~~Documents reviewed (docs, samples, live demo)~~
Internationalization reviewed (strings, unit formatting)
package.json and package-lock.json reviewed
~~Security reviewed (no data URIs, check for nonce leak)~~
Tests reviewed (coverage, legitimacy)

tdurnford · 2023-09-11T14:46:19Z

packages/core/src/types/external/SchemaOrgThing.ts

+ *
+ * @see https://schema.org/Thing
+ */
+export type SchemaOrgThing<T extends string = string> = {


nit: I prefer OrgThemeSchema

tdurnford · 2023-09-11T14:51:08Z

packages/component/src/types/internal/TypeOfArray.ts

@@ -0,0 +1,7 @@
+export type TypeOfArray<T extends unknown[] | Array<unknown> | ReadonlyArray<unknown>> = T extends (infer I)[]


I don't think you need to check T extends (infer I)[] and Array<infer I>. They are the same thing

tdurnford · 2023-09-11T14:54:54Z

packages/component/src/providers/ModalDialog/private/Popover.tsx

+        aria-label={!ariaLabelledBy ? ariaLabel : undefined}
+        aria-labelledby={ariaLabelledBy}
+        className={classNames('webchat__modal-dialog', className, modalDialogStyleSet + '')}
+        onClose={handleClose}


I don' think you need to wrap onDismissRef.current in a callback.

Suggested change

onClose={handleClose}

onClose={onDismissRef.current}

tdurnford · 2023-09-11T14:56:30Z

packages/component/src/providers/ModalDialog/ModalDialogComposer.tsx

+const ModalDialogComposer = memo(({ children }: Props) => {
+  const [renderFunctionAndDialogInit, setRenderFunctionAndDialogInit] = useState<
+    RenderFunctionAndDialogInit | undefined
+  >(undefined);


Suggested change

>(undefined);

>();

tdurnford · 2023-09-11T14:57:55Z

packages/component/src/Styles/StyleSet/ThumbButton.ts

+  };
+}
+
+// .webchat__thumb-button {


What is the plan with this commented out css?

tdurnford · 2023-09-11T14:59:19Z

packages/component/src/Styles/StyleSet/ThumbButton.ts

+
+      '& .webchat__thumb-button__image': {
+        /* TODO: Remove "color" if we want a different hover color. */
+        color: 'var(--webchat__color--accent)',


nit: Having a tokens object similar to fluent would be nice.

Suggested change

color: 'var(--webchat__color--accent)',

color: tokens.colorForegroundAccent,

tdurnford · 2023-09-11T15:00:08Z

packages/component/src/Styles/StyleSet/ThumbButton.ts

+      width: 16,
+
+      '&:active': {
+        // background: 'var(--pva__palette__neutral-light)'


Why is the variable scoped to pva? Should probably remove comment.

Because I forget to remove it. 🤣

tdurnford · 2023-09-11T15:07:31Z

packages/component/src/ActivityStatus/private/Feedback/Feedback.tsx

+
+const Feedback = memo(({ voteActions }: Props) => {
+  const [{ clearTimeout, setTimeout }] = usePonyfill();
+  const [selectedVoteAction, setSelectedVoteAction] = useState<VoteAction | undefined>(undefined);


Suggested change

const [selectedVoteAction, setSelectedVoteAction] = useState<VoteAction | undefined>(undefined);

const [selectedVoteAction, setSelectedVoteAction] = useState<VoteAction | undefined>();

tdurnford · 2023-09-11T15:08:03Z

packages/component/src/ActivityStatus/private/Feedback/Feedback.tsx

+  const [selectedVoteAction, setSelectedVoteAction] = useState<VoteAction | undefined>(undefined);
+  const postActivity = usePostActivity();
+
+  const handleChange = useCallback<(voteAction: VoteAction) => void>(


Use setSelectedVoteAction directly.

packages/bundle/src/markdown/markdownItPlugins/respectCRLF.ts

packages/component/src/ActivityStatus/Slotted.tsx

SAlamuru1 · 2024-06-10T09:19:09Z

Is this feedback reactions works with CDN Bundle for vue.js project

milind045 · 2024-07-13T14:53:30Z

@compulim I want to use VoteAction to capture user's feedback to a message response. I see that clicking upvote or downvote button sends an "event" back to the server. However, where does one find the actvityId that was liked or disliked?
There is clientActvityId in channelData, but it doesn't message the activityId of the message that was sent to the channel. Kindly advise how to correlate the the VoteAction event with the correct activty/ replyActivity

compulim added 30 commits September 1, 2023 23:31

Refactor TextAttachment

f2eab87

Add schema.org

2251f7c

Add @types/jest

2274b79

Refactor Markdown-It plugins

686b880

Fix test

ee27ca1

Add ModalDialogComposer

ff82b1e

Use constants

b2795be

Add link definitions

deb340e

Clean up

42b5166

Cosmetics

03168b0

Cosmetics

871dd35

Add citation, definitions, provenance, vote buttons

62c3573

Use ReplyAction instead of Person

a816fc7

Clean up styles

15cd284

Style slotted

ce8cce2

Localize feedback buttons

175241d

Focus indicator cosmetics

5ad105c

Click on citation should show modal

e78f953

Fix colors

0016d20

Fix accessibility on link definitions

5856ec1

Add open in new window icon

f0d8a9b

Clean up

4aec3e8

Show host if no name on link

bfb13d6

Remove transition

b887beb

Chevron cosmetics

a165d48

Clean up vote buttons

15e685a

Use VoteAction directly

8ed64e4

Fix @context

5298a79

Add onErrorResumeNext

642068d

Allow different type for default value

fac68a7

compulim added 3 commits September 7, 2023 17:40

Update scrollbar size

1b636aa

Update screenshot

911d4d7

Fix tests because DOM go deeper

24c17df

compulim marked this pull request as ready for review September 7, 2023 20:08

compulim requested review from a-b-r-o-w-n, cwhitten, srinaath, tdurnford and beyackle2 as code owners September 7, 2023 20:08

compulim changed the title ~~New UIs for bot response~~ [HIGH] New UIs for better bot response Sep 7, 2023

This was referenced Sep 8, 2023

Fix rendering erroneous Adaptive Card #4852

Merged

[HIGH] Bump deps after release of 4.15.9 #4843

Merged

tdurnford reviewed Sep 11, 2023

View reviewed changes

packages/bundle/src/markdown/markdownItPlugins/respectCRLF.ts Show resolved Hide resolved

tdurnford reviewed Sep 11, 2023

View reviewed changes

packages/component/src/ActivityStatus/Slotted.tsx Show resolved Hide resolved

tdurnford approved these changes Sep 11, 2023

View reviewed changes

compulim mentioned this pull request Sep 11, 2023

Flush new UIs for better bot response #4856

Closed

compulim merged commit d62393e into main Sep 11, 2023

compulim deleted the feat-better-bot-response branch September 11, 2023 17:55

compulim mentioned this pull request Sep 11, 2023

[HIGH] Incorporated PR comments of 4846 #4857

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HIGH] New UIs for better bot response #4846

[HIGH] New UIs for better bot response #4846

compulim commented Sep 2, 2023 •

edited

Loading

tdurnford Sep 11, 2023

tdurnford Sep 11, 2023

tdurnford Sep 11, 2023

tdurnford Sep 11, 2023

tdurnford Sep 11, 2023

tdurnford Sep 11, 2023

tdurnford Sep 11, 2023

compulim Sep 11, 2023

tdurnford Sep 11, 2023

tdurnford Sep 11, 2023

SAlamuru1 commented Jun 10, 2024

milind045 commented Jul 13, 2024

		@@ -0,0 +1,7 @@
		export type TypeOfArray<T extends unknown[] \| Array<unknown> \| ReadonlyArray<unknown>> = T extends (infer I)[]

	color: 'var(--webchat__color--accent)',
	color: tokens.colorForegroundAccent,

	const [selectedVoteAction, setSelectedVoteAction] = useState<VoteAction \| undefined>(undefined);
	const [selectedVoteAction, setSelectedVoteAction] = useState<VoteAction \| undefined>();

[HIGH] New UIs for better bot response #4846

[HIGH] New UIs for better bot response #4846

Conversation

compulim commented Sep 2, 2023 • edited Loading

Changelog Entry

Added

Description

Design

Based on Schema.org

CSS variables

Markdown-it "betterLink" plug-in

Link definitions (a.k.a. reference-style links)

Citation in link definitions

Sample activity

Design

Feedback buttons

Sample activity

Design

Provenance

Sample activity

What is missing

What is breaking

Specific Changes

-

Review Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SAlamuru1 commented Jun 10, 2024

milind045 commented Jul 13, 2024

compulim commented Sep 2, 2023 •

edited

Loading