This repository has been archived by the owner on Jun 10, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 61
Initial Tool and Pen Behavior Functionality #570
Merged
Merged
Changes from 7 commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
771c137
Revert "Merge pull request #497 from microsoft/revert/pen"
eanders-ms dc95048
Merge branch 'red' into tombu/pen-behavior
tombuMS be85e4b
Don't export pen tool for SDK
tombuMS a07e6ae
Merge branch 'red' into tombu/pen-behavior
tombuMS de3836a
Fix bug where action is not found on the behavior
tombuMS 0802530
PR feedback and lint fixes
tombuMS d333810
Additional PR feedback
tombuMS 65ef363
Cleanup
tombuMS 6260655
Merge branch 'red' into tombu/pen-behavior
tombuMS File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,102 @@ | ||
# Behavior Actions | ||
|
||
Behaviors are high level concepts that are made up of one or more actions. These actions make up the interactions | ||
that the user makes with the given behavior. These actions are broken up in to three states: `started`, `performing` and `stopped`. | ||
|
||
- `started` - The state of the initial frame that an action has started.. | ||
- `performing` - The state of all frames after `started` while the action is being performed. | ||
- `stopped` - The state of the frame after the action has been stopped and all other frames after until started again. | ||
|
||
These actions result in events being raised from the runtime to the SDK for app developers code to listen to. These events are based | ||
off of the threee action states above as such: | ||
|
||
- `started` - Fired once when the action has first transitioned to started. | ||
- `performing` - Fired during synchronization updates while the action is being performed. This is an optional event that is exposed only on behaviors that provide them. | ||
- `stopped` - Fired once when the action has first transitioned to the stopped state. | ||
|
||
**Proposed**: In addition, there is an optional update event that can be registered to the action for getting updates while that | ||
action is being performed on the client. | ||
|
||
## Architecture | ||
|
||
### Action State | ||
``` ts | ||
export type ActionState | ||
= 'started' | ||
| 'performing' | ||
| 'stopped' | ||
; | ||
``` | ||
|
||
### Action Handler | ||
``` ts | ||
/** | ||
* The action handler function type. | ||
*/ | ||
export type ActionHandler = (user: User, actionData?: any) => void; | ||
``` | ||
|
||
### Action API | ||
``` ts | ||
/** | ||
* Add a handler for the given action state for when it is triggered. | ||
* @param actionState The action state that the handle should be assigned to. | ||
* @param handler The handler to call when the action state is triggered. | ||
*/ | ||
public on(actionState: ActionState, handler: ActionHandler): this; | ||
|
||
/** | ||
* Gets the current state of the action for the user with the given id. | ||
* @param user The user to get the action state for. | ||
* @returns The current state of the action for the user. | ||
*/ | ||
public getState(user: User): ActionState; | ||
|
||
/** | ||
* Get whether the action is active for the user with the given id. | ||
* @param user - The user to get whether the action is active for, or null | ||
* if active for any user is desired.. | ||
* @returns - True if the action is active for the user, false if it is not. In the case | ||
* that no user is given, the value is true if the action is active for any user, and false | ||
* if not. | ||
*/ | ||
public isActive(user?: User): boolean | ||
|
||
/** | ||
* Add a handler for the performing update call for this action. Callback called on the | ||
* standard actor update cadence while an action is being performed, accompanied by optional | ||
* action data. | ||
* @param handler The handler to call when the performing action update comes in from the client. | ||
* **/ | ||
public onPerformingUpdate((handler: ActionHandler): this; | ||
``` | ||
|
||
## Network | ||
|
||
The `PerformAction` payload contains the type of state that the action is going through, as well as an | ||
optional `actionData` object for the action being performed. There is also a third category for the | ||
`PerformAction` payload that is `performingAction` which uses the payload to send up `actionData` if the | ||
behavior wished to convey this additional data for while the action is being performed. | ||
|
||
### Perform Action Payload | ||
``` ts | ||
/** | ||
* @hidden | ||
* Engine to app. The user is performing an action for a behavior. | ||
*/ | ||
export type PerformAction = Payload & { | ||
type: 'perform-action'; | ||
userId: Guid; | ||
targetId: Guid; | ||
behaviorType: BehaviorType; | ||
actionName: string; | ||
actionState: ActionState & `performing`; | ||
actionData?: any; | ||
}; | ||
``` | ||
|
||
## Client | ||
|
||
The client will be respondible for sending `PerformAction` messages when an action is started or stopped, | ||
as well as performing updates on the standard actor update cadence. Any date that the specific action should | ||
pass along in the context of the behavior will be passed along in the optional action data object in the payload. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
# High Resolution Actor Transform Updates | ||
|
||
There are cases that a developer may want a higher resolution of transform updates that occur for an actor. | ||
To facilitate this and allow for still having a regulated bandwidth, we can add an additional API on the actor | ||
to allow for a developer to register/de-register for high resolution updates that would give an array of transforms | ||
containing the full high resolution path that occured for the actor during the 10 hz regulated network update. | ||
|
||
## API | ||
|
||
A new actor API will be exposed to allow a developer to register and de-register for high resolution | ||
transform updates. | ||
|
||
``` ts | ||
/** | ||
* Add a callback to the actor to receive path updates for while the tool is being used | ||
*/ | ||
public onHighResTransfromUpdate((path: Transform[]) => void) { | ||
// Path callback registration. | ||
// Enable high resolution updates on the client. | ||
} | ||
|
||
/** | ||
* Remove a callback to the actor to receive path updates for while the tool is being used | ||
*/ | ||
public offHighResTransfromUpdate((path: Transform[]) => void) { | ||
// Path callback de-registration. | ||
// Disable high resolution updates on the client. | ||
} | ||
|
||
``` | ||
|
||
## Network | ||
|
||
The actor update structure and payload will now need to be able to accept an array of transforms per update. | ||
This array would contain one transform in the case of normal resolution transform updates, and more in the | ||
case that high resolution transform updates were enabled for the actor. | ||
|
||
## Client Implementation | ||
|
||
The client side changes will require a subscription mechanism on the actor runtime instance to signal that | ||
high resolution updates need to be gathered for the actor during a fixed update interval and queuded to be | ||
sent up with the normal 10 hz update message for the actor. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,87 @@ | ||
# Pen Behavior | ||
|
||
The pen behavior is a method of recording the path by which a user moves a grabbed object that they activate so that various scenearios | ||
can be carried out that wish to capture the virtual drawing of a user. This behavior will carry with it client side visuals | ||
to ensure low-latency feedback, but will supply the complete curve data needed by the MRE app to acto on once the drawing is done. | ||
The `PenBehavior` will be an extension of the `ToolBehavior`. | ||
|
||
## Architecture | ||
|
||
There will be a new `PenBehavior` that extends the `ToolBehavior` that will have the ability to generate a wet ink effect along the use | ||
path on the client. | ||
|
||
### Behavior | ||
``` ts | ||
export interface PenBehaviorInitOptions { | ||
drawOriginActorId: Guid; | ||
} | ||
|
||
/** | ||
* Pen behavior class containing the target behavior actions. | ||
*/ | ||
export class PenBehavior extends ToolBehavior { | ||
public drawOriginActorId: Guid; | ||
|
||
/** @inheritdoc */ | ||
public get behaviorType(): BehaviorType { return 'pen'; } | ||
} | ||
``` | ||
|
||
### Draw Data | ||
|
||
``` ts | ||
export interface DrawDataLike { | ||
transform: Transform; | ||
// Potentially additional data to come, such as: | ||
pressure: number; | ||
} | ||
``` | ||
|
||
## Network | ||
|
||
The network messages for this behavior will use the standard behavior payloads for actions started and stopped. The two actions supported | ||
by the Pen behavior are hold (begin/end) and drawing (begin/end). | ||
|
||
In addition to the standard `PerformAction` start and stop payload, there will be a third `PerformAction` type for `perfoming` which will | ||
contain the draw data in the optional action data of the `PerformAction` payload. | ||
|
||
## Sync layer considerations | ||
|
||
Behaviors already have sync layer filtering to ensure that the action messages are for the client they happen on to the MRE app only. Grab | ||
also has special sync filtering based on the client that the grab is happening and the rest of the clients. | ||
|
||
## Client implementation | ||
|
||
The client will require the addition of a new pen behavior for the host app to implement, as well as a draw recording system that will facilitate | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What about client optimizations (pre-processing) of the recorded data? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you provide a example(s)? This is the first pass to get things working, and there will be more design around this feature, so knowing your requirements would certainly be helpful. 😄 |
||
the recording of the transform data over time while a draw is happening and to send that data in chunks during the draw based on a set frequency. | ||
|
||
## Example Usage | ||
|
||
``` ts | ||
// Create an actor | ||
this.penModel = MRE.Actor.CreateFromGltf(this.assets, { | ||
// from the glTF at the given URL, with box colliders on each mesh | ||
uri: `${this.baseUrl}/pen.glb`, | ||
colliderType: 'mesh', | ||
// Also apply the following generic actor properties. | ||
actor: { | ||
name: 'penTool', | ||
parentId: root.id, | ||
transform: { | ||
local: { | ||
scale: { x: 0.4, y: 0.4, z: 0.4 }, | ||
position: { x: 0, y: 1, z: -1 } | ||
} | ||
} | ||
} | ||
}); | ||
|
||
const penBehavior = Actor.SetBehavior<PenBehavior>(); | ||
|
||
penBehavior.onDrawEventReceived((drawData) => { | ||
// Hypothetical spline generator tool in node. | ||
const path = drawData.map(dd => dd.transform); | ||
const spline = SplineGenerator.generateSpline(path); | ||
// Generate a mesh from the spline and use it to create a new actor at the point of origin of the draw. | ||
}); | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
# Set Behavior API | ||
|
||
An API is needed on the actor for the developer to be able to set a bahavior for the actor to use to | ||
allow fo user interaction with that actor. The API should ensure that the behavior is tied to the one | ||
actor it is being set on and should be flexible enough to allow setting params on the the behavior at the | ||
time of creation or initialization. | ||
|
||
## API | ||
|
||
### In actor.ts | ||
``` ts | ||
/** | ||
* Sets the behavior on this actor. | ||
* @param behavior The type of behavior to set. Pass null to clear the behavior. | ||
* @param initOptions The init options object to pass to the behavior to initialize it. | ||
*/ | ||
public setBehavior<BehaviorT extends Behavior, InitOptionsT>( | ||
behavior: { new(initOptions?: InitOptionsT): BehaviorT }, | ||
initOptions?: InitOptionsT | ||
): BehaviorT { | ||
if (behavior) { | ||
const newBehavior = new behavior(initOptions); | ||
|
||
this.internal.behavior = newBehavior; | ||
this.context.internal.setBehavior(this.id, this.internal.behavior.behaviorType); | ||
return newBehavior; | ||
} | ||
|
||
this.internal.behavior = null; | ||
this.context.internal.setBehavior(this.id, null); | ||
return null; | ||
} | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,81 @@ | ||
# Tool Behavior | ||
|
||
The tool behavior allows the user to assign this behavior to an actor so that the actor can be held and | ||
used. | ||
|
||
## Architecture | ||
|
||
Create a new behavior that is an `ToolBehavior` for being able to add to an actor, that when added will | ||
automatically enable grabbable on that actor. This behavior will then allow the user to execute a primary | ||
action to begin recording the transform changes over time. | ||
|
||
``` ts | ||
/** | ||
* Tool behavior class containing the target behavior actions. | ||
*/ | ||
export class ToolBehavior extends TargetBehavior { | ||
private _holding: DiscreteAction = new DiscreteAction(); | ||
private _using: DiscreteAction = new DiscreteAction(); | ||
|
||
/** @inheritdoc */ | ||
public get behaviorType(): BehaviorType { return 'tool'; } | ||
|
||
public get holding() { return this._holding; } | ||
public get using() { return this._using; } | ||
|
||
/** | ||
* Add a holding handler to be called when the given hover state is triggered. | ||
* @param holdingState The holding state to fire the handler on. | ||
* @param handler The handler to call when the holding state is triggered. | ||
* @return This tool behavior. | ||
*/ | ||
public onHolding(holdingState: 'grabbed' | 'dropped', handler: ActionHandler): this { | ||
const actionState: ActionState = (holdingState === 'grabbed') ? 'started' : 'stopped'; | ||
this._holding.on(actionState, handler); | ||
return this; | ||
} | ||
|
||
/** | ||
* Add a using handler to be called when the given hover state is triggered. | ||
* @param usingState The using state to fire the handler on. | ||
* @param handler The handler to call when the using state is triggered. | ||
* @return This tool behavior. | ||
*/ | ||
public onUsing(usingState: 'started' | 'stopped', handler: ActionHandler): this { | ||
const actionState: ActionState = (hoverState === 'started') ? 'started' : 'stopped'; | ||
this._using.on(actionState, handler); | ||
return this; | ||
} | ||
} | ||
``` | ||
|
||
## Network | ||
|
||
The network messages will use the behavior payloads of SetBehavior and PerformAction payloads the same way all | ||
behaviors do. In addition, there is an additional message that would come in the form of the path of the tool | ||
while it is being used. This will be sent with a higher fidelity than the normal 10 hz transform updates, and | ||
will be in the form of a collection of transform recordings. | ||
|
||
The message payload for the using path updates could look like the following: | ||
|
||
``` ts | ||
export type UsingToolPath = Payload & { | ||
type: 'using-tool-path'; | ||
actorId: toolActorId; | ||
path: Transform[]; | ||
} | ||
``` | ||
|
||
## Sync layer considerations | ||
|
||
Synchronization will follow the same procedure as all other behaviors. Nothing special about this behavior. | ||
The additional sync concerns will center around the path data being sent to the MRE app from the tool behavior | ||
when it is being used. This path data only needs to be sent to the MRE app and not the other clients, as the | ||
MRE app will be responsible for doing what it wants with the path, or behavior that extends `ToolBehavior` may | ||
have its own work it does with this path. | ||
|
||
## Client implementation | ||
|
||
Client implementation is similar to all other behaviors in that a handler will be created for this behavior, as | ||
well as an interface for the client app to implement the actual behavior with. Additionally, a path recording | ||
system will need to be developed to generate the path recording while the tool is in use. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider adding a way to request a different update frequency, if the host supports a way to configure this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently the runtime only operates on a 10 hz frequency. We collect the points based on the fixed update of the host app's update loop, and aggregate in to this 10 hertz update cycle, so the actual fidelity is that of the fixed update. What would be the desired use case for changing the frequency that the updates come in and/or the frequency by which updates are sent across the network?