Skip to content
This repository has been archived by the owner on Aug 21, 2024. It is now read-only.

Commit

Permalink
rewrite reactive UVOL (#8848)
Browse files Browse the repository at this point in the history
* rewrite reactive UVOL1

* Move worker to engine scope, refactor to use hooks, improve playlist mechanism

* Autoplay/paused fixes

* wip: init UniformUVOL

* fetch geometry segments and textures

* unifying non-uniform and uniform component

* Dispose attributes on GPU, track management

* clean up

* Update meshopt decoder module

`meshopt_decoder.module.js` stringifies some functions and spawns workers.
Production build optimizes these files and changes the function names, which causes issues.
It's better if we just hand over the function source, instead of stringifying.

* UVOL1 readability

* Loading effect for UVOL1

* Mirror media element paused

* Add explicit `autoplay` property
Now:
- `handleAutoplay()` calls media.play() synchronously.
- It also sets volumetric.paused to true.
- It cleans up event listeners after the media is played.

* Cleaned up pause/autoplay logic

- Overall improvements to UVOL1 Loading effect
- Show the first frame without waiting for the whole initial buffers

* minor correction to buffering logic

* Implement consistent and reliable loading effect

* revert AvatarDissolveComponent's changes

* Autotune targets

* disable shadows for uniform solve

* improve initial loading logic

* improve blending keyframes

* Improve setAttribute

attributes should not be disposed when setting keyframeB.
Because it'll be used by keyframeA in the next frame.

* Dont play when sufficient buffers not loaded yet

* Improve performance

- Stricter heuristic for autotuning targets
- Upload the textures (with initTexture) to GPU as soon as they're decoded.
  - Doing this at render time will cause some lag

* Move CORTOLoader to AssetLoaderState

* Update shader to r157

* Start UVOL2 at non-zero time

Also fixed the pause being overwritten by snapshot

* Support different segment sizes across targets

---------

Co-authored-by: Gheric Speiginer <gheric.speiginer@gmail.com>
  • Loading branch information
CITIZENDOT and speigg authored Oct 30, 2023
1 parent a502c48 commit 1889fb4
Show file tree
Hide file tree
Showing 15 changed files with 3,584 additions and 372 deletions.
1,089 changes: 1,089 additions & 0 deletions packages/client/public/loader_decoders/corto.js

Large diffs are not rendered by default.

19 changes: 8 additions & 11 deletions packages/editor/src/components/properties/VolumetricNodeEditor.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ import { VolumetricComponent } from '@etherealengine/engine/src/scene/components
import { PlayMode } from '@etherealengine/engine/src/scene/constants/PlayMode'

import VideocamIcon from '@mui/icons-material/Videocam'

import { ItemTypes } from '../../constants/AssetTypes'
import ArrayInputGroup from '../inputs/ArrayInputGroup'
import BooleanInput from '../inputs/BooleanInput'
Expand All @@ -41,7 +40,7 @@ import CompoundNumericInput from '../inputs/CompoundNumericInput'
import InputGroup from '../inputs/InputGroup'
import SelectInput from '../inputs/SelectInput'
import NodeEditor from './NodeEditor'
import { EditorComponentType, commitProperties, commitProperty, updateProperty } from './Util'
import { EditorComponentType, commitProperty, updateProperty } from './Util'

const PlayModeOptions = [
{
Expand Down Expand Up @@ -74,9 +73,7 @@ export const VolumetricNodeEditor: EditorComponentType = (props) => {
const volumetricComponent = useComponent(props.entity, VolumetricComponent)

const toggle = () => {
commitProperties(VolumetricComponent, {
paused: !volumetricComponent.paused.value
})
volumetricComponent.paused.set(!volumetricComponent.paused.value)
}

return (
Expand All @@ -93,13 +90,13 @@ export const VolumetricNodeEditor: EditorComponentType = (props) => {
</InputGroup>

<InputGroup
name="Auto Play"
label={t('editor:properties.media.lbl-paused')}
info={t('editor:properties.media.info-paused')}
name="Autoplay"
label={t('editor:properties.media.lbl-autoplay')}
info={t('editor:properties.media.info-autoplay')}
>
<BooleanInput
value={volumetricComponent.paused.value}
onChange={commitProperty(VolumetricComponent, 'paused')}
onChange={commitProperty(VolumetricComponent, 'autoplay')}
value={volumetricComponent.autoplay.value}
/>
</InputGroup>

Expand Down Expand Up @@ -133,7 +130,7 @@ export const VolumetricNodeEditor: EditorComponentType = (props) => {
/>
{volumetricComponent.paths && volumetricComponent.paths.length > 0 && volumetricComponent.paths[0] && (
<Button style={{ marginLeft: '5px', width: '60px' }} type="submit" onClick={toggle}>
{volumetricComponent.paused
{volumetricComponent.paused.value
? t('editor:properties.media.playtitle')
: t('editor:properties.media.pausetitle')}
</Button>
Expand Down
3 changes: 3 additions & 0 deletions packages/engine/src/assets/functions/createGLTFLoader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,9 @@ export const createGLTFLoader = (keepMaterials = false) => {
loader.register((parser) => new HubsComponentsExtension(parser))
loader.register((parser) => new VRMLoaderPlugin(parser, { helperRoot: new Group(), autoUpdateHumanBones: true }))
loader.register((parser) => new CachedImageLoadExtension(parser))
if (MeshoptDecoder.useWorkers) {
MeshoptDecoder.useWorkers(2)
}
loader.setMeshoptDecoder(MeshoptDecoder)

if (isClient) {
Expand Down
41 changes: 41 additions & 0 deletions packages/engine/src/assets/loaders/corto/CORTOLoader.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@

/*
CPAL-1.0 License
The contents of this file are subject to the Common Public Attribution License
Version 1.0. (the "License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
https://github.com/EtherealEngine/etherealengine/blob/dev/LICENSE.
The License is based on the Mozilla Public License Version 1.1, but Sections 14
and 15 have been added to cover use of software over a computer network and
provide for limited attribution for the Original Developer. In addition,
Exhibit A has been modified to be consistent with Exhibit B.
Software distributed under the License is distributed on an "AS IS" basis,
WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the
specific language governing rights and limitations under the License.
The Original Code is Ethereal Engine.
The Original Developer is the Initial Developer. The Initial Developer of the
Original Code is the Ethereal Engine team.
All portions of the code written by the Ethereal Engine team are Copyright © 2021-2023
Ethereal Engine. All Rights Reserved.
*/


import { BufferGeometry, Loader, LoadingManager } from 'three'

export class CORTOLoader {
constructor()
setDecoderPath(path: string): CORTOLoader
load(
url: string,
byteStart: number,
byteEnd: number,
onLoad: (geometry: BufferGeometry | null) => void,
): void
preload(): Promise<void>
dispose(): CORTOLoader
}
130 changes: 130 additions & 0 deletions packages/engine/src/assets/loaders/corto/CORTOLoader.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@

/*
CPAL-1.0 License
The contents of this file are subject to the Common Public Attribution License
Version 1.0. (the "License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
https://github.com/EtherealEngine/etherealengine/blob/dev/LICENSE.
The License is based on the Mozilla Public License Version 1.1, but Sections 14
and 15 have been added to cover use of software over a computer network and
provide for limited attribution for the Original Developer. In addition,
Exhibit A has been modified to be consistent with Exhibit B.
Software distributed under the License is distributed on an "AS IS" basis,
WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the
specific language governing rights and limitations under the License.
The Original Code is Ethereal Engine.
The Original Developer is the Initial Developer. The Initial Developer of the
Original Code is the Ethereal Engine team.
All portions of the code written by the Ethereal Engine team are Copyright © 2021-2023
Ethereal Engine. All Rights Reserved.
*/


import { BufferAttribute, BufferGeometry, FileLoader } from 'three'

class CORTOLoader {
constructor() {
this.decoderPath = ''
this.decoderPending = null

this.worker = null
this.lastRequest = 0
this.callbacks = {}

this.defaultAttributes = [
{ name: 'position', numComponents: '3' },
{ name: 'normal', numComponents: '3' },
{ name: 'color', numComponents: '4' },
{ name: 'uv', numComponents: '2' }
]
}

setDecoderPath(path) {
this.decoderPath = path
return this
}

load(url, byteStart, byteEnd, onLoad) {
if (!this.decoderPending) {
this.preload()
}

this.decoderPending.then(() => {
const request = this.lastRequest++
this.worker.postMessage({
request: request,
url: url,
byteStart: byteStart,
byteEnd: byteEnd
})
this.callbacks[request] = { onLoad: onLoad }
})
}

preload() {
if (this.decoderPending) return this.decoderPending

let that = this
let callbacks = this.callbacks
let lib = 'corto.js'

this.decoderPending = this._loadLibrary(lib, 'text').then((text) => {
text = URL.createObjectURL(new Blob([text]))
this.worker = new Worker(text)

this.worker.onmessage = function (e) {
var message = e.data
if (!callbacks[message.request]) return

const callback = callbacks[message.request]
const geometry = that._createGeometry(message.geometry)
callback.onLoad(geometry)
delete callbacks[message.request]
}
})

return this.decoderPending
}

dispose() {
if (this.worker) {
this.worker.terminate()
this.worker = null
}
return this
}

_createGeometry(geometry) {
if (!geometry) {
return null
}
var bufferGeometry = new BufferGeometry()

if (geometry.index) bufferGeometry.setIndex(new BufferAttribute(geometry.index, 1))

for (let i = 0; i < this.defaultAttributes.length; i++) {
let attr = this.defaultAttributes[i]
if (!geometry[attr.name]) continue
let buffer = geometry[attr.name]
bufferGeometry.setAttribute(attr.name, new BufferAttribute(buffer, attr.numComponents))
}
return bufferGeometry
}

_loadLibrary(url, responseType) {
var loader = new FileLoader(this.manager)
loader.setPath(this.decoderPath)
loader.setResponseType(responseType)

return new Promise((resolve, reject) => {
loader.load(url, resolve, undefined, reject)
})
}
}

export { CORTOLoader }
4 changes: 2 additions & 2 deletions packages/engine/src/assets/loaders/gltf/KTX2Loader.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ export class KTX2Loader extends CompressedTextureLoader {
load(
url: string,
onLoad: (texture: CompressedTexture) => void,
onProgress: (requrest: ProgressEvent<EventTarget>) => void | undefined,
onError: ((event: ErrorEvent) => void) | undefined
onProgress?: (requrest: ProgressEvent<EventTarget>) => void | undefined,
onError?: ((event: ErrorEvent) => void) | undefined
): CompressedTexture
}
330 changes: 205 additions & 125 deletions packages/engine/src/assets/loaders/gltf/meshopt_decoder.module.js

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion packages/engine/src/assets/state/AssetLoaderState.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,12 @@ Ethereal Engine. All Rights Reserved.

import { defineState } from '@etherealengine/hyperflux'
import { createGLTFLoader } from '../../assets/functions/createGLTFLoader'
import { CORTOLoader } from '../loaders/corto/CORTOLoader'

export const AssetLoaderState = defineState({
name: 'AssetLoaderState',
initial: () => ({
gltfLoader: createGLTFLoader()
gltfLoader: createGLTFLoader(),
cortoLoader: null! as CORTOLoader
})
})
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import { useEffect } from 'react'

import {
defineComponent,
hasComponent,
setComponent,
useComponent,
useOptionalComponent
Expand All @@ -38,6 +39,7 @@ import { useEntityContext } from '../../ecs/functions/EntityFunctions'
import { RendererState } from '../../renderer/RendererState'
import { addObjectToGroup, removeObjectFromGroup } from '../../scene/components/GroupComponent'
import { AudioNodeGroups, MediaComponent, MediaElementComponent } from '../../scene/components/MediaComponent'
import { VolumetricComponent } from '../../scene/components/VolumetricComponent'
import { ObjectLayers } from '../../scene/constants/ObjectLayers'
import { setObjectLayers } from '../../scene/functions/setObjectLayers'

Expand Down Expand Up @@ -71,6 +73,7 @@ export const PositionalAudioComponent = defineComponent({
},

onSet: (entity, component, json) => {
if (hasComponent(entity, VolumetricComponent) || hasComponent(entity, MediaComponent)) return
setComponent(entity, MediaComponent, {})

if (!json) return
Expand Down
26 changes: 0 additions & 26 deletions packages/engine/src/audio/systems/MediaSystem.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ import { EngineRenderer } from '../../renderer/WebGLRendererSystem'
import { StandardCallbacks, setCallback } from '../../scene/components/CallbackComponent'
import { MediaComponent } from '../../scene/components/MediaComponent'
import { VideoComponent, VideoTexturePriorityQueueState } from '../../scene/components/VideoComponent'
import { VolumetricComponent, endLoadingEffect } from '../../scene/components/VolumetricComponent'
import { AudioState, useAudioState } from '../AudioState'
import { PositionalAudioComponent } from '../components/PositionalAudioComponent'

Expand Down Expand Up @@ -102,7 +101,6 @@ globalThis.AudioEffectPlayer = AudioEffectPlayer

const mediaQuery = defineQuery([MediaComponent])
const videoQuery = defineQuery([VideoComponent])
const volumetricQuery = defineQuery([VolumetricComponent])
const audioQuery = defineQuery([PositionalAudioComponent])

const execute = () => {
Expand All @@ -112,30 +110,6 @@ const execute = () => {
setCallback(entity, StandardCallbacks.PAUSE, () => media.paused.set(true))
}

for (const entity of volumetricQuery()) {
const volumetric = getComponent(entity, VolumetricComponent)
const player = volumetric.player
if (player) {
player.update()
const height = volumetric.height
const step = volumetric.height / 150
if (volumetric.loadingEffectActive && player.mesh) {
if (volumetric.loadingEffectTime <= height) {
player.mesh.traverse((child: any) => {
if (child['material']) {
if (child.material.uniforms) {
child.material.uniforms.time = volumetric.loadingEffectTime
}
}
})
volumetric.loadingEffectTime += step
} else {
volumetric.loadingEffectActive = false
endLoadingEffect(entity, player.mesh)
}
}
}
}
for (const entity of audioQuery()) getComponent(entity, PositionalAudioComponent).helper?.update()

const videoPriorityQueue = getState(VideoTexturePriorityQueueState).queue
Expand Down
Loading

0 comments on commit 1889fb4

Please sign in to comment.