Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time-to-first-frame metrics benchmarking and timeline chart #286

Merged
merged 17 commits into from
Dec 21, 2023
Merged

Time-to-first-frame metrics benchmarking and timeline chart #286

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
2bc53b0
data channel benchmarking message
niyatim23 Oct 16, 2023
d092b3f
additional logs
niyatim23 Oct 26, 2023
e47e9fe
granularity in time-to-first-frame
niyatim23 Oct 30, 2023
566028e
master and viewer metrics timeline chart
niyatim23 Nov 8, 2023
f36a203
signaling metrics viewer + master breakdown
niyatim23 Nov 9, 2023
b2a9c45
color coded, dynamic height adjustment, dynamic dataTable row addition
niyatim23 Nov 10, 2023
714a97f
connectAsViewer, enable non-json messages, enable viewing them in mes…
niyatim23 Nov 14, 2023
2deec4e
set timelineChartTestLength to 20
niyatim23 Nov 14, 2023
6d85b1a
cleanup 1
niyatim23 Nov 14, 2023
2ee30d6
pr cleanup as per comments
niyatim23 Nov 20, 2023
0eb2c00
adding a comment
niyatim23 Nov 20, 2023
b2e97de
readme update
niyatim23 Nov 20, 2023
55b1810
rename timestamp
niyatim23 Dec 8, 2023
05113cc
update the readme
niyatim23 Dec 18, 2023
a943e90
rename timestamps
niyatim23 Dec 18, 2023
660fe03
decouple from dqp
niyatim23 Dec 20, 2023
5a146f1
rename
niyatim23 Dec 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -369,6 +369,40 @@ An enum with the following values:
* `MASTER`
* `VIEWER`

## Metrics

The DQP metrics can be enabled in the sample application by checking a box before starting the viewer on the JS side. On the master, there is a flag in the sample application which `(pSampleConfiguration->enableSendingMetricsToViewerViaDc)` can be set to TRUE to send metrics from the master to the [JS](https://awslabs.github.io/amazon-kinesis-video-streams-webrtc-sdk-js/examples/index.html) viewer. This helps get a detailed breakdown of time-to-first-frame and all the processes and API calls on master and the viewer both. This is intended to be used with the KVS WebRTC C SDK running as the master and the JS SDK as the viewer. The master sends peer, ice-agent, signaling and data-channel metrics to the viewer which are plotted ~ 20 seconds after the viewer is started. Since the timeline plot is intended to understand the time-to-first-frame, the sample web page needs to be refreshed and the master needs to be restarted if a new / updated plot is needed. While using the SDK in this mode, it is expected that all datachannel messages are JSON messages. This feature is only meant to be used for a single viewer at a time.

| Category | Metric | Calculation | Description |
|--------------------|--------------------------------|-------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| viewer-wait | viewer-waiting-for-master | Calculated as the time between the signaling client going to `open` state and the time to start the master | Time duration the viewer was waiting for the master to start (time to start the SDK after the viewer signaling channel was connected) |
| viewer-signaling | signaling-viewer | Calculated as the time between creation of the `KinesisVideoClient` and the signaling client going to `open` state | Time taken to establish a signaling connection on the viewer-side |
| viewer-signaling | setup-media-player-viewer | Calculated as the time taken for `getUserMedia` call | Time taken to setup a media player on the viewer-side by seeking permissions for mic / camera (if needed), fetch tracks from the same and add them to the peer connection peer. |
| viewer-signaling | signaling-viewer-describe-channel | Calculated as the time taken for the `describeSignalingChannel` call | Time taken for the API call to describeSignalingChannel on the viewer |
| viewer-signaling | signaling-viewer-describe-media-storage-config | Calculated as the time taken for the `describeMediaStorageConfiguration` call | Time taken for the API call to describeSignalingChannel on the viewer |
| viewer-signaling | signaling-viewer-get-signaling-channel-endpoint | Calculated as the time taken for the `getSignalingChannelEndpoint` | Time taken for the API call to getSignalingChannelEndpoint on the viewer |
| viewer-signaling | signaling-viewer-get-ice-server-config | Calculated as the time taken for the `getIceServerConfig` | Time taken for the API call to getIceServerConfig on the viewer |
| viewer-signaling | signaling-connect-as-viewer | Calculated as the time taken between the signalingClient creation with all the information from the previous steps and it reaching the `open` state | Time taken to open the websocket via connectAsViewer |
| viewer-sdp-exchange | sdp-exchange-viewer | Calculated as the time between setting the local description and the reception of an answer from the master | Time taken to send an offer and receive a response |
| viewer-ice | ice-gathering-viewer | Calculated as the time between `viewer.peerConnection.iceGatheringState` going from `gathering` to `complete` | Time taken to gather all ice candidates on the viewer |
| viewer-peer-connection | pc-establishment-viewer | Calculated as the time between `viewer.peerConnection.connectionState` going from `new` to `connected` | Time taken to establish the peer connection on the viewer |
| viewer-datachannel | datachannel-viewer | Calculated as the time between `timestamp1` (time at which the viewer sends a message) and `timestamp3` (time at which the viewer receives a response from the master) | Time taken to send a message to the master and receive a response back
| viewer-ttff | ttff-after-pc-viewer | Calculated as time between `viewer.peerConnection.connectionState` in `connected` state and `viewer.remoteView.addEventListener` goes to `loadeddata` | Time to first frame after the viewer\'s peer connection has been established |
| master-wait | master-waiting-for-viewer | Calculated as the time between signaling state going to connected and the button to start the viewer | Time duration the master was waiting for the viewer to start (time to click the button after the master signaling channel was connected) |
| master-signaling | signaling-master | Calculated as the time for the entire signaling process on the master | Time taken to establish a signaling connection on the master-side |
| master-signaling | signaling-master-describe-channel | Calculated as the time taken for the `describeSignalingChannel` call | Time taken for the API call to desribeSignalingChannel on the master |
| master-signaling | signaling-master-get-signaling-channel-endpoint | Calculated as the time taken for the `getSignalingChannelEndpoint` | Time taken for the API call to getSignalingChannelEndpoint on the master |
| master-signaling | signaling-master-get-ice-server-config | Calculated as the time taken for the `getIceServerConfig` | Time taken for the API call to getIceServerConfig on the master |
| master-signaling | signaling-master-get-token | Calculated as the time taken for `getCredentialsFn` for authentication | Time taken for the getToken call on the master |
| master-signaling | signaling-master-create-channel | Calculated as the time taken for the `createSignalingChannel` | Time taken createChannel API call on the master |
| master-signaling | signaling-master-connect | Calculated as the time taken for the `connectAsMaster` | Time taken for the signaling connect on the master |
| master-sdp-exchange | sdp-exchange-master | Calculated as the time between the master receiving an offer, processing it and sending an answer to the viewer | Time taken to respond to an offer from the viewer with an answer |
| master-ice | ice-gathering-master | Calculated as the time from starting the gathering to the time its scheduling was stopped | Time taken to gather all ice candidates on the master |
| master-peer-connection | pc-establishment-master | Calculated as the time taken to go to `RTC_PEER_CONNECTION_STATE_CONNECTED` | Time taken to establish the peer connection on the master |
| master-datachannel | datachannel-master | Calculated as the time between `timestamp2` (time at which the master sends a message) and `timestamp4` (time at which the master receives a response from the master) | Time taken to send a message to the viewer and receive a response back |
| master-ttff | ttff-after-pc-master | Calculated as the time between `metrics.master.peerConnection.endTime` and `viewer.remoteView.addEventListener` reaches `loadeddata` | Time to first frame after the master's peer connection has been established |
| ttff | ttff | Calculated as the time taken between the viewer button is clicked and `viewer.remoteView.addEventListener` reaches `loadeddata` | Time to first frame since the viewer button was clicked |

## Compatibility

The SDK is supported in the following browsers / environments:
Expand Down
14 changes: 11 additions & 3 deletions examples/app.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ function getFormValues() {
secretAccessKey: $('#secretAccessKey').val(),
sessionToken: $('#sessionToken').val() || null,
enableDQPmetrics: $('#enableDQPmetrics').is(':checked'),
enableProfileTimeline: $('#enableProfileTimeline').is(':checked'),
sendHostCandidates: $('#send-host').is(':checked'),
acceptHostCandidates: $('#accept-host').is(':checked'),
sendRelayCandidates: $('#send-relay').is(':checked'),
Expand Down Expand Up @@ -137,6 +138,10 @@ function onStop() {
$('#webrtc-live-stats').addClass('d-none');
}

if (getFormValues().enableProfileTimeline) {
$('#timeline-profiling').addClass('d-none');
}

$('#form').removeClass('d-none');
$('#join-storage-session-button').addClass('d-none');
ROLE = null;
Expand Down Expand Up @@ -216,15 +221,17 @@ $('#viewer-button').click(async () => {
$('#webrtc-live-stats').removeClass('d-none');
}

if (formValues.enableProfileTimeline) {
$('#timeline-profiling').removeClass('d-none');
}

$(remoteMessage).empty();
localMessage.value = '';
toggleDataChannelElements();

printFormValues(formValues);

startViewer(localView, remoteView, formValues, onStatsReport, event => {
remoteMessage.append(`${event.data}\n`);
});
startViewer(localView, remoteView, formValues, onStatsReport, remoteMessage);
});

$('#stop-viewer-button').click(onStop);
Expand Down Expand Up @@ -428,6 +435,7 @@ const fields = [
{ field: 'forceTURN', type: 'radio', name: 'natTraversal' },
{ field: 'natTraversalDisabled', type: 'radio', name: 'natTraversal' },
{ field: 'enableDQPmetrics', type: 'checkbox' },
{ field: 'enableProfileTimeline', type: 'checkbox' },
{ field: 'send-host', type: 'checkbox' },
{ field: 'accept-host', type: 'checkbox' },
{ field: 'send-relay', type: 'checkbox' },
Expand Down
50 changes: 37 additions & 13 deletions examples/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
<script src="https://unpkg.com/@ungap/url-search-params"></script>
<script src="https://webrtc.github.io/adapter/adapter-latest.js"></script>
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.2.1/dist/chart.umd.min.js"></script>
<script src="https://www.gstatic.com/charts/loader.js"></script>
<link rel="icon" type="image/png" href="favicon.ico">
</head>
<body>
Expand Down Expand Up @@ -177,6 +178,18 @@ <h4>Amazon KVS WebRTC DQP</h4>
</div>
</div>

<h4>Amazon KVS WebRTC Profiling Timeline chart</h4>
<div class="form-group">
<div class="form-check form-check-inline">
<input class="form-check-input" type="checkbox" id="enableProfileTimeline" value="enableProfileTimeline">
<label for="enableProfileTimeline" class="form-check-label">Enable C SDK master and JS viewer profile timeline<small>(Master + Viewer)</small></label>
<span data-delay="{ &quot;hide&quot;: 1500 }" data-position="auto" tabindex="0" class="text-info ml-1" data-toggle="tooltip" data-html="true" title="
<p>Enables the test and metrics for the Amazon KVS WebRTC by sending the master-side metrics to the viewer via datachannel and display a timeline chart.</p>
<a href=&quot;https://github.com/awslabs/amazon-kinesis-video-streams-webrtc-sdk-js/tree/develop?tab=readme-ov-file#metrics&quot;>Additional information</a>
"><sup>&#9432;</sup></span>
</div>
</div>

<details><summary class="h4">Advanced</summary>
<p><small>Filter settings for which ICE candidates and sent to and received from the peer.</small></p>
<div class="container">
Expand Down Expand Up @@ -338,23 +351,34 @@ <h5>From Master</h5>
</div>
</div>

<div id="dqpmetrics" class="d-none">
<h3 id="dqpmetrics-header">DQP Test Metrics (from Master)</h3>
<div class="row">
<div class="col">
<div class="card bg-light mb-3">
<div id="dqp-test"></div>
<div>
<div id="dqpmetrics" class="d-none">
<h3 id="dqpmetrics-header">DQP Test Metrics (from Master)</h3>
<div class="row">
<div class="col">
<div class="card bg-light mb-3">
<div id="dqp-test"></div>
</div>
</div>
</div>
<div class="col">
<div class="card bg-light mb-3">
<canvas id="metricsChart" style="width:100%"; height="400px"></canvas>
<div class="col">
<div class="card bg-light mb-3">
<canvas id="metricsChart" style="width:100%"; height="400px"></canvas>
</div>
</div>
</div>
<h3 id="live-stats-header">Live Stats (from Master)</h3>
<div class="card bg-light mb-3">
<div id="webrtc-live-stats"></div>
</div>
</div>
<h3 id="live-stats-header">Live Stats (from Master)</h3>
<div class="card bg-light mb-3">
<div id="webrtc-live-stats"></div>
<div id="timeline-profiling" class="d-none" style="padding-top:20px;">
<div class="row">
<div class="col">
<h4 id="timeline-profiling-header"></h4>
<div id="timeline-chart" style="width:100%;height:0px">
</div>
</div>
</div>
</div>
</div>

Expand Down
Loading
Loading