Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document improvements, and using openssl as getting started profile #10

Merged
merged 1 commit into from
Nov 14, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,6 @@ The platform additionally provides the ability to capture important performance

## [Supported Workloads](https://microsoft.github.io/VirtualClient/docs/overview/#supported-benchmark-workloads)

---
## Please bear with us for a second
VirtualClient is an MSFT inner source project we are migrating to GitHub. We are still gradually migrating our code and documents.
Please be patient if the documents have wrong links. We are actively cleaning those up.




## Telemetry Notice
Data Collection.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@
The following profiles run on Windows and will be published to the VirtualClient
NuGet package in the /content/win-x64|win-arm64/profiles folder
-->
<WindowsProfiles Include="profiles\GET-STARTED-OPENSSL.json" />
<WindowsProfiles Include="profiles\BOOTSTRAP-DEPENDENCIES.json" />
<WindowsProfiles Include="profiles\PERF-ASPNETBENCH.json" />
<WindowsProfiles Include="profiles\PERF-COMPRESSION.json" />
Expand All @@ -156,6 +157,7 @@
The following profiles run on Windows and will be published to the VirtualClient
NuGet package in the /content/linux-x64|linux-arm64/profiles folder.
-->
<LinuxProfiles Include="profiles\GET-STARTED-OPENSSL.json" />
<LinuxProfiles Include="profiles\BOOTSTRAP-DEPENDENCIES.json" />
<LinuxProfiles Include="profiles\PERF-ASPNETBENCH.json" />
<LinuxProfiles Include="profiles\PERF-COMPRESSION.json" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
"Parameters": {
"Scenario": "InstallGeekBench5WorkloadPackage",
"BlobContainer": "packages",
"BlobName": "geekbench5.1.0.0-1.zip",
"BlobName": "geekbench5.1.0.0.zip",
"PackageName": "geekbench5",
"Extract": true
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{
"Description": "OpenSSL 3.0 CPU Performance Workload",
"Parameters": { },
"Actions": [
{
"Type": "OpenSslExecutor",
"Parameters": {
"Scenario": "SHA256",
"CommandArguments": "speed -elapsed -seconds 10 sha256",
"PackageName": "openssl",
"Tags": "CPU,OpenSSL,Cryptography"
}
}
],
"Dependencies": [
{
"Type": "DependencyPackageInstallation",
"Parameters": {
"Scenario": "InstallOpenSSLWorkloadPackage",
"BlobContainer": "packages",
"BlobName": "openssl.3.0.0.zip",
"PackageName": "openssl",
"Extract": true
}
}
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"Parameters": {
"Scenario": "InstallLAPACKWorkloadPackage",
"BlobContainer": "packages",
"BlobName": "lapack.3.10.0-1.zip",
"BlobName": "lapack.3.10.0.zip",
"PackageName": "lapack",
"Extract": true
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@
"Parameters": {
"Scenario": "InstallOpenSSLWorkloadPackage",
"BlobContainer": "packages",
"BlobName": "openssl.3.0.0-1.zip",
"BlobName": "openssl.3.0.0.zip",
"PackageName": "openssl",
"Extract": true
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
"Parameters": {
"Scenario": "InstallGraph500WorkloadPackage",
"BlobContainer": "packages",
"BlobName": "graph500.3.0.0-1.zip",
"BlobName": "graph500.3.0.0.zip",
"PackageName": "graph500",
"Extract": true
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -376,7 +376,7 @@
"Parameters": {
"Scenario": "InstallDiskSpdWorkloadPackage",
"BlobContainer": "packages",
"BlobName": "diskspd.2.0.21-1.zip",
"BlobName": "diskspd.2.0.21.zip",
"PackageName": "diskspd",
"Extract": true
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -480,7 +480,7 @@
"Parameters": {
"Scenario": "InstallFIOWorkloadPackage",
"BlobContainer": "packages",
"BlobName": "fio.3.30.0-1.zip",
"BlobName": "fio.3.30.0.zip",
"PackageName": "fio",
"Extract": true
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@
"Parameters": {
"Scenario": "InstallFIOWorkloadPackage",
"BlobContainer": "packages",
"BlobName": "fio.3.30.0-1.zip",
"BlobName": "fio.3.30.0.zip",
"PackageName": "fio",
"Extract": true
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -484,7 +484,7 @@
"Parameters": {
"Scenario": "InstallFIOWorkloadPackage",
"BlobContainer": "packages",
"BlobName": "fio.3.30.0-1.zip",
"BlobName": "fio.3.30.0.zip",
"PackageName": "fio",
"Extract": true
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
"Parameters": {
"Scenario": "InstallLMbenchWorkloadPackage",
"BlobContainer": "packages",
"BlobName": "lmbench.1.0.0-2alpha8-1.zip",
"BlobName": "lmbench.1.0.0-2alpha8.zip",
"PackageName": "lmbench",
"Extract": true
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@
"Parameters": {
"Scenario": "InstallOpenFOAMWorkloadPackage",
"BlobContainer": "packages",
"BlobName": "openfoam.1.4.0-2.zip",
"BlobName": "openfoam.1.4.0.zip",
"PackageName": "openfoam",
"Extract": true
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"Parameters": {
"Scenario": "InstallSPECWorkloadPackage",
"BlobContainer": "packages",
"BlobName": "specjvm.2008.0.0-1.zip",
"BlobName": "specjvm.2008.0.0.zip",
"PackageName": "specjvm2008",
"Extract": true
}
Expand Down
2 changes: 1 addition & 1 deletion website/docs/developing/develop-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ in the Virtual Client derive from the base class 'VirtualClientComponent'.
"Parameters": {
"Scenario": "InstallOpenSSLWorkloadPackage",
"BlobContainer": "packages",
"BlobName": "openssl.3.0.0-1.zip",
"BlobName": "openssl.3.0.0.zip",
"PackageName": "openssl",
"Extract": true
}
Expand Down
57 changes: 28 additions & 29 deletions website/docs/guides/1-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,19 @@ sidebar_position: 1

# Getting Started

In this document, we are going to run a "hello-world" version of VirtualClient: benchmark your system with CoreMark.

:::danger
You are here about one week early. Some of the features might not work. <br/>
This is an MSFT inner source project we are migrating to GitHub and open source. Please make sure to bookmark and come back later.
:::
In this document, we are going to run a "hello-world" version of VirtualClient: benchmark your system's crypotography performance, with OpenSSL Speed, using SHA256 algorithm.

---

## Installation

#### *NuGet package*



- VirtualClient NuGet Package is at https://www.nuget.org/packages/VirtualClient
```powershell
PM> NuGet\Install-Package VirtualClient -Version 0.0.2
PM> NuGet\Install-Package VirtualClient -Version 0.0.3
```
- You could optionally download directly from NuGet https://www.nuget.org/api/v2/package/VirtualClient/0.0.2
- You could optionally download directly from NuGet https://www.nuget.org/api/v2/package/VirtualClient/0.0.3
- VC executable could be find in those paths
```treeview
VirtualClient/
Expand Down Expand Up @@ -61,21 +54,22 @@ VirtualClient\out\bin\Debug\x64\VirtualClient.Main\net6.0\win-x64\publish\Virtua

- Execute this command
```bash
VirtualClient --profile=PERF-CPU-COREMARK.json --profile=MONITORS-NONE.json --iterations=1
sudo ./VirtualClient --profile=GET-STARTED-OPENSSL.json --profile=MONITORS-NONE.json --iterations=1 --packages=https://virtualclient.blob.core.windows.net/packages
```
- `--profile=PERF-CPU-COREMARK.json` tells VC to run a CoreMark benchmark
- `--profile=GET-STARTED-OPENSSL.json` tells VC to run a stripped down version of OpenSSL benchmark.
- VirtualClient has a default profile, `--profile=MONITORS-NONE.json` overrides that behavior in this one-time run.
- `--iteration=1` Tells VC to run this profile once. Default behavior is to run profiles repetatively until timeout.
- `--packages=https://virtualclient.blob.core.windows.net/packages` defines the packages store that VC will download OpenSSL binary from. Not every workload needs binary download. You can also use your own binary and package store if desired.


:::caution
In this profile, VC will install gcc-9 and other development tools, and set gcc-9 as default compiler in your system.<br/>
In this profile, VC will download OpenSSL binaries onto your system, under `/virtualclient/packages/openssl.3.0.0/`.<br/>
If prefered, run in a Virtual Machine to avoid those changes to your system.
:::

## Read results and logs

- You will find three local files under directory `/vc/logs/`
- You will find three local files under directory `/virtualclient/logs/`
```bash
logs
├── events-20221109.log
Expand All @@ -84,29 +78,34 @@ logs
```
- Metrics.log contains the Metrics captured by the benchmark. Columns `metricName`, `metricValue`, `metricUnit` contain some of the most important information
from a benchmark run.
```json {16,18,19}
```json {16,17,18,19}
{
"timestamp": "2022-11-09T04:09:59.3573706+00:00",
"timestamp": "2022-11-14T07:26:18.2717145+00:00",
"level": "Information",
"message": "CoreMark.ScenarioResult",
"agentId": "ExampleClient",
"message": "OpenSSL.ScenarioResult",
"agentId": "testuser",
"appVersion": "1.6.0.0",
"clientId": "ExampleClient",
"executionProfileName": "PERF-CPU-COREMARK.json",
"executionProfilePath": "/home/vcvmadmin/vc/profiles/PERF-CPU-COREMARK.json",
"clientId": "testuser",
"executionProfileName": "GET-STARTED-OPENSSL.json",
"executionProfilePath": "/home/testuser/virtualclient/profiles/GET-STARTED-OPENSSL.json",
"executionSystem": null,
"experimentId": "3a225222-f834-4101-8a81-219a1f4e9587",
"metadata": {"experimentId":"3a225222-f834-4101-8a81-219a1f4e9587","agentId":"ExampleClient"},
"experimentId": "6619e311-e3ee-4727-a082-dc61f1fbb44d",
"metadata": {"experimentId":"6619e311-e3ee-4727-a082-dc61f1fbb44d","agentId":"testuser"},
"metricCategorization": "",
"metricDescription": "",
"metricMetadata": {},
"metricName": "Iterations/Sec",
"metricName": "sha256 16-byte",
"metricRelativity": "HigherIsBetter",
"metricUnit": "iterations/sec",
"metricValue": 93187.139894,
"parameters": {"scenario":"ScoreSystem","packageName":"coremark","profileIteration":1,"profileIterationStartTime":"2022-11-09T04:09:22.3729518Z"},
"metricUnit": "kilobytes/sec",
"metricValue": 323830.14,
"parameters": {"scenario":"SHA256","commandArguments":"speed -elapsed -seconds 10 sha256","packageName":"openssl","tags":"CPU,OpenSSL,Cryptography","profileIteration":1,"profileIterationStartTime":"2022-11-14T07:25:18.1731942Z"},
"platformArchitecture": "linux-arm64",
"scenarioArguments": "XCFLAGS=\"-DMULTITHREAD=4 -DUSE_PTHREAD\" REBUILD=1 LFLAGS_END=-pthread",
"scenarioArguments": "speed -multi 4 -elapsed -seconds 10 sha256",
"scenarioEndTime": "2022-11-14T07:26:18.2470775Z",
"scenarioName": "OpenSSL Speed",
"scenarioStartTime": "2022-11-14T07:25:18.2251103Z",
"systemInfo": {"etc"},
"tags": "CPU,OpenSSL,Cryptography",
"etc": ...
}
```
Expand All @@ -122,4 +121,4 @@ VC is designed for large scale perf testing. Check [Telemetry](./telemetry/telem
:::

## Congratulations !!
You just benchmarked your system with CoreMark.
You just benchmarked your system with OpenSSL.
6 changes: 3 additions & 3 deletions website/docs/guides/2-run-monitors.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,14 @@ sidebar_position: 2

- In the previous tutorial, we ran this command.
```bash
VirtualClient --profile=PERF-CPU-COREMARK.json --profile=MONITORS-NONE.json --iterations=1
VirtualClient --profile=GET-STARTED-OPENSSL.json --profile=MONITORS-NONE.json --iterations=1
```

- `--profile=MONITORS-NONE.json` removes the default monitor.
- Simply remove it to let VC run a default performance counter monitor
```bash
VirtualClient --profile=PERF-CPU-COREMARK.json --iterations=1
VirtualClient --profile=PERF-CPU-COREMARK.json --profile=MONITORS-DEFAULT.json --iterations=1
VirtualClient --profile=GET-STARTED-OPENSSL.json --iterations=1
VirtualClient --profile=GET-STARTED-OPENSSL.json --profile=MONITORS-DEFAULT.json --iterations=1
```
- VC will start collecting performance counters at an interval, and log as metrics.
- `MONITORS-DEFAULT.json` will install atop on Linux system, and use .NET SDK to read perf counters on Windows
Expand Down
2 changes: 1 addition & 1 deletion website/docs/monitors/atop.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ adjusted to change the precision of the counter samples window.

See the link at the top for more details on the exact performance counters that are captured.

[Monitor Profiles](./MonitorProfiles.md)
[Monitor Profiles](./monitor-profiles.md)

### Monitor Parameters
The following parameters are available on the monitor component .
Expand Down
6 changes: 3 additions & 3 deletions website/docs/monitors/monitor-profiles.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ order to capture performance and reliability information from the system while w
Instructs the Virtual Client to not run any monitors at all.


```
```bash
// Do not run any background monitors.
VirtualClient.exe --profile=PERF-CPU-OPENSSL.json --profile=MONITORS-NONE.json --system=Azure --timeout=1440 --packageStore="{BlobConnectionString|SAS Uri}"
```
Expand All @@ -16,8 +16,8 @@ VirtualClient.exe --profile=PERF-CPU-OPENSSL.json --profile=MONITORS-NONE.json -
The default monitor profile for the Virtual Client. This profile captures performance counters on the system using one or more different specialized
toolsets. This monitor profile will be used when no other monitor profiles are specified on the command line.

* [Performance Counters](./PerformanceCounterMetrics.md)
* [Atop](./AtopMonitor.md)
* [Performance Counters](./perf-counter-metrics.md)
* [Atop](./atop.md)

* **OS/Architecture Platform Support**
* Linux
Expand Down
Loading