Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temporary changes to change casing on documents #7

Merged
merged 9 commits into from
Nov 14, 2022
13 changes: 10 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
[![Pull Request Build](https://github.com/microsoft/VirtualClient/actions/workflows/pull-request.yml/badge.svg)](https://github.com/microsoft/VirtualClient/actions/workflows/pull-request.yml)
[![Document Build](https://github.com/microsoft/VirtualClient/actions/workflows/deploy-doc.yml/badge.svg?branch=main)](https://github.com/microsoft/VirtualClient/actions/workflows/deploy-doc.yml)
[![Document Deployment](https://github.com/microsoft/VirtualClient/actions/workflows/pages/pages-build-deployment/badge.svg)](https://github.com/microsoft/VirtualClient/actions/workflows/pages/pages-build-deployment)
[![NuGet Release Status](https://msazure.visualstudio.com/One/_apis/build/status/OneBranch/CRC-AIR-Workloads/microsoft.VirtualClient?branchName=main)](https://msazure.visualstudio.com/One/_build/latest?definitionId=297462&branchName=main)

Virtual Client is a unified workload and system monitoring platform for running customer-representative scenarios on virtual machines or physical hosts/blades in the Azure Cloud.
The platform supports a wide range of different industry standard/benchmark workloads used to measuring various aspects of the system under test (e.g. CPU, I/O, network performance, power consumption). It has been an inner-source project in Microsoft for two years and now it is open sourced on GitHub.
Expand All @@ -20,20 +21,26 @@ The platform additionally provides the ability to capture important performance

## [Supported Workloads](https://microsoft.github.io/VirtualClient/docs/overview/#supported-benchmark-workloads)

---
## Please bear with us for a second
VirtualClient is an MSFT inner source project we are migrating to GitHub. We are still gradually migrating our code and documents.
Please be patient if the documents have wrong links. We are actively cleaning those up.

#### About benchmark examples in source
VirtualClient has example benchmark outputs in source, for unit-testing purpose, to make sure our parsers work correctly.
Those runs might or might not be ran on Azure VMs. The results have also been randomly scrubbed. These examples do not represent Azure VM performance. They are in the source purely for unit testing purposes.



## Telemetry Notice
Data Collection.

The software may collect information about you and your use of the software and send it to Microsoft. Microsoft may use this information to provide services and improve our products and services. You may turn off the telemetry as described in the repository. There are also some features in the software that may enable you and Microsoft to collect data from users of your applications. If you use these features, you must comply with applicable law, including providing appropriate notices to users of your applications together with a copy of Microsoft’s privacy statement. Our privacy statement is located at https://go.microsoft.com/fwlink/?LinkID=824704. You can learn more about data collection and use in the help documentation and our privacy statement. Your use of the software operates as your consent to these practices.

#### VirtualClient does not collect your data by default
VirtualClient does not collect any of your benchmark data and upload to Microsoft. When benchmarking at scale, and leveraging VC's telemetry capabilities, users need to explicitly provide a connection string, that points to a user-owned Azure Data Explorer cluster. VirtualClient does host a Azure storage account to host the benchmark binaries or source. The only information VirtualClient team could infer from usage, is the download traces from Azure storage account.

#### About benchmark examples in source
VirtualClient has example benchmark outputs in source, for unit-testing purpose, to make sure our parsers work correctly.
Those runs might or might not be ran on Azure VMs. The results have also been randomly scrubbed. These examples do not represent Azure VM performance. They are in the source purely for unit testing purposes.

## Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a
Expand Down
4 changes: 2 additions & 2 deletions website/docs/overview/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,8 +255,8 @@ VirtualClient.exe --profile=PERF-CPU-OPENSSL.json --timeout=1440 --experimentId=
"message": "OpenSSLExecutor.Execute",
"severityLevel": "1",
"itemType": "trace",
"operation_Id": "6978d01d-b28d-4f81-8a0e-6296592dad07"
"operation_ParentId": "00000000-0000-0000-0000-000000000000"
"operation_Id": "6978d01d-b28d-4f81-8a0e-6296592dad07",
"operation_ParentId": "00000000-0000-0000-0000-000000000000",
"appName": "VirtualClient",
"appHost": "virtualmachine01",
"sdkVersion": "1.9.0.0",
Expand Down
11 changes: 6 additions & 5 deletions website/docs/overview/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,18 @@ is with automation. That's how VC is born.
2. We need your contribution. If you can make improvements into how VirtualClient runs some benchmarks, we would really appreciate your contribution. This will enable everyone else around the globe

## Why you should (or not) use VirtualClient

1. VC is so much more than just scripts to automate a workload. The expertise that went into automation is more valuable than automation itself. We understand there is so much to learn about every workload and every piece of hardware. VC is positioned to host the collective wisdom of perf engineers from different fields. We studied how to run every benchmark, and the best configurations to use, so that you don't have to spend your time to learn them.
2. VC has the vision to become the united industrial standard of benchmarking and monitoring systems.
3.

### When VirtualClient is probably not a good fit
1. You only need one-off testing with a workload you already understand. You might have some scripts that doesn't need much maintainces.
2.

2. You are fine tuning for an absolutely best score from one particular workload, for benchmarking competitions. VirtualClient is designed to measure **general experience**
from customers' perspective.


## How is VC used at Azure
1. Quality is top priority for Azure.
2. Azure uses VC to protect the customer from upadtes that will regress the reliability or performance.
Quality is top priority for Azure. Azure uses VC to protect the customer from updates that will regress the reliability or performance.

## Will VC support commercial workloads like SPECcpu, GeekBench?
Yes, VC plans to support commercial workloads in the future releases. VirtualClient needs to establish a model to enable the workload automation, without distributing the workloads themselves.
Expand Down
21 changes: 11 additions & 10 deletions website/docs/overview/overview.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
id: overview
sidebar_position: 1
keywords: [benchmark, automation, workload, monitoring, microsoft,
coremark,hpcg, lapack, openfoam,redis, memcached, specjvm]
---

# Overview
Expand All @@ -9,9 +11,9 @@ The platform supports a wide range of different industry standard/benchmark work
The platform additionally provides the ability to capture important performance and reliability measurements from the underlying system. The platform supports all business-critical
Azure environments including guest/VM systems, host/blade systems and data center/DC lab systems. The platform additionally supports both x64 and ARM64 compute architectures.

* [Platform Features](../guides/features)
* [Platform Design](../guides/design)
* [Developer Guide](../developing/develop-guide)
* [Platform Features](../overview/features.md)
* [Platform Design](../overview/design.md)
* [Developer Guide](../developing/develop-guide.md)
* [Additional Usage Examples](../category/usage-scenarios)

## Team Contacts
Expand All @@ -27,7 +29,7 @@ VirtualClient handles the installation and execution of various tools. Individua

| **Benchmark** | **Specialization** | **Supported Platforms/Architectures** | **License(s)** |
|---|---|---|---|
| [7zip](../workloads/compression/Compression.md) | 7zip compression | linux-x64, linux-arm64 | [GNU LGPL](https://www.7-zip.org/faq.html) |
| [7zip](../workloads/compression/7zip.md) | 7zip compression | linux-x64, linux-arm64 | [GNU LGPL](https://www.7-zip.org/faq.html) |
| [AspNetBench](../workloads/aspnetbench/AspNetBench.md) | ASP.NET server | linux-x64, linux-arm64, win-x64, win-arm64 | [**ASP.NET**:MIT](https://github.com/dotnet/aspnetcore/blob/main/LICENSE.txt)<br/>[**Bombardier**:MIT](https://github.com/codesenberg/bombardier/blob/master/LICENSE) |
| [CoreMark](../workloads/coremark/CoreMark.md) | Generic CPU | linux-x64,linux-arm64 | [Apache+Custom](https://github.com/eembc/coremark/blob/main/LICENSE.md) |
| [CPS](../workloads/network/NetworkSuite.md) | Network RoundTripTime | linux-x64, linux-arm64, win-x64, win-arm64 | MSFT developed |
Expand All @@ -48,13 +50,13 @@ VirtualClient handles the installation and execution of various tools. Individua
| [NTttcp](../workloads/network/NetworkSuite.md) | Network bandwidth | linux-x64, linux-arm64, win-x64, win-arm64 | [MIT](https://github.com/microsoft/ntttcp/blob/main/LICENSE) |
| [OpenFOAM](../workloads/openfoam/OpenFOAM.md) | Fluidmechanics | linux-x64, linux-arm64 | [Custom](https://github.com/OpenFOAM/OpenFOAM-10/blob/master/COPYING) |
| [OpenSSL](../workloads/openssl/OpenSSL.md) | Cryptography | linux-x64, linux-arm64, win-x64 | [Apache-2.0](https://github.com/openssl/openssl/blob/master/LICENSE.txt) |
| [pbzip2](../workloads/compression/Compression.md) | pbzip2 compression | linux-x64, linux-arm64 | [BSD](http://compression.great-site.net/pbzip2/) |
| [pbzip2](../workloads/compression/pbzip2.md) | pbzip2 compression | linux-x64, linux-arm64 | [BSD](http://compression.great-site.net/pbzip2/) |
| [Prime95](../workloads/prime95/prime95.md) | Prime number search | linux-x64 | [Custom](https://www.mersenne.org/legal/) |
| [Redis](../workloads/redis/Redis.md) | Redis Performance | linux-x64, linux-arm64 | [**Redis**:BSD-3](https://github.com/redis/redis/blob/unstable/COPYING)<br/>[**Memtier**:GPL-2.0](https://github.com/RedisLabs/memtier_benchmark/blob/master/COPYING) |
| [SockPerf](../workloads/network/NetworkSuite.md) | Network latency | linux-x64, linux-arm64 | [Custom](https://github.com/Mellanox/sockperf/blob/sockperf_v2/copying) |
| [SPECjvm](../workloads/specjvm/SPECjvm.md) | Java Runtime | linux-x64, linux-arm64, win-x64, win-arm64 | [SPEC](https://www.spec.org/spec/docs/SPEC_General_License.pdf) |
| [stress-ng](../workloads/stress-ng/StressNg.md) | Fault Tolerance | linux-x64, linux-arm64 | [GPL-2.0](https://github.com/ColinIanKing/stress-ng/blob/master/COPYING) |
| [SuperBench](../workloads/superbenchmark/Superbenchmark.md) | Machine learning | linux-x64 | [MIT](https://github.com/microsoft/superbenchmark/blob/main/LICENSE) |
| [SuperBench](../workloads/superbenchmark/superbenchmark.md) | Machine learning | linux-x64 | [MIT](https://github.com/microsoft/superbenchmark/blob/main/LICENSE) |



Expand All @@ -75,11 +77,10 @@ VirtualClient is just migrated from MSFT inner-source to GitHub. We are still re
:::

## Current Stable Version(s)
The following section designates the latest stable versions of the Virtual Client. If the version of the Virtual Client is not in this list, then it is NOT recommended
for use anymore.
The following section designates the latest stable versions of the Virtual Client. If the version of the Virtual Client is not in this list, then it is NOT recommended for use anymore.

### Version WIP
Released on ?
### Version 0.0.2
Released

* NuGet Release

Expand Down
21 changes: 0 additions & 21 deletions website/docs/overview/roadmap.md

This file was deleted.

33 changes: 33 additions & 0 deletions website/docs/overview/vision.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
id: vision
sidebar_position: 10
---

# Vision
- VirtualClient becomes the industrial standard to running benchmarks.
- VirtualClient becomes the knowledge hub for performance engineering.

1. No one should spend days learning a benchmark just to run once.
2.

# Roadmap

## Platform features
1. Run VirtualClient in Docker
2. Publish VC in mainstream package managers (apt,dnf,yum,zypper)


## Workloads
1. Commercial workload will come to VirtualClient soon. We will provide a model to release the automation of a workload, with user provided binaries.



## Monitors
1. NvQual

## Dependencies


## Have something you want to see?
Send us a [Feature request](https://github.com/microsoft/VirtualClient/issues/new?assignees=&labels=&template=feature_request.md&title=)

4 changes: 1 addition & 3 deletions website/docs/workloads/aspnetbench/AspNetBenchMetrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,7 @@ Virtual Client for net impact analysis.

[Bombardier output example](https://github.com/codesenberg/bombardier#examples)

### System Metrics
* [Performance Counters](./PerformanceCounterMetrics.md)
* [Power/Temperature Measurements](./PowerMetrics.md)


### Workload-Specific Metrics
The following metrics are captured during the operations of the AspNetBench workload.
Expand Down
3 changes: 1 addition & 2 deletions website/docs/workloads/aspnetbench/AspNetBenchProfiles.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@ The following profiles run customer-representative or benchmarking scenarios usi

* [Workload Details](./AspNetBench.md)
* [Workload Profile Metrics](./AspNetBenchMetrics.md)
* [Workload Packages](./DependencyPackages.md)
* [Usage Scenarios/Examples](./UsageScenarios.md)


-----------------------------------------------------------------------

Expand Down
3 changes: 3 additions & 0 deletions website/docs/workloads/compression/7zip.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
---
id: 7zip
---

# 7zip
7-Zip is a file archiver with a high compression ratio.
Expand Down
4 changes: 1 addition & 3 deletions website/docs/workloads/compression/CompressionMetrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,7 @@
The following document illustrates the type of results that are emitted by the compression/decompression workloads and captured by the
Virtual Client for net impact analysis.

### System Metrics
* [Performance Counters](./PerformanceCounterMetrics.md)
* [Power/Temperature Measurements](./PowerMetrics.md)


### Workload-Specific Metrics
The following metrics are emitted by the compression/decompression workloads itself.
Expand Down
3 changes: 1 addition & 2 deletions website/docs/workloads/compression/CompressionProfiles.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@ The following profiles run customer-representative or benchmarking scenarios usi

* [Workload Details](./Compression.md)
* [Workload Profile Metrics](./CompressionMetrics.md)
* [Workload Packages](./DependencyPackages.md)
* [Usage Scenarios/Examples](./UsageScenarios.md)


-----------------------------------------------------------------------

Expand Down
3 changes: 3 additions & 0 deletions website/docs/workloads/compression/Gzip.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
---
id: gzip
---

# Gzip
gzip is a single-file/stream lossless data compression utility, where the resulting compressed file generally has the suffix .gz.
Expand Down
3 changes: 3 additions & 0 deletions website/docs/workloads/compression/Pbzip2.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
---
id: pbzip2
---

# Pbzip2
PBZIP2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines. The output of this version is fully compatible with bzip2 v1.0.2 or newer (ie: anything compressed with pbzip2 can be decompressed with bzip2). PBZIP2 should work on any system that has a pthreads compatible C++ compiler (such as gcc).
Expand Down
4 changes: 1 addition & 3 deletions website/docs/workloads/coremark/CoreMarkMetrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,7 @@
The following document illustrates the type of results that are emitted by the CoreMark workload and captured by the
Virtual Client for net impact analysis.

### System Metrics
* [Performance Counters](./PerformanceCounterMetrics.md)
* [Power/Temperature Measurements](./PowerMetrics.md)


### Workload-Specific Metrics
The following metrics are emitted by the CoreMark workload itself.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,7 @@
The following document illustrates the type of results that are emitted by the DeathStarBench workload and captured by the
Virtual Client for net impact analysis.

### System Metrics
* [Performance Counters](./PerformanceCounterMetrics.md)
* [Power/Temperature Measurements](./PowerMetrics.md)


### Workload-Specific Metrics
The following metrics are emitted by the DeathStarBench workload itself.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@ The following profiles run customer-representative or benchmarking scenarios usi

* [Workload Details](./DeathStarBench.md)
* [Workload Profile Metrics](./DeathStarBenchMetrics.md)
* [Workload Packages](./DependencyPackages.md)
* [Usage Scenarios/Examples](./UsageScenarios.md)


-----------------------------------------------------------------------

Expand Down
Loading