Skip to content

Commit

Permalink
Merge pull request #6 from mnellemann/doc_updated
Browse files Browse the repository at this point in the history
Updated Documentation readme and docs folder
  • Loading branch information
mnellemann authored Jun 7, 2024
2 parents 8fb7c82 + e2649e4 commit 11772cd
Show file tree
Hide file tree
Showing 6 changed files with 329 additions and 37 deletions.
91 changes: 62 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,28 +25,46 @@ More screenshots can be found in the [doc/screenshots/](doc/screenshots) folder.
There are few steps in the installation.

1. Prepare your Storage Virtualize
2. Installation of InfluxDB and Grafana software
3. Installation and configuration of *SVC Insights* (SVCi)
4. Configure Grafana and import example dashboards
2. Chose if you want to run it Containerized or not.
3. Installation of InfluxDB and Grafana software
4. Installation and configuration of *SVC Insights* (SVCi)
5. Configure Grafana and import example dashboards

### 1 - Prepare Storage Virtualize

- Create a user with the "Monitor" role
- To start extended statistics, the user must be in the "Administrator" role (svctask startstats -interval 30)
- :bulb: Normally the extended statistics is started, but to start or change the intervall
the user must be in the "Administrator" role (svctask startstats -interval 5) 5min

- Firewall access from SVCi.
- The SVCi needs access to the SVC on https/443, if the Virtualize code is below 8.6.+ then it will failback to SCP. same if the SVCi can't reach the system on https

### 2 - InfluxDB and Grafana Installation
- Check out the Doc folder for different installation methods.
- **Containerized**
- [readme-podman.md](doc/readme-podman.md)
- [readme-k8/oc.md](doc/readme-k8/oc.md)
- **OS based**
- [readme-aix.md](doc/readme-aix.md)
- [readme-redhat.md](doc/readme-redhat.md)
- [readme-suse.md](doc/readme-suse.md)
- [readme-suse.md](doc/readme-suse.md)


### 3 - InfluxDB and Grafana Installation - Non-Containerized

Install InfluxDB (v. **1.8** or later) on a host which is network accessible by the SVCi utility (the default InfluxDB port is 8086). You can install Grafana on the same server or any server which are able to connect to the InfluxDB database. The Grafana installation needs to be accessible from your browser (default on port 3000). The default settings for both InfluxDB and Grafana will work fine as a start.

- Create the empty *svci* database by running the **influx** CLI command and type:

```text
```shell
CREATE DATABASE "svci" WITH DURATION 365d REPLICATION 1;
```

See the [Influx documentation](https://docs.influxdata.com/influxdb/v1.8/query_language/manage-database/#create-database) for more information on duration and replication.

### 3 - SVCi Installation & Configuration

### 4 - SVCi Installation & Configuration

Install *SVCi* on a host, that can connect to your SAN Volume Controllers (on port 7443), and is also allowed to connect to the InfluxDB service. This *can be* the same LPAR/VM as used for the InfluxDB installation.

Expand All @@ -59,14 +77,15 @@ Install *SVCi* on a host, that can connect to your SAN Volume Controllers (on po
- Run the **/opt/svci/bin/svci** program in a shell, as a @reboot cron task or configure as a proper service - there are instructions in the [doc/](doc/) folder.
- When started, *svci* expects the InfluxDB database to exist already.

### 4 - Grafana Configuration
### 5 - Grafana Configuration

- Configure Grafana to use InfluxDB as a new datasource
- **NOTE:** set *Min time interval* depending on your SVCi *refresh* setting.
- Import example dashboards from [doc/dashboards/*.json](doc/dashboards/) into Grafana as a starting point and get creative making your own cool dashboards - please share anything useful :)

## Notes


### No data (or past/future data) shown in Grafana

This is most likely due to timezone, date and/or NTP not being configured correctly on the SAN Volune Controller and/or host running SVCi.
Expand All @@ -80,38 +99,47 @@ ALTER RETENTION POLICY "autogen" ON "svci" DURATION 156w
ALTER RETENTION POLICY "autogen" ON "svci" DURATION 90d
```

### Upgrading SVCi
------

On RPM based systems (RedHat, Suse, CentOS), download the latest *svci-x.y.z-n.noarch.rpm* file and upgrade:
```shell
sudo rpm -Uvh svci-x.y.z-n.noarch.rpm
```
<details closed>
<summary><B>Upgrading SVCi - Non-Containerized</B></summary>

On DEB based systems (Debian, Ubuntu and derivatives), download the latest *svci_x.y.z-n_all.deb* file and upgrade:
```shell
sudo dpkg -i svci_x.y.z-n_all.deb
```
-------
### Upgrading SVCi Non-Containerized

Restart the SVCi service on *systemd* based Linux systems:
On RPM based systems (RedHat, Suse, CentOS), download the latest *svci-x.y.z-n.noarch.rpm* file and upgrade:
```shell
sudo rpm -Uvh svci-x.y.z-n.noarch.rpm
```

```shell
systemctl restart svci
journalctl -f -u svci # to check log output
```
On DEB based systems (Debian, Ubuntu and derivatives), download the latest *svci_x.y.z-n_all.deb* file and upgrade:
```shell
sudo dpkg -i svci_x.y.z-n_all.deb
```

### AIX Notes
Restart the SVCi service on *systemd* based Linux systems:

To install (or upgrade) on AIX, you need to pass the *--ignoreos* flag to the *rpm* command:
```shell
systemctl restart svci
journalctl -f -u svci # to check log output
```

```shell
rpm -Uvh --ignoreos svci-x.y.z-n.noarch.rpm
```
### AIX Notes

To install (or upgrade) on AIX, you need to pass the *--ignoreos* flag to the *rpm* command:

## Screenshots
```shell
rpm -Uvh --ignoreos svci-x.y.z-n.noarch.rpm
```


</details>

Screenshots of the provided Grafana dashboard can be found in the [doc/screenshots/](doc/screenshots) folder.

-----

<details closed>
<summary><B>Development and Local Testing</B></summary>

## Development Information

Expand All @@ -125,7 +153,7 @@ Use the gradle build tool, which will download all required dependencies:
```shell
./gradlew clean build
```

-------
### Local Testing

#### InfluxDB v1.x
Expand Down Expand Up @@ -167,3 +195,8 @@ Setup Grafana to connect to the InfluxDB container by defining a new datasource
If you are [connecting](https://docs.influxdata.com/influxdb/v2.7/tools/grafana/) to InfluxDB v2.x, then add a custom http header, enter bucket as database and disable authorization.
- Authorization = Token abcdef_random_token_from_nfluxdb==
- Import dashboards from the [doc/dashboards/](doc/dashboards/) folder.


</details>


11 changes: 7 additions & 4 deletions doc/SVCi.drawio
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<mxfile host="65bd71144e" scale="1" border="15">
<diagram id="a7nnNqAxPxhrp0sOKCIs" name="Page-1">
<mxGraphModel dx="773" dy="772" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" background="#ffffff" math="0" shadow="1">
<mxGraphModel dx="439" dy="702" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" background="#ffffff" math="0" shadow="1">
<root>
<mxCell id="0"/>
<mxCell id="1" parent="0"/>
Expand All @@ -22,9 +22,9 @@
<mxCell id="eK-2ea2avwDUNnB4NnFu-7" style="edgeStyle=orthogonalEdgeStyle;rounded=1;orthogonalLoop=1;jettySize=auto;html=1;sketch=1;startArrow=none;startFill=0;endArrow=classic;endFill=1;strokeWidth=1;strokeColor=#4D4D4D;" parent="1" source="eK-2ea2avwDUNnB4NnFu-6" target="eK-2ea2avwDUNnB4NnFu-2" edge="1">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="3" value="&lt;font style=&quot;background-color: rgb(255, 255, 255);&quot; color=&quot;#000000&quot;&gt;port 7443&lt;/font&gt;" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" parent="eK-2ea2avwDUNnB4NnFu-7" vertex="1" connectable="0">
<mxCell id="3" value="&lt;span style=&quot;color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);&quot;&gt;port https - 7443&lt;/span&gt;" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" parent="eK-2ea2avwDUNnB4NnFu-7" vertex="1" connectable="0">
<mxGeometry x="-0.4631" y="-3" relative="1" as="geometry">
<mxPoint x="-70" y="44" as="offset"/>
<mxPoint x="-86" y="44" as="offset"/>
</mxGeometry>
</mxCell>
<mxCell id="eK-2ea2avwDUNnB4NnFu-6" value="&lt;div&gt;&lt;font color=&quot;#000000&quot;&gt;IBM&lt;br&gt;&lt;/font&gt;&lt;/div&gt;&lt;div&gt;&lt;font color=&quot;#000000&quot;&gt;Storage Flashsystem&lt;/font&gt;&lt;/div&gt;" style="rounded=1;whiteSpace=wrap;html=1;fillColor=none;strokeColor=#6c8ebf;sketch=1;strokeWidth=2;" parent="1" vertex="1">
Expand All @@ -36,7 +36,7 @@
<mxCell id="eK-2ea2avwDUNnB4NnFu-8" value="&lt;div&gt;&lt;font color=&quot;#000000&quot;&gt;IBM&lt;/font&gt;&lt;/div&gt;&lt;div&gt;&lt;font color=&quot;#000000&quot;&gt;Storage Virtualize&lt;/font&gt;&lt;/div&gt;" style="rounded=1;whiteSpace=wrap;html=1;strokeColor=#6c8ebf;dashed=1;fillColor=none;sketch=1;strokeWidth=2;" parent="1" vertex="1">
<mxGeometry x="40" y="145" width="150" height="70" as="geometry"/>
</mxCell>
<mxCell id="il_gvp9eyNAKFmqyvHfq-2" value="&lt;font color=&quot;#000000&quot;&gt;Runs on AIX &amp;amp; Linux&lt;/font&gt;" style="rounded=1;whiteSpace=wrap;html=1;dashed=1;fillColor=none;gradientColor=none;sketch=1;strokeWidth=2;strokeColor=#CCCCFF;" parent="1" vertex="1">
<mxCell id="il_gvp9eyNAKFmqyvHfq-2" value="&lt;font color=&quot;#000000&quot;&gt;Runs on AIX &amp;amp; Linux &amp;amp;&lt;br&gt;Containers&lt;/font&gt;" style="rounded=1;whiteSpace=wrap;html=1;dashed=1;fillColor=none;gradientColor=none;sketch=1;strokeWidth=2;strokeColor=#CCCCFF;" parent="1" vertex="1">
<mxGeometry x="460" y="168.5" width="230" height="40" as="geometry"/>
</mxCell>
<mxCell id="il_gvp9eyNAKFmqyvHfq-3" value="&lt;div&gt;&lt;font color=&quot;#000000&quot;&gt;Java 8+ Runtime on&lt;br&gt;&lt;/font&gt;&lt;/div&gt;&lt;div&gt;&lt;font color=&quot;#000000&quot;&gt;any Java supported platform.&lt;/font&gt;&lt;/div&gt;" style="rounded=1;whiteSpace=wrap;html=1;dashed=1;fillColor=none;gradientColor=none;sketch=1;fontColor=#F0F0F0;strokeColor=#CCCCFF;strokeWidth=2;" parent="1" vertex="1">
Expand Down Expand Up @@ -64,6 +64,9 @@
<mxCell id="HuDfst_B4G-cJXnaIiwU-8" value="" style="shape=umlActor;verticalLabelPosition=bottom;verticalAlign=top;html=1;outlineConnect=0;sketch=1;fillColor=none;strokeColor=#4D4D4D;strokeWidth=2;" parent="1" vertex="1">
<mxGeometry x="810" y="20" width="30" height="60" as="geometry"/>
</mxCell>
<mxCell id="9" value="&lt;div style=&quot;text-align: center;&quot;&gt;&lt;font color=&quot;#000000&quot;&gt;&lt;span style=&quot;font-size: 11px; background-color: rgb(255, 255, 255);&quot;&gt;fallback: SCP/21&lt;/span&gt;&lt;/font&gt;&lt;/div&gt;" style="text;whiteSpace=wrap;html=1;" parent="1" vertex="1">
<mxGeometry x="100" y="108.5" width="90" height="40" as="geometry"/>
</mxCell>
</root>
</mxGraphModel>
</diagram>
Expand Down
Binary file modified doc/SVCi.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
37 changes: 33 additions & 4 deletions doc/TODO.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,40 @@
# TODO

Extended stats

## TODO List

- [] Update docs
- [x] Update install doc for podman,
- [] Update install doc for other os, bump InfluxDB version,
- [] Update main readme with link to doc
- [] Update readme with influxDB info.
- [] Update information about how the data is gather, se below.

- [] Github Action
- [] Check out dockerfile with Chainguard java image for CVE check.
- [] Depandabot with github action
- [] release with with container image

- [] Container
- [] Use best practice for dockerfiles, like run with user for non root.
- [] K8 Deployment?

- [] Grafana.
- [] Updated Grafana dashboard with new stats from XML

- [] SVCi/Virtualize Metric
- [] Logging level



## Details / Drafts


1: XML stats pull and parse. There are dozens of KPIs in there, but they require maths to decode each metric as they are pure counters
2: Real Time Perf stats (using lssystemstats and lsnodestats/lsportstats) - in 8.5.0 you can only have 5 minutes of status history there,
in 8.6.2 this is extended to provide up to 60 data points with roll up for up to 60 days of history).

Extended stats
```shell
svctask stopstats
svctask startstats -interval 5
Expand All @@ -13,7 +45,4 @@ lsdumps -prefix /dumps/iostats
The files generated are written to the /dumps/iostats directory.

https://www.ibm.com/docs/en/flashsystem-5x00/8.4.x?topic=commands-startstats


https://www.ibm.com/support/pages/overview-svc-v510-performance-statistics

121 changes: 121 additions & 0 deletions doc/readme-podman.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Running the SVCi as Container Podman POD stack

Start SVCI, InfluxDB and Grafana as container with podman pod:


## Create a pod with network between the containers and export ports for grafana and influxdb.
> :bulb: Traffic out from container is normally allowed.
```shell
podman pod create -n virtualize_metric_stack -p 8086:8086 -p 3000:3000
podman run --name influxdb -d -p 8086:8086 --pod virtualize_metric_stack influxdb:latest
podman run --name grafana -d --pod virtualize_metric_stack grafana/grafana
```

## Starting the SVCi - Virtualize Metric Container.
Create a folder that contains the svi.toml
Crate the svci.toml file with with your credentials/token for influxdb and flashsystem, se example below.

```shell
# Run interactive to check.
podman run -i --name virtualize_exporter --pod virtualize_metric_stack --volume ${PWD}/data:/opt/app/config/ ghcr.io/mnellemann/svci:main
```
See that you get output `"[main] InfluxClient"`

## Run container in background
```shell
# Run in background
podman run --name virtualize_exporter -d --pod virtualize_metric_stack --volume ${PWD}/data:/opt/app/config/ ghcr.io/mnellemann/svci:main
```


<details closed>
<summary>:bulb: Run container with options</summary>

podman run --name virtualize_exporter -pod virtualize_metric_stack --volume ${PWD}/data:/opt/app/config/ ghcr.io/mnellemann/svci:main java -jar /opt/app/svci.jar/svci-latest.jar -c /opt/app/config/svci.toml -d

</details>



## Check status of the Virtualize_Metric POD:

Check that all containers is running with `podman ps --pods`

```shell
podman ps --pod

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES POD ID PODNAME
2c78533009c1 localhost/podman-pause:5.0.0-dev-8a643c243-1710720000 2 months ago Up About a minute 0.0.0.0:3000->3000/tcp, 0.0.0.0:8086->8086/tcp 19b5b49164c9-infra 19b5b49164c9 virtualize_metric_stack
2e84bf1ee381 docker.io/library/influxdb:latest influxd 2 months ago Up About a minute 0.0.0.0:3000->3000/tcp, 0.0.0.0:8086->8086/tcp influxdb 19b5b49164c9 virtualize_metric_stack
cd8844645b21 docker.io/grafana/grafana:latest 2 months ago Up About a minute 0.0.0.0:3000->3000/tcp, 0.0.0.0:8086->8086/tcp grafana 19b5b49164c9 virtualize_metric_stack
06cdd8cbb9b8 ghcr.io/mnellemann/svci:main java -jar /opt/ap... 28 hours ago Up About a minute 0.0.0.0:3000->3000/tcp, 0.0.0.0:8086->8086/tcp virtualize_exporter 19b5b49164c9 virtualize_metric_stack
```

To get logs from the virtualize_exporter you can use podman logs
```shell
podman logs -f virtualize_exporter
```

To log into the container use: exec -it with /bin/sh
```shell
podman exec -it virtualize_exporter /bin/sh
```

## Example Create the config file for Virtualize_Metric Container.

We are providing a config file to supply the SVCi/Virtualize Metric with an IP and a user. This will enable it to connect to both the Flashsystem/SVC and InfluxDB.



Create a Main folder to store the config file.
in the example we are using mainfolder virtualize-metric and data.
place the config file, svci.toml inside data folder.

```shell
% pwd
-/virtualize-metric
% ls
data
% ls
svci.toml
```
> :bulb: You could also created a container volume and add the config file.


## Example Config Virtualize_Metric Container

Example for influxDB and one storage system.

> :bulb: To monitor more then one storage system just duplicated the [svc.xx] part.
```shell
# SVCi Configuration

# InfluxDB to save metrics
[influx]
url = "http://localhost:8086"
# for InfluxDB V1 use Username and Password
#username = "admin"
#password = "adminadmin24"
bucket = "svci"
# for InfluxDB V2 use token
token = "hiD739k61IhU0Z2zrqQYMR6TCk1Tj3yhgmXZt9-dlNyzxxYyyKC1XXqc1InLYWnQZyaK6tIfj7ATT_feREPMGA=="

## Notice: use hostname without Http and port, default to 7443 and 22 for scp
[svc.fs5200]
hostname = "10.10.10.50"
username = "monitoruser"
password = "SuperDuperPassword!"
refresh = 30
trust = true # Ignore SSL cert. errors

###
### Define one or more SVC's to query for metrics
### Each entry must be named [svc.<something-unique>]
###


```

Loading

0 comments on commit 11772cd

Please sign in to comment.