Skip to content

Commit

Permalink
Add examples and commands for filtering and formatting data
Browse files Browse the repository at this point in the history
  • Loading branch information
nmaguiar committed Feb 11, 2024
1 parent d79e357 commit fa172a5
Showing 1 changed file with 77 additions and 14 deletions.
91 changes: 77 additions & 14 deletions src/docs/EXAMPLES.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,85 @@
# OpenAF processor examples

## 🥉 Basic

### OUTPUT JSON: Creating ndjson file from json files

**Command:**
```bash
# Creates a data.ndjson file where each record is formatted from json files in /some/data
find /some/data -name "*.json" -exec oafp {} output=json \; > data.ndjson
```

### FILTER PATH: Docker ps formatting

**Command:**
```bash
oafp cmd="docker ps --format json" input=ndjson ndjsonjoin=true path="[].{id:ID,name:Names,state:State,image:Image,networks:Networks,ports:Ports,Status:Status}" sql="select * order by networks,state,name" output=ctable
```

**Result:**
```
id │ name │ state │ image │ networks │ ports │ Status
────────────┼────────────────────────┼───────┼──────────────────────────────┼──────────────────────┼───────────────────────────────────────────────────────┼──────────
af3adb5b8349│registry │running│registry:2 │bridge,k3d-k3s-default│0.0.0.0:5000->5000/tcp │Up 2 hours
cba6e3807b44│k3d-k3s-default-server-0│running│rancher/k3s:v1.27.4-k3s1 │k3d-k3s-default │ │Up 2 hours
b775ad480764│k3d-k3s-default-serverlb│running│ghcr.io/k3d-io/k3d-proxy:5.6.0│k3d-k3s-default │80/tcp, 0.0.0.0:1080->1080/tcp, 0.0.0.0:45693->6443/tcp│Up 2 hours
[#3 rows]
```

## 🥈 Medium

### Using a LLM to generate a table
### FILTER PATH: Kubectl get pods formatting

**Command:**
```bash
oafp cmd="kubectl get pods -A -o json" path="items[].{ns:metadata.namespace,kind:metadata.ownerReferences[].kind,name:metadata.name,status:status.phase,restarts:sum(status.containerStatuses[].restartCount),node:spec.nodeName,age:timeago(status.startTime)}" sql="select * order by status,name" output=ctable
```

**Result:**
```
ns │ kind │ name │ status │restarts│ node │ age
───────────┼──────────┼──────────────────────────────────────┼─────────┼────────┼────────────────────────┼──────────────
kube-system│ReplicaSet│coredns-77ccd57875-5m44t │Running │0 │k3d-k3s-default-server-0│66 minutes ago
kube-system│ReplicaSet│local-path-provisioner-957fdf8bc-24hmf│Running │0 │k3d-k3s-default-server-0│66 minutes ago
kube-system│ReplicaSet│metrics-server-648b5df564-hzbwb │Running │0 │k3d-k3s-default-server-0│66 minutes ago
kube-system│ReplicaSet│socks-server-d7c8c4d78-r6jc9 │Running │0 │k3d-k3s-default-server-0│66 minutes ago
kube-system│DaemonSet │svclb-socks-server-78b973ca-zvf58 │Running │0 │k3d-k3s-default-server-0│66 minutes ago
kube-system│DaemonSet │svclb-traefik-e1776788-7z2gf │Running │0 │k3d-k3s-default-server-0│66 minutes ago
kube-system│ReplicaSet│traefik-64f55bb67d-g2vps │Running │0 │k3d-k3s-default-server-0│66 minutes ago
kube-system│Job │helm-install-traefik-6j5zx │Succeeded│1 │k3d-k3s-default-server-0│66 minutes ago
kube-system│Job │helm-install-traefik-crd-z59fs │Succeeded│0 │k3d-k3s-default-server-0│66 minutes ago
[#9 rows]
```

### FILTER PATH:

**Command:**
```bash
oafp cmd="kubectl get nodes -o json" path="items[].{node:metadata.name,totalCPU:status.capacity.cpu,allocCPU:status.allocatable.cpu,totalMem:to_bytesAbbr(from_bytesAbbr(status.capacity.memory)),allocMem:to_bytesAbbr(from_bytesAbbr(status.allocatable.memory)),totalStorage:to_bytesAbbr(from_bytesAbbr(status.capacity.\"ephemeral-storage\")),allocStorage:to_bytesAbbr(to_number(status.allocatable.\"ephemeral-storage\")),conditions:join(\`, \`,status.conditions[].reason)}" output=ctable
```

**Result:**
```bash
node │totalCPU│allocCPU│totalMem│allocMem│totalStorage│allocStorage│ conditions
────────────────────────┼────────┼────────┼────────┼────────┼────────────┼────────────┼───────────────────────────────────────────────────────────────────────────────────────────
k3d-k3s-default-server-0│4 │4 │3.85 GB │3.85 GB │77.6 GB │73.8 GB │KubeletHasSufficientMemory, KubeletHasNoDiskPressure, KubeletHasSufficientPID, KubeletReady
[#1 row]
```
### INPUT LLM: Using a LLM to generate a table
Setting up the LLM model and gather the data into a data.json file:
**Command:**
```bash
export OAFP_MODEL="(type: openai, model: gpt-3.5-turbo, key: ..., timeout: 900000)"
echo "list all United Nations secretaries with their corresponding 'name', their mandate 'begin date', their mandate 'end date' and their corresponding secretary 'numeral'" | oafp input=llm output=json > data.json
```
Checking the obtain data:
**Result:**
```yaml
oafp data.json
─ secretaries ╭ [0] ╭ name : Trygve Lie
Expand Down Expand Up @@ -55,6 +122,7 @@ oafp data.json
Checking the data in a table format:
**Result:**
```
oafp data.json path=secretaries output=ctable
name │begin date│ end date │numeral
Expand All @@ -71,17 +139,17 @@ António Guterres │2017-01-01│present │9
[#9 rows]
```
### Using a LLM to transform an input
### TRANSFORM LLM: Using a LLM to transform an input
Using the data gather in ['Using a LLM to generate a table'](#using-a-llm-to-generate-a-table) use a LLM to transform it:
**Command:**
```bash
export OAFP_MODEL="(type: openai, model: gpt-3.5-turbo, key: ..., timeout: 900000)"
oafp data.json llmprompt="convert the numeral number into a roman number" path=secretaries output=ctable
```
Result:

**Result:**
```
name │begin date│ end date │numeral
───────────────────────┼──────────┼──────────┼───────
Expand All @@ -97,15 +165,16 @@ António Guterres │2017-01-01│present │IX
[#9 rows]
```
### Using a private LLM to generate sample data
### INPUT LLM: Using a private LLM to generate sample data
**Command:**
```bash
export OAFP_MODEL="(type: ollama, model: 'mistral:instruct', url: 'https://models.local', timeout: 900000)"
echo "Output a JSON array with 15 cities where each entry has the 'city' name, the estimated population and the corresponding 'country'" | oafp input=llm output=json > data.json
oafp data.json output=ctable sql="select * order by population desc"
````
Result:
**Result:**
```
city │population│ country
───────────┼──────────┼──────────
Expand All @@ -126,15 +195,9 @@ Dhaka │18568373 │Bangladesh
## 🥇 Advanced
### Creating ndjson file from json files
```bash
# Creates a data.ndjson file where each record is formatted from json files in /some/data
find /some/data -name "*.json" -exec oafp {} output=json \; > data.ndjson
```

### Creating an XLSx file with multiple sheets
### OUTPUT XLS: Creating an XLSx file with multiple sheets
**Command:**
```bash
# Processes each json file in /some/data creating and updating the data.xlsx file with a sheet for each file
find /some/data -name "*.json" | xargs -I '{}' /bin/sh -c 'oafp file={} output=xls xlsfile=data.xlsx xlsopen=false xlssheet=$(echo {} | sed "s/.*\/\(.*\)\.json/\1/g" )'
Expand Down

0 comments on commit fa172a5

Please sign in to comment.