Skip to content

Wicked Cool Usage

Jump to bottom Edit New page
David Celis edited this page Jan 14, 2015 · 83 revisions

Got any wicked cool usage of spark up your sleeve? Drop your code snippets here.

Display precipitation data for the next hour from forecast.io

This requires an API Key for forecast.io, as well as jq and whereami (brew install jq whereami).

curl -Ss https://api.forecast.io/forecast/$FORECAST_API_KEY/$(whereami) | jq ".minutely.data[].precipIntensity * 1000" | spark
▅▇█▆▄▃▃▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Display stock data for S&P 500 (@arawde)

You can replace ^GPSC with other symbols (GOOG, etc) and it should work the same. -A 31 gets one month after given date. Dividing field 5 by varying numbers scales the graph to more readable form.

curl http://www.optiontradingtips.com/resources/historical-data/sp500/%5EGSPC.TXT --silent |
    grep "20100331" -A  31 |
    awk -F' ' '{print $5/20}' |
    spark

Record how much time you spend on daily practice (guitar, etc) (@dbalatero)

Add this script to your $PATH: https://gist.github.com/dbalatero/6443380

practice record 120   # 2 hours, today
practice | spark   # dump out graph

Number of commits in a repo, by author (@holman)

git shortlog -s |
    cut -f1 |
    spark

Number of commits in a repo, by author, over time plus consistent scaling (@kablamo)

Usually you cannot compare spark graphs because the scale not consistent between any two graphs. git-spark solves this problem with its --scale option.

    git spark --days 14 Stegosaurus
    Commits by Stegosaurus over the last 14 days
    total: 95   avg: 7   max: 23
    10 15 6 23 5 0 0 1 15 0 17 3 0 0
    ▄▅▂█▂▁▁▁▅▁▆▁▁▁

Total run time of processes (@daveycrockett)

ps -e | 
    tail -n +2 | 
    cut -c 16-23 | 
    sed -e "s/$/))))/" -e "s/:/ + (60 * (/" -e "s/:/ + (60 *(/" | 
    bc | 
    spark

Letter frequencies in a text file (@daveycrockett)

cat <file> | 
    awk -vFS="" '{for(i=1;i<=NF;i++){ if($i~/[a-zA-Z]/) { w[tolower($i)]++} } }END{for(i in w) print i,w[i]}' | 
    sort |
    cut -c 3- |
    spark

Users' login time since last reboot (@ghedamat)

  # get users for a specific group
  # needs to be improved 
  users=$(getent group uquota | cut -d ':' -f 4 | tr ',' '\n')
  gr=""
  for i in $users
    do
    gr="$gr$(last | sort | grep $i | cut -c 67-71 | tr ":" " " | awk 'BEGIN {sum=0;  } {sum += $1*60+$2; } END { print sum}'),"
  done
  spark $gr

Number of HTTP requests per day (@Eyjafjallajokull)

cat /var/log/nginx/access.log | 
    cut -d\  -f 4 | 
    cut -d/ -f 1 | 
    uniq -c | 
    awk '{print $1}'| 
    spark

Histogram of commits throughout the day by author (@vrish88)

git log --pretty=format:'%an: %at' --author="Bob" |
    awk '{system("date -r "$NF" '+%H'")}' |
    sort |
    uniq -c |
    ruby -e 'puts STDIN.readlines.inject(Hash[Array("00".."23").map{|x| [x,0]}]) {|h, x| h.merge(Hash[*x.split(" ").reverse])}.sort.map(&:last)' |
    spark

Visualize your hg commit history (@sunng87)

I wrote a hg extension to aggregate commits by week and generate summary for spark.

hg summary | spark

Silly example of random data.

seq 0 100 | sort -R | spark

Print a git participation graph of last 52 weeks like in github (@shurizzle)

#!/usr/bin/env ruby

require 'ruby-let'

class Time
  def week
    strftime('%U').to_i
  end
end

puts IO.popen('spark', 'w+') {|f|
  f.write Time.now.let {|now, min=(now - 31536000)|
    `git log --pretty=format:'%an: %at'`.split(/\r?\n/).map {|line|
      Time.at(line.split(' ').last.to_i).let {|c|
        ((c.year == min.year && c.week >= min.week) || (c.year == now.year && c.week <= now.week)) ? c : nil
      }
    }.compact.group_by {|x| x.strftime('%G%U') }.inject({}) {|res, (k, v)|
      res[k] = v.size
      res
    }.let {|act|
      (0...52).map {|i|
        act[(min + 604800 * i).strftime('%G%U')] || 0
      }.join(',')
    }
  }
  f.close_write
  f.read
}

Beijing Air Quality Index, PM2.5 (@freewizard, updated by @djbender)

Note: No longer working because the twitter v1 API is no longer active.

curl -s https://api.twitter.com/1/statuses/user_timeline.rss?screen_name=beijingair | 
    grep /description | 
    perl -nle "print \$1 if /PM2.5;[^;]+; (\d+)/" | spark

One more git related: commits for last 2 weeks

for day in $(seq 14 -1 0); do
    git log --before="${day} days" --after="$[${day}+1] days" --format=oneline |
    wc -l
done | spark

(Based on above) Git commits over the last 8 hours for a given author (~ today's activity) (@trisweb)

for hour in $(seq 8 -1 0); do 
    git log --author "Author Name" --before="${hour} hours" --after="$[${hour}+1] hours" --format=oneline | 
    wc -l;
done | spark

###Visualize filesize inside a directory(@lemen)

    du -BM * | 
    cut -dM -f1 | 
    spark

Animation and colors with Lolcat (@secondplanet)

spark 1 18 9 4 10 | lolcat -p 0.5 # Colored graph
spark 1 18 9 4 10 | lolcat -a # Animated rainbow graph

###Visualize users created by week on a rails project Caleb Thompson

    bundle exec rails r "User.all.group_by{|u| u.created_at.strftime('%W')}.sort.each{|w,u| puts u.count}" |
    spark

###WiFi link quality (@cryptix)

   if [ $(ifconfig wlan0 | grep UP | wc -l) -eq 1 ]
   then
     _linkQual="`iwconfig wlan0 | grep Quality | cut -d'=' -f2 | cut -d' ' -f1 | cut -d'/' -f1`"
 
     if [ $_linkQual -gt 52 ] # >75% link qual
     then
       _linkSparked=$(spark 1 2 3 4)
     elif [ $_linkQual -gt 35 ] # >50% link qual
     then
       _linkSparked=$(spark 1 2 3 0)
     elif [ $_linkQual -gt 17 ] # 25% link qual
     then
       _linkSparked=$(spark 1 2 0 0)
     elif [ $_linkQual -gt 7 ] # 25% link qual
     then
       _linkSparked=$(spark 1 0 0 0)
     else # < 25%
       _linkSparked=$(spark 0 0 0 0)
     fi

     echo $_linkSparked
   fi

Load average (@tsujigiri)

echo "$(cat /proc/loadavg | cut -d ' ' -f 1-3) $(egrep -c '^processor' /proc/cpuinfo)00 0" | sed 's/\(0\.\|0\.0\|\.\)//g' | spark | tail -n 1 | cut -b 1-9

Load history from atop 

atop -P CPL -b 16:00 -e 18:00 -r /var/log/atop/atop_20130215 | 
   grep -v SEP | 
   awk '{print $8}' | 
   spark

Memory usage (@tsujigiri)

total=$(grep 'MemTotal' /proc/meminfo | egrep -o '[0-9]+')
not_apps=0
for mem in $(egrep '(MemFree|Buffers|Cached|Slab|PageTables|SwapCached)' /proc/meminfo | egrep -o '[0-9]+'); do
  not_apps=$((not_apps+mem))
done
spark $((total-not_apps)) $total 0 | tail -n 1 | cut -b 1-3

###Current SVN status in your prompt (Evan Powell) svnstatgraph.sh:

if [ -d .svn ]; then
    GRAPH=`svn stat | awk '{ split($0, a, " ")  arr[a[1]]++ }END{ print arr["M"] ? arr["M"] : "0", arr["A"] ? arr["A"] : "0", arr["?"] ? arr["?"] : "0", arr["D"] ? arr["D"] : "0", arr["!"] ? arr["!"] : "0" }' | spark`
    # More descriptive prompt:
    #echo "[MA?D!|$GRAPH]"
    echo "[$GRAPH]"
fi

~/.bashrc:

PS1 = '<your_prompt_here>`svnstatgraph.sh`\$'

Visualize bubble sort (@onesuper)

#!/bin/bash

array=(4 3 2 5 1)
arrayLen=${#array[@]}

for ((j=0; j<$arrayLen-1; j++)); do
    for ((i=0; i<$arrayLen-$j-1; i++)); do
        if [ "${array[$i]}" -gt "${array[$i+1]}" ]; then
            temp=${array[$i]}
            array[$i]=${array[$i+1]}
            array[$i+1]=$temp
        fi
        spark ${array[@]}
    done
    echo '---------------------'
done

Visualize ping times (@jnovinger)

ping -c 10 google.com | tee >(grep "bytes from" | cut -d " " -f 8 | cut -d "=" -f 2 | spark)

Or perhaps ping -c 10 google.com | tee >(grep -oP 'time=\K\S*' | spark)

Show stats of commits per day by given author (@kstep)

git log --author="Author Name" --format=format:%ad --date=short | uniq -c | awk '{print $1}' | spark

Disk file usage (@gfelisberto)

df -P -x devtmpfs -x tmpfs | grep dev | awk '{print $5}' | sed s'/.$//' | spark

Hourly wind speed data from an NOAA buoy, by author (@efg34)

curl -s http://www.ndbc.noaa.gov/station_page.php?station=wpow1 | grep "<tr bgcolor=\"#f" | awk -F\<td\> '{print $6}' | cut -c -5 | spark

Weather via http://openweathermap.org (@leoboiko)

#!/bin/bash
# dumb json "parsing", could be better with jshon or similar
city="São Paulo,BR" # city="$1"
cityid=$(wget -q -O - "http://openweathermap.org/data/2.1/find/name?q=$city" \
  | python -mjson.tool \
  | sed -n -e 's/.*\<id.: *\([0-9]*\).*/\1/p' \
  | head -n 1)


wget -q -O - "http://openweathermap.org/data/2.1/history/city/?id=$cityid&cnt=80" \
  | python -mjson.tool \
  | sed -n -e "s/.*\<temp.: *\\(([0-9.]*\)).*/\1/p" \
  | spark

Zabbix API history graph results (@christianchristensen)

(more info: https://gist.github.com/christianchristensen/7406950)

curl -s -H 'Content-type: application/json' https://www.zabbix.org/zabbix/api_jsonrpc.php -d '{ "jsonrpc": "2.0", "method": "history.get", "params":{ "output":"extend", "history":0, "time_from":"1384134836", "time_till":"1384138446", "itemids":["16"] }, "id": 123,"auth":"ac905749516dd173668f3dbf7b0cac0a" }' \
| grep -o "\"value\":\"[0-9\.]*\"," \
| awk -F\" '{print $4*100}' \
| spark
▃▄▃▂▃▂▁▁▃▃▂▂▂▁▂▁▄▂▄▄▃▁▂▂▂▂█▇▅▃▂▂▃▂▃▂▁▁▁▅▅▆▄▂▁▁▂▄▄▂▄▂▁▁▃▁▂▂▁▁

a moving wave through the terminal (@lemen)

function sine_wave() {
	
	i=0
	while true
	do 
		SIN=$(python -c "from math import *;print map( lambda x: ceil(6*sin((x+$i)*pi/5)), range($(tput cols)) )" | tr -d '[]' | spark)
		echo -ne $SIN\\r 
		let i=i+1
		sleep 0.05
	done
	echo
}

UptimeRobot response time (@dlc)

$ curl -sS "http://api.uptimerobot.com/getMonitors?apiKey=${UPTIMEROBOT_TOKEN}&format=xml&responseTimes=1&monitors=${MON_ID}" | \
  xmllint --xpath '//responsetime/@value' - | \
  tr -c '0-9 ' "\n" | \
  grep -v "^$" | \
  head -n $COLUMNS | \
  spark 
▃▇▁▂▄▃▂▂▅▁▂█▃▂▃▃▃▁▂▂▃▂▅▁▁▁▃▂▁▃▂▁▄▁▁▃▂▆▁▃▃▃▁▁▃▁▁▂▄▅▅▄▂▂▃▁▄▂▃▁

Bitcoin value over the last 24 hours (@monkeyinmysoup)

Python script in this Gist.

Keen IO open data set visualization (@dzello)

# Bitcoin value over the last 30 minutes
$ gem install keen-cli
$ keen average \
  --project 53f3a8687d8cb95095000001 \
  --collection bitcoin-prices \
  --target-property "USD.averages.last" \
  --timeframe last_30_minutes \
  --interval minutely \
  --read-key 99a06e48fd7fb1279bc40995160eb0b61a9e0efaab8b651b029f0d895f77c0a804ba089282eff28bf8ad07f337422441d0542b7feaac9fea1e92fc153ee7efc51afad3276bda8d7754a338b349d540bfb402cba0dfdc82498c217054efd8abd0f47a0c0bc963bbdf0dc938c91b17d9f2 \
  --spark | spark
▁▂▂▂▂▂▂▂▁▁▁▂▂▂▂▄▄▂▂▄▄▂▂▄▆█████
Add a custom footer
Add a custom sidebar
Clone this wiki locally