Skip to content

Commit

Permalink
Control termshark's cache of pcap files on disk
Browse files Browse the repository at this point in the history
When termshark is run in live capture mode, it saves the pcaps it
creates under $XDG_CACHE_HOME/termshark/pcaps/. Termshark doesn't yet
cap the growth of that directory. Ross reported his cache dir was 40GB!
This change adds a new config option to put a limit on this directory e.g.

[main]
  disk-cache-size-mb = 300

Termshark will check the disk cache once per session, after the first 5s
of inactivity.

Omit this config setting, or set the value to -1, to preserve the
current behavior and let the directory grow. The default is -1 because I
don't want to suddenly delete pcaps that existing users might expect to
remain there based on termshark's current behavior.
  • Loading branch information
gcla committed May 6, 2020
1 parent 958fe26 commit 47d705f
Show file tree
Hide file tree
Showing 3 changed files with 77 additions and 3 deletions.
13 changes: 13 additions & 0 deletions cmd/termshark/termshark.go
Original file line number Diff line number Diff line change
Expand Up @@ -877,12 +877,17 @@ func cmain() int {
inactiveDuration := 30 * time.Second
inactivityTimer := time.NewTimer(inactiveDuration)

checkedPcapCache := false
checkPcapCacheDuration := 5 * time.Second
checkPcapCacheTimer := time.NewTimer(checkPcapCacheDuration)

Loop:
for {
var finChan <-chan time.Time
var opsChan <-chan pcap.RunFn
var tickChan <-chan time.Time
var inactivityChan <-chan time.Time
var checkPcapCacheChan <-chan time.Time
var emptyStructViewChan <-chan time.Time
var emptyHexViewChan <-chan time.Time
var psmlFinChan <-chan struct{}
Expand Down Expand Up @@ -957,6 +962,10 @@ Loop:
inactivityChan = inactivityTimer.C
}

if !checkedPcapCache {
checkPcapCacheChan = checkPcapCacheTimer.C
}

// (User) operations are enabled by default (the test predicate is nil), or if the predicate returns true
// meaning the operation has reached its desired state. Only one operation can be in progress at a time.
if ui.PcapScheduler.IsEnabled() {
Expand Down Expand Up @@ -986,6 +995,9 @@ Loop:

select {

case <-checkPcapCacheChan:
termshark.PrunePcapCache()

case <-inactivityChan:
ui.Fin.Activate()
app.Redraw()
Expand Down Expand Up @@ -1191,6 +1203,7 @@ Loop:
case ev := <-tcellEvents:
app.HandleTCellEvent(ev, gowid.IgnoreUnhandledInput)
inactivityTimer.Reset(inactiveDuration)
checkPcapCacheTimer.Reset(checkPcapCacheDuration)

case ev, ok := <-afterRenderEvents:
// This means app.Quit() has been called, which closes the AfterRenderEvents
Expand Down
3 changes: 0 additions & 3 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@ github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/gcla/deep v1.0.2 h1:qBOx6eepcOSRYnHJ+f2ih4hP4Vca1YnLtXxp73n5KWI=
github.com/gcla/deep v1.0.2/go.mod h1:evE9pbpSGhItmFoBIk8hPOIC/keKTGYhFl6Le1Av+GE=
github.com/gcla/gowid v1.1.1-0.20200129034750-719170c39287 h1:moW/sE8xooyhWU6GTO1XO9eiHESv+y15tarIM5eNDpI=
github.com/gcla/gowid v1.1.1-0.20200129034750-719170c39287/go.mod h1:kwHYNePmuaNa60IAkHfd/OfPeZuoSuz7ww+CeA5q/aQ=
github.com/gcla/gowid v1.1.1-0.20200202174024-45eed270dfd5 h1:e/Jk8oCfaWdXtXHcUC6XbxF26F+3Fqy36+LWfv8NZlI=
github.com/gcla/gowid v1.1.1-0.20200202174024-45eed270dfd5/go.mod h1:kwHYNePmuaNa60IAkHfd/OfPeZuoSuz7ww+CeA5q/aQ=
github.com/gcla/tail v1.0.1-0.20190505190527-650e90873359 h1:3xEhacR7pIJV8daurdBygptxhzTJeYFqJp1V6SDl+pE=
Expand All @@ -31,7 +29,6 @@ github.com/gcla/tcell v1.1.2-0.20200115035344-b90e69b9dbe0 h1:6fMu73gAbCSAYAGFhM
github.com/gcla/tcell v1.1.2-0.20200115035344-b90e69b9dbe0/go.mod h1:vxEiSDZdW3L+Uhjii9c3375IlDmR05bzxY404ZVSMo0=
github.com/gcla/term v0.0.0-20191015020247-31cba2f9f402 h1:d8cpYNgYjXDjvrdnSaFy4+o1D3BpPUKTBIcijrBKv4c=
github.com/gcla/term v0.0.0-20191015020247-31cba2f9f402/go.mod h1:YCPU+G35BFc/575HWMl8oeqq+dTbunufWbWaV0Y2sqY=
github.com/gcla/termshark v1.0.0 h1:3jDyqYHeGIfN2khlAfWmJzoJJTh6Iau2mz81nakLBPk=
github.com/gdamore/encoding v1.0.0 h1:+7OoQ1Bc6eTm5niUzBa0Ctsh6JbMW6Ra+YNuAtDBdko=
github.com/gdamore/encoding v1.0.0/go.mod h1:alR0ol34c49FCSBLjhosxzcPHQbf2trDkoo5dl+VrEg=
github.com/go-test/deep v1.0.1 h1:UQhStjbkDClarlmv0am7OXXO4/GaPdCGiUiMTvi28sg=
Expand Down
64 changes: 64 additions & 0 deletions utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (
"path/filepath"
"regexp"
"runtime"
"sort"
"strconv"
"strings"
"sync"
Expand Down Expand Up @@ -637,6 +638,69 @@ func SaveOffsetToConfig(name string, offsets2 []resizable.Offset) {

//======================================================================

func PrunePcapCache() error {
// This is a new option. Best to err on the side of caution and, if not, present
// assume the cache can grow indefinitely - in case users are now relying on this
// to keep old pcaps around. I don't want to delete any files without the user's
// explicit permission.
var diskCacheSize int64 = int64(ConfInt("main.disk-cache-size-mb", -1))

if diskCacheSize == -1 {
log.Infof("No pcap disk cache size set. Skipping cache pruning.")
return nil
}

// Let user use MB as the most sensible unit of disk size. Convert to
// bytes for comparing to file sizes.
diskCacheSize = diskCacheSize * 1024 * 1024

log.Infof("Pruning termshark's pcap disk cache at %s...", PcapDir())

var totalSize int64
var fileInfos []os.FileInfo
err := filepath.Walk(PcapDir(),
func(path string, info os.FileInfo, err error) error {
if err == nil {
totalSize += info.Size()
fileInfos = append(fileInfos, info)
}
return nil
},
)
if err != nil {
return err
}

sort.Slice(fileInfos, func(i, j int) bool {
return fileInfos[i].ModTime().Before(fileInfos[j].ModTime())
})

filesRemoved := 0
curCacheSize := totalSize
for len(fileInfos) > 0 && curCacheSize > diskCacheSize {
err = os.Remove(filepath.Join(PcapDir(), fileInfos[0].Name()))
if err != nil {
log.Warnf("Could not remove pcap cache file %s while pruning - %v", fileInfos[0].Name(), err)
} else {
curCacheSize = curCacheSize - fileInfos[0].Size()
filesRemoved++
}
fileInfos = fileInfos[1:]
}

if filesRemoved > 0 {
log.Infof("Pruning complete. Removed %d old pcaps. Cache size is now %d MB",
filesRemoved, curCacheSize/(1024*1024))
} else {
log.Infof("Pruning complete. No old pcaps removed. Cache size is %d MB",
curCacheSize/(1024*1024))
}

return nil
}

//======================================================================

var cpuProfileRunning *abool.AtomicBool

func init() {
Expand Down

5 comments on commit 47d705f

@pocc
Copy link
Collaborator

@pocc pocc commented on 47d705f May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgive the ignorance - I'm not seeing where this change is made in this commit?

[main]
  disk-cache-size-mb = 300

@gcla
Copy link
Owner Author

@gcla gcla commented on 47d705f May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

np, it's on line 646

@pocc
Copy link
Collaborator

@pocc pocc commented on 47d705f May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No I meant the default value of 300. I'm assuming that's in another commit with the default config?

@gcla
Copy link
Owner Author

@gcla gcla commented on 47d705f May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh right, sorry - no, I think that was a poor commit message in that case, I meant it as an example. I left the default at -1 out of fear that I would anger someone who has come to depend on files that have collected in that cache directory then discovering that I've deleted all but 1GB of them. I am adding a preferences dialog though that will make this cache option more discoverable.

@pocc
Copy link
Collaborator

@pocc pocc commented on 47d705f May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this problem, why not ask at 1GB (or next time they start it after an update if > 1GB)? Something like "I've noticed that you have 5GB of cached pcaps in $cache_dir. The first couple are
$random_name1
$random_name2
$random_name3

Are you using these pcaps or can I delete them to free up space?"

If yes, then ask if they want cache enabled at all. The way to not anger people is to ask them what they want and then do that.

Please sign in to comment.