Skip to content

Commit

Permalink
update golangci config and cleanup repo
Browse files Browse the repository at this point in the history
  • Loading branch information
mangalaman93 committed Aug 31, 2023
1 parent bdcf5e9 commit e95822a
Show file tree
Hide file tree
Showing 38 changed files with 168 additions and 177 deletions.
17 changes: 0 additions & 17 deletions .deepsource.toml

This file was deleted.

2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# CODEOWNERS info: https://help.github.com/en/articles/about-code-owners
# Owners are automatically requested for review for PRs that changes code
# that they own.
* @akon-dey @nosql22 @joshua-goldstein @skrdgraph
* @joshua-goldstein @mangalaman93 @harshil-goel
29 changes: 0 additions & 29 deletions .github/PULL_REQUEST_TEMPLATE.md

This file was deleted.

4 changes: 2 additions & 2 deletions .github/workflows/ci-aqua-security-trivy-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:
- synchronize
- ready_for_review
branches:
- main
- main
schedule:
- cron: "0 * * * *"
jobs:
Expand All @@ -24,7 +24,7 @@ jobs:
uses: aquasecurity/trivy-action@master
with:
scan-type: 'fs'
scan-ref: '.'
scan-ref: '.'
format: 'sarif'
output: 'trivy-results.sarif'
- name: Upload Trivy scan results to GitHub Security tab
Expand Down
15 changes: 12 additions & 3 deletions .github/workflows/ci-ristretto-lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,26 @@ on:
- cron: "0 * * * *"
jobs:
go-lint:
if: github.event.pull_request.draft == false
name: lint
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3
- name: Get Go Version
run: |
#!/bin/bash
GOVERSION=$({ [ -f .go-version ] && cat .go-version; })
echo "GOVERSION=$GOVERSION" >> $GITHUB_ENV
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: ${{ env.GOVERSION }}
- name: golang-lint
env:
# prevent OOM
GOGC: 10
uses: golangci/golangci-lint-action@v3
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.48
version: latest
only-new-issues: true
args: --timeout=10m
4 changes: 2 additions & 2 deletions .github/workflows/ci-ristretto-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@ name: ci-ristretto-tests
on:
push:
branches:
- main
- main
pull_request_target:
branches:
- main
- main
schedule:
- cron: "30 * * * *"
jobs:
Expand Down
2 changes: 1 addition & 1 deletion .go-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.17.11
1.19
30 changes: 18 additions & 12 deletions .golangci.yml
Original file line number Diff line number Diff line change
@@ -1,23 +1,29 @@
run:
tests: false
skip-dirs:
- contrib
- sim
skip-files:

linters-settings:
lll:
line-length: 120
staticcheck:
checks:
- all
- '-SA1019' # it is okay to use math/rand at times.
gosec:
excludes:
- G404 # it is okay to use math/rand at times.

linters:
disable-all: true
enable:
#- errcheck
#- ineffassign
- gas
#- gofmt
#- golint
#- gosimple
#- govet
- errcheck
- gofmt
- goimports
- gosec
- gosimple
- govet
- ineffassign
- lll
#- varcheck
#- unused
- staticcheck
- unconvert
- unused
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ improve performance and reduce memory requirements.
### Fixed

- Fix the way metrics are handled for deletions. ([#111][])
- Support nil `*Cache` values in `Clear` and `Close`. ([#119][])
- Support nil `*Cache` values in `Clear` and `Close`. ([#119][])
- Delete item immediately. ([#113][])
- Remove key from policy after TTL eviction. ([#130][])

Expand Down
68 changes: 33 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@

Ristretto is a fast, concurrent cache library built with a focus on performance and correctness.

The motivation to build Ristretto comes from the need for a contention-free
cache in [Dgraph][].
The motivation to build Ristretto comes from the need for a contention-free cache in [Dgraph][].

[Dgraph]: https://github.com/dgraph-io/dgraph

Expand All @@ -19,7 +18,7 @@ cache in [Dgraph][].
* **Admission: TinyLFU** - extra performance with little memory overhead (12 bits per counter).
* **Fast Throughput** - we use a variety of techniques for managing contention and the result is excellent throughput.
* **Cost-Based Eviction** - any large new item deemed valuable can evict multiple smaller items (cost could be anything).
* **Fully Concurrent** - you can use as many goroutines as you want with little throughput degradation.
* **Fully Concurrent** - you can use as many goroutines as you want with little throughput degradation.
* **Metrics** - optional performance metrics for throughput, hit ratios, and other stats.
* **Simple API** - just figure out your ideal `Config` values and you're off and running.

Expand All @@ -29,28 +28,27 @@ Ristretto is production-ready. See [Projects using Ristretto](#projects-using-ri

## Table of Contents

* [Usage](#Usage)
* [Example](#Example)
* [Config](#Config)
* [NumCounters](#Config)
* [MaxCost](#Config)
* [BufferItems](#Config)
* [Metrics](#Config)
* [OnEvict](#Config)
* [KeyToHash](#Config)
* [Cost](#Config)
* [Benchmarks](#Benchmarks)
* [Hit Ratios](#Hit-Ratios)
* [Search](#Search)
* [Database](#Database)
* [Looping](#Looping)
* [CODASYL](#CODASYL)
* [Throughput](#Throughput)
* [Mixed](#Mixed)
* [Read](#Read)
* [Write](#Write)
* [Projects using Ristretto](#projects-using-ristretto)
* [FAQ](#FAQ)
- [Ristretto](#ristretto)
- [Features](#features)
- [Status](#status)
- [Table of Contents](#table-of-contents)
- [Usage](#usage)
- [Example](#example)
- [Config](#config)
- [Benchmarks](#benchmarks)
- [Hit Ratios](#hit-ratios)
- [Search](#search)
- [Database](#database)
- [Looping](#looping)
- [CODASYL](#codasyl)
- [Throughput](#throughput)
- [Mixed](#mixed)
- [Read](#read)
- [Write](#write)
- [Projects Using Ristretto](#projects-using-ristretto)
- [FAQ](#faq)
- [How are you achieving this performance? What shortcuts are you taking?](#how-are-you-achieving-this-performance-what-shortcuts-are-you-taking)
- [Is Ristretto distributed?](#is-ristretto-distributed)

## Usage

Expand Down Expand Up @@ -95,31 +93,31 @@ func main() {

### Config

The `Config` struct is passed to `NewCache` when creating Ristretto instances (see the example above).
The `Config` struct is passed to `NewCache` when creating Ristretto instances (see the example above).

**NumCounters** `int64`

NumCounters is the number of 4-bit access counters to keep for admission and eviction. We've seen good performance in setting this to 10x the number of items you expect to keep in the cache when full.
NumCounters is the number of 4-bit access counters to keep for admission and eviction. We've seen good performance in setting this to 10x the number of items you expect to keep in the cache when full.

For example, if you expect each item to have a cost of 1 and MaxCost is 100, set NumCounters to 1,000. Or, if you use variable cost values but expect the cache to hold around 10,000 items when full, set NumCounters to 100,000. The important thing is the *number of unique items* in the full cache, not necessarily the MaxCost value.
For example, if you expect each item to have a cost of 1 and MaxCost is 100, set NumCounters to 1,000. Or, if you use variable cost values but expect the cache to hold around 10,000 items when full, set NumCounters to 100,000. The important thing is the *number of unique items* in the full cache, not necessarily the MaxCost value.

**MaxCost** `int64`

MaxCost is how eviction decisions are made. For example, if MaxCost is 100 and a new item with a cost of 1 increases total cache cost to 101, 1 item will be evicted.
MaxCost is how eviction decisions are made. For example, if MaxCost is 100 and a new item with a cost of 1 increases total cache cost to 101, 1 item will be evicted.

MaxCost can also be used to denote the max size in bytes. For example, if MaxCost is 1,000,000 (1MB) and the cache is full with 1,000 1KB items, a new item (that's accepted) would cause 5 1KB items to be evicted.
MaxCost can also be used to denote the max size in bytes. For example, if MaxCost is 1,000,000 (1MB) and the cache is full with 1,000 1KB items, a new item (that's accepted) would cause 5 1KB items to be evicted.

MaxCost could be anything as long as it matches how you're using the cost values when calling Set.
MaxCost could be anything as long as it matches how you're using the cost values when calling Set.

**BufferItems** `int64`

BufferItems is the size of the Get buffers. The best value we've found for this is 64.
BufferItems is the size of the Get buffers. The best value we've found for this is 64.

If for some reason you see Get performance decreasing with lots of contention (you shouldn't), try increasing this value in increments of 64. This is a fine-tuning mechanism and you probably won't have to touch this.

**Metrics** `bool`

Metrics is true when you want real-time logging of a variety of stats. The reason this is a Config flag is because there's a 10% throughput performance overhead.
Metrics is true when you want real-time logging of a variety of stats. The reason this is a Config flag is because there's a 10% throughput performance overhead.

**OnEvict** `func(hashes [2]uint64, value interface{}, cost int64)`

Expand Down Expand Up @@ -224,8 +222,8 @@ Below is a list of known projects that use Ristretto:

We go into detail in the [Ristretto blog post](https://blog.dgraph.io/post/introducing-ristretto-high-perf-go-cache/), but in short: our throughput performance can be attributed to a mix of batching and eventual consistency. Our hit ratio performance is mostly due to an excellent [admission policy](https://arxiv.org/abs/1512.00727) and SampledLFU eviction policy.

As for "shortcuts," the only thing Ristretto does that could be construed as one is dropping some Set calls. That means a Set call for a new item (updates are guaranteed) isn't guaranteed to make it into the cache. The new item could be dropped at two points: when passing through the Set buffer or when passing through the admission policy. However, this doesn't affect hit ratios much at all as we expect the most popular items to be Set multiple times and eventually make it in the cache.
As for "shortcuts," the only thing Ristretto does that could be construed as one is dropping some Set calls. That means a Set call for a new item (updates are guaranteed) isn't guaranteed to make it into the cache. The new item could be dropped at two points: when passing through the Set buffer or when passing through the admission policy. However, this doesn't affect hit ratios much at all as we expect the most popular items to be Set multiple times and eventually make it in the cache.

### Is Ristretto distributed?

No, it's just like any other Go library that you can import into your project and use in a single process.
No, it's just like any other Go library that you can import into your project and use in a single process.
2 changes: 1 addition & 1 deletion cache_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -824,7 +824,7 @@ func TestDropUpdates(t *testing.T) {
var err error
handler := func(_ interface{}, value interface{}) {
v := value.(string)
lastEvictedSet, err = strconv.ParseInt(string(v), 10, 32)
lastEvictedSet, err = strconv.ParseInt(v, 10, 32)
require.NoError(t, err)

_, ok := droppedMap[int(lastEvictedSet)]
Expand Down
2 changes: 0 additions & 2 deletions contrib/demo/node.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ package main
import (
"fmt"
"runtime"
"unsafe"

"github.com/dgraph-io/ristretto/z"
"github.com/dustin/go-humanize"
Expand All @@ -14,7 +13,6 @@ type node struct {
next *node
}

var nodeSz = int(unsafe.Sizeof(node{}))
var alloc *z.Allocator

func printNode(n *node) {
Expand Down
3 changes: 2 additions & 1 deletion contrib/demo/node_allocator.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
//go:build jemalloc && allocator
// +build jemalloc,allocator

package main
Expand All @@ -10,7 +11,7 @@ import (

// Defined in node.go.
func init() {
alloc = z.NewAllocator(10 << 20, "demo")
alloc = z.NewAllocator(10<<20, "demo")
}

func newNode(val int) *node {
Expand Down
1 change: 1 addition & 0 deletions contrib/demo/node_golang.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
//go:build !jemalloc
// +build !jemalloc

package main
Expand Down
1 change: 1 addition & 0 deletions contrib/demo/node_jemalloc.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
//go:build jemalloc && !allocator
// +build jemalloc,!allocator

package main
Expand Down
2 changes: 1 addition & 1 deletion contrib/memtest/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ func main() {
fill = make([]byte, maxMB<<20)
rand.Read(fill)

c := make(chan os.Signal)
c := make(chan os.Signal, 10)
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
go func() {
<-c
Expand Down
1 change: 1 addition & 0 deletions contrib/memtest/nojemalloc.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ func Calloc(size int) []byte {
}
hdr := reflect.SliceHeader{Data: uintptr(ptr), Len: size, Cap: size}
atomic.AddInt64(&numbytes, int64(size))
//nolint:govet
return *(*[]byte)(unsafe.Pointer(&hdr))
}

Expand Down
17 changes: 11 additions & 6 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,13 +1,18 @@
module github.com/dgraph-io/ristretto

go 1.12
go 1.19

require (
github.com/cespare/xxhash/v2 v2.1.1
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2
github.com/dustin/go-humanize v1.0.0
github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13
github.com/dustin/go-humanize v1.0.1
github.com/pkg/errors v0.9.1
github.com/stretchr/testify v1.4.0
golang.org/x/sys v0.0.0-20221010170243-090e33056c14
github.com/stretchr/testify v1.8.4
golang.org/x/sys v0.11.0
)

require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
Loading

0 comments on commit e95822a

Please sign in to comment.