Skip to content

Commit

Permalink
Merge branch 'main' into box-detector
Browse files Browse the repository at this point in the history
* main: (127 commits)
  Update SaladCloud description (trufflesecurity#3399)
  fix tests (trufflesecurity#3400)
  [chore] Update custom detector default description (trufflesecurity#3398)
  add description to salad (trufflesecurity#3397)
  Add detector for SaladCloud API Keys (trufflesecurity#3273)
  fix(deps): update module github.com/xanzy/go-gitlab to v0.111.0 (trufflesecurity#3393)
  Add SliceContainsString common util (trufflesecurity#3395)
  fix: pr template link to golangci-lint (trufflesecurity#3392)
  fix(deps): update golang.org/x/exp digest to f66d83c (trufflesecurity#3389)
  Separate detector tests into unit/integration (trufflesecurity#3274)
  Manually upgrade github dep (trufflesecurity#3387)
  Updated Fastly Personal Token Detector (trufflesecurity#3386)
  fix(deps): update module google.golang.org/api to v0.200.0 (trufflesecurity#3391)
  [Fix] Snowflake privatelink Support (trufflesecurity#3286)
  Enhanced the easyinsight detector (trufflesecurity#3384)
  Log skipped files on debug level (trufflesecurity#3383)
  build: update retracted bluemonday ver (trufflesecurity#3369)
  Fix git binary handling and add a smoke test (trufflesecurity#3379)
  fix(deps): update module google.golang.org/protobuf to v1.35.1 (trufflesecurity#3382)
  Added Cisco Meraki API Key detector (trufflesecurity#3367)
  ...

# Conflicts:
#	pkg/engine/defaults.go
#	pkg/pb/detectorspb/detectors.pb.go
#	proto/detectors.proto
  • Loading branch information
abmussani committed Oct 14, 2024
2 parents a4c61fc + cf54b71 commit 9c66a31
Show file tree
Hide file tree
Showing 1,124 changed files with 31,012 additions and 9,529 deletions.
11 changes: 11 additions & 0 deletions .captain/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
test-suites:
detectors:
command: gotestsum --jsonfile tmp/go-test.json --raw-command -- go test -tags=detectors -timeout=15m -json -count=1 -vet=off ./pkg/detectors/...
results:
path: tmp/go-test.json
output:
print-summary: true
## No retries right now
# retries:
# attempts: 3
# command: gotestsum --raw-command --jsonfile tmp/go-test.json -- go test -tags=detectors -timeout=15m -json -count=1 -vet=off {{ package }} -run '{{ run }}'
3 changes: 1 addition & 2 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,4 @@ Explain the purpose of the PR.

### Checklist:
* [ ] Tests passing (`make test-community`)?
* [ ] Lint passing (`make lint` this requires [golangci-lint](https://golangci-lint.run/usage/install/#local-installation))?

* [ ] Lint passing (`make lint` this requires [golangci-lint](https://golangci-lint.run/welcome/install/#local-installation))?
2 changes: 1 addition & 1 deletion .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: "1.22"
go-version: "1.23"
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
Expand Down
43 changes: 12 additions & 31 deletions .github/workflows/detector-tests.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: detector test aggregation
name: Detectors Aggregation

on:
workflow_dispatch:
Expand All @@ -14,35 +14,16 @@ jobs:
contents: "read"
id-token: "write"
steps:
- name: Install Go
uses: actions/setup-go@v5
- uses: actions/setup-go@v5
- uses: actions/checkout@v4
- name: Install gotestsum
uses: jaxxstorm/action-install-gh-release@v1.12.0
with:
go-version: "1.22"
- name: Checkout code
uses: actions/checkout@v4
- id: "auth"
uses: "google-github-actions/auth@v2"
with:
workload_identity_provider: "projects/811013774421/locations/global/workloadIdentityPools/github-pool/providers/github-provider"
service_account: "github-ci-external@trufflehog-testing.iam.gserviceaccount.com"
- name: Test integration
continue-on-error: true
run: make test-integration
- name: Set up gotestsum
run: |
go install gotest.tools/gotestsum@latest
mkdir -p tmp/test-results
- name: Test detectors
repo: gotestyourself/gotestsum
- uses: rwx-research/setup-captain@v1
- name: Test Go
run: |
CGO_ENABLED=1 gotestsum --junitfile tmp/test-results/test.xml --raw-command -- go test -json -tags=detectors -timeout=15m $(go list ./... | grep pkg/detectors)
- name: Upload test results to BuildPulse for flaky test detection
if: ${{ !cancelled() }} # Run this step even when the tests fail. Skip if the workflow is cancelled.
uses: buildpulse/buildpulse-action@main
with:
account: 79229934
repository: 694446374
path: |
tmp/test-results/*.xml
key: ${{ secrets.BUILDPULSE_DETECTORS_ACCESS_KEY_ID }}
secret: ${{ secrets.BUILDPULSE_DETECTORS_SECRET_ACCESS_KEY }}
tags: detectors
export CGO_ENABLED=1
captain run detectors
env:
RWX_ACCESS_TOKEN: ${{ secrets.RWX_ACCESS_TOKEN }}
2 changes: 1 addition & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
steps:
- uses: actions/setup-go@v5
with:
go-version: "1.22"
go-version: "1.23"
- uses: actions/checkout@v4
- name: golangci-lint
uses: golangci/golangci-lint-action@v6
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/performance.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: "1.22"
go-version: "1.23"

- name: Checkout code
uses: actions/checkout@v4
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: "1.22"
go-version: "1.23"
- name: Cosign install
uses: sigstore/cosign-installer@4959ce089c160fddf62f7b42464195ba1a56d382 # v3.6.0
uses: sigstore/cosign-installer@dc72c7d5c4d10cd6bcb8cf6e3fd625a9e5e537da # v3.7.0
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v6
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/secrets.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ on:

jobs:
test:
if: github.repository == 'trufflesecurity/trufflehog'
if: ${{ github.repository == 'trufflesecurity/trufflehog' && !github.event.pull_request.head.repo.fork }}
runs-on: ubuntu-latest
steps:
- name: Checkout code
Expand Down
39 changes: 36 additions & 3 deletions .github/workflows/smoke.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,44 @@ jobs:
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: "1.22"
go-version: "1.23"
- name: Checkout code
uses: actions/checkout@v4
- name: Smoke
run: |
set -e
go run . git https://github.com/dustin-decker/secretsandstuff.git
go run . github --repo https://github.com/dustin-decker/secretsandstuff.git
go run . git https://github.com/dustin-decker/secretsandstuff.git > /dev/null
go run . github --repo https://github.com/dustin-decker/secretsandstuff.git > /dev/null
zombies:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: "1.23"
- name: Checkout code
uses: actions/checkout@v4
- name: Run trufflehog
run: |
set -e
go run . git --no-verification file://. > /dev/null
# This case previously had a deadlock issue and left zombies after trufflehog exited #3379
go run . git --no-verification https://github.com/git-test-fixtures/binary.git > /dev/null
- name: Check for running git processes and zombies
run: |
if pgrep -x "git" > /dev/null
then
echo "Git processes are still running"
exit 1
else
echo "No git processes found"
fi
if ps -A -ostat,ppid | grep -e '[zZ]' > /dev/null
then
echo "Zombie processes found"
exit 1
else
echo "No zombie processes found"
fi
2 changes: 1 addition & 1 deletion .github/workflows/snifftest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
steps:
- uses: actions/setup-go@v5
with:
go-version: "1.22"
go-version: "1.23"
- uses: actions/checkout@v4
- name: Run Snifftest
run: make snifftest
6 changes: 3 additions & 3 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: "1.22"
go-version: "1.23"
- name: Checkout code
uses: actions/checkout@v4
- id: "auth"
Expand All @@ -34,7 +34,7 @@ jobs:
mkdir -p tmp/test-results
- name: Test
run: |
CGO_ENABLED=1 gotestsum --junitfile tmp/test-results/test.xml --raw-command -- go test -json -tags=sources $(go list ./... | grep -v /vendor/ | grep -v pkg/detectors)
CGO_ENABLED=1 gotestsum --junitfile tmp/test-results/test.xml --raw-command -- go test -json -tags=sources $(go list ./... | grep -v /vendor/ | grep -v pkg/detectors | grep -v pkg/analyzer/analyzers)
if: ${{ success() || failure() }} # always run this step, even if there were previous errors
- name: Upload test results to BuildPulse for flaky test detection
if: ${{ !cancelled() }} # Run this step even when the tests fail. Skip if the workflow is cancelled.
Expand Down Expand Up @@ -62,7 +62,7 @@ jobs:
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: "1.22"
go-version: "1.23"
- name: Checkout code
uses: actions/checkout@v4
- name: Test
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,7 @@ dist

# binary
trufflehog
tmp/go-test.json
.captain/detectors/timings.yaml
.captain/detectors/quarantines.yaml
.captain/detectors/flakes.yaml
28 changes: 25 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ Checksums are applied to all artifacts, and the resulting checksum file is signe

You need the following tool to verify signature:

- [Cosign](https://docs.sigstore.dev/cosign/installation/)
- [Cosign](https://docs.sigstore.dev/cosign/system_config/installation/)

Verification steps are as follow:

Expand Down Expand Up @@ -309,11 +309,33 @@ The following command will enumerate deleted and hidden commits on a GitHub repo
trufflehog github-experimental --repo https://github.com/<USER>/<REPO>.git --object-discovery
```
In addition to the normal TruffleHog output, the `--object-discovery` flag creates two files in a new `$HOME/.trufflehog` directory: `valid_hidden.txt` and `invalid.txt`. These are used to track state during commit enumeration, as well as to provide users with a complete list of all hidden and deleted commits (`valid_hidden.txt`). If you'd like to automatically remove these files after scanning, please add the flag `--delete-cached-data`.
In addition to the normal TruffleHog output, the `--object-discovery` flag creates two files in a new `$HOME/.trufflehog` directory: `valid_hidden.txt` and `invalid.txt`. These are used to track state during commit enumeration, as well as to provide users with a complete list of all hidden and deleted commits (`valid_hidden.txt`). If you'd like to automatically remove these files after scanning, please add the flag `--delete-cached-data`.

**Note**: Enumerating all valid commits on a repository using this method takes between 20 minutes and a few hours, depending on the size of your repository. We added a progress bar to keep you updated on how long the enumeration will take. The actual secret scanning runs extremely fast.

For more information on Cross Fork Object References, please [read our blog post](https://trufflesecurity.com/blog/anyone-can-access-deleted-and-private-repo-data-github).
For more information on Cross Fork Object References, please [read our blog post](https://trufflesecurity.com/blog/anyone-can-access-deleted-and-private-repo-data-github).

## 16. Scan Hugging Face

### Scan a Hugging Face Model, Dataset or Space

```bash
trufflehog huggingface --model <model_id> --space <space_id> --dataset <dataset_id>
```

### Scan all Models, Datasets and Spaces belonging to a Hugging Face Organization or User

```bash
trufflehog huggingface --org <orgname> --user <username>
```

(Optionally) When scanning an organization or user, you can skip an entire class of resources with `--skip-models`, `--skip-datasets`, `--skip-spaces` OR a particular resource with `--ignore-models <model_id>`, `--ignore-datasets <dataset_id>`, `--ignore-spaces <space_id>`.

### Scan Discussion and PR Comments

```bash
trufflehog huggingface --model <model_id> --include-discussions --include-prs
```

# :question: FAQ

Expand Down
Loading

0 comments on commit 9c66a31

Please sign in to comment.