Question regarding easy caching approach #142
Comments
|
Yes, several people use this option. They build a database on one node, save the database, and then copy the DB to any other node that is running ODC and simply use the |
jeremylong
added
documentation
|
great 😄 thank you for the confirmation |
|
In @cyberblast's example, however, |
|
I added |
|
It seems I forgot about Azure Cache tasks hard wired and strict scoping, which makes sharing of cache between pipes impossible 😞 |
|
Hi, Anyway, I believe it may give a good starting point for anybody implementing a similar aproach. NVD Update Pipe parameters:
- name: purge
displayName: Purge Database
type: boolean
default: false
- name: verbose
displayName: Verbose
type: boolean
default: false
- name: nvdValidForHours
displayName: NVD valid for hours
type: number
default: 23
- name: additionalArguments
displayName: Additional arguments
type: string
default: ' '
- name: nvdApiKey
displayName: NVD API key
type: string
trigger: none
schedules:
- cron: '0 0 * * *'
displayName: Daily midnight run
branches:
include:
- master
stages:
- stage: update_odc_nvd
displayName: nvd.nist.gov
pool:
name: Azure Pipelines
vmImage: ubuntu-latest
jobs:
- job: build
workspace:
clean: outputs
displayName: Update NIST NVD
steps:
- checkout: none
- task: Cache@2
displayName: ODC Cache
inputs:
key: 'ODC | "$(Agent.OS)"'
path: $(Pipeline.Workspace)/odc/app
- task: Cache@2
displayName: NVD Cache
inputs:
key: 'NVD | "$(Agent.OS)"'
path: $(Pipeline.Workspace)/odc/data
- bash: |
set -x # echo on
VERSION=$(curl -s https://jeremylong.github.io/DependencyCheck/current.txt)
if [ ! -d "$(Pipeline.Workspace)/odc/app/$VERSION" ]; then
rm -rf $(Pipeline.Workspace)/odc/app/*
mkdir -p $(Pipeline.Workspace)/odc/app/$VERSION
curl -Ls "https://github.com/jeremylong/DependencyCheck/releases/download/v$VERSION/dependency-check-$VERSION-release.zip" --output dependency-check.zip
unzip -uq ./dependency-check.zip -d $(Pipeline.Workspace)/odc/app/$VERSION
fi
$(Pipeline.Workspace)/odc/app/$VERSION/dependency-check/bin/dependency-check.sh --updateonly --nvdApiKey ${{ parameters.nvdApiKey }} --data $(Pipeline.Workspace)/odc/data --nvdValidForHours ${{ parameters.nvdValidForHours }} $PURGE ${{ parameters.additionalArguments }}
displayName: Update NVD
env:
${{ if eq( parameters.purge, true ) }}:
PURGE: '--purge'
${{ else }}:
PURGE: ''
- task: ArchiveFiles@2
displayName: Compress NVD Artifact
inputs:
rootFolderOrFile: '$(Pipeline.Workspace)/odc/data'
includeRootFolder: false
archiveFile: '$(Build.ArtifactStagingDirectory)/NVD.zip'
- task: PublishPipelineArtifact@1
displayName: Publish NVD Artifact
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)/NVD.zip'
artifact: 'NVD'
publishLocation: 'pipeline'Please be aware that I'm still not 100% sure if this code works well, as NVD DB is currently unavailable. Also, this code doesn't make use of the Azure DevOps Marketplace task "dependency-check-build-task@6". I started with it, but you need to add additional parameters and other mandatory parameters of the task are not needed at all for this use case. So I decided to get rid of it eventually. |
|
Ciao @cyberblast, |
|
Hi @pippolino, |
|
Hi, For completeness I'm also pasting consumer code, executing ODC. It's a task template. parameters:
- name: verbose
type: boolean
default: false
- name: projectName
type: string
default: 'OWASP'
- name: scanPath
type: string
default: './'
- name: warningOnly
type: boolean
default: false
- name: additionalArguments
type: string
default: ''
- name: cvssThreshold
type: number
default: '4'
- name: format
type: string
default: 'HTML, JUNIT, JSON'
- name: publishTestResults
type: boolean
default: true
- name: NistNvdTeamProject
type: string
default: '<Name of DevOps Project>'
- name: NistNvdPipeId
type: string
default: '<Name of NVD DB Pipe>'
- name: NistNvdPipeBranch
type: string
default: 'refs/heads/master'
- name: NistNvdArtifactName
type: string
default: 'NVD'
- name: NistNvdFileName
type: string
default: 'NVD.zip'
steps:
- task: DownloadPipelineArtifact@2
displayName: Download NVD Artifact
continueOnError: ${{ parameters.warningOnly }}
inputs:
source: specific
project: ${{ parameters.NistNvdTeamProject }}
pipeline: ${{ parameters.NistNvdPipeId }}
runVersion: latestFromBranch
runBranch: ${{ parameters.NistNvdPipeBranch }}
artifact: ${{ parameters.NistNvdArtifactName }}
path: '$(Pipeline.Workspace)/odc'
- task: ExtractFiles@1
displayName: Unpack NVD
continueOnError: ${{ parameters.warningOnly }}
inputs:
archiveFilePatterns: '$(Pipeline.Workspace)/odc/${{ parameters.NistNvdFileName }}'
destinationFolder: '$(Pipeline.Workspace)/odc/data'
overwriteExistingFiles: true
- task: dependency-check-build-task@6
displayName: 'OWASP Dependency Check'
continueOnError: ${{ parameters.warningOnly }}
condition: succeeded()
inputs:
projectName: ${{ parameters.projectName }}
scanPath: ${{ parameters.scanPath }}
format: ${{ parameters.format }}
enableVerbose: ${{ parameters.verbose }}
failOnCVSS: ${{ parameters.cvssThreshold }}
warnOnCVSSViolation: ${{ parameters.warningOnly }}
additionalArguments: --noupdate --data $(Pipeline.Workspace)/odc/data ${{ parameters.additionalArguments }}
- ${{ if eq(parameters.publishTestResults, true) }}:
- task: PublishTestResults@2
displayName: 'Publish ODC results'
continueOnError: ${{ parameters.warningOnly }}
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
searchFolder: $(Common.TestResultsDirectory)
testResultsFiles: 'dependency-check/*junit.xml'
failTaskOnFailedTests: ${{ not(parameters.warningOnly) }}To use the task in a pipe it can be done like this (here with pipe in same repo for C#): - template: ../task/test-owasp-dependencies.yml
parameters:
scanPath: '**/*.csproj'
warningOnly: trueand for npm (e.g. react): - template: ../task/test-owasp-dependencies.yml
parameters:
scanPath: '**/yarn.lock'
additionalArguments: '--scan "$(Build.SourcesDirectory)/**/package.json" --scan "$(Build.SourcesDirectory)/**/node_modules" --disableYarnAudit --nodeAuditSkipDevDependencies --nodePackageSkipDevDependencies'
warningOnly: truePlease note that we are here disabling Yarn Audit ( |
|
I also used a similar approach after all of the issues. One suggestion for your pipeline is that you don't need to have tasks to archive and unarchive the files. Azure Pipeline Artifacts does all that already and has optimizations for uploading and downloading to skip redundant files. Here's my pipeline that caches the data files. It runs every 4 hours to always have the latest NVD data while following their recommended best practice for frequency. The nvd and oss variables are stored as secret pipeline variables. appendCommitMessageToRunName: false
trigger:
batch: true
branches:
include:
- '*'
paths:
include:
- OwaspResourceDownload.yml
schedules:
- cron: '0 0,4,8,12,16,20 * * *'
displayName: 'Q.4H Update'
branches:
include:
- main
always: true
variables:
dependencyCheckVersion: latest
pool:
vmImage: 'windows-latest'
stages:
- stage: update
displayName: Update OWASP Dependency Check Data
jobs:
- job: update
displayName: Update OWASP Dependency Check Data
steps:
- checkout: none
- task: PowerShell@2
displayName: Update Build Name
inputs:
targetType: 'inline'
script: |
# OWASP Dependency Check Version
$latestOnlineVersion = Invoke-RestMethod -Uri 'https://jeremylong.github.io/DependencyCheck/current.txt'
$odcVersion = if ($env:dependencyCheckVersion -eq 'latest' -and $latestOnlineVersion) {
$latestOnlineVersion
}
else {
$env:dependencyCheckVersion
}
Write-Host -Object "Dependency Check Version: $odcVersion"
# NVD Last Change
$headers = @{
'Accept' = 'application/json'
'apiKey' = $env:nvdApiKey
}
$startDate = ( Get-Date ).ToUniversalTime().AddHours(-4).ToString('o')
$endDate = ( Get-Date ).ToUniversalTime().ToString('o')
$uri = "https://services.nvd.nist.gov/rest/json/cvehistory/2.0/?changeStartDate=$startDate&changeEndDate=$endDate"
try {
$lastChange = Invoke-RestMethod -Uri $uri -Headers $headers -ErrorAction Stop |
Select-Object -ExpandProperty cveChanges |
Select-Object -Last 1
$nvcLastChangeTime = $lastChange.change.created | Get-Date -Format 'yyyyMMdd.HHmm'
}
catch {
Write-Warning -Message "##[warning] Failed to get NVD Last Change: $($_.Exception.Message)"
$nvcLastChangeTime = $endDate | Get-Date -Format 'yyyyMMdd.HHmm'
}
Write-Host -Object "NVD Last Change: $nvcLastChangeTime"
Write-Host -Object "##vso[task.setvariable variable=nvcLastChangeTime;]$nvcLastChangeTime"
Write-Host -Object "##vso[Build.UpdateBuildNumber]ODC-$($odcVersion)_NVD-$($nvcLastChangeTime)"
- task: Cache@2
inputs:
key: 'owasp-dependency-check | data | "$(nvcLastChangeTime)"'
path: '$(Pipeline.Workspace)/owasp-dependency-check-data'
restoreKeys: 'owasp-dependency-check | data'
- task: dependency-check-build-task@6
displayName: OWASP Dependency Check
retryCountOnTaskFailure: 1
inputs:
dependencyCheckVersion: $(dependencyCheckVersion)
projectName: 'Update'
scanPath: '$(Pipeline.Workspace)'
additionalArguments: >
--nvdApiKey $(nvdApiKey)
--nvdApiDelay 6000
--data "$(Pipeline.Workspace)/owasp-dependency-check-data"
--ossIndexUsername $(ossIndexUsername)
--ossIndexPassword $(ossIndexPassword)
--updateonly
- publish: $(Pipeline.Workspace)/owasp-dependency-check-data
artifact: owasp-dependency-check-dataI then consume it with the following tasks (can't include the whole pipeline for IP reasons): Declare the above pipeline as a resource: resources:
pipelines:
- pipeline: OWASPResources
source: OWASP Resource Download
branch: mainI use variables for CVSS score and ODC version variables:
failOnCVSS: 7 # More info -> https://www.recordedfuture.com/cvss-scores-guide/
dependencyCheckVersion: latestAnd the steps, using the |
|
Hello @cyberblast. After downloading the NVD.zip, i'm trying to run an ODC scan with maven plugin by providing -DautoUpdate=false and also -DdataDirectory=$(Pipeline.Workspace)/odc/data. I also tried to extract the NVD zip to $(Pipeline.Workspace)/.m2/repository/org/owasp/dependency-check-data/9.0.2 but still not working. if by any chance you have an idea ;) |
|
Hi, @thisjustin816 thanks for sharing. Contains some interesting aspects. But also maybe depends a bit on usage scenario/environment. Will also look up again on the artifact topic. I wasn't aware of it. @omgdota123 you need to extract it to the data directory |
|
Hi everyone! parameters:
- name: projectName
type: string
- name: isExperimentalEnabled
type: boolean
steps:
- task: DockerInstaller@0
inputs:
dockerVersion: "$(LATEST_DOCKER_VERSION)"
- script: |
docker pull owasp/dependency-check-action:latest
displayName: Pull OWASP Image
- script: |
experimental=""
if [ "${{ parameters.isExperimentalEnabled }}" ]; then
echo "Experimental analyzers enabled"
experimental="--enableExperimental"
fi
docker run --rm -v $(System.DefaultWorkingDirectory):/workspace owasp/dependency-check-action:latest \
--project ${{ parameters.projectName }} \
--failOnCVSS 7 \
--scan /workspace \
--format HTML --format JUNIT \
--noupdate \
--out /workspace $experimental
displayName: Run dependency check
- task: PublishTestResults@2
inputs:
testResultsFormat: "JUnit"
testResultsFiles: "dependency-check-junit.xml"
searchFolder: "$(System.DefaultWorkingDirectory)"
displayName: "Publish Dependency Check Test Results if Available"
- task: PublishPipelineArtifact@1
inputs:
targetPath: "$(System.DefaultWorkingDirectory)/dependency-check-report.html"
artifact: "OWASP DOCKER HTML"
publishLocation: "pipeline"
displayName: "Publish OWASP Artifact"
# Just in case, removing:
- script: |
docker image rm owasp/dependency-check-action:latest
displayName: Remove OWASP ImageWhat are your thought on this approach? |
|
Hi everyone, @thisjustin816, I recreated your template as bash script(Linux user), worked fine your logic Based by script @thisjustin816 - updated for linux user
appendCommitMessageToRunName: false
trigger:
batch: true
branches:
include:
- '*'
paths:
include:
- update-dpdc-check.yml
schedules:
- cron: '0 0,4,8,12,16,20 * * *'
displayName: 'Q.4H Update'
branches:
include:
- master
- feature/*
always: true
variables:
dependencyCheckVersion: 10.0.4
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: update
displayName: Update OWASP Dependency Check Data
jobs:
- job: update
variables:
- group: DEV
displayName: Update OWASP Dependency Check Data
steps:
- checkout: none
- task: Bash@3
displayName: Update Build Name
inputs:
targetType: 'inline'
script: |
#!/bin/bash
# OWASP Dependency Check Version
latestOnlineVersion=$(curl -s 'https://jeremylong.github.io/DependencyCheck/current.txt')
# Determine the Dependency Check version
if [[ "$(dependencyCheckVersion)" == "latest" && -n "$latestOnlineVersion" ]]; then
export odcVersion=$latestOnlineVersion
else
odcVersion=$(dependencyCheckVersion)
fi
echo "Dependency Check Version: $odcVersion"
# NVD Last Change
headers=(
-H "Accept: application/json"
-H "apiKey: $(nvdApiKey)"
)
startDate=$(date -u -d '-4 hours' +"%Y-%m-%dT%H:%M:%SZ")
endDate=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
uri="https://services.nvd.nist.gov/rest/json/cvehistory/2.0/?changeStartDate=$startDate&changeEndDate=$endDate"
# Fetch the last change
response=$(curl -s "${headers[@]}" "$uri")
lastChange=$(printf '%s\n' "$response" | jq -r '.cveChanges[-1].change.created')
if [[ -n "$lastChange" ]]; then
nvcLastChangeTime=$(date -d "$lastChange" +"%Y%m%d.%H%M")
else
echo "##[warning] Failed to get NVD Last Change"
nvcLastChangeTime=$(date -d "$endDate" +"%Y%m%d.%H%M")
fi
echo "NVD Last Change: $nvcLastChangeTime"
# Set variables for Azure DevOps
echo "##vso[task.setvariable variable=nvcLastChangeTime;]$nvcLastChangeTime"
echo "##vso[Build.UpdateBuildNumber]ODC-$(printf '%s\n' $odcVersion)_NVD-$nvcLastChangeTime"
- task: Cache@2
inputs:
key: 'owasp-dependency-check | data | "$(nvcLastChangeTime)"'
path: '$(Pipeline.Workspace)/owasp-dependency-check-data'
restoreKeys: 'owasp-dependency-check | data'
- task: dependency-check-build-task@6
displayName: OWASP Dependency Check
retryCountOnTaskFailure: 1
inputs:
dependencyCheckVersion: $(dependencyCheckVersion)
projectName: 'Update'
scanPath: '$(Pipeline.Workspace)'
additionalArguments: >
--nvdApiKey $(nvdApiKey)
--nvdApiDelay 6000
--data "$(Pipeline.Workspace)/owasp-dependency-check-data"
--updateonly
--ossIndexUsername $(sonartypeossindex)
--enableExperimental
--ossIndexPassword $(sonarToken)
$(Args_extra)
- publish: $(Pipeline.Workspace)/owasp-dependency-check-data
artifact: owasp-dependency-check-dataI changed the resource instead declared on pipeline, for download task, because I use template for dependency check: - task: DownloadPipelineArtifact@2
inputs:
buildType: 'specific'
project: 'PROJECT'
definition: NNNN
buildVersionToDownload: 'latest'
artifactName: 'owasp-dependency-check-data'
targetPath: '$(Build.Repository.LocalPath)/owasp-dependency-check-data'
displayName: 'Download OWASP Dependency Check Data'Thank you so much for your contrubution, help me a lot ... |
Thank you!!! Out of curiosity , why use the Cron task instead of DevOps trigger schedule in the UI? |
Hi,
obviously loading the whole NVD database for every pipe run is a bad idea. So I thought how to improve it without requiring too much effort or even costs for hosting etc.
Then I came across the --data CLI argument. Using that, we could easily use Azure DevOps Cache task to cache/restore the data.
But I'm wondering if that is a valid approach, as the description for the argument says: "This option should generally not be set."
Also that approach is suggested nowhere.
Anyway, I started to implement it like this, but unfortunately I'm unable to test it currently due to some issues on NVD API side of things (HTTP 503).
Any idea if that should work or not at all or if there is any reason why it should not be done like this?
The text was updated successfully, but these errors were encountered: