Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: reuse local du.zip #56

Merged
merged 18 commits into from
Aug 10, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .eslintrc
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,8 @@
"env": {
"node": true,
"es6": true
},
"rules": {
"prefer-const": "error"
}
}
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.idea/
node_modules/
bin/
.nyc_output/
17 changes: 13 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ Node CLI or module to calculate folder size.

It uses:

- [Sysinternals DU](https://docs.microsoft.com/en-us/sysinternals/downloads/du) on Windows, automatically downloaded at installation time because the license does not allow redistribution. See below about specifying the download location.
- [Sysinternals DU](https://docs.microsoft.com/en-us/sysinternals/downloads/du) on Windows, automatically downloaded at
installation time because the license does not allow redistribution. See below about specifying the download location.
- native `du` on other platforms

## Installation
Expand Down Expand Up @@ -57,9 +58,17 @@ fast-folder-size .

By default the Sysinternals DU.zip is downloaded from https://download.sysinternals.com/files/DU.zip.

If you need to change this, e.g. to download from an internal package repository,
set the **FAST_FOLDER_SIZE_DU_ZIP_LOCATION** environment variable. For example:
If you need to change this, e.g. to download from an internal package repository
or re-use an existing du.zip, you can set the **FAST_FOLDER_SIZE_DU_ZIP_LOCATION** environment variable.

For example:

```shell
export FAST_FOLDER_SIZE_DU_ZIP_LOCATION="https://your.internal.repository/DU.zip"
```

or

```shell
export FAST_FOLDER_SIZE_DU_ZIP_LOCATION=https://your.internal.repository/DU.zip
export FAST_FOLDER_SIZE_DU_ZIP_LOCATION="D://download/du.zip"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why double slash after the drive name? I assume we're targeting Windows in this case, so shouldn't the path separator be backslash instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's personal habit. I tested the C://**/du.zip and C:\**\du.zip, they both pass the test.

Should we add another example like:

or

export FAST_FOLDER_SIZE_DU_ZIP_LOCATION="D:\download\du.zip"

or just replace "D://download/du.zip" with "D:\download\du.zip" ?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's just make sure this works on the windows terminal and not an emulator

```
102 changes: 70 additions & 32 deletions get-sysinternals-du.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,45 +5,83 @@ const path = require('path')
const decompress = require('decompress')
const { HttpsProxyAgent } = require('https-proxy-agent')

// Only run for Windows
if (process.platform !== 'win32') {
process.exit(0)
exports.onDuZipDownloaded = function (tempFilePath, workspace) {
decompress(tempFilePath, path.join(workspace, 'bin'))
}

// check if du is already installed
if (
fs.existsSync(
path.join(__dirname, 'bin', `du${process.arch === 'x64' ? '64' : ''}.exe`)
)
) {
process.exit(0)
}
exports.downloadDuZip = function (mirror, workspace) {
const duZipLocation =
mirror || 'https://download.sysinternals.com/files/DU.zip'

// checks for proxy variables in user environment
const proxyAddress =
process.env.HTTPS_PROXY ||
process.env.https_proxy ||
process.env.HTTP_PROXY ||
process.env.http_proxy

if (proxyAddress) {
https.globalAgent = new HttpsProxyAgent(proxyAddress)
}

const duZipLocation =
process.env.FAST_FOLDER_SIZE_DU_ZIP_LOCATION ||
'https://download.sysinternals.com/files/DU.zip'
console.log(`downloading du.zip from ${duZipLocation}`)
if (!mirror) {
console.log(
`if you have trouble while downloading, try set process.env.FAST_FOLDER_SIZE_DU_ZIP_LOCATION to a proper mirror or local file path`
)
}

// checks for proxy variables in user environment
const proxyAddress =
process.env.HTTPS_PROXY ||
process.env.https_proxy ||
process.env.HTTP_PROXY ||
process.env.http_proxy
https.get(duZipLocation, function (res) {
const tempFilePath = path.join(os.tmpdir(), 'du.zip')

if (proxyAddress) {
const agent = new HttpsProxyAgent(proxyAddress)
const fileStream = fs.createWriteStream(tempFilePath)
res.pipe(fileStream)

https.globalAgent = agent
fileStream.on('finish', function () {
fileStream.close()
exports.onDuZipDownloaded(tempFilePath, workspace)
})
})
}

https.get(duZipLocation, function (res) {
const tempFilePath = path.join(os.tmpdir(), 'du.zip')
exports.default = function (workspace) {
// Only run for Windows
if (process.platform !== 'win32') {
return
}

const fileStream = fs.createWriteStream(tempFilePath)
res.pipe(fileStream)
// check if du is already installed
const duBinFilename = `du${process.arch === 'x64' ? '64' : ''}.exe`
const defaultDuBinPath = path.join(workspace, 'bin', duBinFilename)

fileStream.on('finish', function () {
fileStream.close()
decompress(tempFilePath, path.join(__dirname, 'bin'))
})
})
if (fs.existsSync(defaultDuBinPath)) {
console.log(`${duBinFilename} found at ${defaultDuBinPath}`)
return
}
console.log(`${duBinFilename} not found at ${defaultDuBinPath}`)

const mirrorOrCache = process.env.FAST_FOLDER_SIZE_DU_ZIP_LOCATION

if (
!mirrorOrCache ||
mirrorOrCache.startsWith('http://') ||
mirrorOrCache.startsWith('https://')
) {
exports.downloadDuZip(mirrorOrCache, workspace)
return
}

if (fs.existsSync(mirrorOrCache)) {
exports.onDuZipDownloaded(mirrorOrCache, workspace)
return
}

const message = `du.zip not found at ${mirrorOrCache}`
// this will result the process to exit with code 1
throw Error(message)
}

// only auto execute default() function when its invoked directly
if (require.main === module) {
exports.default(__dirname)
}
88 changes: 88 additions & 0 deletions get-sysinternals-du.test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
if (process.platform !== 'win32') {
// these test cases are only for win32
return
}

const { test, beforeEach } = require('tap')
const path = require('path')
const os = require('os')
const fs = require('fs')
const subject = require('./get-sysinternals-du.js')

const workspace = path.join(os.tmpdir(), 'fast-folder-size-playground')
beforeEach(() => {
if (fs.existsSync(workspace)) fs.rmSync(workspace, { recursive: true })
fs.mkdirSync(workspace)
})

test('it can use local file path as process.env.FAST_FOLDER_SIZE_DU_ZIP_LOCATION', t => {
t.test('C:\\**\\du.zip', t => {
const dummyDuZipPath = path.join(workspace, 'dummy-du.zip')
fs.writeFileSync(dummyDuZipPath, '')
process.env.FAST_FOLDER_SIZE_DU_ZIP_LOCATION = dummyDuZipPath

subject.onDuZipDownloaded = function (tempFilePath) {
t.equal(tempFilePath, dummyDuZipPath)
t.end()
}

subject.default(workspace)
})

t.test('C://**/du.zip', t => {
const dummyDuZipPath = path
.join(workspace, 'dummy-du.zip')
.replaceAll('\\', '/')
.replace(':/', '://')
fs.writeFileSync(dummyDuZipPath, '')
process.env.FAST_FOLDER_SIZE_DU_ZIP_LOCATION = dummyDuZipPath

subject.onDuZipDownloaded = function (tempFilePath) {
t.equal(tempFilePath, dummyDuZipPath)
t.end()
}

subject.default(workspace)
})

t.end()
})

test('it cannot use non-exists local file path as process.env.FAST_FOLDER_SIZE_DU_ZIP_LOCATION', t => {
process.env.FAST_FOLDER_SIZE_DU_ZIP_LOCATION = path.join(
workspace,
'non-exists-dummy-du.zip'
)

t.throws(
() => subject.default(workspace),
error => {
return error.message.startsWith('du.zip not found at')
}
)

t.end()
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when you don't need to assert asynchronously in callbacks, please write the test function as an async function so you don't have to remember calling t.end

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

Once I added the async keyword, eslint shows an error on the parameter t: ESLint: Parsing error: Unexpected token t.
Everything looks fine to me as I'm following the doc https://node-tap.org/docs/api/promises/ , do you know how to fix this ?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably just an eslint configuration to support esm, should be easy to address

})

test('it can use http(s) url as process.env.FAST_FOLDER_SIZE_DU_ZIP_LOCATION', t => {
const dummyUrl = 'https://non-exists.localhost/du.zip'
process.env.FAST_FOLDER_SIZE_DU_ZIP_LOCATION = dummyUrl

subject.downloadDuZip = function (mirror) {
t.equal(mirror, dummyUrl)
t.end()
}

subject.default(workspace)
})

test('when process.env.FAST_FOLDER_SIZE_DU_ZIP_LOCATION not found, then download it directly', t => {
delete process.env.FAST_FOLDER_SIZE_DU_ZIP_LOCATION

subject.downloadDuZip = function (mirror) {
t.equal(mirror, undefined)
t.end()
}

subject.default(workspace)
})
10 changes: 10 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
},
"devDependencies": {
"@types/node": "^20.2.5",
"@types/tap": "^15.0.8",
"eslint": "^8.41.0",
"eslint-config-prettier": "^8.8.0",
"eslint-plugin-prettier": "^4.2.1",
Expand Down