Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Helper to bypass mock FS & expose real files/directories #304

Merged
merged 31 commits into from
Aug 9, 2020
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
26bfdf7
Added mock.bypass() and mock.createDirectoryInfoFromPaths()
nonara Jul 27, 2020
be7bf43
Fix broken `xdescribe`
nonara Jul 27, 2020
a39712e
Node 6 support
nonara Jul 27, 2020
266569b
Investigate missing entry (squash)
nonara Jul 27, 2020
95d3ff2
Fixed + improved test
nonara Jul 27, 2020
34a4503
Added normalize path
nonara Jul 27, 2020
91ab56a
Added coverage for non-string error
nonara Jul 27, 2020
fdeed02
Remove artifact
nonara Jul 27, 2020
0ad9857
prettier
nonara Jul 27, 2020
c3eddcb
LazyLoad Fixes
nonara Jul 27, 2020
decb5de
Updated readme
nonara Jul 27, 2020
135ec12
Make automatically created directories inherit stats (permissions, da…
nonara Jul 27, 2020
61b8ac1
Move badge to top of readme
nonara Jul 27, 2020
b195d8e
Fix: Make non-lazy loaded files retain stats
nonara Jul 27, 2020
3b58f79
Prefer existing permissions const
nonara Jul 27, 2020
6950d39
Re-engineered API
nonara Jul 28, 2020
9f284af
Update readme
nonara Jul 28, 2020
562c6ae
Added tests for integration with mock() & fs.readFileSync
nonara Jul 28, 2020
2ca7d0e
Correct typo
nonara Jul 28, 2020
10f61da
Fix: fixWin32Permissions missing platform check
nonara Jul 28, 2020
92ca9c8
Address 3cp's comments
nonara Jul 28, 2020
172e698
Clarified bypassing methods & added restoration test
nonara Jul 29, 2020
9087580
Update mock.bypass test
nonara Jul 29, 2020
0c07e57
Moved mapping helpers to outside module
nonara Jul 29, 2020
8e11bec
Fix circular issue
nonara Jul 29, 2020
4b579c0
Made modules according to repo convention
nonara Jul 29, 2020
8dfdc85
Applied review fixes
nonara Aug 4, 2020
2cc92f3
Remove async await in test (Node 6/8 support)
nonara Aug 4, 2020
0c7bbbb
Prettier correction
nonara Aug 4, 2020
f431af2
Add withPromise to test
nonara Aug 4, 2020
768224f
Changed to typeof function check
nonara Aug 4, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions lib/filesystem.js
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,8 @@ FileSystem.prototype.getItem = function(filepath) {
if (item) {
if (item instanceof Directory && name !== currentParts[i]) {
nonara marked this conversation as resolved.
Show resolved Hide resolved
// make sure traversal is allowed
// This fails for Windows directories which do not have execute permission, by default. It may be a good idea
// to change this logic to windows-friendly. See notes in mock.createDirectoryInfoFromPaths()
if (!item.canExecute()) {
throw new FSError('EACCES', filepath);
}
Expand Down
123 changes: 123 additions & 0 deletions lib/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,15 @@ const realCreateWriteStream = fs.createWriteStream;
const realStats = realBinding.Stats;
const realStatWatcher = realBinding.StatWatcher;

// File Permissions
const S_IRUSR = 0o400;
const S_IRGRP = 0o40;
const S_IROTH = 0o4;

const S_IXUSR = 0o100;
const S_IXGRP = 0o10;
const S_IXOTH = 0o1;
3cp marked this conversation as resolved.
Show resolved Hide resolved

/**
* Pre-patch fs binding.
* This allows mock-fs to work properly under nodejs v10+ readFile
Expand Down Expand Up @@ -183,3 +192,117 @@ exports.directory = FileSystem.directory;
* Create a symbolic link factory.
*/
exports.symlink = FileSystem.symlink;

/**
* Perform action, bypassing mock FS
* @example
* // This file exists on the real FS, not on the mocked FS
* const filePath = '/path/file.json';
* const data = mock.bypass(() => fs.readFileSync(filePath, 'utf-8'));
*/
exports.bypass = function(fn) {
if (typeof fn !== 'function') {
throw new Error(`Must provide a function to perform for mock.bypass()`);
}

// Deactivate mocked bindings
const binding = process.binding('fs')._mockedBinding;
nonara marked this conversation as resolved.
Show resolved Hide resolved
delete process.binding('fs')._mockedBinding;

// Perform action
const res = fn();
nonara marked this conversation as resolved.
Show resolved Hide resolved

// Reactivate mocked bindings
process.binding('fs')._mockedBinding = binding;

return res;
};

/**
* Populate a DirectoryItems object to use with mock() from a series of paths to files/directories
*/
exports.createDirectoryInfoFromPaths = function(paths, options) {
return exports.bypass(() => {
const res = {};

/* Get options or apply defaults */
let recursive = options && options.recursive;
let lazyLoad = options && options.lazyLoad;
if (recursive === undefined) {
recursive = true;
}
if (lazyLoad === undefined) {
lazyLoad = true;
}

if (Array.isArray(paths)) {
paths.forEach(p => scan(p, true));
} else {
scan(paths, true);
}

return res;

function scan(p, isRoot) {
if (typeof p !== 'string') {
throw new Error(
`Must provide path or array of paths (as strings) to createDirectoryInfoFromPaths()`
);
}

p = path.normalize(p);

const stats = fs.statSync(p);
if (stats.isFile()) {
addFile(p, stats);
} else if ((isRoot || recursive) && stats.isDirectory()) {
const dirStats = Object.assign({}, stats);
// On windows platforms, directories do not have the executable flag, which causes FileSystem.prototype.getItem
// to think that the directory cannot be traversed. This is a workaround, however, a better solution may be to
// re-think the logic in FileSystem.prototype.getItem
// This workaround adds executable privileges if read privileges are found
if (process.platform === 'win32') {
// prettier-ignore
dirStats.mode |=
((dirStats.mode & S_IRUSR) && S_IXUSR) |
((dirStats.mode & S_IRGRP) && S_IXGRP) |
((dirStats.mode & S_IROTH) && S_IXOTH);
}
res[p] = exports.directory(dirStats);
fs.readdirSync(p).forEach(subPath => scan(path.join(p, subPath)));
}
}

function addFile(p, stats) {
res[p] = () => {
const content = lazyLoad
? exports.bypass(() => fs.readFileSync(p))
: '';

const file = exports.file(Object.assign({}, stats, {content}))();

if (lazyLoad) {
Object.defineProperty(file, '_content', {
get() {
const res = exports.bypass(() => fs.readFileSync(p));
Object.defineProperty(file, '_content', {
value: res,
writable: true
});
return res;
},
set(data) {
Object.defineProperty(file, '_content', {
value: data,
writable: true
});
},
configurable: true
});
}

return file;
};
}
});
};
4 changes: 2 additions & 2 deletions lib/item.js
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,8 @@ Item.prototype.canExecute = function() {
let can = false;
if (uid === 0) {
can = true;
} else if (uid === this._uid || uid !== uid) {
// (uid !== uid) means uid is NaN, only for windows
} else if (uid === this._uid || isNaN(uid)) {
// NaN occurs on windows
can = (permissions.USER_EXEC & this._mode) === permissions.USER_EXEC;
} else if (gid === this._gid) {
can = (permissions.GROUP_EXEC & this._mode) === permissions.GROUP_EXEC;
Expand Down
66 changes: 61 additions & 5 deletions readme.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
[![Build Status](https://github.com/tschaub/mock-fs/workflows/Test/badge.svg)](https://github.com/tschaub/mock-fs/actions?workflow=Test)

# `mock-fs`

The `mock-fs` module allows Node's built-in [`fs` module](http://nodejs.org/api/fs.html) to be backed temporarily by an in-memory, mock file system. This lets you run tests against a set of mock files and directories instead of lugging around a bunch of test fixtures.
Expand Down Expand Up @@ -58,7 +60,51 @@ The second (optional) argument may include the properties below.
* `createCwd` - `boolean` Create a directory for `process.cwd()`. This is `true` by default.
* `createTmp` - `boolean` Create a directory for `os.tmpdir()`. This is `true` by default.

### Creating files
### Automatically Creating Files & Directories
nonara marked this conversation as resolved.
Show resolved Hide resolved

You can create files and directories automatically by providing paths to `mock.createDirectoryInfoFromPaths()`. This will
read the filesystem for the paths you provide and automatically create files and directories for them.

### <a id='cdifp'>`mock.createDirectoryInfoFromPaths(paths, options)`</a>

#### <a id='cdifp_paths'>`paths`</a> - `string` or `string[]`

#### <a id='cdifp_options'>`options`</a>

The second (optional) argument may include the properties below.

* `lazyLoad` - `boolean` File content does not get loaded until explicitly read. This is `true` by default.
* `recursive` - `boolean` Load all files and directories recursively. This is `true` by default.

#### Examples

Given the following directory structure
```
- /root/
- subdir/
- file2.txt
- file1.txt
- /lib/
- library.js
- extra.js
```
```js
// Creates files and dirs for all in `/root` and creates the directory `/lib` and the file `/lib/extra.js`
// Notes:
// - /lib/library.js is not included
// - Files are lazy-loaded
mock(mock.createDirectoryInfoFromPaths([ '/root', '/lib/extra.js' ]));

// -------------------------------------------------------------------------------------

// Creates `/root` directory and `/root/file1.txt`
// Notes:
// - subdir and its contents are not loaded (due to recursive=false)
// - Files content is loaded into memory immediately (due to lazyLoad=false)
mock(mock.createDirectoryInfoFromPaths([ '/root' ], { recursive: false, lazyLoad: false }));
```

Copy link
Owner

@tschaub tschaub Jul 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I appreciate all that you've put together here, @nonara.

I have some concern about the size of the change to the API (adding bypass, enable, disable, mapPaths, mapFile, and mapDir). And I think the subtle difference in usage of mapPaths vs mapFile or mapDir is error prone.

It feels like we could try to minimize the changes to the API while still providing most of the same functionality.

All of mapPaths, mapFile, and mapDir look to be about providing a convenient way to load up a mock filesystem with data from the real filesystem. Words like load, clone, and read come to mind for function names.

mock({
  '/path/to/dir': mock.load('/path/to/dir'),
  '/path/to/file': mock.load('/path/to/file')
})

That implies adding a single load function that takes either a path to a directory or a path to a file and copies contents and metadata into a corresponding mock filesystem entry. I assume we should also make it work for symlinks.

If we find that usage patterns make it very awkward to repeat the real path in the mock path, we could add a function that makes that more convenient later.

The bypass, enable, and disable functions feel like they could be boiled down to a single function that works with either an async/promise returning function or a sync function. In the docs below, you warn against using async calls with the mock disabled - that same warning could be included in the bypass docs without adding the enable and disable functions.

Having a single bypass function also gets around the issue that calling enable twice makes disable not work (this is fixable, but could also be avoided by not adding these functions at all).

Copy link
Contributor Author

@nonara nonara Jul 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of mapPaths, mapFile, and mapDir look to be about providing a convenient way to load up a mock filesystem with data from the real filesystem. Words like load, clone, and read come to mind for function names.

I agree that mapDir and mapFile could be merged, and load is a great name.

As for mapPaths, this is a simple way to quickly load multiple real paths. (though we can rename - loadPaths or mount?)

// With (simple)
mock(mock.loadPaths(realPaths));

// Without (simple)
mock(realPaths.reduce((p,c) => { p[c] = map.load(p); return p }, {})));

// With (extended)
mock{{
  ...mock.loadPaths(realPaths),
  'extra': 'content'
});

// Without (extended)
mock({
  ...realPaths.reduce((p,c) => { p[c] = map.load(p; return p) }, {})),
  'extra': 'content'
});

Bear in mind that while reduce is ugly and difficult to read, this is the shortest solution. Most people will end up using multi-line for loops, etc, all to accomplish what many have requested - a feature to simply 'mount' real areas. Worse yet, some will end up doing

{ 
  '/path/to/my/location': map.load('/path/to/my/location'),
  '/path/to/other.location': map.load('/path/to/other/location'),
  ...
}

Tedious.

The decision lies with you. I really have spent more time on this than I am able, so I'll go with whatever you want, but I hope you'll consider it reasonable to allow the API to have three new items: load, bypass, and loadPaths

Otherwise, many (myself included) will end up replicating helper logic to convert an array of real locations for each package we use this in.

The bypass, enable, and disable functions feel like they could be boiled down to a single function that works with either an async/promise returning function or a sync function. In the docs below, you warn against using async calls with the mock disabled - that same warning could be included in the bypass docs without adding the enable and disable functions.

Agreed. The consideration was that many people will simply use the function without noticing the warning. It was set up to be more strict to prevent people using it and filing issues.

But this is entirely your call, if you're good with it, then it's no problem. I'll add async support, and we can add the warning to both the readme and JSDoc for the types package to make sure it's seen.

Having a single bypass function also gets around the issue that calling enable twice makes disable not work (this is fixable, but could also be avoided by not adding these functions at all).

Didn't see that. I was up quite late last night. I'll correct the issue. If you're alright with it, I'd like to leave them attached to exports and simply not document in readme or in the @types package. That way there is zero-footprint, but people who know the source can use them.

I can always replicate the behaviour to do it, but this way if anything changes my code doesn't break.

Let me know! Hopefully we can get this wrapped up. I'll wait until your responses before updating anything.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel tediousness is an issue. I cannot imagine people use mock-fs to load lots of real dirs, it seems defeating the purpose of mocking.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot imagine people use mock-fs to load lots of real dirs, it seems defeating the purpose of mocking.

Not at all. If we have multiple assets directories or even hard-coded paths on the system, this allows mock-fs to be used in a way that lets us 'mount' those areas and do whatever we like without actually modifying them. This is tremendously useful with a broad range of use-cases.

That is, in fact, the reason I started this PR, but that's really all I can say on that. I don't want to belabor it further.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

without actually modifying them

That's a good use case.

### Manually Creating files

When `config` property values are a `string` or `Buffer`, a file is created with the provided content. For example, the following configuration creates a single file with string content (in addition to the two default directories).
```js
Expand Down Expand Up @@ -95,7 +141,7 @@ mock({

Note that if you want to create a file with the default properties, you can provide a `string` or `Buffer` directly instead of calling `mock.file()`.

### Creating directories
### Manually Creating directories

When `config` property values are an `Object`, a directory is created. The structure of the object is the same as the `config` object itself. So an empty directory can be created with a simple object literal (`{}`). The following configuration creates a directory containing two files (in addition to the two default directories):
```js
Expand Down Expand Up @@ -187,6 +233,18 @@ beforeEach(function() {
afterEach(mock.restore);
```

### Bypassing the mock file system

Sometimes you will want to execute calls against the actual file-system. In order to do that, we've provided a helper.

### <a id='mockrestore'>`mock.bypass(fn)`</a>

```js
// This file exists only on the real FS, not on the mocked FS
const realFilePath = '/path/to/real/file.txt';
const myData = mock.bypass(() => fs.readFileSync(realFilePath, 'utf-8'));
```

## Install

Using `npm`:
Expand Down Expand Up @@ -222,6 +280,4 @@ expect(actual).toMatchSnapshot()
```

Note: it's safe to call `mock.restore` multiple times, so it can still be called in `afterEach` and then manually
in test cases which use snapshot testing.

[![Build Status](https://github.com/tschaub/mock-fs/workflows/Test/badge.svg)](https://github.com/tschaub/mock-fs/actions?workflow=Test)
in test cases which use snapshot testing.
1 change: 1 addition & 0 deletions test/assets/dir/file2.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
data2
1 change: 1 addition & 0 deletions test/assets/dir/subdir/file3.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
data3
1 change: 1 addition & 0 deletions test/assets/file1.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
data1
Loading