pyFilesystem support for loading and saving data

[jayvdb]

https://github.com/PyFilesystem

moban currently supports custom filesystem 'git'.

To add more it should adopt a generic fs layer, like hyde.

[chfw]

does it mean wherever moban uses file system, use pyFilesystem?

[jayvdb]

possible, but not a good idea imo.

Also pyfs doesnt support git.

this is only for non-local files, so e.g. deployment could be done with moban.

[jayvdb]

A github driver is at https://github.com/merlink01/fs.github . Not particularly useful as it doesnt support GitLab, but may be use in addition to a git driver, if one is created.

[jayvdb]

When we do switch, there is a jinja template loader, https://github.com/althonos/jinja2-fsloader

[jayvdb]

We could take the Hyde approach and have one custom driver for git, and use pyfs for everything else, and transparently wrap the git repo in a pyfs layer.

[jayvdb]

Created merlink01/fs.github#9 to see if they are interested in GitLab/etc support.

If not, we may need to create a fs.IGitt ourselves.

[chfw]

do we want to expose pyfilesystem2 syntax to our users(downstream projects)?

[jayvdb]

Optional, but yes. Maybe even recommended where a value might be ambiguous without a pyfilesystem2 prefix. It allows users to host their config & templates in any filesystem-like provider, and use a pyflesystem2 URL to tell moban where it is. This is a natural way to say "moban supports any data/file host" without adding/maintaining drivers for all of them. And we inherit the naming conventions of pyfilesystem which prevents any conflicts/ambiguity, and means moban users can use pyfilesystem docs for precise semantics of the URLs. Our users get new file hosts each time pyfilesystem core includes a new one.

The most obvious "win" is that if we can get a git driver for pyfilesystem, we dont need to create special code for mercurial and other VCS: anyone who wants those can create their own pyfilesystem driver.

The standardisation inside moban codebase will also be beneficial. It's support of PyPI packages would be a simple wrapper that can be creating using the core pyfilesystem2 drivers.(I expect it would be an internal driver, not a separately pypi published one).

[chfw]

Yes, pyfilesystem2 actually complements our vision with any location and any format. With jinja2_loader, our template could sit on s3.

I am finding a blocker in practice:

moban scans input and forms a list of things to do, for example: here is a template file, a data file, please render an output. Then moban does it.

moban collects absolute file paths into a data structure. then take them out to read or write.

Essentially with pyfilesystem2: we need it to fulfil two fundamental things:

1. does the file exist?

pseudo code:

with open_fs('pypi://pypi-mobans-pkg/templates') as pypi_fs:
... if pypi_fs.exists('reminder.txt'):
... pypi_fs.abspath('reminder.txt')

2. can I read/write it with an absolute path

fs.write('pypi://pypi-mobans-pkg/templates/reminder.txt', 'my content', 'my encoding')

My concern is:

a. can fs generate a url for any file?
b. can fs action on a single given url?

[chfw]

it is funny: fs.errors.InvalidCharsInPath: path 'C:\Users\VssAdministrator\AppData\Local\moban\moban\Cache\repos\pypi-mobans\config\data.yml' contains invalid characters

https://dev.azure.com/moremoban/moban/_build/results?buildId=484

[chfw]

on python 2: TypeError: paths must be unicode (not str) <-- well, python file system 2 said path should be unicode.

https://travis-ci.org/moremoban/moban/jobs/561675011

[chfw]

@jayvdb , two internal openers for pypi and github were done. but it does not cover:

• pip install your_package
• git clone https://your.repo

because they are overlapping requires command, which does above two. for consistency, I would love to remove requires and simply use "pypi://my-template-package/relative/path" everywhere in .moban.yaml.

[jayvdb]

I think that is ok, as long as it is a new minor release.

I would prefer to keep requires: because it allows aliases which keep the length of references shorter, but I do agree that using full pyfs urls everywhere is very attractive.

We could re-add requires: -like functionality later, possibly by using mountfs and/or multifs.

I'll review the PR #302 later today.

[chfw]

the support is to be released in v0.6.0