Flat-File version

[konobi]

It would be great to get a version of this that works against flat files, available either locally on disk or over http/ftp. This would allow for npm mirroring using rsync and most normal functionality being available without needing to hit the npm servers directly. These files should be updatable from the central registry and allow for small diffs between updates. Would be great for having local, organization wide mirrors.

Of course, all publishing functionality would still need to go to the npm central registry.

[xmojmr]

@konobi this https://github.com/cnpm/cnpmjs.org project solves similar (same?) problem. Cache compatible with npm clients. It uses MySql instead of flat files. I did not try it myself but looks easy to get up and running and stable.

Quote from their README

What can you do with cnpmjs.org

• Build a private npm for your own enterprise. (alibaba is using cnpmjs.org now)
• Build a mirror NPM. (we use it to build a mirror in China: cnpmjs.org)
• Build a completely independent NPM registry to store whatever you like.
  ...

[rlidwka]

If we're talking about npm registry servers, sinopia uses flat files. You can rsync files between its instances.

Works fine as a private registry, but I doubt the entire npm registry would work fast enough if implemented like that. It's just too many files.

[xmojmr]

@rlidwka thanks for the good link (and the links it contains in similar existing things). I think that we might agree that the original requirement

allow for npm mirroring..and most normal functionality..without needing to hit the npm servers directly
..should be updatable from the central registry and allow for small diffs between updates
..local, organization wide mirrors

• 👍 can be satisfied by a decent lightweight NPM mirroring repository implementing the repository API
• 👎 having such a lightweight NPM flat-file based repository inside the NPM client would not bring any benefit other than being built-in
• 👎 would require nonzero man/days coding/testing effort which could be spend for other tasks
• 👍 using NPM repository compatible solution has the benefit that other NPM tools (not based on this NPM client code) can use it as well

---

BTW: I'm looking for a minimal NPM repository REST API specification. What urls does it have to support. What search queries does it have to support. Do you know where such specification is? I don't mean code implementing it that might be reverse-engineered. I mean an approved document.

I'd like to use it for implementing PHP readonly client (the result should be a mixture of http://nipstr.com and http://component.xmojmr.cz)

[rlidwka]

I don't mean code implementing it that might be reverse-engineered. I mean an approved document.

I'm afraid that would be the entire CouchDB API spec. :(

I could draft a de-facto standard which npm uses now, but it would worth nothing since they can change it anytime.

[xmojmr]

@rlidwka CouchDB API spec (http://docs.couchdb.org/en/latest/intro/tour.html) is not quite the answer I'm looking for. It does not seem to contain npm specific things. e.g. I did not find description of the meaning of '/-/all' which in npm terms seems to mean "list of all packages with some info, but the info is not exactly up-to-date but at least the list length should be ok (and you can't SELECT the fields we'll send you)".

Imagine (it really is a fake example) there is an external company creating a fancy 3D visualization tools and their data-interface engine will be written in C# (because they already have a lots of code in the behind available) etc. and you need to pass their developers a specification of the npm registry interface so that they can connect to it. It should be about 10 pages long (not much longer) pdf with some UML sequence diagrams describing all the basic functions so that they can take it, use it and write working database connector compatible with public npm repository, known npm mirroring software and able to work also with private npm repositories. The specification should be in language&platform neutral terms like HTTP, URL, JSON

What would you give them? or what set of articles they should read? (the now-available alternative that they can install vagrant, download some 700MB image of something, configure their npm repository replica and use futon to dig in the internals is not the nice way I was hoping for 😕)

Maybe I'm asking stupid question, but I'm new to this side of npm and you're the first human I've "met" that seems to know what is this repository all about (upside-down)

[othiym23]

Most of the API is embedded in the code for npm-registry-client and npm-registry-couchapp. I don't think that the actual API used by the two of them is documented outside of the code, but extracting it is probably only a day or two project (n.b.: I am not volunteering for that project). Either of those or https://github.com/davglass/registry-static would be a good place to start if you wanted to build your own npm-like registry.

[rlidwka]

@othiym23 , extracting current api isn't a big deal indeed.

But is there any chance that npm won't be changing its behaviour/API in the future making that documentation outdated and without notice? I've seen that a few times before...