Support dynamic includes

[davidreuss]

Hello,

First of all, jsonnet is really great - it's very good at what it does.

Is there any reason why you can't do something like:

import "whatever_%s.json" % my_var

It would be quite helpful for my usecases, so you can avoid writing scripts, with weird jsonnet invocations, to "emulate" that behavior.

[sparkprime]

I thought about it. The cost is that it is much harder to statically analyze the Jsonnet (e.g. to find out what Jsonnet files are affected by a given library refactoring, and running them to ensure they emit the same JSON). I wasn't sure how valuable the ability to compute imports would be, so err'd on the side of caution.

I'd like to know more about your use case so I can think about the tradeoff.

Previously, I have done things like:

local whatever = {
    a: import "whatever_a.jsonnet",
    b: import "whatever_b.jsonnet",
    c: import "whatever_c.jsonnet",
};

And then you can reference it dynamically just with whatever[my_var]. So at least the expansion is done in one place. And of course generating such a thing in a separate file is very easy :)

[davidreuss]

I guess that makes sense, haven't thought about just importing all known into a local, and referencing that.

For my current use-case i have some dynamically generated structure (like a always changing inventory), where i use jsonnet to augment / expand that further.

To include this properly i now have a script which generates the JSON, and invokes jsonnet so it's using stdin as code.

Something like:

data=$(...)
echo 'import "script.jsonnet" + '$data'' | jsonnet -e -

But maybe i need to rethink how i do this.

I've toyed with just changing the import library path for scripts as well, but that feels clunky too.

[sparkprime]

Can't you write your generated data to a json file with a static filename and then import that from script.jsonnet?

[davidreuss]

Yes, i could - but i wanted to know if there was a nifty solution to avoid managing temporary files. My usecase is in an automated context, so i'd love to avoid copying too much around.

But i think with some import path juggling, that it could probably work okay :)

Sent from my iPhone

On 26/08/2015, at 06.49, Dave Cunningham notifications@github.com wrote:

Can't you write your generated data to a json file with a static filename and then import that from script.jsonnet?

—
Reply to this email directly or view it on GitHub.

[sparkprime]

Brain dump of crazy ideas:

1. import "/dev/stdin"

2. add a feature

local foo = importexec ["echo", "{"x": 1}"];
foo.x

yielding 1, but this would only process JSON

3. write a JSON parser in Jsonnet (yes, this is possible)

local foo = std.parseJson(std.extVar("DATA"));

then run jsonnet with -V DATA="$data" (or use an environment variable and -E)

[davidreuss]

If importing /dev/stdin works then that would solve it, but that would just feel pretty wrong.

I'm in total agreement regarding static configuration, and i'd also expect configuration to not change during the execution of jsonnet, even if reading from say a URL.

Writing a JSON parser could indeed be a solution. I like the importexec solution as well, but i guess it depends on how much you want jsonnet to be able to handle itself, and what should be left to other tooling.

On 26/08/2015, at 07.38, Dave Cunningham notifications@github.com wrote:

Brain dump of crazy ideas:

1. import "/dev/stdin"

2. add a feature

local foo = importexec ["echo", "{"x": 1}"];
foo.x

yielding 1, but this would only process JSON

3. write a JSON parser in Jsonnet (yes, this is possible)

local foo = std.parseJson(std.extVar("DATA"));

then run jsonnet with -V DATA="$data" (or use an environment variable and -E)

—
Reply to this email directly or view it on GitHub.

[oconnorr]

For what it's worth, nix-instantiate supports --arg <name> <value> and --argstr <name> <value> command line arguments. In this case the file containing a Nix expression is expected to be a function, and it passes that function a record of name value pairs collected from the command line. --argstr is the same as --arg except that <value> is always a string (rather than an arbitrary expression) and the string doesn't have to be quoted (with shell escaped quotes).

[sparkprime]

What I can do is:

1. add a -R name=value where value is arbitrary jsonnet code (like --arg in Nix)
2. internally, std.extVar will behave the same if the var was defined with -V, otherwise if it was defined with -R it will actually execute the string in the current VM and return the value as a runtime object.

If you need to know the dependencies of the jsonnet execution, you need to add on any dependencies from code you pass in with -R to whatever the dependencies are in the actual jsonnet files, but this is natural and not hard.

On the other hand I could do a full std.eval function so you could just do std.eval(std.getenv()) and keep using -V. But this would mean the code would be impossible to statically analyse for dependencies. It would also mean the code in the file decides whether the variable contains code or string, instead of the user at the commandline.

[sparkprime]

Ok, implemented that: 0299d84

@davidreuss you should be able to do something like this now:

data=$(...)
export data
jsonnet --code-env data script.jsonnet

and then script.jsonnet can do std.extVar("data") which will yield the json struct

Please let me know if that works out for you :)

[davidreuss]

Oh, wow -- thanks, that's really helpful, thanks a lot. 👍

[SimonTheLeg]

awesome content. For those stumbling across this via google years later like me, I thought I'll leave this here:

You can actually also do the following now

jsonnet --ext-code-file "data=<path-to-your-file>" script.jsonnet

This will also work with super long data files, where your shell might throw an Argument list too long error :)