Running opa using a bundle doesn't work likewise doing it with the equivalent folder

[EliuX]

Problem

When I run opa in server mode using a system folder that looks like:

parent
parent/data.json
parent/dm
parent/projects
parent/projects/projects.rego
parent/projects/projects_test.rego
parent/dm/dm.rego
parent/dm/dm_test.rego

All tests run OK and the opa server understand the data namespace (queried with a GET http://localhost:8181/v1/data) as

{
    "result": {
        "parent": {
            "dm": { ... },
            "projects": { ... }
        },
        "employees": [ ... ],
        "organizations": [ ... ]
    }
}

Have in count that the data.json looks like:

{ 
   "employees": [ ... ],
   "organizations": [ ... ]
}

So this is the way I expected in my code for this to work. So far so good.

The problem comes when I compress the parent folder with:

tar --exclude='*_test.rego' -zcvf parent.tar.gz parent

when I query the running server using GET http://localhost:8181/v1/data I realize that all my data.json content was moved into the parent package:

{
    "result": {
        "parent": {
            "dm": { ... },
            "projects": { ... },
            "employees": [ ... ],
            "organizations": [ ... ]
        }
    }
}

So I tried again creating a bundle file that doesn't contain the parent folder:

cd parent
tar --exclude='*test.rego' -zcvf ../parent.tar.gz *
a data.json
a dm
a dm/dm.rego
a projects
a projects/projects.rego

Runned opa in server mode again and realized this time the data.json was not even present

{
    "result": {
        "parent": {
            "dm": { ... },
            "projects": { ... }
        }
    }
}

So basically I realized that I had to create a bundle containing the data.json contained in a folder, but even in that case for the bundle to work I should do one of the following workarounds:

1. Change my whole code, to for instance, use data.parent.organizations instead of data.organizations.
2. Move data.json to the dm package, but the problem persists because when I run it using the bundle I get something like:

{
    "result": {
        "parent": {
            "dm": { 
                  ... ,
                 "employees": [ ... ],
                 "organizations": [ ... ]
          },
            "projects": { ... }
        }
    }
}

but using the folder directly I get something like

{
    "result": {
        "parent": {
            "dm": { ... },
            "projects": { ... }
        }
       "dm": {
                 "employees": [ ... ],
                 "organizations": [ ... ]
          }
    }
}

So the workaround seems to be a change of code again.

Analysis

It doesn't seem a good idea to use 2 versions of the same code, one for running opa in server mode with a folder and another for a bundle, having both the same structure. It would be nice that if you run
tar --exclude='*_test.rego' -zcvf parent.tar.gz parent to any parent folder you get a bundle that runs likewise running it using the folder directly.

[tsandall]

It seems the main issue here is that the data.json file is not being read out of the bundle (for some reason, this seems to be a bug.) Other than that, if you run commands from the directory containign "parent" things should work:

torin:~/loading$ tree
.
└── root
    ├── data.json
    └── dm
        ├── dm.rego
        └── dm_test.rego

2 directories, 3 files
torin:~/loading$ cat root/data.json
{
    "employees": ["bob", "alice"],
    "organizations": ["hr", "it"]
}
torin:~/loading$ cat root/dm/dm.rego
package dm

import data.root.employees

p[x] { x := employees[_] }
torin:~/loading$ cat root/dm/dm_test.rego
package dm

test_foo { p["bob"] }
torin:~/loading$ opa test -v .
data.dm.test_foo: PASS (670ns)
--------------------------------------------------------------------------------
PASS: 1/1
torin:~/loading$ opa run .
OPA 0.9.3-dev (commit 17a3d9c6, built at 2018-10-18T17:49:28Z)

Run 'help' to see a list of commands.

> data
{
  "dm": {
    "p": [
      "bob",
      "alice"
    ],
    "test_foo": true
  },
  "repl": {
    "version": {
      "BuildCommit": "17a3d9c6",
      "BuildHostname": "spacebox.local",
      "BuildTimestamp": "2018-10-18T17:49:28Z",
      "Version": "0.9.3-dev"
    }
  },
  "root": {
    "employees": [
      "bob",
      "alice"
    ],
    "organizations": [
      "hr",
      "it"
    ]
  }
}
>
Do you want to exit ([y]/n)?
torin:~/loading$ tar --exclude="*test.rego" -czvf bundle.tar.gz root
a root
a root/data.json
a root/dm
a root/dm/dm.rego
torin:~/loading$ opa run bundle.tar.gz
OPA 0.9.3-dev (commit 17a3d9c6, built at 2018-10-18T17:49:28Z)

Run 'help' to see a list of commands.

> data
{
  "dm": {
    "p": [
      "bob",
      "alice"
    ]
  },
  "repl": {
    "version": {
      "BuildCommit": "17a3d9c6",
      "BuildHostname": "spacebox.local",
      "BuildTimestamp": "2018-10-18T17:49:28Z",
      "Version": "0.9.3-dev"
    }
  },
  "root": {
    "employees": [
      "bob",
      "alice"
    ],
    "organizations": [
      "hr",
      "it"
    ]
  }
}
>
Do you want to exit ([y]/n)?

[EliuX]

When I running from a the bundle everything is contained in the parent folder (root). In the other hand if I run it from the folder then the data.json is contained outside of parent. Something like:
{

    "result": {
        "root": {
            "dm": { ... },
            "projects": { ... }
        },
        "employees": [ ... ],
        "organizations": [ ... ]
    }
}

Which is not the way it shows to you, which is weird: In your case running opa on the folder takes dm outside root but contains organizations and employees.
I am using this version

Version: 0.9.1
Build Commit: d5ba22b7
Build Timestamp: 2018-08-16T15:46:52Z
Build Hostname: b381b3d1a5ac

[tsandall]

@EliuX can you include the output of something like tree or recursive ls so that I can see how the files are laid out on disk? If you can include the output of tar tzvf on the bundle as well that would be helpful.

[EliuX]

The content of the bundle root.tar.gz is :

drwxr-xr-x  0 eliux  staff       0 Oct 22 15:12 root/
-rw-r--r--  0 eliux  staff    1250 Oct 22 15:15 root/data.json
drwxr-xr-x  0 eliux  staff       0 Oct 22 15:12 root/dm/
drwxr-xr-x  0 eliux  staff       0 Oct 22 15:15 root/projects/
-rw-r--r--  0 eliux  staff    3002 Oct 22 15:21 root/projects/projects.rego
-rw-r--r--  0 eliux  staff    1420 Oct 22 15:12 root/dm/dm.rego

[tsandall]

Can you post similar output for the non-bundle case? In my case I put the data.json file under the root directory and then ran opa run . (where . contains a subdir called root). This is why the data shows up under root for me. If you can post your the files on disk for the non-bundle case it'll help rule out any issues with the normal file loading.

[EliuX]

Can you post similar output for the non-bundle case? In my case I put the data.json file under the root directory and then ran opa run . (where . contains a subdir called root). This is why the data shows up under root for me. If you can post your the files on disk for the non-bundle case it'll help rule out any issues with the normal file loading.

Sorry for the late response @tsandall. Well the content of the bundle is pretty much the content of the folder, except for the test rego files. For that folder I run opa run root and it works the way I expected: the data is in the global scope of data, not under root. This is pretty much the sample scenario I explained at the beginning of the issue, except that I changed the name parent for root. I will check your solution in the next opa release for sure. Thanks.