Adding support for the Julia language

[Nosferican]

Planning to take a look at it in case anyone else has the same idea, we can be aware of the efforts.

[StoneT2000]

I'm not aware of anyone doing this so feel free to work on it! You can look at the C++ simple kit for the basic structure we would like for a kit.

[Nosferican]

I might push a draft soon to get some feedback but I have mostly followed the Python kit. The object-oriented makes it slightly awkward so minor changes to use multiple dispatch but seems to be moving along. I have seen minor differences from the implementation and the public API definition but looking at the kit helped me figure things out.

[Nosferican]

I have opened as a draft PR, so it isn't ready for a complete review, but would regardless appreciate some comments on how to start testing a new bot/port. I guess I would need to get a custom version of the CLI?

[StoneT2000]

Since Julia can be compiled, as long as you can compile the bot code into a file, eg main.out, then the CLI tool will try to execute it by running ./main.out, so no customization necessary

So a test command would be

lux-ai-2021 main.out main.out --replay=out.json

[StoneT2000]

Our team also is not fluent with Julia so if you can document the differences with the python API that would be great, and then feel free to implement however you like that best suits julia programmers.

[Nosferican]

I think I understand the last component of the pipeline now. Probably the easiest solution would be to use PyJulia...
Something like

import julia
julia.include("file.jl")
# other logic
commands = julia.agent(observations)
# other logic

[StoneT2000]

Can you not directly execute compiled Julia code? Or does it not compile to machine code

[Nosferican]

One could potentially use PackageCompiler (see this section) to create an app that can be sent and run on other machines without Julia being installed on that machine. However, at least when looking at the Kaggle submission / environment and the Docker file it seems to be a lot easier to:

1. Install Julia calling a file like install_julia.sh (might need to chmod first)

#!/bin/bash

JULIA_VERSION="1.6.3"

JULIA_VER=`cut -d '.' -f -2 <<< "$JULIA_VERSION"`
BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"
URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"
wget -nv $URL -O /tmp/julia.tar.gz
tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1
rm /tmp/julia.tar.gz

pip3 install --quiet julia

2. Call Julia from Python with these lines

import julia
julia.install()

from julia.api import Julia
jl = Julia(compiled_modules=False)

from julia import Base
from julia import Main

3. You can then add your starter/code and agent implementation like

from julia import Pkg
Pkg.add(url = "https://github.com/owner/LuxAI.jl", rev = "main") # Repo with agent implementation

Main.eval("using LuxAI")
from julia import LuxAI

4. Interact with the Julia process

...
commands = LuxAI.commands(observation["updates"])
...

I guess once I figure out the last kinks in the starter code either way would work.

[StoneT2000]

Ah so compiled julia code still requires some kind of julia runner, in which case the pyjulia looks like a better option.

[Nosferican]

As compiled code, it wouldn't require Julia, but at least for development, it is likely that just calling Julia from Python for that one line in that file to be an easier experience. I am testing making the executable files and as an option and basically people could do the same set up at the end regardless. I was mostly thinking of making it easier for folks to dev / work with the code. That way, they don't need to create the executables but have the option.

[StoneT2000]

Sounds great! I only had concerns over potentially speed and I wasn't sure how pyjulia works (e.g. is it transpiling? is it 100% accurate?)

[Nosferican]

I have had some experience calling Python and R from Julia as well as Julia from R and Python... Not too much of that anymore these days. In general, it's quite useful and works well in most cases with rare exceptions. In this case, I imagine it would be best case scenario since the only thing being passed around is a String.

[Nosferican]

I think I am about done with the startkit... Will push soon. I am building the test suite using

replay_game = JSON3.read(read(joinpath("data", "PySimpleS0.json"), String))

replay_game being the output of

lux-ai-2021 --seed=0 --statefulReplay kits/python/simple/main.py kits/python/simple/main.py

Using that to get verify the state of the game at initiation and how it progresses.
I am also using

observations = readlines(joinpath("data", "inputs.jsonl"))

which isn't technically a JSONL file but is just a capture of the read_input() lines the Python agent got during that game.

Inspecting that game, the first observation that is received from the API has lines

 "r uranium 15 15 341"
 "u 0 0 u_1 2 8 1 60 0 0"
 "u 0 1 u_2 13 8 1 60 0 0"
 "c 0 c_1 0 23"

meaning each player has their initial worker. Based on
https://github.com/Lux-AI-Challenge/Lux-Design-2021/blob/512357eda1fe22600c6281f31f88b386044ce8b6/kits/python/simple/lux/game.py#L55-L65

Worker u_1 has a cooldown at 1 and thus can't take an action. When looking at the commands sent to the API,

{
   "command": "m u_1 e",
   "agentID": 0
}
 {
   "command": "m u_2 w",
   "agentID": 1
}

Worker u_1 did actually take an action to move east. I might have skipped something so wanted to make sure my understanding of how to use the API components thus far was correct. Thanks!

[ziong]

@Nosferican Thanks for your time and effort to create the julia kits.
btw, I have a question, how to deal with the Julia Packages issue?
does it has a list of preinstalled julia packages? but how about if extra packages / different versions are required?

I am not sure but is it possible to create a compile.sh which let us to install required packages?
but it has a prerequisite that the container has the internet access / an internal package repository for downloading the packages.

or can we create a custom DockerFile together with submission to customize the container image that running the bot (agent)?

and for using pyjulia approach, seems that command call via python (main.py) approach, e.g. in the java kits, is more intuitive to me. any benefit using pyhulia instead of command call?

[Nosferican]

Currently for the Julia simple kit I only use one package for parsing JSON (JSON3.jl). When I compile the Julia code into the executable it is embedded as part of the machine code files so everything is self-contained. Technically, depending on the options for compiling the app, you can use the base system image which includes the standard libraries, make a trimmed version including only the standard libraries the code uses or a mix with any additional package you may want the code to depend.

The Julia packages are listed in a Project.toml and Manifest.toml files. The Project.toml gives the dependencies and compatibility requested. The Manifest.toml is the "lock-file" as it allows to get and generate the exact environment. Absent the Manifest.toml, it is generated from the Project.toml file. Users only need to run using Pkg; Pkg.instantiate() in an environment with the Project.toml file to install and set up everything. You can include that as part of a step in the Dockerfile if you want. Currently, I have only included a RUN step to download the Julia binaries / checksum and add to path through a symlink in /usr/bin which is similar to the ones for the other languages.

I guess have warmed up to the command call due to the PyJulia installation quite option not working at the moment for some reason (JuliaPy/pyjulia#450). Without selecting the quite option it prints and pollutes the stdout which messes up with the API instead of keeping it strictly to the commands / actions.

I have tested the command line tool by feeding the observations using the shell and everything seems to be working well. I am able to receive the commands as expected and such. When I try it from the main.py, I get no commands from the Julia agent which is the last piece of the puzzle to solve. I have asked about the issue here. Basically, there is something going on with either eof signals or the flush or something along those lines that for some reason is making the stdin and stdout between Python and the app subprocess differ from the shell experience. Still, I have picked up a few skills in this project... It's almost ready but still missing that last element to fix.

[ziong]

Currently for the Julia simple kit I only use one package for parsing JSON (JSON3.jl). When I compile the Julia code into the executable it is embedded as part of the machine code files so everything is self-contained. Technically, depending on the options for compiling the app, you can use the base system image which includes the standard libraries, make a trimmed version including only the standard libraries the code uses or a mix with any additional package you may want the code to depend.

are you mean to use "PackageCompiler.jl" to build the custom julia sysimage ( or standalone executable?) and included it in the submission zip file? ) (link)
if yes, the sysimage is quite large (~180MB incremental / ~100MB non-incremental in general), will it be an issue for that large submission file?
additionally, actually it may has some problems due to incomplete precompile execute statements set. As you know, the "compile" of julia is depended on the list of execute statements to "cache" the precompiled code into the sysimage, (usually using the test suit). When in run time, if the statement is not "seen", it still requires to look up the local package installation to compile on the fly, and if such package is not installed locally, it will be crashed. This may works if the application executes relatively static but not for all cases, especially for some complex packages, not all logic may be reached in test suit.
so, in my opinion, "PackageComplier.jl" is use for accelerating the warnup time but not intend (or not easily) for application deployment / relocation (although it claims it can do so.)

The Julia packages are listed in a Project.toml and Manifest.toml files. The Project.toml gives the dependencies and compatibility requested. The Manifest.toml is the "lock-file" as it allows to get and generate the exact environment. Absent the Manifest.toml, it is generated from the Project.toml file. Users only need to run using Pkg; Pkg.instantiate() in an environment with the Project.toml file to install and set up everything. You can include that as part of a step in the Dockerfile if you want. Currently, I have only included a RUN step to download the Julia binaries / checksum and add to path through a symlink in /usr/bin which is similar to the ones for the other languages.

yes, Pkg.instantiate() is good idea, but I am not sure the bot container can access the Internet to download the package on the fly. (@StoneT2000, can you clarify the container have internet access when running the match?)
i think you are referring to this Dockerfile , however i cannot notice any change related to the julia kit in the julia-bot branch, but I understand what you are taking about. I think the julia case is quite similar to python, there is 2 preinstalled pyhon packages in the Dockerfile, numpy and scipy, also with these preinstalled libraries. For packages not listed, this post suggested to install locally and included in the submission file. But I think that there is no equivalent local installation method in the julia's package ecosystem.

therefore, summarize all "PackageComplier.jl", "preinstall in Dockerfile", "pkg.instantiate()" & "local installation" approaches, seems still cannot completely resolve the additional packages issue.
I don't know that is it possible to add a post install script to tackle this issue, such as in Dockerfile:
RUN if [ -f "#somewhere#/post_install.sh" ]; then #somewhere#/post_install.sh; fi
#somewhere# should be container mount point of the root of the submission. Please comment.

I guess have warmed up to the command call due to the PyJulia installation quite option not working at the moment for some reason (JuliaPy/pyjulia#450). Without selecting the quite option it prints and pollutes the stdout which messes up with the API instead of keeping it strictly to the commands / actions.

I have tested the command line tool by feeding the observations using the shell and everything seems to be working well. I am able to receive the commands as expected and such. When I try it from the main.py, I get no commands from the Julia agent which is the last piece of the puzzle to solve. I have asked about the issue here. Basically, there is something going on with either eof signals or the flush or something along those lines that for some reason is making the stdin and stdout between Python and the app subprocess differ from the shell experience. Still, I have picked up a few skills in this project... It's almost ready but still missing that last element to fix.

Understand that, thanks for your clarification.
I really excited to use julia to join the competition.
Thanks for your effort again.

[StoneT2000]

There is not internet connection in the production containers. If there is, it's a bug and will probably be removed. I recommend trying to find a way to package all your libraries into your submission (which should be doable afaik)

Moreover this way you don't lose any time in the match.

[StoneT2000]

I would be very surprised to see that Julia does not have any kind of local packaging system, that feels incredibly poorly designed, how would you develop local packages?

[Nosferican]

For the compiled version once everything is running well, the ideal would be to generate the image trimming down any unused standard library (e.g. LinearAlgebra) to make it as small as possible while running the codebase with a test / burn script (feeding it the first two observation lines of a game such that those are also compiled for the image). The packages for Julia would already be embedded in the app so no need to download / install anything at that step. The Pkg.instantiate() would be for the dev / testing experience of users implementing their agents. Once they bake it into the application, there is no need for downloading the packages later on. The steps would be

1. Open repo in a container (e.g., Visual Studio Code for example that generates the same environment as the competition).

• This environment would already have Julia installed as that would get added to the Dockerfile

2. The user would then install any other packages they may need to implement / train their agent.

• They can add any packages they might need using the Julia package manager without issues.

3. Use PackageBuilder to create the executable file which gets compressed and used in the submission.

• In this step, all packages and code for the implementation get compiled so during the execution of the agent in the competition, nothing needs to downloaded / installed. Using a burn script to trigger the compilation of the code will also save a few seconds at initialization.

Hopefully that makes things a bit clearer. If not, let me know and I can try to clarify it.

[ziong]

I would be very surprised to see that Julia does not have any kind of local packaging system, that feels incredibly poorly designed, how would you develop local packages?

sorry for my misleading wordings, the "local" it means in project level / run time level, contrast to the system wise level (global), when u developing / editing an package, it still have a "local" copy in your project directory but not including its' dependences. Those dependence packages still in "global".
actually, I don't think it is poorly design, just because it is not supposing to redistribute without internet / network. The beauty of this design is that you won't have multiple same copies of a package across all installed packages, because each package own its' dependences copies, just like in python. I don't drill down on this, it's a bit off topic. you can find more information here

[ziong]

For the compiled version once everything is running well, the ideal would be to generate the image trimming down any unused standard library (e.g. LinearAlgebra) to make it as small as possible while running the codebase with a test / burn script (feeding it the first two observation lines of a game such that those are also compiled for the image). The packages for Julia would already be embedded in the app so no need to download / install anything at that step. The Pkg.instantiate() would be for the dev / testing experience of users implementing their agents. Once they bake it into the application, there is no need for downloading the packages later on. The steps would be

1. Open repo in a container (e.g., Visual Studio Code for example that generates the same environment as the competition).

• This environment would already have Julia installed as that would get added to the Dockerfile

1. The user would then install any other packages they may need to implement / train their agent.

• They can add any packages they might need using the Julia package manager without issues.

1. Use PackageBuilder to create the executable file which gets compressed and used in the submission.

• In this step, all packages and code for the implementation get compiled so during the execution of the agent in the competition, nothing needs to downloaded / installed. Using a burn script to trigger the compilation of the code will also save a few seconds at initialization.

Hopefully that makes things a bit clearer. If not, let me know and I can try to clarify it.

I have an another idea, I think we can make use of the DEPOT_PATH to point the package repo to running directory.
this will act like "local" package installation.
the submission directory structure like this:

[submission root]
|- main.py
|- main.jl
|- Project.toml
|- Manifest.toml 
|- lux
|    |- src
| - depot  <--- new, optional
|- my_sys.so    <--- optional, custom compiled sysimage

step to create a submission file, it is only needed when additional packages is required:

1. create a docker image with julia (like c++)
2. run a container with the image and mount the submission root directory to the container (e.g. /mnt/simple)
3. set env variable JULIA_DEPOT_PATH to add the local depot directory,
   export JULIA_DEPOT_PATH="/mnt/simple/depot;$JULIA_DEPOT_PATH"
4. run package instantiate / package build (all packages and dependences will be downloaded to the depot directory)
5. compile a new custom sysimage (this step is optional, mainly for acceleration)
6. zip /mnt/simple to a submission file

corresponding changes in main.py,

1. update the DEPOT_PATH (either via the env variable, or update the DEPOT_PATH directly) to the depot directory, if "depot" directory is exist.
2. using the my_sys.so as the sysimage, if it is exist.

Together with adding some selected preinstall packages in the match container Dockerfile, I think this can completely solve the package dependence issue, even it is a bit complicate.
for someone don't need additional packages, they can just make a zip file for submission.
for someone want additional packages or another version, they can add a local "depot".
for someone want compiled the code no matter for efficiency / packed all in one executable, they can make a new sysimage;
The only concern is the submission file size.
@StoneT2000 may I know that any limitation on the submission file size?
@Nosferican do you think it is feasible?

[StoneT2000]

I think submission size can be upwards of a few hundred megabytes

[StoneT2000]

Hi, just bumping this thread up to see if there needs any more help / contributions from the Lux AI team (or anyone else)

No rush though, thanks again for initiating this

[Nosferican]

No worries. Had a few deadlines for work this week that eat some of my time but plan to take a look at it today.

[Nosferican]

Mea culpa. I was out of town the last 10 days or so. Remind me at the end of the week if I haven't pushed an update. Should be getting to it this in the next couple days.

[StoneT2000]

Bumped! @Nosferican

[Nosferican]

Thanks for the bump. I have not forgotten but have had a pretty eventful few days. Will try to get back it soon.