How to share artifacts between jobs when running locally?

[bartekpacia]

I want to create file.txt in the first task and then download it from the second task when running Cirrus CLI locally.

.cirrus.yaml

task:
  name: build
  container:
    image: alpine:3.20
  script: |
    echo "Hello from build task on $(uname -a)"
  script: |
    echo "hello world" > file.txt
  example_artifacts:
    path: file.txt

task:
  name: test
  container:
    image: alpine:3.19
  depends_on:
    - build
  script: |
    echo "Hello from test task on $(uname -a)"

Output

$ cirrus run --output simple
Started 'build' task
Started 'Preparing execution environment...'
Preparing volume to work with...
'Preparing execution environment...' succeeded in 1.9s!
Started 'image pull'
Pulling image alpine:3.20...
'image pull' succeeded in 1.3s!
Started 'main' script
echo "hello world" > file.txt

'main' script succeeded in 1.0s!
Started 'example' artifacts
Failed to upload artifacts after multiple tries: rpc error: code = Unimplemented desc = method GenerateArtifactUploadURLs not implemented
Artifact upload via pre-signed URLs is not supported! Falling back to gRPC...
Uploading 1 artifacts for /tmp/cirrus-ci/working-dir/file.txt
Uploaded /tmp/cirrus-ci/working-dir/file.txt
'example' artifacts succeeded in 0.1s!
'build' task succeeded in 4.8s!
Started 'test' task
Started 'Preparing execution environment...'
Preparing volume to work with...
'Preparing execution environment...' succeeded in 1.2s!
Started 'image pull'
Pulling image alpine:3.19...
'image pull' succeeded in 1.2s!
Started 'main' script
echo "Hello from test task on $(uname -a)"
Hello from test task on Linux b2edfc68d23e 6.6.31-linuxkit #1 SMP Thu May 23 08:36:57 UTC 2024 x86_64 Linux

'main' script succeeded in 1.0s!
'test' task succeeded in 3.8s!

Is it possible? How can I download that artifact in the second task?

[edigaryev]

You can use --dirty.

Also see --artifacts-dir.

[bartekpacia]

Thanks @edigaryev. This is some workaround.

Actually I have a specific use-case in mind and would love to hear how you'd approach it.

I have a Java program that I want to compile with modern JDK 21, but then I want to test it on an older JDK 11. I made the following Cirrus config for this purpose:

build_task:
  container:
    image: azul/zulu-openjdk-alpine:21-latest
  create_file_script: |
    cat >Main.java <<EOF
    public class Main {
      public static void main(String[] args) {
        System.out.println("Hello, World!");
      }
    }
    EOF
  compile_script: javac --release 11 Main.java
  binary_artifacts:
    path: Main.class

test_task:
  name: test
  container:
    image: azul/zulu-openjdk-alpine:11-latest
  depends_on:
    - build
  java_info_script: java --version
  run_classfile_script: |
    cd local_artifacts/build/binary
    java Main

It passes when I run it locally like so:

cirrus run --output plain --artifacts-dir local_artifacts --dirty

Output

$ cirrus run --output plain --artifacts-dir local_artifacts --dirty                                                                                                         exit 1
Started 'build' task
Started 'Preparing execution environment...'
Preparing volume to work with...
'Preparing execution environment...' succeeded in 1.1s!
Started 'image pull'
Pulling image azul/zulu-openjdk-alpine:21-latest...
'image pull' succeeded in 10s!
Started 'create_file' script
cat >Main.java <<EOF
public class Main {
  public static void main(String[] args) {
    System.out.println("Hello, World!");
  }
}
EOF

'create_file' script succeeded in 1.0s!
Started 'compile' script
javac --release 11 Main.java
'compile' script succeeded in 1.4s!
Started 'binary' artifacts
Failed to upload artifacts after multiple tries: rpc error: code = Unimplemented desc = method GenerateArtifactUploadURLs not implemented
Artifact upload via pre-signed URLs is not supported! Falling back to gRPC...
Uploading 1 artifacts for /tmp/cirrus-ci/working-dir/Main.class
Uploaded /tmp/cirrus-ci/working-dir/Main.class
'binary' artifacts succeeded in 0.0s!
'build' task succeeded in 14s!
Started 'test' task
Started 'Preparing execution environment...'
Preparing volume to work with...
'Preparing execution environment...' succeeded in 1.0s!
Started 'image pull'
Pulling image azul/zulu-openjdk-alpine:11-latest...
'image pull' succeeded in 1.2s!
Started 'java_info' script
java --version
openjdk 11.0.24 2024-07-16 LTS
OpenJDK Runtime Environment Zulu11.74+15-CA (build 11.0.24+8-LTS)
OpenJDK 64-Bit Server VM Zulu11.74+15-CA (build 11.0.24+8-LTS, mixed mode)
'java_info' script succeeded in 1.0s!
Started 'run_classfile' script
cd local_artifacts/build/binary
java Main
Hello, World!

'run_classfile' script succeeded in 1.0s!
'test' task succeeded in 4.6s!

My question is: what would be the easiest/recommended way to also make this work on Cirrus CI (the cloud)? I read the docs for artifacts instruction, but didn't find a simple way to do it.

[edigaryev]

My question is: what would be the easiest/recommended way to also make this work on Cirrus CI (the cloud)? I read the docs for artifacts instruction, but didn't find a simple way to do it.

You can use dynamic links to the current build's artifacts, explained in the "URLs to the artifacts section" of the Artifacts Instruction section:

Unfortunately, Cirrus CLI doesn't support such links yet, but a possible solution is to parametrize the api.cirrus-ci.com part as an environment variable (e.g. CIRRUS_ARTIFACTS_HOST) and serve the --artifacts-dir directory from the Cirrus CLI too, while injecting the CIRRUS_ARTIFACTS_HOST environment variable into local tasks, so that things would work seamlessly. WDYT, @fkorotkov?

[bartekpacia]

Thanks @edigaryev! I also like that idea. Being able to use artifacts-type tasks while running Cirrus CLI locally (where, by default, the artifacts would be saved to --artifacts-dir) would be a very welcome feature. I could test that my pipeline works correctly without the venerable git commit + git push cycle.

[fkorotkov]

IMO mimic the REST API is a bit too much for the CLI. @bartekpacia you can use either cache instruction or HTTP cache with the CLI to pass files between tasks. In Cirrus CI cloud cache and artifacts are pretty much identical to an extend that artifacts has a longer and guaranteed TTL. For local execution it shouldn't matter that much IMO.