feat: enable root remote taskfiles #1347

pd93 · 2023-09-23T16:39:51Z

This PR enables the ability to pass a remote Taskfile URI to the --taskfile flag. This means that your root Taskfile no longer needs to exist locally.

When specifying the --taskfile flag, Task usually assumes that the execution directory is the directory that the root Taskfile resides in. Since this isn't possible with a remote Taskfile, the execution directory will default to the current working directory instead when the Task command is executed.

However, I have also added the ability to change this default directory using the existing --dir flag. This now means that you are able to specify BOTH --taskfile and --dir flags simultaneously - Something that was not previously allowed. The following semantics are now applied:

If --taskfile is specified without --dir, the specified Taskfile is used and the directory will default to the directory containing the specified Taskfile (no change).
If --dir is specified without --taskfile, Task will search the specified directory for a Taskfile and use that as the entrypoint. The directory will be the one specified (no change).
If BOTH --dir and --taskfile are specified, the entrypoint will be the specified taskfile and the directory will be the specified directory.
If NEITHER are specified, Task will recursively search up the filesystem for a Taskfile starting in the current working directory. If one is found, the directory is set to the directory containing the found Taskfile.

Finally, this PR also fixes issues related to resolving the location of included Taskfiles inside remote Taskfiles. For example, if you are using a remote Taskfile that includes a "local" file. The included file will now be resolved as a remote file too and the include will work as expected.

wburningham · 2023-11-29T20:59:32Z

Hi @pd93, I would love to use this feature with the other changes that are in v3.30.1. Could you merge this or rebase off of v3.30.1?

wburningham · 2024-01-10T19:59:08Z

@pd93 just following up again on this PR. Is there feedback (workflow usage details) or coding support I could provide to help get this work across the line?

pd93 · 2024-03-04T18:15:06Z

This PR has taken a long time to complete 😆 I did not initially realise the complexity of the changes that would be required to complete this work. Over the last couple of months or so I have spent many hours reworking this from the ground up. The functionality described in the PR description required several major changes to some core code and I wanted to make sure I'm not breaking anything for existing users.

I've spent the last couple of days testing. I suspect there will still be some issues, but the main changes are now complete and ready to be reviewed 🚀

@vmaerten @wburningham Since you've both been trying out remote Taskfiles recently, would you mind kicking the tyres on this for me? I'm pretty sure it's ready to go now. This should resolve both remote root Taskfiles and includes in remote Taskfiles.

pd93 · 2024-03-04T18:20:09Z

@brianbraunstein Since this PR also implements the ability to use both --taskfile and --dir flags as previously requested by you in #655 (comment), it would be great if you are also able to give this a go with regard to the new stdin reader.

vmaerten · 2024-03-04T20:28:46Z

@pd93 For the includes in remote Taskfiles, I've tested it with two differents setup and it works as expected
Thanks a lot 🔥

wburningham · 2024-03-14T23:03:13Z

I got around to testing this. I ran

task --dir . -t https://gist.githubusercontent.com/wburningham/86586099785bcfd4c0289dba09ae2829/raw/b3fa6add37c947b77c435cb07d7257f8f64c6628/taskfile.yaml```

Where the remote taskfile looks like this:

version: 3

tasks:
default:
cmds:
- cmd: echo "CLI_ARGS = '{{.CLI_ARGS}}'"
silent: true
- cmd: echo "CLI_FORCE = '{{.CLI_FORCE}}'"
silent: true
- cmd: echo "TASK = '{{.TASK}}'"
silent: true
- cmd: echo "ROOT_TASKFILE = '{{.ROOT_TASKFILE}}'"
silent: true
- cmd: echo "ROOT_DIR = '{{.ROOT_DIR}}'"
silent: true
- cmd: echo "TASKFILE = '{{.TASKFILE}}'"
silent: true
- cmd: echo "TASKFILE_DIR = '{{.TASKFILE_DIR}}'"
silent: true
- cmd: echo "USER_WORKING_DIR = '{{.USER_WORKING_DIR}}'"
silent: true
- cmd: echo "TASK_VERSION = '{{.TASK_VERSION}}'"
silent: true


Using the command above, should `TASKFILE_DIR` be the value from `--dir` flag?

pd93 · 2024-03-15T02:12:50Z

Hey @wburningham. Special variables for remote Taskfiles are something I haven't worked on yet. It seems they're pretty broken right now. I will add this to the todo list in #1317 and take a look at these in a future PR.

As long as the ability to access root files remotely and to include other remote files in a remote file is working correctly then I'm happy to merge this as-is.

Using the command above, should TASKFILE_DIR be the value from --dir flag?

To answer your question directly. TASKFILE_DIR is supposed to contain the path to the directory containing the Taskfile that holds the current task. Since a remote file is, by definition, not on the same filesystem, I'd expect it to contain a URL to the remote directory containing the file.

wburningham · 2024-03-20T21:00:15Z

I think I found a bug related to the special variables. I can see TASKFILE_DIR being set to the remote taskfile when the taskfile is specified with the --taskfile flag. If the taskfile is references with an includes directive in a local taskfile then TASKFILE_DIR should be the local taskfile dir (I think, or at least that is how it works today).

Steps to reproduce:

❯ cat taskfile.yaml 
version: 3

includes:
  remote: https://gist.githubusercontent.com/wburningham/86586099785bcfd4c0289dba09ae2829/raw/b3fa6add37c947b77c435cb07d7257f8f64c6628/taskfile.yaml

tasks:
  default:
    cmds: 
      - task: remote:default

❯ go install github.com/go-task/task/v3/cmd/task@v3.34.1

❯ task --yes
CLI_ARGS = ''
CLI_FORCE = 'false'
TASK = 'remote:default'
ROOT_TASKFILE = '/private/tmp/test/Taskfile.yaml'
ROOT_DIR = '/private/tmp/test'
TASKFILE = ''
TASKFILE_DIR = '/private/tmp/test'
USER_WORKING_DIR = '/private/tmp/test'
task: [remote:default] echo "TASK_VERSION = 'v3.34.1 (h1:yAAxUM54zoaHv+OtDnGgkWSVeiRuaOCn1lPUXPQQA0o=)'"
TASK_VERSION = 'v3.34.1 (h1:yAAxUM54zoaHv+OtDnGgkWSVeiRuaOCn1lPUXPQQA0o=)'

❯ go install github.com/go-task/task/v3/cmd/task@46470038b134034edf37c0926072fc094822384d

❯ task --yes
CLI_ARGS = ''
CLI_FORCE = 'false'
TASK = 'remote:default'
ROOT_TASKFILE = '/private/tmp/test'
ROOT_DIR = '/private/tmp/test'
TASKFILE = 'https://gist.githubusercontent.com/wburningham/86586099785bcfd4c0289dba09ae2829/raw/b3fa6add37c947b77c435cb07d7257f8f64c6628/taskfile.yaml'
TASKFILE_DIR = 'https:/gist.githubusercontent.com/wburningham/86586099785bcfd4c0289dba09ae2829/raw/b3fa6add37c947b77c435cb07d7257f8f64c6628'
USER_WORKING_DIR = '/private/tmp/test'
task: [remote:default] echo "TASK_VERSION = 'v3.35.2-0.20240310214338-46470038b134 (h1:tHuecP/JTRy2ehr+s0Wo+GilpuUqtFLEHb79pCDVias=)'"
TASK_VERSION = 'v3.35.2-0.20240310214338-46470038b134 (h1:tHuecP/JTRy2ehr+s0Wo+GilpuUqtFLEHb79pCDVias=)'

Notice how TASKFILE and TASKFILE_DIR change between the two versions.

andreynering

🚀

andreynering · 2024-03-21T01:31:24Z

docs/docs/experiments/remote_taskfiles.md

+locally at all. For example:
+
+```shell
+task --taskfile https://taskfile.dev hello


We could consider having a proper Taskfile being served in the website, so copying and pasting would work 🙂

Could be a GitHub or GitHub Gist URL as well...

Suggested change

task --taskfile https://taskfile.dev hello

task --taskfile https://taskfile.dev/taskfiles/hello.yml hello

Just an idea, does not need to be done on this PR...

Not entirely clear what you mean by this

We could consider having a proper Taskfile being served in the website

Is this not already complete in this PR? see https://github.com/go-task/task/pull/1347/files#diff-eea9e58f9afa47c7cdf1393b2d383a4713f91171f6603ced3937cacc09fe9376

I will merge for now and we can adjust later if need be.

Oh, got it, it's already there in the root 🙂

It just surprised me that it automatically fetches the /Taskfile.yml path. I thought the whole path would have to be explicit...

This is what the RemoteExists function is doing. It's similar to what we do with local files where only a directory is given. Task will look around for a file that makes sense. I thought this would be nice for projects that want to use Taskfiles to do some setup work. e.g.

task --taskfile https://myproject.org init or something similar is a really nice, clean way of asking your devs to do run some setup task. Whereas task --taskfile https://myproject.org/Taskfile.yml init is a bit less tidy.

The same applies to the new taskfile.dev/Taskfile.yml file. If a user wants to check that their installation of Task works, they can now run task --taskfile https://taskfile.dev and it will reply with "Hello Task!". This is a bit less clunky than task --taskfile https://taskfile.dev/Taskfile.yml.

Happy to revisit this if you don't like the behaviour though.

It's actually an interesting feature, we can keep it 👍

I did quick review at the time, I so end up not paying attention to that function 🙂

I just suggest to improve the documentation a bit to explain that this is a possibility, just like for local files.

andreynering · 2024-03-21T01:33:38Z

taskfile/taskfile.go

+	}
+
+	// Request the given URL
+	resp, err := http.DefaultClient.Do(req)


Does these HTTP calls have a timeout by default? It would be desired IMO.

Good shout! I've added the existing timeout functionality in the HTTO node.Read(...) to taskfile.RemoteExists(...)

pd93 mentioned this pull request Sep 22, 2023

Remote Taskfiles experiment #1317

Open

15 tasks

pd93 marked this pull request as draft September 24, 2023 01:58

pd93 force-pushed the remote-root-taskfiles branch from 8d03ad3 to f2d66a6 Compare December 20, 2023 23:49

pd93 force-pushed the remote-root-taskfiles branch from f2d66a6 to 796f435 Compare February 13, 2024 01:07

pd93 changed the base branch from main to stdin-node February 13, 2024 01:07

pd93 force-pushed the remote-root-taskfiles branch from 796f435 to aa9e86e Compare February 19, 2024 18:38

pd93 force-pushed the stdin-node branch from db06d84 to 3df49ab Compare February 19, 2024 18:40

pd93 force-pushed the remote-root-taskfiles branch from aa9e86e to 2cf4a41 Compare February 19, 2024 18:41

pd93 mentioned this pull request Feb 19, 2024

Read Taskfile from stdin #655

Closed

Base automatically changed from stdin-node to main February 22, 2024 20:59

pd93 force-pushed the remote-root-taskfiles branch from 2cf4a41 to 75d94ef Compare February 22, 2024 21:11

pd93 added the state: wip A work in progress. label Feb 29, 2024

pd93 force-pushed the remote-root-taskfiles branch from 75d94ef to 6e6e262 Compare March 4, 2024 11:42

vmaerten mentioned this pull request Mar 4, 2024

fix(remote): include a Taskfile in a remote's one #1532

Closed

pd93 marked this pull request as ready for review March 4, 2024 18:05

pd93 removed the state: wip A work in progress. label Mar 4, 2024

pd93 force-pushed the remote-root-taskfiles branch from f420aca to 3de5e97 Compare March 10, 2024 21:31

andreynering approved these changes Mar 21, 2024

View reviewed changes

pd93 added 4 commits March 25, 2024 19:06

feat: root remote taskfiles

ed3bcf4

feat: don't send entire include to node resolvers

62f9d67

refactor: rename node resolver methods

5a67b5a

refactor: slightly improved NewBaseNode API

ad8324f

pd93 added 2 commits March 25, 2024 19:07

docs: remote root taskfiles

ef53e6c

feat: use timeout in RemoteExists function

b346ace

pd93 force-pushed the remote-root-taskfiles branch from 8ef8d45 to b346ace Compare March 25, 2024 19:07

pd93 merged commit 64b7d34 into main Mar 25, 2024
13 checks passed

pd93 deleted the remote-root-taskfiles branch March 25, 2024 19:13

pd93 added a commit that referenced this pull request Mar 25, 2024

chore: changelog for #1347

bb68fb3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: enable root remote taskfiles #1347

feat: enable root remote taskfiles #1347

pd93 commented Sep 23, 2023 •

edited

Loading

wburningham commented Nov 29, 2023

wburningham commented Jan 10, 2024

pd93 commented Mar 4, 2024

pd93 commented Mar 4, 2024 •

edited

Loading

vmaerten commented Mar 4, 2024

wburningham commented Mar 14, 2024

pd93 commented Mar 15, 2024

wburningham commented Mar 20, 2024

andreynering left a comment

andreynering Mar 21, 2024

pd93 Mar 25, 2024

andreynering Mar 25, 2024

pd93 Mar 25, 2024

andreynering Mar 25, 2024

andreynering Mar 21, 2024

pd93 Mar 25, 2024

	task --taskfile https://taskfile.dev hello
	task --taskfile https://taskfile.dev/taskfiles/hello.yml hello

feat: enable root remote taskfiles #1347

feat: enable root remote taskfiles #1347

Conversation

pd93 commented Sep 23, 2023 • edited Loading

wburningham commented Nov 29, 2023

wburningham commented Jan 10, 2024

pd93 commented Mar 4, 2024

pd93 commented Mar 4, 2024 • edited Loading

vmaerten commented Mar 4, 2024

wburningham commented Mar 14, 2024

pd93 commented Mar 15, 2024

wburningham commented Mar 20, 2024

andreynering left a comment

Choose a reason for hiding this comment

andreynering Mar 21, 2024

Choose a reason for hiding this comment

pd93 Mar 25, 2024

Choose a reason for hiding this comment

andreynering Mar 25, 2024

Choose a reason for hiding this comment

pd93 Mar 25, 2024

Choose a reason for hiding this comment

andreynering Mar 25, 2024

Choose a reason for hiding this comment

andreynering Mar 21, 2024

Choose a reason for hiding this comment

pd93 Mar 25, 2024

Choose a reason for hiding this comment

pd93 commented Sep 23, 2023 •

edited

Loading

pd93 commented Mar 4, 2024 •

edited

Loading