Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Point in Time as a virtual dependency #511

Open
hadmut opened this issue Aug 27, 2023 · 6 comments
Open

Feature request: Point in Time as a virtual dependency #511

hadmut opened this issue Aug 27, 2023 · 6 comments

Comments

@hadmut
Copy link

hadmut commented Aug 27, 2023

Hi,

this is a feature request.

When declaring a filetask

file somefile => someother do
   do_something
end

it is executed only if either somefile does not exist, or one of the dependencies is newer.

But quite often I have requirements to not have a file get too old, which is not exactly straightforward in Rake.

Proposal:

Allow to define a point in Time to work as a virtual dependency, e.g. something like

d=DateTime.now.prev_month

file somefile => d do
  # recreate the file
end

to not let the File get older than the given date. Should be easy to implement.

regards

@mblumtritt
Copy link

Just use #file_create:

file_create 'dir/my_file.txt' => 'dir' do |f|
  File.write(f.name, 'It is soo easy!')
end

@hadmut
Copy link
Author

hadmut commented Aug 30, 2023

I haven't found any documentation about what file_create does in context of Rake, https://apidock.com/ruby/v2_2_9/Rake/DSL/file_create doesn't even know it. https://ruby.github.io/rake/ neither.

However, the source code in lib/rake/file_creation_task.rb says that this is definitely not the solution to the problem, since it says

# A FileCreationTask is a file task that when used as a dependency will be
# needed if and only if the file has not been created.  Once created, it is
# not re-triggered if any of its dependencies are newer, nor does trigger
# any rebuilds of tasks that depend on it whenever it is updated.


Which clearly says that this is not what's needed here, because this is about recreation of an existing file.

Furthermore, I do not see how a point in time is involved in your example.

Would it be possible by any means, that you simply did not understand at all what I proposed?

@mblumtritt
Copy link

The proposed syntax will not work because the DSL requires to have the prerequisites after the => :/

But you can create a task which checks for the time:

def (task(:outdated)).timestamp
  ts = File.mtime('test.txt')
  Time.now - 60 > ts ? Rake::LATE : ts
end

des 'creates test.txt when older than a minute'
file 'test.txt' => :outdated do |f|
  File.write(f.name, 'FRESH!')
end

@fatkodima
Copy link

But quite often I have requirements to not have a file get too old, which is not exactly straightforward in Rake.

Can you provide a real-world use case?

@hadmut
Copy link
Author

hadmut commented Nov 9, 2023

There's plenty of.

The use case, where I developed the wish to have this feature was creating virtual machines for kvm using the ubuntu cloud image ubuntu-22.04-server-cloudimg-amd64-disk-kvm.img

from http://cloud-images.ubuntu.com/releases/jammy/release/
(or other versions, 22.04 and jammy are variables).

I defined all steps, i.e. download the cloud image, create the cloud-init-script, create the disk image, create keys, if missing, with a Rakefile, using file tasks and dependencies.

Worked great except for a nasty detail:

These cloud images are updated by ubuntu every few weeks to incorporate latest updates, security fixes etc.

The problem is, that Rake can easily have a file task to download the cloud image if it does not exist, but it requires a completely different task structure if that cloud-image becomes too old. I need to have the Rakefile to download a fresh image from ubuntu in both cases: If the file does not yet exist locally, or if it is too old. It does not make sense to use a cloud image file older than e.g. 3 months and to update every single virtual machine built with this image.

There's lots of other cases where things

  • must not get too old
  • must not get older than something which's age is not a file mod date

e.g. updating package dependencies for Ruby Gems, go, or rust programs, where the dependency versions should be updated every now and then.

Another good example is a docker/podman image. These things need to be rebuilt even if they do exist and all dependencies are fulfilled, just because these things cannot be updated, and need to be recreated every now and then.

I do have applications and do create their docker images with geo data such as openstreetmap excerpts and country border files. These files are rather huge, so you don't want to download them with every run or build. But at the same time, you don't want them to become too old, because data like street maps are getting worse when not properly updated.

So it would be really nice and useful to have something like

cloudimage = "ubuntu-22.04-server-cloudimg-amd64-disk-kvm.img"

file cloudimage => (Time.now - 86400*30) do
sh "wget ..."
end

to download the file if it does not exist or becomes older than 30 days. Just use the given time instead of the mtime of a file dependency. Should not be too difficult.

@hadmut
Copy link
Author

hadmut commented Jul 2, 2024

Simpler:

def (task(:outdated)).timestamp
Time.now - 60
end

file 'test.txt' => :outdated do |f|
File.write(f.name, 'FRESH!')
end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

3 participants