quickemu/quickget refactoring

[zen0bit]

Discussed in issue here
Anyone who have anything to say to topic come here...

[zen0bit]

Come to here @flexiondotorg @philclifford @TuxVinyards @dabrown645 @lj3954

[zen0bit]

The quickemu, a command-line interface (CLI) tool for managing and running VMs on GNU/Linux, has grown significantly in size and complexity. With around 3000 lines of code, maintaining and extending the tool has become challenging. This proposal outlines a strategy to refactor quickget and quickemu into a modular architecture, which will improve maintainability, extensibility, and readability of the codebase.

Current State of quickemu
The current quickemu is a monolithic script that handles a wide range of functionalities. While this has been effective for a small number of distros, the complexity has grown to the point where it is difficult to manage and understand the entire codebase.

Modular Architecture
A modular architecture breaks down the scripts into smaller, independent components, each responsible for a specific functionality. This approach allows for easier maintenance, testing, and extension of individual components without affecting the entire system. Here are the key components that can be modularized:

Application Management: Separate the logic for installing, updating, and removing applications into distinct modules.
Dependency Management: Create a module to handle the installation and management of dependencies required by applications.
User Interface: Abstract the user interface into a separate module to allow for different interfaces (CLI, GUI, etc.).
Configuration Management: Encapsulate configuration settings and preferences in a dedicated module.
Update Mechanism: Implement a module to handle updates for operating systems.
Backup and Restore: Create a module for managing snapshots of VMs.
Integration with External Tools: Modularize the integration with external tools like qqX or q.
Benefits of Modularization
Maintainability: Smaller, focused modules are easier to understand, test, and maintain.
Extensibility: New features and integrations can be added as new modules without modifying existing code.
Reusability: Common functionality can be reused across different modules, reducing code duplication.
Scalability: The system can scale more easily as new components can be added without overwhelming the core application.
Flexibility: Different modules can be developed and maintained independently, allowing for parallel development and contributions from the community.
Implementation Plan
Identify Core Functionalities: Begin by identifying the core functionalities that can be modularized. This includes the installation, removal, and update processes, as well as any other distinct features.
Design Module Interfaces: Define clear interfaces for each module to ensure they can operate independently and interact with each other when necessary.
Refactor Existing Code: Gradually refactor the existing codebase into modules, ensuring that each module is self-contained and has a single responsibility.
Testing: Implement unit tests for each module to ensure that they function correctly in isolation and in conjunction with other modules.
Documentation: Update documentation to reflect the new modular structure and provide clear guidelines for contributors.
Community Involvement: Encourage community contributions to develop and maintain modules for different applications and functionalities.
Example Module Structure
Here's an example of how the installation module might be structured:

#!/usr/bin/env bash

Module: Installation

Dependencies: coreutils, wget, curl, grep, sed

Function: install_app

install_app() {
# Logic for installing an application
}

Function: update_app

update_app() {
# Logic for updating an application
}

Function: remove_app

remove_app() {
# Logic for removing an application
}

Main Entry Point

main() {
# Parse command-line arguments and call the appropriate function
}

Execute the main function

main "$@"
By adopting a modular architecture, Scripts can become a more robust, scalable, and maintainable tool. This proposal outlines the initial steps towards modularization, but the actual implementation will require careful planning, design, and execution. The goal is to create a system where each module can be developed, tested, and maintained independently, leading to a more sustainable and collaborative development process.

Next Steps
Assess Current Codebase: Analyze the current codebase to identify the most critical areas for modularization.
Define Modules: Define the scope and responsibilities of each module.
Design Interfaces: Design the interfaces between modules to ensure they can work together seamlessly.
Refactor: Begin refactoring the codebase into modules, starting with the most critical functionalities.
Test: Implement comprehensive testing for each module to ensure reliability.
Document: Update documentation to reflect the new modular structure.
Community Engagement: Encourage community contributions to develop and maintain modules.

PS: Borrowed from different project (which facing same...)

[zen0bit]

For now work started independently on two sepparated repositories.

With quickget

Thanks to @lj3954 and @dabrown645

Liam branch
DaBrown branch

[TuxVinyards]

Wonder if @lj3954 and @dabrown645 are using their spare time following the American Super Bowl? I am reading that a lot of things are grinding to a halt in America because of it. Or perhaps they are still working on things?

"PS: Borrowed from different project (which facing same...)"

Why not. Makes good sense. I had wondered how your English had suddenly improved so much overnight 👍

[zen0bit]

I know that my english could be improved, trying every day...

[zen0bit]

I am interested in creating UI modules (never done any modularization)

will be studying how to make UI module..

Any suggestions?

[TuxVinyards]

Or the other way, which would mean only having to have one file open at a time, would be to have each language file set as couplets.

In other words, you would tag each pair of Vars as VarName_En_US for the first and just underneath that place the second as VarName_Cz_CZ etc

Hopefully the CZ bit is correct, without looking it up, and hopefully you follow ...

[lj3954]

Quickget is too small of a project to translate just about every word as in the example provided by @TuxVinyards. In addition, I don't believe your message variables will at all work out, @zen0bit, since everything outside of a function (including variable assignation) is run the instant you execute a file, which will cause issues when you're relying on other variables which are yet to be assigned. I've added language support to my branch, which should work for all of the messages included in Quickget (excluding anything printed out by OS plugins, that's probably not happening anytime soon). I added a template, as well as vastly improving the directory structure.

[zen0bit]

Pretend no variables will be in languages files. No problem? (Just don't did it yet) Dont't need to use functions in lang files just text as much as posible.

[lj3954]

Removing variables will likely make things significantly more difficult. Functions being in the language file is a suboptimal solution, but I think it's acceptable for now, considering quickget has so few lines of text to output. Later, I'll try to implement a better solution.

[zen0bit]

Pushed some changes to PR

like printf if variables used in UI message and no better way

[zen0bit]

So which quickget refactor we will use?

• DaBrown branch
  or
• Liam branch
  ???

Wanna see votes...

[TuxVinyards]

Both versions have their merits.

I am beginning to wonder, now, if Liam's code is in fact cleaner and better structured than Dab's - I initially thought it to be the other way around. But -caveat- this is only after a read through on GitHub and I still haven't properly decided.

It's only working with the scripts that gives the real answers & I have been working on other things recently.

Are either of these branch ready to test run in some way or other, at this point?

[zen0bit]

Liam's with few distros looks like working. Not tested extensively. I added void, which is working as expected. Different architectures not yet...

[lj3954]

I don't really think any of this is going to go anywhere. Bash just does not support the features necessary for such a project, especially functions between multiple files which cannot really be used independently. Thus, I'm instead working on rewriting a version of quickget in Rust. I don't expect to ever merge it here, especially considering many people (including Martin Wimpress) do not know the language, but it should be an alternative that allows for a lot more flexibility and improved performance. It's still in very early stages, but you can check it out in my new repository.

[zen0bit]

Can't help with rust.
but will try your fork..

[lj3954]

I've added documentation to attempt to make the addition of distros simpler for someone who doesn't know any Rust. I'd like to hear if it's easy enough to understand.

[zen0bit]

Will try later..

[zen0bit]

Whats your current progress @dabrown645?

Looks like I am left alone on this.

Still wanna tweak this bash version..

[flexiondotorg]

I am open to refactoring Quickemu to make it easier to maintain, as opposed to the monolithic scripts we currently have.

I have even considered re-writing it another language.

[zen0bit]

I am open to refactoring Quickemu to make it easier to maintain, as opposed to the monolithic scripts we currently have.

👍

I have even considered re-writing it another language.

👎
Not against
Just I choosed quickemu BECAUSE is written in bash

For learning bash and how to participate in open source projects in general (And of course because I like it)

PS: Liam already started with quickget rewrite in rust

[zen0bit]

Looks like I am only one here, who knows just bit of bash...

[flexiondotorg]

I'm keen to learn Rust, I've dabbled with a simple project of my own. I'm certainly very interested in reimplementing quickget/quickemu in Rust 👍

Something I've been thinking about is carving out the logic that enumerates releases and editions and creating a new project that just creates configuration files for quickget in CI, everyday.

quickget and Quickgui could fetch that config file and only need to download fully resolved URLs. macOS and Windows might still need in app logic.

So I see a fresh implementation of Quickemu being several things:

• Configuration generator
• quickget replacement that consumes the configuration above
• quickemu replacement

The configuration generator could be a new Rust project and the existing Bash quickget can be adapted to injest the configuration it publishes. This would provide a way to start working on more maintainable code now, without having to throw everything out until we're ready to replace it.

@lj3954 I'm very aware that Quickemu needs to grow excellent support for foreign architectures and also run well on non-x86 Linux and macOS.

[lj3954]

I'm pleased to report that I've achieved a booting x86_64 VM with my Rust rewrite of quickemu. The implementation of many features is still yet to be done, and VMs of other architectures don't work yet, but it's a start.

Screencast.from.2024-04-30.20-55-49.mp4

Screencast.from.2024-04-30.22-01-58.mp4

[lj3954]

riscv64 & aarch64 VMs now can boot. riscv64 VMs require a bootloader in the VM directory, so a quickget supporting multiple architectures would need to either extract it from the image or pull it from somewhere else. The resources linked by @TuxVinyards in his wiki were quite helpful in getting all of this set up.

I have a lot left to implement here, but I hope I can get some feedback on how this works, and possibly for someone to test it on a host with a different architecture (especially aarch64).

[flexiondotorg]

That is fabulous! Nice work ❤️

[TuxVinyards]

@lj3954 Gave your rust work a whirl this morning.

I have spent a couple of days sorting out rustup and looking on and off at your code, of recent. The curve is still steep. I have found things that I would like to fix and PR. But my best 'community contribution' at the present point in time is still just to flag them up. 🤷

1. Your code built without problems 👍

2. But I had no experience here on after. Whether I needed the 'deps' folder etc. Or whether it would work only if package installed, or what ....

I copied the binary to my a folder with a VM and got the following errors:

3. I then copied a VM and its conf to the 'release' folder and got the following:

4. I searched your code for 'socket' and found Option<u16>, socketpath: PathBuf) which I changed to u32. But rust-analyzer threw up its hands in horror and told me that I would have to change all kinds of other references from 16 to 32 if I wanted this to work.

Your work all looks clean and organised . You are doing well and I am finding inspiring to look at how you have done things. And thank you for acknowledging the work that I have done and have posted up in my wiki, BTW.

But at this point I gave up and posted this. 🤣

I expect that I will return back but I do find myself continually asking the question 'why?'

Why do I need to keep hitting my head against the wall. We don't need fast code. Qemu and the Linux core-utils do all the lifting. We don't need to able to target Windows or anything else, do we? And what was wrong with the Bash rewrite you did? I thought that was shaping up quite well?

I do worry about the learning curve and number of community contributions that just won't happen if change to rust. It's definitely a good excuse to learn a new language, though ....

[lj3954]

First, you definitely do not need anything in the dependencies folder, nor do you need to move the binary to any specific location. The binary includes everything you need, and the entire package shouldn't depend on anything other than QEMU. What was in your VM config when you got the error regarding the missing guest OS? My config parsing hasn't been perfect. Try to run it with debug enabled as well (-vv) to see the full output. Also, .conf isn't required in the argument, it'll automatically fill it in for you.

The 16 bit unsigned integer isn't what's causing the problem with the path. My program uses a full path in its arguments to QEMU, and apparently QEMU does not accept arguments longer than 108 bytes. This fixes any issue that the original quickemu script had where launching a VM from outside of its directory would not work due to invalid paths. I've pushed out a fix in lj3954/quickemu-rs@3fa2eed, please rebuild and try this scenario again. Paths should now all be relative, so you should not experience the 108 character limit except for very isolated conditions.

The main reason I rewrote this in a different language was maintainability, not speed. Bash scripts are incredibly hard to maintain when you get to the size that quickemu and quickget are starting to get. Yours, for example, is extremely difficult for me to parse. I essentially need grep or some other tool to find where each function is, and even then, there's plenty of cases of magic numbers and other unexplainable code. Rust does obviously have an advantage in speed as well, but much of that is being lost just by calling qemu during argument creation. For example, on my system, the -device check, used to find whether smartcard is supported, takes over 100ms by itself to run.

I did a rewrite of quickget bash, not quickemu. I found that the implementation of new features was just as difficult as with the original script. My rust quickget rewrite didn't go anywhere. It didn't help that I was much more inexperienced with the language when I tried writing that (let's not talk about how I was rewriting the bash script verbatim and completely forgot to use a map at least twice). Either way, I think quickget, in its current form at least, is not something that can be easily rewritten in another language. The major downsides of bash, that everything is done via string manipulation, that you can call functions that might not exist, that the performance is slow, don't matter very much for a tool like quickget, and the programs used for downloading, like curl, are superior to just about anything you could write quickly elsewhere. Martin has expressed interest in the creation of a version of quickget that functions differently, though, which is an interesting idea.

The majority of changes occurring to bash quickemu, are very minor, or are 1 line changes that could be done even more easily in a well organized repository (for example, your changes to macOS CPU arguments, nobody needs to understand a language particularly well to implement this. Take VMs with different architectures. That feature is completely missing from quickemu, and I think it would be rather difficult (and a mess) to implement, since the arguments passed depending on architecture are different. In most other languages, including Rust, you can use enums to easily implement this in an organized manner.

[TuxVinyards]

Should we use Rust for a rewrite of QuickGUI? A lot more people want to learn Rust than Flutter. And Rust has good bindings for GTK.

How easy/hard would a new gui would be to implement? Quickgui has almost become abandonware ... Would it be, in loose simplistic terms, just a case of unbinding this code from Clap? Or would a separate GUI that used the Quickemu API be a better route?

@lj3954 those fixes worked 👍

Try to run it with debug enabled as well (-vv) to see the full output.

I was a bit lost on than bit. Haven't got that far yet.

Take VMs with different architectures. That feature is completely missing from quickemu, and I think it would be rather difficult (and a mess) to implement, since the arguments passed depending on architecture are different. In most other languages, including Rust, you can use enums to easily implement this in an organized manner.

I wrote a load of code about 6 months ago that switched the 'vm_boot' function between machines on the fly, depending on what was in the .conf file.

Most of that function sets up commonly used things like sockets and display which are needed by all machine types. That bit could be pulled out separately.

I don't see that having selectable modules for the the different architectures as being that hard. The big problem that I found was for the quickGET side of things. Needing to unpack strange IMG files and adding in different EDK Var files and such.

In the end I scaled things back for qqX and just left Machine Types as a selectable option in general settings. I generally concluded that running aarch etc on X86 is bit of an academic exercise. They need to be run on native hardware really.

I essentially need grep or some other tool to find where each function is ..

I don't think this is a bash thing. I don't find that bit too hard. For me, VScode has good search abilities. And there's a very good extension that highlights the functions too. Each to his own, and I don't what IDE (?) you use.

Everything has it's pluses and minuses. And not to say that we shouldn't write in Rust, but installing VS Code or Codium, or some other, to be able to work on modularized Bash is a lot less than going full Rustup.

EDIT

Looking at qemu_args.rs just now, I can't see much "yet to be done" as in the (under)statement from a few days ago:

The implementation of many features is still yet to be done, and VMs of other architectures don't work yet, but it's a start.

@flexiondotorg are we going to wake up tomorrow and find version 5 is suddenly a Rust release?

I am having a sudden sense of shock .....

[flexiondotorg]

@flexiondotorg are we going to wake up tomorrow and find version 5 is suddenly a Rust release?

No 🙂

[lj3954]

Would it be, in loose simplistic terms, just a case of unbinding this code from Clap?

It's unnecessary to "unbind" this code from anything. The program could be modified to build as a library crate as well.

Needing to unpack strange IMG files and adding in different EDK Var files and such.

Aarch64 VMs shouldn't require anything other than extracting the IMG/RAW file (if applicable). EDK vars are automatically copied over just as in x86_64, and quickemu-rs supports raw disk types.

I generally concluded that running aarch etc on X86 is bit of an academic exercise.

Quickemu supports macOS VMs, which are quite similar in that way. I think they can be useful for people who don't have access to the hardware and need to test basic applications. Nevertheless, quickemu-rs should support KVM for aarch64 guests on aarch64 hosts, as well as software virtualization for any VM not of the native architecture.

Looking at qemu_args.rs just now, I can't see much "yet to be done"

Spice display is still entirely unfunctional, and features like delete_vm and delete_disk are not implemented yet. I'm also leaving these features for later on; currently, I'm trying to work on a TUI to simplify modifications of the config file.

[lj3954]

Some more progress on the rewrite, I've added support for multiple disk images, ISOs, etc. Here's an example with 11 disks of various sizes mounted in a VM.

Configuration files are also being changed to TOML for more flexibility and more efficient parsing using already existing libraries. Before I merge this branch, I'll use the logic I previously had to parse the old configuration files in order to implement a tool that can migrate said files to TOML, so every previous quickemu VM will remain compatible.

[ArrayBolt3]

Personally having written a lot of professional Bash at my work place, I don't think we really need to rewrite this in Rust. There are a lot of ways Bash can be wielded so that it is safe, and the fact that the codebase is written in Bash makes contributions easy.

That being said, I do like the idea of a refactor having looked at the code myself. Perhaps we can take a page out of Calamares' book and make each OS its own "module" that handles everything related to that OS, and then the quickget and quickemu scripts can simply call into the modules to do the needed jobs?

[lj3954]

When it comes to quickget, I do think that Rust isn't the right language. The fact that bash scripting is almost entirely composed of string manipulation is a huge disadvantage for writing normal programs, but it's beneficial for a tool like quickget which has to parse data from HTML, RSS feeds, or whatever else. I proposed just about exactly what you're referring to regarding modules in quickget a couple of months ago: https://github.com/lj3954/quickemu/tree/refactorQuickget.

Quickemu, on the other hand, has become an unmaintainable mess. The main vm_boot function, for example, is 1100 lines. Adding other architectures would make it even worse, especially given that each OS may need different arguments in many cases depending on the architecture. Major changes to quickemu are not remotely easy, and you can see that very clearly in the recent changes.

Nearly all the changes in the past year have effectively been 1 line, rather it be changes to arguments or in conditional logic. These are changes that would be very simple to make to a rust program, even for someone who hasn't seen a line of the language before. You can see exactly the opposite with quickget. It's had dozens of major commits within the last few months, and despite its size, is not particularly difficult to maintain.

[shawndfisher]

Hi, I was a bit hesitant about bringing this up, however, today I found this discussion (I'm a bit slow) and I think some of the work I've been doing might be useful. I started working on this project when I saw the arosone and icaros PRs dropped and decided to add them. During the addition of those two distributions, I thought it might be helpful to decouple the main working code from the distribution data. I started by adding a template which had these attributes/methods

1. dist_releases function analogous to release_${OS}
2. dist_editions function analogous to editions_${OS}
3. dist_image function analogous to get_${OS}
4. dist_validate function as a generic handle_missing * I failed to add to my current decouple branch
5. dist_vm_option_tweak to adjust options in the conf file
6. dist_image_handler to [un|b|g]zip files and run additional post d/l transformations

The idea I was going for here was to make it easy to add distributions without having to modify the core functionality of quickget. I think can also carry over to quickemu by allowing us to adjust functionality inside the distribution templates as well, making it easier to concentrate on getting everything working right without having to worry about breaking other distributions.

Some things work nicely with these changes, others need a bit more work, I have an embarrassing macos template that /seems/ to work and windows* distributions are currently broken and what I am working on

I've had a lot of fun working on this, but completely understand if it's not your cup of tea (I've using source to pull in the distribution's functions, which may be seen as a security risk) or if you'd like to change it to another language.

If you are interested: https://github.com/shawndfisher/quickemu/tree/decouple

Oh, I've also added a few distributions: 9front, arkane, dsl, nomadbsd and a few others...

[flexiondotorg]

I like your approach 🙂 I would certainly be interested in seeing a pull request against quickemu master that implements such a decoupling regardless of what we might do with a Rust implementation.

I am hoping to release 4.9.3 in the next day or so; I have a branch of code cleanup I want to test and merge. After that, I'd welcome your work 👍

[lj3954]

My rewrite now supports monitor_cmd (without any external dependencies) as well as --kill, macOS CPU arguments based on CPU support, and (if I'm not mistaken) all of the fixes that should allow for a macOS host. Of course, it uses libraries which already support other platforms for system calls, so not much needed to be implemented there. I believe that support for Spice displays is the only thing needed to be feature-complete with the bash quickemu.

[flexiondotorg]

The work I've been doing on Quickemu and Quickget recently may appear to contradict the discussion here about re-implementation in Rust.

That is not the case. I want to fill in some of the larger gaps in Quickemu using the existing code, so we have a broader reference implementation to work from.

[lj3954]

Of course. Multiple projects can be worked on simultaneously.

By now, I've completed the initial featureset of quickemu-rs. Everything, or just about, from the bash version of quickemu is re-implemented. Therefore, I'm starting work on a config generator for future quickget in a new branch, https://github.com/lj3954/quickemu-rs/tree/implement_quickget. I'd like to lay down some of my ideas and get input on them. You can also instead read the ideas in the code: https://github.com/lj3954/quickemu-rs/blob/implement_quickget/quickget_ci/src/store_data.rs.

First, I think the data should be arranged in one JSON-formatted file. Even with hundreds of operating systems, releases, and editions supported, it shouldn't come out to more than a few dozen KB, which shouldn't be a problem to download. Implementations of quickget should probably store this data on disk, since it's likely that a user will run quickget multiple times in a day, and only re-download the configurations if the file hasn't been modified since the time the CI runs, which needs to be a constant value stored within quickget.

Each entry should contain an architecture, guest os, optional release/edition fields, and arrays of ISO, IMG, Fixed ISO, Floppy, and Disk images, with only the last being mandatory. Each can either use a filename with no creation (like disk.qcow2), a web source, or a source that quickget will have to handle itself (such as Windows ISOs). Disks will also contain an optional size and mandatory format field.

[lj3954]

Screencast_20240527_223159.webm

Here's the first example of a configuration generated by the new quickget_ci project.

[lj3954]

I've also reduced the amount of default fields being serialized. Disks won't be serialized if only the default disk.qcow2 is present, nor will the x86_64 architecture or linux guest os. Doing that reduced the file size of the FreeBSD entries from 6.8KB to 5.3KB, and would have even more of an effect on linux distros which only offer x86_64 releases.

[lj3954]

Seemingly daily progress report on the quickget configuration generator: Any entry with any field containing a URL which cannot be resolved will be discarded. The user will never have to deal with quickget resolving to nonexistent URLs.

Nearly all the code is asynchronous now as well, which probably isn't too important given that it's planned to run it in CI, but I still enjoy the pointless optimizations. On my connection, FreeBSD, which makes 2 sequential requests first, takes <400ms to create and validate all entries. Of course, information will be fetched for all distros simultaneously.

[flexiondotorg]

I am back home after a two-week trip to the USA. Thanks for the updates. Excellent progress and the idea of building configuration in CI is solid.

[lj3954]

Most operating systems aren't implemented yet, but I set up the actions a couple days ago to release configurations daily (~midnight UTC): https://github.com/lj3954/quickget_configs

I've also created a basic quickget-rs which produces TOML configurations that quickemu-rs can read, using the zstd-compressed configurations. https://github.com/lj3954/quickemu-rs/tree/main/quickget/src

[flexiondotorg]

There is a lot of good discussion here, but I am going to close this discussion and ask that we have a more focused discussion about the future direction of the project here:

• Rusty Quickemu and more CI #1322