Add support for uploading to Arweave #163
Conversation
|
Can you find a different solution for uploading? I'm not comfortable adding this many additional dependencies as well as a platform that other users haven't requested. |
That being said, I do believe there is a compromise that might be able to address all of this. Instead of bundling this into the import (
"fmt"
"slices"
"net/url"
"plugin"
)
type Uploader interface {
Upload(*log.Logger, string, string, string, int) error
}
// Upload a pmtiles archive to a bucket.
func Upload(logger *log.Logger, input string, bucket string, key string, pluginPath string, maxConcurrency int) error {
parsedUri, err := url.Parse(bucket)
if err != nil {
return fmt.Errorf("unable to parse bucket uri: %w", err)
}
if slices.Contains(GcloudSupportedSchemes, parsedUri.Scheme) {
return uploadWithGoCloud(logger, input, parsedUri, key, maxConcurrency)
} else if pluginPath != "" {
return uploadWithPlugin(logger, input, parsedUri, key, maxConcurrency)
} else {
return fmt.Errorf("scheme %q is not supported by default and no plugin was provided. Please try again with a supported scheme or a plugin that supportes %q", parsedUri.Scheme, parsedUri.Scheme)
}
return nil
}
// uploadWithPluginexpects loads a go plugin
func uploadWithPlugin(logger *log.Logger, input string, bucket string, key string, pluginPath string, maxConcurrency int) error {
p, err := plugin.Open(pluginPath)
if err != nil {
return fmt.Errorf("unable to open plugin: %w", err)
}
symbol, err := p.Lookup("Uploader")
if err != nil {
return fmt.Errorf("plugin does not expose an Uploader interface: %w", err)
}
uploader, ok := symbol.(Uploader)
if !ok {
return fmt.Errorf("Uploader found in plugin does not implement Uploader interface")
}
return uploader.Upload(logger, input, bucket, key, maxConcurrency
}and the functionality for arweave could be called like The benefits of this are:
|
|
Thanks for the details. I don't see What advantage would adding this have over using any other CLI for uploading files to Arweave? PMTiles is intentionally just a normal file, so any existing tools for that purpose should work. |
|
I guess I don't understand what the hesitation is around my suggested compromise around plugins? It seems pretty in line with the goals you stated. It won't add dependencies, its not a breaking change, and it won't add anything for you to maintain because any actual plugins will be maintained by their developers. I'm also willing to write this because its quite simple. The advantage of these things being directly in the CLI is it makes it easier to onboard people into using PMTiles when the tooling supports the backends. A lot of people using this might be front end focused developers who aren't super comfortable chaining together things on the command line. I have students where this is would be helpful for example. At the end of the day its your call and your repo, I'll just live with a fork if I have to. I guess I can also upstream this to |
|
FWIW, one option would be to define the That would provide the ability for bespoke implementations targeting Arweave or other providers that could exist outside the core package and be imported on an as-needed basis for individual users. This would also involve a potentially "non-trivial" refactoring of the The other alternative would be to implement the |
@thisisaaronland that's essentially what I'm suggesting by using the plugin package. It just makes it so that those other implementations of the interface can be compiled separately than |
|
I was unaware of the plugin system. I can't speak for @bdon but I imagine the lack of Windows support, not to mention all the other caveats around plugins, will be a show-stopper: It does feel like the best approach would be to implement the |
|
@thisisaaronland they don't work on Windows but AFAIK that wouldn't prevent |
Just because it is easy to add does not mean we should add it. The design goal of this program is to be easy for a solo developer to maintain over several years. Adding a plugin system might take 5 minutes, but maintaining that additional surface area over years across other changes in the program could accumulate to days of work of volunteer labor that I need to do in order to accommodate that single use case. I suggest you can maintain a fork highlighting the additional platform and if a significant audience needs those platforms we can evaluate it later, thanks. |
I gave you an implementation that is quite easy for anyone to maintain, given the skillset necessary to maintain the existing code. the surface area was quite small(under 20 lines of code) and isolated to just the |
Rationale
Scope of Changes
Upload()on bucket URI to allow for new upload implementations outside of gocloudReproducing
ar://url or replacear://withhttps://arweave.net/. This will let you download the file from the Arweave networkar://and append it tohttps://viewblock.io/arweave/tx/to see the metadata