Add support for signed upload urls. #1661

erikcarlsson · 2024-12-19T09:19:26Z

No description provided.

encore-cla · 2024-12-19T09:19:41Z

All committers have signed the CLA.

eandre · 2024-12-20T10:58:06Z

runtimes/js/encore.dev/storage/objects/bucket.ts

+export interface UploadUrlOptions {
+  /** The expiration time of the url, in seconds from signing. The maximum
+   * value is seven days */
+  ttl: number;


Options should typically be optional:

Suggested change

ttl: number;

ttl?: number;

eandre · 2024-12-20T10:58:44Z

runtimes/js/src/objects.rs

+    #[napi]
+    pub async fn get_upload_url(
+        &self,
+        options: Option<UploadUrlOptions>, // TODO: can/should this be made non-optional, since ttl is required?


ttl shouldn't be required, right? The user-facing docs state the ttl defaults to 7 days if not provided

eandre · 2024-12-20T10:59:01Z

runtimes/js/src/objects.rs

+impl From<UploadUrlOptions> for core::UploadUrlOptions {
+    fn from(value: UploadUrlOptions) -> Self {
+        Self {
+            ttl: value.ttl as u64,


Convert this to the default of 7 days here

eandre · 2024-12-20T10:59:10Z

runtimes/js/src/objects.rs

+#[napi(object)]
+#[derive(Debug, Default)]
+pub struct UploadUrlOptions {
+    pub ttl: i64,


Suggested change

pub ttl: i64,

pub ttl: Option<i64>,

eandre · 2024-12-20T11:03:18Z

runtimes/go/storage/objects/options.go

+
+// WithTtl is used for signed URLs, to specify the lifetime of the generated
+// URL, in seconds. The max value is seven days.
+func WithTtl(ttl uint64) withTtlOption {


Go convention is TTL not Ttl. Also make it take a time.Duration instead of an uint64

Suggested change

func WithTtl(ttl uint64) withTtlOption {

func WithTTL(ttl time.Duration) withTTLOption {

eandre · 2024-12-20T11:09:06Z

runtimes/js/encore.dev/storage/objects/bucket.ts

+   * Anyone with possession of the URL can write to the given object name
+   * without any additional auth.
+   */
+  async getUploadUrl(name: string, options?: UploadUrlOptions): Promise<string> {


Let's make this signedUploadUrl, and have it return a SignedURL object with just a single field {url: string}

Suggested change

async getUploadUrl(name: string, options?: UploadUrlOptions): Promise<string> {

async signedUploadUrl(name: string, options?: SignedUploadUrlOptions): Promise<SignedURL> {

Done. Although used SignedUploadUrl instead of SignedUrl. I think we ended up agreeing on making upload URLs more flexible, but not so far as folding in signed GET and PUT URLs in one type. But let me know if I got that wrong.

eandre · 2024-12-20T11:12:10Z

tsparser/src/parser/resources/infra/objects.rs

@@ -120,6 +120,7 @@ pub fn resolve_bucket_usage(data: &ResolveUsageData, bucket: Lrc<Bucket>) -> Opt
                "list" => Operation::ListObjects,
                "exists" | "attrs" => Operation::GetObjectMetadata,
                "upload" => Operation::WriteObject,
+                "getUploadUrl" => Operation::WriteObject,


Rename to signedUploadUrl as per above. Also, let's introduce a new SignedUploadUrl operation (needs to be done in legacymeta.rs as well)

eandre · 2024-12-20T11:25:48Z

runtimes/go/storage/objects/internal/providers/s3/bucket.go

-		client: client,
-		cfg:    runtimeCfg,
+		client:         client,
+		presign_client: s3.NewPresignClient(client),


Instantiate in clientForProvider instead so we can cache it

Done, thanks

eandre · 2024-12-20T11:25:58Z

runtimes/go/storage/objects/internal/providers/s3/bucket.go

-	client *s3.Client
-	cfg    *config.Bucket
+	client         *s3.Client
+	presign_client *s3.PresignClient


Suggested change

presign_client *s3.PresignClient

presignClient *s3.PresignClient

eandre · 2024-12-20T11:26:09Z

runtimes/go/storage/objects/internal/providers/s3/bucket.go

+		Bucket: &b.cfg.CloudName,
+		Key:    &object,
+	}
+	sign_opts := func(opts *s3.PresignOptions) {


Suggested change

sign_opts := func(opts *s3.PresignOptions) {

signOpts := func(opts *s3.PresignOptions) {

eandre · 2024-12-20T11:27:02Z

runtimes/go/storage/objects/internal/providers/s3/bucket.go

+	sign_opts := func(opts *s3.PresignOptions) {
+		opts.Expires = time.Duration(data.Ttl) * time.Second
+	}
+	req, err := b.presign_client.PresignPutObject(data.Ctx, &params, sign_opts)


Do the error check here instead

Suggested change

req, err := b.presign_client.PresignPutObject(data.Ctx, &params, sign_opts)

req, err := b.presign_client.PresignPutObject(data.Ctx, &params, sign_opts)

if err != nil {

return "", mapErr(err)

}

return req.URL, nil

Add a PUT handler for the local bucket server.

866929a

erikcarlsson force-pushed the sign-url branch 3 times, most recently from 3d7696f to 6bf10f5 Compare December 19, 2024 10:31

eandre reviewed Dec 20, 2024

View reviewed changes

erikcarlsson force-pushed the sign-url branch 2 times, most recently from 5790283 to 3b6d7b6 Compare December 20, 2024 18:26

Add support for signed upload urls.

f165bfd

erikcarlsson force-pushed the sign-url branch from 3b6d7b6 to f165bfd Compare December 20, 2024 18:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for signed upload urls. #1661

Add support for signed upload urls. #1661

erikcarlsson commented Dec 19, 2024

encore-cla bot commented Dec 19, 2024

eandre Dec 20, 2024

erikcarlsson Dec 20, 2024

eandre Dec 20, 2024

eandre Dec 20, 2024

eandre Dec 20, 2024

eandre Dec 20, 2024

erikcarlsson Dec 20, 2024

eandre Dec 20, 2024

erikcarlsson Dec 20, 2024 •

edited

Loading

eandre Dec 20, 2024

erikcarlsson Dec 20, 2024

eandre Dec 20, 2024

erikcarlsson Dec 20, 2024

eandre Dec 20, 2024

eandre Dec 20, 2024

eandre Dec 20, 2024

	func WithTtl(ttl uint64) withTtlOption {
	func WithTTL(ttl time.Duration) withTTLOption {

	async getUploadUrl(name: string, options?: UploadUrlOptions): Promise<string> {
	async signedUploadUrl(name: string, options?: SignedUploadUrlOptions): Promise<SignedURL> {

	presign_client *s3.PresignClient
	presignClient *s3.PresignClient

	sign_opts := func(opts *s3.PresignOptions) {
	signOpts := func(opts *s3.PresignOptions) {

-	req, err := b.presign_client.PresignPutObject(data.Ctx, &params, sign_opts)
+	req, err := b.presign_client.PresignPutObject(data.Ctx, &params, sign_opts)
+	if err != nil {
+	    return "", mapErr(err)
+	}
+	return req.URL, nil

Add support for signed upload urls. #1661

Are you sure you want to change the base?

Add support for signed upload urls. #1661

Conversation

erikcarlsson commented Dec 19, 2024

encore-cla bot commented Dec 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikcarlsson Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikcarlsson Dec 20, 2024 •

edited

Loading