Implement image hosting directly inside Lemmy

[Nutomic]

Right now we are using a separate software to handle the image hosting. That has some disadvantages, for example anyone can directly open the pictshare site in their browser, and upload images without any authentication.

Implementing this in Rust instead would reduce the number of required Docker containers, and improve the performance compared to php.

It doesn't seem very complicated to implement, so I will probably give it a try.

Edit: Pictshare Features (which we use in Lemmy):

• upload image (as http post with multipart/form-data)
  • https://actix.rs/docs/request/
• store image to filesystem
• have unique id for each image (like m3jbhp.jpg)
• serve thumbnails under /192/m3jbhp.png (i see 192 and 96)
• serve original image under /pictshare/m3jbhp.png
• usa sha1 to detect duplicate files
• remove exif data
  • https://crates.io/crates/kamadak-exif
• convert images to .webp (not working in pictshare)

[dessalines]

I'd prefer to use pictshare ( or any similar self-hosting service ), mainly because its a tool specifically made for image hosting, so it already does things like check for dupes, does thumbnails ( which we could do at some point if we check for pictshare in the domain), gif to mp4 conversion, etc.

I'm a big believer in using tools that are really good at what they do, and for something as general-purpose as image-hosting, its better to not duplicate that work across many projects.

If you wanted to make a general image hosting service in rust, I'd be up for using it with lemmy.

There also might be a way around the account issue, but even if there isn't, it doesn't bother me too much. We could always just hide the pictshare index.html and only have the api available too.

[Nutomic]

Can we at least route pictshare requests through Lemmy, so that we can do proper access control? Right now everyone can upload images without any authentication.

[dessalines]

Sure, first we need to make sure that pictshare has some kind of authentication / protected mode, then we can have that on the back end.

[Nutomic]

I meant that image requests would be routed through Lemmy, so that pictshare isn't even exposed to the internet. But I guess some auth header or parameter would also work.

[dessalines]

Oh I gotcha. Yeah this is probably possible.

[Nutomic]

https://yerbamate.dev/dessalines/lemmy/pulls/13

[bytesnake]

I'm a big believer in using tools that are really good at what they do, and for something as general-purpose as image-hosting, its better to not duplicate that work across many projects.

There is already a Rust library, which supports all you usecases here https://docs.rs/image/0.23.4/image/ It supports all fileformats (except webp encoding), resizing and various filters. You could just add another HTTP endpoint for images and call in the library. And then remove the dependency to PHP completely :)

[Nutomic]

@bytesnake Yes thats my plan now, thanks for the link!

[Nutomic]

https://yerbamate.dev/LemmyNet/lemmy/pulls/46

[asonix]

related: I'm working on a standalone replacement for pictshare: https://git.asonix.dog/asonix/pict-rs

[dessalines]

@asonix Didn't you have code for actix routing to pict-rs? We should add this to lemmy, and not have it handled by nginx.

[asonix]

Yeah, here are the important bits:

        let client = Client::build()
            .header("User-Agent", "pict-rs-frontend, v0.1.0")
            .timeout(Duration::from_secs(30))
            .finish();

        App::new()
            .data(client)
            .service(web::resource("/image/{filename}").route(web::get().to(full_res)))

...

async fn full_res(
    filename: web::Path<String>,
    req: HttpRequest,
    client: web::Data<Client>,
) -> Result<HttpResponse, Error> {
    let url = CONFIG.upstream_image_url(&filename.into_inner());

    image(url, req, client).await
}

async fn image(
    url: String,
    req: HttpRequest,
    client: web::Data<Client>,
) -> Result<HttpResponse, Error> {
    let res = client
        .request_from(url, req.head())
        .if_some(req.head().peer_addr, |addr, req| {
            req.header("X-Forwarded-For", addr.to_string())
        })
        .no_decompress()
        .send()
        .await?;

    if res.status() == StatusCode::NOT_FOUND {
        return Ok(to_404());
    }

    let mut client_res = HttpResponse::build(res.status());

    for (name, value) in res.headers().iter().filter(|(h, _)| *h != "connection") {
        client_res.header(name.clone(), value.clone());
    }

    Ok(client_res.body(BodyStream::new(res)))
}

[asonix]

oh and here's the upload code:

async fn upload(
    req: HttpRequest,
    body: web::Payload,
    client: web::Data<Client>,
) -> Result<HttpResponse, Error> {
    let mut res = client
        .request_from(CONFIG.upstream_upload_url(), req.head())
        .if_some(req.head().peer_addr, |addr, req| {
            req.header("X-Forwarded-For", addr.to_string())
        })
        .send_stream(body)
        .await?;

    let images = res.json::<Images>().await?;

    // respond to the upload
    ...
}

[Nutomic]

https://yerbamate.dev/LemmyNet/lemmy/pulls/77