Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Large Files fail to upload #46

Closed
CrazyDev05 opened this issue Nov 28, 2023 · 33 comments
Closed

Large Files fail to upload #46

CrazyDev05 opened this issue Nov 28, 2023 · 33 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@CrazyDev05
Copy link

when i try to upload a large file (in my case 500MiB)
i get the following error

HeadersTimeoutError: Headers Timeout Error
    at Timeout.onParserTimeout [as _onTimeout] (/var/www/monofile/node_modules/undici/lib/client.js:902:28)
    at listOnTimeout (node:internal/timers:575:11)
    at process.processTimers (node:internal/timers:514:7) {
  code: 'UND_ERR_HEADERS_TIMEOUT'
}
@nbitzz
Copy link
Contributor

nbitzz commented Nov 28, 2023

I don't believe I've seen this before, maybe raise your requestTimeout in config.json?

@CrazyDev05
Copy link
Author

Good point...

@CrazyDev05
Copy link
Author

how large should i make it?

@nbitzz
Copy link
Contributor

nbitzz commented Nov 28, 2023

Whatever you think would work. Also, what branch are you on? Are you on a non-master/stable branch? Are you cloning a file?

@CrazyDev05
Copy link
Author

I'm on the main branch/i cloned the git

@CrazyDev05
Copy link
Author

is the timeout in ms?

@nbitzz
Copy link
Contributor

nbitzz commented Nov 28, 2023

is the timeout in ms?

Yeah. I'm not sure how Discord.js uploads the files, but I do believe I got a different error for timeouts... could be different now, though

If boosting requestTimeout doesn't fix it, maybe decrease your Discord file size? That fixed some issues on the main instances

The only place I know undici is used in is axios, so it could also be an issue with the file you're trying to clone

@CrazyDev05
Copy link
Author

i already decreased maxDiscordFileSize to 10485760

@CrazyDev05
Copy link
Author

Wait I misunderstood the cloning file question, I'm trying to upload a file from my pc (the file just contains Hello \n to test the upload)

@nbitzz
Copy link
Contributor

nbitzz commented Nov 28, 2023

Wait I misunderstood the cloning file question, I'm trying to upload a file from my pc (the file just contains Hello \n to test the upload)

That's... really weird. I'll check DJS's codebase when I have time. How fast is your internet?

@CrazyDev05
Copy link
Author

CrazyDev05 commented Nov 28, 2023 •
edited
Loading

The server has 1gbit/s (up/down), I have 30mbit/s up
and i still have the same error after changing the requestTimeout to 10800000 (failed after 3.5 min)

@nbitzz
Copy link
Contributor

nbitzz commented Nov 28, 2023

The server has 1gbit/s (up/down), I have 30mbit/s up

and i still have the same error after changing the requestTimeout to 10800000 (failed after 3.5 min)

I'll be honest... I have zero idea of what's happening.

git stash, git checkout stable, and git stash apply is the best advice I can give here, but I doubt that'd work; master and stable aren't that different...

@CrazyDev05
Copy link
Author

Maybe some kind of discord limitation?

@nbitzz
Copy link
Contributor

nbitzz commented Nov 28, 2023 •
edited
Loading

Maybe some kind of discord limitation?

I know @Jack5079 got it to work with 10MiB chunks, and I'm not sure what's limiting you. What provider are you using for your server?

I just tested an upload on the main instance, so I know it's still working...

@CrazyDev05
Copy link
Author

CrazyDev05 commented Nov 28, 2023 •
edited
Loading

I'm using a dedicated server from hetzner
Hardware:

  • AMD Ryzen 7 3700X
  • 2x SSD M.2 NVMe 1 TB
  • 4x RAM 16384 MB DDR4 ECC
  • NIC 1 Gbit - Intel I210

@CrazyDev05
Copy link
Author

It should already be fixed discordjs/discord.js#8679

@nbitzz
Copy link
Contributor

nbitzz commented Nov 28, 2023

Hetzner was apparently banned from Discord's API once, but I'm not sure if this is still true, and the fact that you're getting a timeout over an actual proper error is strange. Try running locally or with another provider and see if that works, maybe...?

@CrazyDev05
Copy link
Author

interestingly i can upload small files, but not large ones

@nbitzz
Copy link
Contributor

nbitzz commented Nov 28, 2023

interestingly i can upload small files, but not large ones

Try uploading a file in the 10MiB range, then a file in the 100MiB+ range, and we'll see how that plays out...?

@CrazyDev05
Copy link
Author

CrazyDev05 commented Nov 28, 2023 •
edited by nbitzz
Loading

  • 10MB
  • 40MB
  • 90MB
  • 160MB
  • 250MB
  • 360MB
  • 490MB
    now I'm confused

@CrazyDev05
Copy link
Author

Trying 500MB again

@CrazyDev05
Copy link
Author

CrazyDev05 commented Nov 28, 2023 •
edited
Loading

now i get http error 504 and no error in the console

<html>
   <head>
      <title>504 Gateway Time-out</title>
   </head>
   <body>
      <center>
         <h1>504 Gateway Time-out</h1>
      </center>
      <hr>
      <center>nginx</center>
   </body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page -->

@nbitzz
Copy link
Contributor

nbitzz commented Nov 28, 2023

now i get http error 504 and no error in the console

<html>
   <head>
      <title>504 Gateway Time-out</title>
   </head>
   <body>
      <center>
         <h1>504 Gateway Time-out</h1>
      </center>
      <hr>
      <center>nginx</center>
   </body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page -->

Your server's just taking a while to upload the file after it receives it; fix your timeout in NGINX or just refresh your files page if you have an account until it finishes uploading

FYI, your admin account is admin, with a password of admin. Make sure to reset it if you plan to make this instance public...

@CrazyDev05
Copy link
Author

i already changed the admin password and how long should i make the timeout?

@nbitzz
Copy link
Contributor

nbitzz commented Nov 28, 2023

i already changed the admin password and how long should i make the timeout?

...Longer than it is currently?

@CrazyDev05
Copy link
Author

CrazyDev05 commented Nov 28, 2023 •
edited
Loading

Tried 700MB and error 500 (HeadersTimeoutError) is back...

@nbitzz
Copy link
Contributor

nbitzz commented Nov 28, 2023 •
edited
Loading

Tried 700MB and error 500 (HeadersTimeoutError) is back...

Hmm.... how much RAM do you have? I may be stupid What's the maximum file size on your instance? I know this probably won't help much, but I'd like to know more.
I have no idea what's causing this...

@CrazyDev05
Copy link
Author

it seems to be a discord issue

@CrazyDev05
Copy link
Author

Maybe progressively uploading instead of bulk uploading the files could help?

@nbitzz
Copy link
Contributor

nbitzz commented Nov 28, 2023 •
edited
Loading

Maybe progressively uploading instead of bulk uploading the files could help?

As in, one file per message, or streaming uploads to Discord?
Upload streaming is set for monofile 2.0 (#16).
If you'd like, we can add a maxFilesPerMessage parameter to config.json which would allow you to upload only one chunk at a time, which will likely be available in 2.0.

@CrazyDev05
Copy link
Author

i meant like one file per message and maybe also upload streaming

@nbitzz nbitzz mentioned this issue Nov 28, 2023
Open
@nbitzz nbitzz added bug Something isn't working help wanted Extra attention is needed labels Dec 11, 2023
@nbitzz
Copy link
Contributor

nbitzz commented May 3, 2024

Prooooooooobably fixed with upload streaming i'll close this now

@nbitzz nbitzz closed this as completed May 3, 2024
@nbitzz
Copy link
Contributor

nbitzz commented May 3, 2024

I mean I managed a 10gb file on the api-v1 test instance and pfile is serving 250gb of data now so I think we're good

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@CrazyDev05 @nbitzz