RemoteWebElement.upload: OutOfMemoryError during large file upload to remote selenium server

[lukeis]

Originally reported on Google Code with ID 4442

What steps will reproduce the problem?
1. Write a test case that uploads a large file, e.g.

driver.setFileDetector(new LocalFileDetector()); // needed because the file to upload
is on the machine running the tests and not on the remote host

WebElement uploadInputField = driver.findElement(By.id("myFileUploadField"));
uploadInputField.sendKeys("/path/to/my/large/file");

2. Run the test against a selenium server

3. The test run will fail with an OutOfMemoryError:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
at java.lang.StringBuffer.append(StringBuffer.java:320)
at org.openqa.selenium.internal.Base64Encoder.encode(Base64Encoder.java:65)
at org.openqa.selenium.io.Zip.zipFile(Zip.java:76)
at org.openqa.selenium.remote.RemoteWebElement.upload(RemoteWebElement.java:102)
at org.openqa.selenium.remote.RemoteWebElement.sendKeys(RemoteWebElement.java:91) 

4. Re-run with JVM arguments like -Xmx1024m and verify the same error

What is the expected output? What do you see instead?
The expected output would be a successful file upload instead of an OutOfMemoryError.


Selenium version: 2.25.0
OS: Fedora 16 (executing the JUnit test), VM Windows 7 running the Selenium server
Browser: Firefox and IE (probably any browser, but we only use those)
Browser version: FF10, IE7-IE9

Reported by erooolkoc on 2012-08-20 07:01:58

[lukeis]

Reported by barancev on 2012-09-11 11:02:49

• Labels added: Component-WebDriver, Performance

[lukeis]

Hi,

could you specify what size of file was used?

Reported by a.u.savchuk on 2012-09-22 15:41:58

[lukeis]

We're seeing this OutOfMemoryError exception with a 150MB upload.

Reported by daniel.caminada on 2012-09-24 09:21:23

[lukeis]

By default, Java only assigns a puny amount of memory to use. You should allocate more
using the normal java command line flags. If you're uploading a 150MB file, I'd recommend
>300MB

Reported by simon.m.stewart on 2012-09-24 09:51:52

• Status changed: WorkingAsIntended

[lukeis]

As Erol wrote in the description, using higher defaults (e.g. -Xmx1024m) doesn't solve
the issue...

Reported by daniel.caminada on 2012-09-24 10:00:41

[lukeis]

In order to upload your file, we first read it into memory, convert it to a base64 representation,
then push that across the wire. The server side reads the entire raw stream into memory,
creates a model from it, and then attempts to deal with it. If you're uploading a 150MB
file, you're going to have to leave enough room for many copies of that to exist in
memory simultaneously. If you still experience this issue with 2GB of memory assigned
to Java then this might well be something to investigate.

Reported by simon.m.stewart on 2012-09-24 11:42:58

[lukeis]

Ok, I think that explains the issue.

Unfortunately I'm still stucked here with a 32-bit XP machine, where I can't go higher
than "-Xms512M -Xmx1024M -XX:MaxPermSize=512M" without getting errors at startup ("Could
not reserve enough space for object heap") - but that's my problem not selenium's.

Reported by daniel.caminada on 2012-09-25 17:24:12

[lukeis]

We're having the same Heap Size problem trying to upload a 30MB .tar.gz file.

Selenium version: 2.24.1
OS: Linux executing the test, Windows Server 2008 running the hub and node
Browser: Firefox and IE
Browser version: IE8

Setup for the Linux Build Job (Jenkins) is -Xmx3048m -XX:MaxPermSize=1024m.

Reported by claus.christoph.schaal on 2012-10-29 14:50:47

[lukeis]

It seems that Base64Encoder#encode(byte[]) is using the heap.

For the following comparison a 30MB file was used:
org.openqa.selenium.internal.Base64Encoder#encode(byte[]): 390MB used heap
org.apache.commons.codec.binary.Base64#encodeBase64(byte[]): 218MB used heap

Reported by claus.christoph.schaal on 2012-10-30 14:12:51

[lukeis]

Reported by luke.semerau on 2015-09-17 18:16:01

• Labels added: Restrict-AddIssueComment-Commit