Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export incorrect when QueuedJob takes more than 1 cycle to complete #47

Open
SanderHamaka opened this issue Oct 30, 2020 · 2 comments
Open

Comments

@SanderHamaka
Copy link

Thanks for a very useful module!

Situation:
7000+ rows export on a high load machine

Problem:
QueuedJobService has a default memory limit of 256Mb. Their method runJob has a check isMemoryTooHigh. In our case, that limit was met. The idea behind the way QueuedJobService handles this situation seems to be to release memory and let the next iteration of ProcessJobQueueTask take up where we left it.

It looks like silverstripe-gridfieldqueuedexport can not handle this situation. In the end I have an export with only the records from the last restart. This is easy to spot because the column headers are missing in the first line.

Then the part I am not sure about:
It looks like the problem might be that silverstripe-gridfieldqueuedexport uses write mode and not append mode for file handling in GenerateCSVJob->getCSVWriter():

$csvWriter = Writer::createFromPath($this->getOutputPath(), 'w');

Maybe the module is not build to account for this situation. In that case I think it is worth mentioning it in the README.

@SanderHamaka
Copy link
Author

SanderHamaka commented Mar 8, 2022

I think this issue is still relevant and can be fixed this way:

$csvWriter = Writer::createFromPath($this->getOutputPath(), 'a');
$csvWriter->setNewline("");

Without clearing the newline there will be an empty line between each record since file mode a already inserts a new line.
For my situatie this fix works but it is not tested in any other setup.

@GuySartorelli
Copy link
Member

Would you be willing to create a pull request for this change @SanderHamaka?

SanderHamaka added a commit to SanderHamaka/silverstripe-gridfieldqueuedexport that referenced this issue Mar 9, 2022
… CSV writer overwrites the file resulting in an export only containing the records from the last (re)start and no header row.

By changing the mode to append instead of write every record is appended on a new line. Because of that, the setNewLine has to be reset otherwise there is an empty line between each record in the file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants