You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Situation:
7000+ rows export on a high load machine
Problem:
QueuedJobService has a default memory limit of 256Mb. Their method runJob has a check isMemoryTooHigh. In our case, that limit was met. The idea behind the way QueuedJobService handles this situation seems to be to release memory and let the next iteration of ProcessJobQueueTask take up where we left it.
It looks like silverstripe-gridfieldqueuedexport can not handle this situation. In the end I have an export with only the records from the last restart. This is easy to spot because the column headers are missing in the first line.
Then the part I am not sure about:
It looks like the problem might be that silverstripe-gridfieldqueuedexport uses write mode and not append mode for file handling in GenerateCSVJob->getCSVWriter():
Without clearing the newline there will be an empty line between each record since file mode a already inserts a new line.
For my situatie this fix works but it is not tested in any other setup.
… CSV writer overwrites the file resulting in an export only containing the records from the last (re)start and no header row.
By changing the mode to append instead of write every record is appended on a new line. Because of that, the setNewLine has to be reset otherwise there is an empty line between each record in the file.
Thanks for a very useful module!
Situation:
7000+ rows export on a high load machine
Problem:
QueuedJobService has a default memory limit of 256Mb. Their method runJob has a check isMemoryTooHigh. In our case, that limit was met. The idea behind the way QueuedJobService handles this situation seems to be to release memory and let the next iteration of ProcessJobQueueTask take up where we left it.
It looks like silverstripe-gridfieldqueuedexport can not handle this situation. In the end I have an export with only the records from the last restart. This is easy to spot because the column headers are missing in the first line.
Then the part I am not sure about:
It looks like the problem might be that silverstripe-gridfieldqueuedexport uses write mode and not append mode for file handling in GenerateCSVJob->getCSVWriter():
$csvWriter = Writer::createFromPath($this->getOutputPath(), 'w');
Maybe the module is not build to account for this situation. In that case I think it is worth mentioning it in the README.
The text was updated successfully, but these errors were encountered: