Testing with a real Plone site

[ajayns]

@cekk Do mention any issues you have here! I'm also checking out each feature implemented.

This would help us do any minor fixes and be ready to use the plugin with our proposed gatsby-starter-plone.

[cekk]

i tried to import data from a small plone site (415 items in catalog), and it seems to work very well with the recursion-mode.
The import doesn't take too much and the contents seems to have the correct data.

I have some problems with the pages created from the starter. I'll open some separated issues for this.

[cekk]

ok, the only problem in the starter, is that here I don't have anything (i don't have a docs folder, so the query returns nothing).

Commenting that line, everything works.
Also breadcrumbs and site actions are ok.
Internal links works
Files are served correctly

[ajayns]

Yes currently, the starter is configured to use it's own docs as a source. So here, at the index page, the plugin displays docs/index.md. This is not an error but it's cause of the specific config we have for the gatsby-site for our use.

[cekk]

Another try with 30000 objects and ~15gb of blobs: failed.

After a few hours, I run out of memory.

[datakurre]

That was to be expected. @KyleAMathews Would you be able to give us some generic tips on how to optimize source plugin memory usage?

…

On 22 Jul 2018, 11.48 +0300, Andrea Cecchi ***@***.***>, wrote: Another try with 30000 objects and ~15gb of blobs: failed. After a few hours, I run out of memory. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[datakurre]

As we delegate file nodes to gatsby-source-filesystem, I wonder, how could it be that blobs still way memory? Just JSON should not take that much...

[cekk]

just for reference, this is the traceback:

Creating file nodes

<--- Last few GCs --->

[84778:0x102804c00] 17342052 ms: Mark-sweep 1403.1 (1496.1) -> 1403.1 (1496.1) MB, 2228.2 / 0.0 ms  allocation failure GC in old space requested
[84778:0x102804c00] 17344832 ms: Mark-sweep 1403.1 (1496.1) -> 1402.9 (1464.6) MB, 2779.4 / 0.0 ms  last resort GC in old space requested
[84778:0x102804c00] 17347313 ms: Mark-sweep 1402.9 (1464.6) -> 1402.9 (1464.6) MB, 2480.6 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2b8922725549 <JSObject>
    1: putTask [/Users/cekk/Documents/workspace/gatsby-source-plone/node_modules/better-queue/lib/queue.js:~375] [pc=0x1449a9907b76](this=0x2b89418065d9 <JSGlobal Object>)
    2: updateTask [/Users/cekk/Documents/workspace/gatsby-source-plone/node_modules/better-queue/lib/queue.js:~399] [pc=0x1449a990d2f6](this=0x2b89418065d9 <JSGlobal Object>)
    3: _onTimeout [/Users/cekk/Documents/workspace/...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/cekk/.nvm/versions/node/v9.10.0/bin/node]
 2: node::FatalTryCatch::~FatalTryCatch() [/Users/cekk/.nvm/versions/node/v9.10.0/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/cekk/.nvm/versions/node/v9.10.0/bin/node]
 4: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [/Users/cekk/.nvm/versions/node/v9.10.0/bin/node]
 5: v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity(v8::internal::Handle<v8::internal::NameDictionary>, int, v8::internal::PretenureFlag) [/Users/cekk/.nvm/versions/node/v9.10.0/bin/node]
 6: v8::internal::Dictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add(v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, int*) [/Users/cekk/.nvm/versions/node/v9.10.0/bin/node]
 7: v8::internal::LookupIterator::ApplyTransitionToDataProperty(v8::internal::Handle<v8::internal::JSObject>) [/Users/cekk/.nvm/versions/node/v9.10.0/bin/node]
 8: v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::Object::StoreFromKeyed) [/Users/cekk/.nvm/versions/node/v9.10.0/bin/node]
 9: v8::internal::Runtime_SetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/cekk/.nvm/versions/node/v9.10.0/bin/node]
10: 0x1449a86042fd
make: *** [develop] Abort trap: 6

[ajayns]

I haven't much explored optimization and handling such huge sites yet, I'll do some research and get back.

cc @m-allanson @pieh insights appreciated

[pieh]

This might be something we currently are actually working on (but might not be) - see gatsbyjs/gatsby#6611 and gatsbyjs/gatsby#6656

Nodes itself aren't taking that much memory - but serialization is problematic when there is a lot of nodes. There were some fixes recently published already, so try updating gatsby to beta-55

[datakurre]

@pieh Thanks for the update.

@cekk @ajayns I believe that for now it's good enough for us to acknowledge that with large amount of nodes, memory issues are expected, and target for smaller sites instead, until it is sure that gatsby itself should scale.

Until that, instead of optimizing too early, we probably should look into using gatsby cache #136 to make updating sites with 1-10k pages practical.

[cekk]

@datakurre i agree.
Mine was only a "stress test".
An ideal site candidate is more like the first one.

Do we need more tests or can close this issue?

[datakurre]

@cekk It was a great stress test and found real issues.

Let's close this and run more tests once we have the starter. I hope to have implemented caching support by then.