Skip to content

Commit

Permalink
Merge branch 'master' into feat/contentful-rich-text-tables
Browse files Browse the repository at this point in the history
  • Loading branch information
axe312ger committed Feb 16, 2022
2 parents b334007 + 45cb1f1 commit 9350da1
Show file tree
Hide file tree
Showing 607 changed files with 30,669 additions and 24,123 deletions.
21 changes: 20 additions & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ aliases:
validate_renovate: &validate_renovate
run:
name: Validate renovate-config
command: (node scripts/renovate-config-generator.js && (git status --porcelain renovate.json5 | grep "M renovate.json5")) && (echo "Please run \"node scripts/renovate-config-generator.js\" to update renovate.json5" && exit 1) || npx -p renovate renovate-config-validator .
command: (node scripts/renovate-config-generator.js && (git status --porcelain renovate.json5 | grep "M renovate.json5")) && (echo "Please run \"node scripts/renovate-config-generator.js\" to update renovate.json5" && exit 1) || npx -p renovate@31.28.5 renovate-config-validator .

persist_cache: &persist_cache
save_cache:
Expand Down Expand Up @@ -462,11 +462,28 @@ jobs:
- e2e-test:
test_path: e2e-tests/contentful
test_command: yarn test
# we build a second time to see if warm/cached builds are successful
- e2e-test:
test_path: e2e-tests/contentful
test_command: yarn build
- store_artifacts:
path: e2e-tests/contentful/__diff_output__
- store_test_results:
path: e2e-tests/contentful/cypress/results

e2e_tests_trailing-slash:
<<: *e2e-executor
environment:
<<: *e2e-executor-env
CYPRESS_PROJECT_ID: ofxgw8
CYPRESS_RECORD_KEY: 29c32742-6b85-40e0-9b45-a4c722749d52
steps:
- e2e-test:
test_path: e2e-tests/trailing-slash
test_command: yarn test
- store_test_results:
path: e2e-tests/trailing-slash/cypress/results

starters_validate:
executor: node
steps:
Expand Down Expand Up @@ -669,6 +686,8 @@ workflows:
<<: *e2e-test-workflow
- e2e_tests_contentful:
<<: *e2e-test-workflow
- e2e_tests_trailing-slash:
<<: *e2e-test-workflow
- e2e_tests_development_runtime:
<<: *e2e-test-workflow
- e2e_tests_production_runtime:
Expand Down
1 change: 1 addition & 0 deletions .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ module.exports = {
__ASSET_PREFIX__: true,
_CFLAGS_: true,
__GATSBY: true,
__TRAILING_SLASH__: true,
},
rules: {
"@babel/no-unused-expressions": [
Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
[![A dark purple background with hints of stars. On top of that it says "GatsbyConf 2022" and "March 2-3". The letters "2022" are big, playful, light purple, and have illustrations of stars, people in orbs around them.](https://user-images.githubusercontent.com/3477155/153261605-43e1075b-7889-435b-bd34-a5980b629843.jpg)](https://gatsbyconf.com?utm_source=github&utm_medium=readme-banner&utm_campaign=gatsbyconf22-repo-banner)

---

<p align="center">
<a href="https://www.gatsbyjs.com">
<img alt="Gatsby" src="https://www.gatsbyjs.com/Gatsby-Monogram.svg" width="60" />
Expand Down
23 changes: 23 additions & 0 deletions benchmarks/memory/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
**/.classpath
**/.dockerignore
**/.env
**/.git
**/.gitignore
**/.project
**/.settings
**/.toolstarget
**/.vs
**/.vscode
**/*.*proj.user
**/*.dbmdl
**/*.jfm
**/charts
**/docker-compose*
**/compose*
**/Dockerfile*
**/node_modules
**/npm-debug.log
**/obj
**/secrets.dev.yaml
**/values.dev.yaml
README.md
14 changes: 14 additions & 0 deletions benchmarks/memory/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
FROM node:14-buster
ENV NODE_ENV=production
ENV CI=1
ENV GATSBY_CPU_COUNT=4
RUN apt-get update -y && apt-get upgrade -y && apt-get install git curl npm -y
RUN npm i -g gatsby-cli gatsby-dev-cli
WORKDIR /usr/src/app
RUN echo "\n\necho \"Welcome to the Gatsby Memory benchmark container!\\n - /usr/src/gatsby : Your local gatsby repo\\n - /usr/src/app : The memory benchmark gatsby site\\n\"" > /root/.bashrc

# set up gatsby-dev
RUN gatsby-dev --set-path-to-repo /usr/src/gatsby

# keep the process running
ENTRYPOINT ["tail", "-f", "/dev/null"]
101 changes: 101 additions & 0 deletions benchmarks/memory/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Gatsby Memory Benchmark

The goal of this benchmark is to test Gatsby's memory usage and look for potential optimizations.

## The Docker Container

The docker container used in these tests sets up a Debian instance with node 14 installed (as well as npm/yarn/etc).
It has ports 9000 (for hosting gatsby) and 9229 (for debugging) exposed.

Within the container, two points to your local filesystem are mounted:

- /usr/src/gatsby : Your local gatsby repo
- /usr/src/site : The memory benchmark gatsby site

## Commands

### Docker

These commands are used for interfacing with docker and have built-in utilities for managing the docker container.

#### yarn docker:build

Builds the container used for testing.

#### yarn docker:start

Starts the container built by `yarn docker:build`.

#### yarn docker:connect

Connects to the container started by `yarn docker:start`.

#### yarn docker:start-and-connect

A shorthand for start + connect.

#### yarn docker:stop

Stop the container used for testing.

#### yarn docker:stats

Show a polling display of the container's docker stats.

### Gatsby

These commands are used for interfacing with gatsby.

#### yarn gatsby:build

Simply an alias to `yarn gatsby build`.

#### yarn gatsby:serve

Starts `gatsby serve` on port 9000 and sets the host properly to work inside docker.

#### yarn gatsby:develop

Starts `gatsby develop` on port 9000 and sets the host properly to work inside docker.

#### yarn gatsby:build:debug

Runs `gatsby build` with `inspect-brk` set to start the [debugging process](https://www.gatsbyjs.com/docs/debugging-the-build-process/) on port 9229.

#### yarn gatsby:develop:debug

Runs `gatsby develop` with `inspect-brk` set to start the [debugging process](https://www.gatsbyjs.com/docs/debugging-the-build-process/) on port 9229.

## Setup

Currently we can reproduce builds crashing with out default settings

- Docker container running with 2GB limit
- 300 nodes x ~2MB each = ~600MB of "just" nodes data in each process (number of nodes can be controlled with NUM_NODES env var)
- 3 workers + main process (`GATSBY_CPU_COUNT` set to 4 in docker image, but you can specify different value with env var - for example `GATSBY_CPU_COUNT=6 yarn gatsby:build`)
- `eq_field` template using fast filters (single `eq` specifically)

Goal is to make `eq_field` template to not cause crashes, then add next template (different operator) that cause crashes and repeat until all queries can be handled with set memory limits.

### Workflow

While `gatsby-dev` command is available inside docker, from my testing it seems like it doesn't pick up file changes when run there. Workflow that seems to work reliably:

When starting working with this benchmark:

- start `yarn watch` (possibly with `--scope`) in monorepo
- start `gatsby-dev` outside of docker in benchmark directory (just like with regular site)
- `yarn docker:connect` to get inside docker
- `npm rebuild` to rebuild binaries inside docker

And repeat as many times as you want:

- make changes to `gatsby` source code as you normally would
- run `yarn gatsby:build` inside docker

## Testing

TODO

- How to configure memory limits
- Where to look
3 changes: 3 additions & 0 deletions benchmarks/memory/gatsby-config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
module.exports = {
plugins: [],
}
Loading

0 comments on commit 9350da1

Please sign in to comment.