feat: getting sqlite stuff ready for production

forwardemail · Nov 5, 2023 · dda1be3 · dda1be3
1 parent fe208a6
commit dda1be3
Show file tree

Hide file tree

Showing 46 changed files with 1,191 additions and 145 deletions.
diff --git a/.gitignore b/.gitignore
@@ -87,3 +87,4 @@ temp/
 .ssl-key
 .ssl-dhparam
 .ssl-csr
+ecosystem-sqlite-private.json
diff --git a/README.md b/README.md
@@ -181,7 +181,7 @@ Follow the [Deployment](#deployment) guide below for automatic provisioning and
    vim .env.production
    ```
 
-5. Generate [pm2][] [ecosystem files][ecosystem-files] using our automatic template generator. We created an [ansible-playbook.js](ansible-playbook.js) which loads the `.env.production` environment variables rendered with [@ladjs/env][] into `process.env`, which then gets used in the playbooks.  This is a superior, simple, and the only known dotenv approach we know of in Ansible. Newly created `ecosystem-api.json`, `ecosystem-bree.json`, `ecosystem-web.json`, `ecosystem-smtp.json`, and `ecosystem-imap.json` files will now be created for you in the root of the repository.  If you ever more add or change IP addresses, you can simply re-run this command.
+5. Generate [pm2][] [ecosystem files][ecosystem-files] using our automatic template generator. We created an [ansible-playbook.js](ansible-playbook.js) which loads the `.env.production` environment variables rendered with [@ladjs/env][] into `process.env`, which then gets used in the playbooks.  This is a superior, simple, and the only known dotenv approach we know of in Ansible. Newly created `ecosystem-api.json`, `ecosystem-bree.json`, `ecosystem-web.json`, `ecosystem-smtp.json`, `ecosystem-imap.json`, and `ecosystem-sqlite.json` files will now be created for you in the root of the repository.  If you ever more add or change IP addresses, you can simply re-run this command.
 
    ```sh
    node ansible-playbook ansible/playbooks/ecosystem.yml -l 'localhost'
@@ -253,6 +253,12 @@ Follow the [Deployment](#deployment) guide below for automatic provisioning and
     pm2 deploy ecosystem-imap.json production setup
     ```
 
+    > NOTE: `ecosystem-sqlite-private.json` is ignored in `.gitignore` and not commited to repository as it contains the private IP of the SQLite server.
+
+    ```sh
+    pm2 deploy ecosystem-sqlite-private.json production setup
+    ```
+
 13. Create a SSL certificate at [Namecheap][] (we recommend a 5 year wildcard certificate), set up the certificate, and download and extract the ZIP file with the certificate (emailed to you) to your computer. We do not recommend using tools like [LetsEncrypt][] and `certbot` due to complexity when you have (or scale to) a cluster of servers set up behind load balancers.  In other words, we've tried approaches like `lsyncd` in combination with `crontab` for `certbot` renewals and automatic checking.  Furthermore, using this exposes the server(s) to downtime as ports `80` and `443` may need to be shut down so that `certbot` can use them for certificate generation.  This is not a reliable approach, and simply renewing certificates once a year is vastly simpler and also makes using load balancers trivial.  Instead you can use a provider like [Namecheap][] to get a cheap SSL certificate, then run a few commands as we've documented below. This command will prompt you for an absolute file path to the certificates you downloaded. Renewed your certificate after 1 year? Simply follow this step again.  Do not set a password on the certificate files.  When using the `openssl` command (see Namecheap instructions), you need to use `*.example.com` with an asterisk followed by a period if you are registering a wildcard certificate.
 
     ```sh
@@ -270,6 +276,7 @@ Follow the [Deployment](#deployment) guide below for automatic provisioning and
     pm2 deploy ecosystem-api.json production exec "pm2 reload all"
     pm2 deploy ecosystem-smtp.json production exec "pm2 reload all"
     pm2 deploy ecosystem-imap.json production exec "pm2 reload all"
+    pm2 deploy ecosystem-sqlite-private.json production exec "pm2 reload all"
     ```
 
 14. (Optional) Create a Google application credentials profile file and store it locally.  You only need this if you want to support automatic translation.  The following command will prompt you for the absolute file path (e.g. `/path/to/client-profile.json`).  See the [mandarin][] docs for more information.
@@ -306,6 +313,12 @@ Follow the [Deployment](#deployment) guide below for automatic provisioning and
     pm2 deploy ecosystem-imap.json production
     ```
 
+    > NOTE: `ecosystem-sqlite-private.json` is ignored in `.gitignore` and not commited to repository as it contains the private IP of the SQLite server.
+
+    ```sh
+    pm2 deploy ecosystem-sqlite-private.json production
+    ```
+
 17. Save the process list on the servers so when if the server were to reboot, it will automatically boot back up the processes:
 
     ```sh
@@ -328,6 +341,12 @@ Follow the [Deployment](#deployment) guide below for automatic provisioning and
     pm2 deploy ecosystem-imap.json production exec "pm2 save"
     ```
 
+    > NOTE: `ecosystem-sqlite-private.json` is ignored in `.gitignore` and not commited to repository as it contains the private IP of the SQLite server.
+
+    ```sh
+    pm2 deploy ecosystem-sqlite-private.json production exec "pm2 save"
+    ```
+
 18. Test by visiting your web and API server in your browser (click "proceed to unsafe" site and bypass certificate warning).
 
 19. Configure your DNS records for the web and API server hostnames and respective IP addresses.

diff --git a/ansible/playbooks/security.yml b/ansible/playbooks/security.yml
@@ -22,6 +22,21 @@
           * hard core 0
           * soft core 0
 
+    # https://www.mongodb.com/docs/manual/tutorial/transparent-huge-pages/
+    # https://unix.stackexchange.com/questions/99154/disable-transparent-hugepages
+    # https://stackoverflow.com/questions/51246128/disabling-thp-transparent-hugepages-with-ansible-role
+    - name: "Disable Transparent Huge Pages (THP)"
+      template:
+        src: "{{ playbook_dir }}/templates/thp.j2"
+        dest: /etc/systemd/system/disable-transparent-huge-pages.service
+        owner: root
+        mode: "0644"
+    - name: "Enable THP Systemd Service"
+      service:
+        daemon_reload: yes
+        name: disable-transparent-huge-pages
+        enabled: true
+        state: started
     - name: Ensure fs.suid_dumpable is set to 0 and added in sysctl
       sysctl:
         name: fs.suid_dumpable

diff --git a/ansible/playbooks/sqlite.yml b/ansible/playbooks/sqlite.yml
@@ -0,0 +1,43 @@
+# Copyright (c) Forward Email LLC
+# SPDX-License-Identifier: BUSL-1.1
+
+---
+- name: Import security playbook
+  ansible.builtin.import_playbook: security.yml
+- name: Import Node.js playbook
+  ansible.builtin.import_playbook: node.yml
+- name: Import SSH keys playbook
+  ansible.builtin.import_playbook: ssh-keys.yml
+
+- hosts: sqlite
+  name: SQLITE
+  become: true
+  become_user: root
+  # this was already defined in the ufw role
+  # https://github.com/Oefenweb/ansible-ufw/blob/master/handlers/main.yml
+  handlers:
+    - name: Reload UFW
+      ufw:
+        state: reloaded
+  tasks:
+    # ufw
+    - name: Enable ufw
+      ufw:
+        state: enabled
+        policy: deny
+        direction: incoming
+    - name: Limit ufw ssh
+      ufw:
+        rule: limit
+        port: 22
+        proto: tcp
+    - name: Allow ssh
+      ufw:
+        rule: allow
+        port: 22
+        proto: tcp
+    - name: Allow websocket port
+      ufw:
+        rule: allow
+        port: {{ lookup('env', 'SQLITE_WEBSOCKET_PORT' }}
+        proto: tcp
diff --git a/ansible/playbooks/templates/ecosystem-smtp.json.j2 b/ansible/playbooks/templates/ecosystem-smtp.json.j2
@@ -1,50 +1,15 @@
 {
   "apps": [
     {
-      "name": "smtp-tls",
-      "script": "smtp.js",
+      "name": "sqlite",
+      "script": "sqlite.js",
       "exec_mode": "cluster",
       "wait_ready": true,
       "instances": "max",
       "pmx": false,
-      "env_production": {
-        "NODE_ENV": "production",
-        "SMTP_PORT": 2587
-      }
-    },
-    {
-      "name": "smtp-ssl",
-      "script": "smtp.js",
-      "exec_mode": "cluster",
-      "wait_ready": true,
-      "instances": "max",
-      "pmx": false,
-      "env_production": {
-        "NODE_ENV": "production",
-        "SMTP_PORT": 2465
-      }
-    },
-    {
-      "name": "smtp-bree",
-      "script": "smtp-bree.js",
-      "exec_mode": "fork",
-      "wait_ready": true,
-      "instances": "1",
-      "pmx": false,
       "env_production": {
         "NODE_ENV": "production"
       }
     }
-  ],
-  "deploy": {
-    "production": {
-      "user": "deploy",
-      "host": [{% for host in groups['smtp'] %}"{{ hostvars[host].ansible_host }}"{% if not loop.last %}, {% endif %}{% endfor %}],
-      "ref": "origin/master",
-      "repo": "{{ lookup('env', 'GITHUB_REPO') }}",
-      "path": "/var/www/production",
-      "pre-deploy": "git reset --hard",
-      "post-deploy": "pnpm install && NODE_ENV=production npm start build && pm2 startOrGracefulReload ecosystem-smtp.json --env production --update-env"
-    }
-  }
+  ]
 }
diff --git a/ansible/playbooks/templates/hosts.yml b/ansible/playbooks/templates/hosts.yml
@@ -34,3 +34,7 @@ all:
           ansible_host: 0.0.0.0
         imap-do-am-nl:
           ansible_host: 0.0.0.0
+    sqlite:
+      hosts:
+        sqlite-do-sf-ca:
+          ansible_host: 0.0.0.0
diff --git a/ansible/playbooks/templates/thp.j2 b/ansible/playbooks/templates/thp.j2
@@ -0,0 +1,13 @@
+[Unit]
+Description=Disable Transparent Huge Pages (THP)
+DefaultDependencies=no
+After=sysinit.target local-fs.target
+Before=mongod.service
+
+[Service]
+Type=oneshot
+ExecStart=/bin/sh -c 'echo never | tee /sys/kernel/mm/transparent_hugepage/enabled > /dev/null'
+ExecStart=/bin/sh -c 'echo never | tee /sys/kernel/mm/transparent_hugepage/defrag > /dev/null'
+
+[Install]
+WantedBy=multi-user.target
diff --git a/app/models/aliases.js b/app/models/aliases.js
@@ -94,6 +94,13 @@ const Aliases = new mongoose.Schema({
     type: Number,
     default: 0
   },
+  storageLocation: {
+    type: String,
+    default: 'storage_do_1',
+    enum: ['storage_do_1'],
+    trim: true,
+    lowercase: true
+  },
   retention: {
     type: Number,
     default: 0,
@@ -601,15 +608,22 @@ async function getStorageUsed(wsp, session) {
       i18n.translateError('DOMAIN_DOES_NOT_EXIST_ANYWHERE', 'en')
     );
 
-  const aliasIds = await this.distinct('id', {
+  const aliases = await this.find({
     domain: { $in: domainIds }
-  });
+  })
+    .select({
+      _id: -1,
+      id: 1,
+      storageLocation: 1
+    })
+    .lean()
+    .exec();
 
   // now get all aliases that belong to any of these domains and sum the storageQuota
   const size = await wsp.request({
     action: 'size',
     session: { user: session.user },
-    alias_ids: aliasIds
+    aliases
   });
 
   return size;

diff --git a/app/views/about/index.md b/app/views/about/index.md
@@ -44,6 +44,8 @@ In March 2023, we released [Tangerine](https://github.com/forwardemail/tangerine
 
 In April 2023, we implemented and automated entirely new infrastructure.  Our entire service is now running on globally load-balanced and proximity-based DNS (with health checks and failover) using [Cloudflare](https://cloudflare.com) (before we were using round-robin DNS on Cloudflare).  Additionally we switched to **bare metal servers** across multiple providers – which include [Vultr](https://www.vultr.com/?ref=7429848) and [Digital Ocean](https://m.do.co/c/a7fe489d1b27) (before we solely used Digital Ocean).  Both of these providers are SOC 2 Type 2 compliant – see [Vultr's Compliance](https://www.vultr.com/legal/compliance/) and [Digital Ocean's Certifications](https://www.digitalocean.com/trust/certification-reports) for more insight.  Furthermore, our MongoDB and Redis databases are now running on clusters with primary and standby nodes for high availability, end-to-end SSL encryption, encryption-at-rest, and point-in-time recovery (PITR).
 
-In May 2023, we launched our **outbound SMTP** add-on for [sending email with SMTP](/faq#do-you-support-sending-email-with-smtp) and [sending email with API](/faq#do-you-support-sending-email-with-api) requests.  This feature has built-in safeguards to ensure high deliverability, a modern and robust queue and retry system, and [supports error logs in real-time](/faq#do-you-store-error-logs).
+In May 2023, we launched our **outbound SMTP** feature for [sending email with SMTP](/faq#do-you-support-sending-email-with-smtp) and [sending email with API](/faq#do-you-support-sending-email-with-api) requests.  This feature has built-in safeguards to ensure high deliverability, a modern and robust queue and retry system, and [supports error logs in real-time](/faq#do-you-store-error-logs).
+
+In November 2023, we launched our [**encrypted mailbox storage**](/encrypted-email) feature for [IMAP suppport](/faq#do-you-support-receiving-email-with-imap).
 
 [arc]: https://en.wikipedia.org/wiki/Authenticated_Received_Chain