Installing Storage on a VPS

Here's how I installed Storage on a VPS. I won't go through all the details of setting up a VPS. I'm using Ubuntu 14.04 on Linode and they have great documentation on getting started with a VPS. To install Node.js just follow their instructions.

Software Versions:

nodejs v0.10.33
npm v1.4.28
storage v0.41
apache v2.4.7

Amazon API Keys

To use Storage you will need to have an Amazon Web Services (AWS) account. You can use your normal Amazon login credentials, but may have to activate AWS by going to aws.amazon.com.

Assuming you have an AWS account you will now create a new user with S3 access for Storage to use.

Go to the IAM Management Console and click the button "Create New Users". Enter a descriptive username to represent this user. Make sure the box to generate keys for the user has been checked.
Feel free to download the CSV file with the keys as a backup. Otherwise toggle "Show User Security Credentials" and copy and paste both keys into your notes to be used later in the Storage installation.
Click "Close" and you'll be taken to a list of your users. Click on the user you just created and click "Attach User Policy" and you'll see a list of premade policies you can use. Scroll down and find the policy "Amazon S3 Full Access" and click the button "Select". You'll see a text box with the policy defined in json. Just click the button "Apply Policy" and your user is ready to go.

Amazon S3 Buckets

Now that you have your AWS keys, we'll create two buckets, one private and one public.

Go to the S3 Management Console and click "Create Bucket". For bucket name choose the domain name you will use for the private bucket. For example private.example.com. If you don't have a preference for Region just choose "US Standard" and click the button "Create". If you choose a different Region you have to set the environment variable AWS_REGION accordingly, see Environment Variables paragraph.
There should be a configuration panel for your bucket on the right side of the screen. If there isn't, just click on the magnifying glass next to your bucket in the list on the left.
Click on the heading "Static Website Hosting", toggle "Enable Website Hosting" and enter index.html for both index and error document. Click "Save" and record the value of "Endpoint" in your notes. It should be something like private.example.com.s3-website-us-east-1.amazonaws.com.
Follow the three previous steps again, only this time we're creating a public bucket. The name should be a domain like public.example.com.
To make the public bucket public, we click on the heading "Permissions" and click "Add Bucket Policy".

The bucket policy should be the following, substituting your own domain name in Resource:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "AllowPublicRead",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::public.example.com/*"
        }
    ]
}

Click "Save"

Domain Names and SSL Certificate

You will need to have a domain name configured that points to your Storage installation. You can use an A record or a CNAME since you're pointing to your own VPS. I personally set an A record for the root domain and a generic server subdomain (like web01.example.com) then I pick a subdomain for my storage application (like storage.example.com) and have that be a CNAME pointing at the generic server subdomain.

You will also create CNAME records for the two buckets you created on Amazon S3. The domain names will be the same as the bucket names (ex: public.example.com) and the value of the CNAME record will be the endpoint you recorded in your notes (ex: public.example.com.s3-website-us-east-1.amazonaws.com).

You will also need to have an SSL Certificate. This can either be purchased through your domain name registrar or self generated. This however is outside of the scope of this tutorial. A good tutorial for this is SSL Certificates with Apache 2 on Ubuntu.

In your notes you may want to record the domain name you'll be using for your Storage installation.

Twitter API Keys

Go to dev.twitter.com and make sure you're logged in with your Twitter user. You should see your avatar in the upper right corner of the page.
Scroll down to the bottom of the page and click "Manage Your Apps" under the "Tools" heading.
Click the button "Create New App" most of the fields for this form should be self explanatory. For Callback URL I put in the value https://storage.example.com/callbackfromtwitter substituting your own domain. Make sure you click the checkbox agreeing to their developer agreement and click "Create your Twitter application".
Next click on the "Permissions" tab and change access to "Read and Write" and click "Update Settings".
Click on the "Keys and Access Tokens" tab and record in your notes the values of Consumer Key and Consumer Secret.

Environment Variables

Now you have all the information you need to configure the environment variables for your Storage installation.

Edit ~/.profile with your favorite text editor (I prefer vim)

Add the following lines to your ~/.profile file, substituting values from your notes:

# Settings for scripting/storage
export PORT=8080 # Any number over 1024
export twitterConsumerKey=TwitterConsumerKey
export twitterConsumerSecret=TwitterConsumerSecret
export AWS_ACCESS_KEY_ID=AmazonAccessKey
export AWS_SECRET_ACCESS_KEY=AmazonSecretAccessKey
# Edit/enable the following line only if you select a Region other than US Standard
#export AWS_REGION=us-east-1 # See http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
export s3Path=/public.example.com/
export s3PrivatePath=/private.example.com/
export myDomain=storage.example.com
export TZ=America/Chicago # See http://en.wikipedia.org/wiki/List_of_tz_database_time_zones

Save the ~/.profile file and reapply it by either logging out and logging back in again or by typing source ~/.profile in the terminal.

Installing Storage

Now that we have the environment variables configured for Storage, we'll do the actual installation.

Choose where you want to install Storage. I created the directory /var/www/storage.example.com for this purpose.
Go to that directory with cd /var/www/storage.example.com
Clone the repository with git git clone https://github.com/scripting/storage.git .. Note if you don't have git installed on your VPS just run sudo apt-get install git-core if you're using Debian or Ubuntu. For other distributions just google for instructions.
Install the dependencies with the command npm install this will read packages.json and download all the 3rd-party libraries needed to run Storage.
Test your installation by running nodejs storage.js. You should see something like twStorageServer0.41 running on port 8080.
Hit CTRL-C to quit running Storage

Keeping Storage Running With Forever

Running Storage directly with nodejs is not ideal as it will stop running as soon as you log out of your SSH session. To keep it running I recommend the tool Forever.

Type sudo npm install -g forever which will install the tool forever globally on your system
Run Storage with forever by typing forever start storage.js you'll see some warnings (default values are ok) and a final info line saying Forever processing file: storage.js. If you get the message

/usr/bin/env: node: No such file or directory

this means that the node binary is named nodejs where forever is looking for node. Do this:

  ln -s /usr/bin/nodejs /usr/bin/node

If you ever want to stop Storage just run forever stop storage.js

Keeping Storage Running With pm2

If you want your nodeStorage app to continue running across system reboots, you can use pm2 instead of Forever. This provides some added benefits, including logging unhandled exceptions, bringing your app back up after unexpected machine restarts, and so on.

(Note: you may need to prepend sudo to these commands depending upon your npm setup.)

Install PM2 globally:

npm install pm2 -g
Complete the installation:

pm2 completion install
Start your nodeStorage app using PM2 (substitute the path to the storage.js file for your nodeStorage installation):

pm2 start /var/www/storage.example.com/storage.js
You should see your app running. (Use pm2 list to check anytime.)

image: pm2 output

Set up the pm2 startup hook by running this command:

pm2 startup

Review the output, which will show you what command you will need to execute on your system to install the startup hook.

```
$ [PM2] You have to run this command as root. Execute the following command:
$ sudo su -c "env PATH=$PATH:/home/unitech/.nvm/versions/node/v4.3/bin pm2 startup <distribution> -u <user> --hp <home-path>
```

Note: Do not run the command above. Instead, run the command you see on your system after executing the pm2 startup command.

The startup hook will autoload the process list you create. To create yours, use:

pm2 save

pm2 is quite a robust system with lots of flexibility. Check out the docs to see what else you can do. Some of the basic maintenance tasks include:

Keeping pm2 updated: npm install pm2 -g && pm2 update
Managing logs
Managing your app(s) and setting up monitoring

Proxying Requests Through Apache

You won't want to access Storage through port 8080 all the time and being that this is a VPS you may want to run other domain names off of it as well. The way we handle this, as well as allowing Storage to be served via HTTPS is by proxying requests through the Apache web server.

Create a new virtual hosts file for Storage. You will probably have to do this as root (or sudo). I used the command sudo vim /etc/apache2/sites-available/storage.example.com.conf

Here is an example of what that file should contain, substituting your own values:

<VirtualHost *:443>
        ServerAdmin andrew@andrewshell.org
        ServerName storage.example.com

        ProxyRequests off

        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>

        <Location />
                ProxyPass http://localhost:8080/
                ProxyPassReverse http://localhost:8080/
        </Location>

        SSLEngine on
        SSLOptions +StrictRequire

        SSLCertificateFile /etc/apache2/ssl/20141120.wild.example.com.crt
        SSLCertificateKeyFile /etc/apache2/ssl/20141120.wild.example.com.key
        SSLCertificateChainFile /etc/apache2/ssl/20141120.wild.example.com.ca
</VirtualHost>
<VirtualHost *:80>
  ServerAdmin andrew@andrewshell.org
  ServerName storage.geekity.com
  Redirect permanent / https://storage.example.com/
</VirtualHost>

This will proxy all requests to and from https://storage.example.com/ to your local nodejs app running on port 8080. Save this file.

Next we need to enable the modules that handle proxying. You can do this with the following commands:
```
sudo a2enmod proxy
sudo a2enmod proxy_http
```
Then you enable the virtual host you just defined with:
```
sudo a2ensite storage.example.com.conf
```
Then restart Apache with sudo service apache2 restart
You should now be able to go to https://storage.example.com/version and see 0.41

Proxying Requests Through NGINX

If you install nodeStorage on a VPS or other server that uses nginx instead of Apache, this guide will help you set up nginx to proxy incoming requests for your nodeStorage installation. As with the Apache configuration above, this will allow serving nodeStorage via HTTPS. (You must separately install/configure an SSL/TLS certificate.)

The guide below uses some example values. You should substitute:

The port number for your nodeStorage installation. This example uses port 1229.
Your domain name and/or subdomain. This example uses https://storage.example.com/
The location of your nodeStorage installation. This example uses /var/www/storage.example.com
Syntax of nginx commands for your OS/nginx versions. This guide assumes Ubuntu 16.04.XX LTS and nginx 1.10.X.
Other environment-specific parameters (e.g. HTTP vs. HTTPS, location of SSL/TLS certificates, etc.,)

Create an nginx server block for your nodeStorage installation. I use the nano editor on my Ubuntu VPS, so I created this file with the command:

sudo nano /etc/nginx/sites-available/storage.example.com

Here's an example. Your environment may differ.

# Server configuration: Redirects non-HTTPS traffic to HTTPS traffic and proxies nodeStorage requests to port 1229

server {
    listen 80;
    listen [::]:80;
    server_name storage.example.com;
    return 301 https://$server_name$request_uri;
}

server {

    # SSL configuration
    
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    # Use snippets folder to manage SSL/TLS Certificate paths and various parameters
    include snippets/ssl-storage.example.com.conf;
    include snippets/ssl-params.conf;	

    root /var/www/storage.example.com;

    # Add index.php to the list if you are using PHP
    index index.php index.html index.htm index.nginx-debian.html;

    server_name storage.example.com;

    location = /favicon.ico {
            log_not_found off;
            access_log off;
    }   

    location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
    }

    location / {
        # Disabling my usual PHP declaration by commenting line below
        # try_files $uri $uri/ /index.php?$args;


        # Proxy setup for nodeStorage on port 1229
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://localhost:1229;

    }

    # pass PHP scripts to FastCGI via php-fpm socket
    
    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
    
        # With php7.0-cgi alone:
    #	fastcgi_pass 127.0.0.1:9000;
    #	# With php7.0-fpm:
        fastcgi_pass unix:/run/php/php7.0-fpm.sock;
    }

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    
    location ~ /\.ht {
        deny all;
    }

    location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
        expires max;
        log_not_found off;
    }

    # Permits traffic to .well-known subfolder 
    # useful for LetsEncrypt certificates and other purposes
    # See RFC 5785 https://tools.ietf.org/html/rfc5785
    location ~ /.well-known {
        allow all;
    }

}

This will proxy all requests to and from https://storage.example.com/ to your local nodeStorage app running on port 1229. Save this file.

Activate this server block by creating a symlink in the nginx sites-enabled folder. For example:

sudo ln -s /etc/nginx/sites-available/storage.example.com /etc/nginx/sites-enabled/storage.example.com
Let nginx run its checks before restarting (your environment may require different syntax):

sudo nginx -t
If all tests pass, restart nginx (your environment may require different syntax):

sudo systemctl restart nginx
You should now be able to confirm that everything is working by visiting https://storage.example.com/version in your browser. The nodeStorage app should return its version number, for example:

0.9.6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly