The BlackServer
- 0. Why do such a crazy thing?
- 1. What this document ...
- 2. Basic Configuration
- 3. Install applications on your server
- Caveats, Workarounds, and Bugs
- References
Why pay $$ to rent a crappy computer somewhere else? Host your own website on your terms.
Wouldn't it be awesome to have your computer backup wirelessly, automatically, no matter where you are?
Imagine having a personalised Netflix, filled with your own collection of awesome movies and TV shows and what have you.
Don't let your personal information be the currency with which you pay for essential services on the internet. You shouldn't have to relinquish privacy just to read a stupid email.
Here is a handy table listing services you probably use, alternatives you can host yourself, and the status of the offered solution (if any)
| Service | Example | Alternative | Status |
|---|---|---|---|
| GMail | ?? | ?? | |
| static webserver | DreamHost | MAMP + your own server | working |
| file upload service | Mega | upload + your own server | working |
| git server | GitHub | GitLab + your own server | ?? |
| A read-it-later service | wallabag + your own server | working | |
| web-based YouTube downloader | your favorite spammy site | video + youtube-dl + your own server | working |
| "Cloud" backup | Condi's Dropbox | syncthing + your own server | working |
| torrent tracker | The Pirate Bay | ?? | ?? |
| personal organiser | Evernote | MediaWiki + your own server | working |
| Video streaming service | Netflix | Emby + your own server | working |
| Online collaborative document editor | Google Docs | Mozilla etherpad | working |
This is the configuration of my MacBook server:
- 2GB RAM (upgradable to 6GB)
- 128GB SSD
- .5TB of additional on-machine storage (I swapped out my optical drive)
- 4TB of additional, external storage (1 FireWire port, 2 USB ports)
- the way I did this
- a way for me document what I did
- The best way to do this
- A secure way to do this
Because my computer was rather long in the tooth, and didn't have anything on it I was very attached to, I booted from a Snow Leopard install disc, and did a fresh install. I then updated to the last version of Snow Leopard and then upgraded to Lion, and then patched Lion with all the updates I could lay my hands on. My final configuration is:
Mac OS X 10.7.5 (Lion)
I couldn't update to Mountain Lion because that was the limit of the Black MacbookBook
You need the XCode Command Line Tools to install a number of other things. The last version I could install is Command Line Tools (OS X Lion) for Xcode - April 2013
Here's a link to the disk image. You'll need a free Apple Developer account to get this.
brew is a fantastic pckage manger for Mac OS X. You will definitely need this. Install with this single line:
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"It will ask you for your password. Enter one.
Now that brew is installed, it is super easy to install a huge bunch of things. Note that some of these depend on the order you install them.
brew install youtube-dl # download YouTube videos nicely
brew install ruby # the ruby programming language
brew install git # the best version control software
brew install ffmpeg # the best way to convert video
brew install node # server-side JavaScript
brew install wget # download things easily
brew install syncthing # libre file synchronization
brew install moreutils # installs [moreutils](https://rentes.github.io/unix/utilities/2015/07/27/moreutils-package/)You can also brew install thing1 thing2
Wow. we just installed a bunch of useful command line utilities. What about a better browser and some GUI apps? brew cask has you covered.
brew install caskroom/cask/brew-cask
brew cask install firefox # the best free browser
brew cask install mamp # Apache, mySQL and PHP
brew cask install sublime-text # the best text editor
brew cask install torbrowser # simple anonymity
brew cask install carbon-copy-cloner # backups made easy
brew cask install caffeine # keep your screen onOne of the complications(features?) of using brew is that it installs its own versions of things that Mac OS X already has (like git). To make sure that we always use the brew version, and not the OS X version, set your $PATH so that the brew path occurs before the system path:
echo export PATH='/usr/local/bin:$PATH' >> ~/.bash_profile
Why do this? Because the brew versions are more secure, more recent, and are going to be maintained.
The latest (v1.40) version of NoSleep is broken on Lion, and we'll have to install an older one. See this issue on their repo page. You will have to download v1.3.3 here and manually install it. Make sure you enable "don't check for updates" because as of writing, the latest version is broken, and it will break itself trying to update.
Go to System Preferences > Network and set up a static IP as shown:
What we're doing here is making sure that BlackServer has the same IP address on the local network (your router's network). This means it's at the same place on the network, and that your router can be told to send messages to the BlackServer easily.
We'll need to get in and out of this computer from all over the world soon. And in the beginning at least, it would be nice to have a way to log into the computer and see the screen. Let's set up all this in System Preferences > Sharing
But we're not done yet! For added security, let's configure BlackServer so that we can SSH into it only using public key authentication. This way, there are no passwords to type in, and it's much harder for someone to break into our computer.
Assuming you have made your SSH keys on your local machine (not BlackServer!), copy them to BlackServer
scp ~/.ssh/id_rsa.pub user@blackserver:~/Desktop/Now, on BlackServer, run this:
mkdir ~/.ssh
cat ~/Desktop/id_rsa.pub >> ~/.ssh/authorized_keysOK. BlackServer can now verify your local machine using public key cryptography. Now we need to configure BlackServer to make sure it only does this:
Fire up Sublime Text to edit the SSH config file:
subl /etc/sshd_configand uncomment these lines
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keysAlso, let's prevent authentication using passwords. Change this file to:
# To disable tunneled clear text passwords both PasswordAuthentication and
# ChallengeResponseAuthentication must be set to "no".
PasswordAuthentication no
#PermitEmptyPasswords no
# Change to no to disable s/key passwords
ChallengeResponseAuthentication noWARNING At this point, you can only get into BlackServer via public key authentication. If you lose your SSH keys, or if someone steals them, you have a problem.
Start MAMP (be careful not to run MAMP Pro, which will also be installed) and set it up as follows:
OK, let's see if this works (you will have to enter the local IP address of of your computer instead of black.server):
Excellent. It looks like MAMP works, and we have an Apache server running. Now, MAMP by default does not enable access logging. Access logging allows us to see who is trying to connect to the BlackServer, and is also a useful debugging tool. To enable this,
subl /Applications/MAMP/conf/apache/httpd.conf And remove the comment (the #) from this line
# CustomLog "/Applications/MAMP/logs/apache_access.log" commonRestart your server for this to take effect. You can view the access log using
$ tail -f /Applications/MAMP/logs/apache_access_logIt would be nice to access BlackServer from anywhere in the world. Right now, we can't do that, for two reasons:
- we don't know where in the internet
BlackServeris - the router that
BlackServeris on doesn't know what to do with packets coming from and going toBlackServer. In fact, it is designed to ignore everything by default.
Let's address both problems.
To tell your router how to pass on messages to/from BlackServer, you need to do something called port forwarding. Unfortunately, every router is different, and some routers are so stupid they don't allow you to do that. You have to figure out how to do this on yours. Go to your router's admin page (usually 192.168.1.1) and enter your username and password (usually admin/admin or something silly)
Find a page that looks like this, and add entries as follows:
The specific ports you forward depend on the applications and services you will install on your server, but you get the idea.
Now, we need a service that translates a short name into the IP address of BlackServer (or more precisely, the IP address of the router BlackServer is on). For this, we use a bit of software from no-ip:
brew cask install no-ip-ducConfiguring this piece of software is easy, and you can get it to update a URL you control with the IP address BlackServer is on. Let's assume that the domain name you control is black.server.
composer is a dependency manager for PHP. You will need it if you install wallabag.
Because of the chaos we have unleashed with MAMP, a PHP configuration file called php.ini will not be in the "right" place. To fix that, run
sudo cp /etc/php.ini.default /etc/php.iniAnd fire up Sublime text and add this line
detect_unicode = Offusing
subl /etc/php.inihat tip to Tony Lea
Finally, we can install composer using the installer:
curl -s http://getcomposer.org/installer | phpA hosts file is a file that maps domain names to IP addresses. This is the first thing your computer looks at when resolving names. Think of it as your own local DNS. On Mac OS X, it lives here: /etc/hosts
I recommend copying Steven Black's hosts file, as it in itself is a aggregation of other hosts files, and is frequently updated.
Default installs of Mac OS X ship with a bunch of certificates. If the people issuing these certificates are compromised, or are otherwise untrustworthy, there is a weakness in your system. Consider revoking some certificates using the following guides:
Wouldn't it be nice if you could operate your own file upload service? If people wanted to send you documents, they could simply upload it to your computer. No more messing around with Condi's Dropbox.
I've written a small file upload service that works straight out of the box. Grab it with
cd /Library/Webserver/Documents/
git clone https://github.com/sg-s/upload
chmod 777 uploadsTo check if you've done the right thing, trying going to black.server/upload from another computer. You should see this:
Try uploading something. You should get a green message telling you it worked. You can go see the file for yourself on BlackServer in /Library/Webserver/Documents/upload/uploads/
3.1 wallabag
Wallabag is a read-it-later service, like Pocket or Instapaper, but with 100% less data stored on other people's servers, because your data is stored on your BlackServer.
I found installation extremely difficult, but here is a easy way to get everything working.
Your $PATH variable tells bash where to look for commands. Edit your ~/.bash_profile to look like:
export PATH=/Applications/MAMP/bin/php/php5.6.2/bin:/usr/local/bin:$PATHThis tells the computer to use MAMP's PHP, instead of the crappy version built into OS X.
Assuming you followed the instructions here about the location of your php.ini file, add the following lines to your php.ini
extension_dir = "/Applications/MAMP/bin/php/php5.6.2/lib/php/extensions/no-debug-non-zts-20131226/"
extension = pdo_mysql.so
This tells php to load the PDO extension, and points to the right location.
Check that everything works using php --version. You should see this:
PHP 5.6.2 (cli) (built: Oct 20 2014 16:21:27)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2014 Zend Technologies
If you see an error message about missing PDO, you probably haven't fixed your $PATH.
Navigate to the Documents folder of your webserver (here, /Library/WebServer/Documents/) and run
curl -s http://getcomposer.org/installer | php
composer create-project wallabag/wallabag wallabag
mv composer.phar wallabag/
cd wallabag
php composer.phar installThat's it. Now go to localhost/wallabag/
and choose a SQLite database, and a user name and password.
This is probably due to runaway perl processes. Check top to see which processes are hogging the CPU in this case.
Upgrade perl to the latest version using brew
brew install perl
brew link --force perl
Use this script to kill high-CPU processes. Run this using cron.