Skip to content

Commit

Permalink
Merge in emoncms v8
Browse files Browse the repository at this point in the history 
Merge remote-tracking branch 'upstream/master' into debian-packaging

Conflicts:
	default.settings.php
  • Loading branch information
@Dave-McCraw
Dave-McCraw committed Mar 15, 2014
2 parents a4c98f3 + 616a42d commit bb1c01e
Show file tree
Hide file tree
Showing 134 changed files with 15,134 additions and 9,009 deletions.
1 change: 0 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ settings.php
Modules/sync
Modules/event
Modules/energyaudit
Modules/node
Modules/auto
Modules/energy
Modules/energyform
Expand Down
4 changes: 2 additions & 2 deletions COPYRIGHT.txt
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Emoncms - open source energy visualisation
Part of the openenergymonitor.org project

Copyright (C) 2011,2012,2013 the OpenEnergyMonitor researchers (see individual contributions from github commits)
Copyright (C) 2011,2012,2013,2014 the OpenEnergyMonitor researchers (see individual contributions from github commits)

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
Expand All @@ -21,4 +21,4 @@
Trystan Lea
trystan.lea@gmail.com
http://openenergymonitor.org

6 changes: 6 additions & 0 deletions debian/changelog
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
emoncms (8.0) unstable; urgency=low

* Release of emoncms version 8

-- Dave McCraw <dave@mccraw.co.uk> Sat, 15 Mar 2014 10:05:55 +0000

emoncms (7.0) unstable; urgency=low

* Initial debian packaged release of emoncms - previous versions not packaged but available through git.
Expand Down
279 changes: 98 additions & 181 deletions readme.md
View file
Original file line number Diff line number Diff line change
@@ -1,129 +1,28 @@
# Emoncms v7 (redis)
# Emoncms 8

As part of recent work to improve the performance of emoncms because of high load's on emoncms.org redis was introduced to store feed and input meta data including last feed time and value fields which where causing significant write load on the server. This change benefits all installation types of emoncms whether emoncms.org or a raspberrypi as it siginficantly reduces the amount of disk writes.
# Installation on Linux

Using redis in this way leads to quite a big performance improvement. Enabling almost 5 times the request rate in benchmarking.
## Install dependencies

Blog post: [http://openenergymonitor.blogspot.co.uk/2013/11/improving-emoncms-performance-with_8.html](http://openenergymonitor.blogspot.co.uk/2013/11/improving-emoncms-performance-with_8.html)
You may need to start by updating the system repositories

To upgrade you will need redis server installed and the phpredis client:
sudo apt-get update

sudo apt-get install redis-server
sudo pecl install redis

Add pecl redis module to php5 config

sudo sh -c 'echo "extension=redis.so" > /etc/php5/apache2/conf.d/20-redis.ini'
sudo sh -c 'echo "extension=redis.so" > /etc/php5/cli/conf.d/20-redis.ini'

# Emoncms v6 (timestore+)

See main site: http://emoncms.org

Emoncms is an open source energy visualisation web application, the main feature's include

## Input processing
Input processing allows for conversion and processing before storage, there are over 15 different input processes from simple calibration to power to kWh-per-day data, or histogram data.

## Visualisation
Zoom through large datasets with flot and ajax powered level-of-detail amazing super powered graphs!

## Visual dashboard editor
Create dashboards out of a series of widgets and visualisations with a fully visual drag and drop dashboard editor.

## Open Source
We believe open source is a better way of doing things and that our cloud based web applications should also be open source.

Emoncms is part of the OpenEnergyMonitor project. A project to build a fully open source hardware and software energy monitoring platform.

With Emoncms you can have full control of your data, you can install it on your own server or you can use this hosted service.
Install all dependencies:

Emoncms is licenced under the GPL Affero licence (AGPL)
sudo apt-get install apache2 mysql-server mysql-client php5 libapache2-mod-php5 php5-mysql php5-curl php-pear php5-dev php5-mcrypt git-core redis-server build-essential ufw ntp

# v6 (timestore+)
Install pecl dependencies (serial, redis and swift mailer)

Emoncms version 6 brings in the capability of a new feed storage engine called timestore.
Timestore is time-series database designed specifically for time-series data developed by Mike Stirling.

[mikestirling.co.uk/redmine/projects/timestore](mikestirling.co.uk/redmine/projects/timestore)

Timestore's advantages:

**Faster Query speeds**
With timestore feed data query requests are about 10x faster (2700ms using mysql vs 210ms using timestore).
*Note:* initial benchmarks show timestore request time to be around 45ms need to investigate the slightly slower performance may be on the emoncms end rather than timestore.

**Reduced Disk use**
Disk use is also much smaller, A test feed stored in an indexed mysql table used 170mb, stored using timestore which does not need an index and is based on a fixed time interval the same feed used 42mb of disk space.

**In-built averaging**
Timestore also has an additional benefit of using averaged layers which ensures that requested data is representative of the window of time each datapoint covers.

### Using MYSQL or PHPTimeSeries instead of Timestore

If your a familiar with mysql and want to use mysql to do your own queries and processing of the feed data you may want to select mysql as the default data store rather than timestore. The disadvantage of MYSQL is that it is much slower than timestore for common timeseries queries such as zooming through timeseries data.

There is also another feed engine called PHPTimeSeries which provides improved timeseries query speed than mysql but is still slower than timestore. Its main avantages is that it does not require additional installation of timestore as it uses native php file access, it also stores the data in the same data file .MYD format as mysql which means you can switch from mysql to phptimeseries by copying the .MYD mysql data files directly out of your mysql directory into the PHPTimeSeries directory without additional conversion.

To select either MYSQL or PHPTimeSeries instead of timestore as your default engine set the default engine setting in the emoncms settings.php file to:

$default_engine = Engine::MYSQL;
sudo pear channel-discover pear.swiftmailer.org
sudo pecl install channel://pecl.php.net/dio-0.0.6 redis swift/swift

or:

$default_engine = Engine::PHPTIMESERIES;

If you do not wish to use timestore you can skip to step 2 of the installation process.

If you want to try PHPTimeSeries see optional PHPTimeSeries step below.

# Installation

The following details how to install emoncms on linux from scratch including lamp server install and config.

## 1) Download, make and start timestore

cd /home/username
git clone https://github.com/TrystanLea/timestore
cd timestore
sudo sh install
Add pecl modules to php5 config

**Note the adminkey** at the end as you will want to paste this into the emoncms settings.php file.

If the adminkey could not be found, it may be that timestore failed to start:

To check if timestore is running type:

sudo /etc/init.d/timestore status

Start, stop and restart it with:

sudo /etc/init.d/timestore start
sudo /etc/init.d/timestore stop
sudo /etc/init.d/timestore restart

To read the adminkey manually type:

cat /var/lib/timestore/adminkey.txt

## (Optional) Create PHPTimeSeries data folder

If you wish to try the phptimeseries engine:

sudo mkdir /var/lib/phptimeseries
sudo chown www-data:root /var/lib/phptimeseries

## 2) Install Apache, Mysql and PHP (LAMP Server)

When installing mysql and the blue dialog appears enter a password for root user, note the password down as you will need it later.

$ sudo apt-get install apache2
$ sudo apt-get install mysql-server mysql-client
$ sudo apt-get install php5 libapache2-mod-php5
$ sudo apt-get install php5-mysql
$ sudo apt-get install php5-curl

## 3) Enable mod rewrite
sudo sh -c 'echo "extension=dio.so" > /etc/php5/apache2/conf.d/20-dio.ini'
sudo sh -c 'echo "extension=dio.so" > /etc/php5/cli/conf.d/20-dio.ini'
sudo sh -c 'echo "extension=redis.so" > /etc/php5/apache2/conf.d/20-redis.ini'
sudo sh -c 'echo "extension=redis.so" > /etc/php5/cli/conf.d/20-redis.ini'

Emoncms uses a front controller to route requests, modrewrite needs to be configured:

Expand All @@ -134,17 +33,14 @@ Change (line 7 and line 11), "AllowOverride None" to "AllowOverride All".
That is the sections <Directory /> and <Directory /var/www/>.
[Ctrl + X ] then [Y] then [Enter] to Save and exit.


Restart the lamp server:

$ sudo /etc/init.d/apache2 restart

## 4) Install the emoncms application via git
### Install the emoncms application via git

Git is a source code management and revision control system but at this stage we use it to just download and update the emoncms application.

$ sudo apt-get install git-core

First cd into the var directory:

$ cd /var/
Expand All @@ -157,18 +53,15 @@ Cd into www directory

$ cd www

If you do not yet have emoncms installed, git clone to download:
Download emoncms using git:

$ git clone https://github.com/emoncms/emoncms.git

If you do already have emoncms installed via the git clone command you can download the latest changes with:
Once installed you can pull in updates with:

git pull

Alternatively download emoncms and unzip to your server:
[https://github.com/emoncms/emoncms](https://github.com/emoncms/emoncms)

## 5) Create a MYSQL database

### Create a MYSQL database

$ mysql -u root -p

Expand All @@ -180,8 +73,18 @@ Then enter the sql to create a database:
Exit mysql by:

mysql> exit

### Create data repositories for emoncms feed engine's

## 6) Set emoncms database settings.
sudo mkdir /var/lib/phpfiwa
sudo mkdir /var/lib/phpfina
sudo mkdir /var/lib/phptimeseries

sudo chown www-data:root /var/lib/phpfiwa
sudo chown www-data:root /var/lib/phpfina
sudo chown www-data:root /var/lib/phptimeseries

### Set emoncms database settings.

cd into the emoncms directory where the settings file is located

Expand All @@ -201,29 +104,65 @@ Enter in your database settings.
$password = "PASSWORD";
$server = "localhost";
$database = "emoncms";

Save (Ctrl-X), type Y and exit

### Install add-on emoncms modules

If your using timestore enter the adminkey as copied in step 1 above:

$timestore_adminkey = "";
cd /var/www/emoncms/Modules

If your not using timestore set the default engine to your selected engine:
git clone https://github.com/emoncms/raspberrypi.git
git clone https://github.com/emoncms/event.git
git clone https://github.com/emoncms/openbem.git
git clone https://github.com/emoncms/energy.git
git clone https://github.com/emoncms/notify.git
git clone https://github.com/emoncms/report.git
git clone https://github.com/emoncms/packetgen.git
git clone https://github.com/elyobelyob/mqtt.git

See individual module readme's for further information on individual module installation.

$default_engine = Engine::MYSQL;

or
### In an internet browser, load emoncms:

$default_engine = Engine::PHPTIMESERIES;
[http://localhost/emoncms](http://localhost/emoncms)

Save (Ctrl-X), type Y and exit
The first time you run emoncms it will automatically setup the database and you will be taken straight to the register/login screen.

Create an account by entering your email and password and clicking register to complete.

## 7) In an internet browser, load emoncms:
#### Note: Browser Compatibility

http://IP-ADDRESS/emoncms
**Chrome Ubuntu 23.0.1271.97** - developed with, works great.

The first time you run emoncms it will automatically setup the database and you will be taken straight to the register/login screen.
**Chrome Windows 25.0.1364.172** - quick check revealed no browser specific bugs.

**Firefox Ubuntu 15.0.1** - no critical browser specific bugs, but movement in the dashboard editor is much less smooth than chrome.

**Internet explorer 9** - works well with compatibility mode turned off. F12 Development tools -> browser mode: IE9. Some widgets such as the hot water cylinder do load later than the dial.

**IE 8, 7** - not recommended, widgets and dashboard editor <b>do not work</b> due to no html5 canvas fix implemented but visualisations do work as these have a fix applied.

# Shared Linux Hosting

Your shared hosting provider should already have a LAMP server installed. You may need to ask your hosting provider to enable mod_rewrite. It's unlikely that redis will be available (redis is used to improve performance through caching), but emoncms can be run without it.

To install emoncms on a shared server

1) Download the (currently rework branch) zip file from:

[https://github.com/emoncms/emoncms/archive/rework.zip]([https://github.com/emoncms/emoncms/archive/rework.zip])

Unzip to your shared server's public_html folder, rename the folder to emoncms.

2) Create a mysql database for your emoncms installation, note down its name, username and password.

3) In your shared hosting /home/username folder create a folder called emoncmsdata to hold your emoncms feed data. (Note: NOT public_html as the data files should not be publicly accessible).
Then create three folders within your emoncmsdata folder called: phpfiwa, phpfina and phptimeseries

4) In the emoncms app directory make a copy of default_settings.php and call it settings.php. Open settings.php and enter your mysql username, password and database. In the feed_settings section uncomment the datadir defenitions and set them to the location of each of the feed engine data folders on your system.

5) Thats it, emoncms should now be ready to use!

Create an account by entering your email and password and clicking register to complete.
<br><br>

#### PHP Suhosin module configuration (Debian 6, not required in ubuntu)

Expand All @@ -238,54 +177,31 @@ value (8000, 16000 should be fine).

Follow the guide here step 4 onwards: [http://emoncms.org/site/docs/gettext](http://emoncms.org/site/docs/gettext)

# Upgrading
#### Configure PHP Timezone

If your upgrading from emoncms version 5 up to 6 you will need to:
PHP 5.4.0 has removed the timezone guessing algorithm and now defaults the timezone to "UTC" on some distros (i.e. Ubuntu 13.10). To resolve this:

Install timestore as in step 1 above.
Open php.ini

Install php curl:

sudo apt-get install php5-curl
sudo vi /etc/php5/apache2/php.ini

Run git pull in your emoncms directory
and search for "date.timezone"

cd /var/www/emoncms
git pull

Create a fresh copy of default.settings.php with your mysql database settings and setting the timestore adminkey as in step 6 above.
[Date]
; Defines the default timezone used by the date functions.
; http://php.net/date.timezone
;date.timezone =

Log in with the administrator account (first account created)
edit date.timezone to your appropriate timezone:

Click on the *Admin* tab (top-right) and run database update.

Click on feeds, check that everything is working as expected, if your monitoring equipment is still posting you should see data coming in as usual.

## Converting existing feeds to timestore

So far we've got everything in place for using timestore but any existing feeds are still stored as mysql tables. To convert existing mysql feeds over to timestore a module has been written specifically for managing the conversion of the feeds, to download and run it:

cd /var/www/emoncms/Modules

git clone https://github.com/emoncms/converttotimestore
date.timezone = "Europe/Amsterdam"

Again log in with the administrator account (first account created)
Click on the *Admin* tab (top-right) and run database update.
PHP supported timezones are listed here: http://php.net/manual/en/timezones.php

Navigate to the convert to timestore menu item in the dropdown menu titled Extras and follow the steps outlined.


## Need help?
See timestore forum discussion: [http://openenergymonitor.org/emon/node/2651](http://openenergymonitor.org/emon/node/2651)

## Upgrading from version 4.0
Now save and close and restart your apache.

If your updating from an installation thats older than the 12th of April 2013, the process of upgrading should be much the same as the above. If you cant login in the last step, try adding the line

$updatelogin = true;
sudo /etc/init.d/apache2 restart

to settings.php to enable a special database update only session, be sure to remove this line from settings.php once complete.

# Developers
Emoncms is developed and has had contributions from the following people.

Expand All @@ -306,4 +222,5 @@ Emoncms is developed and has had contributions from the following people.
- thunderace https://github.com/thunderace
- pacaj2am https://github.com/pacaj2am
- Ynyr Edwards https://github.com/ynyreds

- Jerome https://github.com/Jerome-github
- fake-name https://github.com/fake-name
Loading

0 comments on commit bb1c01e

Please sign in to comment.