ETSource

ETSource contains the data used by Quintel applications for modelling energy transition. The files contained herein are a mixture of human-editable "documents", source files used to do offline calculations with Atlas and Refinery, and files containing the results of these calculations for use in ETEngine.

You may also wish to view the Atlas readme for information on loading the ETSource data in a console, importing data from the old InputExcel output files, or for instructions on exporting data for ETEngine.

Open Source

ETSource is released under the MIT License.

The sole exception to this is the energy balance in datasets. These Energy Balances come from the IEA, that does not allow redistribution of their data. You may want to purchase this data with the IEA and put it in the appropiate dataset directory to have a fully functioning ETSource for that region. You will have to adjust and extend the energy balance with the use of ETDataset.

Please contact us when you have questions.

Setup

Prerequisites:

• rbenv
• ruby
• bundler
• rspec

There's no additional setup required for ETSource. There are however optional arguments you can set in a .env file in the root directory of your local ETSource project. See the .env.default which options are available.

"Active" Documents

Files with an ".ad" extension are editable through the Atlas console and are used by ETEngine to set up the graph structure. These files typically contain global data which applies to all regions.

If you prefer, ActiveDocument files can be editted in your favourite text editor. Each document is split into two sections: a comments section where each line begins with a hash ("#"), a section containing attributes values.

# This is the comment section. You can wrap across as many lines as you want,
# but each one should begin with a hash.
#
# Paragraphs are also acceptable.

- use = energetic
- renewability = 1.0

~ demand = EB(something, something)

Comments Section

The comment section is optional, and should be placed at the top of the document. When the file is loaded by Atlas, the comment is available as the description attribute.

document.description
# => "String containing the file comment"

Attributes Section

The attributes section is parsed so that each one is available with a native Ruby datatype. Attributes are specified with one on each line and preceeded with a dash and space.

# - use = energetic
# - renewability = 1.0

document.use           # => "energetic" (String)
document.renewability  # => 1.0 (Float)

Values may also be an array of numbers or strings by wrapping each one in square brackets.

# - range = [1, 2.0, 3, 4.5, 9]
# - strs  = [a, b, c]

document.range  # => [1, 2.0, 3, 4.5, 9]
document.strs   # => ['a', 'b', 'c']

Multi-line Attributes

An attribute value may span multiple lines so long as each line is intended with spaces:

- description =
    Felis catus is your taxonomic nomenclature,
    An endothermic quadruped, carnivorous by nature;
    Your visual, olfactory, and auditory senses
    Contribute to your hunting skills and natural defenses.

The leading spaces will be trimmed:

# - query =
#     SUM(
#       MAX(1, 2),
#       MAX(3, 4)
#     )

puts document.query

# SUM(
#   MAX(1, 2),
#   MAX(3, 4)
# )

Hashes and Namespaces

You can specify "namespaced" attributes which are converted to hashes in the Ruby model:

# - efficiency.gas = 0.5
# - efficiency.electricity = 0.6

document.efficiency  # => { gas: 0.5, electricity: 0.6 }

# - one.two = 2
# - one.three.four = 4

document.one # => { two: 2, three: { four: 4 } }

Dynamic Attributes

All of the attributes described so far are "static" -- they are the same for every region and do not require any extra processing. However some attributes, such as the demand of a node, or the share of an edge, may vary from region-to-region, or depend on external (CSV) sources.

These values are assigned by writing a query which outputs the desired value.

Queries are prefixed with a ~ instead of the usual -, and are followed by the query to be executed:

~ demand =
    EB(residential, gas) +
      EB(residential, electricity) +
      EB(residential, infinite_improbability_drive)

The values output by queries are used during for Refinery calculations. You may set the following attributes using a query:

• Node: demand.
• Edge: parent_share, child_share, or demand.
• Slot: share ("conversion").

Set a slot share by adding an appropriate query in the node document:

~ output.coal = CONVERSION(my_node_outputs, coal)

EB(use, carrier)

The EB() function returns the value of a cell from the energy balance data. This data is stored in a CSV file in "data/energy_balance". Supply the use (the name of a row in the CSV) with a carrier (a column name):

EB(industry, electricity) # => 140673.72

AREA(attribute)

Retrieve a single attribute value from the area data with the AREA() function. The area data is stored in "data/datasets/:area/:area.ad".

AREA(coast_line) # => 451.0

SHARE(file_name, attribute)

Files containing share data (application and technology splits) are found at "data/datasets/:area/shares". Provide the name of the share file, plus the column name, and you'll get the share value:

SHARE(cng, cars) # => 0.5

CSV files containing shares can be given whatever name you want, but it should be consistent across every region, i.e. if you add a "cng.csv" file to "datasets/nl/shares", then a similar file should be added for the other regions also.

CENTRAL_PRODUCTION(node_key)

Like CHP data, the energy output of central production nodes is specified in a separate CSV file. This is at "data/datasets/:area/central_producers.csv". Provide the node key to retrieve the demand:

CENTRAL_PRODUCTION(energy_power_solar_csp_solar_radiation) # => 5678.0

PRIMARY_PRODUCTION(node_key, attribute)

The same as CENTRAL_PRODUCTION except that it reads from the CSV file at "data/dataset/:area/primary_production.csv". It also takes an attribute parameter to identify which column you wish to read.

Before Committing...

After editing documents by hand, you should run the Atlas validation to ensure that you did not introduce any illegal changes:

$ cd ~/code/atlas
$ rake validate[../etsource/data]
OK

Special Attributes

Node

You can set a slot share ("conversion") for a node like so:

- input.gas = 0.4
- input.oil = 0.6

This tells the node that is has two input slots; gas providing 40%, and oil providing 60% of the demanded energy. Swapping "input" for "output" does the same for output slots.

A special "elastic" value will tell the node that this slot should fill up whatever share is required to sum to 1.0:

- output.heat = 0.2
- output.electricity = 0.5
- output.loss = elastic

This node outputs 20% of its energy has heat, and 50% as electricity. The elastic slot will therefore take the remaining 30%. A node may not contain more than one elastic input slot, and one elastic output slot.

Slot share may vary depending on the share of the inputs; this is called "carrier efficiency", and is often used to model converters whose efficiency varies depending on the energy source given to them. In this case, you need to provide the efficiency of the slot if the node were to be given 100% of each input.

- output.electricity.coal = 0.4
- output.electricity.biomass = 0.5

This tells ETSource/Atlas that the "electricity" output slot has a share of 0.4 when only coal is given to the node, and a share of 0.5 when only biomass is given. In reality, your node will likely have a split of inputs; you must provide a share for each input:

- input.coal = 0.7
- input.biomass = 0.3
- output.electricity.coal = 0.4
- output.electricity.biomass = 0.5
- output.loss = elastic

CSV Documents

Throughout the ETSource repo are ".csv" files which contain raw data used by Atlas for creating the processed files for ETEngine. Many are sourced from third-parties (such as energy-balance data from the IEA), while others are created by Quintel staff.

Directory Structure

The ETSource repository is split into many directories, each containing files serving different purposes. Data is either global and affects all regions, or is regional and affects only one area.

./carriers (global)

Contains a document for each type of carrier in the Energy Transition Model. In each file is global data specifying how the carrier works, such as whether the energy is infinite in supply.

./datasets (regional)

Inside the datasets directory is a subdirectory for each region. Inside each of those folders is regional data: an ActiveDocument whose name matches the folder. Each also has a "shares" subdirectory containing CSVs whose values are used by Atlas to set edge shares.

./edges (global)

ActiveDocuments; one for each edge ("link") between nodes in the graph. Each filename contains the supplier ("parent" or "output") node, and the consumer ("child" or "input") node, and the name of the carrier in the format: consumer-supplier@carrier.

Edges are typically organised into subdirectories whose names match the sector of the supplier node. This may change in the future, and presently nothing will break if you put an edge in the wrong subdirectory.

./gqueries (global)

ActiveDocuments detailing the queries which may be performed on ETEngine by application such as ETModel and ETFlex. The subdirectories have no significance except to keep things organised.

./inputs (global)

Documents containing values which may be set on ETEngine by applications like ETModel. One file per input, and the directories are purely for organisation and have no significance.

./nodes (global)

Contains documents for nodes. One document per node, organised into subdirectories whose names indicate the sector to which the node belongs.

./presets (global)

Contains scenarios completed by QI'ers and notable public figures which can be viewed on ETModel.