Execute computing tasks when the grid is powered by renewable energy
The goal of this project is to provide developers with hassle free, easy to use, ready to run tools for carbon aware computing. All libraries and data are open source and open data with unrestricted usage. Within private, open source and commercial software.
The basic idea behind the time-shifting approach is to move the computing load in to a point in time, when the power grid has a maximum of renewable energy. This will result in a lower emission of CO2 of your computing task.
This project will deliver a set of libraries, services and data. There are mostly extensions to other projects and all credits belong to them. To forecast the best execution time the Carbon Aware SDK from the Green Software Foundation is used. The Forecast and actual data is from Energy-Charts provided by Fraunhofer ISE. For UK the data is provided by UK National Grid ESO.
Use the NuGet-package for your .NET project to get the best execution time for a task, in a given Execution-Window for a estimated duration. The lib will calculate the optimal execution time within the provided forecast data.
Use the NuGet-package for your .NET project to get the current Grid Carbon Intensity of a given region.
Just add the package to your project.
Install-Package CarbonAwareComputing Instantiate a CarbonAwareDataProvider and call CalculateBestExecutionTime
// Hold the provider as singleton. The forecast data is cached
var provider = new CarbonAwareDataProviderOpenData();
var forecast = provider.CalculateBestExecutionTime(
ComputingLocations.Germany,
DateTimeOffset.Now,
DateTimeOffset.Now + TimeSpan.FromHours(8),
TimeSpan.FromMinutes(20)
);
var executionTime = forecast.Match(
noForecast =>
{
Console.WriteLine("No forecast available. Use fallback");
return DateTimeOffset.Now;
},
bestExecutionTime =>
{
Console.WriteLine($"Forecast available for a task of {bestExecutionTime.Duration} length");
return bestExecutionTime.ExecutionTime;
});In the above example we will get a optimal execution time for the German Power Grid from now to 8 hours for a task with an estimated duration of 20 minutes. Please have in mind that this a best effort approach. Based on the data or your boundaries a forecast is not available.
// Hold the provider as singleton. The forecast data is cached
var provider = new CarbonAwareDataProviderOpenData();
var intensity = provider.GetCarbonIntensity(
ComputingLocations.Germany,
DateTimeOffset.Now);
intensity.Match(
emissionData =>
{
Console.WriteLine($"Current grid carbon intensity: {emissionData.Value}");
},
_ =>
{
Console.WriteLine($"Data not available);
});In the above example we will get the grid carbon intensity for the German Power Grid for the actual time. Please have in mind that the time range for the intensity is limited. It's mostly for today.
The CarbonAwareDataProviderOpenData has a cache of all forecasts. To improve performance use it as a singleton, to avoid multiple downloads. For a list of locations see below.
Hangfire is one of the most used tools for background processing in .NET. Use the Hangfire.CarbonAwareExecution Extension to enqueue and schedule your jobs.
Hangfire.Community.CarbonAwareExecution is available as a NuGet package. You can install it using the NuGet-Package Console window:
Install-Package Hangfire.Community.CarbonAwareExecutionAfter installation add the extension to the Hangfire configuration.
builder.Services.AddHangfireCarbonAwareExecution(configuration => configuration.
UseCarbonAwareDataProvider(new CarbonAwareDataProviderOpenData(), ComputingLocations.Germany));There are extension to Enqueue and Schedule with WithCarbonAwarenessAsync. For more details check the GitHub Repository.
We provide a live and ready to use subset of the Carbon Aware SDK. The API is available from this location: https://forecast.carbon-aware-computing.com/. Use the Swagger UI https://forecast.carbon-aware-computing.com/swagger/UI to play around with the API.
We also provide an endpoint to get the actual grid carbon intensity. The API is available from this location: https://intensity.carbon-aware-computing.com/. Use the Swagger UI https://intensity.carbon-aware-computing.com/swagger/UI to play around with the API.
To use the API, you must register to the service submitting a valid eMail-Address. Please check the register endpoint in the Swagger UI. The API-Key is send to this email. We will use this address only to inform you about important changes to the service. The registration is suitable for both API (Execution Forecast & Grid Carbon Intensity)
curl -X POST "https://forecast.carbon-aware-computing.com/register" -H "accept: */*" -H "Content-Type: application/json" -d "{\"mailAddress\":\"someone@example.com\"}"We want to support the time-shifting functionality of the SDK and provide only the forecast endpoint for given locations. There are no historically data and the forecast data has only the optimalDataPoints collection set. The emissionsDataPoints with all forecast data is not set due to data efficiency. If you need the forecast data download it directly.
We have fork the Carbon Aware SDK https://github.com/bluehands/carbon-aware-sdk and provide the SDK as a NuGet-Package. The fork has also some modifications for cached data provider. You may use this package for your extensions.
The unofficial Carbon Aware SDK is available form nuget.org. Install it using the Package Manager Console window:
Install-Package GSF.CarbonAware.UnofficialThere is a PowerShell Cmdlets to forecast the best execution time. You may use this Cmdlets in automation scripts to execute tasks with carbon awareness. See the GitHub Repository for more details.
Install-Module -Name CarbonAwareComputing$now=get-date
Get-CarbonAwareExecutionTime -Location at -EarliestExecutionTime $now -LatestExecutionTime ($now).AddHours(10) -EstimatedExecutionDuration "00:10:00"Set the FallbackExecutionTime Parameter to set the execution time when no forecast is available. This command is designed to be used in scripts, therefor no errors are thrown.
There is a Kubernetes Prometheus metrics exporter to report the actual grid carbon intensity. Use this exporter to calculate the carbon emission based on electricity power. This exporter may be combined with data from kepler or Scaphandre.
See the GitHub repository for usage and installation.
To support the Carbon Aware KEDA Operator a compatible grid carbon exporter is provided. The data is exported in to a k8s configmap to adjust the number of scaled nodes. See the GitHub repository for usage and installation.
The forecast data for Europe (without UK) is gathered from Energy Charts provided by Frauenhofer ISE. It is licensed as CC 0 https://creativecommons.org/publicdomain/zero/1.0/. You may use it for any purpose without any credits.
The forecast data for United Kingdom is gathered from UK National Grid ESO. It is licensed as CC BY 4.0 DEED. See terms of usage.
The forecast data is available as json formatted files for every location. The files are directly consumable by the Carbon Aware SDK. Download is publicly available from a Azure Blob Storage.
curl -X GET "https://carbonawarecomputing.blob.core.windows.net/forecasts/{LOCATION}.json" -H "accept: application/json"Replace the {LOCATION} with one of the supported locations two letter country code (e.g. "de").
We support the most countries in Europe, but not all are active. For computing efficiency we start with a Germany, France, Austria, Switzerland and the United Kingdom. If you have a need for some other countries please contact us. We will activate that country. To get a list of all locations see the locations endpoint of the API. Every location has a IsActive-Flag.
curl -X GET "https://forecast.carbon-aware-computing.com/locations" -H "accept: application/json"The forecast data for Europe (without UK) is based on reported energy production (current) and forecast production for Wind (on-shore & off-shore) and Solar. This information's are send to the ENTSO-E Transparency Platform by the power grid Transmission System Operators (TSO). For the additional renewable energy sources like running water, bio mass the forecast is calculated as an interpolation of the last hours. After that the carbon intensity is calculated by the emission factor for every energy source. This forecast is very accurate because it is used by the TSO to manage the power grid. The data is recalculated every hour by Energy Charts. The forecast for next day is available at 19:00+01.
For regions we not support (e.g. US, Asia), a WattTime-DataProvider is implemented. For that you must provide your WattTime Credentials.
Every contribution is warmly welcome. You may contribute to forecast data for other regions than Europe or help to integrate time-shifting in popular processing systems and libraries. Migration to other programming languages and runtime systems is very efficacious as well.
Please drop a message to
Aydin Mir Mohammadi
am@bluehands.de