Feature/billing

[mlshapiro]

Enable budgeting for podpac resources through the aws module.

Budgets can be made for function + associated resources or for all podpac created resources listed under an AWS account. Budget tracking/filtering is done through special tags created at resource generation:

{_podpac_resource: true}
{_podpac_resource_hash: hash}

I anticipate corner cases for resources that were not created at the same time. This will need to get thoroughly tested before we make any promises about budgeting capabilities. For now this more of potential notification than a guaranteed notification.

To use the feature, create the settings:

settings["AWS_BUDGET_AMOUNT"] = 10.0 # budget for all podpac AWS resources, in USD
settings["AWS_BUDGET_EMAIL"] = "user@email.com" # notification e-mail for budget alarms

Also documented here: https://github.com/creare-com/podpac-examples/blob/develop/notebooks/developer/aws-lambda-tutorial.ipynb

Then use the aws.Lambda node as before.

See https://github.com/creare-com/podpac-examples/blob/develop/notebooks/developer/test/test-lambda.ipynb for an example of how to budget for all podpac resources created through the aws module.

[mlshapiro]

@mpu-creare I don't have a test case for this yet. I will try to find one tomorrow, but I think we should include in this release.

[coveralls]

Coverage remained the same at 91.654% when pulling 894dac4 on feature/billing into 2d0a5a9 on develop.

[mlshapiro]

Testing procedure:

• Create a set of function resources

from podpac.managers import aws
from podpac import settings

# set logging to DEBUG to see build process
import logging
logger = logging.getLogger("podpac")
logger.setLevel(logging.DEBUG)

settings["AWS_BUDGET_AMOUNT"] = 0.10 # budget for all podpac AWS resources, in USD
settings["AWS_BUDGET_EMAIL"] = "<email>" # notification e-mail for budget alarms
settings["S3_BUCKET_NAME"] = "podpac-test-budget-bucket"
settings["FUNCTION_NAME"] = "podpac-test-budget-bucket"
settings["FUNCTION_ROLE_NAME"] = "podpac-test-budget-bucket"

node = aws.Lambda()
node.build()

• Confirmed setup through AWS interface
• Confirmed that the sin_coords_node example in https://github.com/creare-com/podpac-examples/blob/develop/notebooks/developer/test/test-lambda.ipynb still works
• Uploaded a 500 MB zip file to S3 bucket
• Download the file and watch the budget increase in the AWS interface

[mlshapiro]

its hard to say how quickly AWS assesses charges and updates the billing, but I have test running and will see if it tracks via tag appropriately

it looks like it is compounded daily. Will check back tomorrow

[mpu-creare]

This seems pretty awesome. Does Python also report the current budget spending?

[mpu-creare]

This looks fine to me. I'm a little worried about all the attributes, the Lambda Node is getting rather bloated. At least its all in one place as opposed to having to search for it on AWS...

[mlshapiro]

it does, but its hidden on the node._budget internal variable. We could add a convenience method to ask.

[mpu-creare]

could you just make it part of the .describe method's output? That would be super useful.

[mlshapiro]

roger

• add current budget to .describe method

[mlshapiro]

See 57b2a52 for cost allocation describe method.

Merging