common: Predefine configuration objects for Compute Engine, App Engine and service account users

[rakyll]

We should provide a config module to allow developers to explicitly set the connection options.

var gcloud = require('gcloud'),
      datastore = gcloud.datastore,
      config = gcloud.config;

// on app engine, it should also discover if running locally
// if process.env.SERVER_SOFTWARE === 'Development'
datastore.dataset(config.appengine());

// compute engine
datastore.dataset(config.compute());

// service account, credentials file is the new developer console file
// that contains private key + service account email.
datastore.dataset(config(credentialsFilename));

[rakyll]

On App Engine and Compute Engine, a request is required to fetch the project/app name. It will make the existing codebase complicated to maintain if we do this lazily before request time.

Two options:

• emit a 'ready' event:

   var ds = datastore.dataset(config.appengine(context));
   ds.on('ready', function() { /* do stuff here */ });

• make config creation async:

  config.appengine(function(conf) {
    var ds = datastore.dataset(conf);
  });

I'd prefer we follow the first option for all. Any ideas?

[silvolu]

+1 for first option

[rakyll]

First looks good, but not useful. You basically want to get done with configuration for once and reuse it for multiple services.

config.compute(function() {
    // start server and don't care about config anymore
    gcloud.datastore.dataset.get(...)
});

This approach is not ideal if you want to authorise for multiple accounts/projects within a single process. We need to avoid global vars and need to pass some configuration object to service constructors which is likely to confuse a user.

config.compute(function(err, conf) {
    gcloud.datastore.dataset(conf).get(...);
});

IMHO, requiring multiple account support is not edgy. For cases where user is migrating data from one project to another, or for running backup jobs, it's a necessity.

[jgeewax]

Just so I can get my head around this, can we update to show a "before and after" type view?

It seems that something like config.compute(...) would lead to different code depending on the environment (correct me if I'm misunderstanding...). I'd like to avoid that if possible.

Using an example, let's say I wanted to get something from GCD. The code I'd like to write is something along the lines of...

var gcloud = require('gcloud'),
      datastore = gcloud.datastore;

gcloud.authenticate();

entity = datastore.get(...);

That authentication line would ideally do a few things:

1. Since we know this is running server-side, we should focus on service accounts (I don't think 3-party auth will be very common).
2. If the user is explicit, try what they intended, raise an exception otherwise:
   1. Specific configuration passed in as code (ie, gcloud.authenticate(<something here>))
   2. Environment variables overridden (ie, DATASTORE_AUTH_JSON or something)
3. If the user is not explicit, look at the environment, and try our best to authenticate. Raise an exception only after none of the options work.
   1. Try GCE with the appropriate scopes (userinfo and datastore in this case)
   2. Try GAE production
   3. Try GAE local dev_appserver
   4. Raise an exception with the log of all things not working.
4. Regardless of whether the user is explicit or implicit, I'd like to have introspection methods to see how exactly we're authenticating. That is:
   1. gcloud.get_authentication_method()
   2. Logging when we first authenticate of what we tried, what failed, and why.

[rakyll]

It seems that something like config.compute(...) would lead to different code depending on the environment (correct me if I'm misunderstanding...). I'd like to avoid that if possible.

True, but only for the configuration step. It's not ideal but falling back to other options (GCE and GAE) confuses people who has never heard of GCE's metadata server. The final fallback option returns the final error message (DNS error for http://metadata). Therefore, I thought user should be explicit about the environment if they are a GCE or GAE user.

Auth layer should also resolve the project ID if it's running on GCE, that's why we can't have gcloud.authenticate() synchronous. It has to look like:

gcloud.authenticate(opt_explicitConf, function(err) {
  // start writing code here
  entity = datastore.get(...);
});

(We can't lazily retrieve project ID during request time. If we do, codebase becomes unnecessarily complicated.)

The flow should be very similar to what you've said above. I'm adding a few other details to the 3rd step.

GAE local server is detectable by an env flag. We will be able to determine if it's local GAE.

i. If GAE local flag is set, use local server and don't authenticate.
ii. Else, try GCE with scopes (GAE prod = GCE).
iii. Return error if nothing has worked.

[stephenplusplus]

I've read through this a few times, and I'm still trying to understand the wires between our code and the various engines, but I thought I'd make a suggestion that hopefully isn't entirely useless, using @jgeewax's code example as a basis.

While you're still considering APIs to go with, I just wanted to mention the Promise system can sometimes ease the pain of having "this before this before this" conditions.

The object passed to gcloud.authenticate can include or omit various properties that are relevant for the intended environment, which would define an implicit connection type that authenticate can use however it needs. The chained success and error functions are only executed after gcloud.authenticate has triggered them, based on the results of the operations, handing them whatever bits of data it wants.

Here would be an example of creating a server that listens for a ping to api/users/:id, which then responds with the json object dump from the datastore.

./index.js

var gcloud = require('gcloud');
var createServer = require('./server');

gcloud
  .authenticate({ /* ... */ })
  .then(createServer, function handleError() { /* ... */ });

./server.js

var http = require('http');
var dataset;

module.exports = function (gcloud) {
  dataset = new gcloud.datastore.dataset({ projectId: 'MyApp' });
  createServer();
};

function createServer() {
  http.createServer(function (req, res) {
    if (req.url.indexOf('/api/user') === 0) {
      getUser(req.params.id, res);
      return;
    }
  });
}

function getUser(id, res) {
  dataset.get(['Users', id], function (err, res) {
    res.end(JSON.stringify(res.data));
  });
}

This is obviously a made-up scenario, but if we can integrate promises in a similar way, the consumer gets the benefit of DI -- so their server.js (or whatever makes sense for their use) module doesn't have to care about striking a connection, and can just proceed knowing it has everything it needs. Just a nice perk 👍

Hopefully this applies at least in some small way. If you see that I'm not grasping something, please let me know!

[jgeewax]

I am hugely in favor of something like this...

gcloud.authenticate({...}).then(function() {
  ...
})

would be really nice.

[stephenplusplus]

Updated to use then 👍

[rakyll]

We don't support promises, I see no reason why we should support promises during auth.

gcloud.authenticate({ /* ... */ }, createServer);

is much slicker and not giving a bad impression that gcloud has promises support.

[stephenplusplus]

gcloud.authenticate({ /* ... */ }, createServer); works just as well for me.

[rakyll]

I've slept over this issue and decided to close it (for a while). We should always require users to set a project ID regardless of which context they run gcloud (at least, in the scope of M1 release). Use a JSON key file for service account authorization, otherwise we'll fallback to metadata server to retrieve an access token if no key file is set.

There is no significant value added if we resolve their project IDs, yet users will need to handle gcloud.authenticate.

Currently, this is how we do auth:

// works everywhere
var ds = gcloud.datastore.dataset({
  projectId: 'bamboo-45-lol',
  keyFilename: '/path/to/keyfile.json'
});
ds.get(...)

// works on GCE
var ds = gcloud.datastore.dataset({
  projectId: 'bamboo-45-lol'
});
ds.get(...)

// works on GAE
var ds = glcoud.datastore.dataset({
  projectId: <appEngineAppName>
});
ds.get(...)

// i may add GAE local server support
// if process.env.SERVER_SOFTWARE === 'Development'

I want to focus on GAE's Node runtime this week to see if we have any problems. Maybe we should include gcloud in their runtime module and doesn't require GAE specific bits in this one.

// cc @rrch

[rakyll]

App Engine exposes project ID as GAE_LONG_APP_ID env variable. We can go from there, we don't need to ask metadata server.