Skip to content

Commit

Permalink
Clean up LocalGcdHelper and docs (#821)
Browse files Browse the repository at this point in the history
* Cleans up local gcd helper and docs to be more consistent with other test helpers
* Renames LocalGcdHelper to LocalDatastoreHelper and RemoteGcsHelper to RemoteStorageHelper
* Updates the gcd.sh script version
  • Loading branch information
ajkannan committed Apr 8, 2016
1 parent 6987b00 commit 53a60d1
Show file tree
Hide file tree
Showing 16 changed files with 254 additions and 314 deletions.
59 changes: 34 additions & 25 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,55 +11,64 @@ This library provides tools to help write tests for code that uses the following

#### On your machine

You can test against a temporary local datastore by following these steps:
You can test against a temporary local Datastore by following these steps:

1. Start the local datastore emulator using `LocalGcdHelper`. This can be done in two ways:
- Run `LocalGcdHelper.java`'s `main` method with arguments `START` and (optionally) `--port=<port number>`. This will create a temporary folder on your computer and bind `localhost:<port number>` for communication with the local datastore. The port number is an optional argument. If no port number is specified, port 8080 will be used.
- Call `LocalGcdHelper.start(<project ID>, <port number>)` before running your tests. Save the `LocalGcdHelper` object returned so that you can stop the emulator later.
1. Start the local Datastore emulator before running your tests using `LocalDatastoreHelper`'s `create` and `start` methods. This will create a temporary folder on your computer and bind a port for communication with the local Datastore. There is an optional argument for `create`: consistency. The consistency setting controls the fraction of Datastore writes that are immediately visible in global queries.
```java
// Use the default consistency setting of 0.9
LocalDatastoreHelper helper = LocalDatastoreHelper.create();
// or explicitly set the consistency
helper = LocalDatastoreHelper.create(0.6);

helper.start(); // Starts the local Datastore emulator in a separate process
```

2. In your program, create and use a datastore whose host is set host to `localhost:<port number>`. For example,
2. Create and use a `Datastore` object with the options given by the `LocalDatastoreHelper` instance. For example:
```java
DatastoreOptions options = DatastoreOptions.builder()
.projectId(PROJECT_ID)
.host("http://localhost:8080")
.build();
Datastore localDatastore = options.service();
Datastore localDatastore = helper.options().service();
```

3. Run your tests.

4. Stop the local datastore emulator.
- If you ran `LocalGcdHelper.java`'s `main` function to start the emulator, run `LocalGcdHelper.java`'s `main` method with arguments `STOP` and (optionally) `--port=<port number>`. If the port is not supplied, the program will attempt to close the last port started.
- If you ran `LocalGcdHelper.start()` to start the emulator, call the `stop()` method on the `LocalGcdHelper` object returned by `LocalGcdHelper.start()`.
4. Stop the local datastore emulator by calling the `stop()` method, like so:
```java
helper.stop();
```

#### On a remote machine

You can test against a remote datastore emulator as well. To do this, set the `DatastoreOptions` project endpoint to the hostname of the remote machine, like the example below.
You can test against a remote Datastore emulator as well. To do this, set the `DatastoreOptions` project endpoint to the hostname of the remote machine, like the example below.

```java
DatastoreOptions options = DatastoreOptions.builder()
.projectId(PROJECT_ID)
.projectId("my-project-id") // must match project ID specified on remote machine
.host("http://<hostname of machine>:<port>")
.authCredentials(AuthCredentials.noAuth())
.build();
Datastore localDatastore = options.service();
```

Note that the remote datastore must be running before your tests are run.
We recommend that you start the emulator on the remote machine using the [Google Cloud SDK](https://cloud.google.com/sdk/gcloud/reference/beta/emulators/datastore/) from command line, as shown below:

```
gcloud beta emulators datastore start --host-port <hostname of machine>:<port>
```

### Testing code that uses Storage

Currently, there isn't an emulator for Google Cloud Storage, so an alternative is to create a test project. `RemoteGcsHelper` contains convenience methods to make setting up and cleaning up the test project easier. To use this class, follow the steps below:
Currently, there isn't an emulator for Google Cloud Storage, so an alternative is to create a test project. `RemoteStorageHelper` contains convenience methods to make setting up and cleaning up the test project easier. To use this class, follow the steps below:
1. Create a test Google Cloud project.
2. Download a JSON service account credentials file from the Google Developer's Console. See more about this on the [Google Cloud Platform Storage Authentication page][cloud-platform-storage-authentication].

3. Create a `RemoteGcsHelper` object using your project ID and JSON key.
Here is an example that uses the `RemoteGcsHelper` to create a bucket.
3. Create a `RemoteStorageHelper` object using your project ID and JSON key.
Here is an example that uses the `RemoteStorageHelper` to create a bucket.
```java
RemoteGcsHelper gcsHelper =
RemoteGcsHelper.create(PROJECT_ID, new FileInputStream("/path/to/my/JSON/key.json"));
Storage storage = gcsHelper.options().service();
String bucket = RemoteGcsHelper.generateBucketName();
RemoteStorageHelper helper =
RemoteStorageHelper.create(PROJECT_ID, new FileInputStream("/path/to/my/JSON/key.json"));
Storage storage = helper.options().service();
String bucket = RemoteStorageHelper.generateBucketName();
storage.create(BucketInfo.of(bucket));
```

Expand All @@ -68,7 +77,7 @@ Here is an example that uses the `RemoteGcsHelper` to create a bucket.
5. Clean up the test project by using `forceDelete` to clear any buckets used.
Here is an example that clears the bucket created in Step 3 with a timeout of 5 seconds.
```java
RemoteGcsHelper.forceDelete(storage, bucket, 5, TimeUnit.SECONDS);
RemoteStorageHelper.forceDelete(storage, bucket, 5, TimeUnit.SECONDS);
```

### Testing code that uses Resource Manager
Expand Down Expand Up @@ -134,4 +143,4 @@ Here is an example that clears the dataset created in Step 3.
```

[cloud-platform-storage-authentication]:https://cloud.google.com/storage/docs/authentication?hl=en#service_accounts
[create-service-account]:https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount
[create-service-account]:https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@
import com.google.gcloud.storage.BlobInfo;
import com.google.gcloud.storage.BucketInfo;
import com.google.gcloud.storage.Storage;
import com.google.gcloud.storage.testing.RemoteGcsHelper;
import com.google.gcloud.storage.testing.RemoteStorageHelper;

import org.junit.AfterClass;
import org.junit.BeforeClass;
Expand Down Expand Up @@ -135,7 +135,7 @@ public class ITBigQueryTest {
private static final String LOAD_FILE = "load.csv";
private static final String JSON_LOAD_FILE = "load.json";
private static final String EXTRACT_FILE = "extract.csv";
private static final String BUCKET = RemoteGcsHelper.generateBucketName();
private static final String BUCKET = RemoteStorageHelper.generateBucketName();
private static final TableId TABLE_ID = TableId.of(DATASET, "testing_table");
private static final String CSV_CONTENT = "StringValue1\nStringValue2\n";
private static final String JSON_CONTENT = "{"
Expand Down Expand Up @@ -172,9 +172,9 @@ public class ITBigQueryTest {
@BeforeClass
public static void beforeClass() throws InterruptedException {
RemoteBigQueryHelper bigqueryHelper = RemoteBigQueryHelper.create();
RemoteGcsHelper gcsHelper = RemoteGcsHelper.create();
RemoteStorageHelper storageHelper = RemoteStorageHelper.create();
bigquery = bigqueryHelper.options().service();
storage = gcsHelper.options().service();
storage = storageHelper.options().service();
storage.create(BucketInfo.of(BUCKET));
storage.create(BlobInfo.builder(BUCKET, LOAD_FILE).contentType("text/plain").build(),
CSV_CONTENT.getBytes(StandardCharsets.UTF_8));
Expand All @@ -200,7 +200,7 @@ public static void afterClass() throws ExecutionException, InterruptedException
RemoteBigQueryHelper.forceDelete(bigquery, DATASET);
}
if (storage != null) {
boolean wasDeleted = RemoteGcsHelper.forceDelete(storage, BUCKET, 10, TimeUnit.SECONDS);
boolean wasDeleted = RemoteStorageHelper.forceDelete(storage, BUCKET, 10, TimeUnit.SECONDS);
if (!wasDeleted && LOG.isLoggable(Level.WARNING)) {
LOG.log(Level.WARNING, "Deletion of bucket {0} timed out, bucket is not empty", BUCKET);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -339,7 +339,7 @@ public RestorableState<AuthCredentials> capture() {
/**
* A placeholder for credentials to signify that requests sent to the server should not be
* authenticated. This is typically useful when using the local service emulators, such as
* {@code LocalGcdHelper} and {@code LocalResourceManagerHelper}.
* {@code LocalDatastoreHelper} and {@code LocalResourceManagerHelper}.
*/
public static class NoAuthCredentials extends AuthCredentials {

Expand Down Expand Up @@ -425,7 +425,7 @@ public static ServiceAccountAuthCredentials createFor(String account, PrivateKey

/**
* Creates a placeholder denoting that no credentials should be used. This is typically useful
* when using the local service emulators, such as {@code LocalGcdHelper} and
* when using the local service emulators, such as {@code LocalDatastoreHelper} and
* {@code LocalResourceManagerHelper}.
*/
public static AuthCredentials noAuth() {
Expand Down
Loading

0 comments on commit 53a60d1

Please sign in to comment.