Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split off the "normal case" example from the "legacy case" one. #839

Merged
6 changes: 6 additions & 0 deletions gcloud-java-contrib/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ Contents
--------

* [gcloud-java-nio](./gcloud-java-nio/): NIO Filesystem Provider for Google Cloud Storage.
* [gcloud-java-nio-examples](./gcloud-java-nio-examples/): How to add GCS NIO after the fact.

See also
--------

* [gcloud-java-examples](../gcloud-java-examples) for an example of how to use NIO normally.

Contributing
------------
Expand Down
41 changes: 41 additions & 0 deletions gcloud-java-contrib/gcloud-java-nio-examples/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
Example of adding the Google Cloud Storage NIO Provider to a legacy jar
=======================================================================

This project shows how to add GCS capabilities to a jar file for a Java 7 application
that uses Java NIO without the need to recompile.

Note that whenever possible, you instead want to recompile the app and use the normal
dependency mechanism to add a dependency to gcloud-java-nio. You can see examples of
this in the [gcloud-java-examples](../../gcloud-java-examples) project.

To run this example:

1. Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled.

2. Log in using gcloud SDK (`gcloud auth login` in command line)

3. Compile the JAR with:
```
mvn package -DskipTests -Dmaven.javadoc.skip=true -Dmaven.source.skip=true
```

4. Run the sample with:

```
java -cp gcloud-java-contrib/gcloud-java-nio/target/gcloud-java-nio-0.1.6-SNAPSHOT-shaded.jar:gcloud-java-contrib/gcloud-java-nio-examples/target/gcloud-java-nio-examples-0.1.6-SNAPSHOT.jar com.google.gcloud.nio.examples.ListFilesystems
```

Notice that it lists gcs, which it wouldn't if you ran it without the nio jar:
```
java -cp gcloud-java-contrib/gcloud-java-nio-examples/target/gcloud-java-nio-examples-0.1.6-SNAPSHOT.jar com.google.gcloud.nio.examples.ListFilesystems
```

The sample doesn't have anything about GCS in it. It gets that ability from the nio jar that
we're adding to the classpath. You can use the nio "fat shaded" jar for this purpose as it also
includes the dependencies for gcloud-java-nio.
The underlying mechanism is Java's standard [ServiceLoader](https://docs.oracle.com/javase/7/docs/api/java/util/ServiceLoader.html)
facility, the [standard way](http://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/filesystemprovider.html) to plug in NIO providers like this one.

If you have access to a project's source code you can also simply add gcloud-java-nio as
a dependency and let Maven pull in the required dependencies (this is what the nio unit tests do).
This approach is preferable as the fat jar approach may waste memory on multiple copies of dependencies.
92 changes: 92 additions & 0 deletions gcloud-java-contrib/gcloud-java-nio-examples/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.google.gcloud</groupId>

This comment was marked as spam.

This comment was marked as spam.

<artifactId>gcloud-java-nio-examples</artifactId>
<packaging>jar</packaging>
<name>GCloud Java NIO example</name>
<description>
Demonstrates how to use the gcloud-java-nio jar to add GCS functionality to legacy code.
</description>
<parent>
<groupId>com.google.gcloud</groupId>
<artifactId>gcloud-java-contrib</artifactId>
<version>0.1.6-SNAPSHOT</version>
</parent>
<properties>
<site.installationModule>nio</site.installationModule>
</properties>
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>gcloud-java-storage</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>19.0</version>
</dependency>
<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
<version>2.0.1</version>
</dependency>
<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
<version>1</version>
</dependency>
<dependency>
<groupId>com.google.auto.service</groupId>
<artifactId>auto-service</artifactId>
<version>1.0-rc2</version>
<scope>provided</scope> <!-- to leave out of the all-deps jar -->
</dependency>
<dependency>
<groupId>com.google.auto.value</groupId>
<artifactId>auto-value</artifactId>
<version>1.1</version>
<scope>provided</scope> <!-- to leave out of the all-deps jar -->
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava-testlib</artifactId>
<version>19.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.truth</groupId>
<artifactId>truth</artifactId>
<version>0.27</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>1.9.5</version>
</dependency>
<dependency>
<groupId>org.apache.maven.plugins</groupId>

This comment was marked as spam.

This comment was marked as spam.

<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.4</version>
</dependency>
</dependencies>
<build>
<plugins>

This comment was marked as spam.

This comment was marked as spam.

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<configuration>
<skip>false</skip>
</configuration>
</plugin>
</plugins>
</build>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
package com.google.gcloud.nio.examples;

This comment was marked as spam.

This comment was marked as spam.


import java.io.IOException;
import java.net.URI;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.spi.FileSystemProvider;

/**
* ListFilesystems is a super-simple program that lists the available NIO filesystems.

*/
public class ListFilesystems {

/**
* See the class documentation.
*/
public static void main(String[] args) throws IOException {
listFilesystems();
}

private static void listFilesystems() {
System.out.println("Installed filesystem providers:");
for (FileSystemProvider p : FileSystemProvider.installedProviders()) {
System.out.println(" " + p.getScheme());
}
}

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

}
1 change: 1 addition & 0 deletions gcloud-java-contrib/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
</properties>
<modules>
<module>gcloud-java-nio</module>
<module>gcloud-java-nio-examples</module>
</modules>
<build>
<plugins>
Expand Down
59 changes: 27 additions & 32 deletions gcloud-java-examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ To run examples from your command line:

2. Set your current project using `gcloud config set project PROJECT_ID`. This step is not necessary for `ResourceManagerExample`.

3. Compile using Maven (`mvn compile` in command line from your base project directory)
3. Compile using Maven: `cd gcloud-java-examples` in command line from your base project directory and then `mvn package appassembler:assemble -DskipTests -Dmaven.javadoc.skip=true -Dmaven.source.skip=true`.

4. Run an example using Maven from command line.
4. Run an example from the command line using the Maven-generated scripts.

* Here's an example run of `BigQueryExample`.

Expand All @@ -55,62 +55,57 @@ To run examples from your command line:
```
Then you are ready to run the following example:
```
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="list tables new_dataset_id"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
target/appassembler/bin/BigQueryExample create dataset new_dataset_id
target/appassembler/bin/BigQueryExample create table new_dataset_id new_table_id field_name:string
target/appassembler/bin/BigQueryExample list tables new_dataset_id
target/appassembler/bin/BigQueryExample load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file
target/appassembler/bin/BigQueryExample query 'select * from new_dataset_id.new_table_id'
```

* Here's an example run of `DatastoreExample`.

Be sure to change the placeholder project ID "your-project-id" with your own project ID. Also note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands.
```
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" -Dexec.args="your-project-id my_name add my\ comment"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" -Dexec.args="your-project-id my_name display"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" -Dexec.args="your-project-id my_name delete"
target/appassembler/bin/DatastoreExample your-project-id my_name add my\ comment
target/appassembler/bin/DatastoreExample your-project-id my_name display
target/appassembler/bin/DatastoreExample your-project-id my_name delete
```

* Here's an example run of `ResourceManagerExample`.

Be sure to change the placeholder project ID "your-project-id" with your own globally unique project ID.
```
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" -Dexec.args="create your-project-id"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" -Dexec.args="list"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" -Dexec.args="get your-project-id"
target/appassembler/bin/ResourceManagerExample create your-project-id
target/appassembler/bin/ResourceManagerExample list
target/appassembler/bin/ResourceManagerExample get your-project-id
```

* Here's an example run of `StorageExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
```
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="list <bucket_name>"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="download <bucket_name> test.txt"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
target/appassembler/bin/StorageExample upload /path/to/test.txt <bucket_name>
target/appassembler/bin/StorageExample list <bucket_name>
target/appassembler/bin/StorageExample download <bucket_name> test.txt
target/appassembler/bin/StorageExample delete <bucket_name> test.txt
```
* Here's an example run of `Stat`, illustrating the use of the gcloud-java-nio jar.

Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket with a file in it.
Compile the JAR with:
```
mvn package -DskipTests -Dmaven.javadoc.skip=true -Dmaven.source.skip=true
```
Then run the sample with:
* Here's an example run of `Stat`, illustrating the use of gcloud-java-nio.

Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket with a file in it.

Run the sample with (from the gcloud-java-examples folder):
```
java -cp gcloud-java-contrib/gcloud-java-nio/target/gcloud-java-nio-0.1.6-SNAPSHOT-shaded.jar:gcloud-java-examples/target/gcloud-java-examples-0.1.6-SNAPSHOT.jar com.google.gcloud.examples.nio.Stat --check
target/appassembler/bin/Stat --check

```
Or, if you have a file in `gs://mybucket/myfile.txt`, you can run:
```
java -cp gcloud-java-contrib/gcloud-java-nio/target/gcloud-java-nio-0.1.6-SNAPSHOT-shaded.jar:gcloud-java-examples/target/gcloud-java-examples-0.1.6-SNAPSHOT.jar com.google.gcloud.examples.nio.Stat gs://mybucket/myfile.txt
target/appassembler/bin/Stat gs://mybucket/myfile.txt
```
The sample doesn't have anything about GCS in it. It gets that ability from the nio jar that
we're adding to the classpath. You can use the nio "fat shaded" jar for this purpose as it also
includes the dependencies for gcloud-java-nio.

If you have access to a project's source code you can also simply add gcloud-java-nio as
a dependency and let Maven pull in the required dependencies (this is what the nio unit tests do).
This approach is preferable as the fat jar approach may waste memory on multiple copies of dependencies.
The sample doesn't have anything special about GCS in it, it just opens files via the NIO API.
It lists gcloud-java-nio as a dependency, and that enables it to interpret `gs://` paths.

Troubleshooting
---------------
Expand Down
37 changes: 37 additions & 0 deletions gcloud-java-examples/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@
<artifactId>gcloud-java</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>gcloud-java-nio</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
Expand All @@ -37,6 +42,38 @@
<skip>false</skip>
</configuration>
</plugin>
<!-- make a script so we can run stuff directly by calling
mvn package appassembler:assemble -DskipTests -Dmaven.javadoc.skip=true -Dmaven.source.skip=true
from the gcloud-java-examples folder. The script is written to target/appassembler/bin/. -->

This comment was marked as spam.

This comment was marked as spam.

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>appassembler-maven-plugin</artifactId>
<configuration>
<programs>
<program>
<mainClass>com.google.gcloud.examples.bigquery.BigQueryExample</mainClass>
<name>BigQueryExample</name>
</program>
<program>
<mainClass>com.google.gcloud.examples.datastore.DatastoreExample</mainClass>
<name>DatastoreExample</name>
</program>
<program>
<mainClass>com.google.gcloud.examples.nio.Stat</mainClass>
<name>Stat</name>
</program>
<program>
<mainClass>com.google.gcloud.examples.resourcemanager.ResourceManagerExample</mainClass>
<name>ResourceManagerExample</name>
</program>
<program>
<mainClass>com.google.gcloud.examples.storage.StorageExample</mainClass>
<name>StorageExample</name>
</program>

</programs>
</configuration>
</plugin>
</plugins>
</build>
</project>
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@
* be logged in (using e.g. the gcloud auth command).
*
* <p>See the README for a command line to run this example.
*

This comment was marked as spam.

This comment was marked as spam.

* <p>In short, this version (in gcloud-java-examples) is in a package that lists gcloud-java-nio
* as a dependency, so it will work directly without having to do any special work.
*/
public class Stat {

Expand Down Expand Up @@ -77,8 +80,8 @@ private static void help() {
"",
"This tool normally knows nothing of Google Cloud Storage. If you pass it --check",
"or a GCS file name (e.g. gs://mybucket/myfile), it will show an error.",
"However, by just adding the gcloud-nio jar in your classpath, this tool is made",
"aware of gs:// paths and can access files on the cloud.",
"However, by just adding the gcloud-nio jar as a dependency and recompiling, this tool is",
"made aware of gs:// paths and can access files on the cloud.",
"",
"The gcloud NIO filesystem provider can similarly enable existing Java 7 programs",
"to read and write cloud files, even if they have no special built-in cloud support."
Expand All @@ -97,7 +100,7 @@ private static void listFilesystems() {

private static void checkGcs() {
FileSystem fs = FileSystems.getFileSystem(URI.create("gs://domain-registry-alpha"));
System.out.println("We seem to be able to instantiate a gs:// filesystem.");
System.out.println("Success! We can instantiate a gs:// filesystem.");
System.out.println("isOpen: " + fs.isOpen());
System.out.println("isReadOnly: " + fs.isReadOnly());
}
Expand Down
3 changes: 3 additions & 0 deletions utilities/update_docs_version.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ if [ "${RELEASED_VERSION##*-}" != "SNAPSHOT" ]; then
sed -ri "s/:[0-9]+\.[0-9]+\.[0-9]+'/:${RELEASED_VERSION}'/g" ${item}/README.md
sed -ri "s/\"[0-9]+\.[0-9]+\.[0-9]+\"/\"${RELEASED_VERSION}\"/g" ${item}/README.md
done

echo "Changing version to $((RELEASED_VERSION + 1))-SNAPSHOT in gcloud-java-nio-example README"

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

sed -ri "s/gcloud-java-nio-examples-[0-9]+\.[0-9]+\.[0-9]+-SNAPSHOT'/gcloud-java-nio-examples-$((RELEASED_VERSION + 1))-SNAPSHOT'/g" gcloud-java-contrib/gcloud-java-nio-examples/README.md

git add README.md */README.md
git config --global user.name "travis-ci"
Expand Down