NIO Filesystem Provider for Google Cloud Storage

Implementation of Java 7 java.nio.file.FileSystem for Google Cloud Storage.

This library allows you to use the standardized Java file system API for interacting with Google Cloud Storage.

• API Documentation

Note: This client is a work-in-progress, and may occasionally make backwards-incompatible changes.

Quickstart

If you are using Maven with a BOM, add this to your pom.xml file.

<dependencyManagement>
 <dependencies>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>libraries-bom</artifactId>
    <version>3.4.0</version>
    <type>pom</type>
    <scope>import</scope>
   </dependency>
 </dependencies>
</dependencyManagement>

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-nio</artifactId>
</dependency>

If you are using Maven without a BOM, add this to your dependencies.

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-nio</artifactId>
  <version>0.120.0-alpha</version>
</dependency>

If you are using Gradle, add this to your dependencies

compile 'com.google.cloud:google-cloud-nio:0.120.0-alpha'

If you are using SBT, add this to your dependencies

libraryDependencies += "com.google.cloud" % "google-cloud-nio" % "0.120.0-alpha"

Example Applications

• Stat shows how to get started with NIO.

• ParallelCountBytes efficiently downloads a file from Google Cloud Storage.

• ListFileSystems retrofit illustrates how NIO can add Google Cloud Storage support to some legacy programs, without having to modify them.

Authentication

See the Authentication section in the base directory's README. This shows how to construct the StorageOptions object, which you can then pass to CloudStorageFileSystem.forBucket.

About Google Cloud Storage

Google Cloud Storage is a durable and highly available object storage service. Google Cloud Storage is almost infinitely scalable and guarantees consistency: when a write succeeds, the latest copy of the object will be returned to any GET, globally.

See the Google Cloud Storage docs for more details on how to activate Cloud Storage for your project.

About Java NIO Providers

Java NIO Providers is an extension mechanism that is part of Java and allows third parties to extend Java's normal File API to support additional filesystems.

Getting Started

Prerequisites

For this tutorial, you will need a Google Developers Console project with "Google Cloud Storage" and "Google Cloud Storage JSON API" enabled via the console's API Manager. You will need to enable billing to use Google Cloud Storage. Follow these instructions to get your project set up. You will also need to set up the local development environment by installing the Google Cloud SDK and running the following commands in command line: gcloud auth login and gcloud config set project [YOUR PROJECT ID].

Installation and setup

You'll need to obtain the google-cloud-nio library.

There are two ways to use this library.

The recommended way is to follow the Quickstart section to add google-cloud-nio as a dependency in your code.

The second way is more complicated, but it allows you to add Google Cloud Storage support to some legacy Java programs. This approach is described in the google-cloud-nio-examples README.

Accessing files

The simplest way to get started is with Paths and Files:

Path path = Paths.get(URI.create("gs://bucket/lolcat.csv"));
List<String> lines = Files.readAllLines(path, StandardCharsets.UTF_8);

If you know the paths will point to Google Cloud Storage, you can also use the direct formulation:

try (CloudStorageFileSystem fs = CloudStorageFileSystem.forBucket("bucket")) {
  Path path = fs.getPath("lolcat.csv");
  List<String> lines = Files.readAllLines(path, StandardCharsets.UTF_8);
}

Once you have a Path you can use it as you would a normal file. For example you can use InputStream and OutputStream for streaming:

try (InputStream input = Files.openInputStream(path)) {
  // ...
}

You can also set various attributes using CloudStorageOptions static helpers:

Files.write(csvPath, csvLines, StandardCharsets.UTF_8,
    withMimeType(MediaType.CSV_UTF8),
    withoutCaching());

Limitations

This library is usable, but not yet complete. The following features are not yet implemented:

• Resuming upload or download
• Generations
• File attributes
• (more - list is not exhaustive)

Some features are not on the roadmap: this library would be a poor choice to mirror a local filesystem onto the cloud because Google Cloud Storage has a different set of features from your local disk. This library, by design, does not mask those differences. Rather, it aims to expose the common subset via a familiar interface.

NOTE: Cloud Storage uses a flat namespace and therefore doesn't support real directories. So this library supports what's known as "pseudo-directories". Any path that includes a trailing slash, will be considered a directory. It will always be assumed to exist, without performing any I/O. Paths without the trailing slash will result in an I/O operation to check a file is present in that "directory". This allows you to do path manipulation in the same manner as you would with the normal UNIX file system implementation. You can disable this feature with CloudStorageConfiguration.usePseudoDirectories().

Complete source code

There are examples in google-cloud-examples for your perusal.

Java Versions

Java 7 or above is required for using this client.

Versioning

This library follows Semantic Versioning.

It is currently in major version zero (0.y.z), which means that anything may change at any time and the public API should not be considered stable.

Contributing

Contributions to this library are always welcome and highly encouraged.

See google-cloud's CONTRIBUTING documentation and the shared documentation for more information on how to get started.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Code of Conduct for more information.

License

Apache 2.0 - See LICENSE for more information.