Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cloud-storage-contrib-nio: Add support for newFileChannel #4718

Merged
merged 11 commits into from
Apr 12, 2019
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.nio.channels.FileChannel;
import java.nio.channels.SeekableByteChannel;
import java.nio.file.AccessMode;
import java.nio.file.AtomicMoveNotSupportedException;
Expand Down Expand Up @@ -300,6 +301,51 @@ public SeekableByteChannel newByteChannel(
}
}

/**
* Open a file for reading OR writing. The {@link FileChannel} that is returned will only allow
* reads or writes depending on the {@link OpenOption}s that are specified. If any of the
* following have been specified, the {@link FileChannel} will be write-only: {@link
* StandardOpenOption#CREATE}
*
* <ul>
* <li>{@link StandardOpenOption#CREATE}
* <li>{@link StandardOpenOption#CREATE_NEW}
* <li>{@link StandardOpenOption#WRITE}
* <li>{@link StandardOpenOption#TRUNCATE_EXISTING}
* </ul>
*
* In all other cases the {@link FileChannel} will be read-only.
*
* @param path The path to the file to open or create
* @param options The options specifying how the file should be opened, and whether the {@link
* FileChannel} should be read-only or write-only.
* @param attrs (not supported, the values will be ignored)
Copy link
Contributor

@JesseLovelace JesseLovelace Apr 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the null check for attrs below if this is the case? attrs is optional in File.createFile

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose to do so to keep it consistent with the other methods in CloudStorageFileSystemProvider that accept an array of FileAtributes as an input parameter. These do the exact same check, even though the input value is ignored.

(See newByteChannel(...)``createDirectory(...))

Furthermore, newFileChannel method can also be used to read an existing file, so it does not always call File.createFile.

* @throws IOException
*/
@Override
public FileChannel newFileChannel(
Path path, Set<? extends OpenOption> options, FileAttribute<?>... attrs) throws IOException {
checkNotNull(path);
initStorage();
CloudStorageUtil.checkNotNullArray(attrs);
if (options.contains(StandardOpenOption.CREATE_NEW)) {
Files.createFile(path, attrs);
} else if (options.contains(StandardOpenOption.CREATE) && !Files.exists(path)) {
Files.createFile(path, attrs);
}
if (options.contains(StandardOpenOption.WRITE)
|| options.contains(StandardOpenOption.CREATE)
|| options.contains(StandardOpenOption.CREATE_NEW)
|| options.contains(StandardOpenOption.TRUNCATE_EXISTING)) {
CloudStorageWriteChannel writeChannel =
(CloudStorageWriteChannel) newWriteChannel(path, options);
return new CloudStorageWriteFileChannel(writeChannel);
} else {
CloudStorageReadChannel readChannel = (CloudStorageReadChannel) newReadChannel(path, options);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need to cast to a CloudStorageReadChannel here? if not perhaps just do this: new CloudStorageReadFileChannel(newReadChannel(path, options));

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, changed.

return new CloudStorageReadFileChannel(readChannel);
}
}

private SeekableByteChannel newReadChannel(Path path, Set<? extends OpenOption> options)
throws IOException {
initStorage();
Expand Down Expand Up @@ -788,8 +834,8 @@ public <A extends BasicFileAttributes> A readAttributes(
@Override
public Map<String, Object> readAttributes(Path path, String attributes, LinkOption... options) {
// TODO(#811): Java 7 NIO defines at least eleven string attributes we'd want to support
// (eg. BasicFileAttributeView and PosixFileAttributeView), so rather than a partial
// implementation we rely on the other overload for now.
// (eg. BasicFileAttributeView and PosixFileAttributeView), so rather than a partial
// implementation we rely on the other overload for now.
throw new UnsupportedOperationException();
}

Expand Down Expand Up @@ -943,15 +989,18 @@ public String getProject() {
*
* <p>Example of listing buckets, specifying the page size and a name prefix.
*
* <pre>{@code
* String prefix = "bucket_";
* Page<Bucket> buckets = provider.listBuckets(BucketListOption.prefix(prefix));
* Iterator<Bucket> bucketIterator = buckets.iterateAll();
* while (bucketIterator.hasNext()) {
* Bucket bucket = bucketIterator.next();
* // do something with the bucket
* <pre>
* {
* &#64;code
* String prefix = "bucket_";
* Page<Bucket> buckets = provider.listBuckets(BucketListOption.prefix(prefix));
* Iterator<Bucket> bucketIterator = buckets.iterateAll();
* while (bucketIterator.hasNext()) {
* Bucket bucket = bucketIterator.next();
* // do something with the bucket
* }
* }
* }</pre>
* </pre>
*
* @throws StorageException upon failure
*/
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
/*
* Copyright 2016 Google LLC
olavloite marked this conversation as resolved.
Show resolved Hide resolved
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.google.cloud.storage.contrib.nio;

import com.google.common.base.Preconditions;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.channels.FileLock;
import java.nio.channels.ReadableByteChannel;
import java.nio.channels.SeekableByteChannel;
import java.nio.channels.WritableByteChannel;

class CloudStorageReadFileChannel extends FileChannel {
private static final String READ_ONLY = "This FileChannel is read-only";
private final SeekableByteChannel readChannel;

CloudStorageReadFileChannel(SeekableByteChannel readChannel) {
Preconditions.checkNotNull(readChannel);
this.readChannel = readChannel;
}

@Override
public int read(ByteBuffer dst) throws IOException {
return readChannel.read(dst);
}

@Override
public long read(ByteBuffer[] dsts, int offset, int length) throws IOException {
long res = 0L;
synchronized (this) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might as well jut put synchronized in the signature.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed.

for (int i = offset; i < offset + length; i++) {
res += readChannel.read(dsts[i]);
}
}
return res;
}

@Override
public int write(ByteBuffer src) throws IOException {
throw new UnsupportedOperationException(READ_ONLY);
}

@Override
public long write(ByteBuffer[] srcs, int offset, int length) throws IOException {
throw new UnsupportedOperationException(READ_ONLY);
}

@Override
public long position() throws IOException {
return readChannel.position();
}

@Override
public FileChannel position(long newPosition) throws IOException {
readChannel.position(newPosition);
return this;
}

@Override
public long size() throws IOException {
return readChannel.size();
}

@Override
public FileChannel truncate(long size) throws IOException {
throw new UnsupportedOperationException(READ_ONLY);
}

@Override
public void force(boolean metaData) throws IOException {
throw new UnsupportedOperationException();
}

@Override
public long transferTo(long position, long count, WritableByteChannel target) throws IOException {
long res = 0L;
synchronized (this) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

synchronized in the signature might be clearer.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed.

long originalPosition = position();
position(position);
int blockSize = (int) Math.min(count, 0xfffffL);
int bytesRead = 0;
ByteBuffer buffer = ByteBuffer.allocate(blockSize);
while (res < count && bytesRead >= 0) {
buffer.position(0);
bytesRead = read(buffer);
if (bytesRead > 0) {
buffer.position(0);
buffer.limit(bytesRead);
target.write(buffer);
res += bytesRead;
}
}
position(originalPosition);
}
return res;
}

@Override
public long transferFrom(ReadableByteChannel src, long position, long count) throws IOException {
throw new UnsupportedOperationException(READ_ONLY);
}

@Override
public int read(ByteBuffer dst, long position) throws IOException {
synchronized (this) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed.

long originalPosition = position();
position(position);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this redundant?

      long originalPosition = position();
      position(position);

Do we actually need position(position);?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we need to position the channel where the read should begin. I've changed the name of the parameter to make it clearer.

int res = readChannel.read(dst);
position(originalPosition);
return res;
}
}

@Override
public int write(ByteBuffer src, long position) throws IOException {
throw new UnsupportedOperationException(READ_ONLY);
}

@Override
public MappedByteBuffer map(MapMode mode, long position, long size) throws IOException {
throw new UnsupportedOperationException();
}

@Override
public FileLock lock(long position, long size, boolean shared) throws IOException {
throw new UnsupportedOperationException();
}

@Override
public FileLock tryLock(long position, long size, boolean shared) throws IOException {
throw new UnsupportedOperationException();
}

@Override
protected void implCloseChannel() throws IOException {
readChannel.close();
}
}
Loading