-
Notifications
You must be signed in to change notification settings - Fork 275
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Azure batch deletes #1365
Azure batch deletes #1365
Conversation
ambry-cloud/src/main/java/com.github.ambry.cloud/azure/AzureBlobDataAccessor.java
Outdated
Show resolved
Hide resolved
ambry-cloud/src/main/java/com.github.ambry.cloud/azure/AzureBlobDataAccessor.java
Outdated
Show resolved
Hide resolved
ambry-cloud/src/main/java/com.github.ambry.cloud/azure/AzureBlobDataAccessor.java
Outdated
Show resolved
Hide resolved
ambry-cloud/src/main/java/com.github.ambry.cloud/azure/AzureBlobDataAccessor.java
Outdated
Show resolved
Hide resolved
try { | ||
statusCode = response.getStatusCode(); | ||
} catch (BlobStorageException bex) { | ||
statusCode = bex.getStatusCode(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can the statuscode returned by exception be one of OK, ACCEPTED, NOT_FOUND or GONE? If not, then maybe we can move the try catch block to include switch as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They throw exception if the http status is not the "expected" value, which in this case is Accepted.
Codecov Report
@@ Coverage Diff @@
## master #1365 +/- ##
=========================================
Coverage ? 71.99%
Complexity ? 6759
=========================================
Files ? 486
Lines ? 38378
Branches ? 4880
=========================================
Hits ? 27629
Misses ? 9415
Partials ? 1334
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM after comments are addressed
ambry-cloud/src/main/java/com.github.ambry.cloud/azure/AzureBlobDataAccessor.java
Outdated
Show resolved
Hide resolved
ambry-cloud/src/main/java/com.github.ambry.cloud/azure/AzureBlobDataAccessor.java
Show resolved
Hide resolved
ambry-cloud/src/main/java/com.github.ambry.cloud/azure/AzureBlobDataAccessor.java
Show resolved
Hide resolved
ambry-cloud/src/main/java/com.github.ambry.cloud/azure/AzureCloudDestination.java
Show resolved
Hide resolved
ambry-cloud/src/main/java/com.github.ambry.cloud/azure/AzureBlobDataAccessor.java
Outdated
Show resolved
Hide resolved
throw new IllegalArgumentException("Invalid batchSize: " + batchSize); | ||
} | ||
List<List<T>> partitionedLists = new ArrayList<>(); | ||
for (int j = 0; j < inputList.size() / batchSize + 1; j++) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could this loop be simplified to:
for (int start = 0; start < inputList.size(); start += batchSize) {
int end = Math.min(start + batchSize, inputList.size());
partitionedLists.add(inputList.subList(start, end));
}
I may be missing an edge case here though
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I was thinking about this partition thing and it seems that we don't really need a full list and just need an iterable that we can do a for loop on. Something like this could work, I think:
public static <T> Iterable<List<T>> partition(List<T> list, int batchSize) {
return () -> new Iterator<List<T>>() {
int nextStart = 0;
@Override
public boolean hasNext() {
return nextStart < list.size();
}
@Override
public List<T> next() {
if (!hasNext()) {
throw new NoSuchElementException();
}
int start = nextStart;
nextStart += batchSize;
return list.subList(start, Math.min(start + batchSize, list.size()));
}
};
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure this is a big improvement. I prefer to mimic the guava utility. Once we upgrade to a newer JDK, we can probably retire the utility and use the Collections one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I made the simplification you suggested.
No description provided.