Prevent building duplicate immediate uploaders

[PtrTeixeira]

There appears to be a threading issue where if the file system poller
happens to run at the same time as an immediate uploaders is running, a
second immediate uploader can be created for the same metadata.
Typically this would be prevented by the metadata-to-uploader map, but
immediate uploaders are removed from or never added to the map, so it
was never checked if there was an existing uploader for a given metadata
file.

This polls the list of immediate uploaders to check their metadata, and
will refuse to create another uploader of there is an existing uploader
with the same metadata file.

[ssalinas]

Overall, this makes the window smaller, but would still allow duplicates in some cases (i.e. if it runs after the check, but before the uploader is added to the map. I think this is on the right track, but we need to see if there is a solution that covers all cases

[ssalinas]

This again makes our window of time for running a duplicate uploader smaller, but doesn't completely eliminate it. I think we can accomplish what we need with a simple locking mechanism and a List of the s3 metadata.

We don't ever actually access the value of the immediate uploader map, so all we really need to store is a list of the metadata that we already have an immediate uploader for.

So, let's try something like a ReentrantLock to avoid multiple things updating our list at once.

We can create our list and the single lock when the uploader driver is started like

private final List<S3UploadMetadata> immediateUploaders = new ArrayList();
private final ReentrantLock lock = new ReentrantLock();

and then have some method to try and add an immediate uploader like:

public boolean addImmediateUploader(S3UploadMetadata metadata) {
  try {
    if (lock.tryLock({some timeout}, {Some timeout TimeUnit}) {
      // If we are here, we successfully acquired the lock, no other thread can be here at the same time
      if (! immediateUploaders.contains(metadata)) {
        // we don't have one yet
        immediateUploaders.add(metadata);
        return true;
      } else {
        // we already have one
        return false;
      }
    } else {
      // couldn't get the lock in time (something probably wrong if we are getting here)
    }
  } finally {
    lock.unlock() // make sure we give it back
  }
}

That way, right at the start of things we can have an if (addImmediateUploader(metadata)) -> do the other stuff

java 8 has some nicer things for working with situations like this, but if I remember correctly they are mostly based around maps, not lists. So, this might be slightly more verbose, but will give a guarantee we aren't doing anything in duplicate.

[ssalinas]

Let's make these statements slightly more useful by adding the actual metadata to them. Like:

LOG.debug("Already have an immediate uploader for {}", metadata);

[ssalinas]

just the small comment on logging, other than that LGTM

[ssalinas]

👍 LGTM