Negative labels , how to get loss ?

[ohjuntack]

Hi !

The OWL-ViT paper specifies that around 50 negative labels should be added for training. I want to find this process in the code.

[ohjuntack]

@dataclasses.dataclass
class AddRandomNegativeLabels(NamedPreprocessOp):
  """Adds randomly sampled labels as additional negative labels.

  This is similar to the Federated Loss proposed in
  https://arxiv.org/pdf/2103.07461.pdf, but samples negatives in proportion to
  their appearance in the dataset, rather than in proportion to the square root
  of their frequency (for simplicity).

  The op works by maintaining a queue of labels seen in the dataset. For each
  dataset example, a number of candidate labels are randomly drawn from the
  queue. Labels that do not appear as positives in the example are added to the
  negatives, up to total_num_negatives.

  To keep the queue full, all text labels of the example, and the candidate
  labels previously sampled from the queue, are enqueued back. After warmup,
  sampled labels from the queue will have the same distribution as in the
  dataset.

  If negative integer labels are present in the features, this op will remove
  them, because they are obsolete after adding randomly sampled negatives.

  Attributes:
    total_num_negatives: Random negatives will be added to the input features to
      bring the total number of negatives to total_num_negatives.
    queue_capacity: Maximal size of the label queue. On average, the queue size
      will be maintained at half of the maximum.
    queue: tf.queue.RandomShuffleQueue. Will be added automatically.
  """

  total_num_negatives: int = 50
  queue_capacity: int = 100_000
  queue: Optional[tf.queue.RandomShuffleQueue] = None

  def __post_init__(self):
    self.queue = tf.queue.RandomShuffleQueue(
        capacity=self.queue_capacity,
        min_after_dequeue=0,
        dtypes=[tf.string],
        shapes=[tf.TensorShape([])],
        shared_name='random_negatives_queue')
    # Initialize with empty strings:
    self.queue.enqueue_many(tf.constant([''] * self.queue_capacity))

  def apply(self, features: image_ops.Features) -> image_ops.Features:
    # Draw candidate negative labels:
    candidate_labels = self.queue.dequeue_many(self.total_num_negatives * 2)

    # Fill queue back up:
    labels_to_enqueue = tf.concat([
        features[modalities.INSTANCE_TEXT_LABELS],
        features[modalities.NEGATIVE_TEXT_LABELS],
        candidate_labels,
    ], axis=0)
    labels_to_enqueue = tf.boolean_mask(
        labels_to_enqueue, tf.not_equal(labels_to_enqueue, PADDING_QUERY),
        name='labels_to_enqueue')
    target_size = self.queue_capacity // 2
    needed_elements = tf.clip_by_value(
        target_size - self.queue.size(), 0, tf.size(labels_to_enqueue))
    enqueue_op = self.queue.enqueue_many(
        tf.slice(labels_to_enqueue, begin=[0], size=[needed_elements]))

    # Get negatives that are not in positives:
    with tf.control_dependencies([enqueue_op]):
      candidate_negatives = tf.sparse.to_dense(
          tf.sets.difference(
              candidate_labels[None, ...],
              features[modalities.INSTANCE_TEXT_LABELS][None, ...]))[0]

    # Set operations sort the labels alphabetically, so we shuffle again:
    candidate_negatives = tf.random.shuffle(candidate_negatives)

    # New negatives contain all the old negatives, plus randomly sampled ones,
    # up to total_num_negatives. In addition, we ensure that padding ('') is not
    # present by including it as first element before applying tf.unique and
    # then slicing it off:
    new_negatives = tf.concat([
        tf.constant([PADDING_QUERY]),
        features[modalities.NEGATIVE_TEXT_LABELS],
        candidate_negatives,
    ], axis=0)
    orig_num_negatives = tf.shape(features[modalities.NEGATIVE_TEXT_LABELS])[0]
    new_num_negatives = tf.maximum(self.total_num_negatives, orig_num_negatives)
    features[modalities.NEGATIVE_TEXT_LABELS] = tf.unique(
        new_negatives)[0][1:(new_num_negatives + 1)]

    # Negative integer labels are now obsolete:
    if modalities.NEGATIVE_LABELS in features:
      logging.info('Removing obsolete field %s from features.',
                   modalities.NEGATIVE_LABELS)
      features.pop(modalities.NEGATIVE_LABELS)

    return features

I've understood how to obtain negative labels using the provided code. However, I'm curious about the process of calculating the negative loss using the code.

thank you !