HyperText: Models for learning Text Representations

This directory contains TensorFlow source code for learning embeddings of text sequences in an unsupervised manner. This is a preliminary implementation with more changes (and documentation) forthcoming.

Contact

Code authors: Bhuwan Dhingra, Chris Shallue

Pull requests and issues: @bdhingra, @cshallue

Getting Started

This code is backed by the open-source TensorFlow implementation of the Skip-Thought vectors model availabel here. Hence, the first step is to download that repository and copy the relevant skip_thoughts directory to this repository.

TENSORFLOW_MODELS_DIR="${HOME}/tensorflow_models/"

git clone https://github.com/tensorflow/models.git "$TENSORFLOW_MODELS_DIR"

cp -r "${TENSORFLOW_MODELS_DIR}/skip_thoughts/skip_thoughts/" .

Follow the instructions in ${TENSORFLOW_MODELS_DIR}/skip_thoughts/README.md and ensure that all the source files from that repository are working properly.

Distributed Skip Thoughts

The skip_thoughts_dist directory contains training and validation scripts which can be used to run the TensorFlow Skip Thoughts model in a distributed setting. These can be used in the following manner:

Training

#!/bin/bash

bazel build -c opt //skip_thoughts_dist:train

bazel-bin/skip_thoughts_dist/train \
  --input_file_pattern <tfrecord_files> \
  --nosync_replicas \
  --train_dir <path_to_save_models>

Validation

Run the validation script for tracking perplexity in a separate process. You may want to pass the CUDA_VISIBLE_DEVICES='' flag to avoid using the GPU for this script.

#!/bin/bash

bazel build -c opt //skip_thoughts_dist:track_perplexity

CUDA_VISIBLE_DEVICES='' bazel-bin/skip_thoughts_dist/track_perplexity \
  --input_file_pattern <tfrecord_files> \
  --checkpoint_dir <path_with_saved_models> \
  --eval_dir <directory_to_log_eval_summaries>

Word Context Models

Word Context models decode each word in a specified window around the source sentence separately as opposed to the full sequence decoder in the original SkipThoughts model. There are several different options on the particular design of the encoder; see the configuration file for details.

In particular, setting decode_strategy = "conditional" and logit_metric = "hyperbolic" will train hyperbolic sentence embeddings.

Training and validation scripts can be run in exactly the same manner as for distributed SkipThoughts above.

Training

#!/bin/bash

bazel build -c opt //word_context:train

bazel-bin/word_context/train \
  --input_file_pattern <tfrecord_files> \
  --nosync_replicas \
  --train_dir <path_to_save_models>

Validation

#!/bin/bash

bazel build -c opt //word_context:track_perplexity

CUDA_VISIBLE_DEVICES='' bazel-bin/word_context/track_perplexity \
  --input_file_pattern <tfrecord_files> \
  --checkpoint_dir <path_with_saved_models> \
  --eval_dir <directory_to_log_eval_summaries>

Evaluation

Once a word context model is trained, it can be evaluated as a feature extractor on downstream tasks. In addition to the tasks described in the SkipThoughts paper, we also provide code to evaluate on NLI tasks. See the readme file in the nli directory for details on how this works.

To run an evaluation on a trained Word Context model (including expanding the vocabulary as described in the SkipThoughts paper):

#!/bin/bash

bazel build -c opt //word_context:evaluate_trec

CUDA_VISIBLE_DEVICES='' bazel-bin/word_context/evaluate_trec \
  --checkpoint_dir <path_to_trained_model> \
  --skip_thought_vocab_file <path_to_training_vocabulary_file> \
  --word2vec_embedding_file <path_to_word2vec embeddings> \
  --word2vec_vocab_file <path_to_word2vec_vocabulary> \
  --eval_tasks CR,MR,SUBJ,MPQA,MultiNLI,SNLI \
  --data_dir <path_to_skipthoughts_eval_data> \
  --multinli_dir <path_to_multinli_data> \
  --nouse_norm \
  --nouse_eos \
  --nli_eval_method mlp \
  --nli_eval_dir <path_to_store_NLI_eval_checkpoints>

Note

This is not an official Google product.