Add Dense Passage Retriever (incl. Training) #513

kolk · 2020-09-03T15:03:58Z

Provide functionality for Dense Passage Retrieval training

TO_DO:

Tokenization

Tokenizer: Include DPRContextEncoderTokenizer and DPRQuestionEncoderTokenizer

DPRProcessor:

_dict_to_samples

clear_text = {'query_text': 'who sings does he love me with reba',
              'passages': [
                                         {'title': 'Does He Love You', 
                                          'text': 'Does He Love You "Does He Love You" is a song written by Sandy Knox and Billy Stritch, and recorded as a duet by American country music artists Reba McEntire and Linda Davis. It was released in August 1993 as the first single from Reba\'s album "Greatest Hits Volume Two". It is one of country music\'s several songs about a love triangle. "Does He Love You" was written in 1982 by Billy Stritch. He recorded it with a trio in which he performed at the time, because he wanted a song that could be sung by the other two members',
                                          'label': 'positive',
                                          'external_id': '11828866'},
                                         {'title': "Why Don't You Love Me (Beyoncé song)",
                                          'text': 'song. According to the lyrics of "Why Don\'t You Love Me", Knowles impersonates a woman who questions her love interest about the reason for which he does not value her fabulousness, convincing him she\'s the best thing for him as she sings: "Why don\'t you love me... when I make me so damn easy to love?... I got beauty... I got class... I got style and I got ass...". The singer further tells her love interest that the decision not to choose her is "entirely foolish". Originally released as a pre-order bonus track on the deluxe edition of "I Am...',
                                          'label': 'hard_negative',
                                          'external_id': '14525568'}
                           ]
             }

with embed_title = False

tokenized = {'query_tokens': ['[CLS]', 'who', 'sings', 'does', 'he', 'love', 'me', 'with', 're', '##ba', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]'], 
             'passages_tokens': [
                                 ['[CLS]', 'does', 'he', 'love', 'you', '"', 'does', 'he', 'love', 'you', '"', 'is', 'a', 'song', 'written', 'by', 'sandy', 'knox', 'and', 'billy', 'st', '##rit', '##ch', ',', 'and', 'recorded', 'as', 'a', 'duet', 'by', 'american', 'country', 'music', 'artists', 're', '##ba', 'mc', '##ent', '##ire', 'and', 'linda', 'davis', '.', 'it', 'was', 'released', 'in', 'august', '1993', 'as', 'the', 'first', 'single', 'from', 're', '##ba', "'", 's', 'album', '"', 'greatest', 'hits', 'volume', 'two', '"', '.', 'it', 'is', 'one', 'of', 'country', 'music', "'", 's', 'several', 'songs', 'about', 'a', 'love', 'triangle', '.', '"', 'does', 'he', 'love', 'you', '"', 'was', 'written', 'in', '1982', 'by', 'billy', 'st', '##rit', '##ch', '.', 'he', 'recorded', 'it', 'with', 'a', 'trio', 'in', 'which', 'he', 'performed', 'at', 'the', 'time', ',', 'because', 'he', 'wanted', 'a', 'song', 'that', 'could', 'be', 'sung', 'by', 'the', 'other', 'two', 'members', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]'],
                                 ['[CLS]', 'song', '.', 'according', 'to', 'the', 'lyrics', 'of', '"', 'why', 'don', "'", 't', 'you', 'love', 'me', '"', ',', 'knowles', 'imp', '##erson', '##ates', 'a', 'woman', 'who', 'questions', 'her', 'love', 'interest', 'about', 'the', 'reason', 'for', 'which', 'he', 'does', 'not', 'value', 'her', 'fabulous', '##ness', ',', 'convincing', 'him', 'she', "'", 's', 'the', 'best', 'thing', 'for', 'him', 'as', 'she', 'sings', ':', '"', 'why', 'don', "'", 't', 'you', 'love', 'me', '.', '.', '.', 'when', 'i', 'make', 'me', 'so', 'damn', 'easy', 'to', 'love', '?', '.', '.', '.', 'i', 'got', 'beauty', '.', '.', '.', 'i', 'got', 'class', '.', '.', '.', 'i', 'got', 'style', 'and', 'i', 'got', 'ass', '.', '.', '.', '"', '.', 'the', 'singer', 'further', 'tells', 'her', 'love', 'interest', 'that', 'the', 'decision', 'not', 'to', 'choose', 'her', 'is', '"', 'entirely', 'foolish', '"', '.', 'originally', 'released', 'as', 'a', 'pre', '-', 'order', 'bonus', 'track', 'on', 'the', 'deluxe', 'edition', 'of', '"', 'i', 'am', '.', '.', '.', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']]}


features = {'passage_input_ids':  [
                            [101, 2515, 2002, 2293, 2017, 1000, 2515, 2002, 2293, 2017, 1000, 2003, 1037, 2299, 2517, 2011, 7525, 11994, 1998, 5006, 2358, 14778, 2818, 1010, 1998, 2680, 2004, 1037, 11979, 2011, 2137, 2406, 2189, 3324, 2128, 3676, 11338, 4765, 7442, 1998, 8507, 4482, 1012, 2009, 2001, 2207, 1999, 2257, 2857, 2004, 1996, 2034, 2309, 2013, 2128, 3676, 1005, 1055, 2201, 1000, 4602, 4978, 3872, 2048, 1000, 1012, 2009, 2003, 2028, 1997, 2406, 2189, 1005, 1055, 2195, 2774, 2055, 1037, 2293, 9546, 1012, 1000, 2515, 2002, 2293, 2017, 1000, 2001, 2517, 1999, 3196, 2011, 5006, 2358, 14778, 2818, 1012, 2002, 2680, 2009, 2007, 1037, 7146, 1999, 2029, 2002, 2864, 2012, 1996, 2051, 1010, 2138, 2002, 2359, 1037, 2299, 2008, 2071, 2022, 7042, 2011, 1996, 2060, 2048, 2372, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            [101, 2299, 1012, 2429, 2000, 1996, 4581, 1997, 1000, 2339, 2123, 1005, 1056, 2017, 2293, 2033, 1000, 1010, 22815, 17727, 18617, 8520, 1037, 2450, 2040, 3980, 2014, 2293, 3037, 2055, 1996, 3114, 2005, 2029, 2002, 2515, 2025, 3643, 2014, 18783, 2791, 1010, 13359, 2032, 2016, 1005, 1055, 1996, 2190, 2518, 2005, 2032, 2004, 2016, 10955, 1024, 1000, 2339, 2123, 1005, 1056, 2017, 2293, 2033, 1012, 1012, 1012, 2043, 1045, 2191, 2033, 2061, 4365, 3733, 2000, 2293, 1029, 1012, 1012, 1012, 1045, 2288, 5053, 1012, 1012, 1012, 1045, 2288, 2465, 1012, 1012, 1012, 1045, 2288, 2806, 1998, 1045, 2288, 4632, 1012, 1012, 1012, 1000, 1012, 1996, 3220, 2582, 4136, 2014, 2293, 3037, 2008, 1996, 3247, 2025, 2000, 5454, 2014, 2003, 1000, 4498, 13219, 1000, 1012, 2761, 2207, 2004, 1037, 3653, 1011, 2344, 6781, 2650, 2006, 1996, 15203, 3179, 1997, 1000, 1045, 2572, 1012, 1012, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 

           'passage_segment_ids': [
                             [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                             [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],
           'passage_attention_mask': [
                            [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
                            [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 
            'query_input_ids': [101, 2040, 10955, 2515, 2002, 2293, 2033, 2007, 2128, 3676, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
            'query_segment_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
            'query_attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
            'label': [1, 0]}

with embed_title=True

tokenized = {'query_tokens': ['[CLS]', 'who', 'sings', 'does', 'he', 'love', 'me', 'with', 're', '##ba', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]'], 
             'passages_tokens': [
                          ['[CLS]', 'does', 'he', 'love', 'you', '[SEP]', 'does', 'he', 'love', 'you', '"', 'does', 'he', 'love', 'you', '"', 'is', 'a', 'song', 'written', 'by', 'sandy', 'knox', 'and', 'billy', 'st', '##rit', '##ch', ',', 'and', 'recorded', 'as', 'a', 'duet', 'by', 'american', 'country', 'music', 'artists', 're', '##ba', 'mc', '##ent', '##ire', 'and', 'linda', 'davis', '.', 'it', 'was', 'released', 'in', 'august', '1993', 'as', 'the', 'first', 'single', 'from', 're', '##ba', "'", 's', 'album', '"', 'greatest', 'hits', 'volume', 'two', '"', '.', 'it', 'is', 'one', 'of', 'country', 'music', "'", 's', 'several', 'songs', 'about', 'a', 'love', 'triangle', '.', '"', 'does', 'he', 'love', 'you', '"', 'was', 'written', 'in', '1982', 'by', 'billy', 'st', '##rit', '##ch', '.', 'he', 'recorded', 'it', 'with', 'a', 'trio', 'in', 'which', 'he', 'performed', 'at', 'the', 'time', ',', 'because', 'he', 'wanted', 'a', 'song', 'that', 'could', 'be', 'sung', 'by', 'the', 'other', 'two', 'members', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]'], 
                          ['[CLS]', 'why', 'don', "'", 't', 'you', 'love', 'me', '(', 'beyonce', 'song', ')', '[SEP]', 'song', '.', 'according', 'to', 'the', 'lyrics', 'of', '"', 'why', 'don', "'", 't', 'you', 'love', 'me', '"', ',', 'knowles', 'imp', '##erson', '##ates', 'a', 'woman', 'who', 'questions', 'her', 'love', 'interest', 'about', 'the', 'reason', 'for', 'which', 'he', 'does', 'not', 'value', 'her', 'fabulous', '##ness', ',', 'convincing', 'him', 'she', "'", 's', 'the', 'best', 'thing', 'for', 'him', 'as', 'she', 'sings', ':', '"', 'why', 'don', "'", 't', 'you', 'love', 'me', '.', '.', '.', 'when', 'i', 'make', 'me', 'so', 'damn', 'easy', 'to', 'love', '?', '.', '.', '.', 'i', 'got', 'beauty', '.', '.', '.', 'i', 'got', 'class', '.', '.', '.', 'i', 'got', 'style', 'and', 'i', 'got', 'ass', '.', '.', '.', '"', '.', 'the', 'singer', 'further', 'tells', 'her', 'love', 'interest', 'that', 'the', 'decision', 'not', 'to', 'choose', 'her', 'is', '"', 'entirely', 'foolish', '"', '.', 'originally', 'released', 'as', 'a', 'pre', '-', 'order', 'bonus', 'track', 'on', 'the', 'deluxe', 'edition', 'of', '"', 'i', 'am', '.', '.', '.', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']
              ]
         }

features = {'passage_input_ids': [
                              [101, 2515, 2002, 2293, 2017, 102, 2515, 2002, 2293, 2017, 1000, 2515, 2002, 2293, 2017, 1000, 2003, 1037, 2299, 2517, 2011, 7525, 11994, 1998, 5006, 2358, 14778, 2818, 1010, 1998, 2680, 2004, 1037, 11979, 2011, 2137, 2406, 2189, 3324, 2128, 3676, 11338, 4765, 7442, 1998, 8507, 4482, 1012, 2009, 2001, 2207, 1999, 2257, 2857, 2004, 1996, 2034, 2309, 2013, 2128, 3676, 1005, 1055, 2201, 1000, 4602, 4978, 3872, 2048, 1000, 1012, 2009, 2003, 2028, 1997, 2406, 2189, 1005, 1055, 2195, 2774, 2055, 1037, 2293, 9546, 1012, 1000, 2515, 2002, 2293, 2017, 1000, 2001, 2517, 1999, 3196, 2011, 5006, 2358, 14778, 2818, 1012, 2002, 2680, 2009, 2007, 1037, 7146, 1999, 2029, 2002, 2864, 2012, 1996, 2051, 1010, 2138, 2002, 2359, 1037, 2299, 2008, 2071, 2022, 7042, 2011, 1996, 2060, 2048, 2372, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
                              [101, 2339, 2123, 1005, 1056, 2017, 2293, 2033, 1006, 20773, 2299, 1007, 102, 2299, 1012, 2429, 2000, 1996, 4581, 1997, 1000, 2339, 2123, 1005, 1056, 2017, 2293, 2033, 1000, 1010, 22815, 17727, 18617, 8520, 1037, 2450, 2040, 3980, 2014, 2293, 3037, 2055, 1996, 3114, 2005, 2029, 2002, 2515, 2025, 3643, 2014, 18783, 2791, 1010, 13359, 2032, 2016, 1005, 1055, 1996, 2190, 2518, 2005, 2032, 2004, 2016, 10955, 1024, 1000, 2339, 2123, 1005, 1056, 2017, 2293, 2033, 1012, 1012, 1012, 2043, 1045, 2191, 2033, 2061, 4365, 3733, 2000, 2293, 1029, 1012, 1012, 1012, 1045, 2288, 5053, 1012, 1012, 1012, 1045, 2288, 2465, 1012, 1012, 1012, 1045, 2288, 2806, 1998, 1045, 2288, 4632, 1012, 1012, 1012, 1000, 1012, 1996, 3220, 2582, 4136, 2014, 2293, 3037, 2008, 1996, 3247, 2025, 2000, 5454, 2014, 2003, 1000, 4498, 13219, 1000, 1012, 2761, 2207, 2004, 1037, 3653, 1011, 2344, 6781, 2650, 2006, 1996, 15203, 3179, 1997, 1000, 1045, 2572, 1012, 1012, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 
            'passage_segment_ids': [
                              [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                              [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 
            'passage_attention_mask': [
                              [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                              [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 
            'query_input_ids': [101, 2040, 10955, 2515, 2002, 2293, 2033, 2007, 2128, 3676, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
            'query_segment_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
            'query_attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
            'label': [1, 0]}

_sample_to_features
file_to_dict: reads json in DPR format and returns list of dictionaries

 [
           {'query': 'who sings does he love me with reba', 
           'passages': [
                             {'title': 'Does He Love You', 
                              'text': 'Does He Love You "Does He Love You" is a song written by Sandy Knox and Billy Stritch, and recorded as a duet by American country music artists Reba McEntire and Linda Davis. It was released in August 1993 as the first single from Reba\'s album "Greatest Hits Volume Two". It is one of country music\'s several songs about a love triangle. "Does He Love You" was written in 1982 by Billy Stritch. He recorded it with a trio in which he performed at the time, because he wanted a song that could be sung by the other two members', 
                            'label': 'positive',
                            'external_id': '11828866'},
                            {'title': 'Does He Love You',
                            'text': 'Does He Love You "Does He Love You" is a song written by Sandy Knox and Billy Stritch, and recorded as a duet by American country music artists Reba McEntire and Linda Davis. It was released in August 1993 as the first single from Reba\'s album "Greatest Hits Volume Two". It is one of country music\'s several songs about a love triangle. "Does He Love You" was written in 1982 by Billy Stritch. He recorded it with a trio in which he performed at the time, because he wanted a song that could be sung by the other two members', 
                         'label': 'positive', 
                         'external_id': '11828866'}, 
                         ....,
                        ]
         },
....]

-Usage:

from farm.data_handler.processor import DPRProcessor
from haystack.retriever.dpr_utils import DPRContextEncoderTokenizer, DPRQuestionEncoderTokenizer
query_tok = "facebook/dpr-question_encoder-single-nq-base"
query_tokenizer = Tokenizer.load(query_tok, use_fast=False)
passage_tok = "facebook/dpr-ctx_encoder-single-nq-base"
context_tokenizer = Tokenizer.load(passage_tok, use_fast=True)
processor = DPRProcessor(tokenizer=query_tokenizer,
             passage_tokenizer=context_tokenizer,
              max_seq_len=256,
              data_dir=DPR/data/data/retriever",
              train_filename="nq-train.json",
              test_filename="nq-dev.json",
              embed_title=True)
dicts = processor.file_to_dicts(os.path.join(processor.data_dir, processor.train_filename))
sample = processor._dict_to_samples(dicts[0])
feat = processor._sample_to_features(sample)

using fast tokenizer:

from farm.data_handler.processor import DPRProcessor
from haystack.retriever.dpr_utils import DPRContextEncoderTokenizerFast, DPRQuestionEncoderTokenizerFast
query_tok = "facebook/dpr-question_encoder-single-nq-base"
query_tokenizer_fast = Tokenizer.load(query_tok, use_fast=True)
passage_tok = "facebook/dpr-ctx_encoder-single-nq-base"
context_tokenizer_fast = Tokenizer.load(passage_tok, use_fast=True)
processor_fast_tokenizer = DPRProcessor(tokenizer=query_tokenizer_fast,
             passage_tokenizer=context_tokenizer_fast,
              max_seq_len=256,
              data_dir=DPR/data/data/retriever",
              train_filename="nq-train.json",
              test_filename="nq-dev.json",
              embed_title=True)
dicts_fast_tokenizer = processor_fast_tokenizer.file_to_dicts(os.path.join(processor_fast_tokenizer.data_dir, processor_fast_tokenizer.train_filename))
sample = processor_fast_tokenizer._dict_to_samples(dicts_fast_tokenizer[0])
feat = processor_fast_tokenizer._sample_to_features(sample)

tholor

Thanks for the first draft for the processor. Let's iterate on it as there are quite some design decisions to be discussed.

farm/data_handler/processor.py

farm/data_handler/input_features.py

farm/data_handler/processor.py

farm/data_handler/input_features.py

…to dpr_processor

tholor

Looking good! Left a few comment for smaller changes.
Plus there's one bigger topic left: refactoring the conversion of DPR from/to transformers to the new FARM style introduced in #576 (see my comment). However, I think we can tackle that one better in a separate PR.

examples/dpr_encoder.py

examples/onnx_question_answering.py

farm/data_handler/processor.py

farm/modeling/biadaptive_model.py

farm/modeling/language_model.py

…to dpr_processor

tholor

Thanks for the changes. Looking good!

DPRProcessor, sample_to_features_dpr added

0dd4959

kolk requested a review from tholor September 3, 2020 15:04

tholor requested changes Sep 4, 2020

View reviewed changes

tholor reviewed Sep 4, 2020

View reviewed changes

farm/data_handler/input_features.py Outdated Show resolved Hide resolved

remove samples_to_features_dpr, featurize in _dict_to_samples

1982b3e

kolk requested a review from tholor September 7, 2020 14:03

kolk added 3 commits September 8, 2020 19:08

DPRProcessor, sample_to_features_dpr added

6066afd

remove samples_to_features_dpr, featurize in _dict_to_samples

b4c79ac

Merge branch 'dpr_processor' of https://github.com/deepset-ai/FARM in…

407b87f

…to dpr_processor

tholor mentioned this pull request Sep 9, 2020

WIP: Add DPR Training deepset-ai/haystack#273

Closed

kolk added 4 commits September 11, 2020 14:52

DPR tokenization and test added

34b7046

DPR question and context models added

76c89f9

DPRProcessor, sample_to_features_dpr added

2ebadf8

remove samples_to_features_dpr, featurize in _dict_to_samples

6c3744b

tholor mentioned this pull request Sep 14, 2020

Combination of extractive QA and FAQ-style deepset-ai/haystack#280

Closed

kolk added 5 commits September 14, 2020 20:17

DPRProcessor, sample_to_features_dpr added

a1bcdf4

remove samples_to_features_dpr, featurize in _dict_to_samples

f3ef0f8

DPR tokenization and test added

265e928

DPR question and context models added

dafff7e

Merge branch 'dpr_processor' of https://github.com/deepset-ai/FARM in…

a390ab4

…to dpr_processor

tholor assigned kolk and tholor Sep 14, 2020

Timoeller mentioned this pull request Sep 17, 2020

fine tuning a dense passage retriever deepset-ai/haystack#388

Closed

kolk added 7 commits September 17, 2020 21:47

DPR training added

dfe4178

biadaptive model and dpr example script added

2eb4aba

DPRProcessor, sample_to_features_dpr added

e4a4e16

remove samples_to_features_dpr, featurize in _dict_to_samples

e738d91

DPRProcessor, sample_to_features_dpr added

eaf21b6

remove samples_to_features_dpr, featurize in _dict_to_samples

e293197

DPR tokenization and test added

6aef273

kolk and others added 24 commits October 14, 2020 13:56

remove samples_to_features_dpr, featurize in _dict_to_samples

80d4549

DPRProcessor, sample_to_features_dpr added

1186186

remove samples_to_features_dpr, featurize in _dict_to_samples

bef17e7

DPR question and context models added

92dacfa

DPRProcessor, sample_to_features_dpr added

4ba91ae

remove samples_to_features_dpr, featurize in _dict_to_samples

35f8703

DPRProcessor, sample_to_features_dpr added

7f65b75

remove samples_to_features_dpr, featurize in _dict_to_samples

3bd5d3a

DPR training added

d4eee5a

dpr eval added

c761b2c

Average rank evaluation added

55fe3e6

dpr eval report creation modified

744d967

DPR eval refactored

9831be5

DPR eval bug fix

71155e4

Saving models fixed, PredictionHead and Processor names refactored

0f36335

DPR language model names refactored

7e0d9c5

bug fix in intitalizing DPR language model

c0e4fa7

merge conflict resolved

ceb6484

conversion to and from transformers added

4664857

DPR tests added

f1d6815

DPR language model config parameter loading added

c9395a4

doc strings modified, metrics refactored

b5cfecf

docstring warnings fixed

1f47f4a

Merge branch 'master' into dpr_processor

f406581

tholor requested changes Oct 15, 2020

View reviewed changes

tholor changed the title ~~WIP Dense Passage Retrieval Support~~ Add Dense Passage Retriever (incl. Training) Oct 15, 2020

kolk added 2 commits October 15, 2020 19:32

Doc string modified, from_scratch() removed from DPR lan models

15889c9

Merge branch 'dpr_processor' of https://github.com/deepset-ai/FARM in…

e82b61e

…to dpr_processor

tholor approved these changes Oct 15, 2020

View reviewed changes

tholor merged commit b9dbd7d into master Oct 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Dense Passage Retriever (incl. Training) #513

Add Dense Passage Retriever (incl. Training) #513

kolk commented Sep 3, 2020 •

edited

Loading

tholor left a comment

tholor left a comment

tholor left a comment

Add Dense Passage Retriever (incl. Training) #513

Add Dense Passage Retriever (incl. Training) #513

Conversation

kolk commented Sep 3, 2020 • edited Loading

tholor left a comment

Choose a reason for hiding this comment

tholor left a comment

Choose a reason for hiding this comment

tholor left a comment

Choose a reason for hiding this comment

kolk commented Sep 3, 2020 •

edited

Loading