Add a token and position embedding layer

[mattdangerw]

For simplicity of our examples, we should add a layer that combines keras.layers.Embedding and keras_nlp.layers.PositionEmbedding into a single offering. Initial take for an API signature...

keras_nlp.layers.TokenAndPositionEmbedding(
    vocabulary_size,
    max_length,
    embedding_dim,
    embeddings_initializer="glorot_uniform",
    embeddings_regularizer=None,
    mask_zero=False,
    **kwargs
)

Internally, this will combine the layers as follows...

# in __init__
self.token_embeddings = layers.Embedding(
    input_dim=vocab_size, output_dim=embedding_dim
)
self.position_embeddings = keras_nlp.layers.PositionEmbedding(
    max_length=max_length
)

# in call
embedded_tokens = self.token_embeddings(inputs)
embedded_positions = self.position_embeddings(embedded_tokens)
outputs = embedded_tokens + embedded_positions

[mattdangerw]

Note that we will do this only for the learned positional embedding (not the sinusoidal encoding). We believe that the learned embedding will be the most common usage for adding position info, so it's the only one we will provide the extra "glue" layer for.

[adhadse]

Can I try to work on this issue? Please assign me.

[mattdangerw]

Please do. Thanks!