Skip to content

Commit

Permalink
update to include the KenLMModel example in README
Browse files Browse the repository at this point in the history
  • Loading branch information
aalok-sathe committed Nov 15, 2023
1 parent 1d962dd commit ee5b226
Show file tree
Hide file tree
Showing 2 changed files with 119 additions and 102 deletions.
10 changes: 9 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@
Compute surprisal from language models!

`surprisal` supports most Causal Language Models (`GPT2`- and `GPTneo`-like models) from Huggingface or local checkpoint,
as well as `GPT3` models from OpenAI using their API!
as well as `GPT3` models from OpenAI using their API! We also support `KenLM` N-gram based language models using the
KenLM Python interface.

Masked Language Models (`BERT`-like models) are in the pipeline and will be supported at a future time.

Expand All @@ -12,6 +13,10 @@ The snippet below computes per-token surprisals for a list of sentences
```python
from surprisal import AutoHuggingFaceModel

from surprisal import KenLMModel
k = KenLMModel(model_path='./literature.arpa')


sentences = [
"The cat is on the mat",
"The cat is on the hat",
Expand All @@ -26,6 +31,9 @@ m.to('cuda') # optionally move your model to GPU!

for result in m.surprise(sentences):
print(result)

for result in k.surprise(sentences):
print(result)
```
and produces output of this sort:
```
Expand Down
Loading

0 comments on commit ee5b226

Please sign in to comment.