This repository combines several algorithms from the natural language processing field (NLP) to determine the similarity of sentences.
The work is funded by an IES grant of Dr. Michael N. Jones and Dr. Jeffrey D. Karpicke. The encoder and classifier components are based on this work from Dr. Jones' group. The server part is inspired by the implementation of Anirudh in Dr. Karpicke's group.
Please see the SETUP.md file about how to install this project.
In order to get a score for the similarity of a response string to a target string, send a get request to /api/score
. Provide the following query parameters:
Parameter | Required | Use Case | Type |
---|---|---|---|
api_key | required | authenticates the api call (authentication can be disabled in the server.py file) |
string |
target | required | target/gold answer to compare the student response to | string |
response | required | response supplied by the student/user | string |
models | required | comma-separated list of encoder models. See below for available models. | string |
classifier | optional | name of a trained classifier (must fit the model-combination). | string |
Here are some example API calls:
/api/score?target=autumn&response=fall&models=quickscore
/api/score?target=plants need water and sunlight to grow&response=plants need the sun&models=infersent
/api/score?target=plants need water and sunlight to grow&response=plants need the sun&models=bow,feature_based&classifier=bow_fb-sick
The server currently supports the following encoder models:
Model Name | Description |
---|---|
infersent | embeds a whole sentence in a 4096 dim. vector; more details here |
bow | "bag of words"; adds up the 300 dim. vector embeddings of all words in the sentence using these pretrained embeddings |
quickscore | calculates the proportion of words in the target that are included in the response, using spell correction, stemming and synonyms |
feature_based | combines the quickscore features with several length and token features of both target and response |
Currently, we have the following pre-trained classifiers:
Classifier Name | Encoder Combination | Target Dataset |
---|---|---|
bow_fb-sick | bow, feature_based | SICK |
infersent-sick | infersent | SICK |
Parameter | Optional | Description | Type |
---|---|---|---|
name | always | used to provide a string which reflect the name of the api, i.e. Automated Scoring | string |
version | always | value representing version of the api called | semantic version number |
errors | optional | a list of errors that occurred | array of strings |
score | optional | the similarity score between target and response | number between 0 and 1 |
models | optional | the models that were used | array of strings |
classifier | optional | the classifier that was used | string |
In order to get a score matrix that compares several targets (idea units) with a student response, send a GET
or POST
request to /api/score-par
. The student response will be split into sentences. Each response sentence will be compared to each idea unit using the same method as in the Score Requests. Provide the following query parameters:
Parameter | Required | Use Case | Type |
---|---|---|---|
api_key | required | authenticates the api call (authentication can be disabled in the server.py file) |
string |
targets | required | JSON string of array of idea units to compare the student response to | string |
response | required | response supplied by the student/user, will be split into sentences | string |
models | required | comma-separated list of encoder models. See below for available models. | string |
classifier | optional | name of a trained classifier (must fit the model-combination). | string |
Here are some example API calls:
/api/score-par?targets=["plants need water to grow", "plants need sunlight to grow"]&response=plants grow in the sun. plants are green.&models=bow,feature_based&classifier=bow_fb-sick
See the previous section for supported encoder models and classifiers.
Parameter | Optional | Description | Type |
---|---|---|---|
name | always | used to provide a string which reflect the name of the api, i.e. Automated Scoring | string |
version | always | value representing version of the api called | semantic version number |
errors | optional | a list of errors that occurred | array of strings |
sentences | optional | student response split into sentences | array of strings |
scores | optional | array of score arrays; the 1st score array contains the similarity scores of the first idea unit to all response sentences | array of arrays of scores between 0 and 1 |
models | optional | the models that were used | array of strings |
classifier | optional | the classifier that was used | string |