local human agent

[jaseweston]

Behaviour:

bash-3.2$ python examples/eval_model.py -m local_human -t babi:Task1k:1 -dt valid
[task:babi:Task1k:1]
[download_path:/Users/jase/src/ParlAI/downloads]
[datatype:valid]
[numthreads:1]
[no_images:False]
[batchsize:1]
[datapath:/Users/jase/src/ParlAI/data]
[model:local_human]
[model_params:]
[model_file:]
[num_examples:1000]
[display_examples:False]
[parlai_home:/Users/jase/src/ParlAI]
LocalHumanAgent
[creating task(s): babi:Task1k:1]
[DialogTeacher initializing.]
[loading fbdialog data:/Users/jase/src/ParlAI/data/bAbI/tasks_1-20_v1-2/en-valid-nosf/qa1_valid.txt]
[babi:Task1k:1]: Daniel journeyed to the kitchen.
Daniel journeyed to the bedroom.
Where is Daniel?
[cands: kitchen|garden|bedroom|bathroom|office|hallway]
Enter Your Reply:bedroom
---
{'total': 1, 'accuracy': 1.0, 'f1': 1.0, 'hits@k': {1: 1.0, 5: 1.0, 10: 1.0, 50: 1.0, 100: 1.0}}
[babi:Task1k:1]: Sandra moved to the garden.
Sandra went back to the kitchen.
Where is Sandra?
[cands: kitchen|garden|bedroom|bathroom|office|hallway]
Enter Your Reply:bathroom
---
{'total': 2, 'accuracy': 0.5, 'f1': 0.5, 'hits@k': {1: 0.5, 5: 0.5, 10: 0.5, 50: 0.5, 100: 0.5}}
[babi:Task1k:1]: Daniel travelled to the office.
Sandra went back to the hallway.
Where is Sandra?
[cands: kitchen|garden|bedroom|bathroom|office|hallway]
Enter Your Reply:hallway
---
{'total': 3, 'accuracy': 0.6666666666666666, 'f1': 0.6666666666666666, 'hits@k': {1: 0.6666666666666666, 5: 0.6666666666666666, 10: 0.6666666666666666, 50: 0.6666666666666666, 100: 0.6666666666666666}}
[babi:Task1k:1]: Daniel journeyed to the hallway.
Daniel went to the bathroom.
Where is Sandra?
[cands: kitchen|garden|bedroom|bathroom|office|hallway]
Enter Your Reply: