Base RunEvaluator Chain #5750
Base RunEvaluator Chain #5750
Conversation
vowelparrot
force-pushed
the
vwp/evaluator_chain_base
branch
3 times, most recently
from
b4fc0e3 to
9ce74cd
Compare
| return self.parse(text) | ||
|
|
||
|
|
||
| class RunEvaluator(Chain): |
There was a problem hiding this comment.
won't this clash with RunEvaluator from the sdk? Also most (all?) subclasses of Chain end with Chain, i.e. RunEvaluatorChain
There was a problem hiding this comment.
Actually I'm a bit confused here. Should this just be a subclass of langchainplus_sdk.evaluation.evaluator.RunEvaluator? It implements evaluate_run
There was a problem hiding this comment.
It should! my oversight - will rename and inherit
There was a problem hiding this comment.
seems like there are a ton of errors in this notebook. Is this expected? Also, should we move this out of experimental?
There was a problem hiding this comment.
I can use da Vinci instead - I think previously the model hallucinated but a recent push made them parsing errors
| feedback = self.output_parser.parse_chain_output(chain_output) | ||
| return {"feedback": feedback} | ||
|
|
||
| def evaluate_run( |
There was a problem hiding this comment.
having async support might be important here so we can run evals faster/concurrently
vowelparrot
force-pushed
the
vwp/evaluator_chain_base
branch
3 times, most recently
from
4bf985f to
658cae0
Compare
vowelparrot
force-pushed
the
vwp/evaluator_chain_base
branch
from
658cae0 to
7ba8f06
Compare
Clean up a bit and only implement the QA and reference free implementations from langchain-ai#5618
Clean up a bit and only implement the QA and reference free implementations from #5618