New Trello document loader #4767

GMartin-dev · 2023-05-16T06:23:13Z

Added New Trello loader class and documentation

Simple Loader on top of py-trello wrapper.
With a board name you can pull cards and to do some field parameter tweaks on load operation.
I included documentation and examples.
Included unit test cases using patch and a fixture for py-trello client class.

Thanks in advance for your feedback @eyurtsev.

eyurtsev

Requesting changes to move configuration information into the initializer.

eyurtsev · 2023-05-16T14:12:24Z

langchain/document_loaders/trello.py

+        self.api_token = api_token
+
+    def load(
+        self,


@GDrupal Thanks for the contribution!

Document loaders require an argumentless load method.

Could you push the configuration into the initializer?

I've commented on the types of some of the parameters in load, so we can update the type signatures as well when moving to __ init __

eyurtsev · 2023-05-16T14:13:19Z

langchain/document_loaders/trello.py

+        self,
+        board_name: str,
+        card_filter: Optional[str] = "all",
+        include_card_name: Optional[bool] = True,


Bools shouldn't have an optional since we only care about True and False, not about a None value. (At least in this case)

Suggested change

include_card_name: Optional[bool] = True,

include_card_name: bool = True,

eyurtsev · 2023-05-16T14:14:07Z

langchain/document_loaders/trello.py

+        include_card_name: Optional[bool] = True,
+        include_comments: Optional[bool] = True,
+        include_checklist: Optional[bool] = True,
+        extra_metadata: Optional[List[str]] = [


Sequence to denote immutability and assign tuple as default value rather than list. No need for an optional.

Suggested change

extra_metadata: Optional[List[str]] = [

extra_metadata: Sequence[str] = ("due_date", "labels", "list", "is_closed")

eyurtsev · 2023-05-16T14:15:51Z

langchain/document_loaders/trello.py

@@ -0,0 +1,124 @@
+"""Loader that loads cards from Trello"""


Do you have any sample data (can be fake data) to help test that the massaging code inside the loader is correct?

Thanks for you feedback!
I updated most of it already. Now, about the fake data....
what do you need exactly?
Are you testing loaders (with external Api request) with mockups endpoints?
Asking because I could not find examples in the codebase.
In this case we have 2 abstraction layers the actual trello api and the objects returned by py-trello, that last one is the one we interact directly from langchain.
We could create a Trello "Lang Chain Test" board from a dummy email, since the free plan gives you api access too.
But I guess you will have to keep those safe, langchain maintainers side.
No sure if that is even possible for your current test setup.

Ideally we could test everything that follows usage of the trello client.

The client can be patched, to have board.get_cards return a fixture

cards = board.get_cards(card_filter=self.card_filter) # <-- from a fixture instead of the web

A fixture would allow testing all the code that follows that statement, without having to rely on an internet connection or configuring trello accounts etc.

Just attached a comment in the main methods that retrieve data. Each comment has a JSON example of the data structure that method returns.
I hope that's enough. The object from py-trello are quite obscure, I just simplified them for this use case.

eyurtsev · 2023-05-16T14:20:20Z

Thanks @GDrupal looking mostly good -- we only need to change the location of configuration to be consistent with the loader interface

eyurtsev · 2023-05-17T16:10:14Z

@GDrupal with some sample card data, a maintain will be able to merge this change in. We'll refactor the initializer to accept the client and create a classmethod that allows instantiation from api keys

eyurtsev · 2023-05-23T01:58:37Z

@GDrupal apologies for the delay -- I'll try to merge in 1-2 days trying to set some time aside to write the tests.

However, if you're familiar with unit test library and how to use mock/patch or use the responses library to get the fixture working feel free to add the tests -- that'll be helpful :)

GMartin-dev · 2023-05-23T05:10:03Z

@eyurtsev
I'm familiar with unit test. I found how are you doing it in the ConfluenceLoader. I will try to have it ready in the next couple of days.
Don't worry.

GMartin-dev · 2023-05-26T09:45:29Z

tests/unit_tests/document_loaders/test_trello.py

@eyurtsev Added unit test cases! let me know if you see something wrong. The Trello objects are quite "weird", it took me some "creative" approach to get it working.

dev2049 · 2023-05-29T23:00:34Z

langchain/document_loaders/trello.py

+                                for item in checklist.items
+                            ]
+                        )
+                        text_content += f"\n{checklist.name}\n" + "\n".join(items)


guessing only new items should be part of the join? otherwise you'll be reading all the items added from a previous checklists

Good catch!
I saw that you did a little refactor, thanks!

updated

dev2049 · 2023-05-30T07:22:57Z

thanks @GDrupal! will mention this feature on twitter, happy to tag you if there's a twitter handle you wanted me to mention!

GMartin-dev · 2023-05-30T09:11:29Z

Go for it! @musicaoriginal2
My tw handler it's messy, because i'm not only do code and I get into a lot local political discussions xD.
I'm working in a project that will need a loader for Asana and utils for both Asana / Trello.
I will be creating more PR's in the next couple of weeks / months.
So, sadly, you guys will probably have to suffer my sloppy code a bit more.

# Added New Trello loader class and documentation Simple Loader on top of py-trello wrapper. With a board name you can pull cards and to do some field parameter tweaks on load operation. I included documentation and examples. Included unit test cases using patch and a fixture for py-trello client class. --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>

GMartin-dev added 2 commits May 16, 2023 03:06

Added TrelloLoader class and updated init file.

94eff6c

Added documentation in trello.ipynb file.

d7f6966

eyurtsev previously requested changes May 16, 2023

View reviewed changes

Moving configuration into initializer, updating docs.

7fbe260

GMartin-dev requested a review from eyurtsev May 17, 2023 05:54

hwchase17 added the 03 enhancement Enhancement of existing functionality label May 22, 2023

Merge remote-tracking branch 'lg/master' into trello_document_loader

a361e75

dev2049 added the Ɑ: doc loader Related to document loader module (not documentation) label May 25, 2023

GMartin-dev added 2 commits May 26, 2023 06:37

Tweaks. Removing redundant code.

2184eb1

Add dummy test data, py-trello fixture and unit test cases.

31bdbb6

GMartin-dev commented May 26, 2023

View reviewed changes

dev2049 reviewed May 29, 2023

View reviewed changes

dev2049 added 2 commits May 29, 2023 17:11

cr

5507063

poetry

d98f6c1

dev2049 added 2 commits May 29, 2023 17:53

merge

d2a05fb

dep

7fe7720

dev2049 merged commit 0b3e0dd into langchain-ai:master May 30, 2023

danielchalef mentioned this pull request Jun 5, 2023

Zep Hybrid Search #5742

Merged

This was referenced Jun 25, 2023

Zep Authentication #6725

Closed

Zep Authentication #6728

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Trello document loader #4767

New Trello document loader #4767

GMartin-dev commented May 16, 2023 •

edited

Loading

eyurtsev left a comment

eyurtsev May 16, 2023

eyurtsev May 16, 2023

eyurtsev May 16, 2023

eyurtsev May 16, 2023

GMartin-dev May 17, 2023

eyurtsev May 17, 2023

GMartin-dev May 18, 2023

eyurtsev commented May 16, 2023

eyurtsev commented May 17, 2023

eyurtsev commented May 23, 2023

GMartin-dev commented May 23, 2023 •

edited

Loading

GMartin-dev May 26, 2023

dev2049 May 29, 2023

GMartin-dev May 30, 2023

dev2049 commented May 30, 2023

GMartin-dev commented May 30, 2023

	include_card_name: Optional[bool] = True,
	include_card_name: bool = True,

	extra_metadata: Optional[List[str]] = [
	extra_metadata: Sequence[str] = ("due_date", "labels", "list", "is_closed")

New Trello document loader #4767

New Trello document loader #4767

Conversation

GMartin-dev commented May 16, 2023 • edited Loading

Added New Trello loader class and documentation

eyurtsev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eyurtsev commented May 16, 2023

eyurtsev commented May 17, 2023

eyurtsev commented May 23, 2023

GMartin-dev commented May 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dev2049 commented May 30, 2023

GMartin-dev commented May 30, 2023

GMartin-dev commented May 16, 2023 •

edited

Loading

GMartin-dev commented May 23, 2023 •

edited

Loading