Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix markdown text splitter horizontal lines #5625

Merged
merged 8 commits into from
Jun 5, 2023
Merged

fix markdown text splitter horizontal lines #5625

merged 8 commits into from
Jun 5, 2023

Conversation

IlyaMichlin
Copy link
Contributor

Fixes #5614

Issue

The *** combination produces an exception when used as a seperator in re.split. Instead \*\*\* should be used for regex exprations.

Who can review?

@eyurtsev

devstein
devstein reviewed Jun 2, 2023
View reviewed changes
Copy link

@devstein devstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this! MO more future proof solution is to use re.escape on all language separators

@IlyaMichlin
Copy link
Contributor Author

@devstein, I appreciate your suggestion, but using re.escape would also escape the '' character in '\n', which isn't the intended behavior. It's important to note that these strings are regular expressions, so they should be explicitly defined for the sake of clarity and to better comprehend their functionality.

@IlyaMichlin IlyaMichlin changed the title Fixed markdown text splitter horizontal lines fix markdown text splitter horizontal lines Jun 3, 2023
hwchase17
hwchase17 reviewed Jun 3, 2023
View reviewed changes
Copy link
Contributor

@hwchase17 hwchase17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add a test for this?

@IlyaMichlin
Copy link
Contributor Author

@hwchase17 I added a test for markdown.
There were a few issues with the regex expressions for RST and Markdown which I fixed and added test covering this.

@IlyaMichlin IlyaMichlin requested a review from hwchase17 June 5, 2023 05:24
hwchase17
hwchase17 approved these changes Jun 5, 2023
View reviewed changes
Copy link
Contributor

@hwchase17 hwchase17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@hwchase17 hwchase17 added the lgtm PR looks good. Use to confirm that a PR is ready for merging. label Jun 5, 2023
@hwchase17 hwchase17 merged commit d5b1608 into langchain-ai:master Jun 5, 2023
Undertone0809 pushed a commit to Undertone0809/langchain that referenced this pull request Jun 19, 2023
@IlyaMichlin @Undertone0809
fix markdown text splitter horizontal lines (langchain-ai#5625)
2163f62 
Fixes langchain-ai#5614 

#### Issue

The `***` combination produces an exception when used as a seperator in
`re.split`. Instead `\*\*\*` should be used for regex exprations.

#### Who can review?

@eyurtsev
This was referenced Jun 25, 2023
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@devstein devstein devstein left review comments

@hwchase17 hwchase17 hwchase17 approved these changes

Assignees
No one assigned
Labels
lgtm PR looks good. Use to confirm that a PR is ready for merging.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

MarkdownTextSplitter: multiple repeat at position 4 (line 3, column 2)
3 participants
@IlyaMichlin @devstein @hwchase17