Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix markdown text splitter horizontal lines #5625

Merged
merged 8 commits into from
Jun 5, 2023
Merged

fix markdown text splitter horizontal lines #5625

Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1d3a9ae
Fixed markdown text splitter horizontal lines
IlyaMichlin Jun 2, 2023
597d451
Fixed rst text splitter section lines
IlyaMichlin Jun 2, 2023
384512c
fix markdown regex
IlyaMichlin Jun 5, 2023
a2af1b0
test markdown split
IlyaMichlin Jun 5, 2023
fca3aa7
fix markdown and rst split regex
IlyaMichlin Jun 5, 2023
c76bba8
test markdown and rst split
IlyaMichlin Jun 5, 2023
a1ad1a3
test markdown and rst split
IlyaMichlin Jun 5, 2023
7d79b12
Merge branch 'master' of https://github.com/hwchase17/langchain
IlyaMichlin Jun 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions langchain/text_splitter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -514,7 +514,7 @@ def get_separators_for_language(language: Language) -> List[str]:
# Split along section titles
"\n===\n",
"\n---\n",
"\n***\n",
"\n\*\*\*\n",
# Split along directive markers
"\n.. ",
# Split by the normal type of lines
Expand Down Expand Up @@ -617,7 +617,7 @@ def get_separators_for_language(language: Language) -> List[str]:
# End of code block
"```\n\n",
# Horizontal lines
"\n\n***\n\n",
"\n\n\*\*\*\n\n",
"\n\n---\n\n",
"\n\n___\n\n",
# Note that this splitter doesn't handle horizontal lines defined
Expand Down