Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exponential backtracking in regex blocks Thread #425

Open
ic0ns opened this issue Jan 6, 2020 · 1 comment · May be fixed by #429
Open

Exponential backtracking in regex blocks Thread #425

ic0ns opened this issue Jan 6, 2020 · 1 comment · May be fixed by #429

Comments

@ic0ns
Copy link

ic0ns commented Jan 6, 2020

Hey,
I am currently facing problem with servers which have regular expressions within their robots.txt which require an unreasonable amount of computing time to match for using RobotstxtServer.java (https://www.regular-expressions.info/catastrophic.html). Eg:

User-agent: *
Disallow: /********************/
Disallow: /*******************

Matching for the url: "/asdjdsfsdfjkhejrhwjerhjkfdhksdjfhksjdfhjksdfhjksfdhjksdfasdasdd/js/jquery/jquery-migrate.min.js" completly freezes the thread. The console just shows:

10:53:34.438 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:53:39.438 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:53:39.438 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:53:44.438 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:53:44.438 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:53:49.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:53:49.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:53:54.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:53:54.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:53:59.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:53:59.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:04.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:04.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:09.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:09.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:14.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:14.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:19.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:19.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:24.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:24.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:29.441 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:29.441 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:34.441 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:34.441 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:39.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:39.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:44.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:44.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:49.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:49.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:54.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:54.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:59.443 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:59.443 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS

Is there a way around this?
@dgoiko
Copy link

dgoiko commented Jan 24, 2020 •
edited
Loading

Fast solution: disable robots.txt

Looking at the code, the problem is with Matcher.matches, and it can't be timed out. You'll have to modify robots.txt server and PathRule.

this post talks about timeoutable regex

I'm prerparing a MR with this functionality. You can take a look at it #429 . It decreases overall regexp performance, but it will do the job.

@dgoiko dgoiko linked a pull request Jan 24, 2020 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Timeoutable regular expressions in RobotstxtServer dgoiko/crawler4j
2 participants
@ic0ns @dgoiko