NUTCH-2959 -- upgrade Tika to 2.9.0 #776

tballison · 2023-09-14T17:51:54Z

Thanks for your contribution to Apache Nutch! Your help is appreciated!

Before opening the pull request, please verify that

there is an open issue on the Nutch issue tracker which describes the problem or the improvement. We cannot accept pull requests without an issue because the change wouldn't be listed in the release notes.
the issue ID (NUTCH-XXXX)
- is referenced in the title of the pull request
- and placed in front of your commit messages surrounded by square brackets ([NUTCH-XXXX] Issue or pull request title)
commits are squashed into a single one (or few commits for larger changes)
Java source code follows Nutch Eclipse Code Formatting rules
Nutch is successfully built and unit tests pass by running ant clean runtime test
there should be no conflicts when merging the pull request branch into the recent master branch. If there are conflicts, please try to rebase the pull request branch on top of a freshly pulled master branch.
if new dependencies are added,
- are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- are LICENSE-binary and NOTICE-binary updated accordingly?

We will be able to faster integrate your pull request if these conditions are met. If you have any questions how to fix your problem or about using Nutch in general, please sign up for the Nutch mailing list. Thanks!

tballison · 2023-09-14T18:39:32Z

I'll merge this in a day or so unless anyone has objections.

tballison · 2023-09-15T11:55:57Z

Converting to draft to manually check for version conflicts.

tballison · 2023-09-18T19:09:58Z

I bumped some of the more common dependencies to match Tika 2.9.0. Let me know what you think.

sebastian-nagel · 2023-09-19T11:16:12Z

afaics, no issues when running in local mode
in pseudo-distributed mode there is an issue with the commons-io versions: 2.8.0 provided by Hadoop (3.3.6) and 2.13.0 requested by Tika (and crawler-commons)
- I've fetched and parsed the Tika standard parsers test documents (see test_tika_parser.sh): the following error appears 44 times in the fetcher (during MIME detection) and 467 during parsing:
```
java.lang.NoSuchMethodError: 'org.apache.commons.io.input.CloseShieldInputStream org.apache.commons.io.input.CloseShieldInputStream.wrap(java.io.InputStream)'
```

@tballison, any idea how to circumvent this error? Hadoop 3.4.0 will upgrade to commons-io 2.11.0 (HADOOP-18301).

tballison · 2023-09-19T13:24:33Z

K. Thank you!

Um... wrap() was introduced in 2.9.0 according to the commons-io docs so 2.11.0 should work.

Can we exclude commons-io from hadoop and then add it as a dependency in the main ivy.xml?

I'll make that change on this PR and then see if I can get pseudo-distributed mode working?

tballison · 2023-09-19T13:45:08Z

I haven't worked with ant in a while. According to ant dependencytree, it looks like we don't have to exclude commons-io everywhere -- placing it in the main ivy.xml has the same effect as putting it in <dependencyManagement/> in maven. It applies throughout.

I wanted to document this in case dependencytree isn't showing what's actually happening. :D

tballison · 2023-09-19T14:23:52Z

I'm guessing that commit won't work if distributed hadoop is bringing its own jars (as you said!). Does hadoop do any custom classloading so that the job jars are isolated from the runtime jars?

Or, more simply, notionally, should the above commit work? Or is that a non-starter?

tballison · 2023-09-19T14:27:35Z

I'm getting a ConnectException when I try to run the tika parser tests via nutch-test-single-node-cluster.

On hadoop startup, I see:

2023-09-19 10:25:15,186 INFO util.GSet: VM type       = 64-bit
2023-09-19 10:25:15,187 INFO util.GSet: 0.029999999329447746% max memory 7.7 GB = 2.4 MB
2023-09-19 10:25:15,187 INFO util.GSet: capacity      = 2^18 = 262144 entries
Re-format filesystem in Storage Directory root= /tmp/hadoop-tallison/dfs/name; location= null ? (Y or N) y
2023-09-19 10:25:17,593 INFO namenode.FSImage: Allocated new BlockPoolId: BP-993805694-127.0.1.1-1695133517585
2023-09-19 10:25:17,594 INFO common.Storage: Will remove files: [/tmp/hadoop-tallison/dfs/name/current/fsimage_0000000000000000000.md5, /tmp/hadoop-tallison/dfs/name/current/VERSION, /tmp/hadoop-tallison/dfs/name/current/fsimage_0000000000000000000, /tmp/hadoop-tallison/dfs/name/current/seen_txid]
2023-09-19 10:25:17,625 INFO common.Storage: Storage directory /tmp/hadoop-tallison/dfs/name has been successfully formatted.
2023-09-19 10:25:17,648 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-tallison/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2023-09-19 10:25:17,705 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-tallison/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 403 bytes saved in 0 seconds .
2023-09-19 10:25:17,718 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2023-09-19 10:25:17,744 INFO namenode.FSNamesystem: Stopping services started for active state
2023-09-19 10:25:17,744 INFO namenode.FSNamesystem: Stopping services started for standby state
2023-09-19 10:25:17,747 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2023-09-19 10:25:17,747 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at tallison-XPS-9320/127.0.1.1
************************************************************/
+ set -e
+ /opt/hadoop/3.3.6/sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: ssh: connect to host localhost port 22: Connection refused
Starting datanodes
localhost: ssh: connect to host localhost port 22: Connection refused
Starting secondary namenodes [tallison-XPS-9320]
tallison-XPS-9320: ssh: connect to host tallison-xps-9320 port 22: Connection refused
+ /opt/hadoop/3.3.6/sbin/start-yarn.sh
Starting resourcemanager
Starting nodemanagers
localhost: ssh: connect to host localhost port 22: Connection refused

I can navigate to http://localhost:8088/cluster/nodes and I see that there are zero nodes.

I cannot navigate to http://localhost:9870/.

When I try to run the test_tika_parser.sh, I see:

+ hadoop fs -mkdir -p crawl/seeds_tika/
mkdir: Call From tallison-XPS-9320/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

This is Ubuntu with java 11.

Any ideas what I'm doing wrong?

sebastian-nagel · 2023-09-19T14:48:20Z

Can we exclude commons-io from hadoop and then add it as a dependency in the main ivy.xml?

When running in distributed or pseudo-distributed mode, commons-io 2.8.0 is first in the classpath, independent from which commons-io version is contained in the Nutch job jar. Using Hadoop methods to write data or for communication may rely on that specific commons-io version and changing the Hadoop classpath is challenging, since even more may break.

Btw., I've just rediscovered that using Tika in (pseudo)distributed mode is broken since the upgrade to Tika 2.3.0, see NUTCH-2937. Although, it didn't seem to have affected the MIME detector.

localhost: ssh: connect to host localhost port 22: Connection refused
Any ideas what I'm doing wrong?

Need to set up passphraseless ssh.

tballison · 2023-09-19T14:51:30Z

Btw., I've just rediscovered that using Tika in (pseudo)distributed mode is broken since the upgrade to Tika 2.3.0, see NUTCH-2937. Although, it didn't seem to have affected the MIME detector.

Oh, dear...

Need to set up passphraseless ssh.

Thank you!

sebastian-nagel · 2023-09-19T15:03:13Z

Oh, dear...

Would need to wait for Hadoop 3.4.0, then we get commons-io 2.11.0 into the classpath. See also the discussion in apache/hadoop#4455.

tballison · 2023-09-19T15:16:36Z

Should we merge on the theory that we're already broken and we aren't breaking things any worse? Or do we just punt until 3.4.0 is available?

sebastian-nagel · 2023-09-19T15:22:19Z

Should we merge on the theory that we're already broken and we aren't breaking things any worse? Or do we just punt until 3.4.0 is available?

Let me check whether the MIME detection works with Tika 2.3.0. I'll run the same test script with master which includes Tika 2.3.0...

…come out with Hadoop 3.4.0.

tballison · 2023-09-19T15:28:31Z

In the latest commit, I downgraded commons-io to be what we're hoping for in Hadoop 2.4.0

tballison · 2023-09-19T15:29:56Z

I checked https://issues.apache.org/jira/browse/TIKA-3655, and it looks like we'd have to downgrade to Tika 2.2.1 (December 2021, released during the log4shell fun) to get a working distributed version.

sebastian-nagel · 2023-09-19T15:58:35Z

Ok, here the results:

Tika 2.3.0 (current Nutch master)
- fetching: not a single NoSuchMethodError
- parsing: only one NoSuchMethodError because of CloseShieldInputStream.wrap(...) (when parsing testTMX.tmx)
```
ParserStatus
    failed=85
    success=624
```
Tika 2.9.0
- fetching: 44 of NoSuchMethodError
- parsing: 467 of NoSuchMethodError
```
ParserStatus
   failed=526
   success=139
```

Or do we just punt until 3.4.0 is available?

This seems the better. Even then (and if successfully tested with 3.4.0): warn users that they need Hadoop 3.4.0 to run Nutch and esp. parse-tika.

tballison · 2023-09-19T17:29:25Z

😭 Y, let's hold off until Hadoop 3.4.0 is released. Tika 3.x may be out by then. 🤣 !

Thank you, again!

lewismc · 2023-09-21T14:14:02Z

I suggest that we downgrade to Tika 2.2.1 to fix that regression.

tballison · 2023-09-26T10:27:24Z

Converting this to draft until Hadoop 3.4.0 is released.

sebastian-nagel · 2023-09-26T17:47:38Z

I suggest that we downgrade to Tika 2.2.1 to fix that regression.

Good point, @lewismc. I've opened NUTCH-3006 for that.

lewismc · 2023-09-28T21:40:15Z

Try the full path Also make sure the directory exists on HDFS

…

On Thu, Sep 28, 2023 at 14:03 Tim Allison ***@***.***> wrote: Paging the nutch-test-single-node-cluster helpdesk.... what do I use for the tika seeds file? Are you using our github repo, or the tika-parsers-common package specifically <https://github.com/apache/tika/tree/main/tika-parsers/tika-parsers-standard>? Or something else? Sorry! + hadoop fs -copyFromLocal seeds_tika.txt crawl/seeds_tika copyFromLocal: `seeds_tika.txt': No such file or directory — Reply to this email directly, view it on GitHub <#776 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI4TF5FHFFNK6PZRRL3FALX4XQYXANCNFSM6AAAAAA4YQQNCM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

sebastian-nagel · 2023-09-29T06:56:48Z

what do I use for the tika seeds file? Are you using our github repo, or the
tika-parsers-common package specifically

see the comments in test_tika_parser.sh

tballison · 2023-09-29T15:05:35Z

With the update to Tika 2.9.1-SNAPSHOT, I get 85 failed parses, most of them are either encrypted documents or "can't retrieve Tika Parser for x"
parse-segment-error-parsing.txt

There is still one NoSuchMethodError, also from commons-io, but this time UnsynchronizedByteArrayInputStream$Builder: at org.apache.tika.parser.pdf.PDFEncodedStringDecoder.decode(PDFEncodedStringDecoder.java:85)

tballison · 2023-09-29T15:06:10Z

see the comments in test_tika_parser.sh

Sorry! Yep, saw that too late.

tballison · 2023-09-29T15:21:22Z

I reverted back to 2.2.1, and that's not far enough back -- there were 222 parse failures many with the wrap problem. I reverted back to 2.0.0, and then had 85 parse failures again. This time, no wrap problems, just the "can't retrieve parser" and encrypted doc exceptions.

tballison · 2023-09-29T16:04:05Z

There is just no winning...

We just upgraded POI to 5.2.4, and it uses a bunch of the newer commons-io methods. If we downgrade POI to 5.2.3, we get a clean build of Tika with commons.io=2.8.0

Can we do anything with shading? I don't think this will work...

So, it looks like we downgrade Tika to 2.0.0 in nutch, or we make a release of Tika 2.9.1 with POI 5.2.3 and then give up until hadoop upgrades

tballison · 2023-09-29T16:09:16Z

Even worse, POI 5.2.4 uses features of commons-io > 2.11.0. This means that POI won't work in hadoop 3.4.0.

tballison · 2023-09-29T16:10:52Z

Stepping away from the keyboard. 😭

tballison · 2023-09-29T17:34:57Z

Alright, the only thing that I think might work is Tika shading commons-io in tika-app, and then Nutch uses tika-app instead of the individual parser-modules etc. for parser-tika.

WDYT?

sebastian-nagel · 2023-09-29T18:04:16Z

Tika shading commons-io in tika-app, and then Nutch uses tika-app instead of the individual parser-modules etc. for parser-tika.

And what about the dependency on tika-core in Nutch core, which is used for MIME detection?

tballison · 2023-09-29T18:08:18Z

That should be ok if we remove CloseShieldInputStream.wrap() from tika-core (and may as well modify all of Tika).

tballison · 2023-10-03T13:55:33Z

Local build of the shim works with 2.9.1-SNAPSHOT, which includes the latest version of POI which will conflict with commons-io in Hadoop 3.4.0 if we don't use this shim (or if Hadoop doesn't upgrade commons-io further).

ParserStatus
failed=82
success=627

I'll downgrade Tika to 2.9.0 and push a shim release to maven central.

tballison · 2023-10-03T15:09:25Z

I published the shim artifacts for: https://github.com/tballison/hadoop-safe-tika

It looks like they haven't made it into the main maven repositories yet. :(

Once they do, I think this PR and the hadoop-safe-tika shim will be ready for review.

This represents an ugly work-around, but I did get a clean build locally, and I ran @sebastian-nagel 's nutch-test-single-node-cluster tika tests.

One area for improvement might be to exclude the dependencies from the shim that Nutch already pulls in, like guava, for example.

The release process for the shim is quite easy. Happy to do that if necessary.

sebastian-nagel · 2023-10-10T12:58:14Z

Hi @tballison, I've tested the shim artifact in local and pseudo-distributed mode: everything looks good.

no more exceptions about CloseShieldInputStream.wrap(...) during fetching (MIME detection) or parsing

+1

Note: I got marginally different results:

ParserStatus
       failed=84
       success=625

... but that could be because I have an outdated set of test documents. I'm using the default http.content.limit = 1 MiB.

tballison · 2023-10-20T18:29:58Z

Thank you so much @sebastian-nagel !

NUTCH-2959 -- bump Tika to 2.9.0

10f7c0c

tballison marked this pull request as draft September 15, 2023 11:55

tballison added 2 commits September 18, 2023 14:46

Merge remote-tracking branch 'upstream/master' into NUTCH-2959

85bdd00

NUTCH-2959 -- bump Tika to 2.9.0, bump common dependencies throughout

6bfeaf4

tballison marked this pull request as ready for review September 18, 2023 19:09

NUTCH-2959 -- bump commons-io

f11d383

NUTCH-2959 -- downgrade commons-io to match the version we expect to …

0f801c1

…come out with Hadoop 3.4.0.

tballison marked this pull request as draft September 26, 2023 10:27

tballison added 3 commits October 3, 2023 10:44

Merge remote-tracking branch 'upstream/master' into NUTCH-2959

e32a0e0

Working now locally and with Seb's single_node_cluster tests

9faf364

update howto_upgrade_tika.txt

9aabc45

tballison marked this pull request as ready for review October 3, 2023 15:04

tballison merged commit 97eb0b5 into apache:master Oct 20, 2023
1 check passed

tballison deleted the NUTCH-2959 branch October 20, 2023 18:29

NUTCH-2959 -- upgrade Tika to 2.9.0 #776

NUTCH-2959 -- upgrade Tika to 2.9.0 #776

Conversation

tballison commented Sep 14, 2023

tballison commented Sep 14, 2023

tballison commented Sep 15, 2023

tballison commented Sep 18, 2023

sebastian-nagel commented Sep 19, 2023

tballison commented Sep 19, 2023

tballison commented Sep 19, 2023 • edited Loading

tballison commented Sep 19, 2023

tballison commented Sep 19, 2023 • edited Loading

sebastian-nagel commented Sep 19, 2023

tballison commented Sep 19, 2023 • edited Loading

sebastian-nagel commented Sep 19, 2023

tballison commented Sep 19, 2023

sebastian-nagel commented Sep 19, 2023

tballison commented Sep 19, 2023

tballison commented Sep 19, 2023

sebastian-nagel commented Sep 19, 2023

tballison commented Sep 19, 2023 • edited Loading

lewismc commented Sep 21, 2023

tballison commented Sep 26, 2023

sebastian-nagel commented Sep 26, 2023

lewismc commented Sep 28, 2023 via email

sebastian-nagel commented Sep 29, 2023

tballison commented Sep 29, 2023

tballison commented Sep 29, 2023

tballison commented Sep 29, 2023

tballison commented Sep 29, 2023

tballison commented Sep 29, 2023

tballison commented Sep 29, 2023

tballison commented Sep 29, 2023

sebastian-nagel commented Sep 29, 2023

tballison commented Sep 29, 2023

tballison commented Oct 3, 2023 • edited Loading

tballison commented Oct 3, 2023

sebastian-nagel commented Oct 10, 2023

tballison commented Oct 20, 2023

tballison commented Sep 19, 2023 •

edited

Loading

tballison commented Sep 19, 2023 •

edited

Loading

tballison commented Sep 19, 2023 •

edited

Loading

tballison commented Sep 19, 2023 •

edited

Loading

tballison commented Oct 3, 2023 •

edited

Loading