Getting java.lang.NoClassDefFoundError: software/amazon/awssdk/transfer/s3/progress/TransferListener

[nageshbasutkar-hue]

I am trying to read a CSV file from S3 in Spark using the following code:

val df = spark.read.option("header", "true").csv("s3a://nagesh-iceberg-demo/taxi.csv")                                        
 
However im encountering these issues,
java.lang.NoClassDefFoundError: software/amazon/awssdk/transfer/s3/progress/TransferListener
  at java.base/java.lang.Class.forName0(Native Method)
  at java.base/java.lang.Class.forName(Class.java:398)
  at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2625)

Environment:
Spark version: 3.5.0
Hadoop version: 3.4.0
AWS SDK version: 1.11.999
Steps Taken:

I’ve tried including both v1 and v2 AWS SDK jars in my spark.jars configuration.
I ensured the correct JARs are included for S3 support

[debora-ito]

Hi @nageshbasutkar-hue, we have a FAQ for that:

FAQ: How do I fix a NoClassDefFoundError, NoSuchMethodError or NoSuchFieldError?

java.lang.NoClassDefFoundError: software/amazon/awssdk/transfer/s3/progress/TransferListener

Packages in the "software.amazon.awssdk.*" path are Java SDK v2 packages, make sure the Java SDK v2 is in your classpath.