You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Dec 20, 2018. It is now read-only.
How do we load multiple AVRO files potentially with different schema(s) in a single directory ? Is there any schema merge facility like we have while loading Parquet files ?
sqlContext.read.option("mergeSchema","true")
following does not work for AVRO files..
var df = sqlContext.read.option("mergeSchema","true").format("com.databricks.spark.avro").avro("/user/demo/")
The text was updated successfully, but these errors were encountered:
this currently doesnt work despite avro having build in support for merging schemas.
the problem is that currently the GenericDatumReader is created without passing in an avro schema.
As of #195, we should now have the ability to use a user-provided Avro schema at read time, so I'm going to mark this as "fixed in 3.1.0". Please re-open if this problem hasn't been fully addressed.
How do we load multiple AVRO files potentially with different schema(s) in a single directory ? Is there any schema merge facility like we have while loading Parquet files ?
sqlContext.read.option("mergeSchema","true")
following does not work for AVRO files..
var df = sqlContext.read.option("mergeSchema","true").format("com.databricks.spark.avro").avro("/user/demo/")
The text was updated successfully, but these errors were encountered: