[SPARK-5138][SQL] Ensure schema can be inferred from a namedtuple

[mulby]

When attempting to infer the schema of an RDD that contains namedtuples, pyspark fails to identify the records as namedtuples, resulting in it raising an error.

Example:

from pyspark import SparkContext
from pyspark.sql import SQLContext
from collections import namedtuple
import os

sc = SparkContext()
rdd = sc.textFile(os.path.join(os.getenv('SPARK_HOME'), 'README.md'))
TextLine = namedtuple('TextLine', 'line length')
tuple_rdd = rdd.map(lambda l: TextLine(line=l, length=len(l)))
tuple_rdd.take(5)  # This works

sqlc = SQLContext(sc)

# The following line raises an error
schema_rdd = sqlc.inferSchema(tuple_rdd)

The error raised is:

  File "/opt/spark-1.2.0-bin-hadoop2.4/python/pyspark/worker.py", line 107, in main
    process()
  File "/opt/spark-1.2.0-bin-hadoop2.4/python/pyspark/worker.py", line 98, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/opt/spark-1.2.0-bin-hadoop2.4/python/pyspark/serializers.py", line 227, in dump_stream
    vs = list(itertools.islice(iterator, batch))
  File "/opt/spark-1.2.0-bin-hadoop2.4/python/pyspark/rdd.py", line 1107, in takeUpToNumLeft
    yield next(iterator)
  File "/opt/spark-1.2.0-bin-hadoop2.4/python/pyspark/sql.py", line 816, in convert_struct
    raise ValueError("unexpected tuple: %s" % obj)
TypeError: not all arguments converted during string formatting

[AmplabJenkins]

Can one of the admins verify this patch?

[pwendell]

/cc @davies

[davies]

@mulby Thanks to fix the bugs, it looks good to me. BTW, could you also fix the bug when raise the exception if obj is tuple (it should be converted to str)?

Jenkins, OK to test.

[SparkQA]

Test build #561 has started for PR 3978 at commit 375d96b.

• This patch merges cleanly.

[SparkQA]

Test build #561 has finished for PR 3978 at commit 375d96b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mulby]

@davies thanks for the prompt review. I've included that fix as well.

[marmbrus]

ok to test

@davies good to go now?

[SparkQA]

Test build #562 has started for PR 3978 at commit 98c61cc.

• This patch merges cleanly.

[SparkQA]

Test build #562 has finished for PR 3978 at commit 98c61cc.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

@marmbrus I think it's good to go, thanks!

[aarondav]

Jenkins, retest this please.

[SparkQA]

Test build #25443 has started for PR 3978 at commit 98c61cc.

• This patch merges cleanly.

[SparkQA]

Test build #25443 has finished for PR 3978 at commit 98c61cc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25443/
Test PASSed.

[marmbrus]

Thanks! Merged to master.