Update Parquet binaryAsString integration to use a static parquet file

[NVnavkumar]

Fixes #5837.

This updates the parquet integration test which tests the functionality of spark.sql.parquet.binaryAsString=true in the plugin to use a static file, so that integration test can run on clustered environments which only have access to HDFS.

Signed-off-by: Navin Kumar navink@nvidia.com

Steps to create the binary as string parquet file (in Python):

import random

import sre_yield
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

base_strings = sre_yield.AllStrings("(.|\n){1,30}", flags=0, charset=sre_yield.CHARSET, max_count=(1 << 64)
strings = [base_strings[random.randint(0, (1 << 64))] for i in range(500)]
records = [(i, s.encode(), s) for i, s in enumerate(strings)]
df = pd.DataFrame.from_records(records)
pq.write_table(pa.Table.from_pandas(df), 'binary_as_string.parquet')

[NVnavkumar]

build

[NVnavkumar]

build

[jlowe]

I'm OK with using a pre-built file to fix the broken nightly build, but shouldn't we be able to create files locally by using the file: schema explicitly? That could be investigated in a followup.

[NVnavkumar]

I'm OK with using a pre-built file to fix the broken nightly build, but shouldn't we be able to create files locally by using the file: schema explicitly? That could be investigated in a followup.

The fundamental issue here was that the integration tests broke running on a cluster (YARN). I think it makes sense if we can set up the test environment to make it easier to create files on the shared HDFS that the Spark executors would have access to.