Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Opening ParquetViewer to an empty view #59

Closed
MCRE-BE opened this issue Aug 23, 2022 · 9 comments
Closed

[BUG] Opening ParquetViewer to an empty view #59

MCRE-BE opened this issue Aug 23, 2022 · 9 comments
Labels
bug Something isn't working

Comments

@MCRE-BE
Copy link
Contributor

MCRE-BE commented Aug 23, 2022

Parquet Viewer Version
What version of Parquet Viewer are you experiencing the issue with?
2.3.6.22567

Where was the parquet file created?
Apache Spark, Hive, Java, C#, pyarrow, etc.
Python through pandas with pyarrow backend

Sample File
Upload a sample file so the issue can be debugged!
DLIx12_test.zip

Describe the bug
A clear and concise description of what the bug is.
Opening file in ParquetViewer gives this as output. No idea what is wrong.
Can it be linked to pyarrow version 9? I have no problem opening same sized file for version 8.

image

Screenshots
If applicable, add screenshots to help explain your problem.

Thrift Metadata

{
"Version": 2,
"Num_rows": 3835405,
"Created_by": "parquet-cpp-arrow version 9.0.0",
"Schema": [
{
"Field_id": 0,
"Name": "Transmission_Number",
"Type": "INT64",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Line_Number",
"Type": "INT64",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "DC",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "GOLD_Article",
"Type": "INT64",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "GOLD_Storage_Loc",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "GOLD_Logistic_Variant",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "SAP_Article",
"Type": "INT64",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Delivery_Quantity_Base_Unit",
"Type": "INT64",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Base_UOM",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Delivery_Quantity_Preparation_Unit",
"Type": "INT64",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Preparation_UOM",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Client",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Shipping_Date",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "GOLD_Shipping_Id",
"Type": "INT64",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Line_Id",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "SAP_OBD_Number",
"Type": "INT64",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Article_Managed_by_Unit",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Shipped_Net_Weight",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Missing_Quantity",
"Type": "INT64",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Missing_Weight",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Pallet_Number",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Mother_Pallet",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Daughter_Pallet",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Code_Picker",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Code_Loader",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Route",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Tour_Rank",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Promo_Number",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Full_Pallet",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Depot_Origine_Transfer",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Depot_Preparation",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Order_Date",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Empty_Included",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Empty_Article_Number_Pallet",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Empty_Article_Number_CV",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Crate_Number",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Warehouse_number",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Child_OBD_Number",
"Type": "INT64",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Missing_Motivation",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Promo_Week",
"Type": "DOUBLE",
"Type_length": 0,
"LogicalType": null,
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Delivery_Type",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
},
{
"Field_id": 0,
"Name": "Source_Name",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(STRING: StringType())",
"Scale": 0,
"Precision": 0,
"Repetition_type": "OPTIONAL",
"Converted_type": "UTF8"
}
]
}

Arrow: Schema

{
"Fields": {
"Transmission_Number": {
"DataType": {
"TypeId": 9,
"Name": "int64",
"BitWidth": 64,
"IsSigned": true,
"IsFixedWidth": true
},
"Name": "Transmission_Number",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Line_Number": {
"DataType": {
"TypeId": 9,
"Name": "int64",
"BitWidth": 64,
"IsSigned": true,
"IsFixedWidth": true
},
"Name": "Line_Number",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"DC": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "DC",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"GOLD_Article": {
"DataType": {
"TypeId": 9,
"Name": "int64",
"BitWidth": 64,
"IsSigned": true,
"IsFixedWidth": true
},
"Name": "GOLD_Article",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"GOLD_Storage_Loc": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "GOLD_Storage_Loc",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"GOLD_Logistic_Variant": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "GOLD_Logistic_Variant",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"SAP_Article": {
"DataType": {
"TypeId": 9,
"Name": "int64",
"BitWidth": 64,
"IsSigned": true,
"IsFixedWidth": true
},
"Name": "SAP_Article",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Delivery_Quantity_Base_Unit": {
"DataType": {
"TypeId": 9,
"Name": "int64",
"BitWidth": 64,
"IsSigned": true,
"IsFixedWidth": true
},
"Name": "Delivery_Quantity_Base_Unit",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Base_UOM": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Base_UOM",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Delivery_Quantity_Preparation_Unit": {
"DataType": {
"TypeId": 9,
"Name": "int64",
"BitWidth": 64,
"IsSigned": true,
"IsFixedWidth": true
},
"Name": "Delivery_Quantity_Preparation_Unit",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Preparation_UOM": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Preparation_UOM",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Client": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Client",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Shipping_Date": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Shipping_Date",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"GOLD_Shipping_Id": {
"DataType": {
"TypeId": 9,
"Name": "int64",
"BitWidth": 64,
"IsSigned": true,
"IsFixedWidth": true
},
"Name": "GOLD_Shipping_Id",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Line_Id": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Line_Id",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"SAP_OBD_Number": {
"DataType": {
"TypeId": 9,
"Name": "int64",
"BitWidth": 64,
"IsSigned": true,
"IsFixedWidth": true
},
"Name": "SAP_OBD_Number",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Article_Managed_by_Unit": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Article_Managed_by_Unit",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Shipped_Net_Weight": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Shipped_Net_Weight",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Missing_Quantity": {
"DataType": {
"TypeId": 9,
"Name": "int64",
"BitWidth": 64,
"IsSigned": true,
"IsFixedWidth": true
},
"Name": "Missing_Quantity",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Missing_Weight": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Missing_Weight",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Pallet_Number": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Pallet_Number",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Mother_Pallet": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Mother_Pallet",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Daughter_Pallet": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Daughter_Pallet",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Code_Picker": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Code_Picker",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Code_Loader": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Code_Loader",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Route": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Route",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Tour_Rank": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Tour_Rank",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Promo_Number": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Promo_Number",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Full_Pallet": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Full_Pallet",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Depot_Origine_Transfer": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Depot_Origine_Transfer",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Depot_Preparation": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Depot_Preparation",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Order_Date": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Order_Date",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Empty_Included": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Empty_Included",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Empty_Article_Number_Pallet": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Empty_Article_Number_Pallet",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Empty_Article_Number_CV": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Empty_Article_Number_CV",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Crate_Number": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Crate_Number",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Warehouse_number": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Warehouse_number",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Child_OBD_Number": {
"DataType": {
"TypeId": 9,
"Name": "int64",
"BitWidth": 64,
"IsSigned": true,
"IsFixedWidth": true
},
"Name": "Child_OBD_Number",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Missing_Motivation": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Missing_Motivation",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Promo_Week": {
"DataType": {
"TypeId": 12,
"Name": "double",
"BitWidth": 64,
"IsSigned": true,
"Precision": 2,
"IsFixedWidth": true
},
"Name": "Promo_Week",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Delivery_Type": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Delivery_Type",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
},
"Source_Name": {
"DataType": {
"TypeId": 13,
"Name": "utf8",
"IsFixedWidth": false
},
"Name": "Source_Name",
"IsNullable": true,
"HasMetadata": false,
"Metadata": null
}
},
"Metadata": {
"pandas": "{"index_columns": [{"kind": "range", "name": null, "start": 0, "stop": 3835405, "step": 1}], "column_indexes": [{"name": null, "field_name": null, "pandas_type": "unicode", "numpy_type": "object", "metadata": {"encoding": "UTF-8"}}], "columns": [{"name": "Transmission_Number", "field_name": "Transmission_Number", "pandas_type": "int64", "numpy_type": "int64", "metadata": null}, {"name": "Line_Number", "field_name": "Line_Number", "pandas_type": "int64", "numpy_type": "int64", "metadata": null}, {"name": "DC", "field_name": "DC", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "GOLD_Article", "field_name": "GOLD_Article", "pandas_type": "int64", "numpy_type": "int64", "metadata": null}, {"name": "GOLD_Storage_Loc", "field_name": "GOLD_Storage_Loc", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "GOLD_Logistic_Variant", "field_name": "GOLD_Logistic_Variant", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "SAP_Article", "field_name": "SAP_Article", "pandas_type": "int64", "numpy_type": "int64", "metadata": null}, {"name": "Delivery_Quantity_Base_Unit", "field_name": "Delivery_Quantity_Base_Unit", "pandas_type": "int64", "numpy_type": "int64", "metadata": null}, {"name": "Base_UOM", "field_name": "Base_UOM", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Delivery_Quantity_Preparation_Unit", "field_name": "Delivery_Quantity_Preparation_Unit", "pandas_type": "int64", "numpy_type": "int64", "metadata": null}, {"name": "Preparation_UOM", "field_name": "Preparation_UOM", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Client", "field_name": "Client", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Shipping_Date", "field_name": "Shipping_Date", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "GOLD_Shipping_Id", "field_name": "GOLD_Shipping_Id", "pandas_type": "int64", "numpy_type": "int64", "metadata": null}, {"name": "Line_Id", "field_name": "Line_Id", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "SAP_OBD_Number", "field_name": "SAP_OBD_Number", "pandas_type": "int64", "numpy_type": "int64", "metadata": null}, {"name": "Article_Managed_by_Unit", "field_name": "Article_Managed_by_Unit", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Shipped_Net_Weight", "field_name": "Shipped_Net_Weight", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Missing_Quantity", "field_name": "Missing_Quantity", "pandas_type": "int64", "numpy_type": "int64", "metadata": null}, {"name": "Missing_Weight", "field_name": "Missing_Weight", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Pallet_Number", "field_name": "Pallet_Number", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Mother_Pallet", "field_name": "Mother_Pallet", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Daughter_Pallet", "field_name": "Daughter_Pallet", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Code_Picker", "field_name": "Code_Picker", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Code_Loader", "field_name": "Code_Loader", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Route", "field_name": "Route", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Tour_Rank", "field_name": "Tour_Rank", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Promo_Number", "field_name": "Promo_Number", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Full_Pallet", "field_name": "Full_Pallet", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Depot_Origine_Transfer", "field_name": "Depot_Origine_Transfer", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Depot_Preparation", "field_name": "Depot_Preparation", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Order_Date", "field_name": "Order_Date", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Empty_Included", "field_name": "Empty_Included", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Empty_Article_Number_Pallet", "field_name": "Empty_Article_Number_Pallet", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Empty_Article_Number_CV", "field_name": "Empty_Article_Number_CV", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Crate_Number", "field_name": "Crate_Number", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Warehouse_number", "field_name": "Warehouse_number", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Child_OBD_Number", "field_name": "Child_OBD_Number", "pandas_type": "int64", "numpy_type": "int64", "metadata": null}, {"name": "Missing_Motivation", "field_name": "Missing_Motivation", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Promo_Week", "field_name": "Promo_Week", "pandas_type": "float64", "numpy_type": "float64", "metadata": null}, {"name": "Delivery_Type", "field_name": "Delivery_Type", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name": "Source_Name", "field_name": "Source_Name", "pandas_type": "unicode", "numpy_type": "object", "metadata": null}], "creator": {"library": "pyarrow", "version": "9.0.0"}, "pandas_version": "1.4.3"}"
},
"HasMetadata": true
}

Additional context
Add any other context about the problem here.

Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.

@MCRE-BE MCRE-BE added the bug Something isn't working label Aug 23, 2022
@MCRE-BE
Copy link
Contributor Author

MCRE-BE commented Aug 23, 2022

Can it be linked to? this comment by @eedokl in #37 (comment)_

@MCRE-BE
Copy link
Contributor Author

MCRE-BE commented Aug 24, 2022

BTW. I saw the 2.3.7 pre-release, same bug
image

@MCRE-BE
Copy link
Contributor Author

MCRE-BE commented Sep 20, 2022

@mukunku is it an easy fix or is it a big issue? Love your app, but more and more unusable as I update my files.
Shamefully I cannot assist in a fix :-(

@mukunku
Copy link
Owner

mukunku commented Sep 21, 2022

@mukunku is it an easy fix or is it a big issue? Love your app, but more and more unusable as I update my files. Shamefully I cannot assist in a fix :-(

Unfortunately there's no easy fix for this. Parquet.net doesn't support v2.4 and I can't find any other good alternatives. I just checked the repo and there was a release literally 10 hours ago after more than a year but no mention of v2.4 support. But maybe there's still hope?

I realize without adding support for these files this tool's relevance will fade so I'm trying my best but not sure how much I can accomplish.

@MCRE-BE
Copy link
Contributor Author

MCRE-BE commented Sep 21, 2022

Shame :-( I'll monitor the repo then and I'll see when you are able to push an update.
Likely someone on their end will have figured that one out as well and will do the needed changes. I'll stick with Python then for the moment.

Thanks for all the efforts !

@mukunku
Copy link
Owner

mukunku commented Sep 26, 2022

@MCRE-BE Looks like Christmas came early! parquet-dotnet released v4.0.0 and I created an experimental release: https://github.com/mukunku/ParquetViewer/releases/tag/v2.4.1

Looks like it's able to open your sample file now. Let me know how it goes with other files!

@MCRE-BE
Copy link
Contributor Author

MCRE-BE commented Sep 26, 2022

Hey indeed !
It seems to be working now. Loading Data is much much slower (to never stopping), but I guess that's because of the major refactor and this being experimental.

Top ! I'll close the bug and report in PR #60 if I see any things that need sharing.

@MCRE-BE MCRE-BE closed this as completed Sep 26, 2022
@mukunku
Copy link
Owner

mukunku commented Sep 27, 2022

Hey indeed ! It seems to be working now. Loading Data is much much slower (to never stopping), but I guess that's because of the major refactor and this being experimental.

Top ! I'll close the bug and report in PR #60 if I see any things that need sharing.

There was a bug with file loading. Try the latest exe out: https://github.com/mukunku/ParquetViewer/releases/tag/v2.4.1

@MCRE-BE
Copy link
Contributor Author

MCRE-BE commented Sep 27, 2022

The new release fixed that bug. Thanks 😃

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants