Add schemas to datasets #7
Conversation
src/synthesized_datasets/_dtypes.py
Outdated
| DType.DOUBLE: st.DoubleType(), | ||
| DType.INTEGER: st.IntegerType(), | ||
| DType.LONG: st.LongType(), | ||
| DType.NULLABLE_LONG: st.FloatType(), |
There was a problem hiding this comment.
should this be st.LongType() ?
There was a problem hiding this comment.
Spark can't read columns that contain [1.0, , 2.0] because of the decimal points :(
There was a problem hiding this comment.
Maybe with an example I can make more sense:
Say you have a column a table with the following columns:
# some csv file
colA,colB,colC
0,1,1.0
1,,
3,2,2.0
0,1,1.0
Both colA and colB can be read into spark as integers (one being non-null and the other being nullable. But Spark can't read colC as an integer column. Well... it does, but it says everything is Null. i.e.,
+----+----+----+
|colA|colB|colC|
+----+----+----+
| 0| 1| NaN|
| 1| | |
| 3| 2| NaN|
| 0| 1| NaN|
+----+----+----+
So whilst pandas can handle the decimal points, spark cannot. The mapping of Nullable_long to FloatType simply means spark creates the following dataframe which I think is good enough for now! (I think we may want to ultimately rewrite the file)
+----+----+----+
|colA|colB|colC|
+----+----+----+
| 0| 1| 1.0|
| 1| | |
| 3| 2| 2.0|
| 0| 1| 1.0|
+----+----+----+
There was a problem hiding this comment.
As discussed, I'll set this to map to long instead of float (+ update the yaml config).
src/synthesized_datasets/_dtypes.py
Outdated
| DType.DOUBLE: st.DoubleType(), | ||
| DType.INTEGER: st.IntegerType(), | ||
| DType.LONG: st.LongType(), | ||
| DType.NULLABLE_LONG: st.FloatType(), |
There was a problem hiding this comment.
Spark can't read columns that contain [1.0, , 2.0] because of the decimal points :(
|
tested loading all the datasets aswell: |
This PR adds dtypes to all of the datasets.
A few datasets were also edited to improve their quality.
removed Unnamed:0:
updated values:
'12.00.00' -> '12:00:00')'?'replaced with''