File-based CSV: infer datatypes

[girarda]

What

Add an option to infer the data types when of a CSV stream even if the user has not provided a schema.

The existing S3 source does this with pyarrow, and most existing users set this option to True so we shouldn't let this be a breaking change.

This ticket is extracted from #28133.

Proposal 1: Use pyarrow to infer the datatype if the option is set to true

Proposal 2: Write our own inference algorithm

We can infer the type of a column by reading the first N rows of a file. For each column:

1. If all values are boolean values, assume the field is a boolean
2. If all values are numbers, assume the field is a number
3. If all values are lists, assume the field is a list
4. If any, but not all values are a list, assume the field is an object
5. If any value is an object, assume the field is an object
6. Else, assume the field is a string

This approach will require validating the output with existing sources to ensure backward compatibility

Acceptance criteria

• The CsvFormat has a configuration option enabling auto schema inferrence
• The schema inference is backward compatible with the existing S3 source with CSV files

[girarda]

grooming notes:

• the schema inference will need to be used in the discover to create the right schema
• the read operation should use the schema that was inferred from the discover step. it will already be available at the read step
• we should limit the number of rows we read when inferring the schema. We can use a limit similar to the one we use for the json format (bytes per file)
• testing: we can use the pyarrow inferrence and compare it's output with our own algorithm for extra security
• we do not want to use pyarrow long-term because it loads the full csv file in memory