Unify adapter benchmark datasets

[stloyd]

Change Log

Added

Fixed

Changed

• Unify adapter benchmark datasets

Removed

Deprecated

Security

---

Description

Script borrowed from #587:

<?php

use Flow\ETL\DSL\CSV;
use Flow\ETL\DSL\From;
use Flow\ETL\DSL\Json;
use Flow\ETL\DSL\Parquet;
use Flow\ETL\Filesystem\SaveMode;
use Flow\ETL\Flow;
use Flow\ETL\DSL\Avro;
use function Flow\ETL\DSL\ref;

include __DIR__ . '/../../vendor/autoload.php';

$faker = Faker\Factory::create();
$orders = array_map(
    static fn (int $i) : array => [
        'order_id' => $faker->uuid,
        'created_at' => $faker->dateTimeThisYear->format(\DateTimeInterface::RFC3339),
        'updated_at' => $faker->dateTimeThisMonth->format(\DateTimeInterface::RFC3339),
        'total_price' => $faker->randomFloat(2, 0, 500),
        'discount' => $faker->randomFloat(2, 0, 50),
        'customer' => [
            'name' => $faker->firstName,
            'last_name' => $faker->lastName,
            'email' => $faker->email,
        ],
        'address' => [
            'street' => $faker->streetAddress,
            'city' => $faker->city,
            'zip' => $faker->postcode,
            'country' => $faker->country,
        ],
        'notes' => \array_map(
            static fn ($i) => $faker->sentence,
            \range(1, $faker->numberBetween(1, 5))
        )
    ],
    range(1, 100)
);

(new Flow())
    ->read(From::array($orders))
    ->mode(SaveMode::Overwrite)
    ->write(Parquet::to(__DIR__ . '/orders_flow.parquet', rows_in_group: 20_000))
    ->write(Json::to(__DIR__ . '/orders_flow.json'))
    ->write(Avro::to(__DIR__ . '/orders_flow.avro'))
    ->withEntry('order_id', ref('order_id')->cast('string'))
    ->withEntry('customer', ref('customer')->cast('string'))
    ->withEntry('address', ref('customer')->cast('string'))
    ->withEntry('notes', ref('customer')->cast('string'))
    ->write(CSV::to(__DIR__ . '/orders_flow.csv'))
    ->run()
;

[norberttech]

How many rows did yo generated? I was thinking about something around 10k since only then parquet might start using more data pages which is pretty important to make sure that there are no memory leaks/bottlenecks there.
Would be also nice to add some optional fields like for example cancelled_at that is not null in like 10% of the entries.
Boolean field could also be a nice addition here.

[norberttech]

You might also want to add location = ['lat', 'lng'] to address to make additional level of nesting.
Could you also put this script into examples so we can reuse it later to adjust those datasets?

[stloyd]

Changed:

• added nullable,
• added boolean,
• added lat & long to address field,
• increased to 10k records,
• included generator in a repository