Migration guide to RailsEventStore::JSONClient

[tomasc]

First, thanks very much for adding the JSON column support for PostgreSQL!

I'd love to migrate to the RailsEventStore::JSONClient. Is that even possible? Would you be able to briefly outline a migration script?

Thank you,
Tomas

[lukaszreszke]

Hey @tomasc, thanks for reaching out.

JSON/JSONB column support for PostgreSQL is nothing new in RailsEventStore. Event serialization formats are described here

RailsEventStore::JSONClient is a result of our experience working with RailsEventStore in projects using the PostgreSQL database. However, the same configuration was possible to build before version 2.9.0 of RES.

We don't plan to develop a migration script for the reasons mentioned above. However, there's a blog post by @pawelpacana that you might find useful (https://blog.arkency.com/how-to-migrate-large-database-tables-without-a-headache/), it should help you build and execute the migration script.

Cheers,
Łukasz

[mostlyobvious]

Is that even possible? Would you be able to briefly outline a migration script?

What you want here is changing data and metadata column types in event_store_events table, along with converting its contents to new format.

You can do it either offline (with downtime, stopping new writes) or mostly online (simultaneous writing in two tables, backfilling missing record and finally performing table switch).

In any case you'd probably need two instances of RailsEventStore::Client at hand. One to read existing events from storage, configured for current database columns and used serialization method (i.e. binary + YAML). Second one to be used in new "writes", transforming event objects by the rules of new serializer.

Example of backfilling:

# given new table for changed events
class CreateEventStoreEventsOfTheFuture < ActiveRecord::Migration[4.2]
  def change
    create_table(:event_store_events_v2, id: :bigserial, force: false) do |t|
      t.references  :event,       null: false, type: :uuid
      t.string      :event_type,  null: false
      t.jsonb       :metadata
      t.jsonb       :data,        null: false
      t.datetime    :created_at,  null: false
      t.datetime    :valid_at,    null: true
    end
    add_index :event_store_events_v2, :event_id, unique: true
    add_index :event_store_events_v2, :created_at
    add_index :event_store_events_v2, :valid_at
    add_index :event_store_events_v2, :event_type
  end
end

# and the model to operate on this table
class EventV2 < ::ActiveRecord::Base
  self.primary_key = :id
  self.table_name = "event_store_events_v2"
end

# and the glue needed to work on the new table
repository = 
  RubyEventStore::ActiveRecord::EventRepository.new(
    serializer: RubyEventStore::NULL, # as seen in RailsEventStore::JSONClient internals
    model_factory: ->{ [EventV2, RubyEventStore::ActiveRecord::EventInStream] }
  )

# then having both clients
current_client = RailsEventStore::Client.new     # YAML, binary
future_client  = RailsEventStore::JSONClient.new(repository: repository) # JSON, jsonb

# we can transform in Ruby from one serialization format to another
current_client.read.in_batches_of(1000).each do |event| 
  future_client.append(event)
end

Results — sequence numbers, event ids and timestamps intact; data and metadata converted:

migrate_data_and_metadata_development=# SELECT * from event_store_events;
 id |               event_id               |     event_type     |                                                        metadata                                                        |                                                                                                                                                                                                      data                                                                                                                                                                                                      |         created_at         | valid_at
----+--------------------------------------+--------------------+------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------+----------
  1 | ccc0bd55-9dc8-4376-9b54-9b828851a1c6 | SomethingSomething | \x2d2d2d0a3a636f7272656c6174696f6e5f69643a2032346336323530312d643932352d346630662d393965612d6164373465643362373562310a | \x2d2d2d207b7d0a                                                                                                                                                                                                                                                                                                                                                                                               | 2023-02-02 15:57:43.385879 |
  2 | 0a39eb80-a8e8-40b6-b98c-614939860edd | SomethingSomething | \x2d2d2d0a3a636f7272656c6174696f6e5f69643a2038623430303861342d363435392d343637352d626536392d3463396638353335303034350a | \x2d2d2d0a3a746f6461793a20323032332d30322d30320a                                                                                                                                                                                                                                                                                                                                                               | 2023-02-02 16:03:00.979878 |
  3 | 20207ea0-d953-477c-8cc1-9441f3d24bea | SomethingSomething | \x2d2d2d0a3a636f7272656c6174696f6e5f69643a2037343230393063642d613864322d343431362d623430632d3331623932663265323738630a | \x2d2d2d0a3a746f6d6f72726f773a2021727562792f6f626a6563743a416374697665537570706f72743a3a54696d65576974685a6f6e650a20207574633a20323032332d30322d30332031363a31363a32302e333838363333303030205a0a20207a6f6e653a2021727562792f6f626a6563743a416374697665537570706f72743a3a54696d655a6f6e650a202020206e616d653a204574632f5554430a202074696d653a20323032332d30322d30332031363a31363a32302e333838363333303030205a0a | 2023-02-02 16:16:20.390303 |
(3 rows)

migrate_data_and_metadata_development=# SELECT * from event_store_events_v2;
 id |               event_id               |     event_type     |                                                                                           metadata                                                                                           |                    data                     |         created_at         | valid_at
----+--------------------------------------+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+----------------------------+----------
  1 | ccc0bd55-9dc8-4376-9b54-9b828851a1c6 | SomethingSomething | {"types": {"data": {}, "metadata": {"correlation_id": ["Symbol", "String"]}}, "correlation_id": "24c62501-d925-4f0f-99ea-ad74ed3b75b1"}                                                      | {}                                          | 2023-02-02 15:57:43.385879 |
  2 | 0a39eb80-a8e8-40b6-b98c-614939860edd | SomethingSomething | {"types": {"data": {"today": ["Symbol", "Date"]}, "metadata": {"correlation_id": ["Symbol", "String"]}}, "correlation_id": "8b4008a4-6459-4675-be69-4c9f85350045"}                           | {"today": "2023-02-02"}                     | 2023-02-02 16:03:00.979878 |
  3 | 20207ea0-d953-477c-8cc1-9441f3d24bea | SomethingSomething | {"types": {"data": {"tomorrow": ["Symbol", "ActiveSupport::TimeWithZone"]}, "metadata": {"correlation_id": ["Symbol", "String"]}}, "correlation_id": "742090cd-a8d2-4416-b40c-31b92f2e278c"} | {"tomorrow": "2023-02-03T16:16:20.388633Z"} | 2023-02-02 16:16:20.390303 |
(3 rows)

Writing simultaneously to both current and future event_store_events tables can be tricky. You'd like to retain sequence numbers in the future table as they are in the current table (id column). But you probably won't be able to use database triggers, as described.

Whatever the method to double-write, here's useful snippet to get raw data and metadata from Event object, suitable for SQL INSERT:

# as seen in RailsEventStore::JSONClient internals
future_mapper = 
          RubyEventStore::Mappers::InstrumentedMapper.new(
            RubyEventStore::Mappers::PipelineMapper.new(
              RubyEventStore::Mappers::Pipeline.new(
                RubyEventStore::Mappers::Transformation::PreserveTypes
                  .new
                  .register(Symbol, serializer: ->(v) { v.to_s }, deserializer: ->(v) { v.to_sym })
                  .register(
                    Time,
                    serializer: ->(v) { v.iso8601(RubyEventStore::TIMESTAMP_PRECISION) },
                    deserializer: ->(v) { Time.iso8601(v) }
                  )
                  .register(
                    ActiveSupport::TimeWithZone,
                    serializer: ->(v) { v.iso8601(RubyEventStore::TIMESTAMP_PRECISION) },
                    deserializer: ->(v) { Time.iso8601(v).in_time_zone },
                    stored_type: ->(*) { "ActiveSupport::TimeWithZone" }
                  )
                  .register(Date, serializer: ->(v) { v.iso8601 }, deserializer: ->(v) { Date.iso8601(v) })
                  .register(DateTime, serializer: ->(v) { v.iso8601 }, deserializer: ->(v) { DateTime.iso8601(v) })
                  .register(BigDecimal, serializer: ->(v) { v.to_s }, deserializer: ->(v) { BigDecimal(v) }),
                RubyEventStore::Mappers::Transformation::SymbolizeMetadataKeys.new
              )
            ),
            ActiveSupport::Notifications
          )

# extract raw values

event
=>
#<RubyEventStore::Event:0x000000010aec7740
 @data={:tomorrow=>Fri, 03 Feb 2023 16:16:20.388633000 UTC +00:00},
 @event_id="20207ea0-d953-477c-8cc1-9441f3d24bea",
 @metadata=#<RubyEventStore::Metadata:0x000000010aec7718 @h={:correlation_id=>"742090cd-a8d2-4416-b40c-31b92f2e278c", :timestamp=>2023-02-02 16:16:20.390303 UTC, :valid_at=>2023-02-02 16:16:20.390303 UTC, :event_type=>"SomethingSomething"}>>

record = future_mapper.event_to_record(event)
=>
#<RubyEventStore::Record:0x000000010ae05230
 @data={"tomorrow"=>"2023-02-03T16:16:20.388633Z"},
 @event_id="20207ea0-d953-477c-8cc1-9441f3d24bea",
 @event_type="SomethingSomething",
 @metadata={:correlation_id=>"742090cd-a8d2-4416-b40c-31b92f2e278c", :event_type=>"SomethingSomething", :types=>{:data=>{:tomorrow=>["Symbol", "ActiveSupport::TimeWithZone"]}, :metadata=>{:correlation_id=>["Symbol", "String"], :event_type=>["Symbol", "String"]}}},
 @serialized_records={},
 @timestamp=2023-02-02 16:16:20.390303 UTC,
 @valid_at=2023-02-02 16:16:20.390303 UTC>

serialized_record = record.serialize(JSON)
=>
#<RubyEventStore::SerializedRecord:0x000000010b184500
 @data="{\"tomorrow\":\"2023-02-03T16:16:20.388633Z\"}",
 @event_id="20207ea0-d953-477c-8cc1-9441f3d24bea",
 @event_type="SomethingSomething",
 @metadata="{\"correlation_id\":\"742090cd-a8d2-4416-b40c-31b92f2e278c\",\"event_type\":\"SomethingSomething\",\"types\":{\"data\":{\"tomorrow\":[\"Symbol\",\"ActiveSupport::TimeWithZone\"]},\"metadata\":{\"correlation_id\":[\"Symbol\",\"String\"],\"event_type\":[\"Symbol\",\"String\"]}}}",
 @timestamp="2023-02-02T16:16:20.390303Z",
 @valid_at="2023-02-02T16:16:20.390303Z">

serialized_record.data
=> "{\"tomorrow\":\"2023-02-03T16:16:20.388633Z\"}"

serialized_record.metadata 
=> "{\"correlation_id\":\"742090cd-a8d2-4416-b40c-31b92f2e278c\",\"event_type\":\"SomethingSomething\",\"types\":{\"data\":{\"tomorrow\":[\"Symbol\",\"ActiveSupport::TimeWithZone\"]},\"metadata\":{\"correlation_id\":[\"Symbol\",\"String\"],\"event_type\":[\"Symbol\",\"String\"]}}}"

I hope this helps a bit.

[mostlyobvious]

We also offer commercial support with RailsEventStore. We can help you with writing and performing such migration on your codebase and environment.

Let's connect at res@arkency.com if you're interested 🙂

[tomasc]

Hello @lukaszreszke and @pawelpacana,

Thank you very much for quick and thorough response. Amazing and much appreciated!

Also big thanks for developing and maintaining RES – I am using it for every project and would not look back. Makes everything so much more flexible, better organised, clearer … I could continue listing positives. Love it!

@pawelpacana I will keep the option of commercial support on my mind. As much as I would love to, we are only very small practice (2–4), so at the moment it is not viable for us. Perhaps sometime down the road.

Again, thank you for all your hard work and for open sourcing this project. Your project, the careful documentation, and all blog posts, had and continue to have a big impact here.

Tomas