Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create an InMemoryDmlHandler and IDmlHandler #791

Closed
Tracked by #754
pkudinov opened this issue Jun 11, 2024 · 1 comment
Closed
Tracked by #754

Create an InMemoryDmlHandler and IDmlHandler #791

pkudinov opened this issue Jun 11, 2024 · 1 comment
Assignees

Comments

@pkudinov
Copy link
Collaborator

pkudinov commented Jun 11, 2024

Create a interface for DmlHandler and implement a new InMemoryDmlHandler which can mock the functionality of DmlHandler using in memory data, for the purposes of local indexer testing. This is necessary because unit testing an indexer is expected to be quick, with high iteration speed. This is not possible when running against an actual Postgres instance in a docker as tear down takes time, set up takes time, and solutions like In Memory PG have various attributes we do not want, such as custom parsers. The first implementation of this can be rough, to cover main features. Later, it can be extended or scrapped based on our needs.

Each DmlHandler function should work, return data, and error out where possible, emulating the behavior of using an actual Postgres instance.

@darunrs
Copy link
Collaborator

darunrs commented Jun 22, 2024

Leaving context on #809 before I go on a 2 week leave.

The code is in a pretty messy in-between state right now. I've done a couple iterations on it to try and find the right abstraction. As a result, most of the code is loose with typing and names. So, I'll first cover things at a high level and then mention the specific decisions made so far in the code.

The data that the InMemoryDmlHandler needs to be stored in a form such that we can:

  1. Enrich input data for fields that need it (Such as serial data, or columns with default values)
  2. Know what columns are primary keys for uniqueness constraints
  3. Search throw rows and column values for select or upsert matching
  4. Return modified data
  5. Be reasonably extendable for any potential features to implement, like type assertions.

So, I broke the problem down into three different problems:

  1. Collecting specification for tables
  2. Enriching data
  3. Performing actions on data

The 1st should be a responsibility of some higher level class which performs this on construciton, with the schema AST as input. The AST contains all information we need, and we just need to continue improving the storage of the information in the AST for our needs. So far, I went with a simple interface, but perhaps we can represent the data in a more extensible form, like configurable classes representing columns.

The 2nd needs too be done at a class level, as things like counters for serial data, need to be available to any row or class when enriching. Maybe this should be its own class since the scope of work here will grow in the future. It could even own validation of input data.

The 3rd I think should be properties of the row, in conjunction with private functions of the data class. The row should have helpful functions like primaryKey() which can be used to compare against other rows. And a private function in the data class which then uses them for things like find.

That is my current state of thought regarding the InMemoryDmlHandler. The code is roughly along those lines, but with somewhat messy abstractions and names that need renaming. But, the insert and select functions to actually work, and I have a unit test for them.

For next steps, I would say this work can just sit back until I come back. But, most of the underlying feature work is there, so implementing the remaining functions shouldn't be too hard. So, if an urgency is there, someone can pick this up and get it to work with a bit of trouble, and start using it for getting a local indexer environment set up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants