4.0.0-rc3

4.0.0-rc3 - 2021-12-20

Fixed

• cli: Fixed missing schema approve --hashes argument.
• codegen: Fixed contract address used instead of an alias when typename is not set.
• tzkt: Fixed processing operations with entrypoint default.
• tzkt: Fixed regression in processing migration originations.
• tzkt: Fixed filtering of big map diffs by the path.

Removed

• cli: Removed deprecated run --oneshot argument and clear-cache command.

4.0.0-rc2

4.0.0-rc2 - 2021-12-11

⚠ Migration

• Run dipdup init command to generate on_synchronized hook stubs.

Added

• hooks: Added on_synchronized hook, which fires each time all indexes reach realtime state.

Fixed

• cli: Fixed config not being verified when invoking some commands.
• codegen: Fixed generating callback arguments for untyped operations.
• index: Fixed incorrect log messages, remove duplicate ones.
• index: Fixed crash while processing storage of some contracts.
• index: Fixed matching of untyped operations filtered by source field (@pravin-d).

Performance

• index: Checks performed on each iteration of the main DipDup loop are slightly faster now.

4.0.0-rc1

4.0.0-rc1 - 2021-12-02

⚠ Migration

• Run dipdup schema approve command on every database you want to use with 4.0.0-rc1. Running dipdup migrate is not necessary since spec_version hasn't changed in this release.

Added

• cli: Added run --early-realtime flag to establish a realtime connection before all indexes are synchronized.
• cli: Addedrun --merge-subscriptions flag to subscribe to all operations/big map diffs during realtime indexing. This flag helps to avoid reaching TzKT subscriptions limit (currently 10000 channels).
• cli: Added status command to print the current status of indexes from the database.
• cli: Added config export [--unsafe] command to print config after resolving all links and variables. Add --unsafe option to substitute environment variables.
• cli: Added cache show command to get information about file caches used by DipDup.
• config: Added first_level and last_level optional fields to TemplateIndexConfig. These limits are applied after ones from the template itself.
• config: Added daemon boolean field to JobConfig to run a single callback indefinitely. Conflicts with crontab and interval fields.
• config: Added advanced top-level section with following fields:

advanced:
  early_realtime: False
  merge_subscriptions: False
  oneshot: False
  postpone_jobs: False
  reindex:
    manual: exception
    migration: wipe
    rollback: ignore
    config_modified: exception
    schema_modified: wipe

ReindexingRequiredError exception is raised by default when reindexing is triggered. CLI flags have priority over self-titled AdvancedConfig fields.

Fixed

• cli: Fixed crashes and output inconsistency when piping DipDup commands.
• cli: Fixed schema wipe --immune flag being ignored.
• codegen: Fixed missing imports in handlers generated during init.
• coinbase: Fixed possible data inconsistency caused by caching enabled for method get_candles.
• hasura: Fixed unnecessary reconfiguration in restart.
• http: Fixed increasing sleep time between failed request attempts.
• index: Fixed invocation of head index callback.
• index: Fixed CallbackError raised instead of ReindexingRequiredError in some cases.
• tzkt: Fixed resubscribing when realtime connectivity is lost for a long time.
• tzkt: Fixed sending useless subscription requests when adding indexes in runtime.
• tzkt: Fixed get_originated_contracts and get_similar_contracts methods whose output was limited to HTTPConfig.batch_size field.
• tzkt: Fixed lots of SignalR bugs by replacing aiosignalrcore library with pysignalr.

Changed

• cli: schema wipe command now requires confirmation when invoked in the interactive shell.
• cli: schema approve command now also causes a recalculation of schema and index config hashes.
• index: DipDup will recalculate respective hashes if reindexing is triggered with config_modified: ignore or schema_modified: ignore in advanced config.

Deprecated

• cli: run --oneshot option is deprecated and will be removed in the next major release. The oneshot mode applies automatically when last_level field is set in the index config.
• cli: clear-cache command is deprecated and will be removed in the next major release. Use cache clear command instead.

Performance

• config: Configuration files are loaded 10x times faster.
• index: Number of operations processed by matcher reduced by 40%-95% depending on the number of addresses and entrypoints used.
• tzkt: Rate limit was increased. Try to set connection_timeout to a higher value if requests fail with ConnectionTimeout exception.
• tzkt: Improved performance of response deserialization.

3.1.3

3.1.3 - 2021-11-15

Fixed

• codegen: Fixed missing imports in operation handlers.
• codegen: Fixed invalid imports and arguments in big_map handlers.

Commits

• 544b477: Fix missing imports in handlers which contain transaction with empty entrypoint (#169) (Lev Gorodetskiy) #169
• 660946e: Fixed invalid imports and arguments in generated big_map handlers (#171) (Lev Gorodetskiy) #171
• 7192153: Bump version: 3.1.2 → 3.1.3 (Lev Gorodetskiy)

3.1.2

3.1.2 - 2021-11-02

Fixed

• Fixed crash occurred during synchronization of big map indexes.

Commits

• 756b8ba: Update CHANGELOG.md (Lev Gorodetskiy)
• 2b09d82: Fix unassigned reference in BigMapFetcher (#166) (Lev Gorodetskiy) #166
• da46b8c: Bump version: 3.1.1 → 3.1.2 (Lev Gorodetskiy)

3.1.1

3.1.1 - 2021-10-18

Fixed

• Fixed loss of real-time subscriptions occurred after TzKT API outage.
• Fixed updating schema hash in schema approve command.
• Fixed possible crash occurred while Hasura is not ready.

Commits

• 950b086: Operation matching tests (#162) (Lev Gorodetskiy) #162
• ea33562: Fixed loss of real-time subscriptions occurred after TzKT API outage (#163) (Lev Gorodetskiy) #163
• 631b573: Bump version: 3.1.0 → 3.1.1 (Lev Gorodetskiy)

3.1.0

3.1.0 - 2021-10-12

Added

• New index class HeadIndex (configuration: dipdup.config.HeadIndexConfig). Use this index type to handle head (limited block header content) updates. This index type is realtime-only: historical data won't be indexed during the synchronization stage.
• Added three new commands: schema approve, schema wipe, and schema export. Run dipdup schema --help command for details.

Changed

• Triggering reindexing won't lead to dropping the database automatically anymore. ReindexingRequiredError is raised instead. --forbid-reindexing option has become default.
• --reindex option is removed. Use dipdup schema wipe instead.
• Values of dipdup_schema.reindex field updated to simplify querying database. See dipdup.enums.ReindexingReason class for possible values.

Fixed

• Fixed ReindexRequiredError not being raised when running DipDup after reindexing was triggered.
• Fixed index config hash calculation. Hashes of existing indexes in a database will be updated during the first run.
• Fixed issue in BigMapIndex causing the partial loss of big map diffs.
• Fixed printing help for CLI commands.
• Fixed merging storage which contains specific nested structures.

Improved

• Raise DatabaseConfigurationError exception when project models are not compatible with GraphQL.
• Another bunch of performance optimizations. Reduced DB pressure, speeded up parallel processing lots of indexes.
• Added initial set of performance benchmarks (run: ./scripts/run_benchmarks.sh)

Commits

• 99a963a: New index type -- 'head' (#153) (Michael Zaikin) #153
• 649ca3c: Fix index config hash calculation, add boilerplate for benchmarking (#152) (Lev Gorodetskiy) #152
• 1bdefa6: Improve models validation (#156) (Lev Gorodetskiy) #156
• 3f34bc2: Bump pytest and other dependencies (#155) (Lev Gorodetskiy) #155
• 285c8d6: Fix ignored Schema.reindex field, add tests (#154) (Lev Gorodetskiy) #154
• possible off-by-1 error (#157) #157 (arrijabba)
• 17f8699: Performance optimizations (#150) (Lev Gorodetskiy) #150
• 41fb7fe: Fix config validation (#160) (Lev Gorodetskiy) #160
• a2ea25d: Fix merging storage which contains specific nested structures (#159) (Lev Gorodetskiy) #159
• 797fa67: Docs and small fixes before 3.1.0 release (#161) (Lev Gorodetskiy) #161
• 6f971ed: Bump changelog (Lev Gorodetskiy)
• 49b0c1e: Bump version: 3.0.4 → 3.1.0 (Lev Gorodetskiy)

3.0.4

3.0.4 - 2021-10-04

Improved

• A significant increase in indexing speed.

Fixed

• Fixed unexpected reindexing caused by the bug in processing zero- and single-level rollbacks.
• Removed unnecessary file IO calls that could cause PermissionError exception in Docker environments.
• Fixed possible violation of block-level atomicity during real-time indexing.

Changes

• Public methods of TzktDatasource now return immutable sequences.

Commits

• ccbb0cb: Fix unnecessary file IO, improve logging (#149) (Lev Gorodetskiy) #149
• 4968ee3: Track datasource level by channel, ignore head rollbacks (#148) (Lev Gorodetskiy) #148
• 7427047: Bump version (Lev Gorodetskiy)
• 3e5bfcc: Bump version: 3.0.3 → 3.0.4 (Lev Gorodetskiy)

3.0.3

3.0.3 - 2021-10-01

Fixed

• Fixed processing of single-level rollbacks emitted before rolled back head.

Commits

• 176b503: Create LICENSE (Lev Gorodetskiy)
• 08b49a0: Allow rollback to the same level (#147) (Lev Gorodetskiy) #147
• 8a03bd6: Update changelog (Lev Gorodetskiy)
• 37a8de9: Bump version: 3.0.2 → 3.0.3 (Lev Gorodetskiy)

3.0.2

Added

• Human-readable CHANGELOG.md 🕺
• Two new options added to dipdup run command:
  • --forbid-reindexing – raise ReindexingRequiredError instead of truncating database when reindexing is triggered for any reason. To continue indexing with existing database run UPDATE dipdup_schema SET reindex = NULL;
  • --postpone-jobs – job scheduler won't start until all indexes are synchronized.

Changed

• Migration to this version requires reindexing.
• dipdup_index.head_id foreign key removed. dipdup_head table still contains the latest blocks from Websocket received by each datasource.

Fixed

• Removed unnecessary calls to TzKT API.
• Fixed removal of PostgreSQL extensions (timescaledb, pgcrypto) by function truncate_database triggered on reindex.
• Fixed creation of missing project package on init.
• Fixed invalid handler callbacks generated on init.
• Fixed detection of existing types in the project.
• Fixed race condition caused by event emitter concurrency.
• Capture unknown exceptions with Sentry before wrapping to DipDupError.
• Fixed job scheduler start delay.
• Fixed processing of reorg messages.

Commits

• 53e8bcb: Add CHANGELOG.md, bump Hasura in examples (#137) (Lev Gorodetskiy) #137
• faster sync, TimescaleDB compatibility (#139) #139 (Lev Gorodetskiy)
• 3007015: Fix creation of missing project package (#140) (Lev Gorodetskiy) #140
• 65ef228: Fix check of existing types, rename arguments with the same name on codegen (#141) (Lev Gorodetskiy) #141
• d178d74: An option to forbid database truncating on reindexing triggered (#143) (Lev Gorodetskiy) #143
• 5808c9a: Fix entering sync state (#142) (Lev Gorodetskiy) #142
• 03ea5e6: Fix some Websocket issues, remove Index.head relation (#144) (Lev Gorodetskiy) #144
• 8902d2b: --postpone-jobs flag (#146) (Lev Gorodetskiy) #146
• 54cd116: Update CHANGELOG.md (Lev Gorodetskiy)
• 993d92f: Bump version: 3.0.1 → 3.0.2 (Lev Gorodetskiy)