diff --git a/CHANGES.md b/CHANGES.md index 80e7b4c3b4..215254ab71 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -6,11 +6,34 @@ This file details changes made to TrueBlocks over time. See the [migration notes ## v1.1.0 (2023/10/01) +Added chifra config show +Updated trueBlocks.toml config file to include scraper configs +Updates block data structure to read withdrawals and adds them to the index +Adds withdrawals to blocks data structure, breaking change to block cache - needs migration +Forces v1.0.0-release into the TrueBlocks.toml file (needs a migration) +Added `--watch_list`, `--commands`, `--batch_size`, and `--run_count` options to `chifra monitors`. +Enables `chifra config edit` +Added `--run_count` to `chifra scrape` (for debugging purposes). +Changed names of some rarely used special blocks in `chifra when`. Breaking, but minimal impact. +Removes a number of previously deprecated options. `chifra abis --sol`, `chifra names --named`, `chifra names --to_custom`, `chifra transactions --trace`, and `chifra --log_level` for all commands. +Better error reporting when running against non-tracing nodes. +Fixed an issue with Content-Type in the server. +Fixed an issue where user could hit cntl+c during caching and corrupt the database. +Fixed an issue where scraper was missing some smart contract addresses created during out of gas transactions. +Better error handling from the RPC. +Added a verbose mode to `chifra when` to include more specials and a description for each special block. +Now disallows running `chifra scrape` if the node is not a tracing archive node. +Near complete re-write of block scraper to fix few bugs and prepare for writing v1.0.0 of the spec. +Add `--run_count` to both the `chifra scrape` and the `chifra monitors --watch`, hides both because they are intended for debugging only. +Added `--publisher` option to `chifra init` and `chifra chunks`. - Removed `apiProvider` field from `Chain` data model as unused. - Changes to `Config` data model to improve clarity and consistency: - changed `GetPathToRootConfig` to `PathToRootConfig` - changed `GetPathToCache` to `PathToCache` - changed `GetPathToIndex` to `PathToIndex` +- Removes `runOnce` from `chifra monitors` (in favor of hidden `runCount` - for `chifra scrape` as well). +- Adds `dryRun` to `chifra scrape`. +- Adds `diff` to `chifra chunks`. ## v1.0.0 (2023/08/20) diff --git a/docs/content/api/openapi.yaml b/docs/content/api/openapi.yaml index eb810041bf..9ffd141bd5 100644 --- a/docs/content/api/openapi.yaml +++ b/docs/content/api/openapi.yaml @@ -622,15 +622,6 @@ paths: schema: type: number format: uint64 - - name: runOnce - description: > - available with --watch option only, only run the monitor --watch commands once then quit - required: false - style: form - in: query - explode: true - schema: - type: boolean - name: sleep description: > available with --watch option only, the number of seconds to sleep between runs @@ -2796,6 +2787,12 @@ components: type: array items: $ref: "#/components/schemas/hash" + description: "a possibly empty array of uncle hashes" + withdrawals: + type: array + items: + $ref: "#/components/schemas/withdrawal" + description: "a possibly empty array of withdrawals (post Shanghai)" transaction: description: "transaction data as returned from the RPC (with slight enhancements)" type: object diff --git a/docs/content/chifra/accounts.md b/docs/content/chifra/accounts.md index 966296a634..b607d0f59f 100644 --- a/docs/content/chifra/accounts.md +++ b/docs/content/chifra/accounts.md @@ -243,7 +243,6 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -251,7 +250,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/docs/content/chifra/admin.md b/docs/content/chifra/admin.md index 644d4b2b18..a6ba748750 100644 --- a/docs/content/chifra/admin.md +++ b/docs/content/chifra/admin.md @@ -213,24 +213,25 @@ Links: Each of the following additional configurable command line options are available. -**Configuration file:** `$CONFIG/$CHAIN/blockScrape.toml` -**Configuration group:** `[settings]` +**Configuration file:** `trueBlocks.toml` +**Configuration group:** `[scrape.]` | Item | Type | Default | Description / Default | | ------------------ | ------------ | ------------ | --------- | -| apps_per_chunk | uint64 | 200000 | the number of appearances to build into a chunk before consolidating it | -| snap_to_grid | uint64 | 100000 | an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index | -| first_snap | uint64 | 0 | the first block at which snap_to_grid is enabled | -| unripe_dist | uint64 | 28 | the distance (in blocks) from the front of the chain under which (inclusive) a block is considered unripe | -| channel_count | uint64 | 20 | number of concurrent processing channels | -| allow_missing | bool | true | do not report errors for blockchains that contain blocks with zero addresses | +| appsPerChunk | uint64 | 200000 | the number of appearances to build into a chunk before consolidating it | +| snapToGrid | uint64 | 100000 | an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index | +| firstSnap | uint64 | 0 | the first block at which snap_to_grid is enabled | +| unripeDist | uint64 | 28 | the distance (in blocks) from the front of the chain under which (inclusive) a block is considered unripe | +| channelCount | uint64 | 20 | number of concurrent processing channels | +| allowMissing | bool | true | do not report errors for blockchains that contain blocks with zero addresses | +Note that for Ethereum mainnet, the default values for appsPerChunk and firstSnap are 2,000,000 and 2,300,000 respectively. See the specification for a justification of these values. These items may be set in three ways, each overridding the preceeding method: --- in the above configuration file under the `[settings]` group, --- in the environment by exporting the configuration item as UPPER_CASE, without underbars, and prepended with TB_SETTINGS_, or --- on the command line using the configuration item with leading dashes (i.e., `--name`). +-- in the above configuration file under the `[scrape.]` group, +-- in the environment by exporting the configuration item as UPPER_CASE (with underbars removed) and prepended with TB_SCRAPE_CHAIN_, or +-- on the command line using the configuration item with leading dashes and in snake case (i.e., `--snake_case`). ### further information @@ -314,6 +315,7 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. ``` Data models produced by this tool: diff --git a/docs/content/data-model/chaindata.md b/docs/content/data-model/chaindata.md index 8649b822a9..6f824b4960 100644 --- a/docs/content/data-model/chaindata.md +++ b/docs/content/data-model/chaindata.md @@ -45,7 +45,8 @@ Blocks consist of the following fields: | date | a calculated field -- the date of the object | datetime | | transactions | a possibly empty array of transactions or transaction hashes | [Transaction[]](/data-model/chaindata/#transaction) | | baseFeePerGas | the base fee for this block | wei | -| uncles | | Hash | +| uncles | a possibly empty array of uncle hashes | Hash | +| withdrawals | a possibly empty array of withdrawals (post Shanghai) | [Withdrawal[]](/data-model/chaindata/#withdrawal) | ## Transaction diff --git a/docs/readmes/accounts-monitors.md b/docs/readmes/accounts-monitors.md index 07916c4da4..bbb29ce36c 100644 --- a/docs/readmes/accounts-monitors.md +++ b/docs/readmes/accounts-monitors.md @@ -63,7 +63,6 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -71,7 +70,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/docs/readmes/admin-chunks.md b/docs/readmes/admin-chunks.md index c778bff2e7..b66806a031 100644 --- a/docs/readmes/admin-chunks.md +++ b/docs/readmes/admin-chunks.md @@ -43,6 +43,7 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. ``` Data models produced by this tool: diff --git a/docs/readmes/admin-scrape.md b/docs/readmes/admin-scrape.md index ed60d67fe7..c5b71565b9 100644 --- a/docs/readmes/admin-scrape.md +++ b/docs/readmes/admin-scrape.md @@ -50,24 +50,25 @@ Links: Each of the following additional configurable command line options are available. -**Configuration file:** `$CONFIG/$CHAIN/blockScrape.toml` -**Configuration group:** `[settings]` +**Configuration file:** `trueBlocks.toml` +**Configuration group:** `[scrape.]` | Item | Type | Default | Description / Default | | ------------------ | ------------ | ------------ | --------- | -| apps_per_chunk | uint64 | 200000 | the number of appearances to build into a chunk before consolidating it | -| snap_to_grid | uint64 | 100000 | an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index | -| first_snap | uint64 | 0 | the first block at which snap_to_grid is enabled | -| unripe_dist | uint64 | 28 | the distance (in blocks) from the front of the chain under which (inclusive) a block is considered unripe | -| channel_count | uint64 | 20 | number of concurrent processing channels | -| allow_missing | bool | true | do not report errors for blockchains that contain blocks with zero addresses | +| appsPerChunk | uint64 | 200000 | the number of appearances to build into a chunk before consolidating it | +| snapToGrid | uint64 | 100000 | an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index | +| firstSnap | uint64 | 0 | the first block at which snap_to_grid is enabled | +| unripeDist | uint64 | 28 | the distance (in blocks) from the front of the chain under which (inclusive) a block is considered unripe | +| channelCount | uint64 | 20 | number of concurrent processing channels | +| allowMissing | bool | true | do not report errors for blockchains that contain blocks with zero addresses | +Note that for Ethereum mainnet, the default values for appsPerChunk and firstSnap are 2,000,000 and 2,300,000 respectively. See the specification for a justification of these values. These items may be set in three ways, each overridding the preceeding method: --- in the above configuration file under the `[settings]` group, --- in the environment by exporting the configuration item as UPPER_CASE, without underbars, and prepended with TB_SETTINGS_, or --- on the command line using the configuration item with leading dashes (i.e., `--name`). +-- in the above configuration file under the `[scrape.]` group, +-- in the environment by exporting the configuration item as UPPER_CASE (with underbars removed) and prepended with TB_SCRAPE_CHAIN_, or +-- on the command line using the configuration item with leading dashes and in snake case (i.e., `--snake_case`). ### further information diff --git a/docs/templates/api/components.txt b/docs/templates/api/components.txt index 4f67493a70..449ca6607a 100644 --- a/docs/templates/api/components.txt +++ b/docs/templates/api/components.txt @@ -424,6 +424,12 @@ components: type: array items: $ref: "#/components/schemas/hash" + description: "a possibly empty array of uncle hashes" + withdrawals: + type: array + items: + $ref: "#/components/schemas/withdrawal" + description: "a possibly empty array of withdrawals (post Shanghai)" transaction: description: "transaction data as returned from the RPC (with slight enhancements)" type: object diff --git a/docs/templates/readme-intros/admin-scrape.config.md b/docs/templates/readme-intros/admin-scrape.config.md index 8619eb92a9..aac4fbb252 100644 --- a/docs/templates/readme-intros/admin-scrape.config.md +++ b/docs/templates/readme-intros/admin-scrape.config.md @@ -1,8 +1,8 @@ | Item | Type | Default | Description / Default | | ------------------ | ------------ | ------------ | --------- | -| apps_per_chunk | uint64 | 200000 | the number of appearances to build into a chunk before consolidating it | -| snap_to_grid | uint64 | 100000 | an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index | -| first_snap | uint64 | 0 | the first block at which snap_to_grid is enabled | -| unripe_dist | uint64 | 28 | the distance (in blocks) from the front of the chain under which (inclusive) a block is considered unripe | -| channel_count | uint64 | 20 | number of concurrent processing channels | -| allow_missing | bool | true | do not report errors for blockchains that contain blocks with zero addresses | +| appsPerChunk | uint64 | 200000 | the number of appearances to build into a chunk before consolidating it | +| snapToGrid | uint64 | 100000 | an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index | +| firstSnap | uint64 | 0 | the first block at which snap_to_grid is enabled | +| unripeDist | uint64 | 28 | the distance (in blocks) from the front of the chain under which (inclusive) a block is considered unripe | +| channelCount | uint64 | 20 | number of concurrent processing channels | +| allowMissing | bool | true | do not report errors for blockchains that contain blocks with zero addresses | diff --git a/sdk/python/src/_monitors.py b/sdk/python/src/_monitors.py index 018c9217bb..a6af4267a3 100644 --- a/sdk/python/src/_monitors.py +++ b/sdk/python/src/_monitors.py @@ -16,7 +16,6 @@ "watchlist": {"hotkey": "-a", "type": "flag"}, "commands": {"hotkey": "-c", "type": "flag"}, "batchSize": {"hotkey": "-b", "type": "flag"}, - "runOnce": {"hotkey": "-r", "type": "switch"}, "sleep": {"hotkey": "-s", "type": "flag"}, "fmt": {"hotkey": "-x", "type": "flag"}, "verbose:": {"hotkey": "-v", "type": "switch"}, diff --git a/sdk/typescript/src/paths/monitors.ts b/sdk/typescript/src/paths/monitors.ts index 39d370c427..4d77d22ac3 100644 --- a/sdk/typescript/src/paths/monitors.ts +++ b/sdk/typescript/src/paths/monitors.ts @@ -18,7 +18,6 @@ export function getMonitors( watchlist?: string, commands?: string, batchSize?: uint64, - runOnce?: boolean, sleep?: double, chain: string, noHeader?: boolean, diff --git a/sdk/typescript/src/types/block.ts b/sdk/typescript/src/types/block.ts index 8f361fab9c..07d5fd1b43 100644 --- a/sdk/typescript/src/types/block.ts +++ b/sdk/typescript/src/types/block.ts @@ -3,7 +3,7 @@ /* * This file was generated with makeClass --sdk. Do not edit it. */ -import { address, blknum, datetime, gas, hash, timestamp, Transaction, uint64, wei } from '.'; +import { address, blknum, datetime, gas, hash, timestamp, Transaction, uint64, wei, Withdrawal } from '.'; export type Block = { author: address @@ -29,4 +29,5 @@ export type Block = { transactions: Transaction[] transactionsRoot: hash uncles?: hash[] + withdrawals?: Withdrawal[] } diff --git a/src/apps/chifra/cmd/chunks.go b/src/apps/chifra/cmd/chunks.go index e893ee6013..1a068b1dc8 100644 --- a/src/apps/chifra/cmd/chunks.go +++ b/src/apps/chifra/cmd/chunks.go @@ -57,7 +57,8 @@ Notes: - The --belongs option is only available in the index mode. - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - - The --publish option requires a private key.` + - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher.` func init() { var capabilities = caps.Default // Additional global caps for chifra chunks @@ -69,16 +70,20 @@ func init() { chunksCmd.Flags().BoolVarP(&chunksPkg.GetOptions().Check, "check", "c", false, "check the manifest, index, or blooms for internal consistency") chunksCmd.Flags().BoolVarP(&chunksPkg.GetOptions().Pin, "pin", "i", false, "pin the manifest or each index chunk and bloom") chunksCmd.Flags().BoolVarP(&chunksPkg.GetOptions().Publish, "publish", "p", false, "publish the manifest to the Unchained Index smart contract") + chunksCmd.Flags().StringVarP(&chunksPkg.GetOptions().Publisher, "publisher", "P", "trueblocks.eth", "for some query options, the publisher of the index (hidden)") chunksCmd.Flags().Uint64VarP(&chunksPkg.GetOptions().Truncate, "truncate", "n", 0, "truncate the entire index at this block (requires a block identifier) (hidden)") chunksCmd.Flags().BoolVarP(&chunksPkg.GetOptions().Remote, "remote", "r", false, "prior to processing, retreive the manifest from the Unchained Index smart contract") chunksCmd.Flags().StringSliceVarP(&chunksPkg.GetOptions().Belongs, "belongs", "b", nil, "in index mode only, checks the address(es) for inclusion in the given index chunk") + chunksCmd.Flags().BoolVarP(&chunksPkg.GetOptions().Diff, "diff", "f", false, "compare two index portions (see notes) (hidden)") chunksCmd.Flags().Uint64VarP(&chunksPkg.GetOptions().FirstBlock, "first_block", "F", 0, "first block to process (inclusive)") chunksCmd.Flags().Uint64VarP(&chunksPkg.GetOptions().LastBlock, "last_block", "L", 0, "last block to process (inclusive)") chunksCmd.Flags().Uint64VarP(&chunksPkg.GetOptions().MaxAddrs, "max_addrs", "m", 0, "the max number of addresses to process in a given chunk") chunksCmd.Flags().BoolVarP(&chunksPkg.GetOptions().Deep, "deep", "d", false, "if true, dig more deeply during checking (manifest only)") chunksCmd.Flags().Float64VarP(&chunksPkg.GetOptions().Sleep, "sleep", "s", 0.0, "for --remote pinning only, seconds to sleep between API calls") if os.Getenv("TEST_MODE") != "true" { + chunksCmd.Flags().MarkHidden("publisher") chunksCmd.Flags().MarkHidden("truncate") + chunksCmd.Flags().MarkHidden("diff") } globals.InitGlobals(chunksCmd, &chunksPkg.GetOptions().Globals, capabilities) diff --git a/src/apps/chifra/cmd/init.go b/src/apps/chifra/cmd/init.go index b814c534bf..dd9435933a 100644 --- a/src/apps/chifra/cmd/init.go +++ b/src/apps/chifra/cmd/init.go @@ -63,8 +63,12 @@ func init() { initCmd.Flags().BoolVarP(&initPkg.GetOptions().All, "all", "a", false, "in addition to Bloom filters, download full index chunks (recommended)") initCmd.Flags().BoolVarP(&initPkg.GetOptions().DryRun, "dry_run", "d", false, "display the results of the download without actually downloading") + initCmd.Flags().StringVarP(&initPkg.GetOptions().Publisher, "publisher", "P", "trueblocks.eth", "the publisher of the index to download (hidden)") initCmd.Flags().Uint64VarP(&initPkg.GetOptions().FirstBlock, "first_block", "F", 0, "do not download any chunks earlier than this block") initCmd.Flags().Float64VarP(&initPkg.GetOptions().Sleep, "sleep", "s", 0.0, "seconds to sleep between downloads") + if os.Getenv("TEST_MODE") != "true" { + initCmd.Flags().MarkHidden("publisher") + } globals.InitGlobals(initCmd, &initPkg.GetOptions().Globals, capabilities) initCmd.SetUsageTemplate(UsageWithNotes(notesInit)) diff --git a/src/apps/chifra/cmd/list.go b/src/apps/chifra/cmd/list.go index 1dd2938cac..9bc4cc9200 100644 --- a/src/apps/chifra/cmd/list.go +++ b/src/apps/chifra/cmd/list.go @@ -66,8 +66,12 @@ func init() { listCmd.Flags().Uint64VarP(&listPkg.GetOptions().FirstRecord, "first_record", "c", 0, "the first record to process") listCmd.Flags().Uint64VarP(&listPkg.GetOptions().MaxRecords, "max_records", "e", 250, "the maximum number of records to process") listCmd.Flags().BoolVarP(&listPkg.GetOptions().Reversed, "reversed", "E", false, "produce results in reverse chronological order") + listCmd.Flags().StringVarP(&listPkg.GetOptions().Publisher, "publisher", "P", "trueblocks.eth", "for some query options, the publisher of the index (hidden)") listCmd.Flags().Uint64VarP(&listPkg.GetOptions().FirstBlock, "first_block", "F", 0, "first block to export (inclusive, ignored when freshening)") listCmd.Flags().Uint64VarP(&listPkg.GetOptions().LastBlock, "last_block", "L", 0, "last block to export (inclusive, ignored when freshening)") + if os.Getenv("TEST_MODE") != "true" { + listCmd.Flags().MarkHidden("publisher") + } globals.InitGlobals(listCmd, &listPkg.GetOptions().Globals, capabilities) listCmd.SetUsageTemplate(UsageWithNotes(notesList)) diff --git a/src/apps/chifra/cmd/monitors.go b/src/apps/chifra/cmd/monitors.go index 436417548b..592bb4b789 100644 --- a/src/apps/chifra/cmd/monitors.go +++ b/src/apps/chifra/cmd/monitors.go @@ -49,7 +49,7 @@ const longMonitors = `Purpose: const notesMonitors = ` Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list).` @@ -70,8 +70,11 @@ func init() { monitorsCmd.Flags().StringVarP(&monitorsPkg.GetOptions().Watchlist, "watchlist", "a", "", "available with --watch option only, a file containing the addresses to watch") monitorsCmd.Flags().StringVarP(&monitorsPkg.GetOptions().Commands, "commands", "c", "", "available with --watch option only, the file containing the list of commands to apply to each watched address") monitorsCmd.Flags().Uint64VarP(&monitorsPkg.GetOptions().BatchSize, "batch_size", "b", 8, "available with --watch option only, the number of monitors to process in each batch") - monitorsCmd.Flags().BoolVarP(&monitorsPkg.GetOptions().RunOnce, "run_once", "r", false, "available with --watch option only, only run the monitor --watch commands once then quit") + monitorsCmd.Flags().Uint64VarP(&monitorsPkg.GetOptions().RunCount, "run_count", "u", 0, "available with --watch option only, run the monitor this many times, then quit (hidden)") monitorsCmd.Flags().Float64VarP(&monitorsPkg.GetOptions().Sleep, "sleep", "s", 14, "available with --watch option only, the number of seconds to sleep between runs") + if os.Getenv("TEST_MODE") != "true" { + monitorsCmd.Flags().MarkHidden("run_count") + } globals.InitGlobals(monitorsCmd, &monitorsPkg.GetOptions().Globals, capabilities) monitorsCmd.SetUsageTemplate(UsageWithNotes(notesMonitors)) diff --git a/src/apps/chifra/cmd/root.go b/src/apps/chifra/cmd/root.go index 99ded4ed1b..b0463aa980 100644 --- a/src/apps/chifra/cmd/root.go +++ b/src/apps/chifra/cmd/root.go @@ -32,7 +32,6 @@ func init() { chifraCmd.SetOut(os.Stderr) chifraCmd.SetFlagErrorFunc(ErrFunc) chifraCmd.Flags().SortFlags = false - chifraCmd.SetUsageTemplate(helpText) } diff --git a/src/apps/chifra/cmd/root_initialize.go b/src/apps/chifra/cmd/root_initialize.go index e8c55c90b8..b2e199d197 100644 --- a/src/apps/chifra/cmd/root_initialize.go +++ b/src/apps/chifra/cmd/root_initialize.go @@ -13,7 +13,9 @@ import ( "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/colors" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config/upgrade" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/version" ) // Initialize makes sure everything is ready to run. These routines don't return if they aren't @@ -45,119 +47,55 @@ func VerifyOs() { } } -const shouldNotExist string = ` - - A configuration file or folder ({0}) exists in an old location. Please follow - all migrations up to {1} to continue. - - See https://github.com/TrueBlocks/trueblocks-core/blob/develop/MIGRATIONS.md - - [{VERSION}] - -` - -const notExist string = ` - - The configuration file or folder ({0}) - does not exist. See https://trueblocks.io/docs/install/install-core/. - - [{VERSION}] - -` - -const noChains string = ` - - The root configuration file ({0}) - does not contain a list of chains. Please follow all migrations up to {1} to continue. - - See https://github.com/TrueBlocks/trueblocks-core/blob/develop/MIGRATIONS.md - - [{VERSION}] - -` - -const backVersion string = ` - - An outdated version of a configration file was found. Please carefully follow - migration {0} before proceeding. - - See https://github.com/TrueBlocks/trueblocks-core/blob/develop/MIGRATIONS.md - - [{VERSION}] - -` - // VerifyMigrations will panic if the installation is not properly migrated func VerifyMigrations() { - // Allow status and config routes to aide user in migrating... - isStatus := false - isConfigPaths := false - for _, arg := range os.Args { - if arg == "status" { - isStatus = true - } else if arg == "config" { - isConfigPaths = true - } else if arg == "help" { - return - } else if isStatus { - isStatus = false // only chifra status with no options is okay - } else if isConfigPaths && arg != "--paths" { - isConfigPaths = false // only chifra config --paths is okay - } - } - if isStatus || isConfigPaths { + if isStatusOrConfig() { + // Allow certain status and config routes to pass so as to aide user in migrating... return } user, _ := user.Current() + const doesNotExist string = `0002. A config item ({0}) is missing. See {https://trueblocks.io/docs/install/install-core/}.` + const shouldNotExist string = `0002. A config item ({0}) exists but should not. See {https://trueblocks.io/docs/install/install-core/}.` + const noChains string = `0003. The configuration file ({0}) contains no chain specifications. See {https://trueblocks.io/docs/install/install-core/}.` + // The old $HOME/.quickBlocks folder should not exist... if _, err := os.Stat(filepath.Join(user.HomeDir, ".quickBlocks")); err == nil { msg := strings.Replace(shouldNotExist, "{0}", "{~/.quickBlocks}", -1) - msg = strings.Replace(msg, "[{VERSION}]", versionText, -1) - msg = strings.Replace(msg, "{1}", "{v0.09.0}", -1) - msg = strings.Replace(msg, "{", colors.Green, -1) - msg = strings.Replace(msg, "}", colors.Off, -1) + msg = colors.ColoredWith(msg, colors.Yellow) logger.Fatal(msg) } // Both the config folder... configFolder := config.PathToRootConfig() if _, err := os.Stat(configFolder); err != nil { - msg := strings.Replace(notExist, "{0}", "{"+configFolder+"}", -1) - msg = strings.Replace(msg, "[{VERSION}]", versionText, -1) - msg = strings.Replace(msg, "{", colors.Green, -1) - msg = strings.Replace(msg, "}", colors.Off, -1) + msg := strings.Replace(doesNotExist, "{0}", "{"+configFolder+"}", -1) + msg = colors.ColoredWith(msg, colors.Yellow) logger.Fatal(msg) } // ...and the config file better exist. configFile := filepath.Join(configFolder + "trueBlocks.toml") if _, err := os.Stat(configFile); err != nil { - msg := strings.Replace(notExist, "{0}", "{"+configFile+"}", -1) - msg = strings.Replace(msg, "[{VERSION}]", versionText, -1) - msg = strings.Replace(msg, "{", colors.Green, -1) - msg = strings.Replace(msg, "}", colors.Off, -1) + msg := strings.Replace(doesNotExist, "{0}", "{"+configFile+"}", -1) + msg = colors.ColoredWith(msg, colors.Yellow) logger.Fatal(msg) } // ...and some chains... - if !config.HasChains() { + chainArray := config.GetChains() + if len(chainArray) == 0 { msg := strings.Replace(noChains, "{0}", "{"+configFile+"}", -1) - msg = strings.Replace(msg, "{1}", "{v0.25.0}", -1) - msg = strings.Replace(msg, "[{VERSION}]", versionText, -1) - msg = strings.Replace(msg, "{", colors.Green, -1) - msg = strings.Replace(msg, "}", colors.Off, -1) + msg = colors.ColoredWith(msg, colors.Yellow) logger.Fatal(msg) } // We need to find the chain configuration path chainConfigPath := config.MustGetPathToChainConfig("") if _, err := os.Stat(chainConfigPath); err != nil { - msg := strings.Replace(notExist, "{0}", "{"+chainConfigPath+"}", -1) - msg = strings.Replace(msg, "[{VERSION}]", versionText, -1) - msg = strings.Replace(msg, "{", colors.Green, -1) - msg = strings.Replace(msg, "}", colors.Off, -1) + msg := strings.Replace(doesNotExist, "{0}", "{"+chainConfigPath+"}", -1) + msg = colors.ColoredWith(msg, colors.Yellow) logger.Fatal(msg) } @@ -174,21 +112,46 @@ func VerifyMigrations() { itemPath := filepath.Join(config.PathToCache(""), item) if _, err := os.Stat(itemPath); err == nil { msg := strings.Replace(shouldNotExist, "{0}", "{"+itemPath+"}", -1) - msg = strings.Replace(msg, "[{VERSION}]", versionText, -1) - msg = strings.Replace(msg, "{1}", "{v1.0.0}", -1) - msg = strings.Replace(msg, "{", colors.Green, -1) - msg = strings.Replace(msg, "}", colors.Off, -1) + msg = colors.ColoredWith(msg, colors.Yellow) logger.Fatal(msg) } } - // We need at least this version... - requiredVersion := "v0.40.0-beta" - if !config.IsAtLeastVersion(requiredVersion) { - msg := strings.Replace(backVersion, "{0}", "{"+requiredVersion+"}", -1) - msg = strings.Replace(msg, "[{VERSION}]", versionText, -1) - msg = strings.Replace(msg, "{", colors.Green, -1) - msg = strings.Replace(msg, "}", colors.Off, -1) - logger.Fatal(msg) + requiredVer := version.NewVersion("v1.0.0-release") + currentVer := version.NewVersion(config.GetVersion().Current) + if currentVer.Uint64() < requiredVer.Uint64() { + _ = upgrade.UpgradeConfigs(requiredVer) // does not return } } + +func isStatusOrConfig() bool { + isStatus := false + isConfig := false + hasPaths := false + hasEdit := false + cnt := len(os.Args) + for _, arg := range os.Args { + if arg == "help" { + return false + } else if arg == "status" { + isStatus = true + } else if arg == "config" { + isConfig = true + } else if arg == "--paths" { + hasPaths = true + } else if arg == "edit" { + hasEdit = true + } else if arg != "--verbose" { + isStatus = false + isConfig = false + } else { + cnt-- // allow --verbose + } + } + + if isStatus && cnt == 2 { + return true + } + + return isConfig && (hasPaths || hasEdit) && cnt < 4 +} diff --git a/src/apps/chifra/cmd/root_initialize_test.go b/src/apps/chifra/cmd/root_initialize_test.go deleted file mode 100644 index 64c4e0bead..0000000000 --- a/src/apps/chifra/cmd/root_initialize_test.go +++ /dev/null @@ -1,17 +0,0 @@ -// Copyright 2021 The TrueBlocks Authors. All rights reserved. -// Use of this source code is governed by a license that can -// be found in the LICENSE file. - -package cmd - -//import ( -// "testing" -//) - -//func TestInitialize(t *testing.T) { -// Initialize() -//} - -// func TestMigrations(t *testing.T) { -// VerifyMigrations() -// } diff --git a/src/apps/chifra/cmd/scrape.go b/src/apps/chifra/cmd/scrape.go index 204becc965..d60e3d20e1 100644 --- a/src/apps/chifra/cmd/scrape.go +++ b/src/apps/chifra/cmd/scrape.go @@ -62,13 +62,19 @@ func init() { scrapeCmd.Flags().BoolVarP(&scrapePkg.GetOptions().Remote, "remote", "r", false, "pin new chunks to the gateway (requires pinning service keys)") scrapeCmd.Flags().Float64VarP(&scrapePkg.GetOptions().Sleep, "sleep", "s", 14, "seconds to sleep between scraper passes") scrapeCmd.Flags().Uint64VarP(&scrapePkg.GetOptions().StartBlock, "start_block", "l", 0, "first block to visit when scraping (snapped back to most recent snap_to_grid mark)") - scrapeCmd.Flags().Uint64VarP(&scrapePkg.GetOptions().Settings.Apps_per_chunk, "apps_per_chunk", "", 200000, "the number of appearances to build into a chunk before consolidating it (hidden)") - scrapeCmd.Flags().Uint64VarP(&scrapePkg.GetOptions().Settings.Snap_to_grid, "snap_to_grid", "", 100000, "an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden)") - scrapeCmd.Flags().Uint64VarP(&scrapePkg.GetOptions().Settings.First_snap, "first_snap", "", 0, "the first block at which snap_to_grid is enabled (hidden)") - scrapeCmd.Flags().Uint64VarP(&scrapePkg.GetOptions().Settings.Unripe_dist, "unripe_dist", "", 28, "the distance (in blocks) from the front of the chain under which (inclusive) a block is considered unripe (hidden)") - scrapeCmd.Flags().Uint64VarP(&scrapePkg.GetOptions().Settings.Channel_count, "channel_count", "", 20, "number of concurrent processing channels (hidden)") - scrapeCmd.Flags().BoolVarP(&scrapePkg.GetOptions().Settings.Allow_missing, "allow_missing", "", false, "do not report errors for blockchains that contain blocks with zero addresses (hidden)") + scrapeCmd.Flags().Uint64VarP(&scrapePkg.GetOptions().RunCount, "run_count", "u", 0, "run the scraper this many times, then quit (hidden)") + scrapeCmd.Flags().StringVarP(&scrapePkg.GetOptions().Publisher, "publisher", "P", "trueblocks.eth", "for some query options, the publisher of the index (hidden)") + scrapeCmd.Flags().BoolVarP(&scrapePkg.GetOptions().DryRun, "dry_run", "d", false, "show the configuration that would be applied if run,no changes are made (hidden)") + scrapeCmd.Flags().Uint64VarP(&scrapePkg.GetOptions().Settings.AppsPerChunk, "apps_per_chunk", "", 200000, "the number of appearances to build into a chunk before consolidating it (hidden)") + scrapeCmd.Flags().Uint64VarP(&scrapePkg.GetOptions().Settings.SnapToGrid, "snap_to_grid", "", 100000, "an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden)") + scrapeCmd.Flags().Uint64VarP(&scrapePkg.GetOptions().Settings.FirstSnap, "first_snap", "", 0, "the first block at which snap_to_grid is enabled (hidden)") + scrapeCmd.Flags().Uint64VarP(&scrapePkg.GetOptions().Settings.UnripeDist, "unripe_dist", "", 28, "the distance (in blocks) from the front of the chain under which (inclusive) a block is considered unripe (hidden)") + scrapeCmd.Flags().Uint64VarP(&scrapePkg.GetOptions().Settings.ChannelCount, "channel_count", "", 20, "number of concurrent processing channels (hidden)") + scrapeCmd.Flags().BoolVarP(&scrapePkg.GetOptions().Settings.AllowMissing, "allow_missing", "", false, "do not report errors for blockchains that contain blocks with zero addresses (hidden)") if os.Getenv("TEST_MODE") != "true" { + scrapeCmd.Flags().MarkHidden("run_count") + scrapeCmd.Flags().MarkHidden("publisher") + scrapeCmd.Flags().MarkHidden("dry_run") scrapeCmd.Flags().MarkHidden("apps_per_chunk") scrapeCmd.Flags().MarkHidden("snap_to_grid") scrapeCmd.Flags().MarkHidden("first_snap") diff --git a/src/apps/chifra/internal/blocks/handle_show.go b/src/apps/chifra/internal/blocks/handle_show.go index 890494771d..16b17cd675 100644 --- a/src/apps/chifra/internal/blocks/handle_show.go +++ b/src/apps/chifra/internal/blocks/handle_show.go @@ -83,7 +83,6 @@ func (opts *BlocksOptions) HandleShow() error { extra := map[string]interface{}{ "hashes": opts.Hashes, - "count": opts.Count, "uncles": opts.Uncles, "articulate": opts.Articulate, } diff --git a/src/apps/chifra/internal/blocks/handle_show_hashes.go b/src/apps/chifra/internal/blocks/handle_show_hashes.go index 37a88ba728..820f5f442c 100644 --- a/src/apps/chifra/internal/blocks/handle_show_hashes.go +++ b/src/apps/chifra/internal/blocks/handle_show_hashes.go @@ -83,7 +83,6 @@ func (opts *BlocksOptions) HandleHashes() error { extra := map[string]interface{}{ "hashes": opts.Hashes, - "count": opts.Count, "uncles": opts.Uncles, "articulate": opts.Articulate, } diff --git a/src/apps/chifra/internal/blocks/handle_traces.go b/src/apps/chifra/internal/blocks/handle_traces.go index 18dc0f16eb..7130d82ea5 100644 --- a/src/apps/chifra/internal/blocks/handle_traces.go +++ b/src/apps/chifra/internal/blocks/handle_traces.go @@ -51,7 +51,6 @@ func (opts *BlocksOptions) HandleTraces() error { } extra := map[string]interface{}{ - "count": opts.Count, "uncles": opts.Uncles, "logs": opts.Logs, "traces": opts.Traces, diff --git a/src/apps/chifra/internal/blocks/handle_uncles.go b/src/apps/chifra/internal/blocks/handle_uncles.go index dd9b29d804..636e1f88ff 100644 --- a/src/apps/chifra/internal/blocks/handle_uncles.go +++ b/src/apps/chifra/internal/blocks/handle_uncles.go @@ -85,7 +85,6 @@ func (opts *BlocksOptions) HandleUncles() error { extra := map[string]interface{}{ "hashes": opts.Hashes, - "count": opts.Count, "uncles": opts.Uncles, "articulate": opts.Articulate, } diff --git a/src/apps/chifra/internal/chunks/README.md b/src/apps/chifra/internal/chunks/README.md index ca8daf4f7b..498cd63bd5 100644 --- a/src/apps/chifra/internal/chunks/README.md +++ b/src/apps/chifra/internal/chunks/README.md @@ -43,6 +43,7 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. ``` Data models produced by this tool: diff --git a/src/apps/chifra/internal/chunks/handle_check.go b/src/apps/chifra/internal/chunks/handle_check.go index 23c1cbc1f8..b8f63c8321 100644 --- a/src/apps/chifra/internal/chunks/handle_check.go +++ b/src/apps/chifra/internal/chunks/handle_check.go @@ -12,7 +12,6 @@ import ( "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config/scrapeCfg" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/file" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/index" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" @@ -71,12 +70,12 @@ func (opts *ChunksOptions) HandleCheck(blockNums []uint64) error { return fileNames[i] < fileNames[j] }) - cacheManifest, err := manifest.ReadManifest(chain, manifest.FromCache) + cacheManifest, err := manifest.ReadManifest(chain, opts.PublisherAddr, manifest.FromCache) if err != nil { return err } - remoteManifest, err := manifest.ReadManifest(chain, manifest.FromContract) + remoteManifest, err := manifest.ReadManifest(chain, opts.PublisherAddr, manifest.FromContract) if err != nil { return err } @@ -111,7 +110,8 @@ func (opts *ChunksOptions) HandleCheck(blockNums []uint64) error { reports := []simpleReportCheck{} - allowMissing := scrapeCfg.AllowMissing(chain) + allowMissing := config.GetScrape(chain).AllowMissing + seq := simpleReportCheck{Reason: "Filenames sequential"} if err := opts.CheckSequential(fileNames, cacheArray, remoteArray, allowMissing, &seq); err != nil { return err diff --git a/src/apps/chifra/internal/chunks/handle_diff.go b/src/apps/chifra/internal/chunks/handle_diff.go new file mode 100644 index 0000000000..f79c73d7e0 --- /dev/null +++ b/src/apps/chifra/internal/chunks/handle_diff.go @@ -0,0 +1,116 @@ +package chunksPkg + +import ( + "context" + "fmt" + "io" + "os" + "path/filepath" + "sort" + "strings" + + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/file" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/index" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/output" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/types" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/walk" +) + +func (opts *ChunksOptions) HandleDiff(blockNums []uint64) error { + chain := opts.Globals.Chain + + ctx, cancel := context.WithCancel(context.Background()) + fetchData := func(modelChan chan types.Modeler[types.RawModeler], errorChan chan error) { + showDiff := func(walker *walk.CacheWalker, path string, first bool) (bool, error) { + return opts.handleDiff(modelChan, walker, path, first) + } + + walker := walk.NewCacheWalker( + chain, + opts.Globals.TestMode, + 10000, /* maxTests */ + showDiff, + ) + + if err := walker.WalkBloomFilters(blockNums); err != nil { + errorChan <- err + cancel() + } + } + + return output.StreamMany(ctx, fetchData, opts.Globals.OutputOpts()) +} + +func (opts *ChunksOptions) handleDiff(modelChan chan types.Modeler[types.RawModeler], walker *walk.CacheWalker, path string, first bool) (bool, error) { + path = index.ToIndexPath(path) + folder, name := filepath.Split(path) + diffPath := os.Getenv("TB_CHUNKS_DIFFPATH") + diffPath = filepath.Join(strings.Replace(folder, config.PathToIndex(opts.Globals.Chain), diffPath+"/", -1), name) + if len(diffPath) > 0 && diffPath[0] != '/' { + diffPath = "./" + diffPath + } + wd, _ := os.Getwd() + diffPath = strings.Replace(diffPath, "./", wd+"/", -1) + + if !file.FileExists(diffPath) { + logger.Fatal(fmt.Sprintf("The diff path does not exist: %s", diffPath)) + } + + logger.Info("Comparing:") + logger.Info(fmt.Sprintf(" existing: %s (%d)", path, file.FileSize(path))) + logger.Info(fmt.Sprintf(" current: %s (%d)", diffPath, file.FileSize(diffPath))) + + _, _ = opts.exportTo("one", path) + _, _ = opts.exportTo("two", diffPath) + return true, nil +} + +func (opts *ChunksOptions) exportTo(dest, source string) (bool, error) { + indexChunk, err := index.NewChunkData(source) + if err != nil { + return false, err + } + defer indexChunk.Close() + + _, err = indexChunk.File.Seek(int64(index.HeaderWidth), io.SeekStart) + if err != nil { + return false, err + } + + apps := make([]types.SimpleAppearance, 0, 500000) + for i := 0; i < int(indexChunk.Header.AddressCount); i++ { + s := simpleAppearanceTable{} + err := s.AddressRecord.ReadAddress(indexChunk.File) + if err != nil { + return false, err + } + if s.Appearances, err = indexChunk.ReadAppearanceRecordsAndResetOffset(&s.AddressRecord); err != nil { + return false, err + } + for _, app := range s.Appearances { + apps = append(apps, types.SimpleAppearance{ + Address: s.AddressRecord.Address, + BlockNumber: app.BlockNumber, + TransactionIndex: app.TransactionId, + }) + } + } + + sort.Slice(apps, func(i, j int) bool { + if apps[i].BlockNumber == apps[j].BlockNumber { + if apps[i].TransactionIndex == apps[j].TransactionIndex { + return apps[i].Address.Hex() < apps[j].Address.Hex() + } + return apps[i].TransactionIndex < apps[j].TransactionIndex + } + return apps[i].BlockNumber < apps[j].BlockNumber + }) + + for _, app := range apps { + fmt.Printf("%s\t%d\t%d\t%s\n", dest, app.BlockNumber, app.TransactionIndex, app.Address) + } + + return true, nil +} diff --git a/src/apps/chifra/internal/chunks/handle_manifest.go b/src/apps/chifra/internal/chunks/handle_manifest.go index 279ff2ed87..49a2343476 100644 --- a/src/apps/chifra/internal/chunks/handle_manifest.go +++ b/src/apps/chifra/internal/chunks/handle_manifest.go @@ -20,7 +20,7 @@ var sourceMap = map[bool]manifest.Source{ func (opts *ChunksOptions) HandleManifest(blockNums []uint64) error { chain := opts.Globals.Chain testMode := opts.Globals.TestMode - man, err := manifest.ReadManifest(chain, sourceMap[opts.Remote]) + man, err := manifest.ReadManifest(chain, opts.PublisherAddr, sourceMap[opts.Remote]) if err != nil { return err } diff --git a/src/apps/chifra/internal/chunks/handle_pin.go b/src/apps/chifra/internal/chunks/handle_pin.go index 1697f0c6f4..5d175d1ec2 100644 --- a/src/apps/chifra/internal/chunks/handle_pin.go +++ b/src/apps/chifra/internal/chunks/handle_pin.go @@ -21,12 +21,12 @@ import ( ) func (opts *ChunksOptions) HandlePin(blockNums []uint64) error { - firstBlock := mustParseUint(os.Getenv("TB_CHUNK_PIN_FIRST_BLOCK")) + firstBlock := mustParseUint(os.Getenv("TB_CHUNKS_PINFIRSTBLOCK")) chain := opts.Globals.Chain ctx, cancel := context.WithCancel(context.Background()) fetchData := func(modelChan chan types.Modeler[types.RawModeler], errorChan chan error) { - man := simpleChunkPinReport{ + report := simpleChunkPinReport{ Version: version.ManifestVersion, Chain: chain, Schemas: unchained.Schemas, @@ -35,14 +35,14 @@ func (opts *ChunksOptions) HandlePin(blockNums []uint64) error { if len(blockNums) == 0 { var err error tsPath := config.PathToIndex(chain) + "ts.bin" - if man.TsHash, err = pinning.PinItem(chain, "timestamps", tsPath, opts.Remote); err != nil { + if report.TsHash, err = pinning.PinItem(chain, "timestamps", tsPath, opts.Remote); err != nil { errorChan <- err cancel() return } manPath := filepath.Join(config.MustGetPathToChainConfig(chain), "manifest.json") - if man.ManifestHash, err = pinning.PinItem(chain, "manifest", manPath, opts.Remote); err != nil { + if report.ManifestHash, err = pinning.PinItem(chain, "manifest", manPath, opts.Remote); err != nil { errorChan <- err cancel() return @@ -66,17 +66,19 @@ func (opts *ChunksOptions) HandlePin(blockNums []uint64) error { result, err := pinning.PinChunk(chain, index.ToBloomPath(path), index.ToIndexPath(path), opts.Remote) if err != nil { - return false, err + errorChan <- err + cancel() // keep going... + return true, nil } if pinning.LocalDaemonRunning() { - man.Pinned = append(man.Pinned, result.Local.BloomHash) - man.Pinned = append(man.Pinned, result.Local.IndexHash) + report.Pinned = append(report.Pinned, result.Local.BloomHash) + report.Pinned = append(report.Pinned, result.Local.IndexHash) } if opts.Remote { - man.Pinned = append(man.Pinned, result.Remote.BloomHash) - man.Pinned = append(man.Pinned, result.Remote.IndexHash) + report.Pinned = append(report.Pinned, result.Remote.BloomHash) + report.Pinned = append(report.Pinned, result.Remote.IndexHash) } if !result.Matches { @@ -87,6 +89,13 @@ func (opts *ChunksOptions) HandlePin(blockNums []uint64) error { } else if opts.Remote && pinning.LocalDaemonRunning() { logger.Info(colors.BrightGreen+"Matches: "+result.Remote.BloomHash.String(), "-", result.Remote.IndexHash, colors.Off) } + if opts.Globals.Verbose { + if opts.Remote { + fmt.Println("result.Remote:", result.Remote.String()) + } else { + fmt.Println("result.Local:", result.Local.String()) + } + } sleep := opts.Sleep if sleep > 0 { @@ -114,7 +123,7 @@ func (opts *ChunksOptions) HandlePin(blockNums []uint64) error { } } - modelChan <- &man + modelChan <- &report } return output.StreamMany(ctx, fetchData, opts.Globals.OutputOpts()) diff --git a/src/apps/chifra/internal/chunks/handle_truncate.go b/src/apps/chifra/internal/chunks/handle_truncate.go index 3e7873e50b..57f682a733 100644 --- a/src/apps/chifra/internal/chunks/handle_truncate.go +++ b/src/apps/chifra/internal/chunks/handle_truncate.go @@ -22,6 +22,8 @@ import ( "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/walk" ) +// TODO: We need to make sure, when we truncate, that we truncate the corresponding maps files as well. + func (opts *ChunksOptions) HandleTruncate(blockNums []uint64) error { chain := opts.Globals.Chain @@ -34,8 +36,7 @@ func (opts *ChunksOptions) HandleTruncate(blockNums []uint64) error { return nil } - indexPath := config.PathToIndex(chain) - _ = index.CleanTemporaryFolders(indexPath, true) + _ = index.CleanTempIndexFolders(chain, []string{"ripe", "unripe", "staging"}) ctx, cancel := context.WithCancel(context.Background()) fetchData := func(modelChan chan types.Modeler[types.RawModeler], errorChan chan error) { @@ -63,7 +64,7 @@ func (opts *ChunksOptions) HandleTruncate(blockNums []uint64) error { testRange := base.FileRange{First: opts.Truncate, Last: utils.NOPOS} if rng.Intersects(testRange) { - if err = manifest.RemoveChunk(chain, index.ToBloomPath(path), index.ToIndexPath(path)); err != nil { + if err = manifest.RemoveChunk(chain, opts.PublisherAddr, index.ToBloomPath(path), index.ToIndexPath(path)); err != nil { return false, err } nChunksRemoved++ diff --git a/src/apps/chifra/internal/chunks/options.go b/src/apps/chifra/internal/chunks/options.go index 001ebc0a3d..c68e750133 100644 --- a/src/apps/chifra/internal/chunks/options.go +++ b/src/apps/chifra/internal/chunks/options.go @@ -30,9 +30,11 @@ type ChunksOptions struct { Check bool `json:"check,omitempty"` // Check the manifest, index, or blooms for internal consistency Pin bool `json:"pin,omitempty"` // Pin the manifest or each index chunk and bloom Publish bool `json:"publish,omitempty"` // Publish the manifest to the Unchained Index smart contract + Publisher string `json:"publisher,omitempty"` // For some query options, the publisher of the index Truncate uint64 `json:"truncate,omitempty"` // Truncate the entire index at this block (requires a block identifier) Remote bool `json:"remote,omitempty"` // Prior to processing, retreive the manifest from the Unchained Index smart contract Belongs []string `json:"belongs,omitempty"` // In index mode only, checks the address(es) for inclusion in the given index chunk + Diff bool `json:"diff,omitempty"` // Compare two index portions (see notes) FirstBlock uint64 `json:"firstBlock,omitempty"` // First block to process (inclusive) LastBlock uint64 `json:"lastBlock,omitempty"` // Last block to process (inclusive) MaxAddrs uint64 `json:"maxAddrs,omitempty"` // The max number of addresses to process in a given chunk @@ -42,10 +44,12 @@ type ChunksOptions struct { Conn *rpc.Connection `json:"conn,omitempty"` // The connection to the RPC server BadFlag error `json:"badFlag,omitempty"` // An error flag if needed // EXISTING_CODE + PublisherAddr base.Address // EXISTING_CODE } var defaultChunksOptions = ChunksOptions{ + Publisher: "trueblocks.eth", Truncate: utils.NOPOS, LastBlock: utils.NOPOS, MaxAddrs: utils.NOPOS, @@ -58,9 +62,11 @@ func (opts *ChunksOptions) testLog() { logger.TestLog(opts.Check, "Check: ", opts.Check) logger.TestLog(opts.Pin, "Pin: ", opts.Pin) logger.TestLog(opts.Publish, "Publish: ", opts.Publish) + logger.TestLog(len(opts.Publisher) > 0 && opts.Publisher != "trueblocks.eth", "Publisher: ", opts.Publisher) logger.TestLog(opts.Truncate != utils.NOPOS, "Truncate: ", opts.Truncate) logger.TestLog(opts.Remote, "Remote: ", opts.Remote) logger.TestLog(len(opts.Belongs) > 0, "Belongs: ", opts.Belongs) + logger.TestLog(opts.Diff, "Diff: ", opts.Diff) logger.TestLog(opts.FirstBlock != 0, "FirstBlock: ", opts.FirstBlock) logger.TestLog(opts.LastBlock != 0 && opts.LastBlock != utils.NOPOS, "LastBlock: ", opts.LastBlock) logger.TestLog(opts.MaxAddrs != utils.NOPOS, "MaxAddrs: ", opts.MaxAddrs) @@ -100,6 +106,8 @@ func chunksFinishParseApi(w http.ResponseWriter, r *http.Request) *ChunksOptions opts.Pin = true case "publish": opts.Publish = true + case "publisher": + opts.Publisher = value[0] case "truncate": opts.Truncate = globals.ToUint64(value[0]) case "remote": @@ -109,6 +117,8 @@ func chunksFinishParseApi(w http.ResponseWriter, r *http.Request) *ChunksOptions s := strings.Split(val, " ") // may contain space separated items opts.Belongs = append(opts.Belongs, s...) } + case "diff": + opts.Diff = true case "firstBlock": opts.FirstBlock = globals.ToUint64(value[0]) case "lastBlock": @@ -126,8 +136,10 @@ func chunksFinishParseApi(w http.ResponseWriter, r *http.Request) *ChunksOptions } } opts.Conn = opts.Globals.FinishParseApi(w, r, opts.getCaches()) + opts.Publisher, _ = opts.Conn.GetEnsAddress(opts.Publisher) // EXISTING_CODE + opts.PublisherAddr = base.HexToAddress(opts.Publisher) // EXISTING_CODE opts.Belongs, _ = opts.Conn.GetEnsAddresses(opts.Belongs) @@ -152,8 +164,10 @@ func chunksFinishParse(args []string) *ChunksOptions { defFmt := "txt" opts := GetOptions() opts.Conn = opts.Globals.FinishParse(args, opts.getCaches()) + opts.Publisher, _ = opts.Conn.GetEnsAddress(opts.Publisher) // EXISTING_CODE + opts.PublisherAddr = base.HexToAddress(opts.Publisher) if len(args) > 0 { opts.Mode = args[0] for i, arg := range args { diff --git a/src/apps/chifra/internal/chunks/output.go b/src/apps/chifra/internal/chunks/output.go index ffeacee48a..12d323685c 100644 --- a/src/apps/chifra/internal/chunks/output.go +++ b/src/apps/chifra/internal/chunks/output.go @@ -72,7 +72,10 @@ func (opts *ChunksOptions) ChunksInternal() (err error, handled bool) { blockNums = blockNums[:200] } - if opts.Pin { + if opts.Diff { + err = opts.HandleDiff(blockNums) + + } else if opts.Pin { err = opts.HandlePin(blockNums) } else if opts.Publish { diff --git a/src/apps/chifra/internal/chunks/validate.go b/src/apps/chifra/internal/chunks/validate.go index c6c2599122..dbff840f28 100644 --- a/src/apps/chifra/internal/chunks/validate.go +++ b/src/apps/chifra/internal/chunks/validate.go @@ -7,8 +7,10 @@ package chunksPkg import ( "errors" "fmt" + "os" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/file" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/index" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/pinning" @@ -38,6 +40,19 @@ func (opts *ChunksOptions) validateChunks() error { return err } + if opts.Diff { + if opts.Mode != "index" { + return validate.Usage("The {0} option is only available in {1} mode.", "--diff", "index") + } + path := os.Getenv("TB_CHUNKS_DIFFPATH") + if path == "" { + return validate.Usage("The {0} option requires {1}.", "--diff", "TB_CHUNKS_DIFFPATH to be set") + } + if !file.FolderExists(path) { + return fmt.Errorf("the path TB_CHUNKS_DIFFPATH=%s does not exist", path) + } + } + isIndexOrManifest := opts.Mode == "index" || opts.Mode == "manifest" if !isIndexOrManifest { if err = opts.isDisallowed(!isIndexOrManifest /* i.e., true */, opts.Mode); err != nil { @@ -52,7 +67,7 @@ func (opts *ChunksOptions) validateChunks() error { if opts.Mode == "manifest" { if opts.Pin { if opts.Remote { - pinataKey, pinataSecret := config.GetPinningKeys(chain) + pinataKey, pinataSecret := config.GetKey("pinata").ApiKey, config.GetKey("pinata").Secret if pinataKey == "" || pinataSecret == "" { return validate.Usage("The {0} option requires {1}.", "--pin --remote", "an api key") } @@ -76,8 +91,8 @@ func (opts *ChunksOptions) validateChunks() error { if opts.Mode == "index" { // do nothing } else if opts.Mode == "manifest" { - if !pinning.LocalDaemonRunning() { - return validate.Usage("The {0} option requires {1}.", "manifest --deep", "a locally running IPFS daemon") + if !opts.Remote && !pinning.LocalDaemonRunning() { + return validate.Usage("The {0} option requires {1}.", "manifest --deep", "a locally running IPFS daemon or --remote") } } else { return validate.Usage("The {0} option requires mode {1}.", "--deep", "index or manifest") @@ -118,6 +133,13 @@ func (opts *ChunksOptions) validateChunks() error { return err } + if len(opts.Publisher) > 0 { + err := validate.ValidateExactlyOneAddr([]string{opts.Publisher}) + if err != nil { + return err + } + } + err = validate.ValidateIdentifiers( chain, opts.Blocks, @@ -135,6 +157,10 @@ func (opts *ChunksOptions) validateChunks() error { return err } + if opts.Diff && len(opts.BlockIds) != 1 { + return validate.Usage("The {0} option requires exactly one block identifier.", "--diff") + } + if opts.FirstBlock != 0 || opts.LastBlock != utils.NOPOS || opts.MaxAddrs != utils.NOPOS { if opts.FirstBlock >= opts.LastBlock { msg := fmt.Sprintf("first_block (%d) must be strictly earlier than last_block (%d).", opts.FirstBlock, opts.LastBlock) diff --git a/src/apps/chifra/internal/config/handle_show.go b/src/apps/chifra/internal/config/handle_show.go new file mode 100644 index 0000000000..566f9c8695 --- /dev/null +++ b/src/apps/chifra/internal/config/handle_show.go @@ -0,0 +1,14 @@ +package configPkg + +import ( + "fmt" + + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/file" +) + +func (opts *ConfigOptions) HandleShow() error { + fn := config.PathToRootConfig() + "trueBlocks.toml" + fmt.Printf("%s", file.AsciiFileToString(fn)) + return nil +} diff --git a/src/apps/chifra/internal/config/output.go b/src/apps/chifra/internal/config/output.go index ef932a8e62..0e0516f0f6 100644 --- a/src/apps/chifra/internal/config/output.go +++ b/src/apps/chifra/internal/config/output.go @@ -62,8 +62,8 @@ func (opts *ConfigOptions) ConfigInternal() (err error, handled bool) { err = opts.HandlePaths() } else if opts.Mode == "edit" { err = opts.HandleEdit() - } else { - logger.Warn("The config tool is current unavailable. Please use the 'chifra status' tool instead.") + } else if opts.Mode == "show" { + err = opts.HandleShow() } // EXISTING_CODE timer.Report(msg) diff --git a/src/apps/chifra/internal/config/validate.go b/src/apps/chifra/internal/config/validate.go index 2361beffdc..75cb1f6759 100644 --- a/src/apps/chifra/internal/config/validate.go +++ b/src/apps/chifra/internal/config/validate.go @@ -24,8 +24,7 @@ func (opts *ConfigOptions) validateConfig() error { return validate.Usage("chain {0} is not properly configured.", chain) } - err := validate.ValidateEnum("modes", opts.Mode, "[show|edit]") - if err != nil { + if err := validate.ValidateEnum("modes", opts.Mode, "[show|edit]"); err != nil { return err } @@ -33,5 +32,9 @@ func (opts *ConfigOptions) validateConfig() error { return validate.Usage("You must set the EDITOR environment variable to use the 'edit' mode.") } + // if len(opts.Mode) == 0 || !opts.Paths { + // return validate.Usage("You must supply either {0} or {1}.", "a mode", "--paths") + // } + return opts.Globals.Validate() } diff --git a/src/apps/chifra/internal/daemon/options.go b/src/apps/chifra/internal/daemon/options.go index d0bfbdd1a1..d13a988cbc 100644 --- a/src/apps/chifra/internal/daemon/options.go +++ b/src/apps/chifra/internal/daemon/options.go @@ -39,8 +39,8 @@ var defaultDaemonOptions = DaemonOptions{ // testLog is used only during testing to export the options for this test case. func (opts *DaemonOptions) testLog() { - logger.TestLog(len(opts.Port) > 0, "Port: ", opts.Port) - logger.TestLog(len(opts.Api) > 0, "Api: ", opts.Api) + logger.TestLog(len(opts.Port) > 0 && opts.Port != ":8080", "Port: ", opts.Port) + logger.TestLog(len(opts.Api) > 0 && opts.Api != "on", "Api: ", opts.Api) logger.TestLog(len(opts.Scrape) > 0, "Scrape: ", opts.Scrape) logger.TestLog(opts.Monitor, "Monitor: ", opts.Monitor) logger.TestLog(opts.Grpc, "Grpc: ", opts.Grpc) diff --git a/src/apps/chifra/internal/daemon/output.go b/src/apps/chifra/internal/daemon/output.go index 99683f3515..87f885e1bb 100644 --- a/src/apps/chifra/internal/daemon/output.go +++ b/src/apps/chifra/internal/daemon/output.go @@ -70,7 +70,7 @@ func (opts *DaemonOptions) DaemonInternal() (err error, handled bool) { } chain := opts.Globals.Chain - provider, _ := config.GetRpcProvider(chain) + provider := config.GetChain(chain).RpcProvider logger.InfoTable("Server URL: ", apiUrl) logger.InfoTable("RPC Provider: ", provider) diff --git a/src/apps/chifra/internal/explore/output.go b/src/apps/chifra/internal/explore/output.go index ee2d69349c..e1e6ade57e 100644 --- a/src/apps/chifra/internal/explore/output.go +++ b/src/apps/chifra/internal/explore/output.go @@ -112,7 +112,7 @@ func (u *ExploreUrl) getUrl(opts *ExploreOptions) string { return query } - url := config.GetRemoteExplorer(chain) + url := config.GetChain(chain).RemoteExplorer query := "" switch u.termType { case ExploreNone: @@ -128,7 +128,7 @@ func (u *ExploreUrl) getUrl(opts *ExploreOptions) string { } if opts.Local { - url = config.GetLocalExplorer(chain) + url = config.GetChain(chain).LocalExplorer query = strings.Replace(query, "tx/", "explorer/transactions/", -1) query = strings.Replace(query, "block/", "explorer/blocks/", -1) } diff --git a/src/apps/chifra/internal/export/handle_neighbors.go b/src/apps/chifra/internal/export/handle_neighbors.go index 660da7b519..c7be7b75bc 100644 --- a/src/apps/chifra/internal/export/handle_neighbors.go +++ b/src/apps/chifra/internal/export/handle_neighbors.go @@ -141,7 +141,7 @@ bool isSame(const CReverseAppMapEntry* a, const CReverseAppMapEntry* b) { //----------------------------------------------------------------------- bool assignReason(const CName& accountedFor, CAppearance& app, const CTransaction& trans) { - if (app.transactionIndex > 99996) { // leave this here for searching: 99999 + if (app.transactionIndex > types.ExternalReward) { // leave this here for searching: types.BlockReward app.reason = "miner"; return true; } diff --git a/src/apps/chifra/internal/export/validate.go b/src/apps/chifra/internal/export/validate.go index d71d23d947..2c06ae40ed 100644 --- a/src/apps/chifra/internal/export/validate.go +++ b/src/apps/chifra/internal/export/validate.go @@ -28,7 +28,7 @@ func (opts *ExportOptions) validateExport() error { return validate.Usage("chain {0} is not properly configured.", chain) } - key := config.GetRootConfig().Keys["trueblocks"].License + key := config.GetKey("trueblocks").License if opts.Neighbors && !strings.Contains(key, "+neighbors") { return validate.Usage("The {0} option requires a license key. Please contact us in our discord.", "--neighbors") } diff --git a/src/apps/chifra/internal/globals/options.go b/src/apps/chifra/internal/globals/options.go index 62a960a056..61e0d2aef6 100644 --- a/src/apps/chifra/internal/globals/options.go +++ b/src/apps/chifra/internal/globals/options.go @@ -15,6 +15,7 @@ import ( "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/output" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/rpc" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/tslib" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/unchained" "github.com/spf13/cobra" ) @@ -35,7 +36,7 @@ type GlobalOptions struct { func (opts *GlobalOptions) TestLog() { logger.TestLog(opts.Verbose, "Verbose: ", opts.Verbose) logger.TestLog(opts.NoHeader, "NoHeader: ", opts.NoHeader) - logger.TestLog(len(opts.Chain) > 0 && opts.Chain != config.GetDefaultChain(), "Chain: ", opts.Chain) + logger.TestLog(len(opts.Chain) > 0 && opts.Chain != config.GetSettings().DefaultChain, "Chain: ", opts.Chain) logger.TestLog(opts.Wei, "Wei: ", opts.Wei) logger.TestLog(opts.Ether, "Ether: ", opts.Ether) logger.TestLog(opts.Help, "Help: ", opts.Help) @@ -55,7 +56,7 @@ func (opts *GlobalOptions) TestLog() { func SetDefaults(opts *GlobalOptions) { if len(opts.Chain) == 0 { - opts.Chain = config.GetDefaultChain() + opts.Chain = config.GetSettings().DefaultChain } if opts.ShowRaw { @@ -192,12 +193,15 @@ func (opts *GlobalOptions) FinishParseApi(w http.ResponseWriter, r *http.Request } if len(opts.Chain) == 0 { - opts.Chain = config.GetDefaultChain() + opts.Chain = config.GetSettings().DefaultChain } if config.IsChainConfigured(opts.Chain) { - if err := tslib.EstablishTsFile(opts.Chain); err != nil { - logger.Error("Could not establish ts file:", err) + // TODO: #3219 Either this needs to be an option or PreferredPublisher needs to be configurable + // TODO: Why do we need to do this here? + publisher := unchained.GetPreferredPublisher() + if err := tslib.EstablishTsFile(opts.Chain, publisher); err != nil { + logger.Warn(err) } return rpc.NewConnection(opts.Chain, opts.Cache && !opts.ShowRaw, caches) } else { @@ -218,12 +222,15 @@ func (opts *GlobalOptions) FinishParse(args []string, caches map[string]bool) *r } if len(opts.Chain) == 0 { - opts.Chain = config.GetDefaultChain() + opts.Chain = config.GetSettings().DefaultChain } if config.IsChainConfigured(opts.Chain) { - if err := tslib.EstablishTsFile(opts.Chain); err != nil { - logger.Error("Could not establish ts file:", err) + // TODO: #3219 Either this needs to be an option or PreferredPublisher needs to be configurable + // TODO: Why do we need to do this here? + publisher := unchained.GetPreferredPublisher() + if err := tslib.EstablishTsFile(opts.Chain, publisher); err != nil { + logger.Warn(err) } return rpc.NewConnection(opts.Chain, opts.Cache && !opts.ShowRaw, caches) } else { diff --git a/src/apps/chifra/internal/init/handle_dryrun.go b/src/apps/chifra/internal/init/handle_dryrun.go index 8bc4ad9fcf..513b4f8690 100644 --- a/src/apps/chifra/internal/init/handle_dryrun.go +++ b/src/apps/chifra/internal/init/handle_dryrun.go @@ -16,7 +16,7 @@ import ( func (opts *InitOptions) HandleDryRun() error { chain := opts.Globals.Chain - remoteManifest, err := manifest.ReadManifest(chain, manifest.FromContract) + remoteManifest, err := manifest.ReadManifest(chain, opts.PublisherAddr, manifest.FromContract) if err != nil { return err } diff --git a/src/apps/chifra/internal/init/handle_init.go b/src/apps/chifra/internal/init/handle_init.go index 8209f9c27f..3cdc52f150 100644 --- a/src/apps/chifra/internal/init/handle_init.go +++ b/src/apps/chifra/internal/init/handle_init.go @@ -30,9 +30,9 @@ func (opts *InitOptions) HandleInit() error { // TODO: BOGUS - IF THE SCRAPER IS RUNNING, THIS WILL CAUSE PROBLEMS // Make sure that the temporary scraper folders are empty, so that, when the // scraper starts, it starts on the correct block. - _ = index.CleanTemporaryFolders(config.PathToIndex(chain), true) + _ = index.CleanTempIndexFolders(chain, []string{"ripe", "unripe", "staging"}) - remoteManifest, err := manifest.ReadManifest(chain, manifest.FromContract) + remoteManifest, err := manifest.ReadManifest(chain, opts.PublisherAddr, manifest.FromContract) if err != nil { return err } diff --git a/src/apps/chifra/internal/init/options.go b/src/apps/chifra/internal/init/options.go index bcb563f0d1..250c3e0d77 100644 --- a/src/apps/chifra/internal/init/options.go +++ b/src/apps/chifra/internal/init/options.go @@ -12,6 +12,7 @@ import ( "net/http" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/internal/globals" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/caps" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/rpc" @@ -22,21 +23,26 @@ import ( type InitOptions struct { All bool `json:"all,omitempty"` // In addition to Bloom filters, download full index chunks (recommended) DryRun bool `json:"dryRun,omitempty"` // Display the results of the download without actually downloading + Publisher string `json:"publisher,omitempty"` // The publisher of the index to download FirstBlock uint64 `json:"firstBlock,omitempty"` // Do not download any chunks earlier than this block Sleep float64 `json:"sleep,omitempty"` // Seconds to sleep between downloads Globals globals.GlobalOptions `json:"globals,omitempty"` // The global options Conn *rpc.Connection `json:"conn,omitempty"` // The connection to the RPC server BadFlag error `json:"badFlag,omitempty"` // An error flag if needed // EXISTING_CODE + PublisherAddr base.Address // EXISTING_CODE } -var defaultInitOptions = InitOptions{} +var defaultInitOptions = InitOptions{ + Publisher: "trueblocks.eth", +} // testLog is used only during testing to export the options for this test case. func (opts *InitOptions) testLog() { logger.TestLog(opts.All, "All: ", opts.All) logger.TestLog(opts.DryRun, "DryRun: ", opts.DryRun) + logger.TestLog(len(opts.Publisher) > 0 && opts.Publisher != "trueblocks.eth", "Publisher: ", opts.Publisher) logger.TestLog(opts.FirstBlock != 0, "FirstBlock: ", opts.FirstBlock) logger.TestLog(opts.Sleep != float64(0.0), "Sleep: ", opts.Sleep) opts.Conn.TestLog(opts.getCaches()) @@ -61,6 +67,8 @@ func initFinishParseApi(w http.ResponseWriter, r *http.Request) *InitOptions { opts.All = true case "dryRun": opts.DryRun = true + case "publisher": + opts.Publisher = value[0] case "firstBlock": opts.FirstBlock = globals.ToUint64(value[0]) case "sleep": @@ -72,8 +80,10 @@ func initFinishParseApi(w http.ResponseWriter, r *http.Request) *InitOptions { } } opts.Conn = opts.Globals.FinishParseApi(w, r, opts.getCaches()) + opts.Publisher, _ = opts.Conn.GetEnsAddress(opts.Publisher) // EXISTING_CODE + opts.PublisherAddr = base.HexToAddress(opts.Publisher) // EXISTING_CODE return opts @@ -97,8 +107,10 @@ func initFinishParse(args []string) *InitOptions { defFmt := "txt" opts := GetOptions() opts.Conn = opts.Globals.FinishParse(args, opts.getCaches()) + opts.Publisher, _ = opts.Conn.GetEnsAddress(opts.Publisher) // EXISTING_CODE + opts.PublisherAddr = base.HexToAddress(opts.Publisher) if len(args) > 0 { opts.BadFlag = validate.Usage("Invalid argument ({0}).", args[0]) } diff --git a/src/apps/chifra/internal/init/validate.go b/src/apps/chifra/internal/init/validate.go index 9fbb8dedfd..59c509cbc2 100644 --- a/src/apps/chifra/internal/init/validate.go +++ b/src/apps/chifra/internal/init/validate.go @@ -26,5 +26,12 @@ func (opts *InitOptions) validateInit() error { return validate.Usage("integration testing was skipped for chifra init") } + if len(opts.Publisher) > 0 { + err := validate.ValidateExactlyOneAddr([]string{opts.Publisher}) + if err != nil { + return err + } + } + return opts.Globals.Validate() } diff --git a/src/apps/chifra/internal/list/handle_freshen.go b/src/apps/chifra/internal/list/handle_freshen.go index cb70729064..68c576f5bd 100644 --- a/src/apps/chifra/internal/list/handle_freshen.go +++ b/src/apps/chifra/internal/list/handle_freshen.go @@ -271,7 +271,7 @@ func (updater *MonitorUpdate) visitChunkToFreshenFinal(fileName string, resultCh indexFilename := index.ToIndexPath(fileName) if !file.FileExists(indexFilename) { - _, err := index.EstablishIndexChunk(updater.Options.Globals.Chain, bl.Range) + _, err := index.EstablishIndexChunk(updater.Options.Globals.Chain, updater.Options.PublisherAddr, bl.Range) if err != nil { results = append(results, index.AppearanceResult{Range: bl.Range, Err: err}) return diff --git a/src/apps/chifra/internal/list/list_integration_test.go b/src/apps/chifra/internal/list/list_integration_test.go index 4610ee60d6..e82dfbb680 100644 --- a/src/apps/chifra/internal/list/list_integration_test.go +++ b/src/apps/chifra/internal/list/list_integration_test.go @@ -20,7 +20,7 @@ import ( ) func Test_HandleFreshenMonitors(t *testing.T) { - tslib.EstablishTsFile(utils.GetTestChain()) + tslib.EstablishTsFile(utils.GetTestChain(), base.GetTestPublisher()) opts := globals.GlobalOptions{} opts.Chain = "mainnet" listOpts := ListOptions{ diff --git a/src/apps/chifra/internal/list/options.go b/src/apps/chifra/internal/list/options.go index b4fe7f6c60..3643c06331 100644 --- a/src/apps/chifra/internal/list/options.go +++ b/src/apps/chifra/internal/list/options.go @@ -32,17 +32,20 @@ type ListOptions struct { FirstRecord uint64 `json:"firstRecord,omitempty"` // The first record to process MaxRecords uint64 `json:"maxRecords,omitempty"` // The maximum number of records to process Reversed bool `json:"reversed,omitempty"` // Produce results in reverse chronological order + Publisher string `json:"publisher,omitempty"` // For some query options, the publisher of the index FirstBlock uint64 `json:"firstBlock,omitempty"` // First block to export (inclusive, ignored when freshening) LastBlock uint64 `json:"lastBlock,omitempty"` // Last block to export (inclusive, ignored when freshening) Globals globals.GlobalOptions `json:"globals,omitempty"` // The global options Conn *rpc.Connection `json:"conn,omitempty"` // The connection to the RPC server BadFlag error `json:"badFlag,omitempty"` // An error flag if needed // EXISTING_CODE + PublisherAddr base.Address // EXISTING_CODE } var defaultListOptions = ListOptions{ MaxRecords: 250, + Publisher: "trueblocks.eth", LastBlock: utils.NOPOS, } @@ -57,6 +60,7 @@ func (opts *ListOptions) testLog() { logger.TestLog(opts.FirstRecord != 0, "FirstRecord: ", opts.FirstRecord) logger.TestLog(opts.MaxRecords != 250, "MaxRecords: ", opts.MaxRecords) logger.TestLog(opts.Reversed, "Reversed: ", opts.Reversed) + logger.TestLog(len(opts.Publisher) > 0 && opts.Publisher != "trueblocks.eth", "Publisher: ", opts.Publisher) logger.TestLog(opts.FirstBlock != 0, "FirstBlock: ", opts.FirstBlock) logger.TestLog(opts.LastBlock != 0 && opts.LastBlock != utils.NOPOS, "LastBlock: ", opts.LastBlock) opts.Conn.TestLog(opts.getCaches()) @@ -100,6 +104,8 @@ func listFinishParseApi(w http.ResponseWriter, r *http.Request) *ListOptions { opts.MaxRecords = globals.ToUint64(value[0]) case "reversed": opts.Reversed = true + case "publisher": + opts.Publisher = value[0] case "firstBlock": opts.FirstBlock = globals.ToUint64(value[0]) case "lastBlock": @@ -111,8 +117,10 @@ func listFinishParseApi(w http.ResponseWriter, r *http.Request) *ListOptions { } } opts.Conn = opts.Globals.FinishParseApi(w, r, opts.getCaches()) + opts.Publisher, _ = opts.Conn.GetEnsAddress(opts.Publisher) // EXISTING_CODE + opts.PublisherAddr = base.HexToAddress(opts.Publisher) // EXISTING_CODE opts.Addrs, _ = opts.Conn.GetEnsAddresses(opts.Addrs) @@ -137,8 +145,10 @@ func listFinishParse(args []string) *ListOptions { defFmt := "txt" opts := GetOptions() opts.Conn = opts.Globals.FinishParse(args, opts.getCaches()) + opts.Publisher, _ = opts.Conn.GetEnsAddress(opts.Publisher) // EXISTING_CODE + opts.PublisherAddr = base.HexToAddress(opts.Publisher) for _, arg := range args { if base.IsValidAddress(arg) { opts.Addrs = append(opts.Addrs, arg) diff --git a/src/apps/chifra/internal/list/validate.go b/src/apps/chifra/internal/list/validate.go index 76b72654d6..11a1f272b0 100644 --- a/src/apps/chifra/internal/list/validate.go +++ b/src/apps/chifra/internal/list/validate.go @@ -67,6 +67,13 @@ func (opts *ListOptions) validateList() error { } } + if len(opts.Publisher) > 0 { + err := validate.ValidateExactlyOneAddr([]string{opts.Publisher}) + if err != nil { + return err + } + } + // Note that this does not return if the index is not initialized if err := index.IndexIsInitialized(chain); err != nil { if opts.Globals.IsApiMode() { diff --git a/src/apps/chifra/internal/monitors/README.md b/src/apps/chifra/internal/monitors/README.md index 5378165537..ee59c5f1c9 100644 --- a/src/apps/chifra/internal/monitors/README.md +++ b/src/apps/chifra/internal/monitors/README.md @@ -63,7 +63,6 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -71,7 +70,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/src/apps/chifra/internal/monitors/handle_watch.go b/src/apps/chifra/internal/monitors/handle_watch.go index 7ca9e09a94..e927da6a39 100644 --- a/src/apps/chifra/internal/monitors/handle_watch.go +++ b/src/apps/chifra/internal/monitors/handle_watch.go @@ -46,6 +46,7 @@ func (opts *MonitorsOptions) RunMonitorScraper(wg *sync.WaitGroup, s *Scraper) { s.ChangeState(true, tmpPath) + runCount := uint64(0) for { if !s.Running { s.Pause() @@ -61,11 +62,16 @@ func (opts *MonitorsOptions) RunMonitorScraper(wg *sync.WaitGroup, s *Scraper) { logger.Error(err) return } else { - if canceled || opts.RunOnce { + if canceled { return } } + runCount++ + if opts.RunCount != 0 && runCount >= opts.RunCount { + return + } + sleep := opts.Sleep if sleep > 0 { ms := time.Duration(sleep*1000) * time.Millisecond @@ -155,7 +161,7 @@ func (opts *MonitorsOptions) Refresh(monitors []monitor.Monitor) (bool, error) { continue } - fmt.Printf("Processing item %d in batch %d: %d %d\n", j, i, countsBefore[j], countAfter) + logger.Info(fmt.Sprintf("Processing item %d in batch %d: %d %d\n", j, i, countsBefore[j], countAfter)) for _, cmd := range theCmds { countBefore := countsBefore[j] @@ -313,7 +319,7 @@ func (opts *MonitorsOptions) getMonitorList() []monitor.Monitor { for result := range monitorChan { switch result.Address { - case monitor.SentinalAddr: + case base.SentinalAddr: close(monitorChan) default: if result.Count() > 500000 { diff --git a/src/apps/chifra/internal/monitors/options.go b/src/apps/chifra/internal/monitors/options.go index 03e9047f02..4c5db06621 100644 --- a/src/apps/chifra/internal/monitors/options.go +++ b/src/apps/chifra/internal/monitors/options.go @@ -32,7 +32,7 @@ type MonitorsOptions struct { Watchlist string `json:"watchlist,omitempty"` // Available with --watch option only, a file containing the addresses to watch Commands string `json:"commands,omitempty"` // Available with --watch option only, the file containing the list of commands to apply to each watched address BatchSize uint64 `json:"batchSize,omitempty"` // Available with --watch option only, the number of monitors to process in each batch - RunOnce bool `json:"runOnce,omitempty"` // Available with --watch option only, only run the monitor --watch commands once then quit + RunCount uint64 `json:"runCount,omitempty"` // Available with --watch option only, run the monitor this many times, then quit Sleep float64 `json:"sleep,omitempty"` // Available with --watch option only, the number of seconds to sleep between runs Globals globals.GlobalOptions `json:"globals,omitempty"` // The global options Conn *rpc.Connection `json:"conn,omitempty"` // The connection to the RPC server @@ -57,7 +57,7 @@ func (opts *MonitorsOptions) testLog() { logger.TestLog(len(opts.Watchlist) > 0, "Watchlist: ", opts.Watchlist) logger.TestLog(len(opts.Commands) > 0, "Commands: ", opts.Commands) logger.TestLog(opts.BatchSize != 8, "BatchSize: ", opts.BatchSize) - logger.TestLog(opts.RunOnce, "RunOnce: ", opts.RunOnce) + logger.TestLog(opts.RunCount != 0, "RunCount: ", opts.RunCount) logger.TestLog(opts.Sleep != float64(14), "Sleep: ", opts.Sleep) opts.Conn.TestLog(opts.getCaches()) opts.Globals.TestLog() @@ -74,6 +74,7 @@ func monitorsFinishParseApi(w http.ResponseWriter, r *http.Request) *MonitorsOpt copy := defaultMonitorsOptions opts := © opts.BatchSize = 8 + opts.RunCount = 0 opts.Sleep = 14 for key, value := range r.URL.Query() { switch key { @@ -100,8 +101,8 @@ func monitorsFinishParseApi(w http.ResponseWriter, r *http.Request) *MonitorsOpt opts.Commands = value[0] case "batchSize": opts.BatchSize = globals.ToUint64(value[0]) - case "runOnce": - opts.RunOnce = true + case "runCount": + opts.RunCount = globals.ToUint64(value[0]) case "sleep": opts.Sleep = globals.ToFloat64(value[0]) default: diff --git a/src/apps/chifra/internal/monitors/validate.go b/src/apps/chifra/internal/monitors/validate.go index daaa9542c3..adbc7e6ea8 100644 --- a/src/apps/chifra/internal/monitors/validate.go +++ b/src/apps/chifra/internal/monitors/validate.go @@ -77,18 +77,18 @@ func (opts *MonitorsOptions) validateMonitors() error { } if opts.BatchSize < 1 { - return validate.Usage("The {0} option must be greater than zero.", "--batch-size") + return validate.Usage("The {0} option must be greater than zero.", "--batch_size") } } else { if opts.BatchSize != 8 { - return validate.Usage("The {0} option is not available{1}.", "--batch-size", " without --watch") + return validate.Usage("The {0} option is not available{1}.", "--batch_size", " without --watch") } else { opts.BatchSize = 0 } - if opts.RunOnce { - return validate.Usage("The {0} option is not available{1}.", "--run-once", " without --watch") + if opts.RunCount > 0 { + return validate.Usage("The {0} option is not available{1}.", "--run_count", " without --watch") } if opts.Sleep != 14 { diff --git a/src/apps/chifra/internal/names/output_test.go b/src/apps/chifra/internal/names/output_test.go index 194534d560..111f52329e 100644 --- a/src/apps/chifra/internal/names/output_test.go +++ b/src/apps/chifra/internal/names/output_test.go @@ -157,7 +157,7 @@ func TestNamesOptions_getCrudDataEnv(t *testing.T) { // invalid envs expected = &CrudData{ Address: crudDataField[base.Address]{ - Value: base.HexToAddress("0x0"), + Value: base.ZeroAddr, Updated: true, }, Name: crudDataField[string]{ diff --git a/src/apps/chifra/internal/scrape/README.md b/src/apps/chifra/internal/scrape/README.md index ffcd0f28cd..f6c6333f00 100644 --- a/src/apps/chifra/internal/scrape/README.md +++ b/src/apps/chifra/internal/scrape/README.md @@ -45,24 +45,25 @@ Data models produced by this tool: Each of the following additional configurable command line options are available. -**Configuration file:** `$CONFIG/$CHAIN/blockScrape.toml` -**Configuration group:** `[settings]` +**Configuration file:** `trueBlocks.toml` +**Configuration group:** `[scrape.]` | Item | Type | Default | Description / Default | | ------------------ | ------------ | ------------ | --------- | -| apps_per_chunk | uint64 | 200000 | the number of appearances to build into a chunk before consolidating it | -| snap_to_grid | uint64 | 100000 | an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index | -| first_snap | uint64 | 0 | the first block at which snap_to_grid is enabled | -| unripe_dist | uint64 | 28 | the distance (in blocks) from the front of the chain under which (inclusive) a block is considered unripe | -| channel_count | uint64 | 20 | number of concurrent processing channels | -| allow_missing | bool | true | do not report errors for blockchains that contain blocks with zero addresses | +| appsPerChunk | uint64 | 200000 | the number of appearances to build into a chunk before consolidating it | +| snapToGrid | uint64 | 100000 | an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index | +| firstSnap | uint64 | 0 | the first block at which snap_to_grid is enabled | +| unripeDist | uint64 | 28 | the distance (in blocks) from the front of the chain under which (inclusive) a block is considered unripe | +| channelCount | uint64 | 20 | number of concurrent processing channels | +| allowMissing | bool | true | do not report errors for blockchains that contain blocks with zero addresses | +Note that for Ethereum mainnet, the default values for appsPerChunk and firstSnap are 2,000,000 and 2,300,000 respectively. See the specification for a justification of these values. These items may be set in three ways, each overridding the preceeding method: --- in the above configuration file under the `[settings]` group, --- in the environment by exporting the configuration item as UPPER_CASE, without underbars, and prepended with TB_SETTINGS_, or --- on the command line using the configuration item with leading dashes (i.e., `--name`). +-- in the above configuration file under the `[scrape.]` group, +-- in the environment by exporting the configuration item as UPPER_CASE (with underbars removed) and prepended with TB_SCRAPE_CHAIN_, or +-- on the command line using the configuration item with leading dashes and in snake case (i.e., `--snake_case`). ### further information diff --git a/src/apps/chifra/internal/scrape/handle_blaze_blaze.go b/src/apps/chifra/internal/scrape/handle_blaze_blaze.go index a2e7395207..589a536e2b 100644 --- a/src/apps/chifra/internal/scrape/handle_blaze_blaze.go +++ b/src/apps/chifra/internal/scrape/handle_blaze_blaze.go @@ -33,7 +33,7 @@ func (bm *BlazeManager) HandleBlaze() (ok bool, err error) { } func (bm *BlazeManager) HandleBlaze1(blocks []base.Blknum) (ok bool, err error) { - nChannels := int(bm.opts.Settings.Channel_count) + nChannels := int(bm.opts.Settings.ChannelCount) // We need three pipelines...we shove into blocks, blocks shoves into appearances and timestamps blockChannel := make(chan base.Blknum) diff --git a/src/apps/chifra/internal/scrape/handle_scrape.go b/src/apps/chifra/internal/scrape/handle_scrape.go index ea85eef037..956238e4ec 100644 --- a/src/apps/chifra/internal/scrape/handle_scrape.go +++ b/src/apps/chifra/internal/scrape/handle_scrape.go @@ -72,8 +72,8 @@ func (opts *ScrapeOptions) HandleScrape() error { // than 'UnripeDist.' If it is, the `ripeBlock` is 'UnripeDist' behind the // head (i.e., 28 blocks usually - six minutes) ripeBlock = blazeMan.meta.Latest - if ripeBlock > opts.Settings.Unripe_dist { - ripeBlock = blazeMan.meta.Latest - opts.Settings.Unripe_dist + if ripeBlock > opts.Settings.UnripeDist { + ripeBlock = blazeMan.meta.Latest - opts.Settings.UnripeDist } blazeMan = BlazeManager{ diff --git a/src/apps/chifra/internal/scrape/handle_scrape_blaze.go b/src/apps/chifra/internal/scrape/handle_scrape_blaze.go index 324a9593d4..33ab85a6a5 100644 --- a/src/apps/chifra/internal/scrape/handle_scrape_blaze.go +++ b/src/apps/chifra/internal/scrape/handle_scrape_blaze.go @@ -26,7 +26,7 @@ func (bm *BlazeManager) HandleScrapeBlaze() error { // Do the actual scrape, wait until it finishes, clean up and return on failure if _, err := bm.HandleBlaze(); err != nil { - _ = index.CleanTemporaryFolders(config.PathToIndex(chain), false) + _ = index.CleanTempIndexFolders(chain, []string{"ripe", "unripe"}) return err } @@ -37,7 +37,7 @@ func (bm *BlazeManager) HandleScrapeBlaze() error { if !bm.processedMap[bn] { // At least one block was not processed. This would only happen in the event of an // error, so clean up, report the error and return. The loop will repeat. - _ = index.CleanTemporaryFolders(config.PathToIndex(chain), false) + _ = index.CleanTempIndexFolders(chain, []string{"ripe", "unripe"}) msg := fmt.Sprintf("A block %d was not processed%s", bn, strings.Repeat(" ", 50)) return errors.New(msg) } @@ -66,7 +66,7 @@ func (bm *BlazeManager) WriteTimestamps(endPoint uint64) error { } defer func() { - tslib.DeCache(chain) + tslib.ClearCache(chain) fp.Close() // sigintTrap.Disable(trapCh) // writeMutex.Unlock() diff --git a/src/apps/chifra/internal/scrape/options.go b/src/apps/chifra/internal/scrape/options.go index 8006bf95f4..86244f01f3 100644 --- a/src/apps/chifra/internal/scrape/options.go +++ b/src/apps/chifra/internal/scrape/options.go @@ -12,8 +12,9 @@ import ( "net/http" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/internal/globals" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/caps" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config/scrapeCfg" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/rpc" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/validate" @@ -21,21 +22,26 @@ import ( // ScrapeOptions provides all command options for the chifra scrape command. type ScrapeOptions struct { - BlockCnt uint64 `json:"blockCnt,omitempty"` // Maximum number of blocks to process per pass - Pin bool `json:"pin,omitempty"` // Pin new chunks (requires locally-running IPFS daemon or --remote) - Remote bool `json:"remote,omitempty"` // Pin new chunks to the gateway (requires pinning service keys) - Sleep float64 `json:"sleep,omitempty"` // Seconds to sleep between scraper passes - StartBlock uint64 `json:"startBlock,omitempty"` // First block to visit when scraping (snapped back to most recent snap_to_grid mark) - Settings scrapeCfg.ScrapeSettings `json:"settings,omitempty"` // Configuration items for the scrape - Globals globals.GlobalOptions `json:"globals,omitempty"` // The global options - Conn *rpc.Connection `json:"conn,omitempty"` // The connection to the RPC server - BadFlag error `json:"badFlag,omitempty"` // An error flag if needed + BlockCnt uint64 `json:"blockCnt,omitempty"` // Maximum number of blocks to process per pass + Pin bool `json:"pin,omitempty"` // Pin new chunks (requires locally-running IPFS daemon or --remote) + Remote bool `json:"remote,omitempty"` // Pin new chunks to the gateway (requires pinning service keys) + Sleep float64 `json:"sleep,omitempty"` // Seconds to sleep between scraper passes + StartBlock uint64 `json:"startBlock,omitempty"` // First block to visit when scraping (snapped back to most recent snap_to_grid mark) + RunCount uint64 `json:"runCount,omitempty"` // Run the scraper this many times, then quit + Publisher string `json:"publisher,omitempty"` // For some query options, the publisher of the index + DryRun bool `json:"dryRun,omitempty"` // Show the configuration that would be applied if run,no changes are made + Settings config.ScrapeSettings `json:"settings,omitempty"` // Configuration items for the scrape + Globals globals.GlobalOptions `json:"globals,omitempty"` // The global options + Conn *rpc.Connection `json:"conn,omitempty"` // The connection to the RPC server + BadFlag error `json:"badFlag,omitempty"` // An error flag if needed // EXISTING_CODE + PublisherAddr base.Address // EXISTING_CODE } var defaultScrapeOptions = ScrapeOptions{ - BlockCnt: 2000, + BlockCnt: 2000, + Publisher: "trueblocks.eth", } // testLog is used only during testing to export the options for this test case. @@ -45,6 +51,9 @@ func (opts *ScrapeOptions) testLog() { logger.TestLog(opts.Remote, "Remote: ", opts.Remote) logger.TestLog(opts.Sleep != float64(14), "Sleep: ", opts.Sleep) logger.TestLog(opts.StartBlock != 0, "StartBlock: ", opts.StartBlock) + logger.TestLog(opts.RunCount != 0, "RunCount: ", opts.RunCount) + logger.TestLog(len(opts.Publisher) > 0 && opts.Publisher != "trueblocks.eth", "Publisher: ", opts.Publisher) + logger.TestLog(opts.DryRun, "DryRun: ", opts.DryRun) opts.Settings.TestLog(opts.Globals.Chain, opts.Globals.TestMode) opts.Conn.TestLog(opts.getCaches()) opts.Globals.TestLog() @@ -63,11 +72,13 @@ func scrapeFinishParseApi(w http.ResponseWriter, r *http.Request) *ScrapeOptions opts.BlockCnt = 2000 opts.Sleep = 14 opts.StartBlock = 0 - opts.Settings.Apps_per_chunk = 200000 - opts.Settings.Snap_to_grid = 100000 - opts.Settings.First_snap = 0 - opts.Settings.Unripe_dist = 28 - opts.Settings.Channel_count = 20 + opts.RunCount = 0 + opts.Settings.AppsPerChunk = 200000 + opts.Settings.SnapToGrid = 100000 + opts.Settings.FirstSnap = 0 + opts.Settings.UnripeDist = 28 + opts.Settings.ChannelCount = 20 + configs := make(map[string]string, 10) for key, value := range r.URL.Query() { switch key { case "blockCnt": @@ -80,18 +91,24 @@ func scrapeFinishParseApi(w http.ResponseWriter, r *http.Request) *ScrapeOptions opts.Sleep = globals.ToFloat64(value[0]) case "startBlock": opts.StartBlock = globals.ToUint64(value[0]) + case "runCount": + opts.RunCount = globals.ToUint64(value[0]) + case "publisher": + opts.Publisher = value[0] + case "dryRun": + opts.DryRun = true case "appsPerChunk": - opts.Settings.Apps_per_chunk = globals.ToUint64(value[0]) + fallthrough case "snapToGrid": - opts.Settings.Snap_to_grid = globals.ToUint64(value[0]) + fallthrough case "firstSnap": - opts.Settings.First_snap = globals.ToUint64(value[0]) + fallthrough case "unripeDist": - opts.Settings.Unripe_dist = globals.ToUint64(value[0]) + fallthrough case "channelCount": - opts.Settings.Channel_count = globals.ToUint64(value[0]) + fallthrough case "allowMissing": - opts.Settings.Allow_missing = true + configs[key] = value[0] default: if !copy.Globals.Caps.HasKey(key) { opts.BadFlag = validate.Usage("Invalid key ({0}) in {1} route.", key, "scrape") @@ -99,8 +116,10 @@ func scrapeFinishParseApi(w http.ResponseWriter, r *http.Request) *ScrapeOptions } } opts.Conn = opts.Globals.FinishParseApi(w, r, opts.getCaches()) + opts.Publisher, _ = opts.Conn.GetEnsAddress(opts.Publisher) // EXISTING_CODE + opts.PublisherAddr = base.HexToAddress(opts.Publisher) // EXISTING_CODE return opts @@ -124,16 +143,16 @@ func scrapeFinishParse(args []string) *ScrapeOptions { defFmt := "txt" opts := GetOptions() opts.Conn = opts.Globals.FinishParse(args, opts.getCaches()) + opts.Publisher, _ = opts.Conn.GetEnsAddress(opts.Publisher) // EXISTING_CODE + opts.PublisherAddr = base.HexToAddress(opts.Publisher) if len(args) == 1 && (args[0] == "run" || args[0] == "indexer") { // these options have been deprecated, so do nothing } else if len(args) > 1 { opts.BadFlag = validate.Usage("Invalid argument {0}", args[0]) } - configFn := "blockScrape.toml" // EXISTING_CODE - opts.Settings, _ = scrapeCfg.GetSettings(opts.Globals.Chain, configFn, &scrapeCfg.Unset) if len(opts.Globals.Format) == 0 || opts.Globals.Format == "none" { opts.Globals.Format = defFmt } diff --git a/src/apps/chifra/internal/scrape/scrape_consolidate.go b/src/apps/chifra/internal/scrape/scrape_consolidate.go index b4c433093b..06f10dc1b5 100644 --- a/src/apps/chifra/internal/scrape/scrape_consolidate.go +++ b/src/apps/chifra/internal/scrape/scrape_consolidate.go @@ -10,7 +10,6 @@ import ( "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config/scrapeCfg" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/file" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/index" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" @@ -57,12 +56,12 @@ func (bm *BlazeManager) Consolidate() (bool, error) { // Check to see if we got as many ripe files as we were expecting. In the case when AllowMissing is true, we // can't really know, but if AllowMissing is false, then the number of files should be the same as the range width ripeCnt := len(ripeFileList) - unripeDist := bm.opts.Settings.Unripe_dist + unripeDist := bm.opts.Settings.UnripeDist if uint64(ripeCnt) < (bm.BlockCount() - unripeDist) { // Then, if they are not at least sequential, clean up and try again... - allowMissing := scrapeCfg.AllowMissing(chain) + allowMissing := config.GetScrape(chain).AllowMissing if err := isListSequential(chain, ripeFileList, allowMissing); err != nil { - _ = index.CleanTemporaryFolders(config.PathToCache(chain), false) + _ = index.CleanTempIndexFolders(chain, []string{"ripe", "unripe"}) return true, err } } @@ -106,8 +105,8 @@ func (bm *BlazeManager) Consolidate() (bool, error) { ripeRange := base.RangeFromFilename(ripePath) curRange.Last = ripeRange.Last - isSnap := (curRange.Last >= bm.opts.Settings.First_snap && (curRange.Last%bm.opts.Settings.Snap_to_grid) == 0) - isOvertop := (curCount >= uint64(bm.opts.Settings.Apps_per_chunk)) + isSnap := (curRange.Last >= bm.opts.Settings.FirstSnap && (curRange.Last%bm.opts.Settings.SnapToGrid) == 0) + isOvertop := (curCount >= uint64(bm.opts.Settings.AppsPerChunk)) if isSnap || isOvertop { // we're consolidating... @@ -126,7 +125,7 @@ func (bm *BlazeManager) Consolidate() (bool, error) { } indexPath := config.PathToIndex(chain) + "finalized/" + curRange.String() + ".bin" - if report, err := index.WriteChunk(chain, indexPath, appMap, len(appearances), bm.opts.Pin, bm.opts.Remote); err != nil { + if report, err := index.WriteChunk(chain, bm.opts.PublisherAddr, indexPath, appMap, len(appearances), bm.opts.Pin, bm.opts.Remote); err != nil { return false, err } else if report == nil { logger.Fatal("Should not happen, write chunk returned empty report") diff --git a/src/apps/chifra/internal/scrape/scrape_manager.go b/src/apps/chifra/internal/scrape/scrape_manager.go index f7dc1f4064..6a5118089f 100644 --- a/src/apps/chifra/internal/scrape/scrape_manager.go +++ b/src/apps/chifra/internal/scrape/scrape_manager.go @@ -42,17 +42,17 @@ func (bm *BlazeManager) Report(nAppsThen, nAppsNow int) { settings := bm.opts.Settings msg := "Block={%d} have {%d} appearances of {%d} ({%0.1f%%}). Need {%d} more. Added {%d} records ({%0.2f} apps/blk)." - need := settings.Apps_per_chunk - utils.Min(settings.Apps_per_chunk, uint64(nAppsNow)) + need := settings.AppsPerChunk - utils.Min(settings.AppsPerChunk, uint64(nAppsNow)) seen := nAppsNow if nAppsThen < nAppsNow { seen = nAppsNow - nAppsThen } - pct := float64(nAppsNow) / float64(settings.Apps_per_chunk) + pct := float64(nAppsNow) / float64(settings.AppsPerChunk) pBlk := float64(seen) / float64(bm.BlockCount()) height := bm.StartBlock() + bm.BlockCount() - 1 msg = strings.Replace(msg, "{", colors.Green, -1) msg = strings.Replace(msg, "}", colors.Off, -1) - logger.Info(fmt.Sprintf(msg, height, nAppsNow, settings.Apps_per_chunk, pct*100, need, seen, pBlk)) + logger.Info(fmt.Sprintf(msg, height, nAppsNow, settings.AppsPerChunk, pct*100, need, seen, pBlk)) } // Pause goes to sleep for a period of time based on the settings. @@ -61,7 +61,7 @@ func (bm *BlazeManager) Pause() { time.Sleep(250 * time.Millisecond) isDefaultSleep := bm.opts.Sleep >= 13 && bm.opts.Sleep <= 14 distanceFromHead := bm.meta.Latest - bm.meta.Staging - shouldSleep := !isDefaultSleep || distanceFromHead <= (2*bm.opts.Settings.Unripe_dist) + shouldSleep := !isDefaultSleep || distanceFromHead <= (2*bm.opts.Settings.UnripeDist) if shouldSleep { sleep := bm.opts.Sleep // this value may change elsewhere allow us to break out of sleeping???? if sleep > 1 { diff --git a/src/apps/chifra/internal/scrape/scrape_prepare.go b/src/apps/chifra/internal/scrape/scrape_prepare.go index ed4871e729..0aa7a4c148 100644 --- a/src/apps/chifra/internal/scrape/scrape_prepare.go +++ b/src/apps/chifra/internal/scrape/scrape_prepare.go @@ -23,7 +23,7 @@ func (opts *ScrapeOptions) Prepare() (ok bool, err error) { chain := opts.Globals.Chain // We always clean the temporary folders (other than staging) when starting - _ = index.CleanTemporaryFolders(config.PathToIndex(chain), false) + _ = index.CleanTempIndexFolders(chain, []string{"ripe", "unripe"}) // If the file already exists, we're done. bloomPath := config.PathToIndex(chain) + "blooms/000000000-000000000.bloom" @@ -56,7 +56,7 @@ func (opts *ScrapeOptions) Prepare() (ok bool, err error) { logger.Info("Writing block zero allocations for", len(prefunds), "prefunds, nAddresses:", len(appMap)) indexPath := index.ToIndexPath(bloomPath) - if report, err := index.WriteChunk(chain, indexPath, appMap, len(prefunds), opts.Pin, opts.Remote); err != nil { + if report, err := index.WriteChunk(chain, opts.PublisherAddr, indexPath, appMap, len(prefunds), opts.Pin, opts.Remote); err != nil { return false, err } else if report == nil { logger.Fatal("Should not happen, write chunk returned empty report") diff --git a/src/apps/chifra/internal/scrape/validate.go b/src/apps/chifra/internal/scrape/validate.go index ef2ddd5a55..a38d00152a 100644 --- a/src/apps/chifra/internal/scrape/validate.go +++ b/src/apps/chifra/internal/scrape/validate.go @@ -10,7 +10,6 @@ import ( "path/filepath" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config/scrapeCfg" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/file" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/index" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/pinning" @@ -25,9 +24,6 @@ import ( func (opts *ScrapeOptions) validateScrape() error { chain := opts.Globals.Chain - // First, we need to pick up the settings TODO: Should be auto-generated code somehow - opts.Settings, _ = scrapeCfg.GetSettings(chain, "blockScrape.toml", &opts.Settings) - opts.testLog() if opts.BadFlag != nil { @@ -62,7 +58,7 @@ func (opts *ScrapeOptions) validateScrape() error { if opts.Pin { if opts.Remote { - pinataKey, pinataSecret := config.GetPinningKeys(chain) + pinataKey, pinataSecret := config.GetKey("pinata").ApiKey, config.GetKey("pinata").Secret if pinataKey == "" || pinataSecret == "" { return validate.Usage("The {0} option requires {1}.", "--pin --remote", "an api key") } @@ -76,6 +72,13 @@ func (opts *ScrapeOptions) validateScrape() error { // Note this does not return if a migration is needed index.CheckBackLevelIndex(chain) + if len(opts.Publisher) > 0 { + err := validate.ValidateExactlyOneAddr([]string{opts.Publisher}) + if err != nil { + return err + } + } + ret := opts.Globals.Validate() pidPath := filepath.Join(config.PathToCache(chain), "tmp/scrape.pid") diff --git a/src/apps/chifra/internal/status/types_status.go b/src/apps/chifra/internal/status/types_status.go index 887ea464ac..cd4fac7a38 100644 --- a/src/apps/chifra/internal/status/types_status.go +++ b/src/apps/chifra/internal/status/types_status.go @@ -124,7 +124,7 @@ func (s *simpleStatus) Model(verbose bool, format string, extraOptions map[strin } chains = append(chains, ch) } else { - chainArray := config.GetChainArray() + chainArray := config.GetChains() for _, chain := range chainArray { ch := types.SimpleChain{ Chain: chain.Chain, @@ -178,7 +178,7 @@ func (opts *StatusOptions) GetSimpleStatus() (*simpleStatus, error) { return nil, err } - provider, _ := config.GetRpcProvider(chain) + provider := config.GetChain(chain).RpcProvider s := &simpleStatus{ ClientVersion: vers, Version: version.LibraryVersion, @@ -192,8 +192,8 @@ func (opts *StatusOptions) GetSimpleStatus() (*simpleStatus, error) { IsApi: opts.Globals.IsApiMode(), IsArchive: opts.Conn.IsNodeArchive(), IsTracing: opts.Conn.IsNodeTracing(), - HasEsKey: config.HasEsKeys(chain), - HasPinKey: config.HasPinningKeys(chain), + HasEsKey: len(config.GetKey("etherscan").ApiKey) > 0, + HasPinKey: len(config.GetKey("pinata").ApiKey) > 0 || len(config.GetKey("pinata").Secret) > 0, Chain: chain, NetworkId: fmt.Sprint(meta.NetworkId), ChainId: fmt.Sprint(meta.ChainId), diff --git a/src/apps/chifra/internal/tokens/handle_parts.go b/src/apps/chifra/internal/tokens/handle_parts.go index 48b905b145..0c49d035a8 100644 --- a/src/apps/chifra/internal/tokens/handle_parts.go +++ b/src/apps/chifra/internal/tokens/handle_parts.go @@ -8,6 +8,7 @@ import ( "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/names" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/output" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/tslib" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/types" "github.com/ethereum/go-ethereum" ) @@ -20,6 +21,8 @@ func (opts *TokensOptions) HandleParts() error { fetchData := func(modelChan chan types.Modeler[types.RawToken], errorChan chan error) { for _, address := range opts.Addrs { addr := base.HexToAddress(address) + currentBn := uint64(0) + currentTs := base.Timestamp(0) for _, br := range opts.BlockIds { blockNums, err := br.ResolveBlocks(chain) if err != nil { @@ -41,6 +44,13 @@ func (opts *TokensOptions) HandleParts() error { TotalSupply: state.TotalSupply, Decimals: uint64(state.Decimals), } + if opts.Globals.Verbose { + if bn == 0 || bn != currentBn { + currentTs, _ = tslib.FromBnToTs(chain, bn) + } + s.Timestamp = currentTs + currentBn = bn + } modelChan <- s } } diff --git a/src/apps/chifra/internal/transactions/validate.go b/src/apps/chifra/internal/transactions/validate.go index d7f60dfb73..5fbac3c5b6 100644 --- a/src/apps/chifra/internal/transactions/validate.go +++ b/src/apps/chifra/internal/transactions/validate.go @@ -26,7 +26,7 @@ func (opts *TransactionsOptions) validateTransactions() error { return validate.Usage("chain {0} is not properly configured.", chain) } - key := config.GetRootConfig().Keys["trueblocks"].License + key := config.GetKey("trueblocks").License if len(opts.AccountFor) > 0 && !strings.Contains(key, "+accounting") { return validate.Usage("The {0} option requires a license key. Please contact us in our discord.", "--accounting") } diff --git a/src/apps/chifra/pkg/abi/download.go b/src/apps/chifra/pkg/abi/download.go index 7aac8ede81..5c09f4ce5e 100644 --- a/src/apps/chifra/pkg/abi/download.go +++ b/src/apps/chifra/pkg/abi/download.go @@ -33,7 +33,7 @@ func DownloadAbi(chain string, address base.Address, destination *FunctionSyncMa // C++ code used do check if the address is contract in 2 places: here and in handle_addresses. We // check only in handle_addresses. - key := config.GetRootConfig().Keys["etherscan"].ApiKey + key := config.GetKey("etherscan").ApiKey if key == "" { return errors.New("cannot read Etherscan API key") } diff --git a/src/apps/chifra/pkg/abi/load_abi.go b/src/apps/chifra/pkg/abi/load_abi.go index b75312bbcd..3e7689ebde 100644 --- a/src/apps/chifra/pkg/abi/load_abi.go +++ b/src/apps/chifra/pkg/abi/load_abi.go @@ -373,15 +373,9 @@ func readFunction(reader *bufio.Reader) (function *types.SimpleFunction, err err return } -// getCacheAndChainPath returns path to cache for given chain -func getCacheAndChainPath(chain string) string { - cacheDir := config.GetRootConfig().Settings.CachePath - return path.Join(cacheDir, chain) -} - // getAbis reads all ABIs stored in the cache func getAbis(chain string) ([]types.SimpleFunction, error) { - fullPath := path.Join(getCacheAndChainPath(chain), walk.CacheTypeToFolder[walk.Cache_Abis], "known.bin") + fullPath := path.Join(config.PathToCache(chain), walk.CacheTypeToFolder[walk.Cache_Abis], "known.bin") if f, err := os.Open(fullPath); err != nil { return nil, err @@ -609,7 +603,7 @@ func SetAbis(chain string, abis []types.SimpleFunction) (err error) { // save writes contents of `content` Reader to a file func save(chain string, filePath string, content io.Reader) (err error) { - cacheDir := getCacheAndChainPath(chain) + cacheDir := config.PathToCache(chain) fullPath := path.Join(cacheDir, filePath) var f *os.File @@ -750,7 +744,7 @@ func LoadAbiFromAddress(chain string, address base.Address, destination *Functio if err = fromJson(localFile, destination); err != nil { return } - // File is correct + // File is correct, cache it if err = insertAbi(chain, address, localFile); err != nil { return } @@ -760,7 +754,7 @@ func LoadAbiFromAddress(chain string, address base.Address, destination *Functio // insertAbi copies file (e.g. opened local file) into cache func insertAbi(chain string, address base.Address, inputReader io.Reader) error { - fullPath := path.Join(getCacheAndChainPath(chain), walk.CacheTypeToFolder[walk.Cache_Abis], address.Hex()+".json") + fullPath := path.Join(config.PathToCache(chain), walk.CacheTypeToFolder[walk.Cache_Abis], address.Hex()+".json") if file, err := os.OpenFile(fullPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0666); err != nil { return err } else { @@ -775,7 +769,7 @@ func insertAbi(chain string, address base.Address, inputReader io.Reader) error // GetAbi returns single ABI per address. ABI-per-address are stored as JSON, not binary. func GetAbi(chain string, address base.Address) (simpleAbis []types.SimpleFunction, err error) { filePath := path.Join(walk.CacheTypeToFolder[walk.Cache_Abis], address.Hex()+".json") - f, err := os.Open(path.Join(getCacheAndChainPath(chain), filePath)) + f, err := os.Open(path.Join(config.PathToCache(chain), filePath)) if err != nil { return } diff --git a/src/apps/chifra/pkg/base/address.go b/src/apps/chifra/pkg/base/address.go index cbc7372e5e..d4a02ce99b 100644 --- a/src/apps/chifra/pkg/base/address.go +++ b/src/apps/chifra/pkg/base/address.go @@ -19,6 +19,12 @@ type Address struct { common.Address } +// A few well known address. ZeroAddr, of course, is 0x0. SentinalAddr is a marker to signify the end of the monitor list produced by ListMonitors +var ( + SentinalAddr = HexToAddress("0xdeaddeaddeaddeaddeaddeaddeaddeaddeaddead") + ZeroAddr = HexToAddress("0x0") +) + // Hex returns string representation of an address func (a *Address) Hex() string { if a.IsZero() { @@ -162,3 +168,8 @@ func IsValidAddressE(val string) (bool, error) { // FAKE_ETH_ADDRESS is the address we use to represent ETH in the ledgers var FAKE_ETH_ADDRESS = HexToAddress("0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee") + +// GetTestPublisher does not get customized per chain. We can only test against mainnet currently +func GetTestPublisher() Address { + return HexToAddress("0xf503017d7baf7fbc0fff7492b751025c6a78179b") +} diff --git a/src/apps/chifra/pkg/base/address_test.go b/src/apps/chifra/pkg/base/address_test.go index 6bf6df28bd..6f8e6ff4e0 100644 --- a/src/apps/chifra/pkg/base/address_test.go +++ b/src/apps/chifra/pkg/base/address_test.go @@ -65,8 +65,7 @@ func TestAddress_IsZero(t *testing.T) { t.Fatal("wrong result for zero value") } - zeroAddr := HexToAddress("0x0") - if result := zeroAddr.IsZero(); result != true { + if result := ZeroAddr.IsZero(); result != true { t.Fatal("wrong result for zero address") } } diff --git a/src/apps/chifra/pkg/base/fileRange.go b/src/apps/chifra/pkg/base/fileRange.go index 83749cc770..bb2ced8bed 100644 --- a/src/apps/chifra/pkg/base/fileRange.go +++ b/src/apps/chifra/pkg/base/fileRange.go @@ -134,3 +134,7 @@ func (r *FileRange) LaterThanB(blk uint64) bool { func (r *FileRange) Equals(needle FileRange) bool { return r.First == needle.First && r.Last == needle.Last } + +func (r *FileRange) Span() uint64 { + return r.Last - r.First + 1 +} diff --git a/src/apps/chifra/pkg/base/known_blocks.go b/src/apps/chifra/pkg/base/known_blocks.go new file mode 100644 index 0000000000..8b3147916d --- /dev/null +++ b/src/apps/chifra/pkg/base/known_blocks.go @@ -0,0 +1,9 @@ +package base + +// TODO: This is not correct per chain... +const ( + ByzantiumBlock = Blknum(4370000) + ConstantinopleBlock = Blknum(7280000) + LondonBlock = Blknum(12965000) + ShanghaiBlock = Blknum(17034870) +) diff --git a/src/apps/chifra/pkg/cache/cache_test.go b/src/apps/chifra/pkg/cache/cache_test.go index fcfb5e584b..0031973e22 100644 --- a/src/apps/chifra/pkg/cache/cache_test.go +++ b/src/apps/chifra/pkg/cache/cache_test.go @@ -42,11 +42,7 @@ func (e *ExampleBlock) MarshalCache(writer io.Writer) (err error) { var minimalVersion version.Version func init() { - var err error - minimalVersion, err = version.NewVersion("GHC-TrueBlocks//0.10.0-beta") - if err != nil { - panic(err) - } + minimalVersion = version.NewVersion("GHC-TrueBlocks//0.10.0-beta") } // Now we make ExampleBlock implement CacheUnmarshaler interface, which is like diff --git a/src/apps/chifra/pkg/cache/item.go b/src/apps/chifra/pkg/cache/item.go index d568e4393f..d2d66f65ba 100644 --- a/src/apps/chifra/pkg/cache/item.go +++ b/src/apps/chifra/pkg/cache/item.go @@ -26,10 +26,7 @@ func init() { // Set currentHeader, so that we don't have to parse version // and build a header each time we want to encode/decode cache // item. - ver, err := version.NewVersion(version.LibraryVersion) - if err != nil { - panic(err) - } + ver := version.NewVersion(version.LibraryVersion) currentHeader = &header{ Magic: Magic, Version: ver.Uint64(), diff --git a/src/apps/chifra/pkg/colors/colors.go b/src/apps/chifra/pkg/colors/colors.go index aabb454d04..2f9af46e8c 100644 --- a/src/apps/chifra/pkg/colors/colors.go +++ b/src/apps/chifra/pkg/colors/colors.go @@ -4,6 +4,9 @@ package colors +import "strings" + +var None = "" var Off = "\033[0m" var Red = "\033[31m" var Green = "\033[32m" @@ -43,3 +46,19 @@ func ColorsOff() { BrightWhite = "" BrightBlack = "" } + +func Colored(s string) string { + s = strings.Replace(s, "{", Green, -1) + s = strings.Replace(s, "@", BrightYellow, -1) + s = strings.Replace(s, "}", Off, -1) + return s +} + +func ColoredWith(s string, c string) string { + s = c + s + s = strings.Replace(s, "{", Green, -1) + s = strings.Replace(s, "@", BrightYellow, -1) + s = strings.Replace(s, "}", c, -1) + s += Off + return s +} diff --git a/src/apps/chifra/pkg/config/chainGroup.go b/src/apps/chifra/pkg/config/chainGroup.go new file mode 100644 index 0000000000..ee5a224f7b --- /dev/null +++ b/src/apps/chifra/pkg/config/chainGroup.go @@ -0,0 +1,79 @@ +// Copyright 2021 The TrueBlocks Authors. All rights reserved. +// Use of this source code is governed by a license that can +// be found in the LICENSE file. + +package config + +import ( + "strings" + "sync" +) + +type chainGroup struct { + Chain string `toml:"chain,omitempty"` + ChainId string `toml:"chainId"` + IpfsGateway string `toml:"ipfsGateway,omitempty"` + LocalExplorer string `toml:"localExplorer,omitempty"` + RemoteExplorer string `toml:"remoteExplorer,omitempty"` + RpcProvider string `toml:"rpcProvider"` + Symbol string `toml:"symbol"` + Scrape ScrapeSettings `toml:"scrape"` +} + +var chainMutex sync.Mutex + +// GetChain returns the chain for a given chain +func GetChain(chain string) chainGroup { + ch := GetRootConfig().Chains[chain] + if ch.Chain == "" { + ch.Chain = chain + ch.LocalExplorer = cleanUrl(ch.LocalExplorer) + ch.RemoteExplorer = cleanUrl(ch.RemoteExplorer) + ch.RpcProvider = cleanPrefix(ch.RpcProvider) + if len(ch.IpfsGateway) == 0 { + ch.IpfsGateway = GetSettings().DefaultGateway + } + ch.IpfsGateway = strings.Replace(ch.IpfsGateway, "[{CHAIN}]", chain, -1) + // cache it... + chainMutex.Lock() + GetRootConfig().Chains[chain] = ch + chainMutex.Unlock() + } + return ch +} + +// GetChains returns a list of all chains configured in the config file. Note, there is no "official" +// list. Users may add their own chains. +func GetChains() []chainGroup { + chainArray := make([]chainGroup, 0, len(GetRootConfig().Chains)) + for k, v := range GetRootConfig().Chains { + v.Chain = k + if len(v.IpfsGateway) == 0 { + v.IpfsGateway = GetSettings().DefaultGateway + } + chainArray = append(chainArray, v) + } + return chainArray +} + +// IsChainConfigured returns true if the chain is configured in the config file. +func IsChainConfigured(needle string) bool { + return GetRootConfig().Chains[needle] != chainGroup{} +} + +// cleanUrl cleans the user's input trying to make a valid Url. +func cleanUrl(url string) string { + url = cleanPrefix(url) + if !strings.HasSuffix(url, "/") { + url += "/" + } + return url +} + +// cleanPrefix cleans the user's input trying to make a valid Url. +func cleanPrefix(url string) string { + if !strings.HasPrefix(url, "http") { + url = "https://" + url + } + return url +} diff --git a/src/apps/chifra/pkg/config/root_config.go b/src/apps/chifra/pkg/config/config.go similarity index 56% rename from src/apps/chifra/pkg/config/root_config.go rename to src/apps/chifra/pkg/config/config.go index 91d2826d5b..f387a62d52 100644 --- a/src/apps/chifra/pkg/config/root_config.go +++ b/src/apps/chifra/pkg/config/config.go @@ -10,69 +10,21 @@ import ( "path/filepath" "runtime" "strings" - "time" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/file" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/usage" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/version" "github.com/spf13/viper" ) var trueBlocksViper = viper.New() var trueBlocksConfig ConfigFile -type versionGroup struct { - Current string `toml:"current"` -} - -type chainGroup struct { - Chain string `toml:"chain"` - ChainId string `toml:"chainId"` - LocalExplorer string `toml:"localExplorer"` - RemoteExplorer string `toml:"remoteExplorer"` - RpcProvider string `toml:"rpcProvider"` - IpfsGateway string `toml:"ipfsGateway"` - Symbol string `toml:"symbol"` -} - -type keyGroup struct { - License string `toml:"license"` - ApiKey string `toml:"apiKey"` - Secret string `toml:"secret"` - Jwt string `toml:"jwt"` -} - -// TODO: This needs to be documented -type grpcGroup struct { - UdsTimeout time.Duration `toml:"udsTimeout"` -} - -type settingsGroup struct { - CachePath string `toml:"cachePath"` - IndexPath string `toml:"indexPath"` - DefaultChain string `toml:"defaultChain"` - DefaultGateway string `toml:"defaultGateway"` -} - type ConfigFile struct { - Version versionGroup - Settings settingsGroup - Grpc grpcGroup - Keys map[string]keyGroup - Chains map[string]chainGroup -} - -func GetChainArray() []chainGroup { - var result []chainGroup - for k, v := range GetRootConfig().Chains { - v.Chain = k - if len(v.IpfsGateway) == 0 { - v.IpfsGateway = GetRootConfig().Settings.DefaultGateway - } - result = append(result, v) - } - return result + Version versionGroup `toml:"version"` + Settings settingsGroup `toml:"settings"` + Keys map[string]keyGroup `toml:"keys"` + Chains map[string]chainGroup `toml:"chains"` } // init sets up default values for the given configuration @@ -89,7 +41,16 @@ func init() { func GetRootConfig() *ConfigFile { if len(trueBlocksConfig.Settings.CachePath) == 0 { configPath := PathToRootConfig() - MustReadConfig(trueBlocksViper, &trueBlocksConfig, configPath) + trueBlocksViper.AddConfigPath(configPath) + trueBlocksViper.SetEnvPrefix("TB") + trueBlocksViper.AutomaticEnv() + trueBlocksViper.SetEnvKeyReplacer(strings.NewReplacer(".", "_")) + if err := trueBlocksViper.ReadInConfig(); err != nil { + logger.Fatal(err) + } + if err := trueBlocksViper.Unmarshal(&trueBlocksConfig); err != nil { + logger.Fatal(err) + } user, _ := user.Current() @@ -118,7 +79,7 @@ func GetRootConfig() *ConfigFile { // only then can we complete these paths. At this point these paths // only point to the top-levl of the cache or index. Also note that // these two calls do not return if they fail, so no need to handle errors - defaultChains := []string{GetDefaultChain()} + defaultChains := []string{GetSettings().DefaultChain} _ = file.EstablishFolders(trueBlocksConfig.Settings.CachePath, defaultChains) _ = file.EstablishFolders(trueBlocksConfig.Settings.IndexPath, defaultChains) } @@ -126,23 +87,9 @@ func GetRootConfig() *ConfigFile { return &trueBlocksConfig } -func IsAtLeastVersion(needle string) bool { - var current, desired version.Version - var err error - if current, err = version.NewVersion(GetRootConfig().Version.Current); err != nil { - return true - } - - if desired, err = version.NewVersion(needle); err != nil { - return true - } - - return !current.IsEarlierThan(desired) -} - // PathToRootConfig returns the path where to find configuration files func PathToRootConfig() string { - configPath, err := PathFromXDG("XDG_CONFIG_HOME") + configPath, err := pathFromXDG("XDG_CONFIG_HOME") if err != nil { logger.Fatal(err) } else if len(configPath) > 0 { @@ -164,11 +111,7 @@ func PathToRootConfig() string { return filepath.Join(user.HomeDir, osPath) + "/" } -func GetDefaultChain() string { - return GetRootConfig().Settings.DefaultChain -} - -func PathFromXDG(envVar string) (string, error) { +func pathFromXDG(envVar string) (string, error) { // If present, we require both an existing path and a fully qualified path xdg := os.Getenv(envVar) if len(xdg) == 0 { @@ -185,43 +128,3 @@ func PathFromXDG(envVar string) (string, error) { return filepath.Join(xdg, "") + "/", nil } - -// MustReadConfig calls Viper's ReadInConfig and fills values in the -// given targetStruct. Any error will result in a call to logger.Fatal -func MustReadConfig(v *viper.Viper, targetStruct interface{}, path string) { - v.AddConfigPath(path) - v.SetEnvPrefix("TB") - v.AutomaticEnv() - v.SetEnvKeyReplacer(strings.NewReplacer(".", "_")) - - err := v.ReadInConfig() - if err != nil { - _, ok := err.(viper.ConfigFileNotFoundError) - // We only require some files to be present - if !ok { - logger.Fatal(err) - } - } - - err = v.Unmarshal(targetStruct) - if err != nil { - logger.Fatal(err) - } -} - -func GetPinningKeys(chain string) (string, string) { - keys := GetRootConfig().Keys - a := keys["pinata"].ApiKey - b := keys["pinata"].Secret - return a, b -} - -func HasPinningKeys(chain string) bool { - a, b := GetPinningKeys(chain) - return len(a)+len(b) > 0 -} - -func HasEsKeys(chain string) bool { - keys := GetRootConfig().Keys - return len(keys["etherscan"].ApiKey) > 0 -} diff --git a/src/apps/chifra/pkg/config/root_config_integration_test.go b/src/apps/chifra/pkg/config/config_integration_test.go similarity index 100% rename from src/apps/chifra/pkg/config/root_config_integration_test.go rename to src/apps/chifra/pkg/config/config_integration_test.go diff --git a/src/apps/chifra/pkg/config/root_config_test.go b/src/apps/chifra/pkg/config/config_test.go similarity index 71% rename from src/apps/chifra/pkg/config/root_config_test.go rename to src/apps/chifra/pkg/config/config_test.go index 13f4c9295a..32705fd426 100644 --- a/src/apps/chifra/pkg/config/root_config_test.go +++ b/src/apps/chifra/pkg/config/config_test.go @@ -11,34 +11,33 @@ import ( ) func Test_ReadTrueBlocks(t *testing.T) { - values := GetRootConfig() - if len(values.Settings.CachePath) == 0 { + if len(GetSettings().CachePath) == 0 { t.Error("CachePath is empty") } - if values.Settings.CachePath[len(values.Settings.CachePath)-1] != '/' { + if GetSettings().CachePath[len(GetSettings().CachePath)-1] != '/' { // the raw path may not end with a slash, but the returned path better ret := PathToCache(utils.GetTestChain()) if len(ret) > 0 && ret[len(ret)-1] != '/' { t.Error("CachePath does not end with a '/'") } } - if values.Settings.CachePath[0] != '/' { + if GetSettings().CachePath[0] != '/' { t.Error("CachePath is not an absolute path") } - if len(values.Settings.IndexPath) == 0 { + if len(GetSettings().IndexPath) == 0 { t.Error("IndexPath is empty") } - if values.Settings.IndexPath[len(values.Settings.IndexPath)-1] != '/' { + if GetSettings().IndexPath[len(GetSettings().IndexPath)-1] != '/' { // the raw path may not end with a slash, but the returned path better ret := PathToIndex(utils.GetTestChain()) if len(ret) > 0 && ret[len(ret)-1] != '/' { t.Error("IndexPath does not end with a '/'") } } - if values.Settings.IndexPath[0] != '/' { + if GetSettings().IndexPath[0] != '/' { t.Error("IndexPath is not an absolute path") } - if len(values.Settings.DefaultChain) == 0 { + if len(GetSettings().DefaultChain) == 0 { t.Error("DefaultChain is empty.") } } diff --git a/src/apps/chifra/pkg/config/file.go b/src/apps/chifra/pkg/config/file.go new file mode 100644 index 0000000000..a2d1b30792 --- /dev/null +++ b/src/apps/chifra/pkg/config/file.go @@ -0,0 +1,26 @@ +package config + +import ( + "os" + + "github.com/BurntSushi/toml" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/version" +) + +func ReadConfigFile(inFile string, contents interface{}) error { + _, err := toml.DecodeFile(inFile, contents) + return err +} + +func (cfg *ConfigFile) WriteConfigFile(outFn string) error { + cfg.SetVersionStr(version.VersionString()) + if f, err := os.Create(outFn); err != nil { + return err + } else { + defer f.Close() + if err := toml.NewEncoder(f).Encode(cfg); err != nil { + return err + } + } + return nil +} diff --git a/src/apps/chifra/pkg/config/keyGroup.go b/src/apps/chifra/pkg/config/keyGroup.go new file mode 100644 index 0000000000..1fc26ea7cd --- /dev/null +++ b/src/apps/chifra/pkg/config/keyGroup.go @@ -0,0 +1,16 @@ +// Copyright 2021 The TrueBlocks Authors. All rights reserved. +// Use of this source code is governed by a license that can +// be found in the LICENSE file. + +package config + +type keyGroup struct { + License string `toml:"license,omitempty"` + ApiKey string `toml:"apiKey"` + Secret string `toml:"secret,omitempty"` + Jwt string `toml:"jwt,omitempty"` +} + +func GetKey(set string) keyGroup { + return GetRootConfig().Keys[set] +} diff --git a/src/apps/chifra/pkg/config/config_paths.go b/src/apps/chifra/pkg/config/paths.go similarity index 82% rename from src/apps/chifra/pkg/config/config_paths.go rename to src/apps/chifra/pkg/config/paths.go index 78f2cfa03a..960c3c9cd4 100644 --- a/src/apps/chifra/pkg/config/config_paths.go +++ b/src/apps/chifra/pkg/config/paths.go @@ -15,15 +15,15 @@ import ( // MustGetPathToChainConfig returns the chain-specific config folder ignoring errors func MustGetPathToChainConfig(chain string) string { - path, _ := GetPathToChainConfig(chain) + path, _ := PathToChainConfig(chain) return path } -// GetPathToChainConfig returns the chain-specific config folder -func GetPathToChainConfig(chain string) (string, error) { +// PathToChainConfig returns the chain-specific config folder +func PathToChainConfig(chain string) (string, error) { // We always need a chain if len(chain) == 0 { - chain = GetDefaultChain() + chain = GetSettings().DefaultChain } ret := PathToRootConfig() @@ -36,11 +36,11 @@ func GetPathToChainConfig(chain string) (string, error) { // PathToIndex returns the one and only indexPath func PathToIndex(chain string) string { // We need the index path from either XDG which dominates or the config file - indexPath, err := PathFromXDG("XDG_CACHE_HOME") + indexPath, err := pathFromXDG("XDG_CACHE_HOME") if err != nil { logger.Fatal(err) } else if len(indexPath) == 0 { - indexPath = GetRootConfig().Settings.IndexPath + indexPath = GetSettings().IndexPath } // We want the index folder to be named `unchained` and be in @@ -51,7 +51,7 @@ func PathToIndex(chain string) string { // We always have to have a chain... if len(chain) == 0 { - chain = GetDefaultChain() + chain = GetSettings().DefaultChain } // We know what we want, create it if it doesn't exist and return it @@ -63,11 +63,11 @@ func PathToIndex(chain string) string { // PathToCache returns the one and only cachePath func PathToCache(chain string) string { // We need the index path from either XDG which dominates or the config file - cachePath, err := PathFromXDG("XDG_CACHE_HOME") + cachePath, err := pathFromXDG("XDG_CACHE_HOME") if err != nil { logger.Fatal(err) } else if len(cachePath) == 0 { - cachePath = GetRootConfig().Settings.CachePath + cachePath = GetSettings().CachePath } // We want the cache folder to be named `cache` and be in @@ -78,7 +78,7 @@ func PathToCache(chain string) string { // We always have to have a chain... if len(chain) == 0 { - chain = GetDefaultChain() + chain = GetSettings().DefaultChain } // We know what we want, create it if it doesn't exist and return it @@ -118,13 +118,3 @@ func EstablishIndexPaths(indexPath string) { logger.Fatal(err) } } - -func IsChainConfigured(needle string) bool { - haystack := GetChainArray() - for _, chain := range haystack { - if chain.Chain == needle { - return true - } - } - return false -} diff --git a/src/apps/chifra/pkg/config/config_paths_integration_test.go b/src/apps/chifra/pkg/config/paths_integration_test.go similarity index 100% rename from src/apps/chifra/pkg/config/config_paths_integration_test.go rename to src/apps/chifra/pkg/config/paths_integration_test.go diff --git a/src/apps/chifra/pkg/config/per_chain.go b/src/apps/chifra/pkg/config/per_chain.go deleted file mode 100644 index bf340c790f..0000000000 --- a/src/apps/chifra/pkg/config/per_chain.go +++ /dev/null @@ -1,71 +0,0 @@ -// Copyright 2021 The TrueBlocks Authors. All rights reserved. -// Use of this source code is governed by a license that can -// be found in the LICENSE file. - -package config - -import ( - "strings" -) - -// HasChains returns the expected chain id for a given chain -func HasChains() bool { - return len(GetRootConfig().Chains) > 0 -} - -// GetChainId returns the expected chain id for a given chain -func GetChainId(chain string) string { - ch := GetRootConfig().Chains[chain] - return ch.ChainId -} - -// GetLocalExplorer returns the expected chain id for a given chain -func GetLocalExplorer(chain string) string { - ch := GetRootConfig().Chains[chain] - return cleanUrl(ch.LocalExplorer) -} - -// GetRemoteExplorer returns the expected chain id for a given chain -func GetRemoteExplorer(chain string) string { - ch := GetRootConfig().Chains[chain] - return cleanUrl(ch.RemoteExplorer) -} - -// GetIpfsGateway returns the ipfs gateway for a chain -func GetIpfsGateway(chain string) string { - def := GetRootConfig().Settings.DefaultGateway - gateway := GetRootConfig().Chains[chain].IpfsGateway - if len(gateway) == 0 { - gateway = def - } - gateway = strings.Replace(gateway, "[{CHAIN}]", chain, -1) - return cleanUrl(gateway) -} - -// GetRpcProvider returns the RPC provider for a chain -func GetRpcProvider(chain string) (string, error) { - ch := GetRootConfig().Chains[chain] - cleaned := cleanPrefix(ch.RpcProvider) - return cleaned, nil -} - -// GetSymbol returns the expected chain id for a given chain -func GetSymbol(chain string) string { - ch := GetRootConfig().Chains[chain] - return ch.Symbol -} - -func cleanUrl(url string) string { - url = cleanPrefix(url) - if !strings.HasSuffix(url, "/") { - url += "/" - } - return url -} - -func cleanPrefix(url string) string { - if !strings.HasPrefix(url, "http") { - url = "https://" + url - } - return url -} diff --git a/src/apps/chifra/pkg/config/scrapeCfg/doc.go b/src/apps/chifra/pkg/config/scrapeCfg/doc.go deleted file mode 100644 index e6b24e194e..0000000000 --- a/src/apps/chifra/pkg/config/scrapeCfg/doc.go +++ /dev/null @@ -1,2 +0,0 @@ -// Package scrapeCfg carries information on the scraper's configuration -package scrapeCfg diff --git a/src/apps/chifra/pkg/config/scrapeCfg/scrapeCfg.go b/src/apps/chifra/pkg/config/scrapeCfg/scrapeCfg.go deleted file mode 100644 index 085cff36bf..0000000000 --- a/src/apps/chifra/pkg/config/scrapeCfg/scrapeCfg.go +++ /dev/null @@ -1,262 +0,0 @@ -// Copyright 2021 The TrueBlocks Authors. All rights reserved. -// Use of this source code is governed by a license that can -// be found in the LICENSE file. - -package scrapeCfg - -import ( - "os" - "path/filepath" - "reflect" - "strconv" - "strings" - - "github.com/BurntSushi/toml" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/file" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/utils" -) - -// TODO: We need a better way to handle configuration files -// TODO: See issue #2259 and OLD_BLOCKSCRAPE_CONFIG_CODE below - -// ScrapeSettings are configuration items needed by the scraper -type ScrapeSettings struct { - Apps_per_chunk uint64 `json:"appsPerChunk"` // The number of appearances to build into a chunk before consolidating it - Snap_to_grid uint64 `json:"snapToGrid"` // An override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index - First_snap uint64 `json:"firstSnap"` // The first block at which snap_to_grid is enabled - Unripe_dist uint64 `json:"unripeDist"` // The distance (in blocks) from the front of the chain under which (inclusive) a block is considered unripe - Channel_count uint64 `json:"-"` // Number of concurrent block processing channels - Allow_missing bool `json:"allowMissing,omitempty"` // Do not report errors for blockchain that contain blocks with zero addresses - // EXISTING_CODE - // EXISTING_CODE -} - -var defaultSettings = ScrapeSettings{ - Apps_per_chunk: 200000, - Snap_to_grid: 100000, - First_snap: 0, - Unripe_dist: 28, - Channel_count: 20, - Allow_missing: false, - // EXISTING_CODE - // EXISTING_CODE -} - -var Unset = ScrapeSettings{ - Apps_per_chunk: utils.NOPOS, - Snap_to_grid: utils.NOPOS, - First_snap: utils.NOPOS, - Unripe_dist: utils.NOPOS, - Channel_count: utils.NOPOS, - Allow_missing: false, - // EXISTING_CODE - // EXISTING_CODE -} - -func (s *ScrapeSettings) isDefault(chain, fldName string) bool { - def := GetDefault(chain) - switch fldName { - case "Apps_per_chunk": - return s.Apps_per_chunk == def.Apps_per_chunk - case "Snap_to_grid": - return s.Snap_to_grid == def.Snap_to_grid - case "First_snap": - return s.First_snap == def.First_snap - case "Unripe_dist": - return s.Unripe_dist == def.Unripe_dist - case "Channel_count": - return s.Channel_count == def.Channel_count - case "Allow_missing": - return s.Allow_missing == def.Allow_missing - } - - // EXISTING_CODE - // EXISTING_CODE - - return false -} - -func (s *ScrapeSettings) TestLog(chain string, test bool) { - logger.TestLog(!s.isDefault(chain, "Apps_per_chunk"), "Apps_per_chunk: ", s.Apps_per_chunk) - logger.TestLog(!s.isDefault(chain, "Snap_to_grid"), "Snap_to_grid: ", s.Snap_to_grid) - logger.TestLog(!s.isDefault(chain, "First_snap"), "First_snap: ", s.First_snap) - logger.TestLog(!s.isDefault(chain, "Unripe_dist"), "Unripe_dist: ", s.Unripe_dist) - logger.TestLog(!s.isDefault(chain, "Channel_count"), "Channel_count: ", s.Channel_count) - logger.TestLog(!s.isDefault(chain, "Allow_missing"), "Allow_missing: ", s.Allow_missing) - // EXISTING_CODE - // EXISTING_CODE -} - -func GetDefault(chain string) ScrapeSettings { - base := defaultSettings - // EXISTING_CODE - if chain == "mainnet" { - base.Apps_per_chunk = 2000000 - base.First_snap = 2300000 - } - // EXISTING_CODE - return base -} - -// GetSettings retrieves scrape config from (in order) default, config, environment, optionally provided cmdLine -func GetSettings(chain, configFn string, cmdLine *ScrapeSettings) (ScrapeSettings, error) { - // Start with the defalt values... - base := GetDefault(chain) - - tt := reflect.TypeOf(defaultSettings) - fieldList, _, _ := utils.GetFields(&tt, "txt", true) - - if strings.Contains(configFn, "trueBlocks.toml") { - configFn = filepath.Join(config.PathToRootConfig(), configFn) - } else { - configFn = filepath.Join(config.MustGetPathToChainConfig(chain), configFn) - } - - if file.FileExists(configFn) { - type TomlFile struct { - Settings ScrapeSettings - } - var t TomlFile - t.Settings = Unset - // ...pick up values from toml file... - if _, err := toml.Decode(file.AsciiFileToString(configFn), &t); err != nil { - logger.Error("Could not load", configFn) - logger.Fatal("Error:", err) - return ScrapeSettings{}, err - } - base.overlay(chain, t.Settings) - } - - // ...check the environment... - for _, field := range fieldList { - envKey := toEnvStr(field) - envValue := os.Getenv(envKey) - if envValue != "" { - fName := utils.MakeFirstUpperCase(field) - fld := reflect.ValueOf(&base).Elem().FieldByName(fName) - if fld.Kind() == reflect.String { - fld.SetString(envValue) - } else if fld.Kind() == reflect.Bool { - if envValue == "true" { - fld.SetBool(true) - } - } else { - if v, err := strconv.ParseUint(envValue, 10, 64); err == nil { - fld.SetUint(v) - } - } - } - } - - if cmdLine != nil { - base.overlay(chain, *cmdLine) - } - - return base, nil -} - -func (s *ScrapeSettings) overlay(chain string, overlay ScrapeSettings) { - if !overlay.isDefault(chain, "Apps_per_chunk") && overlay.Apps_per_chunk != 0 && overlay.Apps_per_chunk != utils.NOPOS { - s.Apps_per_chunk = overlay.Apps_per_chunk - } - if !overlay.isDefault(chain, "Snap_to_grid") && overlay.Snap_to_grid != 0 && overlay.Snap_to_grid != utils.NOPOS { - s.Snap_to_grid = overlay.Snap_to_grid - } - if !overlay.isDefault(chain, "First_snap") && overlay.First_snap != 0 && overlay.First_snap != utils.NOPOS { - s.First_snap = overlay.First_snap - } - if !overlay.isDefault(chain, "Unripe_dist") && overlay.Unripe_dist != 0 && overlay.Unripe_dist != utils.NOPOS { - s.Unripe_dist = overlay.Unripe_dist - } - if !overlay.isDefault(chain, "Channel_count") && overlay.Channel_count != 0 && overlay.Channel_count != utils.NOPOS { - s.Channel_count = overlay.Channel_count - } - if !overlay.isDefault(chain, "Allow_missing") && overlay.Allow_missing { - s.Allow_missing = overlay.Allow_missing - } - - // EXISTING_CODE - // EXISTING_CODE -} - -// EXISTING_CODE -// - -func AllowMissing(chain string) bool { - s, _ := GetSettings(chain, "blockScrape.toml", nil) - return s.Allow_missing -} - -func toEnvStr(name string) string { - return "TB_SETTINGS_" + strings.ToUpper(strings.Replace(name, "_", "", -1)) -} - -/* -OLD_BLOCKSCRAPE_CONFIG_CODE -type blockScrapeSettings struct { - Block_chan_cnt int - Addr_chan_cnt int - Apps_per_chunk int - Unripe_dist int - Snap_to_grid int - First_snap int - Allow_missing bool - Pinata_api_key string - Pinata_secret_api_key string - Pinata_jwt string -} - -var defaultSettings = blockScrapeSettings{ - Block_chan_cnt: 10, - Addr_chan_cnt: 20, - Apps_per_chunk: 200000, - Unripe_dist: 28, - Snap_to_grid: 100000, - First_snap: 0, - Allow_missing: false, - Pinata_api_key: "", - Pinata_secret_api_key: "", - Pinata_jwt: "", -} - -type BlockScrape struct { - Settings blockScrapeSettings -} - -func GetBlockScrapeSettings(chain string) blockScrapeSettings { - str := utils.AsciiFileToString(filepath.Join(MustGetPathToChainConfig(chain), "blockScrape.toml")) - conf := BlockScrape{ - Settings: defaultSettings, - } - if _, err := toml.Decode(str, &conf); err != nil { - // TODO: Don't panic here, just report and return defaults - logger.Fatal(err) - } - - tt := reflect.TypeOf(conf.Settings) - fields, _, _ := utils.GetFields(&tt, "txt", true) - - for _, field := range fields { - key := "TB_SETTINGS_" + strings.ToUpper(field) - val := os.Getenv(key) - if val != "" { - if strings.HasPrefix(field, "pinata") { - reflect.ValueOf(&conf.Settings).Elem().FieldByName(utils.MakeFirstUpperCase(field)).SetString(val) - } else if field == "allow_missing" { - if val == "true" { - reflect.ValueOf(&conf.Settings).Elem().FieldByName(utils.MakeFirstUpperCase(field)).SetBool(true) - } - } else { - if v, err := strconv.ParseInt(val, 10, 32); err == nil { - reflect.ValueOf(&conf.Settings).Elem().FieldByName(utils.MakeFirstUpperCase(field)).SetInt(v) - } - } - } - } - - return conf.Settings -} -*/ -// EXISTING_CODE diff --git a/src/apps/chifra/pkg/config/scrapeGroup.go b/src/apps/chifra/pkg/config/scrapeGroup.go new file mode 100644 index 0000000000..264cae8fcd --- /dev/null +++ b/src/apps/chifra/pkg/config/scrapeGroup.go @@ -0,0 +1,84 @@ +// Copyright 2021 The TrueBlocks Authors. All rights reserved. +// Use of this source code is governed by a license that can +// be found in the LICENSE file. + +package config + +import ( + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/utils" +) + +// ScrapeSettings carries config information for the scraper +type ScrapeSettings struct { + AppsPerChunk uint64 `toml:"appsPerChunk" json:"appsPerChunk"` + SnapToGrid uint64 `toml:"snapToGrid" json:"snapToGrid"` + FirstSnap uint64 `toml:"firstSnap" json:"firstSnap"` + UnripeDist uint64 `toml:"unripeDist" json:"unripeDist"` + AllowMissing bool `toml:"allowMissing" json:"allowMissing"` + ChannelCount uint64 `toml:"channelCount" json:"channelCount"` +} + +// GetScrape returns the scraper settings per chain +func GetScrape(chain string) ScrapeSettings { + empty := ScrapeSettings{} + if GetRootConfig().Chains[chain].Scrape == empty { + settings := ScrapeSettings{ + AppsPerChunk: 500000, + SnapToGrid: 100000, + FirstSnap: 500000, + UnripeDist: 28, + ChannelCount: 20, + AllowMissing: false, + } + if chain == "mainnet" { + settings.AppsPerChunk = 2000000 + settings.FirstSnap = 2300000 + } + ch := GetRootConfig().Chains[chain] + ch.Scrape = settings + GetRootConfig().Chains[chain] = ch + } + + return GetRootConfig().Chains[chain].Scrape +} + +func (s *ScrapeSettings) TestLog(chain string, test bool) { + logger.TestLog(false, "AppsPerChunk: ", s.AppsPerChunk) + logger.TestLog(false, "SnapToGrid: ", s.SnapToGrid) + logger.TestLog(false, "FirstSnap: ", s.FirstSnap) + logger.TestLog(false, "UnripeDist: ", s.UnripeDist) + logger.TestLog(false, "ChannelCount: ", s.ChannelCount) + logger.TestLog(false, "AllowMissing: ", s.AllowMissing) +} + +func SetScrapeArgs(chain string, args map[string]string) { + ch := trueBlocksConfig.Chains[chain] + + empty := ScrapeSettings{} + if trueBlocksConfig.Chains[chain].Scrape == empty { + ch.Scrape = GetScrape(chain) + + } else { + settings := trueBlocksConfig.Chains[chain].Scrape + for key, value := range args { + switch key { + case "appsPerChunk": + settings.AppsPerChunk = utils.MustParseUint(value) + case "snapToGrid": + settings.SnapToGrid = utils.MustParseUint(value) + case "firstSnap": + settings.FirstSnap = utils.MustParseUint(value) + case "unripeDist": + settings.UnripeDist = utils.MustParseUint(value) + case "channelCount": + settings.ChannelCount = utils.MustParseUint(value) + case "allowMissing": + settings.AllowMissing = true + } + } + ch.Scrape = settings + } + + trueBlocksConfig.Chains[chain] = ch +} diff --git a/src/apps/chifra/pkg/config/settingsGroup.go b/src/apps/chifra/pkg/config/settingsGroup.go new file mode 100644 index 0000000000..406de365a0 --- /dev/null +++ b/src/apps/chifra/pkg/config/settingsGroup.go @@ -0,0 +1,16 @@ +// Copyright 2021 The TrueBlocks Authors. All rights reserved. +// Use of this source code is governed by a license that can +// be found in the LICENSE file. + +package config + +type settingsGroup struct { + CachePath string `toml:"cachePath"` + IndexPath string `toml:"indexPath"` + DefaultChain string `toml:"defaultChain"` + DefaultGateway string `toml:"defaultGateway"` +} + +func GetSettings() settingsGroup { + return GetRootConfig().Settings +} diff --git a/src/apps/chifra/pkg/config/upgrade/doc.go b/src/apps/chifra/pkg/config/upgrade/doc.go new file mode 100644 index 0000000000..3bc438760a --- /dev/null +++ b/src/apps/chifra/pkg/config/upgrade/doc.go @@ -0,0 +1,2 @@ +// Package upgrade reads and manages configuration files +package upgrade diff --git a/src/apps/chifra/pkg/config/upgrade/upgrade.go b/src/apps/chifra/pkg/config/upgrade/upgrade.go new file mode 100644 index 0000000000..089ebbd08d --- /dev/null +++ b/src/apps/chifra/pkg/config/upgrade/upgrade.go @@ -0,0 +1,88 @@ +package upgrade + +import ( + "fmt" + + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/colors" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/file" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/version" +) + +// upgradeConfigs will upgrade the config files to the latest versions +func UpgradeConfigs(newVersion version.Version) error { + fn := config.PathToRootConfig() + "trueBlocks.toml" + var cfg config.ConfigFile + if err := config.ReadConfigFile(fn, &cfg); err != nil { + return err + } + + for chain, group := range cfg.Chains { + group.Chain = chain + ch := cfg.Chains[chain] + scrape := config.ScrapeSettings{ + AppsPerChunk: 500000, + SnapToGrid: 100000, + FirstSnap: 500000, + UnripeDist: 28, + AllowMissing: false, + ChannelCount: 20, + } + if chain == "mainnet" { + scrape.AppsPerChunk = 2000000 + scrape.FirstSnap = 2300000 + } + + fn := config.PathToRootConfig() + "config/" + chain + "/blockScrape.toml" + if file.FileExists(fn) { + _ = MergeScrapeConfig(fn, &scrape) + } + ch.Scrape = scrape + cfg.Chains[chain] = ch + } + + // Re-write the file (after making a backup) with the new version + _, _ = file.Copy(fn, fn+".bak") + _ = cfg.WriteConfigFile(fn) // updates the version + logger.Fatal(colors.Colored(fmt.Sprintf("Your configuration files were upgraded to {%s}. Rerun your command.", newVersion.String()))) + + return nil +} + +type oldScrapeGroup struct { + AppsPerChunk uint64 `toml:"apps_per_chunk"` + SnapToGrid uint64 `toml:"snap_to_grid"` + FirstSnap uint64 `toml:"first_snap"` + UnripeDist uint64 `toml:"unripe_dist"` + AllowMissing bool `toml:"allow_missing"` + ChannelCount uint64 `toml:"channel_count"` +} + +type OldScrape struct { + Settings oldScrapeGroup `toml:"settings"` +} + +func MergeScrapeConfig(fn string, scrape *config.ScrapeSettings) error { + var sCfg OldScrape + if err := config.ReadConfigFile(fn, &sCfg); err != nil { + return err + } + if sCfg.Settings.AppsPerChunk > 0 { + scrape.AppsPerChunk = sCfg.Settings.AppsPerChunk + } + if sCfg.Settings.SnapToGrid > 0 { + scrape.SnapToGrid = sCfg.Settings.SnapToGrid + } + if sCfg.Settings.FirstSnap > 0 { + scrape.FirstSnap = sCfg.Settings.FirstSnap + } + if sCfg.Settings.UnripeDist > 0 { + scrape.UnripeDist = sCfg.Settings.UnripeDist + } + if sCfg.Settings.ChannelCount > 0 { + scrape.ChannelCount = sCfg.Settings.ChannelCount + } + scrape.AllowMissing = sCfg.Settings.AllowMissing + return nil +} diff --git a/src/apps/chifra/pkg/config/versionGroup.go b/src/apps/chifra/pkg/config/versionGroup.go new file mode 100644 index 0000000000..f72a1f4710 --- /dev/null +++ b/src/apps/chifra/pkg/config/versionGroup.go @@ -0,0 +1,17 @@ +// Copyright 2021 The TrueBlocks Authors. All rights reserved. +// Use of this source code is governed by a license that can +// be found in the LICENSE file. + +package config + +type versionGroup struct { + Current string `toml:"current"` +} + +func GetVersion() versionGroup { + return GetRootConfig().Version +} + +func (cfg *ConfigFile) SetVersionStr(current string) { + cfg.Version.Current = current +} diff --git a/src/apps/chifra/pkg/file/asciifiles.go b/src/apps/chifra/pkg/file/asciifiles.go index a2fe8393fc..0c613393f3 100644 --- a/src/apps/chifra/pkg/file/asciifiles.go +++ b/src/apps/chifra/pkg/file/asciifiles.go @@ -74,7 +74,7 @@ func LinesToAsciiFile(filename string, value []string) error { } defer file.Close() - lines := strings.Join(value, "\n") + "\n" + lines := strings.Join(value[:], "\n") + "\n" _, err = io.WriteString(file, lines) if err != nil { return err diff --git a/src/apps/chifra/pkg/file/count.go b/src/apps/chifra/pkg/file/count.go index 7b1b1db779..13fe49b2df 100644 --- a/src/apps/chifra/pkg/file/count.go +++ b/src/apps/chifra/pkg/file/count.go @@ -2,14 +2,9 @@ package file import ( "os" - - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" ) func NFilesInFolder(path string) int { - files, err := os.ReadDir(path) - if err != nil { - logger.Fatal(err) - } + files, _ := os.ReadDir(path) return len(files) } diff --git a/src/apps/chifra/pkg/index/chunk_write.go b/src/apps/chifra/pkg/index/chunk_write.go index 63bcca02de..fcb05e6420 100644 --- a/src/apps/chifra/pkg/index/chunk_write.go +++ b/src/apps/chifra/pkg/index/chunk_write.go @@ -7,7 +7,6 @@ import ( "os" "path/filepath" "sort" - "strings" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/colors" @@ -29,24 +28,26 @@ type WriteChunkReport struct { Range base.FileRange nAddresses int nAppearances int + FileSize int64 Snapped bool Pinned bool PinRecord manifest.ChunkRecord } func (c *WriteChunkReport) Report() { - str := fmt.Sprintf("%sWrote %d address and %d appearance records to $INDEX/%s.bin%s%s", colors.BrightBlue, c.nAddresses, c.nAppearances, c.Range, colors.Off, spaces20) + report := `Wrote {%d} address and {%d} appearance records to {$INDEX/%s.bin}` if c.Snapped { - str = fmt.Sprintf("%sWrote %d address and %d appearance records to $INDEX/%s.bin %s%s%s", colors.BrightBlue, c.nAddresses, c.nAppearances, c.Range, colors.Yellow, "(snapped to grid)", colors.Off) + report += ` @(snapped to grid)}` } - logger.Info(str) + report += " (size: {%d} , span: {%d})" + logger.Info(colors.ColoredWith(fmt.Sprintf(report, c.nAddresses, c.nAppearances, c.Range, c.FileSize, c.Range.Span()), colors.BrightBlue)) if c.Pinned { str := fmt.Sprintf("%sPinned chunk $INDEX/%s.bin (%s,%s)%s", colors.BrightBlue, c.Range, c.PinRecord.IndexHash, c.PinRecord.BloomHash, colors.Off) logger.Info(str) } } -func WriteChunk(chain, fileName string, addrAppearanceMap AddressAppearanceMap, nApps int, pin, remote bool) (*WriteChunkReport, error) { +func WriteChunk(chain string, publisher base.Address, fileName string, addrAppearanceMap AddressAppearanceMap, nApps int, pin, remote bool) (*WriteChunkReport, error) { // We're going to build two tables. An addressTable and an appearanceTable. We do this as we spin // through the map @@ -167,7 +168,7 @@ func WriteChunk(chain, fileName string, addrAppearanceMap AddressAppearanceMap, report.PinRecord.BloomHash = rec.BloomHash report.PinRecord.IndexSize = rec.IndexSize report.PinRecord.BloomSize = rec.BloomSize - return &report, manifest.UpdateManifest(chain, rec) + return &report, manifest.UpdateManifest(chain, publisher, rec) } else { return nil, err @@ -178,8 +179,6 @@ func WriteChunk(chain, fileName string, addrAppearanceMap AddressAppearanceMap, } } -var spaces20 = strings.Repeat(" ", 20) - func ResultToRecord(result *pinning.PinResult) manifest.ChunkRecord { if len(result.Local.BloomHash) > 0 { return manifest.ChunkRecord{ diff --git a/src/apps/chifra/pkg/index/download.go b/src/apps/chifra/pkg/index/download.go index 281fe0400c..c82adcb7a5 100644 --- a/src/apps/chifra/pkg/index/download.go +++ b/src/apps/chifra/pkg/index/download.go @@ -186,7 +186,7 @@ func DownloadChunks(chain string, chunksToDownload []manifest.ChunkRecord, chunk ctx: ctx, progressChannel: progressChannel, downloadWg: &downloadWg, - gatewayUrl: config.GetIpfsGateway(chain), + gatewayUrl: config.GetChain(chain).IpfsGateway, writeChannel: writeChannel, nRetries: 8, } diff --git a/src/apps/chifra/pkg/index/establish.go b/src/apps/chifra/pkg/index/establish.go index ac953e1004..de61f3de9f 100644 --- a/src/apps/chifra/pkg/index/establish.go +++ b/src/apps/chifra/pkg/index/establish.go @@ -25,10 +25,10 @@ import ( // EstablishIndexChunk a filename to an index portion, finds the correspoding CID (hash) // entry in the manifest, and downloads the index chunk to the local drive -func EstablishIndexChunk(chain string, fileRange base.FileRange) (bool, error) { +func EstablishIndexChunk(chain string, publisher base.Address, fileRange base.FileRange) (bool, error) { exists, fileName := fileRange.RangeToFilename(chain) - chunkManifest, err := manifest.ReadManifest(chain, manifest.FromCache) + chunkManifest, err := manifest.ReadManifest(chain, publisher, manifest.FromCache) if err != nil { return exists, err } @@ -70,19 +70,24 @@ func EstablishIndexChunk(chain string, fileRange base.FileRange) (bool, error) { return file.FileExists(fileName), nil } -// CleanTemporaryFolders removes any files that may be partial or incomplete -func CleanTemporaryFolders(indexPath string, incStaging bool) error { - folders := []string{"ripe", "unripe", "maps", "staging"} - if !incStaging { - folders = folders[:len(folders)-2] - } +// CleanEphemeralIndexFolders removes files in ripe and unripe +func CleanEphemeralIndexFolders(chain string) error { + return CleanTempIndexFolders(chain, []string{"ripe", "unripe"}) +} + +// CleanTempIndexFolders removes any files that may be partial or incomplete +func CleanTempIndexFolders(chain string, subFolders []string) error { + indexPath := config.PathToIndex(chain) - for _, f := range folders { + for _, f := range subFolders { folder := filepath.Join(indexPath, f) + // We want to remove whatever is there... err := os.RemoveAll(folder) if err != nil { return err } + // ...but put it back + _ = file.EstablishFolder(folder) } return nil diff --git a/src/apps/chifra/pkg/index/establish_integration_test.go b/src/apps/chifra/pkg/index/establish_integration_test.go index 330aba7aeb..dcfa48a2c0 100644 --- a/src/apps/chifra/pkg/index/establish_integration_test.go +++ b/src/apps/chifra/pkg/index/establish_integration_test.go @@ -14,7 +14,7 @@ package index // } // func IndexPathFromRange(fileRange base.FileRange) (string, error) { -// tmp := cache CachePath("mainnet", walk.Index_Final) +// tmp := cache C achePath("mainnet", walk.Index_Final) // path := tmp.Get FullPath(walk.FilenameFromRange(fileRange, "")) // if file.FileExists(path) { // return path, nil diff --git a/src/apps/chifra/pkg/index/uniq_appearances.go b/src/apps/chifra/pkg/index/uniq_appearances.go index 38ac914bec..896e96361c 100644 --- a/src/apps/chifra/pkg/index/uniq_appearances.go +++ b/src/apps/chifra/pkg/index/uniq_appearances.go @@ -11,6 +11,15 @@ import ( "github.com/ethereum/go-ethereum/common/hexutil" ) +// UniqFromWithdrawals extracts addresses from an array of receipts +func UniqFromWithdrawals(chain string, withdrawals []types.SimpleWithdrawal, bn base.Blknum, addrMap AddressBooleanMap) (err error) { + for _, withdrawal := range withdrawals { + // logger.Info("Adding withdrawl", withdrawal.Address.Hex(), bn, types.Withdrawal) + addAddressToMaps(withdrawal.Address.Hex(), bn, types.Withdrawal, addrMap) + } + return nil +} + // UniqFromReceipts extracts addresses from an array of receipts func UniqFromReceipts(chain string, receipts []types.SimpleReceipt, addrMap AddressBooleanMap) (err error) { for _, receipt := range receipts { @@ -74,11 +83,11 @@ func UniqFromTraces(chain string, traces []types.SimpleTrace, addrMap AddressBoo // Early clients allowed misconfigured miner settings with address // 0x0 (reward got burned). We enter a false record with a false tx_id // to account for this. - author = "0xdeaddeaddeaddeaddeaddeaddeaddeaddeaddead" - addAddressToMaps(author, bn, 99997, addrMap) + author = base.SentinalAddr.Hex() + addAddressToMaps(author, bn, types.MisconfigReward, addrMap) } else { - addAddressToMaps(author, bn, 99999, addrMap) + addAddressToMaps(author, bn, types.BlockReward, addrMap) } @@ -89,18 +98,18 @@ func UniqFromTraces(chain string, traces []types.SimpleTrace, addrMap AddressBoo // Early clients allowed misconfigured miner settings with address // 0x0 (reward got burned). We enter a false record with a false tx_id // to account for this. - author = "0xdeaddeaddeaddeaddeaddeaddeaddeaddeaddead" - addAddressToMaps(author, bn, 99998, addrMap) + author = base.SentinalAddr.Hex() + addAddressToMaps(author, bn, types.UncleReward, addrMap) } else { - addAddressToMaps(author, bn, 99998, addrMap) + addAddressToMaps(author, bn, types.UncleReward, addrMap) } } else if trace.Action.RewardType == "external" { // This only happens in xDai as far as we know... author := trace.Action.Author.Hex() - addAddressToMaps(author, bn, 99996, addrMap) + addAddressToMaps(author, bn, types.ExternalReward, addrMap) } else { fmt.Println("Unknown reward type", trace.Action.RewardType) diff --git a/src/apps/chifra/pkg/logger/logger.go b/src/apps/chifra/pkg/logger/logger.go index 1fc68813d0..ef861e4ad3 100644 --- a/src/apps/chifra/pkg/logger/logger.go +++ b/src/apps/chifra/pkg/logger/logger.go @@ -93,6 +93,16 @@ func toLog(sev severity, a ...interface{}) { } fmt.Fprintln(os.Stderr, "") + } else if sev == warning { + defer fmt.Fprint(os.Stderr, colors.Off) + fmt.Fprint(os.Stderr, colors.Yellow) + fmt.Fprintln(os.Stderr, a...) + + } else if sev == err { + defer fmt.Fprint(os.Stderr, colors.Off) + fmt.Fprint(os.Stderr, colors.Red) + fmt.Fprintln(os.Stderr, a...) + } else { fmt.Fprintln(os.Stderr, a...) } diff --git a/src/apps/chifra/pkg/manifest/download.go b/src/apps/chifra/pkg/manifest/download.go index 677d33ad35..4a897f6db2 100644 --- a/src/apps/chifra/pkg/manifest/download.go +++ b/src/apps/chifra/pkg/manifest/download.go @@ -16,43 +16,19 @@ import ( "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/articulate" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/call" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/rpc" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/types" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/unchained" ) -// fromRemote gets the CID from the smart contract, calls -// the gateway and returns the parsed manifest -func fromRemote(chain string) (*Manifest, error) { - cid, err := ReadUnchainedIndex(chain, "", unchained.GetPreferredPublisher()) - if err != nil { - return nil, err - } - - gatewayUrl := config.GetIpfsGateway(chain) - - logger.InfoTable("Chain:", chain) - logger.InfoTable("Gateway:", gatewayUrl) - logger.InfoTable("CID:", cid) - - return downloadManifest(chain, gatewayUrl, cid) -} - // ReadUnchainedIndex calls UnchainedIndex smart contract to get the current manifest IPFS CID as // published by the given publisher -func ReadUnchainedIndex(chain, reason string, publisher base.Address) (string, error) { +func ReadUnchainedIndex(chain string, publisher base.Address, database string) (string, error) { cid := os.Getenv("TB_OVERRIDE_CID") if cid != "" { return cid, nil } - database := chain - if reason != "" { - database += ("-" + reason) - } - unchainedChain := "mainnet" // the unchained index is on mainnet theCall := fmt.Sprintf("manifestHashMap(%s, \"%s\")", publisher, database) conn := rpc.TempConnection(unchainedChain) diff --git a/src/apps/chifra/pkg/manifest/manifest.go b/src/apps/chifra/pkg/manifest/manifest.go index bd8a08d36f..07c82945c9 100644 --- a/src/apps/chifra/pkg/manifest/manifest.go +++ b/src/apps/chifra/pkg/manifest/manifest.go @@ -11,8 +11,8 @@ import ( "strings" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/colors" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config/scrapeCfg" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/file" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/unchained" @@ -33,7 +33,7 @@ type Manifest struct { Schemas base.IpfsHash `json:"schemas"` // An IPFS hash pointing to documentation describing the binary format of the files in the index - Config scrapeCfg.ScrapeSettings `json:"config"` + Config config.ScrapeSettings `json:"config"` // A list of pinned chunks (see ChunkRecord) detailing the location of all chunks in the index and associated bloom filters Chunks []ChunkRecord `json:"chunks"` @@ -63,9 +63,25 @@ const ( var ErrManifestNotFound = errors.New("could not find manifest.json or it was empty") // ReadManifest reads the manifest from either the local cache or the Unchained Index smart contract -func ReadManifest(chain string, source Source) (*Manifest, error) { +func ReadManifest(chain string, publisher base.Address, source Source) (*Manifest, error) { if source == FromContract { - man, err := fromRemote(chain) + database := chain + cid, err := ReadUnchainedIndex(chain, publisher, database) + if err != nil { + return nil, err + } else if len(cid) == 0 { + return nil, fmt.Errorf("no record found in the Unchained Index for database %s from publisher %s", database, publisher.Hex()) + } + + gatewayUrl := config.GetChain(chain).IpfsGateway + + logger.InfoTable("Chain:", chain) + logger.InfoTable("Gateway:", gatewayUrl) + logger.InfoTable("Publisher:", publisher) + logger.InfoTable("Database:", database) + logger.InfoTable("CID:", cid) + + man, err := downloadManifest(chain, gatewayUrl, cid) if man != nil { man.LoadChunkMap() } @@ -97,8 +113,8 @@ func (m *Manifest) LoadChunkMap() { // TODO: Protect against overwriting files on disc -func UpdateManifest(chain string, chunk ChunkRecord) error { - man, err := ReadManifest(chain, FromCache) +func UpdateManifest(chain string, publisher base.Address, chunk ChunkRecord) error { + man, err := ReadManifest(chain, publisher, FromCache) if err != nil { if err != ErrManifestNotFound { return err @@ -130,12 +146,14 @@ func UpdateManifest(chain string, chunk ChunkRecord) error { }) } - logger.Info("Updating manifest with", len(man.Chunks), "chunks", spaces) + logger.Info(colors.Magenta+"Updating manifest with", len(man.Chunks), "chunks", spaces, colors.Off) return man.SaveManifest(chain) } // SaveManifest writes the manifest to disc in JSON func (m *Manifest) SaveManifest(chain string) error { + m.Config = config.GetScrape(chain) + fileName := filepath.Join(config.MustGetPathToChainConfig(chain), "manifest.json") w, err := os.OpenFile(fileName, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0644) if err != nil { diff --git a/src/apps/chifra/pkg/manifest/remove_chunk.go b/src/apps/chifra/pkg/manifest/remove_chunk.go index b58b6b0cbc..7a75f437c7 100644 --- a/src/apps/chifra/pkg/manifest/remove_chunk.go +++ b/src/apps/chifra/pkg/manifest/remove_chunk.go @@ -13,7 +13,7 @@ import ( // update the manifest by removing all chunks at or after the given path. Note that if this // function aborts due to error and the backup files still exist, the function will attempt // to restore the backup files before returning. -func RemoveChunk(chain, bloomFn, indexFn string) (err error) { +func RemoveChunk(chain string, publisher base.Address, bloomFn, indexFn string) (err error) { manifestFn := filepath.Join(config.MustGetPathToChainConfig(chain), "manifest.json") manifestBackup := manifestFn + ".backup" @@ -59,7 +59,7 @@ func RemoveChunk(chain, bloomFn, indexFn string) (err error) { } var man *Manifest - man, err = ReadManifest(chain, FromCache) + man, err = ReadManifest(chain, publisher, FromCache) removedRange, err1 := base.RangeFromFilenameE(bloomFn) if err1 != nil { diff --git a/src/apps/chifra/pkg/monitor/monitor.go b/src/apps/chifra/pkg/monitor/monitor.go index 774e39b955..10cedb8fa3 100644 --- a/src/apps/chifra/pkg/monitor/monitor.go +++ b/src/apps/chifra/pkg/monitor/monitor.go @@ -170,14 +170,11 @@ func (mon *Monitor) Remove() (bool, error) { return !file.FileExists(mon.Path()), nil } -// SentinalAddr is a marker to signify the end of the monitor list produced by ListMonitors -var SentinalAddr = base.HexToAddress("0xdeaddeaddeaddeaddeaddeaddeaddeaddeaddead") - // ListMonitors puts a list of Monitors into the monitorChannel. The list of monitors is built from // a file called addresses.tsv in the current folder or, if not present, from existing monitors func ListMonitors(chain, watchList string, monitorChan chan<- Monitor) { defer func() { - monitorChan <- Monitor{Address: SentinalAddr} + monitorChan <- Monitor{Address: base.SentinalAddr} }() if watchList != "existing" { @@ -250,7 +247,7 @@ func GetMonitorMap(chain string) (map[base.Address]*Monitor, []*Monitor) { for mon := range monitorChan { mon := mon switch mon.Address { - case SentinalAddr: + case base.SentinalAddr: close(monitorChan) default: monMap[mon.Address] = &mon diff --git a/src/apps/chifra/pkg/pinning/fetch.go b/src/apps/chifra/pkg/pinning/fetch.go index 225fc34c6b..29d3e74d08 100644 --- a/src/apps/chifra/pkg/pinning/fetch.go +++ b/src/apps/chifra/pkg/pinning/fetch.go @@ -33,7 +33,7 @@ func FetchFromGateway(ctx context.Context, gateway, hash string) (*FetchResult, } if response.StatusCode != 200 { // logger.Fatalln("DefaultClient.Do returned StatusCode not equal to 200 in FetFromGateway with", url) - return nil, fmt.Errorf("wrong status code: %d", response.StatusCode) + return nil, fmt.Errorf("fetch to %s returned status code: %d", url, response.StatusCode) } body := response.Body diff --git a/src/apps/chifra/pkg/pinning/pin_file.go b/src/apps/chifra/pkg/pinning/pin_file.go index 6b1a17da0c..701ca70a5c 100644 --- a/src/apps/chifra/pkg/pinning/pin_file.go +++ b/src/apps/chifra/pkg/pinning/pin_file.go @@ -36,7 +36,6 @@ func PinItem(chain string, dbName, path string, isRemote bool) (hash base.IpfsHa } if isLocal { - logger.Info("Pinning", dbName, "to local service") localService, _ := NewPinningService(chain, Local) if hash, err = localService.PinFile(path, true); err != nil { err = fmt.Errorf("error pinning locally: %s", err) @@ -45,10 +44,9 @@ func PinItem(chain string, dbName, path string, isRemote bool) (hash base.IpfsHa } if isRemote { - logger.Info("Pinning", dbName, "to remote service") remoteService, _ := NewPinningService(chain, Pinata) var remoteHash base.IpfsHash - if remoteHash, err = remoteService.PinFile(path, true); err != nil { + if remoteHash, err = remoteService.PinFile(path, false); err != nil { err = fmt.Errorf("error pinning remotely: %s", err) return } @@ -60,7 +58,7 @@ func PinItem(chain string, dbName, path string, isRemote bool) (hash base.IpfsHa } } - logger.Info("Pinned", dbName, "file", path, "to", hash) + logger.Info(colors.Magenta+"Pinned", dbName, "file", path, "to", hash, colors.Off) return } @@ -77,7 +75,6 @@ func PinChunk(chain, bloomFile, indexFile string, isRemote bool) (PinResult, err isLocal := LocalDaemonRunning() if isLocal { - // logger.Info(colors.Magenta+"Pinning locally...", colors.Off) if result.Local.BloomHash, result.err = localService.PinFile(bloomFile, true); result.err != nil { return PinResult{}, result.err } @@ -90,7 +87,6 @@ func PinChunk(chain, bloomFile, indexFile string, isRemote bool) (PinResult, err } if isRemote { - logger.Info(colors.Magenta+"Pinning remotely...", colors.Off) if result.Remote.BloomHash, result.err = remoteService.PinFile(bloomFile, false); result.err != nil { return PinResult{}, result.err } diff --git a/src/apps/chifra/pkg/pinning/service.go b/src/apps/chifra/pkg/pinning/service.go index 63aa18a3c4..80c2fcb469 100644 --- a/src/apps/chifra/pkg/pinning/service.go +++ b/src/apps/chifra/pkg/pinning/service.go @@ -37,8 +37,7 @@ const ( ) func NewPinningService(chain string, which ServiceType) (Service, error) { - pinataKey, pinataSecret := config.GetPinningKeys(chain) - + pinataKey, pinataSecret := config.GetKey("pinata").ApiKey, config.GetKey("pinata").Secret switch which { case Local: return Service{}, nil diff --git a/src/apps/chifra/pkg/prefunds/prefunds.go b/src/apps/chifra/pkg/prefunds/prefunds.go index 329bc382c1..9ae14ba747 100644 --- a/src/apps/chifra/pkg/prefunds/prefunds.go +++ b/src/apps/chifra/pkg/prefunds/prefunds.go @@ -21,7 +21,7 @@ type Allocation struct { } // emptyAllocs is a list of empty allocations. We use this to return at least one allocation -var emptyAllocs = []Allocation{{Address: base.HexToAddress("0x0"), Balance: *big.NewInt(0)}} +var emptyAllocs = []Allocation{{Address: base.ZeroAddr, Balance: *big.NewInt(0)}} type allocCallback func(*Allocation, *any) (bool, error) diff --git a/src/apps/chifra/pkg/pricing/maker.go b/src/apps/chifra/pkg/pricing/maker.go index 385b6a19c6..0c849ca598 100644 --- a/src/apps/chifra/pkg/pricing/maker.go +++ b/src/apps/chifra/pkg/pricing/maker.go @@ -66,10 +66,10 @@ func PriceUsdMaker(conn *rpc.Connection, testMode bool, statement *types.SimpleS blockNumber: statement.BlockNumber, source1: makerMedianizer, theCall1: theCall, - source2: base.HexToAddress("0x0"), + source2: base.ZeroAddr, theCall2: "0x" + rawHex, - first: base.HexToAddress("0x0"), - second: base.HexToAddress("0x0"), + first: base.ZeroAddr, + second: base.ZeroAddr, reversed: false, int0: int0, int1: int1, diff --git a/src/apps/chifra/pkg/pricing/reporting.go b/src/apps/chifra/pkg/pricing/reporting.go index ae04bfab0f..aa8b6ec0bd 100644 --- a/src/apps/chifra/pkg/pricing/reporting.go +++ b/src/apps/chifra/pkg/pricing/reporting.go @@ -48,14 +48,14 @@ func (r *priceDebugger) report(msg string, testMode bool) { logger.TestLog(true, "blockNumber: ", r.blockNumber) logger.TestLog(true, "source1 (uni/mkr): ", r.source1) logger.TestLog(true, "str0 (call): ", r.theCall1) - if r.source2 != base.HexToAddress("0x0") { + if r.source2 != base.ZeroAddr { logger.TestLog(true, "source2 (pair): ", r.source2) } logger.TestLog(true, "str1 (call/data): ", r.theCall2) - if r.first != base.HexToAddress("0x0") { + if r.first != base.ZeroAddr { logger.TestLog(true, "first: ", r.first) } - if r.second != base.HexToAddress("0x0") { + if r.second != base.ZeroAddr { logger.TestLog(true, "second: ", r.second) logger.TestLog(true, "reversed: ", r.reversed) } diff --git a/src/apps/chifra/pkg/rpc/client.go b/src/apps/chifra/pkg/rpc/client.go index 6ccb4d3540..b9d2873619 100644 --- a/src/apps/chifra/pkg/rpc/client.go +++ b/src/apps/chifra/pkg/rpc/client.go @@ -53,14 +53,14 @@ func (conn *Connection) GetClientIDs() (uint64, uint64, error) { // GetLatestBlockNumber returns the block number at the front of the chain (i.e. latest) func (conn *Connection) GetLatestBlockNumber() uint64 { if ec, err := conn.getClient(); err != nil { - logger.Error("Could not connect to RPC client: %w", err) + logger.Error("Could not connect to RPC client", err) return 0 } else { defer ec.Close() r, err := ec.BlockNumber(context.Background()) if err != nil { - logger.Error("Could not connect to RPC client: %w", err) + logger.Error("Could not connect to RPC client", err) return 0 } @@ -75,7 +75,7 @@ var clientMutex sync.Mutex var perProviderClientMap = map[string]*ethclient.Client{} func (conn *Connection) getClient() (*ethclient.Client, error) { - provider, _ := config.GetRpcProvider(conn.Chain) + provider := config.GetChain(conn.Chain).RpcProvider if provider == "https://" { var noProvider = ` diff --git a/src/apps/chifra/pkg/rpc/contract_integration_test.go b/src/apps/chifra/pkg/rpc/contract_integration_test.go index 11225c0839..68a752697d 100644 --- a/src/apps/chifra/pkg/rpc/contract_integration_test.go +++ b/src/apps/chifra/pkg/rpc/contract_integration_test.go @@ -43,7 +43,7 @@ func TestGetContractDeployBlock(t *testing.T) { } // Error when the address is not a contract - _, err = conn.GetContractDeployBlock(base.HexToAddress("0x0")) + _, err = conn.GetContractDeployBlock(base.ZeroAddr) if err != ErrNotAContract { t.Fatal("expected ErrNotAContract, but got", err) } diff --git a/src/apps/chifra/pkg/rpc/ens.go b/src/apps/chifra/pkg/rpc/ens.go index c65ef2f973..5aaa75d08e 100644 --- a/src/apps/chifra/pkg/rpc/ens.go +++ b/src/apps/chifra/pkg/rpc/ens.go @@ -14,6 +14,20 @@ import ( // GetEnsAddresses converts an array of strings, if they contains .eth, into addresses. Note, we take // chain parameter, but ignore it choosing to look at mainnet ENS only func (conn *Connection) GetEnsAddresses(addrs []string) (out []string, found bool) { + has := false + for _, addr := range addrs { + if strings.Contains(addr, ".eth") { + has = true + break + } + } + if !has { + for _, term := range addrs { + out = append(out, utils.LowerIfHex(term)) + } + return out, false + } + // Note: we use ENS on mainnet always tc := TempConnection("mainnet") if ec, err := tc.getClient(); err != nil { diff --git a/src/apps/chifra/pkg/rpc/etherscan.go b/src/apps/chifra/pkg/rpc/etherscan.go index 374aaab77d..108d74b8c7 100644 --- a/src/apps/chifra/pkg/rpc/etherscan.go +++ b/src/apps/chifra/pkg/rpc/etherscan.go @@ -115,25 +115,25 @@ func (conn *Connection) rawToSimple(addr, requestType string, rawTx *types.RawSl if requestType == "int" { // We use a weird marker here since Etherscan doesn't send the transaction id for internal txs and we don't want to make another RPC call // We tried (see commented code), but EtherScan balks with a weird message - s.TransactionIndex = 80809 + s.TransactionIndex = types.EsInternalTx // s.BlockHash = base.HexToHash("0xdeadbeef") // got, err := conn.GetESTransactionByHash(s.Hash) // if err != nil { // logger.Warn("error getting transaction from etherscan:", err) - // s.TransactionIndex = 80809 + // s.TransactionIndex = EsInternalTx // } else { // s.TransactionIndex = utils.MustParseUint(got.TransactionIndex) // } } else if requestType == "miner" { s.BlockHash = base.HexToHash("0xdeadbeef") - s.TransactionIndex = 99999 + s.TransactionIndex = types.BlockReward s.From = base.BlockRewardSender s.Value.SetString("5000000000000000000", 0) s.To = base.HexToAddress(addr) } else if requestType == "uncles" { s.BlockHash = base.HexToHash("0xdeadbeef") - s.TransactionIndex = 99998 + s.TransactionIndex = types.UncleReward s.From = base.UncleRewardSender s.Value.SetString("3750000000000000000", 0) s.To = base.HexToAddress(addr) @@ -172,7 +172,7 @@ func getEtherscanUrl(value string, requestType string, paginator *Paginator) (st logger.Fatal("Should not happen in getEtherscanUrl", requestType) } - key := config.GetRootConfig().Keys["etherscan"].ApiKey + key := config.GetKey("etherscan").ApiKey if key == "" { return "", errors.New("cannot read Etherscan API key") } diff --git a/src/apps/chifra/pkg/rpc/get_block.go b/src/apps/chifra/pkg/rpc/get_block.go index 0cccd29d48..fb1e1eaff8 100644 --- a/src/apps/chifra/pkg/rpc/get_block.go +++ b/src/apps/chifra/pkg/rpc/get_block.go @@ -52,11 +52,6 @@ func (conn *Connection) GetBlockBodyByNumber(bn uint64) (types.SimpleBlock[types return block, err } - block.Uncles = make([]base.Hash, 0, len(rawBlock.Uncles)) - for _, uncle := range rawBlock.Uncles { - block.Uncles = append(block.Uncles, base.HexToHash(uncle)) - } - ts, _ := strconv.ParseInt(rawBlock.Timestamp, 0, 64) block.Transactions = make([]types.SimpleTransaction, 0, len(rawBlock.Transactions)) for _, rawTx := range rawBlock.Transactions { @@ -105,11 +100,6 @@ func (conn *Connection) GetBlockHeaderByNumber(bn uint64) (block types.SimpleBlo return block, err } - block.Uncles = make([]base.Hash, 0, len(rawBlock.Uncles)) - for _, uncle := range rawBlock.Uncles { - block.Uncles = append(block.Uncles, base.HexToHash(uncle)) - } - block.Transactions = make([]string, 0, len(rawBlock.Transactions)) for _, txHash := range rawBlock.Transactions { block.Transactions = append(block.Transactions, fmt.Sprint(txHash)) @@ -273,9 +263,9 @@ func (conn *Connection) getBlockRaw(bn uint64, withTxs bool) (*types.RawBlock, e func (conn *Connection) getBlockReward(bn uint64) *big.Int { if bn == 0 { return big.NewInt(0) - } else if bn < byzantiumBlock { + } else if bn < base.ByzantiumBlock { return big.NewInt(5000000000000000000) - } else if bn < constantinopleBlock { + } else if bn < base.ConstantinopleBlock { return big.NewInt(3000000000000000000) } else { return big.NewInt(2000000000000000000) diff --git a/src/apps/chifra/pkg/rpc/get_meta.go b/src/apps/chifra/pkg/rpc/get_meta.go index 03b4861a19..69df0123a3 100644 --- a/src/apps/chifra/pkg/rpc/get_meta.go +++ b/src/apps/chifra/pkg/rpc/get_meta.go @@ -105,3 +105,8 @@ func (m *MetaData) NextIndexHeight() base.Blknum { func (m *MetaData) ChainHeight() base.Blknum { return m.Latest } + +// StageHieght returns the highest block that's been staged +func (m *MetaData) StageHeight() base.Blknum { + return m.Staging +} diff --git a/src/apps/chifra/pkg/rpc/get_traces.go b/src/apps/chifra/pkg/rpc/get_traces.go index 80fc0cecb1..1f413e483f 100644 --- a/src/apps/chifra/pkg/rpc/get_traces.go +++ b/src/apps/chifra/pkg/rpc/get_traces.go @@ -27,7 +27,7 @@ func (conn *Connection) GetTracesByBlockNumber(bn uint64) ([]types.SimpleTrace, curTs := conn.GetBlockTimestamp(bn) var idx uint64 - // TODO: This could be loadTrace in the same way loadBlocks works + // TODO: This could be loadTrace in the same way load Blocks works var ret []types.SimpleTrace for _, rawTrace := range rawTraces { traceAction := types.SimpleTraceAction{ @@ -163,7 +163,7 @@ func (conn *Connection) GetTracesByTransactionHash(txHash string, transaction *t result.SetRaw(rawTrace.Result) } - // TODO: This could be loadTrace in the same way loadBlocks works + // TODO: This could be loadTrace in the same way load Blocks works trace := types.SimpleTrace{ Error: rawTrace.Error, BlockHash: base.HexToHash(rawTrace.BlockHash), @@ -222,7 +222,7 @@ func (conn *Connection) GetTracesByFilter(filter string) ([]types.SimpleTrace, e curTs := conn.GetBlockTimestamp(utils.MustParseUint(f.FromBlock)) var idx uint64 - // TODO: This could be loadTrace in the same way loadBlocks works + // TODO: This could be loadTrace in the same way load Blocks works for _, rawTrace := range rawTraces { // Note: This is needed because of a GoLang bug when taking the pointer of a loop variable rawTrace := rawTrace diff --git a/src/apps/chifra/pkg/rpc/get_transaction.go b/src/apps/chifra/pkg/rpc/get_transaction.go index c827d2c399..2ab84d18f1 100644 --- a/src/apps/chifra/pkg/rpc/get_transaction.go +++ b/src/apps/chifra/pkg/rpc/get_transaction.go @@ -74,12 +74,12 @@ func (conn *Connection) GetTransactionByAppearance(appearance *types.RawAppearan if tx, err = conn.GetTransactionPrefundByApp(appearance); err != nil { return nil, err } - } else if txid == 99999 || txid == 99997 || txid == 99996 { - if tx, err = conn.GetTransactionRewardByTypeAndApp(BLOCK_REWARD, appearance); err != nil { + } else if txid == types.BlockReward || txid == types.MisconfigReward || txid == types.ExternalReward { + if tx, err = conn.GetTransactionRewardByTypeAndApp(types.BlockReward, appearance); err != nil { return nil, err } - } else if txid == 99998 { - if tx, err = conn.GetTransactionRewardByTypeAndApp(UNCLE_REWARD, appearance); err != nil { + } else if txid == types.UncleReward { + if tx, err = conn.GetTransactionRewardByTypeAndApp(types.UncleReward, appearance); err != nil { return nil, err } } @@ -218,15 +218,6 @@ func (conn *Connection) GetTransactionPrefundByApp(appearance *types.RawAppearan return nil, errors.New("not found") } -type RewardType int - -const ( - BLOCK_REWARD RewardType = iota - NEPHEW_REWARD - UNCLE_REWARD - TXFEE_REWARD -) - // // This data was taken from Geth ./params/config.go // blknum_t hardForkBlock(const string_q& hf) { // if (hf == "byzantium") { @@ -249,14 +240,9 @@ const ( // return 0; // } -const ( - byzantiumBlock = uint64(4370000) - constantinopleBlock = uint64(7280000) -) - // TODO: This is not cross-chain correct -func (conn *Connection) GetTransactionRewardByTypeAndApp(rt RewardType, appearance *types.RawAppearance) (*types.SimpleTransaction, error) { +func (conn *Connection) GetTransactionRewardByTypeAndApp(rt base.Txnum, appearance *types.RawAppearance) (*types.SimpleTransaction, error) { if block, err := conn.GetBlockBodyByNumber(uint64(appearance.BlockNumber)); err != nil { return nil, err } else { @@ -272,7 +258,7 @@ func (conn *Connection) GetTransactionRewardByTypeAndApp(rt RewardType, appearan bn := uint64(appearance.BlockNumber) blockReward = conn.getBlockReward(bn) switch rt { - case BLOCK_REWARD: + case types.BlockReward: if block.Miner.Hex() == appearance.Address { sender = base.BlockRewardSender nUncles := len(uncles) @@ -288,7 +274,7 @@ func (conn *Connection) GetTransactionRewardByTypeAndApp(rt RewardType, appearan } else { blockReward = big.NewInt(0) } - case UNCLE_REWARD: + case types.UncleReward: for _, uncle := range uncles { if uncle.Miner.Hex() == appearance.Address { sender = base.UncleRewardSender @@ -302,7 +288,7 @@ func (conn *Connection) GetTransactionRewardByTypeAndApp(rt RewardType, appearan if block.Miner.Hex() == appearance.Address { sender = base.BlockRewardSender // if it's both, it's the block reward // The uncle miner may also have been the miner of the block - if minerTx, err := conn.GetTransactionRewardByTypeAndApp(BLOCK_REWARD, appearance); err != nil { + if minerTx, err := conn.GetTransactionRewardByTypeAndApp(types.BlockReward, appearance); err != nil { return nil, err } else { blockReward = &minerTx.Rewards.Block @@ -312,9 +298,9 @@ func (conn *Connection) GetTransactionRewardByTypeAndApp(rt RewardType, appearan } else { blockReward = big.NewInt(0) } - case NEPHEW_REWARD: + case types.NephewReward: fallthrough - case TXFEE_REWARD: + case types.TxFeeReward: fallthrough default: return nil, errors.New("invalid reward type") diff --git a/src/apps/chifra/pkg/rpc/get_uncle.go b/src/apps/chifra/pkg/rpc/get_uncle.go index ae9114f6c3..b82b42f967 100644 --- a/src/apps/chifra/pkg/rpc/get_uncle.go +++ b/src/apps/chifra/pkg/rpc/get_uncle.go @@ -18,7 +18,9 @@ import ( func (conn *Connection) GetUncleBodiesByNumber(bn uint64) ([]types.SimpleBlock[types.SimpleTransaction], error) { if count, err := conn.GetUnclesCountInBlock(bn); err != nil { return nil, err - } else if count > 0 { + } else if count == 0 { + return []types.SimpleBlock[types.SimpleTransaction]{}, nil + } else { ret := make([]types.SimpleBlock[types.SimpleTransaction], count) for i := uint64(0); i < count; i++ { method := "eth_getUncleByBlockNumberAndIndex" @@ -46,8 +48,6 @@ func (conn *Connection) GetUncleBodiesByNumber(bn uint64) ([]types.SimpleBlock[t } } return ret, nil - } else { - return []types.SimpleBlock[types.SimpleTransaction]{}, nil } } @@ -55,7 +55,9 @@ func (conn *Connection) GetUncleBodiesByNumber(bn uint64) ([]types.SimpleBlock[t func (conn *Connection) GetUnclesHashesByNumber(bn uint64) ([]base.Hash, error) { if count, err := conn.GetUnclesCountInBlock(bn); err != nil { return nil, err - } else if count > 0 { + } else if count == 0 { + return []base.Hash{}, nil + } else { ret := make([]base.Hash, count) for i := uint64(0); i < count; i++ { method := "eth_getUncleByBlockNumberAndIndex" @@ -72,8 +74,6 @@ func (conn *Connection) GetUnclesHashesByNumber(bn uint64) ([]base.Hash, error) } } return ret, nil - } else { - return []base.Hash{}, nil } } @@ -84,6 +84,8 @@ func (conn *Connection) GetUnclesCountInBlock(bn uint64) (uint64, error) { if count, err := query.Query[string](conn.Chain, method, params); err != nil { return 0, err + } else if count == nil || *count == "" { + return 0, nil } else { return strconv.ParseUint(fmt.Sprint(*count), 0, 64) } diff --git a/src/apps/chifra/pkg/rpc/get_withdrawal.go b/src/apps/chifra/pkg/rpc/get_withdrawal.go new file mode 100644 index 0000000000..eb1215a8f5 --- /dev/null +++ b/src/apps/chifra/pkg/rpc/get_withdrawal.go @@ -0,0 +1,51 @@ +// Copyright 2021 The TrueBlocks Authors. All rights reserved. +// Use of this source code is governed by a license that can +// be found in the LICENSE file. + +package rpc + +import ( + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/types" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/utils" +) + +// GetWithdrawalsByNumber returns all withdrawals in a block +func (conn *Connection) GetWithdrawalsByNumber(bn base.Blknum) ([]types.SimpleWithdrawal, error) { + if conn.StoreReadable() { + withdrawalGroup := &types.SimpleWithdrawalGroup{ + BlockNumber: bn, + TransactionIndex: utils.NOPOS, + } + if err := conn.Store.Read(withdrawalGroup, nil); err == nil { + return withdrawalGroup.Withdrawals, nil + } + } + + if withdrawals, ts, err := conn.getWithdrawalsSimple(bn); err != nil { + return withdrawals, err + } else { + if conn.StoreWritable() && conn.EnabledMap["withdrawals"] && base.IsFinal(conn.LatestBlockTimestamp, ts) { + withdrawalGroup := &types.SimpleWithdrawalGroup{ + Withdrawals: withdrawals, + BlockNumber: bn, + TransactionIndex: utils.NOPOS, + } + if err = conn.Store.Write(withdrawalGroup, nil); err != nil { + logger.Warn("Failed to write withdrawals to cache", err) + } + } + + return withdrawals, err + } +} + +// getWithdrawalsSimple fetches the withdrawals from a block +func (conn *Connection) getWithdrawalsSimple(bn base.Blknum) ([]types.SimpleWithdrawal, base.Timestamp, error) { + if block, err := conn.GetBlockHeaderByNumber(bn); err != nil { + return []types.SimpleWithdrawal{}, utils.NOPOSI, err + } else { + return []types.SimpleWithdrawal{}, block.Timestamp, nil + } +} diff --git a/src/apps/chifra/pkg/rpc/query/query.go b/src/apps/chifra/pkg/rpc/query/query.go index 99d9ccc878..980a20b652 100644 --- a/src/apps/chifra/pkg/rpc/query/query.go +++ b/src/apps/chifra/pkg/rpc/query/query.go @@ -6,7 +6,9 @@ import ( "fmt" "io" "net/http" + "os" "runtime" + "strings" "sync/atomic" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" @@ -50,7 +52,7 @@ func Query[T any](chain string, method string, params Params) (*T, error) { Params: params, } - provider, _ := config.GetRpcProvider(chain) + provider := config.GetChain(chain).RpcProvider err := FromRpc(provider, &payload, &response) if err != nil { return nil, err @@ -64,15 +66,40 @@ func Query[T any](chain string, method string, params Params) (*T, error) { var rpcCounter uint32 -// fromRpc Returns all traces for a given block. -func FromRpc(rpcProvider string, payload *Payload, ret interface{}) error { - type rpcPayload struct { - Jsonrpc string `json:"jsonrpc"` - Method string `json:"method"` - Params `json:"params"` - ID int `json:"id"` +type rpcPayload struct { + Jsonrpc string `json:"jsonrpc"` + Method string `json:"method"` + Params `json:"params"` + ID int `json:"id"` +} + +var devDebug = false +var devDebugMethod = "" + +func init() { + devDebugMethod = os.Getenv("TB_DEBUG_CURL") + devDebug = len(devDebugMethod) > 0 +} + +func debugCurl(payload rpcPayload, rpcProvider string) { + if !devDebug { + return + } + + if devDebugMethod != "true" && payload.Method != devDebugMethod { + return } + bytes, _ := json.MarshalIndent(payload, "", "") + payloadStr := strings.Replace(string(bytes), "\n", " ", -1) + var curlCmd = `curl -X POST -H "Content-Type: application/json" --data '[{payload}]' [{rpcProvider}]` + curlCmd = strings.Replace(curlCmd, "[{payload}]", payloadStr, -1) + curlCmd = strings.Replace(curlCmd, "[{rpcProvider}]", rpcProvider, -1) + fmt.Println(curlCmd) +} + +// fromRpc Returns all traces for a given block. +func FromRpc(rpcProvider string, payload *Payload, ret any) error { payloadToSend := rpcPayload{ Jsonrpc: "2.0", Method: payload.Method, @@ -80,6 +107,8 @@ func FromRpc(rpcProvider string, payload *Payload, ret interface{}) error { ID: int(atomic.AddUint32(&rpcCounter, 1)), } + debugCurl(payloadToSend, rpcProvider) + plBytes, err := json.Marshal(payloadToSend) if err != nil { return err @@ -113,7 +142,7 @@ func QuerySlice[T any](chain string, method string, params Params) ([]T, error) Params: params, } - provider, _ := config.GetRpcProvider(chain) + provider := config.GetChain(chain).RpcProvider err := FromRpc(provider, &payload, &response) if err != nil { return nil, err @@ -144,7 +173,7 @@ func QueryBatch[T any](chain string, batchPayload []BatchPayload) (map[string]*T payloads = append(payloads, *config.Payload) } - provider, _ := config.GetRpcProvider(chain) + provider := config.GetChain(chain).RpcProvider err := fromRpcBatch(provider, payloads, &response) if err != nil { return nil, err diff --git a/src/apps/chifra/pkg/rpc/state_integration_test.go b/src/apps/chifra/pkg/rpc/state_integration_test.go index 28bc0122ff..971009075e 100644 --- a/src/apps/chifra/pkg/rpc/state_integration_test.go +++ b/src/apps/chifra/pkg/rpc/state_integration_test.go @@ -4,6 +4,7 @@ package rpc import ( + "fmt" "math/big" "reflect" "strconv" @@ -146,7 +147,35 @@ func TestGetState(t *testing.T) { } if !reflect.DeepEqual(gotState, tt.wantState) { - t.Errorf("GetState() = %+v, want %+v", gotState, tt.wantState) + msg := "" + if gotState.AccountType != tt.wantState.AccountType { + msg += fmt.Sprintf("AccountType: %s -- %s\n", gotState.AccountType, tt.wantState.AccountType) + } + if gotState.Address != tt.wantState.Address { + msg += fmt.Sprintf("Address: %s -- %s\n", gotState.Address, tt.wantState.Address) + } + if gotState.Balance.String() != tt.wantState.Balance.String() { + msg += fmt.Sprintf("Balance: %s -- %s\n", gotState.Balance.String(), tt.wantState.Balance.String()) + } + if gotState.BlockNumber != tt.wantState.BlockNumber { + msg += fmt.Sprintf("BlockNumber: %d -- %d\n", gotState.BlockNumber, tt.wantState.BlockNumber) + } + if gotState.Code != tt.wantState.Code { + msg += fmt.Sprintf("Code: %s -- %s\n", gotState.Code, tt.wantState.Code) + } + if gotState.Deployed != tt.wantState.Deployed { + msg += fmt.Sprintf("Deployed: %d -- %d\n", gotState.Deployed, tt.wantState.Deployed) + } + if gotState.Nonce != tt.wantState.Nonce { + msg += fmt.Sprintf("Nonce: %d -- %d\n", gotState.Nonce, tt.wantState.Nonce) + } + if gotState.Proxy != tt.wantState.Proxy { + msg += fmt.Sprintf("Proxy: %s -- %s\n", gotState.Proxy, tt.wantState.Proxy) + } + if gotState.Timestamp != tt.wantState.Timestamp { + msg += fmt.Sprintf("Timestamp: %d -- %d\n", gotState.Timestamp, tt.wantState.Timestamp) + } + t.Errorf(fmt.Sprintf("Test %s\n", tt.name), msg) } }) } diff --git a/src/apps/chifra/pkg/tslib/append.go b/src/apps/chifra/pkg/tslib/append.go index f7b0eda1ae..7451128e3b 100644 --- a/src/apps/chifra/pkg/tslib/append.go +++ b/src/apps/chifra/pkg/tslib/append.go @@ -13,7 +13,7 @@ func Append(chain string, tsArray []TimestampRecord) error { tsFn := config.PathToIndex(chain) + "ts.bin" tmpPath := filepath.Join(config.PathToCache(chain), "tmp") if backupFn, err := file.MakeBackup(tmpPath, tsFn); err == nil { - DeCache(chain) + ClearCache(chain) defer func() { if file.FileExists(backupFn) { _ = os.Rename(backupFn, tsFn) diff --git a/src/apps/chifra/pkg/tslib/establish.go b/src/apps/chifra/pkg/tslib/establish.go index 6ec5708393..7a63e6dbf3 100644 --- a/src/apps/chifra/pkg/tslib/establish.go +++ b/src/apps/chifra/pkg/tslib/establish.go @@ -2,7 +2,6 @@ package tslib import ( "context" - "errors" "fmt" "io" "net/http" @@ -11,32 +10,35 @@ import ( "path/filepath" "time" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/colors" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/file" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/manifest" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/sigintTrap" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/unchained" ) -func EstablishTsFile(chain string) error { +func EstablishTsFile(chain string, publisher base.Address) error { tsPath := filepath.Join(config.PathToIndex(chain), "ts.bin") if file.FileExists(tsPath) { return nil } - cid, err := manifest.ReadUnchainedIndex(chain, "ts", unchained.GetPreferredPublisher()) + database := chain + "-ts" + cid, err := manifest.ReadUnchainedIndex(chain, publisher, database) if err != nil { return err + } else if len(cid) == 0 { + return fmt.Errorf("no record found in the Unchained Index for database %s from publisher %s", database, publisher.Hex()) } - return downloadCidToBinary(chain, cid, tsPath) + return downloadCidToBinary(chain, database, tsPath, cid) } -// downloadCidToBinary downloads a timestamp database file -func downloadCidToBinary(chain, cid, fileName string) error { - gatewayUrl := config.GetIpfsGateway(chain) +// downloadCidToBinary downloads a CID to a binary file +func downloadCidToBinary(chain, database, outputFn, cid string) error { + gatewayUrl := config.GetChain(chain).IpfsGateway url, err := url.Parse(gatewayUrl) if err != nil { @@ -46,20 +48,19 @@ func downloadCidToBinary(chain, cid, fileName string) error { logger.InfoTable("Chain:", chain) logger.InfoTable("Gateway:", gatewayUrl) - logger.InfoTable("CID:", cid) logger.InfoTable("URL:", url.String()) - msg := colors.Yellow + "Downloading timestamp file. This may take a moment..." + colors.Off - logger.Info(msg) + logger.InfoTable("CID:", cid) + logger.Info(fmt.Sprintf("%sDownloading database %s (%s). This may take a moment...%s", colors.Yellow, database, cid, colors.Off)) header, err := http.Head(url.String()) if err != nil { return err } if header.StatusCode != 200 { - return errors.New("received non-200 response code") + return fmt.Errorf("CID not found: %d status", header.StatusCode) } - if header.ContentLength <= file.FileSize(fileName) { + if header.ContentLength <= file.FileSize(outputFn) { // The file on disc is larger than the one we will download which means it has more // timestamps in it, so we don't download it. return nil @@ -72,18 +73,18 @@ func downloadCidToBinary(chain, cid, fileName string) error { } defer response.Body.Close() if response.StatusCode != 200 { - return errors.New("received non-200 response code") + return fmt.Errorf("CID not found: %d status", header.StatusCode) } - logger.Info(colors.Yellow + "Download complete. Saving timestamps..." + colors.Off) + logger.Info(fmt.Sprintf("%sDownloading complete %s (%s). Writing file...%s", colors.Yellow, database, cid, colors.Off)) userHitsCtrlC := false go func() { for { - if file.FileSize(fileName) >= header.ContentLength { + if file.FileSize(outputFn) >= header.ContentLength { break } - pct := int(float64(file.FileSize(fileName)) / float64(header.ContentLength) * 100) + pct := int(float64(file.FileSize(outputFn)) / float64(header.ContentLength) * 100) msg := colors.Yellow + fmt.Sprintf("Downloading timestamps. Please wait... %d%%", pct) + colors.Off if userHitsCtrlC { msg = colors.Yellow + fmt.Sprintf("Finishing work. please wait... %d%% ", pct) + colors.Off @@ -101,7 +102,7 @@ func downloadCidToBinary(chain, cid, fileName string) error { trapChannel := sigintTrap.Enable(ctx, cancel, cleanOnQuit) defer sigintTrap.Disable(trapChannel) - ff, err := os.Create(fileName) + ff, err := os.Create(outputFn) if err != nil { return err } diff --git a/src/apps/chifra/pkg/tslib/list.go b/src/apps/chifra/pkg/tslib/list.go index a6be724cf4..9c76af52aa 100644 --- a/src/apps/chifra/pkg/tslib/list.go +++ b/src/apps/chifra/pkg/tslib/list.go @@ -7,9 +7,11 @@ package tslib import ( "encoding/csv" "errors" + "fmt" "io" "os" "path/filepath" + "strings" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger" @@ -19,8 +21,8 @@ import ( // GetSpecials returns a chain-specific list of special block names and numbers func GetSpecials(chain string) (specials []types.SimpleNamedBlock, err error) { - specialsPath := filepath.Join(config.MustGetPathToChainConfig(chain), "specials.csv") - _, err = os.Stat(specialsPath) + path := filepath.Join(config.MustGetPathToChainConfig(chain), "specials.csv") + _, err = os.Stat(path) if err != nil { // It's okay if there are no specials for a certain chain if chain == "mainnet" { @@ -30,42 +32,16 @@ func GetSpecials(chain string) (specials []types.SimpleNamedBlock, err error) { return specials, nil } - file, err := os.OpenFile(specialsPath, os.O_RDONLY, 0) - if err != nil { - return - } - reader := csv.NewReader(file) - // component,block/epoch,name,timestamp,date,description - reader.FieldsPerRecord = 6 - - for { - record, err1 := reader.Read() - if err1 == io.EOF { - break - } - if err1 != nil { - return specials, err1 - } - if len(record) == 6 { - bn := utils.MustParseUint(record[1]) - name := record[2] - ts := utils.MustParseInt(record[3]) - if bn == 0 && name != "frontier" { - continue + if specials, err = readSpecials(path, 6); err != nil { + if errors.Is(err, csv.ErrFieldCount) { + if specials, err = readSpecials(path, 4); err != nil { + return specials, err } - s := types.SimpleNamedBlock{ - BlockNumber: bn, - Name: name, - Timestamp: ts, - Component: record[0], - Description: record[5], - } - specials = append(specials, s) } } if len(specials) == 0 { - err = errors.New("found no special blocks") + err = fmt.Errorf("found no special blocks in file %s", path) } return @@ -76,3 +52,65 @@ func IsSpecialBlock(chain, needle string) bool { _, err := FromNameToBn(chain, needle) return err == nil } + +func readSpecials(path string, nFields int) (specials []types.SimpleNamedBlock, err error) { + file, err := os.OpenFile(path, os.O_RDONLY, 0) + if err != nil { + return + } + + reader := csv.NewReader(file) + // block,name,timestamp,date + reader.FieldsPerRecord = nFields + for { + if record, err := reader.Read(); err == io.EOF { + break + + } else if err != nil { + return specials, err + + } else { + if nFields == len(record) { + locs := map[string]int{ + "bn": 1, + "name": 2, + "ts": 3, + "component": 0, + // "date": 4, // skipped since we have timestamp + "description": 5, + } + if nFields == 4 { + locs = map[string]int{ + "bn": 0, + "name": 1, + "ts": 2, + // "date": 3, // skipped since we have timestamp + } + } + if strings.Contains(record[locs["bn"]], "-") { + // before block zero + continue + } + s := types.SimpleNamedBlock{ + BlockNumber: utils.MustParseUint(record[locs["bn"]]), + Name: record[locs["name"]], + Timestamp: utils.MustParseInt(record[locs["ts"]]), + } + // is this the header? + if s.BlockNumber == 0 && s.Name == "name" { + continue + } + + if nFields == 6 { + s.Component = record[locs["component"]] + s.Description = record[locs["description"]] + } else { + s.Component = "execution" + } + + specials = append(specials, s) + } + } + } + return +} diff --git a/src/apps/chifra/pkg/tslib/repair.go b/src/apps/chifra/pkg/tslib/repair.go index f78f1a8da6..736e3d9982 100644 --- a/src/apps/chifra/pkg/tslib/repair.go +++ b/src/apps/chifra/pkg/tslib/repair.go @@ -29,7 +29,7 @@ func Repair(chain string, bn uint64) error { tmpPath := filepath.Join(config.PathToCache(chain), "tmp") if backupFn, err := file.MakeBackup(tmpPath, tsFn); err == nil { defer func() { - DeCache(chain) + ClearCache(chain) if file.FileExists(backupFn) { // If the backup file exists, something failed, so we replace the original file. _ = os.Rename(backupFn, tsFn) diff --git a/src/apps/chifra/pkg/tslib/truncate.go b/src/apps/chifra/pkg/tslib/truncate.go index 89591eb07c..022af262b2 100644 --- a/src/apps/chifra/pkg/tslib/truncate.go +++ b/src/apps/chifra/pkg/tslib/truncate.go @@ -34,7 +34,7 @@ func Truncate(chain string, maxBn uint64) error { tmpPath := filepath.Join(config.PathToCache(chain), "tmp") if backupFn, err := file.MakeBackup(tmpPath, tsFn); err == nil { defer func() { - DeCache(chain) + ClearCache(chain) if file.FileExists(backupFn) { // If the backup file exists, something failed, so we replace the original file. _ = os.Rename(backupFn, tsFn) diff --git a/src/apps/chifra/pkg/tslib/tsdb.go b/src/apps/chifra/pkg/tslib/tsdb.go index 12f0f2402b..a25c1e31cb 100644 --- a/src/apps/chifra/pkg/tslib/tsdb.go +++ b/src/apps/chifra/pkg/tslib/tsdb.go @@ -121,7 +121,7 @@ func FromTs(chain string, ts base.Timestamp) (*TimestampRecord, error) { return &perChainTimestamps[chain].memory[index], nil } -func DeCache(chain string) { +func ClearCache(chain string) { perChainTimestamps[chain] = TimestampDatabase{ loaded: false, count: 0, diff --git a/src/apps/chifra/pkg/tslib/tslib_integration_test.go b/src/apps/chifra/pkg/tslib/tslib_integration_test.go index b63f8b3443..cbb2fc9112 100644 --- a/src/apps/chifra/pkg/tslib/tslib_integration_test.go +++ b/src/apps/chifra/pkg/tslib/tslib_integration_test.go @@ -27,7 +27,7 @@ func TestLoadTimestampsPass(t *testing.T) { {name: "Block 1 Mil", bn: 1000000, ts: 1455404053, date: "2016-02-13 22:54:13"}, } - EstablishTsFile(utils.GetTestChain()) + EstablishTsFile(utils.GetTestChain(), base.GetTestPublisher()) for _, e := range expected { bn, err := FromTsToBn(utils.GetTestChain(), e.ts) if err != nil { diff --git a/src/apps/chifra/pkg/types/types_appearance.go b/src/apps/chifra/pkg/types/types_appearance.go index 0d350b60fb..548d0ed287 100644 --- a/src/apps/chifra/pkg/types/types_appearance.go +++ b/src/apps/chifra/pkg/types/types_appearance.go @@ -16,6 +16,19 @@ import ( "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/utils" ) +const ( + // below are stored in cache, do not change + BlockReward base.Txnum = 99999 // the largest five-digit decimal number + UncleReward base.Txnum = 99998 + MisconfigReward base.Txnum = 99997 + ExternalReward base.Txnum = 99996 + Withdrawal base.Txnum = 99995 + // above are stored in cache, do not change + NephewReward base.Txnum = 99994 + TxFeeReward base.Txnum = 99993 + EsInternalTx base.Txnum = 99992 +) + // EXISTING_CODE type RawAppearance struct { diff --git a/src/apps/chifra/pkg/types/types_block.go b/src/apps/chifra/pkg/types/types_block.go index fdfe27520c..271ad4ba39 100644 --- a/src/apps/chifra/pkg/types/types_block.go +++ b/src/apps/chifra/pkg/types/types_block.go @@ -23,45 +23,47 @@ import ( // EXISTING_CODE type RawBlock struct { - Author string `json:"author"` - BaseFeePerGas string `json:"baseFeePerGas"` - BlockNumber string `json:"number"` - Difficulty string `json:"difficulty"` - ExtraData string `json:"extraData"` - GasLimit string `json:"gasLimit"` - GasUsed string `json:"gasUsed"` - Hash string `json:"hash"` - LogsBloom string `json:"logsBloom"` - Miner string `json:"miner"` - MixHash string `json:"mixHash"` - Nonce string `json:"nonce"` - ParentHash string `json:"parentHash"` - ReceiptsRoot string `json:"receiptsRoot"` - Sha3Uncles string `json:"sha3Uncles"` - Size string `json:"size"` - StateRoot string `json:"stateRoot"` - Timestamp string `json:"timestamp"` - TotalDifficulty string `json:"totalDifficulty"` - Transactions []any `json:"transactions"` - TransactionsRoot string `json:"transactionsRoot"` - Uncles []string `json:"uncles"` + Author string `json:"author"` + BaseFeePerGas string `json:"baseFeePerGas"` + BlockNumber string `json:"number"` + Difficulty string `json:"difficulty"` + ExtraData string `json:"extraData"` + GasLimit string `json:"gasLimit"` + GasUsed string `json:"gasUsed"` + Hash string `json:"hash"` + LogsBloom string `json:"logsBloom"` + Miner string `json:"miner"` + MixHash string `json:"mixHash"` + Nonce string `json:"nonce"` + ParentHash string `json:"parentHash"` + ReceiptsRoot string `json:"receiptsRoot"` + Sha3Uncles string `json:"sha3Uncles"` + Size string `json:"size"` + StateRoot string `json:"stateRoot"` + Timestamp string `json:"timestamp"` + TotalDifficulty string `json:"totalDifficulty"` + Transactions []any `json:"transactions"` + TransactionsRoot string `json:"transactionsRoot"` + Uncles []string `json:"uncles"` + Withdrawals []RawWithdrawal `json:"withdrawals"` // EXISTING_CODE // EXISTING_CODE } type SimpleBlock[Tx string | SimpleTransaction] struct { - BaseFeePerGas base.Wei `json:"baseFeePerGas"` - BlockNumber base.Blknum `json:"blockNumber"` - Difficulty uint64 `json:"difficulty"` - GasLimit base.Gas `json:"gasLimit"` - GasUsed base.Gas `json:"gasUsed"` - Hash base.Hash `json:"hash"` - Miner base.Address `json:"miner"` - ParentHash base.Hash `json:"parentHash"` - Timestamp base.Timestamp `json:"timestamp"` - Transactions []Tx `json:"transactions"` - Uncles []base.Hash `json:"uncles,omitempty"` - raw *RawBlock `json:"-"` + BaseFeePerGas base.Wei `json:"baseFeePerGas"` + BlockNumber base.Blknum `json:"blockNumber"` + Difficulty uint64 `json:"difficulty"` + GasLimit base.Gas `json:"gasLimit"` + GasUsed base.Gas `json:"gasUsed"` + Hash base.Hash `json:"hash"` + Miner base.Address `json:"miner"` + ParentHash base.Hash `json:"parentHash"` + Timestamp base.Timestamp `json:"timestamp"` + Transactions []Tx `json:"transactions"` + Uncles []base.Hash `json:"uncles,omitempty"` + Withdrawals []SimpleWithdrawal `json:"withdrawals,omitempty"` + raw *RawBlock `json:"-"` // EXISTING_CODE // EXISTING_CODE } @@ -79,19 +81,6 @@ func (s *SimpleBlock[Tx]) Model(verbose bool, format string, extraOptions map[st var order = []string{} // EXISTING_CODE - if extraOptions["count"] == true { - return Model{ - Data: map[string]interface{}{ - "blockNumber": s.BlockNumber, - "transactionsCnt": len(s.Transactions), - }, - Order: []string{ - "blockNumber", - "transactionsCnt", - }, - } - } - if extraOptions["hashes"] == true { txHashes := make([]string, 0, len(s.Transactions)) // Check what type Tx is @@ -284,6 +273,16 @@ func (s *SimpleBlock[Tx]) MarshalCache(writer io.Writer) (err error) { return err } + // Withdrawals + withdrawals := make([]cache.Marshaler, 0, len(s.Withdrawals)) + for _, withdrawal := range s.Withdrawals { + withdrawal := withdrawal + withdrawals = append(withdrawals, &withdrawal) + } + if err = cache.WriteValue(writer, withdrawals); err != nil { + return err + } + return nil } @@ -345,6 +344,12 @@ func (s *SimpleBlock[string]) UnmarshalCache(version uint64, reader io.Reader) ( return err } + // Withdrawals + s.Withdrawals = make([]SimpleWithdrawal, 0) + if err = cache.ReadValue(reader, &s.Withdrawals, version); err != nil { + return err + } + s.FinishUnmarshal() return nil diff --git a/src/apps/chifra/pkg/types/types_chunkrecord.go b/src/apps/chifra/pkg/types/types_chunkrecord.go index dc224f9d00..2401ac2881 100644 --- a/src/apps/chifra/pkg/types/types_chunkrecord.go +++ b/src/apps/chifra/pkg/types/types_chunkrecord.go @@ -10,6 +10,8 @@ package types // EXISTING_CODE import ( + "encoding/json" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base" ) @@ -72,4 +74,9 @@ func (s *SimpleChunkRecord) Model(verbose bool, format string, extraOptions map[ } // EXISTING_CODE +func (s *SimpleChunkRecord) String() string { + bytes, _ := json.MarshalIndent(s, "", " ") + return string(bytes) +} + // EXISTING_CODE diff --git a/src/apps/chifra/pkg/types/types_name.go b/src/apps/chifra/pkg/types/types_name.go index a9ea3505cf..5319ad5fc3 100644 --- a/src/apps/chifra/pkg/types/types_name.go +++ b/src/apps/chifra/pkg/types/types_name.go @@ -104,7 +104,7 @@ func (s *SimpleName) Model(verbose bool, format string, extraOptions map[string] "petname", } - if len(s.Address.Bytes()) > 0 && s.Address != base.HexToAddress("0x0") { + if len(s.Address.Bytes()) > 0 && s.Address != base.ZeroAddr { model["address"] = strings.ToLower(s.Address.String()) } diff --git a/src/apps/chifra/pkg/types/types_namedblock.go b/src/apps/chifra/pkg/types/types_namedblock.go index 8234d1a520..d3de571866 100644 --- a/src/apps/chifra/pkg/types/types_namedblock.go +++ b/src/apps/chifra/pkg/types/types_namedblock.go @@ -81,7 +81,7 @@ func (s *SimpleNamedBlock) Model(verbose bool, format string, extraOptions map[s if verbose { model["component"] = s.Component - order = append([]string{"component"}, order...) + order = append(order, "component") model["description"] = s.Description order = append(order, "description") diff --git a/src/apps/chifra/pkg/types/types_slurp.go b/src/apps/chifra/pkg/types/types_slurp.go index bb101caef9..e3b362d348 100644 --- a/src/apps/chifra/pkg/types/types_slurp.go +++ b/src/apps/chifra/pkg/types/types_slurp.go @@ -154,7 +154,7 @@ func (s *SimpleSlurp) Model(verbose bool, format string, extraOptions map[string if s.BlockHash != base.HexToHash("0xdeadbeef") && !s.BlockHash.IsZero() { model["blockHash"] = s.BlockHash } - if s.TransactionIndex != 80809 { + if s.TransactionIndex != EsInternalTx { model["transactionIndex"] = s.TransactionIndex } diff --git a/src/apps/chifra/pkg/types/types_transaction.go b/src/apps/chifra/pkg/types/types_transaction.go index 6814a942da..c9168bbdb6 100644 --- a/src/apps/chifra/pkg/types/types_transaction.go +++ b/src/apps/chifra/pkg/types/types_transaction.go @@ -206,12 +206,9 @@ func (s *SimpleTransaction) Model(verbose bool, format string, extraOptions map[ if s.Receipt != nil && !s.Receipt.IsDefault() { contractAddress := s.Receipt.ContractAddress.Hex() - // TODO: this should not be hardcoded here. We have tslib.GetSpecials(), but there - // TODO: are 2 issues with it: 1. circular dependency with types package, 2. every - // TODO: call to GetSpecials parses CSV file, so we need to call it once and cache - byzantiumBlock := uint64(4370000) + // TODO: This is quite odd. Why? status := &s.Receipt.Status - if s.BlockNumber < byzantiumBlock || *status == 4294967295-1 { + if s.BlockNumber < base.ByzantiumBlock || *status == 4294967295-1 { status = nil } diff --git a/src/apps/chifra/pkg/types/types_transaction_test.go b/src/apps/chifra/pkg/types/types_transaction_test.go index 60bb9d7b93..0695676465 100644 --- a/src/apps/chifra/pkg/types/types_transaction_test.go +++ b/src/apps/chifra/pkg/types/types_transaction_test.go @@ -22,7 +22,7 @@ func TestTransactionCache(t *testing.T) { Input: "0x", Nonce: 2377519, Receipt: &SimpleReceipt{ - ContractAddress: base.HexToAddress("0x0"), + ContractAddress: base.ZeroAddr, EffectiveGasPrice: 21000000000, GasUsed: 22966, Logs: []SimpleLog{ @@ -93,7 +93,7 @@ func TestTransactionCacheArticulated(t *testing.T) { Input: "0x1fee5cd200000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000a7365706f6c69612d747300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002e516d5441623866625233674d6f777876624e466f584339337a6762386f66395567756d3363554b564366414e3172000000000000000000000000000000000000", Nonce: 551, Receipt: &SimpleReceipt{ - ContractAddress: base.HexToAddress("0x0"), + ContractAddress: base.ZeroAddr, EffectiveGasPrice: 0, GasUsed: 41738, Logs: []SimpleLog{ diff --git a/src/apps/chifra/pkg/types/types_withdrawal.go b/src/apps/chifra/pkg/types/types_withdrawal.go index 30f9eeb631..6e3420f376 100644 --- a/src/apps/chifra/pkg/types/types_withdrawal.go +++ b/src/apps/chifra/pkg/types/types_withdrawal.go @@ -53,6 +53,19 @@ func (s *SimpleWithdrawal) Model(verbose bool, format string, extraOptions map[s var order = []string{} // EXISTING_CODE + model = map[string]interface{}{ + "address": s.Address, + "amount": s.Amount, + "index": s.Index, + "validatorIndex": s.ValidatorIndex, + } + + order = []string{ + "address", + "amount", + "index", + "validatorIndex", + } // EXISTING_CODE return Model{ diff --git a/src/apps/chifra/pkg/uniq/uniq_appearances_details.go b/src/apps/chifra/pkg/uniq/uniq_appearances_details.go index 237ed19e98..810cc9774b 100644 --- a/src/apps/chifra/pkg/uniq/uniq_appearances_details.go +++ b/src/apps/chifra/pkg/uniq/uniq_appearances_details.go @@ -36,12 +36,12 @@ func GetUniqAddressesInBlock(chain, flow string, conn *rpc.Connection, procFunc return err } else { miner := block.Miner.Hex() - txid := uint64(99999) + txid := types.BlockReward if block.Miner.IsZero() { // Early clients allowed misconfigured miner settings with address 0x0 (reward got // burned). We enter a false record with a false tx_id to account for this. - miner = "0xdeaddeaddeaddeaddeaddeaddeaddeaddeaddead" - txid = 99997 + miner = base.SentinalAddr.Hex() + txid = types.MisconfigReward } streamAppearance(procFunc, flow, "miner", miner, bn, txid, utils.NOPOS, ts, addrMap) @@ -50,12 +50,12 @@ func GetUniqAddressesInBlock(chain, flow string, conn *rpc.Connection, procFunc } else { for _, uncle := range uncles { unc := uncle.Miner.Hex() - txid = uint64(99998) + txid = types.UncleReward if uncle.Miner.IsZero() { // Early clients allowed misconfigured miner settings with address 0x0 (reward got // burned). We enter a false record with a false tx_id to account for this. - unc = "0xdeaddeaddeaddeaddeaddeaddeaddeaddeaddead" - txid = 99998 // do not change this! + unc = base.SentinalAddr.Hex() + txid = types.UncleReward // do not change this! } streamAppearance(procFunc, flow, "uncle", unc, bn, txid, utils.NOPOS, ts, addrMap) } @@ -201,32 +201,32 @@ func uniqFromTracesDetails(chain string, procFunc UniqProcFunc, flow string, tra } else if trace.TraceType == "reward" { if trace.Action.RewardType == "block" { author := trace.Action.Author.Hex() - falseTxid := uint64(99999) + falseTxid := types.BlockReward // Early clients allowed misconfigured miner settings with address 0x0 (reward got // burned). We enter a false record with a false tx_id to account for this. a := base.HexToAddress(author) if a.IsZero() { - author = "0xdeaddeaddeaddeaddeaddeaddeaddeaddeaddead" - falseTxid = uint64(99997) + author = base.SentinalAddr.Hex() + falseTxid = types.MisconfigReward } streamAppearance(procFunc, flow, "miner", author, bn, falseTxid, traceid, ts, addrMap) } else if trace.Action.RewardType == "uncle" { author := trace.Action.Author.Hex() - falseTxid := uint64(99998) + falseTxid := types.UncleReward // Early clients allowed misconfigured miner settings with address 0x0 (reward got // burned). We enter a false record with a false tx_id to account for this. a := base.HexToAddress(author) if a.IsZero() { - author = "0xdeaddeaddeaddeaddeaddeaddeaddeaddeaddead" - falseTxid = uint64(99998) // do not change! it will break the index + author = base.SentinalAddr.Hex() + falseTxid = types.UncleReward // do not change! it will break the index } streamAppearance(procFunc, flow, "uncle", author, bn, falseTxid, traceid, ts, addrMap) } else if trace.Action.RewardType == "external" { // This only happens in xDai as far as we know... author := trace.Action.Author.Hex() - falseTxid := uint64(99996) + falseTxid := types.ExternalReward streamAppearance(procFunc, flow, "external", author, bn, falseTxid, traceid, ts, addrMap) } else { diff --git a/src/apps/chifra/pkg/validate/block_integration_test.go b/src/apps/chifra/pkg/validate/block_integration_test.go index e86802ca77..c2f021dcef 100644 --- a/src/apps/chifra/pkg/validate/block_integration_test.go +++ b/src/apps/chifra/pkg/validate/block_integration_test.go @@ -8,6 +8,7 @@ import ( "strings" "testing" + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/colors" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/identifiers" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/tslib" @@ -15,7 +16,7 @@ import ( ) func Test_BlockRanges(t *testing.T) { - tslib.EstablishTsFile(utils.GetTestChain()) + tslib.EstablishTsFile(utils.GetTestChain(), base.GetTestPublisher()) for _, item := range testBlocks { if !item.enabled { continue diff --git a/src/apps/chifra/pkg/version/version.go b/src/apps/chifra/pkg/version/version.go index b09a63bbb6..6ee84f5d7e 100644 --- a/src/apps/chifra/pkg/version/version.go +++ b/src/apps/chifra/pkg/version/version.go @@ -5,8 +5,10 @@ package version import ( - "strconv" + "fmt" "strings" + + "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/utils" ) type Version struct { @@ -16,49 +18,40 @@ type Version struct { Aspect string `json:"aspect"` } -func NewVersion(str string) (vers Version, err error) { - if str[:3] == "GHC" { - // First, remove the part that cannot be parsed with ParseInt - str = str[len("GHC-TrueBlocks//"):] - } +func NewVersion(str string) Version { + str = strings.Replace(str, "GHC-TrueBlocks//", "", -1) str = strings.Replace(strings.Replace(str, "-", ".", -1), "v", "", -1) parts := strings.Split(str, ".") + + var vers Version if len(parts) > 0 { - if vers.Major, err = strconv.ParseInt(parts[0], 10, 32); err != nil { - return vers, err - } + vers.Major = utils.MustParseInt(parts[0]) } if len(parts) > 1 { - if vers.Minor, err = strconv.ParseInt(parts[1], 10, 32); err != nil { - return vers, err - } + vers.Minor = utils.MustParseInt(parts[1]) } + if len(parts) > 2 { - if vers.Build, err = strconv.ParseInt(parts[2], 10, 32); err != nil { - return vers, err - } + vers.Build = utils.MustParseInt(parts[2]) } + if len(parts) > 3 { vers.Aspect = parts[3] } - return vers, nil -} - -func (ref *Version) IsEarlierThan(test Version) bool { - if ref.Major < test.Major { - return true - } - if ref.Minor < test.Minor { - return true - } - if ref.Build < test.Build { - return true - } - return false + return vers } // Uint64 returns version as a single uint64 func (ref *Version) Uint64() uint64 { return uint64((ref.Major * 1000000) + (ref.Minor * 1000) + ref.Build) } + +// String prints the version to a string +func (ref *Version) String() string { + return fmt.Sprintf("v%d.%d.%d-%s", ref.Major, ref.Minor, ref.Build, ref.Aspect) +} + +func VersionString() string { + return strings.Replace(LibraryVersion, "GHC-TrueBlocks//", "v", -1) +} diff --git a/src/apps/chifra/pkg/version/version_test.go b/src/apps/chifra/pkg/version/version_test.go index 6c52d9589b..bb4d9e931f 100644 --- a/src/apps/chifra/pkg/version/version_test.go +++ b/src/apps/chifra/pkg/version/version_test.go @@ -23,11 +23,11 @@ func TestVersion(t *testing.T) { Aspect: "", } - if !early.IsEarlierThan(late) { + if early.Uint64() >= late.Uint64() { t.Error("early is not earlier than late") } - if late.IsEarlierThan(early) { + if late.Uint64() < early.Uint64() { t.Error("late is earlier than early") } } @@ -40,7 +40,6 @@ func TestNewVersion(t *testing.T) { name string args args wantVers Version - wantErr bool }{ { name: "library version string", @@ -54,14 +53,22 @@ func TestNewVersion(t *testing.T) { Aspect: "beta", }, }, + { + name: "version 1.0.0", + args: args{ + str: "v1.0.0-release", + }, + wantVers: Version{ + Major: 1, + Minor: 0, + Build: 0, + Aspect: "release", + }, + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - gotVers, err := NewVersion(tt.args.str) - if (err != nil) != tt.wantErr { - t.Errorf("NewVersion() error = %v, wantErr %v", err, tt.wantErr) - return - } + gotVers := NewVersion(tt.args.str) if !reflect.DeepEqual(gotVers, tt.wantVers) { t.Errorf("NewVersion() = %v, want %v", gotVers, tt.wantVers) } diff --git a/src/apps/chifra/pkg/walk/walk.go b/src/apps/chifra/pkg/walk/walk.go index f8c93eab69..4791d6ae73 100644 --- a/src/apps/chifra/pkg/walk/walk.go +++ b/src/apps/chifra/pkg/walk/walk.go @@ -43,6 +43,8 @@ const ( Index_Staging Index_Unripe Index_Maps + + Config ) var cacheTypeToName = map[CacheType]string{ @@ -66,6 +68,7 @@ var cacheTypeToName = map[CacheType]string{ Index_Staging: "staging", Index_Unripe: "unripe", Index_Maps: "neighbors", + Config: "config", } // CacheTypeToFolder is a map of cache types to the folder name (also, it acts as the mode in chifra status) @@ -90,6 +93,7 @@ var CacheTypeToFolder = map[CacheType]string{ Index_Staging: "staging", Index_Unripe: "unripe", Index_Maps: "maps", + Config: "config", } var cacheTypeToExt = map[CacheType]string{ @@ -113,18 +117,11 @@ var cacheTypeToExt = map[CacheType]string{ Index_Staging: "txt", Index_Unripe: "txt", Index_Maps: "bin", + Config: "toml", } func (ct CacheType) String() string { return cacheTypeToName[ct] - /* - 207: func (s *SimpleBlock[Tx]) CacheLocation() (directory string, extension string) { - 171: func (s *SimpleLogGroup) CacheLocation() (directory string, extension string) { - 206: func (s *SimpleSlurpGroup) CacheLocation() (directory string, extension string) { - 200: func (s *SimpleStatementGroup) CacheLocation() (directory string, extension string) { - 216: func (s *SimpleTraceGroup) CacheLocation() (directory string, extension string) { - 322: func (s *SimpleTransaction) CacheLocation() (directory string, extension string) { - */ } func IsCacheType(path string, cT CacheType, checkExt bool) bool { @@ -179,6 +176,8 @@ func GetRootPathFromCacheType(chain string, cacheType CacheType) string { fallthrough case Index_Maps: return filepath.Join(config.PathToIndex(chain), CacheTypeToFolder[cacheType]) + "/" + case Config: + return config.PathToRootConfig() case Cache_NotACache: fallthrough default: @@ -194,6 +193,11 @@ func WalkCacheFolder(ctx context.Context, chain string, cacheType CacheType, dat walkFolder(ctx, path, cacheType, data, filenameChan) } +func WalkConfigFolders(ctx context.Context, data interface{}, filenameChan chan<- CacheFileInfo) { + path := config.PathToRootConfig() + walkFolder(ctx, path, Config, data, filenameChan) +} + func walkFolder(ctx context.Context, path string, cacheType CacheType, data interface{}, filenameChan chan<- CacheFileInfo) { defer func() { filenameChan <- CacheFileInfo{Type: Cache_NotACache} diff --git a/src/apps/chifra/proto/client.go b/src/apps/chifra/proto/client.go index 96a64c9e4e..d45c4038ac 100644 --- a/src/apps/chifra/proto/client.go +++ b/src/apps/chifra/proto/client.go @@ -7,13 +7,14 @@ import ( "path" "time" - "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config" "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/file" "google.golang.org/grpc" "google.golang.org/grpc/credentials/insecure" ) -const udsDefaultTimeout = 5 * time.Millisecond +// TODO: we can put this config item back if we need it. Put it in the +// settings group not its own group +// const udsDefaultTimeout = 5 * time.Millisecond var ErrServerNotRunning = errors.New("gRPC server not running") @@ -22,10 +23,12 @@ func SocketAddress() string { } func GetContext() (context.Context, context.CancelFunc) { - timeout := config.GetRootConfig().Grpc.UdsTimeout - if timeout == 0 { - timeout = udsDefaultTimeout - } + // TODO: we can put this config item back if we need it. Put it in the + // settings group not its own group + timeout := 5 * time.Millisecond // config.GetRootConfig().Grpc.UdsTimeout + // if timeout == 0 { + // timeout = udsDefaultTimeout + // } return context.WithTimeout(context.Background(), timeout) } diff --git a/src/cmd-line-options.csv b/src/cmd-line-options.csv index 3b2ca47d55..1980477b02 100644 --- a/src/cmd-line-options.csv +++ b/src/cmd-line-options.csv @@ -50,11 +50,11 @@ num,group,tags,api_route,tool,longName,hotKey,def_val,is_required,is_customizabl 11288,apps,Accounts,monitors,acctExport,watchlist,a,,false,false,true,true,gocmd,flag,,available with --watch option only, a file containing the addresses to watch 11292,apps,Accounts,monitors,acctExport,commands,c,,false,false,true,true,gocmd,flag,,available with --watch option only, the file containing the list of commands to apply to each watched address 11294,apps,Accounts,monitors,acctExport,batch_size,b,8,false,false,true,true,gocmd,flag,,available with --watch option only, the number of monitors to process in each batch -11296,apps,Accounts,monitors,acctExport,run_once,r,,false,false,true,true,gocmd,switch,,available with --watch option only, only run the monitor --watch commands once then quit +11296,apps,Accounts,monitors,acctExport,run_count,u,0,false,false,false,false,gocmd,flag,,available with --watch option only, run the monitor this many times, then quit 11298,apps,Accounts,monitors,acctExport,sleep,s,14,false,false,true,true,gocmd,flag,,available with --watch option only, the number of seconds to sleep between runs 11094,apps,Accounts,monitors,acctExport,,,,false,false,true,true,--,description,,Add, remove, clean, and list address monitors. 11095,apps,Accounts,monitors,acctExport,n1,,,false,false,false,false,--,note,,An `address` must be either an ENS name or start with '0x' and be forty-two characters long. -11096,apps,Accounts,monitors,acctExport,n2,,,false,false,false,false,--,note,,If no address is presented to the --clean command, all existing monitors are be cleaned. +11096,apps,Accounts,monitors,acctExport,n2,,,false,false,false,false,--,note,,If no address is presented to the --clean command, all existing monitors will be cleaned. 11096,apps,Accounts,monitors,acctExport,n3,,,false,false,false,false,--,note,,The --watch option requires two additional parameters to be specified: `--watchlist` and `--commands`. 11097,apps,Accounts,monitors,acctExport,n4,,,false,false,false,false,--,note,,Addresses provided on the command line are ignored in `--watch` mode. 11098,apps,Accounts,monitors,acctExport,n5,,,false,false,false,false,--,note,,Providing the value `existing` to the `--watchlist` monitors all existing monitor files (see --list). @@ -68,6 +68,7 @@ num,group,tags,api_route,tool,longName,hotKey,def_val,is_required,is_customizabl 12035,apps,Accounts,list,acctExport,first_record,c,0,false,false,true,true,gocmd,flag,,the first record to process 12040,apps,Accounts,list,acctExport,max_records,e,250,false,false,true,true,gocmd,flag,,the maximum number of records to process 12043,apps,Accounts,list,acctExport,reversed,E,,false,false,true,true,gocmd,switch,,produce results in reverse chronological order +12044,apps,Accounts,list,acctExport,publisher,P,trueblocks.eth,false,false,false,false,gocmd,flag,
,for some query options, the publisher of the index 12045,apps,Accounts,list,acctExport,first_block,F,0,false,false,true,true,gocmd,flag,,first block to export (inclusive, ignored when freshening) 12050,apps,Accounts,list,acctExport,last_block,L,NOPOS,false,false,true,true,gocmd,flag,,last block to export (inclusive, ignored when freshening) 12055,apps,Accounts,list,acctExport,,,,false,false,true,true,--,description,,List every appearance of an address anywhere on the chain. @@ -223,6 +224,9 @@ num,group,tags,api_route,tool,longName,hotKey,def_val,is_required,is_customizabl 12132,apps,Admin,scrape,blockScrape,remote,r,,false,false,true,true,gocmd,switch,,pin new chunks to the gateway (requires pinning service keys) 12115,apps,Admin,scrape,blockScrape,sleep,s,14,false,false,true,true,gocmd,flag,,seconds to sleep between scraper passes 12185,apps,Admin,scrape,blockScrape,start_block,l,0,false,false,true,true,gocmd,flag,,first block to visit when scraping (snapped back to most recent snap_to_grid mark) +12187,apps,Admin,scrape,blockScrape,run_count,u,0,false,false,false,false,gocmd,flag,,run the scraper this many times, then quit +12188,apps,Admin,scrape,blockScrape,publisher,P,trueblocks.eth,false,false,false,false,gocmd,flag,
,for some query options, the publisher of the index +12189,apps,Admin,scrape,blockScrape,dry_run,d,,false,false,false,false,gocmd,switch,,show the configuration that would be applied if run,no changes are made 12160,apps,Admin,scrape,blockScrape,apps_per_chunk,,200000,false,false,false,false,config,flag,,the number of appearances to build into a chunk before consolidating it 12170,apps,Admin,scrape,blockScrape,snap_to_grid,,100000,false,false,false,false,config,flag,,an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index 12175,apps,Admin,scrape,blockScrape,first_snap,,0,false,false,false,false,config,flag,,the first block at which snap_to_grid is enabled @@ -258,9 +262,11 @@ num,group,tags,api_route,tool,longName,hotKey,def_val,is_required,is_customizabl 31920,apps,Admin,chunks,chunkMan,check,c,,false,false,true,true,gocmd,switch,,check the manifest, index, or blooms for internal consistency 31935,apps,Admin,chunks,chunkMan,pin,i,,false,false,true,true,gocmd,switch,,pin the manifest or each index chunk and bloom 31940,apps,Admin,chunks,chunkMan,publish,p,,false,false,true,true,gocmd,switch,,publish the manifest to the Unchained Index smart contract +31942,apps,Admin,chunks,chunkMan,publisher,P,trueblocks.eth,false,false,false,false,gocmd,flag,
,for some query options, the publisher of the index 31925,apps,Admin,chunks,chunkMan,truncate,n,,false,false,false,false,gocmd,flag,,truncate the entire index at this block (requires a block identifier) 31945,apps,Admin,chunks,chunkMan,remote,r,,false,false,true,true,gocmd,switch,,prior to processing, retreive the manifest from the Unchained Index smart contract 31950,apps,Admin,chunks,chunkMan,belongs,b,,false,false,true,true,gocmd,flag,list,in index mode only, checks the address(es) for inclusion in the given index chunk +31950,apps,Admin,chunks,chunkMan,diff,f,,false,false,false,false,gocmd,switch,,compare two index portions (see notes) 31951,apps,Admin,chunks,chunkMan,first_block,F,0,false,false,true,true,gocmd,flag,,first block to process (inclusive) 31953,apps,Admin,chunks,chunkMan,last_block,L,NOPOS,false,false,true,true,gocmd,flag,,last block to process (inclusive) 31955,apps,Admin,chunks,chunkMan,max_addrs,m,NOPOS,false,false,true,true,gocmd,flag,,the max number of addresses to process in a given chunk @@ -275,9 +281,11 @@ num,group,tags,api_route,tool,longName,hotKey,def_val,is_required,is_customizabl 31980,apps,Admin,chunks,chunkMan,n5,,,false,false,false,false,--,note,,The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. 31990,apps,Admin,chunks,chunkMan,n7,,,false,false,false,false,--,note,,The --pin option requires a locally running IPFS node or a pinning service API key. 31995,apps,Admin,chunks,chunkMan,n8,,,false,false,false,false,--,note,,The --publish option requires a private key. +31997,apps,Admin,chunks,chunkMan,n9,,,false,false,false,false,--,note,,The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. 11905,apps,Admin,init,init,all,a,,false,false,true,true,gocmd,switch,,in addition to Bloom filters, download full index chunks (recommended) -11910,apps,Admin,init,init,dry_run,d,,false,false,true,true,gocmd,switch,,display the results of the download without actually downloading +11907,apps,Admin,init,init,dry_run,d,,false,false,true,true,gocmd,switch,,display the results of the download without actually downloading +11910,apps,Admin,init,init,publisher,P,trueblocks.eth,false,false,false,false,gocmd,flag,
,the publisher of the index to download 11915,apps,Admin,init,init,first_block,F,0,false,false,true,true,gocmd,flag,,do not download any chunks earlier than this block 11920,apps,Admin,init,init,sleep,s,0.0,false,false,true,true,gocmd,flag,,seconds to sleep between downloads 11925,apps,Admin,init,init,,,,false,false,true,true,--,description,,Initialize the TrueBlocks system by downloading the Unchained Index from IPFS. diff --git a/src/dev_tools/makeClass/commandoption.cpp b/src/dev_tools/makeClass/commandoption.cpp index c11124a87d..bde2353aeb 100644 --- a/src/dev_tools/makeClass/commandoption.cpp +++ b/src/dev_tools/makeClass/commandoption.cpp @@ -519,7 +519,7 @@ string_q nextCommandoptionChunk_custom(const string_q& fieldIn, const void* data case 'v': if (fieldIn % "variable") { if (com->isConfig) { - return firstUpper(com->Format("[{LOWER}]")); + return firstUpper(toCamelCase(com->longName)); } return substitute(toProper(com->longName), "_", ""); } diff --git a/src/dev_tools/makeClass/handle_datamodel_type.cpp b/src/dev_tools/makeClass/handle_datamodel_type.cpp index 47f5c58b71..04a0ef1b42 100644 --- a/src/dev_tools/makeClass/handle_datamodel_type.cpp +++ b/src/dev_tools/makeClass/handle_datamodel_type.cpp @@ -199,6 +199,9 @@ string_q specialCase(const CClassDefinition& model, const CMember& field, const } else if (name % "Logs") { ret = isRaw ? "[]RawLog" : "[]SimpleLog"; + } else if (name % "Withdrawals") { + ret = isRaw ? "[]RawWithdrawal" : "[]SimpleWithdrawal"; + } else if (name % "Action") { ret = isRaw ? "RawTraceAction" : "*SimpleTraceAction"; diff --git a/src/dev_tools/makeClass/handle_gocmds.cpp b/src/dev_tools/makeClass/handle_gocmds.cpp index befa8f1d87..42215039ab 100644 --- a/src/dev_tools/makeClass/handle_gocmds.cpp +++ b/src/dev_tools/makeClass/handle_gocmds.cpp @@ -18,7 +18,6 @@ extern string_q get_aliases(const CCommandOption& cmd); extern string_q get_optfields(const CCommandOption& cmd); extern string_q get_requestopts(const CCommandOption& cmd); extern string_q get_defaults_apis(const CCommandOption& cmd); -extern string_q get_config_override(const CCommandOption& cmd); extern string_q get_ens_convert1(const CCommandOption& cmd); extern string_q get_ens_convert2(const CCommandOption& cmd); extern string_q get_config_package(const CCommandOption& cmd); @@ -142,7 +141,6 @@ bool COptions::handle_gocmds_options(const CCommandOption& p) { replaceAll(source, "[{PROPER}]", toProper(p.api_route)); replaceAll(source, "[{OPT_FIELDS}]", get_optfields(p)); replaceAll(source, "[{DEFAULTS_API}]", get_defaults_apis(p)); - replaceAll(source, "[{CONFIG_OVERRIDE}]", get_config_override(p)); replaceAll(source, "[{ENS_CONVERT1}]", get_ens_convert1(p)); replaceAll(source, "[{ENS_CONVERT2}]", get_ens_convert2(p)); replaceAll(source, "[{CONFIGPKG}]", get_config_package(p)); @@ -384,6 +382,7 @@ string_q get_godefaults(const CCommandOption& cmd) { for (auto p : *((CCommandOptionArray*)cmd.members)) { if (!isDef(p)) { string_q val = substitute(p.def_val, "NOPOS", "utils.NOPOS"); + val = substitute(val, "trueblocks.eth", "\"trueblocks.eth\""); os << "\t" << padRight(p.Format("[{VARIABLE}]") + ": ", wid + 2, ' ') << val << "," << endl; } } @@ -408,9 +407,17 @@ string_q get_testlogs(const CCommandOption& cmd) { } else if (startsWith(p.data_type, "list<") || p.data_type == "" || p.data_type == "
" || contains(p.data_type, "enum")) { - const char* STR_TESTLOG_STRING = - "\tlogger.TestLog(len(opts.[{VARIABLE}]) > 0, \"[{VARIABLE}]: \", opts.[{VARIABLE}])"; - os << p.Format(STR_TESTLOG_STRING) << endl; + if (!p.def_val.empty() && p.def_val != "\"\"" && p.def_val != "utils.NOPOS") { + const char* STR_TESTLOG_STRING = + "\tlogger.TestLog(len(opts.[{VARIABLE}]) > 0 && opts.[{VARIABLE}] != \"[{DEF_VAL}]\", " + "\"[{VARIABLE}]: \", opts.[{VARIABLE}])"; + p.def_val = substitute(p.def_val, "\"", ""); + os << p.Format(STR_TESTLOG_STRING) << endl; + } else { + const char* STR_TESTLOG_STRING = + "\tlogger.TestLog(len(opts.[{VARIABLE}]) > 0, \"[{VARIABLE}]: \", opts.[{VARIABLE}])"; + os << p.Format(STR_TESTLOG_STRING) << endl; + } } else if (p.data_type == "" || p.data_type == "") { const char* STR_TESTLOG_UINT = @@ -440,17 +447,16 @@ string_q get_optfields(const CCommandOption& cmd) { string_q configDocs = getDocsPathTemplates(n); ::remove(configDocs.c_str()); // remove it if it exists, we will replace it - bool hasConfig = 0; size_t varWidth = 0, typeWidth = 0; for (auto p : *((CCommandOptionArray*)cmd.members)) { + string_q var = p.Format("[{VARIABLE}]"); if (p.generate == "config") { - string_q var = p.Format("Settings"); + var = toCamelCase(var); varWidth = max(var.length(), varWidth); - string_q type = cmd.Format("[{API_ROUTE}]Cfg.[{PROPER}]Settings"); + string_q type = cmd.Format("config.[{PROPER}]Settings"); typeWidth = max(type.length(), typeWidth); continue; } - string_q var = p.Format("[{VARIABLE}]"); varWidth = max(var.length(), varWidth); string_q type = p.Format("[{GO_INTYPE}]"); typeWidth = max(type.length(), typeWidth); @@ -479,6 +485,7 @@ string_q get_optfields(const CCommandOption& cmd) { os << "// " << firstUpper(c) << endl; ostringstream os; + bool hasConfig = 0; for (auto p : *((CCommandOptionArray*)cmd.members)) { if (p.generate == "config") { ostringstream dd; @@ -488,7 +495,7 @@ string_q get_optfields(const CCommandOption& cmd) { dd << "| " << string_q(18, '-') << " | " << string_q(12, '-') << " | " << string_q(12, '-') << " | --------- |" << endl; } - string_q x = substitute(p.Format("[{LONGNAME}]"), "_", "_"); + string_q x = toCamelCase(p.Format("[{LONGNAME}]")); dd << "| " << padRight(x, 18) << " | " << padRight(p.Format("[{GO_INTYPE}]"), 12) << " | " << padRight(p.Format("[{DEF_VAL}]"), 12) << " | " << p.Format("[{DESCRIPTION}]") << " |" << endl; appendToAsciiFile(configDocs, dd.str()); @@ -507,7 +514,7 @@ string_q get_optfields(const CCommandOption& cmd) { } if (hasConfig) { - string type = cmd.Format("[{API_ROUTE}]Cfg.[{PROPER}]Settings"); + string type = cmd.Format("config.[{PROPER}]Settings"); ONE(os, "Settings", varWidth, type, typeWidth, "Configuration items for the " + cmd.api_route); } @@ -544,22 +551,10 @@ string_q get_ens_convert2(const CCommandOption& cmd) { return os.str(); } -string_q get_config_override(const CCommandOption& cmd) { - for (auto p : *((CCommandOptionArray*)cmd.members)) - if (p.generate == "config") { - ostringstream os; - os << "\t" - << "opts.Settings, _ = " << cmd.api_route - << "Cfg.GetSettings(opts.Globals.Chain, configFn, &" + cmd.api_route + "Cfg.Unset)\n"; - return os.str(); - } - return ""; -} - string_q get_config_package(const CCommandOption& cmd) { for (auto p : *((CCommandOptionArray*)cmd.members)) if (p.generate == "config") - return "\t\"github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config/" + cmd.api_route + "Cfg\"\n"; + return "\t\"github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/config\"\n"; return ""; } @@ -587,6 +582,13 @@ string_q get_index_package(const string_q& fn) { } string_q get_defaults_apis(const CCommandOption& cmd) { + string_q last = ""; + for (auto p : *((CCommandOptionArray*)cmd.members)) { + if (p.generate == "config") { + last = p.longName; + } + } + ostringstream os; for (auto p : *((CCommandOptionArray*)cmd.members)) { if (p.isDeprecated) { @@ -608,11 +610,21 @@ string_q get_defaults_apis(const CCommandOption& cmd) { } os << p.Format(fmt) << endl; } + if (p.longName == last) { + os << "\tconfigs := make(map[string]string, 10)" << endl; + } } return os.str(); } string_q get_requestopts(const CCommandOption& cmd) { + string_q last = ""; + for (auto p : *((CCommandOptionArray*)cmd.members)) { + if (p.generate == "config") { + last = p.longName; + } + } + ostringstream os; for (auto p : *((CCommandOptionArray*)cmd.members)) { string_q low = toCamelCase(p.Format("[{LOWER}]")); @@ -623,9 +635,16 @@ string_q get_requestopts(const CCommandOption& cmd) { fmt = substitute(STR_REQUEST_CASE1, "++LOWER++", low); } if (p.generate == "config") { - fmt = substitute(fmt, "opts.", "opts.Settings."); + if (p.longName == last) { + os << "\t\tcase \"" + low + "\":" << endl; + os << "\t\t\tconfigs[key] = value[0]" << endl; + } else { + fmt = substitute(fmt, "opts.[{VARIABLE}] = [{ASSIGN}]", "fallthrough"); + os << p.Format(fmt) << endl; + } + } else { + os << p.Format(fmt) << endl; } - os << p.Format(fmt) << endl; } return os.str(); } @@ -640,7 +659,11 @@ string_q get_goDefault(const CCommandOption& p) { return p.def_val; return "0.0"; } else if (p.go_intype == "string") { - return p.def_val; + if (contains(p.def_val, ".eth")) { // an address + return "\"" + p.def_val + "\""; + } else { + return p.def_val; + } } else if (p.go_intype == "uint64") { if (contains(p.def_val, "NOPOS")) { return "0"; @@ -714,8 +737,12 @@ string_q get_setopts(const CCommandOption& cmd) { if (p.option_type != "positional") { os << "\t[{ROUTE}]Cmd.Flags()."; os << p.go_flagtype; - os << "(&[{ROUTE}]Pkg.GetOptions()." << (p.isConfig ? "Settings." : ""); - os << p.Format("[{VARIABLE}]") << ", "; + os << "(&[{ROUTE}]Pkg.GetOptions()."; + if (p.isConfig) { + os << "Settings." + p.Format("[{VARIABLE}]") << ", "; + } else { + os << p.Format("[{VARIABLE}]") << ", "; + } os << p.Format("\"[{LONGNAME}]\", "); os << p.Format("\"[{HOTKEY}]\", "); os << get_goDefault(p) << ", "; diff --git a/src/dev_tools/makeClass/handle_readmes.cpp b/src/dev_tools/makeClass/handle_readmes.cpp index 3151e277be..ed91dcca66 100644 --- a/src/dev_tools/makeClass/handle_readmes.cpp +++ b/src/dev_tools/makeClass/handle_readmes.cpp @@ -194,17 +194,20 @@ const char* STR_CONFIG = "\n" "Each of the following additional configurable command line options are available.\n" "\n" - "**Configuration file:** `$CONFIG/$CHAIN/blockScrape.toml` \n" - "**Configuration group:** `[settings]` \n" + "**Configuration file:** `trueBlocks.toml` \n" + "**Configuration group:** `[scrape.]` \n" "\n" "[{CONFIGS}]\n" + "Note that for Ethereum mainnet, the default values for appsPerChunk and firstSnap are 2,000,000 and 2,300,000 " + "respectively. See the specification for a justification of these values.\n" "\n" "These items may be set in three ways, each overridding the preceeding method:\n" "\n" - "-- in the above configuration file under the `[settings]` group, \n" - "-- in the environment by exporting the configuration item as UPPER_CASE, without " - "underbars, and prepended with TB_SETTINGS_, or \n" - "-- on the command line using the configuration item with leading dashes (i.e., `--name`). "; + "-- in the above configuration file under the `[scrape.]` group, \n" + "-- in the environment by exporting the configuration item as UPPER_CASE (with underbars removed) and " + "prepended with TB_SCRAPE_CHAIN_, or \n" + "-- on the command line using the configuration item with leading dashes and in snake case (i.e., `--snake_case`). " + " "; const char* STR_README_BEGPARTS = "## [{NAME}]\n\n"; const char* STR_README_ENDPARTS = "\n[{USAGE}][{MODELS}][{LINKS}][{CONFIG}][{NOTES}][{FOOTER}]\n"; diff --git a/src/dev_tools/makeClass/templates/blank_options.go.tmpl b/src/dev_tools/makeClass/templates/blank_options.go.tmpl index cbfc278ec7..2a7c68d148 100644 --- a/src/dev_tools/makeClass/templates/blank_options.go.tmpl +++ b/src/dev_tools/makeClass/templates/blank_options.go.tmpl @@ -83,7 +83,7 @@ func [{LOWER}]FinishParse(args []string) *[{PROPER}]Options { [{ENS_CONVERT1}] // EXISTING_CODE // EXISTING_CODE -[{ENS_CONVERT2}][{CONFIG_OVERRIDE}] if len(opts.Globals.Format) == 0 || opts.Globals.Format == "none" { +[{ENS_CONVERT2}] if len(opts.Globals.Format) == 0 || opts.Globals.Format == "none" { opts.Globals.Format = defFmt } diff --git a/src/dev_tools/testRunner/main.cpp b/src/dev_tools/testRunner/main.cpp index 5c5d80e08a..b19984e947 100644 --- a/src/dev_tools/testRunner/main.cpp +++ b/src/dev_tools/testRunner/main.cpp @@ -241,8 +241,7 @@ void COptions::doTests(CMeasure& total, CTestCaseArray& testArray, const string_ CStringArray envLines; for (auto f : fileLines) { if (!startsWith(f, "#")) { - for (auto line : fileLines) - prepender << "Env: " << line << endl; + prepender << "Env: " << f << endl; envLines.push_back(f); } } diff --git a/src/dev_tools/testRunner/testCases/apps/acctExport.csv b/src/dev_tools/testRunner/testCases/apps/acctExport.csv index ed33ad46f1..e24cded770 100644 --- a/src/dev_tools/testRunner/testCases/apps/acctExport.csv +++ b/src/dev_tools/testRunner/testCases/apps/acctExport.csv @@ -77,9 +77,9 @@ on ,both ,fast ,monitors ,apps/acctExport ,monitors_err_not_both1 ,y on ,both ,fast ,monitors ,apps/acctExport ,monitors_err_not_both2 ,y ,addrs = 0xf503017d7baf7fbc0fff7492b751025c6a78179b & fmt = json & delete & undelete on ,both ,fast ,monitors ,apps/acctExport ,monitors_err_sleep ,y ,addrs = 0xf503017d7baf7fbc0fff7492b751025c6a78179b & fmt = json & sleep on ,both ,fast ,monitors ,apps/acctExport ,monitors_err_batch_size2 ,y ,addrs = 0xf503017d7baf7fbc0fff7492b751025c6a78179b & fmt = json & batch_size = 1 -on ,both ,fast ,monitors ,apps/acctExport ,monitors_err_run_once ,y ,addrs = 0xf503017d7baf7fbc0fff7492b751025c6a78179b & fmt = json & run_once +on ,both ,fast ,monitors ,apps/acctExport ,monitors_err_run_once ,y ,addrs = 0xf503017d7baf7fbc0fff7492b751025c6a78179b & fmt = json & run_count = 1 on ,both ,fast ,monitors ,apps/acctExport ,monitors_err_watch ,y ,watch & batch_size = 0 & fmt = json -local ,both ,fast ,monitors ,apps/acctExport ,monitors_watch ,y ,watch & commands = ./command.fil & watchlist = ./watches.txt & run_once & fmt = json +local ,both ,fast ,monitors ,apps/acctExport ,monitors_watch ,y ,watch & commands = ./command.fil & watchlist = ./watches.txt & run_count = 1 & fmt = json on ,both ,fast ,list ,apps/acctExport ,list_prepare_1 ,y ,addrs = 0x001d14804b399c6ef80e64576f657660804fec0b & fmt = json & last_block = 1501460 on ,both ,fast ,monitors ,apps/acctExport ,monitors_clean ,y ,addrs = 0x001d14804b399c6ef80e64576f657660804fec0b & clean diff --git a/src/dev_tools/testRunner/testCases/apps/blockScrape.csv b/src/dev_tools/testRunner/testCases/apps/blockScrape.csv index 1a0d752155..c8060cb975 100644 --- a/src/dev_tools/testRunner/testCases/apps/blockScrape.csv +++ b/src/dev_tools/testRunner/testCases/apps/blockScrape.csv @@ -41,6 +41,11 @@ off ,both ,fast ,scrape ,apps/blockScrape ,real_run2 ,n on ,both ,fast ,scrape ,apps/blockScrape ,by_file_bad ,n ,file = cmd_file_2 +local ,both ,fast ,scrape ,apps/blockScrape ,options_defaults ,n ,dry_run +local ,both ,fast ,scrape ,apps/blockScrape ,options_with_env ,n ,dry_run +local ,both ,fast ,scrape ,apps/blockScrape ,options_cmd_line ,n ,dry_run & apps_per_chunk 10000 & snap_to_grid 10000 & first_snap 10000 & unripe_dist 10000 & channel_count 10000 +local ,both ,fast ,scrape ,apps/blockScrape ,options_combo ,n ,dry_run & apps_per_chunk 10000 & snap_to_grid 10000 & first_snap 10000 + # Capabilities # chain & fmt & help & nocolor & noop & version & verbose & no_header & file & output & append & cache & decache & raw & ether & wei on ,both ,fast ,scrape ,apps/blockScrape ,caps_allowed ,y ,chain & nocolor & noop & version & verbose & fail_on_purpose diff --git a/src/dev_tools/testRunner/testCases/apps/config.csv b/src/dev_tools/testRunner/testCases/apps/config.csv index 636deb6844..0fe5d82c14 100644 --- a/src/dev_tools/testRunner/testCases/apps/config.csv +++ b/src/dev_tools/testRunner/testCases/apps/config.csv @@ -4,7 +4,7 @@ on ,cmd ,fast ,config ,apps/config ,help ,n ,@h on ,cmd ,fast ,config ,apps/config ,help_long ,n ,help on ,cmd ,fast ,config ,apps/config ,invalid_param_2 ,y ,@l & junk -on ,both ,fast ,config ,apps/config ,config_show ,y ,modes = show +local ,both ,fast ,config ,apps/config ,config_show ,y ,modes = show on ,both ,fast ,config ,apps/config ,config_edit ,y ,modes = edit # This fails because the timestamp is not test proof - simple fix diff --git a/src/dev_tools/testRunner/testCases/tools/getBlocks.csv b/src/dev_tools/testRunner/testCases/tools/getBlocks.csv index 14803d3070..47d21e73c7 100644 --- a/src/dev_tools/testRunner/testCases/tools/getBlocks.csv +++ b/src/dev_tools/testRunner/testCases/tools/getBlocks.csv @@ -59,6 +59,8 @@ on ,both ,fast ,blocks ,tools/getBlocks ,without_logs_fail_e2 ,y ,bloc on ,both ,fast ,blocks ,tools/getBlocks ,without_logs_fail_t1 ,y ,blocks = 4012000-4012001 & topic = 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef on ,both ,fast ,blocks ,tools/getBlocks ,without_logs_fail_t2 ,y ,blocks = 4012000-4012001 & topic = 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a +sep ,both ,fast ,blocks ,tools/getBlocks ,withdrawals_sepolia ,y ,blocks = 4001001 & chain = sepolia & hashes + on ,both ,fast ,blocks ,tools/getBlocks ,with_logs ,y ,blocks = 4012000-4012001 & logs on ,both ,fast ,blocks ,tools/getBlocks ,with_logs_csv ,n ,blocks = 4012000-4012001 & logs & fmt = csv on ,both ,fast ,blocks ,tools/getBlocks ,with_logs_topic ,y ,blocks = 4012000-4012001 & logs & topic = 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef diff --git a/src/dev_tools/testRunner/testCases/tools/getTokens.csv b/src/dev_tools/testRunner/testCases/tools/getTokens.csv index 6bdc80bdb2..a8dff4b54f 100644 --- a/src/dev_tools/testRunner/testCases/tools/getTokens.csv +++ b/src/dev_tools/testRunner/testCases/tools/getTokens.csv @@ -56,7 +56,7 @@ on ,both ,slow ,tokens ,tools/getTokens ,info_version ,y on ,both ,slow ,tokens ,tools/getTokens ,info_version2 ,n ,addrs = 0xd26114cd6ee289accf82350c8d8487fedb8a0c07 0xb9e7f8568e08d5659f5d29c4997173d84cdf2607 & parts = version & fmt = csv on ,both ,slow ,tokens ,tools/getTokens ,info_version3 ,y ,addrs = 0xd26114cd6ee289accf82350c8d8487fedb8a0c07 0xb9e7f8568e08d5659f5d29c4997173d84cdf2607 & parts = version & blocks = 4001001-4001002 & fmt = json on ,both ,slow ,tokens ,tools/getTokens ,proxy_no ,y ,addrs = 0x00000000441378008ea67f4284a57932b1c000a5 & fmt = json & blocks = 7442048 & parts = all -on ,both ,slow ,tokens ,tools/getTokens ,proxy_yes ,y ,addrs = 0x00000000441378008ea67f4284a57932b1c000a5 & fmt = json & blocks = 8000000 & parts = all +on ,both ,slow ,tokens ,tools/getTokens ,proxy_yes ,y ,addrs = 0x00000000441378008ea67f4284a57932b1c000a5 & fmt = json & blocks = 8000000 & parts = all & verbose on ,both ,slow ,tokens ,tools/getTokens ,by_account ,y ,addrs = 0xb9e7f8568e08d5659f5d29c4997173d84cdf2607 0xa25d04fa348f107428708ba19c95d1c79b3744bf & by_acct & blocks = 4000000 on ,both ,slow ,tokens ,tools/getTokens ,by_account_long ,y ,addrs = 0xb9e7f8568e08d5659f5d29c4997173d84cdf2607 0xa25d04fa348f107428708ba19c95d1c79b3744bf & by_acct & blocks = 4000000 diff --git a/src/dev_tools/testRunner/testCases/tools/whenBlock.csv b/src/dev_tools/testRunner/testCases/tools/whenBlock.csv index a331f168b6..c12c5f5805 100644 --- a/src/dev_tools/testRunner/testCases/tools/whenBlock.csv +++ b/src/dev_tools/testRunner/testCases/tools/whenBlock.csv @@ -29,6 +29,7 @@ on ,both ,fast ,when ,tools/whenBlock ,invalid_option_2 ,y on ,both ,fast ,when ,tools/whenBlock ,invalid_option_3 ,y ,blocks = 0-0 on ,both ,fast ,when ,tools/whenBlock ,invalid_verbose_syntax ,y ,verbose on ,both ,slow ,when ,tools/whenBlock ,list_dates ,n ,list & fmt = txt +on ,both ,slow ,when ,tools/whenBlock ,list_dates_sepolia ,n ,list & fmt = txt & chain = sepolia on ,both ,slow ,when ,tools/whenBlock ,list_dates_long ,y ,list on ,both ,fast ,when ,tools/whenBlock ,long_verbose_valid_block ,y ,verbose & blocks = 1000 on ,both ,slow ,when ,tools/whenBlock ,mixed_block_and_date ,y ,blocks = 2017-03-02 & blocks = 123123 diff --git a/src/other/build_assets/remote_testing/build_and_test.sh b/src/other/build_assets/remote_testing/build_and_test.sh index 5a741374a6..851bc6b7a3 100644 --- a/src/other/build_assets/remote_testing/build_and_test.sh +++ b/src/other/build_assets/remote_testing/build_and_test.sh @@ -40,6 +40,10 @@ then exit 1 fi + # If there are any config migrations, this will trigger it.. + chifra blocks 1 2>&1 >/dev/null + + # start the server... TEST_MODE=true chifra daemon -p ":$SRV_PORT" 2>/dev/null & echo "Waiting for chifra daemon -p :$SRV_PORT to start..." diff --git a/src/other/data-models/classDefinitions/fields/block.csv b/src/other/data-models/classDefinitions/fields/block.csv index 50a3d2aece..d2867a782a 100644 --- a/src/other/data-models/classDefinitions/fields/block.csv +++ b/src/other/data-models/classDefinitions/fields/block.csv @@ -21,4 +21,5 @@ date ,datetime , , , , , , baseFeePerGas ,wei , , , , , , , , 10 , 7 ,120911 ,the base fee for this block transactions ,Transaction , ,true ,true , , , , , 9 , , ,a possibly empty array of transactions or transaction hashes transactionsRoot ,hash , , , , , ,rawonly , , , , , -uncles ,hash , , ,true ,true ,true , , , 11 , , , +uncles ,hash , , ,true ,true ,true , , , 11 , , ,a possibly empty array of uncle hashes +withdrawals ,Withdrawal , ,true ,true , ,true , , , 12 , , ,a possibly empty array of withdrawals (post Shanghai) diff --git a/src/other/install/trueBlocks.toml b/src/other/install/trueBlocks.toml index 20c0702175..0cfab76ffb 100644 --- a/src/other/install/trueBlocks.toml +++ b/src/other/install/trueBlocks.toml @@ -1,5 +1,5 @@ [version] - current = "v0.40.0-beta" + current = "v1.0.0-release" [settings] cachePath = "" diff --git a/test/gold/apps/acctExport/acctExport_100_list_n.txt b/test/gold/apps/acctExport/acctExport_100_list_n.txt index dff230d681..5332a5d136 100644 --- a/test/gold/apps/acctExport/acctExport_100_list_n.txt +++ b/test/gold/apps/acctExport/acctExport_100_list_n.txt @@ -1,6 +1,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --max_records 100 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] MaxRecords: 100 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt address blockNumber transactionIndex 0x05a56e2d52c817161883f50c441c3228cfe54d9f 1 99999 diff --git a/test/gold/apps/acctExport/acctExport_102_list_mr.txt b/test/gold/apps/acctExport/acctExport_102_list_mr.txt index 8c1b97156a..df2dabe85d 100644 --- a/test/gold/apps/acctExport/acctExport_102_list_mr.txt +++ b/test/gold/apps/acctExport/acctExport_102_list_mr.txt @@ -1,6 +1,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --max_records 10 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] MaxRecords: 10 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt address blockNumber transactionIndex 0x05a56e2d52c817161883f50c441c3228cfe54d9f 1 99999 diff --git a/test/gold/apps/acctExport/acctExport_104_list_fr.txt b/test/gold/apps/acctExport/acctExport_104_list_fr.txt index 17dd325493..52acc2c998 100644 --- a/test/gold/apps/acctExport/acctExport_104_list_fr.txt +++ b/test/gold/apps/acctExport/acctExport_104_list_fr.txt @@ -2,6 +2,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_record TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] FirstRecord: 3 TEST[DATE|TIME] MaxRecords: 100 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt address blockNumber transactionIndex 0x05a56e2d52c817161883f50c441c3228cfe54d9f 235 99999 diff --git a/test/gold/apps/acctExport/acctExport_106_list_fr_mr.txt b/test/gold/apps/acctExport/acctExport_106_list_fr_mr.txt index 99444f0e64..60e52b3991 100644 --- a/test/gold/apps/acctExport/acctExport_106_list_fr_mr.txt +++ b/test/gold/apps/acctExport/acctExport_106_list_fr_mr.txt @@ -2,6 +2,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_record TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] FirstRecord: 3 TEST[DATE|TIME] MaxRecords: 10 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt address blockNumber transactionIndex 0x05a56e2d52c817161883f50c441c3228cfe54d9f 235 99999 diff --git a/test/gold/apps/acctExport/acctExport_108_list_lb.txt b/test/gold/apps/acctExport/acctExport_108_list_lb.txt index 9253fdb570..2053cd9ff2 100644 --- a/test/gold/apps/acctExport/acctExport_108_list_lb.txt +++ b/test/gold/apps/acctExport/acctExport_108_list_lb.txt @@ -1,5 +1,6 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --last_block 100 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 100 TEST[DATE|TIME] Format: txt address blockNumber transactionIndex diff --git a/test/gold/apps/acctExport/acctExport_110_list_lb_mr.txt b/test/gold/apps/acctExport/acctExport_110_list_lb_mr.txt index 95653618e9..6d6ea3613b 100644 --- a/test/gold/apps/acctExport/acctExport_110_list_lb_mr.txt +++ b/test/gold/apps/acctExport/acctExport_110_list_lb_mr.txt @@ -1,6 +1,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --last_block 100 --max_records 10 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] MaxRecords: 10 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 100 TEST[DATE|TIME] Format: txt address blockNumber transactionIndex diff --git a/test/gold/apps/acctExport/acctExport_110_list_lb_mr_2.txt b/test/gold/apps/acctExport/acctExport_110_list_lb_mr_2.txt index 0ff5ae2982..714945ba8c 100644 --- a/test/gold/apps/acctExport/acctExport_110_list_lb_mr_2.txt +++ b/test/gold/apps/acctExport/acctExport_110_list_lb_mr_2.txt @@ -1,6 +1,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --last_block 100 --max_records 1 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] MaxRecords: 1 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 100 TEST[DATE|TIME] Format: txt address blockNumber transactionIndex diff --git a/test/gold/apps/acctExport/acctExport_112_list_lb_fr.txt b/test/gold/apps/acctExport/acctExport_112_list_lb_fr.txt index 7088e1772f..c04f64c023 100644 --- a/test/gold/apps/acctExport/acctExport_112_list_lb_fr.txt +++ b/test/gold/apps/acctExport/acctExport_112_list_lb_fr.txt @@ -1,6 +1,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --last_block 100 --first_record 1 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] FirstRecord: 1 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 100 TEST[DATE|TIME] Format: txt address blockNumber transactionIndex diff --git a/test/gold/apps/acctExport/acctExport_114_list_lb_fr_mr.txt b/test/gold/apps/acctExport/acctExport_114_list_lb_fr_mr.txt index f2f6339b74..3f730b1780 100644 --- a/test/gold/apps/acctExport/acctExport_114_list_lb_fr_mr.txt +++ b/test/gold/apps/acctExport/acctExport_114_list_lb_fr_mr.txt @@ -2,6 +2,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --last_block 1 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] FirstRecord: 1 TEST[DATE|TIME] MaxRecords: 10 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 100 TEST[DATE|TIME] Format: txt address blockNumber transactionIndex diff --git a/test/gold/apps/acctExport/acctExport_114_list_lb_fr_mr_2.txt b/test/gold/apps/acctExport/acctExport_114_list_lb_fr_mr_2.txt index ef8514013a..09bcaa7074 100644 --- a/test/gold/apps/acctExport/acctExport_114_list_lb_fr_mr_2.txt +++ b/test/gold/apps/acctExport/acctExport_114_list_lb_fr_mr_2.txt @@ -2,6 +2,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --last_block 1 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] FirstRecord: 1 TEST[DATE|TIME] MaxRecords: 1 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 100 TEST[DATE|TIME] Format: txt address blockNumber transactionIndex diff --git a/test/gold/apps/acctExport/acctExport_116_list_fb.txt b/test/gold/apps/acctExport/acctExport_116_list_fb.txt index 0378141b0f..d415f53106 100644 --- a/test/gold/apps/acctExport/acctExport_116_list_fb.txt +++ b/test/gold/apps/acctExport/acctExport_116_list_fb.txt @@ -1,6 +1,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block 90 --max_records 100 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] MaxRecords: 100 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 90 TEST[DATE|TIME] Format: txt address blockNumber transactionIndex diff --git a/test/gold/apps/acctExport/acctExport_118_list_fb_mr.txt b/test/gold/apps/acctExport/acctExport_118_list_fb_mr.txt index 29aeda9742..bf3771dde0 100644 --- a/test/gold/apps/acctExport/acctExport_118_list_fb_mr.txt +++ b/test/gold/apps/acctExport/acctExport_118_list_fb_mr.txt @@ -1,6 +1,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block 90 --max_records 10 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] MaxRecords: 10 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 90 TEST[DATE|TIME] Format: txt address blockNumber transactionIndex diff --git a/test/gold/apps/acctExport/acctExport_120_list_fb_fr.txt b/test/gold/apps/acctExport/acctExport_120_list_fb_fr.txt index e94bfa741c..6b16e672b8 100644 --- a/test/gold/apps/acctExport/acctExport_120_list_fb_fr.txt +++ b/test/gold/apps/acctExport/acctExport_120_list_fb_fr.txt @@ -2,6 +2,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] FirstRecord: 3 TEST[DATE|TIME] MaxRecords: 100 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 90 TEST[DATE|TIME] Format: txt address blockNumber transactionIndex diff --git a/test/gold/apps/acctExport/acctExport_122_list_fb_fr_mr.txt b/test/gold/apps/acctExport/acctExport_122_list_fb_fr_mr.txt index c8b4759977..4222f490b3 100644 --- a/test/gold/apps/acctExport/acctExport_122_list_fb_fr_mr.txt +++ b/test/gold/apps/acctExport/acctExport_122_list_fb_fr_mr.txt @@ -2,6 +2,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] FirstRecord: 3 TEST[DATE|TIME] MaxRecords: 10 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 90 TEST[DATE|TIME] Format: txt address blockNumber transactionIndex diff --git a/test/gold/apps/acctExport/acctExport_124_list_fb_lb.txt b/test/gold/apps/acctExport/acctExport_124_list_fb_lb.txt index 6379a598ff..7536118e70 100644 --- a/test/gold/apps/acctExport/acctExport_124_list_fb_lb.txt +++ b/test/gold/apps/acctExport/acctExport_124_list_fb_lb.txt @@ -1,5 +1,6 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block 55 --last_block 236 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 55 TEST[DATE|TIME] LastBlock: 236 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_124_list_fb_lb_2.txt b/test/gold/apps/acctExport/acctExport_124_list_fb_lb_2.txt index b2f38f188f..ff72309409 100644 --- a/test/gold/apps/acctExport/acctExport_124_list_fb_lb_2.txt +++ b/test/gold/apps/acctExport/acctExport_124_list_fb_lb_2.txt @@ -1,5 +1,6 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block 56 --last_block 236 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 56 TEST[DATE|TIME] LastBlock: 236 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_124_list_fb_lb_3.txt b/test/gold/apps/acctExport/acctExport_124_list_fb_lb_3.txt index f812ce0954..1b68d7a14a 100644 --- a/test/gold/apps/acctExport/acctExport_124_list_fb_lb_3.txt +++ b/test/gold/apps/acctExport/acctExport_124_list_fb_lb_3.txt @@ -1,5 +1,6 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block 56 --last_block 235 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 56 TEST[DATE|TIME] LastBlock: 235 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_124_list_fb_lb_4.txt b/test/gold/apps/acctExport/acctExport_124_list_fb_lb_4.txt index ad0b86eae2..a952afa520 100644 --- a/test/gold/apps/acctExport/acctExport_124_list_fb_lb_4.txt +++ b/test/gold/apps/acctExport/acctExport_124_list_fb_lb_4.txt @@ -1,5 +1,6 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block 55 --last_block 235 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 55 TEST[DATE|TIME] LastBlock: 235 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_126_list_fb_lb_mr.txt b/test/gold/apps/acctExport/acctExport_126_list_fb_lb_mr.txt index f5a2ed8926..4d4aaf2273 100644 --- a/test/gold/apps/acctExport/acctExport_126_list_fb_lb_mr.txt +++ b/test/gold/apps/acctExport/acctExport_126_list_fb_lb_mr.txt @@ -1,6 +1,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block 56 --last_block 2516 --max_records 10 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] MaxRecords: 10 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 56 TEST[DATE|TIME] LastBlock: 2516 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_126_list_fb_lb_mr_2.txt b/test/gold/apps/acctExport/acctExport_126_list_fb_lb_mr_2.txt index 77e2a807fb..ffb30a3634 100644 --- a/test/gold/apps/acctExport/acctExport_126_list_fb_lb_mr_2.txt +++ b/test/gold/apps/acctExport/acctExport_126_list_fb_lb_mr_2.txt @@ -1,6 +1,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block 56 --last_block 2516 --max_records 8 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] MaxRecords: 8 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 56 TEST[DATE|TIME] LastBlock: 2516 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_128_list_fb_lb_fr.txt b/test/gold/apps/acctExport/acctExport_128_list_fb_lb_fr.txt index 8934e24f8b..fa676f0f7c 100644 --- a/test/gold/apps/acctExport/acctExport_128_list_fb_lb_fr.txt +++ b/test/gold/apps/acctExport/acctExport_128_list_fb_lb_fr.txt @@ -1,6 +1,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block 56 --last_block 2516 --first_record 3 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] FirstRecord: 3 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 56 TEST[DATE|TIME] LastBlock: 2516 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_128_list_fb_lb_fr_2.txt b/test/gold/apps/acctExport/acctExport_128_list_fb_lb_fr_2.txt index 036168d067..1e5dd3c88d 100644 --- a/test/gold/apps/acctExport/acctExport_128_list_fb_lb_fr_2.txt +++ b/test/gold/apps/acctExport/acctExport_128_list_fb_lb_fr_2.txt @@ -1,6 +1,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block 56 --last_block 235 --first_record 1 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] FirstRecord: 1 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 56 TEST[DATE|TIME] LastBlock: 235 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_128_list_fb_lb_fr_3.txt b/test/gold/apps/acctExport/acctExport_128_list_fb_lb_fr_3.txt index 0284ce39ad..bdd83d3fd1 100644 --- a/test/gold/apps/acctExport/acctExport_128_list_fb_lb_fr_3.txt +++ b/test/gold/apps/acctExport/acctExport_128_list_fb_lb_fr_3.txt @@ -1,6 +1,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block 56 --last_block 235 --first_record 4 TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] FirstRecord: 4 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 56 TEST[DATE|TIME] LastBlock: 235 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_130_list_fb_lb_fr_mr.txt b/test/gold/apps/acctExport/acctExport_130_list_fb_lb_fr_mr.txt index ddb48b2a18..3e34bde18e 100644 --- a/test/gold/apps/acctExport/acctExport_130_list_fb_lb_fr_mr.txt +++ b/test/gold/apps/acctExport/acctExport_130_list_fb_lb_fr_mr.txt @@ -2,6 +2,7 @@ chifra list 0x05a56e2d52c817161883f50c441c3228cfe54d9f --fmt txt --first_block TEST[DATE|TIME] Addrs: [0x05a56e2d52c817161883f50c441c3228cfe54d9f] TEST[DATE|TIME] FirstRecord: 1 TEST[DATE|TIME] MaxRecords: 10 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 56 TEST[DATE|TIME] LastBlock: 235 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_accounting_list_0.txt b/test/gold/apps/acctExport/acctExport_accounting_list_0.txt index d464cab207..97514383bc 100644 --- a/test/gold/apps/acctExport/acctExport_accounting_list_0.txt +++ b/test/gold/apps/acctExport/acctExport_accounting_list_0.txt @@ -1,5 +1,6 @@ chifra list 0x001d14804b399c6ef80e64576f657660804fec0b --fmt json TEST[DATE|TIME] Addrs: [0x001d14804b399c6ef80e64576f657660804fec0b] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: json { "data": [ diff --git a/test/gold/apps/acctExport/acctExport_accounting_list_1.txt b/test/gold/apps/acctExport/acctExport_accounting_list_1.txt index d464cab207..97514383bc 100644 --- a/test/gold/apps/acctExport/acctExport_accounting_list_1.txt +++ b/test/gold/apps/acctExport/acctExport_accounting_list_1.txt @@ -1,5 +1,6 @@ chifra list 0x001d14804b399c6ef80e64576f657660804fec0b --fmt json TEST[DATE|TIME] Addrs: [0x001d14804b399c6ef80e64576f657660804fec0b] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: json { "data": [ diff --git a/test/gold/apps/acctExport/acctExport_caps_allowed_l.txt b/test/gold/apps/acctExport/acctExport_caps_allowed_l.txt index 93f536fa8b..95ed3429a5 100644 --- a/test/gold/apps/acctExport/acctExport_caps_allowed_l.txt +++ b/test/gold/apps/acctExport/acctExport_caps_allowed_l.txt @@ -17,6 +17,7 @@ Flags: -c, --first_record uint the first record to process -e, --max_records uint the maximum number of records to process (default 250) -E, --reversed produce results in reverse chronological order + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -F, --first_block uint first block to export (inclusive, ignored when freshening) -L, --last_block uint last block to export (inclusive, ignored when freshening) -x, --fmt string export format, one of [none|json*|txt|csv] diff --git a/test/gold/apps/acctExport/acctExport_caps_allowed_m.txt b/test/gold/apps/acctExport/acctExport_caps_allowed_m.txt index bcbbf8b664..087beeeaca 100644 --- a/test/gold/apps/acctExport/acctExport_caps_allowed_m.txt +++ b/test/gold/apps/acctExport/acctExport_caps_allowed_m.txt @@ -18,7 +18,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -26,7 +26,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_caps_disallowed_1_l.txt b/test/gold/apps/acctExport/acctExport_caps_disallowed_1_l.txt index e9ed6b5a3f..026f58a94f 100644 --- a/test/gold/apps/acctExport/acctExport_caps_disallowed_1_l.txt +++ b/test/gold/apps/acctExport/acctExport_caps_disallowed_1_l.txt @@ -17,6 +17,7 @@ Flags: -c, --first_record uint the first record to process -e, --max_records uint the maximum number of records to process (default 250) -E, --reversed produce results in reverse chronological order + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -F, --first_block uint first block to export (inclusive, ignored when freshening) -L, --last_block uint last block to export (inclusive, ignored when freshening) -x, --fmt string export format, one of [none|json*|txt|csv] diff --git a/test/gold/apps/acctExport/acctExport_caps_disallowed_1_m.txt b/test/gold/apps/acctExport/acctExport_caps_disallowed_1_m.txt index 4108f7bdcc..86b64a307f 100644 --- a/test/gold/apps/acctExport/acctExport_caps_disallowed_1_m.txt +++ b/test/gold/apps/acctExport/acctExport_caps_disallowed_1_m.txt @@ -18,7 +18,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -26,7 +26,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_caps_disallowed_2_l.txt b/test/gold/apps/acctExport/acctExport_caps_disallowed_2_l.txt index 08a7fb7346..b2bdafcccc 100644 --- a/test/gold/apps/acctExport/acctExport_caps_disallowed_2_l.txt +++ b/test/gold/apps/acctExport/acctExport_caps_disallowed_2_l.txt @@ -17,6 +17,7 @@ Flags: -c, --first_record uint the first record to process -e, --max_records uint the maximum number of records to process (default 250) -E, --reversed produce results in reverse chronological order + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -F, --first_block uint first block to export (inclusive, ignored when freshening) -L, --last_block uint last block to export (inclusive, ignored when freshening) -x, --fmt string export format, one of [none|json*|txt|csv] diff --git a/test/gold/apps/acctExport/acctExport_caps_disallowed_2_m.txt b/test/gold/apps/acctExport/acctExport_caps_disallowed_2_m.txt index 1c125d966c..a34e618b44 100644 --- a/test/gold/apps/acctExport/acctExport_caps_disallowed_2_m.txt +++ b/test/gold/apps/acctExport/acctExport_caps_disallowed_2_m.txt @@ -18,7 +18,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -26,7 +26,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_caps_disallowed_3_l.txt b/test/gold/apps/acctExport/acctExport_caps_disallowed_3_l.txt index 62993a81c2..9711a7c478 100644 --- a/test/gold/apps/acctExport/acctExport_caps_disallowed_3_l.txt +++ b/test/gold/apps/acctExport/acctExport_caps_disallowed_3_l.txt @@ -17,6 +17,7 @@ Flags: -c, --first_record uint the first record to process -e, --max_records uint the maximum number of records to process (default 250) -E, --reversed produce results in reverse chronological order + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -F, --first_block uint first block to export (inclusive, ignored when freshening) -L, --last_block uint last block to export (inclusive, ignored when freshening) -x, --fmt string export format, one of [none|json*|txt|csv] diff --git a/test/gold/apps/acctExport/acctExport_caps_disallowed_3_m.txt b/test/gold/apps/acctExport/acctExport_caps_disallowed_3_m.txt index 07eb06c33b..8fae8c57d7 100644 --- a/test/gold/apps/acctExport/acctExport_caps_disallowed_3_m.txt +++ b/test/gold/apps/acctExport/acctExport_caps_disallowed_3_m.txt @@ -18,7 +18,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -26,7 +26,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_caps_disallowed_4_l.txt b/test/gold/apps/acctExport/acctExport_caps_disallowed_4_l.txt index 7cc186c9a2..93fb0b4d64 100644 --- a/test/gold/apps/acctExport/acctExport_caps_disallowed_4_l.txt +++ b/test/gold/apps/acctExport/acctExport_caps_disallowed_4_l.txt @@ -17,6 +17,7 @@ Flags: -c, --first_record uint the first record to process -e, --max_records uint the maximum number of records to process (default 250) -E, --reversed produce results in reverse chronological order + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -F, --first_block uint first block to export (inclusive, ignored when freshening) -L, --last_block uint last block to export (inclusive, ignored when freshening) -x, --fmt string export format, one of [none|json*|txt|csv] diff --git a/test/gold/apps/acctExport/acctExport_caps_disallowed_4_m.txt b/test/gold/apps/acctExport/acctExport_caps_disallowed_4_m.txt index b59473a2e9..270843cb81 100644 --- a/test/gold/apps/acctExport/acctExport_caps_disallowed_4_m.txt +++ b/test/gold/apps/acctExport/acctExport_caps_disallowed_4_m.txt @@ -18,7 +18,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -26,7 +26,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_caps_disallowed_5_l.txt b/test/gold/apps/acctExport/acctExport_caps_disallowed_5_l.txt index 3991971123..f3040fad79 100644 --- a/test/gold/apps/acctExport/acctExport_caps_disallowed_5_l.txt +++ b/test/gold/apps/acctExport/acctExport_caps_disallowed_5_l.txt @@ -17,6 +17,7 @@ Flags: -c, --first_record uint the first record to process -e, --max_records uint the maximum number of records to process (default 250) -E, --reversed produce results in reverse chronological order + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -F, --first_block uint first block to export (inclusive, ignored when freshening) -L, --last_block uint last block to export (inclusive, ignored when freshening) -x, --fmt string export format, one of [none|json*|txt|csv] diff --git a/test/gold/apps/acctExport/acctExport_caps_disallowed_5_m.txt b/test/gold/apps/acctExport/acctExport_caps_disallowed_5_m.txt index 3991971123..f3040fad79 100644 --- a/test/gold/apps/acctExport/acctExport_caps_disallowed_5_m.txt +++ b/test/gold/apps/acctExport/acctExport_caps_disallowed_5_m.txt @@ -17,6 +17,7 @@ Flags: -c, --first_record uint the first record to process -e, --max_records uint the maximum number of records to process (default 250) -E, --reversed produce results in reverse chronological order + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -F, --first_block uint first block to export (inclusive, ignored when freshening) -L, --last_block uint last block to export (inclusive, ignored when freshening) -x, --fmt string export format, one of [none|json*|txt|csv] diff --git a/test/gold/apps/acctExport/acctExport_export_relevant_list.txt b/test/gold/apps/acctExport/acctExport_export_relevant_list.txt index a5640b6935..1bcce67237 100644 --- a/test/gold/apps/acctExport/acctExport_export_relevant_list.txt +++ b/test/gold/apps/acctExport/acctExport_export_relevant_list.txt @@ -1,6 +1,7 @@ chifra list 0x9531c059098e3d194ff87febb587ab07b30b1306 --count --fmt json --last_block 12000000 TEST[DATE|TIME] Addrs: [0x9531c059098e3d194ff87febb587ab07b30b1306] TEST[DATE|TIME] Count: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 12000000 TEST[DATE|TIME] Format: json { diff --git a/test/gold/apps/acctExport/acctExport_list_appearances.txt b/test/gold/apps/acctExport/acctExport_list_appearances.txt index a441382368..1a567fbc5a 100644 --- a/test/gold/apps/acctExport/acctExport_list_appearances.txt +++ b/test/gold/apps/acctExport/acctExport_list_appearances.txt @@ -1,5 +1,6 @@ chifra list 0x001d14804b399c6ef80e64576f657660804fec0b --first_block 100000 --last_block 2000000 TEST[DATE|TIME] Addrs: [0x001d14804b399c6ef80e64576f657660804fec0b] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 100000 TEST[DATE|TIME] LastBlock: 2000000 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_list_appearances_count.txt b/test/gold/apps/acctExport/acctExport_list_appearances_count.txt index 276c53ad19..451c646bac 100644 --- a/test/gold/apps/acctExport/acctExport_list_appearances_count.txt +++ b/test/gold/apps/acctExport/acctExport_list_appearances_count.txt @@ -1,6 +1,7 @@ chifra list 0x001d14804b399c6ef80e64576f657660804fec0b --first_block 100000 --last_block 2000000 --count TEST[DATE|TIME] Addrs: [0x001d14804b399c6ef80e64576f657660804fec0b] TEST[DATE|TIME] Count: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 100000 TEST[DATE|TIME] LastBlock: 2000000 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_list_appearances_csv.txt b/test/gold/apps/acctExport/acctExport_list_appearances_csv.txt index 2deda9092d..09b71da8fc 100644 --- a/test/gold/apps/acctExport/acctExport_list_appearances_csv.txt +++ b/test/gold/apps/acctExport/acctExport_list_appearances_csv.txt @@ -1,5 +1,6 @@ chifra list 0x001d14804b399c6ef80e64576f657660804fec0b --first_block 100000 --last_block 2000000 --fmt csv --no_header TEST[DATE|TIME] Addrs: [0x001d14804b399c6ef80e64576f657660804fec0b] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 100000 TEST[DATE|TIME] LastBlock: 2000000 TEST[DATE|TIME] NoHeader: true diff --git a/test/gold/apps/acctExport/acctExport_list_appearances_dates.txt b/test/gold/apps/acctExport/acctExport_list_appearances_dates.txt index cdcfe4aa98..5eb5b860f5 100644 --- a/test/gold/apps/acctExport/acctExport_list_appearances_dates.txt +++ b/test/gold/apps/acctExport/acctExport_list_appearances_dates.txt @@ -1,5 +1,6 @@ chifra list 0x001d14804b399c6ef80e64576f657660804fec0b --first_block 100000 --last_block 2000000 --verbose TEST[DATE|TIME] Addrs: [0x001d14804b399c6ef80e64576f657660804fec0b] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 100000 TEST[DATE|TIME] LastBlock: 2000000 TEST[DATE|TIME] Verbose: true diff --git a/test/gold/apps/acctExport/acctExport_list_appearances_nozero1.txt b/test/gold/apps/acctExport/acctExport_list_appearances_nozero1.txt index 3aa084703d..bffaad8f09 100644 --- a/test/gold/apps/acctExport/acctExport_list_appearances_nozero1.txt +++ b/test/gold/apps/acctExport/acctExport_list_appearances_nozero1.txt @@ -1,6 +1,7 @@ chifra list 0x007f7f58d3eb5b7510a301ecc749fc1fcddbe14d 0x4444DEADdeadDEADdeadDEADdeadDEADdeadDEAD 0x2910543af39aba0cd09dbb2d50200b3e800a63d2 --first_block 100000 --last_block 100100 --fmt txt --count TEST[DATE|TIME] Addrs: [0x007f7f58d3eb5b7510a301ecc749fc1fcddbe14d 0x4444deaddeaddeaddeaddeaddeaddeaddeaddead 0x2910543af39aba0cd09dbb2d50200b3e800a63d2] TEST[DATE|TIME] Count: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 100000 TEST[DATE|TIME] LastBlock: 100100 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_list_appearances_nozero2.txt b/test/gold/apps/acctExport/acctExport_list_appearances_nozero2.txt index ebf4615639..6c2b52bdbc 100644 --- a/test/gold/apps/acctExport/acctExport_list_appearances_nozero2.txt +++ b/test/gold/apps/acctExport/acctExport_list_appearances_nozero2.txt @@ -2,6 +2,7 @@ chifra list 0x007f7f58d3eb5b7510a301ecc749fc1fcddbe14d 0x4444DEADdeadDEADdeadDE TEST[DATE|TIME] Addrs: [0x007f7f58d3eb5b7510a301ecc749fc1fcddbe14d 0x4444deaddeaddeaddeaddeaddeaddeaddeaddead 0x2910543af39aba0cd09dbb2d50200b3e800a63d2] TEST[DATE|TIME] Count: true TEST[DATE|TIME] NoZero: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 100000 TEST[DATE|TIME] LastBlock: 100100 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_list_appearances_rev.txt b/test/gold/apps/acctExport/acctExport_list_appearances_rev.txt index 02c9304ec8..e7fef34bbb 100644 --- a/test/gold/apps/acctExport/acctExport_list_appearances_rev.txt +++ b/test/gold/apps/acctExport/acctExport_list_appearances_rev.txt @@ -1,6 +1,7 @@ chifra list 0x001d14804b399c6ef80e64576f657660804fec0b --first_block 100000 --last_block 2000000 --reversed TEST[DATE|TIME] Addrs: [0x001d14804b399c6ef80e64576f657660804fec0b] TEST[DATE|TIME] Reversed: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 100000 TEST[DATE|TIME] LastBlock: 2000000 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_list_bounds_1a.txt b/test/gold/apps/acctExport/acctExport_list_bounds_1a.txt index 15c4e7731e..5e66a8daf6 100644 --- a/test/gold/apps/acctExport/acctExport_list_bounds_1a.txt +++ b/test/gold/apps/acctExport/acctExport_list_bounds_1a.txt @@ -1,6 +1,7 @@ chifra list trueblocks.eth --fmt json --last_block 9000000 --bounds TEST[DATE|TIME] Addrs: [0xf503017d7baf7fbc0fff7492b751025c6a78179b] TEST[DATE|TIME] Bounds: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 9000000 TEST[DATE|TIME] Format: json { diff --git a/test/gold/apps/acctExport/acctExport_list_bounds_1b.txt b/test/gold/apps/acctExport/acctExport_list_bounds_1b.txt index c474705817..b068cce7b2 100644 --- a/test/gold/apps/acctExport/acctExport_list_bounds_1b.txt +++ b/test/gold/apps/acctExport/acctExport_list_bounds_1b.txt @@ -2,6 +2,7 @@ chifra list trueblocks.eth --count --fmt csv --last_block 9000000 --bounds TEST[DATE|TIME] Addrs: [0xf503017d7baf7fbc0fff7492b751025c6a78179b] TEST[DATE|TIME] Count: true TEST[DATE|TIME] Bounds: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 9000000 TEST[DATE|TIME] Format: csv address,nRecords,fileSize diff --git a/test/gold/apps/acctExport/acctExport_list_bounds_2a.txt b/test/gold/apps/acctExport/acctExport_list_bounds_2a.txt index 28ebace082..dfbcef4189 100644 --- a/test/gold/apps/acctExport/acctExport_list_bounds_2a.txt +++ b/test/gold/apps/acctExport/acctExport_list_bounds_2a.txt @@ -1,6 +1,7 @@ chifra list trueblocks.eth --fmt txt --first_block 8854723 --last_block 9000000 --bounds TEST[DATE|TIME] Addrs: [0xf503017d7baf7fbc0fff7492b751025c6a78179b] TEST[DATE|TIME] Bounds: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 8854723 TEST[DATE|TIME] LastBlock: 9000000 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_list_bounds_2b.txt b/test/gold/apps/acctExport/acctExport_list_bounds_2b.txt index f2fb535c9d..e629e6bcce 100644 --- a/test/gold/apps/acctExport/acctExport_list_bounds_2b.txt +++ b/test/gold/apps/acctExport/acctExport_list_bounds_2b.txt @@ -2,6 +2,7 @@ chifra list trueblocks.eth --count --fmt txt --first_block 8854723 --last_block TEST[DATE|TIME] Addrs: [0xf503017d7baf7fbc0fff7492b751025c6a78179b] TEST[DATE|TIME] Count: true TEST[DATE|TIME] Bounds: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 8854723 TEST[DATE|TIME] LastBlock: 9000000 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_list_bounds_3a.txt b/test/gold/apps/acctExport/acctExport_list_bounds_3a.txt index 7468724c0e..b358c402e8 100644 --- a/test/gold/apps/acctExport/acctExport_list_bounds_3a.txt +++ b/test/gold/apps/acctExport/acctExport_list_bounds_3a.txt @@ -1,6 +1,7 @@ chifra list trueblocks.eth --fmt json --first_block 8854725 --last_block 9000000 --bounds TEST[DATE|TIME] Addrs: [0xf503017d7baf7fbc0fff7492b751025c6a78179b] TEST[DATE|TIME] Bounds: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 8854725 TEST[DATE|TIME] LastBlock: 9000000 TEST[DATE|TIME] Format: json diff --git a/test/gold/apps/acctExport/acctExport_list_bounds_3b.txt b/test/gold/apps/acctExport/acctExport_list_bounds_3b.txt index 0e4e1cea87..46b35a9b5d 100644 --- a/test/gold/apps/acctExport/acctExport_list_bounds_3b.txt +++ b/test/gold/apps/acctExport/acctExport_list_bounds_3b.txt @@ -2,6 +2,7 @@ chifra list trueblocks.eth --count --fmt json --first_block 8854725 --last_bloc TEST[DATE|TIME] Addrs: [0xf503017d7baf7fbc0fff7492b751025c6a78179b] TEST[DATE|TIME] Count: true TEST[DATE|TIME] Bounds: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 8854725 TEST[DATE|TIME] LastBlock: 9000000 TEST[DATE|TIME] Format: json diff --git a/test/gold/apps/acctExport/acctExport_list_ens.txt b/test/gold/apps/acctExport/acctExport_list_ens.txt index 4734435625..f05655dbc9 100644 --- a/test/gold/apps/acctExport/acctExport_list_ens.txt +++ b/test/gold/apps/acctExport/acctExport_list_ens.txt @@ -1,5 +1,6 @@ chifra list rhyslindmark.eth --first_block 4037786 --last_block 4081406 TEST[DATE|TIME] Addrs: [0xff9387a9aae1f5daab1cd8eb0e92113ea9d19ca3] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 4037786 TEST[DATE|TIME] LastBlock: 4081406 TEST[DATE|TIME] Format: txt diff --git a/test/gold/apps/acctExport/acctExport_list_err_nozero.txt b/test/gold/apps/acctExport/acctExport_list_err_nozero.txt index a349c0a59a..2de193c1f0 100644 --- a/test/gold/apps/acctExport/acctExport_list_err_nozero.txt +++ b/test/gold/apps/acctExport/acctExport_list_err_nozero.txt @@ -1,6 +1,7 @@ chifra list 0x007f7f58d3eb5b7510a301ecc749fc1fcddbe14d 0x4444DEADdeadDEADdeadDEADdeadDEADdeadDEAD 0x2910543af39aba0cd09dbb2d50200b3e800a63d2 --first_block 100000 --last_block 100100 --fmt txt --no_zero TEST[DATE|TIME] Addrs: [0x007f7f58d3eb5b7510a301ecc749fc1fcddbe14d 0x4444deaddeaddeaddeaddeaddeaddeaddeaddead 0x2910543af39aba0cd09dbb2d50200b3e800a63d2] TEST[DATE|TIME] NoZero: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 100000 TEST[DATE|TIME] LastBlock: 100100 TEST[DATE|TIME] Format: txt @@ -20,6 +21,7 @@ Flags: -c, --first_record uint the first record to process -e, --max_records uint the maximum number of records to process (default 250) -E, --reversed produce results in reverse chronological order + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -F, --first_block uint first block to export (inclusive, ignored when freshening) -L, --last_block uint last block to export (inclusive, ignored when freshening) -x, --fmt string export format, one of [none|json*|txt|csv] diff --git a/test/gold/apps/acctExport/acctExport_list_help.txt b/test/gold/apps/acctExport/acctExport_list_help.txt index 88679cc955..cbc5886fb2 100644 --- a/test/gold/apps/acctExport/acctExport_list_help.txt +++ b/test/gold/apps/acctExport/acctExport_list_help.txt @@ -17,6 +17,7 @@ Flags: -c, --first_record uint the first record to process -e, --max_records uint the maximum number of records to process (default 250) -E, --reversed produce results in reverse chronological order + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -F, --first_block uint first block to export (inclusive, ignored when freshening) -L, --last_block uint last block to export (inclusive, ignored when freshening) -x, --fmt string export format, one of [none|json*|txt|csv] diff --git a/test/gold/apps/acctExport/acctExport_list_list.txt b/test/gold/apps/acctExport/acctExport_list_list.txt index ba83377b48..e65bb737af 100644 --- a/test/gold/apps/acctExport/acctExport_list_list.txt +++ b/test/gold/apps/acctExport/acctExport_list_list.txt @@ -1,5 +1,6 @@ chifra list 0x001d14804b399c6ef80e64576f657660804fec0b --fmt txt --last_block 2000000 TEST[DATE|TIME] Addrs: [0x001d14804b399c6ef80e64576f657660804fec0b] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 2000000 TEST[DATE|TIME] Format: txt address blockNumber transactionIndex diff --git a/test/gold/apps/acctExport/acctExport_list_long_help.txt b/test/gold/apps/acctExport/acctExport_list_long_help.txt index 4e7192cfdf..b352ec10e9 100644 --- a/test/gold/apps/acctExport/acctExport_list_long_help.txt +++ b/test/gold/apps/acctExport/acctExport_list_long_help.txt @@ -17,6 +17,7 @@ Flags: -c, --first_record uint the first record to process -e, --max_records uint the maximum number of records to process (default 250) -E, --reversed produce results in reverse chronological order + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -F, --first_block uint first block to export (inclusive, ignored when freshening) -L, --last_block uint last block to export (inclusive, ignored when freshening) -x, --fmt string export format, one of [none|json*|txt|csv] diff --git a/test/gold/apps/acctExport/acctExport_list_no_params.txt b/test/gold/apps/acctExport/acctExport_list_no_params.txt index 860e0874df..d2b40c0a8e 100644 --- a/test/gold/apps/acctExport/acctExport_list_no_params.txt +++ b/test/gold/apps/acctExport/acctExport_list_no_params.txt @@ -1,4 +1,5 @@ chifra list +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: Please specify at least one valid Ethereum address. Usage: @@ -16,6 +17,7 @@ Flags: -c, --first_record uint the first record to process -e, --max_records uint the maximum number of records to process (default 250) -E, --reversed produce results in reverse chronological order + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -F, --first_block uint first block to export (inclusive, ignored when freshening) -L, --last_block uint last block to export (inclusive, ignored when freshening) -x, --fmt string export format, one of [none|json*|txt|csv] diff --git a/test/gold/apps/acctExport/acctExport_list_prepare_1.txt b/test/gold/apps/acctExport/acctExport_list_prepare_1.txt index 033273266b..3108707fc1 100644 --- a/test/gold/apps/acctExport/acctExport_list_prepare_1.txt +++ b/test/gold/apps/acctExport/acctExport_list_prepare_1.txt @@ -1,5 +1,6 @@ chifra list 0x001d14804b399c6ef80e64576f657660804fec0b --fmt json --last_block 1501460 TEST[DATE|TIME] Addrs: [0x001d14804b399c6ef80e64576f657660804fec0b] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 1501460 TEST[DATE|TIME] Format: json { diff --git a/test/gold/apps/acctExport/acctExport_list_prepare_2.txt b/test/gold/apps/acctExport/acctExport_list_prepare_2.txt index 033273266b..3108707fc1 100644 --- a/test/gold/apps/acctExport/acctExport_list_prepare_2.txt +++ b/test/gold/apps/acctExport/acctExport_list_prepare_2.txt @@ -1,5 +1,6 @@ chifra list 0x001d14804b399c6ef80e64576f657660804fec0b --fmt json --last_block 1501460 TEST[DATE|TIME] Addrs: [0x001d14804b399c6ef80e64576f657660804fec0b] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 1501460 TEST[DATE|TIME] Format: json { diff --git a/test/gold/apps/acctExport/acctExport_list_prepare_3.txt b/test/gold/apps/acctExport/acctExport_list_prepare_3.txt index 79c07d36d6..2b2129f9ee 100644 --- a/test/gold/apps/acctExport/acctExport_list_prepare_3.txt +++ b/test/gold/apps/acctExport/acctExport_list_prepare_3.txt @@ -1,5 +1,6 @@ chifra list 0x00ad1a7b0ef4d3c5070d73b9412b6ed22efb7533 --fmt json --last_block 2873700 TEST[DATE|TIME] Addrs: [0x00ad1a7b0ef4d3c5070d73b9412b6ed22efb7533] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] LastBlock: 2873700 TEST[DATE|TIME] Format: json { diff --git a/test/gold/apps/acctExport/acctExport_monitors_decache.txt b/test/gold/apps/acctExport/acctExport_monitors_decache.txt index 9316815ad0..8c6057d8f3 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_decache.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_decache.txt @@ -18,7 +18,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -26,7 +26,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_decache_again.txt b/test/gold/apps/acctExport/acctExport_monitors_decache_again.txt index 9316815ad0..8c6057d8f3 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_decache_again.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_decache_again.txt @@ -18,7 +18,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -26,7 +26,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_delete_1_fail.txt b/test/gold/apps/acctExport/acctExport_monitors_delete_1_fail.txt index a3e73d28db..ecf4501e94 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_delete_1_fail.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_delete_1_fail.txt @@ -19,7 +19,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -27,7 +27,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_err_batch_size2.txt b/test/gold/apps/acctExport/acctExport_monitors_err_batch_size2.txt index ab694aadef..359b9cbcee 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_err_batch_size2.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_err_batch_size2.txt @@ -2,7 +2,7 @@ chifra monitors 0xf503017d7baf7fbc0fff7492b751025c6a78179b --fmt json --batch_s TEST[DATE|TIME] Addrs: [0xf503017d7baf7fbc0fff7492b751025c6a78179b] TEST[DATE|TIME] BatchSize: 1 TEST[DATE|TIME] Format: json -Error: The --batch-size option is not available without --watch. +Error: The --batch_size option is not available without --watch. Usage: chifra monitors [flags]
[address...] @@ -19,7 +19,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -27,7 +27,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_err_not_both1.txt b/test/gold/apps/acctExport/acctExport_monitors_err_not_both1.txt index 9e2f2b992e..5296ee9ef9 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_err_not_both1.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_err_not_both1.txt @@ -20,7 +20,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -28,7 +28,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_err_not_both2.txt b/test/gold/apps/acctExport/acctExport_monitors_err_not_both2.txt index 7c60a80a66..97dc2b5860 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_err_not_both2.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_err_not_both2.txt @@ -20,7 +20,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -28,7 +28,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_err_run_once.txt b/test/gold/apps/acctExport/acctExport_monitors_err_run_once.txt index 72f995aca3..f83fd21aec 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_err_run_once.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_err_run_once.txt @@ -1,8 +1,8 @@ -chifra monitors 0xf503017d7baf7fbc0fff7492b751025c6a78179b --fmt json --run_once +chifra monitors 0xf503017d7baf7fbc0fff7492b751025c6a78179b --fmt json --run_count 1 TEST[DATE|TIME] Addrs: [0xf503017d7baf7fbc0fff7492b751025c6a78179b] -TEST[DATE|TIME] RunOnce: true +TEST[DATE|TIME] RunCount: 1 TEST[DATE|TIME] Format: json -Error: The --run-once option is not available without --watch. +Error: The --run_count option is not available without --watch. Usage: chifra monitors [flags]
[address...] @@ -19,7 +19,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -27,7 +27,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_err_sleep.txt b/test/gold/apps/acctExport/acctExport_monitors_err_sleep.txt index 85776d3a66..2900228d7f 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_err_sleep.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_err_sleep.txt @@ -18,7 +18,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -26,7 +26,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_err_watch.txt b/test/gold/apps/acctExport/acctExport_monitors_err_watch.txt index 62aef639ab..f603369cbd 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_err_watch.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_err_watch.txt @@ -19,7 +19,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -27,7 +27,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_help.txt b/test/gold/apps/acctExport/acctExport_monitors_help.txt index 831bd90b20..bc4a945b20 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_help.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_help.txt @@ -18,7 +18,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -26,7 +26,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_long_help.txt b/test/gold/apps/acctExport/acctExport_monitors_long_help.txt index 0867f532da..d227a75c0b 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_long_help.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_long_help.txt @@ -18,7 +18,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -26,7 +26,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_no_params.txt b/test/gold/apps/acctExport/acctExport_monitors_no_params.txt index 8c8df94fbf..3edd3535ff 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_no_params.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_no_params.txt @@ -17,7 +17,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -25,7 +25,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_remove_fail.txt b/test/gold/apps/acctExport/acctExport_monitors_remove_fail.txt index f0b2f8b4c1..7d784762b7 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_remove_fail.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_remove_fail.txt @@ -19,7 +19,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -27,7 +27,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_undelete2.txt b/test/gold/apps/acctExport/acctExport_monitors_undelete2.txt index 215be6719d..362291cf3b 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_undelete2.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_undelete2.txt @@ -19,7 +19,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -27,7 +27,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/acctExport_monitors_undelete_fail.txt b/test/gold/apps/acctExport/acctExport_monitors_undelete_fail.txt index 046185fce4..cd099e2b93 100644 --- a/test/gold/apps/acctExport/acctExport_monitors_undelete_fail.txt +++ b/test/gold/apps/acctExport/acctExport_monitors_undelete_fail.txt @@ -19,7 +19,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -27,7 +27,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/acctExport/api_tests/acctExport_monitors_err_batch_size2.txt b/test/gold/apps/acctExport/api_tests/acctExport_monitors_err_batch_size2.txt index a0ee5db153..a52ba50515 100644 --- a/test/gold/apps/acctExport/api_tests/acctExport_monitors_err_batch_size2.txt +++ b/test/gold/apps/acctExport/api_tests/acctExport_monitors_err_batch_size2.txt @@ -1,6 +1,6 @@ monitors?addrs=0xf503017d7baf7fbc0fff7492b751025c6a78179b&fmt=json&batchSize=1 { "errors": [ - "The --batch-size option is not available without --watch." + "The --batch_size option is not available without --watch." ] } diff --git a/test/gold/apps/acctExport/api_tests/acctExport_monitors_err_run_once.txt b/test/gold/apps/acctExport/api_tests/acctExport_monitors_err_run_once.txt index 3c021f1e2f..c23a93e44a 100644 --- a/test/gold/apps/acctExport/api_tests/acctExport_monitors_err_run_once.txt +++ b/test/gold/apps/acctExport/api_tests/acctExport_monitors_err_run_once.txt @@ -1,6 +1,6 @@ -monitors?addrs=0xf503017d7baf7fbc0fff7492b751025c6a78179b&fmt=json&runOnce +monitors?addrs=0xf503017d7baf7fbc0fff7492b751025c6a78179b&fmt=json&runCount=1 { "errors": [ - "The --run-once option is not available without --watch." + "The --run_count option is not available without --watch." ] } diff --git a/test/gold/apps/blockScrape/blockScrape_by_file_bad.txt b/test/gold/apps/blockScrape/blockScrape_by_file_bad.txt index 60af34af7c..166e5092e4 100644 --- a/test/gold/apps/blockScrape/blockScrape_by_file_bad.txt +++ b/test/gold/apps/blockScrape/blockScrape_by_file_bad.txt @@ -9,6 +9,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_caps_allowed.txt b/test/gold/apps/blockScrape/blockScrape_caps_allowed.txt index 6783e6645e..e63efe5b72 100644 --- a/test/gold/apps/blockScrape/blockScrape_caps_allowed.txt +++ b/test/gold/apps/blockScrape/blockScrape_caps_allowed.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_1.txt b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_1.txt index f3eca432f3..f99a1a4700 100644 --- a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_1.txt +++ b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_1.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_10.txt b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_10.txt index 6a002f00ae..fa19bcdbeb 100644 --- a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_10.txt +++ b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_10.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_3.txt b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_3.txt index 397914c844..08a8b487d7 100644 --- a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_3.txt +++ b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_3.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_4.txt b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_4.txt index 9b00d04ccf..5c66b5152c 100644 --- a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_4.txt +++ b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_4.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_5.txt b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_5.txt index 0ff9aded76..66149ceffc 100644 --- a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_5.txt +++ b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_5.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_6.txt b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_6.txt index b9660b443f..a9f5ee82a4 100644 --- a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_6.txt +++ b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_6.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_7.txt b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_7.txt index 5098d60039..188b0a93e4 100644 --- a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_7.txt +++ b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_7.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_8.txt b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_8.txt index a53c5be0ad..5ef18c8e07 100644 --- a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_8.txt +++ b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_8.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_9.txt b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_9.txt index e85013bd3a..8171d834f5 100644 --- a/test/gold/apps/blockScrape/blockScrape_caps_disallowed_9.txt +++ b/test/gold/apps/blockScrape/blockScrape_caps_disallowed_9.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_help.txt b/test/gold/apps/blockScrape/blockScrape_help.txt index e856038a3b..1178c198a5 100644 --- a/test/gold/apps/blockScrape/blockScrape_help.txt +++ b/test/gold/apps/blockScrape/blockScrape_help.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_help_long.txt b/test/gold/apps/blockScrape/blockScrape_help_long.txt index 2a41400e4d..b2213e96d8 100644 --- a/test/gold/apps/blockScrape/blockScrape_help_long.txt +++ b/test/gold/apps/blockScrape/blockScrape_help_long.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_invalid.txt b/test/gold/apps/blockScrape/blockScrape_invalid.txt index c2366e5308..99f13b2646 100644 --- a/test/gold/apps/blockScrape/blockScrape_invalid.txt +++ b/test/gold/apps/blockScrape/blockScrape_invalid.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_listpins.txt b/test/gold/apps/blockScrape/blockScrape_listpins.txt index 1f4d1a7318..6c0463864d 100644 --- a/test/gold/apps/blockScrape/blockScrape_listpins.txt +++ b/test/gold/apps/blockScrape/blockScrape_listpins.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_mode_01_fail.txt b/test/gold/apps/blockScrape/blockScrape_mode_01_fail.txt index 7a2b64e743..b0f3f49bbe 100644 --- a/test/gold/apps/blockScrape/blockScrape_mode_01_fail.txt +++ b/test/gold/apps/blockScrape/blockScrape_mode_01_fail.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_nAProcs_invalid.txt b/test/gold/apps/blockScrape/blockScrape_nAProcs_invalid.txt index ba80013097..68f0846f7a 100644 --- a/test/gold/apps/blockScrape/blockScrape_nAProcs_invalid.txt +++ b/test/gold/apps/blockScrape/blockScrape_nAProcs_invalid.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_nBProcs_invalid.txt b/test/gold/apps/blockScrape/blockScrape_nBProcs_invalid.txt index ba80013097..68f0846f7a 100644 --- a/test/gold/apps/blockScrape/blockScrape_nBProcs_invalid.txt +++ b/test/gold/apps/blockScrape/blockScrape_nBProcs_invalid.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_nBlocks.txt b/test/gold/apps/blockScrape/blockScrape_nBlocks.txt index 68ddea0867..1f4742e219 100644 --- a/test/gold/apps/blockScrape/blockScrape_nBlocks.txt +++ b/test/gold/apps/blockScrape/blockScrape_nBlocks.txt @@ -1,5 +1,6 @@ chifra scrape --block_cnt 12 TEST[DATE|TIME] BlockCnt: 12 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Caps: -fmt,-noHeader,-file,-output,-append TEST[DATE|TIME] Format: txt Error: Cannot test block scraper @@ -12,6 +13,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_nBlocks_invalid.txt b/test/gold/apps/blockScrape/blockScrape_nBlocks_invalid.txt index aaf35726c9..98307111c3 100644 --- a/test/gold/apps/blockScrape/blockScrape_nBlocks_invalid.txt +++ b/test/gold/apps/blockScrape/blockScrape_nBlocks_invalid.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/blockScrape/blockScrape_real_run2.txt b/test/gold/apps/blockScrape/blockScrape_real_run2.txt index b902fd982d..3122ce85a0 100644 --- a/test/gold/apps/blockScrape/blockScrape_real_run2.txt +++ b/test/gold/apps/blockScrape/blockScrape_real_run2.txt @@ -1,5 +1,6 @@ chifra scrape --sleep .5 TEST[DATE|TIME] Sleep: 0.5 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Caps: -fmt,-noHeader,-file,-output,-append TEST[DATE|TIME] Format: txt Error: Cannot test block scraper @@ -12,6 +13,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/chifra/chifra_help_chunks.txt b/test/gold/apps/chifra/chifra_help_chunks.txt index d234f14524..efb23bbb7f 100644 --- a/test/gold/apps/chifra/chifra_help_chunks.txt +++ b/test/gold/apps/chifra/chifra_help_chunks.txt @@ -14,9 +14,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,3 +37,4 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chifra/chifra_help_init.txt b/test/gold/apps/chifra/chifra_help_init.txt index a79014affa..76c822d1ba 100644 --- a/test/gold/apps/chifra/chifra_help_init.txt +++ b/test/gold/apps/chifra/chifra_help_init.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/chifra/chifra_help_list.txt b/test/gold/apps/chifra/chifra_help_list.txt index 1628d5b3ee..d12827b42d 100644 --- a/test/gold/apps/chifra/chifra_help_list.txt +++ b/test/gold/apps/chifra/chifra_help_list.txt @@ -17,6 +17,7 @@ Flags: -c, --first_record uint the first record to process -e, --max_records uint the maximum number of records to process (default 250) -E, --reversed produce results in reverse chronological order + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -F, --first_block uint first block to export (inclusive, ignored when freshening) -L, --last_block uint last block to export (inclusive, ignored when freshening) -x, --fmt string export format, one of [none|json*|txt|csv] diff --git a/test/gold/apps/chifra/chifra_help_rm.txt b/test/gold/apps/chifra/chifra_help_rm.txt index 101d45c165..14992538db 100644 --- a/test/gold/apps/chifra/chifra_help_rm.txt +++ b/test/gold/apps/chifra/chifra_help_rm.txt @@ -18,7 +18,7 @@ Flags: -a, --watchlist string available with --watch option only, a file containing the addresses to watch -c, --commands string available with --watch option only, the file containing the list of commands to apply to each watched address -b, --batch_size uint available with --watch option only, the number of monitors to process in each batch (default 8) - -r, --run_once available with --watch option only, only run the monitor --watch commands once then quit + -u, --run_count uint available with --watch option only, run the monitor this many times, then quit (hidden) -s, --sleep float available with --watch option only, the number of seconds to sleep between runs (default 14) -x, --fmt string export format, one of [none|json*|txt|csv] -v, --verbose enable verbose output @@ -26,7 +26,7 @@ Flags: Notes: - An address must be either an ENS name or start with '0x' and be forty-two characters long. - - If no address is presented to the --clean command, all existing monitors are be cleaned. + - If no address is presented to the --clean command, all existing monitors will be cleaned. - The --watch option requires two additional parameters to be specified: --watchlist and --commands. - Addresses provided on the command line are ignored in --watch mode. - Providing the value existing to the --watchlist monitors all existing monitor files (see --list). diff --git a/test/gold/apps/chifra/chifra_help_scrape.txt b/test/gold/apps/chifra/chifra_help_scrape.txt index bb4e379b5a..4070d15b43 100644 --- a/test/gold/apps/chifra/chifra_help_scrape.txt +++ b/test/gold/apps/chifra/chifra_help_scrape.txt @@ -11,6 +11,9 @@ Flags: -r, --remote pin new chunks to the gateway (requires pinning service keys) -s, --sleep float seconds to sleep between scraper passes (default 14) -l, --start_block uint first block to visit when scraping (snapped back to most recent snap_to_grid mark) + -u, --run_count uint run the scraper this many times, then quit (hidden) + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") + -d, --dry_run show the configuration that would be applied if run,no changes are made (hidden) --apps_per_chunk uint the number of appearances to build into a chunk before consolidating it (hidden) (default 200000) --snap_to_grid uint an override to apps_per_chunk to snap-to-grid at every modulo of this value, this allows easier corrections to the index (hidden) (default 100000) --first_snap uint the first block at which snap_to_grid is enabled (hidden) diff --git a/test/gold/apps/chunkMan/chunkMan_addresses_default.txt b/test/gold/apps/chunkMan/chunkMan_addresses_default.txt index f963e167cb..297834abd1 100644 --- a/test/gold/apps/chunkMan/chunkMan_addresses_default.txt +++ b/test/gold/apps/chunkMan/chunkMan_addresses_default.txt @@ -1,5 +1,6 @@ chifra chunks addresses TEST[DATE|TIME] Mode: addresses +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt address range offset count 0x000d836201318ec6899a67540690382780743280 000000000-000000000 0 1 diff --git a/test/gold/apps/chunkMan/chunkMan_addresses_json.txt b/test/gold/apps/chunkMan/chunkMan_addresses_json.txt index 2b266a8161..f9226660a7 100644 --- a/test/gold/apps/chunkMan/chunkMan_addresses_json.txt +++ b/test/gold/apps/chunkMan/chunkMan_addresses_json.txt @@ -1,5 +1,6 @@ chifra chunks addresses --fmt json TEST[DATE|TIME] Mode: addresses +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: json { "data": [ diff --git a/test/gold/apps/chunkMan/chunkMan_appearances_default.txt b/test/gold/apps/chunkMan/chunkMan_appearances_default.txt index af878a5849..b9d3c42924 100644 --- a/test/gold/apps/chunkMan/chunkMan_appearances_default.txt +++ b/test/gold/apps/chunkMan/chunkMan_appearances_default.txt @@ -1,5 +1,6 @@ chifra chunks appearances TEST[DATE|TIME] Mode: appearances +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt blockNumber transactionIndex 0 0 diff --git a/test/gold/apps/chunkMan/chunkMan_appearances_txt.txt b/test/gold/apps/chunkMan/chunkMan_appearances_txt.txt index 9b68162a68..0656022295 100644 --- a/test/gold/apps/chunkMan/chunkMan_appearances_txt.txt +++ b/test/gold/apps/chunkMan/chunkMan_appearances_txt.txt @@ -1,5 +1,6 @@ chifra chunks appearances --fmt txt TEST[DATE|TIME] Mode: appearances +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt blockNumber transactionIndex 0 0 diff --git a/test/gold/apps/chunkMan/chunkMan_belongs.txt b/test/gold/apps/chunkMan/chunkMan_belongs.txt index f389363895..9349b29450 100644 --- a/test/gold/apps/chunkMan/chunkMan_belongs.txt +++ b/test/gold/apps/chunkMan/chunkMan_belongs.txt @@ -1,6 +1,7 @@ chifra chunks index --belongs 0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9 2287625 --fmt csv TEST[DATE|TIME] Mode: index TEST[DATE|TIME] Blocks: [2287625] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Belongs: [0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9] TEST[DATE|TIME] Format: csv Error: The --belongs option only works with --fmt json @@ -16,9 +17,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -37,4 +40,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_belongs2.txt b/test/gold/apps/chunkMan/chunkMan_belongs2.txt index 20169a53ad..5ebc7ff761 100644 --- a/test/gold/apps/chunkMan/chunkMan_belongs2.txt +++ b/test/gold/apps/chunkMan/chunkMan_belongs2.txt @@ -1,6 +1,7 @@ chifra chunks index --belongs 0x0000000000000214156104345760006001905490 0x12 TEST[DATE|TIME] Mode: index TEST[DATE|TIME] Blocks: [0x12] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Belongs: [0x0000000000000214156104345760006001905490] TEST[DATE|TIME] Format: json { diff --git a/test/gold/apps/chunkMan/chunkMan_block_not_found.txt b/test/gold/apps/chunkMan/chunkMan_block_not_found.txt index 7833bf54f5..05a1ec3379 100644 --- a/test/gold/apps/chunkMan/chunkMan_block_not_found.txt +++ b/test/gold/apps/chunkMan/chunkMan_block_not_found.txt @@ -1,6 +1,7 @@ chifra chunks index --belongs 0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9 0x0f1217b92276cd17608d4212879739e6a5ec388bd7a03bef9798655234afd2b2 TEST[DATE|TIME] Mode: index TEST[DATE|TIME] Blocks: [0x0f1217b92276cd17608d4212879739e6a5ec388bd7a03bef9798655234afd2b2] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Belongs: [0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9] TEST[DATE|TIME] Format: json { diff --git a/test/gold/apps/chunkMan/chunkMan_blooms_default.txt b/test/gold/apps/chunkMan/chunkMan_blooms_default.txt index dac6ecbbf8..7eeb8b5f78 100644 --- a/test/gold/apps/chunkMan/chunkMan_blooms_default.txt +++ b/test/gold/apps/chunkMan/chunkMan_blooms_default.txt @@ -1,5 +1,6 @@ chifra chunks blooms TEST[DATE|TIME] Mode: blooms +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt range magic hash nBlooms nInserted size byteWidth 000000000-000000000 0xdead 0x81ae14ba68e372bc9bd4a295b844abd8e72b1de10fcd706e624647701d911da1 1 8893 131114 131072 diff --git a/test/gold/apps/chunkMan/chunkMan_blooms_txt.txt b/test/gold/apps/chunkMan/chunkMan_blooms_txt.txt index 519d6eb826..4f92fa2167 100644 --- a/test/gold/apps/chunkMan/chunkMan_blooms_txt.txt +++ b/test/gold/apps/chunkMan/chunkMan_blooms_txt.txt @@ -1,5 +1,6 @@ chifra chunks blooms --fmt txt TEST[DATE|TIME] Mode: blooms +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt range magic hash nBlooms nInserted size byteWidth 000000000-000000000 0xdead 0x81ae14ba68e372bc9bd4a295b844abd8e72b1de10fcd706e624647701d911da1 1 8893 131114 131072 diff --git a/test/gold/apps/chunkMan/chunkMan_caps_allowed.txt b/test/gold/apps/chunkMan/chunkMan_caps_allowed.txt index 8f64486fb2..bcdb5105f1 100644 --- a/test/gold/apps/chunkMan/chunkMan_caps_allowed.txt +++ b/test/gold/apps/chunkMan/chunkMan_caps_allowed.txt @@ -14,9 +14,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,4 +37,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_caps_disallowed_1.txt b/test/gold/apps/chunkMan/chunkMan_caps_disallowed_1.txt index 0f790f2ae6..99d3ae5623 100644 --- a/test/gold/apps/chunkMan/chunkMan_caps_disallowed_1.txt +++ b/test/gold/apps/chunkMan/chunkMan_caps_disallowed_1.txt @@ -14,9 +14,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,4 +37,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_caps_disallowed_2.txt b/test/gold/apps/chunkMan/chunkMan_caps_disallowed_2.txt index 037a018acf..d395c8e657 100644 --- a/test/gold/apps/chunkMan/chunkMan_caps_disallowed_2.txt +++ b/test/gold/apps/chunkMan/chunkMan_caps_disallowed_2.txt @@ -14,9 +14,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,4 +37,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_caps_disallowed_3.txt b/test/gold/apps/chunkMan/chunkMan_caps_disallowed_3.txt index b1a0d19161..a68d664906 100644 --- a/test/gold/apps/chunkMan/chunkMan_caps_disallowed_3.txt +++ b/test/gold/apps/chunkMan/chunkMan_caps_disallowed_3.txt @@ -14,9 +14,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,4 +37,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_caps_disallowed_4.txt b/test/gold/apps/chunkMan/chunkMan_caps_disallowed_4.txt index 9c8fe0aac3..7d6bf6ac03 100644 --- a/test/gold/apps/chunkMan/chunkMan_caps_disallowed_4.txt +++ b/test/gold/apps/chunkMan/chunkMan_caps_disallowed_4.txt @@ -14,9 +14,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,4 +37,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_caps_disallowed_5.txt b/test/gold/apps/chunkMan/chunkMan_caps_disallowed_5.txt index 74dd6e27cc..38d71ca742 100644 --- a/test/gold/apps/chunkMan/chunkMan_caps_disallowed_5.txt +++ b/test/gold/apps/chunkMan/chunkMan_caps_disallowed_5.txt @@ -14,9 +14,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,4 +37,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_check_bad.txt b/test/gold/apps/chunkMan/chunkMan_check_bad.txt index 60b15ca7d3..798a8c855b 100644 --- a/test/gold/apps/chunkMan/chunkMan_check_bad.txt +++ b/test/gold/apps/chunkMan/chunkMan_check_bad.txt @@ -1,6 +1,7 @@ chifra chunks addresses --check TEST[DATE|TIME] Mode: addresses TEST[DATE|TIME] Check: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: The --check option is not available in addresses mode. Usage: @@ -15,9 +16,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -36,4 +39,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_clean_bad.txt b/test/gold/apps/chunkMan/chunkMan_clean_bad.txt index efd5b583f9..01b99c7620 100644 --- a/test/gold/apps/chunkMan/chunkMan_clean_bad.txt +++ b/test/gold/apps/chunkMan/chunkMan_clean_bad.txt @@ -14,9 +14,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,4 +37,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_fail_belongs_addr_wrong.txt b/test/gold/apps/chunkMan/chunkMan_fail_belongs_addr_wrong.txt index 8581d1e3f4..87c9967633 100644 --- a/test/gold/apps/chunkMan/chunkMan_fail_belongs_addr_wrong.txt +++ b/test/gold/apps/chunkMan/chunkMan_fail_belongs_addr_wrong.txt @@ -1,5 +1,6 @@ chifra chunks index 0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9 TEST[DATE|TIME] Mode: index +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Belongs: [0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9] TEST[DATE|TIME] Format: json Error: You must specify at least one block identifier with the --belongs option @@ -15,9 +16,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -36,4 +39,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_fail_belongs_no_addr.txt b/test/gold/apps/chunkMan/chunkMan_fail_belongs_no_addr.txt index eca1a00b63..066ce222ee 100644 --- a/test/gold/apps/chunkMan/chunkMan_fail_belongs_no_addr.txt +++ b/test/gold/apps/chunkMan/chunkMan_fail_belongs_no_addr.txt @@ -1,5 +1,6 @@ chifra chunks index --belongs 2287625 TEST[DATE|TIME] Mode: index +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Belongs: [2287625] TEST[DATE|TIME] Format: json Error: Please specify at least one valid Ethereum address. @@ -15,9 +16,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -36,4 +39,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_fail_belongs_no_blocks.txt b/test/gold/apps/chunkMan/chunkMan_fail_belongs_no_blocks.txt index 43f8a94832..9edc1421d1 100644 --- a/test/gold/apps/chunkMan/chunkMan_fail_belongs_no_blocks.txt +++ b/test/gold/apps/chunkMan/chunkMan_fail_belongs_no_blocks.txt @@ -1,5 +1,6 @@ chifra chunks index --belongs 0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9 TEST[DATE|TIME] Mode: index +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Belongs: [0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9] TEST[DATE|TIME] Format: json Error: You must specify at least one block identifier with the --belongs option @@ -15,9 +16,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -36,4 +39,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_fail_belongs_wrong_mode.txt b/test/gold/apps/chunkMan/chunkMan_fail_belongs_wrong_mode.txt index 07b0cca752..158193b400 100644 --- a/test/gold/apps/chunkMan/chunkMan_fail_belongs_wrong_mode.txt +++ b/test/gold/apps/chunkMan/chunkMan_fail_belongs_wrong_mode.txt @@ -1,5 +1,6 @@ chifra chunks addresses --belongs 0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9 TEST[DATE|TIME] Mode: addresses +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Belongs: [0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9] TEST[DATE|TIME] Format: json Error: The --belongs option requires the index mode. @@ -15,9 +16,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -36,4 +39,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_fail_both.txt b/test/gold/apps/chunkMan/chunkMan_fail_both.txt index f57396fd51..451d8683d8 100644 --- a/test/gold/apps/chunkMan/chunkMan_fail_both.txt +++ b/test/gold/apps/chunkMan/chunkMan_fail_both.txt @@ -1,5 +1,6 @@ chifra chunks --check TEST[DATE|TIME] Check: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: Please choose at least one of [manifest|index|blooms|addresses|appearances|stats]. Usage: @@ -14,9 +15,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,4 +38,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_fail_detail_belong.txt b/test/gold/apps/chunkMan/chunkMan_fail_detail_belong.txt index 62e12e6443..761f26f886 100644 --- a/test/gold/apps/chunkMan/chunkMan_fail_detail_belong.txt +++ b/test/gold/apps/chunkMan/chunkMan_fail_detail_belong.txt @@ -1,6 +1,7 @@ chifra chunks index --belongs 0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9 2287625 --verbose TEST[DATE|TIME] Mode: index TEST[DATE|TIME] Blocks: [2287625] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Belongs: [0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9] TEST[DATE|TIME] Verbose: true TEST[DATE|TIME] Format: json @@ -17,9 +18,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -38,4 +41,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_fail_neither.txt b/test/gold/apps/chunkMan/chunkMan_fail_neither.txt index 2abb699da2..677b4c26f3 100644 --- a/test/gold/apps/chunkMan/chunkMan_fail_neither.txt +++ b/test/gold/apps/chunkMan/chunkMan_fail_neither.txt @@ -1,4 +1,5 @@ chifra chunks +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: Please choose at least one of [manifest|index|blooms|addresses|appearances|stats]. Usage: @@ -13,9 +14,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -34,4 +37,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_header_csv.txt b/test/gold/apps/chunkMan/chunkMan_header_csv.txt index eaa12a3267..36311b7a3c 100644 --- a/test/gold/apps/chunkMan/chunkMan_header_csv.txt +++ b/test/gold/apps/chunkMan/chunkMan_header_csv.txt @@ -1,5 +1,6 @@ chifra chunks header --fmt csv TEST[DATE|TIME] Mode: header +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: csv Error: The mode option (header) must be one of [ manifest | index | blooms | addresses | appearances | stats ] Usage: @@ -14,9 +15,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,4 +38,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_header_default.txt b/test/gold/apps/chunkMan/chunkMan_header_default.txt index 05c543b640..97740b17b6 100644 --- a/test/gold/apps/chunkMan/chunkMan_header_default.txt +++ b/test/gold/apps/chunkMan/chunkMan_header_default.txt @@ -1,5 +1,6 @@ chifra chunks header TEST[DATE|TIME] Mode: header +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: The mode option (header) must be one of [ manifest | index | blooms | addresses | appearances | stats ] Usage: @@ -14,9 +15,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,4 +38,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_help.txt b/test/gold/apps/chunkMan/chunkMan_help.txt index 881330d563..566fb39ac0 100644 --- a/test/gold/apps/chunkMan/chunkMan_help.txt +++ b/test/gold/apps/chunkMan/chunkMan_help.txt @@ -14,9 +14,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,3 +37,4 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_help_long.txt b/test/gold/apps/chunkMan/chunkMan_help_long.txt index dada649588..1ea52502b3 100644 --- a/test/gold/apps/chunkMan/chunkMan_help_long.txt +++ b/test/gold/apps/chunkMan/chunkMan_help_long.txt @@ -14,9 +14,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,3 +37,4 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_index_default.txt b/test/gold/apps/chunkMan/chunkMan_index_default.txt index 921cf19065..a3f0cd381b 100644 --- a/test/gold/apps/chunkMan/chunkMan_index_default.txt +++ b/test/gold/apps/chunkMan/chunkMan_index_default.txt @@ -1,5 +1,6 @@ chifra chunks index TEST[DATE|TIME] Mode: index +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt range magic hash nAddresses nAppearances fileSize 000000000-000000000 0xdeadbeef 0x81ae...1da1 8893 8893 320192 diff --git a/test/gold/apps/chunkMan/chunkMan_index_json.txt b/test/gold/apps/chunkMan/chunkMan_index_json.txt index d3af12e40f..ff737dca60 100644 --- a/test/gold/apps/chunkMan/chunkMan_index_json.txt +++ b/test/gold/apps/chunkMan/chunkMan_index_json.txt @@ -1,5 +1,6 @@ chifra chunks index --fmt json TEST[DATE|TIME] Mode: index +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: json { "data": [ diff --git a/test/gold/apps/chunkMan/chunkMan_invalid_param_1.txt b/test/gold/apps/chunkMan/chunkMan_invalid_param_1.txt index 9f675d573d..99c0706ba3 100644 --- a/test/gold/apps/chunkMan/chunkMan_invalid_param_1.txt +++ b/test/gold/apps/chunkMan/chunkMan_invalid_param_1.txt @@ -1,5 +1,6 @@ chifra chunks junk TEST[DATE|TIME] Mode: junk +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: The mode option (junk) must be one of [ manifest | index | blooms | addresses | appearances | stats ] Usage: @@ -14,9 +15,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,4 +38,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_list_check.txt b/test/gold/apps/chunkMan/chunkMan_list_check.txt index 705fb735ea..95fc16923e 100644 --- a/test/gold/apps/chunkMan/chunkMan_list_check.txt +++ b/test/gold/apps/chunkMan/chunkMan_list_check.txt @@ -1,6 +1,7 @@ chifra chunks monitors --check TEST[DATE|TIME] Mode: monitors TEST[DATE|TIME] Check: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: The mode option (monitors) must be one of [ manifest | index | blooms | addresses | appearances | stats ] Usage: @@ -15,9 +16,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -36,4 +39,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_manifest_csv.txt b/test/gold/apps/chunkMan/chunkMan_manifest_csv.txt index b143ecdde9..66b01e709e 100644 --- a/test/gold/apps/chunkMan/chunkMan_manifest_csv.txt +++ b/test/gold/apps/chunkMan/chunkMan_manifest_csv.txt @@ -1,5 +1,6 @@ chifra chunks manifest --fmt csv TEST[DATE|TIME] Mode: manifest +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: csv range,bloomHash,bloomSize,indexHash,indexSize 000000000-000000000,QmYhuaJu9bHAGpSsuaQug7bjnLcoE6B5PCZsg4XZGVFbKy,131114,QmaKUsfH5AXqPJgjGAQFZWRheF1jtc5PqGi8cYrwmMCXdu,320192 diff --git a/test/gold/apps/chunkMan/chunkMan_manifest_default.txt b/test/gold/apps/chunkMan/chunkMan_manifest_default.txt index 1b0b095a52..51ec4c568f 100644 --- a/test/gold/apps/chunkMan/chunkMan_manifest_default.txt +++ b/test/gold/apps/chunkMan/chunkMan_manifest_default.txt @@ -1,5 +1,6 @@ chifra chunks manifest TEST[DATE|TIME] Mode: manifest +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt range bloomHash bloomSize indexHash indexSize 000000000-000000000 QmYhuaJu9bHAGpSsuaQug7bjnLcoE6B5PCZsg4XZGVFbKy 131114 QmaKUsfH5AXqPJgjGAQFZWRheF1jtc5PqGi8cYrwmMCXdu 320192 diff --git a/test/gold/apps/chunkMan/chunkMan_no_params.txt b/test/gold/apps/chunkMan/chunkMan_no_params.txt index 2abb699da2..677b4c26f3 100644 --- a/test/gold/apps/chunkMan/chunkMan_no_params.txt +++ b/test/gold/apps/chunkMan/chunkMan_no_params.txt @@ -1,4 +1,5 @@ chifra chunks +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: Please choose at least one of [manifest|index|blooms|addresses|appearances|stats]. Usage: @@ -13,9 +14,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -34,4 +37,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_pin_chunks_bad.txt b/test/gold/apps/chunkMan/chunkMan_pin_chunks_bad.txt index 6020cd67f9..c4ade82118 100644 --- a/test/gold/apps/chunkMan/chunkMan_pin_chunks_bad.txt +++ b/test/gold/apps/chunkMan/chunkMan_pin_chunks_bad.txt @@ -1,6 +1,7 @@ chifra chunks addresses --pin TEST[DATE|TIME] Mode: addresses TEST[DATE|TIME] Pin: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: The --pin option is not available in addresses mode. Usage: @@ -15,9 +16,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -36,4 +39,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_pin_chunks_bad2.txt b/test/gold/apps/chunkMan/chunkMan_pin_chunks_bad2.txt index 6020cd67f9..c4ade82118 100644 --- a/test/gold/apps/chunkMan/chunkMan_pin_chunks_bad2.txt +++ b/test/gold/apps/chunkMan/chunkMan_pin_chunks_bad2.txt @@ -1,6 +1,7 @@ chifra chunks addresses --pin TEST[DATE|TIME] Mode: addresses TEST[DATE|TIME] Pin: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: The --pin option is not available in addresses mode. Usage: @@ -15,9 +16,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -36,4 +39,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_pin_data_bad.txt b/test/gold/apps/chunkMan/chunkMan_pin_data_bad.txt index e72bdaadcf..409ac8b039 100644 --- a/test/gold/apps/chunkMan/chunkMan_pin_data_bad.txt +++ b/test/gold/apps/chunkMan/chunkMan_pin_data_bad.txt @@ -1,6 +1,7 @@ chifra chunks manifest --publish TEST[DATE|TIME] Mode: manifest TEST[DATE|TIME] Publish: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: The --publish option is not available in test mode. Usage: @@ -15,9 +16,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -36,4 +39,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_pin_data_bad2.txt b/test/gold/apps/chunkMan/chunkMan_pin_data_bad2.txt index 2625843f07..f497b162a3 100644 --- a/test/gold/apps/chunkMan/chunkMan_pin_data_bad2.txt +++ b/test/gold/apps/chunkMan/chunkMan_pin_data_bad2.txt @@ -1,6 +1,7 @@ chifra chunks addresses --publish TEST[DATE|TIME] Mode: addresses TEST[DATE|TIME] Publish: true +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: The --publish option is not available in addresses mode. Usage: @@ -15,9 +16,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -36,4 +39,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_redir_output.txt b/test/gold/apps/chunkMan/chunkMan_redir_output.txt index 45eab117b6..c9452dc524 100644 --- a/test/gold/apps/chunkMan/chunkMan_redir_output.txt +++ b/test/gold/apps/chunkMan/chunkMan_redir_output.txt @@ -1,6 +1,7 @@ chifra chunks index 2000000 --output output_test_file TEST[DATE|TIME] Mode: index TEST[DATE|TIME] Blocks: [2000000] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] OutputFn: output_test_file TEST[DATE|TIME] Format: txt ---- diff --git a/test/gold/apps/chunkMan/chunkMan_redir_output_append.txt b/test/gold/apps/chunkMan/chunkMan_redir_output_append.txt index 7f3a341d72..0b6bd90a44 100644 --- a/test/gold/apps/chunkMan/chunkMan_redir_output_append.txt +++ b/test/gold/apps/chunkMan/chunkMan_redir_output_append.txt @@ -1,6 +1,7 @@ chifra chunks index 3000000 --output output_test_file --no_header --append TEST[DATE|TIME] Mode: index TEST[DATE|TIME] Blocks: [3000000] +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] NoHeader: true TEST[DATE|TIME] OutputFn: output_test_file TEST[DATE|TIME] Append: true diff --git a/test/gold/apps/chunkMan/chunkMan_remote_fail.txt b/test/gold/apps/chunkMan/chunkMan_remote_fail.txt index 8fe79ca8ed..93645c8796 100644 --- a/test/gold/apps/chunkMan/chunkMan_remote_fail.txt +++ b/test/gold/apps/chunkMan/chunkMan_remote_fail.txt @@ -1,5 +1,6 @@ chifra chunks remote TEST[DATE|TIME] Mode: remote +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt Error: The mode option (remote) must be one of [ manifest | index | blooms | addresses | appearances | stats ] Usage: @@ -14,9 +15,11 @@ Flags: -c, --check check the manifest, index, or blooms for internal consistency -i, --pin pin the manifest or each index chunk and bloom -p, --publish publish the manifest to the Unchained Index smart contract + -P, --publisher string for some query options, the publisher of the index (hidden) (default "trueblocks.eth") -n, --truncate uint truncate the entire index at this block (requires a block identifier) (hidden) -r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract -b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk + -f, --diff compare two index portions (see notes) (hidden) -F, --first_block uint first block to process (inclusive) -L, --last_block uint last block to process (inclusive) -m, --max_addrs uint the max number of addresses to process in a given chunk @@ -35,4 +38,5 @@ Notes: - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode. - The --pin option requires a locally running IPFS node or a pinning service API key. - The --publish option requires a private key. + - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher. diff --git a/test/gold/apps/chunkMan/chunkMan_stats_default.txt b/test/gold/apps/chunkMan/chunkMan_stats_default.txt index 91be3a0348..a899949bf6 100644 --- a/test/gold/apps/chunkMan/chunkMan_stats_default.txt +++ b/test/gold/apps/chunkMan/chunkMan_stats_default.txt @@ -1,5 +1,6 @@ chifra chunks stats TEST[DATE|TIME] Mode: stats +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt range rangeEnd nAddrs nApps nBlocks nBlooms recWid bloomSz chunkSz addrsPerBlock appsPerBlock appsPerAddr ratio 000000000-000000000 2015-07-30 15:26:15 UTC 8893 8893 1 1 131076 131114 320192 8893 8893 1 2.4420885641502816 diff --git a/test/gold/apps/chunkMan/chunkMan_stats_txt.txt b/test/gold/apps/chunkMan/chunkMan_stats_txt.txt index 19955c3614..4a5ad84d56 100644 --- a/test/gold/apps/chunkMan/chunkMan_stats_txt.txt +++ b/test/gold/apps/chunkMan/chunkMan_stats_txt.txt @@ -1,5 +1,6 @@ chifra chunks stats --fmt txt TEST[DATE|TIME] Mode: stats +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Format: txt range rangeEnd nAddrs nApps nBlocks nBlooms recWid bloomSz chunkSz addrsPerBlock appsPerBlock appsPerAddr ratio 000000000-000000000 2015-07-30 15:26:15 UTC 8893 8893 1 1 131076 131114 320192 8893 8893 1 2.4420885641502816 diff --git a/test/gold/apps/init/init_by_file_bad.txt b/test/gold/apps/init/init_by_file_bad.txt index f62e45e6e6..1228c00bbd 100644 --- a/test/gold/apps/init/init_by_file_bad.txt +++ b/test/gold/apps/init/init_by_file_bad.txt @@ -6,6 +6,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_caps_allowed.txt b/test/gold/apps/init/init_caps_allowed.txt index 9943a6054a..dfd9b9613a 100644 --- a/test/gold/apps/init/init_caps_allowed.txt +++ b/test/gold/apps/init/init_caps_allowed.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_caps_disallowed_1.txt b/test/gold/apps/init/init_caps_disallowed_1.txt index a58cfe94cf..1133900df6 100644 --- a/test/gold/apps/init/init_caps_disallowed_1.txt +++ b/test/gold/apps/init/init_caps_disallowed_1.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_caps_disallowed_2.txt b/test/gold/apps/init/init_caps_disallowed_2.txt index d9113c9dd1..1e0bcb8de3 100644 --- a/test/gold/apps/init/init_caps_disallowed_2.txt +++ b/test/gold/apps/init/init_caps_disallowed_2.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_caps_disallowed_3.txt b/test/gold/apps/init/init_caps_disallowed_3.txt index 3057e11442..91638a9303 100644 --- a/test/gold/apps/init/init_caps_disallowed_3.txt +++ b/test/gold/apps/init/init_caps_disallowed_3.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_caps_disallowed_4.txt b/test/gold/apps/init/init_caps_disallowed_4.txt index 426e4be9c2..187d31a08f 100644 --- a/test/gold/apps/init/init_caps_disallowed_4.txt +++ b/test/gold/apps/init/init_caps_disallowed_4.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_caps_disallowed_5.txt b/test/gold/apps/init/init_caps_disallowed_5.txt index ac40f20e5c..ebd4c6feae 100644 --- a/test/gold/apps/init/init_caps_disallowed_5.txt +++ b/test/gold/apps/init/init_caps_disallowed_5.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_caps_disallowed_6.txt b/test/gold/apps/init/init_caps_disallowed_6.txt index cf1d10ae53..598170e42d 100644 --- a/test/gold/apps/init/init_caps_disallowed_6.txt +++ b/test/gold/apps/init/init_caps_disallowed_6.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_caps_disallowed_7.txt b/test/gold/apps/init/init_caps_disallowed_7.txt index 1f02c7d4cf..891010a05d 100644 --- a/test/gold/apps/init/init_caps_disallowed_7.txt +++ b/test/gold/apps/init/init_caps_disallowed_7.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_first_block.txt b/test/gold/apps/init/init_first_block.txt index 3c649b237a..f01c19c1da 100644 --- a/test/gold/apps/init/init_first_block.txt +++ b/test/gold/apps/init/init_first_block.txt @@ -1,4 +1,5 @@ chifra init --first_block 12 +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] FirstBlock: 12 TEST[DATE|TIME] Caps: -fmt,-noHeader,-file,-output,-append TEST[DATE|TIME] Format: txt @@ -9,6 +10,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_fmt_api.txt b/test/gold/apps/init/init_fmt_api.txt index 6aa1546a68..db9156d3ee 100644 --- a/test/gold/apps/init/init_fmt_api.txt +++ b/test/gold/apps/init/init_fmt_api.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_fmt_csv.txt b/test/gold/apps/init/init_fmt_csv.txt index 38db782e02..466c7768d9 100644 --- a/test/gold/apps/init/init_fmt_csv.txt +++ b/test/gold/apps/init/init_fmt_csv.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_fmt_default.txt b/test/gold/apps/init/init_fmt_default.txt index f976d41d67..dc2069cf25 100644 --- a/test/gold/apps/init/init_fmt_default.txt +++ b/test/gold/apps/init/init_fmt_default.txt @@ -1,4 +1,5 @@ chifra init +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Caps: -fmt,-noHeader,-file,-output,-append TEST[DATE|TIME] Format: txt Error: integration testing was skipped for chifra init @@ -8,6 +9,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_fmt_json.txt b/test/gold/apps/init/init_fmt_json.txt index 9cec668489..7817f0ec92 100644 --- a/test/gold/apps/init/init_fmt_json.txt +++ b/test/gold/apps/init/init_fmt_json.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_fmt_junk.txt b/test/gold/apps/init/init_fmt_junk.txt index f143e3b63a..cdafd44e48 100644 --- a/test/gold/apps/init/init_fmt_junk.txt +++ b/test/gold/apps/init/init_fmt_junk.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_fmt_txt.txt b/test/gold/apps/init/init_fmt_txt.txt index d7cc7769b1..a5b324bb24 100644 --- a/test/gold/apps/init/init_fmt_txt.txt +++ b/test/gold/apps/init/init_fmt_txt.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_help.txt b/test/gold/apps/init/init_help.txt index ca0de64a7d..e31fb1c120 100644 --- a/test/gold/apps/init/init_help.txt +++ b/test/gold/apps/init/init_help.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_help_long.txt b/test/gold/apps/init/init_help_long.txt index 2636510ebf..b09297c535 100644 --- a/test/gold/apps/init/init_help_long.txt +++ b/test/gold/apps/init/init_help_long.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_invalid_param_1.txt b/test/gold/apps/init/init_invalid_param_1.txt index c2175021ad..3f1cd20db9 100644 --- a/test/gold/apps/init/init_invalid_param_1.txt +++ b/test/gold/apps/init/init_invalid_param_1.txt @@ -8,6 +8,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_list_local.txt b/test/gold/apps/init/init_list_local.txt index f976d41d67..dc2069cf25 100644 --- a/test/gold/apps/init/init_list_local.txt +++ b/test/gold/apps/init/init_list_local.txt @@ -1,4 +1,5 @@ chifra init +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Caps: -fmt,-noHeader,-file,-output,-append TEST[DATE|TIME] Format: txt Error: integration testing was skipped for chifra init @@ -8,6 +9,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/apps/init/init_no_params.txt b/test/gold/apps/init/init_no_params.txt index f976d41d67..dc2069cf25 100644 --- a/test/gold/apps/init/init_no_params.txt +++ b/test/gold/apps/init/init_no_params.txt @@ -1,4 +1,5 @@ chifra init +TEST[DATE|TIME] Publisher: 0xf503017d7baf7fbc0fff7492b751025c6a78179b TEST[DATE|TIME] Caps: -fmt,-noHeader,-file,-output,-append TEST[DATE|TIME] Format: txt Error: integration testing was skipped for chifra init @@ -8,6 +9,7 @@ Usage: Flags: -a, --all in addition to Bloom filters, download full index chunks (recommended) -d, --dry_run display the results of the download without actually downloading + -P, --publisher string the publisher of the index to download (hidden) (default "trueblocks.eth") -F, --first_block uint do not download any chunks earlier than this block -s, --sleep float seconds to sleep between downloads -v, --verbose enable verbose output diff --git a/test/gold/tools/ethNames/api_tests/ethNames_edit_add.txt b/test/gold/tools/ethNames/api_tests/ethNames_edit_add.txt index 0496946c91..236fd66e97 100644 --- a/test/gold/tools/ethNames/api_tests/ethNames_edit_add.txt +++ b/test/gold/tools/ethNames/api_tests/ethNames_edit_add.txt @@ -5,48 +5,6 @@ Env: TB_NAME_SOURCE = SOURCE Env: TB_NAME_SYMBOL = SYMBOL Env: TB_NAME_DECIMALS = 11 Env: TB_NAME_DESCR = DESCRIPTION -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME -Env: TB_NAME_TAG = TAGS -Env: TB_NAME_SOURCE = SOURCE -Env: TB_NAME_SYMBOL = SYMBOL -Env: TB_NAME_DECIMALS = 11 -Env: TB_NAME_DESCR = DESCRIPTION -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME -Env: TB_NAME_TAG = TAGS -Env: TB_NAME_SOURCE = SOURCE -Env: TB_NAME_SYMBOL = SYMBOL -Env: TB_NAME_DECIMALS = 11 -Env: TB_NAME_DESCR = DESCRIPTION -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME -Env: TB_NAME_TAG = TAGS -Env: TB_NAME_SOURCE = SOURCE -Env: TB_NAME_SYMBOL = SYMBOL -Env: TB_NAME_DECIMALS = 11 -Env: TB_NAME_DESCR = DESCRIPTION -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME -Env: TB_NAME_TAG = TAGS -Env: TB_NAME_SOURCE = SOURCE -Env: TB_NAME_SYMBOL = SYMBOL -Env: TB_NAME_DECIMALS = 11 -Env: TB_NAME_DESCR = DESCRIPTION -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME -Env: TB_NAME_TAG = TAGS -Env: TB_NAME_SOURCE = SOURCE -Env: TB_NAME_SYMBOL = SYMBOL -Env: TB_NAME_DECIMALS = 11 -Env: TB_NAME_DESCR = DESCRIPTION -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME -Env: TB_NAME_TAG = TAGS -Env: TB_NAME_SOURCE = SOURCE -Env: TB_NAME_SYMBOL = SYMBOL -Env: TB_NAME_DECIMALS = 11 -Env: TB_NAME_DESCR = DESCRIPTION CURL: -X POST CURL: -H "Content-Type: application/x-www-form-urlencoded" CURL: -d "address=0x0000000000000000000000000000000010101010&name=NAME&tag=TAGS&source=SOURCE&symbol=SYMBOL&decimals=11&description=DESCRIPTION" diff --git a/test/gold/tools/ethNames/api_tests/ethNames_edit_edit.txt b/test/gold/tools/ethNames/api_tests/ethNames_edit_edit.txt index 14a54dbdec..71c2d3cb05 100644 --- a/test/gold/tools/ethNames/api_tests/ethNames_edit_edit.txt +++ b/test/gold/tools/ethNames/api_tests/ethNames_edit_edit.txt @@ -5,48 +5,6 @@ Env: TB_NAME_SOURCE = SOURCE_CHANGED Env: TB_NAME_SYMBOL = SYMBOL_CHANGED Env: TB_NAME_DECIMALS = 13 Env: TB_NAME_DESCR = DESCRIPTION_CHANGED -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME_CHANGED -Env: TB_NAME_TAG = TAGS_CHANGED -Env: TB_NAME_SOURCE = SOURCE_CHANGED -Env: TB_NAME_SYMBOL = SYMBOL_CHANGED -Env: TB_NAME_DECIMALS = 13 -Env: TB_NAME_DESCR = DESCRIPTION_CHANGED -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME_CHANGED -Env: TB_NAME_TAG = TAGS_CHANGED -Env: TB_NAME_SOURCE = SOURCE_CHANGED -Env: TB_NAME_SYMBOL = SYMBOL_CHANGED -Env: TB_NAME_DECIMALS = 13 -Env: TB_NAME_DESCR = DESCRIPTION_CHANGED -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME_CHANGED -Env: TB_NAME_TAG = TAGS_CHANGED -Env: TB_NAME_SOURCE = SOURCE_CHANGED -Env: TB_NAME_SYMBOL = SYMBOL_CHANGED -Env: TB_NAME_DECIMALS = 13 -Env: TB_NAME_DESCR = DESCRIPTION_CHANGED -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME_CHANGED -Env: TB_NAME_TAG = TAGS_CHANGED -Env: TB_NAME_SOURCE = SOURCE_CHANGED -Env: TB_NAME_SYMBOL = SYMBOL_CHANGED -Env: TB_NAME_DECIMALS = 13 -Env: TB_NAME_DESCR = DESCRIPTION_CHANGED -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME_CHANGED -Env: TB_NAME_TAG = TAGS_CHANGED -Env: TB_NAME_SOURCE = SOURCE_CHANGED -Env: TB_NAME_SYMBOL = SYMBOL_CHANGED -Env: TB_NAME_DECIMALS = 13 -Env: TB_NAME_DESCR = DESCRIPTION_CHANGED -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME_CHANGED -Env: TB_NAME_TAG = TAGS_CHANGED -Env: TB_NAME_SOURCE = SOURCE_CHANGED -Env: TB_NAME_SYMBOL = SYMBOL_CHANGED -Env: TB_NAME_DECIMALS = 13 -Env: TB_NAME_DESCR = DESCRIPTION_CHANGED CURL: -X PUT CURL: -H "Content-Type: application/x-www-form-urlencoded" CURL: -d "address=0x0000000000000000000000000000000010101010&name=NAME_CHANGED&tag=TAGS_CHANGED&source=SOURCE_CHANGED&symbol=SYMBOL_CHANGED&decimals=13&description=DESCRIPTION_CHANGED" diff --git a/test/gold/tools/ethNames/ethNames_edit_add.txt b/test/gold/tools/ethNames/ethNames_edit_add.txt index 7182e600c7..edd157c8df 100644 --- a/test/gold/tools/ethNames/ethNames_edit_add.txt +++ b/test/gold/tools/ethNames/ethNames_edit_add.txt @@ -5,48 +5,6 @@ Env: TB_NAME_SOURCE = SOURCE Env: TB_NAME_SYMBOL = SYMBOL Env: TB_NAME_DECIMALS = 11 Env: TB_NAME_DESCR = DESCRIPTION -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME -Env: TB_NAME_TAG = TAGS -Env: TB_NAME_SOURCE = SOURCE -Env: TB_NAME_SYMBOL = SYMBOL -Env: TB_NAME_DECIMALS = 11 -Env: TB_NAME_DESCR = DESCRIPTION -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME -Env: TB_NAME_TAG = TAGS -Env: TB_NAME_SOURCE = SOURCE -Env: TB_NAME_SYMBOL = SYMBOL -Env: TB_NAME_DECIMALS = 11 -Env: TB_NAME_DESCR = DESCRIPTION -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME -Env: TB_NAME_TAG = TAGS -Env: TB_NAME_SOURCE = SOURCE -Env: TB_NAME_SYMBOL = SYMBOL -Env: TB_NAME_DECIMALS = 11 -Env: TB_NAME_DESCR = DESCRIPTION -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME -Env: TB_NAME_TAG = TAGS -Env: TB_NAME_SOURCE = SOURCE -Env: TB_NAME_SYMBOL = SYMBOL -Env: TB_NAME_DECIMALS = 11 -Env: TB_NAME_DESCR = DESCRIPTION -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME -Env: TB_NAME_TAG = TAGS -Env: TB_NAME_SOURCE = SOURCE -Env: TB_NAME_SYMBOL = SYMBOL -Env: TB_NAME_DECIMALS = 11 -Env: TB_NAME_DESCR = DESCRIPTION -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME -Env: TB_NAME_TAG = TAGS -Env: TB_NAME_SOURCE = SOURCE -Env: TB_NAME_SYMBOL = SYMBOL -Env: TB_NAME_DECIMALS = 11 -Env: TB_NAME_DESCR = DESCRIPTION chifra names --create --fmt json TEST[DATE|TIME] Create: true TEST[DATE|TIME] Format: json diff --git a/test/gold/tools/ethNames/ethNames_edit_edit.txt b/test/gold/tools/ethNames/ethNames_edit_edit.txt index c689e8fbc1..bc1d9dec83 100644 --- a/test/gold/tools/ethNames/ethNames_edit_edit.txt +++ b/test/gold/tools/ethNames/ethNames_edit_edit.txt @@ -5,48 +5,6 @@ Env: TB_NAME_SOURCE = SOURCE_CHANGED Env: TB_NAME_SYMBOL = SYMBOL_CHANGED Env: TB_NAME_DECIMALS = 13 Env: TB_NAME_DESCR = DESCRIPTION_CHANGED -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME_CHANGED -Env: TB_NAME_TAG = TAGS_CHANGED -Env: TB_NAME_SOURCE = SOURCE_CHANGED -Env: TB_NAME_SYMBOL = SYMBOL_CHANGED -Env: TB_NAME_DECIMALS = 13 -Env: TB_NAME_DESCR = DESCRIPTION_CHANGED -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME_CHANGED -Env: TB_NAME_TAG = TAGS_CHANGED -Env: TB_NAME_SOURCE = SOURCE_CHANGED -Env: TB_NAME_SYMBOL = SYMBOL_CHANGED -Env: TB_NAME_DECIMALS = 13 -Env: TB_NAME_DESCR = DESCRIPTION_CHANGED -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME_CHANGED -Env: TB_NAME_TAG = TAGS_CHANGED -Env: TB_NAME_SOURCE = SOURCE_CHANGED -Env: TB_NAME_SYMBOL = SYMBOL_CHANGED -Env: TB_NAME_DECIMALS = 13 -Env: TB_NAME_DESCR = DESCRIPTION_CHANGED -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME_CHANGED -Env: TB_NAME_TAG = TAGS_CHANGED -Env: TB_NAME_SOURCE = SOURCE_CHANGED -Env: TB_NAME_SYMBOL = SYMBOL_CHANGED -Env: TB_NAME_DECIMALS = 13 -Env: TB_NAME_DESCR = DESCRIPTION_CHANGED -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME_CHANGED -Env: TB_NAME_TAG = TAGS_CHANGED -Env: TB_NAME_SOURCE = SOURCE_CHANGED -Env: TB_NAME_SYMBOL = SYMBOL_CHANGED -Env: TB_NAME_DECIMALS = 13 -Env: TB_NAME_DESCR = DESCRIPTION_CHANGED -Env: TB_NAME_ADDRESS = 0x0000000000000000000000000000000010101010 -Env: TB_NAME_NAME = NAME_CHANGED -Env: TB_NAME_TAG = TAGS_CHANGED -Env: TB_NAME_SOURCE = SOURCE_CHANGED -Env: TB_NAME_SYMBOL = SYMBOL_CHANGED -Env: TB_NAME_DECIMALS = 13 -Env: TB_NAME_DESCR = DESCRIPTION_CHANGED chifra names --update --fmt json TEST[DATE|TIME] Update: true TEST[DATE|TIME] Format: json diff --git a/test/gold/tools/getBlocks/api_tests/getBlocks_decache.txt b/test/gold/tools/getBlocks/api_tests/getBlocks_decache.txt index f032196c49..377b5b351e 100644 --- a/test/gold/tools/getBlocks/api_tests/getBlocks_decache.txt +++ b/test/gold/tools/getBlocks/api_tests/getBlocks_decache.txt @@ -2,7 +2,7 @@ blocks?blocks=1152642&decache { "data": [ { - "msg": "5 items totaling 1897 bytes were removed from the blocks cache." + "msg": "5 items totaling 1905 bytes were removed from the blocks cache." } ], "meta": { diff --git a/test/gold/tools/getBlocks/api_tests/getBlocks_decache_one.txt b/test/gold/tools/getBlocks/api_tests/getBlocks_decache_one.txt index a9e9699ac0..d87e4d4471 100644 --- a/test/gold/tools/getBlocks/api_tests/getBlocks_decache_one.txt +++ b/test/gold/tools/getBlocks/api_tests/getBlocks_decache_one.txt @@ -2,7 +2,7 @@ blocks?blocks=12&decache { "data": [ { - "msg": "1 items totaling 161 bytes were removed from the blocks cache." + "msg": "1 items totaling 169 bytes were removed from the blocks cache." } ], "meta": { diff --git a/test/gold/tools/getBlocks/api_tests/getBlocks_decache_traces.txt b/test/gold/tools/getBlocks/api_tests/getBlocks_decache_traces.txt index a9e9699ac0..d87e4d4471 100644 --- a/test/gold/tools/getBlocks/api_tests/getBlocks_decache_traces.txt +++ b/test/gold/tools/getBlocks/api_tests/getBlocks_decache_traces.txt @@ -2,7 +2,7 @@ blocks?blocks=12&decache { "data": [ { - "msg": "1 items totaling 161 bytes were removed from the blocks cache." + "msg": "1 items totaling 169 bytes were removed from the blocks cache." } ], "meta": { diff --git a/test/gold/tools/getBlocks/api_tests/getBlocks_decache_txs.txt b/test/gold/tools/getBlocks/api_tests/getBlocks_decache_txs.txt index a9e9699ac0..d87e4d4471 100644 --- a/test/gold/tools/getBlocks/api_tests/getBlocks_decache_txs.txt +++ b/test/gold/tools/getBlocks/api_tests/getBlocks_decache_txs.txt @@ -2,7 +2,7 @@ blocks?blocks=12&decache { "data": [ { - "msg": "1 items totaling 161 bytes were removed from the blocks cache." + "msg": "1 items totaling 169 bytes were removed from the blocks cache." } ], "meta": { diff --git a/test/gold/tools/getBlocks/api_tests/getBlocks_raw_01.txt b/test/gold/tools/getBlocks/api_tests/getBlocks_raw_01.txt index cfdeb9121a..f0fea4468e 100644 --- a/test/gold/tools/getBlocks/api_tests/getBlocks_raw_01.txt +++ b/test/gold/tools/getBlocks/api_tests/getBlocks_raw_01.txt @@ -58,7 +58,8 @@ blocks?blocks=2768801%201995314%201958017&raw } ], "transactionsRoot": "0xb82a3089b32828fdf00c1dc690253c4c171cf5e31cde2a19818a219d991841fd", - "uncles": [] + "uncles": [], + "withdrawals": null }, { "author": "", @@ -134,7 +135,8 @@ blocks?blocks=2768801%201995314%201958017&raw } ], "transactionsRoot": "0x976cfa98e970746ea6c254552dd31cb2a3f7cc38b0e08a744c91aaf1fbbbd84e", - "uncles": [] + "uncles": [], + "withdrawals": null }, { "author": "", @@ -195,7 +197,8 @@ blocks?blocks=2768801%201995314%201958017&raw } ], "transactionsRoot": "0x431798971e1b33824842c82e8ae24c3078863f5fd47e373a068f22cab63bab5f", - "uncles": [] + "uncles": [], + "withdrawals": null } ], "meta": { diff --git a/test/gold/tools/getBlocks/api_tests/getBlocks_terse_yes_raw.txt b/test/gold/tools/getBlocks/api_tests/getBlocks_terse_yes_raw.txt index df430d46a8..48b681f108 100644 --- a/test/gold/tools/getBlocks/api_tests/getBlocks_terse_yes_raw.txt +++ b/test/gold/tools/getBlocks/api_tests/getBlocks_terse_yes_raw.txt @@ -117,7 +117,8 @@ blocks?blocks=4001001&hashes&raw "0xa600632d3a6b343ecd51795291818dbd8c59e4588c08c0cfc5a9f2614edfe567" ], "transactionsRoot": "0x6494993a4b201058be9dc1cf005a7b7795b4c7452b0b66af7e82a97549cbed49", - "uncles": [] + "uncles": [], + "withdrawals": null } ], "meta": { diff --git a/test/gold/tools/getBlocks/getBlocks_decache.txt b/test/gold/tools/getBlocks/getBlocks_decache.txt index 964fb70165..6c633e0a74 100644 --- a/test/gold/tools/getBlocks/getBlocks_decache.txt +++ b/test/gold/tools/getBlocks/getBlocks_decache.txt @@ -6,7 +6,7 @@ TEST[DATE|TIME] Format: json { "data": [ { - "msg": "5 items totaling 1897 bytes were removed from the blocks cache." + "msg": "5 items totaling 1905 bytes were removed from the blocks cache." } ] } diff --git a/test/gold/tools/getBlocks/getBlocks_decache_one.txt b/test/gold/tools/getBlocks/getBlocks_decache_one.txt index 9ae67f22a9..18eef012a3 100644 --- a/test/gold/tools/getBlocks/getBlocks_decache_one.txt +++ b/test/gold/tools/getBlocks/getBlocks_decache_one.txt @@ -6,7 +6,7 @@ TEST[DATE|TIME] Format: json { "data": [ { - "msg": "1 items totaling 161 bytes were removed from the blocks cache." + "msg": "1 items totaling 169 bytes were removed from the blocks cache." } ] } diff --git a/test/gold/tools/getBlocks/getBlocks_decache_traces.txt b/test/gold/tools/getBlocks/getBlocks_decache_traces.txt index 9ae67f22a9..18eef012a3 100644 --- a/test/gold/tools/getBlocks/getBlocks_decache_traces.txt +++ b/test/gold/tools/getBlocks/getBlocks_decache_traces.txt @@ -6,7 +6,7 @@ TEST[DATE|TIME] Format: json { "data": [ { - "msg": "1 items totaling 161 bytes were removed from the blocks cache." + "msg": "1 items totaling 169 bytes were removed from the blocks cache." } ] } diff --git a/test/gold/tools/getBlocks/getBlocks_decache_txs.txt b/test/gold/tools/getBlocks/getBlocks_decache_txs.txt index 9ae67f22a9..18eef012a3 100644 --- a/test/gold/tools/getBlocks/getBlocks_decache_txs.txt +++ b/test/gold/tools/getBlocks/getBlocks_decache_txs.txt @@ -6,7 +6,7 @@ TEST[DATE|TIME] Format: json { "data": [ { - "msg": "1 items totaling 161 bytes were removed from the blocks cache." + "msg": "1 items totaling 169 bytes were removed from the blocks cache." } ] } diff --git a/test/gold/tools/getBlocks/getBlocks_raw_01.txt b/test/gold/tools/getBlocks/getBlocks_raw_01.txt index cbfbf96c11..c947d2a3e6 100644 --- a/test/gold/tools/getBlocks/getBlocks_raw_01.txt +++ b/test/gold/tools/getBlocks/getBlocks_raw_01.txt @@ -62,7 +62,8 @@ TEST[DATE|TIME] Format: json } ], "transactionsRoot": "0xb82a3089b32828fdf00c1dc690253c4c171cf5e31cde2a19818a219d991841fd", - "uncles": [] + "uncles": [], + "withdrawals": null }, { "author": "", @@ -138,7 +139,8 @@ TEST[DATE|TIME] Format: json } ], "transactionsRoot": "0x976cfa98e970746ea6c254552dd31cb2a3f7cc38b0e08a744c91aaf1fbbbd84e", - "uncles": [] + "uncles": [], + "withdrawals": null }, { "author": "", @@ -199,7 +201,8 @@ TEST[DATE|TIME] Format: json } ], "transactionsRoot": "0x431798971e1b33824842c82e8ae24c3078863f5fd47e373a068f22cab63bab5f", - "uncles": [] + "uncles": [], + "withdrawals": null } ] } \ No newline at end of file diff --git a/test/gold/tools/getBlocks/getBlocks_terse_yes_raw.txt b/test/gold/tools/getBlocks/getBlocks_terse_yes_raw.txt index a693a6e15a..d96868b247 100644 --- a/test/gold/tools/getBlocks/getBlocks_terse_yes_raw.txt +++ b/test/gold/tools/getBlocks/getBlocks_terse_yes_raw.txt @@ -122,7 +122,8 @@ TEST[DATE|TIME] Format: json "0xa600632d3a6b343ecd51795291818dbd8c59e4588c08c0cfc5a9f2614edfe567" ], "transactionsRoot": "0x6494993a4b201058be9dc1cf005a7b7795b4c7452b0b66af7e82a97549cbed49", - "uncles": [] + "uncles": [], + "withdrawals": null } ] } \ No newline at end of file diff --git a/test/gold/tools/getTokens/api_tests/getTokens_proxy_yes.txt b/test/gold/tools/getTokens/api_tests/getTokens_proxy_yes.txt index 065edc661d..e9d85968d0 100644 --- a/test/gold/tools/getTokens/api_tests/getTokens_proxy_yes.txt +++ b/test/gold/tools/getTokens/api_tests/getTokens_proxy_yes.txt @@ -1,12 +1,14 @@ -tokens?addrs=0x00000000441378008ea67f4284a57932b1c000a5&fmt=json&blocks=8000000&parts=all +tokens?addrs=0x00000000441378008ea67f4284a57932b1c000a5&fmt=json&blocks=8000000&parts=all&verbose { "data": [ { "address": "0x00000000441378008ea67f4284a57932b1c000a5", "blockNumber": 8000000, + "date": "2019-06-21 06:55:49 UTC", "decimals": 18, "name": "TrueGBP", "symbol": "TGBP", + "timestamp": 1561100149, "totalSupply": "1080020.89" } ], diff --git a/test/gold/tools/getTokens/getTokens_proxy_yes.txt b/test/gold/tools/getTokens/getTokens_proxy_yes.txt index a993c9c760..ca41dc2f0f 100644 --- a/test/gold/tools/getTokens/getTokens_proxy_yes.txt +++ b/test/gold/tools/getTokens/getTokens_proxy_yes.txt @@ -1,7 +1,8 @@ -chifra tokens 0x00000000441378008ea67f4284a57932b1c000a5 --fmt json 8000000 --parts all +chifra tokens 0x00000000441378008ea67f4284a57932b1c000a5 --fmt json 8000000 --parts all --verbose TEST[DATE|TIME] Addrs: [0x00000000441378008ea67f4284a57932b1c000a5] TEST[DATE|TIME] Blocks: [8000000] TEST[DATE|TIME] Parts: [all] +TEST[DATE|TIME] Verbose: true TEST[DATE|TIME] Caps: cache,decache TEST[DATE|TIME] Format: json { @@ -9,9 +10,11 @@ TEST[DATE|TIME] Format: json { "address": "0x00000000441378008ea67f4284a57932b1c000a5", "blockNumber": 8000000, + "date": "2019-06-21 06:55:49 UTC", "decimals": 18, "name": "TrueGBP", "symbol": "TGBP", + "timestamp": 1561100149, "totalSupply": "1080020.89" } ] diff --git a/test/gold/tools/whenBlock/api_tests/whenBlock_decache_one.txt b/test/gold/tools/whenBlock/api_tests/whenBlock_decache_one.txt index 9b34d4744a..45e60e6b21 100644 --- a/test/gold/tools/whenBlock/api_tests/whenBlock_decache_one.txt +++ b/test/gold/tools/whenBlock/api_tests/whenBlock_decache_one.txt @@ -2,7 +2,7 @@ when?blocks=12&decache { "data": [ { - "msg": "1 items totaling 161 bytes were removed from the blocks cache." + "msg": "1 items totaling 169 bytes were removed from the blocks cache." } ], "meta": { diff --git a/test/gold/tools/whenBlock/whenBlock_combined_options_1.txt b/test/gold/tools/whenBlock/whenBlock_combined_options_1.txt index 8602da8984..f6ae075c34 100644 --- a/test/gold/tools/whenBlock/whenBlock_combined_options_1.txt +++ b/test/gold/tools/whenBlock/whenBlock_combined_options_1.txt @@ -3,5 +3,5 @@ TEST[DATE|TIME] Blocks: [10000] TEST[DATE|TIME] Verbose: true TEST[DATE|TIME] Caps: cache,decache TEST[DATE|TIME] Format: txt -component blockNumber timestamp date name description - 10000 1438334627 2015-07-31 09:23:47 UTC +blockNumber timestamp date name component description +10000 1438334627 2015-07-31 09:23:47 UTC diff --git a/test/gold/tools/whenBlock/whenBlock_decache_one.txt b/test/gold/tools/whenBlock/whenBlock_decache_one.txt index cbc8fba0b6..dc8be72a32 100644 --- a/test/gold/tools/whenBlock/whenBlock_decache_one.txt +++ b/test/gold/tools/whenBlock/whenBlock_decache_one.txt @@ -3,4 +3,4 @@ TEST[DATE|TIME] Blocks: [12] TEST[DATE|TIME] Decache: true TEST[DATE|TIME] Caps: cache,decache TEST[DATE|TIME] Format: txt -1 items totaling 161 bytes were removed from the blocks cache. +1 items totaling 169 bytes were removed from the blocks cache. diff --git a/test/gold/tools/whenBlock/whenBlock_long_verbose_valid_block.txt b/test/gold/tools/whenBlock/whenBlock_long_verbose_valid_block.txt index f38d726f8c..3550be0e57 100644 --- a/test/gold/tools/whenBlock/whenBlock_long_verbose_valid_block.txt +++ b/test/gold/tools/whenBlock/whenBlock_long_verbose_valid_block.txt @@ -3,5 +3,5 @@ TEST[DATE|TIME] Blocks: [1000] TEST[DATE|TIME] Verbose: true TEST[DATE|TIME] Caps: cache,decache TEST[DATE|TIME] Format: txt -component blockNumber timestamp date name description - 1000 1438272138 2015-07-30 16:02:18 UTC +blockNumber timestamp date name component description +1000 1438272138 2015-07-30 16:02:18 UTC diff --git a/test/gold/tools/whenBlock/whenBlock_mixed_options.txt b/test/gold/tools/whenBlock/whenBlock_mixed_options.txt index 04157f6197..0aabe4412e 100644 --- a/test/gold/tools/whenBlock/whenBlock_mixed_options.txt +++ b/test/gold/tools/whenBlock/whenBlock_mixed_options.txt @@ -3,43 +3,43 @@ TEST[DATE|TIME] List: true TEST[DATE|TIME] Verbose: true TEST[DATE|TIME] Caps: cache,decache TEST[DATE|TIME] Format: txt -component blockNumber timestamp date name description -beacon 1 1606824028 2020-12-01 12:00:28 UTC beacon_genesis The Beacon Chain needed 16384 deposits of 32 staked ETH to ship securely. This happened on November 27, meaning the Beacon Chain started producing blocks on December 1, 2020. This is an important first step in achieving the Ethereum vision. -beacon 74240 1635332183 2021-10-27 10:56:23 UTC altair The Altair upgrade was the first scheduled upgrade for the Beacon Chain. It added support for sync committees—enabling light clients, and increased validator inactivity and slashing penalties as development progressed towards The Merge. -beacon 144896 1662464087 2022-09-06 11:34:47 UTC bellatrix The Bellatrix upgrade was the second scheduled upgrade for the Beacon Chain, preparing the chain for The Merge. It brings validator penalties to their full values for inactivity and slashable offenses. Bellatrix also includes an update to the fork choice rules to prepare the chain for The Merge and the transition from the last proof-of-work block to the first proof-of-stake block. This includes making consensus clients aware of the terminal total difficulty of 58750000000000000000000. -beacon 194048 1681338455 2023-04-12 22:27:35 UTC capella The Capella upgrade was the third major upgrade to the consensus layer (Beacon Chain) and enabled staking withdrawals. Capella occurred synchronously with the execution layer upgrade, Shanghai, and enabled staking withdrawal functionality. This consensus layer upgrade brought the ability for stakers who did not provide withdrawal credentials with their initial deposit to do so, thereby enabling withdrawals. The upgrade also provided automatic account sweeping functionality, which continuously processes validator accounts for any available rewards payments or full withdrawals. -execution 0 1438269975 2015-07-30 15:26:15 UTC frontier Frontier was a live, but barebone implementation of the Ethereum project. It followed the successful Olympic testing phase. It was intended for technical users, specifically developers. Blocks had a gas limit of 5,000. This ‘thawing’ period enabled miners to start their operations and for early adopters to install their clients without having to ‘rush’. -execution 46147 1438918233 2015-08-07 03:30:33 UTC firstTrans The block containing the first transaction on the Ethereum Mainnet blockchain. -execution 50111 1438984075 2015-08-07 21:47:55 UTC firstContract The block containing the first contract on the Ethereum Mainnet blockchain. -execution 200000 1441661589 2015-09-07 21:33:09 UTC iceage The frontier thawing fork lifted the 5,000 gas limit per block and set the default gas price to 51 gwei. This allowed for transactions – transactions require 21,000 gas. The difficulty bomb was introduced to ensure a future hard-fork to proof-of-stake. -execution 543626 1447577992 2015-11-15 08:59:52 UTC devcon1 The first Ethereum developer conference. -execution 1150000 1457981393 2016-03-14 18:49:53 UTC homestead The Homestead fork that looked to the future. It included several protocol changes and a networking change that gave Ethereum the ability to do further network upgrades. -execution 1428756 1461980573 2016-04-30 01:42:53 UTC daofund The first block people were able to fund the infamous DAO. -execution 1718497 1466134488 2016-06-17 03:34:48 UTC daohack The first blocks of the DAO hack. -execution 1920000 1469020840 2016-07-20 13:20:40 UTC daofork The DAO fork was in response to the 2016 DAO attack where an insecure DAO contract was drained of over 3.6 million ETH in a hack. The fork moved the funds from the faulty contract to a new contract with a single function: withdraw. Anyone who lost funds could withdraw 1 ETH for every 100 DAO tokens in their wallets. This course of action was voted on by the Ethereum community. Any ETH holder was able to vote via a transaction on a voting platform. The decision to fork reached over 85% of the votes. Some miners refused to fork because the DAO incident wasn't a defect in the protocol. They went on to form Ethereum Classic. -execution 2286910 1474275599 2016-09-19 08:59:59 UTC devcon2 The second Ethereum developer conference. -execution 2463000 1476796771 2016-10-18 13:19:31 UTC tangerine_whistle The Tangerine Whistle fork was the first response to the denial of service (DoS) attacks on the network (September/October 2016) including: addressing urgent network health issues concerning underpriced operation codes. -execution 2675000 1479831344 2016-11-22 16:15:44 UTC spurious_dragon The Spurious Dragon fork was the second response to the denial of service (DoS) attacks on the network (September/October 2016) including: tuning opcode pricing to prevent future attacks on the network. enabling “debloat” of the blockchain state. adding replay attack protection. -execution 2717576 1480448309 2016-11-29 19:38:29 UTC stateclear The block of the state clearing soft fork. -execution 3265360 1488272400 2017-02-28 09:00:00 UTC eea The establishment of the Ethereum Enterprise Association. -execution 3327417 1489165544 2017-03-10 17:05:44 UTC ens2 The second ENS deployment of ENS. -execution 4041179 1500417203 2017-07-18 22:33:23 UTC parityhack1 The block of the first Parity hack. -execution 4370000 1508131331 2017-10-16 05:22:11 UTC byzantium The Byzantium fork: Reduced block mining rewards from 5 to 3 ETH. Delayed the difficulty bomb by a year. Added ability to make non-state-changing calls to other contracts. Added certain cryptography methods to allow for layer 2 scaling. -execution 4469339 1509526798 2017-11-01 08:59:58 UTC devcon3 The third Ethereum developer conference. -execution 4501969 1509981921 2017-11-06 15:25:21 UTC parityhack2 The block of the second Parity hack. -execution 4605167 1511415679 2017-11-23 05:41:19 UTC kitties The block of the CryptoKitties contract. -execution 4620855 1511634257 2017-11-25 18:24:17 UTC makerdao The block when MakerDAO was deployed. -execution 6610517 1540890000 2018-10-30 09:00:00 UTC devcon4 The fourth Ethereum developer conference. -execution 6627917 1541137042 2018-11-02 05:37:22 UTC uniswap The block when Uniswap was deployed. -execution 7280000 1551383524 2019-02-28 19:52:04 UTC constantinople The Constantinople fork: Ensured the blockchain didn't freeze before proof-of-stake was implemented. Optimised the gas cost of certain actions in the EVM. Added the ability to interact with addresses that haven't been created yet. -execution 8700401 1570525179 2019-10-08 08:59:39 UTC devcon5 The fifth Ethereum developer conference. -execution 8928158 1573672677 2019-11-13 19:17:57 UTC mcdai The block when Multi-Collateral Dai was deployed. -execution 9069000 1575764709 2019-12-08 00:25:09 UTC istanbul The Istanbul fork: Optimised the gas cost of certain actions in the EVM. Improved denial-of-service attack resilience. Made Layer 2 scaling solutions based on SNARKs and STARKs more performant. Enabled Ethereum and Zcash to interoperate. Allowed contracts to introduce more creative functions. -execution 9200000 1577953849 2020-01-02 08:30:49 UTC muir_glacier The Muir Glacier fork introduced a delay to the difficulty bomb. Increases in block difficulty of the proof-of-work consensus mechanism threatened to degrade the usability of Ethereum by increasing wait times for sending transactions and using dapps. -execution 11052984 1602667372 2020-10-14 09:22:52 UTC deposit_contract The staking deposit contract introduced staking to the Ethereum ecosystem. Although a Mainnet contract, it had a direct impact on the timeline for launching the Beacon Chain, an important Ethereum upgrade. -execution 12244000 1618481223 2021-04-15 10:07:03 UTC berlin The Berlin upgrade optimized gas cost for certain EVM actions, and increases support for multiple transaction types. -execution 12965000 1628166822 2021-08-05 12:33:42 UTC london The London upgrade introduced EIP-1559 , which reformed the transaction fee market, along with changes to how gas refunds are handled and the Ice Age schedule. -execution 13773000 1639079723 2021-12-09 19:55:23 UTC arrow_glacier The Arrow Glacier network upgrade pushed back the difficulty bomb by several months. This is the only change introduced in this upgrade, and is similar in nature to the Muir Glacier upgrade. Similar changes have been performed on the Byzantium, Constantinople and London network upgrades. -execution 15050000 1656586444 2022-06-30 10:54:04 UTC gray_glacier The Gray Glacier network upgrade pushed back the difficulty bomb by three months. This is the only change introduced in this upgrade, and is similar in nature to the Arrow Glacier and Muir Glacier upgrades. Similar changes have been performed on the Byzantium, Constantinople and London network upgrades. -execution 15537393 1663224162 2022-09-15 06:42:42 UTC paris_merge The Paris upgrade was triggered by the proof-of-work blockchain passing a terminal total difficulty of 58750000000000000000000. This happened at block 15537393 on 15th September 2022, triggering the Paris upgrade the next block. Paris was The Merge transition - its major feature was switching off the proof-of-work mining algorithm and associated consensus logic and switching on proof-of-stake instead. Paris itself was an upgrade to the execution clients (equivalent to Bellatrix on the consensus layer) that enabled them to take instruction from their connected consensus clients. This required a new set of internal API methods, collectively known as the Engine API, to be activated. This was arguably the most significant upgrade in Ethereum history since Homestead! -execution 17034870 1681338479 2023-04-12 22:27:59 UTC shanghai The Shanghai upgrade brought staking withdrawals to the execution layer. In tandem with the Capella upgrade, this enabled blocks to accept withdrawal operations, which allows stakers to withdraw their ETH from the Beacon Chain to the execution layer. +blockNumber timestamp date name component description +1 1606824028 2020-12-01 12:00:28 UTC beacon_genesis beacon The Beacon Chain needed 16384 deposits of 32 staked ETH to ship securely. This happened on November 27, meaning the Beacon Chain started producing blocks on December 1, 2020. This is an important first step in achieving the Ethereum vision. +74240 1635332183 2021-10-27 10:56:23 UTC altair beacon The Altair upgrade was the first scheduled upgrade for the Beacon Chain. It added support for sync committees—enabling light clients, and increased validator inactivity and slashing penalties as development progressed towards The Merge. +144896 1662464087 2022-09-06 11:34:47 UTC bellatrix beacon The Bellatrix upgrade was the second scheduled upgrade for the Beacon Chain, preparing the chain for The Merge. It brings validator penalties to their full values for inactivity and slashable offenses. Bellatrix also includes an update to the fork choice rules to prepare the chain for The Merge and the transition from the last proof-of-work block to the first proof-of-stake block. This includes making consensus clients aware of the terminal total difficulty of 58750000000000000000000. +194048 1681338455 2023-04-12 22:27:35 UTC capella beacon The Capella upgrade was the third major upgrade to the consensus layer (Beacon Chain) and enabled staking withdrawals. Capella occurred synchronously with the execution layer upgrade, Shanghai, and enabled staking withdrawal functionality. This consensus layer upgrade brought the ability for stakers who did not provide withdrawal credentials with their initial deposit to do so, thereby enabling withdrawals. The upgrade also provided automatic account sweeping functionality, which continuously processes validator accounts for any available rewards payments or full withdrawals. +0 1438269975 2015-07-30 15:26:15 UTC frontier execution Frontier was a live, but barebone implementation of the Ethereum project. It followed the successful Olympic testing phase. It was intended for technical users, specifically developers. Blocks had a gas limit of 5,000. This ‘thawing’ period enabled miners to start their operations and for early adopters to install their clients without having to ‘rush’. +46147 1438918233 2015-08-07 03:30:33 UTC firstTrans execution The block containing the first transaction on the Ethereum Mainnet blockchain. +50111 1438984075 2015-08-07 21:47:55 UTC firstContract execution The block containing the first contract on the Ethereum Mainnet blockchain. +200000 1441661589 2015-09-07 21:33:09 UTC iceage execution The frontier thawing fork lifted the 5,000 gas limit per block and set the default gas price to 51 gwei. This allowed for transactions – transactions require 21,000 gas. The difficulty bomb was introduced to ensure a future hard-fork to proof-of-stake. +543626 1447577992 2015-11-15 08:59:52 UTC devcon1 execution The first Ethereum developer conference. +1150000 1457981393 2016-03-14 18:49:53 UTC homestead execution The Homestead fork that looked to the future. It included several protocol changes and a networking change that gave Ethereum the ability to do further network upgrades. +1428756 1461980573 2016-04-30 01:42:53 UTC daofund execution The first block people were able to fund the infamous DAO. +1718497 1466134488 2016-06-17 03:34:48 UTC daohack execution The first blocks of the DAO hack. +1920000 1469020840 2016-07-20 13:20:40 UTC daofork execution The DAO fork was in response to the 2016 DAO attack where an insecure DAO contract was drained of over 3.6 million ETH in a hack. The fork moved the funds from the faulty contract to a new contract with a single function: withdraw. Anyone who lost funds could withdraw 1 ETH for every 100 DAO tokens in their wallets. This course of action was voted on by the Ethereum community. Any ETH holder was able to vote via a transaction on a voting platform. The decision to fork reached over 85% of the votes. Some miners refused to fork because the DAO incident wasn't a defect in the protocol. They went on to form Ethereum Classic. +2286910 1474275599 2016-09-19 08:59:59 UTC devcon2 execution The second Ethereum developer conference. +2463000 1476796771 2016-10-18 13:19:31 UTC tangerine_whistle execution The Tangerine Whistle fork was the first response to the denial of service (DoS) attacks on the network (September/October 2016) including: addressing urgent network health issues concerning underpriced operation codes. +2675000 1479831344 2016-11-22 16:15:44 UTC spurious_dragon execution The Spurious Dragon fork was the second response to the denial of service (DoS) attacks on the network (September/October 2016) including: tuning opcode pricing to prevent future attacks on the network. enabling “debloat” of the blockchain state. adding replay attack protection. +2717576 1480448309 2016-11-29 19:38:29 UTC stateclear execution The block of the state clearing soft fork. +3265360 1488272400 2017-02-28 09:00:00 UTC eea execution The establishment of the Ethereum Enterprise Association. +3327417 1489165544 2017-03-10 17:05:44 UTC ens2 execution The second ENS deployment of ENS. +4041179 1500417203 2017-07-18 22:33:23 UTC parityhack1 execution The block of the first Parity hack. +4370000 1508131331 2017-10-16 05:22:11 UTC byzantium execution The Byzantium fork: Reduced block mining rewards from 5 to 3 ETH. Delayed the difficulty bomb by a year. Added ability to make non-state-changing calls to other contracts. Added certain cryptography methods to allow for layer 2 scaling. +4469339 1509526798 2017-11-01 08:59:58 UTC devcon3 execution The third Ethereum developer conference. +4501969 1509981921 2017-11-06 15:25:21 UTC parityhack2 execution The block of the second Parity hack. +4605167 1511415679 2017-11-23 05:41:19 UTC kitties execution The block of the CryptoKitties contract. +4620855 1511634257 2017-11-25 18:24:17 UTC makerdao execution The block when MakerDAO was deployed. +6610517 1540890000 2018-10-30 09:00:00 UTC devcon4 execution The fourth Ethereum developer conference. +6627917 1541137042 2018-11-02 05:37:22 UTC uniswap execution The block when Uniswap was deployed. +7280000 1551383524 2019-02-28 19:52:04 UTC constantinople execution The Constantinople fork: Ensured the blockchain didn't freeze before proof-of-stake was implemented. Optimised the gas cost of certain actions in the EVM. Added the ability to interact with addresses that haven't been created yet. +8700401 1570525179 2019-10-08 08:59:39 UTC devcon5 execution The fifth Ethereum developer conference. +8928158 1573672677 2019-11-13 19:17:57 UTC mcdai execution The block when Multi-Collateral Dai was deployed. +9069000 1575764709 2019-12-08 00:25:09 UTC istanbul execution The Istanbul fork: Optimised the gas cost of certain actions in the EVM. Improved denial-of-service attack resilience. Made Layer 2 scaling solutions based on SNARKs and STARKs more performant. Enabled Ethereum and Zcash to interoperate. Allowed contracts to introduce more creative functions. +9200000 1577953849 2020-01-02 08:30:49 UTC muir_glacier execution The Muir Glacier fork introduced a delay to the difficulty bomb. Increases in block difficulty of the proof-of-work consensus mechanism threatened to degrade the usability of Ethereum by increasing wait times for sending transactions and using dapps. +11052984 1602667372 2020-10-14 09:22:52 UTC deposit_contract execution The staking deposit contract introduced staking to the Ethereum ecosystem. Although a Mainnet contract, it had a direct impact on the timeline for launching the Beacon Chain, an important Ethereum upgrade. +12244000 1618481223 2021-04-15 10:07:03 UTC berlin execution The Berlin upgrade optimized gas cost for certain EVM actions, and increases support for multiple transaction types. +12965000 1628166822 2021-08-05 12:33:42 UTC london execution The London upgrade introduced EIP-1559 , which reformed the transaction fee market, along with changes to how gas refunds are handled and the Ice Age schedule. +13773000 1639079723 2021-12-09 19:55:23 UTC arrow_glacier execution The Arrow Glacier network upgrade pushed back the difficulty bomb by several months. This is the only change introduced in this upgrade, and is similar in nature to the Muir Glacier upgrade. Similar changes have been performed on the Byzantium, Constantinople and London network upgrades. +15050000 1656586444 2022-06-30 10:54:04 UTC gray_glacier execution The Gray Glacier network upgrade pushed back the difficulty bomb by three months. This is the only change introduced in this upgrade, and is similar in nature to the Arrow Glacier and Muir Glacier upgrades. Similar changes have been performed on the Byzantium, Constantinople and London network upgrades. +15537393 1663224162 2022-09-15 06:42:42 UTC paris_merge execution The Paris upgrade was triggered by the proof-of-work blockchain passing a terminal total difficulty of 58750000000000000000000. This happened at block 15537393 on 15th September 2022, triggering the Paris upgrade the next block. Paris was The Merge transition - its major feature was switching off the proof-of-work mining algorithm and associated consensus logic and switching on proof-of-stake instead. Paris itself was an upgrade to the execution clients (equivalent to Bellatrix on the consensus layer) that enabled them to take instruction from their connected consensus clients. This required a new set of internal API methods, collectively known as the Engine API, to be activated. This was arguably the most significant upgrade in Ethereum history since Homestead! +17034870 1681338479 2023-04-12 22:27:59 UTC shanghai execution The Shanghai upgrade brought staking withdrawals to the execution layer. In tandem with the Capella upgrade, this enabled blocks to accept withdrawal operations, which allows stakers to withdraw their ETH from the Beacon Chain to the execution layer. diff --git a/test/gold/tools/whenBlock/whenBlock_special_block_daofork.txt b/test/gold/tools/whenBlock/whenBlock_special_block_daofork.txt index 5c36e968a4..00d239eac1 100644 --- a/test/gold/tools/whenBlock/whenBlock_special_block_daofork.txt +++ b/test/gold/tools/whenBlock/whenBlock_special_block_daofork.txt @@ -3,5 +3,5 @@ TEST[DATE|TIME] Blocks: [daofork] TEST[DATE|TIME] Verbose: true TEST[DATE|TIME] Caps: cache,decache TEST[DATE|TIME] Format: txt -component blockNumber timestamp date name description -execution 1920000 1469020840 2016-07-20 13:20:40 UTC daofork The DAO fork was in response to the 2016 DAO attack where an insecure DAO contract was drained of over 3.6 million ETH in a hack. The fork moved the funds from the faulty contract to a new contract with a single function: withdraw. Anyone who lost funds could withdraw 1 ETH for every 100 DAO tokens in their wallets. This course of action was voted on by the Ethereum community. Any ETH holder was able to vote via a transaction on a voting platform. The decision to fork reached over 85% of the votes. Some miners refused to fork because the DAO incident wasn't a defect in the protocol. They went on to form Ethereum Classic. +blockNumber timestamp date name component description +1920000 1469020840 2016-07-20 13:20:40 UTC daofork execution The DAO fork was in response to the 2016 DAO attack where an insecure DAO contract was drained of over 3.6 million ETH in a hack. The fork moved the funds from the faulty contract to a new contract with a single function: withdraw. Anyone who lost funds could withdraw 1 ETH for every 100 DAO tokens in their wallets. This course of action was voted on by the Ethereum community. Any ETH holder was able to vote via a transaction on a voting platform. The decision to fork reached over 85% of the votes. Some miners refused to fork because the DAO incident wasn't a defect in the protocol. They went on to form Ethereum Classic. diff --git a/test/gold/tools/whenBlock/whenBlock_special_block_daofund.txt b/test/gold/tools/whenBlock/whenBlock_special_block_daofund.txt index 10eebd26d6..abd54c9133 100644 --- a/test/gold/tools/whenBlock/whenBlock_special_block_daofund.txt +++ b/test/gold/tools/whenBlock/whenBlock_special_block_daofund.txt @@ -3,5 +3,5 @@ TEST[DATE|TIME] Blocks: [daofund] TEST[DATE|TIME] Verbose: true TEST[DATE|TIME] Caps: cache,decache TEST[DATE|TIME] Format: txt -component blockNumber timestamp date name description -execution 1428756 1461980573 2016-04-30 01:42:53 UTC daofund The first block people were able to fund the infamous DAO. +blockNumber timestamp date name component description +1428756 1461980573 2016-04-30 01:42:53 UTC daofund execution The first block people were able to fund the infamous DAO. diff --git a/test/gold/tools/whenBlock/whenBlock_special_block_daohack.txt b/test/gold/tools/whenBlock/whenBlock_special_block_daohack.txt index 8f0a4cbaba..e33d612931 100644 --- a/test/gold/tools/whenBlock/whenBlock_special_block_daohack.txt +++ b/test/gold/tools/whenBlock/whenBlock_special_block_daohack.txt @@ -3,5 +3,5 @@ TEST[DATE|TIME] Blocks: [daohack] TEST[DATE|TIME] Verbose: true TEST[DATE|TIME] Caps: cache,decache TEST[DATE|TIME] Format: txt -component blockNumber timestamp date name description -execution 1718497 1466134488 2016-06-17 03:34:48 UTC daohack The first blocks of the DAO hack. +blockNumber timestamp date name component description +1718497 1466134488 2016-06-17 03:34:48 UTC daohack execution The first blocks of the DAO hack. diff --git a/test/gold/tools/whenBlock/whenBlock_special_block_first.txt b/test/gold/tools/whenBlock/whenBlock_special_block_first.txt index 3da9e89fef..2dc5f56d8f 100644 --- a/test/gold/tools/whenBlock/whenBlock_special_block_first.txt +++ b/test/gold/tools/whenBlock/whenBlock_special_block_first.txt @@ -3,5 +3,5 @@ TEST[DATE|TIME] Blocks: [frontier] TEST[DATE|TIME] Verbose: true TEST[DATE|TIME] Caps: cache,decache TEST[DATE|TIME] Format: txt -component blockNumber timestamp date name description -execution 0 1438269975 2015-07-30 15:26:15 UTC frontier Frontier was a live, but barebone implementation of the Ethereum project. It followed the successful Olympic testing phase. It was intended for technical users, specifically developers. Blocks had a gas limit of 5,000. This ‘thawing’ period enabled miners to start their operations and for early adopters to install their clients without having to ‘rush’. +blockNumber timestamp date name component description +0 1438269975 2015-07-30 15:26:15 UTC frontier execution Frontier was a live, but barebone implementation of the Ethereum project. It followed the successful Olympic testing phase. It was intended for technical users, specifically developers. Blocks had a gas limit of 5,000. This ‘thawing’ period enabled miners to start their operations and for early adopters to install their clients without having to ‘rush’. diff --git a/test/gold/tools/whenBlock/whenBlock_special_block_multi1.txt b/test/gold/tools/whenBlock/whenBlock_special_block_multi1.txt index 91f60731c7..1986feb504 100644 --- a/test/gold/tools/whenBlock/whenBlock_special_block_multi1.txt +++ b/test/gold/tools/whenBlock/whenBlock_special_block_multi1.txt @@ -3,7 +3,7 @@ TEST[DATE|TIME] Blocks: [stateclear iceage 4001001] TEST[DATE|TIME] Verbose: true TEST[DATE|TIME] Caps: cache,decache TEST[DATE|TIME] Format: txt -component blockNumber timestamp date name description -execution 2717576 1480448309 2016-11-29 19:38:29 UTC stateclear The block of the state clearing soft fork. -execution 200000 1441661589 2015-09-07 21:33:09 UTC iceage The frontier thawing fork lifted the 5,000 gas limit per block and set the default gas price to 51 gwei. This allowed for transactions – transactions require 21,000 gas. The difficulty bomb was introduced to ensure a future hard-fork to proof-of-stake. - 4001001 1499651592 2017-07-10 01:53:12 UTC +blockNumber timestamp date name component description +2717576 1480448309 2016-11-29 19:38:29 UTC stateclear execution The block of the state clearing soft fork. +200000 1441661589 2015-09-07 21:33:09 UTC iceage execution The frontier thawing fork lifted the 5,000 gas limit per block and set the default gas price to 51 gwei. This allowed for transactions – transactions require 21,000 gas. The difficulty bomb was introduced to ensure a future hard-fork to proof-of-stake. +4001001 1499651592 2017-07-10 01:53:12 UTC diff --git a/test/gold/tools/whenBlock/whenBlock_special_block_stateclear.txt b/test/gold/tools/whenBlock/whenBlock_special_block_stateclear.txt index 5c1b380e95..ddcbca4c35 100644 --- a/test/gold/tools/whenBlock/whenBlock_special_block_stateclear.txt +++ b/test/gold/tools/whenBlock/whenBlock_special_block_stateclear.txt @@ -3,5 +3,5 @@ TEST[DATE|TIME] Blocks: [stateclear] TEST[DATE|TIME] Verbose: true TEST[DATE|TIME] Caps: cache,decache TEST[DATE|TIME] Format: txt -component blockNumber timestamp date name description -execution 2717576 1480448309 2016-11-29 19:38:29 UTC stateclear The block of the state clearing soft fork. +blockNumber timestamp date name component description +2717576 1480448309 2016-11-29 19:38:29 UTC stateclear execution The block of the state clearing soft fork. diff --git a/test/gold/tools/whenBlock/whenBlock_special_block_tangerine.txt b/test/gold/tools/whenBlock/whenBlock_special_block_tangerine.txt index 2665254ef4..f6e7908c69 100644 --- a/test/gold/tools/whenBlock/whenBlock_special_block_tangerine.txt +++ b/test/gold/tools/whenBlock/whenBlock_special_block_tangerine.txt @@ -3,5 +3,5 @@ TEST[DATE|TIME] Blocks: [tangerine_whistle] TEST[DATE|TIME] Verbose: true TEST[DATE|TIME] Caps: cache,decache TEST[DATE|TIME] Format: txt -component blockNumber timestamp date name description -execution 2463000 1476796771 2016-10-18 13:19:31 UTC tangerine_whistle The Tangerine Whistle fork was the first response to the denial of service (DoS) attacks on the network (September/October 2016) including: addressing urgent network health issues concerning underpriced operation codes. +blockNumber timestamp date name component description +2463000 1476796771 2016-10-18 13:19:31 UTC tangerine_whistle execution The Tangerine Whistle fork was the first response to the denial of service (DoS) attacks on the network (September/October 2016) including: addressing urgent network health issues concerning underpriced operation codes.