v2.16.0

Added

• Error handling when arbitrum sequencer is not accessible
• Error handle for gnosis chiado for seen tx
• Gas limit estimation feature to EVM gas estimators. Introduced a new config EVM.GasEstimator.EstimateLimit to toggle this feature
• Report new heads as a telemetry to OTI
• Optimism Stack L1 Oracle to add support for Mantle
• Allow workflows to run without external registry configured
• Astar TerminallyUnderpriced error mapping
• Custom client error messages for Mantle to capture InsufficientEth and Fatal errors
• New health check that ensures RPC provides new finalized heads at least every NoNewFinalizedHeadsThreshold

Updated

• Sync feeds-manager wsrpc proto
• Add DB syncing for registry syncer
• Polygon configs to match PIP-35
• Simplify how token and gas prices are stored in the database - user upsert instead of insert/delete flow

v2.16.0 Changelog

v2.15.0

Added

• Support for EIP-1559 transactions for Scroll

• Protocol-level support for preventing bid/ask variant violations in mercury

• Add “VerboseLogging” option to mercury. Off by default, can be enabled like so:

  [Mercury]
  VerboseLogging = true
  

Updated

• Dequeue minimum guaranteed upkeeps as a priority
• Rename the InBackupHealthReport to StartUpHealthReport and enable it for DB migrations as well. This will enable health report to be available during long start-up tasks (db backups and migrations).
• VRFV2Plus coordinator and wrapper split contracts between L1 and L2 chains
• Fix: Set LatestFinalizedBlock for finalized blocks saved by logpoller

v2.15.0 Changelog

v2.14.0

Added

• Added Aptos Keystore to Core. This includes Aptos Key which uses ED25519, Keystore, Relevant tests
• zkSync L1 GasPrice calculation

Updated

• Refactored the BlockHistoryEstimator check to prevent excessively bumping transactions. Check no longer waits for CheckInclusionBlocks to pass before assessing an attempt.
• Fixed a bug that would use the oldest blocks in the cached history instead of the latest to perform gas estimations.
• Updated L1 gas price calculations for Optimism Ecotone and Fjord upgrades
• Fixed local finality violation caused by an RPC lagging behind on latest finalized block.
• Fixed unreachable code bug which could result in stuck txns

DB Update

• CCIP capability specs migration

v2.14.0 Changelog

v2.13.0

Breaking Change

• Remove the xdai ChainType config option. Moving forward, only gnosis can be used.

Added

• Added a mechanism to validate forwarders for OCR2 and fallback to EOA if necessary
• Added an auto-purge feature to the EVM TXM that identifies terminally stuck transactions either through a chain specific method or heurisitic then purges them to unblock the nonce. Included 4 new toml configs under Transactions.AutoPurge to configure this new feature: Enabled, Threshold, MinAttempts, and DetectionApiUrl.
• Add option to include GasPriceSubunits pipeline to include gasPriceSubunits in median ocr2 transmission (only to be used with Starknet chain for now)
• Added config option HeadTracker.FinalityTagBypass to force HeadTracker to track blocks up to FinalityDepth even if FinalityTagsEnabled = true. This option is a temporary measure to address high CPU usage on chains with extremely large actual finality depth (gap between the current head and the latest finalized block).
• Added config option HeadTracker.MaxAllowedFinalityDepth maximum gap between current head to the latest finalized block that HeadTracker considers healthy.

Updated

• Fixed CPU usage issues caused by inefficiencies in HeadTracker.

HeadTracker's support of finality tags caused a drastic increase in the number of tracked blocks on the Arbitrum chain (from 50 to 12,000), which has led to a 30% increase in CPU usage.

The fix improves the data structure for tracking blocks and makes lookup more efficient. BenchmarkHeadTracker_Backfill shows 40x time reduction.

• Fixed metric description on mercury_transmit_queue_load

DB Update

• Improve handling of postgres connection settings and driver versions

v2.13.0 Changelog

v2.12.0

Added

• Added a mechanism to validate forwarders for OCR2 and fallback to EOA if necessary

• Added a new CLI command, blocks find-lca, which finds the latest block that is available in both the database and on the chain for the specified chain. Added a new CLI command, node remove-blocks, which removes all blocks and logs greater than or equal to the specified block number.

• Re-enable abandoned transaction tracker

• Enable configurable client error regexes for error classification. New toml configuration options for [EVM.NodePool.Errors] to pass regexes on NonceTooLow, NonceTooHigh, ReplacementTransactionUnderpriced, LimitReached, TransactionAlreadyInMempool, TerminallyUnderpriced, InsufficientEth, TxFeeExceedsCap, L2FeeTooLow, L2FeeTooHigh, L2Full, TransactionAlreadyMined, Fatal, and ServiceUnavailable.

• Validate user email before asking for a password in the chainlink CLI.

• Add configurability to mercury transmitter

  [Mercury.Transmitter]
  TransmitQueueMaxSize = 10_000 # Default
  TransmitTimeout = "5s" # Default

Updated

• core/chains/evm/logpoller: Stricter finality checks in LogPoller, to be more robust during rpc failover events
• Reducing the scope of 0233 migration to include only 5th word index which is required for CCIP
• Fixed CPU usage issues caused by inefficiencies in HeadTracker. HeadTracker's support of finality tags caused a drastic increase in the number of tracked blocks on the Arbitrum chain (from 50 to 12,000), which has led to a 30% increase in CPU usage. The fix improves the data structure for tracking blocks and makes lookup more efficient. BenchmarkHeadTracker_Backfill shows 40x time reduction.
• Validate support for postgresql-client 16, and update docker image's bundled postgresql-client from 15 to 16.

Upcoming change in 2.13.0

• The xdai ChainType has been renamed to gnosis to match the chain's new name. The old value is still supported but has been deprecated and will be removed in v2.13.0.

v2.12.0 Changelog

v2.11.0

Added

• Added a tx simulation feature to the chain client to enable testing for zk out-of-counter (OOC) errors
• Add the pool_rpc_node_highest_finalized_block metric that tracks the highest finalized block seen per RPC. If FinalityTagEnabled = true, a positive NodePool.FinalizedBlockPollInterval is needed to collect the metric. If the finality tag is not enabled, the metric is populated with a calculated latest finalized block based on the latest head and finality depth.

Updated

• Moved JuelsPerFeeCoinCacheDuration under the JuelsPerFeeCoinCache struct in config. Rename JuelsPerFeeCoinCacheDuration to updateInterval. Add stalenessAlertThreshold to JuelsPerFeeCoinCache config. The StalenessAlertThreshold config option has a default of 24 hours which means that it doesn't have to be set unless we want to override the duration after which a stale cache should start throwing errors.
• Updated config for Polygon zkEVM chains.
• HeadTracker now respects the FinalityTagEnabled config option. If the flag is enabled, HeadTracker backfills blocks up to the latest finalized block provided by the corresponding RPC call. To address potential misconfigurations, HistoryDepth is now calculated from the latest finalized block instead of the head. Note that consumers like TXM and LogPoller do not fully use the Finality Tag yet.
• Change LimitTransfer gasLimit type from uint32 to uint64

Upcoming change in v2.13.0

• The xdai ChainType has been renamed to gnosis to match the chain's new name. The old value is still supported but has been deprecated and will be removed in v2.13.0.

v2.11.0 Changelog

v2.10.0

Added

• Gas bumping logic to the SuggestedPriceEstimator. The bumping mechanism for this estimator refetches the price from the RPC and adds a buffer on top using the greater of BumpPercent and BumpMin.
• Added a new configuration field named NodeIsSyncingEnabled for EVM.NodePool that will check on every reconnection to an RPC if it's syncing and should not be transitioned to Alive state. Disabled by default.
• Add preliminary support for "llo" job type (Data Streams V1)
• Add LogPrunePageSize parameter to the EVM configuration. This parameter controls the number of logs removed during prune phase in LogPoller. Default value is 0, which deletes all logs at once - exactly how it used to work, so it doesn't require any changes on the product's side.
• Add Juels Fee Per Coin data source caching for OCR2 Feeds. Cache is time based and is turned on by default with default cache refresh of 5 minutes. Cache can be configured through pluginconfig using "juelsPerFeeCoinCacheDuration" and "juelsPerFeeCoinCacheDisabled" tags. Duration tag accepts values between "30s" and "20m" with default of "0s" that is overridden on cache startup to 5 minutes.
• Add rebalancer support for feeds manager ocr2 plugins.

Fixed

• P2P.V2 is required in configuration when either OCR or OCR2 are enabled. The node will fail to boot if P2P.V2 is not enabled.
• Removed unnecessary gas price warnings in gas estimators when EIP-1559 mode is enabled.

Changed

• Minimum required version of Postgres is now >= 12. Postgres 11 was EOL'd in November 2023. Added a new version check that will prevent Chainlink from running on EOL'd Postgres. If you are running Postgres <= 11 you should upgrade to the latest version. The check can be forcibly overridden by setting SKIP_PG_VERSION_CHECK=true.
• Updated the LimitDefault and LimitMax configs types to uint64

v2.9.1

Changed

• eth_call RPC requests are now sent with both input and data fields to increase compatibility with servers that recognize only one.
• GasEstimator will now include Type 0x3 (Blob) transactions in the gas calculations to estimate it more accurately.

v2.9.0

Added

• chainlink health CLI command and HTML /health endpoint, to provide human-readable views of the underlying JSON health data.
• New job type stream to represent streamspecs. This job type is not yet used anywhere but will be required for Data Streams V1.
• Environment variables CL_MEDIAN_ENV, CL_SOLANA_ENV, and CL_STARKNET_ENV for setting environment variables in LOOP Plugins with an .env file.

  echo "Foo=Bar" >> median.env
  echo "Baz=Val" >> median.env  
  CL_MEDIAN_ENV="median.env"
  

Fixed

• Fixed the encoding used for transactions when resending in batches

Removed

• P2P.V1 is no longer supported and must not be set in TOML configuration in order to boot. Use P2P.V2 instead. If you are using both, V1 can simply be removed.
• Removed TelemetryIngress.URL and TelemetryIngress.ServerPubKey from TOML configuration, these fields are replaced by [[TelemetryIngress.Endpoints]]:

  [[TelemetryIngress.Endpoints]]
  Network = '...' # e.g. EVM. Solana, Starknet, Cosmos
  ChainID = '...' # e.g. 1, 5, devnet, mainnet-beta
  URL = '...'
  ServerPubKey = '...'

v2.8.0

Added

• Added distributed tracing in the OpenTelemetry trace format to the node, currently focused at the LOOPP Plugin development effort. This includes a new set of Tracing TOML configurations. The default for collecting traces is off - you must explicitly enable traces and setup a valid OpenTelemetry collector. Refer to .github/tracing/README.md for more details.
• Added a new, optional WebServer authentication option that supports LDAP as a user identity provider. This enables user login access and user roles to be managed and provisioned via a centralized remote server that supports the LDAP protocol, which can be helpful when running multiple nodes. See the documentation for more information and config setup instructions. There is a new [WebServer].AuthenticationMethod config option, when set to ldap requires the new [WebServer.LDAP] config section to be defined, see the reference docs/core.toml.
• New prom metrics for mercury transmit queue:
  mercury_transmit_queue_delete_error_count
mercury_transmit_queue_insert_error_count
mercury_transmit_queue_push_error_count
  Nops should consider alerting on these.
• Mercury now implements a local cache for fetching prices for fees, which ought to reduce latency and load on the mercury server, as well as increasing performance. It is enabled by default and can be configured with the following new config variables:

  [Mercury]
  
  # Mercury.Cache controls settings for the price retrieval cache querying a mercury server
  [Mercury.Cache]
  # LatestReportTTL controls how "stale" we will allow a price to be e.g. if
  # set to 1s, a new price will always be fetched if the last result was
  # from 1 second ago or older.
  # 
  # Another way of looking at it is such: the cache will _never_ return a
  # price that was queried from now-LatestReportTTL or before.
  # 
  # Setting to zero disables caching entirely.
  LatestReportTTL = "1s" # Default
  # MaxStaleAge is that maximum amount of time that a value can be stale
  # before it is deleted from the cache (a form of garbage collection).
  # 
  # This should generally be set to something much larger than
  # LatestReportTTL. Setting to zero disables garbage collection.
  MaxStaleAge = "1h" # Default
  # LatestReportDeadline controls how long to wait for a response from the
  # mercury server before retrying. Setting this to zero will wait indefinitely.
  LatestReportDeadline = "5s" # Default
  

• New prom metrics for the mercury cache:
  mercury_cache_fetch_failure_count
mercury_cache_hit_count
mercury_cache_wait_count
mercury_cache_miss_count
• Added new EVM.OCR TOML config fields DeltaCOverride and DeltaCJitterOverride for overriding the config DeltaC.
• Mercury v0.2 has improved consensus around current block that uses the most recent 5 blocks instead of only the latest one
• Two new prom metrics for mercury, nops should consider adding alerting on these:
  • mercury_insufficient_blocks_count
  • mercury_zero_blocks_count

Changed

• PromReporter no longer directly reads txm related status from the db, and instead uses the txStore API.
• L2Suggested mode is now called SuggestedPrice
• Console logs will now escape (non-whitespace) control characters
• Following EVM Pool metrics were renamed:
  • evm_pool_rpc_node_states → multi_node_states
  • evm_pool_rpc_node_num_transitions_to_alive → pool_rpc_node_num_transitions_to_alive
  • evm_pool_rpc_node_num_transitions_to_in_sync → pool_rpc_node_num_transitions_to_in_sync
  • evm_pool_rpc_node_num_transitions_to_out_of_sync → pool_rpc_node_num_transitions_to_out_of_sync
  • evm_pool_rpc_node_num_transitions_to_unreachable → pool_rpc_node_num_transitions_to_unreachable
  • evm_pool_rpc_node_num_transitions_to_invalid_chain_id → pool_rpc_node_num_transitions_to_invalid_chain_id
  • evm_pool_rpc_node_num_transitions_to_unusable → pool_rpc_node_num_transitions_to_unusable
  • evm_pool_rpc_node_highest_seen_block → pool_rpc_node_highest_seen_block
  • evm_pool_rpc_node_num_seen_blocks → pool_rpc_node_num_seen_blocks
  • evm_pool_rpc_node_polls_total → pool_rpc_node_polls_total
  • evm_pool_rpc_node_polls_failed → pool_rpc_node_polls_failed
  • evm_pool_rpc_node_polls_success → pool_rpc_node_polls_success

Removed

• Removed Optimism2 as a supported gas estimator mode

Fixed

• Corrected Ethereum Sepolia LinkContractAddress to 0x779877A7B0D9E8603169DdbD7836e478b4624789
• Fixed a bug that caused the Telemetry Manager to report incorrect health

Upcoming Required Configuration Changes

Starting in v2.9.0:

• TelemetryIngress.URL and TelemetryIngress.ServerPubKey will no longer be allowed. Any TOML configuration that sets this fields will prevent the node from booting. These fields will be replaced by [[TelemetryIngress.Endpoints]]
• P2P.V1 will no longer be supported and must not be set in TOML configuration in order to boot. Use P2P.V2 instead. If you are using both, V1 can simply be removed.