You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently cache supports either sharded or replicated modes. However, there is no sense to replicate very large (e.g. several gigabytes like DVD images) entries to each NUMA node, rather the entries should be stored in one particular node (also the entries are sent in many TDB lookups in sense of #391). In particular, web cache is a bad candidate for replicated data, since there is eviction process and large data set. From the other hand, small and written-once on the configuration time data (#910 and #1350) is a good candidate for replication.
UNIFORM - just pretend that we work on uniform memory (interleaving default NUMA mode)
REPLICATED - inserts and deletes happen on all the nodes, lookup on local node only. At the moment we see configuration time databases only, so no need for lock-free
When a web resource is saved into cache, it can be saved only on current or on all NUMA nodes. All cache-related tests must validate correctness of cache behaviour:
In sharding mode, resource musn't be copied across all the NUMA nodes,
In replicated mode, resource must be copied across all NUMA nodes.
Currently we run tests on CI on single NUMA node, so CI must also be updated to use different number of NUMA nodes. Probably there is more tests that must be run on different NUMA topologies
The text was updated successfully, but these errors were encountered:
We must keep in mind, NUMA node is not always consist of memory and cpu. Sometimes node can have only memory or only cpu. Now we ignore such cases for cache's REPLICA mode, but prohibit to use SHARD cache's mode on setups where some node doesn't have at least one cpu. Also in this case we reserve memory at boot stage for each online node, even if node doesn't have cpus. Doing this we waste some space, however naive solution with using for_each_node_with_cpus is not applicable here, because at this boot stage for_each_node_with_cpus is not available for use.
Here we doing __cache_add_node(), however more proper way to do it is to call only tfw_cache_copy_resp() and tdb_entry_alloc_unique() for each node, instead of calling whole __cache_add_node().
Introduction
Currently cache supports either sharded or replicated modes. However, there is no sense to replicate very large (e.g. several gigabytes like DVD images) entries to each NUMA node, rather the entries should be stored in one particular node (also the entries are sent in many TDB lookups in sense of #391). In particular, web cache is a bad candidate for replicated data, since there is eviction process and large data set. From the other hand, small and written-once on the configuration time data (#910 and #1350) is a good candidate for replication.
Also see Challenges of Memory Management on Modern NUMA System: Optimizing NUMA systems applications with Carrefour for modern NUMA problems and optimization methods, page replication in particular.
Modes of operation
There must be 3 modes fully resolved on TDB side:
Testing
(This is from tempesta-tech/tempesta-test#61 , reopen the issue if necessary).
When a web resource is saved into cache, it can be saved only on current or on all NUMA nodes. All cache-related tests must validate correctness of cache behaviour:
Currently we run tests on CI on single NUMA node, so CI must also be updated to use different number of NUMA nodes. Probably there is more tests that must be run on different NUMA topologies
The text was updated successfully, but these errors were encountered: