Add CMake build system for valkey #1082

…lkey-io#1057) moduleTimerHandler is aeTimeProc handler and event loop gets created with this. However, found that the function return type is int but actually returns "long long" value(i.e., next_period). and return value being assigned to int variable in processTimeEvents(where time events are processed), this might cause an overflow of the timer values. So changed the return type of the function to long long. And also updated other callback function return type to be consistent. I found this when I was checking functions reported in valkey-io#1054 issue stacktrace. (FYI, this is just to update the return type to be consistent and it will not the fix for the issue reported) Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>

Fix typo in test_helper.tcl: even driven => event driven Signed-off-by: chx9 <cheng.huan@icloud.com>

There is no ethernet style virtual device (like lo 127.0.0.1) for RDMA, however a connection with the same local address and peer address are considered as local. Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>

fixes valkey-io#1088 Signed-off-by: Ran Shidlansik <ranshid@amazon.com>

These two test cases run in a loop: * AOF rewrite during write load: RDB preamble=yes * AOF rewrite during write load: RDB preamble=no Both of the test cases build up a lot of data (3-4 million keys when I run locally) so we should empty the data before the second test case. Otherwise, the second test cases adds keys on top of the keys added in the first test case, resulting in the double number of keys and takes more time. Before this commit: [ok]: AOF rewrite during write load: RDB preamble=yes (18225 ms) [ok]: AOF rewrite during write load: RDB preamble=no (37249 ms) After: [ok]: AOF rewrite during write load: RDB preamble=yes (18777 ms) [ok]: AOF rewrite during write load: RDB preamble=no (19940 ms) Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

…ssible (valkey-io#1092) Currently, we create a Lua table without initial capacity even when the capacity is known. As a result, we need to resize the Lua tables repeatedly when converting RESP serialized object to Lua object and it consumes extra cpu resources a bit when we need to transfer RESP-serialized data to Lua world. This patch try to remove this extra resize to reduce (re-)allocation overhead. | name | unstable bb57dfe (rps) | this patch(rps) | improvements | | --------------- | -------- | --------- | -------------- | | evalsha - hgetall h1 | 60565.68 | 64487.01 | 6.47% | | evalsha - hgetall h10 | 47023.41 | 50602.17 | 7.61% | | evalsha - hgetall h25 | 33572.82 | 37345.48 | 11.23% | | evalsha - hgetall h50 | 24206.63 | 25276.14 | 4.42% | | evalsha - hgetall h100 | 15068.87 | 15656.8 | 3.90% | | evalsha - hgetall h300 | 5948.56 | 6094.74 | 2.46% | Signed-off-by: Masahiro Ide <masahiro.ide@lycorp.co.jp> Co-authored-by: Masahiro Ide <masahiro.ide@lycorp.co.jp>

…ey-io#1100) As title description ![image](https://github.com/user-attachments/assets/655324e6-b042-4c2f-b558-b912a7d2c10c) Signed-off-by: hwware <wen.hui.ware@gmail.com>

Since we paused the primary node earlier, the replica may enter cluster down due to primary node pfail. Here set allow read to prevent subsequent read errors. Signed-off-by: Binbin <binloveplay1314@qq.com>

Introduce a `size_t` field into the rax struct to track allocation size. Update the allocation size on rax insert and deletes. Return the allocation size when `raxAllocSize` is called. This size tracking is now used in MEMORY USAGE and MEMORY STATS in place of the previous method based on sampling. The module API allows to create sorted dictionaries, which are backed by rax. Users now also get precise memory allocation for them (through `ValkeyModule_MallocSizeDict`). Fixes valkey-io#677. For the release notes: * MEMORY USAGE and MEMORY STATS are now exact for streams, rather than based on sampling. --------- Signed-off-by: Guillaume Koenig <knggk@amazon.com> Signed-off-by: Guillaume Koenig <106696198+knggk@users.noreply.github.com> Co-authored-by: Joey <yzhaon@amazon.com> Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

…ases (valkey-io#1106) As discussed here: https://github.com/orgs/valkey-io/discussions/1103#discussioncomment-10814006 `cp` can't be used anymore, `rsync` is more powerful and allow to exclude files. Alternatively: 1. Remove the c, d and o files. Which isn't ideal either. 2. Improve the build. Eg. by building inside a `build` directory instead of in the src folder. Ps. I know these workflows aren't trigger in this PR. Only via "Build Release Packages" workflow action: https://github.com/valkey-io/valkey/actions/workflows/build-release-packages.yml.. So I can't fully test in this PR. But it should work ^^ Ps. ps. I did test `rsync -av --exclude='*.c' --exclude='*.d' --exclude='*.o' src/valkey-*` command in isolation and that works as expected! --------- Signed-off-by: Melroy van den Berg <melroy@melroy.org>

- Add systemd support to the build artifact tarballs, so people can use it under systemd compatible distros. As discussed here: https://github.com/orgs/valkey-io/discussions/1103#discussioncomment-10815549. Adding `libsystemd-dev` to install and add `USE_SYSTEMD=yes` to the build. - Cleanup & bring the arm & x86 workflow files in-sync. It was a bit of a mess ;) (removing `jq wget awscli` from the 'Tarball' step) Signed-off-by: Melroy van den Berg <melroy@melroy.org>

…alkey-io#1115) Applying the CVEs against mainline. (CVE-2024-31449) Lua library commands may lead to stack overflow and potential RCE. (CVE-2024-31227) Potential Denial-of-service due to malformed ACL selectors. (CVE-2024-31228) Potential Denial-of-service due to unbounded pattern matching. Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>

Fix the warning introduced in valkey-io#688: ``` unit/test_rax.c:168:15: runtime error: left shift of 36625 by 16 places cannot be represented in type 'int' SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior unit/test_rax.c:168:15 in Fuzz test in mode 1 [7504]: ``` Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

This commit adds initialization code for the fields `io_last_reply_block` and `io_last_bufpos` of the `client` struct. While in the current code flow, these fields are only accessed after being written in the `trySendWriteToIOThreads`, I discovered that they were not being initialized while doing some changes to the code flow of IO threads. I believe it's good pratice to initialize all fields of a struct upon creation, and will avoid future bugs which are usually hard to debug. Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>

This change counts both solo test executions to give an accurate total number of tests being run. --------- Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>

…io#1102) Improved documentation and readability of lua code as well as removed references to Redis. --------- Signed-off-by: Parth Patel <661497+parthpatel@users.noreply.github.com>

Correct the typo in valkey.conf file Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>

Since a7cbca4 ("RDMA: Support .is_local method (valkey-io#1089)"), valkey-server started to support auto-detect local connection, then we can use protected mode for local RDMA device for test. Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>

ctags is used widely on a linux platform, add tags into .gitignore. Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>

…alkey-io#1128) fixes: valkey-io#1123 As per github documentation below is core information on runners. **Linux:** public repositories: 4 cores private repositories: 2 cores **Macos:** its 3 or 4 based on both and its depends on the Processor. **Reference details for more information:** Discussion in valkey-io#1123 - Public repo: https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories - Private repo: https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for--private-repositories Suggested-by: zhenwei pi <pizhenwei@bytedance.com> Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>

Until now, this flag only dumped logs on a failed assert in test case. It is useful that this flag dumps logs on a crash as well. Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

…o#1101) I’ve prepared a minor fix for `processCommand()` function. In `processCommand()`, the `obey_client` variable is created, but some conditional statements call the `mustObeyClient()` function instead of reusing `obey_client`. I’ve modified these statements to `reuse obey_client`. Since I’m relatively new to Redis, please let me know if there are any reasons why the conditional statements need to call `mustObeyClient()` again. Thank you for taking the time to review my PR. Signed-off-by: otheng03 <07c00h@gmail.com>

…valkey-io#1105) Signed-off-by: Masahiro Ide <masahiro.ide@lycorp.co.jp> Signed-off-by: Masahiro Ide <imasahiro9@gmail.com> Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech> Co-authored-by: Masahiro Ide <masahiro.ide@lycorp.co.jp>

There is a lot of bad legacy usage of `default:` with enums, which is an anti-pattern. If you omit the default, the compiler will tell you if a new enum value was added and that it is missing from a switch statement. Someone mentioned on another PR they used `default:` because of this warning, so just removing it, but might create an issue to do a wider cleanup. Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>

Signed-off-by: chx9 <cheng.huan@icloud.com>

Currently when module loading fails due to busy name, we don't have a clean way to assist to troubleshooting. Case 1: when loading the same module multiple times, we can not detemine the cause of its failure without referring to the module list or the earliest module load log. The log may not exist and sometimes it is difficult for people to associate module list. Case 2: when multiple modules use the same module name, we can not quickly associate the busy name without referring to the module list and the earliest module load log. Different people wrote modules with the same module name, they don't easily associate module name. So in this PR, when doing module onload, we will try to print a busy name log if this happen. Currently we check ctx.module since if it is NULL it means the Init call failed, and Init currently only fails with busy name. It's kind of ugly. It would have been nice if we could have had a better way for onload to signal why the load failed. Signed-off-by: Binbin <binloveplay1314@qq.com>

A new option for diskless replication on the replica side. After a network failure, the replica may need to perform a full sync. The other option for diskless full sync is `swapdb`, but it uses twice as much memory, temporarily. In situations where this is not acceptable, and where losing data is acceptable, the `flush-before-load` can be useful. If the full sync fails, the old data is lost though. Therefore, the new option is marked as "dangerous". --------- Signed-off-by: kronwerk <ca11e5e22g@gmail.com> Signed-off-by: kronwerk <kronwerk@users.noreply.github.com> Co-authored-by: kronwerk <ca11e5e22g@gmail.com>

… categories (valkey-io#1140) The module commands which were added to acl categories were getting skipped when `ACL CAT category` command was executed. This PR fixes the bug. Before: ``` 127.0.0.1:6379> ACL CAT foocategory (empty array) ``` After: ``` 127.0.0.1:6379> ACL CAT foocategory aclcheck.module.command.test.add.new.aclcategories ``` --------- Signed-off-by: Roshan Khatri <rvkhatri@amazon.com> Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>

@melroy89

…effect. (valkey-io#1138) this fixes: valkey-io#1116 _Issue details from valkey-io#1116 by @zuiderkwast_ > This config is undocumented since valkey-io#758. The default was changed to "yes" and it is quite useless to set it to "no". Yet, it can happen that some user has an old config file where it is explicitly set to "no". The result will be bad performace, since I/O threads will not do all the I/O. > > It's indeed confusing. > > 1. Either remove the whole option from the code. And thus no need for documentation. _OR:_ > 2. Introduce the option back in the configuration, just as a comment is fine. And showing the default value "yes": `# io-threads-do-reads yes` with additional text. > > _Originally posted by @melroy89 in [valkey-io#1019 (reply in thread)](https://github.com/orgs/valkey-io/discussions/1019#discussioncomment-10824778)_ --------- Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>

Minor typo. Signed-off-by: Binbin <binloveplay1314@qq.com>

…alkey-io#1150) Remove explicit reference to removal and just indicate to avoid using it. Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>

…1119) Similar to valkey-io#860 but this is for HGETALL families (HGETALL/HKEYS/HVALS). This patch moves `prepareClientToWrite` out of the loop to reduce the function overhead. Signed-off-by: Masahiro Ide <imasahiro9@gmail.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>

Ci report this failure: ``` *** [err]: SHUTDOWN NOSAVE can kill a timedout script anyway in tests/unit/scripting.tcl Expected 'BUSY Valkey is busy running a script. *' to match '*connection refused*' (context: type eval line 8 cmd {assert_match {*connection refused*} $e} proc ::test) ``` We can see the logs the shutdown got rejected because there is an AOFRW pending: ``` Writing initial AOF, can't exit. Errors trying to shut down the server. Check the logs for more information. ``` The reason is that the previous test enabled the aof. Signed-off-by: Binbin <binloveplay1314@qq.com>

Typo, comment cleanups. Signed-off-by: Binbin <binloveplay1314@qq.com>

in case of valgrind run, the replica might get disconnected from the primary due to repl-timeout reached. Fix is to configure larger timeout in case of valgrind test. **Partially** fixes: valkey-io#1152 Signed-off-by: Ran Shidlansik <ranshid@amazon.com>

…el-replication (valkey-io#1164) Sometimes when dual-channel is turned off the tested replica might disconnect on COB overrun. disable the replica COB limit in order to prevent such cases. Fixes: valkey-io#1153 Signed-off-by: Ran Shidlansik <ranshid@amazon.com> Signed-off-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Binbin <binloveplay1314@qq.com>

Currently in our daily, if a job fails, it will cancel the other jobs in the same matrix, we want to avoid this so that all jobs in a matrix can eventually run to completion. Docs: jobs.<job_id>.strategy.fail-fast applies to the entire matrix. If jobs.<job_id>.strategy.fail-fast is set to true or its expression evaluates to true, GitHub will cancel all in-progress and queued jobs in the matrix if any job in the matrix fails. This property defaults to true. Signed-off-by: Binbin <binloveplay1314@qq.com>

Minor cleanup, listLast do the same thing and is widely used and easier to understand (less code). Signed-off-by: Binbin <binloveplay1314@qq.com>

1. Make sure to assert the ERR prefix. 2. Match "Syntax error*" in case of the message change. Signed-off-by: Binbin <binloveplay1314@qq.com>

…oc,free} (valkey-io#1169) The zcalloc symbol is a symbol name already used by zlib, which is defining other names using the "z" prefix specific to zlib. In practice, linking valkey with a static openssl, which itself might depend on a static libz will result in link time error rejecting multiple symbol definitions. Fixes: valkey-io#1157 Signed-off-by: Romain Geissler <romain.geissler@amadeus.com>

…o#977) Currently in conf we describe activerehashing as: Active rehashing uses 1 millisecond every 100 milliseconds of CPU time. This is the case for hz = 10. If we change hz, the description in conf will be inaccurate. Users may notice that the server spends some CPU (used in activerehashing) at high hz but don't know why, since our cron calls are fixed to 1ms. This PR takes hz into account and fixed the CPU usage at 1% (this may not be accurate in some cases because we do 100 step rehashing in dictRehashMicroseconds but it can avoid CPU spikes in this case). This PR also improves the description of the activerehashing configuration item to explain this change. Signed-off-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

…y-io#1171) The client that was killed by FUNCTION KILL received a reply of SCRIPT KILL and the server log also showed SCRIPT KILL. Signed-off-by: Binbin <binloveplay1314@qq.com>

Consolidate the cleanup of local variables to a single point within the method, ensuring proper resource management and p reventing memory leaks or double-free issues. Previoslly descused here: - valkey-io#60 (comment) - valkey-io#60 (comment) --------- Signed-off-by: naglera <anagler123@gmail.com> Signed-off-by: Amit Nagler <58042354+naglera@users.noreply.github.com> Co-authored-by: Ping Xie <pingxie@outlook.com>

Remove reference to "the mailing list". We don't have a mailing list.

**Overview** This PR introduces the use of [MurmurHash3](https://en.wikipedia.org/wiki/MurmurHash) as the hashing function for Lua's luaS_newlstr function, replacing the previous simple hash function. The change aims to improve performance, particularly for large strings. **Changes** Implemented MurmurHash3 algorithm in lstring.c Updated luaS_newlstr to use MurmurHash3 for string hashing **Performance Testing:** Test Setup: 1. Ran a valkey server 2. Loaded 1000 keys with large values (100KB each) to the server using a Lua script ``` local numKeys = 1000 for i = 1, numKeys do local key = "large_key_" .. i local largeValue = string.rep("x", 1024*100) redis.call("SET", key, largeValue) end ``` 3. Used a Lua script to randomly select and retrieve keys ``` local randomKey = redis.call("RANDOMKEY") local result = redis.call("GET", randomKey) ``` 4. Benchmarked using valkey-benchmark: `./valkey-benchmark -n 100000 evalsha c157a37967e69569339a39a953c046fc2ecb4258 0` Results: A | Unstable | This PR | Change -- | -- | -- | -- Throughput | 6,835.74 requests per second | 17,061.94 requests per second | **+150% increase** Avg Latency | 7.218 ms | 2.838 ms | **-61% decrease** Min Latency | 3.144 ms | 1.320 ms | **-58% decrease** P50 Latency | 8.463 ms | 3.167 ms | **-63% decrease** P95 Latency | 8.863 ms | 3.527 ms | **-60% decrease** P99 Latency | 9.063 ms | 3.663 ms | **-60% decrease** Max Latency | 63.871 ms | 55.327 ms | **-13% decrease** Summary: * Throughput: Improved by 150%. * Latency: Significant reductions in average, minimum, and percentile latencies (P50, P95, P99), leading to much faster response times. * Max Latency: Slightly decreased by 13%, indicating fewer outlier delays after the fix. --------- Signed-off-by: Shai Zarka <zarkash@amazon.com> Signed-off-by: zarkash-aws <zarkash@amazon.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

…o#1155) valkey-io#1145 First part of a two-step effort to add `WithSlot` API for expiry. This PR is to fix a crash that occurs when a RANDOMKEY uses a different slot than the cached slot of a client during a multi-exec. The next part will be to utilize the new API as an optimization to prevent duplicate work when calculating the slot for a key. --------- Signed-off-by: Nadav Levanoni <nadavl@amazon.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Nadav Levanoni <nadavl@amazon.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>

…primary nodes (valkey-io#1075) There is no limitation in Valkey to create a cluster with 1 or 2 primaries, only that it cannot do automatic failover. Remove this restriction and add `are you sure` prompt to prompt the user. This allow we use it to create a test cluster by cli or by create-cluster. Signed-off-by: Binbin <binloveplay1314@qq.com>

…s `used_memory_thread`. (valkey-io#1179) When profiling some workloads with `io-threads` enabled. We found the false sharing issue is heavy. This patch try to split the the elements accessed by main thread and io-threads into different cache line by padding the elements in the head of `used_memory_thread_padded` array. This design helps mitigate the false sharing between main thread and io-threads, because the main thread has been the bottleneck with io-threads enabled. We didn't put each element in an individual cache line is that we don't want to bring the additional cache line fetch operation (3 vs 16 cache line) when call function like `zmalloc_used_memory()`. --------- Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com> Signed-off-by: Lipeng Zhu <zhu.lipeng@outlook.com> Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech> Co-authored-by: Wangyang Guo <wangyang.guo@intel.com> Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

Hide 'unixsocketgroup' and 'unixsocketperm' into a Unix socket specific data structure. A single opaque pointer 'void *priv' is enough for a listener. Once any new config is added, we don't need 'void *priv2', 'void *priv3' and so on. Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>

…y-io#1182) This special pattern '#' is used to get the element itself, it does not actually participate in the slot check. In this case, passing `GET #` will cause '#' to participate in the slot check, causing the command to get an `pattern may be in different slots` error. Signed-off-by: Binbin <binloveplay1314@qq.com>

Signed-off-by: Eran Ifrah <eifrah@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CMake build system for valkey #1082

Add CMake build system for valkey #1082

Commits on Oct 20, 2024