[META] Module API V1

[yossigo]

Module API V1

This is an attempt to scope out all missing or broken Redis Module API capabilities, and create an exahustive list that defines a stable V1 interface.

The Module API was in continuous state of development since Redis 4. Although it maintains ABI backwards compatibility, this does make things more difficult for module developers to mange.

General

Better documentation is desparately needed. Things that come in mind:

1. A step by step / tutorial-like stuff.
2. Better organize the growing reference for better navigation, etc.
3. Some real world examples.

Big Topics

• New, significant capabilities.
• Need work to define the concept or reach a solid API design.
• Potentially have a system-wide effect.

• Asynchronous RM_Call support (RedisModule API for async RM_Call #7992)
• An API to query command flags. This should be done in tandem with improved command introspection, like A better approach for COMMAND INFO for movablekeys commands #7297 and can also provide help as described in help text and cli syntax hints for module commands #4175 and an alternative to Module: add module desc subcommand #3700 (Module API to get command flags #7991)
• I/O APIs for use cases such as encryption or compression of AOF/RDB files, streaming files directly to and from non-file sink or source
• RESP3 Support (A few things that came up #6860, No. 1) (Module api support for RESP3 #8521, Unified Lua and modules reply parsing and added resp3 support to RM_Call #9202)
• Opt-in ACL support for modules
• APIs to extend ACLs and built-in users ([NEW] module api for specifying alternate password hashing algorithms #8329)
• Better support for ACL and module commands (categories?)
• A way for in-place upgrade of a module (old version pass state to new version)

Native Key Access

• Provide better coverage for accessing native Redis data types.
• We should not try to cover all commands and just provide the necessary primitives.

• Streams support as a native type (Stream API in Module #5760)
• Missing hash functions:
  • RM_HashDel (using RM_HashSet with REDISMODULE_HASH_DELETE).
  • RM_HashLen (using RM_ValueLength).
  • Field iteration API, similar to the one provided by sorted sets.
• Missing list functions (see Modules: Add remaining list API functions #8439):
  • RM_ListGet
  • RM_ListInsert
  • RM_ListLen (using RM_ValueLength).
  • RM_ListRemove
  • RM_ListSet
  • Iteration API
• Missing set functions:
  • RM_SetAdd
  • RM_SetCard
  • RM_SetIsMember
  • RM_SetRemove
  • Iteration API
• Optimize the API to provide zero-copy (but safe) access to elements. see [NEW] Avoid allocations for RedisModuleString #8473

Minor Improvements / Fixes

• Need some level of API design or decision making.
• Impact has a limited scope.

• Module Pub/Sub subscribe support (Modules API: Allow modules to Subscribe for Pub/Sub #7298)
• RM_HashSet return code is ambigious (Redis module RM_HashSet() return confusing 0 for hash creation #6914)
• All variadic API functions should also have a non-variadic version ([NEW] Add non-varargs implementations of RedisModule_HashGet and RedisModule_HashSet #7860)
• Support opt-in keyspace notification on native data type access (Modules: Keyspace notifications with native types API #6544)
• Better way to handle RM_BlockClient when called in a context that does not allow that (Module functions not stopped after trying to block client from MULTI #5567)
• Add samples support to mem_usage callback ([MODULES API] RedisModuleTypeMemUsageFunc should accept a samples argument #4177)
• Module command validation step for better MULTI/EXEC handling (Unexpected module command behaviour in multi #3635)
• Add RM_KeyExists function (A few things that came up #6860, No. 11)
• Additional hooks: Before fork, before keyspace iteration on main thread, before/after AOF rewrite to allow aux data storage (Module auxiliary data broken when using AOF and aof-use-rdb-preamble set to no #7350, A few things that came up #6860 No. 12+13))
• Extend defrag API to handle late-defrag of globals and defrag of RAX structures.
• Create more unit tests to improve module API coverage
• Graduate from experimental API some of the stuff (A few things that came up #6860, No. 4)

Trivial

• Optimize network usage when REDISMODULE_POSTPONED_ARRAY_LEN is used (Issue with REDISMODULE_POSTPONED_ARRAY_LEN on module API #6444)
• SendClusterMessage crashes thread safe context (Using 'RedisModule_SendClusterMessage' with ThreadSafeCtx cause redis to crash #5500)
• Get hash slots for a Cluster Node (Module experimental cluster api #5371)
• Concurrent MODULE UNLOAD and a module command in MULTI/EXEC will crash (CRITICAL BUG: redis crashes inside MULTI/EXEC if modules unloaded #4826)
• Provide module file name in MODULE LIST (Module info path #4413)
• Fix RM_StringTruncate() with a zero length which currently does nothing, or document the otherwise unexpected behavior (modules RM_StringTruncate with newlen == 0 #3717)
• Fix RM_WrongArity() crash on thread safe context ([MODULES] Calling RedisModule_WrongArity from a thread-safe context crashes redis #4756)
• Add NOTIFY_MODULE (Need to better understand the use case) (add NOTIFY_MODULE flag #6963)
• Fix RM_BlockClient misuse leading to crashes (Modules: handle NULL pointer in BlockClient #4366)
• Optimize RM_Call by recycling a single client (Optimize RM_Call by recycling the client #4623)
• Refuse to load a module without aof_rewrite callback if preamble is disabled, or disable preamble if such modules are loaded

Module API V2 and beyond

Big things beyond the current Module API V1 effort.

• Expose event loop for arbitrary file descriptor I/O (Event driven networking interfaces for module system #4158)
• Expose the connection abstraction layer to allow modules to do arbitrary TCP and TLS

[zuiderkwast]

@yossigo Please mark as done above:

• RM_HashLen → already covered by RM_ValueLength ✔️ Marked
• RM_HashDel → already possible by setting a field to the special value REDISMODULE_HASH_DELETE using RM_HashSet. ✔️ Marked
• RM_ListLen → already covered by RM_ValueLength ✔️ Marked
• RM_HashSet return code is ambigious (PR: Modules: In RM_HashSet, add COUNT_ALL flag and set errno #8446, issue Redis module RM_HashSet() return confusing 0 for hash creation #6914) ✔️ Marked
• RM_ZsetRem may leave empty sorted set keys, leading to crashes (done in a different PR: RM_ZsetRem: Delete key if empty #8453) -- ❓
• Optimize RM_Call by recycling a single client (done in a different PR: Shared reusable client for RM_Call() #8516) ✔️ Marked
• Missing list functions (done in Modules: Add remaining list API functions #8439) ✔️ Marked
• Add RM_KeyExists function (done in Add RedisModule_KeyExists #9600) ✔️ Marked

[hwware]

Is there anyone working on the missing list and set functions? If not then we'd like to take a look at it and contribute.

[zuiderkwast]

I started working on the list functions, but it was unclear what they (Oran and Yossi) want from the API, so it was paused. I think agreeing on the API is more difficult than the implementation. If you @hwware start working on set and list functions, I will help review and discuss it.