Skip to content

40. tuyaDAEMOM global.alldevices

Jump to bottom
Marco Sillano edited this page Dec 24, 2023 · 8 revisions

'global.alldevices': a OO dB implemented as JSON tree object for fast access.

The goal of 'global.alldevices' JSON structure is to store in a unique place all information required by tuyaDAEMON to get a full insulation layer from the tuya communication details and to have a safe preventive control on allowed operations.

This data structure must be user-maintained, containing info on all controlled devices (tuya-compatible or custom).

tuyaDAEMON devices

  • A real device is a tuya device standard, WiFi and AC-powered (i.e. permanently connected to WiFi), compatible with the _smart-tuya-device_ node. See note.

  • A virtual device is a tuya device accessible from a smart-tuya-device node that connects not directly to the device, but via a gateway (e.g. Zigbee Gateway), shared by many virtual devices. Tuya defines these devices as 'subdevices'.

  • A fake device is a real or artificial device, not handled by smart-tuya-device nodes, but maintained by some ad hoc node-red flow. Also, all fake devices use global.tuyastatus and the dB 'messages' table to share values. Some fake devices:

    • 'mirror' devices: are tuya devices existing in smartlife, but for some reason not found using the smart-tuya-device node, e.g. a battery-powered WiFi sensor. These devices can send/receive status and events to/from node-red using TRIGGERS, via automation in tuya-cloud. See tuyaTRIGGERS flow, e.g. WiFi PIR motion sensor, IR TV remotes, etc. To keep the tuyaDAEMON light, we will implement this not in an extended manner, but only on real exigences basis. Example: Smoke_Detector
      [Since v. 2.2.4] If a mirror device is accessed using core_OPENAPI it becomes a 'real' device, see here.

    • 'custom' devices, real HW devices not tuya-compatibles, using any protocol of communication, with dedicated node-red flows doing the required transformations. Example the "PM_detector", or 433 MHz gateway.

    • 'software' devices, handled by tuyaDEAMON extensions. Example: _system, that reports some parameters about the tuyaDAEMON run, or "watering_sys", a derived device that uses 3 real tuya devices.

devices definition: JSON 'global.alldevices'

JSON alldevices structure: device examples

For 'real' devices (minimal):
           {
        (1)    "id": "123301602cf4325eae00",  
        (2)    "name": "tuya_bridge",
        (14)   "device": "switch-1CH",
               "dps": [
                         {
        (3)         "dp": "1"
                         }
                      ]
           }
           
For 'virtual' devices (minimal)    
// accessed using gateway and cid:
           {
        (1)    "id": "123910468caab5e75887",
        (4)    "cid": "1233acfffe526223",
        (5)    "gateway": "123093b1b788b5992cro7p",
        (14)   "device": "LED_700ml_Humidifier",
        (2)    "name": "umidificatore",
               "dps": [
                         {
        (3)         "dp": "1",
        (2)         "name": "spray",
                         }
                      ]
           }
  
For 'mirror' devices (fake)
//  handled by tuyaTRIGGER, never accessed by the user.  
           {
        (1)    "id": "_PIR_sensor#01",
        (2)    "name": "Sensore di movimento",
       (14)    "device": "PIR_motion",
               "dps": [
                         {
        (10)        "dp": "1010",
                    "name": "Alarm",
                         }
                      ]
            }
  
For `_system` (software fake device)
// handled by tuyaDEAMON extensions: some GET/SET can be defined using capabilities
           {
        (9)    "id": "_system",
        (9)    "name": "HAL@home",
        (14)   "device": "_system",
                "dps": [
                         {
        (10)        "dp": "_laststart",
        (10)        "name": "start"
                         },
                         {
        (10)        "dp": "_ACpower"
                         }
                      ]
           }

Rich optional structure (used by CORE, _system):

           {
        (1)    "id": "123301602cf4325eae00",  
        (2)    "name": "umidificatore",
        (7)    "capability": ["SET","SCHEMA"],
        (14)   "device": "LED_700ml_Humidifier",
        (11)   "power":"AC",
               "dps": [
                          {
        (3)          "dp": "1",
                     "name": "spray",
        (12)         "typefield": "BOOLEANONOFF"
                          },
                          {
                     "dp": "6",
                     "name": "led mode",
        (13)         "capability": "WO",
        (8)          "type": "string",
        (6)          "comment": "values: 'coulour','colourful1' "
                          }
                      ]
           }

notes

  • Any device handled by tuyaDEAMON MUST be present in global.alldvices. If a DP is missed, tuyaDAEMON emits a warning, but the command is processed with defaults.
  • The global.alldvices structure presents 3 branches at the first level: 'real', 'virtual' and 'fake', to keep separate the devices.

  1. for real and virtual devices: the Tuya deviceId, as found using tuya-cli wizard, mandatory, used as an index by CORE.

  2. friendly names, user-defined. Optional but strongly recommended. Any language, utf8. If missed, id or dp are used instead.
    note: a device has multiple names, used in different contexts (by default they can also be the same):

    • The SmartLife name, is used in the APP, and returned by tuya_cli wizard. Not used in TuyaDEAMON.
    • The node-red node name, is used only by node-red to identify the node-red-contrib-tuya-smart-device node. Not used in TuyaDEAMON.
    • The TuyaDAEMON user-defined name, in global.alldevices. Rules:
      • used by CORE in external outputs: global.tuyastatus, debug pad, etc..
      • used by CORE in commands, shares, etc, in place of the ID (the "ID" is not portable, the "name" is portable only if predefined).
      • can almost always be changed by the user at any time (any language accepted).
      • if a name cannot be changed (predefined or for whatever reason) that name starts with an underscore (e.g. "_system")
      • for MQTT compatibility, avoid the chars '%', '$', '+', '#' in names.
      • for file system compatibility, avoid the chars '-', '/', '', ':' in names.
      • Max 40 char (DB 'messages' table limit). (simple rule: use only a-z,A-Z,0-9 and _)

  3. real dp: you can find it in messages from the device (e.g. capturing status change after a command by smartlife), string, mandatory, used by CORE.

  4. cid index: only for virtual devices, in tuya-cli wizard output and in all messages from the device (e.g. capturing status change after a command by smartlife), mandatory, used by CORE.

  5. gateway id of the associated gateway device, only for virtual devices, mandatory, used by CORE.

  6. free comment, allowed in any place, for private use. For multiline comments use 'comment01', 'comment02'.., optional.

  7. device capability, to filter the user commands. Array [ one or more of ('SET','GET','SCHEMA','MULTIPLE') or 'NONE' or 'ALL' plus 'REFRESH'], optional (default ['ALL']), used by CORE.

    • 'SET' == some dp have the SET capability.
    • 'GET' == some dp have the GET capability.
    • 'SCHEMA' == the device honors SCHEMA command.
    • 'MULTIPLE'== the device accepts MULTIPLE command.
    • 'NONE' == the device doesn't accept any command.
    • 'ALL'== the device accepts all commands (SET,GET,SCHEMA,MULTIPLE: only REFRESH must always be specified)
    • 'REFRESH' == the device accepts REFRESH command (since ver. 2.0).

  8. the type defines what values the dp accepts. The type can be: 'boolean' | 'enum' | 'int'| 'string'| 'binary', optional. Used by CORE to force the sent 'set' data type.

    Since ver. 1.4: added "numeric"|"see note"

    Default coding rules for values are:

    • the null ("") string and the "NULL" string becomes NULL (used to replace GET(x) with SET(x):null, see capability 'WW', 'GW').
    • only the strings "false" and "true" and boolean becomes boolean values
    • integer (4) and int-strings ("4") are sent as 'int'.
    • not boolean-string ('true','false') , not int-string ("102") and not null ("","NULL"): data is 'string'|'object'.

    With 'type', the coding rules are:

    • the null ("") string and the "NULL" string becomes NULL.

    • 'boolean'- are converted to false: false, "false", "FALSE", 0. Else true.

    • 'enum' - number-strings ("4") are converted to enum ('int'). Note: take care of cases where is required a string, e.g. "4", for tuyayDAEMON the type is 'string'.

    • 'int' - number-strings ("4") are converted to 'int'.

    • 'string' - numbers (6) are converted to strings: ("6"). Note: sometimes you can see defined as 'enum' a limited choice of strings: e.g. "slow"|"fast". For tuyaDEAMON this is a 'string' type.

    • 'binary' - data is usually a string code64, and data are handled by dedicated decode() and encode() functions (see 'typefield').

    • 'numeric' like 'string': "5.24" or "5,24". Note: not a Tuya type, added for better handling the decimals in any locale.

    • 'see note' - other cases, e.g. a structured object. In tuyadaemontoolkit use the 'DPvalues' field for more info.

      _note: For the 'MULTIPLE' command it is the user's responsibility to give the right coded values, as an object or a JSON string.

  9. a 'fake' device must use any unique id. Suggestions:

    • 'software' devices: uses a string starting with an underscore ("_system"), defined in code.

    • 'mirror' devices: uses a string starting with an underscore ("_siren"), defined in code.

      note: the same device can exist as a 'real' (or 'virtual') device (id from Tuya) for test purposes, and as a 'mirror' device (id from code) in production.

  10. a fake device can use any dp. Suggestions:

    • 'mirror' devices: equal to TUYATRG number (1 - 86500, see core_TRIGGER.triggerMAP node for details), used by tuyaTRIGGER extension, defined in code so cannot be changed.
    • 'software' and 'custom' devices: uses numbers or a string starting with an underscore ("_mode"), defined in code, the user can't change it.
    • for MQTT compatibility, avoid the chars '%', '$', '+', '#' in names.
    • Max 40 char (DB 'messages' table limit).

  11. classifies the real devices using the type of power supply, values: 'BAT' | 'AC' | 'UPS', optional (default 'BAT'), used by SYSTEM.

    • 'BAT' == Battery powered.

    • 'AC' == AC grid powererd.

    • 'UPS' == Uninterruptible Power Source, e.g. AC power with buffer battery or power bank.

      note: If the 'power' is missing, the device is NOT used in connections statistics done by the _system device (see).

  12. identifies the decode() and encode() functions used to convert data from/to devices. Values are user defined in 'format command' 'FastFormat' and 'OUT data process' nodes, optional (default 'no convertion'), used by CORE.
    Since 2.2.0: encode/decode functions are defined in the 'core.*ENCODE/DECODE user library' node.

    _note: If there is a 'typefield' it is the responsibility of the encode()/decode() functions to return the required type (case 'SET'/'GET'). The value of 'type', if it exists, is ignored. _

  13. single dp capability, to filter or change the user commands, one of 'RO'|'WO'|'WW'|'GW'|'RW'|'TRG'|'PUSH'|'SKIP', optional (default 'RW'), used by CORE and by tuyaTRIGGERS flows:

    • note: in some devices GET(dp) don't works, but 'SET(dp):null' get the value (like GET(dp)).
    • 'RW' == read-write, i.e. SET and GET are ok. Don't use SET(dp):null not useful and maybe not allowed.
    • 'WW' == SET is ok, GET(dp) becomes SET(dp):null. This is mandatory if GET(dp) has a non-standard behavior. SET and GET are ok.
    • 'RO' == read-only, i.e. only GET for this dp.
    • 'GW' == GET(dp) is implemented as SET(dp):null. Other SETs are not allowed, i.e. only GET is ok.
    • 'WO' == write-only, i.e. only real SET for this DPs, not GET, not SET(dp):null
    • 'PUSH' == only data PUSHed from the device, i.e. SET and GET not allowed. note: Proactive PUSHed data are compatible with any other capability.
    • 'TRG' == only internal TRIGGERS, i.e. user SET and GET are not allowed (e.g. _system._proxy, in some 'mirror devices'). Access via ´share´ and ´fast_IN´ are allowed.
    • 'SKIP' == commands are not sent to the device, but transformed as an answer and sent directly 'to logging'. Used to process some pseudoDP, not accepted by the tuya device (e.g. '_connected') or to add new features (methods) defined only by tuyaDAEMON-chain (share): only user SET (as a trigger) is allowed.

Tuya defines data transfer type for a DP to report only, send only, or send and report: - Report only: The device reports the status when the DP value changes, without receiving the DP control command. - Send only: The device receives and acts on the DP control command, without reporting the DP status. This leaves you uninformed of the current status of the DP. - Send and report: The device receives and acts on the DP control command, and then reports the DP status.

  1. since 2.2.0: added "device", the device class name from here. Must be a string or a JSON array. Used by tuyaDAEMON.toolkit and in the documentation of known devices.
  • for MQTT compatibility, avoid the chars '%', '$', '+', '#' in names.
  • for file system compatibility, avoid the chars '-', '/', '', ':' in names.
    (simple rule: use only a-z,A-Z,0-9 and _)

This alldevices structure is 'expandible additive': if some custom extension requires info on a device/dp basis, that information can be added to alldevices, provided that the pre-existing definitions are not changed.

tuyaDAEMOM toolkit can help users to manage the global.alldevices structure and to create some useful artifacts in the device installation process.

Output control

Since ver. 2.2.0

Extension: a 'hide' field is defined to increase user control over tuyaDEAMON outputs (global.tuyastatus object, node-red debugpad + MQTT, if installed, DB 'tuyathome.messages' table, if enabled).

The 'hide' strings (optional, default "") are built with 1 or more of the chars:

           "C": no Commands to debugpad + MQTT
           "E": no Event/response to debugpad + MQTT
           "T": no TX (commands) records to DB
           "R": no RX (event/response) records to DB
           "K": Kill, like "CERT" + no `global.tuyastatus` update.

The 'hide' strings can be added to any device and/or property for fine tuning tuyaDAEMON: the device.hide and property.hide are ORed to route any log.

Example:

              {
              id: "_core"
              name: "core"                               // example, user defined
              capability: ["GET", "SET", "SCHEMA"]
              hide: "T"                  // all '_core' Commands not stored on DB
              dps: [
                     {
                     dp: "_version"
                     name: "version"
                     capability: "RW"                  // "SET" and "GET" allowed
                     }
                     {
                     dp: "_heartbeat"    
                     capability: "RO"           // 'PUSH' and 'GET' capabilities.
                     hide: "R"            // '_heartbeat' Events not stored on DB
                     }]}

devices structuration

Since ver. 2.0

Version 2.0 introduces the device structuration concept (see ver.-2.0--Network-and-OO), a powerful and fast mechanism of interaction between devices. This way is easier to define 'derived' devices in OO style, that specialize some base tuya devices for custom tasks, and to create powerful 'chains' of commands.

SHARE actions

Any event can fire one or many tuyaDAEMON commands, and any commands can have one or more conditions to be tested before executing them. The command can be directed also to a remote tuyaDAEMON instance. This mechanism is more flexible than Tuya's 'automation' because the use of 'eval()' allows dynamic commands and extended conditions.

A share can be used also alone, without a firing action but on the user control, on 'core.share IN' node or '_system._doShare' property.

The advantages of "share" are:

  • powerful distributed logic, can solve many problems without custom code.
  • essential to implement 'inheritance' between devices.
  • fast implementation: commands are directly sent via 'fast IN'.
  • Easy creation and maintenance, without node-red changes ("share" are user-defined by JSON in '*Global CORE config'.alldevices node).
  • used alone: the user gets the conditional control and the fork function (using standard commands new extra properties are required).

The drawbacks of the current "share" implementation are:

  • global.alldevices greatly increases in size.
  • greater complexity of global.alldevices, mixing device features with user extensions.
  • loss of portability of global.alldevices; 'shares' are very context-sensitive.

In the global.alldvices structure, any DP can define a 'share' array like this:

         {
   (1)    "share": [{
   (3)        "test": [
                   "tuyastatus[\"HAL@home\"][\"_ACpower\"] == true",
                   "msg.info.value === \"ON\"",
                   "var xnow = new Date(); (xnow.getHours() < 10)"
                         ... more test strings ...
                 ],
   (2)        "action": [                     // like a standard command
                    {
                   "remote" = "NAMEXX",       // optional, send to a remote tuyaDAEMON instance
                   "device": "_system",
                   "property": "_trigger",    
                   "value": "@4000+1000"
                    } ... more action {}...
                 ]
             } ... more {(test[],) action[]} ...
         ]
        }

The same, used alone:

          {
   (4)    "info": {
                  "device"  : "_system"                              // user defined: later used
                  <custom>  : "ON"                               // more user defined (optional)  
                  },
   (1)    "share": [{       
   (3)        "test": [
                   "tuyastatus[\"HAL@home\"][\"_ACpower\"] == true", 
                   "msg.info.value.<custom> === \"ON\"",                // a string: internal (") escaped
                   "var xnow = new Date(); (xnow.getHours() < 10)"
                         ... more test strings ...
                 ],
   (2)        "action": [                                  // any action like a standard command
                    {
                   "remote" = "NAMEXX",        // optional, send to a remote tuyaDAEMON instance
                   "property": "_trigger",            // here missed 'device': default from info
                   "value": "@4000+1000"
                    } ... more action {}...
                 ]
             } ... more {(test[],) action[]} ...
         ]
        }

notes

  1. "share" (optional) defines one or more commands to be sent via 'fast_cmds'. The 'shares' in global.alldevices are processed when an event occurs and the answer message is ready. Alone, a share can be send to 'core.'share IN' node or via '_system._toShare'.
    share[] is an array of {(test[],) action[]} objects plus an optional info{} object.

  2. "action[]" is a mandatory array of 1...n tuyaDAEMON commands, defined using the standard extended format: {"remote", "device", "property", "value"}

    • 'static' mode: the definitions are const objects or strings not starting with '@'.
    • 'dynamic' strings: if the definitions is a string and begins with '@', the rest must be a js code fragment, processed using eval() (e.g. '@4000 + 1000' => eval(4000+1000) => 5000).
    • 'dynamic' objects: in the case of an object, 'dynamic' is recursive, i.e. any property of an object can have a 'dynamic' string value. Example:
                   "share": [{
                              "action": [{
                                    "device": "_system",
                                    "property": "_timerON",
                                    "value": {
                                        "timeout": "@msg.info.value.timeout",
                                        "id": "_testPing24H",
                                        ... more ... }}]}]

action rules:

  • 'remote' is optional and static. It is the name of a tuyaDEAMON instance defined in DEAMONmap.
  • device (none|null|device-name|deviceCid|deviceId), can not be undefined if remote is missed.

For device, property and value:

  • if any is missed (none) or if the property is not a string, or is undefined, the default is from info (msg.info.device, msg.info.property, msg.info.value) or from the event, else default is undefined.
  • if any is === null becomes undefined, also if it exists in info or event (e.g. 'SCHEMA' from local 'switch' device: { device:'switch', property: null, value:null}).
  • else any can be an object (only value) or a string (dynamic or static).

note: property MUST be a string, else it is used the default from info (can be dangerous)!

All commands in the action array are executed, but only if the related test[] eval() to true.

  1. optional "test[]" is an array of expressions that can verify some conditions required to fire all commands in action[]. Any expression MUST eval() to true|false, examples:

    • testing the 'value' of the answer message (it is decoded in msg.info.value). (e.g 'msg.info.value.count > 0')
    • testing any value in tuyastatus, or in global or 'core' flow environments. (e.g. "tuyastatus.core._heartbeat > '11.00.00'")
    • eval() accepts also multiline code, ending with a condition. (e.g. 'let xnow = new Date(); (xnow.getHours() < 10)' => true until 9:59)
    • 'test[]' is optional, the default is true.
    • In the case of more than one test, the tests are evaluated in AND (i.e. all tests must be true to execute the actions).

  2. In the alone share the info{} object can replace the data from the firing event. Required only if some action:{device|property|value} is missed or if msg.info.xxx is used in any eval() expression. You can think of them as 'parameters' for the 'share'.
    Example:

       {
   (4)    "info": {
                "device":"_core",
                "start"  : "11:00:00"                              // user defined: later used
                },
   (1)    "share": [{ 
                "test":["tuyastatus.core._heartbeat > msg.info.start"],     // uses info.start
                "action":[{                      // in action the 'device' is missed => default from info
                       "property":"_info"        // 'value' missed and not in 'info' => undefined
                                                 // action := GET(local.core._info) (using defaults in info)
                       }]
                }]
        }

Implementation notes

'There is no built-in way to store your Function node code in external files.', says knolleary. Really, the unique built-in way to include a library requires from the user the update of the settings.js file and spread a project on many files. Very bad.
Recent Node-red adds some facilities but the problems remain.

Libraries of utility functions are very useful: the code maintenance is easier, and function nodes are not full of copy/paste sections. In tuyaDAEMON some getter functions, for the global.alldevices data, are used in many places. A nightmare.

since 2.0

As a partial solution I found a way to use functions stored in a JSON string. Not so easy to use, it requires JSON stringification (hard debug) and a wrapper function in any node, but it is working.

since 2.2.0

Later I found a different and simpler way to build a singleton object. Thanks to knolleary, using this way we get many advantages:

  • alldevices JSON tree (data) still is in the 'global Config' node for easy user updates.
  • At CORE startup, after all 'On Start' global updates, the 'CORE.Global Objects constructor' node constructs a singleton 'context.global.alldevices', adding all required functions (methods) to data (for details see TuyaDAEMON startup).
  • The function code is in plain JScript inside the 'Global Objects constructor' function node: easy debug.
  • Nothing extra to add to function nodes to use singleton objects, like 'require()' etc...
  • Standard OO use: context.global.alldevices.a_method(params) (memory-only context access style).
  • You can use 'this' in methods, getting reduced and clean code.
  • Faster than 2.0 JSON strategy: 31s vs 106s in 10000 loop test.
  • implemented inside the CORE flow, without external files or changes to node-red settings file (very good).

note: the same strategy is used in ver. 2.2.0 to build more singleton objects, e.g. to encapsulate in a singleton all decode(device-value)/encode(user-value) functions, or for public libraries.

context.global.alldevices methods

method parameters description
getODev (id, limit = null) Find a device object (ODev) in alldevices. Params: id := usr-dev-name > cid > id, limit := real|virtual|fake|null (= all)
getDevName (ODev) Get the device name, using the priority: usr-dev-name > cid > id
getConnectName (ODev) Get the the gateway|device usr-name|id
getODps (ODev, property) Get a DP object (ODps). Params: ODev, property := usr-dp-name|dp
getDpsName (ODev, property) Get a pd name: usr-dp-name > dp
normalize (msg, ODev, ODps) Normalize the msg (see also 'core.fake_cmds' node documentation)
encodeValue (value, ODps) Encodes a value as defined by ODps.type and ODps.typefield, see alldevices-note[8].

devices

Watchdog02 (custom)

Clone this wiki locally