S2 Qubino Mini dimmers randomly go to "Is Failed"

Discussions about RaZberry - Z-Wave board for Raspberry computer
Post Reply
CorC
Posts: 10
Joined: 10 Aug 2017 10:34

S2 Qubino Mini dimmers randomly go to "Is Failed"

Post by CorC »

I've a number of Qubino Mini Dimmers, support S2 security, in my new home setup.

I've been running this setup since November 2019, and have had intermittent random failure of some of these Qubino S2 devices.

Now I didn't have the perfect reception setup antenna wise yet, so I figured it would improve when I would have a better setup.

Now that 3.06 z-way has stabilised my system, and the antenna reception is also greatly improved by adding a u.fl and external antenna, I still have the random failure of some of the Mini Dimmers.

It's hard for me to pin down when this occurs exactly, it can be fixed by cutting power to the Qubino Module. However I'd like to understand what causes this "Is Failed" state.

Is it the razberry or the qubino device having an issue, and how should I go about debugging this?

Thing worth noting:

I don't have any other S2 devices except the Qubino Mini Dimmers.
It's not the same module failing all the time, It seems random a bit
Although it seems like the modules that tend to be furthest away from the controller suffer this issue the most.
Also, but I don't have enough empirical data, it's just a hunch, this tends to happen when my raspberrypi is busy / having a high CPU load.

By analysing the logs, it looks like the razberry is waiting for a reply on the "S2 Nonce Get" command, but never gets it when the module is on "Is Failed" state.

When I trigger an interview when the module is in failed state, and explicitly ask SecurityS2 interview, you'll see the request command being scheduled again:

[2020-06-12 08:45:08.510] [D] [zway] SETDATA devices.26.instances.0.commandClasses.159.data.grantedKeys.S0 = False
[2020-06-12 08:45:08.511] [D] [zway] SETDATA devices.26.instances.0.commandClasses.159.data.grantedKeys.S2Unauthenticated = False
[2020-06-12 08:45:08.513] [D] [zway] SETDATA devices.26.instances.0.commandClasses.159.data.grantedKeys.S2Authenticated = True
[2020-06-12 08:45:08.515] [D] [zway] SETDATA devices.26.instances.0.commandClasses.159.data.grantedKeys.S2Access = False
[2020-06-12 08:45:08.516] [D] [zway] SETDATA devices.26.instances.0.commandClasses.159.data.security = True
[2020-06-12 08:45:08.518] [D] [zway] SETDATA devices.26.instances.0.commandClasses.159.data.secureNodeInfoFrames.S2Authenticated = byte[0]
[2020-06-12 08:45:08.520] [D] [zway] ( zero-length buffer )
[2020-06-12 08:45:08.521] [zway] Node 26:0 CC SecurityS2: requesting secure NIF for security class S2 Authenticated
[2020-06-12 08:45:08.523] [zway] Using security scheme S2 Authenticated
[2020-06-12 08:45:08.524] [zway] Adding job: Secure S2 NIF Get
[2020-06-12 08:45:08.526] [zway] Job 0x13 (Security S2 Nonce Get): Cancelling job: Security S2 Nonce Get
[2020-06-12 08:45:08.527] [zway] Removing job: Security S2 Nonce Get
[2020-06-12 08:45:08.527] [zway] Adding job: Security S2 Nonce Get
[2020-06-12 08:45:08.527] [D] [zway] Security S2 send timer set to 100.000000
[2020-06-12 08:45:13.758] [D] [zway] Job 0x3b: deleted from queue

followed by

[2020-06-12 08:46:56.057] [D] [zway] Security S2 send timeout event
[2020-06-12 08:46:56.059] [zway] Job 0x13 (Secure S2 NIF Get): Security S2 failed to prepare S2 packet
[2020-06-12 08:46:56.071] [zway] Job 0x13 (Security S2 Nonce Get): Cancelling job: Security S2 Nonce Get
[2020-06-12 08:46:56.072] [zway] Removing job: Security S2 Nonce Get
[2020-06-12 08:46:56.074] [zway] Adding job: Security S2 Nonce Get
[2020-06-12 08:46:56.076] [D] [zway] Security S2 send timer set to 100.000000

Controller info:

Firmware

Library Type: Static Controller
SDK Version: 6.81.01
Serial API Version: 05.37

Software Information

Version number: v3.0.6
Compile-ID: 82a4fd59993dc610c06507351714f95f21475262
Compile-Date: 2020-04-28 13:52:20 +0300
UI

UI version 1.5.1
Built date 03-04-2020 13:45:40

User avatar
PoltoS
Posts: 5887
Joined: 26 Jan 2011 19:36

Re: S2 Qubino Mini dimmers randomly go to "Is Failed"

Post by PoltoS »

Looks like unreachable device. May be it hangs?

Few things to check:
1. Does the device send something in this failed state when you click in it? Do you see anything from it in the log?
2. Are neighboring devices reachable from the controller? Try to update routes to devices in the vicinity of the failed one and then update routes to the failed one.

This will help to check if it is a device issue or routing issue

CorC
Posts: 10
Joined: 10 Aug 2017 10:34

Re: S2 Qubino Mini dimmers randomly go to "Is Failed"

Post by CorC »

I had to wait a bit for one of the devices to go into failed state again, currently have 2 in failed state.

I checked the logs for any incoming packets to the controller when hitting the switch button (turning lights on / off or dimming), no incoming packets what so ever, although when you check the "timing info" in expert interface, it shows about 4 to 6 (Depending on the device) of packets that were received, not OK though, and with a ~400 * 10ms packets delay.
Screenshot 2020-06-26 at 08.39.34.png
Screenshot 2020-06-26 at 08.39.34.png (174.29 KiB) Viewed 499 times
Might be an indication as to the routing not being optimal?

I've tried restarting the z-way controller software, and tried a re-organization (the latter should update routes to all devices right? also the ones in vicinity)

I tried to create a manual route to one of the failed devices, however looks like that's not an option when the device is failed?

One of the failed devices should be able to directly communicate with the controller.

Both devices are the same type Mini Dimmers from Qubino with the S2 security specs.
Screenshot 2020-06-26 at 08.43.06.png
Screenshot 2020-06-26 at 08.43.06.png (98.73 KiB) Viewed 499 times

User avatar
PoltoS
Posts: 5887
Joined: 26 Jan 2011 19:36

Re: S2 Qubino Mini dimmers randomly go to "Is Failed"

Post by PoltoS »

Timing info shows sent packets. Red means undelivered, so your device never got it.

Looks like a device issue. If it does not answer, you can not do anything. Please contact Qubino support and show them logs and screenshots. They are familiar with Z-Way

CorC
Posts: 10
Joined: 10 Aug 2017 10:34

Re: S2 Qubino Mini dimmers randomly go to "Is Failed"

Post by CorC »

Update on the issue, Qubino has been able to reproduce the issue, and one of their developers is looking into it, full issue description below:

Qubino Support

11:15 AM (10 hours ago)

to me
Hello Cor,

We've managed to replicate the issue by including three Mini Dimmers, in S2unauth, in ZwaveMe and toggling the inputs. After some attempts, the first Mini Dimmer was marked as failed. After playing around with all three for some more time, all three were marked as failed.

Our current understanding is, that this happens due to the routed packets (range) and due to the packets being encrypted (secure inclusion). With secure inclusion, the device and the gateway have to exchange additional packets (Nonce Get/Report) before the actual encrypted application packets is sent. Each of the packet also has to be acknowledged. It appears that is some cases, some of the packets are not acknowledged, so the gateway resend the mentioned packets multiple times - first directly, then to all the other known neighbours of the node (it tries to send these packets three times, via each node). We currently believe that these retries result in the Mini Dimmer being blocked.

A bug ticket was opened for this issue, so one of the developers will have a more detailed look at it, to confirm that this is indeed the issue and to see, what could be done about it. This is a functionality, that is handled by the zwave sdk. Currently, we can only recommend, that you try to include your devices in unsecure mode, to see if that will resolve the issue.

We thank you for your patience and your help.

Post Reply