V4.1.2: steadily increasing cpu load

[piet66]

One week ago, I upgraded from 3.2.3 to 4.1.2.
Unfortunately, I noticed that the CPU load is constantly increasing, just like in 4.1.0.

cpuK.png (61.22 KiB) Viewed 25448 times

CPU load of z-way-server in 3.2.3: 5.9%, in 4.1.2 at the moment:38.5%.

In my opinion it's caused by zway/core, which takes in 3.2.3 about 45 percent of the z-way-server cpu, and in 4.1.2 about 87 percent.

CPU load in 4.1.2:

Code: Select all

             COMMAND  %CPU       TIME    SECONDS    PERCENT 
        z-way-server  38.5    3224:29     193469     100.00 
                   *******
        z-way-server   0.0       0:18         18       0.01 
     OptimizingCompi   0.0       0:16         16       0.01 
     v8:SweeperThrea   0.0       0:49         49       0.03 
     v8:SweeperThrea   0.0       0:35         35       0.02 
     v8:SweeperThrea   0.0       0:21         21       0.01 
     v8:SweeperThrea   0.0       0:10         10       0.01 
           zway/core  33.5    2811:38     168698      87.20    !!!!!!!!!!!!!
      zway/webserver   0.7      61:13       3673       1.90 
         zway/timers   0.2      18:07       1087       0.56 
     zway/dev/ttyAMA   3.8     325:50      19550      10.10 
        zway/sockets   0.0       0:59         59       0.03 
     zway/http/async   0.0       0:00          0          0 

For comparison in 3.2.3:

Code: Select all

             COMMAND  %CPU       TIME    SECONDS    PERCENT 
        z-way-server   5.9     215:09      12909     100.00 
                   *******
        z-way-server   0.0       0:08          8       0.06 
     OptimizingCompi   0.0       0:12         12       0.09 
     v8:SweeperThrea   0.0       0:20         20       0.15 
     v8:SweeperThrea   0.0       0:12         12       0.09 
     v8:SweeperThrea   0.0       0:06          6       0.05 
     v8:SweeperThrea   0.0       0:03          3       0.02 
           zway/core   2.6      97:34       5854      45.35       !!!!!!!!!!!!!!
      zway/webserver   0.5      21:08       1268       9.82 
         zway/timers   0.2       7:14        434       3.36 
     zway/dev/ttyAMA   2.3      86:33       5193      40.23 
        zway/sockets   0.0       0:26         26       0.20 

I will revert to 3.2.3.

[seattleneil]

Sigh, increasing CPU load in the zway/core thread has been an ongoing problem. Although my having the same problem as you does nothing to fix the problem, I'm hoping that having multiple users reporting this problem will cause @PoltoS to pay more attention to the problem and provide a fix, as he did with the 4.1.0 job queue problem.

I tend to think the solution will involve running a multi-thread profiling tool such as gperftools. Several users have made educated guesses about the cause such as the http device app, but nothing conclusive.

My fingers are crossed that a solution will be provided. If not, we may need to consider other Z-Wave platforms. This post looks interesting:

Code: Select all

https://www.home-assistant.io/blog/2023/12/12/silicon-labs-official-partnership-nabu-casa/

[michap]

Unfortunately, I noticed that the CPU load is constantly increasing, just like in 4.1.0.

Hi,
what tool you have used for this (CPU load table)? (for compare the values)

My htop does not show such high usage - but maybe really depend on the used modules...

Michael

[aLiEnHeAd]

Same problem here.
Version 4.1.2 - load of 1.0
z-way-server process uses 100% CPU

Code: Select all

top - 21:02:54 up 2 days,  1:37,  2 users,  load average: 1.08, 1.01, 1.00
Tasks: 121 total,   1 running, 120 sleeping,   0 stopped,   0 zombie
%Cpu(s): 26.1 us,  0.0 sy,  0.0 ni, 73.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :    922.1 total,    553.2 free,    155.7 used,    213.2 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.    712.0 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  427 root      20   0  246508 117340  21612 S 100.0  12.4   2058:11 z-way-server

I suspect a new HTTP-Device as the cause (Polling the motion status of my reolink camera)

[piet66]

Unfortunately, I noticed that the CPU load is constantly increasing, just like in 4.1.0.

Hi,
what tool you have used for this (CPU load table)? (for compare the values)

My htop does not show such high usage - but maybe really depend on the used modules...

Michael

My own tool, MxBash V2.0.0 . You can download it from ZWay store. I've just uploaded it.
I'm not sure that the zway team is able to verify it speedy. Till that you have to add the token 'MxBash' to your Smarthome Settings/App Store Access to see it.

[seattleneil]

You can also use the top command to see similar info, as follows:

Code: Select all

$ top -b -n 1 -H -p `pidof z-way-server`
top - 17:11:34 up 36 days,  5:27,  1 user,  load average: 0.13, 0.11, 0.09
Threads:  10 total,   0 running,  10 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.2 us,  0.0 sy,  0.0 ni, 96.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   1848.2 total,    649.6 free,    190.6 used,   1008.0 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.   1499.1 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
24456 root      20   0  242868 100904  21640 S   6.7   5.3 267:09.56 zway/core
24450 root      20   0  242868 100904  21640 S   0.0   5.3   0:15.83 z-way-server
24451 root      20   0  242868 100904  21640 S   0.0   5.3   0:00.76 OptimizingCompi
24452 root      20   0  242868 100904  21640 S   0.0   5.3   0:00.81 v8:SweeperThrea
24453 root      20   0  242868 100904  21640 S   0.0   5.3   0:00.55 v8:SweeperThrea
24454 root      20   0  242868 100904  21640 S   0.0   5.3   0:00.40 v8:SweeperThrea
24455 root      20   0  242868 100904  21640 S   0.0   5.3   0:00.22 v8:SweeperThrea
24457 root      20   0  242868 100904  21640 S   0.0   5.3  31:39.69 zway/webserver
24458 root      20   0  242868 100904  21640 S   0.0   5.3  11:41.00 zway/timers
24459 root      20   0  242868 100904  21640 S   0.0   5.3  69:12.23 zway/dev/ttyAMA

A useful experiment would be for @aLiEnHeAd to disable the reolink http device and then restart z-way-server. Is the zway/core CPU load much lower after running 24 hours? If so, this information should help the Z-Way developers isolate the problem.

On a tangent, my sense is that doing an http get every minute within Z-Way's javascript engine is asking for trouble since the Z-Way software architecture does not isolate a poorly-behaved javascript app from a well-behaved javascript app. For example, a long-running javascript app will impact the entire core thread. A safer approach would be to create a shell script that does the polling and when motion is detected, the shell script pushes the alert to a way http sensor device (i.e., move the polling logic outside of z-way and use a shell script that calls the curl command in a shell script loop). The benefit of Z-Way's software architecture is that it can run more efficiently and with less RAM than more complicated home automation platforms such as Home Assistant.

[aLiEnHeAd]

A useful experiment would be for @aLiEnHeAd to disable the reolink http device and then restart z-way-server. Is the zway/core CPU load much lower after running 24 hours? If so, this information should help the Z-Way developers isolate the problem.
[...]A safer approach would be to create a shell script [...]

I deactivated the device yesterday and the load is 0.01 with z-way-server using 2-3% CPU.
So at least in my case it solves the problem. I'll try creating a shell script for my task, thanks for your idea.

[seattleneil]

Eureka! Based on the information provided by @aLiEnHeAd, I replaced the HTTP devices with instances of the app called "Virtual Device (JavaScript)". As a result, the z-way-server process on my Raspberry Pi has a CPU load of 4% instead of 12%.

Here's what I did:

1. Added the following line to the file /opt/z-way-server/automation: sh /home/pi/cgi-bin/tasmota_switch.sh

2. Added a file called "tasmota_switch.sh" in directory /home/pi/cgi-bin with the following contents:

Code: Select all

#!/usr/bin/bash

ERR_MSG="ERROR - need 2 arguments: LR_Xmas_Tree|Xmas_Outside_Lights|Railing_Xmas_Light on|off|stat"

case $1 in
   LR_Xmas_Tree)
     IP=tasmota-1329.lan
     ;;
   Xmas_Outside_Lights)
     IP=tasmota-1141.lan
     ;;
   Railing_Xmas_Light)
     IP=tasmota-7253.lan
     ;;
   *)
     echo "$ERR_MSG"
     exit
     ;;
esac

case $2 in
   on)
     wget -t 2 -T 5 --connect-timeout=5 -w 2 --no-http-keep-alive --max-redirect=0 -q -O /dev/null "http://$IP/cm?cmnd=Power%20On"
     echo -n "on"
     ;;
   off)
     wget -t 2 -T 5 --connect-timeout=5 -w 2 --no-http-keep-alive --max-redirect=0 -q -O /dev/null "http://$IP/cm?cmnd=Power%20Off"
     echo -n "off"
     ;;
   stat)
     if `wget -t 2 -T 5 --connect-timeout=5 -w 2 --no-http-keep-alive --max-redirect=0 -q -O - "http://$IP/cm?cmnd=Power" | grep -q OFF`; then echo -n off; else echo -n on; fi
     ;;
   *)
     echo "$ERR_MSG"
     exit
     ;;
esac

I used wget instead of curl since wget consumes less resources. I also set various wget command options to limit the amount of time the HTTP GET took and to close the TCP connection.

3. Created 3 devices using the "Virtual Device (JavaScript)" app. Here's a screenshot for 1 of the devices:

Virtual Device (JavaScript).png (47.07 KiB) Viewed 23714 times

4. Deleted the corresponding HTTP device.

In addition to binary switches, I created a similar shell script for multilevel sensors and multilevel switches. Does anyone sees a downside to this approach?

[piet66]

Now I have a question of understanding: do you think, that the http.request() command is buggy?

I'm also using this command in my apps. And that could explain my problem too.
But then the bug would have to be new in ZWay release 4, as I don't have this problem in release 3.

Can anyone confirm that the issue is new?

[rowlockterms]

Now I have a question of understanding: do you think, that the http.request() command is buggy?

I'm also using this command in my apps. And that could explain my problem too.
But then the bug would have to be new in ZWay release 4, as I don't have this problem in release 3.

Can anyone confirm that the issue is new?

Experiencing the same CPU load increase with http.request() in V4.1.2. Never encountered this in release 3. Anyone else facing this new bug?