job queue runs full after upgrade to v4.1.0

[piet66]

If you are looking for a more convenient method to supervise the CPU load, I can promote my Linux script collection 'MxBash'.
If you want to have a try, download it from Z-Way store. For to see it, you have to add the token 'MxBash' to smarthome/settings/App Store Access.

[piet66]

Since no one else on this forum has reported the z-way-server process consuming anything close to 65% %CPU,

I think we must distinguish between 2 different kinds of CPU values:
top displays the current CPU consumption, which may be very high for short moments. That's not a problem. ps computes the average consumption over a certain interval. This should be relatively constant. In my system about 7%.

[AlesKO]

ss.jpg (3.13 KiB) Viewed 21143 times

Just returned from one week vacations

installed v4.1.0-27-g589c9812 version.

[harre]

4.1.1 solved my issues with long queues that have been bothering me for a while.

[feuersee]

running the commands: ps -L -p `pidof z-way-server`; ps -F -p `pidof z-way-server`

Screenshot 2023-08-05 084844.gif (33.52 KiB) Viewed 21112 times

[feuersee]

sorry, second screenshot for ps -F -p `pidof z-way-server`:

[seattleneil]

The output from the ps commands you provided confirms the zway/core thread is consuming an excessive amount of CPU time. Specifically, the z-way-server process has been running for 11 days (since July 27) and the zway/core thread has consumed nearly 5 days of CPU time. The output for all of the other threads look normal which suggests the problem is related to your Z-Way configuration. Assuming the expert UI does not show an excessive amount of jobs, then you'll need to do some digging as your issue is different from the job queue issue.

The hard part is determining the culprit. There are several linux commands that provide low-level information such as strace and gdb, but the output is difficult to interpret. Instead of using these low-level linux commands, I suggest starting at the Z-Way application layer since it's much friendlier. For example, one possible explanation is that you've accidentally created an automation rule that's a loop. So start by disabling all of the rules using the web UI (http://[IP ADDRESS}:8083/smarthome/#/rules) by selecting the stop icon. If you still have high CPU usage, another explanation is you may have a mis-behaving app. So continue by pausing all of the non-essential apps (http://[IP ADDRESS}:8083/smarthome/#/apps/instance) by selecting the pause icon for your local apps (do not pause "System Cron", "Inband Notifier" and "Z-Wave Network Access"). If you still have high CPU usage, check the log file (/var/log/z-way-server.log) to see it something unexpected is being logged. If top continues to show high CPU usage of the z-way-server process, a draconian approach is make a full backup of your Z-Way configuration and a Z-Wave network backup, stop the Z-Way process, rename the /opt/z-way-server directory (run: sudo mv /opt/z-way-server /opt/z-way-server-KEEP), download and reinstall the latest Z-Way software (run: PKG=z-way-4.1.1-lws16_armhf.deb && wget -O /tmp/$PKG https://storage.z-wave.me/z-way-server/$PKG && sudo dpkg -i /tmp/$PKG) and then restore the .zbk backup (using the expert web UI - http://[IP ADDRESS}:8083/expert/#/network/control). The idea behind this draconian approach is to see whether a clean Z-Way install fixes your excessive CPU utilization problem. Note that you should be able to control your Z-Wave devices using the expert UI. Also note that all of your automaton settings will be undefined (those settings are contained in the .zak full backup file made from the smarthome UI). With a clean Z-Way installation, I'm assuming/hoping the CPU utilization of the z-way-server process will become normal. If this assumption proves to be correct, then you can either rebuild your automation info and add-on apps or compare the contents of /opt/z-way-server/automation/storage/configjson-*.json with the original file from your renamed version (opt/z-way-server-KEEP). You can also try restoring the .zak file, but it's reasonably likely that using the .zak backup file will restore the excessive CPU usage problem).

If you want to try isolating the culprit using a low-level linux command, to trace what's going on with the z-way-server process zway/core thread, run:
sudo strace -fp `pidof z-way-server` 2>&1 | grep XXXXX | grep -v nanosleep (where XXXXX is the zway/core LWP value as shown by the ps -L -p `pidof z-way-server` command).

Good luck.

[feuersee]

Hello and thank you,

i stopped all apps i do not need and now the cpu-load is down again! There are some apps that not longer work under 4.1.1 (i lost philips hue app too )

Greetings Stephan

[PoltoS]

Which app was causing it? You can stop apps one by one to locate it.

[piet66]

You could also provide the list of your stopped apps. That way I can check if I am using any of them.