Z-way-server occasionally goes down

[lanbrown]

@5d 5h 54m
It's been a while that the beast has been running that long without crashing.

I've ran into the same thing. I did upgrade to 4.0.2 from 4.0.1 since mine was up for so long. Clearly running it under gdb is doing something different than when it runs natively.

[Edwin Benavidez]

It's difficult to pinpoint the exact issue without further information and access to your system. Here are some steps you can take to try to troubleshoot the problem:

1. Check the system logs for any error messages that may provide clues as to what is causing the crashes. You can use the "sudo journalctl -u z-way-server" command to view the logs.

2. Check if there are any issues with your z-way devices or HTTP devices by performing basic tests on them to see if they are functioning properly.

3. Make sure that your Raspberry Pi has enough resources (e.g. CPU, memory) to handle the workload of running z-way and all of your devices. If necessary, you may need to upgrade your hardware or reduce the number of devices connected to your system.

4. Check if there are any software conflicts or compatibility issues that could be causing the crashes. This may involve updating the software on your devices or the Raspberry Pi itself.

[piet66]

5. Maybe the lifetime of your SD card slowly is going to end. Consider moving to a new one.

[lanbrown]

It's difficult to pinpoint the exact issue without further information and access to your system. Here are some steps you can take to try to troubleshoot the problem:

1. Check the system logs for any error messages that may provide clues as to what is causing the crashes. You can use the "sudo journalctl -u z-way-server" command to view the logs.

2. Check if there are any issues with your z-way devices or HTTP devices by performing basic tests on them to see if they are functioning properly.

3. Make sure that your Raspberry Pi has enough resources (e.g. CPU, memory) to handle the workload of running z-way and all of your devices. If necessary, you may need to upgrade your hardware or reduce the number of devices connected to your system.

4. Check if there are any software conflicts or compatibility issues that could be causing the crashes. This may involve updating the software on your devices or the Raspberry Pi itself.

1) All good suggestions, but won't solve the issue. Nothing is in the log messages and Z-Way support also cannot find anything. Whatever is causing the crash is not indicated in the log.

2) In my case, it is just a single system out of six that has the issue. The issue has been on more than one system; a VM as well as on a Raspberry Pi

3) CM4 with 8GB of RAM; 1 to 2 % CPU usage and not even 400MB of RAM used out of the 8GB. It is not a resource issue.

4) Only the Z-Way software is installed. All using the latest updates and as previously mentioned, happens under a VM as well as on a Raspberry Pi.

5. Maybe the lifetime of your SD card slowly is going to end. Consider moving to a new one.

5) Not an SD card issue either.

If it was any of the above, then running it under gdb would result in the same issue, which so far has not been the case.

[JohannesF]

Hi All,

still no crash after > 8d. Exciting!
I can more or less echo what Ianbrown said:
1) I do see a lot of error messages in the log file but can't identify what made the system go down. I'd appreciate if you could look into the extract of the log which I had posted with the question. I inserted a blank line to show when the system crashed.
2) I've been facing this issue fo a long time. Have tried different Pis, reinstalled the whole system, removed any other task from the machine (like AirPrint server...), swapped the SD card. No fundamental change.
3) Monit tells that the system uses 1.6% of CPU and around 10% of the 1GB memory. That's probably not the root cause either.
4) System is now dedicated to Z-way. And Monti and dbg to debug. I have also updated the system regularly.
5) Already tried that. New raspi, new sd card (32GB), system and Z-way freshly installed. Settings recovered from backup.

At one point I had the impression that the system would run less unstable after inactivation most of the HTTP devices.
I felt some improvement also after a fresh setup and restore. I was wondering if some garbage had been left behind when e.g. installing a device failed.

Does anyone have any experience with "garbage removal"? Could this be the reason for so many error messages in the log?

Thanks and regards
Johannes

[lanbrown]

I do believe the issue is related to HTTP in general. The system I have an issue with talks to five other Z-Way systems that have Z-Way devices. This system also is what integrates into Home Assistant. So the system is right in the middle so it sees a lot of HTTP traffic. The remote z-way server app uses polling via HTTP whereas the Home Assistant integration is using WebSockets.

I can speed the issue up by changing the polling from 5 seconds to the "recommended" setting of 1 second. Running z-way under gdb has resulted in no crashing on the same exact system, with the only change being running z-way-server under gdb. If it was a hardware or CPU issue, then it would still be present when running under gdb. If it was a memory related, then I would expect that it would be worse under gdb since gdb would need to consume memory.

I have disabled monitoring via monit as that would be counterproductive. While the z-way-server process is running, it is not running natively but running under gdb. So the starting and stopping of it as well as just monitoring the process is not compatible with monit.

[JohannesF]

Hmmm? Just tried to see what gdb is doing.

I had started the debugger 9 days ago from an SSH connection from my MacBook to the Raspi. Obviously, the terminal session has been closed meanwhile.
Listing now all processes on the Raspi with ps -e doesn't bring up any gdb process. Z-way-server is listed with PID 17939 and running smoothly.
(If I start gdb and list processes in a second terminal session, gdb is listed.)

I wonder if gdb is running at all.

[lanbrown]

What I did was VNC from my Mac to the Raspberry Pi. I opened a terminal on the Raspberry Pi and ran the commands from it. Then I disconnected from the VNC session. If I want to see what has been outputted in the terminal from gdb, I just VNC to the Raspberry Pi. If you want to continue to run z-way-server under gdb in the hopes that it will crash again so you can provide information to support so they can fix the crash, then I would make sure all updates are applied, you have VNC configured, do a reboot, upon the system coming up after the reboot, disable monit monitoring (this can be done via the GUI and the disable is only temporary until the next reboot, shutdown the z-way-server and then run the gdb commands in a terminal on the Raspberry Pi started via VNC from you Mac.

If you need help in setting up VNC so your Mac can work with it:

Code: Select all

Enable VNC:
1) sudo systemctl enable vncserver-x11-serviced 
2) Generate the password you wish to use in screen with vncpasswd -service
sudo vncpasswd -service 
3) Create and edit the following file here: /etc/vnc/config.d/common.custom
Enter the following config: Authentication=VncAuth
4) Restart the vnc service:
sudo systemctl restart vncserver-x11-serviced 
5) Open Screen on your mac with the instructions below, and use the password you provided to vncpasswd.

[JohannesF]

Hi,

That's the output from gdb when I run it in a VNC session.
Needless to say that Z-Way-Server doesn't start.

I am completely lost...

Code: Select all

pi@raspi4Zway:~ $ cd /opt/z-way-server
pi@raspi4Zway:/opt/z-way-server $ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./libs:./modules gdb ./z-way-server
GNU gdb (Raspbian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./z-way-server...
(No debugging symbols found in ./z-way-server)
(gdb) handle SIGPIPE nostop
Signal        Stop	Print	Pass to program	Description
SIGPIPE       No	Yes	Yes		Broken pipe
(gdb) handle SIGILL nostop
Signal        Stop	Print	Pass to program	Description
SIGILL        No	Yes	Yes		Illegal instruction
(gdb) handle SIGHUP nostop
Signal        Stop	Print	Pass to program	Description
SIGHUP        No	Yes	Yes		Hangup
(gdb) set pagination off
(gdb) run
Starting program: /opt/z-way-server/z-way-server 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Z-Way version v4.0.2 from 2022-12-25 18:20:13 +0300 (build 0445874bba9f9ee7fd9dba8c8a52660764098ff5)
Using default configuration file config.xml
Looking for modules in ./modules
Module modhttp.so loaded
Module modinstaller.so loaded
Module modsockets.so loaded
Module modzbw.so loaded
Module modenocean.so loaded
Module modnotification2ext.so loaded
Module modws.so loaded
Module modiconinstaller.so loaded
Module modhomekit.so loaded
Module modmqtt.so loaded
Module modzwave.so loaded
Module modskininstaller.so loaded
Module moddhs.so loaded
Module modcrypto.so loaded
Error: 'debug-port' number is out of range. Skipping it.
Failed to open log file: 13
[New Thread 0x73066440 (LWP 3271)]
[New Thread 0x769d2440 (LWP 3272)]
[New Thread 0x72865440 (LWP 3273)]
[New Thread 0x72854440 (LWP 3274)]
[New Thread 0x72843440 (LWP 3275)]
[New Thread 0x72832440 (LWP 3276)]
[2023-01-07 20:49:10.459] [E] [core] result: Error: Error: Permission denied
    at Error (native)
    at automation/updateBackendConfig.js:608:4
    at automation/updateBackendConfig.js:672:3
    at automation/main.js:23:1
[2023-01-07 20:49:10.461] [E] [core] result: Error: Error: Uncaught Error: Permission denied
    at Error (native)
    at automation/main.js:23:1
                          

[lanbrown]

Make this:

Code: Select all

pi@raspi4Zway:/opt/z-way-server $ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./libs:./modules gdb ./z-way-server

to this:

Code: Select all

pi@raspi4Zway:/opt/z-way-server $ sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./libs:./modules gdb ./z-way-server

You need to use sudo