Z-way-server occasionally goes down

Discussions about Z-Way software and Z-Wave technology in general
lanbrown
Posts: 279
Joined: 01 Jun 2021 08:06

Re: Z-way-server occasionally goes down

Post by lanbrown »

I have that running on two systems. Only one that was the intermediate between Home Assistant and the other Z-Way systems would go to exited. The interesting thing is, that the process was still somewhat running. I noticed an automation didn't kick off and saw another z-way system was exited. Not many devices on it but a multi-sensor on it was still getting updates to Home Assistant though, just commands getting sent to it were not working. So I did the gdb process on this node as well. I should probably just go through them all (only four others) and just run them all through gdb.

I do have the new systems ready to go, so maybe I should do the swap first. Guaranteed it isn't a MicroSD card as these are using NVME drives and can handle .7 DWPD. They even have a battery in them as a backup.
JohannesF
Posts: 36
Joined: 04 Jan 2021 13:20

Re: Z-way-server occasionally goes down

Post by JohannesF »

Thanks - will try.
JohannesF
Posts: 36
Joined: 04 Jan 2021 13:20

Re: Z-way-server occasionally goes down

Post by JohannesF »

With your help I've been able to start Z-Way under gdb. Thanks for you help and your patience!

I am seeing lots of messages saying "New Thread..." and then "Thread ... exited".
Waiting now for the next crash...
lanbrown
Posts: 279
Joined: 01 Jun 2021 08:06

Re: Z-way-server occasionally goes down

Post by lanbrown »

If it is saying exited, then that is probably going to be a problem. I had the first crash and I think I got everything for support. I still think you need to keep your running under gdb as well. Just in case they need something else. I'm getting all of the data collected for support that I think they'll need. I did just try to tell gdb to run again and I was getting a lot of exiting messages as well. I'm going to do some OS patching, restart the system, disable monit, stop the z-way-server process and run it under gdb again.
lanbrown
Posts: 279
Joined: 01 Jun 2021 08:06

Re: Z-way-server occasionally goes down

Post by lanbrown »

I just got my second crash. I'm awaiting to hear from @PoltoS if they need any further information.
JohannesF
Posts: 36
Joined: 04 Jan 2021 13:20

Re: Z-way-server occasionally goes down

Post by JohannesF »

Congrats ;-)
Mine is still running.
gdb keeps firing pairs of "[New Thread 0x...(LWP nnnn)]" and "[Thread 0x... (LWP nnnn) exited]"
nnnn is then incremented.
These messages are generated approximately 1 per second.
Every now and then other messages are displayed.

Code: Select all

[New Thread 0x6a9fb440 (LWP 7263)]
[New Thread 0x6c1fe440 (LWP 7264)]
[Thread 0x6a9fb440 (LWP 7263) exited]
[Thread 0x6c1fe440 (LWP 7264) exited]
[New Thread 0x6c1fe440 (LWP 7265)]
[Thread 0x6c1fe440 (LWP 7265) exited]
Got NULL from _zdata_find(eventDH, "status")
Got NULL from _zdata_find(eventDH, "parameters")
lanbrown
Posts: 279
Joined: 01 Jun 2021 08:06

Re: Z-way-server occasionally goes down

Post by lanbrown »

I won but I don't like the prize.

If I were you, I would restart your system and start from scratch.
JohannesF
Posts: 36
Joined: 04 Jan 2021 13:20

Re: Z-way-server occasionally goes down

Post by JohannesF »

Stay optimistic! Perhaps PoltoS can identify and fix the root cause of your crashes and everything will be alright.

I was already thinking to set up a second raspi with a new razpberry and a fresh SD card. But as a first attempt I would restore the configuration from a backup. Configuring from scratch would be a major endeavour.
lanbrown
Posts: 279
Joined: 01 Jun 2021 08:06

Re: Z-way-server occasionally goes down

Post by lanbrown »

If your issue is the same as mine, then all of that won't resolve the issue. Same issue with a Ubuntu VM as I have had with it running on a Raspberry Pi. This also goes back to 3.x days.

Before I moved my Z-Wave devices to Z-Way, I had a Ubuntu VM running and I was playing around with it talking to a Raspberry Pi Z-Way instance with one device on it. I had this issue and it was only on the VM. When I moved to the Raspberry Pi, it was a clean install and still had the issue.
lanbrown
Posts: 279
Joined: 01 Jun 2021 08:06

Re: Z-way-server occasionally goes down

Post by lanbrown »

It did it again and it even after a reboot I kept seeing the new threads. The source still looks to be the same.

Code: Select all

cat gdb_output_20230109b
sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./libs:./modules gdb ./z-way-server
GNU gdb (Raspbian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./z-way-server...
(No debugging symbols found in ./z-way-server)
(gdb) handle SIGPIPE nostop
Signal        Stop	Print	Pass to program	Description
SIGPIPE       No	Yes	Yes		Broken pipe
(gdb) handle SIGILL nostop
Signal        Stop	Print	Pass to program	Description
SIGILL        No	Yes	Yes		Illegal instruction
(gdb) handle SIGHUP nostop
Signal        Stop	Print	Pass to program	Description
SIGHUP        No	Yes	Yes		Hangup
(gdb) set pagination off
(gdb) run
Starting program: /opt/z-way-server/z-way-server 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Z-Way version v4.0.2 from 2022-12-25 18:20:13 +0300 (build 0445874bba9f9ee7fd9dba8c8a52660764098ff5)
Using default configuration file config.xml
Looking for modules in ./modules
Module modcrypto.so loaded
Module modsockets.so loaded
Module modenocean.so loaded
Module moddhs.so loaded
Module modhomekit.so loaded
Module modzbw.so loaded
Module modiconinstaller.so loaded
Module modinstaller.so loaded
Module modnotification2ext.so loaded
Module modzwave.so loaded
Module modws.so loaded
Module modhttp.so loaded
Module modskininstaller.so loaded
Module modmqtt.so loaded
Error: 'debug-port' number is out of range. Skipping it.
[New Thread 0xb2fff440 (LWP 1477)]
[New Thread 0xb69d2440 (LWP 1478)]
[New Thread 0xb3066440 (LWP 1479)]
[New Thread 0xb3055440 (LWP 1480)]
[New Thread 0xb3044440 (LWP 1481)]
[New Thread 0xb27fe440 (LWP 1482)]
[New Thread 0xb1bff440 (LWP 1483)]
[New Thread 0xb13fe440 (LWP 1484)]
[New Thread 0xb07ff440 (LWP 1485)]
[New Thread 0xafbff440 (LWP 1486)]
[Thread 0xafbff440 (LWP 1486) exited]
[New Thread 0xafbff440 (LWP 1487)]
[Thread 0xafbff440 (LWP 1487) exited]
[New Thread 0xafbff440 (LWP 1488)]
[Thread 0xafbff440 (LWP 1488) exited]
[New Thread 0xafbff440 (LWP 1489)]
[Thread 0xafbff440 (LWP 1489) exited]
[New Thread 0xafbff440 (LWP 1490)]
[Thread 0xafbff440 (LWP 1490) exited]
[New Thread 0xafbff440 (LWP 1491)]
[Thread 0xafbff440 (LWP 1491) exited]
[New Thread 0xafbff440 (LWP 1492)]
[Thread 0xafbff440 (LWP 1492) exited]
[New Thread 0xafbff440 (LWP 1493)]
[New Thread 0xaccdf440 (LWP 1494)]
[New Thread 0xac32d440 (LWP 1495)]
[New Thread 0xabb2c440 (LWP 1496)]
[New Thread 0xab32b440 (LWP 1497)]
[Thread 0xafbff440 (LWP 1493) exited]
[Thread 0xabb2c440 (LWP 1496) exited]
[Thread 0xab32b440 (LWP 1497) exited]
[Thread 0xac32d440 (LWP 1495) exited]
[Thread 0xaccdf440 (LWP 1494) exited]
[New Thread 0xac32d440 (LWP 1498)]
[New Thread 0xab32b440 (LWP 1499)]
[New Thread 0xaccdf440 (LWP 1500)]
[New Thread 0xabb2c440 (LWP 1501)]
[New Thread 0xafbff440 (LWP 1502)]
[Thread 0xac32d440 (LWP 1498) exited]
[Thread 0xab32b440 (LWP 1499) exited]
[Thread 0xafbff440 (LWP 1502) exited]
[Thread 0xaccdf440 (LWP 1500) exited]
[Thread 0xabb2c440 (LWP 1501) exited]
[New Thread 0xabb2c440 (LWP 1503)]
[New Thread 0xaccdf440 (LWP 1504)]
[New Thread 0xafbff440 (LWP 1505)]
[New Thread 0xab32b440 (LWP 1506)]
[New Thread 0xa8a50440 (LWP 1507)]
[Thread 0xafbff440 (LWP 1505) exited]
[Thread 0xabb2c440 (LWP 1503) exited]
[Thread 0xab32b440 (LWP 1506) exited]
[Thread 0xaccdf440 (LWP 1504) exited]
[Thread 0xa8a50440 (LWP 1507) exited]
[New Thread 0xa8a50440 (LWP 1508)]
[New Thread 0xaccdf440 (LWP 1509)]
[New Thread 0xab32b440 (LWP 1510)]
[New Thread 0xabb2c440 (LWP 1511)]
[New Thread 0xafbff440 (LWP 1512)]
[Thread 0xa8a50440 (LWP 1508) exited]
[Thread 0xabb2c440 (LWP 1511) exited]
[Thread 0xaccdf440 (LWP 1509) exited]
[Thread 0xab32b440 (LWP 1510) exited]
[Thread 0xafbff440 (LWP 1512) exited]
[New Thread 0xafbff440 (LWP 1513)]
[New Thread 0xab32b440 (LWP 1514)]
[New Thread 0xaccdf440 (LWP 1515)]
[Thread 0xaccdf440 (LWP 1515) exited]
[New Thread 0xabb2c440 (LWP 1516)]
[New Thread 0xaccdf440 (LWP 1517)]
[Thread 0xafbff440 (LWP 1513) exited]
[Thread 0xab32b440 (LWP 1514) exited]
[Thread 0xabb2c440 (LWP 1516) exited]
[Thread 0xaccdf440 (LWP 1517) exited]
[New Thread 0xaccdf440 (LWP 1518)]
[New Thread 0xabb2c440 (LWP 1519)]
[New Thread 0xab32b440 (LWP 1520)]
[Thread 0xaccdf440 (LWP 1518) exited]
[New Thread 0xaccdf440 (LWP 1521)]
[Thread 0xab32b440 (LWP 1520) exited]
[New Thread 0xab32b440 (LWP 1522)]
[Thread 0xabb2c440 (LWP 1519) exited]
[Thread 0xaccdf440 (LWP 1521) exited]
[Thread 0xab32b440 (LWP 1522) exited]
[New Thread 0xab32b440 (LWP 1523)]
[New Thread 0xaccdf440 (LWP 1524)]
[New Thread 0xabb2c440 (LWP 1525)]
[Thread 0xab32b440 (LWP 1523) exited]
[Thread 0xaccdf440 (LWP 1524) exited]
[Thread 0xabb2c440 (LWP 1525) exited]
[New Thread 0xabb2c440 (LWP 1526)]
[New Thread 0xaccdf440 (LWP 1527)]
[Thread 0xabb2c440 (LWP 1526) exited]
[Thread 0xaccdf440 (LWP 1527) exited]
[New Thread 0xaccdf440 (LWP 1533)]
[New Thread 0xabb2c440 (LWP 1534)]
[New Thread 0xab32b440 (LWP 1535)]
[Thread 0xabb2c440 (LWP 1534) exited]
[New Thread 0xabb2c440 (LWP 1536)]
[Thread 0xaccdf440 (LWP 1533) exited]
[Thread 0xab32b440 (LWP 1535) exited]
[New Thread 0xab32b440 (LWP 1537)]
[Thread 0xabb2c440 (LWP 1536) exited]
[Thread 0xab32b440 (LWP 1537) exited]
[New Thread 0xab32b440 (LWP 1538)]
[New Thread 0xabb2c440 (LWP 1539)]
[New Thread 0xaccdf440 (LWP 1540)]
[Thread 0xab32b440 (LWP 1538) exited]
[Thread 0xabb2c440 (LWP 1539) exited]
[New Thread 0xabb2c440 (LWP 1541)]
[Thread 0xaccdf440 (LWP 1540) exited]
[Thread 0xabb2c440 (LWP 1541) exited]
[New Thread 0xabb2c440 (LWP 1542)]
[Thread 0xabb2c440 (LWP 1542) exited]
[New Thread 0xabb2c440 (LWP 1543)]
[New Thread 0xaccdf440 (LWP 1544)]
[New Thread 0xab32b440 (LWP 1545)]
[Thread 0xabb2c440 (LWP 1543) exited]
[Thread 0xaccdf440 (LWP 1544) exited]
[New Thread 0xaccdf440 (LWP 1546)]
[Thread 0xab32b440 (LWP 1545) exited]
[Thread 0xaccdf440 (LWP 1546) exited]
[New Thread 0xaccdf440 (LWP 1547)]
[Thread 0xaccdf440 (LWP 1547) exited]
[New Thread 0xaccdf440 (LWP 1548)]
[New Thread 0xab32b440 (LWP 1549)]
[New Thread 0xabb2c440 (LWP 1550)]
[Thread 0xab32b440 (LWP 1549) exited]
[New Thread 0xab32b440 (LWP 1551)]
[Thread 0xabb2c440 (LWP 1550) exited]
[Thread 0xaccdf440 (LWP 1548) exited]
[Thread 0xab32b440 (LWP 1551) exited]
[New Thread 0xab32b440 (LWP 1552)]
[Thread 0xab32b440 (LWP 1552) exited]
[New Thread 0xab32b440 (LWP 1553)]
[New Thread 0xaccdf440 (LWP 1554)]
[New Thread 0xabb2c440 (LWP 1555)]
[Thread 0xab32b440 (LWP 1553) exited]
[Thread 0xaccdf440 (LWP 1554) exited]
[Thread 0xabb2c440 (LWP 1555) exited]
[New Thread 0xaccdf440 (LWP 1556)]
[Thread 0xaccdf440 (LWP 1556) exited]
[New Thread 0xaccdf440 (LWP 1557)]
[Thread 0xaccdf440 (LWP 1557) exited]
[New Thread 0xaccdf440 (LWP 1558)]
[New Thread 0xabb2c440 (LWP 1559)]
[New Thread 0xab32b440 (LWP 1560)]
[Thread 0xaccdf440 (LWP 1558) exited]
[Thread 0xabb2c440 (LWP 1559) exited]
[New Thread 0xabb2c440 (LWP 1561)]
[Thread 0xab32b440 (LWP 1560) exited]
[Thread 0xabb2c440 (LWP 1561) exited]
[New Thread 0xabb2c440 (LWP 1562)]
[Thread 0xabb2c440 (LWP 1562) exited]
^C[New Thread 0xabb2c440 (LWP 1563)]

Thread 1 "z-way-server" received signal SIGINT, Interrupt.
0xb6302aec in __GI___clock_nanosleep_time64 (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0xbefff3a8, req@entry=0xbefff3a0, rem=0xbefff3b8, rem@entry=0xbefff3b0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:52
52	../sysdeps/unix/sysv/linux/clock_nanosleep.c: No such file or directory.
(gdb) bt
#0  0xb6302aec in __GI___clock_nanosleep_time64 (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0xbefff3a8, req@entry=0xbefff3a0, rem=0xbefff3b8, rem@entry=0xbefff3b0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:52
#1  0xb6302be0 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0xbefff3ec, rem=rem@entry=0xbefff3ec) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:92
#2  0xb6309390 in __GI___nanosleep (requested_time=requested_time@entry=0xbefff3ec, remaining=remaining@entry=0xbefff3ec) at nanosleep.c:27
#3  0xb630927c in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55
#4  0x0001352c in main ()
(gdb) 
Hopefully this is what support will need to figure out the issue.
Post Reply