z-way-server doesn't start with init.d

[piet66]

Hello,
I have an effect for a few days that I don't understand. Maybe someone can help me.

It has happened now twice that the server has stopped running for no apparent reason. OK, this happened from time to time and it could always be restarted.

But the last times, it was not possible to start it with init.d. No message I've found in the syslog.
But I could run it manually with the command 'LD_LIBRARY_PATH=libs ./z-way-server'. And it worked fine.

And after a reboot, the problem seems to be solved.

Any idea where I can have a look?

[seattleneil]

But the last times, it was not possible to start it with init.d.

Controlling the z-way-server process by invoking "/etc/init.d/z-way-server stop|start" is incorrect as most modern linux distributions have switched to using systemd for managing linux services. This transition occurred with the jessie version of Raspberry Pi OS. In the new/systemd approach, users start and stop services using the systemctl command. You can see this for yourself. To check the status of the z-way-server process, run: systemctl status z-way-server. To stop the z-way-server process, run: systemctl stop z-way-server. To start the z-way-server process, run: systemctl start z-way-server. For the question/issue you've asked, before using systemctl to start the z-way-server process, check that there isn't a z-way-server process started outside of systemd, run: ps -ef | grep z-way-server. Then ask systemd about the z-way-server process to see why the process exited, run: systemctl status z-way-server. Rather than start a new z-way-server process, I suggest using the systemd restart command to clean up any lock files, run: systemctl restart z-way-server. If there's a problem with libraries, the problem should show up on the systemctl status z-way-server output.

The transition to systemd has caused a lot of confusion since it retains backward compatibility with init.d scripts. This means users still see the familiar init.d scripts and calling the init.d scripts the legacy way still works, but doing so can easily interfere with systemd. For example, when you reboot your Pi, systemd will automatically start the z-way-server process because systemd retains backward compatibility with the init.d scripts. All is well until a user invokes "/etc/init.d/z-way-server start" as this can create a second instance of the z-way-server process that runs outside of systemd. In theory, the Z-Way developers could address this confusion by replacing the z-way-server init.d script with a systemd service file. This is not a criticism of the Z-Way developers since lots of services still use init.d scripts. Perhaps the Z-Way developers might consider making the transition to a systemd service file as systemd is able to automatically restart processes that have failed. This would solve the first issue you raised:

It has happened now twice that the server has stopped running for no apparent reason. OK, this happened from time to time and it could always be restarted.

Another benefit is that systemd will log startup processing and log the reason a process terminated. To see systemd's history for the z-way-server process, run: journalctl -u z-way-server. Yes, systemd is much more powerful than the legacy init.d approach, but the power of systemd comes at the cost of being new, different and more complicated than the init.d approach. For example, systemd also handles mounting of filesystems and scheduled tasks (similar to cron, but does not replace cron).

Good luck.

[piet66]

Hello seattleneil,
Thank you for your detailed information. I still have a few things to add:

ZWay still doesn't provide a service configuration file for systemd. It only provides an init.d file. For starting z-way-server with systemctl, systemd takes this init.d file. IMO, it's quite the same as starting the program directly with init.d.

● z-way-server.service - LSB: RaZberry Z-Wave service
Loaded: loaded (/etc/init.d/z-way-server; generated)
Active: active (running) since Sat 2023-11-11 13:23:19 CET; 20h ago
Docs: man:systemd-sysv-generator(8)
Tasks: 11 (limit: 2059)
CGroup: /system.slice/z-way-server.service
└─442 z-way-server

Systemd offers a whole range of additional possibilities. But in order to benefit from these, you must have a service configuration file. In case anyone is interested: systemd.service.
I would appreciate it very much if someone would provide a configuration file for z-way-server.

Unfortunately, systemd cannot help with my problem. If the program terminates silently without giving a reason, systemd cannot output anything more:

● z-way-server.service - LSB: RaZberry Z-Wave service
Loaded: loaded (/etc/init.d/z-way-server; generated)
Active: active (exited) since Tue 2023-11-07 17:44:39 CET; 3 days ago
Docs: man:systemd-sysv-generator(8)
Process: 28844 ExecStart=/etc/init.d/z-way-server start (code=exited, status=0/SUCCESS)

My problem is still unsolved:
- Why does the ZWay service stop working from time to time silently without giving a reason?
- Why was the service not started again. The output of systemctl stayed the same:

● z-way-server.service - LSB: RaZberry Z-Wave service
Loaded: loaded (/etc/init.d/z-way-server; generated)
Active: active (exited) since Tue 2023-11-07 17:44:39 CET; 3 days ago
Docs: man:systemd-sysv-generator(8)
Process: 28844 ExecStart=/etc/init.d/z-way-server start (code=exited, status=0/SUCCESS)

[seattleneil]

You wrote:

My problem is still unsolved:
- Why does the ZWay service stop working from time to time silently without giving a reason?
- Why was the service not started again. The output of systemctl stayed the same:

For problem 1, if nothing is logged and the z-way-server process randomly exits for no discernable reason, the only idea I have for figuring out why the z-way-server process terminated unexpectedly is to use gdb. Unfortunately, gdb has a pretty steep learning curve which makes this is a difficult problem to solve. To use gdb, stop the z-way-server process and run the following sequence of shell commands as root:
cd /opt/z-way-server; LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./libs:./modules; gdb ./z-way-server

At the gdb prompt, enter "run". When the z-way-server process stops, enter "backtrace" at the gdb prompt to see the frames that were active when the process exited. gdb should also display a reason why the z-way-server process exited. Ideally, gdb will reveal the reason the z-way-server process stopped.

Although not directly related to solving problem 1, your signature shows you are running version Z-Way version 3.2.3. I assume you have good reasons for not upgrading to the latest version. However, I tend to think you'll get more attention from the Z-Way developers if you upgrade to the latest version of Z-Way. Another observation is that no other users on this forum have recently reported a similar problem which suggests there's something unique about your configuration. One of the dangers of the Z-Way software architecture is that it allows users to extend functionality using its javascript engine but it does not protect itself from extensions that could interfere with Z-Way's use of the javascript engine. For example, perhaps you've added an app that's misbehaving, created a javascript virtual device that's misbehaving or accidentally defined a loop in an automation rule.

For problem 2, the solution is much easier. When systemd processes an init.d script, there is no support for restarting a process. One work-around is to run something like monit to detect when the z-way-server process exits and then restart it. IMHO, a preferable solution that's consistent with your approach is to replace the z-way-server init.d script with a service file. Copy something like the following to file /etc/systemd/system/z-way-server.service:

Code: Select all

# systemd configuration for z-way-server

[Unit]
Description=RaZberry Z-Way server
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/opt/z-way-server/z-way-server --daemon --pidfile=/var/run/z-way-server.pid
ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/opt/z-way-server
Type=simple
PIDFile=/var/run/z-way-server.pid
Restart=always
RestartSec=120
User=root
Group=root

[Install]
WantedBy=multi-user.target

It takes a few steps to use the service file, as follows:
1. Stop the current z-way-server process. Run ps -ef | grep z-way to confirm.
2. Move /etc/init.d/z-way-server to /etc/init.d/z-way-server-OLD. This makes it easy to revert back to the init.d method.
3. Create the file /etc/systemd/system/z-way-server.service as shown above.
4. Run: systemctl daemon-reload
5. Run: systemctl start z-way-server. Run systemctl status z-way-server to confirm.
6. To make the z-way-server service start automatically, run: systemctl enable z-way-server

Note that I have not done any testing on the service script; proceed as you see fit.

[piet66]

Hi seattleneil,
thank you for your help.

Problem 1 occurs a few times per year and is not reproducible. I have resigned myself to this, as my watchdog keeps restarting it. At least until the recent events. Frankly, the gdb is too complicated for me.
The Problem exists since years and is already known by the ZWay team, because other users have reported the same. I suspect that the problem is also difficult for the ZWay team to analyze.

I will take a look into your service file. This is probably the only way to get closer to the matter.

[seattleneil]

Problem 1 occurs a few times per year and is not reproducible.

I agree, that's going to be a very difficult problem to solve. If you have not already done so, please consider upgrading to version 4.1.2.

@PoltoS - any chance Z-Way software will provide a systemd z-way-server service file? For compatibility with linux systems that don't support systemd, the installer could check for directory /etc/systemd/system and install the service file if the directory is found (and remove the init.d script) or install the init.d script if not found.

[lanbrown]

The problem deals with the HTTP service and a large number of calls. I still run into the exited issue from time to time.

I plan on testing out using the z-way add for HA and then using the HACS add-on to have multiple HA systems reporting into one. Since the issue deals with HTTP and not websockets, this would resolve my issue since Z-Way to HA uses websockets. The only system that I run into this is where multiple Z-Way systems are reporting into one which then feeds HA.

[piet66]

I can announce that I am now one step further .

I'm now running z-way-server with a systemd configuration file. And it brings up error messages, if it stops running:

-- Logs begin at Sat 2023-11-25 23:16:01 CET, end at Tue 2023-11-28 16:24:23 CET. --
Nov 28 06:03:14 raspberrypi z-way-server[23685]: Got NULL from _zdata_find(paramDH, "readonly")
Nov 28 15:57:40 raspberrypi systemd[1]: z-way-server.service: Main process exited, code=killed, status=11/SEGV
Nov 28 15:57:40 raspberrypi systemd[1]: z-way-server.service: Failed with result 'signal'.
Nov 28 16:00:05 raspberrypi systemd[1]: Started Z-Way Server.

It's a segmentation fault, so far I understand. Now you just have to find the right place. Can anyone help me?

[PoltoS]

SegFault should be pretty easy to chase. If we can have access to your system, we would love to analyse it with gdb in order to fix it. Looks like there is some issue with many HTTP requests....

[piet66]

Hi PoltoS,
thank you for your offer.
But I don't think that's possible in that way. The server stop happens so seldom, that I don't expect anything from an examination with gdb.

I hope that next time a core dump is created. Would you be able to analyze that?

And the problem with not working restart has happened only twice. Since that time, no more.