Highlighted

Fix for "New main PID <nlserver_watchdog_pid> does not belong to service, and PID file is not owned by root. Refusing." error

selvandhanm3541

01-11-2019

Hello there,

In the last couple weeks, the nlserver6 service fails to start with the following error:

systemd[1]: Starting LSB: Neolane Communication Suite...

...

...

nlserver6[...]: Starting nlserver6: [  OK  ]

systemd[1]: New main PID <nlserver_watchdog_pid> does not belong to service, and PID file is not owned by root. Refusing.

systemd[1]: New main PID <nlserver_watchdog_pid> does not belong to service, and PID file is not owned by root. Refusing.

systemd[1]: Failed to start LSB: Neolane Communication Suite.

systemd[1]: Unit nlserver6.service entered failed state.

systemd[1]: nlserver6.service failed.

Upon investigation, it is found that this was caused by a recent systemd package update to version systemd-219-67.el7_7.1.x86_64/systemd-219-67.el7_7.2.x86_64 which includes the following change which in turn introduced stricter PID files and MAINPID handling (as /var/run/nlserver6.pid is owned by 'neolane' user, but systemd expects it to be owned by root).

core: be stricter when handling PID files and MAINPID sd_notify() messages (#1663143) -> https://github.com/systemd/systemd/commit/db256aab13d8a89d583ecd2bacf0aca87c66effc

Excerpt from https://github.com/systemd/systemd/commit/db256aab13d8a89d583ecd2bacf0aca87c66effc#diff-ab78220e1270... wherein the above error originates from:

if (st.st_uid != 0) {

    log_unit_error(UNIT(s), "New main PID "PID_FMT" does not belong to service, and PID file is not owned by root. Refusing.", pid);

    return -EPERM;

}

Looks like a bug has been requested https://github.com/systemd/systemd/issues/8085 to fallback the above change as it breaks several daemons and a PR https://github.com/systemd/systemd/pull/8098 also made to relax the restriction. But, this doesn’t seem to be included in the CentOS 7.7.1908 update yet or probably planned in a future update and so we have downgraded systemd to the last version 219-62.el7_6.9 which doesn't include the above (PID restriction) change.

Steps to downgrade systemd to 219-62.el7_6.9 (from 219-67.el7_7.1/219-67.el7_7.2😞

- Downgrade systemd to 219-62.el7_6.9:

  >> yum downgrade http://vault.centos.org/7.6.1810/updates/x86_64/Packages/systemd-219-62.el7_6.9.x86_64.rpm http://vault.centos.org/7.6.1810/updates/x86_64/Packages/systemd-libs-219-62.el7_6.9.x86_64.rpm http://vault.centos.org/7.6.1810/updates/x86_64/Packages/systemd-sysv-219-62.el7_6.9.x86_64.rpm

  Note: Downgrade other libs (like libgudev1) which requires latest systemd udpate (219-67.el7_7.1/219-67.el7_7.2)

- Disable systemd package on /etc/yum.conf:

  # Disable systemd packages for install/update temporarily to workaround `core: be stricter when handling PID files and MAINPID sd_notify() messages (#1663143)` (https://github.com/systemd/systemd/commit/db256aab13d8a89d583ecd2bacf0aca87c66effc) included in 219-67.el7_7.1/219-67.el7_7.2

  exclude=systemd*

Hope this helps for others who are facing the above issue.

Adobe Campaign Support Team - Suggest you to implement a fix for this on nlserver6 service script (possibly?) (/etc/init.d/nlserver6) as the above systemd PID file handling restriction is likely to be included in a future release again. Thanks!

Cheers,

Selva