More openvswitch woes

[mike-nguyen]

openvswitch %pre scriptlet adds the openvswitch user to the hugetblfs group. Since %pre runs without set -e by default the failures are ignored resulting in worker nodes that do not come online during a cluster install.

These errors are showing up during the rpm-ostree compose:

14:30:05  openvswitch3.1.prein: usermod.rpmostreesave: /etc/passwd.6: lock file already used
14:30:05  openvswitch3.1.prein: usermod.rpmostreesave: cannot lock /etc/passwd; try again later.

[mike-nguyen]

This was addressed by: #1275. It only prevents us from shipping with broken groups for openvswitch

[cgwalters]

We have a new problem now which is is rhel9 (?) builds picked up https://src.fedoraproject.org/rpms/openvswitch/c/a17c9d439da4f7e3bfec0ce4c3b178232d28d3fb?branch=rawhide it sounds like, and I believe today the problem is that use of sysusers.d just clashes with our previously hardcoded bits here https://github.com/openshift/os/blob/master/passwd#L27 and most importantly here:

os/group

Line 45 in 1f2c0eb

openvswitch:x:800:

(note openvswitch has no groups).

Basically we can do one of two things:

• Have the user and group data injected via sysusers
• Hardcode the user data in /usr

But it doesn't make sense to do both. At this point, we could try dropping the hardcoded user/group files from this repo and rely on sysusers (i.e. per machine state). Or we could hardcode the hugetlbfs group.

Now honestly, I think the real fix here is to move openvswithch to use DynamicUser=yes and open the hugetlbfs bits as an earlier privileged operation instead of relying on group access.

[dcbw]

Or we could hardcode the hugetlbfs group.

@cgwalters note that openvswitch being in hugetlbfs only happens on x86-64 and ARM where we support DPDK. Not on POWER or s390.

[dcbw]

@cgwalters I'm curious what's actually clashing here though. sysusers.d(5) says it'll do the things it's asked if the group/user doesn't exist yet:

       g
           Create a system group of the specified name should it not exist yet. Note that u implicitly creates a matching group. The group will be created with no password set.

       m
           Add a user to a group. If the user or group do not exist yet, they will be implicitly created.

But we don't get any error logs out of systemd-sysusers about why it's not doing what it's asked... If openvswitch already exists, shouldn't it just ignore /usr/lib/sysusers.d/openvswitch.conf but since hugetlbfs doesn't exist and OVS isn't in it, it should still do all of that?

[dustymabe]

I did a little investigation on this today.

I tried dropping the hardcoded bits:

diff --git a/group b/group
index e86d91b..1fb1db8 100644
--- a/group
+++ b/group
@@ -42,5 +42,3 @@ nfsnobody:x:65534:
 kube:x:994:
 sshd:x:74:
 chrony:x:992:
-openvswitch:x:800:
-hugetlbfs:x:801:
diff --git a/passwd b/passwd
index 673a3d5..893fd8a 100644
--- a/passwd
+++ b/passwd
@@ -24,4 +24,3 @@ nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin
 kube:x:996:994:Kubernetes user:/:/sbin/nologin
 sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
 chrony:x:994:992::/var/lib/chrony:/sbin/nologin
-openvswitch:x:800:800::/:/sbin/nologin

but that didn't work either (the openvswitch user still doesn't get in the hugetlbfs group). I cracked open the resulting qemu qcow (i.e. without booting it) and I see the openvswitch user and group and the hugetlbfs group in the resulting built image (although with different UID/GID so I know the removal of the hardcoded bits had effect).

I think after we remove the hardcoded bits now what's kicking in is this:

%sysusers_create_compat %{SOURCE2}
%ifarch %{dpdkarches}
%sysusers_create_compat %{SOURCE3}
%endif

where

Source2: openvswitch.sysusers
Source3: openvswitch-hugetlbfs.sysusers

and are defined as:

cat openvswitch.sysusers
#Type Name         ID         GECOS                   Home directory  Shell
u     openvswitch  -          "Open vSwitch Daemons"  /               /sbin/nologin

cat openvswitch-hugetlbfs.sysusers
#Type Name         ID         GECOS                   Home directory  Shell
m     openvswitch  hugetlbfs

So that looks normal.. but what is sysusers_create_compat? It's just a macro that calls a bash script. So I imagine some of that logic inside that bash script (i.e. maybe it works during rpm install for dnf, but not rpm-ostree?) is why we end up with openvswitch with no group.

I will note that if on a running instance I remove openvswitch and hugetlbfs entries and rerun SYSTEMD_LOG_LEVEL=debug systemd-sysusers it does create things appropriately.

[dustymabe]

WDYT of #1317 at least to get us unblocked for now?

[dustymabe]

At this point we can either merge #1317, which drops the hardcoded bits user/group assignments, or #1318, which just works around the RPM scriptlet not working.

IIUC merging #1317 will cause systems that upgrade to have a different (new) UID/GID for the openvswitch user/group and hugetlbfs group on the next reboot. I'm not sure if this is OK or not.

[LorbusChris]

It looks like https://src.fedoraproject.org/rpms/systemd/blob/rawhide/f/sysusers.generate-pre.sh and https://pkgs.devel.redhat.com/cgit/rpms/systemd/tree/sysusers.generate-pre.sh?h=rhel-9.2.0 have diverged, and the latter is missing multiple updates to the script.

Specifically, this change that is missing in RHEL looks like the culprit to me: https://src.fedoraproject.org/rpms/systemd/c/f27d461663bec17ad64422682f260f0020ccc7f7?branch=rawhide