Cobra's bits (Posts about centos)

Bad mojo

Cobra — Wed, 16 Dec 2020 13:50:18 GMT

There are many examples of companies that didn't last very long after having been acquired by IBM. In a process called blue washing, IBM replaces the processes, corporate culture and philosophy of the new acquisition with those of its own. This routine inevitably leads to the complete assimilation and absorption of the company's original identity until nothing is left of it.

It is thus no surprise that IBM's spectacular takeover of Redhat in 2018 was met with considerable scepticism and sometimes outright concern. And rightly so: a few days ago Redhat announced the end of CentOS as we know it. Ironically, those who recently upgraded from CentOS 7 to CentOS 8 to get support until 2029 instead of 2024 now have only one year left.

CentOS Stream is not a viable replacement for the most common use case of CentOS, namely, running software suites certified for Redhat as I've described previously on this blog (part I, part II). New distributions filling the gap that CentOS will leave have been announced (Rocky Linux, Project Lenix), and we will see if and when they materialize, and how long they last. In the meantime, I'm glad that I generally avoided Linux distributions with a commercial background.

Opposite extremes

Cobra — Sun, 30 Jun 2019 10:41:20 GMT

I have a CentOS virtual machine because I had to install it on a compute server in my office, and I keep it since it's such an interesting antithesis to the rolling release distribution I prefer for my daily computing environment. CentOS is by a large margin the most conservative of all Linux distributions, and it's sometime useful for me to have access to older software in its natural habitat. Just look at this table comparing the versions of some of the major packages on fully updated Arch and CentOS 7 installations:

                Current                         Arch                            CentOS
linux           5.1.15                          5.1.15                          3.10
libc            2.29                            2.29                            2.17
gcc             9.1.0                           9.1.0                           4.8.5
systemd         242                             242                             219
bash            5.0                             5.0                             4.2
openssh         8.0p1                           8.0p1                           7.4p1
python          3.7.3                           3.7.3                           2.7.5
perl            5.30.2                          5.30.2                          5.16.3
texlive         2019                            2019                            2012
vim             8.1                             8.1                             7.4
xorg            1.20.5                          1.20.5                          1.20.1
firefox         67.0.1                          67.0.1                          60.2.2
chromium        75.0.3770.100                   75.0.3770.100                   73.0.3683.86

You can easily see why I prefer Arch over CentOS as a desktop system.

But CentOS has it's merits, particularly for servers. There's no other distribution (except, of course, its commercial sibling RHEL) with a longer support span: CentOS 7 was released in July 2014 and is supported till end of June 2024. And that's not just a partial support as for the so-called LTS versions of Ubuntu.

Now, I've noticed that CentOS keeps old kernels after updates, befitting its highly conservative attitude. However, in view of the very limited hard disk space I typically give my virtual machines (8 GB), I got a bit nervous when I saw that kernels really seemed to pile up after a few updates. There were five of them! Turned out that's the default, giving the word “careful” an entirely new meaning.

But am I supposed to remove some of these kernels manually? No. I was glad to find that the RHEL developers had already recognized the need for a more robust solution:

yum install yum-utils
package-cleanup --oldkernels --count=3

And to make this limit permanent, I just had to edit /etc/yum.conf and set

installonly_limit=3

Well thought out. 😉

Search package providing a certain command

Cobra — Wed, 24 Apr 2019 16:10:46 GMT

I've posted a short note on this topic almost exactly ten years ago, and it's time for an update. The situation: you've heard or read about a certain tool and want to install it, but you can't find it no matter how hard you try.

My first advice: don't search and install via graphical applications. ”Software centers” popular in consumer distributions may not show command line applications at all, so if you've read my last post and search for dc, you won't find what you are looking for.

Second: not every command comes in a package with the same name. For example, in Archlinux, dc is bundled with bc, and it is the latter (much more popular) application which gives the package its name.

To master such situations, it's time to leave graphical software centers behind and to learn a few basics about the actual package manager underneath. As an example, I'm showing a search for dig and drill, each of which is contained in a differently named package, with the name of these packages depending (as always) on the distribution.

Archlinux

pacman -Fs dig
        bind-tools
pacman -Fs drill
        ldns

Debian

wajig whichpkg /usr/bin/dig
        dnsutils
wajig whichpkg /usr/bin/drill
        ldnsutils

CentOS/Fedora

yum/dnf provides /usr/bin/dig
        bind-utils
yum/dnf provides /usr/bin/drill
        ldns

OpenSUSE

Reportedly, zypper offers the same functionality.

Of the big six, that leaves Gentoo and Slackware. If you use these or a distribution whose package manager is not covered here, while it offers the desired funtionality, send me a note.

Update: An attentive reader reminded me of the Pacman Rosetta, which includes OpenSUSE and Gentoo. 😎

TCAD station, part II

Cobra — Fri, 19 May 2017 17:47:04 GMT

As a testbed for commercial TCAD software, we will use a standard desktop PC equipped with an i4790 CPU and 32 GB RAM, and an Nvidia 710 GT to connect the monitor via DVI. As a substitute for Redhat Enterprise Linux (RHEL) required by all of the packages we're about to evaluate, I'm going to install CentOS.

I create a bootable USB stick by issuing

dd bs=4M if=CentOS-7-x86_64-Minimal.iso of=/dev/sdc status=progress && sync

The first stick doesn't work, and it takes us perhaps half an hour to realize that it's the fault of the stick, not the system. A second one works right away. I manually partition the disk, as in my installation in a virtual machine, choosing btrfs as filesystem for both system and home partitions. I add a UEFI boot partition as well as a swap partition.

During the first boot from hard disk, the system throws an error message concerning nouveau (the open-source driver for the Nvidia graphics card) and subsequently hangs with a kernel soft lock signified by the repeated message that CPU #i stuck for 120s. After a hard reset, the system boots and behaves as expected, but when I boot again to activate the new kernel installed by yum, the system hangs again. Since the hangup is always preceded by the error message from the nouveau driver, we remove the Nvidia card and reboot. Now the systems hangs without any message. Great! A cold boot (with the Nvidia card installed again) seems to help at first, but later the system hangs repeatedly and finally refuses to boot to the login screen at all.

What I thought to be done within an hour has already taken most of this Friday morning. Since the Nvidia card does not seem to responsible for these problems, I start to suspect the btrfs filesystem, which Redhat still considers to be a technology preview. In the afternoon, I thus reinstall the system, this time choosing the default filesystem XFS. And indeed, while I still get the error message from nouveau, the system boots up without any of the previous symptoms. I go home with the conviction that I've solved the problem.

Saturday morning, curiosity gets the better of me and I decide to check if the system still runs. It does, but htop shows that the uptime is only 49 min instead of the expected 13 h. Weird! I check again Sunday morning, and again the uptime is less than an hour. At the same time, I notice that the filesystem usage seems to increase by roughly 1 GB per day. Aha! An 'll /var/crash' confirms my suspicion: the system crashes and dumps the kernel roughly every 2 or 3 hours.

Core dumps can be analyzed to determine their cause. Following the Redhat tutorial and issuing the command

crash /usr/lib/debug/lib/modules/3.10.0-514.16.1.el7.x86_64/vmlinux /var/crash/127.0.0.1-2017-05-01-07\:35\:53/vmcore

I get the following crash report:

KERNEL: /usr/lib/debug/lib/modules/3.10.0-514.16.1.el7.x86_64/vmlinux
        DUMPFILE: /var/crash/127.0.0.1-2017-05-01-07:35:53/vmcore  [PARTIAL DUMP]
                CPUS: 8
                DATE: Mon May  1 07:35:42 2017
          UPTIME: 08:27:43
LOAD AVERAGE: 0.05, 0.03, 0.05
           TASKS: 224
        NODENAME: testbed.de
         RELEASE: 3.10.0-514.16.1.el7.x86_64
         VERSION: #1 SMP Wed Apr 12 15:04:24 UTC 2017
         MACHINE: x86_64  (3600 Mhz)
          MEMORY: 31.9 GB
           PANIC: "Kernel panic - not syncing: Hard LOCKUP"
                 PID: 0
         COMMAND: "swapper/6"
                TASK: ffff880174a9edd0  (1 of 8)  [THREAD_INFO: ffff880174ab8000]
                 CPU: 6
           STATE: TASK_RUNNING (PANIC)

This diagnosis lets me find the cause of the core dumps very easily: it's an unresolved bug reported in September 2016 and given high priority by the developers. The bug has been found in version 7.2.1511 but has not been fixed even in the current version 7.3.1611. However, the bug reporter and others have identified the nouveau driver to be the culprit. And indeed: after removing the GT710 and rebooting, the system does not suffer from any further lockups and core dumps:

 ob@testbed:~$ uptime
19:00:08 up 17 days,  9:00,  2 users,  load average: 0.00, 0.01, 0.05

I have to admit that this experience was an unexpected surprise. CentOS, as a binary compatible clone of RHEL, has the reputation of being the very model of a conservative Linux distribution, and thus to be a paragon of stability and reliability. Consequently, I had expected outdated software, but not buggy implementations of core packages, and critical bugs that are open for 8 month without eliciting any response from a developer.

Thinking twice, however, I realize that I should not have been surprised at all. About ten years ago, we decided to switch our core servers from SUSE Enterprise Linux (SLES) to OpenSUSE since we were entirely frustrated with the support and bug fixing policy of SLES, despite the fact that we payed a handsome amount to Novell every single year. Personally, I'm not too fond of OpenSUSE either, but the core servers don't overly concern me. Our compute servers, for which I'm responsible, are running Debian Testing, and in view of the minimal administrative effort required over the past ten years, I congratulate myself for this decision.

TCAD station, part I

Cobra — Mon, 01 May 2017 12:00:17 GMT

You certainly know the old tune on the lack of “professional” software for Linux, with “professional“ being usually an implicit synonym for Microsoft Office and the Adobe Creative Suite. For a scientist or engineer, people joining this chorus appear to be misinformed and to be motivated by ideology rather than reality. In technically oriented fields, software is in fact mostly cross-platform or even developed primarily for Linux. That's true in particular for multithreaded software with non-negligible demands for computational resources and five- to six-figures price tags. Examples include Maxwell solvers, multiphysics solutions, and TCAD packages.

Commercial software for Linux is usually certified only for one of the enterprise Linux distributions, namely, Redhat and SUSE Enterprise Linux (RHEL and SLES). Some of this products turn out to be actually distribution-agnostic, meaning that they run without any problems also on, e.g., Debian. But in many other cases, the software only runs and even only installs on systems with the distribution it was developed for. I've learned that the hard way, wasting an entire day trying to get the commercial finite-difference time-domain simulation package FDTD solutions to work under Debian Stretch. In the end, we've used Meep instead.

I've vowed that this wouldn't happen again, and since we are in the process to evaluate some selected TCAD solutions (all of which are certified for RHEL only), it seemed to be wise to set up a test server running CentOS (a binary-compatible clone of RHEL). The TCAD software requires a graphical interface, but I did not intend to perform a standard installation, which results in a complete Gnome desktop suitable for a workstation rather than a compute server. For servers, I prefer the installation to be as lightweight as possible. For example, I usually install the tiling window manager wmii if a graphical interface is required or desirable on a server.

Since I'm not at all familiar with CentOS and the available software, I decided to first set up a virtual machine to look for possible pitfalls. For the base instalIation, I've downloaded and installed the minimal ISO which I've then, upon first boot, updated with:

yum update
grub2-mkconfig -o /boot/grub2/grub.cfg

Why the reconfiguration of grub? Well, the update installed a new kernel, but CentOS did not automatically update the grub configuration file. Weird, but true.

CentOS offers three tiling window manager, two of which I'm familiar with: i3 and xmonad (never even heard about spectrwm). On second thought, however, I realized that it may be a better idea to install a more conventional desktop to give the TCAD testers an environment they feel comfortable with. XFCE seemed to be a reasonable compromise and can be installed with these few steps:

yum install epel-release -y
yum groupinstall "X Window System" -y
yum groupinstall "Xfce" -y
systemctl get-default
systemctl set-default graphical.target
systemctl isolate graphical.target

The last step seamlessly starts the X Window system and thus catapults one into the graphical desktop. Slick! Now we only want a resolution higher than the oldfashioned 1024x768 offered by the default (VESA) driver. In other words, we need to install the Virtualbox guest additions:

yum install dkms
yum groupinstall "Development Tools"

To install the guest additions, I first inserted the 'Guest Additions CD Image' in the 'Devices' menu and then downloaded the image. After the download, I mounted the image and compiled the guest additions:

mkdir /media/vboxadditions
mount /dev/cdrom /media/vboxadditions
cd `/media/vboxadditions <file:///home/cobra/Documents/media/vboxadditions>`_
./VBoxLinuxAdditions.run
reboot

Still no higher display resolutions available? The script in this thread enabled me to activate the 'Auto-resize Guest Display' option in the 'View' menu, which finally allowed me to use the desired full HD resolution (on a WQHD monitor):

#!/bin/bash
Diplay_Name=`xrandr | grep connected | cut -d' ' -f1`
Display_Spec=`cvt 1920 1080 | grep Modeline | cut -d' ' -f2 |cut -d '"' -f2`
Display_Params=`cvt 1920 1080 | grep Modeline | cut -d' ' -f2-18 | sed s/'"'//g`

xrandr --newmode $Display_Params
xrandr --addmode $Diplay_Name $Display_Spec
xrandr --output $Diplay_Name --mode $Display_Spec

In the second part of this post, I will write about the installation of CentOS on the physical server we have reserved for the evaluation stage. From the (largely) pleasant experience with the virtual machine, I expected this task to be entirely straightforward. I thought to be done in an hour, including configuration of user accounts as well as the sshd and vncd daemons for remote access. Well ... it took more than one day. Stay tuned. 😉