SysAdmin Blog

• Blog Actions
• RSS

When the installation of an VMware NSX controller fails and it's locked in the UI, you can just delete it from the API:



(from https://mastodon.infra.de/@galaxis/102376746230112269 (cache))

The standard setup for Mastodon is to use nginx as a reverse proxy. After one too many missing features I recently switched my installation over to using good old Apache.

There's an example Apache config (cache) in the unmaintained old documentation archive for Mastodon, and since I assume it's useless to try to update that, I'll quickly dump my current config here. There's no guarantee for correctness, but it currently seems to work for me. Note that this configuration does not do any caching for requests to static content retrieved through the reverse proxy.

The following Apache modules are used:

• proxy
• proxy_http
• http2
• proxy_http2
• proxy_wstunnel
• headers
• socache_shmcb
• ssl

General SSL configuration (personal preference, CipherSuite selection is probably going to age badly). TLS v1.3 is disabled since Ubuntu bionic ships an Apache version that's too old for that:

<IfModule mod_ssl.c>

        SSLCertificateFile     <path to combined public key / certificate chain file>
        SSLCertificateKeyFile  <path to private key>
        #   the referenced file can be the same as SSLCertificateFile
        #   when the CA certificates are directly appended to the server
        #   certificate for convinience.
        SSLCertificateChainFile <path to combined public key / certificate chain file>

        # SSLProtocol -all +TLSv1.2 +TLSv1.3
        SSLProtocol -all +TLSv1.2 +TLSv1.1
        SSLHonorCipherOrder on
        SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:EECDH+AESGCM:AES256+EECDH:AES128+EECDH
        SSLCompression off
        SSLSessionTickets off
        SSLSessionCache "shmcb:logs/session-cache(512000)"
        SSLStaplingResponderTimeout 5
        SSLStaplingReturnResponderErrors off
        SSLUseStapling on
        SSLStaplingCache "shmcb:logs/stapling-cache(150000)"

        # needs to be generated first, see https://weakdh.org/sysadmin.html
        SSLOpenSSLConfCmd DHParameters /etc/ssl/dhparam.pem

</IfModule>

Mastodon vhost configuration:

<VirtualHost *:443>
        ServerAdmin webmaster@example.com
        ServerName mastodon.example.com

        SSLEngine on

        Protocols h2 http/1.1

        # fetch static files directly from local file system (adapt to installation path)
        DocumentRoot /home/mastodon/live/public

        Header always set Strict-Transport-Security "max-age=31536000"

        <LocationMatch "^/(assets|avatars|emoji|headers|packs|sounds|system)">
                Header always set Cache-Control "public, max-age=31536000, immutable"
                Require all granted
        </LocationMatch>

        <Location "/">
                Require all granted
        </Location>

        ProxyPreserveHost On
        RequestHeader set X-Forwarded-Proto "https"
        ProxyAddHeaders On

        # these files / pathes don't get proxied and are retrieved from DocumentRoot
        ProxyPass /500.html !
        ProxyPass /sw.js !
        ProxyPass /robots.txt !
        ProxyPass /manifest.json !
        ProxyPass /browserconfig.xml !
        ProxyPass /mask-icon.svg !
        ProxyPassMatch ^(/.*\.(png|ico)$) !
        ProxyPassMatch ^/(assets|avatars|emoji|headers|packs|sounds|system) !
        # everything else is either going to the streaming API or the web workers
        ProxyPass /api/v1/streaming ws://localhost:4000
        ProxyPassReverse /api/v1/streaming ws://localhost:4000
        ProxyPass / http://localhost:3000/
        ProxyPassReverse / http://localhost:3000/

        ErrorDocument 500 /500.html
        ErrorDocument 501 /500.html
        ErrorDocument 502 /500.html
        ErrorDocument 503 /500.html
        ErrorDocument 504 /500.html

</VirtualHost>

The trailing / on the websocket ProxyPass directive is missing by design (it's there in the old example config): Some API requests seen in the wild will not match /api/v1/streaming/ and will get lost.

The iPXE open source boot firmware project provides an CD image that boots the iPXE binary using isolinux.

Over on the Fediverse, the topic of bootstraping a system from a floppy disk came up, and with the iPXE binary being a mere 330KB, there's really no reason why it shouldn't be possible to boot that from a floppy disk. And it actually does work, with a few simple steps (on a Debian-ish Linux):

• format floppy disk and create FAT filesystem

  fdformat /dev/fd0
  mkfs -t fat /dev/fd0

• get syslinux and install to floppy

  apt install syslinux syslinux-utils
  syslinux --install /dev/fd0

• get iPXE ISO

  curl -O http://boot.ipxe.org/ipxe.iso

• mount both iPXE ISO and floppy, copy over required files, rename isolinux.cfg to syslinux.cfg

  mkdir fd iso
  mount /dev/fd0 fd
  mount -o ro ipxe.iso iso
  cp iso/ipxe.krn fd/
  cp iso/boot.cat fd/
  cp iso/isolinux.cfg fd/syslinux.cfg
  umount fd
  umount iso
  rmdir fd iso
  

That's all! Take your floppy and boot a system

Once iPXE has been started, hit Ctrl-B to call the shell. If you have a DHCP server on your network and a web server with a bootable ISO image, it's just two iPXE commands:

dhcp
sanboot http://<webserver>/<filename>.iso

We've been playing with a SolidFire flash storage cluster for some time, and recently wanted to update the nodes to the current ElementOS 10.1 release.

Unfortunately, our FDVA management node installation was borked, so we decided to just roll a new one from the current VM appliance template - easy.
As it turns out though, the FDVA appliance only ships with the latest software release files, and the individual SolidFire nodes check back for a repository with their current version before starting the update, which consequently fails (it's all very Ubuntu-ish):

admin@SF-7323:~$ sudo sfinstall 192.168.10.21 -u admin -p password -l
2017-12-20 17:27:52: sfinstall Release Version: 10.1.0.83 Revision:  Build date: 2017-11-23 01:27
2017-12-20 17:27:52: Checking connectivity to MVIP 192.168.10.21
2017-12-20 17:27:52: Successfully connected to cluster MVIP
2017-12-20 17:27:53: PrintRepositoryPackages failed: SolidFireApiError server=[192.168.10.10] method=[AptUpdate], params=[{'quiet': 2}] - error name=[xCheckFailure], 
message=[cmdResult={ rc=255 stdout="W: Failed to fetch http://192.168.10.10/fluorine-updates/ubuntu/dists/precise/main/binary-amd64/Packages  404  Not Found
[..]
W: Failed to fetch http://192.168.10.10/fluorine-updates/security-ubuntu/dists/precise-security/universe/binary-amd64/Packages  404  Not Found

The SolidFire docs don't really mention what to do from there, so we tinkered around for some time and found this:

Any older version of the repository can be fetched using the update-fdva tool with the currently used ElementOS release version as command line (version number can be seen on the cluster web UI or when asking the cluster nodes for their mnode repository using sfinstall). In our case, the active version was 9.2.0.43 -

admin@SF-7323:~$ sudo update-fdva 9.2.0.43
Get: 1 http://localhost precise Release.gpg [490 B]
Get: 2 http://localhost precise-updates Release.gpg [490 B]
[..]

This will fetch the 9.2.0.43 version of the SolidFire repository, but will also downgrade to the matching (old) versions of solidfire-fdva-tools and solidfire-python-framework...

admin@SF-7323:~$ dpkg -l | grep fdva
ii  solidfire-fdva-tools-fluorine-patch2-9.2.0.43               9.2.0.43                          SolidFire FDVA Tools 9 [fluorine-patch2]

...so we immediately reinstalled the current versions, using update-fdva again, this time with the current release version number:

admin@SF-7323:~$ sudo update-fdva 10.1.0.83

With all that in place, we could just run the update routine using the usual sfinstall command.

After dist-upgrading a Debian system recently, I wondered which packages might have been left over from previous releases (the system in question has been through several dist-upgrades over its lifetime), even after running apt-get autoremove and deborphan. After dropping that question on Mastodon (cache), I got an answer pointing to apt-show-versions, which I didn't know of up to now.

This totally does what I've been looking for. From the man page:

NAME
       apt-show-versions - Lists available package versions with distribution

DESCRIPTION
       apt-show-versions parses the dpkg status file and the APT lists for the installed and available package
       versions and distribution and shows upgrade options within the specific distribution of the selected package.

       This is really useful if you have a mixed stable/testing environment and want to list all packages which are
       from testing and can be upgraded in testing.

Since I didn't have a package cache for apt-show-versions from the older release, all old packages are currently just shown with a No available version in archive comment. But since current packages are being tagged with the release, I can just exclude those with a simble grep:

# apt-show-versions | egrep -vc jessie
58

Another of those "just so I find my own post the next time I'm looking for this" things...

After the replacing the CPU fan on an old Intel D945GCLF2 Atom board, I "optimized" the BIOS settings by enabling automatic fan control (instead of having the fan at a fixed speed).

Currently, I have lots of messages like this in my kernel log, and a kworker thread using 100% CPU:

ACPI Error: Method parse/execution failed [\_SB_.PCI0.LPC_.SMBR] (Node ffff88007ec3f900), AE_AML_INFINITE_LOOP (20140424/psparse-536)
ACPI Error: Method parse/execution failed [\_SB_.PCI0.LPC_.INIT] (Node ffff88007ec3f928), AE_AML_INFINITE_LOOP (20140424/psparse-536)
ACPI Error: Method parse/execution failed [\_GPE._L00] (Node ffff88007ec35bd0), AE_AML_INFINITE_LOOP (20140424/psparse-536)
ACPI Exception: AE_AML_INFINITE_LOOP, while evaluating GPE method [_L00] (20140424/evgpe-580)

So, an hour or so of searching later, I finally hit this comment on the Novell bugzilla (cache) - and then I promptly remembered that I used to have known about this problem, and that it was exactly the reason why the fan was set to a fixed speed:

I've recently tried to reconfigure a mailinglist that I run on one of my systems to make less problems with recipients that use DMARC.

To that end, I wanted to implement the first option mentioned in the corresponding DMARC FAQ for mailinglist administrators (cache): Don't do any changes to the message body and headers potentially covered by a DKIM signature.

On the mailinglist configuration side, this means not adding a list tag to the message subject, and not changing the body with an additional header or footer. (Adding the usual RFC2369/RFC2919 list headers is no problem.)

After doing that it turned out that my setup, using sendmail, still changed the body on some messages, thanks to the automatic MIME autoconversion that sendmail does.

Getting rid of that actually required some digging into sendmail configuration and documentation:

Responsible for message delivery to local programs is the aptly named "prog" mailer. Unfortunately, as far as the m4 configuration statements are concerned, all the variables for this mailer are called "SHELL"... The default flags for the prog/shell mailer are hardcoded to "[eu9]", according to the README (though these defaults are changed in some of the OSTYPE definitions in the m4 macro collection):


The meaning of the individual flags is documented in the OP manual:



So it seems I want to get rid of the 9 in the LOCAL_SHELL_FLAGS, and replace it by a 8...

Adding the following two statements to my sendmail m4 configuration source does exactly that:

dnl # disable MIME-Autoconversion for prog mailer
MODIFY_MAILER_FLAGS(`SHELL', `-9')
MODIFY_MAILER_FLAGS(`SHELL', `+8')

On a headless Raspberry Pi running raspbian/jessie, the /var/log/messages file is filling up with entries like these:

 rsyslogd-2007: action 'action 17' suspended, next retry is [..date..] [ try http://www.rsyslog.com/e/2007 ] 

It seems this message is generated when rsyslogd isn't able to deliver syslog messages to one of the destinations in rsyslog.conf

In the case a raspbian, it's obviously the entry at the end of the config that tries to pipe messages to |/dev/xconsole - which doesn't exist on a system that doesn't run X11...

The messages disappear after commenting out or deleting the corresponding lines:

#daemon.*;mail.*;\
#       news.err;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       |/dev/xconsole

I really should file a bug report for this...

I'm pretty certain I used to know this - but for the next time I'm putting this into a search engine and don't find it in the Splunk docs:

One of our data sources writes structured data into our Splunk installation which contains variable names with dashes - in this particular case, access-time

It's no problem using such a variable in a lot of Splunk operations, but it fails in an eval, as it will be interpreted as a mathematical operation (access minus time).

There's two options to work around that:

1. the one mentioned in the Splunk documentation: Put the variable name in single quotes, i.e. | eval newtime='access-time' - constant
2. the other one is to simply rename the variable before working on it: | rename access-time AS accesstime | eval newtime=accesstime - constant

Mostly as a reminder to myself when I'm looking to solve this kind of problem the next time: Since the March 22, 2017 version of the FortiClient VPN Android app kept crashing on my mobile (still running the last Cyanogenmod 13 snapshot) as soon as I tried to switch away to the launcher, I wanted to downgrade the app.

Unfortunately, there's no copy on apkmirror.com or F-Droid, and I don't know about any other reasonably trustworthy sources. I also already had removed and reinstalled the app, so recovering the old version on the phone didn't seem an option either.

Fortunately, I take TWRP backups now and then, so I tried looking at one of those. For once, having unencrypted backups turned out real convenient: A TWRP data.ext4.win file is just a tar.gz, so I was able recover the app/com.fortinet.forticlient_vpn-1/base.apk file (using 7Zip on Windows), and copy that over to my phone. After uninstalling the current version of the FortiClient app, I just reinstalled the program with the CM file manager using the restored base.apk as a source. Done.