SysAdmin Blog

find obsolete packages on a Debian system

Alexander Bochmann Saturday 08 of July, 2017
After dist-upgrading a Debian system recently, I wondered which packages might have been left over from previous releases (the system in question has been through several dist-upgrades over its lifetime), even after running apt-get autoremove and deborphan. After dropping that question on Mastodon (cache), I got an answer pointing to apt-show-versions, which I didn't know of up to now.

This totally does what I've been looking for. From the man page:

       apt-show-versions - Lists available package versions with distribution

       apt-show-versions parses the dpkg status file and the APT lists for the installed and available package
       versions and distribution and shows upgrade options within the specific distribution of the selected package.

       This is really useful if you have a mixed stable/testing environment and want to list all packages which are
       from testing and can be upgraded in testing.

Since I didn't have a package cache for apt-show-versions from the older release, all old packages are currently just shown with a No available version in archive comment. But since current packages are being tagged with the release, I can just exclude those with a simble grep:

# apt-show-versions | egrep -vc jessie


Alexander Bochmann Wednesday 10 of May, 2017
Another of those "just so I find my own post the next time I'm looking for this" things...

After the replacing the CPU fan on an old Intel D945GCLF2 Atom board, I "optimized" the BIOS settings by enabling automatic fan control (instead of having the fan at a fixed speed).

Currently, I have lots of messages like this in my kernel log, and a kworker thread using 100% CPU:

ACPI Error: Method parse/execution failed [\_SB_.PCI0.LPC_.SMBR] (Node ffff88007ec3f900), AE_AML_INFINITE_LOOP (20140424/psparse-536)
ACPI Error: Method parse/execution failed [\_SB_.PCI0.LPC_.INIT] (Node ffff88007ec3f928), AE_AML_INFINITE_LOOP (20140424/psparse-536)
ACPI Error: Method parse/execution failed [\_GPE._L00] (Node ffff88007ec35bd0), AE_AML_INFINITE_LOOP (20140424/psparse-536)
ACPI Exception: AE_AML_INFINITE_LOOP, while evaluating GPE method [_L00] (20140424/evgpe-580)

So, an hour or so of searching later, I finally hit this comment on the Novell bugzilla (cache) - and then I promptly remembered that I used to have known about this problem, and that it was exactly the reason why the fan was set to a fixed speed:

In BIOS I have DISABLED auto fan speed. It is now set at 90%. It seems to have fixed it

sendmail MIME conversion vs. DMARC+DKIM

Alexander Bochmann Saturday 15 of April, 2017
I've recently tried to reconfigure a mailinglist that I run on one of my systems to make less problems with recipients that use DMARC.

To that end, I wanted to implement the first option mentioned in the corresponding DMARC FAQ for mailinglist administrators (cache): Don't do any changes to the message body and headers potentially covered by a DKIM signature.

On the mailinglist configuration side, this means not adding a list tag to the message subject, and not changing the body with an additional header or footer. (Adding the usual RFC2369/RFC2919 list headers is no problem.)

After doing that it turned out that my setup, using sendmail, still changed the body on some messages, thanks to the automatic MIME autoconversion that sendmail does.

Getting rid of that actually required some digging into sendmail configuration and documentation:

Responsible for message delivery to local programs is the aptly named "prog" mailer. Unfortunately, as far as the m4 configuration statements are concerned, all the variables for this mailer are called "SHELL"... The default flags for the prog/shell mailer are hardcoded to "[eu9]", according to the README (though these defaults are changed in some of the OSTYPE definitions in the m4 macro collection):

sendmail cf.README wrote:
LOCAL_SHELL_FLAGS [eu9] The flags used by the shell mailer. The flags lsDFM are always included.

The meaning of the individual flags is documented in the OP manual:

sendmail OP manual wrote:
   Strip all output to seven bits. This is the default if the L flag is set. Note that clearing this option is not sufficient to get full eight bit data passed through sendmail. If the 7 option is set, this is essentially always set, since the eighth bit was stripped on input. Note that this option will only impact messages that didn't have 8->7 bit MIME conversions performed.
   If set, it is acceptable to send eight bit data to this mailer; the usual attempt to do 8->7 bit MIME conversions will be bypassed.
   If set, do limited 7->8 bit MIME conversions. These conversions are limited to text/plain data.

So it seems I want to get rid of the 9 in the LOCAL_SHELL_FLAGS, and replace it by a 8...

Adding the following two statements to my sendmail m4 configuration source does exactly that:

dnl # disable MIME-Autoconversion for prog mailer

raspbian jessie - rsyslogd-2007: action 'action 17' suspended, next retry ...

Alexander Bochmann Monday 10 of April, 2017
On a headless Raspberry Pi running raspbian/jessie, the /var/log/messages file is filling up with entries like these:

 rsyslogd-2007: action 'action 17' suspended, next retry is [..date..] [ try http://www.rsyslog.com/e/2007 ] 

It seems this message is generated when rsyslogd isn't able to deliver syslog messages to one of the destinations in rsyslog.conf

In the case a raspbian, it's obviously the entry at the end of the config that tries to pipe messages to |/dev/xconsole - which doesn't exist on a system that doesn't run X11...

The messages disappear after commenting out or deleting the corresponding lines:

#       news.err;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       |/dev/xconsole

I really should file a bug report for this...

Splunk eval vs. variable names with dashes

Alexander Bochmann Wednesday 05 of April, 2017
I'm pretty certain I used to know this - but for the next time I'm putting this into a search engine and don't find it in the Splunk docs:

One of our data sources writes structured data into our Splunk installation which contains variable names with dashes - in this particular case, access-time

It's no problem using such a variable in a lot of Splunk operations, but it fails in an eval, as it will be interpreted as a mathematical operation (access minus time).

There's two options to work around that:

  1. the one mentioned in the Splunk documentation: Put the variable name in single quotes, i.e. | eval newtime='access-time' - constant
  2. the other one is to simply rename the variable before working on it: | rename access-time AS accesstime | eval newtime=accesstime - constant

downgrading Android apps using data from TWRP backups

Alexander Bochmann Tuesday 28 of March, 2017
Mostly as a reminder to myself when I'm looking to solve this kind of problem the next time: Since the March 22, 2017 version of the FortiClient VPN Android app kept crashing on my mobile (still running the last Cyanogenmod 13 snapshot) as soon as I tried to switch away to the launcher, I wanted to downgrade the app.

Unfortunately, there's no copy on apkmirror.com or F-Droid, and I don't know about any other reasonably trustworthy sources. I also already had removed and reinstalled the app, so recovering the old version on the phone didn't seem an option either.

Fortunately, I take TWRP backups now and then, so I tried looking at one of those. For once, having unencrypted backups turned out real convenient: A TWRP data.ext4.win file is just a tar.gz, so I was able recover the app/com.fortinet.forticlient_vpn-1/base.apk file (using 7Zip on Windows), and copy that over to my phone. After uninstalling the current version of the FortiClient app, I just reinstalled the program with the CM file manager using the restored base.apk as a source. Done.

Cisco ASA logging: Disable hiding of usernames in failed admin logins

Alexander Bochmann Thursday 23 of March, 2017
Cisco ASA firewalls don't log, by default, the username used in a failed administrator login. Instead, the login is masked out using "*" characters:

%ASA-6-113005: AAA user authentication Rejected : reason = AAA failure : server = : user = ***** : user IP =

The rationale is that users sometimes enter their password instead of the username, and the password will then end up in logs. As we're using two-factor authentication for admin logins, that doesn't apply to us.

That behaviour was actually tracked as a bug in Cisco's bug database (cache), and while the article mentions that a command was introduced to change this behaviour, the command itself isn't mentioned.

After some fiddling on the ASA command line I found this statement:

no logging hide username

The corresponding button in the ASDM GUI is in Device Management -> Logging -> Syslog Setup: "Hide username if its validity cannot be determined"

so I didn't notice that my OpenBSD vserver had broken IPv6 for quite some time...

Alexander Bochmann Sunday 19 of February, 2017
...until I had a look at the DNS server log, which showed errors contacting other servers via IPv6.

The hoster I'm using has a somewhat strange IPv6 setup where you get a /64 for your system, but the default gateway is just fe80::1 - when I originally set up the system, I put that into /etc/mygate whithout thinking much about it.

This initially was ok for quite some time, but it seems the default route vanished at some point. (In retrospect I don't quite understand why the setup ever worked at all, as the lo0 lookback interface has fe80::1 auto-assigned too...)

Then I remembered that fe80:: carries interface tags, since it exists on any IPv6-enabled interface, and the OS needs some way to decide which fe80:: it has to deal with right now.

Edited /etc/mygate accordingly, and things are back to normal (vio is OpenBSD's VirtIO network device driver, so my virtual ethernet device is vio0):


Linux ATA bus errors with ASMedia ASM1062 PCIe card

Alexander Bochmann Saturday 22 of October, 2016
I recently added a cheap ASM1062 2-port SATA card to my Linux box at home, since it's Asus C8HM70-I board only has two SATA ports, and I wanted to use an additional small SSD as boot device.

With my disks hooked up to the new card, I started to get SATA errors when there was moderate write load:

kernel log
ata5.00: exception Emask 0x10 SAct 0x7c000000 SErr 0x400000 action 0x6 frozen
ata5.00: irq_stat 0x08000000, interface fatal error
ata5: SError: { Handshk }
ata5.00: failed command: WRITE FPDMA QUEUED
ata5.00: cmd 61/00:d0:00:2b:6f/0a:00:ac:00:00/40 tag 26 ncq 1310720 out
         res 40/00:f4:00:53:6f/00:00:ac:00:00/40 Emask 0x10 (ATA bus error)
ata5.00: status: { DRDY }
ata5.00: failed command: WRITE FPDMA QUEUED
ata5.00: cmd 61/00:d8:00:35:6f/0a:00:ac:00:00/40 tag 27 ncq 1310720 out
         res 40/00:f4:00:53:6f/00:00:ac:00:00/40 Emask 0x10 (ATA bus error)
ata5: hard resetting link
ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata5.00: configured for UDMA/133
ata5: EH complete

I'm not yet ready to blame the card itself, since I remembered I recycled a pair of rather old SATA cables to connect the drives, and the card supports SATA 6G... The mainboard itself has just one SATA 6G connector, and with that I used different cables that clip into the port, but the clip mechanic doesn't work with the connectors on the ASMedia card.

For now, I turned the SATA link speed down to 3G by adding an libata.force parameter to the kernel command line:


(5 and 6 are corresponding to ata5 and ata6 from the libata kernel messages.)

This seems to work as a stopgap measure - the bus errors haven't reappeared since.


ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)

With libata.force:

ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 320)

syslog-ng and RcvbufErrors on Linux

Alexander Bochmann Tuesday 10 of May, 2016
We're running a syslog-ng installation to collect syslog data from quite a lot of systems (and then selectively feed them into our Splunk installation). Almost all of these send syslog via UDP.

Recently, when adding a couple more machines, I noticed that the syslog server is dropping UDP datagrams:

udp RcvbufErrors
# netstat -su | grep -A6 "^Udp:"
    518026364 packets received
    36078 packets to unknown port received.
    23164168 packet receive errors
    1248583 packets sent
    RcvbufErrors: 23164167


This is mentioned in the syslog-ng OSE docs, but it seems no one here ever got to that section, including myself.

So, in that context I learned about the so-rcvbuf() parameter to the udp() source in syslog-ng, and the Linux kernel net.core.rmem_max sysctl...

Kernel configuration
# sysctl -w net.core.rmem_max=16777216

(add the same parameter to /etc/sysctl.conf)

source s_net {  
                udp(ip( port(514) so-rcvbuf(8388608)); 

(There's no reason why so-rcvbuf() couldn't be the same as rmem_max, and neither needs to be a multiple of 1024 - both just bad habits of mine...)

Don't increase net.core.rmem_default, as that would make the Linux kernel use a bigger buffer for every UDP socket being created on the system.

The RcvbufErrors counter hasn't been increasing since that change, but I'll add monitoring for that, so drops won't go unnoticed in the future.