adangel.org

PDF to image conversion workaround

Sun, 22 Feb 2026 00:00:00 +0100

Motivation

I’m using ReadEra to read magazines in PDF format. It usually works fine. But yesterday, one PDF didn’t show up correctly. It looks like some font displaying problem. The text is there, but not visible. You could select it, but nothing is displayed. Such text is a bit hard to read…

cover of the magazin before applying the workaround

The workaround

Since the PDF looks good in other applications, I don’t think it is a bad PDF. The PDF has lots of fonts embedded, but so did the earlier issues of the magazines in the past, which were displayed correctly.

The simple try to print the PDF with another application again to PDF didn’t fix the problem: The text was still missing.

So there is another workaround needed: We just convert the whole PDF into a series of images and then create a new PDF from these images. That way, there should be no problems in displaying the PDF since it consists only of images. The downside however is, that the PDF is now entirely made up of images and no text. But as a human, I’m still good at converting text on images into text myself.

Step 1

You’ll need ghostscript, which is often already installed on Linux systems.

mkdir pdfimages
gs -sDEVICE=png16m -sOutputFile=pdfimages/page-%03d.jpg -r300x300 -dNOPAUSE -dBATCH -f input.pdf

This will convert input.pdf into a series of jpeg images into the folder pdfimages. You can simply verify, that the pages are displayed correctly. In case you see the same missing text, then this workaround won’t help you - then Ghostscript has the same issue as ReadEra.

Step 2

Now we’ll combine all the pages back into a new handy PDF using img2pdf.

img2pdf pdfimages/*.jpg -o input_images.pdf

Now we have again a single PDF and since it only consists of images, there won’t be a display problem.

Result

The result looks like this:

cover of the magazin after applying the workaround

The file size of the new PDF is much bigger now. You might experiment with more options when generating the jpegs or the PDF (e.g. resolution, compression). But as a workaround, this is good enough.

References

glibc trickery - part 2

Tue, 16 Dec 2025 00:00:00 +0100

It turned out, that the last post Using glibc trickery to run nodejs isn’t sufficient in one special case: If you are using yarn…

Yarn

Yarn is another package manager for JavaScript projects. It can replace npm, which is bundled with node nowadays. Yarn has a specific command: node. This allows you to run node again - but not some node. Yarn tries to make sure, that the exact same node version is used as the one that is used to run Yarn itself.

So, one could think, that yarn achieves this, by looking at “argv0” - the zero argument of the argument list of the process. We can actually run a small node script to check, which value we get.

Given the same setup from the previous blog post, we can run this command now:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    docker.io/opensuse/leap:42 \
    node -e 'console.log(process.argv0);'
/root/node-test/node-v24.6.0-linux-x64/bin/node

Hm, ok. This looks ok. However, if we would try to start a new process, we would run in the same GLIBC version issues again. It would be better, if this would have been the wrapper script already.

Changing argv0

process.argv0 is documented here: https://nodejs.org/docs/latest/api/process.html#processargv0. There is also process.argv, which contains the whole argument array, that has been passed to the nodejs process. Let’s see, what we get:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    docker.io/opensuse/leap:42 \
    node -e 'console.log(`argv0=${process.argv0}\nargv=${process.argv}\n`);'
argv0=/root/node-test/node-v24.6.0-linux-x64/bin/node
argv=/root/node-test/glibc-2.38/lib64/ld-linux-x86-64.so.2

Let’s try to change it to the wrapper script. The wrapper script is located at /root/node-test/node and is using the ld.loader. This has an argument called --argv0, which let’s you overwrite exactly this piece of information. This feature has been implemented with glibc 2.33, see bug 16124 and man ld.so.8.

The new wrapper script looks like this:

#!/bin/bash
NODE_HOME=/root/node-test/node-v24.6.0-linux-x64
GLIBC_HOME=/root/node-test/glibc-2.38

exec ${GLIBC_HOME}/lib64/ld-linux-x86-64.so.2 \
  --argv0 /root/node-test/node \
  --library-path ${GLIBC_HOME}/lib64 \
  ${NODE_HOME}/bin/node "$@"

Running node again, we get now this:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    docker.io/opensuse/leap:42 \
    node -e 'console.log(`argv0=${process.argv0}\nargv=${process.argv}\n`);'
argv0=/root/node-test/node
argv=/root/node-test/glibc-2.38/lib64/ld-linux-x86-64.so.2

Much better. “argv0” looks correct. Hopefully, argv doesn’t matter. But wait, where are node’s actual arguments like “-e” and “console.log(….)”? It turns out, these are in another variable: process.execArgv. There is also process.execPath which we’ll need later. Let’s have a look at all of these variables now:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    docker.io/opensuse/leap:42 \
    node -e 'console.log(`argv0=${process.argv0}\nargv=${process.argv}\nexecPath=${process.execPath}\nexecArgv=${process.execArgv}\n`);'
argv0=/root/node-test/node
argv=/root/node-test/glibc-2.38/lib64/ld-linux-x86-64.so.2
execPath=/root/node-test/glibc-2.38/lib64/ld-linux-x86-64.so.2
execArgv=-e,console.log(`argv0=${process.argv0}\nargv=${process.argv}\nexecPath=${process.execPath}\nexecArgv=${process.execArgv}\n`);

Ok, so far so good.

Executing yarn

Let’s see, whether we can run “yarn node –version”. First of all, we need to get yarn.

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    docker.io/opensuse/leap:42 \
    npm install -g corepack
$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    docker.io/opensuse/leap:42 \
    yarn init -2

First command works, second failed with executable file `yarn` not found in $PATH. Usually, it should install it into the same directory as node/corepack, which would have been /root/node-test/node-v24.6.0-linux-x64/bin, but these symlinks actually ended up in /root/node-test/glibc-2.38/bin/… Let’s ignore this for now and install yarn manually by downloading the single js file:

$ wget -O yarn-4.10.3.js https://repo.yarnpkg.com/4.10.3/packages/yarnpkg-cli/bin/yarn.js
$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    -w /root/node-test \
    docker.io/opensuse/leap:42 \
    node yarn-4.10.3.js --version
4.10.3

This works - so we can at least start “yarn” now.

Note: I’ve added -w /root/node-test to set the current working directory, so that “yarn-4.10.3.js” can be referenced relatively.

What is the result of yarn node --version?

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    -w /root/node-test \
    docker.io/opensuse/leap:42 \
    node yarn-4.10.3.js node --version
Usage Error: No project found in /root/node-test

Ok, we first need to initialize a project via yarn init and yarn install:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    -w /root/node-test \
    docker.io/opensuse/leap:42 \
    node yarn-4.10.3.js init
...
$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    -w /root/node-test \
    docker.io/opensuse/leap:42 \
    node yarn-4.10.3.js install
...
$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    -w /root/node-test \
    docker.io/opensuse/leap:42 \
    node yarn-4.10.3.js node --version
ld.so (GNU libc) stable release version 2.38.
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

That’s worrying: Instead of node’s version, we see some output from “ld.so”. This means, yarn wasn’t able to start a new node process.

How yarn determines the node used to run itself

This tooks some digging into yarn’s source code, which is available on GitHub under yarnpkg/berry.

When you execute “yarn node”, yarn internally actually executes “yarn exec node” (see node.ts). This is implemented in exec.ts and calls the method executePackageShellcode in scriptUtils.ts which sets up an environment, where executing just “node” will execute the correct node process. This is done through wrapper scripts by method makeScriptEnv and several calls to makePathWrapper. The argument “argv0” that is used to execute, is passed from process.execPath.

So, yarn actually is going to execute process.execPath in order to start a new node process… But we have seen above, that this value is the loader… and not our node wrapper script.

This explains, why “yarn node” doesn’t work:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    -w /root/node-test \
    docker.io/opensuse/leap:42 \
    node yarn-4.10.3.js node
/root/node-test/glibc-2.38/lib64/ld-linux-x86-64.so.2: missing program name
Try '/root/node-test/glibc-2.38/lib64/ld-linux-x86-64.so.2 --help' for more information.

So, the question now became: “How to change process.execPath?”

How does node determine process.execPath?

Let’s have a look at the implementation of node. In node_process_object.cc there is some code for handling execPath. It reads the value from “env”, which is of type Environment, which is implemented in env.cc. This contains a lot of “exec_path” references. In the end, it seems to be initialized in the constructor by calling the method Environment::GetExecPath: This is calling the platform dependent uv_exepath, which is implemented for linux in procfs-exepath.c, which just reads the symlink /proc/self/exe. By the way, if uv_exepath would return an error, then Environment::GetExecPath would fall back to argv0…

Ok, can we somehow change, what the symlink in /proc/self/exe it pointing at?

Changing /proc/self/exe

Apparently this is very difficult, for good reasons. See e.g. Hiding in plain sight - Abusing the dynamic linker.

So, there is the LD_PRELOAD mechanism, also known as the ld preload trick. With glibc you can load a shared library early on (“preload”) and this can intercept other library methods. Since node is using readlink, we can intercept this call and return whatever path we want for /proc/self/exe… that should do the trick.

So, how does this work? We need a shared library. It is based on How could I intercept linux syscalls?. Let’s call it change_exe.c:

// Based on https://stackoverflow.com/questions/69859/how-could-i-intercept-linux-sys-calls
#define _GNU_SOURCE
#include <dlfcn.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>

extern int errorno;

ssize_t __thread (*_readlink)(const char *restrict path, char * buf, size_t bufsiz) = NULL;

ssize_t readlink(const char *restrict path, char * buf, size_t bufsiz)
{
    if (NULL == _readlink) {
        _readlink = (ssize_t (*)(const char *restrict path, char * buf, size_t bufsiz)) dlsym(RTLD_NEXT, "readlink");
    }
    if (strcmp(path, "/proc/self/exe") == 0) {
        const char * env_path = getenv("CHANGE_EXE_PATH");
        if (env_path) {
            const size_t env_path_len = strlen(env_path);
            strncpy(buf, env_path, bufsiz);
            if (bufsiz > env_path_len + 1) {
                return env_path_len;
            }
            return bufsiz;
        }
    }
    return _readlink(path, buf, bufsiz);
}

// gcc -fPIC -shared -ldl -o change_exe.so change_exe.c
// Test 1: LD_PRELOAD="./change_exe.so" readlink /proc/self/exe
// Test 2: CHANGE_EXE_PATH="/foo" LD_PRELOAD="./change_exe.so" readlink /proc/self/exe

This library will check whether the environment variable CHANGE_EXE_PATH exists and will use this one when readlink is called for /proc/self/exe. This way, we can configure the actual path.

Final wrapper script

We can now change our wrapper script to include not only “argv0” but also a preloaded library:

#!/bin/bash
NODE_HOME=/root/node-test/node-v24.6.0-linux-x64
GLIBC_HOME=/root/node-test/glibc-2.38
export CHANGE_EXE_PATH=/root/node-test/node

exec ${GLIBC_HOME}/lib64/ld-linux-x86-64.so.2 \
  --argv0 ${CHANGE_EXE_PATH} \
  --preload /root/node-test/change_exe.so \
  --library-path ${GLIBC_HOME}/lib64 \
  ${NODE_HOME}/bin/node "$@"

Does it work?

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    -w /root/node-test \
    docker.io/opensuse/leap:42 \
    node yarn-4.10.3.js node --version
v24.6.0

Yes! The whole chain node -> yarn -> node is working now. Let’s see, what the values process.execPath and process.argv and so on are now:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    docker.io/opensuse/leap:42 \
    node -e 'console.log(`argv0=${process.argv0}\nargv=${process.argv}\nexecPath=${process.execPath}\nexecArgv=${process.execArgv}\n`);'
argv0=/root/node-test/node
argv=/root/node-test/node
execPath=/root/node-test/node
execArgv=-e,console.log(`argv0=${process.argv0}\nargv=${process.argv}\nexecPath=${process.execPath}\nexecArgv=${process.execArgv}\n`);

Summary

While researching this topic, I remembered proot which I occasionally use for backups (see Backups with restic and proot). proot as well intercepts system calls and returns “adjusted” values - it especially intercepts all calls that deal with path names. But do they do anything about /proc/self/exe? Let’s see:

$ ./proot node-v24.6.0-linux-x64/bin/node \
    -e 'console.log(`argv0=${process.argv0}\nargv=${process.argv}\nexecPath=${process.execPath}\nexecArgv=${process.execArgv}\n`);'
argv0=node-v24.6.0-linux-x64/bin/node
argv=/root/node-test/node-v24.6.0-linux-x64/bin/node
execPath=/root/node-test/node-v24.6.0-linux-x64/bin/node
execArgv=-e,console.log(`argv0=${process.argv0}\nargv=${process.argv}\nexecPath=${process.execPath}\nexecArgv=${process.execArgv}\n`);

So, they obviously thought about this… However, proot works a bit different: While “change_exe.c” only intercepts a glibc library method “readlink”, which itself is wrapper function around the actual syscall, proot uses ptrace to intercept the actual syscalls. And it seems to follow processes: if you run a shell with proot and start another process from there, this child process also have its syscall intercepted.

Also note, that all this with wrapper scripts is only necessary, if you a) cannot upgrade the system (hey, why not using a modern operating system?) and b) you cannot change the node binaries.

Because if you can change the node binaries, you can use patchelf to set the loader and library path directly in the node binary. Then you don’t need to call node explicitly via the loader anymore - it will just work. This is explained in Multiple glibc libraries on a single host.

References

Here are some pages in used during research in no particular order:

Bicycle Tour 2025

Wed, 15 Oct 2025 00:00:00 +0200

This year’s tour went from From Munich via Claudia to Engadin. First leg goes from Munich to river Lech, then following the Lech until Füssen/Reutte and switching to river Loisach. After the Fern Pass the last river is the Inn.

It’s a bit similar to the bicycle tour of 2021 but with a different starting direction. Hoped to see Engadin with more sun, but the weather went worse unfortunately (even with rain warnings in the region).

You can see my tracks on the interactive map I created:

Bicycle Tour 2025: From Munich via Claudia to Engadin

Bicycle Tour 2025: From Munich via Claudia to Engadin

I could now test my travel eSIM in Switzerland, as Switzerland is not part of the EU and therefore most mobile carries only offer expensive roaming prices. I took Yesim and after activating the eSIM on the mobile phone (and also online in Yesim’s account), it worked quite good. Interestingly, the public IP you get, is from Poland…

You can also see, that you won’t get a IPv6 address, which means, that IPv6 only services wouldn’t be accessible.

The fact, that the IP traffic is routed through Poland seems to be the normal case: Yesim is not acting as a local Swiss mobile provider, but as some foreign mobile provider, which also roams in Switzerland. This has the advantage, that it can roam through any available network with the best signal and is not limited to one provider. It just happens to be the case, that Yesim has chosen a Polish provider. Other eSIM providers like Holafly route their traffic through China.

See this news article Travel eSIMs secretly route traffic over Chinese and undisclosed networks: study and the corresponding study eSIMplicity or eSIMplification? Privacy and Security Risks in the eSIM Ecosystem and the reddit post Researchers discover eSIMs routing traffic through China.

Note, that this is only the public IP address, that is visible - how the traffic is actually internally routed, is not visible. But your traffic appears to the visited sites with your public IP. And if there is some geolocated restriction implemented on the sites, you might be surprised that these travel eSIMs act as if they are a VPN. So, depending on what you want to use, you might also need an actual VPN provider to finally appear in the country you are physically located… amazing. Something like NordVPN.

The fact, that the traffic is routed through maybe far distant countries definitely affects the latency of your connection - and most like also the throughput. For me, it was ok, I didn’t need much internet. If I had used more, I probably would have noticed Polish ads and I think, I wouldn’t have been able to by train tickets online for sbb (Schweizerische Bundesbahnen). But luckily, they still have ticket machines.

Using glibc trickery to run nodejs

Sat, 16 Aug 2025 00:00:00 +0200

Let’s assume, you need to run a recent nodejs binary on a very old linux system. More specifically, your old linux system is a SUSE Linux Enterprise. The old version 12 is still supported, there is a service pack 5 available, even still this year. Version 12 was released more than 10 years ago…

What happens, if you try to run a recent, more or less statically linked, binary on such an old system? You might experience the following error:

node-v24.6.0-linux-x64/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by node-v24.6.0-linux-x64/bin/node)
node-v24.6.0-linux-x64/bin/node: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by node-v24.6.0-linux-x64/bin/node)
node-v24.6.0-linux-x64/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by node-v24.6.0-linux-x64/bin/node)
node-v24.6.0-linux-x64/bin/node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by node-v24.6.0-linux-x64/bin/node)

SUSE Linux Enterprise

This distribution is a commercial one, but it is based on a free distribution by SUSE: openSUSE leap. So, you can use openSUSE leap in the matching version to test and reproduce this yourself.

According to the German wikipedia page SUSE Linux Enterprise Server SLES 12 is based on openSUSE Leap 42. More specifically, SLES 12 SP3 is based on openSUSE Leap 42.3. For the most recent service pack SLES 12 SP5, there doesn’t seem to be an openSUSE Leap equivalent version, but for our tests, 42.3 will do fine. Of course, you shouldn’t use 42.3 in production, as this won’t have any security fixes.

The next generation of SLES (SUSE Linux Enterprise Server) is SLES 15, which is based on openSUSE Leap 15. It’s convenient, that the version numbers now match. The latest version is SLES 15 SP 6 or openSUSE Leap 15.6.

The openSUSE Leap distribution is available on docker.io: https://hub.docker.com/r/opensuse/leap. We can simply start a docker image with the specific version:

podman run -it --rm --pull newer -v $(pwd):/root/node-test docker.io/opensuse/leap:42 bash

Or the newer version:

podman run -it --rm --pull newer -v $(pwd):/root/node-test docker.io/opensuse/leap:15 bash 

Note: It bind-mounts the current directory into /root/node-test, so that we can download nodejs distributions there and run it later inside the container. These images are very basic and e.g. don’t provide many packages, e.g. no “curl” or “wget”.

How to reproduce

Let’s download the current version of nodejs from https://nodejs.org/en/download/current as a standalone binary and extract it:

$ mkdir node-test
$ cd node-test
$ curl -O https://nodejs.org/dist/v24.6.0/node-v24.6.0-linux-x64.tar.xz
$ tar xfJv node-v24.6.0-linux-x64.tar.xz
$ node-v24.6.0-linux-x64/bin/node --version
v24.6.0

The last command runs nodejs and prints the version, you should see something like “v24.6.0”.

Now, let’s run this inside Leap 42:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test docker.io/opensuse/leap:42 /root/node-test/node-v24.6.0-linux-x64/bin/node --version
/root/node-test/node-v24.6.0-linux-x64/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /root/node-test/node-v24.6.0-linux-x64/bin/node)
/root/node-test/node-v24.6.0-linux-x64/bin/node: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /root/node-test/node-v24.6.0-linux-x64/bin/node)
/root/node-test/node-v24.6.0-linux-x64/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /root/node-test/node-v24.6.0-linux-x64/bin/node)
/root/node-test/node-v24.6.0-linux-x64/bin/node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /root/node-test/node-v24.6.0-linux-x64/bin/node)

And voila, you get the errors about “version GLIBC… not found” instead of the version number printed.

However, it works inside Leap 15 (and therefore should work in SLES 15):

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test docker.io/opensuse/leap:15 /root/node-test/node-v24.6.0-linux-x64/bin/node --version
v24.6.0

Libc versions of SUSE Linux Enterprise

So, whats the difference? The “libc” version seems to be too old. Which version the system is running, you can see by executing ldd --version:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test docker.io/opensuse/leap:42 ldd --version
ldd (GNU libc) 2.22
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test docker.io/opensuse/leap:15 ldd --version
ldd (GNU libc) 2.38
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

So, openSUSE Leap 42 (SLES 12) is built with and running with GNU libc 2.22. The newer openSUSE Leap 15 (SLES 15) is instead built with and running with GNU libc 2.38. And nodejs seems to require at least version 2.28.

glibc and the loader

The GNU C library provides the basic functionality for all programs running on Linux. This includes the kernel interface via syscalls. Even a simple hello world program uses it:

#include <stdio.h>

int main() {
  printf("Hello world!\n");
  return 0;
}

If you compile it, it is linked automatically against libc:

$ gcc -o hello hello.c
$ ./hello
Hello world!
$ ldd hello
	linux-vdso.so.1 (0x00007f53b7d57000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f53b7b2d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f53b7d59000)

If you statically link it, the you won’t see it anymore:

$ gcc -static -o hello hello.c
$ ./hello 
Hello world!
$ ldd hello
	not a dynamic executable

All the used methods are now embedded into the binary instead of dynamically linked in from libc.so.6. This makes the binary bigger, but you don’t need to have the correct libc version available.

When running a program under Linux, there are some environment variables, with which you can control from where dynamic libraries are loaded. These are e.g. LD_LIBRARY_PATH. With that, you can kind of “override” which dynamic/shared libraries are actually used. Usually, the shared libraries from /lib or /lib64 are used, but you might have a special program with special needs. This is documented in the man page for ld.so.

And there are other interesting options: E.g. LD_PRELOAD, which allows you to basically override specific methods without changing the original program. So you can inject or wrap specific system calls to e.g. figure out where some resource leaks coming from.

The man page ld.so documents the loader, which is the part, that loads a binary executable at the first place and the resolves the symbols finding and loading the shared objects. And you can call it explicitly via /lib64/ld-linux-x86-64.so.2 ...:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test docker.io/opensuse/leap:15 /lib64/ld-linux-x86-64.so.2 /root/node-test/node-v24.6.0-linux-x64/bin/node --version
v24.6.0

We’ll use this technique later on to create a wrapper script for nodejs…

Updating glibc

When a distribution updates to a new glibc version, this usually means that every package needs to be rebuilt, which is a huge effort and is risky. That’s why such updates are usually done in major versions (e.g. SLES 12 vs. SLES 15).

But maybe we can use the glibc version from SLES 15 (Leap 15) back in SLES 12 (Leap 42) and force nodejs to use it via the environment variable LD_LIBRARY_PATH. Let’s try.

First step is, to find the package repository of Leap 15, so that we can download the glibc rpm package from there: The repositories urls are listed on the page Package repositories and the URL for Leap 15.6 is: https://download.opensuse.org/distribution/leap/15.6/repo/oss/.

We actually need glibc-2.38-150600.12.1.x86_64.rpm. This saves us from compiling it ourselves. Hopefully, this new version will somehow run on the old Leap 42 environment.

$ curl -LO https://download.opensuse.org/distribution/leap/15.6/repo/oss/x86_64/glibc-2.38-150600.12.1.x86_64.rpm
$ rpm2cpio glibc-2.38-150600.12.1.x86_64.rpm
$ cpio -div -D glibc-2.38 < glibc-2.38-150600.12.1.x86_64.rpm.tar

Now we have it extracted to the local directory ./glibc-2.38/lib64/libc.so.6 (and many more files).

Using LD_LIBRARY_PATH like so, doesn’t work:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e LD_LIBRARY_PATH=/root/node-test/glibc-2.38/lib64 \
    docker.io/opensuse/leap:42 /root/node-test/node-v24.6.0-linux-x64/bin/node --version
/root/node-test/node-v24.6.0-linux-x64/bin/node: relocation error: /root/node-test/glibc-2.38/lib64/libc.so.6: symbol _dl_signal_error, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

Let’s try with LD_PRELOAD, as it is described on A Solution to Version GLIBC_2.XX Not Found:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e LD_PRELOAD=/root/node-test/glibc-2.38/lib64/libc.so.6 \
    docker.io/opensuse/leap:42 /root/node-test/node-v24.6.0-linux-x64/bin/node --version
/root/node-test/node-v24.6.0-linux-x64/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /root/node-test/node-v24.6.0-linux-x64/bin/node)

Almost, let’s try the combination:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e LD_PRELOAD=/root/node-test/glibc-2.38/lib64/libc.so.6 \
    -e LD_LIBRARY_PATH=/root/node-test/glibc-2.38/lib64 \
    docker.io/opensuse/leap:42 /root/node-test/node-v24.6.0-linux-x64/bin/node --version
/root/node-test/node-v24.6.0-linux-x64/bin/node: relocation error: /root/node-test/glibc-2.38/lib64/libc.so.6: symbol _dl_signal_error, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

No. But maybe if we preload both libc and libm?

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e LD_PRELOAD=/root/node-test/glibc-2.38/lib64/libc.so.6:/root/node-test/glibc-2.38/lib64/libm.so.6 \
    docker.io/opensuse/leap:42 /root/node-test/node-v24.6.0-linux-x64/bin/node --version
/root/node-test/node-v24.6.0-linux-x64/bin/node: relocation error: /root/node-test/glibc-2.38/lib64/libc.so.6: symbol _dl_signal_error, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

Maybe libc and libm and additionally LD_LIBRARY_PATH?

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e LD_PRELOAD=/root/node-test/glibc-2.38/lib64/libc.so.6:/root/node-test/glibc-2.38/lib64/libm.so.6 \
    -e LD_LIBRARY_PATH=/root/node-test/glibc-2.38/lib64 \
    docker.io/opensuse/leap:42 /root/node-test/node-v24.6.0-linux-x64/bin/node --version
/root/node-test/node-v24.6.0-linux-x64/bin/node: relocation error: /root/node-test/glibc-2.38/lib64/libc.so.6: symbol _dl_signal_error, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

No luck. It complains about a missing symbol in the loader ld-linux-x86-64.so.2 itself.

Now it’s time to also replace the loader and calling the loader explicitly:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e LD_LIBRARY_PATH=/root/node-test/glibc-2.38/lib64 \
    docker.io/opensuse/leap:42 \
    /root/node-test/glibc-2.38/lib64/ld-linux-x86-64.so.2 \
    /root/node-test/node-v24.6.0-linux-x64/bin/node --version
v24.6.0

There we go! It is important to have both the loader and LD_LIBRARY_PATH pointing to the new libc version. Otherwise it won’t work.

The wrapper script

In order to make it simpler to call nodejs without always going through the loader explicitly, we’ll create a small bash script to run it:

#!/bin/bash
NODE_HOME=/root/node-test/node-v24.6.0-linux-x64
GLIBC_HOME=/root/node-test/glibc-2.38

exec ${GLIBC_HOME}/lib64/ld-linux-x86-64.so.2 \
  --library-path ${GLIBC_HOME}/lib64 \
  ${NODE_HOME}/bin/node "$@"

Save this as /root/node-test/node and make it executable.

Then, you can run nodejs as follows:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    docker.io/opensuse/leap:42 \
    /root/node-test/node --version
v24.6.0

Instead of calling /root/node-test/node-v24.6.0-linux-x64/bin/node directly, you are simply executing /root/node-test/node, which takes care of the details. If you put that onto your PATH, it’s even simpler:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    docker.io/opensuse/leap:42 \
    node --version
v24.6.0

And you can even run npm:

$ podman run -it --rm --pull newer -v $(pwd):/root/node-test \
    -e PATH=/root/node-test:/root/node-test/node-v24.6.0-linux-x64/bin:/bin:/usr/bin \
    docker.io/opensuse/leap:42 \
    npm --version
11.5.1

The npm executable is found in the PATH under /root/node-test/node-v24.6.0-linux-x64/bin/npm. This is actually a shebang script to run “node” from the PATH. Since the wrapper script /root/node-test/node exists, this will be used to run node, which in turn runs the npm js script.

Summary

You can replace the dynamically loaded shared objects with LD_LIBRARY_PATH and LD_PRELOAD. But sometimes, this is not enough - you need to replace the dynamic loader as well.

With this trick (executing the loader /lib64/ld-linux-x86-64.so.2 explicitly), it was possible to run a recent nodejs version on an old linux distribution.

There are some caveats: As the libc is a wrapper around syscalls and the syscalls are provided by the running kernel version, this could be a problem: If nodejs uses a new libc function that is implemented using a new syscall, that is not available in the old kernel, this won’t work. However, this seems not to be the case.

Using this trick is therefore not recommended to use in production environments. Having said that, it is still fascinating, that this even is possible and works.

References

Local WordPress copy with docker compose

Sun, 29 Jun 2025 00:00:00 +0200

Introduction

When you have a WordPress page and you want to test something without testing in production… what do you need?

A separate testing stage. Using docker, it is simple to start up some LAMPish (linux, apache, mysql, php) stack locally and start testing without the fear to break something.

Docker Compose

There are kind of official docker images for WordPress available on docker hub, e.g. https://hub.docker.com/_/wordpress. In order to use it, you also need a database, e.g. https://hub.docker.com/_/mysql. For easy access to the database, we’ll use another image: https://hub.docker.com/_/phpmyadmin. The complete docker compose file looks like this:

services:

  wordpress:
    image: wordpress
    restart: unless-stopped
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
      WORDPRESS_TABLE_PREFIX: prefix_
    volumes:
      - ./wordpress:/var/www/html

  db:
    image: mysql:8.0
    restart: unless-stopped
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - ./db:/var/lib/mysql

  phpmyadmin:
    image: phpmyadmin
    restart: unless-stopped
    ports:
      - 8081:80
    environment:
      - PMA_ARBITRARY=1

Save this as a file docker-compose.yaml.

Now you can start all three services with docker compose up. The WordPress installation will be available at http://localhost:8080 and the phpmyadmin is available at http://localhost:8081.

There are some important configuration options to know: The service wordpress uses bind mounts to access the local directory ./wordpress inside the container. When the container starts up the first time, WordPress is extracted and installed into this directory - and this directory is then served by the included apache webserver. If you start it the first time, you are presented with the usual WordPress installation wizard where you need to setup a user and password. If the container is restarted, this directory is reused. So the data is preserved. This is then also a simple way, to replace this default WordPress installation with your production copy, see below.

Similar thing is happening with the data of the mysql database: This is written down into the bind mounted local directory ./db and is preserved through restarts.

phpmyadmin doesn’t use any volume, it simply serves the application. As it is included in the same docker compose file, it can directly access the database via the service name “db”.

There are some configuration options, which are provided as environment variables. E.g. the WordPress instance needs to know the database, which must match the values for the mysql service. For phpmyadmin we use “PMA_ARBITRARY=1” which allows to use this phpmyadmin instance to connect to any reachable mysql server. You can simply enter “db” as server name and use “exampleuser/examplepass” to login.

You can stop the services with Ctrl+C, as docker compose up runs in the foreground. Start them again with docker compose up or destroy the with docker compose down. Note: Only the containers are destroyed, not the data, that is preserved in the local directories.

Copying the data

A WordPress instance consists of the PHP files itself (including any additionally installed plugin), the uploaded files and the content of the pages which is stored in the database. In order to replicate a production instance locally, you need to copy all.

Stop the (docker compose) services if they are still running.
Notice the userid/groupid ownership of the files. You’ll need this later. Then delete or move away all files in the local ./wordpress.
Copy all files from the production instance into local ./wordpress. And adjust the file ownership accordingly, e.g. with chown -R 100032:100032 ./wordpress.
Create a database dump from the production instance, e.g. via phpymadmin’s export functionality.

Adjust the WordPress configuration for the local stage, that means: Edit the file ./wordpress/wp-config.php and change the following properties:

define( 'DB_NAME', 'exampledb' );
define( 'DB_USER', 'exampleuser' );
define( 'DB_PASSWORD', 'examplepass' );
define( 'DB_HOST', 'db' );

// https://stackoverflow.com/questions/10578369/wordpress-wp-admin-redirects-to-https
// allow to use wp-admin locally
define( 'FORCE_SSL_ADMIN', false );

Then start the services with docker compose up.
Go to phpyadmin: http://localhost:8081 and import your production database dump into database “exampledb”.
Go to the table “options” (it might be prefixed with some table prefix) and change the options “siteurl” and “home” to be “http://localhost:8080”.
Then go to WordPress: http://localhost:8080. It should display the production page. You can also log in to wp-admin: http://localhost:8080/wp-admin using the production credentials.
Update WordPress, install new plugins, test. This should all work now locally.

Note: Some themes might have stored the production URLs for images, so keep an eye on the network requests to see, if all requests really end up on localhost. E.g. I noticed that the configured background image for “Avant Portfolio” was using a absolute production url. But configuring the background image again solved the issue.

Note 2: Maybe you have some absolute links in use for some custom navigation. You’ll need to change those manually to point to “http://localhost:8080/…” (or just use relative links if possible).

Note 3: If the file permissions are messed up, WordPress updates or plugin updates / installations might not work. So be sure, to use the correct userid.

Summary

This article shows an easy way to get a production copy of a WordPress instance running locally. You can simply copy the local folders and create another WordPress instance locally for another test. And all this without directly impacting the production instance.

Casio Wave Ceptor 3354 WVA-430TE battery replacement

Mon, 28 Apr 2025 00:00:00 +0200

I have this wristwatch, which is also viewable in the online watch museum: Casio Wave Ceptor 3354 Wva - 430te Funkuhr Tough Solar Titanium Herren Armbanduhr. They have much better pictures than I have.

It is a solar watch that didn’t need any maintenance for many years. I needed a new watchstrap once. But since one or two years, every now and then the watch stopped. After a while, it started to work again; I just had to adjust the hands because it stopped in the middle. And after it started to move again, it didn’t display the correct analog time - while the digital time was already good. But that is possible with this clock - it’s even described in the manual under “Adjust analog time”.

This happened more often lately. It’s winter and cold (not good for a chemically based battery) and my sleeves usually cover the clock almost all the day. So it probably just ran out of power from the rechargeable battery - until it stopped. It couldn’t even go to save mode properly (moving the hands to 12:00). It just stopped.

The watch has a small solar panel, which is usually sufficient to charge the battery if enough sun light hits the surface during the day. And I had no problems at all for many years.

But now, after about 17 years in constant use, it’s time for a new battery it seems. Kudos to Casio, which made this easy repairable. The backplane uses 4 screws and then you are in. You replace the battery, there is a sign to use only the correct battery type - which I confirmed, I ordered the correct one: CTL1616. When putting in the new battery, the watch showed the same behavior as the before: It didn’t run, it was in some kind of emergency power save mode. The one indicator bottom left in the display was blinking, but no other signs were shown. If you short-circuit point AC with ground for two seconds - as noted on the sticker - it started to run. This behavior indicates, that the theory of a worn out battery after 17 years could be correct. I measured the voltage of the old battery - it showed about 2.4V. But so did the new battery. I guess, voltage on its own is just not a helpful indicator of the health of a rechargeable battery.

Anyway, I switched the battery now. Let’s see, how long this one lasts.

Here are some pictures:

watch - front side

watch - back side

inside of watch: sticker on the battery

inside of watch: sticker removed, battery visible

the new battery - looks like the correct type CTL1616 for casio

new battery left, old battery right: both are even from Panasonic

the old battery had an additional insulation which I reused

Glitch in the matrix?

Sun, 23 Mar 2025 00:00:00 +0100

Just to manage expectations - it is probably explainable, how such things happen in the real world. But it’s nevertheless something strange, if you take signposts seriously.

Berg am Laim

So, first things first. Let me explain the situation: In Munich there is an underpass in Berg am Laim, that goes below the railways. In the last year or so, this underpass has been a bit refurbished and new signs have been installed.

The signs that have been added show the shortest exit way. That’s probably a new regulation that such signs need to be installed and glowing if the underpass has a certain length and could be considered a tunnel. This underpass is probably the gray area, as you can see the light at the end of the tunnel already. So it is not very long.

Underpass... (100m before)

or tunnel?

Anyway, here is the sign at the beginning of the tunnel:

Sign 1

As you can see, it has been added after 20 meters and you can exit the tunnel to either side: The nearest exit would be 20 meters away, the other end is 150 meters away. That allows you to deduce that the total length of the underpass is 170 meters. So far, so good.

After another 30 meters, you see the next sign:

Sign 2

We see, it says either 50 meters or 120 meters - in total 170 meters. Still makes sense, no problem so far.

But wait, until we walked another 20 meters to the next sign:

Sign 3

Here we can clearly read, that the exit is either 70 meters or 90 meters away. In total our tunnel is now 160 meters long. It shrunk 10 meters. Interesting. Between the last sign and this one the tunnel lost 10 meters. Good for us, who want to go to the other side, we need to walk 10 meters less.

Let’s see, how it goes on. After another 20 meters, we see the next sign:

Sign 4

It says 90/70 meters, which is in total still 160 meters. As we see from the numbers, we passed already more than half of the tunnel. The far end is now nearer.

Let’s walk another 30 meters to the next sign. But wait, what does it show?

Sign 5

It says 120/50 meters, in total it’s again 170 meters. It seems we left the weird area, when we were pleased already that we need to walk 10 meters less. The tunnel is back to its original length. That’s very strange.

Just to be sure, let’s walk to the last sign, 30 more meters:

Sign 6

It says logically 150/20 meters, in total 170 meters. All makes sense.

Just to double check, let’s walk in the same direction but on the other side of the street and observe the exit signs there:

Sign Right 1: 150m

Sign Right 2: 150m

Sign Right 3: 160m

Sign Right 4: 160m

Sign Right 5: 160m

Sign Right 6: 160m

The right side seems to be more consistent: It just expands 10m from one end to the other. Interestingly, the tunnel seems to be 10m longer on the left side than on the right side.

Another look at this right side of the tunnel reveals another strange thing: The pictures are taken in order, so you should expect, that the distance from one end (where we started) is strictly monotonically increasing, while the distance to the other end (where we are heading to) is strictly monotonically decreasing. The figure on the sign should always look into the direction of the nearest exit. The distance to the starting end of the tunnel is: 10m, 40m, then suddenly 90m, then back to 70m (I definitely didn’t walk backwards…) and then jumping up to 120m. Something strange is happening here.

In summary, this tunnel is a very interesting construction: If we use this effect on purpose for other tunnels or maybe even streets, we could save so much time. Just shorten the distance in the middle of a route and we’ll be faster arriving at the destination. Nice.

There’s a catch though: If you travel in reverse direction, the length of the tunnel might be longer. Which gives the routing algorithm another way to optimize routes.

Java by Comparison and PMD

Sun, 23 Feb 2025 00:00:00 +0100

This is now a different kind of blog post. I want to present a little bit the book “Java by Comparison” by Simon Harrer, Jörg Lenhard, and Linus Dietz. But not just present it, instead evaluate, which examples could be automated to be detected by static code analysis and maybe which PMD rules already exist.

The book

The book itself is not new, it is from 2018. It contains 70 examples, which are presented in a concise form: The left page shows the example with the “bad” code and the right page shows the improved code. So you can see the example side-by-side. Additionally, there are lots of explanations of course.

I like the approach taken, to explain common programming “rules” and best practices with examples. That is much more catchy than just a written down explanation.

The book is available from the Pragmatic Bookshelf and has an own page: https://pragprog.com/titles/javacomp/java-by-comparison/.

The structure

Organized by wider topics. I’ll follow the same structure. Maybe sometimes I’ll combine two examples.

Small code cleanups

This first chapter is about small code style cleanups, like superfluous code or inconvenient written code, that is hard to read.

“Avoid Unnecessary Comparisons” are conditional expressions, that are compared against a boolean, such as if (a == true) .... This can be written more concise as just if (a), no need for this extra comparison. It turns out, there is already a PMD rule for that: SimplifyBooleanExpressions. The reasoning behind this is: readability and code clutter.
“Avoid Negations” makes the point, that an expression is easier to read and understand, if there is no negation involved. E.g. instead of if (!isEmpty()) use better if (isNotEmpty()). There is no direct PMD rule that finds such code, but it could be easily written as an XPath expression. E.g. //UnaryExpression[@Operator = '!']. There is however a related rule, that is ConfusingTernary. This makes the point, if you have an if-else-expression and you use a negated condition, just swap them around and use a positive condition.
“Return Boolean Expressions Directly” is about avoiding clutter again. Instead of return true inside a if condition, you can directly return the condition and avoid the if statement entirely. There is already a PMD rule: SimplifyBooleanReturns.
“Simplify Boolean Expressions” is about readability and easier understanding. There is a PMD rule with the same name, but that rule does something different, see above. If the boolean expression is a bit more complicated, it makes sense to introduce some local variables to give the parts of the expression a name and make it easier to comprehend. This has currently no equivalent PMD rule. However, we could count the number of parts the expression consists of (literally how many && and || we have). The suggestion would be to split this up and introduce named local variables or methods. It would be similar like CyclomaticComplexity but focused on a single expression.
“Avoid NullPointerException in Conditionals” talks about the issue of forgetting to check the input parameters of public (or protected or default) methods against null before using them. There are some related PMD rules, that help to avoid such NPEs: LiteralsFirstInComparisons, BrokenNullCheck, EqualsNull (which is a bit silly), MisplacedNullCheck, UnusedNullCheckInEquals. But there is no rule that e.g. checks whether a method’s parameter is used without a null check. This rule seems to be possible, but might yield a lot of false positives. It might be a code style rule, as the rule of thumb would be then: always check parameters for null before using them.
“Avoid Switch Fallthrough” is self-explanatory. And there is already a PMD rule available: ImplicitSwitchFallThrough.
“Always Use Braces” is a code-style topic. PMD used to have multiple rules for each statement that checked, whether braces are used. Since PMD 6.2.0, there is one rule for that: ControlStatementBraces.
“Ensure Code Symmetry” is an interesting one. This is about readability and understandability of the code. The examples shows a if-else-if structure, where some branches do deny access and other branches grant access. In order to make the code more symmetrical, these two different concerns should be split up and separated, so that when reading the code, one has to understand only one part at a time and not both concerns at once. Detecting such an asymmetry sounds not so easy. We could use some heuristics like (in case of if statements) the number of statements in each branch or the accessed variables in each branch. It’s definitively a rule, that would only give a hint, that something could be improved in the code, but there are no hard indicators.

Improved code style

The next chapter is about even better code style.

“Replace Magic Numbers with Constants” is a simple rule that helps in understanding the code. If you use only literals for the numbers, that you would need to explain the meaning of the number. This can be improved by defining a constant with a name for the number and use the constant. Now the number has a name and a meaning and the code is easier to understand. Surprisingly, PMD doesn’t have such a rule. I know however, that Checkstyle has one, e.g. MagicNumber.
“Favor Enums Over Integer Constants” is a follow-up on the previous example - enums might even be better in some circumstances. There is no rule in PMD for that. The rule would need to be a bit more intelligent than just pointing to constants (static final) - it would need to identify the usages of these constants. In the example in the book, there is a method which takes an input parameter and compares this against a bunch of constants and only doing something, if the input parameter matches one of the constants. This essentially limits the possible values for the input parameter by using the constants, which makes it a perfect case for enums. That means, a PMD rule would need to identify such a method, which probably would only work reliably in simple cases such as the given example. It could look a bit broader and consider the cohesion between constants and methods and if there is a single method where a bunch of constants are used (and the constants are not used somewhere else), then we could suggest to introduce an enum. All this would require, that the constants are not public, as otherwise we would need to check the usages across the whole project.
“Favor For-Each Over For Loops” is an easy one and there is a PMD rule: ForLoopCanBeForeach. The rule description however misses an explanation, why a foreach loop should be used instead. The reason is, that the code is simpler, because it doesn’t expose unnecessary variables such as the index variable. That makes the code less prone to mistakes (e.g. using the wrong variable for accessing the element by index, etc.).
“Avoid Collection Modification During Iteration” finds calls to remove or add or any other modification of a java.util.Collection, while it is iterated over in a foreach loop. In most implementations this will lead to a “ConcurrentModificationException”. In case we need to remove an element during iteration, we should use the Iterator explicitly, which has a remove-method (given this operation is actually supported). Another way would be, to iterate over a copy of the collection, so that we can modify the original collection. PMD doesn’t have such a rule, but it should be easy to implement this. It would be in the category “error prone”.
“Avoid Compute-Intense Operations During Iteration” belongs to the category “performance”. In the example, a call to Pattern.matches is done inside a loop over and over again, always with the same regex pattern. The method internally compiles the regex and this is an “expensive” operation, which should be done only once. The resulting pattern should be reused. There are other methods around regex, that suffer from the same “not cached pattern”: e.g. String.replaceAll(String, String) - if this is called within a loop with the first argument always being the same, the same pattern would be compiled over and over again. There are probably more cases of “expensive” operation, that should be avoided inside a loop. For the Apex language, there is the rule OperationWithHighCostInLoop, which serves a similar purpose (but finds other method calls). For Java, we have already AvoidInstantiatingObjectsInLoop, which is at least loop-related. But there is no rule yet, that checks specifically for the mentioned regex compilation calls.
“Group With New Lines” suggests to use empty new lines in the code, to form code blocks, that belong together semantically. I guess, that would be very difficult for a static analyzer to figure this out, as it would need to really understand the code. It goes into the concept of cohesion. Theoretically, your class would be so small, that it does only one thing, that there would be only one group and no need for new lines. But that’s not the reality. In that sense, it might be easier to just detect big classes like GodClass or just ExcessivePublicCount or TooManyMethods.
“Favor Format Over Concatenation” reminds us, that java nowadays also supports C-like format strings. If you are building a big string combining labels and values, it quickly becomes a mess and is very difficult to read. A format string is most likely easier to read and should be used instead. While PMD has a couple of rules that deal with StringBuilder and the concatenation strings in this context, it doesn’t have such a rule. It shouldn’t be too difficult to detect InfixExpressions (“+”) of type String (e.g. //InfixExpression[@Operator='+'][pmd-java:typeIs("java.lang.String")]).
“Favor Java API Over DIY” suggest to use the Java API and the ready to use methods. The book brings two examples: Instead of doing a null check manually and throwing a NullPointerException, just use Objects.requireNonNull and instead of manually counting how often a given element is in a collection, use Collections.frequency. PMD doesn’t have such a generic rule, that would find all the cases. And such a rule would needed to be updated regularly and would also need to consider the used Java version, so that not a suggestion of a method is made, that cannot be used, as it was only added in a later Java version. There is however one rule, that goes into this: AvoidArrayLoops, which suggests to use System.arraycopy or Arrays.copyOf. And the PMD rule UseArraysAsList suggest to use Arrays.asList instead of a manual loop. So, there are already two PMD rules in this area, but for sure there could be much more cases.

Commenting code

This chapter is all about comments. And while comments are welcomed in most cases, unhelpful or outdated comments are bad.

“Remove Superfluous Comments” addresses old comments (with wrong content) and comments, that don’t add information, as they merely repeat the code (e.g. comments telling what is done instead of why is something done). This reminds me of JAutodoc which can generate JavaDoc comments without any useful content. PMD has some rules in the category documentation but these rules mostly do the opposite: They demand a comment. It would be difficult to write a rule, that interprets the content and figures out whether it is superfluous (and can be removed) or not. But we could create a rule, that identifies comments, that just repeats the method name and parameters and doesn’t add anything additional (finding the generated JAutodoc comments that have never been changed). Another thing is “TODO” comments: The recommendation in the book is - either fix the TODO immediately and remove the TODO comment, and if this is not possible right away, create a new issue in the issue tracker and remove the TODO comment. We could create a rule that find the TODO comments, like the checkstyle rule TodoComment.
“Remove Commented-Out Code” is about using your VCS to remember old implementations rather than using comments. There is also no PMD rules for that. Commented out code has the problem, that it often doesn’t work anymore when it is commented in, as the surrounding projects has evolved (e.g. refactored) and the commented out code has been updated. In the end, it’s very likely, that the commented out code cannot be used anyway. In regard to detecting this, there could be some regular expressions that detect method calls. And the position of such a comment can also give a hint - e.g. inside an argument list.
“Replace Comments with Constants” is a bit similar to “Replace Magic Numbers with Constants”. In the example given in the book, the comments explained the meaning of the magic numbers. The comment could be used as an additional information/suggestion when creation a violation message for the magic number, like “Did you mean to create a constant named XYZ?”. In that sense, it would be an enhancement to the magic number rule, but not a rule on its own.
“Replace Comments with Utility Methods” gives an example, that sometimes a code block (some contiguous code lines) are preceded by a single line comment. That seems to be an opportunity to refactor out this code block and give it a name - exactly the name from the comment. I think, it would be quite difficult to reliably find such comments, where such a refactoring would make sense, while ignoring the other comments.
“Document Implementation Decisions” is about expanding short comments (like “fast implementation”) into a more detailed comments, explaining the what, why, how of the decision that is made. There is no PMD rule that can identify such comments and it would probably be quite difficult to implement such a rule.
“Document Using Examples” makes the point, that the JavaDoc for a regex pattern is much more easier to understand when examples are provided of some strings, that match, and some, that doesn’t. This however is again about interpreting the content of a comment, which is out of scope for a static analyzer like PMD. Of course, we could say, that a java.util.Pattern should be preceded with a comment, that has the two keywords “Valid examples” and “Invalid examples”. That could identify exactly one special case (regex patterns), but wouldn’t find other missing examples.
“Structure JavaDoc of Packages” is about the content of the JavaDoc for packages. It should mention the most important classes of the package and maybe give a usage example of those. All this is again out of scope for PMD. There is the rule CommentRequired which can detect missing comments, but doesn’t consider, whether the comment contains useful content. And also, this rule is able to find missing JavaDocs on packages (e.g. in package-info.java).
“Structure JavaDoc of Classes and Interfaces” is about ensuring a specific structure for the JavaDoc of types. It should start with a summary sentence. While PMD itself can only ensure that there is a JavaDoc at all (see CommentRequired), it doesn’t check the contents. Checkstyle has a rule called SummaryJavadoc which does some checks on the first (summary) sentence. While a static analyzer can definitely verify the there is a first sentence and that this is separated by a blank line from the rest of the doc, it can’t verify that the content is useful at all.
“Structure JavaDoc of Methods” and “Structure JavaDoc of Constructors” are similar like the one before: PMD can only verify the existence of such Javadocs, but can’t check the contents. Checkstyle has some more sophisticated checks for completeness, like all method parameters need to be documented (see JavadocMethod), but it can’t verify the content.

Naming

Naming is said to be one of the hardest things to get right in computer science. If you want to know the other, see Two Hard Things. So, this chapter is all about naming things in Java.

“Use Java Naming Conventions” is about “correct” casing of types, fields, variables, methods and all that. Luckily PMD has a rule already, actually several rules: ClassNamingConventions, FieldNamingConventions, FormalParameterNamingConventions, GenericsNaming, LocalVariableNamingConventions, MethodNamingConventions, PackageCase. Most of these rules can be configured, but that’s not the goal of the example the book makes: You should simply use the Java Code Conventions as defined by Oracle, and don’t invent your own conventions.
“Follow Getter/Setter Conventions for Frameworks” says that if you create a JavaBean, you should follow the bean conventions. The getter and setter methods and the field names must be connected. There are two PMD rules for that purpose: InvalidJavaBean and BooleanGetMethodName. Following such conventions spares you from surprises, that something doesn’t work as expected, when your bean is used in a “standard” Java framework.
“Avoid Single-Letter Names” is about naming of parameter names and local variables. Today, with the IDE’s help, it not difficult anymore to avoid any too short names. The IDEs can most times help you with autocompleting the variable names. So you often don’t have to type the full name. Longer names can provide so much more information about the variables, that the code is much easier to understand. PMD has a couple of rules for that case as well: ShortClassName, ShortMethodName, ShortVariable.
“Avoid Abbreviations” might be picked up by the other short naming rules, since an abbreviation might be short enough. But not always, e.g. “bufW” is a correct variable name and 4 letters long, but it isn’t helpful. In the book’s example, it’s used as an abbreviation for “bufferedWriter” - which contains too much detail - so “writer” is the sensible solution here. There is no PMD rule for that, but a similar rule in Checkstyle: AbbreviationAsWordInName. In general, we could split the name into single words (e.g. a case change starts a new word) and verify the length of the single words. The “bufW” would become “buf” and “W” and we have a short name. Whether that’s an abbreviation can only be a guess.
“Avoid Meaningless Terms” is a nice example about bad naming. In today’s software, you often find a bunch of “Manager” classes or “Impl” suffixes. The most famous example referenced in the book is with the following quote:

Roses are red, leaves are green but only Java has AbstractSingletonProxyFactoryBean.

It’s from http://bash.org/?962108 (wayback machine) and it relates to the AbstractSingletonProxyFactoryBean in the spring framework.

There is no rule in PMD that finds such things. We could write a rule, that searches for some given meaningless terms, but whether they are really meaningless can’t be decided. We could only highlight suspicious names.
“Use Domain Terminology” is about using the correct terms when modeling the classes and methods. That way the software is easier to understand (if you now the domain at least). Don’t invent new names for something, that is already known. This is again very difficult for static analyzer, as it would require some level of understanding, what the software really does.

Error handling

This chapter is about proper error handling using exceptions and other means.

“Fail Fast” is a simple code style, that avoids nested structures and keeps the logic flat. If you check for an error condition at the start of a method and return or throw an exception directly, then you can be sure, that for the remaining part of the method, this particular condition is already handled. This makes the normal path of execution again more visible, as it is not hidden between other error condition checks. There is no PMD rule for this, but it shouldn’t be too difficult to write a rule, that detects methods that start with an “if” statement with multiple branches and each branch (except one) ends with a throw statement.
“Always Catch Most Specific Exception” is about the problem, when you simply catch “Exception”, which is very unspecific: It could be any problem - and you handle all the different problems the same way. And especially with unchecked exception, you might handle more cases than you wanted. But you want to have your program crash when e.g. there is a NullPointerException, as this is a programming error. There exists already the PMD rule AvoidCatchingGenericException.
“Explain Cause in Message” addresses the difficulty of good error messages. The simple text “An unexpected error occurred” is not helpful - and this is already clear from the exception anyway. The book suggests a simple pattern that one can follow: “Expected [EXPECTED], but got [ACTUAL] in [CONTEXT]”. This is much more helpful in understand later on, what actually led to the error. For this, there is no direct PMD rule. We could check however, that an exception isn’t created without a message at all. But the rule would be very inaccurate. I guess, this means, that such a rule would be a bit more difficult to implement.
“Avoid Breaking the Cause Chain” is about passing the original exception, when throwing a new exception. This is easy: PMD has already a rule for that: PreserveStackTrace.
“Expose Cause in Variable” reminds us, that exceptions are classes on its own and can contain more structured information other than just the message. They can contain fields and methods as well. This can be useful for other parts of the application, where one needs to handle the exceptional situation. The suggestion is to create a custom exception subclass, that contains the interesting information pieces as fields. There is a PMD rule AvoidThrowingRawExceptionTypes, but it doesn’t match this case exactly. I guess, it will be difficult to automatically detect, when a custom exception would be useful. One heuristic could be: If the message is built as a string concatenation out of multiple single components, then this would be an indication to just pass the components into the exception’s constructor and have variables for these components.
“Always Check Type Before Cast” suggests to always do a instanceof check before you do the cast. Not checking the type before might result in a ClassCastException at runtime. There is no PMD rule for that, but it shouldn’t be too difficult, to write a rule. The cast should be happening as the first statement in a if block, that checks the type using “instanceof” - then it’s ok. A cast without such an if might be problematic.
“Always Close Resources” suggest to simply use try-with-resources statement for dealing with resources. This is available since Java 7 and ensures, that the resource is closed at the end. Luckily there is already a PMD rule for that: CloseResource. Actually, there is another rule: UseTryWithResources.
“Always Close Multiple Resources” is a variation of the case before and makes an example when you manually close multiple resources in a finally block. You better just use a try-with-resources statement with declaring multiple resources. The following PMD rule applies here: UseTryWithResources.
“Explain Empty Catch” focuses on catch blocks, that contain no statements and just swallowing the exception. It is unclear then, whether the exception handling has been forgotten to implement or whether this is intended behavior in that case. PMD has the rule EmptyCatchBlock for this.

Unit testing

This chapter gives some examples on what to pay attention to when writing unit tests.

“Structure Tests Into Given-When-Then” gives advice how to make unit tests a bit more readable. Often you first need to setup some test preconditions like the data. After that, you are executing the functionality under test. And then you need to verify the final state and compare it your expectations. There is no PMD rule to check for this example. And I think, it would be difficult to write one: PMD can’t understand, which lines of the unit test belong to which semantic part (“given”, “when”, or “then”). We could check for the length of a unit test - if it is too long (whatever this means), it is suspicious to be testing too much (e.g. several functionalities). But that would be a different case.
“Use Meaningful Assertions” is about using the correct assert-helper-methods when possible. E.g. instead of assertTrue(result == 1), one should use the more appropriate method assertEquals(1, result). This method also gives a better failure message, when the test fails: “Expected 1 but was xyz” instead of “Expected true but was false”. PMD has a rule for that, that checks a couple of cases: SimplifiableTestAssertion.
“Expected Before Actual Value” is about calling assertEquals but using the wrong parameter order. For JUnit5 Assertions and JUnit4 Assert, the first parameter should be the expected value and the second parameter should be the actual value. Mixing up the two parameters results in confusing test failure messages. Note, that TestNG Assert uses a different argument order. There is no PMD rule to detect this. To be 100% reliable, it would be very difficult to implement. But for simple cases, where the expected value is a compile time constant, while the actual value is the result of a method call or a variable, we could actually find such cases.
“Use Reasonable Tolerance Values” suggests to use a delta when comparing floating point values. It’s easy to forget this and there won’t be a compiler error, since the two double values would be boxed to Double and then compared using equals. But exact comparisons of two double values aren’t reliable. JUnit provides an overloaded assertEquals method for doubles that takes a third double argument as delta. There is no PMD rule for this, but thanks to type resolution, it should be simple to implement a rule, that finds assertEquals calls with doubles or floats, that only use 2 parameters.
“Let JUnit Handle Exceptions” covers the old-style way of testing exception, before assertThrows() was available or manually failing the test in a catch-block with fail(). That has the downside, that you don’t see the exception and therefore it might not be clear, why the test was failing. Interestingly, PMD doesn’t have a rule for that, but it should be quite easy to implement.
“Describe Your Tests” is about two similar things, and it is about naming. First, you test names should have a meaningful name. With JUnit, you can be more descriptive by adding a @DisplayName annotation. This helps to understand, which functionality exactly is broken in case a unit test fails. However, PMD won’t be able to assess whether the description is good enough and is exactly correct. It could only require that every test is additionally annotated with @DisplayName and that the test methods e.g. don’t start with the prefix test - as this is not necessary anymore since JUnit4.
The other aspect of this example is, to give a clear reason, why a test is ignored. You can ignore tests with the @Disabled annotation - and this annotation allows you to specify some explanation, why the test has been disabled. PMD could require, that such an explanation is always there for disabled tests, but it won’t be able to asses, whether the explanation makes sense.
“Favor Standalone Tests” is about a problem that arises, when setup methods like @BeforeEach are used: In order to understand a test, you also need to read the setup method in addition to the test itself. That means you need to jump around in the test source. The tests are better comprehensible, if they have the complete pattern “given-when-then” contained in each test. The given-part would otherwise be separated out into the setup-method. So, the recommendation here is, to simply avoid @BeforeEach and @BeforeAll. The setup-method could still be shared, but just needs to be called explicitly. We could easily write a PMD rule to detect any usages of these setup-methods.
“Parametrize Your Tests” looks at a test pattern, where you test inside one method the same code, but with different values. This means, you usually have already factored out the actual testcode, e.g. in a custom assert method, which is called repeatedly with varying values. For this, @ParameterizedTest is a good option: It makes the test report more understandable, and you directly see, which of the different values is causing a problem. There is no PMD rule for that, and I think it wouldn’t be so easy to identify a test, that would be suitable to be refactored into a parameterized test. PMD could maybe detect the very simple cases, but you immediately need to have some understanding of control flow, as there are some method calls involved.
“Cover the Edge Cases” is about test coverage and writing not only one test (“the happy case”) but also considering failure cases and - as the title mentions - edge cases. As this all depends on the subject under test, it seems to be very difficult to write a rule for that. We could write a rule, that detects test classes, that only have one test case defined - but that’s probably a too simplistic heuristic.

Design examples

This chapter is about design and how functions should not only be functional, but also beautiful and convenient.

“Split Method with Boolean Parameters” focuses on methods, that take a boolean parameter and contain a big if-statement with two execution branches. This is a opportunity to make two methods out of this one method, give a better name and avoid the ugly boolean parameter. If a PMD rule would exist for that, it would be in the category “Design”. But there is no rule. Simple cases like the example in the book can easily be detected, though.
“Split Method with Optional Parameters” looks at methods, which take a parameter and that parameter is part of a null check and early return. This might also be a candidate for a simple rule, that finds such simple cases like in the book.
“Favor Abstract Over Concrete Types” addresses the same problem, that the existing PMD rule LooseCoupling does. Interestingly, this rule is not in the “Design” category, but in “Best Practices”.
“Favor Immutable Over Mutable State” is about designing value objects. If you objects are immutable, you avoid lots of the problems in multi threading and consistency. Instead of mutating the object, a new instance is returned instead, similar like BigDecimal or LocalDate. PMD has some related rules, like ImmutableField or UselessOperationOnImmutable, but no rule that suggests to write such immutable value classes in the first place. It might be a bit difficult, to limit this rule and avoid too many false-positives. And it would be a rule, that requires more refactoring and not a simple quick fix.
“Combine State and Behavior” is about object oriented design and looks out for classes without methods but only fields. Then there would be a second class, which contains only methods and take the first class always as a parameter. First class represents the “State” and the second class represents the “Behavior”. Another way to look at this is, that the methods could be static, as they don’t use any instance variables of their own class. So, maybe that is an indication, that the method should be moved somewhere else. Right now, there is no PMD rule for that. A first try could be rule, that finds methods, that could be static.
“Avoid Leaking References” is about encapsulating and protecting the inner state of a class. PMD has a rule for arrays, that are leaking: MethodReturnsInternalArray. But the book is right, this is not only about array, but also about collections.
“Avoid Return Null”: PMD has a rule that suggests to return a empty collection, rather than null: ReturnEmptyCollectionRatherThanNull. The book is more strict about this and suggests, that you should think about this, whenever you return null from a method. Avoiding null and returning a null object instead is a common pattern, that can avoid null pointer exception later.

Using streams

Streams have been introduced in Java 8 with JEP 107. These provide a functional way to express, what should happen in data processing instead of how. Using streams often means, that lambdas and method references are used. And there are a couple of things to consider in terms of readability and maintainability.

“Favor Lambdas Over Anonymous Classes” shows an example with a simple anonymous class, that implements a a functional interface. This can be written more concise without the boilerplate as a lambda expression. The specific example here is by the way not directly related to streams - it uses the method computeIfAbsent, which is part of a Map and takes a single Function<T, R> as an argument. There is no PMD rule for that. But it should be simple thanks to type resolution, to figure out whether the implemented anonymous class is actually implementing a single abstract method (or a functional interface) and suggest to use a lambda instead.
“Favor Functional Over Imperative Style” is now really an example, that suggest to use streams. The example uses a for-each loop with an if-condition inside and collecting the result in a list after performing a mapping of the values. There is no PMD rule for this. Creating such a rule might result in lots of false positives, so that rule could only say: “This for-each loop probably can be converted into a stream pipeline, but I’m not 100% sure”. I guess, this rule would be a bit more difficult to implement reliably.
“Favor Method References Over Lambdas” is easy: There is a PMD rule already existing: LambdaCanBeMethodReference. Using a method reference can express the intention more concise and is therefore simpler to read and understand. The used methods themselves can be tested independently of their usage in the stream pipeline.
“Avoid Side Effects” provides an example of a stream pipeline, that uses as the last stage the forEach method. In this last stage, a object defined outside of the stream is modified - the side effect. And this is not pure functional programming style anymore. There is no PMD rule for that and it seems, that a reliable rule is not so easy to implement. We could start with verifying the forEach call - it can be used as a terminating (last) operation. But if it modifies some other object, this might cause problems, if the stream is executed in parallel, as the terminating operation might not be executed in the original order or even in multiple threads.
“Use Collect for Terminating Complex Streams” suggests to use instead of a forEach operation, which then collects each element of the stream manually into a collection, to use collect instead with the various provided collectors, such as grouping. In the given example, the end result of the method is a Map of some kind, but the stream doesn’t create this map with collect, but rather using forEach with side effects. The Map is a local variable outside of the stream. Similar to the last example, there is no PMD rule for this. And it seems to so simple to implement a good rule for that. We could only point to suspicious stream pipelines.
“Avoid Exceptions in Streams” gives an example on how to deal with (checked) exceptions that need to be handled in the lambda expressions used for the stream operations. In functional style programming, functions should produce an output given a specific input - exceptions don’t fit into that scheme. Throwing a RuntimeException inside a stream pipeline make the whole stream not producing a result anymore and just end with an exception. To avoid this, in this example it is suggested to ignore the element of the stream, that caused the exception and let the stream continue to compute the result. This can be achieved by using flatMap and returning a single element Stream via Stream.of() in the successful case and an empty stream (Stream.empty()) in the exceptional case - effectively skipping the element if there is a problem with that. This of course depends heavily on the actual use case, whether you can ignore an exception or not. There is no PMD rule for that, but one could be written that detects throwing of exceptions inside a lambda expression, that is used in a stream pipeline.
“Favor Optional Over Null” is about communicating explicitly, if a return value of a method could return nothing (aka. an empty optional or null). Just returning null would require to do a null check by the caller of the method, but that could be easily forgotten and resulting in a NullPointerException. Returning an Optional<T> instead forces the caller to think about, what to do, if there is no result from the method call, and thus can avoid NullPointerExceptions. There doesn’t seem to be a PMD rule for detecting this. We could start with simple cases, where a method explicitly returns null in some execution path. But this wouldn’t detect all potential cases. The rule would error on the false negative side, which might be ok for the beginning. Another angle for detection could be to look at method calls. If there is directly after the method call a null-check, then the code assumes that the method could return null, so we could report a violation here.
“Avoid Optional Fields or Parameters” is about a different usage of Optional. The last example suggested to use Optional as a return type of methods. When returning an Optional, you never return null for that method. But if you have a method parameter with Optional, a caller could provide null - and now you need to deal with three possible values: null, empty Optional, and a value wrapped in Optional. This just makes implementing the method just more complicated. It would be simpler, if the method just allowed to be called with a non-null value and throws an exception, if it is called with null (e.g. via Objects.requireNonNull). The same applies for fields, as they can then have potentially the same three possible values and every access to the field in the class is more complicated. There is no PMD rule for this, but the rule is straightforward. We just need to look for usages of Optional in field declarations and in the argument list of method declarations.
“Use Optionals as Streams” suggests a more functional approach when using Optionals: Instead of checking with isPresent() and then getting the value explicitly, the Optional class has a lot of convenience functions that are very similar to the stream functions: like map, filter. Additionally, it as terminating functions like orElseThrow. There is no PMD rule for this. A potential rule could look for a “isPresent” method call followed by “get” method call to find suspicious usages of Optional. The rule might find usages, that maybe are not so easily adaptable to the suggested functional style.

Production readiness

This chapter is partially about general advice, such as “Use Static Code Analysis Tools”. Even PMD is suggested, among other tools like SpotBugs, Checkstyle, Error Prone, or IntelliJ IDEAs Code Inspections. Other general advice is: Use a common code format, use automated builds and continuous integration. But there also some more concrete examples.

“Favor Logging Over Console Output” targets the usage of System.out and System.err. Instead of these, you should use a proper logging framework. Such a framework can easily be configured with a log level to output debug messages only in certain situations and they also provide a way to log exception stacktraces properly. PMD has a couple of related rules for using logging, such as GuardLogStatement, InvalidLogMessageFormat, MoreThanOneLogger, ProperLogger, and UseCorrectExceptionLogging. All these rules already assume, that a logging framework is used. There is two additional rules: AvoidPrintStackTrace finds calls to exception.printStackTrace() and it says “use a logger call instead”. And there is the rule SystemPrintln which flags any usages of System.out/err. Exactly what this is about. It simply suggests to use a logger.
“Speed Up Your Program” is a simple tip when using Streams: Streams might be sped up when using parallel streams. This however is only possible, if the single stream operations are indeed free of side effects and depending on the stream (if it is sorted), a parallel stream might not always be faster. We don’t have a rule in PMD for that. Creating a rule, that identifies streams, that are not yet parallelized, is simple. But the rule can only give the suggestion, to try out, whether using a parallel stream would make your program faster - as it depends not only on the stream operations but also on the amount of data, that is being processed.
“Know Your Falsehoods” is the last concrete example in the book. It is about making too simplistic assumptions about the real world. The example show some code, that parses a given name of a person and extracts the surname. However, the surname might not always be provided as the last word - and it could be composed of actually two words separated by a space. Not to mention other regions (i18n, l10n). The solution for this concrete problem is, to make no assumption at all and just ask for the surname directly. This is definitely an example, that can’t be formalized as a PMD rule.

Summary

The book contains a lot of examples. And if you like to learn from examples (bad and good), that’s are very useful book. My goal here was to map the examples and suggestions to existing PMD rules and to rules, that could be potentially implemented. That way, you have an easy reminder, if you don’t follow the suggested practices.

Even if you are not new anymore to developing software, it’s still a good book to remind you of the good practices, that are maybe forgotten in the hectic everyday life.

Im closing with a couple of tables, that contain the mapping between the examples from the book and the PMD rules. It would be great to update the rule documentations, to mention the book and improve the reasoning behind the rules.

Existing PMD rules

Example Title	PMD Rule	Category in PMD
Avoid Unnecessary Comparisons	SimplifyBooleanExpressions	Design
Return Boolean Expressions Directly	SimplifyBooleanReturns	Design
Avoid Switch Fallthrough	ImplicitSwitchFallThrough	Error Prone
Always Use Braces	ControlStatementBraces	Code Style
Favor For-Each Over For Loops	ForLoopCanBeForeach	Best Practices
Use Java Naming Conventions	ClassNamingConventions, FieldNamingConventions, FormalParameterNamingConventions, GenericsNaming, LocalVariableNamingConventions, MethodNamingConventions, PackageCase	Code Style
Follow Getter/Setter Conventions for Frameworks	InvalidJavaBean, BooleanGetMethodName	Design and Code Style
Avoid Single-Letter Names	ShortClassName, ShortMethodName, ShortVariable	Code Style
Always Catch Most Specific Exception	AvoidCatchingGenericException	Design
Avoid Breaking the Cause Chain	PreserveStackTrace	Best Practices
Always Close Resources	CloseResource, UseTryWithResources	Error Prone and Best Practices
Always Close Multiple Resources	UseTryWithResources	Best Practices
Explain Empty Catch	EmptyCatchBlock	Error Prone
Use Meaningful Assertions	SimplifiableTestAssertion	Best Practices
Favor Abstract Over Concrete Types	LooseCoupling	Best Practices
Favor Method References Over Lambdas	LambdaCanBeMethodReference	Code Style
Favor Logging Over Console Output	AvoidPrintStackTrace, SystemPrintln	Best Practices

Potential PMD rules

These are the examples, for which no direct counterpart in PMD exists. But on first glance, it sounds feasibly to implement PMD rules in the future.

Example Title	Related existing rules
Avoid Negations	ConfusingTernary
Simplify Boolean Expressions	CyclomaticComplexity
Avoid NullPointerException in Conditionals	LiteralsFirstInComparisons, BrokenNullCheck, EqualsNull , MisplacedNullCheck, UnusedNullCheckInEquals
Replace Magic Numbers with Constants	Checkstyle MagicNumber
Avoid Collection Modification During Iteration	n/a
Avoid Compute-Intense Operations During Iteration	OperationWithHighCostInLoop (Apex), AvoidInstantiatingObjectsInLoop
Favor Format Over Concatenation	n/a
Favor Java API Over DIY	AvoidArrayLoops, UseArraysAsList
Remove Superfluous Comments	Checkstyle TodoComment, JAutodoc
Remove Commented-Out Code	n/a
Avoid Abbreviations	Checkstyle AbbreviationAsWordInName
Fail Fast	n/a
Expose Cause in Variable	AvoidThrowingRawExceptionTypes
Always Check Type Before Cast	n/a
Expected Before Actual Value	n/a
Use Reasonable Tolerance Values	n/a
Let JUnit Handle Exceptions	n/a
Favor Standalone Tests	n/a
Split Method with Boolean Parameters	n/a
Split Method with Optional Parameters	n/a
Avoid Leaking References	MethodReturnsInternalArray
Avoid Return Null	ReturnEmptyCollectionRatherThanNull
Favor Lambdas Over Anonymous Classes	n/a
Avoid Exceptions in Streams	n/a
Favor Optional Over Null	n/a
Avoid Optional Fields or Parameters	n/a
Use Optionals as Streams	n/a

Difficult rules

These are the examples, for which is sounds very difficult to implement a reliable PMD rule for detection.

Example Title
Ensure Code Symmetry
Favor Enums Over Integer Constants
Group With New Lines
Replace Comments with Constants
Replace Comments with Utility Methods
Document Implementation Decisions
Document Using Examples
Structure JavaDoc of Packages
Structure JavaDoc of Classes and Interfaces
Structure JavaDoc of Methods
Structure JavaDoc of Constructors
Avoid Meaningless Terms
Use Domain Terminology
Explain Cause in Message
Structure Tests Into Given-When-Then
Describe Your Tests
Parametrize Your Tests
Cover the Edge Cases
Favor Immutable Over Mutable State
Combine State and Behavior
Favor Functional Over Imperative Style
Avoid Side Effects
Use Collect for Terminating Complex Streams
Speed Up Your Program
Know Your Falsehoods

Repairing broken EDID of monitor

Sun, 26 Jan 2025 00:00:00 +0100

Introduction

Recently I tried out a HDMI stick at my monitor. It’s the same monitor, I repaired back then in Repairing Hyundai W240D monitor. After I was finished with the HDMI stick I reconnected my laptop again, but - uh oh: The resolution was not recognized anymore.

I was presented with an ugly 640x480 resolution. The first things like plugging the monitor in again or switching it off and on again didn’t help. I could select between 640x480, 800x600 and 1024x768. And the monitor was recognized as “unnamed monitor”.

So, looking at the logs - maybe I see some error. And I got lucky: The kernel logged the following line:

kernel: EDID block 0 (tag 0x00) checksum is invalid, remainder is 66

That’s strange. Seems like the monitor doesn’t identify itself anymore. Searching the interwebs brought me to a wonderful page, which exactly explains, what I need to do: https://wiki.debian.org/RepairEDID.

The following is a step-by-step description of what I did to get it working again. In case, it happens again, I hopefully remember that this issue can be fixed.

The commands

Reading the EDID EEPROM

First step is to read out the EDID EEPROM. We use two kernel modules to get access to this:

$ modprobe i2c-dev
$ modprobe eeprom

Then we can search for EDID EEPROMs. These have the property, that they are at address 0x0050 on the i2c bus:

$ ls -al /sys/bus/i2c/devices/*-0050/eeprom
-r--r--r-- 1 root root 256 26. Nov 12:09 /sys/bus/i2c/devices/6-0050/eeprom
-r--r--r-- 1 root root 256 26. Nov 12:09 /sys/bus/i2c/devices/7-0050/eeprom

These are the two EEPROMs I have. Both are 256 bytes big. One is for the internal laptop display, the other is the broken monitor. We can verify the contents of this EEPROMs by using parse-edid:

$ apt install read-edid
$ cat /sys/bus/i2c/devices/6-0050/eeprom|parse-edid
Checksum Correct

Section "Monitor"
	Identifier ""
	ModelName ""
	VendorName "SHP"
	# Monitor Manufactured week 50 of 2014
	# EDID version 1.4
	# Digital Display
	DisplaySize 290 170
	Gamma 2.20
	Option "DPMS" "false"
	Modeline 	"Mode 0" -hsync -vsync 
EndSection
$ cat /sys/bus/i2c/devices/7-0050/eeprom|parse-edid
WARNING: Checksum failed
Trying to continue...
Section "Monitor"
	Identifier "W240D DVI"
	ModelName "W240D DVI"
	VendorName "HIT"
	# Monitor Manufactured week 30 of 2007
	# EDID version 1.3
	# Digital Display
	DisplaySize 520 320
	Gamma 2.20
	Option "DPMS" "false"
	Horizsync 30-83
	VertRefresh 59-76
	# Maximum pixel clock is 180MHz
	#Not giving standard mode: 1920x1200, 60Hz
	#Not giving standard mode: 1600x1200, 60Hz
	#Not giving standard mode: 1680x1050, 60Hz
	#Not giving standard mode: 1280x1024, 60Hz

	#Extension block found. Parsing...
	Modeline 	"Mode 11" -hsync -vsync 
	Modeline 	"Mode 0" +hsync -vsync 
	Modeline 	"Mode 1" +hsync +vsync 
	Modeline 	"Mode 2" 148.500 1920 2008 2052 2200 1080 1084 1089 1125 +hsync +vsync
	Modeline 	"Mode 3" 74.250 1920 2448 2492 2640 1080 1082 1089 1125 +hsync +vsync interlace
	Modeline 	"Mode 4" 74.250 1280 1720 1760 1980 720 725 730 750 +hsync +vsync
	Modeline 	"Mode 5" 27.000 720 732 796 864 576 581 586 625 -hsync -vsync
	Modeline 	"Mode 6" 74.250 1920 2008 2052 2200 1080 1082 1087 1125 +hsync +vsync interlace
	Modeline 	"Mode 7" 74.250 1280 1390 1420 1650 720 725 730 750 +hsync +vsync
	Modeline 	"Mode 8" 27.027 720 736 798 858 480 489 495 525 -hsync -vsync
	Modeline 	"Mode 9" 27.027 720 736 798 858 480 489 495 525 -hsync -vsync
	Modeline 	"Mode 10" 25.200 640 656 752 800 480 490 492 525 -hsync -vsync
	Modeline 	"Mode 12" +hsync +vsync interlace
	Modeline 	"Mode 13" +hsync +vsync 
	Modeline 	"Mode 14" -hsync -vsync 
	Option "PreferredMode" "Mode 11"
EndSection

The second one is the problematic one, as indicated by “WARNING: Checksum failed”. But most data seems still to be there, so not too bad. The EEPROM is still there and didn’t loose all the data.

A hexdump and the difference

Let’s have a look at the hexdump:

$ cat /sys/bus/i2c/devices/7-0050/eeprom|hd
00000000  00 ff ff ff ff ff ff 00  21 34 03 7d 43 41 32 01  |........!4.}CA2.|
00000010  1e 11 01 03 80 34 20 78  0a ef 95 a3 54 4c 9b 26  |.....4 x....TL.&|
00000020  00 50 54 ad cf 00 d1 00  a9 40 b3 00 81 80 01 01  |.PT......@......|
00000030  01 01 01 01 01 01 28 3c  80 a0 70 b0 23 40 30 20  |......(<..p.#@0 |
00000040  36 00 08 40 21 00 00 1a  48 3f 40 30 62 b0 32 40  |6..@!...H?@0b.2@|
00000050  40 c0 13 00 08 40 21 00  00 1e 00 00 00 fd 00 3b  |@....@!........;|
00000060  4c 1e 53 12 00 0a 20 20  20 20 20 20 00 00 00 fc  |L.S...      ....|
00000070  00 57 32 34 30 44 20 44  56 49 0a 20 20 20 01 33  |.W240D DVI.   .3|
00000080  02 03 1c 71 49 90 14 13  12 05 04 03 02 01 23 09  |...qI.........#.|
00000090  07 07 83 01 00 00 65 03  0c 00 10 00 8c 0a d0 90  |......e.........|
000000a0  20 40 31 20 0c 40 55 00  13 8e 21 00 00 18 01 1d  | @1 .@U...!.....|
000000b0  80 18 71 1c 16 20 58 2c  25 00 c4 8e 21 00 00 9e  |..q.. X,%...!...|
000000c0  01 1d 00 72 51 d0 1e 20  6e 28 55 00 c4 8e 21 00  |...rQ.. n(U...!.|
000000d0  00 1e 8c 0a d0 8a 20 e0  2d 10 10 3e 96 00 c4 8e  |...... .-..>....|
000000e0  21 00 00 18 8c 0a d0 8a  20 e0 2d 10 10 3e 96 00  |!....... .-..>..|
000000f0  13 8e 21 00 00 18 00 00  00 00 00 00 00 00 00 c4  |..!.............|

These are the 256 bytes of the EEPROM for which the checksum is not correct.

Luckily, I also backed up the EDID data when I repaired the monitor the last time. This backup is in the form of a blog post, see Repairing Hyundai W240D monitor. Now, taking the correct EEPROM content at that time:

and cat /sys/class/drm/card0/card0-DP-1/edid | hd:

00000000  00 ff ff ff ff ff ff 00  21 34 03 7d 43 41 32 01  |........!4.}CA2.|
00000010  1e 11 01 03 80 34 20 78  0a ef 95 a3 54 4c 9b 26  |.....4 x....TL.&|
00000020  0f 50 54 ad cf 00 d1 00  a9 40 b3 00 81 80 01 01  |.PT......@......|
00000030  01 01 01 01 01 01 28 3c  80 a0 70 b0 23 40 30 20  |......(<..p.#@0 |
00000040  36 00 08 40 21 00 00 1a  48 3f 40 30 62 b0 32 40  |6..@!...H?@0b.2@|
00000050  40 c0 13 00 08 40 21 00  00 1e 00 00 00 fd 00 3b  |@....@!........;|
00000060  4c 1e 53 12 00 0a 20 20  20 20 20 20 00 00 00 fc  |L.S...      ....|
00000070  00 57 32 34 30 44 20 44  56 49 0a 20 20 20 01 33  |.W240D DVI.   .3|
00000080  02 03 1c 71 49 90 14 13  12 05 04 03 02 01 23 09  |...qI.........#.|
00000090  07 07 83 01 00 00 65 03  0c 00 10 00 8c 0a d0 90  |......e.........|
000000a0  20 40 31 20 0c 40 55 00  13 8e 21 00 00 18 01 1d  | @1 .@U...!.....|
000000b0  80 18 71 1c 16 20 58 2c  25 00 c4 8e 21 00 00 9e  |..q.. X,%...!...|
000000c0  01 1d 00 72 51 d0 1e 20  6e 28 55 00 c4 8e 21 00  |...rQ.. n(U...!.|
000000d0  00 1e 8c 0a d0 8a 20 e0  2d 10 10 3e 96 00 c4 8e  |...... .-..>....|
000000e0  21 00 00 18 8c 0a d0 8a  20 e0 2d 10 10 3e 96 00  |!....... .-..>..|
000000f0  13 8e 21 00 00 18 00 00  00 00 00 00 00 00 00 c4  |..!.............|

Can you spot the difference? It’s actually only one single byte, that’s bad: it’s at offset 0x20: The bad one has 0x00 but it should be 0x0f. No idea, what cause this change…

Writing to the EEPROM

First step: Let’s backup the current state of the EEPROM in case we mess up, it will be handy:

$ cat /sys/bus/i2c/devices/7-0050/eeprom > hyundai-monitor-bad-edid.bin

Next step: We will use the i2c utils to write to it. So, let’s first use this tool to dump the content, so that we are sure, we have the correct EEPROM… In the docs I read, there are many warnings about writing to the wrong EEPROM - e.g. the DIMMs also have a EEPROM which store the timing data. If you accidentally write to the wrong EEPROM and write to the DIMMs of your laptop, you practically have bricked your laptop. So, better to be sure.

It should be i2c bus number 7 and address 0x50:

$ i2cdump 7 0x50
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-7, address 0x50, mode byte
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 00 ff ff ff ff ff ff 00 21 34 03 7d 43 41 32 01    ........!4?}CA2?
10: 1e 11 01 03 80 34 20 78 0a ef 95 a3 54 4c 9b 26    ?????4 x????TL?&
20: 00 50 54 ad cf 00 d1 00 a9 40 b3 00 81 80 01 01    .PT??.?.?@?.????
30: 01 01 01 01 01 01 28 3c 80 a0 70 b0 23 40 30 20    ??????(<??p?#@0 
40: 36 00 08 40 21 00 00 1a 48 3f 40 30 62 b0 32 40    6.?@!..?H?@0b?2@
50: 40 c0 13 00 08 40 21 00 00 1e 00 00 00 fd 00 3b    @??.?@!..?...?.;
60: 4c 1e 53 12 00 0a 20 20 20 20 20 20 00 00 00 fc    L?S?.?      ...?
70: 00 57 32 34 30 44 20 44 56 49 0a 20 20 20 01 33    .W240D DVI?   ?3
80: 02 03 1c 71 49 90 14 13 12 05 04 03 02 01 23 09    ???qI?????????#?
90: 07 07 83 01 00 00 65 03 0c 00 10 00 8c 0a d0 90    ????..e??.?.????
a0: 20 40 31 20 0c 40 55 00 13 8e 21 00 00 18 01 1d     @1 ?@U.??!..???
b0: 80 18 71 1c 16 20 58 2c 25 00 c4 8e 21 00 00 9e    ??q?? X,%.??!..?
c0: 01 1d 00 72 51 d0 1e 20 6e 28 55 00 c4 8e 21 00    ??.rQ?? n(U.??!.
d0: 00 1e 8c 0a d0 8a 20 e0 2d 10 10 3e 96 00 c4 8e    .????? ?-??>?.??
e0: 21 00 00 18 8c 0a d0 8a 20 e0 2d 10 10 3e 96 00    !..????? ?-??>?.
f0: 13 8e 21 00 00 18 00 00 00 00 00 00 00 00 00 c4    ??!..?.........?

So, we see the same data. And at offset 0x20 there is still the wrong byte 0x00.

Now let’s fix it:

$ i2cset -y 7 0x50 0x20 0x0F

This sets the byte at offset 0x20 to 0x0F for the EEPROM on bus 7 and address 0x50.

Dump it again, to verify it worked:

$ i2cdump 7 0x50
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-7, address 0x50, mode byte
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 00 ff ff ff ff ff ff 00 21 34 03 7d 43 41 32 01    ........!4?}CA2?
10: 1e 11 01 03 80 34 20 78 0a ef 95 a3 54 4c 9b 26    ?????4 x????TL?&
20: 0f 50 54 ad cf 00 d1 00 a9 40 b3 00 81 80 01 01    ?PT??.?.?@?.????
30: 01 01 01 01 01 01 28 3c 80 a0 70 b0 23 40 30 20    ??????(<??p?#@0 
40: 36 00 08 40 21 00 00 1a 48 3f 40 30 62 b0 32 40    6.?@!..?H?@0b?2@
50: 40 c0 13 00 08 40 21 00 00 1e 00 00 00 fd 00 3b    @??.?@!..?...?.;
60: 4c 1e 53 12 00 0a 20 20 20 20 20 20 00 00 00 fc    L?S?.?      ...?
70: 00 57 32 34 30 44 20 44 56 49 0a 20 20 20 01 33    .W240D DVI?   ?3
80: 02 03 1c 71 49 90 14 13 12 05 04 03 02 01 23 09    ???qI?????????#?
90: 07 07 83 01 00 00 65 03 0c 00 10 00 8c 0a d0 90    ????..e??.?.????
a0: 20 40 31 20 0c 40 55 00 13 8e 21 00 00 18 01 1d     @1 ?@U.??!..???
b0: 80 18 71 1c 16 20 58 2c 25 00 c4 8e 21 00 00 9e    ??q?? X,%.??!..?
c0: 01 1d 00 72 51 d0 1e 20 6e 28 55 00 c4 8e 21 00    ??.rQ?? n(U.??!.
d0: 00 1e 8c 0a d0 8a 20 e0 2d 10 10 3e 96 00 c4 8e    .????? ?-??>?.??
e0: 21 00 00 18 8c 0a d0 8a 20 e0 2d 10 10 3e 96 00    !..????? ?-??>?.

Looks good! Now we have again “0x0f” at offset “0x20”.

If we now parse the EDID data again:

$ cat /sys/bus/i2c/devices/7-0050/eeprom|parse-edid
Checksum Correct

Section "Monitor"
	Identifier "W240D DVI"
	ModelName "W240D DVI"
	VendorName "HIT"
	# Monitor Manufactured week 30 of 2007
	# EDID version 1.3
	# Digital Display
	DisplaySize 520 320
	Gamma 2.20
	Option "DPMS" "false"
	Horizsync 30-83
	VertRefresh 59-76
	# Maximum pixel clock is 180MHz
	#Not giving standard mode: 1920x1200, 60Hz
	#Not giving standard mode: 1600x1200, 60Hz
	#Not giving standard mode: 1680x1050, 60Hz
	#Not giving standard mode: 1280x1024, 60Hz

	#Extension block found. Parsing...
	Modeline 	"Mode 11" -hsync -vsync 
	Modeline 	"Mode 0" +hsync -vsync 
	Modeline 	"Mode 1" +hsync +vsync 
	Modeline 	"Mode 2" 148.500 1920 2008 2052 2200 1080 1084 1089 1125 +hsync +vsync
	Modeline 	"Mode 3" 74.250 1920 2448 2492 2640 1080 1082 1089 1125 +hsync +vsync interlace
	Modeline 	"Mode 4" 74.250 1280 1720 1760 1980 720 725 730 750 +hsync +vsync
	Modeline 	"Mode 5" 27.000 720 732 796 864 576 581 586 625 -hsync -vsync
	Modeline 	"Mode 6" 74.250 1920 2008 2052 2200 1080 1082 1087 1125 +hsync +vsync interlace
	Modeline 	"Mode 7" 74.250 1280 1390 1420 1650 720 725 730 750 +hsync +vsync
	Modeline 	"Mode 8" 27.027 720 736 798 858 480 489 495 525 -hsync -vsync
	Modeline 	"Mode 9" 27.027 720 736 798 858 480 489 495 525 -hsync -vsync
	Modeline 	"Mode 10" 25.200 640 656 752 800 480 490 492 525 -hsync -vsync
	Modeline 	"Mode 12" +hsync +vsync interlace
	Modeline 	"Mode 13" +hsync +vsync 
	Modeline 	"Mode 14" -hsync -vsync 
	Option "PreferredMode" "Mode 11"
EndSection

Now we get again “Checksum Correct”.

Testing

Ok, next step: Replug display cable. Then something unexpected happend: The monitor didn’t display anything at all. In the OSD (on screen display) menu, it said the resolution was “1921x1200”, which doesn’t work of course.

I simply chose the option “recall” in the OSD, which I think is kind of a reset to factory defaults. And voila - the monitor works again.

Cleanup

Since writing to i2c devices can be dangerous, it’s better to unload the i2c-dev module again afterwards:

$ modprobe -r i2c-dev
$ modprobe -r eeprom

Further thoughts

According to the specification in wikipedia of Extended Display Identification Data the wrong byte at offset 0x20 is part of the chromatic values. This means, that this broken EDID was probably an accident, as why would the HDMI stick change this? I’ve read, that sometimes these EEPROMs can be messed up, when the monitor is started or plugged in or out. So it’s just bad luck.

The original source was this EDID Repair which mentions another project: EDID Inserter. This sounds like an interesting project. In the office, I also use a KVM switch and sometimes it takes time until the monitor shows an image. Sometimes the monitor is not detected at all. This might be an explanation: When the KVM switch changes the display, it might disconnect also the DDC (Display Data Channel) which transports the EDID. And this inserter tries to solve the problem by connecting a fixed EEPROM directly so that the graphics card also sees a monitor connected and doesn’t switch the video mode.

Today, there seem to be commercial solutions available, just search for “EDID Emulator”. I would need such a thing for Display Port (not HDMI), but that’s maybe not so easy, as the i2c lines are maybe not directly exposed, see DDC over Display Port.

There are maybe other solutions possible - at least for linux. Search for “Linux Bypass EDID” or https://www.kernel.org/doc/Documentation/EDID/HOWTO.txt and look at this very old question How can I make Linux behave better when EDID is unavailable?.

Maybe it’s worth a try?

iCal and Google Calendar synchronization

Sun, 15 Dec 2024 00:00:00 +0100

This post is a about a small detail I learned while investigating, why the changes in my (self written) ical calendar export didn’t synchronize to Google calendar. In other words, the changes to an event, e.g. the updated start/end times or repeating options didn’t show up on Google.

The problem

So, caching definitely plays a role in that. Google itself states that changes to the imported ical calendar don’t show up immediately, but take up to 12 hours. Actually, I only find this statement in a community post Intergrated iCal calendar not syncing in 8 - 12 hours but not in the official documentation. But, that’s fine.

The scenario is, that I added a calendar via URL. This URL is a RFC 5545 compliant iCalendar resource that returns a couple of events, including some recurring events. The resource is valid according to the validator on icalendar.org. And indeed, the very first import in Google calendar looked correct. At some point, I noticed that new events didn’t appear anymore. The reason was, that I got the folding wrong (for the description) and the actual resource was invalid and therefore Google calendar didn’t add the new events. It just ignored the whole ical resource. Once I fixed this the new events started to appear again after a while. So far so good.

But some time later, I noticed, that a specific change to an event didn’t show up. The change was a different recurrent rule, so it was obvious that something was not working: The old event was from Monday to Thursday. But it has been changed to repeat weekly now from Monday to Friday. And the Friday calendar entry was missing in the Google calendar. Which is pretty obvious in the weekly view.

Caching?

I checked the access logs and there is a regular entry from a User Agent called “Google-Calendar-Importer” that accesses the calendar. The interval was between 4 hours and 7 hours. So, really within the boundary of 8-12 hours mentioned above. But I double checked, whether I return any caching related HTTP headers - and I didn’t. So in theory, it could very well be the case, that the whole resource is somehow cached. Although then I should see any access at all. But just to be sure, I added the headers Last-Modified and Cache-Control: max-age=3600, must-revalidate. That means, it can be cached, but it needs to be revalidated on every request (e.g. via a HEAD request). If the last modification timestamp didn’t change, the cached value can be used. I also implemented the possibility to execute a HEAD request, which wasn’t possible before.

I experimented a bit: When I change the URL and add some query parameter, in theory I should by-pass all caching infrastructure. A GET request with a query-parameter shouldn’t be cached at all. And indeed, this worked: The imported events from that modified URL appeared correctly.

I also tried to remove the subscription and add it back again (with the same URL). But that didn’t show success: It appeared the same way as before - the one event with the changed recurrent rule was still the old version. All other events showed up as before. I have even seen a fresh request in the access log. That means, that Google calendar actually requested the calender, but didn’t use it? Because there is another cache going on?

Caching single events… change management

It took me a while to notice that Google calendar seems to cache at the level of a single event. Clearly the calendar is requested and downloaded. New events appear in the calendar, but previously existing events are not updated.

That led to back to the specification again for iCalendar. There is actually a section on 3.8.7 Change Management Properties. This explained a lot: For each event, a couple of properties can be defined, such as “CREATED” (when the event was created), “DTSTAMP” (a timestamp of creation or last revision), “LAST-MODIFIED” (when the event was last revised). And lastly “SEQUENCE” - which is kind of a incrementing version counter for every revision of the event.

I checked, which of these properties I change when the event is changed. And first I only changed the “LAST-MODIFIED” time stamp. I kept the “DTSTAMP” the same, since I interpreted the specification in that way, that this time stamp says when this event was first published in the calendar (the METHOD I used is publish…). I experimented with that to use the same value as LAST-MODIFIED (since it could be, that the Google devs interpreted this section of the specification in a different way), but that didn’t work. But I still now return the last modification time stamp for that.

The last property is “SEQUENCE” and I implemented it that way, that I always returned “0”. And I never returned anything else. So, as last step, I implemented a simple change counter that gets incremented whenever the event is modified in some way. And - lo and behold - it immediately started working. Immediately meaning the next time Google calendar synchronized, which was a couple of hours later. But then, the event suddenly appeared additionally on Friday, as it was intended.

Note, that Google Calendar seems to manage the events based on a combination of the URL of the imported calendar and the event’s UID (Unique Identifier). That means, you really need to make sure, that the UID of an event doesn’t change and is stable, but still unique. Otherwise you might see duplicated events or you override existing events during import.

I would also expect, that this caching/event management that Google Calendar does in the background is actually independent of the current user’s calendar: If two (or more) users import a calendar from the same URL, they would need to import this calendar only once. But that’s just an idea and I didn’t try to verify this. If that’s true, then - in theory - you shouldn’t see additional access requests on the URL if more Google users import that calendar into their own. As Google would then do only one request for all users.

Summary

This means, that maybe some of the complaints about Google calendar not syncing in time or never syncing like Google Calendar does not sync URL-linked calendars within 12 hours as stated or Why is my Google Calendar sync slow or not updating at all? or Intergrated iCal calendar not syncing in 8 - 12 hours might actually be a result of a not compliant iCalendar source, that doesn’t properly maintain the change management properties. And Google Calendar might actually just work correctly.

There is of course one challenge still open, that’s asked here: How do I adjust sync frequency in google calendar?. When you import a iCalender via URL, you have no control over how often Google Calender checks for updated / new events. If it is not so time critical, then the stated “8-12 hours” time frame might be ok. If not, there is another solution available as GAS-ICS-Sync. This is a script that runs on Google’s App Script. I can be configured to run at an arbitrary interval and it actively downloads the iCalendar resources and uses Google Calendar API to add/update calendar events.

For me, it was enough to fix my custom built iCalendar generator to correctly export the events with the correct property SEQUENCE.