The XZ Backdoor on Linux and the importance of Open Source Software

What happened?

Sit down and buckle up, because this story is nuts.

In March of 2024, Andreas Freund a Microsoft employee was looking into some performance problems facing Debian-sid. He noticed that it was generating a large number of SSH connections which were eating up an abnormal amount of CPU resources. Upon further inspection he reported this issue to the OpenWall Project and it was then picked up by a number of software vendors for further inspection.

Lo and behold, Mr. Freund had discovered a hidden piece of code which had been planted in the latest versions 5.6.0 and 5.6.1 of XZ by the long time contributor Jian Tan.

The changes included in these versions compromise the security of the operating system giving an attacker complete administrative access.

Mr. Freund described the discovery of this major supply chain attack, which is akin to that of SolarWinds, as having "really required a lot of coincidences" and that "It seems to me we just got very, very lucky here."

What the XZ backdoor code does

The malicious code which was stealthily added to the XZ Utils project targets sshd which is an executable file that is used for making SSH connections remotely. If an attacker has access to the correct encryption key, they will then be able to add it to SSH login certificates, push them, and then execute them through the backdoored devices.

The link between the compression software XZ and SSH is due to a patch in Debian, among other distros, which creates a link between SSH and systemd. Systemd in turn is linked to XZ's LZMA library which creates the path from the XZ backdoor all the way to SSH access.

A detailed infographic explaining the exact steps in the process was made by Thomas Roccia:

What is particularly startling about this backdoor is how sophisticated it is and the steps taken by the attacker to embed themselves in the XZ project over the course of years in order to gain enough access to try and execute the attack. In the aftermath of this discovery the eyes of threat researchers have focused on one main contributor using the alias Jian Tan.

Who is Jian Tan?

At present, we do not know the true identity of the user operating under the name Jian Tan. But researchers have uncovered some leads. Jian Tan, operating under the username JiaT75, first appeared to be working on the XZ Utils code in 2021. This isn't a major cause for alarm, as XZ is an open source project and anyone can offer their help in improving it or squashing bugs. Throughout the years, Tan continued making small contributions to the project which increased their reputation and trust within the community built around the project.

In May of 2022, an unknown user using the alias Dennis Ens began complaining about a recent update, this was followed by another unknown figure using the name Jigar Kumar, to push the primary developer of XZ Utils to add a new maintainer (someone who builds and organizes the source code for open source projects) so that development can move faster. This pressure ultimately led to Jian Tan being added as a maintainer to the project.

In mid 2023 Tan requested that a public vulnerability checker be disabled, which would make it possible to introduce the backdoor code.

Screenshot of Tan's GitHub request to have ifunc disabled.

After continuing to work on the project, in February of 2024, Tan made the final version commits for XZ Utils 5.6.0 and 5.6.1 which contain the malicious backdoor.

This long-term social engineering attack gained the trust of an open source community which gave Tan the access they needed to try and pull off the attack. The amount of time invested and the high technical level of the backdoor itself is leading some security researchers to believe that there is a state sponsored actor behind it.

What's next for XZ?

The website for XZ Utils is fully in control of its primary developer Lasse Collin and Jian Tan has been completely removed from the project. The GitHub page for the project is back online and Collin has reported that they are currently reviewing the repository and once this is complete he plans to publish an article about the incident.

At the time of discovery the following Linux distributions were confirmed to be impacted by the backdoor:

• Arch Linux
• Debian testing, unstable and experimental versions 5.5.1alpha-01 through 5.6.1-1
• Fedora 40 Beta, Fedora 41, and Fedora Rawhide
• Kali Linux
• openSUSE Tumbleweed and openSUSE MicroOS.

Homebrew, the package manager for MacOS, was using 5.6.x but has reverted back to XZ Utils 5.4.6.

If you wish to check if you have a backdoored version of XZ Utils on your device you can run "xz --version" in your terminal. If version 5.6.0 or 5.6.1 is installed it is recommended that you revert back to version 5.4.x.

Open Source is the only way to provide secure software

We got extremely lucky. Had the XZ Utils project been privately maintained and Freund had been forced to submit a support ticket, we might be in a very different position today. The XZ backdoor incident is a perfect example of why open source software publishing provides major security benefits. Only by allowing everyone access to review source code, do we give security researchers the chance to discover these kind of attacks before things go horribly wrong.

Open Source Software boosts everyone's security:

• Allowing public security code review brings constructive criticism from experts of all expertise and backgrounds.
• A community can spot bugs and vulnerabilities faster than a closed source development team.
• Boosting trust through transparency lets end users know that they can trust in the products they choose to protect themselves and their data.

Open source software is a crucial aspect of all digital security. The XZ Utils project is used by enterprise products and without allowing for community scrutiny this insider attack may have gone unnoticed. With closed source software, only selected persons have the privilege of viewing the source code which ultimately makes overlooking changes like these more likely.

While large tech companies have lots of developers on their teams who are viewing the code, there’s never a guarantee that one employee is just like Jian Tan, trying to sneak in malicious code. Open source makes sure that more eyes are on the code – which means that in addition to strong security protocols and repeated code reviews, more eyes can spot more issues. According to Linus’ law: Given enough eyeballs, all bugs are shallow.

At Tuta we release our entire encryption protocols and client code open source to ensure our users that they can trust that we are delivering on the security promises we make. Not only is our encryption developed and tested by cryptography experts, but it is also made available for all security researchers to test and review. Trust is earned, and by publishing our code fully open source we are proving that Tuta provides the best email security available.

Stay safe and happy encrypting.