On March 29, 2024, Andres Freund reported a backdoor in the xz 5.6.0 and 5.6.1 release tarballs. He had not begun with a supply-chain investigation. He was chasing excess CPU use and roughly half a second of latency in SSH authentication on a Debian Sid system.
His original oss-security message is still the best place to start. It separates what he had reproduced from what remained uncertain and shows the commands that moved the investigation forward.
The incident matters because the attacker crossed several trust boundaries without having to compromise them all. Public Git history, generated release artifacts, distribution packaging, and the process that eventually loaded liblzma were distinct stages. Defenders who verified only one stage could miss what happened in the next.
Scope before mechanism
The upstream maintainer’s incident page confirms the narrow source scope: the xz 5.6.0 and 5.6.1 release tarballs contained the backdoor and were created and signed by Jia Tan. The final trigger code was not present in the Git repository. Some suspicious and payload-bearing test artifacts were in Git, but they were inert without the tarball-only build logic. Tukaani’s incident record preserves those distinctions.
An installed xz version was not, by itself, enough for remote exploitation. The malicious build path checked aspects of the build environment, architecture, compiler and linker behavior, and packaging context. The resulting liblzma then had to be loaded into a useful process. The path Freund observed involved a distribution-patched OpenSSH build that loaded libsystemd, which in turn loaded liblzma.
flowchart LR
G[Reviewed Git tree] --> T[Generated release tarball]
T --> P[Distribution package build]
P --> L[Installed liblzma]
L --> S[Selected sshd process]
T -. tarball-only build logic .-> PThat is why statements such as “all Linux systems with xz 5.6.1 were remotely exploitable” are wrong. The affected set was the intersection of a tainted source artifact, a compatible build path, an affected package, and runtime reachability from sshd.
How the payload entered the build
The release tarball contained m4/build-to-host.m4 logic that was absent from the corresponding public Git history. During a matching build, that logic decoded several stages from files presented as test data and produced a prebuilt object that was incorporated into liblzma.
The shell path was deliberately difficult to read. It used standard command-line tools, character substitution, AWK, decompression, and byte carving. Gynvael Coldwind’s analysis, preserved on oss-security, walks through the stages and the extraction of the final object.
The attacker did not hide a malicious C file in the repository. Instead, the attacker placed innocuous-looking ingredients in version control and supplied the assembly instructions through the generated release artifact. A reviewer comparing ordinary source files could miss the behavior; a reviewer comparing the release tarball with a clean artifact generated from the tag had a chance to see the extra build logic.
The activation conditions reduced accidental discovery. A normal build that did not match the selected environment could produce an ordinary library. Official distribution infrastructure was one intended setting, but it was not a magical requirement: another build with matching conditions could also activate the path.
From liblzma to pre-authentication code execution
On the affected systems, sshd did not link to liblzma because SSH needs compression from xz. It arrived through another library, commonly a distribution’s systemd integration:
sshd -> libsystemd -> liblzmaThe injected object used glibc’s indirect-function machinery and dynamic-linker behavior to redirect the RSA signature-verification path. Freund later showed execution flowing from OpenSSH’s RSA verification into code inside the compromised liblzma, then back into libcrypto. His follow-up trace documents that path.
Subsequent analysis found a gated remote-code-execution mechanism rather than a generic authentication bypass. The malicious code inspected attacker-controlled RSA public-key material reachable before authentication, extracted and decrypted a payload, verified it against a fixed Ed448 public key, and could pass a command to system(). The private key corresponding to that Ed448 public key was the attacker’s authorization secret. A random client could not activate the command path merely by knowing the backdoor existed. The early reverse-engineering summary is archived in the oss-security discussion.
This distinction matters. The compromised library contained an RCE facility, but public reporting did not establish that the operator’s private activation key was recovered or that the backdoor was used against victims.
Who actually received affected packages
The compromised releases reached development, testing, and rolling channels before discovery. Exposure varied by distribution and package revision.
Fedora reported that Rawhide users were likely to have received the tainted package and that Fedora 40 pre-release users might have received it from updates-testing. Fedora 38 and 39 were not affected, and the compromised build never became a Fedora 40 final stable update. Fedora’s March 29 advisory records the package revisions and rollback.
Other distributions had their own combinations of package version, build flags, and OpenSSH linkage. A responsible fleet check therefore starts with the vendor’s advisory and package revision, not a universal shell one-liner.
Useful local evidence includes:
xz --version
dpkg-query -W xz-utils liblzma5 2>/dev/null || true
rpm -q xz xz-libs 2>/dev/null || true
ldd "$(command -v sshd)" 2>/dev/null | grep -E 'libsystemd|liblzma' || trueInterpret these together:
xz --versionidentifies an upstream version but may not reveal a distribution’s reverted or patched revision.- The package manager records the exact distribution build.
lddcan show the dependency path on a dynamically linked binary, but absence of a directliblzmaline is not a complete historical compromise assessment.- A clean package today does not prove that an affected package was never installed earlier.
For a host that actually ran a distribution-confirmed tainted build, replacing the package is necessary but not enough to prove the host was never accessed. Follow the vendor’s incident guidance. Fedora advised potentially affected pre-release users to update and, where compromise was plausible, consider reinstalling rather than treating a downgrade as forensic proof of safety.
Why signatures did not save the release
The malicious tarballs were official artifacts signed by the person authorized to release them. The signature answered one question correctly: the artifact came from that signing identity. It could not answer whether the signer was trustworthy or whether the tarball corresponded to reviewed source.
Digital signatures did their job here; provenance still has several separate claims:
- identity: who authorized the artifact;
- source: which reviewed tree and generated inputs it represents;
- build: which process and environment produced it;
- distribution: which repository delivered it;
- deployment: which process loaded it.
The xz release had evidence for the first claim and a dangerous gap in the second.
What reproducible builds would and would not catch
“Reproducible builds would have caught xz” is incomplete. Rebuilding the same malicious release tarball can reproduce the same malicious output perfectly. Reproducibility tells you whether declared inputs deterministically produce an output. It does not tell you that the inputs are the ones reviewers intended.
The useful comparison is between independently established source origins. A pipeline can generate a release artifact from a signed Git tag, compare that result with the published source archive, and then reproduce the binary package from the verified source. A mismatch between the tag-derived artifact and the release tarball is the evidence that matters here.
Tukaani’s later review notes explicitly compare Git tags and tarballs. The broader lesson is to make generated source archives reproducible and traceable, not merely to rebuild whichever archive arrived at the package builder.
What the incident changed for me
The elegant part of the attack was the attacker’s understanding of where the ecosystem stopped comparing evidence, not any single obfuscation trick.
Maintainers reviewed commits, release consumers verified signatures, package builders trusted release tarballs, and operators trusted signed packages. Each practice was reasonable in isolation. The attacker lived in the seams between them.
The controls I would now ask for on a security-sensitive dependency are correspondingly plain:
- at least two people can review and authorize releases;
- generated tarballs are reproducible from a precise source revision;
- package builders verify source provenance before building;
- build logs and inputs are retained for independent comparison;
- privileged processes minimize optional dependency chains;
- incident inventories preserve package history, not only current state.
An SBOM helps locate liblzma after disclosure. A signature helps identify the release authority. Reproducibility helps compare build outputs. None substitutes for the others.
Freund found the backdoor because a performance anomaly did not make sense and he kept following it. The engineering response is to connect the evidence across the boundaries the attacker crossed, not to hope for another observer with that same luck.