Few people are aware of it, but in the last few days, an astute developer has thwarted a multi-year plan to install remote backdoors across the Internet.
Two years ago, a programmer known as Jia Tan (JiaT75) began helping develop a little-known compression library known as xz. For those who don’t know, software today is not monolithic. All software you use is built from a collection of tools. library, programming becomes easier. For example, most programmers do not need to write the details of a sorting algorithm. Because there is a library somewhere that performs the sorting. This allows programmers to focus on higher-level tasks, such as making the software actually do what the user wants. However, these libraries don’t just spring up out of thin air. Interested programmers should maintain and extend its functionality. This is a lot of work, so it’s a huge relief when new programmers volunteer to help.
However, Jia Tan’s motives were far from pure. xz is not used directly by many software, but is included by other libraries and used by other programs. In particular, sshd, the popular remote login service used by system administrators everywhere, can optionally include integration with third-party libraries, including the xz library. This is the configuration used by most server operating systems on the Internet. Therefore, as Jia Tan gained the trust of his xz maintainers, Jia indirectly also gained access to other parts of the operating system.
Software often includes code that tests itself. This is a way for software maintainers to avoid introducing obvious bugs when making changes. Test code is typically not incorporated into the final shipped software. Jia Tan embedded a backdoor in a file ostensibly used to test xz’s compression techniques. However, once he builds the system and tests it with a backdoor, the case is combined into the final software, which is then deployed. Backdoors work by overwriting standard encryption/decryption functions with their own versions of these functions.
The software was already integrated with test versions of several standard operating systems, so cutting-edge developers were already starting to use it. This was discovered because Andres Freund, a software engineer at Microsoft, was doing performance testing. To perform these tests, he was trying to minimize the CPU time used by other tools on the system. He noticed that sshd was using an abnormal amount of processing power and began investigating the cause of the slow performance. His analysis tools showed that sshd was spending a lot of time in the xz library, and further investigation revealed that the xz library was replacing some of the standard encryption and decryption functionality.
Thankfully, this was discovered before it had any major impact. No one knows who Jia Tan is, and it will probably never be found out. However, this makes software developers wary of the fact that they are willing to play a long game to get malicious software installed on anything, whether they are individuals or part of nation-state organizations.
People who use this incident to criticize open source software and say that this is part of the problem: If everyone didn’t rely on open source software, none of this would have happened. There are some too. But to be honest, I think it’s the opposite. The only reason this happened is found It was because of open source software. We were able to find and diagnose problems so quickly because we had developers who were familiar not only with our own software, but with all the code bases that the software ran on. I have used closed source software from time to time, and there is zero transparency in those situations. If someone were to introduce malicious code into critical closed-source software, no one would know or even have the ability to discover it. If discovered, the companies responsible will likely try to avoid disclosing the extent of the problem. However, because it is open source, there is a transparent record of everything that happened: every message, every commit, every artifact uploaded. Everything can be inspected and inspected, the full extent of the damage determined, and root causes discussed publicly so everyone can be on guard.
So, thanks to the watchful eye of open source software developers, we all but lost the internet this week.
