Apache had to scramble to get ready to release patches for Log4Shell in early December 2021 when it made the situation public on December 9 last year. As a result, researchers quickly found edge cases and workarounds for the patches, and Apache was forced to release multiple iterations, adding to the confusion.
“This thing was everywhere, really everywhere,” said Jonathan Leitschuh, an open source security researcher. “Attackers jumped on it, the security community jumped on it, payloads flew all over the place.”
However, researchers say Apache’s overall response has been solid. Nalley adds that Apache has made changes and improvements in response to the Log4Shell saga and has hired dedicated staff to expand the security support it can provide to open source projects to catch bugs before they are submitted in code and to respond to incidents when necessary.
“In a short amount of time, two weeks, we had repairs, which is great,” says Nalley. “In some ways this is not a new situation for us, and I would like to say that we handled it perfectly. But the reality is, even at the Apache Software Foundation, this emphasized what a responsibility we have to everyone who uses our software.”
Going forward, the more concerning aspect of the situation is that, even a year later, about a quarter or more of Log4j downloads from the Apache repository Maven Central and other repository servers are still full of vulnerable versions of Log4j. In other words, software developers are still actively maintaining systems with vulnerable versions of the tool or even building new software that is vulnerable.
“The reality is that most of the time people choose a vulnerable open-source software component, a solution is already available,” said Brian Fox, co-founder and chief technology officer of the software vendor Sonatype, which manages Maven. Central and is also a remote Apache repository provider. “I’ve been at it for a long time and I’m jaded, but that’s really shocking. And the only explanation is that people really don’t understand what’s in their software.” .”
Fox says that after the initial battle to tackle Log4Shell, version downloads in Maven Central and other repositories hit a shelf where about 60 percent of downloads were patched versions and 40 percent were still vulnerable versions. In the past three months, Fox and Apache’s Nalley say they’ve seen the numbers fall to a split of about 75/25 percent for the first time. However, as Fox puts it, “After a year, a quarter of downloads are still pretty awful.”
“Some people feel that Log4j was a big wake-up for the industry, a collective freak-out and awakening,” he says. “And it helped us really get the message out about software supply chain security because people were no longer in denial. What we were all talking about was now real. We lived it all. But the peer pressure from Log4j alone should have forced everyone to upgrade, so if we can’t get this one to 100 percent, what about all the others?”
For security researchers, the question of how to tackle the long tail of a vulnerability is always there. And the problem applies not only to open-source software, but also to proprietary systems. Think how many years it took to get the last 10 percent of Windows users off XP.
“With these worst case scenarios – black swan events in open source – you just know they’re going to keep happening because the community has gotten much better at responding, but the pace of open source development is even faster,” Lorenc of ChainGuard says. “So we need to find the balance between prevention and mitigation, and keep coming up with efforts to reduce the frequency as much as possible. It’s like The Simpsons meme when Bart says “This is the worst day of my life.” And Homer says no, “The worst day of your life.” so far.’”
