Background

A vulnerability was recently discovered in a programming library called Log4j.  This software library is used to provide logging functions to many popular Java applications.  The vulnerability, named Log4Shell by security researchers, can provide an attacker complete control over any system that has a Java application installed.

This vulnerability has received widespread publicity and is being actively attacked. Because the issue is with internal logging software, typical protections such as firewalls may not provide adequate protection for this attack.

Successful attacks could result in data breaches and ransomware. Urgent action is required to prevent damage.

 Response roadmap

The University’s Chief Information Security Officer is asking all teams to prioritize their response to this issue. The CISO has released the following  guideline:

  1. Any system exhibiting indications of compromise or with a high likelihood of compromise will be blocked at the network edge until it has been properly triaged and mitigated. This action will extend to enterprise applications and major systems.
  2. Any system exhibiting signs of compromise must be reinstalled/re-imaged. This vulnerability provides full remote code execution that can lead to full remote access to the system.

Any system exhibiting signs of compromise with high-risk data must be triaged assuming a data breach. Report any critical issues immediately to security.response@utoronto.ca and ciso@utoronto.ca.

Details

If an attacker sends a request against a server running a vulnerable version of Log4j, the server will make the request through the Java Naming and Directory Interface (JNDI), resulting in the execution of malicious code. Any attacker-controlled data logged through Log4j (such as username/search strings) can potentially result in Remote Code Execution (RCE).

Vulnerable versions (as of Dec. 17, 2021):

  • 2.0 <= Apache log4j <= 2.15.0
  • Log4j is embedded in many Java applications and frameworks which may need updates.

 

What should I do?

Updated Dec. 17, 2021

Identify

Is your software vulnerable? A number of different agencies are maintaining lists. See the links at the end of References.

Is your system vulnerable?

  1. For identifying the vulnerability on systems:
    • Linux/Unix/OSX systems:

For Linux/Unix/OSX systems, the Fenrir local scanning tool runs as a bash script. (https://github.com/Neo23x0/Fenrir). The tool scans for Indicators of Compromise (IOC) and also for vulnerable log4j versions.

    • Windows:

For Windows systems, there are a couple of Powershell scripts now available that search for JndiLookup.class in Jar files:

  1. Other options:
    • You can test if a server is vulnerable by using the canarytoken service:
        • visit https://canarytokens.org
        • choose the Log4shell token
        • enter the email address at which you wish to be notified
        • copy/use the returned string
    • Alternatively, you can search for vulnerable versions of the Log4j library on each server:find / -type f -iname log4j-core\*.jar
        (versions 2.0 to 2.15.0 are vulnerable)
    • War files
        • Some applications may have the library embedded in the WAR file. If you find the JndiLookup.class file in it, you are likely running Log4j 2.x and are also vulnerable to this.  You may be able to test for that via:
      strings MyJavaApp.war | grep 'org/apache/logging/log4j/core/lookup/JndiLookup.class'
    • Fenrir Log4Shell IOC scanner

New tool to scan Fenrir Log4Shell IOC scanner bash script: https://github.com/Neo23x0/Fenrir/releases/tag/v0.9.0

Basic characteristics:

      • Bash Script
      • No installation or agent needed
      • Uses common tools to extract attributes (e.g., md5sum, grep, stat in different modes)
      • Intended to run on any Linux / Unix / OS X with Bash – Low footprint
      • Ansible playbook with RAM drive solution
      • Smart exclusions (file size, extension, certain directories) speed up the scan process

Detect 

Have your systems been compromised / attacked?

Log4j attack attempts will show up in your logs with some variation of a string starting with

${jndi:

The following tools should help you search your logs for these indicators, but attackers are constantly changing their patterns in order to make it more difficult to detect.

  1. example search commands: https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b
  2. The Fenrir tool listed above in the Identify section (https://github.com/Neo23x0/Fenrir)

Please reach out to security.response@utoronto.ca if you are unsure how to look through your logs or find anything suspicious when you do.

Mitigate

Options:

  1.  Upgrade to Log4j v2.16.0
  2. Alternatively, you can remove the vulnerable JndiLookup.class within the Log4j core library entirely if you have an older version that does not support the mitigation flag and you are not using JNDI features.  Ensure you have a backup of the original log4j-core jar in case your application breaks and you need to revert!
find / -type f -iname log4j-core\*.jar -exec zip -q -d {​​​​​​​​​​​​​​​​​​}​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​ org/apache/logging/log4j/core/lookup/JndiLookup.class \;

Note:

A previous recommendation was to set the following

log4j2.formatMsgNoLookups = true

This is no longer deemed sufficient. This also affects log4j v2.15.0.  See  CVE-2021-45046

 

Firewall mitigation

Attacks against the Log4j vulnerability appear to follow a common pattern. The attacker first establishes that a system is vulnerable by scanning broad network ranges with the JNDI request, which then causes a vulnerable system to connect to a second server specified by the initial request to download a malicious payload.

If your local system or network firewall is configured so that systems are only allowed to make outbound requests to required servers (i.e., for downloading patches, NTP, DNS, etc.) and drop all other outbound connections, the server would be protected from responding to the initial request (which could lead an attacker to believe the initial attack was not successful and move on) as well as prevent the server from downloading the malicious payload.

Please review your outbound firewall rules and ensure that they drop all but traffic essential to the functioning of the services hosted on your servers.

 

References

Affected software lists (These will be incomplete, and constantly change)