Information Systems:Network incident - April 4 2021: Loss of internet connectivity to main LAN

Summary

There was a network outage on April 4, 2021. However, there was no network breach as initially suspected. The cause of the outage was a downed port on the core switch, the other end of which is connected to the main LAN interface (172.30.16.1) of the Sophos firewall (node 1 of the HA-pair). The downed port appears to have "locked up" (physical layer). Despite the firewall being configured for hardware redundancy, no automatic failover to the other node occurred because a “link-down” event is a normal network occurrence (e.g. physically disconnecting the cable from a port), and this is how this event appeared to the firewall end of the connection.

Prior to and up until the time of outage, there was evidence of ongoing malformed packet attacks against the Sophos’ IPSEC VPN service. Though this traffic occurred on the WAN port, firewall events are forwarded to the syslog collector (Kiwi) via the LAN interface, so a correlating flood of packets from the attack was sent through the switch port that went down. Also, both WAN and LAN traffic pass through the same unit in the switch stack where the port went down. These represent a possible basis for correlation between the attack and the port failure, but no conclusive evidence could be found to support causation. The IPSEC VPN service has been turned off for the time being, since the configured site-to-site IPSEC VPN connections are not being used currently. Additional firewall rules protecting the VPN are recommended. A replacement of the core switch stack is highly recommended. No problems were found with the Sophos firewall.

Context

On Saturday (day prior to outage), I received an unusual alert (i.e. I can’t remember the last time I received this type of email) that Kiwi Syslog was receiving an alarmingly high level of alerts. The threshold is set to alert on over 250k messages per hour; the actual number: 250,838 messages. The breakdown of senders pinpointed two primary source IPs: the vCenter appliance (18.23), and the Sophos firewall (16.1):

The vCenter appliance was investigated first, and a minor issue was discovered to be the cause of excessive logging. This issue was resolved. The logging configuration of the appliance was also tweaked to prevent forwarding of less critical log levels. This drastically lowered the number of forwarded log entries thereafter.
The Sophos firewall appliance was investigated next, and the logs revealed an ongoing malformed packet attack attempt against the Sophos IPSEC VPN service. Provided is an example of this traffic:

Malformed packet attacks are intended to cause a Denial of Service or otherwise cause network malfunction in some capacity. Ample literature exists online regarding this type of threat. Here is a good summary from this Comodo article:

The malformed packet attack is another wide-spread type of DoS attack. The purpose of this attack is to send ill-formed packets to hosts and take advantage of the bad design of the code that processes the packets. Effects range from unacceptable degradation of performance to system crashes.

Since the firewall was not the primary cause of the alert, and since the packet rate (from the timestamps) was not significantly high to be deemed a DoS threat, this traffic was dismissed as to be investigated later.
The Kiwi Syslog alert threshold was lowered to 100k MPH, as that was deemed more appropriate than 250k.
The following day (day of the outage), another alert was received:

This suggested that the malicious traffic was increasing. Analyzing the logs confirmed this, and revealed different/new IPs being used in the attack.

Incident and suspicion of network breach

At 10PM on Sunday, several alerts were generated by our monitoring service, all for internet services hosted on-premises. Specifically, iNotes, Infonet, Traveler, and Web Orders were down.
At the same time, I was disconnected from VPN.
The firewall admin portal was still accessible through the WAN address. However, attempts to authenticate using my usual credentials now returned “Invalid credentials”. A few common admin credentials were also tried to no avail.
The workstations listed in FixMe.IT were now reporting as offline. Some machines like the RiseVision players and Loomis shipping terminals – connected to a different WAN - were online.
The 3CX PBX (also on another network) was still functional and reachable through the app. I was able to call warehouse staff extensions.

With the main network and critical services seemingly offline and unreachable, remote access disconnected, and given the malicious traffic observed earlier, a network attack was suspected. Also now being unable to access the firewall, a network breach was also suspected.

Response

With no ability to access the firewall remotely, and given the suspicion that the firewall itself had been compromised (i.e. no access), investigation had to be done on site.
The firewall’s physical network connections were immediately disconnected in case an attacker was still connected.
A possible cryptolocker attack was investigated by surveying the file server and other Windows shares. No files appeared to be compromised.
Websites and services were tested from within the LAN and were accessible. Infonet, wiki, telnet/Mocha.
The firewall was accessed via serial console. Login using the built-in shell user ‘loginuser’ was attempted and successful. Elevation to root using the known password was also successful.
A survey of the system (config, logs etc.) revealed no malicious activity or unauthorized entry.
The only piece of information that was helpful was this log entry which coincided with the time of the outage and pointed at a downed NIC (eth0), which was then determined to be the LAN interface:

There were no network activity lights on either end of this connection (neither the port on the switch nor the firewall). Some physical reconnections were attempted with no change. The switch admin page did not indicate any anomalies with the port. It reported the link as enabled (on this switch, equates to being up).
The LCD panel on node 1 was unresponsive (could not cycle through the displayed stats). At this point, node 1 was rebooted, causing a failover to node 2.
When node 2 became the master, alerts were received that the downed systems were back online.
The firewall uses Active Directory authentication, and the domain controllers are on the main LAN. The downed main LAN port prevented this authentication, explaining the inability to log in with a domain user account (AD SSO). With this knowledge, the password for the local admin user was changed and tested.
Investigation was deferred to the next morning.

Problem reoccurrence / additional issues

Jeremy performed some testing and reported that the systems were back online except for the phone system, which was now inaccessible. There is precedence for this behavior when firewall changes are made and/or there is a node failover. The usual fix is to reboot both nodes simultaneously to flush the connection tracking table (conntrack).
A pending firewall update was installed and the entire system was rebooted (remotely). Node 1 became the master following the reboot. At this time, the original problem reoccurred: VPN was disconnected, and logging in with a domain user was not possible. Logging in via local admin was, however, now possible.
Switching to Node 2 as the master fixed the problem, but made the phone system inaccessible. Switching to Node 1 again reproduced the problem.
The inaccessibility of the phone system with Node 2 as the master was traced to a VLAN misconfiguration on the correlating switch port. This was fixed, which brought the phone system back up. Node 2 was left as the active master as it yielded a fully operational network.
Investigation of Node 1 was deferred to the next morning.

Resolution/Outcome

The cause of the outage was ultimately traced to a "lock-up" of the switch port connected to the LAN interface of Node 1 of the Sophos. This was found by ruling out the cable and the Sophos LAN port (node 1, eth0). The port on the switch was unresponsive (appeared electrically dead) but finally negotiated a connection after a few attempts (blinking network activity LEDs). This points to a hardware or other layer 1 issue, (such as autonegotiation).
The original network connections between Sophos and the switch were restored, but Node 2 was left as the master.
IPSEC VPN was disabled. The previous attack traffic is no longer observed.
There was no impact to operations during the next business day.

Takeaways and recommendations

It is difficult to rule out the malformed packet attack as a contributing factor, although there is no direct evidence in the logs to support a correlation. An important point to consider is that the same unit of the switch stack (unit 4) where the downed port is located also contains the WAN ports - the ports that bridge the Telus router and Sophos WAN interfaces. Thus, the malformed-packet attack traffic is processed by the switch CPU in multiple ways (forwarding the actual packets, forwarding the log packets, as well as having to deal with the rest of the WAN and LAN traffic). Again, with not much stats on the switch, it is difficult to tell what happened on the switch at the time.
The core switch stack should be upgraded. Other than causing the outage, there are many limitations: no logging (or not easily accessible), sluggish and unintuitive interface that compromises network troubleshooting efforts, questionable performance (old hardware, not a true switch stack etc.), lacking features (e.g CDP, SPAN, VTP, RMON).
This situation proved that Sophos logins via Active Directory authentication are not 100% reliable (e.g. no cached logins). The local admin account credentials needs to be remembered as the fallback.
An out-of-band serial console server for the firewall is recommended. A secondary WAN (Shaw) was available as a means of access, but is currently not configured for backdoor administrative access to the main network.
With new switches, Link Aggregation can be configured between the LAN interface and the switch so that there is multiple-port redundancy for such a critical connection.
A DoS attack is not the same as a brute force attack. The latter entails system access if successful, the former does not. Panic makes you assume the worst sometimes. Being locked out of a system suddnely also does that to you.

Other Notes

Pictured is the syslog collector at the point of the outage. Traffic from the malformed-packet attack can be seen, which suddenly stops when the switch port to that interface goes down:

Syslog stats showing the spike in traffic leading up to the outage (this was after the first alert and fixes to vCenter; most of the alerts are from the Sophos). The 3 highest bars represent 6PM-8PM, and all 3 hours generated email alerts. Interestingly, between 9-10PM, the attack seems to have lessened. The baseline is shown on the right half of the graph, after the IPSEC VPN service was stopped. The average is approximately 3-5K messages per hour.

Source IPs and number of hits. Parsed from ipsec.log using SpectX: