How everything started…

  1. noticed I can’t access my VM via ssh;
  2. checked log, CPU usage 100% from Apr 5, and network bandwidth stayed at 2Mbps;

Quick fix

  1. reset server password with the tips provided here: “https://vultrcoupons.com/reset-root-password-vultr-vps/";
  2. checked auth.log with cat /var/log/auth.log | grep "Accepted" command, found entry:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    Apr  5 **:**:47 vultr sshd[23636]: Accepted password for root from *[/*Attacker IP From China*/]* port 59464 ssh2
    Apr  5 **:**:47 vultr sshd[23636]: pam_unix(sshd:session): session opened for user root by (uid=0)
    Apr  5 **:**:47 vultr systemd-logind[706]: New session 17302 of user root.
    Apr  5 **:**:48 vultr sshd[23638]: Failed password for root from *[/*Attacker IP From France*/]* port 48796 ssh2
    Apr  5 **:**:49 vultr sshd[23638]: Received disconnect from *[/*Attacker IP From France*/]* port 48796:11: Bye Bye [preauth]
    Apr  5 **:**:49 vultr sshd[23638]: Disconnected from authenticating user root *[/*Attacker IP From France*/]* port 48796 [preauth]
    Apr  5 **:**:50 vultr sshd[23666]: Invalid user nproc from *[/*Attacker IP From China*/]* port 59976
    Apr  5 **:**:50 vultr sshd[23666]: pam_unix(sshd:auth): check pass; user unknown
    Apr  5 **:**:50 vultr sshd[23666]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=*[/*Attacker IP From China*/]*
    Apr  5 **:**:51 vultr sshd[23666]: Failed password for invalid user nproc from *[/*Attacker IP From China*/]* port 59976 ssh2
    Apr  5 **:**:52 vultr sshd[23666]: Received disconnect from *[/*Attacker IP From China*/]* port 59976:11: Bye Bye [preauth]
    Apr  5 **:**:52 vultr sshd[23666]: Disconnected from invalid user nproc *[/*Attacker IP From China*/]* port 59976 [preauth]
    Apr  5 **:**:53 vultr chpasswd[23755]: pam_unix(chpasswd:chauthtok): password changed for root

Root of all evil

  1. checked files changed; found two directories: 1 and .configrc under root folder;

  2. inspected folder structure:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    .configrc/
    ├── a
    │   ├── a
    │   ├── bash.pid
    │   ├── dir.dir
    │   ├── kswapd0
    │   ├── run
    │   ├── stop
    │   └── upd
    ├── b
    │   ├── a
    │   ├── dir.dir
    │   ├── run
    │   ├── stop
    │   └── sync
    ├── cron.d
    └── dir2.dir

  3. The attacker first executes cronjobs inside cron.d file, specifically:

  • execute update every 12 hour or after reboot;

  • execute sync every Sunday 0805 a.m.; more on that later;

  • execute aptitude every month.

  1. To inspect, stop and delete all the cronjobs:

    1
    2
    3
    4
    cd /var/spool/cron/crontabs
    ls -lhat
    crontab -r
    ls

  2. Back to the sync script. The sync basically calls run under the same directory, which consists of a script converting a base64 string into a perl script and executes it as a process called rsync, then removes user’s ssh directory and add a ssh public key for a user called mdrfckr🤣 ;

6.
What’s the content of this Perl script? after the base 64 conversion, it seems it was obfuscated like this: eval unpack u=>q{/*random characters that don't make sense at all*/}

some google search later, discovered it has more functionalities like a bot net. alt text

7.
In the kswapd0, found json string like:

{

...

"pools": [
    {
        "coin": "monero",
        "algo": null,
        "url": "*xxx.xxx.xxx.xxx:80*",
        "user": "*User*",
        "pass": "x",
        "tls": false,
        "keepalive": true,
        "nicehash": true
    },]
}

combining the fact that this is the main program consuming all the CPU power plus the discovery of another shell includes some CPU checking and executing wrmsr Linux command to disable hardware prefetching on AMD and intel platforms, it’s clearly a cryptominer, specifically, it’s XMrig and the exploiter set up to pool in two servers in Netherland.

Summary:

  1. Use fail2ban to add multiple offenders to iptables drop table;
  2. Disable root login;
  3. Disable password login and use only SSH keys for login purpose; if you must use a user-password combination, use a robust password;
  4. If the server is already infected, migrate your important stuff to another server as soon as possible.