Using iptables to find spamming website in shared hosting environment

Recently I’ve received an abuse report from aol_dot_com.  After checked the email header attached with the report, and found that the spam sent from our shared hosting server.

Return-Path: <[email protected]>
Received: from mysharehosting.server (mysharehosting.server [12.34.56.78])
(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(No client certificate requested)
by mxserver.aol_dot_com(Internet Inbound) with ESMTPS id 7063B700000B1
for <redacted>; Fri, 12 Aug 2016 04:56:54 -0400 (EDT)
Received: (qmail 13675 invoked from network); 12 Aug 2016 16:56:49 +0800
Received: from localhost (HELO fakedomain) (127.0.0.1)
by localhost with ESMTPA; 12 Aug 2016 16:56:48 +0800
Date: Fri, 12 Aug 2016 08:56:48 +0000 (UTC)
From: info <[email protected]>
To: [email protected]_dot_com
Message-ID: <[email protected]>
Subject: Hiya.

But, the server has more than 500 websites, and the spams inject to mail queue through localhost port 25. The question is how to find the real victim website?

Luckily, it is pretty straightforward, thanks to the Netfilter team, we can log the uid who access the network resource using iptables.

  1. Create a custom chain BAND_SPAM and append log packet & reject. Then, a return rule in order to return to original the iptables chains when not match the condition.
    $ iptables -N BAND_SPAM
    $ iptables -A BAND_SPAM -p tcp -d 127.0.0.1 --dport 25 -j LOG --log-prefix "BAND_SPAM UID:" --log-uid
    $ iptables -A BAND_SPAM -p tcp -d 127.0.0.1 --dport 25 -j REJECT
    $ iptables -A BAND_SPAM -j RETURN
  2. Insert an OUTPUT rule, when gid=505 (our web user group), jump into BAND_SPAM chain
    $ iptables -I OUTPUT -m owner --gid-owner 505 -j BAND_SPAM

So, what happen after this? Here is the result when spammer attempt to send emails.

[[email protected] ~]# dmesg|grep BAND_SPAM
BAND_SPAM UID:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=8980 DF PROTO=TCP SPT=43892 DPT=25 WINDOW=32792 RES=0x00 SYN URGP=0 UID=10014
BAND_SPAM UID:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=8981 DF PROTO=TCP SPT=43892 DPT=25 WINDOW=32792 RES=0x00 SYN URGP=0 UID=10014
BAND_SPAM UID:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=44438 DF PROTO=TCP SPT=44452 DPT=25 WINDOW=32792 RES=0x00 SYN URGP=0 UID=10014
BAND_SPAM UID:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=44439 DF PROTO=TCP SPT=44452 DPT=25 WINDOW=32792 RES=0x00 SYN URGP=0 UID=10014

And we can find the web user from passwd file.
[[email protected] ~]# grep 10014 /etc/passwd
realvictim:x:10014:505::/vhosts/realvictim:/bin/false

Viola! We can call client and assist them to harden the website.

 

Kernel tuning – sysctl

Here is some of my kernel tuning

# Tune network memory
net.core.wmem_max = 4194304
net.core.rmem_max = 4194304
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_abort_on_overflow = 1
# Disable IPV6 if no use.
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

# Shorten nf_conntrack timeout values
net.netfilter.nf_conntrack_generic_timeout = 180
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 30
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 30
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 40
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 30
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 40
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 60
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 60
# Need more buckets in nf_conntrack
net.nf_conntrack_max = 200000

Completely disable swap on CentOS 7

Our services are no longer on bare metal, instead, new services are on VPS. Recently we have a problem on CE7 when reboot and it waiting for swap partition, which we have removed completely to free-up space.

Since hypervisors will manage disk IO and memory of VPS, default i/o scheduler & memory paging will double the I/O. Hypervisor allow overcommit on memory, and it will try to take free memory from other idled VPS, before it start paging to disk. Therefore, best practice are I/O scheduler set to noop and disable memory paging by turn it off & remove swap.

CE7 auto tuned i/o schedule to noop when detect itself are running under hypervisor, but it won’t turn off swap and allocate plenty amount of disk space for swap partition.

Turn off swap is simple.

$ sudo swapoff -a

and remove it to free-up space.

$ sudo lvremove -Ay /dev/centos/swap

of course reassign it to /dev/centos/root

$ sudo lvextend -l +100%FREE centos/root

But one point we are missing, grub2.cfg need to be regenerate, but modification needed before regenerate.

$ sudo vi /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
##GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet"
GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root crashkernel=auto rhgb quiet"
GRUB_DISABLE_RECOVERY="true"

$ sudo cp /etc/grub2.cfg /etc/grub2.cfg.bak
$ sudo grub2-mkconfig >/etc/grub2.cfg

Viola! No more swap partition waiting for next reboot!