Using iptables to find spamming website in shared hosting environment

Recently I’ve received an abuse report from aol_dot_com. ¬†After checked the email header attached with the report, and found that the spam sent from our shared hosting server.

Return-Path: <[email protected]>
Received: from mysharehosting.server (mysharehosting.server [12.34.56.78])
(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(No client certificate requested)
by mxserver.aol_dot_com(Internet Inbound) with ESMTPS id 7063B700000B1
for <redacted>; Fri, 12 Aug 2016 04:56:54 -0400 (EDT)
Received: (qmail 13675 invoked from network); 12 Aug 2016 16:56:49 +0800
Received: from localhost (HELO fakedomain) (127.0.0.1)
by localhost with ESMTPA; 12 Aug 2016 16:56:48 +0800
Date: Fri, 12 Aug 2016 08:56:48 +0000 (UTC)
From: info <[email protected]>
To: [email protected]_dot_com
Message-ID: <[email protected]>
Subject: Hiya.

But, the server has more than 500 websites, and the spams inject to mail queue through localhost port 25. The question is how to find the real victim website?

Luckily, it is pretty straightforward, thanks to the Netfilter team, we can log the uid who access the network resource using iptables.

  1. Create a custom chain BAND_SPAM and append log packet & reject. Then, a return rule in order to return to original the iptables chains when not match the condition.
    $ iptables -N BAND_SPAM
    $ iptables -A BAND_SPAM -p tcp -d 127.0.0.1 --dport 25 -j LOG --log-prefix "BAND_SPAM UID:" --log-uid
    $ iptables -A BAND_SPAM -p tcp -d 127.0.0.1 --dport 25 -j REJECT
    $ iptables -A BAND_SPAM -j RETURN
  2. Insert an OUTPUT rule, when gid=505 (our web user group), jump into BAND_SPAM chain
    $ iptables -I OUTPUT -m owner --gid-owner 505 -j BAND_SPAM

So, what happen after this? Here is the result when spammer attempt to send emails.

[[email protected] ~]# dmesg|grep BAND_SPAM
BAND_SPAM UID:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=8980 DF PROTO=TCP SPT=43892 DPT=25 WINDOW=32792 RES=0x00 SYN URGP=0 UID=10014
BAND_SPAM UID:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=8981 DF PROTO=TCP SPT=43892 DPT=25 WINDOW=32792 RES=0x00 SYN URGP=0 UID=10014
BAND_SPAM UID:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=44438 DF PROTO=TCP SPT=44452 DPT=25 WINDOW=32792 RES=0x00 SYN URGP=0 UID=10014
BAND_SPAM UID:IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=44439 DF PROTO=TCP SPT=44452 DPT=25 WINDOW=32792 RES=0x00 SYN URGP=0 UID=10014

And we can find the web user from passwd file.
[[email protected] ~]# grep 10014 /etc/passwd
realvictim:x:10014:505::/vhosts/realvictim:/bin/false

Viola! We can call client and assist them to harden the website.

 

Additional work on plugins after upgraded to WP 3.6

I’ve just finished upgrade WP 3.6 and related plugins. There are two plugins generate Error message

PHP Strict Standards: call_user_func_array() expects parameter 1 to be a valid callback, non-static method GoogleSitemapGeneratorLoader::Enable() should not be called statically in /wp-includes/plugin.php on line 406, referer: http://www.refmanual.com/wp-admin/plugin-editor.php?file=google-xml-sitemaps-v3-for-qtranslate%2Fsitemap-core.php&plugin=google-xml-sitemaps-v3-for-qtranslate%2Fsitemap-ui.php

PHP Strict Standards: Non-static method GoogleSitemapGeneratorLoader::GetBaseName() should not be called statically in /wp-content/plugins/google-xml-sitemaps-v3-for-qtranslate/sitemap.php on line 114, referer: http://www.refmanual.com/wp-admin/plugin-editor.php?file=google-xml-sitemaps-v3-for-qtranslate%2Fsitemap-core.php&plugin=google-xml-sitemaps-v3-for-qtranslate%2Fsitemap-ui.php

PHP Strict Standards: call_user_func_array() expects parameter 1 to be a valid callback, non-static method GoogleSitemapGeneratorLoader::CallHtmlShowHelpList() should not be called statically in /wp-includes/plugin.php on line 173, referer: http://www.refmanual.com/wp-admin/plugin-editor.php?file=google-xml-sitemaps-v3-for-qtranslate%2Fsitemap-core.php&plugin=google-xml-sitemaps-v3-for-qtranslate%2Fsitemap-ui.php

PHP Strict Standards: Declaration of W3_Cache_Memcached::delete() should be compatible with W3_Cache_Base::delete($key, $group = '') in /wp-content/plugins/w3-total-cache/lib/W3/Cache/Memcached.php on line 15, referer: http://www.refmanual.com/wp-admin/plugins.php Continue reading

Configuring LVS-TUN for web service

There are many way to spread load to server farm, but I think LVS-TUN can do the best job. Since LVS-TUN will forward the requests to REAL servers and they response client request through their own public interface. Also REAL servers can locate in different data center or even different Geographical location. This mechanism can prevent uplink flooding, compare to LVS-NAT and LVS-DR that cannot work outside a physical network segment.

Our sample config is a startup model with two failover load balancers running ldirectord, heartbeat and LVS-TUN IP-IP tunneling configured on two web nodes in different datacenter.

Requirement

You’ll need a virtual IP address (VIP) that can float between load balancers, and setup in all web nodes. The way to work with that VIP is that, load balancers will receive traffic on VIP and forward request to a web node, and it will reply using that IP address as a source. Heartbeat will determine which load balancer take control of VIP at one time. Web nodes need to configure silent and won’t send ARP.
Install necessary packages

# yum -y install \
heartbeat.i386 \
heartbeat-ldirectord.i386 \
heartbeat-pils.i386 \
heartbeat-stonith.i386 \
ipvsadm.i386

Configuring the load balancers

Copy default config files…

[[email protected]_lvs ~]# cp /usr/share/doc/heartbeat-2.1.3/ha.cf /etc/ha.d/
[[email protected]_lvs ~]# cp /usr/share/doc/heartbeat-2.1.3/authkeys /etc/ha.d/
[[email protected]_lvs ~]# cp /usr/share/doc/heartbeat-2.1.3/haresources /etc/ha.d/
[[email protected]_lvs ~]# cp /usr/share/doc/heartbeat-ldirectord-2.1.3/ldirectord.cf /etc/ha.d/

Setup ldirectord

#cat >>/etc/heartbeat/ldirectord.cf<<EOF
checktimeout=10
checkinterval=2
autoreload=yes
logfile="local0"
quiescent=yes


virtual=1.2.3.4:80
real=192.168.0.10:80 ipip
real=172.19.0.10:80 ipip
service=http
request="httpcheck.html"
receive="OK"
scheduler=rr
protocol=tcp
checktype=negotiate

Create a html file called httpcheck.html, content is “OK”. ldirectord will GET /httpcheck.html and see if it receive “OK” every two seconds, and it will drop the REAL server after 10 seconds disconnect. Noted that, the real ip is different subnet, i.e. real server don’t need to sit together in once place, one could be in USA and another one could be in Singapore. Load balance type was simply round-robin style.

Setup Heartbeat Continue reading