You can, however, easily compile your own RPM and manually upgrade OpenSSH.  The commands below are the ones I used to install version 5.8 (the latest stable at the time of this post), but can essentially be used for any compatible version.

First, download the OpenSSH source tarball from the vendor and unpack it. You can find the tarballs at http://www.openssh.com/portable.html

wget http://mirror.mcs.anl.gov/openssh/portable/openssh-5.8p1.tar.gz

tar -xvzf openssh-5.8p1.tar.gz

Copy the spec file and tarball:

cp ./openssh-5.8p1/contrib/redhat/openssh.spec /usr/src/redhat/SPECS/

cp openssh-5.8p1.tar.gz /usr/src/redhat/SOURCES/

Do a little magic:

cd /usr/src/redhat/SPECS

perl -i.bak -pe ‘s/^(%define no_(gnome|x11)_askpass)\s+0$/$1 1/’ openssh.spec

…and build your RPM:

rpmbuild -bb openssh.spec

Now if you go back into /usr/src/redhat/RPMS/<arch> , you should see three RPMs. Go ahead and install them:

rpm -Uvh *.rpm

To verify the installed version, just type ‘ssh -v localhost’ and you should see the banner come up, indicating the new version.

Related posts:

1. cPanel Updates on Redhat 9 Can Break Stuff
2. Manually Upgrading MySQL
3. Upgrading or Downgrading MySQL

rpm -qa |grep -i exim

The version should be 4.69-25 – if it’s not, you need to upgrade. You can simply run:

/scripts/eximup

Click the link below to read cPanel’s advisory:

http://mail.cpanel.net/pipermail/news_cpanel.net/2010-December/000060.html

Related posts:

1. Using Custom RBL’s with Exim and cPanel
2. Changing Exim’s Sending IP
3. Exim ACL Ratelimit Database Not Available

A two-year-old kernel issue in Redhat distributions has surfaced in the form of a nasty exploit byAc1db1tch3z.  Basically, a 32-bit binary is compiled and loaded to the server, and when run by any users (even non-root users), it uses a bug in the 32/64-bit compatibility layer to open a root shell.  Here’s a copy and paste of one that I ran on a test server:

user1@server [~]# ./badscript
Ac1dB1tCh3z VS Linux kernel 2.6 kernel 0d4y
$$$ Kallsyms +r
$$$ K3rn3l r3l3as3: 2.6.18-194.11.3.el5
??? Trying the F0PPPPPPPPPPPPPPPPpppppppppp_____ m3th34d
$$$ L00k1ng f0r kn0wn t4rg3tz..
$$$ c0mput3r 1z aqu1r1ng n3w t4rg3t...
$$$ selinux_ops->ffffffff80327ac0
$$$ dummy_security_ops->ffffffff804b9540
$$$ capability_ops->ffffffff80329380
$$$ selinux_enforcing->ffffffff804bc2a0
$$$ audit_enabled->ffffffff804a7124
$$$ Bu1ld1ng r1ngzer0c00l sh3llc0d3 - F0PZzzZzZZ/LSD(M) m3th34d
$$$ Prepare: m0rn1ng w0rk0ut b1tch3z
$$$ Us1ng st4nd4rd s3ash3llz
$$$ 0p3n1ng th3 m4giq p0rt4l
$$$ bl1ng bl1ng n1gg4 :PppPpPPpPPPpP
# whoami
root

Scary, eh?  And to think, Redhat has still not released a fix for this, and it’s been two days.  Therefore, if you’re running a 64-bit CentOS or RHEL 5 server, you may be a sitting duck. All it takes is one site on your server to be prone to remote file injection, and the hack is in.

No reason to fret, though, you have a couple options.

1. Download a working patch from here, and install the kernel, kernel-devel, and kernel-headers RPMs, then reboot
2. Ksplice really came through on this one, and by nature, provided an update that does not require a reboot. Take a look at their post here, and download their diagnostic tool to any of your 64-bit servers running CentOS or RHEL 5, to make sure they haven’t been compromised. They are also offering a 30-day trial so you can secure your servers.  $4/mo is a worthy investment, and I’m 100% sure that you’ll be happy with your results.

Update: on 9/12, Redhat released a patch for 64-bit RHEL systems: https://rhn.redhat.com/errata/RHSA-2010-0704.html, and a similar one exists now for CentOS: http://bugs.centos.org/view.php?id=4518

Related posts:

1. Exim Privilege Escalation in 4.69-23
2. Upgrading OpenSSH on CentOS 5
3. cPanel Updates on Redhat 9 Can Break Stuff

It’s really hard to install ClamAV on cPanel – if you’re too lazy to click a couple buttons.  Simply go to WHM > Manage Plugins and enable the ClamAV Connector, and you’re good to go.

After this is done, you can go to WHM > Configure ClamAV Scanner and set scanning options for the entire server or specific users:

You should now see a Virus Scanner option in cPanel. If you don’t, you may need to enable it in Feature Manager.

From here, cPanel users can run scans on any permitted items.

For administrators, here are a few quick commands that may be useful:

Update antivirus database:

freshclam

Scan a directory and print out infected files:

clamav -ri /home

Scan a directly and remove infected files and emails:

clamav -ri –remove /home

Related posts:

1. Securing FTP Access on a cPanel Server
2. Fixing Quotas on a cPanel Server
3. 11 Ways to Free Up Disk Space on a cPanel Server

FTP Attacks on the Rise

A lot of hosts have reported continual issues with FTP hacks, where hackers are logging into FTP accounts with the user’s credentials and uploading malicious or “spammy” code. Here are a few examples:

http://support.inmotionhosting.com/ftp_exploits.html
http://mediumcube.com/mctalk/tag/ftp-password-hack/
http://www.34sp.com/blog/official-news/ftp-exploits-and-account-hacks/

No one seems to have identified the exact cause of the hacks, other than that they originate from the client’s computer. Originally this was attributed to viruses like Gumblar, but it’s hard to grasp that thousands of users all had the same problem – many of which had antivirus software capable of detecting the virus, or that were running various operating systems that didn’t have Adobe products installed. All we know is that someone hackers were accessing user login information from end users.

After a bit of research, it was found out that many popular FTP clients, like FileZilla, don’t encrypt the passwords when they’re stored, so it was recommended by security professionals to avoid storing passwords in FTP clients and browsers.

Need help on this? See: Securing FTP Access on a cPanel Server

Verifying the Extent of Damage

Most hosting providers are lucky to have never really had to deal with a major security breach. Most hacked site reports I’ve gotten in the last three years were standalone attacks or attacks targeted to vulnerabilities in specific software. Then there’s the few actual server issues that you have to take a look at…the ones involving root exploits and destroyed servers. Your job is to take that hacked site report and determine whether it’s an application issue, a server issue, or a client issue.

The Script Kiddie Attacks (Application Issues)

I categorize any targeted attack to a specific type of software, a script kiddie attack. In this situation, a software (usually open-source) has a discovered vulnerability that is posted on an advisory site, with a full synopsis of how the attack is performed. Then, an ignorant self-proclaimed “hacker” uses something like Google code search to look for sites running the vulnerable software, sometimes only targeting a specific server, and launches the attack against them.

These tend to be easier to handle since they’re easier to track. When someone comes to me about their hacked OsCommerce site, the first thing I know to check is the version, the security advisory for that software, and Google for similar attacks. Outdated software is the #1 factor in hacked sites nowadays, and they are also the easiest to deal with. Your action in this case only really needs to be as far as recommending to the user to keep their software up to date.

The Server Issue

You know something was out of place when you saw those suspicious processes running from /tmp on your server. You already traced it, and you may or may now know where it came from. That doesn’t mean your server was rooted, but don’t quite rule it out. If you see any malicious processes running as root on your server, you should assume that your server was rooted. And based on what you know about the powers of the root user, you probably already know that you need evacuate the users from the machine and reinstall the OS. Don’t take a chance by not doing this – you have no idea what the hacker could have done to your server, and it’s a lot less painful to suck up a few hours of moving data than having something ten times worse happen the second time around. Hopefully you already have a failover or backup plan in place to do emergency migrations in situations of a security breach or server failure.

The Initial Reaction / A Little Bit of Background

Quite honestly, I get really tired of the typical customer reaction when their site gets hacked. Since you’re the hosting provider, the customer almost always assumes that it’s your fault. “What can you do to protect my site” or “Did you fix the problem with your server” or “How did you let this happen”.

While you’d be doing your customers an excellent service to come up with ways to counteract attacks, it’s an unrealistic expectation that any hosting provider can prevent and/or detect every possible attack that can occur, especially when a majority of them are at the fault of the user. What does this mean? Well, let’s look at some simply statistics shall we? Out of 1,890 hacked sites (from various hosting providers) we studied in 2009 (not specific to a certain hosting provider):

• 31% were affected by the Gumblar virus or its variants (586 total)
• 42% were running severely outdated open-source software (794 total)
• 23% were running custom or unidentified applications that were exploited due to insecure coding (435 total)
• 4% were affected by a script executed on the server itself, or a server-side exploit (75 total)

Even in the 7% category, where the problem on the server end resulted in a hacked site, 52 out of the 75 users were using self-managed hosting services and [after a little prodding] self-admittedly failed to secure their server properly. The others in this category were determined as residing on a shared hosting server that was not secured properly to prevent rogue scripts in a common folder (like /tmp) from executing against everyone else on the server. Even then, this seemed to only affect servers running PHP as an Apache module, where there existed a ton of files owned by the webserver user or set to 777 permissions.

Dealing with your customers

If you’re part of a larger hosting organization you probably have a department that deals with these kinds of things.  How far you need to go into communication depends on the particular issue, but here’s a metric that I feel is appropriate:

• If single, unrelated sites are hacked, deal with them separately
• If multiple sites running the same kind of software are getting similar attacks, consider adding a security advisory to your support center to notify users of a potential exploit
• If a server is hacked but only a handful of sites are hacked, deal with those users only, and keep it under wraps assuming you are able to fix the problem or seamlessly move everyone to another server
• If there’s a major issue going around, consider sending your customer base a notification, or making an official statement in your support center, blog, or mailing list

Basically, the key is communication, but you also don’t want to overshare. Be informative but brief. Those of you who have dealt with things like this before know that there’s really only three things the the customer wants to know:

• How did it happen
• Who did it
• How do we prevent this in the future

Whether you’re dealing with one customer or 50,000, you want to  be able to answer those questions. Here is an excellent example:

http://www.34sp.com/blog/official-news/ftp-exploits-and-account-hacks/

Put simply, you have to assume that most of your customers are going to be idiots less knowledgeable, and realize that when they don’t know what’s going one, they either pretend like they know everything, or blame someone else. And while us hot-headed sysadmins just love to break down the ego of a know-it-all, the last thing you want to do is provoke a rash response. However, you also don’t want to beat around the bush.

Related posts:

1. Securing FTP Access on a cPanel Server
2. Tips to Reduce Your Customer Support Costs
3. How to Change Your FTP Port

While it’s really not something you as the hosting provider can control, there are measures you can take to secure your server against FTP hacks.

The Things You Probably Noticed

You probably started thinking about FTP security when you started getting complaints about hacked sites. After looking into it further, you notice that the hack on most all the sites are very similar, and that all the hack files were uploaded via FTP by someone that is not the actual user of the account.  When this happens to a bunch of users on the same server, the first inclination is…Ok, wow, the server got hacked.

That’s what I initially thought, but it became very obvious that someone was out there, spreading passwords around like wildfire. I still don’t actually know where the password were obtained from, but the most common theory out there is that a security hole in Adobe software, named “The Gumblar Virus” used the plain-text storage mechanisms of common FTP clients (like Filezilla, CuteFTP, etc) to capture passwords and send them to hackers.  These hackers then used scripts to automatically download files from a user’s account, modify them, and re-upload them.

To make matters worse, the hack eventually evolved to where other servers were hacking other servers using this attack method.  One of our sysadmins once found a file uploaded to a compromised account that had a list of over 100 usernames, password, and server combinations, that was clearly used to automate attacks to other servers.

Enforcing Encrypted Logins

You can force your users to use encrypted connections in order to connect via FTP to the server. To do this, go to WHM > FTP Server Configuration and set the option for TLS Encryption Support to Required.  The cipher suite should also be set to something like:

ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM:!SSLv2

Note that this is for FTP over TLS, which is the general preference since SFTP requires enabling port 22 and SSH access, which most hosting providers tends to swing against in a shared hosting environment. After enabling TLS, you can install a SSL certificate for FTP in WHM > Manage Service SSL Certificates, which is the hostname that your users should connect to over Passive/Explicity TLS in their FTP clients. By requiring SSL your customers are less likely to have their password sniffed out by network packet sniffers.

FTP Upload Scanning

One thing I noticed is that most hosting providers are reacting to the malicious file uploads by enabling upload scanning to catch certain strings within the files.  In the situation with IMH (as released in our official statement about the FTP hack problem), if our scanner detects a user uploading malicious files, the scanner will block that person’s IP, change the password for the user, and automatically email the customer to let them know what’s up.  While our solution is custom-written to allow for complete control and customization, there are other options out there for those of you that are less programming-savvy or just don’t have the time to deal with it:

http://www.configserver.com/cp/cxs.html

http://www.serverprogress.com/upload_guardian.php

Changing the FTP Port

You may want to consider running FTP on a different port which can create a deference for automated scripts. Instructions for doing this can be found here.

Enforcing Password Policies

Later versions of cPanel 11.25 allow you to enforce password policies for your customers, including password strength and timeouts, requiring users to select secure passwords and change them at intervals you specify. You can find this option in WHM > Security Center . The password life settings are not enabled until 11.25.1.

Brute-Force Detection

Keep in mind that the recent issues with FTP hacks in 2009 and 2010 were from Gumblar-related attacks, where the hacker obtained or sniffed out FTP passwords from a combination of exploitable software on the client’s PC. Therefore, brute-force protection isn’t going to help you here since the attacker already had the FTP login credentials before they even reached your server.  However, in general it’s advisable to monitor any authenticated service for excessive login failures to prevent password guessing and accounts being compromised as a result. cPanel has built-in brute-force detection in a service called cpHulkd, which you can enable in WHM > Security Center. Or you can use BFD, which is an optional addon to APF.

Educating Your Customers

Most importantly, you need to be upfront with your customers about what’s going on.  If you’re running a couple single small servers you can probably address this behind the scenes, but if you’re a larger hosting provider you might as well not even try to hide it. All this is going to do is prompt a lot of chatter that can damage your business’s reputation and generate more support contact from your customers.

Make sure your customers know what’s going on, and what you’re doing to help the situation. Even though most FTP hacks are probably not your fault, most of your customers probably don’t even care – all they want to know is what you are doing to protect their accounts. You also want to educate them on common security practices when it comes to managing their accounts.  Here are some examples of hosting providers that promptly identified an issue with FTP security and notified their customers accordingly:

http://support.inmotionhosting.com/ftp_exploits.html

http://www.34sp.com/blog/official-news/ftp-exploits-and-account-hacks/

http://weblog.mediatemple.net/weblog/category/system-incidents/1026-gs-security-advisory/

Hopefully this helps you in securing your FTP services.  With a combined effort, the use of common security practices can help keep the widespread FTP hacks at bay, and make all our jobs easier.

Related posts:

1. Dealing With Hacked Sites
2. 11 Ways to Free Up Disk Space on a cPanel Server
3. Installing ClamAV on a cPanel Server

Pure-FTP

1) Edit /etc/pure-ftpd.conf and look for the following line:

Bind <addr> <port>

Where <addr> is a publically routable IP.  The default example, 127.0.0.1, will cause the socket to bind locally but then this connection won’t serve externally. <port> is the new port you wish to put the service on.

2) If necessary, add the new FTP port to your server’s firewall

3) Edit /etc/chkserv.d/ftpd and change the port, which is the first comma-separated entry to the right of ‘=’ on the line in the file, to match the port you put the service on.

Finally:

/etc/init.d/pure-ftpd restart
/etc/init.d/cpanel restart (restarts tailwatchd/chkservd)

ProFTP

Edit /etc/proftpd.conf and change:

Port 21

Then make sure to test to make sure FTP is working before notifying your users!

Related posts:

1. How to Enable Passive FTP
2. Opening an Additional Exim Port
3. Adding Services to Chksrvd for Monitoring

You’ll want DomainKeys and SPF records if your users have trouble sending email to certain providers, or they are having issues with spoofed (forged) email. CPanel currently allows two easy ways for you or your users to set up email verification. This is supported at least from cPanel 11.18 onward.

User-Level:

You can enable the “Email Authentication” feature in WHM ~> Feature Manager, which will enable the Email Authentication icon in the users’ cPanels where they can create DomainKeys and SPF records for their domain(s).

Root-level:

There are scripts in /usr/local/cpanel/bin that can install these on a per-user basis:

/usr/local/cpanel/bin/domain_keys_installer $user

/usr/local/cpanel/bin/spf_installer $user

(and corresponding scripts to remove, like spf_uninstaller and domain_keys_uninstaller)

If you want to hit up everyone on the server, you can run my for loop one-liner:

for user in `ls -A /var/cpanel/users` ; do /usr/local/cpanel/bin/domain_keys_installer $user && /usr/local/cpanel/bin/spf_installer $user ; done

Now what about new users? cPanel already though of that, and has options to create hooks for when after an account is created. To set up the server to automatically create an SPF record and DomainKey for new accounts, edit /scripts/postwwwacct and paste in the following code:

#!/usr/bin/perl

my %OPTS = @ARGV;
$ENV{USER} = “$OPTS{‘user’}”;
system q(/usr/local/cpanel/bin/domain_keys_installer $USER);
system q(/usr/local/cpanel/bin/spf_installer $USER);

To verify an SPF record and/or DomainKey, you can run a DNS check:

dig default._domainkey.$domain TXT

dig $domain TXT

A technical note about DKIM:

You might know that DKIM is actually a generated key pair, similar to an SSH or SSL Certificate’s RSA key. CPanel stores these files in /var/cpanel/domain_keys, where the public folder contains the key reflected in the DNS zone, and the private folder contains the private key. You may have users that actually authenticate via DKIM in their mail clients, in which case you may need to provide them with the private key in order for them to sent email.

Related posts:

1. Using DKIM with Exim and cPanel
2. 10 Tips for Improving Email Delivery
3. Installing an SSL Certificate for MySQL