Apr
7
2014

OpenSSL Heartbleed Bug and What You Need to Know

Over the past few days, we have received an overwhelming number of questions about the OpenSSL Heartbleed bug and how cPanel system administrators should be handling this.

First of all, if you haven’t read Codenomicon’s write-up on the bug, which thoroughly explains what it is, you should look visit heartbleed.com. Because their website already covers just about everything you need to know, I don’t feel the need to rehash all the nitty-gritty details here.  I’m only going to address how you, a cPanel administrator, should address this on your server.

Only CentOS 6.5 is vulnerable to this bug.  Obviously this affects other OS’s as well, but as a cPanel administrator you’re only dealing with CentOS (and perhaps FreeBSD).  If you use a version of CentOS older than 6.5, read no further – you’re not affected and can rest easy.

You can test whether you’re vulnerable by using this tool against a website on your server that has an SSL certificate installed.

To clear this up real quick: OpenSSL is a vendor-supplied package that, in your case, is provided by CentOS.  It is not supplied by cPanel, so there’s no point in asking them to “fix” this.  There is nothing for them to fix.  However, if you have system package upgrades enabled for cpupdate, and cPanel updates run automatically on your server, chances are the OpenSSL updates have already been installed.  To configure your cPanel update settings, check out this link, or go to WHM -> Update Preferences.

Check to see if you’re running the latest version as so:

root@server [~]# rpm -qa |grep -i openssl
openssl-1.0.1e-16.el6_5.7.x86_64
openssl-devel-1.0.1e-16.el6_5.7.x86_64

At this point you’re looking for version 1.0.1e or newer (at the time of this writing, 1.0.1e is the latest).  This update does not necessarily fix the bug in question, but rather disables the TLS heartbeat extensions that are vulnerable.  A later version will likely fix the problem altogether, but such is not available yet because RHEL has not released a fix.

After OpenSSL is updated, you need to restart services.  This is necessary whether you updated OpenSSL yourself, or let cPanel do it during its update process.  The following services should be restarted:

  • cPanel
  • Apache
  • Exim
  • Dovecot/Courier
  • Pure-ftpd/Proftp
  • MySQL
  • any other services that use SSL (Tip: you can use the following command to find a list of services to restart)

lsof -n | grep ssl | grep DEL | awk ‘{print $1}’ | sort | uniq

(Suggestion provided by Lucas Rolff)

Once this is done, you should be good to go.  It is recommended, however, that you re-key your SSL certificates and have then re-issued and re-installed to prevent security problems resulting from a compromised private key.

You may also wish to follow this conversation from the cPanel forums.

Mar
18
2014

Fix for “Missing owner for domain X, force lookup to root”

On recent cPanel versions, rebuilding the Apache conf returns the following non-fatal error:


root@savannah [~]# /scripts/rebuildhttpdconf
info [rebuildhttpdconf] Missing owner for domain savannah.tcaserversolutions.com, force lookup to root
Built /usr/local/apache/conf/httpd.conf OK

 

The error does not adversely effect the way the httpd.conf file is built, but the presence of the unnecessary info message can be annoying.  And by “can be annoying”, I mean “is annoying”.

To fix this, you will need to edit the main vhost template for the ‘nobody’ user, which owns the primary hostname of your server.  Edit /var/cpanel/userdata/nobody/$hostname and add this line to the bottom of the file:


owner: 'nobody'

 

Then run /scripts/rebuildhttpdconf again and the “error” should be gone.

 

This fix was affectionately provided by Mohammed Naser from Vexxhost.

 

 

Mar
12
2014

Implementing Mandrill with Exim on cPanel

Mandrill is a transactional email service run by MailChimp, comparable to SendGrid.  It comes stock with a powerful API for fast implementation into applications for sending email reliably over SMTP, but it can also be used as a smart host for all of your server’s outgoing email.

The below instructions cover how to do this via command line.  If you prefer WHM, simply go to WHM -> Exim Configuration Manager -> Advanced Editor and alter the sections indicated.

First, open up your /etc/exim.conf.local file in an editor and look for the @AUTH@ section.  Modify it to look like this:

 

@AUTH@
auth_login:
driver = plaintext
public_name = LOGIN
hide client_send = : your@email : api_key

 

The value of api_key is not your Mandrill account password – it’s your API key.  You can find it in your account settings under “SMTP & API Credentials”

Also, replace your@email with your Mandrill account email address.  If you already have something in the AUTH section, simply add this block of text below it.

Now look for the @PREROUTERS@ section, and modify it to look like this:

 

@PREROUTERS@
smart_route:
driver = manualroute
domains = ! +local_domains
ignore_target_hosts = 127.0.0.0/8
transport = auth_relay
route_list = * smtp.mandrillapp.com
no_more

 

 

The last modification should be to the @TRANSPORTSTART@ section:

 

@TRANSPORTSTART@
auth_relay:
driver = smtp
port = 587
hosts_require_auth = $host_address
hosts_require_tls = $host_address

 

Save this file.  Now open /etc/exim.conf.localopts and add this line (or modify it and append the hostname to the existing line)

 

smarthost_routelist=*: smtp.mandrillapp.com

 

If you’re using WHM, this is under Basic Editor -> Mail -> Smarthost support.

Save the file, and apply the changes:

/scripts/buildeximconf

service exim restart

 

To test whether things are working, send an email out from your server, and look for it in /var/log/exim_mainlog. You should see something like this in your log entry:

 

2014-03-11 21:59:50 1WNbGg-0006ud-2d => to@email R=smart_route T=auth_relay H=smtp.us-east-1.mandrillapp.com [x.x.x.x] X=UNKNOWN:ECDHE-RSA-AES256-GCM-SHA384:256 A=auth_login C="250 2.0.0 Ok: queued as 3529E193E178"

 

If you have SPF records, you’ll need to add the hostname of the SMTP server to the record itself to allow the third-party mail server to send email on behalf of your domain.  Mandrill will provide the hostname you should use under Settings -> Sending Domains, when you do an SPF test.

 

 

Mar
11
2014

Unable to Change Shell via WHM or CLI

When trying to change a user’s shell via WHM, you may occasionally see the following error:

Shell could not be modified

As shown below:

WHM Shell Fail

Attempting to change the shell via command line fails as well:

# chsh -s /bin/bash iviews5
Changing shell for iviews5.
setpwnam: File exists
Shell *NOT* changed.  Try again later.

This means there is a lock on the /etc/passwd file.  Simply deleting /etc/ptmp fixes this problem:

rm /etc/ptmp

Feb
13
2014

cPanel 11.42 Cheat Sheet

In 2010, we released our first cPanel cheat sheet for version 11.25.  Admittedly, we haven’t been keeping up with the times.  A lot has changed with cPanel over the last few years and an update is well overdue.

Though labeled for version 11.42, the new cheat sheet is verified to be accurate for the following versions:

  • 11.36
  • 11.38
  • 11.40
  • 11.42

Hop on over to our Cheat Sheets page to download the newest version.  Due to the popularity of our 11.25 cheat sheet and by popular request from our visitors and clients, we plan to release a new cheat sheet for every new minor cPanel version.  We will also be developing more cheat sheets in the near future for various parts of your system that we couldn’t squeeze into the cPanel one.

(in case you’re wondering how many times I can say “cheat sheet” in a single post, the answer is “8″)

http://www.thecpaneladmin.com/wp-content/plugins/downloads-manager/img/icons/pdf.gif File: cPanel 11.42 Cheat Sheet (48.87KB)
Added: 02/14/2014
Downloads: 543
Description: cPanel ports, file/folder locations, scripts, and log locations, for cPanel 11.36, 11.38, 11.40, and 11.42.

 

Jan
17
2014

How to FSCK a Linux Filesystem

There comes a time in every sysadmin’s life where filesystem errors just…happen.  Luckily, these are somewhat easy to fix, assuming you don’t have a greater problem involving physical hardware damage.

First, you need to know the name of the disk device having the problem.  Do a quick df to see what device the affected partition is on:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       2.7T  2.6T  106G  97% /
/usr/tmpDSK     4.0G  1.7G  2.2G  44% /tmp

Look under the “Filesystem” column to see the device name for the partition in question.  Now, if this is any filesystem but “/”, your job is probably going to be easy.  Simply unmount the file system and run a fsck against it.  For example, if you have a separate /home partition listed as /dev/sda3, you would do:

umount /home
fsck -yC /dev/sda3

There are a number of options for fsck, but the above combination is my personal preference.  The ‘y’ tells the fsck to fix whatever error sees, which is preferable unless you feel that your index finger has the stamina to hit ‘y’ 500 times in a row, and the ‘C’ prints out a pretty little progress bar so you can keep an eye on it.  Ext4 fiesystems fsck rather quickly – typically less than an hour for a 2TB filesystem.  Ext3 takes significantly longer.

Now, unmounting a filesystem may not be straight-forward – if any services are actively using files on that partition, the OS will refuse to unmount it.  Doing a lazy unmount (umount -l) won’t work here either – you need to unmount it cleanly.  To see what processes are using the filesystem in question, use lsof.  From the above example:

lsof -p |grep /var/

Then stop any services or processes using it.

If the filesystem issue is on your primary partition, you have a little more work ahead of you.  You’re going to need to boot into a rescue image.  To do this, simply use a Netinstall image and boot to the CentOS installation screen, then type:

linux rescue nomount

You can skip networking and all that jazz, then run the shell.  From there, you’ll need to find the partition in question and run the same fsck command.  Do note that on CentOS 6+, the device name may be incremented since it will count the rescue image as the first device in most cases.  So your /dev/sda3 might be /dev/sdb3 now, or even /dev/sdc3.

Once the fsck is done, reboot and confirm your filesystem is clean:

dumpe2fs -h /dev/sda3

The “Filesystem state” line should read “clean”.  If it doesn’t, the fsck either didn’t complete correctly, or you have a larger problem on your hands.

Dec
28
2013

How to Upgrade Python on CentOS

If you’re running CentOS, you’re probably a few versions behind on Python.  Currently, the version packaged for CentOS 5 and 6 is 2.6.  Contrary to what the title of this post implies, you actually cannot safely upgrade Python on any Redhat distribution.  If you’re feeling brave, try this to see why:

yum remove python

Warning, do not pass a ‘-y’ to the above command.  All you’re doing here is viewing output.  Hit CTRL+C once it prompts to continue, and no changes will be made to your system.

If you ran the command above, you’ll see all the packages that depend on Python, most all of which will break (including Yum itself) if you alter the system-installed package or attempt to upgrade.  Therefore, if you need a newer version of Python, the only safe way to do this is to install it alongside the system version.

This example is for installing Python 2.7, but you can easily do similar steps for version 3.3, etc.  Just download your preferred version from python.org.

cd /usr/src

wget http://python.org/ftp/python/2.7.6/Python-2.7.6.tgz

tar -xvzf Python-2.7.6.tgz

cd Python-2.7.6

./configure –prefix=/usr/local

make

make install

 

And that’s all that’s to it!  Now any scripts that require the alternate version should have the following shebang:

#!/usr/local/bin/python2.7

 

If you need to use pip to install modules for your alternate version, make sure to use the correct one:

/usr/local/bin/pip2.7

 

Oct
8
2013

Implementing SendGrid with Exim on cPanel

Setting up your cPanel server to send through a third-party mail server is very easy to do, if you understand the basics of how cPanel builds its Exim configs. You never want to edit your exim.conf file directly – your changes will be wiped out any time a cPanel update runs or someone makes a change via the Exim Configuration Editor in WHM.  Below is a quick guide on setting up a custom mail router with a provider like SendGrid, which in turn will route all outbound email through the external mail service.

The below instructions cover how to do this via command line.  If you prefer WHM, simply go to WHM -> Exim Configuration Manager -> Advanced Editor and alter the sections indicated.

First, open up your /etc/exim.conf.local file in an editor and look for the @AUTH@ section.  Modify it to look like this:

 

@AUTH@
sendgrid_login:
driver = plaintext
public_name = LOGIN
client_send = : <user> : <password>

 

Of course, replace <user> with your SendGrid username and <password> with your SendGrid account password.  If you already have something in the AUTH section, simply add this block of text below it.

Now look for the @PREROUTERS@ section, and modify it to look like this:

 

@PREROUTERS@
send_via_sendgrid:
driver = manualroute
domains = ! +local_domains
transport = sendgrid_smtp
route_list = "* smtp.sendgrid.net::587 byname"
host_find_failed = defer
no_more

 

The last modification should be to the @TRANSPORTSTART@ section:

 

@TRANSPORTSTART@
sendgrid_smtp:
driver = smtp
hosts = smtp.sendgrid.net
hosts_require_auth = smtp.sendgrid.net
hosts_require_tls = smtp.sendgrid.net

 

Now save the file, and apply the changes:

/scripts/buildeximconf

service exim restart

 

To test whether things are working, send an email out from your server, and look for it in /var/log/exim_mainlog. You should see something like this in your log entry:

 

2013-10-08 19:37:29 1VTjeS-0000Ac-O3 -> my@email R=send_via_sendgrid T=sendgrid_smtp H=smtp.sendgrid.net [x.x.x.x] X=TLSv1:DHE-RSA-AES256-SHA:256

 

If you have SPF records, you’ll need to add the hostname of the SMTP server to the record itself to allow the third-party mail server to send email on behalf of your domain.  Sendgrid will provide the hostname you should use.

 

 

Sep
9
2013

How to Easily Monitor Cluster Status

If you are running cPanel 11.28 or higher (which we hope you are, considering the current release at the time of this writing is 11.40!), you have the option in WHM to automatically disable DNS clustering if too many connection failures occur:

 

WHM cluster status options

 

While this can be a handy feature, if you have a high-capacity system that frequently makes DNS updates, this could create complications.  The easy way out is to of course select the third option to keep the cluster members online, however, this could slow down DNS changes (adding/removing domains, etc) if your cluster members are unavailable.  Choosing to receive notifications may be a viable option, but some administrators receive so much email to the WHM admin contact address that the email alerts may not be as visible as they need to be.

 

If you have a monitoring system that supports custom plugins, such as Nagios, you can monitor the status of the cluster simply be looking for a few files.  When DNS clustering is globally enabled, a file called /var/cpanel/useclusteringdns will exist, and all cluster members will be enabled by default. The status files for specific cluster members are stored conveniently in /var/cpanel/clusterqueue/status/, and will be named as follows:

 

  • $IP: This file always exists, and has binary contents.  If it contains all 1′s, everything is fine.  If it has 0′s in it, connection failures have occurred. Of course, $IP represents the actual IP address of the remote cluster member
  • $IP-down: This file will exists when a cluster member is down.  Deleting this file will re-activate the cluster member.

You can also monitor the following folders which may help ore-emptively identify a DNS problem or overload within your clustering system:

 

  • /var/cpanel/clusterqueue/retry/:  Contains a single file for each request to the cluster that has failed.  Having too many of these will indicate a problem reaching one or more members of the cluster
  • /var/cpanel/clusterqueue/requests/: Contains a single file for each request to the cluster that has not yet synchronized.  Having too many of these at once will indicate heavy activity on the server.  If this activity is unusual, it may need to be investigated.

 

Knowing what files to look for can allow you to write your own monitoring scripts and/or Nagios plugins to periodically check for clustering problems, or attempt to automatically fix them, if you don’t want to rely on email notifications.

Jul
18
2013

How to Set up Multiple Shared IPs on a cPanel Server

There are some situations where you may need to set up multiple shared IP addresses on a server.  Some reasons may include grouping accounts per IP or lowering the effect of DDoS attacks if it becomes necessary to block traffic to a specific interface.  Whatever the reason, multiple shared IP addresses on a cPanel server is rather easy to set up.

 

First you’ll want to reserve the IP address via WHM -> Reserved IPs.  Alternatively, simply drop them into /etc/reservedips, separated by line, ie:

1.2.3.4

4.5.6.7

7.8.9.10

If you want to label the reserved IPs, set them up in /etc/reservedips:

1.2.3.4=shared ip

4.5.6.7=shared ip

… and so on

Now rebuild the IP pool:

/scripts/rebuildippool

This will prevent the IPs from being used as dedicated IPs for individual cPanel accounts, but for multiple accounts to be able to use them as shared IPs you’ll need to add them to a pool.  Simply use the /var/cpanel/mainips/root file to list each IP on its own line.  When root creates an account on the server, the first IP address in that file will be used.  If you want a reseller to have multiple shared IPs as well, simply replace ‘root’ with the reseller’s username.

 

To take this configuration further, you may want to have the main IP rotate automatically so each IP in the pool gets equal use.  We’re written a script to automate this task rather seamlessly via cron.

 

Click here to download the script

 

Installation steps:

wget -O /usr/bin/rotate_main_ips http://www.thecpaneladmin.com/files/rotate_main_ips.py

chmod 755 rotate_main_ips

echo ‘*/30 * * * * /usr/bin/rotate_main_ips’ > /etc/cron.d/rotate_main_ips

service crond reload