After switching several servers over from APF to CSF, my colleagues and I noticed that there are times during the day that LFD just chews away CPU, even while the process claims it’s “sleeping”:

root 15785 35.6 0.1 194496 93104 ? Ss 00:00 474:54 lfd – sleeping

35.6% CPU? What is it possibly doing?

In one of their FAQs, one of the developers from ConfigServers recommended disabling the following features in the csf.conf file, one by one to help narrow it down:

LF_DIRWATCH = “0″
CT_LIMIT = “0″
LF_HTACCESS = “0″
LF_MODSEC = “0″
LF_DIRWATCH_FILE = “0″
PT_LIMIT = “0″

Now, we already had these disabled – our use of LFD is limited to the downloading of global allow/deny lists only, as we don’t wish to use CSF for login failure detection at this time.  So it was a bit puzzling at why, even with the main functionality of LFD disabled, it was using so much CPU.

An strace against LFD’s PID gave us the answer – it showed that LFD was continually parsing the various log files on the server, seemingly in a never-ending loop.  This appeared to correlate with the “LD_PARSE” value of 5 seconds.  When your servers are as large and busy as ours, your log files tend to be huge – and there’s no way that CSF is parsing those logs in 5 seconds or less.

To resolve this, we set the LF_PARSE value a bit higher to allow it to rest a bit. Since we aren’t using LFD for login failure tracking, we essentially do not have a need for it to frequently scan the server’s logs. We haven’t had a problem with LFD since.

Similar to the intentions of SPF records and DomainKeys, DKIM is intended to authenticate a sender by using a public/private key pair to digitally sign email, therefore increasing deliverability. At this point in time, while SPF records and DomainKeys are supported in cPanel, outgoing authentication with DKIM is not. That being said, please keep the following points in mind while following this tutorial:

1. The cPanel developers indicated that DKIM auth will be a feature in 11.32, therefore, when 11.32 is released it’s possible that a few steps in this guide will become obsolete
2. cPanel does not recommend or provide support for some of the changes this tutorial will ask you to make.  As usual, make these changes at your own risk or use our server consultation services for help

Upgrade Exim

DKIM is supported in Exim 4.70 and higher – the version current at the time this article was written is 4.69. Therefore, you will need to upgrade.  Hop over to the below URL, locate the latest version of Exim 4.7x, and navigate to the folder that corresponds with your operating system to find the RPM:

http://httpupdate.cpanel.net/exim/

In this case, we downloaded the RPM from:

http://httpupdate.cpanel.net/exim/4.70-4/centos/5.6/exim-4.70-4_cpanel_maildir.i386.rpm

On your server, run the following command (replace the URL with the one corresponding to the RPM you found)

rpm -Uvh http://httpupdate.cpanel.net/exim/4.70-4/centos/5.6/exim-4.70-4_cpanel_maildir.i386.rpm --nodeps

Now, verify the version:

root@server [~]# rpm -qa |grep -i exim
exim-4.70-4_cpanel_maildir

You’ll also want to keep cPanel from running an Exim update and reverting the version back to 4.69:

touch /etc/eximupdisable

 

If you have existing domains on this server, make sure to move local and remote domains files back:

mv -f /etc/localdomains.rpmsave /etc/localdomains
mv -f /etc/remotedomains.rpmsave /etc/remotedomains

*If you have trouble installing the RPM due to fetchmail or a conflict with the previously-install Exim package, do

yum remove fetchmail

rpm -e exim-4.69-29_cpanel_maildir (replace with the actual name of the old Exim package)

Install DomainKeys

The topic of how to install DomainKeys was previously covered in this article. Install the keys for each user, or run a loop as shown in the article to install them for all users.  The keys will be in: /var/cpanel/domain_keys/[public|private]/$domain .

Configure Exim

Open /etc/exim.conf and near the top, add the following lines:

DKIM_DOMAIN = ${lc:${domain:$h_from:}}
DKIM_FILE = /var/cpanel/domain_keys/private/${lc:${domain:$h_from:}}
DKIM_PRIVATE_KEY = ${if exists{DKIM_FILE}{DKIM_FILE}{0}}

Then scroll down until you see these lines:

remote_smtp:
driver = smtp

Replace this with:

remote_smtp:
driver = smtp
dkim_domain = DKIM_DOMAIN
dkim_selector = default
dkim_private_key = DKIM_PRIVATE_KEY
dkim_canon = relaxed
dkim_strict = 0

Now, restart Exim:

service exim restart

The additions to exim.conf will use a domain’s key, if the domain has one, to sign each message with DKIM.

Now, keep in mind that cPanel will overwrite these changes during cPanel updates. You have the following options:

• Run chattr +ia /etc/exim.conf, OR:
• Copy /etc/exim.conf to /root/exim.conf and create a file called /scripts/posteximup (and chmod to 700) containing the following code:

scp -p /root/exim.conf /etc/exim.conf
service exim restart

Verify

The easiest way to verify the functionality of your new DKIM setup is to send an email from your server to check-auth@verifier.port25.com . You’ll get an autoresponse back letting you now that everything is working (note that with DKIM, DomainKeys are obsolete so a status of “neutral” is normal)

SPF check:          pass
DomainKeys check:   neutral
DKIM check:         pass
Sender-ID check:    pass
SpamAssassin check: ham

If at all possible, you should use /scripts/mysqlup to do a MySQL upgrade, however, if you need a version of MySQL outside of what they are offering in current cPanel versions, it is possible to upgrade manually.

These are the instructions for performing a manual upgrade of MySQL to 5.1.30, but this typically applies to all versions over 4.1 and higher. Typically it is best to upgrade to whatever the latest stable branch of MySQL is and apply these instructions to install the RPMS.

If this is on a live system, make sure to make backups of the databases.


mkdir /root/dbbackups

touch /root/dbbackups/list
for db in `mysql -e 'show databases' |awk '{print $1}' | grep -v Database`


do

mysqldump --add-drop-table $db > /root/dbbackups/$db.sql && echo $db >> list

done
mkdir /root/sqllibs
cp /usr/lib/libmysqlclient.* /root/sqllibs

Do a search for all installed MySQL packages to be removed:
rpm -qa | grep -i mysql-

This should present a list of 5 or 6 MySQL RPM’s that are installed, likely. Uninstall them one by one with the rpm -e like so:
rpm -e MySQL-devel-4.1.21-0.glibc23

Some packages are dependencies of each other, so you may need to remove one before being able to remove another.

Download the binaries for the OS version and MySQL version you are using from the cPanel repo:
http://httpupdate.cpanel.net/mysqlinstall/

execute rpm -i for each package (ex: pm -i MySQL-client-5.1.30-0.glibc23.i386.rpm)

Edit /var/cpanel/cpanel.config and change the MySQL version to 5.1 (or whatever version you’re running now). Then run this command:
touch /etc/mysqlupdisable

This will keep cPanel from resetting or reverting back to its original MySQL version during updates.

Restart MySQL first to make sure that it can at least start (which it should). Once you’ve verified this, run mysql -V to make sure that the correct version is installed. You should see output similar as below:
root@server [~]# mysql -V
mysql Ver 14.13 Distrib 5.1.30-beta, for pc-linux-gnu (i686) using readline 5.0


MySQL should now be up and running, but it’s likely that trying to log in directly as the root user will fail. Before Following the below steps, make sure you can do a ‘mysql -u root’ and log in successfully. If not, you’ll need to set the root password. By default, MySQL has a blank root password which will need to be set to ours…twice.
Recopy the libs, but don’t overwrite any that already exist:
mv /root/sqllibs/libmysqlclient.* /usr/lib/mysql/

Next update cPanel’s PHP and DBD MySQL bundle for perl by executing the following:
/scripts/perlinstaller --force Bundle::DBD::mysql
/scripts/makecpphp

Verify that phpMyAdmin and php -m load properly and without errors.

If your server is approaching 99% disk space, your users may be aware of this before you are.  At the 1% free margin, the following error will pop up when accessing certain features in cPanel and WHM:

Sorry for the inconvience!
The filesystem mounted at / on this server is running out of disk space. cPanel operation has been temporarily suspended to prevent something bad from happening. Please ask your system admin to remove any files not in use on that partition

While 99% may seem like a dangerous limit, servers nowadays are getting rather large. If you have a server with over 3T of disk space, that 1% free could be well over 100G of space – space that would seemingly become waste.

Until this limit is removed in a future release, you can actually trick cPanel into thinking that the server has more space available. The key is the cache files located in /root/.cpanel/datastore, specifically the file _bin_df_-P_-k_-l.

If you edit that file and change the occurrence of 99% to a lower value, you’ll see the restricted cPanel and WHM features return to normal.  Keep in mind that the cache regenerates, so in some cases it may be appropriate to create a cron job to periodically change the file.

Of course, if 99% disk usage on your server leaves you with a minuscule amount of space, you’ll want to remove some files or add additional storage. The following articles may be useful:

• http://www.thecpaneladmin.com/11-ways-to-free-up-disk-space-on-a-cpanel-server/
• http://www.thecpaneladmin.com/huge-cpan-folder-home/
• http://www.thecpaneladmin.com/managing-multiple-hard-drives-cpanel/
• http://www.thecpaneladmin.com/manually-rearranging-accounts/

Thanks to my colleague Lee for sharing this tidbit of info

If you’ve taken a peek at your PCI scan results lately, you may have noticed that your scan provider is now requiring OpenSSH 4.5 or higher – a version that is not currently available in the CentOS 5 repositories.  A Yum update isn’t going to help you much there.

You can, however, easily compile your own RPM and manually upgrade OpenSSH.  The commands below are the ones I used to install version 5.8 (the latest stable at the time of this post), but can essentially be used for any compatible version.

First, download the OpenSSH source tarball from the vendor and unpack it. You can find the tarballs at http://www.openssh.com/portable.html

wget http://mirror.mcs.anl.gov/openssh/portable/openssh-5.8p1.tar.gz

tar -xvzf openssh-5.8p1.tar.gz

Copy the spec file and tarball:

cp ./openssh-5.8p1/contrib/redhat/openssh.spec /usr/src/redhat/SPECS/

cp openssh-5.8p1.tar.gz /usr/src/redhat/SOURCES/

Do a little magic:

cd /usr/src/redhat/SPECS

perl -i.bak -pe ‘s/^(%define no_(gnome|x11)_askpass)\s+0$/$1 1/’ openssh.spec

…and build your RPM:

rpmbuild -bb openssh.spec

Now if you go back into /usr/src/redhat/RPMS/<arch> , you should see three RPMs. Go ahead and install them:

rpm -Uvh *.rpm

To verify the installed version, just type ‘ssh -v localhost’ and you should see the banner come up, indicating the new version.

With spam continuing to be an increasing problem, hosting providers are always looking for ways to filter out junk so their customers remain as spam free as possible.  SpamAssassin, the anti-spam software bundled with cPanel, will do a sufficient job in filtering some spam, but to truly get bragging rights as a spam-free host, you need something stronger.

The people over at SpamExperts provide a valuable service in protecting you from unwanted junk email.  It smoothly integrates with cPanel and communicates with the SpamExperts servers via API to  allow your users to set up and configure their own spam protection.  As with most hosted spam solutions, you will need to set up your domains’ MX records to point to their servers so they can begin filtering your email:

Upon switching our email to use SpamExperts, we noticed that all of our “legitimate” email was delivered, and we received absolutely no spam!  Email delivery was also very fast.  Some of the features we also found useful are:

• Availability of raw logs tracking every email that was process by their system
• Quarantine area, for viewing email that was captured as spam
• Statistics
• Ability to instantly report spam
• Extension filtering
• Sender and recipient blacklists and whitelists
• API for cPanel integration

Pricing is also very reasonable to fit any hosting provider’s budget – as low as $.41USD per domain per year, with bulk purchase options available. It’s very easy to offer the additional spam protection to your customers as either a paid or free service.

cPanel Integration

The email firewall products come with a cPanel plugin called “ProSpamFilter2″ which allow the administrator to manage the server-wide settings, as well as end users in cPanel to customize spam protection for their own domains:

We at The cPanel Admin fully recommend SpamExperts as an excellent anti-spam addition to your server.  You can offer it as part of your hosting package, or as a value-add upgrade feature.

A common request we get is how a user can automatically BCC outgoing email to another address. This is most often requested by law firms that need to be able to retain copies of communication. Forwarding incoming email is easy – there’s already a section in cPanel for it. However, if you want to forward outgoing email there’s a small twist.

First, go to WHM > Exim Configuration Editor, then click “Advanced Editor” at the bottom. Don’t make any changes, just click “save”. This should create a file on the server called /etc/exim.conf.local.  From command line as root, you’ll need to edit /etc/exim.conf.local and add some options to the @CONFIG@ and @TRANSPORTSTART@ sections. The end result would look like this (plus any edits you may have already made):


@AUTH@

@BEGINACL@

@CONFIG@
system_filter_directory_transport = local_copy_outgoing

@DIRECTOREND@

@DIRECTORMIDDLE@

@DIRECTORSTART@

@ENDACL@

@RETRYEND@

@RETRYSTART@

@REWRITE@

@ROUTEREND@

@ROUTERSTART@

@TRANSPORTEND@

@TRANSPORTMIDDLE@

@TRANSPORTSTART@
local_copy_outgoing:
driver = appendfile
delivery_date_add
envelope_to_add
return_path_add
group = cpaneleximfilter
user = cpaneleximfilter
mode = 0660
maildir_format = true
create_directory = true

Now, make a copy of the existing system filter file to a custom location, so cPanel updates don’t mess with it:

cp  /etc/cpanel_exim_system_filter /etc/cpanel_exim_system_filter_custom

Edit /etc/exim.conf.localops and change the “systemfilter” option to match our new file:

systemfilter=/etc/cpanel_exim_system_filter_custom

Now, open /etc/cpanel_exim_system_filter_custom and add the following block of code to the end of the file:

if $header_from: contains "@senderdomain.com"
then
unseen deliver "other@emailaddress"
endif

Replace the blue text with the sender domain (or email address)  and the email address that outgoing mail should be copied to.

Apply these new settings to the exim.conf and restart:

/scripts/buildeximconf
service exim restart

Now when you send email from the email address or domain specified in the filter file, it will be copied to the other email address.

I was working a case last week for a new client that’s been hosting local business websites for over three years. As with all of my case studies, I did a short survey of the client’s infrastructure, processes, and security. As I progressed through the survey I started to notice that every aspect of his business and network were organized and documented almost compulsively, indicative of someone who takes pride in what their business offers. I wondered – why does this guy even need me?

The answer to my question became clear when I got to the disaster recovery portion of the survey. When I asked, “What backup measures do you have in place for internal and client data?” To my dismay, he replied:

“Backups aren’t all that important right now”

OK, rewind. You’re a hosting provider, and you’re hosting data for businesses. You’re not hosting some college kid’s basketball website, or your mom’s quilting club pictures. You’re hosting real data, for real companies that have a reasonable expectation of reliability and security. So my next question was, “What are you going to do if one of your servers explodes?” Silence.

The fact that this guy was lucky to have gone three years without any sort of server disaster is a miracle. It happens to the best of us – hard drives fail, RAID controllers fault, and servers sometimes just – die. It doesn’t matter whether you paid $500 for your server or $5000 – a failure can happen to anyone. Would you rather wait until you have a disaster to start planning for it?

While most people agree that having a reliable backup system for their data is a concept that can go unsaid, you’d be surprised how many business don’t back up data at all.  Think about how much it would cost you to purchase an additional server to back up your data to. Now think about how much it would cause you if all your data suddenly disappeared.

“Well, it’s their responsibility to back up their own websites,” said my client.

I was talking about this with Vanessa today hoping that she could change the client’s mind about the importance of data backups.  This is an excerpt from her actual email:

While most hosting providers would agree that it is indeed the responsibility of a site owner to retain backups of their own data, you may notice that they still back up their servers on a regular basis.  It’s a known fact that most website owners don’t back up their own sites.  Some assume you’re doing it, some think it’s not important, and others just don’t know how.  You may think it’s their fault if they lose everything, but when it comes down to it, when your server crashes and you lose all your customers’ data, it’s you they are going to blame. Whether you agree or not is irrelevant – they are going to find another host.

According to Vanessa, InMotion Hosting has their backup recovery plan down to a science.  They back up their customers’ data on a regular basis, and have a specific recovery plan in place that allows them to restore service and data to customers within minutes in the event of an unexpected server failure.   When explaining this to my client, she said:

Even if the data loss is at the hand of the customer, I can’t tell you how many times they’ve been relieved to find out that we have a backup.  It’s the difference between exceeding the customer’s expectations, and letting them down. You don’t want to let them down, do you?

I’d have to agree.

When running a cPanel update, you may get the following error:

/scripts/upcp
/scripts/upcp syntax OK
Running Futex Check/Fix……Done
Undefined subroutine &Cpanel::Update::automatic_updates_enabled called at /scripts/upcp line 273.

To fix this, run the following command:

wget -O /root/updatenow.static http://httpupdate.cpanel.net/cpanelsync/RELEASE/scripts/updatenow.static && perl /root/updatenow.static --manual

Then you can proceed running a cPanel update:

/scripts/upcp --force

If you go into WHM > List Accounts and all your accounts are showing an unlimited quota and 0 disk space usage, you probably have a quota problem.

First thing, try running

/scripts/fixquotas

This’ll take a while depending on your current disk usage, but will usually fix the problem. If it doesn’t, make sure that quotas are on for your filesystem.  Look at /etc/fstab, and in the fourth column (the options column), make sure that ‘usrquota’ is specified for the file system that contains your user home folders.  For example, this is how ours looks:

LABEL=/     /           ext3    defaults,noatime,usrquota 0  0
LABEL=/home    /home          ext3    defaults,noatime,usrquota 0  0

If /home is part of the / partition, then the quotas need to be applied to / .

Now, remount the partitions with new options.  For the root ( / ) partition for example, you’d run the following command:

mount -o remount /

Then run the following command to update quotas:

/scripts/initquotas

If the problem indicates a problem writing to /aquota.user or similar, make sure the file isn’t set to immutable:

chattr -ia /aquota.user

After following these steps, if you’re still having a problem feel free to submit a support ticket and we’ll help you out!