Reverse Proxy Add Forward Module for Apache (mod_rpaf)

When your apache web server is behind a load balancer (or reverse proxy), the most common complaint is that all the requests come from the same IP address.

Although you can use %{X-Forwarded-For}i instead of %h on your apache LogFormat and in case you are using php http server vars, you can use $_SERVER[‘HTTP_X_FORWARDED_FOR’]  instead of $_SERVER[‘REMOTE_ADDR’] , a great solution to solve it without changing your website code or the apache LogFormat  is by using mod_rpaf.

Furthermore mod_rpaf allows you to define what are the proxy/load balancer IP addresses and it takes the last IP from X-Forwarded-For header, discarding the IP addresses you’ve defined.

The installation and configuration  is pretty easy on Debian:

The installation enables the module automatically, so you only have to define your proxy/load balancer IP addresses in /etc/apache2/mods-enabled/rpaf.conf and restart apache

After that, your backend Apache server will work with the real IP address and not with the proxy/load balancer IP.

Highly Available Shared Storage with NFS,Heartbeat and DRBD

The following is a possible solution for those who need a highly available shared storage.

1 – What you are going to need
2 – Scenario

3 – Installing and Configuring DRBD

3.1 – Install drbd8-utils in both nfs servers:

 

3.2 – Configure drbd conf files in both nfs servers:

  • /etc/drbd.d/global_common.conf:

 

  • /etc/drbd.d/data.res:

*I previously created an LVM volume named data (/dev/vg/data), but you can also put another disk (/dev/sdX), or a dedicated partition for drbd(/dev/sdXn)

3.3 – Setting up DRBD on the active nfs server:

 
3.3.1 – Check out whether drbd is working

*Note that the passive nfs server is in “Unknown state” because it’s not configured yet.

3.4 – Give format to the drbd device

Now you can try to mount it to verify that it’s alright:

3.5 – Setting up DRBD on the passive nfs server:

3.5.1 – Check out whether drbd is working

*Depending on your drdb device size, it could take a while to be fully synchronized

4 – Setting up NFS Kernel Server

4.1 – Install all the NFS Server required packages:

4.2 – Configure NFS server conf file in both nfs servers:

Just add this line in /etc/exports:

*I’ve just added rw and no_subtree_check to improve the reliability but of course you can add all the options you need.

And apply the changes in both nfs servers

5 – The Heartbeat setup

5.1 – Install heartbeat package in both nfs servers:

5.2 – Configuring heartbeat conf files:

  • /etc/ha.d/authkeys (the same configuration in both servers):

*This file must have only read/write permissions for root (chmod 600 /etc/ha.d/authkeys && chown root:root /etc/ha.d/authkeys)

  • /etc/ha.d/haresources (the same configuration in both servers):

*10.0.3.10 is the cluster virtual IP

  • /etc/ha.d/ha.cf (nfs-server01):

  • /etc/ha.d/ha.cf (nfs-server02):

5.3 – Start heartbeat on both servers:

Now, you can try the high availability rebooting the active nfs server and checking out whether passive server is taking the control.

6 – Finally, configure your nfs clients

6.1 – Install nfs-common and autofs packages on your nfs clients:

6.2 – Configure /etc/auto.nfs file:
Add this line (just an example):

That’s all!

Next time I’ll explain another highly available shared storage solution with GlusterFS which has also worked fine for me.

MySQL unauthenticated user connections

If your mysql server is full of unauthenticated user connections, check that the DNS you are using (/etc/resolv.conf) are working fine. Most probably there’s an issue with them.

However, I suggest you disable the DNS hostname lookup to solve it, and furthermore, to optimze your MySQL response time.

To disable it, add skip-name-resolve in your my.cnf file (inside [mysqld]) and restart mysql.

Finally, take into account that when DNS hostname lookup is disabled, you can only use IP addresses on mysql grant tables.

How to figure out what is causing an Apache Segmentation Fault

Having an apache segmentation fault on your server can drive you crazy trying to solve it… So, I’m gonna explain you how I figured out what was causing the last apache segfault I had.

My server was running under Debian Squeeze, with Apache 2.2 and PHP 5.3, and I got randomly segmentation faults:

I supposed the problem was on some php extension, so I deactivated some of them…but the problem was still there :(

After spending a lot of time looking for what was causing it, I decided to debug apache with gdb. I installed it through backports because I needed an upper version than gdb 7.0 (7.0.1-2 is the current version on squeeze stable repository) otherwise I got this: “warning: The current binary is a PIE (Position Independent Executable), which GDB does NOT currently support. Most debugger features will fail if used in this session.” when I was trying to debug apache:

Well, I had the debugger installed, but I also needed the php and apache debug symbols:

Once I had installed all I needed, I configured apache to create a core dump file every time it had a segfault by adding in /etc/apache2/httpd.conf the following line:

Previously, I had created the /tmp/apache-coredumps directory

By default, my system core file size was 0, so I had to change it through ulimit:

I restarted apache, and I only had to wait for the next Segmentation fault… and I didn’t have to wait too much …

Then, as the message said, I had a core file in /tmp/apache-coredumps, and I was closer to finding out what was causing segfaults :)

Therefore, it was time to use gdb:

I had a problem with php as I thought, and I was able to find more information about it. And after Googling for a few minutes, I found there is a bug with the PHP and APC version that I was using! (more info here)

After that, I could solve the problem and until now I haven’t got more segfaults on this server :)

Using gdb will not solve your segmentation faults, but at least, you will be able to figure out what’s causing it, and you’ll get more info to solve it.

Server Monitoring with Munin

Munin is a very useful application for anyone who wants to monitor their servers. It works with nodes (clients) and a master (server) which connects to all the nodes getting all the data to make the graphs (which can be seen through any web browser)

There are a lot of available plugins, and you can also write your own plugins. Therefore, you’ll be able to monitor almost everything you want. So I totally recommend it :)

Server Installation

Assuming you already have a web server installed (which is necessary to see the graphs through a browser), you have to install munin server using the following command (on Debian/Ubuntu):

or by using this one if you’re under FreeBSD:

Perl, rrdtool, and other libraries will be installed if you don’t have them yet.

Node Installation

Now it’s the turn to install munin-node on your clients. It’s as simple as:

*on Debian/Ubuntu

*On FreeBSD. Remember to enable it by adding munin_node_enable = YES in /etc/rc.conf

Server Configuration

The main munin server config file is munin.conf (located in /etc/munin/ or /usr/local/etc depending on whether you are using linux or freebsd) and in this file you have to configure the paths where munin will save the html,rrd and log files:

In this file, you also have to add all the nodes you want to monitor:

You don’t have to restart any daemon when you modify this file because there’s a cron (munin-cron) which runs all the necessary programs (munin-update, munin-graph and munin-html) every 5 minutes, although you can change that according to your needs.

Node Configuration

The node config file is munin-node.conf and you have to allow the master ip address in this file (by default, only localhost is allowed) by adding:

*Supposing the munin server ip is 172.20.52.4

Munin plugins are located in /usr/share/munin/plugins/ but the enabled plugins are in /etc/munin/plugins:

*By defalut, all basic plugins should be enabled, but you can use munin-node-configure to enable more plugins

Once your node configuration is ready, you have to restart munin-node daemon to apply the changes:

or if you are under freebsd:

Once all is configured..

After a few minutes, in my case in http://172.20.52.4/munin, there will be an index with all your nodes and their graphs


*Example of one of my servers cpu graph