We are in a process to replace our Internet security infrastructure. It’s currently based on windows 2000 proxy server and Websense web filter 6.3 to “monitor” and control users internet browsing. For those who don’t know Websense, it’s on of the leader in internet security solutions. It’s main product is Websense security gateway, which provides a complete infrastructure to manage internet web access. We always had trouble with Websense filter, especially the reporting tool as it’s not really reliable… And as you would expect in a project to replace 3 proxy servers and all the stuff we use around, it’s not straightforward. First of all, the gateway can be installed only on Linux Red Hat 4.5 (yes, 4.5 not 5.4!) We thought it was a minimum requirement but it’s actually not! if everything seems to work fine, you can’t download the database because you have the wrong kernel (can’t see the relation between a download and a kernel though :p ) So it basically means that something on the code check that you have the specific kernel… I don;t really understand but fair enough and let’s go for a RH4.5!

They also told us that their support for MySQL was crap so they highly recommend to use MSSQL! Great! what if you install MySQL then? you’re not supported ? If it is so bad, why do they even give the customer the possibility to choose? And why someone from the company dare to say that their support of this database is bad? It’s not really a plus for the company… You tend to hide your weakness, not to expose them :) but whatever…

But the main problem we have now is that you can’t stream a video with real player! I don’t know if it’s a problem that affect more than this type of stream but it’s not cool… We raised a ticket with them regarding this issue and as the could reproduce it every time, they stated that it was a bug. They’ve been investigating the bug for more than two weeks now and they apparently haven’t figured out why this is happening!!!

Therefore, as long as the bug is not fixed, the project is frozen and we can’t go live ^_^. I love bugs. I hope they are going to come with a nicer answer than with the last we had with the reporting system! (not supported anymore because of too much bug… Thanks I noticed ^_^)

Sounds like it’s going to be a lot of  fun to support this system at work :p

I’ve had some testing to do lately. We want to see what 2008 server could bring to us, and more specifically, if we could replace our current very old NAS header by a 2008 Storage server cluster. And to carry out that piece of work, I decided to go for virtualization.

So what do you have to know when you want to go for a cluster of two Windows server virtualized on ESXi 3.5? You probably already know that the quorum is a disk shared between the nodes of the cluster and contain the cluster config data (of course there is different way for clustering a windows server but her I’m talking about the only one you can do with 2 servers: the standard quorum cluster). In order to share the quorum, you’ve got two possibilities.

You an either go for provisioning a SAN disk an attach it to your hosts, or use a virtual disk created in ESX. The second way got two path. If you use Windows 2003 server, you can directly share do it by using a flat disk and a LSI Logic SCSI controller. But if you want to do this on a Windows 2008 sever, because it support only SCSI 3 Persistent Reservations and ESXi3.5 only V2, yo have to use an iSCSI initiator within the guest as well as a ISCSI storage target.

I won’t describe how to attach a SAN disk to your windows as there is no point to do so, if you don’t have an IBM SAN with SVC and mcdata switches, it will be completely different…

So I’ll describe the first and easiest way on that post, and the win 2008 way in another post.

Method 1: use a shared virtual disk

you first need to create a flat disk. You can’t do that within the infrastructure client, so we’ll use the remote CLI. The command is:

vmkfstools.pl –c <Size of Volume in Bytes> –d thick –a lsilogic <path to Virtual disk>
which gives
vmkfstools.pl –c 4G –d thick –a lsilogic [DATASTORE1] cluster/quorum.vmdk

Make sure you’re on the good directory before you type the command (C:\Program Files\VMware\VMware VI Remote CLI\bin by default).

You may also want to add your profile –sessionfile <path_of_profile> or if you don’t have a profile, you can either specify the server you want to reach in the command –server <esxi_server> or by setting an environment variable: set VI_SERVER=<your_server_name>

Then you’ll just have to attach your new disk (add > hard disk > use an existing Vdisk …) and an LSI logic SCSI controller to both of the VM. Make sure your disk is connected to the good controller if you’ve got more than one. The bus sharing of the controller must be set to virtual or physical, depending on your needs.

There you are! You can start configuring your windows 2003 cluster! But as I said before this doesn’t work for windows 2008… So the 2nd par will explain you the workaround.

On Windows 2008, hibernation is activated by default for system with less than 4GB of ram. It’s not like if microsoft thought your server may want to take a nape but in case that there is a power failure and switch to UPS. It make sense, why not, but it’s not useful for everyone. The file hiberfil.sys is created by default on the system drive even if it’s not used in order for the system to make sure there will be always enouth room. You may want to just move the file to another drive, but unfortunately you can’t… Or at list i didn’t manage to do it. You can still turn hivernation off though. En even change the size of the file (doesn’t really make sense as it is created by default at the size of your ram). Here the command you can run on a command prompt with admin right:

change the size of the file (this is a percentage of your ram – min 50%)

POWERCFG -H -Size <PercentSize>

Desactivate hibernation

POWERCFG -H OFF

Learn more about power management

POWERCFG /?

Red Hat has published it’s results from the first quarter and the only thing we can say is that RH is doing well! results up 11% from the year ago quarter. The net income for the quarter is 18.5 millions, a 7.1% growth.
This is realy good and show that the open sources technologies get more and more interest from companies, especially on recession !
An other interesting figure is that now 61% of the sales are made by the business partner network. It show that the can rely on more that just themself and they need it to keep growing.

An interesting annonce was that they annonced the beta release of a portfolio of products around virtualisation (Red Hat hypervisor is KVM).

Do you have a Windows DNS? and you need to add 300 A record on it… So do I!

In order do quickly do so, I wrote a small script: dnsupdate

you just need a txt file next to it called dnsupdate.txt populated by NAME,IP_ADDRESS (only 1 per line), and the script will go through the list and add an A entry for each of them.

Hope this will be useful for someone else :D

If you still use VMware GSX and/or VMware server 1, and some of your guest are Linux, you may have already encountered this issue. This might seems not useful as those VMware products are quite old now, but when it’s on production, you don’t really get rid of something easily :D

The clock of the system is always de-sync even with a ntp server configured on it. This is a big issue when you run automated script with cron, and you’re system backup is done at 10am instead of 5am… That’s because your CPU clock is running too fast or too slowly. This is a known issue on those product as they don’t support host power management features on Linux properly.

To sort this out, you just to do the following:

you first need to know your CPU speed

~cat /proc/cpuinfo | grep MHz

cpu MHz         : 3601.737

edit the file /etc/vmware/config and add

host.cpukHz = 3601737
host.noTSC = TRUE
ptsc.noTSC = TRUE

host.cpukHz change depending the output of the fisrt command. It’s basically the CPU speed in Hz.

if not already done, you need to install the VMware tools on the guest and configure the time synchronisation. You can do it either by editing a file or in the options of the VM tools in a graphical interface.
The file to edit is the .vmx file and the line to add is tools.syncTime = “TRUE”

Once it’s all done, you’ll have to reboot the guest and restart VMware on the host.

If the problem is not fixed, this may be addressed by disabling power management altogether. This can be done by adding “apm=off acpi=off noapic” to the boot entry of the host’s /etc/grub.conf file.

I’ve got 2 TS-409 NAS appliance from QNAP. They are configured in raid5 and data are synchronised between them. As there is no possibilities to monitor the disk usage (no SNMP or any other things like that), I decided to create a little script which send an email when the disk usage reach 90%

This is not a big deal as it only contain few lines and is really straightforward, but I actually spent 2 hours to figure out how I could send this bloody email… Again, nothing difficult, just that I started in the wrong direction. i wanted to send it through a telnet connection on port 25 of the mail server… The only problem is that it was fine on any linux with a proper telnet binary, but the telnet implementation on busybox is different and doesn’t work when I try to telnet the smtp port of our exchange… So I just used ssmtp, already included and it works like a charm.

The script could obviously report a lot more but that’s what I’ve been asked :D

#!/bin/sh

FROM="FROM: Nas1@xxx.yyy"
TO="someone@xxx.yyy"
RCPT="TO: $TO"
SUB="Subject: Test Alert"
BODY="The NAS is filled at XXX%"

dfresult=`df -h /dev/md0 | grep /dev`
position=`expr index "$dfresult" %`
let position=$position-3

if [ ! ${dfresult:$position:1} == "" ];then
   value=${dfresult:$position:2}
   if [ $value -ge 90 ];then
      BODY=${BODY/XXX/$value}
      (echo "$RCPT";echo "$FROM";echo "$SUB";echo "";echo "$BODY") | ssmtp $TO
   else
      echo "not critical"
   fi
else
   echo "less than 10%"
fi

exit 0

It wouldn’t be difficult to add few more things like add how many GB left, and things like that but I’ll let you do it :D

Here we are! The new kernel has been released today after 8 release candidate and bring some interesting new features along driver update and bugfix. I won’t go into details but here are the main thinks:

• - Improvement on Ext3/4
• - The file system NILFS (New Implementation of a Log-structured File System) has entered in the mainline. Developped by the Japanese company NTT, this FS improve writs on the detriment of reads. the general performances seems even better than BTRFS though
• - Exofs (Extended Object File System) is an other “new” FS for SCSI disks, who is based on object and metadata instead of low level blocks an sectors and is developed by SUN
• - POHMELFS (Parallel Optimized Host Message Exchange Layered File System) is again a new FS, but it’s purpose is having a high performances network file system (like NFS, CIFS, …) but with paralleled access to nodes.
• - FS-Cache has finally been ported on the kernel and allow to improve network file systems performances by caching the data.
• - TOMOYO is the new security module created to be much easier to manage than SELinux.
• - IBM has pushed a set of patch to port Integrity management in the kernel. This provide some sort of assurance that the files on the system have not been messed with.
• - The mutex code has been amended to introduce a new mechanism in it. the patch changes mutex contention behaviour such that it will sometimes busy wait on acquisition to move its behaviour closer to that of spinlocks.
• - Reliable Datagram Sockets (RDS) protocol is now supported and improve clusters performances by bringing improvement on process when the talks to other processes on the cluster.
• - The asynchronous function call infrastructure is now activated. It allows to query storage devices asynchronously on kernel boot, which makes it quicker.
• - The new MicroBlaze architecture is now suported
• - The integrity controle on block devices introduced on 2.6.27 has been extended to raid configurations.
• - The kernel can be compressed with bzip2 and lzma algorithms to gain between 10% and 33% of the gziped sizw
• - Improvements on dm-crypt for multi-core CPUs
• - Harware acceleration AES encryption for futur Intel Westmere CPU
• - Support for AMD/ATI R6xx/R7xx graphic cards
• - Extend of the 4GB ram limit for ARM architecture
• - General improvements of ftrace (Linux Dtrace like)
• - Enhancement of cpufreq
• - SSP (Secure Simple Pairing) for bluetooth 2.1 is now suported
• - Enhancement of SSD drives performances through the implementation of the TRIM command
• - Support of the SEGA Dreamcast console improved
• - Driver update for ALSA (1.0.20) and V4L

As you can see there is quite a few new thinks about file systems. I might post something about all that as it’s been a quite busy area lately with EXT4, Btrfs, …
The Kernel 2.6.30 represents 11912 patches and one million and half lines of code modified, which makes it a quite big development cycle.

More details about the new drivers
This is just a quick look at what’s new on the Kernel, you can get much more informations there

I recently found a tool that help me a lot on my duties at work! It’s called PuTTy Connection Manager. No it’s not just the putty that everyone knows. It’s an application that actually come on top of putty to enhance it. As you can see below, the main feature is tabs. But don’t tell me, if you are an unfortunate user of putty like me (not that putty is not good but I’d prefer to use a Linux OS at work. Therefore, I wouldn’t have to use putty…), that you never thought that putty would be much better with tabs!!!

PuTTy Connection manager

PuTTy Connection Manager require the .NET framwork V2 and Encryption library 0.6.0.0 if you want to encrypt your database.

Unfortunately, the project (French :D) seems to be dead and it’s not open sources… But it got to a point where I don’t really need much more features and there is no important bug. If you know any PuTTy-like tools better than this one, and preferably still alive, let me know, it might be interesting :D

You can download the latest version here
And the website is here

Nothing new here but I’ll use this post as a quick reminder to enable kernel dump on red hat enterprise 4. Quite useful when you don’t know why your box keep crashing this week when it was perfectly fine for the last 2 years :D

Red Hat historically use netdump in order to save a kernel dump file but since RHEL 4, they introduced diskdump. Which is good as I don’t want to bother set-up a netdump server (even if there is few interesting thinks in favour of netdump, like “what if your system doesn’t recover…”). You can easily understand the main difference between them… The first one create the file over the network using a server (like if you have a syslog server) while the other one use a local disk.
Here the few steps to follow:

Everything should be already installed, but in case it’s not on your system:

# up2date crash diskdumputils

crash is an utility used to analyse the dump file.

Check the loaded kernel modules, if not already loaded, then do it.

# lsmod | grep diskdump
# modprob diskdump

Configure what partition you want to use. 2 choices: 1st you have a partition only for this purpose, 2nd you can use a swap partition. if you don’t use a swap partition, you’ll have to format it. The size of the partition you choose must be at least the size of your ram.

# cat /proc/swaps
# vi /etc/sysconfig/diskdump

You can specify more than one partition if you separate them by ‘:’

DEVICE=/dev/sda4:/dev/sda5

If you didn’t choose a swap partition, then you have to format your device in order use it as a dump device.

# service diskdump initialformat

Tell your system to start diskdump automatically at the next reboot and start it.

# chkconfig diskdump on
# service diskdump start

If you get a warning, it may just means that your partition is not good (wrong partition specified in /etc/sysconfig/diskdump, or forgot to format, …)

check that the module works fine. the output must be something like the following:

# cat /proc/diskdump

# sample_rate: 8
# block_order: 2
# fallback_on_err: 1
# allow_risky_dumps: 1
# dump_level: 0
# compress: 0
# total_blocks: 2097059
#
sda4 4401810 4192965
sda5 8594838 4192902

When a crash occures, the data from the disk dump partition are not directly readable. You need to gather them in a file on a readable partition. This is done with the following command:

# savecore -vD /dev/sda4

You can add this command to /etc/rc.local if you want to run it automatically after a crash.

This is it! Next time your kernel crashes, you’ll get a file called vmcore stored in /var/crash/127.0.0.1-\/. If for any reason the dump is not complete, the file will then be named vmcore-incomplete

You can test if it works fine by doing one of the following things. It will result on you server crashing therfore creating a dumpfile.

# Alt-SysRq-C or
# echo c > /proc/sysrq-trigger or
# echo 1 > /proc/sys/kernel/sysrq

you can also quickly compile the following code [cc -c -I/usr/src/linux/include panic.c]
and load this nice module :D [insmod panic.o]

#### panic.c #####

#define __KERNEL__
# MODULE
# include init_module(void)

int init_module (void)
	{
	panic(" panic has been called");
	return 0;
	}

now in order to analyse your dump, crash require the kernel debuginfo package corresponding to the kernel you are running. You can find it there. Then you need to install it.

wget http://updates.redhat.com/enterprise/4AS/en/os/Debuginfo/i386/RPMS/
kernel-debuginfo-2.6.9-89.EL.i686.rpm && \
     su -c "rpm -Uvh kernel-debuginfo-2.6.9-89.EL.i686.rpm"

After your kernel panic and a reboot, you can now run crash with the following arguments (vmlinux_path vmcore_path).

# crash /usr/lib/debug/lib/modules/2.6.9-89.EL/vmlinux \
        /var/crash/127.0.0.1-2009-06-07-14\:44/vmcore

OUTPUT
      KERNEL: /usr/lib/debug/lib/modules/2.6.9-89.EL/vmlinux
    DUMPFILE: /var/crash/127.0.0.1-2009-06-07-14:44/vmcore
        CPUS: 1
        DATE: Tue Jun  8 15:02:44 2009
      UPTIME: 00:51:24
LOAD AVERAGE: 0.05, 0.05, 0.00
       TASKS: 81
    NODENAME: rhxxx.test.xxx.org.uk
     RELEASE: 2.6.9-89.EL
     VERSION: #1 Fri Feb 24 16:44:51 EST 2006
     MACHINE: i686  (3600 Mhz)
      MEMORY: 1.5 GB
         PID: 3514
     COMMAND: "crash"
        TASK: f562ecd0  [THREAD_INFO: f4b7d000]
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)

I leave you find out all the things you can do with crash (try a “man crash” or help once you ran crash) but the command ‘log’ and ’sys’ will be the one you are going to use the most.

you can find much more details here
Enjoy :D