Saskaweb SysAdmin Tales

I was (un)pleasantly surprised to find one of our hosted Exchange customers uses a third-party hosted app that sends email to coworkers. The application is very important to this customer, but it’s been a bit of a challenge to support the email application for me.

It looks to use the gd library to craft a pdf and then attach it a la base64 to an email. The customer uses their email address for the from field (had to change the spf record) and sends it to a co-worker.

The main problem is that the attachment is corrupted and can not be opened on the Exchange server. The workaround was to setup a mailserver just for this one customer. Email sent to non-Exchange users on our standard hosted email server is relayed to the proper server.  Email sent to the Exchange users are held in an IMAP/POP server and then pulled into the information store by the Exchange server.  Far from optimal, but it works.

Hopefully the developers with the third party app can help keep this attachment sound.

Hurray - the cluster has finally landed!

The tests showed promising results so I decided on it for our webhosting and offsite backup stores.

We now have two fileservers with a number of clients using client-side automatic file replication (AFR). The performance has been adequate.

There is still room for improvement with our gluster deployment. I couldn’t get the custom fuse compiled on Ubuntu, so there’s no NFS re-export at present. As I determined that FreeBSD can not run glusterfs with the stability required, I have confined the cluster to only our Linux servers.

Most of our spam filters use FreeBSD and currently our spam corpus is stored on independent disks. It’d be nice to consolidate them but our demands for stability dictate that we’ll have to wait until either nfs re-export is available or the glusterfs client can run smoothly on FreeBSD.

I’m liking it tho — getting writes at gigE wire speed with full redundancy is nice

Isn’t that just how it goes?

It looks like glusterfs might have fit my new fileserver requirements perfectly — and I could’ve even stayed with FreeBSD.

For those who don’t know about glusterfs (like me a week ago) - I highly recommend giving it a look.  It is a lot easier to comprehend than other cluster filesystems.  I bailed on gfs and ocfs2 because the idea of maintaining volumes in some sort of drbd netherworld  with a collection of scripts seemed a bit too risky.  I’d be willing to give the filesystems a try if I had a nice shared storage, but it’s either a lot of money or else buying an older surestore or powervault scsi das and have to live with the obscene power usage.  And from what I’ve read on the scsi powervaults, I don’t want one.

But there’s glusterfs… It’s single greatest thing for me is that it uses fuse and sits on top of a standard filesystem.  Yep.  If everything went to pot you still have the data in the filesystem that you’re most familiar working with — it doesn’t look as easy to back out from gfs

I haven’t installed it yet but I should have some new boxes arriving soon.  I’m curious about quota management, backups, and what to do with the few iscsi volumes I have.  Maybe glusterfs will make it into the next round of hardware updates.

For a few days I’ve been running the 8021q module on a fileserver.  I do have multiple NICs in that machine as I wanted a dedicated interface for ISCSI and another one for pulling backups.

I had a few weird experiences and it might be due to the fact that most of my servers are on VLAN 1.  When I was running a backup, I found that most of the bandwidth seemed to be going through the interface with VLANs attached to it, and not the dedicated interface.  I also found some nice ARP messages on a FreeBSD box that an IP address was changing MACs.  I’d restrict the trunk but of course VLAN 1 can’t be restricted on a trunk.

So I’m not sure if its a problem with the 8021q module or just the fact that I have too much on VLAN 1, but I ended up ditching VLANs on the fileserver.

I’ve just finished moving data over to a new fileserver.  And for the first time, I’m trusting file storage to Linux… Ubuntu Hardy Heron, to be precise.

After much deliberation regarding clustering filesystems and block-level replication, I’ve decided to keep it simple.  The fileserver is a 2TB array on an XFS filesystem.  Replication is passed to a read-only failover server by rsync and failover is handled by scripts and proxies.  Simple and effective.  The last thing I want is a system that introduces downtime when dealing with complexities that were supposed to save me from downtime.

I needed a high performance fs with quotas and nfs support, so I decided on XFS.  I’ve never really liked ext(x) filesystems and I’ve heard a lot of good things about XFS.

Moving hundreds of gigabytes from a UFS FreeBSD system to an Ubuntu XFS system was not the easiest thing.  I had moved most of the data over before quotas were enabled on the XFS (you have to edit GRUB to enable XFS quotas on / — who knew??).  I mounted the old data via NFS and then rsynced (-av) it over.  I needed root access to the source and I decided to do it this way instead of changing my sshd on the source.  So anyway, not sure what happened, but after I created the uid’s my xfs_quota gave some pretty alarming stats.

xfs_quota -x -c report / showed that most of my uid’s were using 8E (yep — exabytes) of data!  Being that their quotas were 30G and the whole array is only 2TB, I found this hard to be believe.  The fix?  Well, I created a new user with a new uid, set the quota, and then did a chown -R.  It worked — block usage was back below 30GB.  Don’t ask — just move on

Although it’s pretty hard to beat apt-get for simplicity, I still kind of know why I stick to FreeBSD.  There are somethings that are vastly superior with this new fileserver:

1)  FreeBSD 7.0 couldn’t get the right geometry from the 3ware RAID controller.  I tried and tried and then decided to move to Ubuntu with XFS.

2)  MySQL queries are still a million times faster than FreeBSD — even in comparison to FBSD 7.0 with its new malloc().  Even though it’s a fileserver, I still need to access the ndbcluster.

3)  With LVM and XFS, I should be able to resize the partitions without too much pain.  On FreeBSD, I’m still doing very time-consuming dump and restore’s

4)  Look at all the stuff I can install!  And no compiling either!

I don’t think the weaknesses are enough to make me switch back to a FBSD fileserver, but here they are:

1)  Damn Linux binaries don’t seem to have the same options as FBSD.  Sure — sometimes they have better features, but Murphy makes it so only the bins I regularly use are weaker on Linux.  Where’s the -d option for du?

2)  xfs_quota support is a bit alien on Ubuntu — I do not have feelings of great confidence

3) The NFS-quota support on FBSD 7.0 worked flawlessly for me… from what I’ve read I’m not even going to try it with Linux.

4)  Well I guess it’s not a big deal that the NFS-quota wouldn’t work — the fscking Linux fileserver can’t seem to use the bi-map from my FBSD router as well as the old FBSD fileserver did.  Suddenly, ftp can’t connect — srry, it can connect but can’t do an ls or send data.

5)  I set up iptables (still not a big fan of the syntax), did iptables-save, rebooted and watched them all disappear.  Nice.  Thanks Ubuntu.  I would’ve stuck with ufw, but finding useful documentation took more time than it was worth.  Fix?  Script in iptables-restore during ifup.

6) What did Ubuntu do to pure-ftpd?  Do you seriously have to put each variable as a separate file in the conf directory?  Wow — that sucks — what was the logic here??

Anywho, the fileserver is up and happy and I like looking at all that free space in my cacti graphs.  All in all, I’m pleased with Ubuntu and XFS and I managed to do the moving without causing any downtime.

I’ve abandoned using FreeBSD’s gmirror with two ggate drives as a storage system. The performance was adequate, but there was no failover. When testing, I unplugged one of the ggate targets and the client froze all activity on that mount. I became less and less pumped over client-side redundant systems like this and am now focused on server-side redundancy.

DRBD? Can something be done with ZFS? Will the performance of GFS or OCFS2 be adequate?

Lots of testing to be done!

Offering offsite backup and now managed dedicated Exchange (each server is allocated 85 GB) is once again starting to push our storage capacity.  I’ve started bringing in parts so the hardware decision portion is largely completed, but the implementation probably won’t be done for another month.

With demands for space rapidly rising (a tenfold increase in the past year), we need to look for a system that can be massively expanded for a reasonable cost.  I’ve decided to migrate our large file storage (most things except standard email) to SATA drives.  You really can’t beat the price and the performance is adequate for our hosting environment.  Sure, I’d love to move to FC, but as a small, upstart company we can’t justify the immediate costs.  SNAP systems are affordable, but I’m sure that I can design a system that is faster, bigger, cheaper, and just as reliable.   Or, I’m up to challenge anyway

More to come…

I have a customer who is customizing some forms in Microsoft Word 2007.  I stopped by for on an onsite service call and found she was having some problems with managing whitespace so I told her to hit the INSERT key to toggle overlay.  To my surprise, it didn’t work.

It turns out that Microsoft has decided the INSERT key is more trouble than it’s worth.  It is disabled by default now in Office 2007.  To enable it (and it only took a quick google), just go to the Office button > Word options > Advanced > Editing options > Use the Insert key to control overtype mode.

The offsite backup business has been doing well so it wasn’t too much of a surprise to find I was running low on diskspace.

In the past, I’ve been adding new disks and expanding volumes but I’m getting pretty tired of how expensive SCSI disks are. Every home computer comes with at least an 80GB drive while I’m still stuck shelling out big coin for smaller SCSI disks.

So why haven’t I moved my storage servers over to SATA? Well, cost is the main issue. Reliable servers that use SATA RAID’s (and I’m not talking about the onboard junk like the Intel Storage Manager) are only available on the latest generation and are still fairly expensive. True, you can get a simple 1u from Dell or HP with some SATA disks, but I want redundant hot-swappable fans and power supplies. Even the relatively cheap Poweredge 2950 still starts at $2300. What I need is a way to use big cheap IDE or SATA disks with my existing servers.

Enter ggate

Ggate has been available for FreeBSD for a few releases now so I thought I’d finally give it a try.  And why not today?  Saskatoon is a rainy mess today, so it’s a good a day as any.

Ggate is similar to ISCSI in that it exports an entire slice at the block-level. A client mounts the slice remotely and it behaves more or less like it was on a local volume. The process turned out to be a snap.

On a test server I installed FreeBSD 7.0 i386 REL on a 250 GB IDE hard drive. I set up the slices with the bulk of it in a large slice at the end. I used sysinstall.

After reboot, I edited /etc/fstab and commented out the big slice (ad0s1g) so it wouldn’t mount on boot. Ggate is not a cluster filesystem: It can only be accessed by one operating system at a time. Then setup the /etc/gg.exports and cat /sbin/ggated >> /etc/rc.local. Done. Reboot and ggated now exports ad0s1g.

For a client, I chose a test webserver node. I mounted the standard /www partition to /www2. I then used ggatec -o rw [ggated server] /dev/ad0s1g. The command output was: ggate0. Nice! Then mount /dev/ggate0 /www. Done! Now I copy some of the web content from /www2 to /www.

At first, disk i/o wasn’t very promising. I was getting around 3MB/s over 100Mbit LAN. I googled around a bit and found some advice:

sysctl net.inet.tcp.sendspace=131072
sysctl net.inet.tcp.recvspace=262144
sysctl kern.ipc.maxsockbuf=1048576
sysctl kern.ipc.nmbclusters=35584

The page where I found this didn’t specify if this was for the server or client, so I entered it on both. (Perhaps it should be tweaked for server and client roles by sendspace, recvspace???) I also found suggestions to bump up the send and receive buffers when launching ggatec.

I tried copying again and was quite pleased to get 11 MB/s — not bad for 100Mbit LAN!

But you’re still using a home pc caliber hard drive

I know, I know, and this was just a trial. I’m going to pick up another one of my lighter 1u servers. Once I have two of these small 1u’s, I can put a nice big drive into each of them and then try my luck at gmirror over two ggate mounted drives. If this works and performance is acceptable then I’ll have a large RAID 1 array giving an existing storage server the benefit of cheap and big hard drives without sacrificing redundancy or performance.

More to come!!!

I’ve been doing custom web applications with MySQL databases for quite a few years now, but I’ve just realized that I’ve never used MySQL to do a good old fashioned mail merge. Today’s project will be to create a mail merge using MySQL server 5.0 and OpenOffice for Windows.

First I created a new table for business contacts. I created an auto-incrementing id field as the primary key (I might want to join with a personal contact table in the future) and fleshed it out with the standards: street address, postal code, city, and province. I guess I could have normalized city and province and probably would have a few years ago, but experience has weathered that away. Ah well — don’t feel too bad if you normalize like a madmen, even the remarkable Linus Pauling went a bit overboard on that vitamin C thing. I digress….

Okay the table is created. I have a username / password that will work from my local subnet. Now what?

I open up OpenOffice Writer and find mail merge. Hmmmm here are the options for csv’s and the like… How about File -> New -> Database… Aha! Does OpenOffice install the Java MySQL connector? Nope. Okay then: http://mysql.org. Beta? Nope. 3.1 it is. Download the msi, double-click and then start clicking the “Next” buttons. Done. Start -> Control Panel -> Administrative Settings -> ODBC. Create new connection, use MySQL driver, set host, username and password. Back to OpenOffice. Hey an SQL query editor that looks remarkably similar to MSSQL. SELECT * FROM business_contacts; hit F5 and voila. Good. Save the query and exit. Back to Writer, follow the mailmerge wizard and it’s pretty well done. Arrange the query output (didn’t love OpenOffice’s interface here). Resize the box. Set the font and then print. Sweet. I just sent a test batch of 15 addresses to be printed on envelopes. Crap…. I forgot how my old laserjet insists on being hand-fed envelopes. Another post I guess