We’ve recently made changes to how our bandwidth is managed.  Saskaweb hosted services will now be restricted to fifty percent of our internet connection.  The other fifty percent has been allocated to client virtual private servers and colocated clients.

VPS and colocated servers are capped at 2Mbit upload and 2Mbit download. We’ll be adding a proxy server to utilize our high download speed connections in the near future so download restrictions will soon disappear.

How did we come up with 2Mbit?  We chose the 2Mbit limit after carefully going over all our client usage graphs.  We found that with all our clients, 95% of their bandwidth usage over five minute intervals was well under 2Mbit.  We chose 2Mbit allowing for growth.

Why did we choose to limit the bandwidth?  We haven’t had any problems yet but the risk of one client using too much bandwidth is too great to ignore.  Our previous limit was much more generous but upon review it was found that one client pushing to this limit would affect other clients.  So instead we chose a flat 2Mbit limit that will be applied to all clients.

Our average peak utilization is around 40% after our recent bandwidth upgrade so we do have room to grow and of course we can always buy more of this expensive resource as required.

Over the weekend we completed a number of upgrades:

• Main storage is now based on Nexenta Core 3.0
• zpools were upgraded
• System memory on main storage was increased to 12GB
• Additional gigabit ethernet interfaces were installed and aggregated (LACP layer 4) for the storage area network
• A new storage switch was deployed
• STP root was reassigned
• CPUs were upgraded on one virtual host server

The performance increase has been fantastic and we’re still not done!  Our new array (two seven U320 disk raidz2 arrays) isn’t in use yet.  We have four SSDs to play around with and lots of benchmarks to run.  We’ll be trying various ZFS tricks like using an SSD for the ZIL and also playing with L2ARC.

A client was having some problems sending to @hotmail.com addresses.  They weren’t on any RBLs or anything that bad, but nonetheless ALL their messages were going to the Junk Folder.  At an onsite consulting visit with this Saskatoon business, Saskaweb found the following resources  to check when delivering email to @hotmail.com:

http://mail.live.com/mail/troubleshooting.aspx

http://mail.live.com/mail/services.aspx

In short – make sure you have an SPF record!  We’re still waiting for hotmail to realize that we’ve added an SPF, but all of our checks now look good.  Fingers crossed!

A few days ago we started using hyperdb for this blog.  Hyperdb is a plugin from WordPress to improve scalability with WordPress sites.  Our MySQL ‘master’ server is replicated over multiple slaves.  Because the replication is only in one direction, writes must occur at the master. 

The majority of traffic on our WordPress sites is read only:  The websites have more visitors than people who create the content.  Wouldn’t it be nice to spread all of this read-only traffic over our multiple MySQL slave servers?

This is where hyperdb steps in.  Hyperdb directs writes to the MySQL master server and reads to the slaves.  This way you can scale out popular sites by splitting the load over multiple database servers.

After changing over this blog as a test, we’re pleased to report no issues with implementing hyperdb.  We will be extending hyperdb to all of our client WordPress sites and implementing it with all new sites to improve their performance.

Days after we ordered parts to optimize our OpenSolaris system (including 4 new SSDs), Oracle has pulled the plug.  This is quite a letdown for Saskaweb.

Where to next?  Should we move over to Nexenta?  Although we’ve been very happy with our OSOL server, ongoing support (at least community driven) is a a requisite.

We’ve just finished purchasing additional bandwidth and should have it activated within a couple days. 

Our 8:30 am spike peaks around 80% of our capacity and averages around 25% capacity for the rest of the business day.  We’re purchasing more bandwidth to provide faster downloads for all our customers and hopefully take the bite out of that 8:30 spike.

This is my first post after adding hyperdb to this blog.  Doesn’t look any different?  Good!

Our master MySQL server is now averaging 30 queries per second and I’m looking to spread some of that load over replicated spares.  (We switched from a cluster to a replicated setup about a month ago.)  After a bit of reading, I decided to give hyperdb a try.  The process wasn’t too bad – though I wish there was a bit more documentation.  Hint:  Make sure PHP error reporting is sent to your apache logs and keep an eye.  After changing the config, hyperdb works fine and is (hopefully) spreading the load over our replicated spares.

We’ve been impressed with ZFS and will add another terabyte and lots of spindles. We’ll take advantage of the downtime to add two more network interfaces which will be bonded for greater bandwidth. With luck, that’ll be online by the end of August.

Our current storage switch is an older Cisco which doesn’t support LACP, so that’ll have to be replaced as well. From what I’ve read, it doesn’t look like PaGP is really supported with OpenSolaris. It’d be nice to stick with Cisco but a new gigabit Cisco just isn’t in the budget.  Ah well, it’s a single VLAN so it shouldn’t be that hard to handle for a non-Cisco switch.  We have a nice 3com on the shelf that should be up to the challenge.

So after the expansion, our ZFS rig will have 4x1TB SATA in a raidz1, 7x73GB 15k u320 in a raidz2, and 7x143GB 10k u320 in another raidz2.

The plan will be:

Large files will relatively little i/o (some of our offsite backup files) and zfs backups from some client vms will be stored on the 3TB SATA array.  Our little four SATA disk raidz can’t produce a lot of iops so we don’t want to stick things like live vm’s on it.

We’ll be moving one client’s small business server over to the 7x73GB array and see how the performance is.  It’d be nice to get more than one exchange server per array, but Exchange is a real pig for disk i/o.  Even when nothing seems to be happening, the lights are all blinking.  We’ll probably move our webstore to the 7x146GB array although it’s not really hurting now so we’ll have to see.

If this all works out, it’ll be a pretty inexpensive and fast little SAN.  A terabyte of storage over 14 u320 spindles with all the benefits of ZFS for under $1000.  I don’t want to get too far ahead of myself, but I’d really like to beat a Dell MD3000i with this ZFS rig.  Can’t wait until the parts arrive!

We’ve switched from MySQL cluster to replication.  We found that although the cluster gave us excellent throughput, a few cluster crashes combined with lengthy startup times forced the issue.

During our move, we changed to a master/slave replicated structure.  We will be trying out various technologies to load balance sql queries in the near future.

It certainly wasn’t as seamless as we wanted, but the server move has been completed.

Most of our servers have been moved but we are maintaining some networking gear and a backup server in the original space.

Although we had a lot of trouble with bringing an Exchange server online, our legacy email was operational within two hours.  This included moving locations and changing our database structure from clustered to replicated.  So that part went well.

What didn’t go well was Exchange panicking over some active directory work we decided to do within our scheduled downtime.  That was a mistake — we shouldn’t have tried to get too much accomplished within the maintenance window.  The result was too many variables which made for difficult troubleshooting.

Another mistake that we made was focusing all of our attention on this Exchange problem.  We should’ve taken a break from this and bring other services online.  It took only thirty minutes to bring legacy flat web pages online, but this process wasn’t started until we had turned the corner on our Exchange problem.  We hadn’t prepared a triage plan, which was a mistake.

Back to the good news:

The cooling in the new space is much better.

This server was in a stubborn hot spot with our old room.  Notice the end of daily temperature fluctuations — our old system relied on building air circulation which was shut off at night.  The new system uses chilled water and the temperature control is much tighter.  With our climate the chilled water system will get ‘free cooling’ in the winter which means lower electricity use which is good for everyone!