NetworkGod's Blog

Firstly, I know it’s been a while… over a year in fact. So, thanks for hanging in there. I’ve been busy being a father and husband, running a hosting company, and changing jobs.

Anyway, I’ve decided to write here today out of shear frustration. I’m sick of paying hardware vendors crazy amounts of money for crap that just doesn’t work. I’ve been a long-time advocate for WatchGuard. They’re stuff used to be rock solid, but it’s just not doing it for me anymore. I have an x55e at home and an x550e at the hosting company I run. Now, the x55e retailed for nearly $1,000 and the x550e was around $1,300. That DOES NOT include the annual “subscription” that I’ve had to pay. For that amount of money, you would think that the user interface would be easy and you wouldn’t even know they were there. Not so. Okay, so I bring those guys up to say that I have been looking around at other vendors. Well, I *thought* I found one. There is a new company called Meraki that looked REALLY cool. Imagine if Apple built APs, firewalls and switches. Yea, they’re beautiful. Apparently, some MIT guys put together some equipment to make networking what it SHOULD be. Well, the only problem here is the price. Holy crap they are proud of their stuff. Not only do they want top-dollar, but they want a (very expensive) annual subscription as well. On top of all this… VMware, which my hosting company runs on, comes out with a new version as well! Guess what? They want that annual fee too! Even though I *never* call support; That’s just not relevant.

So… I’m done. I have the skill set to handle this stuff… I’m going to do all this crap myself.

I’m starting by replacing my firewalls. I’ve bought a couple of Soekris boxes and will be installing m0n0wall on them for my home firewall and firewall at the office. I’m a FreeBSD guy… I can make the thing sing. There’s no reason I should have to shell out $2,000 for pretty hardware plus $1,000 annual fee when I can do it myself for a one-time $400 cost. Did I mention NO annual fees? Also, I’ve just found a hypervisor for FreeBSD called VirtualBox. Turns out it’s developed by Sun and it’s OPEN SOURCE! So my plan is to install it on one of my current VMware hypervisors and see if it runs stable. If so, there won’t be ANY bullshit hardware fees this year.

Thanks for listening… I feel better already!

In today’s world of the non-stop flow of information, having an Internet Service Provider outage is like being stranded alone on an island. If you happen to be at home when this happens, you know for sure that the entire world is out there, catching up on the latest news, streaming TV shows, and Tweeting about what’s for dinner, but you are simply no longer part of the conversation. Not the end of the world… go catch up on some TV. But what happens when there is an ISP outage at work? It can mean thousands of dollars in lost revenue, not to mention productivity. The way I normally combat this situation is with a WatchGuard e-Series firewall. Beginning with the x55e model, you can do something called “WAN Failover” which does exactly what it sounds like. You connect two different ISPs to the firewall, which acts as your gateway. Whichever ISP is working at that moment, will be your route out to the Internet. You can also load-balance between the two working ISPs. I actually use this exact scenario at my office with AT&T DSL and ComCast Cable. Yes, you do have to pay for two different Internet connections but in the grand scheme of business, an extra $100 /month is a small price to pay.

** Thanks to Tony Downs for suggesting the topic of this post.

If you are an admin for any number of users, chances are that you have been asked to block certain web sites for any number of reasons. There is no shortage of companies that will gladly take your money to filter your network traffic (my personal favorite is using a WatchGuard Firebox with the WebBlocker service) however, these services are usually expensive, and there are times when you don’t actually need a filter, you just need to be able to stop one or two sites.

As an example, I was asked by a company the other day to block Facebook. The administration feels that they are losing large amounts of productivity due to this particular social networking site, so I implemented a very simple solution.

On our internal company DNS server, I created a zone for facebook.com and then created an CNAME record to point www.facebook.com to notatwork.greenfoxhosting.com. Basically, what this does is to cause anyone using our DNS server to see a “This website is not allowed at work!” page instead of Facebook.

You can see an example by going to http://notatwork.greenfoxhosting.com.

Now, for all you techies out there, I realize there are multiple ways around this such as proxies and the like, but the bottom line is that this will stop 99% of the employees from getting to Facebook or any other site you so desire.

Oh, and the main selling point: It’s FREE!

Props: A colleague of mine, Sam, suggested this and I implemented it for the rest of the world to use freely…

A buddy of mine asked me how to update his FreeBSD 6.2 server to something more recent, and I told him it was too difficult for me to explain so I did it for him. I am now officially ashamed of myself. I should be explaining on my blog how to do this stuff and pass on this wealth of knowledge I have acquired over the years. So, here we go!

Let me start by saying that I rarely use anything but the generic FreeBSD kernel. I know it makes the server faster and all that, but I’ve found that for my needs, it’s not worth the trouble you have during updating. Also, let me start by saying that I am in no way responsible for anything you screw up using these instructions and/or for any data loss you may have. This is a guide to help you along a particular path and nothing else.

1) Let’s get your ports up to date by running: portsnap fetch extract update. After you do this the first time, you won’t need the extract anymore. Portsnap is an AWESOME way to update your ports. Much better than the CVS hassle from earlier years.

2) Speaking of cvsup, let’s update the source. I know there is a freebsd-update command but it always seems to fail on me and I haven’t taken the time to make it work yet. For source, I still like cvsup. So, if you don’t have the cvsup port, go get it and install it. On a related now, when going from one major release to another as I am (from 6.2 to 7.3) you need to take baby steps. Trust me, it may take longer, but it will help the whole process go smoother in the long run. Since I was starting at 6.2, I’m doing this process updating to 6.4-RELEASE, then doing it again to 7.0-RELEASE, then doing it one last time to 7.3-RELEASE. Yes, it takes forever, but it works when you’re done.

3) Create a file called src-supfile and use it to update your source. Mine looks like this:

*default host=cvsup.freebsd.org [edit to your nearest server for speed]
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_7_0 [edit to the release you would like to update to]
*default delete use-rel-suffix
src-all

4) Run cvsup src-supfile and wait for it to finish. Once this is done, the latest source for your release will be in /usr/src. Change to that directory using cd /usr/src

5) Now comes the time consuming part. It should all be pretty much non-interactive except for the mergemaster commands which will compare some of your files with the new ones. Basically, it will bring up a diff between each file and you can pick which one you want to use or if you want to merge the files line by line. Take care on this part as files like /etc/master.passwd can be set to default and you won’t be able to login after the reboot. Do these commands one at a time in case of errors:

make buildworld
make buildkernel
make installkernel
mergemaster -p
make installworld
mergemaster
shutdown -r now

6) After the reboot, you should be on the new kernel which you can verify by using the uname -a command. Take the time to check and update any packages/daemons that you’re running, as sometimes they don’t like major changes. Fix as needed.

*) The FreeBSD Handbook are Google are invaluable references if you get stuck.

I’ve been using network monitoring software called Nagios for years. It’s a really cool, web-based software that runs as a daemon on FreeBSD. It watches my networks and servers 24/7/365 so I don’t have to. It e-mails, SMSes, etc when anything is wrong… very cool stuff (even if my wife hates the notifications.)

I’ve modified Nagios to run the configuration from a MySQL database, so if I need to add or modify anything, I just do it from the MySQL server rather than from the text-based config files. I’m one step away from making a new web interface for the MySQL database, but I have too many irons in the fire as it is. Anyway, today I had to do something I had never done before, so I thought I’d write about it.

There was a situation at the office today that used 100% of the bandwidth for nearly 3 hours. The problem is, I didn’t know about it until it was over… technically, nothing was down, but my customers were experiencing a problem and I didn’t know about it… totally unacceptable. So, I came up with the solution to use Nagios to watch the bandwidth of my router and let me know if it hits a certain threshold.

Pretty simple stuff, but worth mentioning… all I really did was install MRTG on my Nagios server to watch my router, then had it update every 5 minutes as usual. Next I added a Nagios service that checks the data in the MRTG log against the threshold I set in Nagios using the check_mrtgtraf command.

I’m very impressed that Nagios has this command already ready for use… I just needed a reason to use it. I’ve included the configs below in case anyone needs it.

Command:

define command{
command_name check_mrtgtraf
command_line $USER1$/check_mrtgtraf -F $ARG1$ -a $ARG2$ -w $ARG3$ -c $ARG4$ -e $ARG5$
}

Service:

define service{
host_name router
service_description BANDWIDTH
check_command check_mrtgtraf!/usr/local/etc/mrtg/logfile.log!AVG!250000,374999!375000,375000!10
active_checks_enabled 1
passive_checks_enabled 1
parallelize_check 1
obsess_over_service 1
check_freshness 1
notifications_enabled 1
event_handler_enabled 1
flap_detection_enabled 1
failure_prediction_enabled 1
process_perf_data 1
retain_status_information 1
retain_nonstatus_information 1
is_volatile 0
check_period 24×7
max_check_attempts 3
normal_check_interval 5
retry_check_interval 1
contact_groups admins
notification_options w,u,c,r
notification_interval 60
notification_period 24×7
}

I fought this battle for a little while this weekend, so I thought I would post the procedure to help others. I had three VMWare 3.5 servers that I needed to upgrade to ESX 4.0. Plus I wanted to get all the VMWare Tools up to date as well. These are the steps I took:

1) Login into each FreeBSD VM
2) If you have a version of VMTools/Guestd running, stop it and delete the startup script (mine was in /usr/local/etc/rc.d/)
3) In the console, shut down the VM by running “shutdown now”
4) After the OS has halted, Power Off the VM
5) After all the VMs are powered down, enter Maintenance Mode in VCenter
6) Go to VMWare’s website and download the latest Update Zip
7) Run the Host Update Utility and choose the Zip file you downloaded
8) After the update is complete, make sure you are no longer in Maintenance Mode
9) In VCenter, right-click on each VM and choose “Upgrade Virtual Hardware”
10) Edit the settings for the VM and make sure the OS is listed as “FreeBSD (32-bit)” – (all mine are 32-bit)
11) Power On the VM
12) In the VCenter, right-click on the VM -> Guest -> Install/Upgrade VMware Tools
13) Login to the console (as root – or su) and run “mount_cd9660 /dev/acd0 /mnt” to mount the install disk
14) Copy the install file to some dir (I just used /root) “cp /mnt/vmware-freebsd-tools.tar.gz /root/”
15) You can then unmount the disk “umount /mnt”
16) Now extract the file “tar -xzvf vmware-freebsd-tools.tar.gz”
17) Now go into the folder “cd vmware-tools-distrib/”
18) Run the install file “./vmware-install.pl”
19) I suggest you read each option that it gives you, but all the defaults were fine for me, so I just hit Enter until it was complete
20) All done! Your Tools status in Vcenter should now read “OK”

If this guide helped you at all, or you find any errors, please leave a comment.

Thanks! I hope it saves someone some time!

I rarely do on-site work anymore, and now I remember why. I’ve been coding for so long, that when I got an “emergency” call this morning, I was actually excited to get out amongst the living. One of my clients had a vendor fly down from Ohio or somewhere and spent all weekend doing an upgrade on one of their systems. They promptly left this morning, with everything supposedly working. So I get the panic call around 9AM saying that no one can get into anything, and will I PLEASE come help them out… I’m their only hope. So I get there and I find all kinds of problems… it’s no surprise that their network isn’t working. So I fix each problem one by one, but there is still TONS of packet loss on the network with seemingly no explanation. Next thing I know, something is tapping me on the shoulder, so I turn around… no one is there. My imagination? NO! There’s a freaking river flowing down the wall, and through the tile above my head. Turns out there’s a flooded bathroom upstairs and it’s getting all over me. Nice. I keep working on the problem, when I feel another tapping… this time on the other side! Yes, another leak! Maybe I should just go throw my head in the toilet and get it over with. To make matters worse, I go about the rest of my day to find out that all my problems (with the exception of the water) are because someone saw a cable hanging out of a switch and decided to plug it in… to the same switch. Fun times… I vow to stay behind my desk coding for fear of another adventure of this sort.

Due to the nature of my many jobs, I am all over the place, all the time, and having a laptop with me is a must. Whether it’s on-site at a customer location while consulting, or sitting in the library coding while the kids check out a few new books as a programmer, I must have a reliable laptop with internet access. Now the internet access I have is handled with a Verizon card. I have yet to find a place that it doesn’t work. It’s not always blazing fast, but it will usually get me by. For the laptop, I have a early-2008 15″ MacBook Pro. It’s reliable, fast, and since I’ve been using FreeBSD (the UNIX operating system Apple OSX was built on top of) for over 10 years now, it was a match made in heaven. So now to the things I DON’T like about my MBP of which there are only two. First, it gets CRAZY hot. I was sitting outside the other day and the internal temp hit around 154 deg F. It’s so bad I’ve added a piece of software called Fan Control just to keep the fans spinning at 100%. I’ve read that this is due to an overage of cooling paste on the processors, but whatever… it get’s hot, and as a general rule, I don’t crack open laptops. The next thing I don’t like is the battery-life. Even with the fans spun down, and the power settings optimized, I don’t think I’ve ever gotten over 2 hours out of the battery. Sometimes, while on battery, the laptop will even shut off in the middle of working while the battery indicator says I have 75% battery. This has led me to seriously considering a new laptop (a newer model MBP of course) but for now, I’m going to try an external battery called a HyperMac. It’s a little on the expensive side, but it will work with any MacBook, so when I do decide to upgrade, it will still be useful. I guess the MBP can’t be 100% perfect… they have to give Microsoft *some* chance of keeping the doors open.

Why do all of my problems end up being stories? I guess that makes for a good blog, right? Last week, I noticed that my mail server VM was taking up a lot of space. I found that I had taken a snapshot some time ago that was no longer valid, so I deleted it. There was also a disk file that hadn’t been modified in about 6 months, so I moved it thinking that it was no longer needed. Notice that I moved it rather that deleted in just in case. A few days went by, and everything was still fine, so I deleted the file. Well, this morning, about 10am, my mail server crashed and crashed hard. Turns out it did need that file after all. So I spent the majority of my day setting up my NEW mail server. Something that needed to be done anyway, but I would rather have done it during maintenance hours.

Okay… I don’t want to catch any grief if I’m like the last person on the planet to get this… but here it goes. I am by no means an audiophile. I don’t get into the cool speakers, the awesome surround sound, or the latest noise-reduction headphones. I do, however, listen to a lot of mp3s while I code and have over the years ripped my CDs into what I think is a decent collection. So while listening to some old school Alice in Chains, I decided to try going back to the CDs and try re-ripping them as Apple Lossless (instead my normal 160kbps mp3) and the difference is night and day… even to my untrained ears. I am discovering wonderful new sounds in the music that I have been listening to for more than 20 years now. I can’t wait to listen to my entire collection for the first time again…