Recent Web Site Outages
Some of our viewers may have noticed the web sites for WATVA and Trail Patrol were not accessible Saturday to Monday. So I thought I would give the low down on what happened. If you have no interest in the inner workings of the internet, you can stop here. Might I suggest this page instead: http://metobs.ssec.wisc.edu/aoss/cameras/hawkcam-flash.html
So where to start? I think i will start at where the problem came from. The web servers upgrade. A lot of people love to update, but those of us that deal with IT dread upgrades. Why? It usually means something is not going to work like it supposed to, and that means sheer panic for the folks the depend on the said technology.
A few weeks ago I noticed that our web server, which is sitting a comfortable air conditioned warehouse in San Jose, California (we share a rented dedicated server from HostGator.com), had some things that just didn't work right on it. However, they were not pressing issue, just things that made life a little harder to deal with on the server. That is until I noticed the server was running a Linux operating system that was a few versions old. A few versions isn't so bad actually, as it was a solid operating system in terms of reliability. However the upgrades offered some things that I really wanted to make my job a little easier and more reliable. How often does our web server go down or get restarted? In the last year, the web server has been restarted once. 364 solid days of running! I can't go more than a week without restarting my desktop. So I decided that the benefits outweighed the potential drawbacks to upgrading, even though I knew some headaches would come up.
The server was taken offline Friday May 11th at 11pm CST. The update, from HostGator, would take about 5 hours. They sent me emails during critical moments of the upgrade, which was nice. Around 4am the update was complete. HostGator sent me an email saying they would retain the old server for 48 hours before recycling the hard drive.
On our systems down in Sheboygan, all the web sites came back without an issue. Not only did they come back without a problem, they loaded a lot faster. I thought wow, no problem, this is rare! Then Sunday came. Sunday afternoon we received an email from a great WATVA member in the north woods that said she was unable to access the site. We sent off some emails to our Regional Coordinators in the north to confirm this. Most said they could not access it, while one said if he used his wireless data he could but not cable internet. To me, that made no sense. So, where did we go wrong?
I started to look at the setup of the server. Everything seemed to be in order. Hmm. So I had the person send me a screen shot of what error they were getting or what their screen looked like. This helps me to get pointed in the right direction. I figured out it wasn't the hardware of the server, this was a software glitch, most likely a DNS issue.
DNS? What is that?!? DNS, or Domain Name Server, is like the internet's white pages. They are computers setup all over the world that keep the physical address of a web site in a directory. Physical Address? Let me explain! When you go to the web site www.watva.org, you're not really viewing a page from the web address of watva.org. Nope, sorry. The fancy letters that make up www.watva.org are actually a mask! A mask for what? This: 50.23.12.151, and IP address. That's the true address for this server (among other sites that are sitting on this server). Computers and such on the internet have no idea what www.watva.org means, but they do know what the IP address means. An IP address is much like your house address or phone number, more like your phone number. If you really want to learn more about IP addresses, see the wikipedia document here: http://en.wikipedia.org/wiki/IP_address. I tried to find a nice graphic of DNS and the internet, but they made my head hurt, so I shall spare you.
So where does DNS fit in with IP addresses? Right here! A domain name server is a giant list that says if you want to visit www.watva.org, your computer needs to send it's request the web server at the address of 50.23.125.151. If your going to a web server that has multiple web sites on it, such as the server WATVA has (it shares its web site with National Motorsports, Rider Network, NOHVIS, etc.), the DNS server knows what page to show you based on your request. Again, a giant directory is all DNS is.
Back to the main topic here, the settings for the DNS on our server were incorrect, but only slightly. Each internet service is different, and each domain name server is different. Some understood that the error with WATVA was only partial, so it did find a way to send its requests here. Domain name servers in the north, did not understand this and gave users a web site not found page. So, with some researching and pulling of some hairs, I resolved those issues on the server. Only, users were still not getting there, and furthermore I now could not access the site.
Not only did I find that some settings were wrong on the web server, but there were some incorrect settings with some of the domain name setup itself (which we use GoDaddy for our domains)! That's a larger problem however, in terms of trouble shooting. Why? Each time you visit www.watva.org your computer remembers that the web site watva.org goes to the specific IP address. It remembers this for around 12-24 hours unless a setting says to forget it at a different time. So if I made a change, I would have to wait 12-24 hours to see if it worked. That's a pain. I was able to get around this a little by using some handy tools that dont make me wait. However, all of my users didnt have these tools and I wouldn't expect anyone to have them either.
In the end, we got the issues resolved with both the domain server and the domain name. It took a few days but that's the nature of the beast with DNS issues.
For those that read this and are thinking:
Don't worry. The problem was fixed!