- Yesterday was technology day.  We ended up severing ties with our dear ‘friends’ Verizon; we switched to vonage.  Along with that, I followed the instructions on their site and made our house into a closed loop.  If you cut your physical connection to Verizon, you can plug your vonage router into a phone jack in your house and suddenly all of your phones are vonage phones; it’s very cool.  We also bought an iMac this year; yes, an iMac.  Last night, after some struggle, I finally finished configuring it (it always is a struggle in a paradigm shift). We are owners of a computer manufactured by Apple…wow, I said it. I hear admitting it is a first step.

 - Lord willing, we’ll have some big news to announce soon. Pray for us if you think about it, we’re trying to make something logistically work that will be…big.

 - Along with that, Sunday was one of those Spirit filled days at Church where God really affirmed alot of things in our hearts.  The talk is available here, and it was about God’s will for us, and death and doing your life well.  Great stuff; we sang ‘We Rest On Thee’ afterwards, which, if one could have them, is our theme song as a couple (like Over The Rhine’s music, that song factors in a couple of key moments for us in our marriage).  Anyway, couldn’t sing the second or third verse just for weeping; the Pastor led and I have to wonder what he was thinking. It’s a balm to have God really challenge, convict and encourage you all at the same time. 

We go in faith, our own great weakness feeling,
And needing more each day Thy grace to know:
Yet from our hearts a song of triumph pealing,
“We rest on Thee, and in Thy Name we go.”
Yet from our hearts a song of triumph pealing,
“We rest on Thee, and in Thy Name we go.”

You may remember the purpose of this is to give a bit of a primer and ethnography, of sorts, from a primairly Microsoft, IIS world to a Linux/Apache world.

Well, recently the second server came in and I’ve been getting heartbeat up and rolling.  The complex thing about this is that the nics that service the shared IP address must be in the same slot and referenced the same (i.e. eth0) otherwise sadness will occur. 

After the fold, a brief overview of heartbeat and a plea for questions.

Currently, I have two front ends servicing a shared IP address and an apache instance (see here for more detail).  Heartbeat2 is running in between those two boxes on a crossover cable and privately addressed IPs.  A third connection services the rsync link to a privately addressed staging server with the staging server only accepting connections between the front ends. 

What’s nice is that once I figured out that the nics that share the resources needed to be in the same slot and once the heartbeat was running over a crossover cable, life was strangely easy.  Apache was trivial to setup since I just followed the instructions in the link above.  There are several good links off of that site and you would do well with exploring it if you’re in need of that information. 

In regards to heartbeat, I can’t emphasize enough that STONITH should be used for the cluster.  When I was creating some other resources before I turned it on, the box whigged out and decided to not come back up; STONITH fixed that.

Also, one final note on rsync.  I want to share my rsync command because I discovered a few things.  First, if you set AD permissions on a directory on your staging server and rsync it over, the perms will not copy successfully.  You must throw the –owner no and –group no parameters in there so that the rsync server won’t touch ownership on the frontend.

With that said, here’s my rsync with notes as to what each line does (borrowed from overtone.org’s long defunct article):

#!/bin/sh
/usr/bin/rsync –password-file /root/bin/rsync.pass \ #my password file I created for the rsync user
                     -avz user@1.2.3.4::share #avz \ (recursive, verbose, compressed) user@1.2.3.4 (user@staging server) and share is the rsync share I setup
                     –address 1.2.3.5 \ # the private interface on the front end I’m using to communicate with the staging server
                     –exclude /directory \ # a directory I’m excluding from the copy because I was having problems getting it up there
                     –owner no \ # put to keep the front end’s permissions from attempting to be copied over.
                     –group no \ # put to keep the group ownership from being changed
                     –delete \ # anything different?  change it.
                     /srv/www/htdocs # the directory I’m copying.

Even though this will go on into production shortly, please e-mail me if you have questions about my setup and want to emulate it; the people I’ve talked to with this have all been extremely helpful and I want to further the love.

I’m redoing rsync today and wanted to get a couple of things down for you cats.  This is the way we mirror our website:

Staging Server (privately addressed)

I have /etc/rsyncd.conf setup and reads as follows

gid = users
read only = true
use chroot = true
max connections = 3
transfer logging = true
log format = %h %o %f %l %b
log file = /var/log/rsyncd.log
hosts allow = private side of the DMZ (allows for only one connection from this IP addy only…)
slp refresh = 300

[module name for rsync]
path = /srv/www/htdocs
comment = My Workplace’s website
list = yes
auth users = the name of the dummy account I chose to make the move with.  Doesn’t need to ‘exist’
secrets file = /etc/rsyncd.secrets

and I have rsyncd running

Front ends:

cront job that runs the following every minute:

rsync –password-file=’/home/rsync.pass’ -avz account@ipaddress of target server::rsyncentry for module –address private side of DMZ –delete /srv/www/htdocs

*sigh*

UPDATE: add one more step on. You need to remote into the IP Address that you’re adding when you put in N+1 IP Addresses and turn back on the default gateway. And who said Linux wasn’t quirky…

There were two things that made tonight really a unique experience.  At one point, Aidan was acting up and I had to take him back to the back of the room.  As I looked around the room watching people enjoy, listen, and worship to the music that was being played I noticed something beautiful.  There are so many people that come to the Jesus house that come from different economic backgrounds and are down on their luck.  What the Jesus house people do is love well and give of all that they are and expect nothing in return.  What was beautiful was that I sat there and watched the diversity of God’s kingdom in the people from the west side of bloomington sitting next to people from the east side and it was a beautifully painted picture of how we are all desperately in need of a savior.

The other unique thing was that during one the mini sermons that Glen was giving he mentioned Bill Gates and how us not making our faith active is kind of like Bill giving someone 100 bucks; it’s really nothing for him to do that.  Faith calls for action, real action.  Well, after going on for a second about the richest man in the world, Glen commented ‘Maybe that’s why I like Linux’.  I really wonder about how well Linux is penetrating the market and culture when a blues guy starts talking about using Linux…I wonder if a change is a comin? 

http://overtone.org/articles/rsync.php is a simple, wonderfully written article on how to setup a basic rsync setup on your host to mirror out a website to different boxes. Granted, this is a bit chatty and time consuming to do (I could imagine how it would be with a larger site), but for the size of the site I’m doing and the importance of the data, I’m pretty pleased with the results.

The end result is I now have a privately addressed ’staging server’ that will have samba enabled and users will map to shares with. The content rsyncs itself out to my mirrors via a static route to the private address of the mirror (I haven’t done static routes before and it was cool and fairly easy to set that up); because I set up a DMZ (which was really easy to do in SUSE), the private has a whole ton of ports open and the public end only has two (I’ll let you guess which two those are ).

One small edit since I was redoing the server today and I’d forgotten how to make those edits.  For you GUI ppl, it’s under YAST, network devices and properties for the network card.  You basically need to setup a connection on both ends of the link in order to make routing work.  On the server, the link looks like this:

IP:  Management console or staging server
GW: private address gateway

SM: 255.255.255.255
Pick the private NIC for the interface.

The on the client:

IP:  Server IP Address
GW: default gateway

SM: 255.255.255.255

I’m finding that the further I get into this setup I’m *choke* loving Linux a ton more. Everything has a hosts allow option on it which means that I can control who specifically gets access to what in the setup. In a situation like this where security is a concern those options are proving to be their weight in Gold. I also get a measure of failover if the staging server goes down, it’s a few minutes of reconfiguration and I can turn a mirror into the staging server and have it be mappable.

Too….much….fun.

–pete

After piecing together rsync and getting it operational, I found a couple of problems.  In the course of getting SUSE 10.0 up and operational, I discovered that the machine got compromised through a ssh hole that wasn’t patched.  Sucks to be me.  I then tweaked the model a little bit to setup a DMZ for the site.  Each of my mirrors is going to have a public interface only having port 80 and 445 open and a private interface that is only open to the private network (with the variety of ports open). 

This involved setting up two interfaces/gateways/subnet masks and configuring the firewall to attach to the appropriate interface.  The only problem was that SUSE does not allow more than one default gateway; the end result is that I could only open either the public interface or the private interface and couldn’t open both. 

My first idea was to look for a different distro for the mirrors, one specifically that could support two default gateways (similar to what I have setup in Server 2003).  After looking at Ubuntu, great distro for a desktop – sucky server, and FreeBSD, what the heck is this crap?, I ended up finding out that the paid version of SUSE did this.  I downloaded the evaulation copy of SUSE that ended up not doing this in 10.0.  I was dedjected, but I had an epiphany when I realized that I could use this situation to my advantage.  What I ended up doing was setting up a static route between the mirror and the staging server sitting in the private network and viola, I have connectivity for rsync and connectivity for the web on the other end. 

After doing some initial research, rsync works really well and does what it does incredibly efficently.  More on this when I can get to my bookmarks list @ work.

I’m one for asthetics, and this server is super cool looking when it’s plugged in.  Dell boxes have this status indicator on the front to let you know the health of the box at a glance and usually these are blue if they are plugged in and happy and a bright orange if they are not.  Well, the blue light on this box is incredibly bright and small and it reminds me of HAL in 2001.  So, now if you walk into the server room there is the 1425 at the top of the rack is the bright blue eye that beckons you to enter.

Well, today’s project was getting the OS installed and getting samba/AD authentication up and rolling as well.

The OS:

It was incredibly easy.  For this box, I downloaded SUSE OSS 10.0 64 bit edition and it screams once it’s running.  There were two small issues.  First, is that I set up a hardware array of RAID 1 before installing an OS and when it came to partitioning, the system still allowed me to utilize the second 250GB drive.  After working with it for a little while, I just assumed that creating the partition on the first drive would do the job.  Once I get the server how I want it, I may do some testing to make sure that assumption was correct.  The second weird aspect to the install was that my 4 network interfaces (long story…) were duplicated in the available NIC connections.  When I picked a interface to plug into, I had basically a 1 in 8 chance of getting it right (realistically a 1 in 4 because two of the nics were different chipsets than the other two). 

There were a couple of things I verified I wanted in the install as well: Apache (of course), rsync, mysql, PHP, samba, and windbind.  For now, I just verified that the os was installing those and I am going with the built in copies of each of those pieces of software.  The better choice may be to install them from scratch, but getting going, I’m going with the version that I can figure out the best within the framework of the OS; as I get better, I can tweak it as necessary.

Samba,AD authentication:

Again, fairly easy.  It suprises me how well things get rolling in this setup when you have a basic knowledge and a little patience.  There are some steps that you need to follow, however.  There were two main websites that I used to get this rolling.  First, you have to get kerberos up and rolling and a really great article at http://www.windowsnetworking.com/articles_tutorials/Authenticating-Linux-Active-Directory.html
 shows how to do that.  You may want to note that as you put in the domain name, you type it in all caps; it seems to not work if it is lower case; do the tests that the article mentions.  The only real problem is that you’re tied to one specific domain controller (we have 5 available, but if one goes down…well…) for kerberos authentication. 

From my understanding, rather than monkeying with all the different pam authentications, I followed the directions at http://www.enterprisenetworkingplanet.com/netos/article.php/3487081 which turned out to be stellar as well.  Once I got that up and rolling, I can and have created shares though smb.conf (it’s a bit to specific to talk about here and there’s mounds of documentation on it) and in the allow list I can type domainname\usergroup and it works. 

The next step will be rsync.

EDIT:  One file I forgot (and forgot to mention) is /etc/nsswitch.conf which tells the system where to get authentication information from.  You need to add winbind to it – the relevant part of the file for this project is:

passwd: compat winbind
group:  compat winbind
shadow: compat

hosts:  files dns wins
networks:       files dns

services:       db files
protocols:      db files
rpc:    db files
ethers: db files
netmasks:       files
netgroup:       files
publickey:      files

bootparams:     files
automount:      files nis
aliases:        files

Part of my ethos, I believe, as a Christian geek is to do my job well.  My desire with this particular project is do it detailed…there’s going to be a lot to it and I want to make sure I know what I’m talking about.

I’ve been a professional now for about a decade and some change (4 weeks to be exact) and in that time there have been moments of my career that have been, by definition, paradigm shifts; these have been moments of transition where a body of knowledge that I’d gained is now no longer needed and a new phase is beginning.

This particular one I’d like to chronicle, if you will, in this format for a couple of reasons.  First, I think it’s useful information and contexts like this have helped me figure some things out from time to time as I’ve hit google looking for answers.  Secondly, for my own benefit I can provide some internal documentation for myself as to how I reproduce what I’ve been building.

The project that I’m about to start is taking our web presence from IIS 6.0 (which I have loved dearly) to Apache.  Like any long term relationship that ends, there are a couple of reasons for doing this. 

First, my experience with Open Source Software (OSS) has been stellar.  The applications that we’ve used it for have been ones that have been robust and ’set and forget’ type setups.  One particular one that I’m fond of is using CUPS (Common Unix Printing System) to serve out a dozen or so printers that I’ve set up on timers for our classroom labs.  Students come in, and at the start of each class a crontab on the server issues a command for the printer to start its queue and accept jobs and at the stop of the scheduled class the printer turns off the queue automatically.  It’s saved us a ton on printing costs and it’s been a nice service to students who need to use printing for class.

My second reason for switching right now really has to do with scaling an application.  I’ve been really challenging myself to start thinking in terms of services for our LAN instead of servers and it’s been fun to start building out horizontally.  We’ve clustered our main data presence and provided a couple of layers of backup for it.  I’d like to do the same with our web presence.  The model we have currently with IIS 6 I feel doesn’t allow me to do that without some significant hardware purchases.

With that said, I’ve already been setting up some tests with different pieces of the model and I’ll be reviewing those as time goes on.  For now, here’s my model:

Machines

Four seperate boxes will be involved in this setup.  Two of the three main servers will be a Dell PowerEdge SC 1425 with the following specs:

Dual 3.2 Ghz Intel Xeon processors
2GB of RAM
2 250GB 7200 RPM SATA drives in a RAID 1 configuration

I’ll have an existing PowerEdge 2650 (the link is to a 2850) which has been consistently my favorite server of choice for a few years running.  This particular box is:

Dual 2.8Ghz Xeon processors
3GB of RAM
146GB of storage in a RAID5 array

Software

After looking at a few distros out there, I’m choosing the 64bit version of SUSE OSS.  There is just something to having a professional corporation throw its resources behind a distro; they have a bit more of an idea how to put together a GUI and even for a geek, how to put tools out there.

The Model

Basically it boils down to this.  I’m privately addressing a box that I’ve yet to determine and I’m going to setup Samba, Winbind, and apache on it.  I’m then going to rsync out my data to the other three severs, two of which will be located in our datacenter and a third to be located elsewhere.

Why I think this is a compellling read for you is that I’m juding myself a relative apache novice.  I understand it, I appreciate it, and these next few months I’d like to move my knowledge over to it from IIS.  There’s a fairly complicated IIS setup that we run that each piece of the puzzle will need to have an apache counterpart.  For my own sanity and keeping track of things, this is where this blog will come into play.  Hope you enjoy the ride…