Archive Yourself
I've been thinking a lot about archives.
I'm an amateur historian, so I spend a lot of time looking into the past. Part of looking into the past is field work - in my case, going out into the Pine Barrens and exploring where these ghost towns were. Another big part of that is doing research. What was this town? Who lived there? How does that place fit into the broader context of the region?
The Pine Barrens historian is lucky in some ways, and unlucky in others. From the 1920's through the 1960's, a historian and folklorist named Henry Charlton Beck spent a lot of time in the region, documenting the history and culture of the Pine Barrens. He saw things that don't exist anymore, and in a lot of ways, he paved the way for other historians through the ages (myself included) to follow on through his research. We're lucky because of all of the places in the United States, how many have had generations of historians doing research? I'm sure there are dozens of cellar holes and half-hidden clearings in the wilds of Pennsylvania, or Oregon, or wherever that nobody knows about. The people who lived there are long dead, and nobody cared much to archive the knowledge of the place.
The thing is, we only know the history of the Pine Barrens through the context of those who documented things. So while Beck was a great folklorist, he didn't spend a lot of time describing cellar holes, or what he found at his forgotten towns. He described the people who lived there, maybe a short anecdote or two, and not much else. Historians who have followed since have documented things better or worse depending on their own interests. Right off the bat we don't have an accurate historical record anymore.
Maybe that's the problem with history? How faithful are our archives - our history - of things when we gloss over the mundane and the humdrum and focus on the more exciting things? Are they intrinsically worth more? Of our past, what is worth saving? Will there be some future relative, two centuries from now, who will wonder what I ate for breakfast today, December 28, 2007?
I also wonder about the trend towards digital archives. My website is a pretty useful research tool for those interested in the history of the Pine Barrens. There's a lot documented there that isn't mentioned in any book. Personal stories that are fascinating, but would never be published. How do you go about archiving a website for scholarly use five centuries down the road? Sure, you can say that the Internet Archive project spiders sites and saves them, but the IA has problems fetching old versions of my site from 2002 -- can we trust it to be useful in five hundred years? Would anybody have a HTML browser in five hundred years? Will people even know the HTML specification then? If the answer is no, then what we have is a lot of history held in a very transient medium, that will likely not be accessible to anybody in the future. The problem is, however, that websites generally don't translate well into other mediums. So, what do you do?
One of the things I want to do is research my family tree some more. I've not really been able to trace my ancestry back further than 1860 or so, which in my opinion is terribly recent. Is my family information gone forever? What about the lives of my ancestors in the 1600's? If there is no record of them, what of their lives? Were they spent in vain, only to serve as breeding stock for the next generation? What about their hopes, dreams, and fears?
I've been meaning to sit down and document my life for Dana. Not to be pretentious, but because there's so many experiences that I've had - good and bad - that it might be good to write them down, and maybe after I'm gone, some ancestor of mine will come along and be interested in my life. The question I wrestle with, though, is what medium do I use? I would love to install a copy of WordPress on a local server and blog my way through my life. The problem is, of course, the transient nature of web published data, as well as the danger of hard drive failures, fire, etc. I could write, but again, a fire could come along and destroy my journal. Plus, handwriting is not really in a friendly format for future researchers. I could make up a bunch of Word files, get a Mozy account, and back them up somewhere -- and maybe that's the best idea, although who knows if MS Word will be readable in the future, and I can't stand writing in a plain text editor under Windows.
I like to think that I'm not alone here. We all, in a way, live our lives in bubbles. If people wrote more, documented more, left more for their future, it not only would put our lives in the best context of all - our own - but preserves what we all do, what we live for, for future generations. I don't want to be a name and a date on a family tree in five hundred years. I want someone to know about my experiences growing up, my impressions of things, my mode of thinking.
VMWare Server + Win2k3 64 bit + Linux NFS = Not Fun
I haven't written a tech blog post in a while, but I've been working on an interesting, albeit frustrating, problem over the last few days.
At work I have 12 Dell PowerEdge 1950 servers, each with dual quad core Xeons (ranging from 1.8ghz to 2.3ghz), 16GB of RAM, and 138GB SAS drives. They're running VMWare Server 1.0.3 on CentOS 4.4, with all of the latest OS level updates installed.
We're virtualizing about 120+ Red Hat Enterprise Linux 4 U2, U4, Windows 2000, and Windows 2003 Server nodes, both 32 and 64 bit. These nodes would be running my company's software, Oracle, and MS SQL Server.
The bulk of those VM's live on a Dell Poweredge 2900 server with 8 x 500GB SATA drives, and a Dell PERC 5/i RAID controller in a RAID 5 config. The CPU is a quad core 1.8ghz Xeon. It has 2GB of RAM. The server is running CentOS 5 and is sharing it's disks with NFS v3. There's a 2GB bonded ethernet connection using the onboard Broadcom nic's and a Dell Powerconnect 5324 switch.
We were seeing that Windows 2003 64 bit nodes, when under moderate to heavy load, would experience massive packet loss. Additionally, the VMWare Server Client would not redraw the servers screens reliably. Finally, the node would bluescreen with a KERNEL_DATA_INPAGE_ERROR. This would happen when our software was copying SQL Server media to the node in preparation to provision a database. This would only happen with 64 bit Windows - 32/64 bit Linux would be fine, and 32 bit Windows would be fine.
The Windows Event Log would be littered with warnings and errors about "The device, \Device\Scsi\symmpi1, is not ready for access yet." It didn't take a rocket scientist to figure out that something was happening to make these machines try to access swap, fail, and bluescreen.
Now, I had been told by users that this was happening on nodes that were on local disk as well as our remote NFS server. I did extensive testing and was not able to reproduce the problem when the nodes were on local disk. It turns out that I was given erroneous information, and that nodes that people thought were local were in fact on NFS. Once I moved my test nodes over to NFS, I could reproduce the problem.
VMWare has a KB article that addresses this issue. In fact, it seems fairly common for people who run their VM's over an iSCSI SAN. Once I applied the registry change, my VM's stopped bluescreening, but our file copy operation would still fail.
Looking on the VMWare Server, you would see load averages of ~20-30, and iowait's around 25%. Looking at the NFS box, you could see that i/o to /dev/sda2 was eating up about 100% of CPU.
I changed our NFS mount options. No dice. I turned on Jumbo Frames on the bridged nic on my test VMware server. No dice. Each step would make things a "little" better, but not solve the problems.
Then, I moved the VM images over to our Netapp, which was no small feat since most of the space is used. I finally freed up about 120gb, enough for my 5 test VM's and their snapshots, and went to testing. I fired the VM's back up ran through another provisioning event.
Not only did my packet loss issues seem to go away, but for once I was able to run a Windows 2003 64 bit node on NFS and provision MS SQL instances without bluescreening.
Our Netapp isn't the newest model. It's a FAS 270 with 1.2tb of space. It's connected to another Dell switch in another rack, with a 1GB uplink to my core switches. The Netapp does not even have Jumbo Frames enabled. Somehow, though, it's kicking the crap out of my Dell NFS box, despite being seemingly "inferior."
My questions at the moment are:
- Is my config on this NFS box fundamentally broken somehow?
- Is Linux's NFS server really bad? Would I be better off with BSD or Solaris?
- Is something up with the driver for the PERC/5i? Is write caching enabled?
- Is there something up with the LSI driver in Win64 that does not show up in Win32 or in Linux?
- If I have to rebuild this NFS box, where do I put 1TB worth of VMWare images while I rebuild the box?
More Adventures near Lower Mill
This Saturday I hiked the bottom portion of the canal from Mount Misery Brook to Bisphams Mill Creek.
On the George Cook Topo Map of 1887 you can plainly see the canal, and the darkened area where it crossed Mount Misery Brook.
The lake down by Lower Mill does not exist anymore. When Henry Beck visited in the late 1920's he found the canal emptying out into the lake right in front of a dam. Looking at the NJDEP i-MapNJ aerial photography of 1931, you can see the lake is gone. Today Bisphams Mill Creek flows through normally, and the only thing that betrays the existence of that lake is a cedar swamp and a deep depression.
I won't re-write what I wrote on my site, so if you're curious to read about the days adventure, you should mash your mouse button on this link. Everyone that is, except for Jae. He's not allowed.