I thought about just putting this in a comment to my last site problems post, but I figured it would be more visible as a new post. It seems every time I think I have things settled around here, something else pops up that goes wrong. An observant Faultie e-mailed to inform me of a few bugs that have cropped up in the move to WordPress. Investigating his issues led me to discover a few more, and hopefully by now they’ve all been squashed. We can all thank Tim from Germany for his attentiveness. (Thanks, Tim!)
<script> tag to make sure it was valid and tried again. IE failed once again. After a good deal more researching I found that while Firefox will let you have an empty script tag (i.e.,
Now to get to what Tim actually e-mailed me about. He stated that when he tried to switch the Atom feed to the new URL, the feed broke. The only way he was able to get the feed to work was to put the IP address and port number instead of the domain name into URL. NetVibes required this change to work, but it worked as originally stated in Firefox. He theorized correctly that this had to do with how my dynamic DNS uses HTML frames to redirect HTTP requests, as previous described. Undoubtedly Firefox could work around that issue but NetVibes couldn’t. (And of course, I tested the feed through Firefox.) The obvious solution of using the IP instead of the domain is not a good one, of course, since I’ve already stated that I’m on a dynamic IP block, which would break all these links at my ISP’s whim. The best solution that I can think of to this problem is to use the domain but leave off the “www” and add the port. This would make the RSS 2.0 feed URL
http://jeffdarlington.com:8081/?feed=rss2 and the Atom feed URL
http://jeffdarlington.com:8081/?feed=atom. This still isn’t ideal, as I’m hoping to eventually rectify this whole DNS-forwarding, framing, port-forwarding mess and make it more normal, but it should work for now even if my ISP changes my IP. Of course, the old URLs should hopefully still work; the rewrite rules automatically forward to the new URL.
Next came commenting. Tim originally intended to add all this in a comment, but couldn’t find the link to register. Silly me apparently forgot to check the checkbox (which, in my defense, is rather tiny and buried deep in an options page) that allows people to register. I checked the box, updated the settings, and loaded up another browser to check the registration link. Unfortunately, I’ve now found out that WP sends you a temporary password via e-mail when you register. Being behind a dynamic IP makes sending server generated e-mail a hit-or-miss ordeal, as many spam filters and relay servers block dynamic IP blocks. I’ve already encountered this problem with the Store several times, and as yet don’t have a good workaround. I tried registering a comment login with a different e-mail address, but so far I haven’t received a response. There’s a good chance this e-mail verification will be the nail in the coffin for comments here, so I’m afraid I’ll have to disable them completely for the time being.
I’ll keep debugging and see what else crops up. Thanks everyone for your patience.
In addition to switching blog software, I’ve been quietly making a few other changes on my little Linux box lately. For one thing, I’m changing the authentication scheme for SSH from simple passwords to public key authentication. Only a few friends and family have direct access to my Linux box, so it only affects a few people. However, given recent events, I feel somewhat justified in ramping up my security.
On the off chance you might have read this into the above statement, no, my network hasn’t been hacked. No advance GPF strips or story notes have been swiped from my hard drives. (Somehow, I detect a definite sense of mixed feelings from some of you, including relief and disappointment….) But my oh my, have they been trying. Not that I think it’s a directed attack by any means, but I have had a number of break-in attempts over the past few months, and they seem to escalate frequently.
Like I’m sure many Linux admins do, I run logwatch daily. (Actually, it seems to come pretty standard with an install of Fedora Core, but if I ever switch distros I’ll make sure to install it if it’s not a default.) logwatch runs through most of your system logs and generates a single, combined report summarizing the health and status of your computer and plants an e-mail in your root mail inbox every morning. It covers just about everything, from crons, Web and FTP logs (especially errors), mail, system messages, automatic updates, and (especially for the topic of this article) security logs. It was because of logwatch that I discovered that Demeter’s hard drive was failing last year and had to be replaced (more SMART failures, surprisingly undocumented either here or in the GPF News, and completely unrelated to Apollo’s crash; last year was not a good hard drive year for me). It’s through logwatch that I watch various IIS-targeted worms periodically bounce harmlessly off of Apache (which I take a small dose of perverse pleasure in). And it’s through logwatch that I sit nervously and watch the almost daily hack attempts to break into my system.
Having any computer on the Internet is a dangerous thing these days. I certainly don’t want to alarm the less tech-savvy of you out there (the last thing I want is to discourage people from getting online), but any computer connected to the Internet is under a constant barrage of probes and pings. Most of these are harmless and often routine, but a growing number of them are malicious. Hackers are constantly scanning the Net for connected systems that may have vulnerabilities that can be exploited. Over the years I’ve grown more and more curious about Internet security, and I’m finding myself get more and more paranoid about how insecure much of the Internet really is.
Having a server on the Net is just asking for trouble. They are under a constant assault, being poked and prodded from all sides and on every port. Most regular users have a small amount of anonymity through the dynamic IPs of their ISP. Servers, on the other hand, are meant to be found, waiting for requests (as their very name states) to serve. By their very nature they advertise themselves with domain names saying, “Here I am! I’m here to help you.” Being so public, they make themselves more visible to attack. While some black hat hackers target specific hosts looking for various bits of sensitive personal or corporate information, most of these hacks are actually pretty casual: simple probes looking for any random vulnerable system that can be broken into. Perhaps they may find something useful to exploit, like financial information or blackmail material. But my theory is that they are mostly looking for new weapons to add to their arsenal, additional zombies to add to their growing botnet collection.
Virtually every day, my logwatch report comes up with at least several hundred break-in attempts via SSH. On especially nasty days, the attacks can range in the thousands. Gaining access via SSH is more advantageous than other means, as it gives you direct command-line access to the system which is also encrypted end-to-end, preventing eavesdropping by anyone in the middle. Needless to say, unencrypted telnet is completely disabled on my system.
The good news for me, at least, is that I’ve taken quite a few precautions to prevent these break-ins. The first and probably most important step is to prevent root from logging in via SSH. In order to gain root access, an attacker must first compromise another login, requiring them to break two passwords instead of just one. This adds an extra step for me when it comes to my admin duties, but that’s an extra step I’m definitely willing to make. I’ve also limited which logins can access SSH to a specific group, into which are placed only users that I want to explicitly grant such access. This greatly reduces the number of available logins that can be compromised. (Most of these attacks try various combinations of common user names, so odds are they have no clue what user names may exist on the machine.) The next big step, though, is to require public-key authentication. This means that those with console access not only need a password but a digital private key, and only the combination of the two will allow someone in. This is the form of authentication SourceForge uses for access to their build and test environments, and it’s probably the most secure method freely available (usually implemented via OpenSSH).
Setting up GnuPG keys, securely wiping old hard drives, and now locking down my Linux box with strong-grade encryption… I’m quite the paranoid little thing, now aren’t I? Does everyone get this paranoid in their old age, or is it just me? If you know the answer, you know where my public key is….
Edit April 2, 2007: I tried to make a follow-up comment to this, but that obviously didn’t work. So I’ll just edit this post to include a link to this analysis malicious SSH login attempts article I just found. It made my flesh crawl a little. In addition, check out the following Linux.Com articles on beefing up SSH security. Thankfully, I’ve already discovered that I’m doing most of these tricks already, which made me feel just a tiny bit more secure.
No, you’re not seeing things. Neural Core Dump should be looking quite a bit different now. After a lot of investigation and some pretty complicated Apache configuration file tweaking, I’ve moved the blog from using Movable Type to WordPress. Not that I’ve had any serious issues with MT; it has served me rather well the past couple years, and I’ve been reasonably happy with it. I primarily picked MT over the other options out there because (a) I wanted to host my own blog and not have it on some big mega site like Blogger or TypePad and (b) I had heard good things about MT (as one person put it, it’s “real blogging software”). However, the recent problems with the site (admittedly not MT’s fault in any way) have caused secondary issues (like the inability to post comments) that have made MT less functional for me. Plus, I like to think of myself as a closet Open Source advocate (see the advocacy question in this old NewsForge interview), and WP is Open Source all the way. (MT’s source is available, of course, seeing as it’s primarily Perl with a dash of PHP, but strictly speaking it’s not Open Source.)
Now for the nitty gritty. I’m sure you’re asking, “How does this affect me?” Hopefully, it won’t affect you much, if at all. The aforementioned Apache tweaking has been mostly mod_rewrite directives to allow old links (both internal and external) to continue to work. If you have old links or bookmarks that point to posts made on the old site, those links should continue to work. (I can’t, of course, guarantee that they’ll work indefinitely, but I tend to be more obsessive about maintaining old links that most webmasters.) However, I’d recommend that you use the new permalinks generated by WP in new links and perhaps update old links to ensure the best compatibility. Note that this also affects the various XML feeds like Atom and RSS; WP puts them in different places than MT and the rewrites should allow the old MT feeds to work, but I’d suggesting changing your feed URLs.
Commenters will notice that we’re no longer using TypePad authentication. Thus, any old logins you may have created to comment here in the past won’t work here anymore. (They should, of course, continue to work on other TypePad-authenticated sites, so I wouldn’t abandon them unless you only used them to comment here.) You’ll need to create a new “subscriber” account here on this site in order to comment. Comments are still moderated, but once you’ve made your first moderated post you should be able to make additional posts without any trouble. WP has a number of comment spam features, so I may have to dredge the filter occasionally to make sure no comments get inadvertently filtered.
WP seems to be noticeably slower than MT. This is likely because WP is purely dynamic PHP while MT builds static HTML pages. I’m not sure how much I like that, but I plan to keep using WP for a while to see how well it seems to handle the load.
Expect plenty of hammering, sawing, and general construction work as I continue to fiddle and tweak. While the conversion should be largely complete, I’ve got a whole new system to figure out, so things are bound to change.
Well, I just performed a new version of the GPF production schedule estimation that I previously mentioned. For review, I took the dates of recently finished comics (i.e. since the last estimation calculation), converted those dates to week numbers, and calculated the average number of comics I’m producing per week. Unfortunately, this copy of the calculation includes the Christmas and New Years holidays, so the estimation is abysmal: 0.58 comics per week (hereafter referred to as “CPW”). Thus, I’ve effectively produced half a comic per week, or a comic every two weeks, since mid-December. That’s not good. However, when I ignore the holidays (which are usually light on comic production anyway, but that’s typically hidden from you by my once legendary buffer) and only focus on the month of January thus far, the outlook is a bit better: 1.67 CPW. That’s technically better than the previous 1.27 CPW I calculated the last time, but it only covers three weeks, whereas the previous calculation covered almost four months. When I extend the previous calculation to encompass September 2006 through January 2007 (a nice five month average), the total average comes to 1.44 CPW.
Bottom line: I’m doing better, perhaps even increasing my production more than the last time. But there still won’t be a schedule frequency upgrade in February. Sorry, but GPF will remain weekly next month. Bummer. However, sometime in March is my next scheduled Keenspot Newsbox, so hopefully I can increase the schedule by then.
I was beginning to wonder why there weren’t very many comments lately. Not that I tend to post all that frequently, and I’ll readily admit my recent posts may very well have had little meat to them worth commenting on, but I thought at least the Geeklet Lullaby would have drawn some attention from some GPF forum regulars. Now I think I know why you guys have been so quiet.
I went to post an amendment comment to the recent site problems post to state why I think things were happening a certain way when I noticed that TypeKey was not remembering my login. Naturally, I clicked the link to sign in. I was directed to the TypeKey site, put in my login credentials, told it to remember me for two weeks, and clicked the button. I was redirected back to the blog… where it told me I wasn’t logged in and it politely asked me to sign in again. Some deep, dense Neanderthal-laden DNA entrenched somewhere in my genetic code responded with, “Duh… I’ll try it again.” (Okay, maybe it’s not so much of a dunderhead response; I’ve worked with computers long enough to know that some things are unpredictable–even when they’re coded not to be–and you ought to try it at least a couple times first to make sure the problem is reproducible.) So I clicked the link again. TypeKey dutifully remembered I was logged in and didn’t prompt me for my credentials this time. Instead, I was sent directly back to the blog… where I was told I wasn’t logged in.
I’m hoping that some of you by now are saying this sounds familiar. (If not, then I’ll start feeling very lonely.) Here’s what I think has happened, which in part will include the aforementioned amendment. I’ll assume you’ve already read the previous recent site problems post. (If not, you know where to find it.)
As previously mentioned, my dynamic DNS service is currently set up to forward the domain “www.jeffdarlington.com” to my Linux box on the port 8081 since my ISP has blocked access to port 80. Well, I found out how they make this work. If you try to view the HTML source of the main blog page, you’ll discover that it consists of a frameset with only one frame, whose source attribute points to the raw “IP:port” address. This frameset HTML is served by the dynamic DNS. What you see appears to be the main page of the blog filling the entire browser window; what you get is a page from the dynamic DNS holding a frame that points to the blog page. Inside that frame, the browser thinks everything is referenced by the “IP:port” address, not “www.jeffdarlington.com”. TypeKey is set to respond to the domain name, however, so it doesn’t recognize the raw address. Thus, when you get back to the blog after signing in, TypeKey says your authenticated to post at “www.jeffdarlington.com” but not at “IP:port”, so it continues to insist that you must sign in.
I’m not sure if there’s a workaround for this issue. I’ll see what I can do, but I can’t promise anything. I could see if TypeKey will allow authentication on the IP, but if the ISP changes my IP we’ll be back to square one. I think for the time being that we’ll just have to put up with having no comments until a more permanent solution can be found. That permanent solution may require me to buy Web space somewhere, but I like the flexibility of hosting this site myself. I might be able to convince Keenspot to host the blog, but then I’ll need to include ads to offset their bandwidth. (I’d also have to come up with a way for Keen’s automated ad rotations to work with Movable Type’s static pages.)
The good news, at least, is that it appears that the various RSS and Atom feeds seem to work correctly, at least when I access them directly from Firefox. This should mean that those of you watching this via the feeds should still be getting posts. However, if you need to comment, though, I’m afraid you’ll be stuck with the GPF forum or regular ol’ e-mail for now.
A couple years ago we replaced our old laptop, Zeus (a ThinkPad A21m), with a newer one, Apollo (a ThinkPad R51). Shortly after we purchased Apollo, Zeus started acting very strangely. While the LCD was definitely beginning to fail (it would occasionally suddenly lose luminance, which would come back intermittently), there was something else that was causing the system to mysteriously crash. The screen would go blank, the system fan (or so I thought) seemed to start making weird pulsating noises, and the keyboard and TrackPoint would go unresponsive. The only way to recover from this crash was to remove all sources of power, including the battery. It seemed to happen predictably after the system had been on for a period of time; the first time after a cold boot it would take about a half-hour to an hour, and on subsequent reboots it would occur within minutes. If I let the system sit overnight and cool down, it would seem to be okay again for an hour, and then the process would restart.
I naturally assumed because of the apparent “warm up” and “cool down” nature of the problem that it may have been something causing the system to overheat. I had encountered a similar (or maybe opposite) problem with my old college desktop, Pandora, where she wouldn’t boot until her main hard drive had warmed up enough for an apparently broken contact to heat up and expand until it connected. Unfortunately, since Zeus had already been replaced, we weren’t exactly in any hurry to repair him. My original plan for Zeus was to turn him into a Linux machine simply to toy around with, but I wasn’t exactly in a hurry to spend a truckload of money to ship him back to IBM and have him repaired out of warranty. So Zeus has spent the last year or two gathering dust on various shelves in various closets. Occasionally I would take him down and boot him up for a while to see if the symptoms persisted, and every now and then even gather the courage to pry open his case (he was, after all, out of warranty) to see if I could identify the problem. However, this became a rarer and rarer occurrence until Zeus was practically all but forgotten.
When we started making the move to West Virginia, we began to clean house on some of our old hardware. Pandora and Minerva, the two oldest desktops, eventually made there way to the Guilford County hazardous waste facility to be recycled. (Old computer hardware should never be simply tossed into the garbage, as it contains numerous toxic and precious materials that should either be safely disposed of or recycled.) This occurred, of course, after both hard drives were securely wiped to prevent the recovery of any personal data. IBM has a very nice “secure data disposal” utility available to its customers that performs up to the DoD recommended seven-pass overwrite to clear the disk. Since Zeus was apparently in no condition to perform this task (which could theoretically take hours), we decided to take him to WV with us until we could safely wipe or dispose of his hard drive. The only way I could think of to accomplish the wipe would be to put Zeus’ hard drive into Apollo, boot using the secure data disposal CD, and let Apollo do the hard part of overwriting the disk. Since this would mean Apollo, my primary Internet lifeline and comic-making workhorse, would be unavailable for potentially up to a full day, it’s obvious to say that finding the time to perform this task has been… difficult.
Well, this weekend was my monthly computer maintenance weekend, where I run the usual gauntlet of tasks to keep all our systems up-to-date: downloading and installing security and software patches, scanning for spyware and viruses, and doing hard drive error checking and optimization. (Some of these tasks, of course, are done more frequently, but they also tend to occur during the monthly batch.) The usual disk maintenance includes running Disk Cleanup, CHKDSK, and Defrag. I run this batch of jobs on all our Windows machines, including my wife’s work laptop, to try and keep everything in tip-top shape.
Saturday morning, I was just finishing up a few maintenance tasks that I had let run overnight. Ben and my wife had both already been up, eaten breakfast, and gone back to bed, so I effectively had the morning to myself. Zeus had drifted back into my consciousness again for some unknown reason and, since it looked like it wouldn’t be a busy day, I decided sacrifice having Apollo for the day to finally take care of wiping that disk. I swapped out Apollo’s drive (something I got rather good at after last year’s crash) and popped in Zeus’. As I started to boot, I immediately noticed that the weird, pulsating “fan noise” I remembered from before was now coming from Apollo. I proceeded with the disk wipe–cringing each time I heard the “click of death” coming from my “good” laptop–and began to wonder if Zeus’ problems were really hard drive related and not heat related.
I rummaged around the office, found my old KNOPPIX CD, and threw it into Zeus’ drive to see what would happen. I immediately found one distinction between these two machines. While I was able to successfully run Apollo last January entirely off the KNOPPIX CD with no hard drive installed, Zeus absolutely refused to boot. There must be something about that particular model (or series of models) where it doesn’t like operating without a hard drive. Curiosity overcame caution as I inserted Apollo’s good drive–with all my mail, Palm backups, comics, GPF-related notes, etc.–into Zeus’ chassis. It took a minute to convince Zeus to boot from the CD instead of the hard drive (which, of course, was filled with Apollo’s drivers that subsequently made Windows XP screech to a halt), but it wasn’t long before the old machine had a new lease on life. He remained up, active, and connected to the wireless network all day; in fact, portions of this blog post were actually typed in from Zeus via a rather old copy of Firefox. Fancy that.
It took Apollo an estimated 15 hours or so to completely wipe that drive, so I checked in on his progress Sunday morning. I swapped the drives back and was quite relieved to see Apollo boot as normal with his old drive returned. Zeus, unfortunately, did not fare as well. Even though his hard drive was returned and the KNOPPIX disc was still in the optical drive, the absence of a formatted hard drive with a valid operating system completely threw him for a loop. The boot order specifically said to look at the CD-ROM before the hard disk, but he wouldn’t go any further than to print a warning that the drive had been successfully wiped with the secure data disposal utility.
Zeus now sits back on his closet shelf, ready to gather dust again. If he won’t boot even from a CD without an OS on the hard drive, it’s unlikely he could be resurrected for anything useful. So now I come to the task of finding somewhere in this area where a computer can be properly disposed of. I really didn’t expect any miracles from this little bit of tinkering, but it’s always good to solve a little mystery every now and then.
To the tune of “Hush Little Baby”:
Hush little baby, don’t say a word,
Daddy’s gonna download Thunderbird.
And if Thunderbird won’t catch your spam,
Daddy’s gonna set up SpamAssassin.
And if SA won’t trap the lot,
Daddy’ll catch those jerks with a honeypot.
And once he’s blacklisted their IPs,
He’ll report their crimes to their ISPs.
And once the feds are knocking on their door,
Daddy’ll DDoS them back to 1984.
And once that scum succumbs to their fate,
You and daddy can deathmatch to celebrate.
If you’re reading this post, congratulations. That means our recent site problems have been resolved… at least for the moment. Below is the rough rundown of what happened this weekend, for those of you who care. There’s lots of technical jargon below, so if you don’t care, feel free to blindly ignore the rest of this post and live in ignorant bliss. Trust me, it’s probably a safer place to be.
For those who didn’t already know, this site and the gpf-comics.net domain are both hosted on my personal Linux box behind our cable modem. This was originally intended as an experiment to teach myself the ins and outs of running a Web server (specifically Apache), but over time it morphed into a cheap form of hosting that gave me complete control over the hardware and software involved. Keenspot hosts the main GPF site and they do a very good job of it; but to be perfectly honest, I have no clue what hardware they use, only a hunch about the software (because I have SSH access), and I couldn’t tell you for the life of me where the server is located. I like having directly physical control over the GPF Store (which is back online, BTW), which gives me a lot more confidence in what happens to my customer’s data. I also like knowing how things work.
My Apache setup is moderately complicated. I briefly touched on it before, but I’ve been using a number of virtual hosts to serve several sites on the same machine. The blog and the Store are the main sites I run, but I also have a number of other domains that I’m currently sitting on and forwarding to other places with the intent of using them for future purposes. Apache happily listened to all these requests and parceled out the right stuff to the right people without breaking a sweat. Until this weekend, that is.
Unfortunately, I think the success of this blog was part of its undoing. I haven’t looked at the logs to confirm it, but I think the recent “Ben vs. Darth Vader” post brought a bit of unwanted attention from my cable service. The increased download traffic probably caught their attention and they blocked port 80 to my IP. One reason I strongly suspect this is that my SSH access didn’t go away and I could still access the Store using the HTTPS port, so I knew Apache and my Linux box were up and they were still online. I tried a number of workarounds throughout the weekend, working in small batches here and there during stolen moments, but each fix seemed to break something else.
At the risk of revealing to my ISP what I’ve done, here was the winning solution. My dynamic DNS service lets me forward HTTP requests to different ports transparently. Since port 80 was blocked, I forwarded gpf-comics.net to port 8080 and jeffdarlington.com to 8081. (I couldn’t forward them both to the same port, because this redirect seems to strip the domain name from the request and only feeds Apache the IP, thus making name based virtual hosts moot.) The other domains, which were originally just being bounced by my Apahce, are now being forwarded directly by DNS2Go so they never reach my box. Since HTTPS seems unaffected, it’s currently still pointing to the same place. The good news is that most of your bookmarks and links should continue to work, so don’t change them (in case I need to tweak the configuration again). The caveat is that if you’ve been using “jeffdarlington.com” to access this site instead of “www.jeffdarlington.com” the redirecting will probably break. The redirect only works with the “www.” so please use that URL.
Sorry for the recent quietness, folks, but with the holidays often come extreme business, and I’ve definitely been busy. However, I thought my first post of the year should be an interesting one, and it definitely is. A word of warning: extreme baby cuteness lies ahead.
My wife and I started last night with the long, arduous task of taking down all the Christmas decorations. Of course, putting them up this season took longer than in the past because of our new then-four-month-old distraction, and I’ll bet taking them down will face the same temporal challenges. I took the job of babysitter (literally, as I was making it him sit up) while my wife started taking ornaments off the tree. This was largely uneventful, as Ben was distracted by the TV most of the time.
Then my wife took down my Darth Vader Hallmark ornament. I seem to receive a bunch of Star Trek and Star Wars ornaments every year, and I’ll be that over the years I’ve probably ended up with almost every ornament in Hallmark’s collections from these series. Well, she set Lord Vader down next to me while she started opening his box. I was exhausted and doped up on several prescription medicines (I’m currently battling my yearly bout of bronchitis), so that was probably affecting my sense of humor. I picked up Vader and started marching him through the air toward Ben, doing the Vader breath mask sound. In my best James Earl Jones impersonation (which is admittedly pretty bad), I said, “Ben… I am your father.” And something amazing happened.
Ben laughed. He laughed hysterically.
Since Vader’s lightsabre was fully extended (yes, there are occasional jokes about the comparative lengths of Vader’s and Luke’s ornament’s lightsabres), I started making the lightsabre swooshing sound. Having grown up during the golden years of Star Wars, I got really good at making that sound growing up. Once I started that, though, Ben laughed even more. Each swoosh brought more giggles, which were incredibly infectious. Mind you, we’ve heard Ben laugh before, but never have we found anything that predictably made him laugh on cue.