For the full effect, though, you’ll also need to listen to/read the previous two non-Q&A episodes of the show, #149 and #151. (Low-bandwidth and trascriptions can be found here and here.) The entire dialog concerns the recent trend of ISPs selling out their customers to allow third-party advertisers to come in and install hardware at the ISP to facilitate tracking the ISPs’ customers’ surfing habits across sites. While the ad companies in question claim to not be recording personally identifyable information about the ISPs’ customers, the capability is there and the possibilities for abuse are enormous. It brings back many shades of the DoubleClick controversies of the late 1990s-early 2000s, only much more ominous. I provided a unqiue standpoint to the discussion: that of a Web developer hosting a site and encountering similiar mysterious “first party” cookies set for my domain but not set by me.

The full body my question is present, but I’m not completely satisfied with the answer. Let’s just say I think Steve Gibson made an assumption about the GPF site that’s not 100% true. I’ve replied to his response with additional information. I don’t necessarily expect another response (he does, after all, have his own agenda to follow on his show), and even if he does it will likely be in episode #154, the next scheduled Q&A episode. If anyone is interested, I’ll post updates if and when this occurs. If I don’t get a response, I’ll post my response here, especially since it contains some disturbing observations about “first party” cookies that have mildly paranoid folks like me nervous. (I’d hate to see what it does to really paranoid people.)

I’m not sure how many of you out there follow the Security Now! podcast over at TWiT, but it’s probably obvious by now that I do, given recent posts. This past week’s episode, #119, exposes a rather unsettling fact that shouldn’t be ignored. (The high quality 64kbps MP3 can be found at that link, while a 16kbps MP3, a transcript in various formats, and additional notes can be found here.) While I encourage you to download and listen/read the facts for yourself, I’ll see if I can summarize it below for the attention-span impaired.

For a long time, I’ve defended PayPal as a method of monetary transfer. They’ve always been good to me personally, even during the stormy periods where some GPF readers boycotted them for “questionable” practices. (See the PayPal Wikipedia entry for an abbreviated history.) For that matter, many online comics wouldn’t be able to monetize themselves in any fashion if it weren’t for PayPal, as many webcomics use the service for donations and online stores. (PayPal has always been an acceptable form of payment in every incarnation of the GPF Store.) They’ve always had issues with customer service, but they’ve also been champions in anti-phishing campaigns.

But Steve Gibson and Leo Laporte have helped disclose a rather shady new practice: In a previous Security Now! episode, a listener mentioned problems downloading a software service from PayPal, only to discover that the download link was sending him to a server over at DoubleClick rather than PayPal. Since he was locally blocking access to the domain “doubleclick.net” in his hosts file, the link failed and the software would not download. Gibson promised to investigate the incident and after a number of side-tracks finally presented his results.

DoubleClick, for the few out there unfamiliar with it, is one of the Internet’s largest online advertising agencies, serving ad banners to millions of Web sites (including, indirectly, GPF). DoubleClick has long been unpopular among netizens for its questionable policies of tracking Web surfers across multiple sites, using a trick with tracking cookies to follow you from site to site. Privacy concerns were raised even further when Google, a company that itself stores and indexes a lot of personal information about its users of GMail, Ad-Sense, and other services, recently purchased DoubleClick. DoubleClick eventually bowed to pressure from the Net at large and created an opt-out page so their tracking cookie would contain “non-personally-identifiable information” and thus negate some of the tracking cookie’s effectiveness. (This opt-out page is still linked to (now indirectly, as the URL has changed) from the GPF privacy policy page.) Many folks these days, however, including myself, simply run spyware scanners like Spybot: Search & Destroy or Ad-Aware and periodically delete such tracking cookies, or just block the “doubleclick.net” domain and its subdomains using the hosts file trick mentioned above. (This is how, in part, Spybot’s immunization against cookies works.) This eliminates or at least minimizes the opportunity for your Web surfing habits to be linked personally to you.

However, PayPal’s new links bypass many of these anti-drive-by-cookie-ing techniques by sending you directly to DoubleClick’s servers, rather than inlining content like Flash or images from their site. Since these are internal PayPal URLs and not links that are expected to send you to the outside, they should be immediately suspicious. What’s even worse is that if you examine the URL closely, there appears to be some sort of “user ID” like number included that may personally identify you if you click on it. What’s even more disturbing is the number of these links you run across as you surf the PayPal site; while some obviously ad-like images contain the “doubleclick.net” URL, many links in the site bar that look like ordinary navigational links contain it as well. While Gibson points out–quite rightly–that there is no evidence to support any sort of conspiracy theories that many come to mind, it is obvious enough that some sort of information sharing is going on between the two companies, and that if a unique user identifier is indeed being passed along with the URL, there’s a likelihood that both companies can link your potential spending habits with PayPal to your surfing habits tracked by DoubleClick.

Now it’s easy to be alarmist and to say everyone should boycott PayPal. Unfortunately, so many of us in webcomics depend on PayPal for survival, so there’s no way we can easily remove ourselves from it. And there’s no competitor out there with enough critical mass to really challenge PayPal for dominance, so there aren’t many viable alternatives. Thus the only current immunization option is diligent observation.

The good news is that the DoubleClick URLs within PayPal’s site all contain at the end PayPal URL you will eventually be redirected to. It’s trivial to copy the URL, paste it into your address bar, crop out the DoubleClick portion, and go directly the the PayPal internal destination. Laporte even suggested that it won’t be long before someone comes up with a Firefox plugin that does that for you on the fly. The problem I see with this is that it won’t be long before the diabolical duo figures out savvy users are bypassing the links and they find a better way to obscure the redirection target URL so the copy/paste/edit trick will no longer work. While true encryption might be a bit too much server load for them to handle en masse, a simple ROT13 or Base64 encode might be enough to thwart all but the most stalwart gearheads.

So… should you avoid PayPal? That’s up to you. I can’t, but I’ll be a lot more careful of where I click on their site from now on.

First and foremost, I can’t claim full credit for this idea. It borrows some from Steve Gibson‘s roaming authentication scheme outlined in episode #113 of the Security Now! podcast. In that show (and subsequently continued in episode #115), Gibson outlines his method of allowing his employees to access secure portions of his site while traveling. The method described here is not quite as secure as his, as I’m forcing things to happen at the Web server software layer as opposed to the application layer and thus don’t have the same fine granularity of control he has. However, it uses many of the same ideas.

It’s relatively easy with mod_rewrite to protect certain resources of a site by restricting access to certain IP addresses. Consider the following:

RewriteCond %{REQUEST_URI} ^/store/admin/.*
RewriteCond %{REMOTE_ADDR} !^192\.168\.13\.
RewriteCond %{REMOTE_ADDR} !^127\.0\.0\.1$
RewriteRule ^/store/admin/.* /store/ [R,L]

This rule set essentially says: (1) if the requested URL starts with the string “/store/admin/” and (2) the IP address of the requesting client does not begins with “192.168.13.” or (3) is not exactly “127.0.0.1″ then (4) redirect all requests for URLs starting with “/store/admin/” to the root URL of the store, “/store/”. Essentially, we’re only allowing access to what is apparently the administrative portions of an online store to a very limited number of IP addresses, one of which is fully qualified (the “loop-back” address of 127.0.0.1) and the rest belonging to a range (192.168.13.0 through 192.168.13.255). Anyone outside these IPs will be transparently redirected to the front page of the store. (Redirecting is much friendlier than outright forbidding access.) All of this takes place in Apache itself, before we even get to the application and any potential security flaws it might have. There are no worries about hacking the store software itself to deny access. Of course, we can list any number of REMOTE_ADDR entries that we wish; each condition is a regular expression (which are negated here by the “bang” at the front) so we can filter on any octet we want and can easily specify real, outside IPs rather than private ones. For example, for this site I limit access to my various admin sections to the IP of my cable modem and our outside IP at work.

However, what happens when you are required to go on a trip and need to access the administrative parts of the site while on the go? Obviously, you can’t add the hotel’s outside IP to this rule set in advance (imagine asking the front desk for that information), and you probably won’t be able to add it easily once you get there. Sure, WordPress and the store front software have login security on their various admin interfaces, but we’re trying to protect those from hackers, right? Aside from reopening them to the entire Internet before the trip and closing them again once we get back, there aren’t very many options. How then can we identify approved “roaming” users and/or machines so they can access the admin sites without being inside a hard-coded list of IPs?

Gibson’s answer was to optionally set a secure cookie in the user’s browser if they access the admin site within one of the approved IPs first. Being within an approved IP, they aren’t restricted by the access rule and they are allowed to reach the login prompt. During login, they are prompted on whether or not they want to enable roaming access on this particular machine. If they agree, a secure cookie is set in the browser and set to expire at some date in the future. Later, when the user attempts to access the admin site outside of the approved IP list, the site checks to see if the cookie has been set. If present, the user is allowed to log in, just as if they were within one of the approved IPs. The cookie acts as a kind of two-factor authentication: the first factor being “something you know”, the user name and password, and the second being “something you have”, the cookie. Since the cookie is set in secure mode (HTTPS), it will only be sent back to the site over a secure connection. And since (well behaved) browsers only allow a site to read the cookies it has itself set, no other site should be able to read it.

This is all well and good… if you have access to the source of the application you’re trying to secure and you’re willing to hack it. Gibson wrote his own store front, so this was relatively easy for him to integrate. But I want to secure WordPress, a third-party store app, and a few random subdirectories that are pretty much statically built HTML. As much as I like running Open Source software, I usually prefer not to muck around with things if I can help it, lest I screw something up. Thus, I don’t particularly want to hack WP and the store to add this extra layer of functionality. Fortunately, though, mod_rewrite gives us a mechanism through which we can accomplish basically the same thing without modifying the underlying application. In theory, since all this occurs before we even reach the application, one could argue it may even be more secure than the application’s authentication mechanisms themselves.

You can actually set browser cookies via mod_rewrite rules. Consider what happens if we insert the following before the rules we defined above:

RewriteCond %{REMOTE_ADDR} ^192\.168\.13\.
RewriteCond %{HTTP_COOKIE} "!(.+; )*admincookie=uniqueval(; .+)*"
RewriteRule .* - [CO=admincookie:uniqueval:.domainname.tld:43200:/store/]

This rule set essentially says: (1) if the remote IP starts with “192.168.13.” and (2) there isn’t a cookie already set by the name “admincookie” then (3) set a cookie named “admincookie” with the value “uniqueval” for the domain “.domainname.tld” (assuming that’s our real domain name) for a period of 30 days (60 minutes x 24 hours x 30 days = 43,200 minutes) restricted to the path “/store/” and its subdirectories. Now let’s modify the rule set from before:

RewriteCond %{REQUEST_URI} ^/store/admin/.*
RewriteCond %{REMOTE_ADDR} !^192\.168\.13\.
RewriteCond %{REMOTE_ADDR} !^127\.0\.0\.1$
RewriteCond %{HTTP_COOKIE} "!(.+; )*admincookie=uniqueval(; .+)*"
RewriteRule ^/store/admin/.* /store/ [R,L]

Note that we’ve added a new condition. In addition to checking for the approved IP list, we also check to see if the “admincookie” has been set and that its value is what we expect (“uniqueval”). Note the parenthetical parts at the beginning and end of the cookie regex; these should make sure we match the unique cookie name/value pair, regardless of how many cookies are present. (Also note the quotes around this regex; since whitespace delimits the parts of the rewrite statements, the quotes are required to include the spaces after the semicolons in the regex. Without the quotes, the regex produces a “bad flag delimiters” error when Apache parses the configuration file.) Since each approved item’s entry is negated, the rule is only applied if none of them match. So now we should be able to get into the site remotely if and only if we’re inside an approved IP or we have the secret cookie, which we know is only set if we’ve been in one of the approved IPs first. Instant roaming authentication!

To summarize, the primary advantages to this scheme are:

• Restricts access to specific requested URIs to specific IP addresses and/or ranges and to machines outside those addresses that have a special roaming authentication cookie.
• The roaming cookie can only be set from within one or more of the authorized IPs.
• The cookie is set and checked at the Web server level, before the request reaches application code, so this scheme can be placed on top of third-party applications as an additional layer of security. No changes to the application layer are required.

There are, of course, a few caveats:

• Based on these rules alone, none of the information transmitted back and forth is encrypted; it’s all sent in the clear, which may potentially be sniffed by a man in the middle. Then again, you can always use mod_rewrite to force certain URLs to always use SSL (assuming you have a secure certificate), thereby securing the connection first. All WP admin functions, the GPF Store, and my other secured admin locales here on this site are all secured via SSL, so that helps in keeping my site secure by eliminating sniffing. (Of course, if you go this route, don’t forget to copy any necessary rules from the main Apache configuration file to the SSL config file, as the secure site will be treated as a different virtual host with its own set of rewriting rules. This little hiccup is what was keeping me from publishing this for quite a while.)
• Forcing SSL, however, doesn’t necessarily protect the cookie itself. Gibson’s roaming authentication cookie required an SSL connection. This is called a secure mode cookie. While I’m still doing research into this, as far as I can tell so far mod_rewrite does not have the facility to specify secure mode in a cookie set by a rewrite rule. Thus, the above cookie is not secure and will be sent with each request in or below the specified path, encrypted or not. The cookie is then theoretically susceptible to sniffing attacks. Setting a secure mode cookie is easy enough to do in application code, but not apparently so in mod_rewrite.
• The value of the cookie is currently hard-coded to a set value, and every browser accessing it within the approved IP will receive the same hard-coded cookie value. Ideally, the cookie should be unique to each browser and somehow obscured or, better yet, encrypted. Unfortunately, while I’ve been researching this also, so far I haven’t come up with a way to create such a unique token natively within mod_rewrite. (Remember, all this is occurring in Apache before we even reach application code.) Right now, I’m using a rather large hash of a unique pass phrase string I use a small command-line script to create a very long, highly random string using random numbers, several hashes, a little bit of Blowfish encryption, and some random string manipulation, but the cookie value is still technically hard coded. It may be possible to write a cron that will periodically create a new value token, update the Apache config file, and restart Apache. This will update the value every so often, but it seems quite a hassle. (Plus, the user under which the cron runs has to be root in order to modify both the config file and to restart Apache.)
• Similarly, there’s no way to distinguish between browsers behind the approved IP. My desktop is unlikely to roam anywhere, so it technically doesn’t need the cookie. Meanwhile, if my parents bring their laptop over and visit this site from that machine while within my network, they’d get the roaming cookie as well. Neither of these scenarios are ideal. In application code, it would be simple to set the cookie only after the user has been authenticated by the app’s internal mechanism first. But we’re not working with application code; we’re rewriting URLs in the server. Thus, this becomes a different security concern: controlling access to the approved IPs, which could be both a physical (who can patch directly in via Ethernet) and a logical issue (who can access the wireless LAN). One thing I’ve done to get around this problem is to require the browser to be authorized for roaming access to first navigate to a very large (~50 characters), randomly-generated alphanumeric subdirectory alias (generated using the same command-line script mentioned above) that is highly improbably that some might guess. This alias can only be accessed within a single authorized IP address that I directly control. It is only then that the cookie set. This eliminates the unintentional cookie setting by casual browsing of the site from other machines behind the approved IP.
• The regular expression to match the cookie should probably be more precise. For example, the expression as stated above could also match the string “myadmincookie=uniquevalnum2″, which technically isn’t what we want. Since we’re only dealing with cookies that should be set by our domain, it may not be that big of a deal, but it’s still a vulnerability nonetheless. If nothing else, there’s always the potential of colliding with cookies sent by other applications running on your site, so picking a unique cookie name and value is important. The %{HTTP_COOKIE} variable gets all the cookies for a given site/path as one big string, with each name/value pair delimited by a semi-colon and a space (“; “) and the name and value are glued together with an equal sign. I’m looking into a better regex to match this more precisely and I’ll update this post if I find one. I’ve updated the regex so it should match the cookie name/value pair more exactly.
• Of course, none of this by itself can completely secure a site. In addition to this scheme, I force SSL on certain paths, deny access for all users to other paths that should never be accessed directly, and even explicitly block certain IP ranges that have attempted to hack the site. It’s not fool proof by any means, but combined with many other secure practices and mechanisms, it adds one more layer of protection, and sometimes one added layer can make all the difference.

I welcome any feedback on how to improve this, especially if anyone knows how to get around the secure and unique cookie caveats.

Appendium: I should also point out that this scheme should be equally usable if you place the code in your master Apache configuration file (usually something like /etc/httpd/conf/httpd.conf on UNIX clones) or in per-directory .htaccess files. I usually prefer to put such rules in the master config file, mostly because it’s more secure (outside of the document root) and only gets parsed and loaded once while .htaccess files are read and parsed each time there’s a request in that directory (or any of its subdirectories). However, that only works if you have access to the master config, which most shared hosting services don’t provide. Of course, such rules placed in an .htaccess file will only apply to that directory and its subdirectories, so you’d have to tweak the rules (such as file paths and the cookie path) as necessary.

Update 11/20/2007: Updated cookie regex to better match the exactly name/value pair; added notes about rotating cookie values.

Update 11/30/2007: Put cookie regex in quotes to correct avoid “bad flag delimiters” parsing errors; added advantage summary to better showcase the advantages of the scheme; updated my cookie value scheme; added highly-random subdirectory alias to avoid unintentional cookie-ing