By now, I’m assuming most of you have read Mondays GPF News item. (If you haven’t, shame on you.) GPF is leaving Keenspot, and I’m neck-deep in unit testing the new site with hopes of releasing it to beta testers soon. If you’re interested in beta testing, you can volunteer in this thread on the old forum.
However, I’ve hit upon one little programming snag, so I thought I’d put out an appeal for help. I thought the blog would be more appropriate venue for this than the forum; that assumption could be wrong, but I’ll go with it anyway. For those of you with some Web-based programming knowledge, especially in the areas of PHP and cookies, please put on your thinking caps.
As part of the new site, I’m implementing my own version of Keenspot’s PREMIUM service, reusing the old relabeling of GPF Premium. Keenspot PREMIUM is going away (for several reasons I won’t go into here), but as the service’s biggest proponent and largest beneficiary, I’d hate to lose that functionality. So the new site will launch with its own independent Premium functionality including all the old service’s features (optional ad-free surfing, weekly archives, High-Def archives, tons of exclusives like Jeff’s Sketchbook, etc.) plus a few new features that I’ve been wanting to implement but haven’t had the time or technological hoop-jumping expertise to work on at Keen.
For security reasons, I want to secure Premium sign-ups and account management via secure HTTP (HTTPS). The benefits should be obvious. By encrypting account creation & management pages, you eliminate sniffing attacks and protect user privacy. While these pages may still be susceptible to other forms of attacks (and I’ve coded them to be as resilient as I know how), encrypting the traffic end-to-end can go a long way to cutting off those vectors of attack.
However, I seem to have hit a brick wall when it comes to setting the Premium authentication cookie. Like Keenspot’s implementation, the subscriber’s browser will be “enabled” by “branding” it with a cookie, which will be read and authenticated each time the page is loaded. If valid, Premium features for that page will be turned on; if invalid, the page will default to a non-enabled state, which could be a simple as showing all ads or as complex as denying access to the content within. Unlike Keenspot’s implementation, which was JavaScript based, mine is scripted server-side in PHP, meaning it should be more accessible to a wider range of browsers and in theory more secure (no Premium content is sent at all if Premium is not enabled, rather than letting the client browser decide). My implementation has been thoroughly tested and appears to work pretty much flawlessly… with one hitch.
The problem occurs when I set the cookie over the encrypted HTTPS connection, then try to read it over unencrypted HTTP. I appears that none of my test browsers send the cookie back when the encryption state changes. The reverse is the same; if I change the URL and set the cookie over HTTP, then try to access a page via HTTPS, the encrypted page can’t see the cookie either. It works like an either-or situation, when what I really want is both. If I set a cookie over HTTPS, I want to see it in both HTTP and HTTPS mode.
PHP’s primary cookie interface is the setcookie() method (for setting) and the $_COOKIE array (for reading). setcookie() includes a boolean parameter for secure cookies, i.e. cookies that will only be sent via HTTPS. What’s annoying is that even when I set this flag to false to force it to be insecure, the scripts continue to exhibit the same behavior: cookies set via HTTP can only be read via HTTP and vice versa. I’ve also tried setting the same cookie both ways–first in one protocol, then the other, without erasing the first cookie–but that didn’t seem to work. The second cookie overwrites the first one, effectively turning it off.
I had heard that IE 6 exhibited this behavior as a bug. However, I tried the exact same tests in Firefox 2.0.0.11, Opera 9.24, and Safari 3.0.4 (all on Windows) as well as IE 7, and all reacted the same way. Cookies set over HTTP could not be read over HTTPS and vice versa. It’s a bit frustrating. Obviously, I don’t want my Premium folks to be forced to use the new site in encrypted mode all the time, as this would slow down all the pages and put a significant extra load on the server as the number of subscribers increases. But I want to protect my users’ privacy and settings (and one of my important revenue streams) by encrypting their account access.
So I guess I’m looking for answers to two questions:
- Is this some new standard of behavior that I’m missing? I was operating under the assumption that secure cookies (those set with the boolean secure flag) were restricted to HTTPS, but otherwise all other cookies should be sent regardless of whether it was encrypted or not. When I use the awesome Firefox Web Developer plugin, it tells me that the cookie should be there regardless of the encrypted state. Yet it still doesn’t get sent. But am I asking too much? Am I not understanding the specifications, or has something changed that I wasn’t aware of? Can I not have my cake and eat it too?
-
- If I’m right and this should be working, what am I doing wrong? Is there a feature of PHP I should be use other than setcookie() to do this? Is it a bug, either in PHP or the browsers?
- If I’m wrong and this behavior is expected, is there a workaround I can use to let the same Premium features work over both HTTP and HTTPS? Or should I just give up on the encrypted account management and assume my stuff isn’t worth stealing enough to bother?
Any responses via e-mail or (preferred) comments below will be appreciated.
Update March 5, 2008: Thanks to the input of many commentors below, it looks like I’ve got a solution. The problem, as usual, was somewhere between the chair and the keyboard and the faulty component has been sufficiently flogged with a wet noodle. Immense thanks to everyone who provided feedback and suggestions.
There seems to be an obvious (to me?) simple solution. Your site needs to send two cookies when turning premium on/off. One unique cookie is https, the other is http. They can’t be the same cookie with the bit flipped, as you observed that one overwrites the other. So, give them slightly different “names” (”premium-s” and “premium”?) so they both live afterwards. If either cookie is returned, then enable premium.
Or is this too simple, and I don’t understand the underlying concepts?
Well, that thought occurred to me already and I actually tried to implement it. The only problem is, the same situation occurs. If I set both cookies while in HTTPS, I’ll get the secure cookie (”premium-s” in your example) via HTTPS but I won’t get the insecure cookie when I use HTTP. Same thing vice versa: if I set both cookies via HTTP, I’ll get the insecure cookie in HTTP but I won’t get the secure cookie in HTTPS. It’s totally bizarre.
The only other thing that’s occurred to me was to still use two cookies, set the secure one via HTTPS, bounce the user to insecure HTTP and set the insecure cookie, then bounce them back to secure mode to do other things. But that sort of convoluted mess sounds like a jumble of security holes waiting to happen and I’m reluctant to even try it.
I’m honestly at a loss here. It’s as if everything I thought I knew about cookies isn’t working anymore. I’m leaning toward ditching the entire HTTPS mode, although I’d hate to expose my readers to potential privacy concerns. It’s not like Premium records anything dangerous; at most it gets your registered e-mail address, an arbitrary user name you define, and a password that gets hashed so not even I can read it in the DB. But still….
yes, it’s a convoluted mess, but you do what you have to do. One option may be to set the “other” kind of cookie on your way out of that space (assuming you can get the user to do something ordinary like go back to browsing the comic or clicking on a “logout” button, instead of them just closing their browser).
Another thought — maybe the premium cookie *only* needs to be dealt with while in ordinary HTTP: mode, not HTTPS:. What I mean is — make sure your site is tightly segregated into https: and http: sections, with no crossover (as much as possible) — you should only need to use HTTPS: for when a person is entering their password, or creating/editing their profile, or invoking the part of the site when they try to give you a credit card number or other payment information. (Maybe one or two other things I didn’t think of.)That’s it — the rest of the site lives in HTTP: mode. Therefore, to set the cookie, they click on a button which flips them out of https: to http: via redirect, sets the cookie; perhaps that’s only done because they press a button marked “logout and set cookie”.
(Hopefully the thought will spark ideas as to how you’d want it to actually behave, given the circumstances. Good luck.)
Dumb suggestion. Sometime, the newer features are not supported properly.
You might check other cookie vars such as $_REQUEST, or $HTTP_COOKIE_VARS. $_COOKIE is supposedly new. It might have be a php bug.
Worth a shot.
Alternativly, when the login succeeds, you could create a session id good for one ping, and store it. Then redirect to a non-secure URL with the session id as a paramter. Once checked one time, the session id is delete and the non-secure channel sends a cookie back to be set. Somewhat annoying but not too out there.
Not sure if this will work (ymmv) …
In my previous job as a Perl e-commerce guy, we needed a script that could set & get cookies from both the secure & non-secure versions of a given domain. I seem to recall we accomplished this by leaving out the “http” & “https” portions of the domain name set in the cookie. If the problem you’re having is related to the use of PHP’s “setcookie” function, you may want to try bypassing it and set cookies manually using the php “header” function instead - you’ll have more control over your cookies this way.
Thanks for the input, guys, and sorry for taking so long to respond. Unfortunately, none of your suggestions are working. I’ve tried writing the Set-Cookie header manually, checking all possible avenues for returning the cookie to PHP ($_COOKIE, $_REQUEST, $HTTP_COOKIE_VARS), setting the cookie twice (once in HTTP and again in HTTPS), using separate cookies for HTTP and HTTPS… none of which have worked. I still get the same situation: it sets it for one but won’t read it for the other.
At this point, I’m at a total loss. I’ll be launching the site without HTTPS on the account management stuff for now. The only solution I can come up with something along GreatScott’s line: when it comes to actually setting the cookie, bounce them out of secure mode, set the cookie, and bounce them back. Unfortunately, I don’t have time to implement a solution that complex.
This is a long-shot, but try explicitly setting the cookie path to ‘/’. I seem to recall reading about a similar problem that was solved by doing that even though it wasn’t clear why that worked as the URL path was supposed to be identical for the http and the https URLs.
I juat signed up for GPF Premium and took a look at the cookie you use. It already has path=’/’ so my suggestion to try that won’t help.
But here might be a clue. Whatever you are doing for the GPF_NEWS cookie works fine. I can go to any of http://www.gpf-comics.com http://gpf-comics.com https://www.gpf-comics.com https://gpf-comics.com and they all see my GPF_NEWS cookie.
However, with GPF Premium, not only does https not work to see the gpf_premium cookie, but here is a new clue: http://gpf-comics.com/premium does not use the cookie made by going to http://www.gpf-comic.com and vice-versa, even though the cookies look the same when I look at them in Firefox and they do overwrite each other. So maybe you can try to figure out why the premium cookie for gpf-comics.com overwrites but is not compatible with the one from http://www.gpf-comics.com
More clues: I installed the editcookie addon in Firefox to make it easier to look at the cookies, and did the following experiment:
1. Go to http://gpf-comics.com/premium log in to my Premium account, click on the button to enable Premium in my browser, then look at and save the Content of the gpf_premium cookie.
2. Then go to http://www.gpf-comics.com/premium, notice that I am not logged in, login to my Premium account, click on the button to enable Premium in my browser, then look at and save the Content of the gpf_premium cookie. It is different than in step 1.
3. Repeat step 1. Verify that the content of the cookie is the same as in step 1.
4. Repeat step 2. Verify that the Content of the cooke is the same as in step 2.
The problem with the News Notification cookie is that, depending on when it was last set, it was probably set by JavaScript (the only option available to me at Keenspot) where as Premium is being set via PHP. The News cookie is now also being set by PHP setcookie(). It looks like the News cookie properly updated with today’s News update, according to a quick check of my cookies. As far as I know, there shouldn’t be any difference between how the News and Premium cookies are now being set, so it might be interesting to try your test again, now that the News cookie is being set via PHP.
On “gpf-comics.com” vs. “www.gpf-comics.com”: I should point out that before long URLs at “gpf-comics.com” will eventually be redirected to the comparable URL at “www.gpf-comics.com”. This is primarily to help search engine optimization and such, and since I actually do have sub-domains the “www.” is still significant. Those redirects aren’t in place yet because they will conflict with supporting the beta test domain, but once the beta test domain goes away and all its URLs redirect to “www.gpf-comics.com” URLs, the “gpf-comics.com” URLs will redirect as well.
That said, all the cookies should be set for “.gpf-comics.com”, meaning it should work for any sub-domain. Thus it should work regardless of whether the “www.” is there or not. Unfortunately, I haven’t had time to test this at all. I’ll see what I can do, but now that we’ve launched the site it will probably be a lower priority for a bit.
I noticed the change in the GPF_NEWS cookie during my tests, and my results are with the new PHP version of the cookie. The http and https, both with and without ‘www.’ all work fine and together with a single cookie for GPF_NEWS but not for Premium. So there is something you are doing differently between the two.
In Premium, the cookie domain is properly set to ‘.gpf-comics.com’ and works with and without www, but the Content field of the cookie is being set differently with and without www, making the cookies different. I suspect that if you figure out why Premium’s www and non-www cookie Content fields are different you will find out what the problem is with https.
Without giving away too many details, the Premium cookie contains a signed hash of several values to provide a validation check to prevent tampering or stealing of the cookie. However, the HTTP host value is part of that hash, so “www.gpf-comics.com” and “gpf-comics.com” would produce different hashes, and thus a cookie set by “www.gpf-comics.com” would not be valid if read by “gpf-comics.com”. So in that place the validation would indeed fail. It might be a short term problem to contend with, but as previously stated HTTP requests to “gpf-comics.com” will eventually automatically redirect to “www.gpf-comics.com”, so this shouldn’t be an issue by the end of the month.
That, however, shouldn’t be an issue with HTTP vs. HTTPS. The protocol isn’t included in this host value, so it shouldn’t matter whether you access it via “www.gpf-comics.com” or “gpf-comics.com”. For that matter, the SSL certificate is for “www.gpf-comics.com”, so you shouldn’t even be able to access the site in HTTPS mode at “gpf-comics.com” without producing certificate errors.
However, also included in the hash as part of the “hidden” data (to add some entropy and salt to the value) is
$_SERVER['SERVER_SIGNATURE'], which is different when accessed via HTTP or HTTPS (”Port 80″ vs. “Port 443″). That would be just enough to throw the hash off and make it different. This could be the magic bullet. I’ll run a couple tests as soon as I can to see if that solves the problem and report back with the findings.That did it, bugstomper. When I removed SERVER_SIGNATURE out of the mix, the hash became consistent across the HTTP/HTTPS barrier. I’ve written a new cookie hashing algorithm that should be more stable when switching modes and was able to test it. I successfully set a cookie in HTTPS mode and then read it via HTTP, which is exactly what I wanted.
I’ll need to modify my code to get this to work, which should be pretty simple as the function exists in a include file that gets called everywhere it’s needed. The bad part is that this change will break all the existing Premium cookies, so subscribers will have to re-enable their browsers once it goes into effect. I’ll make the changes, test them, then notify the subscribers before I make the change.
Thanks everyone for your assistance. You came up with a lot of good ideas for what went wrong and solutions to fix them. Unfortunately, my problem was somewhere between the chair and the keyboard.
Great news! I figured from the length and look of the field that was different that it was a SHA-256 hash of something and that it had to include a field that was sensitive to the http/https difference, but of course with no way to guess what you are hashing together there was no way to reverse engineer past that point.
Coincidentally, I’m just starting a job for a client in which I’m doing my first https form, and investigating this was a big help for me. Thanks Jeff!
A signed (HMAC) SHA-256 hash to be precise, encoded with a modified Base64. Again, hopefully not giving too much away about the implementation, the hash is of rest of the cookie contents, scrambled in various ways, and salted with a number of unknown (to the end user) elements. The salt should prevent an attacker from trying various permutations of the raw data to fake the hash. It’s a really good way of storing the data in the open and still preventing tampering. Of course, I was already using a separate constant as part of the salt, so pulling information from the
$_SERVERarray was probably overkill and obviously introduced problems. Then again, sometimes good security is all about overkill.I’ve since tested the modified code in Firefox 2, IE 6, Opera 9, and Safari 3(?), all for Windows. That should probably be enough to call it a success. I’d still prefer to check it in IE 7 as well as a couple browsers in Linux, but it should be sufficient to move forward. Again, many thanks to all.