It is possible to configure a web server to detect the type of browser contacting it, by looking at the User-Agent header field sent as part of the request, and then serve content based on that type. This is used by some sites (particularly banks' web interfaces, I've heard?) to ‘force’ their customers to use a particular make of browser (because the bank knows only that that one works), or even a particular operating system. This is very irritating for the following reasons:

1. Receiving a message saying that one must switch software, or even ‘upgrade’, is slap in the face for ones (possibly commercial) choice of browser or operating system. It's just plain rude.

2. Who says that the rejected browser isn't capable of using the bank's web interface? A large number of thoughtful people have spent a great deal of time working out international standards for interoperability between servers and browsers, and frequently design graceful degradation into them so that servers don't have to do tricks to get a working and widely compatible interface, while still exploiting advanced features. Isn't the bank sticking to these standards?

3. The User-Agent HTTP field is merely advisory. You can't really determine what browser is being used from this string (although there are other, sneaky ways to make a good guess). When faced with a website's failure to process a perfectly decent request due to the value of User-Agent, she might simply turn to a feature of her browser's configuration that allows it to pretend to be something acceptable to the server. Consequently, when you check your server statistics, you might think that everyone is using a certain browser, when really you're deluding yourself. Furthermore, if this spoofing worked, then the test was worthless (going back to the previous point).

Because of the advisory nature of User-Agent, I advise that you do not use it, unless it serves the needs of the visitor. For example, if you have a web interface to some facility that generates XML with a reference to an XSLT transformation to be applied to it to generate HTML, you could try detecting browsers that don't support that transformation, do it for them, and supply the HTML directly. It's not your problem if their visit is marred because they don't supply an accurate User-Agent string.

One final point, which is less of an irritation than it is a source of amusement, is the use of User-Agent as an authentication system. The story goes like this… You use Google to search for something. It finds a page, and you can clearly see from the summary which Google provides that it contains your sought content. You visit the site but find no such content – you need to log in first. No account? Well, registration turns out to be free, though you'll probably have to give certain personal details, like your email address.

How did Google get the content if you didn't? Easy – the server detected that Google's robot, Googlebot, was searching its site, and let it through without a challenge. (Why? Because they want money from you, or the ability to advertise to you, or sell that ability, yet still expect their content to be indexed.)

You can test this theory by reconfiguring your browser to pretend to be Googlebot, with a string such as this:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Can you get in now? Of course you can, and without having to reveal private details or pay a subscription.

A variation of this scheme is to check that the HTTP header Referer contains a URI from Google, i.e., the visitor apparently found the content through a Google search.

If the webmaster really wants to control access to his content while still allowing indexing, he should stop wasting his time lying to the users of search engines, and instead consider splitting up articles so that the main bulk of each article is genuinely hidden behind proper security, while an abstract of the article is made public.