How to track affiliate referals to your insurance website without problems

There are a number of reasons that tracking referrals to your website from third-party websites can be problematic if you attempt to do so with server-side code.

This is a technical article intended for insurance software developers but the problem described affects all of us who work with affiliates and resellers and attempt to track referrals server-side.

Let me break it down so you see the process, and the flaws.

Before I start, let me just make it clear: REFERRER and REFERER are alternative spellings of the same word, I think probably UK and US respectively. Since I’m in the UK right now I’ll be sticking to ‘referrer’ most of the time but forgive me if I lapse and switch 🙂

The tracking process we usually use to capture referrer is as follows:

  1. The affiliate (e.g. an aggregator like CompareTheMarket or MoneySupermarket, or an independent broker or authorised representative, or even a shopping affiliate like Froogle, Kelkoo, SiteRunner, Ciao) publishes a link with their ID and our ID on their website.
  2. A visitor clicks on the link with the intention to buy from your site.
  3. The link takes the visitor back to the site of the affiliate that logs the visit, referrer, etc. and sets a cookie in the visitor’s browser.
  4. The visitor is redirected to your site, where you log the new visitor as coming from the affiliate either by reading an ID in the querystring or by reading the *referrer value* (HTTP_REFERER) server side.
  5. The visitor proceeds to shop while we track them around the site with an id binding them to the original affiliate.
  6. The visitor either leaves or places an order, in which case we record the affiliate details in the order/quote/policy record.

How affiliates usually track us:

Since affiliates don’t trust their merchants to report this information back to them, there are two very similar methods they ask us to use to help them to capture more information:

  1. They give us a snippet of code to be embedded in the source code of the ‘Thank you’ page that connects to the affiliate site notifying them the details of the successful transaction.
  2. They give us an IMG tag to embed instead, linking to a server-side script on their site.

Note that this is not reliable, since we could easily randomize or fake the reporting of click-through. They are very much at our mercy, just as – in the other direction – we are at theirs, as you will see…

The industry-wide problem with all of this is:

There is absolutely no way to guarantee the referral data, for a number of reasons. In some respects HTTP itself is a flawed protocol in this context: namely the HTTP_REFERER server variable that we all use is not reliable – often empty or containing fraudulent data – and this server variable is the only way to capture the origin of a click event required to credit an affiliate referrer.

It can be empty because:

  1. Some affiliates use a meta-refresh to redirect people to the merchant site (a-la Kelkoo), allowing them to deliver an extra banner ad, but often ditching the all important referrer info.
  2. Some browsers don’t pass this variable at all.
  3. Some browsers discard this variable if the site is opened in a new window, or if you drag a link and drop it on another window, or use a desktop/start menu shortcut, bookmark, etc.
  4. Some people disable it manually, for privacy (usually outside of a single domain).
  5. Some firewalls block it (e.g. Norton, usually outside of a single domain).
  6. Many e-mail programs like Outlook filter it.
  7. Most download managers don’t send it.

Furthermore, in order to track the referrer as people move around the site it has to be stored in the session (usually the only solution for storing information about someone who hasn’t yet registered or logged-in, but a very volatile mechanism which can easily expire), or a cookie (which people often disable, firewalls block etc.).

Also, if people enter the domain name of the site manually or set a bookmark once they found us on the affiliate site and then return later, there is nothing to track or detect.

Are there any workarounds?

Unfortunately the entire system is based on trust between affiliate and merchant, and worse still, trust of the browser/software/network/pc of the visitor, all of which is unreliable, and data – however stored – can be faked, since there is not actual authentication between the affiliate and us.

However, there is a dim light at the end of the tunnel, if you squint and look hard enough…

  • Javascript, client based tracking like Google Analytics, Mint or BLVDStatus do a good job of telling you where visitors are coming from.  They’re not so good at showing clearly when visitors who originally came from a referrer return at a later date, because of the ‘last click wins behaviour‘.  To get around this when using Google Analytics, you need to override the last click wins behaviour.  You may also like to track all of your outgoing links.
  • Alternatively or in addition, affiliates could provide authentication by SSL or a similar mechanism to prove to us that they are who they say they are and visa-versa, making the whole process of delivering customers a secure transaction.  I’ve not seen anything like this in practice but I’m sure it goes on.  If anyone has any examples, let me know!
  • Often the only practical short-term solution is to create a different URL for each affiliate, which forces them to visit a different link to their counterparts, (e.g. http://www.yoursite.com/froogle ), but this can lead to resubmissions by fraudulent affiliates, since there is no way for us to stop it.

In my experience, use every method you can and compare the data coming from each approach which will help you highlight any grey areas or holes, and pinpoint fraudulent activity by any web partners you are working with.

As ever, your comments are very welcome!

As always, based on any comments received I will update this article so it remains authoritative.

2 comments

  1. I have just read your article with interest and would like your opinion advice on a recurring problem we have had over the years with several affiliate networking companies. We provide a service whereby we conduct an online search for our clients and arrange insurance on their behalf. We are FSA approved and authorised to transact our business in this way and this should give insurers peace of mind that by checking and validating customer details before arranging the insurance we are reducing the risk of fraud or error for them. The problem we have been having is when we highlight a tracking problem i.e. sales not validating (because we are arranging the insurance we are aware of all sales that go through our site) invariably the insurer gets taken off our website. We are aware that what the networking companies are doing is fraudulent and are in the position to expose this but so far have declined to do so. Obviously our business is suffering and there is no doubt that if our sales aren’t being credited neither are others, the only difference is we have the knowledge that others don’t. What would you advise??

  2. Hi Alex,

    Thanks for your comment. It’s a pretty little problem and exactly the sort of thing that’s hard to fix using standard tracking processes. The crux of the problem is the redirect from your site to another. If you could somehow capture the full details of the quote and then pass that securely, perhaps via an API or other mechanism which authenticates with the insurer, then you could no doubt avoid this problem altogether. You would keep the customer on your own site and feed the insurer all the information they need to quote and bind, all the while keeping the customer up-to-date on the current position.

    I’m wary of self promotion but genuinely feel you should take a look at another project we’re involved in, http://www.SchemeServe.com, which is software for running your own insurance quotation website which could potentially be used to create such a process.

    It would mean however significant onus on your own organisation to maintain the application forms and more formal relationships with the insurers but I imagine would mean a better service all round and certainly greater accountability for you.

    Best wishes,

    Adam

Leave a Reply