Link rot is the bane of directory editors. If you run a niche directory site, such as I do, one of the things you want to make sure of is that your links are current ie. that they do not give 404s or other errors.
There is no point in fooling yourself into thinking that you can take on the likes of Google, but you can offer good resources for your chosen subject.
The backend of Browse.ie is powered by Gossamer Threads Links SQL (now known as Glinks) which has a link checker builtin. Unfortunately, as I discovered, the UserAgent string it was reporting was generic and had been blocked by some website owners.
With the help of Niall O’Broin I finally fixed the UserAgent to be something a little more descriptive.
If you see the following string in your logs, then you’ll know that it has visited your site:
"Browse.ie Link Checker - See http://www.browse.ie"
An earlier version reported itself as:
"Browse-ie Link Checker - See http://www.browse.ie"
The robot will simply check that the link is valid. Nothing more. Nothing less.
As long as it gets a valid response, such as a 200, it will mark your link as being “ok”.
I’m not sure how often the robot will do the check, but I expect to run it once a week or so to ensure that there are no dead links ie. avoid link rot
Avoiding Link Rot – Browse.ie link checker
