The Sparo Linkchecker User Agent
Information for website admins who have spotted the Sparo Link Checker user agent in their site logs.The Sparo Linkchecker bot is an entirely harmless bot. It checks that links stored within a Sparo CMS website (such as this one) are not broken.
At the moment it is a fairly basic robot, simply checking that an attempt to download a linked URL does not result in the downloader function timing out, throwing an exception or receiving a non-successful HTML return code. In future I may borrow a few features from other projects and incorporate malware checking and more sophisticated error sensing.
I usually take a super-considerate approach with this sort of program. However, this bot will take no notice of robots.txt directives (or similar metadata, etc). This is by design - it has to be able to check access to any link a human could access, otherwise it wouldn't be doing its job.
It also moves at a faster pace than other downloaders and robots I've made, so if a Sparo website happens to contain a lot of links to one domain, then that domain may receive a lot of requests in quick succession during a checking process. However, I haven't completely forgotten my manners - the download speed should be well within the capabilities of almost any webserver.
In the unlikely event that the SLC bot causes your website any problems, please contact me with as much information as possible to help me pinpoint the source of the problem so that I can fix it more quickly.
