Boitho distributed crawler (BETA).
This is a distributed crawler, where everybody can
donate there superfluous computer resources and spare bandwidth,
to help us create a bigger and better search engine.
Our goal is to make a general internet search engine
with a thumbnail picture of all the pages. The problem is that the
bandwidth and computer resources needed to make a thumbnail of a
internet page is several times the resources needed just to download
a HTML page. This means that Boitho has to spend more resources
on crawling than other search engines.
To make the most of our available resources, and to
allow volunteers to donate their superfluous bandwidth and idle
CPU time, we have developed a distributed crawler for Boitho, like
seti@home and Grub. That way people can install a program on their
computers and help us with the crawling.
If you have any questions you can contact Runar Buvik,
e-mail: runarb (at) boitho (dot) com, phone: +47.93030627
How to install
If you are running an
old version of the crawler, you will have to close it, and then
wait until the taskbar icon disappears.
Download and run the file BoithoCrawler.exe
Follow the instructions on the screen
Start the crawler by clicking on the start-menu
-> All programs -> BoithoCrawler -> Start Boitho Crawler
If you want to you can create an account. That way
you get bether statistics over how many URLs an robots.txt you have
crawled. Go to Tools -> Options -> User Info -> Create
Minimum system requirements
Operating system: Windows XP, Windows 2000 or Windows
Hard disk: 1 MB for the program, and 30 Mb for cachefiles.
Blocking us from accessing you site
You may block access by create/modify you robots.txt file. Please refer to the crawlers homepage for blocking instructions.
The bandwidth meter does not always work properly if you
have two or more network cards.