A crawler or spider is an intelligent agent that gathers information on the Internet without a user's intervention. Crawlers and spiders are e.g. used by search engines to index the internet.
