WWW::Robot A configurable web traversal engine
Download

WWW::Robot Ranking & Summary

WWW::Robot Tags

web perl Robots web robots web traversal engine web agents

WWW::Robot Description

A configurable web traversal engine WWW::Robot is a configurable web traversal engine (for web robots & agents).SYNOPSIS use WWW::Robot; $robot = new WWW::Robot( 'NAME' => 'MyRobot', 'VERSION' => '1.000', 'EMAIL' => 'fred@foobar.com' ); # ... configure the robot's operation ... $robot->run( 'http://www.foobar.com/' );This module implements a configurable web traversal engine, for a robot or other web agent. Given an initial web page (URL), the Robot will get the contents of that page, and extract all links on the page, adding them to a list of URLs to visit.Features of the Robot module include: * Follows the Robot Exclusion Protocol. * Supports the META element proposed extensions to the Protocol. * Implements many of the Guidelines for Robot Writers. * Configurable. * Builds on standard Perl 5 modules for WWW, HTTP, HTML, etc.A particular application (robot instance) has to configure the engine using hooks, which are perl functions invoked by the Robot engine at specific points in the control loop.The robot engine obeys the Robot Exclusion protocol, as well as a proposed addition. See "SEE ALSO" for references to documents describing the Robot Exclusion protocol and web robots. Requirements: · Perl

WWW::Robot Related Software