Filewatcher File Search File Search
Catalog
Content Search
» » » » »

libwww-robotrules-perl

database of robots.txt-derived permissions

WWW::RobotRules parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html>. Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site.

The parsed files are kept in a WWW::RobotRules object, and this object provides methods to check if access to a given URL is prohibited. The same WWW::RobotRules object can be used for one or more parsed /robots.txt files on any number of hosts.

Homepage:
Package version:6.01-1
Architecture:all
Distribution:Debian
Filename:libwww-robotrules-perl_6.01-1_all.deb

/usr/share/man/man3/WWW::RobotRules.3pm.gz

WWW::RobotRules(3User Contributed Perl DocumentaWWW::RobotRules(3pm)



NAME
       WWW::RobotRules - database of robots.txt-derived permissions

SYNOPSIS
        use WWW::RobotRules;
        my $rules = WWW::RobotRules->new('MOMspider/1.0');

        use LWP::Simple qw(get);

        {
          my $url = "http://some.place/robots.txt";
          my $robots_txt = get $url;
          $rules->parse
more»

/usr/share/man/man3/WWW::RobotRules::AnyDBM_File.3pm.gz

WWW::RobotRules::UserBContributed PWWW::RobotRules::AnyDBM_File(3pm)



NAME
       WWW::RobotRules::AnyDBM_File - Persistent RobotRules

SYNOPSIS
        require WWW::RobotRules::AnyDBM_File;
        require LWP::RobotUA;

        # Create a robot useragent that uses a diskcaching RobotRules
        my $rules = WWW::RobotRules::AnyDBM_File->new( 'my-robot/1.0', 'cachefile' );
        my $ua = WWW
more»

/usr/share/doc/libwww-robotrules-perl/changelog.Debian.gz

libwww-robotrules-perl (6.01-1) unstable; urgency=low

  * Initial Release (Closes: #618281).

 -- Nicholas Bamber <nicholas@periapt.co.uk>  Sun, 13 Mar 2011 23:20:24 +0000

/usr/share/doc/libwww-robotrules-perl/changelog.gz

_______________________________________________________________________________
2011-03-13 WWW-Robot
more»

/usr/share/doc/libwww-robotrules-perl/copyright

Format-Specification: http://svn.debian.org/wsvn/dep/web/deps/dep5.mdwn?op=file&rev=135
Maintainer: 
more»

Browse inside libwww-robotrules-perl_6.01-1_all.deb

         [DIR]DEBIAN/ (2)  65535+ mirrors
         [DIR]usr/ (1)  65535+ mirrors

Download libwww-robotrules-perl_6.01-1_all.deb

Results 1 - 1 of 1
Help - FTP Sites List - Software Dir.
Search over 15 billion files
© 1997-2016 FileWatcher.com