Class Robotex
In: lib/robotex.rb
Parent: Object

Methods

allowed?   delay   delay!   get_robots_txt   new   parse_host   timeout   timeout=  

Classes and Modules

Class Robotex::ParsedRobots

Constants

VERSION = '1.0.0'
DEFAULT_TIMEOUT = 3

Attributes

user_agent  [R] 

Public Class methods

Public Instance methods

Download the server‘s robots.txt, and return try if we are allowed to acces the url, false otherwise

Return the value of the Crawl-Delay directive, or nil if none

Sleep for the amount of time necessary to obey the Crawl-Delay specified by the server

[Validate]