rubygem-robotex - Ruby library to obey robots.txt
| Website: | http://www.github.com/chriskite/robotex |
|---|---|
| License: | MIT |
| Vendor: | Fedora Project |
- Description:
With one line of code, Robotex (pronounced like “robotics”) will download and parse the robots.txt file and let you know if your program is allowed to visit a given link.
Packages
| rubygem-robotex-1.0.0-12.el6.noarch [8 KiB] |
Changelog
by Miroslav Suchý (2013-03-18):
- use correct path (msuchy@redhat.com) |