django-robots Robots exclusion application for Django, complementing Sitemaps.
Download

django-robots Ranking & Summary

django-robots Tags

Django manage robots.txt robots exclusion Django Sitemap Sitemap robots.txt

django-robots Description

Robots exclusion application for Django, complementing Sitemaps. django-robots is a basic Django application to manage robots.txt files following the robots exclusion protocol, complementing the Django Sitemap contrib app.The robots exclusion application consists of two database models which are tied together with a m2m relationship: * Rules * URLsInstallationGet the source from the application site at:http://code.google.com/p/django-robots/To install the sitemap app, follow these steps:1. Follow the instructions in the INSTALL.txt file2. Add 'robots' to your INSTALLED_APPS setting.3. Make sure 'django.template.loaders.app_directories.load_template_source' is in your TEMPLATE_LOADERS setting. It's in there by default, so you'll only need to change this if you've changed that setting.4. Make sure you've installed the sites framework.InitializationTo activate robots.txt generation on your Django site, add this line to your URLconf:(r'^robots.txt$', include('robots.urls')),This tells Django to build a robots.txt when a robot accesses /robots.txt. Then, please sync your database to create the necessary tables and create Rule objects in the admin interface or via the shell.RulesRule - defines an abstract rule which is used to respond to crawling web robots, using the robots exclusion protocol, a.k.a. robots.txt.You can link multiple URL pattern to allows or disallows the robot identified by its user agent to access the given URLs.The crawl delay field is supported by some search engines and defines the delay between successive crawler accesses in seconds. If the crawler rate is a problem for your server, you can set the delay up to 5 or 10 or a comfortable value for your server, but it's suggested to start with small values (0.5-1), and increase as needed to an acceptable value for your server. Larger delay values add more delay between successive crawl accesses and decrease the maximum crawl rate to your web server.The sites framework is used to enable multiple robots.txt per Django instance. If no rule exists it automatically allows every web robot access to every URL.Please have a look at the database of web robots for a full list of existing web robots user agent strings.URLsUrl - defines a case-sensitive and exact URL pattern which is used to allow or disallow the access for web robots. Case-sensitive.A missing trailing slash does also match files which start with the name of the given pattern, e.g., '/admin' matches /admin.html too.Some major search engines allow an asterisk (*) as a wildcard to match any sequence of characters and a dollar sign ($) to match the end of the URL, e.g., '/*.jpg$' can be used to match all jpeg files.CachingYou can optionally cache the generation of the robots.txt. Add or change the ROBOTS_CACHE_TIMEOUT setting with a value in seconds in your Django settings file:ROBOTS_CACHE_TIMEOUT = 60*60*24This tells Django to cache the robots.txt for 24 hours (86400 seconds). The default value is None (no caching). Requirements: · Django · Python

django-robots Related Software