Issues: Redundancy

• Many spiders are performing essentially the same tasks.
• There is no way to ask an HTTP server "what has changed?"
• Complete detection of identical documents is difficult (symbolic links, slightly altered versions, etc.).