Apache Nutch is an extensible and scalable web crawler
Support for crawling and searching Creative-Commons licensed content.