NWebCrawlerThis is a web crawler program written in C#. | |
Download |
NWebCrawler Ranking & Summary
Advertisement
NWebCrawler Tags
- grab grabber Java crawler Crawler web crawler data crawler Image Crawler LAN crawler share crawler file crawler website crawler link crawler internet crawler GDS crawler written email crawler Crawler framework crawler bar SiteMap Crawler Web Grabber thread count profile crawler crawler profile E-mail crawler email web crawler
NWebCrawler Description
The NWebCrawler application was developed to be a web crawler program written in C#. Main features: Configuable: thread count, waiting time, connection timeout, allow MIME types and priorities, download folders. Statstics information: URL count, total downloaded files, total downloaded bytes, CPU utility and available memory. Preferential crawler: user can set priority for MIME types (high, above, normal, below, low). Robust: 10+ URL normalization rules, crawler trap avoiding rules.
NWebCrawler Related Software