arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Using different credential for different urls with the same crawler

rated by 0 users
Answered (Verified) This post has 1 verified answer | 1 Reply | 2 Followers

Top 100 Contributor
4 Posts
rakmagik posted on Mon, Jul 12 2010 8:17 PM

If I have 10 sites to crawl and each of then have a different username /pwd to access restricted content, how can I configure the crawl. The demo code creates one crawler and adds multiple crawl Requests for each url, but how do you managed the credentials for each url?

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

Look at the CredentialCache (a parameter in the Crawler constructor overload): http://arachnode.net/search/SearchResults.aspx?q=credentialcache

The CredentialCache is consumed in WebClient.cs.

-Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (2 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC