arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Difference between AbsoluteUri1 and AbsoluteUri2

rated by 0 users
Answered (Verified) This post has 1 verified answer | 7 Replies | 1 Follower

posted on Mon, Nov 30 2009 1:32 AM

Hello, 

Anybody can tell me what is a difference between AbsoluteUri1 and AbsoluteUri2. I tried to add some url by stored procedure, sometimes i have constraint exceptions.

Best regards

k1

 

 

 

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

1 is the parent.  2 is the child.

1 and 2 need to match if submitting a request to crawl a site, rather than to crawl an image.

1 and 2 are named as such to encourage you to use the API.  (You really should be using the API as the Engine uses this table and you may inadvertantly interfere with crawling...)

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

All Replies

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

1 is the parent.  2 is the child.

1 and 2 need to match if submitting a request to crawl a site, rather than to crawl an image.

1 and 2 are named as such to encourage you to use the API.  (You really should be using the API as the Engine uses this table and you may inadvertantly interfere with crawling...)

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

replied on Mon, Nov 30 2009 8:38 AM

Hello,

When i'm using api i have a lot of trouble with updating table while crawling.  Can u give me example of parent / child - for domain http://sports.yahoo.com/golf ?

 

Btw. great job :)

Top 10 Contributor
1,905 Posts

I'm not sure what you mena by having trouble while crawling?

http://sports.yahoo.com/golf, http://sports.yahoo.com/golf if you want to submit a request for this site.

http://sports.yahoo.com/golf, http://sports.yahoo.com/golf if you want to submit a request for this image./club.gif

-Mike

Thank you!  Version 1.4 is very, very close.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

replied on Tue, Dec 1 2009 1:05 AM

Hello,

I having a trouble with create new CrawlRequests while other crawl requests are in progress. 

 

Best regards

 

Top 10 Contributor
1,905 Posts

What is the error?  What is the difficulty?

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

replied on Tue, Dec 1 2009 7:17 AM

Stackoverflow, in my opinion problem exists in DataSets. There are some conflicts (maybe deadlock i dont know)

Top 10 Contributor
1,905 Posts

Please post your code.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (8 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC