Bug Report Update : Redirect functionality...

Reply

arachnode.net Posted: Thu, May 5 2011 7:28 PM

rated by 0 users

I recently had a bug come into my email that discussed the following:

http://home.nzcity.co.nz/go.aspx?u=http://www.nzdating.com/

It then goes to http://www.NZDating.com & finds links like http://www.nzdating.com/members/wo.aspx 

And appears to try to index

http://home.nzcity.co.nz/members/wo.aspx (this link belongs to nzdating.com - /members/wo.aspx)

which should actually be

http://www.nzdating.com/members/wo.aspx

I made a change to DataManager.cs and to DiscoveryManager.cs that will check for redirects and automatically update the crawlRequest.Discovery to the crawlRequest.WebClient.HttpWebResponse.ResponseUri if the crawlRequest.WebClient.HttpWebResponse.StatusCode is 300, 301, 302 or 307.

If your application needs to track the submitted AbsoluteUri, check the crawlRequest.WebClient.HttpWebRequest.RequestUri property.

(in DiscoveryManager.cs)