I recently had a bug come into my email that discussed the following:
http://home.nzcity.co.nz/go.aspx?u=http://www.nzdating.com/
It then goes to http://www.NZDating.com & finds links like http://www.nzdating.com/members/wo.aspx
And appears to try to index
http://home.nzcity.co.nz/members/wo.aspx (this link belongs to nzdating.com - /members/wo.aspx)
which should actually be
http://www.nzdating.com/members/wo.aspx
I made a change to DataManager.cs and to DiscoveryManager.cs that will check for redirects and automatically update the crawlRequest.Discovery to the crawlRequest.WebClient.HttpWebResponse.ResponseUri if the crawlRequest.WebClient.HttpWebResponse.StatusCode is 300, 301, 302 or 307.
If your application needs to track the submitted AbsoluteUri, check the crawlRequest.WebClient.HttpWebRequest.RequestUri property.
(in DiscoveryManager.cs)
For best service when you require assistance:
Skype: arachnodedotnet