Hi, i'm trying to crawl some sites using a disallowed words , but the sites are stored, the words exists into disallowedwords table
Or, i need activate some else?
Thanks for your support
Thisis correct - the Source.cs Plugin works with the cfg.DisallowedWords database table.
You can always clear this table out, or, use your own plugin to filter as you see fit.
if(crawlRequest.DecodedHtml.Contains("thisforbiddenword"))
{
crawlRequest.IsDisallowed = true;
}
Definitely look at AbsoluteUri.cs and Source.cs
Thanks!Mike
For best service when you require assistance:
Skype: arachnodedotnet
