arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Crawling of webpage where data comes with scrollbar

rated by 0 users
Not Answered This post has 0 verified answers | 1 Reply | 1 Follower

Top 75 Contributor
Male
5 Posts
Abhishek Gahlout posted on Thu, Apr 11 2013 2:23 AM

Hi , I need to know , how should I crawl a webpage where the information is loading when page scrolls. eg: please visit the link

http://www.wikido.com/us/ny/ny

Though there are thousand of records for this page but at a time only 60 records are shown. further records are shown when we scroll the vertical scroll bar.Crawler only fetches data foar 60 records at a time as when page first loads it only shows 60 records. So to crawl and get complete information , how should I crawl these kind of webpages.

All Replies

Top 10 Contributor
1,905 Posts

As you solved it: Check Fiddler for the requested AbsoluteUri - in this case additional AbsoluteUris are generated with QueryStrings, which makes it VERY easy.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (2 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC