arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release
Key Principles in Software Development

Ever since I updated http://arachnode.net to CommunityServer 2008.5, I feel prolific, and more importantly, proficient when it comes to writing blog posts.  It's likely true that CommunityServer 2008.5 has inspired these changes in the willingness to write and share, and it is perhaps a little less likely that my other observations are true.  Nonetheless...

In developing arachnode.net I did my best to adhere to several key principles, the principles of which I will now pull off the top of my head.  :)

Code for the average man/woman

  • Try and keep your code accessible to as many developer skill levels as possible.  While it won't be possible to write a search engine that is instantly comprehended by those recently introduced to casting, it should be possible to write a web crawler that those who can cast can easily understand.  Coding for the average man/woman means, among other things, coding transparently, naming objects with descriptive names, even if a little too long, avoiding endless inheritance, consolidating configuration settings and commenting your code.

Avoid optimizing early and over-optimization

  • It's fun to optimize code and reap the egotistical benefits of doing so, until you find yourself having to re-work the code you just optimized, thereby rendering your optimizations obsolete or nearly so.  Before you start throwing out words like 'distributed', 'balanced', or my personal un-favorite 'enterprise', ensure your code works flawlessly on one machine for the task it's supposed to do.

...perhaps someday I will finish this post...


Posted Thu, Jan 1 2009 8:01 PM by arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC