Crawling Google Docs secure website

lizatt posted on Tue, Mar 8 2011 1:19 PM

rated by 0 users

Tried crawling Google Docs using my credentials. Get following error:

System.Net.WebException was caught
  Message=The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
  Source=System
  StackTrace:
       at System.Net.HttpWebRequest.GetResponse()
       at Arachnode.Next.Crawler`1.CrawlThread(Object o) in E:\dev331c\ad\ediscovery\Connectors\WebCrawler\arachnode\trunk\Next\Crawler.cs:line 466
  InnerException: System.Security.Authentication.AuthenticationException
       Message=The remote certificate is invalid according to the validation procedure.
       Source=System
       StackTrace:
            at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
            at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
            at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
            at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
            at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
            at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
            at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
            at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
            at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
            at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
            at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
            at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
            at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
            at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
            at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
            at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
            at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
            at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
            at System.Net.ConnectStream.WriteHeaders(Boolean async)
       InnerException:

I am setting the credentials at around line 437 of Crawler.cs with the following:

httpWebRequest.Credentials = new NetworkCredential("username", "password", "");
crawlRequest.HttpWebRequest = httpWebRequest;