Recurring Error pinging Panorama server Roman Sakson  2020-05-04 01:04:52
 

Dear Panorama Team,

we seem to have recurring troubles pinging the Panorama server (it started already before Easter and goes on). Remarkably, it seems to work every now and then, so that we can upload our QC files, however, delays do occur and I would like to understand where this error comes from (to my knowledge, we did not change anything regarding our internet connection). A typical error message looks like this (could go on for hours or even days):

[5/4/2020 5:20:42 AM] ERROR: [5/4/2020 5:20:42 AM] Error pinging Panorama server https://panoramaweb.org/
System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 128.95.55.201:443
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.WebClient.UploadValues(Uri address, String method, NameValueCollection data)
at AutoQC.WebClientWithCredentials.Post(Uri uri, NameValueCollection postData)
at AutoQC.WebPanoramaClient.PingPanorama(String folderPath, String username, String password)
at AutoQC.PanoramaPinger.PingPanoramaServer()

Thank you very much in advance!

Roman

 
 
Vagisha Sharma responded:  2020-05-04 06:58:27

Hi Roman,

Thank you for posting to the support board. The error stack trace indicates that the problem is due to the server not responding in time. If this is happening very frequently then we should investigate. Could you please attach your AutoQC.log file? You will find it in the same folder on your computer as your Skyline document.

Thanks,
Vagisha

 
Roman Sakson responded:  2020-05-04 07:38:22

Hi Vagisha,

sure, I attached the log file.

Is there a way to notify several accounts or emails when there is a respond to a particular thread? I will receive an email when you answer but there is a colleague who is also working on the system, could she somehow be added to the thread followers?

Thank you!

Roman

 
jeckels responded:  2020-05-04 09:53:42

Hi Roman,

Please ask your colleague to go to this message thread and be sure that she's logged in

https://panoramaweb.org/announcements/home/support/thread.view?rowId=4512

and then click the subscribe link in the upper right to get email notifications. She can either subscribe to just this one thread, or the whole forum.

https://www.labkey.org/Documentation/wiki-page.view?name=createMessage#subscribe

Thanks,
Josh

 
Roman Sakson responded:  2020-05-04 11:41:42

Hi Josh,

great, this is exactly what I was looking for, thank you!

Roman

 
Roman Sakson responded:  2020-05-11 16:44:58

Dear Panorama team,

I just wanted to ask whether the AutoQC log file I provided was of any help? Unfortunately, we still have those issues with pinging the Panorama server...

Best regards,

Roman

 
Vagisha Sharma responded:  2020-05-12 19:52:09

Hi Roman,

Thanks for sharing the log. We've tried to correlate the errors ("Error pinging Panorama server..") in your log with our server logs. But we weren't able to find requests in the server logs that corresponded with the ping errors in your log. So it appears that the server did not receive the requests. I pulled up the AutoQC Loader log from one of the instrument computers in our lab and did not find the volume of errors that I see in your log. There were some errors but they happened on the dates that we were doing server maintenance. So I don't know why you are seeing so many errors.

The AutoQC ping action is what drives the colored AutoQC status indicator in the top right corner of the "QC Summary" panel. It is meant to help users keep track of whether AutoQC Loader is running or not when viewing their data in Panorama. Even if it fails every now and then it will not interfere with the main task of uploading Skyline documents to Panorama. In your log I also saw a couple of Skyline document upload failures that occurred at the same time as AutoQC ping failed. If you keep AutoQC Loader running and new raw files are added to the folder, eventually the document will be uploaded but there will be delays as you noted in your initial post. So we need to make AutoQC Loader handle intermittent network interruptions better. I have added this to our issues list: https://skyline.ms/issues/home/issues/details.view?issueId=728.

Thanks,
Vagisha

 
Roman Sakson responded:  2020-05-14 11:10:19

Hi Vagisha,

thank you for your reply and also for opening the issue, I think that this would make AutoQC even better than it is already, indeed. At the moment it seems to run more stably, by the way.

Thank you,

Roman