19 Apr 2021 05:47:32,346 INFO : Re-running LINCS pipeline for: LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43
19 Apr 2021 05:47:32,347 INFO : =======================================
19 Apr 2021 05:59:29,988 INFO : Starting to run task 'org.labkey.lincs.cromwell.CromwellGctTask' at location 'webserver-high-priority'
19 Apr 2021 05:59:29,988 INFO : Starting task to create L2 GCT for LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43.sky.zip
19 Apr 2021 05:59:29,994 INFO : Submitting job to http://m002.grid.gs.washington.edu:8000/api/workflows/v1
19 Apr 2021 05:59:30,018 INFO : Checking status of job at http://m002.grid.gs.washington.edu:8000/api/workflows/v1/99bec771-fec3-41ae-9184-8845712302ce/status
19 Apr 2021 05:59:50,033 INFO : Cromwell job status: Submitted
19 Apr 2021 06:00:10,049 INFO : Cromwell job status: Running
19 Apr 2021 06:00:50,089 INFO : Cromwell job status: Succeeded
19 Apr 2021 06:00:50,089 INFO : Cromwell job completed successfully
19 Apr 2021 06:00:50,091 INFO : Successfully completed task 'org.labkey.lincs.cromwell.CromwellGctTask'
19 Apr 2021 06:47:37,837 INFO : Starting to run task 'org.labkey.lincs.psp.LincsPspTask' at location 'webserver-high-priority'
19 Apr 2021 06:47:37,839 INFO : Running LINCS PSP pipeline for LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43
19 Apr 2021 06:47:37,839 INFO : Resubmitting job for runId: 30547, name: LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43
19 Apr 2021 06:47:37,839 INFO : Old job details:
19 Apr 2021 06:47:37,839 INFO : Id: 102
19 Apr 2021 06:47:37,839 INFO : Pipeline job Id: 68227
19 Apr 2021 06:47:37,839 INFO : Run Id: 30547
19 Apr 2021 06:47:37,839 INFO : PSP Job Id: 5cfd5720344c151e38c98da2
19 Apr 2021 06:47:37,839 INFO : PSP Job name: LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43
19 Apr 2021 06:47:37,839 INFO : PSP Job status: POUR : succeeded in uploading all GCTs to Panorama
19 Apr 2021 06:47:37,839 INFO : JSON: {"level 4":{"s3":{"url":"s3://proteomics.clue.io/psp/level4/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL4.gct"},"panorama":{"method":"PUT","url":"https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL4.gct"}},"assay":"GCP","created":"2019-06-09T18:59:44.412Z","name":"LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43","id":"5cfd5720344c151e38c98da2","last_modified":"2019-06-09T19:00:15.031Z","config":{"s3":{"url":"s3://proteomics.clue.io/psp/config/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43.cfg"},"panorama":{"method":"PUT","url":"https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43.cfg"}},"level 2":{"s3":{"url":"s3://proteomics.clue.io/psp/level2/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL2.gct"},"panorama":{"method":"GET","url":"https://panoramaweb.org/lincs/LINCS/GCP/runGCTReportApi.view?runId=30547&remote=true&reportName=GCT%20File%20GCP"}},"status":"POUR : succeeded in uploading all GCTs to Panorama","level 3":{"s3":{"url":"s3://proteomics.clue.io/psp/level3/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL3.gct"},"panorama":{"method":"PUT","url":"https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL3.gct"}}}
19 Apr 2021 06:47:37,839 INFO : Resubmitting...
19 Apr 2021 06:47:37,841 INFO : Looking for existing job with name LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43
19 Apr 2021 06:47:38,128 INFO : Job with name LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43exists.
19 Apr 2021 06:47:38,128 INFO : Looking for existing job with name LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_2
19 Apr 2021 06:47:38,202 INFO : Failed to complete task 'org.labkey.lincs.psp.LincsPspTask'
19 Apr 2021 06:47:38,206 ERROR: Error submitting PSP job
org.labkey.api.pipeline.PipelineJobException: Error submitting PSP job
at org.labkey.lincs.psp.LincsPspTask.postToPsp(LincsPspTask.java:96)
at org.labkey.lincs.psp.LincsPspTask.run(LincsPspTask.java:36)
at org.labkey.api.pipeline.PipelineJob.runActiveTask(PipelineJob.java:818)
at org.labkey.api.pipeline.PipelineJob.run(PipelineJob.java:1055)
at org.labkey.pipeline.mule.PipelineJobRunner.run(PipelineJobRunner.java:40)
at jdk.internal.reflect.GeneratedMethodAccessor1683.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.mule.impl.model.resolvers.DynamicEntryPoint.invokeMethod(DynamicEntryPoint.java:312)
at org.mule.impl.model.resolvers.DynamicEntryPoint.invoke(DynamicEntryPoint.java:259)
at org.mule.impl.DefaultLifecycleAdapter.intercept(DefaultLifecycleAdapter.java:193)
at org.mule.impl.InterceptorsInvoker.execute(InterceptorsInvoker.java:47)
at org.mule.impl.model.DefaultMuleProxy.run(DefaultMuleProxy.java:470)
at org.mule.impl.work.WorkerContext.run(WorkerContext.java:310)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1486)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:391)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:865)
at org.mule.impl.work.ScheduleWorkExecutor.doExecute(ScheduleWorkExecutor.java:39)
at org.mule.impl.work.MuleWorkManager.executeWork(MuleWorkManager.java:277)
at org.mule.impl.work.MuleWorkManager.scheduleWork(MuleWorkManager.java:244)
at org.mule.impl.model.seda.SedaComponent.run(SedaComponent.java:483)
at org.mule.impl.work.WorkerContext.run(WorkerContext.java:310)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.labkey.lincs.psp.LincsPspException: Error looking for job with name LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_2. URL: https://api.clue.io/api/psp?filter=%7B%22where%22%3A%7B%22name%22%3A%22LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_2%22%7D%7D
at org.labkey.lincs.psp.LincsPspUtil.jobExists(LincsPspUtil.java:251)
at org.labkey.lincs.psp.LincsPspUtil.getJobName(LincsPspUtil.java:201)
at org.labkey.lincs.psp.LincsPspTask.postToPsp(LincsPspTask.java:89)
... 24 more
Caused by: java.io.IOException: Server returned HTTP response code: 502 for URL: https://api.clue.io/api/psp?filter=%7B%22where%22%3A%7B%22name%22%3A%22LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_2%22%7D%7D
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1982)
at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1977)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:554)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1976)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1544)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1528)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
at org.labkey.lincs.psp.LincsPspUtil.jobExists(LincsPspUtil.java:229)
... 26 more
Caused by: java.io.IOException: Server returned HTTP response code: 502 for URL: https://api.clue.io/api/psp?filter=%7B%22where%22%3A%7B%22name%22%3A%22LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_2%22%7D%7D
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1932)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1528)
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308)
at org.labkey.lincs.psp.LincsPspUtil.jobExists(LincsPspUtil.java:227)
... 26 more
19 Apr 2021 08:41:13,136 INFO : Retrying job.
19 Apr 2021 08:41:13,137 DEBUG: Database indicates active task ID is org.labkey.lincs.psp.LincsPspTask
19 Apr 2021 08:41:13,137 DEBUG: Retry details: Old Job ID: 1bf6e506-833b-1039-a447-fea6ddf5a122
19 Apr 2021 08:41:41,673 INFO : Starting to run task 'org.labkey.lincs.psp.LincsPspTask' at location 'webserver-high-priority'
19 Apr 2021 08:41:41,673 INFO : Running LINCS PSP pipeline for LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43
19 Apr 2021 08:41:41,674 INFO : Resubmitting job for runId: 30547, name: LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43
19 Apr 2021 08:41:41,674 INFO : Old job details:
19 Apr 2021 08:41:41,674 INFO : Id: 102
19 Apr 2021 08:41:41,674 INFO : Pipeline job Id: 68227
19 Apr 2021 08:41:41,674 INFO : Run Id: 30547
19 Apr 2021 08:41:41,674 INFO : PSP Job Id: 5cfd5720344c151e38c98da2
19 Apr 2021 08:41:41,674 INFO : PSP Job name: LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43
19 Apr 2021 08:41:41,674 INFO : PSP Job status: POUR : succeeded in uploading all GCTs to Panorama
19 Apr 2021 08:41:41,674 INFO : JSON: {"level 4":{"s3":{"url":"s3://proteomics.clue.io/psp/level4/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL4.gct"},"panorama":{"method":"PUT","url":"https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL4.gct"}},"assay":"GCP","created":"2019-06-09T18:59:44.412Z","name":"LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43","id":"5cfd5720344c151e38c98da2","last_modified":"2019-06-09T19:00:15.031Z","config":{"s3":{"url":"s3://proteomics.clue.io/psp/config/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43.cfg"},"panorama":{"method":"PUT","url":"https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43.cfg"}},"level 2":{"s3":{"url":"s3://proteomics.clue.io/psp/level2/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL2.gct"},"panorama":{"method":"GET","url":"https://panoramaweb.org/lincs/LINCS/GCP/runGCTReportApi.view?runId=30547&remote=true&reportName=GCT%20File%20GCP"}},"status":"POUR : succeeded in uploading all GCTs to Panorama","level 3":{"s3":{"url":"s3://proteomics.clue.io/psp/level3/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL3.gct"},"panorama":{"method":"PUT","url":"https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL3.gct"}}}
19 Apr 2021 08:41:41,674 INFO : Resubmitting...
19 Apr 2021 08:41:41,675 INFO : Looking for existing job with name LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43
19 Apr 2021 08:41:41,996 INFO : Job with name LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43exists.
19 Apr 2021 08:41:41,996 INFO : Looking for existing job with name LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_2
19 Apr 2021 08:41:42,156 INFO : Job with name LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_2exists.
19 Apr 2021 08:41:42,156 INFO : Looking for existing job with name LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_3
19 Apr 2021 08:41:42,293 INFO : PSP job name: LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_3
19 Apr 2021 08:41:42,293 INFO : Sending JSON:
19 Apr 2021 08:41:42,293 INFO : {
"level 4": {"panorama": {
"method": "PUT",
"url": "https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL4.gct"
}},
"assay": "GCP",
"name": "LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_3",
"config": {"panorama": {
"method": "PUT",
"url": "https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43.cfg"
}},
"level 2": {"panorama": {
"method": "GET",
"url": "https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43.gct"
}},
"level 3": {"panorama": {
"method": "PUT",
"url": "https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL3.gct"
}}
}
19 Apr 2021 08:41:42,301 INFO : Sending POST request to https://api.clue.io/api/psp
19 Apr 2021 08:41:42,542 INFO : Response code - 200
19 Apr 2021 08:41:42,542 INFO : Response from server: {"name":"LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_3","assay":"GCP","status":"Waiting_To_Download","id":"607da4b6e1c96f0017d8b797","created":"2021-04-19T15:41:42.380Z","last_modified":"2021-04-19T15:41:42.380Z","level 2":{"panorama":{"method":"GET","url":"https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43.gct"}},"level 3":{"panorama":{"method":"PUT","url":"https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL3.gct"}},"level 4":{"panorama":{"method":"PUT","url":"https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43_LVL4.gct"}},"config":{"panorama":{"method":"PUT","url":"https://panoramaweb.org/_webdav/LINCS/GCP/%40files/GCT/LINCS_GCP_Plate60_annotated_minimized_2018-04-20_14-36-43.cfg"}}}
19 Apr 2021 08:41:42,543 INFO : Sent POST request to https://api.clue.io/api/psp
19 Apr 2021 08:41:42,545 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:41:42,680 INFO : Status: Waiting_To_Download
19 Apr 2021 08:42:02,684 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:42:02,971 INFO : Status: created LVL2 GCT
19 Apr 2021 08:42:22,974 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:42:23,262 INFO : Status: created LVL2 GCT
19 Apr 2021 08:42:43,266 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:42:43,560 INFO : Status: created LVL2 GCT
19 Apr 2021 08:43:03,563 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:43:03,934 INFO : Status: created LVL2 GCT
19 Apr 2021 08:43:23,935 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:43:24,228 INFO : Status: created LVL2 GCT
19 Apr 2021 08:43:44,234 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:43:44,522 INFO : Status: created LVL2 GCT
19 Apr 2021 08:44:04,525 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:44:04,815 INFO : Status: created LVL2 GCT
19 Apr 2021 08:44:24,822 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:44:25,114 INFO : Status: created LVL2 GCT
19 Apr 2021 08:44:45,116 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:44:45,406 INFO : Status: created LVL2 GCT
19 Apr 2021 08:45:05,408 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:45:05,751 INFO : Status: created LVL2 GCT
19 Apr 2021 08:45:25,753 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:45:26,038 INFO : Status: created LVL2 GCT
19 Apr 2021 08:45:46,039 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:45:46,411 INFO : Status: created LVL 3 GCT
19 Apr 2021 08:46:06,414 INFO : Checking status of job 607da4b6e1c96f0017d8b797
19 Apr 2021 08:46:06,709 INFO : Status: POUR : succeeded in uploading all GCTs to Panorama
19 Apr 2021 08:46:06,709 INFO : PSP job completed with status: POUR : succeeded in uploading all GCTs to Panorama
19 Apr 2021 08:46:06,709 INFO : Finished running LINCS PSP pipeline.
19 Apr 2021 08:46:06,709 INFO : Successfully completed task 'org.labkey.lincs.psp.LincsPspTask'