2026-06-05 10:25:22,980 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:25:22,980 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:25:22,980 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:25:22,980 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:25:22,980 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:25:22,980 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:25:22,979 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:25:22,979 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:25:22,979 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:25:22,979 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:22:30,103 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:22:30,103 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:22:30,103 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:22:30,103 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:22:30,103 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:22:30,102 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:18:43,594 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37854 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:18:43,594 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59162 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:18:43,594 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33400 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:18:43,594 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:53596 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:18:43,594 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35956 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:18:43,594 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55590 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:18:43,594 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58050 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:18:43,594 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45494 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:18:43,594 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45592 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:18:43,593 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:38490 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:18:43,593 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54130 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:18:43,593 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50126 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:18:43,593 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59122 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:18:43,593 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59580 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:18:43,593 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:49292 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:18:43,593 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50206 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55568 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37824 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59146 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33376 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:53574 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35940 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55564 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58036 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45466 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45580 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:38484 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54110 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:18:43,592 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50120 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59104 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59556 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:49286 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50192 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37812 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59142 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33370 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:53562 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35926 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55550 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58030 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45456 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45564 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:18:43,591 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:38470 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54094 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50112 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59092 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59552 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:49276 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50178 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37800 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59140 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33356 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:53558 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35920 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55548 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:18:43,590 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58026 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45442 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45562 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:38462 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50098 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59080 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59536 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:49260 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50166 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37786 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59124 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33348 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:53552 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55546 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:18:43,589 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58012 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45432 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45546 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:38456 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59524 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:49248 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50164 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37772 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59120 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33332 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55542 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57996 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45416 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:18:43,588 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:38446 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:18:43,587 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59510 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:18:43,587 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50150 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:16:42,754 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:16:42,754 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:16:42,754 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:16:42,754 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:16:42,754 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:16:42,754 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:16:42,754 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:16:42,754 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:16:42,754 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:16:42,754 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:16:42,754 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:16:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:16:42,752 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:16:42,751 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:16:42,751 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:16:42,751 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:16:42,393 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:40732 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:16:42,393 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58078 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:16:42,393 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52484 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:16:42,393 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50960 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:16:42,393 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43458 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:16:42,393 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37770 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:16:42,393 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:56086 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:16:42,393 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58136 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:16:42,393 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:40250 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:16:42,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55098 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:16:42,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44778 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:16:42,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:42798 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:16:42,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34638 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:16:42,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44364 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:16:42,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35322 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:16:42,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39816 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:40714 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58064 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52462 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50948 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43432 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37756 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:56056 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58128 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:40234 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55088 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44754 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:42782 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:16:42,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34630 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:16:42,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44352 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:16:42,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35314 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:16:42,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39802 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:16:42,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:40696 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:16:42,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58046 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:16:42,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52454 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50930 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43408 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37732 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:56040 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58112 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:40220 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55078 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44742 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:42772 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34618 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44338 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35294 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39780 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:16:42,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:40692 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58036 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52452 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50922 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43404 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37724 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:56034 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58106 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:40218 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55070 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44726 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:42768 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34612 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44330 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:16:42,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35292 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39766 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:40680 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58030 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52448 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50916 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43390 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37714 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:56032 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58100 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:40214 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55068 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44722 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:42752 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:16:42,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34600 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44322 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35286 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39752 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:40664 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58018 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52446 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50912 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43388 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37710 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:56016 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58096 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:40198 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55056 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44714 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:16:42,386 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:42748 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:16:42,385 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34590 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:16:42,385 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44310 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:16:42,385 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35270 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:16:42,385 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39744 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:16:30,660 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:16:30,659 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:16:30,658 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:16:30,657 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:16:30,657 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:16:30,657 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:16:30,657 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:16:30,657 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:16:30,657 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:16:30,657 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:16:30,657 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:16:30,657 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:15:47,324 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:60998 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:15:47,324 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58012 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:15:47,324 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43502 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:15:47,324 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:42202 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:15:47,324 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48112 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:15:47,324 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:49058 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:15:47,324 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41236 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:15:47,323 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45810 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:15:47,323 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44060 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:15:47,323 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54704 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:15:47,323 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52428 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:15:47,323 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45502 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:15:47,323 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43646 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:15:47,323 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54348 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:15:47,323 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35422 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:15:47,323 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52900 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:15:47,322 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:60994 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:15:47,322 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57998 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:15:47,322 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43486 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:15:47,322 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:42172 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:15:47,322 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48102 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:15:47,322 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:49036 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:15:47,322 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41212 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:15:47,322 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45788 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:15:47,322 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44032 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:15:47,322 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54686 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:15:47,321 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52416 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:15:47,321 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45494 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:15:47,321 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43614 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:15:47,321 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54336 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:15:47,321 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35404 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:15:47,321 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52890 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:15:47,321 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:42156 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:15:47,320 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:49012 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:15:47,320 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45776 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:15:47,320 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44016 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:15:47,320 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52412 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:15:47,320 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45486 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:15:47,320 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35382 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:15:47,320 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48994 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:15:47,318 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:15:47,318 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:15:47,318 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:15:47,318 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out trying to connect to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:15:47,318 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out trying to connect to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:15:47,318 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:15:47,318 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:15:47,318 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:15:47,318 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:15:47,318 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out trying to connect to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out trying to connect to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out trying to connect to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out trying to connect to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out trying to connect to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out trying to connect to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out trying to connect to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out trying to connect to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out trying to connect to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out trying to connect to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out trying to connect to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:15:47,317 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out trying to connect to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out trying to connect to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out trying to connect to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:15:47,316 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:15:47,315 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out trying to connect to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:15:17,289 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:15:16,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:15:16,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:15:16,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:15:16,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out trying to connect to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:15:16,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out trying to connect to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:15:16,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:15:16,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out trying to connect to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out trying to connect to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out trying to connect to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out trying to connect to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out trying to connect to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out trying to connect to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out trying to connect to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:15:16,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out trying to connect to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out trying to connect to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out trying to connect to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out trying to connect to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out trying to connect to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out trying to connect to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out trying to connect to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:15:16,390 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out trying to connect to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out trying to connect to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out trying to connect to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out trying to connect to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out trying to connect to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out trying to connect to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:15:16,389 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out trying to connect to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:15:16,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out trying to connect to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:15:16,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:15:16,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:15:16,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:14:55,167 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:14:55,166 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:14:55,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:14:55,164 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:14:07,478 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out trying to connect to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:14:07,478 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:14:07,478 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:14:07,478 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out trying to connect to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:14:07,478 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:14:06,938 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:14:06,938 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:14:06,938 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:14:06,938 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:14:06,938 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:14:06,938 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:14:06,937 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:14:06,937 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:14:06,919 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:14:06,919 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:14:06,919 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:14:06,919 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:14:06,919 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:14:06,918 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:14:06,918 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:14:06,918 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:14:06,918 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:14:06,918 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:14:06,918 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:14:06,918 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:14:06,918 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:14:06,918 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:14:06,918 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:12:23,286 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:12:23,286 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:12:23,286 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:12:23,286 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:12:23,286 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:12:23,286 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:10:19,620 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:10:19,620 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:10:19,620 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:09:03,060 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:47294 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:09:03,060 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33642 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:09:03,060 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41770 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:09:03,060 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58586 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:09:03,060 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43550 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:09:03,060 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35024 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:09:03,060 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44728 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:09:03,060 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59352 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:09:03,059 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:42542 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:09:03,059 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:49960 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:09:03,059 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:36468 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:09:03,059 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:51550 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:09:03,059 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39508 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:09:03,059 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46496 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:09:03,059 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44348 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:09:03,059 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58876 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:47280 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33616 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41742 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58558 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43532 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35002 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44710 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59336 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:42516 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:49954 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:36436 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:51530 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:09:03,058 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39488 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:09:03,057 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46490 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:09:03,057 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44338 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:09:03,057 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58872 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:09:03,056 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:47254 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:09:03,056 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33604 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:09:03,056 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:43520 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:09:03,056 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34982 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:09:03,056 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44672 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:09:03,056 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59314 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:09:03,056 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:36414 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:09:03,056 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:51508 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:09:03,056 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:44318 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:07:44,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34304 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:07:44,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41346 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:07:44,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35470 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:07:44,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:51790 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:07:44,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59512 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:07:44,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33582 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:07:44,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35064 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:07:44,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50322 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:07:44,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:56366 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:07:44,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:47322 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:07:44,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46422 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:07:44,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41460 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:07:44,734 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48340 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:07:44,734 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46274 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:07:44,734 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37434 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:07:44,734 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34290 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:07:44,734 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41336 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:07:44,734 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35448 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:07:44,734 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59488 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:07:44,734 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33570 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:07:44,733 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35040 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:07:44,733 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50296 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:07:44,733 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:56342 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:07:44,733 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:47312 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:07:44,733 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46404 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:07:44,733 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41446 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:07:44,733 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48324 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:07:44,733 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37422 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:07:44,732 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34284 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:07:44,732 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41324 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:07:44,732 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35426 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:07:44,732 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:51774 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:07:44,732 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59474 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:07:44,732 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33548 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:07:44,732 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35008 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:07:44,732 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50282 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:07:44,732 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:56332 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:07:44,732 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:47290 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:07:44,732 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46384 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:07:44,732 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41432 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48300 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46254 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37404 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41470 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34268 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41310 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35418 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:51764 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59460 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:33534 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34998 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50276 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 10:07:44,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:56320 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 10:07:44,730 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:47286 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 10:07:44,730 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46374 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 10:07:44,730 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41426 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 10:07:44,730 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48294 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 10:07:44,730 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46238 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 10:07:44,730 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:37400 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 10:07:44,730 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41458 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:07:34,258 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:07:34,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:07:34,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:07:34,255 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:07:34,254 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 10:07:34,253 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:07:34,252 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:07:34,252 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 10:07:34,252 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 10:07:34,252 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 10:07:34,252 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 10:07:34,252 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 10:07:34,252 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 10:06:45,482 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 10:06:45,482 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 10:06:45,481 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 10:06:45,481 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:58:45,700 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52604 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 09:58:45,700 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:49006 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39250 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59938 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52056 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54614 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57220 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52624 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41506 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54414 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58100 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41186 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46490 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55050 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 09:58:45,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55834 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 09:58:45,698 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50896 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 09:58:45,698 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52588 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 09:58:45,698 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48996 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 09:58:45,698 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39238 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 09:58:45,698 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59920 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 09:58:45,698 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52032 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 09:58:45,698 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54590 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 09:58:45,697 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57214 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 09:58:45,697 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52602 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 09:58:45,697 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41478 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 09:58:45,697 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54400 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 09:58:45,697 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58080 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 09:58:45,697 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41160 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 09:58:45,697 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46480 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 09:58:45,697 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55028 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 09:58:45,697 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55806 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 09:58:45,697 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50876 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 09:58:45,696 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52558 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 09:58:45,696 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48980 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 09:58:45,696 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39222 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 09:58:45,696 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59900 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 09:58:45,696 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52022 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 09:58:45,696 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54576 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 09:58:45,696 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57182 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 09:58:45,696 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52592 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 09:58:45,696 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41468 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 09:58:45,696 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54388 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 09:58:45,696 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58062 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 09:58:45,695 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41148 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 09:58:45,695 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46464 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 09:58:45,695 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55006 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 09:58:45,695 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55780 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 09:58:45,695 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50850 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 09:58:45,694 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52542 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 09:58:45,694 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48972 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 09:58:45,694 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39204 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 09:58:45,694 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59886 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 09:58:45,694 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52004 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 09:58:45,694 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54554 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 09:58:45,694 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57156 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 09:58:45,694 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52570 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 09:58:45,694 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41450 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 09:58:45,694 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54370 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 09:58:45,694 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58048 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 09:58:45,694 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41122 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 09:58:45,693 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46440 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 09:58:45,693 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54998 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 09:58:45,693 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55762 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 09:58:45,693 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50834 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 09:58:45,691 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52480 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 09:58:45,691 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48904 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 09:58:45,691 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39180 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 09:58:45,691 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59856 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 09:58:45,691 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:51964 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 09:58:45,691 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54502 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 09:58:45,691 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57136 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 09:58:45,691 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52560 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 09:58:45,691 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41408 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 09:58:45,691 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54318 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 09:58:45,691 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:58012 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41080 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46400 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54934 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55734 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50790 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52464 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48896 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39178 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59842 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:51948 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54494 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57130 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52554 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 09:58:45,690 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41392 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54314 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57998 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41066 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46398 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54930 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55720 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50784 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52462 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48888 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39162 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59836 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:51946 remote=tcp://172.21.160.14:33823>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54484 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57126 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 09:58:45,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52544 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 09:58:45,688 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41384 remote=tcp://172.21.159.225:34523>: Stream is closed
2026-06-05 09:58:45,688 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54304 remote=tcp://172.21.159.222:38709>: Stream is closed
2026-06-05 09:58:45,688 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57986 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 09:58:45,688 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41052 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 09:58:45,688 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46382 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 09:58:45,688 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54926 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 09:58:45,688 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55712 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 09:58:45,688 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50770 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 09:58:45,687 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52444 remote=tcp://172.21.25.54:40907>: Stream is closed
2026-06-05 09:58:45,687 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:48884 remote=tcp://172.21.25.34:45563>: Stream is closed
2026-06-05 09:58:45,687 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:39146 remote=tcp://172.21.25.32:35467>: Stream is closed
2026-06-05 09:58:45,687 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:59796 remote=tcp://172.21.25.29:39127>: Stream is closed
2026-06-05 09:58:45,687 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54456 remote=tcp://172.21.159.253:40151>: Stream is closed
2026-06-05 09:58:45,686 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57080 remote=tcp://172.21.159.246:40119>: Stream is closed
2026-06-05 09:58:45,686 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52520 remote=tcp://172.21.159.230:43433>: Stream is closed
2026-06-05 09:58:45,686 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:57956 remote=tcp://172.21.159.214:46479>: Stream is closed
2026-06-05 09:58:45,686 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:41024 remote=tcp://172.21.159.207:39369>: Stream is closed
2026-06-05 09:58:45,686 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:46356 remote=tcp://172.21.159.197:35713>: Stream is closed
2026-06-05 09:58:45,686 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:54906 remote=tcp://172.21.151.183:37023>: Stream is closed
2026-06-05 09:58:45,686 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:55676 remote=tcp://172.21.151.168:43677>: Stream is closed
2026-06-05 09:58:45,686 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:50746 remote=tcp://172.21.151.150:39501>: Stream is closed
2026-06-05 09:58:32,993 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:58:32,993 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:58:32,993 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:58:32,988 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out trying to connect to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:58:32,988 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:58:32,988 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:58:32,988 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:58:32,988 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:58:32,988 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:58:32,988 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out trying to connect to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out trying to connect to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out trying to connect to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out trying to connect to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out trying to connect to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out trying to connect to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out trying to connect to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out trying to connect to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:58:32,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out trying to connect to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:58:32,986 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,725 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,724 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,723 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,722 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,721 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,715 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,714 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,713 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40907 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.54:40907 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45563 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.34:45563 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:35467 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.32:35467 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:39127 failed: OSError: Timed out during handshake while connecting to tcp://172.21.25.29:39127 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33823 failed: OSError: Timed out during handshake while connecting to tcp://172.21.160.14:33823 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:40151 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.253:40151 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:40119 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.246:40119 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43433 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.230:43433 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:34523 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:34523 after 30 s
2026-06-05 09:55:32,712 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:38709 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.222:38709 after 30 s
2026-06-05 09:55:32,711 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.214:46479 after 30 s
2026-06-05 09:55:32,711 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39369 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:39369 after 30 s
2026-06-05 09:55:32,711 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35713 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.197:35713 after 30 s
2026-06-05 09:55:32,711 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,711 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:55:32,711 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:39501 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.150:39501 after 30 s
2026-06-05 09:55:32,711 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:55:32,711 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-06-05 09:54:35,322 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:37023 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:37023 after 30 s
2026-06-05 09:54:35,322 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:43677 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.168:43677 after 30 s
2026-04-22 11:01:06,873 - distributed.scheduler - INFO - Receive client connection: Client-worker-8fe49a9e-3e3a-11f1-8fcc-b6f02dd406e9
2026-04-22 11:01:06,756 - distributed.scheduler - INFO - Receive client connection: Client-worker-8fd4c93f-3e3a-11f1-8f9d-566b4744c386
2026-04-22 11:01:05,655 - distributed.scheduler - INFO - Receive client connection: Client-worker-8f2e88fc-3e3a-11f1-8f4c-8288bfee4388
2026-04-22 11:01:05,358 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:45563
2026-04-22 11:01:05,356 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:45563', status: running, memory: 9, processing: 0>
2026-04-22 11:01:05,310 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:46479
2026-04-22 11:01:05,309 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:46479', status: running, memory: 0, processing: 0>
2026-04-22 11:01:05,275 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:38709
2026-04-22 11:01:05,273 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:38709', status: running, memory: 0, processing: 0>
2026-04-22 11:01:05,219 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:34523
2026-04-22 11:01:05,217 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:34523', status: running, memory: 0, processing: 0>
2026-04-22 11:01:05,104 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:43677
2026-04-22 11:01:05,103 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:43677', status: running, memory: 1, processing: 0>
2026-04-22 11:01:05,075 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:39369
2026-04-22 11:01:05,074 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:39369', status: running, memory: 1, processing: 0>
2026-04-22 11:01:04,810 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:35467
2026-04-22 11:01:04,809 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:35467', status: running, memory: 5, processing: 0>
2026-04-22 11:01:04,804 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:39127
2026-04-22 11:01:04,803 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:39127', status: running, memory: 11, processing: 0>
2026-04-22 11:01:04,755 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:40151
2026-04-22 11:01:04,754 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:40151', status: running, memory: 7, processing: 0>
2026-04-22 11:01:04,680 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:40907
2026-04-22 11:01:04,678 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:40907', status: running, memory: 36, processing: 0>
2026-04-22 11:01:04,644 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:37023
2026-04-22 11:01:04,642 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:37023', status: running, memory: 32, processing: 0>
2026-04-22 11:01:04,638 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:43433
2026-04-22 11:01:04,636 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:43433', status: running, memory: 18, processing: 0>
2026-04-22 11:01:04,517 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:33823
2026-04-22 11:01:04,515 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:33823', status: running, memory: 32, processing: 0>
2026-04-22 11:01:04,453 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:35713
2026-04-22 11:01:04,452 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:35713', status: running, memory: 28, processing: 0>
2026-04-22 11:01:04,329 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:39501
2026-04-22 11:01:04,327 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:39501', status: running, memory: 2, processing: 0>
2026-04-22 11:01:04,297 - distributed.scheduler - INFO - Close client connection: Client-worker-7aa8bd1c-3e3a-11f1-8f4c-2a51d3dc178e
2026-04-22 11:01:04,296 - distributed.scheduler - INFO - Close client connection: Client-worker-5c281dea-3e36-11f1-8f64-fae0fcb74df3
2026-04-22 11:01:04,295 - distributed.scheduler - ERROR - SchedulerState.transition_processing_erred() missing 1 required keyword-only argument: 'worker' Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/utils.py", line 752, in wrapper return await func(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 4870, in remove_worker self.transitions(recommendations, stimulus_id=stimulus_id) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 7308, in transitions self._transitions(recommendations, client_msgs, worker_msgs, stimulus_id) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 1955, in _transitions new_recs, new_cmsgs, new_wmsgs = self._transition(key, finish, stimulus_id) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 1851, in _transition recommendations, client_msgs, worker_msgs = func( TypeError: SchedulerState.transition_processing_erred() missing 1 required keyword-only argument: 'worker'
2026-04-22 11:01:04,294 - distributed.scheduler - ERROR - Error transitioning "('simple-shuffle-06cc96759edeb7097969c2374ef22039', 6)" from 'processing' to 'erred' Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 1851, in _transition recommendations, client_msgs, worker_msgs = func( TypeError: SchedulerState.transition_processing_erred() missing 1 required keyword-only argument: 'worker'
2026-04-22 11:01:04,279 - distributed.scheduler - INFO - Task ('from_pandas-5e71892f0e34621cf2d740f0c2b1f96f', 8) marked as failed because 3 workers died while trying to run it
2026-04-22 11:01:04,279 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:35927', status: closed, memory: 0, processing: 2>
2026-04-22 11:01:04,274 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:40849', status: closed, memory: 0, processing: 0>
2026-04-22 11:01:04,274 - distributed.scheduler - INFO - Remove client Client-worker-7aa8bd1c-3e3a-11f1-8f4c-2a51d3dc178e
2026-04-22 11:01:04,271 - distributed.scheduler - ERROR - SchedulerState.transition_processing_erred() missing 1 required keyword-only argument: 'worker' Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/utils.py", line 752, in wrapper return await func(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 4870, in remove_worker self.transitions(recommendations, stimulus_id=stimulus_id) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 7308, in transitions self._transitions(recommendations, client_msgs, worker_msgs, stimulus_id) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 1955, in _transitions new_recs, new_cmsgs, new_wmsgs = self._transition(key, finish, stimulus_id) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 1851, in _transition recommendations, client_msgs, worker_msgs = func( TypeError: SchedulerState.transition_processing_erred() missing 1 required keyword-only argument: 'worker'
2026-04-22 11:01:04,270 - distributed.scheduler - ERROR - Error transitioning "('simple-shuffle-06cc96759edeb7097969c2374ef22039', 6)" from 'processing' to 'erred' Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 1851, in _transition recommendations, client_msgs, worker_msgs = func( TypeError: SchedulerState.transition_processing_erred() missing 1 required keyword-only argument: 'worker'
2026-04-22 11:01:04,268 - distributed.scheduler - INFO - Task ('from_pandas-5e71892f0e34621cf2d740f0c2b1f96f', 14) marked as failed because 3 workers died while trying to run it
2026-04-22 11:01:04,268 - distributed.scheduler - INFO - Task ('from_pandas-6d59010172bbd4429cbe00e2a674eb65', 9) marked as failed because 3 workers died while trying to run it
2026-04-22 11:01:04,268 - distributed.scheduler - INFO - Task ('from_pandas-6d59010172bbd4429cbe00e2a674eb65', 12) marked as failed because 3 workers died while trying to run it
2026-04-22 11:01:04,268 - distributed.scheduler - INFO - Task ('from_pandas-6d59010172bbd4429cbe00e2a674eb65', 15) marked as failed because 3 workers died while trying to run it
2026-04-22 11:01:04,268 - distributed.scheduler - INFO - Task ('from_pandas-5e71892f0e34621cf2d740f0c2b1f96f', 15) marked as failed because 3 workers died while trying to run it
2026-04-22 11:01:04,267 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:46131', status: closed, memory: 0, processing: 32>
2026-04-22 11:01:04,254 - distributed.scheduler - INFO - Remove client Client-worker-5c281dea-3e36-11f1-8f64-fae0fcb74df3
2026-04-22 11:01:04,247 - distributed.scheduler - INFO - Close client connection: Client-worker-7b235ff0-3e3a-11f1-8fcd-86acc56fd09f
2026-04-22 11:01:04,221 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:37891', status: closed, memory: 0, processing: 0>
2026-04-22 11:01:04,220 - distributed.scheduler - INFO - Remove client Client-worker-7b235ff0-3e3a-11f1-8fcd-86acc56fd09f
2026-04-22 11:01:04,183 - distributed.scheduler - INFO - Close client connection: Client-worker-5bdedb4d-3e36-11f1-8f5c-566b4744c386
2026-04-22 11:01:04,175 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:39731', status: closed, memory: 0, processing: 0>
2026-04-22 11:01:04,174 - distributed.scheduler - INFO - Remove client Client-worker-5bdedb4d-3e36-11f1-8f5c-566b4744c386
2026-04-22 11:01:04,123 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:32835', status: closed, memory: 0, processing: 0>
2026-04-22 11:01:03,788 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:41105', status: closed, memory: 0, processing: 0>
2026-04-22 11:01:03,708 - distributed.scheduler - INFO - Close client connection: Client-worker-5c084c35-3e36-11f1-8f2b-c2349ba99f21
2026-04-22 11:01:03,701 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:34733', status: closed, memory: 0, processing: 0>
2026-04-22 11:01:03,700 - distributed.scheduler - INFO - Remove client Client-worker-5c084c35-3e36-11f1-8f2b-c2349ba99f21
2026-04-22 11:01:03,682 - distributed.scheduler - INFO - Close client connection: Client-worker-5bd1ab09-3e36-11f1-8f5f-7e353d94dc10
2026-04-22 11:01:03,678 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:33301', status: closed, memory: 0, processing: 0>
2026-04-22 11:01:03,677 - distributed.scheduler - INFO - Remove client Client-worker-5bd1ab09-3e36-11f1-8f5f-7e353d94dc10
2026-04-22 11:01:03,668 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:33161', status: closed, memory: 0, processing: 0>
2026-04-22 11:01:03,629 - distributed.scheduler - INFO - Close client connection: Client-worker-5ae4dbd5-3e36-11f1-900e-4ef3162b8ac2
2026-04-22 11:01:03,621 - distributed.scheduler - INFO - Remove client Client-worker-5ae4dbd5-3e36-11f1-900e-4ef3162b8ac2
2026-04-22 11:01:03,611 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:44927', status: closed, memory: 0, processing: 0>
2026-04-22 11:01:03,566 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:45149', status: closed, memory: 0, processing: 0>
2026-04-22 11:01:03,520 - distributed.scheduler - INFO - Close client connection: Client-worker-7b30097a-3e3a-11f1-8f85-ee7435aaba1f
2026-04-22 11:01:03,506 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:45329', status: closed, memory: 0, processing: 0>
2026-04-22 11:01:03,504 - distributed.scheduler - INFO - Remove client Client-worker-7b30097a-3e3a-11f1-8f85-ee7435aaba1f
2026-04-22 11:01:03,409 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:39555', status: closed, memory: 0, processing: 0>
2026-04-22 11:01:03,364 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:45021', status: closed, memory: 0, processing: 0>
2026-04-22 11:00:32,146 - distributed.scheduler - INFO - Receive client connection: Client-worker-7b30097a-3e3a-11f1-8f85-ee7435aaba1f
2026-04-22 11:00:32,086 - distributed.scheduler - INFO - Receive client connection: Client-worker-7b235ff0-3e3a-11f1-8fcd-86acc56fd09f
2026-04-22 11:00:31,305 - distributed.scheduler - INFO - Receive client connection: Client-worker-7aa8bd1c-3e3a-11f1-8f4c-2a51d3dc178e
2026-04-22 10:31:31,305 - distributed.scheduler - INFO - Receive client connection: Client-6d94be75-3e36-11f1-8007-924c5ad31268
2026-04-22 10:31:28,378 - distributed.scheduler - INFO - Close client connection: Client-4b9b8742-3e1d-11f1-8007-924c5ad31268
2026-04-22 10:31:28,376 - distributed.scheduler - INFO - Remove client Client-4b9b8742-3e1d-11f1-8007-924c5ad31268
2026-04-22 10:31:28,375 - distributed.scheduler - INFO - Remove client Client-4b9b8742-3e1d-11f1-8007-924c5ad31268
2026-04-22 10:31:02,813 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:45329
2026-04-22 10:31:02,813 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:45329', status: closed, memory: 0, processing: 0>
2026-04-22 10:31:02,788 - distributed.scheduler - INFO - Receive client connection: Client-worker-5c948c6b-3e36-11f1-8f7d-ae00acbfc48e
2026-04-22 10:31:02,357 - distributed.scheduler - INFO - Receive client connection: Client-worker-5c281dea-3e36-11f1-8f64-fae0fcb74df3
2026-04-22 10:31:02,339 - distributed.scheduler - INFO - Receive client connection: Client-worker-5c084c35-3e36-11f1-8f2b-c2349ba99f21
2026-04-22 10:31:02,328 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:37891
2026-04-22 10:31:02,328 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:37891', status: closed, memory: 0, processing: 0>
2026-04-22 10:31:01,649 - distributed.scheduler - INFO - Close client connection: Client-worker-472a1c56-3e36-11f1-8f45-ee7435aaba1f
2026-04-22 10:31:01,637 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:40627', status: closed, memory: 0, processing: 0>
2026-04-22 10:31:01,607 - distributed.scheduler - INFO - Remove client Client-worker-472a1c56-3e36-11f1-8f45-ee7435aaba1f
2026-04-22 10:31:01,594 - distributed.scheduler - INFO - Receive client connection: Client-worker-5bdedb4d-3e36-11f1-8f5c-566b4744c386
2026-04-22 10:31:01,498 - distributed.scheduler - INFO - Receive client connection: Client-worker-5bd1ab09-3e36-11f1-8f5f-7e353d94dc10
2026-04-22 10:31:01,269 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:33161
2026-04-22 10:31:01,267 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:33161', status: closed, memory: 0, processing: 0>
2026-04-22 10:31:01,190 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:35099', status: closed, memory: 0, processing: 0>
2026-04-22 10:31:00,580 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:45021
2026-04-22 10:31:00,579 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:45021', status: closed, memory: 0, processing: 0>
2026-04-22 10:31:00,565 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:40119
2026-04-22 10:31:00,564 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:40119', status: running, memory: 22, processing: 0>
2026-04-22 10:31:00,490 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:34733
2026-04-22 10:31:00,488 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:34733', status: closed, memory: 0, processing: 0>
2026-04-22 10:31:00,382 - distributed.scheduler - ERROR - SchedulerState.transition_processing_erred() missing 1 required keyword-only argument: 'worker' Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/utils.py", line 752, in wrapper return await func(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 4870, in remove_worker self.transitions(recommendations, stimulus_id=stimulus_id) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 7308, in transitions self._transitions(recommendations, client_msgs, worker_msgs, stimulus_id) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 1955, in _transitions new_recs, new_cmsgs, new_wmsgs = self._transition(key, finish, stimulus_id) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 1851, in _transition recommendations, client_msgs, worker_msgs = func( TypeError: SchedulerState.transition_processing_erred() missing 1 required keyword-only argument: 'worker'
2026-04-22 10:31:00,381 - distributed.scheduler - ERROR - Error transitioning "('simple-shuffle-06cc96759edeb7097969c2374ef22039', 6)" from 'processing' to 'erred' Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 1851, in _transition recommendations, client_msgs, worker_msgs = func( TypeError: SchedulerState.transition_processing_erred() missing 1 required keyword-only argument: 'worker'
2026-04-22 10:31:00,347 - distributed.scheduler - INFO - Task ('from_pandas-5e71892f0e34621cf2d740f0c2b1f96f', 13) marked as failed because 3 workers died while trying to run it
2026-04-22 10:31:00,346 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:37921', status: closed, memory: 0, processing: 0>
2026-04-22 10:31:00,301 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:32835
2026-04-22 10:31:00,300 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:32835', status: closed, memory: 0, processing: 0>
2026-04-22 10:31:00,246 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:46131
2026-04-22 10:31:00,245 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:46131', status: closed, memory: 0, processing: 32>
2026-04-22 10:31:00,076 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:45149
2026-04-22 10:31:00,075 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:45149', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:59,971 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:41105
2026-04-22 10:30:59,969 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:41105', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:59,952 - distributed.scheduler - INFO - Receive client connection: Client-worker-5ae4dbd5-3e36-11f1-900e-4ef3162b8ac2
2026-04-22 10:30:59,860 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:39555
2026-04-22 10:30:59,859 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:39555', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:59,701 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:35927
2026-04-22 10:30:59,700 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:35927', status: closed, memory: 0, processing: 2>
2026-04-22 10:30:59,671 - distributed.scheduler - INFO - Close client connection: Client-worker-477972dc-3e36-11f1-8f3c-ae00acbfc48e
2026-04-22 10:30:59,669 - distributed.scheduler - ERROR - SchedulerState.transition_processing_erred() missing 1 required keyword-only argument: 'worker' Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/utils.py", line 752, in wrapper return await func(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 4870, in remove_worker self.transitions(recommendations, stimulus_id=stimulus_id) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 7308, in transitions self._transitions(recommendations, client_msgs, worker_msgs, stimulus_id) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 1955, in _transitions new_recs, new_cmsgs, new_wmsgs = self._transition(key, finish, stimulus_id) File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 1851, in _transition recommendations, client_msgs, worker_msgs = func( TypeError: SchedulerState.transition_processing_erred() missing 1 required keyword-only argument: 'worker'
2026-04-22 10:30:59,667 - distributed.scheduler - ERROR - Error transitioning "('simple-shuffle-06cc96759edeb7097969c2374ef22039', 6)" from 'processing' to 'erred' Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 1851, in _transition recommendations, client_msgs, worker_msgs = func( TypeError: SchedulerState.transition_processing_erred() missing 1 required keyword-only argument: 'worker'
2026-04-22 10:30:59,640 - distributed.scheduler - INFO - Task ('from_pandas-7257ad7e7e1317b17a9f46f3bd9f975f', 10) marked as failed because 3 workers died while trying to run it
2026-04-22 10:30:59,639 - distributed.scheduler - INFO - Task ('from_pandas-5e71892f0e34621cf2d740f0c2b1f96f', 2) marked as failed because 3 workers died while trying to run it
2026-04-22 10:30:59,638 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:38983', status: closed, memory: 0, processing: 8>
2026-04-22 10:30:59,632 - distributed.scheduler - INFO - Remove client Client-worker-477972dc-3e36-11f1-8f3c-ae00acbfc48e
2026-04-22 10:30:59,573 - distributed.scheduler - INFO - Close client connection: Client-worker-476ed5e9-3e36-11f1-908b-f28f846166bd
2026-04-22 10:30:59,559 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:32895', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:59,558 - distributed.scheduler - INFO - Remove client Client-worker-476ed5e9-3e36-11f1-908b-f28f846166bd
2026-04-22 10:30:59,335 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:40849
2026-04-22 10:30:59,334 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:40849', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:59,332 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:33301
2026-04-22 10:30:59,330 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:33301', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:59,306 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:41371', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:59,278 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:34747', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:59,262 - distributed.scheduler - INFO - Close client connection: Client-worker-4772ef5c-3e36-11f1-8f28-fae0fcb74df3
2026-04-22 10:30:59,257 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:41545', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:59,254 - distributed.scheduler - INFO - Remove client Client-worker-4772ef5c-3e36-11f1-8f28-fae0fcb74df3
2026-04-22 10:30:59,156 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:42513', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:59,144 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:44927
2026-04-22 10:30:59,142 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:44927', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:59,074 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:39731
2026-04-22 10:30:59,072 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:39731', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:58,913 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:34761', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:58,696 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:33241', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:58,634 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:42777', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:58,386 - distributed.scheduler - INFO - Close client connection: Client-worker-4a632198-3e32-11f1-8f17-7e353d94dc10
2026-04-22 10:30:58,377 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:37411', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:58,339 - distributed.scheduler - INFO - Remove client Client-worker-4a632198-3e32-11f1-8f17-7e353d94dc10
2026-04-22 10:30:58,279 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:37761', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:58,237 - distributed.scheduler - INFO - Close client connection: Client-worker-46f8746b-3e36-11f1-8fd2-4ef3162b8ac2
2026-04-22 10:30:58,217 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:37645', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:58,215 - distributed.scheduler - INFO - Remove client Client-worker-46f8746b-3e36-11f1-8fd2-4ef3162b8ac2
2026-04-22 10:30:58,140 - distributed.scheduler - INFO - Close client connection: Client-worker-47069750-3e32-11f1-8f18-566b4744c386
2026-04-22 10:30:58,117 - distributed.scheduler - INFO - Remove client Client-worker-47069750-3e32-11f1-8f18-566b4744c386
2026-04-22 10:30:58,104 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:36529', status: closed, memory: 0, processing: 0>
2026-04-22 10:30:27,386 - distributed.scheduler - INFO - Receive client connection: Client-worker-477972dc-3e36-11f1-8f3c-ae00acbfc48e
2026-04-22 10:30:27,372 - distributed.scheduler - INFO - Receive client connection: Client-worker-4772ef5c-3e36-11f1-8f28-fae0fcb74df3
2026-04-22 10:30:27,349 - distributed.scheduler - INFO - Receive client connection: Client-worker-476ed5e9-3e36-11f1-908b-f28f846166bd
2026-04-22 10:30:26,840 - distributed.scheduler - INFO - Receive client connection: Client-worker-472a1c56-3e36-11f1-8f45-ee7435aaba1f
2026-04-22 10:30:26,525 - distributed.scheduler - INFO - Receive client connection: Client-worker-46f8746b-3e36-11f1-8fd2-4ef3162b8ac2
2026-04-22 10:30:22,567 - distributed.scheduler - INFO - Receive client connection: Client-449b0cea-3e36-11f1-8001-daa6852bd5a7
2026-04-22 10:20:57,878 - distributed.scheduler - INFO - Close client connection: Client-de187e15-3e34-11f1-8001-daa6852bd5a7
2026-04-22 10:20:57,877 - distributed.scheduler - INFO - Remove client Client-de187e15-3e34-11f1-8001-daa6852bd5a7
2026-04-22 10:20:57,875 - distributed.scheduler - INFO - Remove client Client-de187e15-3e34-11f1-8001-daa6852bd5a7
2026-04-22 10:20:21,086 - distributed.scheduler - INFO - Receive client connection: Client-de187e15-3e34-11f1-8001-daa6852bd5a7
2026-04-22 10:13:20,447 - distributed.scheduler - INFO - Close client connection: Client-cfa49bd4-3e33-11f1-8001-daa6852bd5a7
2026-04-22 10:13:20,446 - distributed.scheduler - INFO - Remove client Client-cfa49bd4-3e33-11f1-8001-daa6852bd5a7
2026-04-22 10:13:20,444 - distributed.scheduler - INFO - Remove client Client-cfa49bd4-3e33-11f1-8001-daa6852bd5a7
2026-04-22 10:12:47,341 - distributed.scheduler - INFO - Receive client connection: Client-cfa49bd4-3e33-11f1-8001-daa6852bd5a7
2026-04-22 10:01:54,258 - distributed.scheduler - INFO - Receive client connection: Client-worker-4a632198-3e32-11f1-8f17-7e353d94dc10
2026-04-22 10:01:48,631 - distributed.scheduler - INFO - Receive client connection: Client-worker-47069750-3e32-11f1-8f18-566b4744c386
2026-04-22 10:01:48,225 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:33241
2026-04-22 10:01:48,214 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:33241', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:48,137 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:42513
2026-04-22 10:01:48,135 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:42513', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:48,108 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:37921
2026-04-22 10:01:48,107 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:37921', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:48,080 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:32895
2026-04-22 10:01:48,079 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:32895', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:48,072 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:38983
2026-04-22 10:01:48,071 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:38983', status: closed, memory: 0, processing: 8>
2026-04-22 10:01:48,059 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:35099
2026-04-22 10:01:48,058 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:35099', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:48,055 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:41371
2026-04-22 10:01:48,054 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:41371', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:48,052 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:40627
2026-04-22 10:01:48,051 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:40627', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:48,009 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:34747
2026-04-22 10:01:48,008 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:34747', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,978 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:37411
2026-04-22 10:01:47,977 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:37411', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,968 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:41545
2026-04-22 10:01:47,967 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:41545', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,962 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:37761
2026-04-22 10:01:47,961 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:37761', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,949 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:42777
2026-04-22 10:01:47,948 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:42777', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,814 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:34761
2026-04-22 10:01:47,813 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:34761', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,801 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:37645
2026-04-22 10:01:47,799 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:37645', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,754 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:36529
2026-04-22 10:01:47,752 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:36529', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,225 - distributed.scheduler - INFO - Close client connection: Client-worker-e0406a48-3e2d-11f1-8ef2-2e20a4008324
2026-04-22 10:01:47,225 - distributed.scheduler - INFO - Lost all workers
2026-04-22 10:01:47,224 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:43055', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,223 - distributed.scheduler - INFO - Remove client Client-worker-e0406a48-3e2d-11f1-8ef2-2e20a4008324
2026-04-22 10:01:47,179 - distributed.scheduler - INFO - Close client connection: Client-1115f0f1-3e32-11f1-8001-daa6852bd5a7
2026-04-22 10:01:47,178 - distributed.scheduler - INFO - Remove client Client-1115f0f1-3e32-11f1-8001-daa6852bd5a7
2026-04-22 10:01:47,178 - distributed.scheduler - INFO - Remove client Client-1115f0f1-3e32-11f1-8001-daa6852bd5a7
2026-04-22 10:01:47,137 - distributed.scheduler - INFO - Task ('from_pandas-fece414c141f919c42a3acf6b286a9c7', 4) marked as failed because 3 workers died while trying to run it
2026-04-22 10:01:47,136 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:42667', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,066 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:41467', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,052 - distributed.scheduler - INFO - Close client connection: Client-worker-c431968b-3e29-11f1-8ef0-ae00acbfc48e
2026-04-22 10:01:47,051 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:35675', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,050 - distributed.scheduler - INFO - Remove client Client-worker-c431968b-3e29-11f1-8ef0-ae00acbfc48e
2026-04-22 10:01:47,019 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:38013', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,012 - distributed.scheduler - INFO - Close client connection: Client-worker-13a59a04-3e32-11f1-9046-f28f846166bd
2026-04-22 10:01:47,011 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:40035', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:47,009 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:40055', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:46,976 - distributed.scheduler - INFO - Remove client Client-worker-13a59a04-3e32-11f1-9046-f28f846166bd
2026-04-22 10:01:46,933 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:37159', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:46,929 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:44859', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:46,926 - distributed.scheduler - INFO - Close client connection: Client-worker-c2f26e55-3e29-11f1-8f96-4ef3162b8ac2
2026-04-22 10:01:46,925 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:39675', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:46,925 - distributed.scheduler - INFO - Remove client Client-worker-c2f26e55-3e29-11f1-8f96-4ef3162b8ac2
2026-04-22 10:01:46,923 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:42181', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:46,922 - distributed.scheduler - INFO - Close client connection: Client-worker-e0186738-3e2d-11f1-8ed9-2a51d3dc178e
2026-04-22 10:01:46,922 - distributed.scheduler - INFO - Remove client Client-worker-e0186738-3e2d-11f1-8ed9-2a51d3dc178e
2026-04-22 10:01:46,918 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:40057', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:46,911 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:39689', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:46,904 - distributed.scheduler - INFO - Close client connection: Client-worker-c65b21a2-3e29-11f1-8ede-fae0fcb74df3
2026-04-22 10:01:46,900 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:41711', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:46,894 - distributed.scheduler - INFO - Remove client Client-worker-c65b21a2-3e29-11f1-8ede-fae0fcb74df3
2026-04-22 10:01:46,890 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:35691', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:46,819 - distributed.scheduler - INFO - Close client connection: Client-worker-c339e3ca-3e29-11f1-8ed2-566b4744c386
2026-04-22 10:01:46,804 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:44855', status: closed, memory: 0, processing: 0>
2026-04-22 10:01:46,803 - distributed.scheduler - INFO - Remove client Client-worker-c339e3ca-3e29-11f1-8ed2-566b4744c386
2026-04-22 10:00:49,388 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:44859
2026-04-22 10:00:49,386 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:44859', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:48,846 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:37159
2026-04-22 10:00:48,845 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:37159', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:48,459 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:38013
2026-04-22 10:00:48,458 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:38013', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:48,279 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:41711
2026-04-22 10:00:48,277 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:41711', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:48,177 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:35541', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:48,124 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:41467
2026-04-22 10:00:48,123 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:41467', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:48,121 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:40057
2026-04-22 10:00:48,120 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:40057', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:48,118 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:40035
2026-04-22 10:00:48,116 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:40035', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:47,789 - distributed.scheduler - INFO - Close client connection: Client-worker-c39ef368-3e29-11f1-8f14-2eb33e86e1bd
2026-04-22 10:00:47,783 - distributed.scheduler - INFO - Remove client Client-worker-c39ef368-3e29-11f1-8f14-2eb33e86e1bd
2026-04-22 10:00:47,779 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:38119', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:47,455 - distributed.scheduler - INFO - Close client connection: Client-worker-13a28bb9-3e32-11f1-8e67-8288bfee4388
2026-04-22 10:00:47,450 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:38415', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:47,431 - distributed.scheduler - INFO - Remove client Client-worker-13a28bb9-3e32-11f1-8e67-8288bfee4388
2026-04-22 10:00:47,322 - distributed.scheduler - INFO - Close client connection: Client-worker-c3e6adb9-3e29-11f1-8eed-b6f02dd406e9
2026-04-22 10:00:47,314 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:33213', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:47,313 - distributed.scheduler - INFO - Remove client Client-worker-c3e6adb9-3e29-11f1-8eed-b6f02dd406e9
2026-04-22 10:00:47,078 - distributed.scheduler - INFO - Close client connection: Client-worker-e05cd836-3e2d-11f1-8f17-86acc56fd09f
2026-04-22 10:00:47,075 - distributed.scheduler - INFO - Close client connection: Client-worker-13c5dd72-3e32-11f1-8e81-c2349ba99f21
2026-04-22 10:00:47,063 - distributed.scheduler - INFO - Remove client Client-worker-e05cd836-3e2d-11f1-8f17-86acc56fd09f
2026-04-22 10:00:47,062 - distributed.scheduler - INFO - Close client connection: Client-worker-c386f87f-3e29-11f1-8eba-6ec63a5ab21e
2026-04-22 10:00:47,057 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:33987', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:47,048 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:35409', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:47,034 - distributed.scheduler - INFO - Remove client Client-worker-13c5dd72-3e32-11f1-8e81-c2349ba99f21
2026-04-22 10:00:47,019 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:38067', status: closed, memory: 0, processing: 0>
2026-04-22 10:00:46,890 - distributed.scheduler - INFO - Remove client Client-worker-c386f87f-3e29-11f1-8eba-6ec63a5ab21e
2026-04-22 10:00:22,663 - distributed.scheduler - INFO - Receive client connection: Client-worker-13c5dd72-3e32-11f1-8e81-c2349ba99f21
2026-04-22 10:00:22,434 - distributed.scheduler - INFO - Receive client connection: Client-worker-13a59a04-3e32-11f1-9046-f28f846166bd
2026-04-22 10:00:22,427 - distributed.scheduler - INFO - Receive client connection: Client-worker-13a28bb9-3e32-11f1-8e67-8288bfee4388
2026-04-22 10:00:18,144 - distributed.scheduler - INFO - Receive client connection: Client-1115f0f1-3e32-11f1-8001-daa6852bd5a7
2026-04-22 09:50:55,761 - distributed.scheduler - INFO - Close client connection: Client-aa8f9903-3e30-11f1-8001-daa6852bd5a7
2026-04-22 09:50:55,761 - distributed.scheduler - INFO - Remove client Client-aa8f9903-3e30-11f1-8001-daa6852bd5a7
2026-04-22 09:50:55,759 - distributed.scheduler - INFO - Remove client Client-aa8f9903-3e30-11f1-8001-daa6852bd5a7
2026-04-22 09:50:16,639 - distributed.scheduler - INFO - Receive client connection: Client-aa8f9903-3e30-11f1-8001-daa6852bd5a7
2026-04-22 09:40:54,236 - distributed.scheduler - INFO - Close client connection: Client-4407da7a-3e2f-11f1-8001-daa6852bd5a7
2026-04-22 09:40:54,235 - distributed.scheduler - INFO - Remove client Client-4407da7a-3e2f-11f1-8001-daa6852bd5a7
2026-04-22 09:40:54,233 - distributed.scheduler - INFO - Remove client Client-4407da7a-3e2f-11f1-8001-daa6852bd5a7
2026-04-22 09:40:15,129 - distributed.scheduler - INFO - Receive client connection: Client-4407da7a-3e2f-11f1-8001-daa6852bd5a7
2026-04-22 09:31:26,255 - distributed.scheduler - INFO - Close client connection: Client-dd83b5ce-3e2d-11f1-8001-daa6852bd5a7
2026-04-22 09:31:26,247 - distributed.scheduler - INFO - Remove client Client-dd83b5ce-3e2d-11f1-8001-daa6852bd5a7
2026-04-22 09:31:26,245 - distributed.scheduler - INFO - Remove client Client-dd83b5ce-3e2d-11f1-8001-daa6852bd5a7
2026-04-22 09:30:18,405 - distributed.scheduler - INFO - Receive client connection: Client-worker-e05cd836-3e2d-11f1-8f17-86acc56fd09f
2026-04-22 09:30:18,208 - distributed.scheduler - INFO - Receive client connection: Client-worker-e0406a48-3e2d-11f1-8ef2-2e20a4008324
2026-04-22 09:30:17,949 - distributed.scheduler - INFO - Receive client connection: Client-worker-e0186738-3e2d-11f1-8ed9-2a51d3dc178e
2026-04-22 09:30:13,635 - distributed.scheduler - INFO - Receive client connection: Client-dd83b5ce-3e2d-11f1-8001-daa6852bd5a7
2026-04-22 09:20:51,743 - distributed.scheduler - INFO - Close client connection: Client-7701fbe5-3e2c-11f1-8001-daa6852bd5a7
2026-04-22 09:20:51,741 - distributed.scheduler - INFO - Remove client Client-7701fbe5-3e2c-11f1-8001-daa6852bd5a7
2026-04-22 09:20:51,739 - distributed.scheduler - INFO - Remove client Client-7701fbe5-3e2c-11f1-8001-daa6852bd5a7
2026-04-22 09:20:12,155 - distributed.scheduler - INFO - Receive client connection: Client-7701fbe5-3e2c-11f1-8001-daa6852bd5a7
2026-04-22 09:13:03,269 - distributed.scheduler - INFO - Close client connection: Client-63000cbf-3e2b-11f1-8001-daa6852bd5a7
2026-04-22 09:13:03,268 - distributed.scheduler - INFO - Remove client Client-63000cbf-3e2b-11f1-8001-daa6852bd5a7
2026-04-22 09:13:03,266 - distributed.scheduler - INFO - Remove client Client-63000cbf-3e2b-11f1-8001-daa6852bd5a7
2026-04-22 09:12:29,097 - distributed.scheduler - INFO - Receive client connection: Client-63000cbf-3e2b-11f1-8001-daa6852bd5a7
2026-04-22 09:01:58,617 - distributed.scheduler - INFO - Close client connection: Client-aa1a3252-3e29-11f1-8001-daa6852bd5a7
2026-04-22 09:01:58,615 - distributed.scheduler - INFO - Remove client Client-aa1a3252-3e29-11f1-8001-daa6852bd5a7
2026-04-22 09:01:58,614 - distributed.scheduler - INFO - Remove client Client-aa1a3252-3e29-11f1-8001-daa6852bd5a7
2026-04-22 09:00:57,566 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 14) NoneType: None
2026-04-22 09:00:57,566 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 12) NoneType: None
2026-04-22 09:00:57,565 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 11) NoneType: None
2026-04-22 09:00:57,565 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 13) NoneType: None
2026-04-22 09:00:57,565 - distributed.scheduler - ERROR - Couldn't gather keys {"('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 13)": [], "('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 11)": [], "('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 12)": [], "('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 14)": []} state: ['waiting', 'waiting', 'waiting', 'waiting'] workers: [] NoneType: None
2026-04-22 09:00:56,943 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:43055
2026-04-22 09:00:56,942 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:43055', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:56,801 - distributed.scheduler - INFO - Receive client connection: Client-worker-c65b21a2-3e29-11f1-8ede-fae0fcb74df3
2026-04-22 09:00:56,728 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:33987
2026-04-22 09:00:56,727 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:33987', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:56,475 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:39689
2026-04-22 09:00:56,474 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:39689', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:55,974 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:35409
2026-04-22 09:00:55,972 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:35409', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:55,861 - distributed.scheduler - INFO - Close client connection: Client-worker-79a6903c-3e25-11f1-8eb2-2e20a4008324
2026-04-22 09:00:55,847 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:44527', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:55,844 - distributed.scheduler - INFO - Remove client Client-worker-79a6903c-3e25-11f1-8eb2-2e20a4008324
2026-04-22 09:00:55,725 - distributed.scheduler - INFO - Close client connection: Client-worker-79848d3e-3e25-11f1-8ed9-86acc56fd09f
2026-04-22 09:00:55,719 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:46227', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:55,717 - distributed.scheduler - INFO - Remove client Client-worker-79848d3e-3e25-11f1-8ed9-86acc56fd09f
2026-04-22 09:00:55,533 - distributed.scheduler - INFO - Close client connection: Client-worker-739453ef-3e21-11f1-8ed0-ee7435aaba1f
2026-04-22 09:00:55,520 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:37711', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:55,518 - distributed.scheduler - INFO - Remove client Client-worker-739453ef-3e21-11f1-8ed0-ee7435aaba1f
2026-04-22 09:00:54,818 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: ['tcp://172.21.25.29:41447'], ('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 15) NoneType: None
2026-04-22 09:00:54,813 - distributed.scheduler - ERROR - Couldn't gather keys {"('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 15)": ['tcp://172.21.25.29:41447']} state: ['waiting'] workers: ['tcp://172.21.25.29:41447'] NoneType: None
2026-04-22 09:00:54,810 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:41447', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:53,799 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 5) NoneType: None
2026-04-22 09:00:53,799 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 7) NoneType: None
2026-04-22 09:00:53,799 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 4) NoneType: None
2026-04-22 09:00:53,799 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 0) NoneType: None
2026-04-22 09:00:53,799 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 1) NoneType: None
2026-04-22 09:00:53,798 - distributed.scheduler - ERROR - Couldn't gather keys {"('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 1)": [], "('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 0)": [], "('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 4)": [], "('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 7)": [], "('query-9194dd9074c8e3d9d2f0ada71a35fa0b', 5)": []} state: ['waiting', 'processing', 'waiting', 'memory', 'waiting'] workers: [] NoneType: None
2026-04-22 09:00:53,149 - distributed.scheduler - INFO - Receive client connection: Client-worker-c431968b-3e29-11f1-8ef0-ae00acbfc48e
2026-04-22 09:00:52,824 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:40055
2026-04-22 09:00:52,823 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:40055', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:52,723 - distributed.scheduler - INFO - Receive client connection: Client-worker-c3e6adb9-3e29-11f1-8eed-b6f02dd406e9
2026-04-22 09:00:52,269 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:35541
2026-04-22 09:00:52,269 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:35541', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:52,190 - distributed.scheduler - INFO - Receive client connection: Client-worker-c39ef368-3e29-11f1-8f14-2eb33e86e1bd
2026-04-22 09:00:52,134 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:35691
2026-04-22 09:00:52,133 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:35691', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:52,052 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:42181
2026-04-22 09:00:52,051 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:42181', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:52,046 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:38415
2026-04-22 09:00:52,045 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:38415', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:52,040 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:42667
2026-04-22 09:00:52,039 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:42667', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:52,035 - distributed.scheduler - INFO - Receive client connection: Client-worker-c386f87f-3e29-11f1-8eba-6ec63a5ab21e
2026-04-22 09:00:51,859 - distributed.scheduler - INFO - Close client connection: Client-worker-7a1be5fa-3e25-11f1-9004-f28f846166bd
2026-04-22 09:00:51,854 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:44291', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:51,828 - distributed.scheduler - INFO - Remove client Client-worker-7a1be5fa-3e25-11f1-9004-f28f846166bd
2026-04-22 09:00:51,574 - distributed.scheduler - INFO - Receive client connection: Client-worker-c339e3ca-3e29-11f1-8ed2-566b4744c386
2026-04-22 09:00:51,199 - distributed.scheduler - INFO - Close client connection: Client-worker-99822a3e-3e25-11f1-8e62-7e353d94dc10
2026-04-22 09:00:51,189 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:37725', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:51,155 - distributed.scheduler - INFO - Remove client Client-worker-99822a3e-3e25-11f1-8e62-7e353d94dc10
2026-04-22 09:00:51,106 - distributed.scheduler - INFO - Close client connection: Client-worker-696074d8-3e21-11f1-8e90-2a51d3dc178e
2026-04-22 09:00:51,101 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:42845', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:51,084 - distributed.scheduler - INFO - Remove client Client-worker-696074d8-3e21-11f1-8e90-2a51d3dc178e
2026-04-22 09:00:51,080 - distributed.scheduler - INFO - Close client connection: Client-worker-7a123dd2-3e25-11f1-8e9a-fae0fcb74df3
2026-04-22 09:00:51,070 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:34845', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:51,068 - distributed.scheduler - INFO - Remove client Client-worker-7a123dd2-3e25-11f1-8e9a-fae0fcb74df3
2026-04-22 09:00:51,057 - distributed.scheduler - INFO - Receive client connection: Client-worker-c2f26e55-3e29-11f1-8f96-4ef3162b8ac2
2026-04-22 09:00:50,998 - distributed.scheduler - INFO - Close client connection: Client-worker-ac9dcfeb-3e29-11f1-8e0f-0e4c34ff5e38
2026-04-22 09:00:50,992 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:44347', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:50,991 - distributed.scheduler - INFO - Remove client Client-worker-ac9dcfeb-3e29-11f1-8e0f-0e4c34ff5e38
2026-04-22 09:00:50,967 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:33391', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:50,352 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:38119
2026-04-22 09:00:50,350 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:38119', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:50,345 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:35675
2026-04-22 09:00:50,344 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:35675', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:50,118 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:33213
2026-04-22 09:00:50,117 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:33213', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:50,114 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:39675
2026-04-22 09:00:50,113 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:39675', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:50,111 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:38067
2026-04-22 09:00:50,109 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:38067', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:49,805 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:44855
2026-04-22 09:00:49,803 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:44855', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:49,277 - distributed.scheduler - INFO - Close client connection: Client-worker-79570d14-3e25-11f1-8eb1-ae00acbfc48e
2026-04-22 09:00:49,269 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:39637', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:49,269 - distributed.scheduler - INFO - Remove client Client-worker-79570d14-3e25-11f1-8eb1-ae00acbfc48e
2026-04-22 09:00:49,258 - distributed.scheduler - INFO - Close client connection: Client-worker-ad38713d-3e29-11f1-8ed3-2eb33e86e1bd
2026-04-22 09:00:49,249 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:44709', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:49,240 - distributed.scheduler - INFO - Remove client Client-worker-ad38713d-3e29-11f1-8ed3-2eb33e86e1bd
2026-04-22 09:00:49,123 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:41129', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:49,112 - distributed.scheduler - INFO - Close client connection: Client-worker-ad4913ef-3e29-11f1-8eb1-b6f02dd406e9
2026-04-22 09:00:49,106 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:40567', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:49,104 - distributed.scheduler - INFO - Remove client Client-worker-ad4913ef-3e29-11f1-8eb1-b6f02dd406e9
2026-04-22 09:00:48,971 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:39409', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:48,941 - distributed.scheduler - INFO - Close client connection: Client-worker-6ea112ce-3e21-11f1-8e84-566b4744c386
2026-04-22 09:00:48,914 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:45045', status: closed, memory: 0, processing: 0>
2026-04-22 09:00:48,913 - distributed.scheduler - INFO - Remove client Client-worker-6ea112ce-3e21-11f1-8e84-566b4744c386
2026-04-22 09:00:14,776 - distributed.scheduler - INFO - Receive client connection: Client-worker-ad4913ef-3e29-11f1-8eb1-b6f02dd406e9
2026-04-22 09:00:14,667 - distributed.scheduler - INFO - Receive client connection: Client-worker-ad38713d-3e29-11f1-8ed3-2eb33e86e1bd
2026-04-22 09:00:13,591 - distributed.scheduler - INFO - Receive client connection: Client-worker-ac9dcfeb-3e29-11f1-8e0f-0e4c34ff5e38
2026-04-22 09:00:09,392 - distributed.scheduler - INFO - Receive client connection: Client-aa1a3252-3e29-11f1-8001-daa6852bd5a7
2026-04-22 08:50:49,824 - distributed.scheduler - INFO - Close client connection: Client-006d0c7f-3e20-11f1-8001-daa6852bd5a7
2026-04-22 08:50:49,824 - distributed.scheduler - INFO - Close client connection: Client-430fc6e4-3e21-11f1-8001-daa6852bd5a7
2026-04-22 08:50:49,823 - distributed.scheduler - INFO - Close client connection: Client-fdb05cf9-3e22-11f1-8001-daa6852bd5a7
2026-04-22 08:50:49,823 - distributed.scheduler - INFO - Close client connection: Client-1018d22b-3e24-11f1-8001-daa6852bd5a7
2026-04-22 08:50:49,823 - distributed.scheduler - INFO - Close client connection: Client-76a30fa6-3e25-11f1-8001-daa6852bd5a7
2026-04-22 08:50:49,822 - distributed.scheduler - INFO - Close client connection: Client-dd27a42c-3e26-11f1-8001-daa6852bd5a7
2026-04-22 08:50:49,820 - distributed.scheduler - INFO - Remove client Client-006d0c7f-3e20-11f1-8001-daa6852bd5a7
2026-04-22 08:50:49,819 - distributed.scheduler - INFO - Remove client Client-430fc6e4-3e21-11f1-8001-daa6852bd5a7
2026-04-22 08:50:49,816 - distributed.scheduler - INFO - Remove client Client-fdb05cf9-3e22-11f1-8001-daa6852bd5a7
2026-04-22 08:50:49,815 - distributed.scheduler - INFO - Remove client Client-1018d22b-3e24-11f1-8001-daa6852bd5a7
2026-04-22 08:50:49,814 - distributed.scheduler - INFO - Remove client Client-76a30fa6-3e25-11f1-8001-daa6852bd5a7
2026-04-22 08:50:49,813 - distributed.scheduler - INFO - Remove client Client-dd27a42c-3e26-11f1-8001-daa6852bd5a7
2026-04-22 08:50:48,978 - distributed.scheduler - INFO - Close client connection: Client-43a72fd7-3e28-11f1-8001-daa6852bd5a7
2026-04-22 08:50:48,976 - distributed.scheduler - INFO - Remove client Client-43a72fd7-3e28-11f1-8001-daa6852bd5a7
2026-04-22 08:50:48,974 - distributed.scheduler - INFO - Remove client Client-43a72fd7-3e28-11f1-8001-daa6852bd5a7
2026-04-22 08:50:08,001 - distributed.scheduler - INFO - Receive client connection: Client-43a72fd7-3e28-11f1-8001-daa6852bd5a7
2026-04-22 08:40:06,545 - distributed.scheduler - INFO - Receive client connection: Client-dd27a42c-3e26-11f1-8001-daa6852bd5a7
2026-04-22 08:31:03,552 - distributed.scheduler - INFO - Receive client connection: Client-worker-99822a3e-3e25-11f1-8e62-7e353d94dc10
2026-04-22 08:30:10,911 - distributed.scheduler - INFO - Receive client connection: Client-worker-7a1be5fa-3e25-11f1-9004-f28f846166bd
2026-04-22 08:30:10,910 - distributed.scheduler - INFO - Receive client connection: Client-worker-7a123dd2-3e25-11f1-8e9a-fae0fcb74df3
2026-04-22 08:30:10,204 - distributed.scheduler - INFO - Receive client connection: Client-worker-79a6903c-3e25-11f1-8eb2-2e20a4008324
2026-04-22 08:30:09,875 - distributed.scheduler - INFO - Receive client connection: Client-worker-79848d3e-3e25-11f1-8ed9-86acc56fd09f
2026-04-22 08:30:09,578 - distributed.scheduler - INFO - Receive client connection: Client-worker-79570d14-3e25-11f1-8eb1-ae00acbfc48e
2026-04-22 08:30:05,049 - distributed.scheduler - INFO - Receive client connection: Client-76a30fa6-3e25-11f1-8001-daa6852bd5a7
2026-04-22 08:20:03,522 - distributed.scheduler - INFO - Receive client connection: Client-1018d22b-3e24-11f1-8001-daa6852bd5a7
2026-04-22 08:12:23,137 - distributed.scheduler - INFO - Receive client connection: Client-fdb05cf9-3e22-11f1-8001-daa6852bd5a7
2026-04-22 08:01:21,977 - distributed.scheduler - INFO - Receive client connection: Client-worker-739453ef-3e21-11f1-8ed0-ee7435aaba1f
2026-04-22 08:01:13,647 - distributed.scheduler - INFO - Receive client connection: Client-worker-6ea112ce-3e21-11f1-8e84-566b4744c386
2026-04-22 08:01:05,195 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:46227
2026-04-22 08:01:05,194 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:46227', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:04,923 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:37711
2026-04-22 08:01:04,922 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:37711', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:04,879 - distributed.scheduler - INFO - Receive client connection: Client-worker-696074d8-3e21-11f1-8e90-2a51d3dc178e
2026-04-22 08:01:04,837 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:41447
2026-04-22 08:01:04,835 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:41447', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:04,011 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:42707', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:03,803 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:44051', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:03,692 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:46695', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,925 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:42845
2026-04-22 08:01:01,924 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:42845', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,813 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:44709
2026-04-22 08:01:01,812 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:44709', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,720 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:44291
2026-04-22 08:01:01,719 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:44291', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,401 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:39637
2026-04-22 08:01:01,400 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:39637', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,393 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:39409
2026-04-22 08:01:01,392 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:39409', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,384 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:33391
2026-04-22 08:01:01,383 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:33391', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,368 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:37725
2026-04-22 08:01:01,367 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:37725', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,364 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:44347
2026-04-22 08:01:01,363 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:44347', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,305 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:41129
2026-04-22 08:01:01,304 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:41129', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,248 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:34845
2026-04-22 08:01:01,247 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:34845', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,200 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:44527
2026-04-22 08:01:01,200 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:44527', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,178 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:45045
2026-04-22 08:01:01,177 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:45045', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:01,152 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:40567
2026-04-22 08:01:01,151 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:40567', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,889 - distributed.scheduler - INFO - Close client connection: Client-worker-45c51959-3e21-11f1-8e94-2eb33e86e1bd
2026-04-22 08:01:00,887 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:40051', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,886 - distributed.scheduler - INFO - Remove client Client-worker-45c51959-3e21-11f1-8e94-2eb33e86e1bd
2026-04-22 08:01:00,879 - distributed.scheduler - INFO - Close client connection: Client-worker-4634c359-3e21-11f1-8e4d-2a51d3dc178e
2026-04-22 08:01:00,873 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:42275', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,873 - distributed.scheduler - INFO - Remove client Client-worker-4634c359-3e21-11f1-8e4d-2a51d3dc178e
2026-04-22 08:01:00,771 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:34793', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,439 - distributed.scheduler - INFO - Close client connection: Client-worker-45908146-3e21-11f1-8e24-7e353d94dc10
2026-04-22 08:01:00,434 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:41999', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,433 - distributed.scheduler - INFO - Remove client Client-worker-45908146-3e21-11f1-8e24-7e353d94dc10
2026-04-22 08:01:00,380 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:41905', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,372 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:43797', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,358 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:43043', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,334 - distributed.scheduler - INFO - Close client connection: Client-worker-45db18dd-3e21-11f1-8f20-4ef3162b8ac2
2026-04-22 08:01:00,322 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:35901', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,320 - distributed.scheduler - INFO - Remove client Client-worker-45db18dd-3e21-11f1-8f20-4ef3162b8ac2
2026-04-22 08:01:00,279 - distributed.scheduler - INFO - Close client connection: Client-worker-45c6fb5a-3e21-11f1-8dce-0e4c34ff5e38
2026-04-22 08:01:00,276 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:45851', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,275 - distributed.scheduler - INFO - Remove client Client-worker-45c6fb5a-3e21-11f1-8dce-0e4c34ff5e38
2026-04-22 08:01:00,197 - distributed.scheduler - INFO - Close client connection: Client-worker-46203f96-3e21-11f1-8e77-b6f02dd406e9
2026-04-22 08:01:00,196 - distributed.scheduler - INFO - Close client connection: Client-worker-46165c08-3e21-11f1-8e58-fae0fcb74df3
2026-04-22 08:01:00,194 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:42957', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,194 - distributed.scheduler - INFO - Remove client Client-worker-46203f96-3e21-11f1-8e77-b6f02dd406e9
2026-04-22 08:01:00,190 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:44711', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,169 - distributed.scheduler - INFO - Remove client Client-worker-46165c08-3e21-11f1-8e58-fae0fcb74df3
2026-04-22 08:01:00,139 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:43509', status: closed, memory: 0, processing: 0>
2026-04-22 08:01:00,131 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:40399', status: closed, memory: 0, processing: 0>
2026-04-22 08:00:05,803 - distributed.scheduler - INFO - Receive client connection: Client-worker-4634c359-3e21-11f1-8e4d-2a51d3dc178e
2026-04-22 08:00:05,683 - distributed.scheduler - INFO - Receive client connection: Client-worker-46203f96-3e21-11f1-8e77-b6f02dd406e9
2026-04-22 08:00:05,601 - distributed.scheduler - INFO - Receive client connection: Client-worker-46165c08-3e21-11f1-8e58-fae0fcb74df3
2026-04-22 08:00:05,227 - distributed.scheduler - INFO - Receive client connection: Client-worker-45db18dd-3e21-11f1-8f20-4ef3162b8ac2
2026-04-22 08:00:05,090 - distributed.scheduler - INFO - Receive client connection: Client-worker-45c6fb5a-3e21-11f1-8dce-0e4c34ff5e38
2026-04-22 08:00:05,083 - distributed.scheduler - INFO - Receive client connection: Client-worker-45c51959-3e21-11f1-8e94-2eb33e86e1bd
2026-04-22 08:00:04,726 - distributed.scheduler - INFO - Receive client connection: Client-worker-45908146-3e21-11f1-8e24-7e353d94dc10
2026-04-22 08:00:00,537 - distributed.scheduler - INFO - Receive client connection: Client-430fc6e4-3e21-11f1-8001-daa6852bd5a7
2026-04-22 07:50:59,254 - distributed.scheduler - INFO - Receive client connection: Client-006d0c7f-3e20-11f1-8001-daa6852bd5a7
2026-04-22 07:41:35,745 - distributed.scheduler - INFO - Close client connection: Client-99f0b7a8-3e1e-11f1-8001-daa6852bd5a7
2026-04-22 07:41:35,744 - distributed.scheduler - INFO - Remove client Client-99f0b7a8-3e1e-11f1-8001-daa6852bd5a7
2026-04-22 07:41:35,742 - distributed.scheduler - INFO - Remove client Client-99f0b7a8-3e1e-11f1-8001-daa6852bd5a7
2026-04-22 07:40:57,813 - distributed.scheduler - INFO - Receive client connection: Client-99f0b7a8-3e1e-11f1-8001-daa6852bd5a7
2026-04-22 07:32:34,599 - distributed.scheduler - INFO - Close client connection: Client-33702eb2-3e1d-11f1-8001-daa6852bd5a7
2026-04-22 07:32:34,598 - distributed.scheduler - INFO - Remove client Client-33702eb2-3e1d-11f1-8001-daa6852bd5a7
2026-04-22 07:32:34,596 - distributed.scheduler - INFO - Remove client Client-33702eb2-3e1d-11f1-8001-daa6852bd5a7
2026-04-22 07:31:36,934 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:46695
2026-04-22 07:31:36,933 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:46695', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:36,892 - distributed.scheduler - INFO - Receive client connection: Client-4b9b8742-3e1d-11f1-8007-924c5ad31268
2026-04-22 07:31:36,877 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:40399
2026-04-22 07:31:36,876 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:40399', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:36,726 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:44051
2026-04-22 07:31:36,725 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:44051', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:35,866 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:43527', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:35,840 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: ['tcp://172.21.25.32:34455'], ('query-4db47129767f6fe48c7be2865a352f1a', 9) NoneType: None
2026-04-22 07:31:35,840 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('query-4db47129767f6fe48c7be2865a352f1a', 10) NoneType: None
2026-04-22 07:31:35,840 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('query-4db47129767f6fe48c7be2865a352f1a', 8) NoneType: None
2026-04-22 07:31:35,839 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('query-4db47129767f6fe48c7be2865a352f1a', 4) NoneType: None
2026-04-22 07:31:35,839 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: ['tcp://172.21.25.32:34455'], ('query-4db47129767f6fe48c7be2865a352f1a', 12) NoneType: None
2026-04-22 07:31:35,837 - distributed.scheduler - ERROR - Couldn't gather keys {"('query-4db47129767f6fe48c7be2865a352f1a', 12)": ['tcp://172.21.25.32:34455'], "('query-4db47129767f6fe48c7be2865a352f1a', 4)": [], "('query-4db47129767f6fe48c7be2865a352f1a', 8)": [], "('query-4db47129767f6fe48c7be2865a352f1a', 10)": [], "('query-4db47129767f6fe48c7be2865a352f1a', 9)": ['tcp://172.21.25.32:34455']} state: ['waiting', 'processing', 'waiting', 'processing', 'waiting'] workers: ['tcp://172.21.25.32:34455'] NoneType: None
2026-04-22 07:31:35,836 - distributed.scheduler - INFO - Close client connection: Client-worker-02d06288-3e19-11f1-8e38-2e20a4008324
2026-04-22 07:31:35,822 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:34455', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:35,822 - distributed.scheduler - INFO - Remove client Client-worker-02d06288-3e19-11f1-8e38-2e20a4008324
2026-04-22 07:31:35,617 - distributed.scheduler - INFO - Close client connection: Client-worker-0278e694-3e19-11f1-8e50-ee7435aaba1f
2026-04-22 07:31:35,611 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:45081', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:35,609 - distributed.scheduler - INFO - Remove client Client-worker-0278e694-3e19-11f1-8e50-ee7435aaba1f
2026-04-22 07:31:35,315 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:34793
2026-04-22 07:31:35,313 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:34793', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:35,037 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:41905
2026-04-22 07:31:35,036 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:41905', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,995 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:43043
2026-04-22 07:31:34,993 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:43043', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,962 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:45851
2026-04-22 07:31:34,960 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:45851', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,934 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:42275
2026-04-22 07:31:34,933 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:42275', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,869 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:41999
2026-04-22 07:31:34,868 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:41999', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,783 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:44711
2026-04-22 07:31:34,781 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:44711', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,760 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:42957
2026-04-22 07:31:34,758 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:42957', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,747 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:40051
2026-04-22 07:31:34,746 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:40051', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,729 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:42707
2026-04-22 07:31:34,728 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:42707', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,726 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:43797
2026-04-22 07:31:34,725 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:43797', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,698 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:35901
2026-04-22 07:31:34,697 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:35901', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,676 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:43509
2026-04-22 07:31:34,674 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:43509', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,431 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:32941', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,430 - distributed.scheduler - INFO - Close client connection: Client-worker-cef67a2b-3e14-11f1-8f85-f28f846166bd
2026-04-22 07:31:34,429 - distributed.scheduler - INFO - Remove client Client-worker-cef67a2b-3e14-11f1-8f85-f28f846166bd
2026-04-22 07:31:34,213 - distributed.scheduler - INFO - Close client connection: Client-e4ba9517-3dff-11f1-8007-924c5ad31268
2026-04-22 07:31:34,212 - distributed.scheduler - INFO - Remove client Client-e4ba9517-3dff-11f1-8007-924c5ad31268
2026-04-22 07:31:34,211 - distributed.scheduler - INFO - Remove client Client-e4ba9517-3dff-11f1-8007-924c5ad31268
2026-04-22 07:31:34,193 - distributed.scheduler - INFO - Close client connection: Client-worker-cf853320-3e14-11f1-8e39-ae00acbfc48e
2026-04-22 07:31:34,188 - distributed.scheduler - INFO - Task run-fbad1848d1ff3d5a571bc283b409ba5a marked as failed because 3 workers died while trying to run it
2026-04-22 07:31:34,187 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:40315', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,157 - distributed.scheduler - INFO - Remove client Client-worker-cf853320-3e14-11f1-8e39-ae00acbfc48e
2026-04-22 07:31:34,060 - distributed.scheduler - INFO - Close client connection: Client-worker-ceeffd8e-3e14-11f1-8db6-8288bfee4388
2026-04-22 07:31:34,048 - distributed.scheduler - INFO - Task ('sjoin-f7e724a9485d517e60cbe0de77064215', 66) marked as failed because 3 workers died while trying to run it
2026-04-22 07:31:34,047 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:37817', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:34,046 - distributed.scheduler - INFO - Remove client Client-worker-ceeffd8e-3e14-11f1-8db6-8288bfee4388
2026-04-22 07:31:33,962 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:39641', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:33,953 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:40645', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:33,953 - distributed.scheduler - INFO - Close client connection: Client-worker-029a6d9d-3e19-11f1-8ddb-7e353d94dc10
2026-04-22 07:31:33,952 - distributed.scheduler - INFO - Remove client Client-worker-029a6d9d-3e19-11f1-8ddb-7e353d94dc10
2026-04-22 07:31:33,930 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:38357', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:33,894 - distributed.scheduler - INFO - Close client connection: Client-worker-cf0dfb62-3e14-11f1-8e35-b6f02dd406e9
2026-04-22 07:31:33,885 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:40389', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:33,884 - distributed.scheduler - INFO - Remove client Client-worker-cf0dfb62-3e14-11f1-8e35-b6f02dd406e9
2026-04-22 07:31:33,880 - distributed.scheduler - INFO - Close client connection: Client-worker-cf7b5c66-3e14-11f1-8e0e-fae0fcb74df3
2026-04-22 07:31:33,866 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:34981', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:33,865 - distributed.scheduler - INFO - Remove client Client-worker-cf7b5c66-3e14-11f1-8e0e-fae0fcb74df3
2026-04-22 07:31:33,824 - distributed.scheduler - INFO - Close client connection: Client-worker-cf4a0f45-3e14-11f1-8e0a-6ec63a5ab21e
2026-04-22 07:31:33,821 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:38153', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:33,820 - distributed.scheduler - INFO - Remove client Client-worker-cf4a0f45-3e14-11f1-8e0a-6ec63a5ab21e
2026-04-22 07:31:33,791 - distributed.scheduler - INFO - Close client connection: Client-worker-03030e8e-3e19-11f1-8e52-2eb33e86e1bd
2026-04-22 07:31:33,784 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:41179', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:33,767 - distributed.scheduler - INFO - Remove client Client-worker-03030e8e-3e19-11f1-8e52-2eb33e86e1bd
2026-04-22 07:31:33,757 - distributed.scheduler - INFO - Close client connection: Client-worker-c134982b-3e10-11f1-8ede-4ef3162b8ac2
2026-04-22 07:31:33,754 - distributed.scheduler - INFO - Close client connection: Client-worker-ced249d8-3e14-11f1-8e35-566b4744c386
2026-04-22 07:31:33,752 - distributed.scheduler - INFO - Remove client Client-worker-c134982b-3e10-11f1-8ede-4ef3162b8ac2
2026-04-22 07:31:33,749 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:44519', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:33,744 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:35419', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:33,713 - distributed.scheduler - INFO - Remove client Client-worker-ced249d8-3e14-11f1-8e35-566b4744c386
2026-04-22 07:31:33,619 - distributed.scheduler - INFO - Close client connection: Client-worker-364d645c-3e1d-11f1-8e64-86acc56fd09f
2026-04-22 07:31:33,603 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:45455', status: closed, memory: 0, processing: 0>
2026-04-22 07:31:33,600 - distributed.scheduler - INFO - Remove client Client-worker-364d645c-3e1d-11f1-8e64-86acc56fd09f
2026-04-22 07:31:01,183 - distributed.scheduler - INFO - Receive client connection: Client-worker-364d645c-3e1d-11f1-8e64-86acc56fd09f
2026-04-22 07:30:56,345 - distributed.scheduler - INFO - Receive client connection: Client-33702eb2-3e1d-11f1-8001-daa6852bd5a7
2026-04-22 07:21:35,062 - distributed.scheduler - INFO - Close client connection: Client-ccebacf5-3e1b-11f1-8001-daa6852bd5a7
2026-04-22 07:21:35,061 - distributed.scheduler - INFO - Remove client Client-ccebacf5-3e1b-11f1-8001-daa6852bd5a7
2026-04-22 07:21:35,059 - distributed.scheduler - INFO - Remove client Client-ccebacf5-3e1b-11f1-8001-daa6852bd5a7
2026-04-22 07:20:54,851 - distributed.scheduler - INFO - Receive client connection: Client-ccebacf5-3e1b-11f1-8001-daa6852bd5a7
2026-04-22 07:13:56,721 - distributed.scheduler - INFO - Close client connection: Client-bff22687-3e1a-11f1-8001-daa6852bd5a7
2026-04-22 07:13:56,719 - distributed.scheduler - INFO - Remove client Client-bff22687-3e1a-11f1-8001-daa6852bd5a7
2026-04-22 07:13:56,718 - distributed.scheduler - INFO - Remove client Client-bff22687-3e1a-11f1-8001-daa6852bd5a7
2026-04-22 07:13:23,589 - distributed.scheduler - INFO - Receive client connection: Client-bff22687-3e1a-11f1-8001-daa6852bd5a7
2026-04-22 07:02:02,177 - distributed.scheduler - INFO - Close client connection: Client-fff0d1e2-3e18-11f1-8001-daa6852bd5a7
2026-04-22 07:02:02,176 - distributed.scheduler - INFO - Remove client Client-fff0d1e2-3e18-11f1-8001-daa6852bd5a7
2026-04-22 07:02:02,174 - distributed.scheduler - INFO - Remove client Client-fff0d1e2-3e18-11f1-8001-daa6852bd5a7
2026-04-22 07:00:57,124 - distributed.scheduler - INFO - Receive client connection: Client-worker-03030e8e-3e19-11f1-8e52-2eb33e86e1bd
2026-04-22 07:00:56,905 - distributed.scheduler - INFO - Receive client connection: Client-worker-02d06288-3e19-11f1-8e38-2e20a4008324
2026-04-22 07:00:56,433 - distributed.scheduler - INFO - Receive client connection: Client-worker-029a6d9d-3e19-11f1-8ddb-7e353d94dc10
2026-04-22 07:00:56,188 - distributed.scheduler - INFO - Receive client connection: Client-worker-0278e694-3e19-11f1-8e50-ee7435aaba1f
2026-04-22 07:00:51,964 - distributed.scheduler - INFO - Receive client connection: Client-fff0d1e2-3e18-11f1-8001-daa6852bd5a7
2026-04-22 06:51:29,534 - distributed.scheduler - INFO - Close client connection: Client-65e1d4fc-3e13-11f1-8001-daa6852bd5a7
2026-04-22 06:51:29,534 - distributed.scheduler - INFO - Close client connection: Client-cc66711b-3e14-11f1-8001-daa6852bd5a7
2026-04-22 06:51:29,534 - distributed.scheduler - INFO - Close client connection: Client-5a963abc-3e12-11f1-8001-daa6852bd5a7
2026-04-22 06:51:29,533 - distributed.scheduler - INFO - Close client connection: Client-32ea720b-3e16-11f1-8001-daa6852bd5a7
2026-04-22 06:51:29,532 - distributed.scheduler - INFO - Remove client Client-65e1d4fc-3e13-11f1-8001-daa6852bd5a7
2026-04-22 06:51:29,531 - distributed.scheduler - INFO - Remove client Client-cc66711b-3e14-11f1-8001-daa6852bd5a7
2026-04-22 06:51:29,530 - distributed.scheduler - INFO - Remove client Client-5a963abc-3e12-11f1-8001-daa6852bd5a7
2026-04-22 06:51:29,529 - distributed.scheduler - INFO - Remove client Client-32ea720b-3e16-11f1-8001-daa6852bd5a7
2026-04-22 06:51:28,970 - distributed.scheduler - INFO - Close client connection: Client-996ac5bd-3e17-11f1-8001-daa6852bd5a7
2026-04-22 06:51:28,969 - distributed.scheduler - INFO - Remove client Client-996ac5bd-3e17-11f1-8001-daa6852bd5a7
2026-04-22 06:51:28,968 - distributed.scheduler - INFO - Remove client Client-996ac5bd-3e17-11f1-8001-daa6852bd5a7
2026-04-22 06:50:50,449 - distributed.scheduler - INFO - Receive client connection: Client-996ac5bd-3e17-11f1-8001-daa6852bd5a7
2026-04-22 06:40:48,984 - distributed.scheduler - INFO - Receive client connection: Client-32ea720b-3e16-11f1-8001-daa6852bd5a7
2026-04-22 06:30:52,747 - distributed.scheduler - INFO - Receive client connection: Client-worker-cf853320-3e14-11f1-8e39-ae00acbfc48e
2026-04-22 06:30:52,659 - distributed.scheduler - INFO - Receive client connection: Client-worker-cf7b5c66-3e14-11f1-8e0e-fae0fcb74df3
2026-04-22 06:30:52,331 - distributed.scheduler - INFO - Receive client connection: Client-worker-cf4a0f45-3e14-11f1-8e0a-6ec63a5ab21e
2026-04-22 06:30:51,936 - distributed.scheduler - INFO - Receive client connection: Client-worker-cf0dfb62-3e14-11f1-8e35-b6f02dd406e9
2026-04-22 06:30:51,791 - distributed.scheduler - INFO - Receive client connection: Client-worker-cef67a2b-3e14-11f1-8f85-f28f846166bd
2026-04-22 06:30:51,756 - distributed.scheduler - INFO - Receive client connection: Client-worker-ceeffd8e-3e14-11f1-8db6-8288bfee4388
2026-04-22 06:30:51,543 - distributed.scheduler - INFO - Receive client connection: Client-worker-ced249d8-3e14-11f1-8e35-566b4744c386
2026-04-22 06:30:47,497 - distributed.scheduler - INFO - Receive client connection: Client-cc66711b-3e14-11f1-8001-daa6852bd5a7
2026-04-22 06:20:46,001 - distributed.scheduler - INFO - Receive client connection: Client-65e1d4fc-3e13-11f1-8001-daa6852bd5a7
2026-04-22 06:13:17,560 - distributed.scheduler - INFO - Receive client connection: Client-5a963abc-3e12-11f1-8001-daa6852bd5a7
2026-04-22 06:01:54,256 - distributed.scheduler - INFO - Close client connection: Client-98e3420e-3e10-11f1-8001-daa6852bd5a7
2026-04-22 06:01:54,253 - distributed.scheduler - INFO - Remove client Client-98e3420e-3e10-11f1-8001-daa6852bd5a7
2026-04-22 06:01:54,252 - distributed.scheduler - INFO - Remove client Client-98e3420e-3e10-11f1-8001-daa6852bd5a7
2026-04-22 06:01:50,762 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:34455
2026-04-22 06:01:50,762 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:34455', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:50,729 - distributed.scheduler - INFO - Receive client connection: Client-worker-c134982b-3e10-11f1-8ede-4ef3162b8ac2
2026-04-22 06:01:50,701 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:45455
2026-04-22 06:01:50,699 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:45455', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:50,309 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:34981
2026-04-22 06:01:50,292 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:34981', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:50,225 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:37817
2026-04-22 06:01:50,224 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:37817', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:50,181 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:40645
2026-04-22 06:01:50,180 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:40645', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:50,169 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:39641
2026-04-22 06:01:50,167 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:39641', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:50,139 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:32941
2026-04-22 06:01:50,138 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:32941', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:50,122 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:40315
2026-04-22 06:01:50,121 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:40315', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:50,097 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:38153
2026-04-22 06:01:50,096 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:38153', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:50,072 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:41179
2026-04-22 06:01:50,071 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:41179', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:50,060 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:38357
2026-04-22 06:01:50,058 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:38357', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,970 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:45081
2026-04-22 06:01:49,968 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:45081', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,945 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:40389
2026-04-22 06:01:49,944 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:40389', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,939 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:43527
2026-04-22 06:01:49,938 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:43527', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,825 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:35419
2026-04-22 06:01:49,824 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:35419', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,759 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:44519
2026-04-22 06:01:49,757 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:44519', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,635 - distributed.scheduler - INFO - Close client connection: Client-worker-9b52821f-3e10-11f1-8e23-86acc56fd09f
2026-04-22 06:01:49,632 - distributed.scheduler - INFO - Lost all workers
2026-04-22 06:01:49,624 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:38819', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,623 - distributed.scheduler - INFO - Remove client Client-worker-9b52821f-3e10-11f1-8e23-86acc56fd09f
2026-04-22 06:01:49,616 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:33361', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,173 - distributed.scheduler - INFO - Task ('hash-join-0878ad0b59064ab99a6235ac6c7270e1', 15) marked as failed because 3 workers died while trying to run it
2026-04-22 06:01:49,173 - distributed.scheduler - INFO - Task ('hash-join-0878ad0b59064ab99a6235ac6c7270e1', 12) marked as failed because 3 workers died while trying to run it
2026-04-22 06:01:49,173 - distributed.scheduler - INFO - Task ('hash-join-0878ad0b59064ab99a6235ac6c7270e1', 5) marked as failed because 3 workers died while trying to run it
2026-04-22 06:01:49,172 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:41521', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,162 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:32909', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,155 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:42817', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,089 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:36195', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,087 - distributed.scheduler - INFO - Close client connection: Client-worker-9b29b4de-3e10-11f1-8dce-6ec63a5ab21e
2026-04-22 06:01:49,074 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:42741', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,073 - distributed.scheduler - INFO - Remove client Client-worker-9b29b4de-3e10-11f1-8dce-6ec63a5ab21e
2026-04-22 06:01:49,058 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:36311', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,038 - distributed.scheduler - INFO - Close client connection: Client-worker-9b68f7a5-3e10-11f1-8f44-f28f846166bd
2026-04-22 06:01:49,032 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:43537', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:49,031 - distributed.scheduler - INFO - Remove client Client-worker-9b68f7a5-3e10-11f1-8f44-f28f846166bd
2026-04-22 06:01:49,014 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:33439', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:48,997 - distributed.scheduler - INFO - Close client connection: Client-worker-9b30e21d-3e10-11f1-8dd3-2a51d3dc178e
2026-04-22 06:01:48,989 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:36159', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:48,988 - distributed.scheduler - INFO - Remove client Client-worker-9b30e21d-3e10-11f1-8dd3-2a51d3dc178e
2026-04-22 06:01:48,976 - distributed.scheduler - INFO - Close client connection: Client-worker-a62c7676-3e0c-11f1-8dfc-ae00acbfc48e
2026-04-22 06:01:48,970 - distributed.scheduler - INFO - Remove client Client-worker-a62c7676-3e0c-11f1-8dfc-ae00acbfc48e
2026-04-22 06:01:48,961 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:43057', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:48,921 - distributed.scheduler - INFO - Close client connection: Client-worker-9c181a48-3e10-11f1-8e0e-ee7435aaba1f
2026-04-22 06:01:48,914 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:42661', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:48,913 - distributed.scheduler - INFO - Remove client Client-worker-9c181a48-3e10-11f1-8e0e-ee7435aaba1f
2026-04-22 06:01:48,890 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:41747', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:48,879 - distributed.scheduler - INFO - Close client connection: Client-worker-9b9c0085-3e10-11f1-8d94-c2349ba99f21
2026-04-22 06:01:48,874 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:33291', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:48,873 - distributed.scheduler - INFO - Remove client Client-worker-9b9c0085-3e10-11f1-8d94-c2349ba99f21
2026-04-22 06:01:48,832 - distributed.scheduler - INFO - Close client connection: Client-worker-9be65fdb-3e10-11f1-8ea5-4ef3162b8ac2
2026-04-22 06:01:48,821 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:35313', status: closed, memory: 0, processing: 0>
2026-04-22 06:01:48,819 - distributed.scheduler - INFO - Remove client Client-worker-9be65fdb-3e10-11f1-8ea5-4ef3162b8ac2
2026-04-22 06:00:48,473 - distributed.scheduler - INFO - Receive client connection: Client-worker-9c181a48-3e10-11f1-8e0e-ee7435aaba1f
2026-04-22 06:00:48,140 - distributed.scheduler - INFO - Receive client connection: Client-worker-9be65fdb-3e10-11f1-8ea5-4ef3162b8ac2
2026-04-22 06:00:47,653 - distributed.scheduler - INFO - Receive client connection: Client-worker-9b9c0085-3e10-11f1-8d94-c2349ba99f21
2026-04-22 06:00:47,322 - distributed.scheduler - INFO - Receive client connection: Client-worker-9b68f7a5-3e10-11f1-8f44-f28f846166bd
2026-04-22 06:00:47,185 - distributed.scheduler - INFO - Receive client connection: Client-worker-9b52821f-3e10-11f1-8e23-86acc56fd09f
2026-04-22 06:00:46,946 - distributed.scheduler - INFO - Receive client connection: Client-worker-9b30e21d-3e10-11f1-8dd3-2a51d3dc178e
2026-04-22 06:00:46,908 - distributed.scheduler - INFO - Receive client connection: Client-worker-9b29b4de-3e10-11f1-8dce-6ec63a5ab21e
2026-04-22 06:00:43,096 - distributed.scheduler - INFO - Receive client connection: Client-98e3420e-3e10-11f1-8001-daa6852bd5a7
2026-04-22 05:51:14,325 - distributed.scheduler - INFO - Close client connection: Client-326e287d-3e0f-11f1-8001-daa6852bd5a7
2026-04-22 05:51:14,322 - distributed.scheduler - INFO - Remove client Client-326e287d-3e0f-11f1-8001-daa6852bd5a7
2026-04-22 05:51:14,321 - distributed.scheduler - INFO - Remove client Client-326e287d-3e0f-11f1-8001-daa6852bd5a7
2026-04-22 05:50:41,707 - distributed.scheduler - INFO - Receive client connection: Client-326e287d-3e0f-11f1-8001-daa6852bd5a7
2026-04-22 05:41:13,733 - distributed.scheduler - INFO - Close client connection: Client-cbf7955d-3e0d-11f1-8001-daa6852bd5a7
2026-04-22 05:41:13,733 - distributed.scheduler - INFO - Remove client Client-cbf7955d-3e0d-11f1-8001-daa6852bd5a7
2026-04-22 05:41:13,732 - distributed.scheduler - INFO - Remove client Client-cbf7955d-3e0d-11f1-8001-daa6852bd5a7
2026-04-22 05:40:40,299 - distributed.scheduler - INFO - Receive client connection: Client-cbf7955d-3e0d-11f1-8001-daa6852bd5a7
2026-04-22 05:32:27,537 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:33291
2026-04-22 05:32:27,536 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:33291', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:27,385 - distributed.scheduler - INFO - Receive client connection: Client-worker-a62c7676-3e0c-11f1-8dfc-ae00acbfc48e
2026-04-22 05:32:27,369 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:38819
2026-04-22 05:32:27,368 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:38819', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:27,329 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:42661
2026-04-22 05:32:27,328 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:42661', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,555 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:33361
2026-04-22 05:32:26,554 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:33361', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,537 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:36311
2026-04-22 05:32:26,535 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:36311', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,528 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:33685', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,504 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:43537
2026-04-22 05:32:26,503 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:43537', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,501 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:32909
2026-04-22 05:32:26,500 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:32909', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,421 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:36159
2026-04-22 05:32:26,420 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:36159', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,323 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:42817
2026-04-22 05:32:26,322 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:42817', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,315 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:33439
2026-04-22 05:32:26,313 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:33439', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,312 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:41747
2026-04-22 05:32:26,310 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:41747', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,283 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:41521
2026-04-22 05:32:26,282 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:41521', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,275 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:42741
2026-04-22 05:32:26,274 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:42741', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,272 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:43057
2026-04-22 05:32:26,271 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:43057', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,263 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:36195
2026-04-22 05:32:26,261 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:36195', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,204 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:43385', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,184 - distributed.scheduler - INFO - Close client connection: Client-worker-4d254393-3e08-11f1-8dd7-86acc56fd09f
2026-04-22 05:32:26,180 - distributed.scheduler - INFO - Remove client Client-worker-4d254393-3e08-11f1-8dd7-86acc56fd09f
2026-04-22 05:32:26,179 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:40107', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:26,081 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:35313
2026-04-22 05:32:26,079 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:35313', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,516 - distributed.scheduler - INFO - Close client connection: Client-worker-7e6a5cf0-3e0c-11f1-8d8d-2a51d3dc178e
2026-04-22 05:32:25,514 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:40215', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,511 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:46537', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,479 - distributed.scheduler - INFO - Remove client Client-worker-7e6a5cf0-3e0c-11f1-8d8d-2a51d3dc178e
2026-04-22 05:32:25,471 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:45173', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,468 - distributed.scheduler - INFO - Close client connection: Client-worker-7ec710d9-3e0c-11f1-8d40-8288bfee4388
2026-04-22 05:32:25,466 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:37801', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,464 - distributed.scheduler - INFO - Remove client Client-worker-7ec710d9-3e0c-11f1-8d40-8288bfee4388
2026-04-22 05:32:25,450 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:46861', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,415 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:44113', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,376 - distributed.scheduler - INFO - Close client connection: Client-worker-7da3488e-3e0c-11f1-8d91-6ec63a5ab21e
2026-04-22 05:32:25,374 - distributed.scheduler - INFO - Remove client Client-worker-7da3488e-3e0c-11f1-8d91-6ec63a5ab21e
2026-04-22 05:32:25,373 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:38799', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,357 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:37659', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,351 - distributed.scheduler - INFO - Close client connection: Client-worker-7f313e74-3e0c-11f1-8d69-7e353d94dc10
2026-04-22 05:32:25,348 - distributed.scheduler - INFO - Remove client Client-worker-7f313e74-3e0c-11f1-8d69-7e353d94dc10
2026-04-22 05:32:25,345 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:44789', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,328 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:38201', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,323 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:39123', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,285 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:33997', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,126 - distributed.scheduler - INFO - Close client connection: Client-worker-7da6fdd2-3e0c-11f1-8e66-4ef3162b8ac2
2026-04-22 05:32:25,121 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:37721', status: closed, memory: 0, processing: 0>
2026-04-22 05:32:25,119 - distributed.scheduler - INFO - Remove client Client-worker-7da6fdd2-3e0c-11f1-8e66-4ef3162b8ac2
2026-04-22 05:32:21,264 - distributed.scheduler - INFO - Close client connection: Client-657e19b7-3e0c-11f1-8001-daa6852bd5a7
2026-04-22 05:32:21,261 - distributed.scheduler - INFO - Remove client Client-657e19b7-3e0c-11f1-8001-daa6852bd5a7
2026-04-22 05:32:21,260 - distributed.scheduler - INFO - Remove client Client-657e19b7-3e0c-11f1-8001-daa6852bd5a7
2026-04-22 05:31:51,046 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-513baa04b7682687bc326531ea747614 NoneType: None
2026-04-22 05:31:51,045 - distributed.scheduler - ERROR - Couldn't gather keys {'run-513baa04b7682687bc326531ea747614': []} state: ['processing'] workers: [] NoneType: None
2026-04-22 05:31:22,519 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('len-agg-332d61b7798eae49e463c4f55df7aa1b', 0) NoneType: None
2026-04-22 05:31:22,519 - distributed.scheduler - ERROR - Couldn't gather keys {"('len-agg-332d61b7798eae49e463c4f55df7aa1b', 0)": []} state: ['waiting'] workers: [] NoneType: None
2026-04-22 05:31:22,010 - distributed.scheduler - INFO - Receive client connection: Client-worker-7f313e74-3e0c-11f1-8d69-7e353d94dc10
2026-04-22 05:31:21,373 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:40215
2026-04-22 05:31:21,373 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:40215', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:21,312 - distributed.scheduler - INFO - Receive client connection: Client-worker-7ec710d9-3e0c-11f1-8d40-8288bfee4388
2026-04-22 05:31:20,688 - distributed.scheduler - INFO - Receive client connection: Client-worker-7e6a5cf0-3e0c-11f1-8d8d-2a51d3dc178e
2026-04-22 05:31:20,396 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:46861
2026-04-22 05:31:20,395 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:46861', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:20,310 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:42447', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:20,171 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:43385
2026-04-22 05:31:20,169 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:43385', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:20,167 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:45173
2026-04-22 05:31:20,165 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:45173', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:20,162 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:33997
2026-04-22 05:31:20,161 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:33997', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:19,899 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:44113
2026-04-22 05:31:19,893 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:44113', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:19,669 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:37659
2026-04-22 05:31:19,668 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:37659', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:19,625 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:38201
2026-04-22 05:31:19,624 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:38201', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:19,426 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:33685
2026-04-22 05:31:19,425 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:33685', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:19,411 - distributed.scheduler - INFO - Receive client connection: Client-worker-7da6fdd2-3e0c-11f1-8e66-4ef3162b8ac2
2026-04-22 05:31:19,402 - distributed.scheduler - INFO - Receive client connection: Client-worker-7da3488e-3e0c-11f1-8d91-6ec63a5ab21e
2026-04-22 05:31:19,393 - distributed.scheduler - INFO - Close client connection: Client-worker-68ab6b42-3e0c-11f1-8da1-2eb33e86e1bd
2026-04-22 05:31:19,386 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:36633', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:19,372 - distributed.scheduler - INFO - Remove client Client-worker-68ab6b42-3e0c-11f1-8da1-2eb33e86e1bd
2026-04-22 05:31:19,181 - distributed.scheduler - INFO - Close client connection: Client-worker-6828524a-3e0c-11f1-8ce5-0e4c34ff5e38
2026-04-22 05:31:19,173 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:35193', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:19,172 - distributed.scheduler - INFO - Remove client Client-worker-6828524a-3e0c-11f1-8ce5-0e4c34ff5e38
2026-04-22 05:31:19,123 - distributed.scheduler - INFO - Close client connection: Client-worker-4da79f9b-3e08-11f1-8d94-ee7435aaba1f
2026-04-22 05:31:19,116 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:38343', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:19,114 - distributed.scheduler - INFO - Remove client Client-worker-4da79f9b-3e08-11f1-8d94-ee7435aaba1f
2026-04-22 05:31:19,063 - distributed.scheduler - INFO - Close client connection: Client-worker-6826f3ea-3e0c-11f1-8ecb-f28f846166bd
2026-04-22 05:31:19,063 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:44789
2026-04-22 05:31:19,062 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:44789', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:19,059 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:34819', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:19,037 - distributed.scheduler - INFO - Remove client Client-worker-6826f3ea-3e0c-11f1-8ecb-f28f846166bd
2026-04-22 05:31:19,022 - distributed.scheduler - INFO - Close client connection: Client-worker-4dcfc950-3e08-11f1-8d8a-566b4744c386
2026-04-22 05:31:19,009 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:46001', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:19,008 - distributed.scheduler - INFO - Remove client Client-worker-4dcfc950-3e08-11f1-8d8a-566b4744c386
2026-04-22 05:31:18,709 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:46537
2026-04-22 05:31:18,708 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:46537', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:18,657 - distributed.scheduler - INFO - Close client connection: Client-worker-6806a2d6-3e0c-11f1-8d87-b6f02dd406e9
2026-04-22 05:31:18,637 - distributed.scheduler - INFO - Task ('sjoin-fa0424d783adbfed9fca4e1397719f19', 165) marked as failed because 3 workers died while trying to run it
2026-04-22 05:31:18,636 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:38387', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:18,635 - distributed.scheduler - INFO - Remove client Client-worker-6806a2d6-3e0c-11f1-8d87-b6f02dd406e9
2026-04-22 05:31:18,598 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:38799
2026-04-22 05:31:18,597 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:38799', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:18,589 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:37801
2026-04-22 05:31:18,588 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:37801', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:18,559 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:41105', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:18,309 - distributed.scheduler - INFO - Close client connection: Client-worker-4d087fa3-3e08-11f1-8d19-c2349ba99f21
2026-04-22 05:31:18,284 - distributed.scheduler - INFO - Remove client Client-worker-4d087fa3-3e08-11f1-8d19-c2349ba99f21
2026-04-22 05:31:18,265 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:34465', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:18,247 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:37721
2026-04-22 05:31:18,245 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:37721', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:17,978 - distributed.scheduler - INFO - Close client connection: Client-worker-4e8d559f-3e08-11f1-8d2c-7e353d94dc10
2026-04-22 05:31:17,961 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:36479', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:17,960 - distributed.scheduler - INFO - Remove client Client-worker-4e8d559f-3e08-11f1-8d2c-7e353d94dc10
2026-04-22 05:31:17,563 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:38747', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:17,485 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:36827', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:17,481 - distributed.scheduler - INFO - Close client connection: Client-worker-4dfdc6b7-3e08-11f1-8cfe-8288bfee4388
2026-04-22 05:31:17,464 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:38775', status: closed, memory: 0, processing: 0>
2026-04-22 05:31:17,461 - distributed.scheduler - INFO - Remove client Client-worker-4dfdc6b7-3e08-11f1-8cfe-8288bfee4388
2026-04-22 05:31:17,378 - distributed.scheduler - INFO - Close client connection: Client-worker-4d08a1f5-3e08-11f1-8e29-4ef3162b8ac2
2026-04-22 05:31:17,358 - distributed.scheduler - INFO - Remove client Client-worker-4d08a1f5-3e08-11f1-8e29-4ef3162b8ac2
2026-04-22 05:31:17,325 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:40379', status: closed, memory: 0, processing: 0>
2026-04-22 05:30:44,281 - distributed.scheduler - INFO - Receive client connection: Client-worker-68ab6b42-3e0c-11f1-8da1-2eb33e86e1bd
2026-04-22 05:30:43,334 - distributed.scheduler - INFO - Receive client connection: Client-worker-6828524a-3e0c-11f1-8ce5-0e4c34ff5e38
2026-04-22 05:30:43,323 - distributed.scheduler - INFO - Receive client connection: Client-worker-6826f3ea-3e0c-11f1-8ecb-f28f846166bd
2026-04-22 05:30:43,117 - distributed.scheduler - INFO - Receive client connection: Client-worker-6806a2d6-3e0c-11f1-8d87-b6f02dd406e9
2026-04-22 05:30:38,880 - distributed.scheduler - INFO - Receive client connection: Client-657e19b7-3e0c-11f1-8001-daa6852bd5a7
2026-04-22 05:21:08,147 - distributed.scheduler - INFO - Close client connection: Client-ff02490e-3e0a-11f1-8001-daa6852bd5a7
2026-04-22 05:21:08,146 - distributed.scheduler - INFO - Remove client Client-ff02490e-3e0a-11f1-8001-daa6852bd5a7
2026-04-22 05:21:08,145 - distributed.scheduler - INFO - Remove client Client-ff02490e-3e0a-11f1-8001-daa6852bd5a7
2026-04-22 05:20:37,447 - distributed.scheduler - INFO - Receive client connection: Client-ff02490e-3e0a-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,400 - distributed.scheduler - INFO - Close client connection: Client-30c2ef74-3dfa-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,400 - distributed.scheduler - INFO - Close client connection: Client-1f6af356-3df9-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,399 - distributed.scheduler - INFO - Close client connection: Client-97468abf-3dfb-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,399 - distributed.scheduler - INFO - Close client connection: Client-fdcb942e-3dfc-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,399 - distributed.scheduler - INFO - Close client connection: Client-6453d236-3dfe-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,399 - distributed.scheduler - INFO - Close client connection: Client-cae14e6e-3dff-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,399 - distributed.scheduler - INFO - Close client connection: Client-75822cf9-3e01-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,398 - distributed.scheduler - INFO - Close client connection: Client-97dffd50-3e02-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,398 - distributed.scheduler - INFO - Close client connection: Client-fe610130-3e03-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,398 - distributed.scheduler - INFO - Close client connection: Client-64e753b6-3e05-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,398 - distributed.scheduler - INFO - Close client connection: Client-cb6fb7b2-3e06-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,397 - distributed.scheduler - INFO - Close client connection: Client-31f6d105-3e08-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,385 - distributed.scheduler - INFO - Remove client Client-30c2ef74-3dfa-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,384 - distributed.scheduler - INFO - Remove client Client-1f6af356-3df9-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,381 - distributed.scheduler - INFO - Remove client Client-97468abf-3dfb-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,380 - distributed.scheduler - INFO - Remove client Client-fdcb942e-3dfc-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,379 - distributed.scheduler - INFO - Remove client Client-6453d236-3dfe-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,378 - distributed.scheduler - INFO - Remove client Client-cae14e6e-3dff-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,377 - distributed.scheduler - INFO - Remove client Client-75822cf9-3e01-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,377 - distributed.scheduler - INFO - Remove client Client-97dffd50-3e02-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,375 - distributed.scheduler - INFO - Remove client Client-fe610130-3e03-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,374 - distributed.scheduler - INFO - Remove client Client-64e753b6-3e05-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,373 - distributed.scheduler - INFO - Remove client Client-cb6fb7b2-3e06-11f1-8001-daa6852bd5a7
2026-04-22 05:13:16,372 - distributed.scheduler - INFO - Remove client Client-31f6d105-3e08-11f1-8001-daa6852bd5a7
2026-04-22 05:13:15,808 - distributed.scheduler - INFO - Close client connection: Client-e7a4bdbd-3e09-11f1-8001-daa6852bd5a7
2026-04-22 05:13:15,807 - distributed.scheduler - INFO - Remove client Client-e7a4bdbd-3e09-11f1-8001-daa6852bd5a7
2026-04-22 05:13:15,806 - distributed.scheduler - INFO - Remove client Client-e7a4bdbd-3e09-11f1-8001-daa6852bd5a7
2026-04-22 05:12:48,734 - distributed.scheduler - INFO - Receive client connection: Client-e7a4bdbd-3e09-11f1-8001-daa6852bd5a7
2026-04-22 05:01:53,758 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-e7b9b9a922a92f3350130efe7864769e NoneType: None
2026-04-22 05:01:53,758 - distributed.scheduler - ERROR - Couldn't gather keys {'run-e7b9b9a922a92f3350130efe7864769e': []} state: ['processing'] workers: [] NoneType: None
2026-04-22 05:01:22,399 - distributed.scheduler - INFO - Receive client connection: Client-worker-4e8d559f-3e08-11f1-8d2c-7e353d94dc10
2026-04-22 05:01:21,466 - distributed.scheduler - INFO - Receive client connection: Client-worker-4dfdc6b7-3e08-11f1-8cfe-8288bfee4388
2026-04-22 05:01:21,163 - distributed.scheduler - INFO - Receive client connection: Client-worker-4dcfc950-3e08-11f1-8d8a-566b4744c386
2026-04-22 05:01:20,901 - distributed.scheduler - INFO - Receive client connection: Client-worker-4da79f9b-3e08-11f1-8d94-ee7435aaba1f
2026-04-22 05:01:20,033 - distributed.scheduler - INFO - Receive client connection: Client-worker-4d254393-3e08-11f1-8dd7-86acc56fd09f
2026-04-22 05:01:19,858 - distributed.scheduler - INFO - Receive client connection: Client-worker-4d08a1f5-3e08-11f1-8e29-4ef3162b8ac2
2026-04-22 05:01:19,841 - distributed.scheduler - INFO - Receive client connection: Client-worker-4d087fa3-3e08-11f1-8d19-c2349ba99f21
2026-04-22 05:01:19,754 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:42447
2026-04-22 05:01:19,752 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:42447', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,423 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:36827
2026-04-22 05:01:19,422 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:36827', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,420 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:39123
2026-04-22 05:01:19,419 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:39123', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,418 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:36479
2026-04-22 05:01:19,387 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:36479', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,379 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:34819
2026-04-22 05:01:19,378 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:34819', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,376 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:38747
2026-04-22 05:01:19,374 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:38747', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,266 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:35193
2026-04-22 05:01:19,266 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:35193', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,246 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:38387
2026-04-22 05:01:19,245 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:38387', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,243 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:41105
2026-04-22 05:01:19,241 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:41105', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,168 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:36633
2026-04-22 05:01:19,167 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:36633', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,165 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:40107
2026-04-22 05:01:19,164 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:40107', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,119 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:38775
2026-04-22 05:01:19,118 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:38775', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,109 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:34465
2026-04-22 05:01:19,109 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:34465', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:19,097 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:38343
2026-04-22 05:01:19,095 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:38343', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,930 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:40379
2026-04-22 05:01:18,929 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:40379', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,902 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:46001
2026-04-22 05:01:18,900 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:46001', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,642 - distributed.scheduler - INFO - Close client connection: Client-worker-3492e2f8-3e08-11f1-8d50-2e20a4008324
2026-04-22 05:01:18,641 - distributed.scheduler - INFO - Lost all workers
2026-04-22 05:01:18,629 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:41695', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,626 - distributed.scheduler - INFO - Remove client Client-worker-3492e2f8-3e08-11f1-8d50-2e20a4008324
2026-04-22 05:01:18,379 - distributed.scheduler - INFO - Close client connection: Client-worker-e3616cf1-3dff-11f1-8d0c-6ec63a5ab21e
2026-04-22 05:01:18,372 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:43467', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,370 - distributed.scheduler - INFO - Remove client Client-worker-e3616cf1-3dff-11f1-8d0c-6ec63a5ab21e
2026-04-22 05:01:18,287 - distributed.scheduler - INFO - Close client connection: Client-worker-e315c2a7-3dff-11f1-8e78-f28f846166bd
2026-04-22 05:01:18,285 - distributed.scheduler - INFO - Remove client Client-worker-e315c2a7-3dff-11f1-8e78-f28f846166bd
2026-04-22 05:01:18,283 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:32993', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,282 - distributed.scheduler - INFO - Close client connection: Client-worker-e4040df3-3dff-11f1-8d19-2a51d3dc178e
2026-04-22 05:01:18,277 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:34081', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,276 - distributed.scheduler - INFO - Remove client Client-worker-e4040df3-3dff-11f1-8d19-2a51d3dc178e
2026-04-22 05:01:18,273 - distributed.scheduler - INFO - Close client connection: Client-worker-00c66818-3e04-11f1-8cef-7e353d94dc10
2026-04-22 05:01:18,272 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:46025', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,270 - distributed.scheduler - INFO - Remove client Client-worker-00c66818-3e04-11f1-8cef-7e353d94dc10
2026-04-22 05:01:18,228 - distributed.scheduler - INFO - Close client connection: Client-worker-169a972b-3e04-11f1-8d47-b6f02dd406e9
2026-04-22 05:01:18,227 - distributed.scheduler - INFO - Close client connection: Client-worker-e3192817-3dff-11f1-8d5e-fae0fcb74df3
2026-04-22 05:01:18,218 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:44527', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,206 - distributed.scheduler - INFO - Remove client Client-worker-169a972b-3e04-11f1-8d47-b6f02dd406e9
2026-04-22 05:01:18,205 - distributed.scheduler - INFO - Remove client Client-worker-e3192817-3dff-11f1-8d5e-fae0fcb74df3
2026-04-22 05:01:18,191 - distributed.scheduler - INFO - Task ('sjoin-c1d04ffc4dd32b8081e833d5ca345962', 250) marked as failed because 3 workers died while trying to run it
2026-04-22 05:01:18,191 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:34475', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,185 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:41343', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,174 - distributed.scheduler - INFO - Close client connection: Client-worker-00b9084c-3e04-11f1-8ca4-0e4c34ff5e38
2026-04-22 05:01:18,161 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:34757', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,158 - distributed.scheduler - INFO - Remove client Client-worker-00b9084c-3e04-11f1-8ca4-0e4c34ff5e38
2026-04-22 05:01:18,085 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:39829', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,079 - distributed.scheduler - INFO - Close client connection: Client-worker-1b817183-3e04-11f1-8d9b-86acc56fd09f
2026-04-22 05:01:18,071 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:44569', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,069 - distributed.scheduler - INFO - Remove client Client-worker-1b817183-3e04-11f1-8d9b-86acc56fd09f
2026-04-22 05:01:18,026 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:33593', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,018 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:43131', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,012 - distributed.scheduler - INFO - Close client connection: Client-worker-00f9d2ae-3e04-11f1-8cd8-c2349ba99f21
2026-04-22 05:01:18,003 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:41177', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:18,001 - distributed.scheduler - INFO - Remove client Client-worker-00f9d2ae-3e04-11f1-8cd8-c2349ba99f21
2026-04-22 05:01:17,979 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:35059', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:17,949 - distributed.scheduler - INFO - Close client connection: Client-worker-010fabf0-3e04-11f1-8dde-4ef3162b8ac2
2026-04-22 05:01:17,929 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:44001', status: closed, memory: 0, processing: 0>
2026-04-22 05:01:17,927 - distributed.scheduler - INFO - Remove client Client-worker-010fabf0-3e04-11f1-8dde-4ef3162b8ac2
2026-04-22 05:00:38,838 - distributed.scheduler - INFO - Receive client connection: Client-worker-3492e2f8-3e08-11f1-8d50-2e20a4008324
2026-04-22 05:00:34,435 - distributed.scheduler - INFO - Receive client connection: Client-31f6d105-3e08-11f1-8001-daa6852bd5a7
2026-04-22 04:50:32,924 - distributed.scheduler - INFO - Receive client connection: Client-cb6fb7b2-3e06-11f1-8001-daa6852bd5a7
2026-04-22 04:40:31,405 - distributed.scheduler - INFO - Receive client connection: Client-64e753b6-3e05-11f1-8001-daa6852bd5a7
2026-04-22 04:31:18,775 - distributed.scheduler - INFO - Receive client connection: Client-worker-1b817183-3e04-11f1-8d9b-86acc56fd09f
2026-04-22 04:31:10,591 - distributed.scheduler - INFO - Receive client connection: Client-worker-169a972b-3e04-11f1-8d47-b6f02dd406e9
2026-04-22 04:30:34,406 - distributed.scheduler - INFO - Receive client connection: Client-worker-010fabf0-3e04-11f1-8dde-4ef3162b8ac2
2026-04-22 04:30:34,252 - distributed.scheduler - INFO - Receive client connection: Client-worker-00f9d2ae-3e04-11f1-8cd8-c2349ba99f21
2026-04-22 04:30:33,927 - distributed.scheduler - INFO - Receive client connection: Client-worker-00c66818-3e04-11f1-8cef-7e353d94dc10
2026-04-22 04:30:33,824 - distributed.scheduler - INFO - Receive client connection: Client-worker-00b9084c-3e04-11f1-8ca4-0e4c34ff5e38
2026-04-22 04:30:29,907 - distributed.scheduler - INFO - Receive client connection: Client-fe610130-3e03-11f1-8001-daa6852bd5a7
2026-04-22 04:20:28,433 - distributed.scheduler - INFO - Receive client connection: Client-97dffd50-3e02-11f1-8001-daa6852bd5a7
2026-04-22 04:12:21,274 - distributed.scheduler - INFO - Receive client connection: Client-75822cf9-3e01-11f1-8001-daa6852bd5a7
2026-04-22 04:01:08,876 - distributed.scheduler - INFO - Receive client connection: Client-e4ba9517-3dff-11f1-8007-924c5ad31268
2026-04-22 04:01:07,705 - distributed.scheduler - INFO - Receive client connection: Client-worker-e4040df3-3dff-11f1-8d19-2a51d3dc178e
2026-04-22 04:01:07,343 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:41695
2026-04-22 04:01:07,342 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:41695', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:07,080 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:43131
2026-04-22 04:01:07,080 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:43131', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:06,643 - distributed.scheduler - INFO - Receive client connection: Client-worker-e3616cf1-3dff-11f1-8d0c-6ec63a5ab21e
2026-04-22 04:01:06,300 - distributed.scheduler - INFO - Close client connection: Client-worker-cd618df2-3dff-11f1-8d0f-2e20a4008324
2026-04-22 04:01:06,297 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:40181', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:06,295 - distributed.scheduler - INFO - Remove client Client-worker-cd618df2-3dff-11f1-8d0f-2e20a4008324
2026-04-22 04:01:06,146 - distributed.scheduler - INFO - Receive client connection: Client-worker-e3192817-3dff-11f1-8d5e-fae0fcb74df3
2026-04-22 04:01:06,115 - distributed.scheduler - INFO - Receive client connection: Client-worker-e315c2a7-3dff-11f1-8e78-f28f846166bd
2026-04-22 04:01:06,044 - distributed.scheduler - INFO - Close client connection: Client-924843fd-3df7-11f1-8007-924c5ad31268
2026-04-22 04:01:06,041 - distributed.scheduler - INFO - Remove client Client-924843fd-3df7-11f1-8007-924c5ad31268
2026-04-22 04:01:06,038 - distributed.scheduler - INFO - Remove client Client-924843fd-3df7-11f1-8007-924c5ad31268
2026-04-22 04:01:06,022 - distributed.scheduler - INFO - Close client connection: Client-worker-83bf43c1-3df7-11f1-8d20-ee7435aaba1f
2026-04-22 04:01:06,014 - distributed.scheduler - INFO - Task run-36cf509e3c2db9a20146cda43b7b1838 marked as failed because 3 workers died while trying to run it
2026-04-22 04:01:06,014 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:34127', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:06,012 - distributed.scheduler - INFO - Remove client Client-worker-83bf43c1-3df7-11f1-8d20-ee7435aaba1f
2026-04-22 04:01:05,980 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:41177
2026-04-22 04:01:05,979 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:41177', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,934 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:33593
2026-04-22 04:01:05,933 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:33593', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,876 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:39829
2026-04-22 04:01:05,876 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:39829', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,874 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:44569
2026-04-22 04:01:05,852 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:44569', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,842 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:46025
2026-04-22 04:01:05,841 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:46025', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,759 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:35059
2026-04-22 04:01:05,758 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:35059', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,692 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:44527
2026-04-22 04:01:05,691 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:44527', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,636 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:41343
2026-04-22 04:01:05,635 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:41343', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,524 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:44001
2026-04-22 04:01:05,523 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:44001', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,513 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:32993
2026-04-22 04:01:05,512 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:32993', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,391 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:34757
2026-04-22 04:01:05,390 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:34757', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,346 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:34081
2026-04-22 04:01:05,345 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:34081', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,321 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:34475
2026-04-22 04:01:05,320 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:34475', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:05,309 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:43467
2026-04-22 04:01:05,307 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:43467', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,958 - distributed.scheduler - INFO - Close client connection: Client-worker-9a3450e1-3dfb-11f1-8d18-566b4744c386
2026-04-22 04:01:04,944 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:33339', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,943 - distributed.scheduler - INFO - Remove client Client-worker-9a3450e1-3dfb-11f1-8d18-566b4744c386
2026-04-22 04:01:04,855 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:45219', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,810 - distributed.scheduler - INFO - Close client connection: Client-worker-99cf44b3-3dfb-11f1-8c9a-c2349ba99f21
2026-04-22 04:01:04,807 - distributed.scheduler - INFO - Close client connection: Client-worker-cd2718a1-3dff-11f1-8c7d-8288bfee4388
2026-04-22 04:01:04,804 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:33111', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,803 - distributed.scheduler - INFO - Remove client Client-worker-99cf44b3-3dfb-11f1-8c9a-c2349ba99f21
2026-04-22 04:01:04,802 - distributed.scheduler - INFO - Close client connection: Client-worker-cd1cdd6d-3dff-11f1-8caf-7e353d94dc10
2026-04-22 04:01:04,798 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:39363', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,783 - distributed.scheduler - INFO - Remove client Client-worker-cd2718a1-3dff-11f1-8c7d-8288bfee4388
2026-04-22 04:01:04,777 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:42055', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,757 - distributed.scheduler - INFO - Remove client Client-worker-cd1cdd6d-3dff-11f1-8caf-7e353d94dc10
2026-04-22 04:01:04,725 - distributed.scheduler - INFO - Close client connection: Client-worker-8043dc0f-3df7-11f1-8d51-86acc56fd09f
2026-04-22 04:01:04,720 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:33303', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,720 - distributed.scheduler - INFO - Remove client Client-worker-8043dc0f-3df7-11f1-8d51-86acc56fd09f
2026-04-22 04:01:04,709 - distributed.scheduler - INFO - Close client connection: Client-worker-807a6d67-3df7-11f1-8d05-b6f02dd406e9
2026-04-22 04:01:04,698 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:46491', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,697 - distributed.scheduler - INFO - Remove client Client-worker-807a6d67-3df7-11f1-8d05-b6f02dd406e9
2026-04-22 04:01:04,696 - distributed.scheduler - INFO - Close client connection: Client-worker-cdd2e965-3dff-11f1-8d11-ae00acbfc48e
2026-04-22 04:01:04,683 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:34089', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,672 - distributed.scheduler - INFO - Remove client Client-worker-cdd2e965-3dff-11f1-8d11-ae00acbfc48e
2026-04-22 04:01:04,652 - distributed.scheduler - INFO - Close client connection: Client-worker-800c3bc7-3df7-11f1-8e38-f28f846166bd
2026-04-22 04:01:04,647 - distributed.scheduler - INFO - Remove client Client-worker-800c3bc7-3df7-11f1-8e38-f28f846166bd
2026-04-22 04:01:04,640 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:35995', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,534 - distributed.scheduler - INFO - Close client connection: Client-worker-7f7d704d-3df7-11f1-8d94-4ef3162b8ac2
2026-04-22 04:01:04,531 - distributed.scheduler - INFO - Remove client Client-worker-7f7d704d-3df7-11f1-8d94-4ef3162b8ac2
2026-04-22 04:01:04,522 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:44075', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,502 - distributed.scheduler - INFO - Close client connection: Client-worker-9a30f22d-3dfb-11f1-8c62-0e4c34ff5e38
2026-04-22 04:01:04,494 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:35153', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,465 - distributed.scheduler - INFO - Remove client Client-worker-9a30f22d-3dfb-11f1-8c62-0e4c34ff5e38
2026-04-22 04:01:04,383 - distributed.scheduler - INFO - Close client connection: Client-worker-99de21bd-3dfb-11f1-8ccb-6ec63a5ab21e
2026-04-22 04:01:04,379 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:38125', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,378 - distributed.scheduler - INFO - Remove client Client-worker-99de21bd-3dfb-11f1-8ccb-6ec63a5ab21e
2026-04-22 04:01:04,369 - distributed.scheduler - INFO - Close client connection: Client-worker-833bb9e7-3df7-11f1-8cd9-2a51d3dc178e
2026-04-22 04:01:04,353 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:37187', status: closed, memory: 0, processing: 0>
2026-04-22 04:01:04,352 - distributed.scheduler - INFO - Remove client Client-worker-833bb9e7-3df7-11f1-8cd9-2a51d3dc178e
2026-04-22 04:01:04,284 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:33731', status: closed, memory: 0, processing: 0>
2026-04-22 04:00:30,475 - distributed.scheduler - INFO - Receive client connection: Client-worker-cdd2e965-3dff-11f1-8d11-ae00acbfc48e
2026-04-22 04:00:29,743 - distributed.scheduler - INFO - Receive client connection: Client-worker-cd618df2-3dff-11f1-8d0f-2e20a4008324
2026-04-22 04:00:29,337 - distributed.scheduler - INFO - Receive client connection: Client-worker-cd2718a1-3dff-11f1-8c7d-8288bfee4388
2026-04-22 04:00:29,249 - distributed.scheduler - INFO - Receive client connection: Client-worker-cd1cdd6d-3dff-11f1-8caf-7e353d94dc10
2026-04-22 04:00:25,510 - distributed.scheduler - INFO - Receive client connection: Client-cae14e6e-3dff-11f1-8001-daa6852bd5a7
2026-04-22 03:50:23,962 - distributed.scheduler - INFO - Receive client connection: Client-6453d236-3dfe-11f1-8001-daa6852bd5a7
2026-04-22 03:40:22,445 - distributed.scheduler - INFO - Receive client connection: Client-fdcb942e-3dfc-11f1-8001-daa6852bd5a7
2026-04-22 03:30:25,856 - distributed.scheduler - INFO - Receive client connection: Client-worker-9a3450e1-3dfb-11f1-8d18-566b4744c386
2026-04-22 03:30:25,832 - distributed.scheduler - INFO - Receive client connection: Client-worker-9a30f22d-3dfb-11f1-8c62-0e4c34ff5e38
2026-04-22 03:30:25,286 - distributed.scheduler - INFO - Receive client connection: Client-worker-99de21bd-3dfb-11f1-8ccb-6ec63a5ab21e
2026-04-22 03:30:25,196 - distributed.scheduler - INFO - Receive client connection: Client-worker-99cf44b3-3dfb-11f1-8c9a-c2349ba99f21
2026-04-22 03:30:20,948 - distributed.scheduler - INFO - Receive client connection: Client-97468abf-3dfb-11f1-8001-daa6852bd5a7
2026-04-22 03:20:19,455 - distributed.scheduler - INFO - Receive client connection: Client-30c2ef74-3dfa-11f1-8001-daa6852bd5a7
2026-04-22 03:12:40,872 - distributed.scheduler - INFO - Receive client connection: Client-1f6af356-3df9-11f1-8001-daa6852bd5a7
2026-04-22 03:01:34,586 - distributed.scheduler - INFO - Receive client connection: Client-924843fd-3df7-11f1-8007-924c5ad31268
2026-04-22 03:01:32,314 - distributed.scheduler - INFO - Close client connection: Client-76722d87-3db4-11f1-8007-924c5ad31268
2026-04-22 03:01:32,310 - distributed.scheduler - INFO - Remove client Client-76722d87-3db4-11f1-8007-924c5ad31268
2026-04-22 03:01:32,308 - distributed.scheduler - INFO - Remove client Client-76722d87-3db4-11f1-8007-924c5ad31268
2026-04-22 03:01:10,236 - distributed.scheduler - INFO - Receive client connection: Client-worker-83bf43c1-3df7-11f1-8d20-ee7435aaba1f
2026-04-22 03:01:09,993 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:40181
2026-04-22 03:01:09,991 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:40181', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:09,367 - distributed.scheduler - INFO - Receive client connection: Client-worker-833bb9e7-3df7-11f1-8cd9-2a51d3dc178e
2026-04-22 03:01:08,919 - distributed.scheduler - INFO - Close client connection: Client-worker-665cd09f-3df7-11f1-8cd6-2e20a4008324
2026-04-22 03:01:08,912 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:35933', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:08,911 - distributed.scheduler - INFO - Remove client Client-worker-665cd09f-3df7-11f1-8cd6-2e20a4008324
2026-04-22 03:01:04,730 - distributed.scheduler - INFO - Receive client connection: Client-worker-807a6d67-3df7-11f1-8d05-b6f02dd406e9
2026-04-22 03:01:04,388 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:33111
2026-04-22 03:01:04,387 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:33111', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:04,370 - distributed.scheduler - INFO - Receive client connection: Client-worker-8043dc0f-3df7-11f1-8d51-86acc56fd09f
2026-04-22 03:01:03,984 - distributed.scheduler - INFO - Receive client connection: Client-worker-800c3bc7-3df7-11f1-8e38-f28f846166bd
2026-04-22 03:01:03,491 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:45219
2026-04-22 03:01:03,489 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:45219', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:03,309 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:37051', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:03,281 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:35995
2026-04-22 03:01:03,280 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:35995', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:03,052 - distributed.scheduler - INFO - Receive client connection: Client-worker-7f7d704d-3df7-11f1-8d94-4ef3162b8ac2
2026-04-22 03:01:02,943 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:35153
2026-04-22 03:01:02,942 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:35153', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,932 - distributed.scheduler - INFO - Close client connection: Client-e86aea01-3db2-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,931 - distributed.scheduler - INFO - Close client connection: Client-4eeb8bd1-3db4-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,931 - distributed.scheduler - INFO - Close client connection: Client-1befbc60-3db7-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,931 - distributed.scheduler - INFO - Close client connection: Client-23d54afc-3db6-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,931 - distributed.scheduler - INFO - Close client connection: Client-826ee8e1-3db8-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,931 - distributed.scheduler - INFO - Close client connection: Client-e8f0a7a6-3db9-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,931 - distributed.scheduler - INFO - Close client connection: Client-4f7034e6-3dbb-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,931 - distributed.scheduler - INFO - Close client connection: Client-b5f59eaf-3dbc-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,931 - distributed.scheduler - INFO - Close client connection: Client-6cf001b2-3dbe-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,930 - distributed.scheduler - INFO - Close client connection: Client-82ff14d5-3dbf-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,930 - distributed.scheduler - INFO - Close client connection: Client-e982f36f-3dc0-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,930 - distributed.scheduler - INFO - Close client connection: Client-5006fa35-3dc2-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,930 - distributed.scheduler - INFO - Close client connection: Client-b688c753-3dc3-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,930 - distributed.scheduler - INFO - Close client connection: Client-1d05ea77-3dc5-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,930 - distributed.scheduler - INFO - Close client connection: Client-c2dc6c6d-3dc6-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,929 - distributed.scheduler - INFO - Close client connection: Client-ea11c76e-3dc7-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,929 - distributed.scheduler - INFO - Close client connection: Client-509a6d3b-3dc9-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,929 - distributed.scheduler - INFO - Close client connection: Client-b71f7b22-3dca-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,929 - distributed.scheduler - INFO - Close client connection: Client-1d9bf34b-3dcc-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,929 - distributed.scheduler - INFO - Close client connection: Client-8421e40d-3dcd-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,929 - distributed.scheduler - INFO - Close client connection: Client-512a71f2-3dd0-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,928 - distributed.scheduler - INFO - Close client connection: Client-2f7f8dac-3dcf-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,928 - distributed.scheduler - INFO - Close client connection: Client-b7b7bfe1-3dd1-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,928 - distributed.scheduler - INFO - Close client connection: Client-1e3ae033-3dd3-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,928 - distributed.scheduler - INFO - Close client connection: Client-84abab04-3dd4-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,928 - distributed.scheduler - INFO - Close client connection: Client-eb29dfce-3dd5-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,928 - distributed.scheduler - INFO - Close client connection: Client-b825f5c9-3dd8-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,928 - distributed.scheduler - INFO - Close client connection: Client-86337ce5-3dd7-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,927 - distributed.scheduler - INFO - Close client connection: Client-1eaae4ba-3dda-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,927 - distributed.scheduler - INFO - Close client connection: Client-8534c685-3ddb-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,927 - distributed.scheduler - INFO - Close client connection: Client-ebb9c103-3ddc-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,927 - distributed.scheduler - INFO - Close client connection: Client-523b7cfd-3dde-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,927 - distributed.scheduler - INFO - Close client connection: Client-1f46465d-3de1-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,927 - distributed.scheduler - INFO - Close client connection: Client-007883e2-3de0-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,926 - distributed.scheduler - INFO - Close client connection: Client-85d7ced1-3de2-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,926 - distributed.scheduler - INFO - Close client connection: Client-ec620886-3de3-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,926 - distributed.scheduler - INFO - Close client connection: Client-52e00e73-3de5-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,926 - distributed.scheduler - INFO - Close client connection: Client-b96685b8-3de6-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,926 - distributed.scheduler - INFO - Close client connection: Client-6299aaf3-3de9-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,926 - distributed.scheduler - INFO - Close client connection: Client-73482ecd-3de8-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,926 - distributed.scheduler - INFO - Close client connection: Client-c91812c5-3dea-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,925 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:37187
2026-04-22 03:01:02,925 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:37187', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,923 - distributed.scheduler - INFO - Close client connection: Client-2f9ed531-3dec-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,923 - distributed.scheduler - INFO - Close client connection: Client-9635d8fa-3ded-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,923 - distributed.scheduler - INFO - Close client connection: Client-fcc13812-3dee-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,923 - distributed.scheduler - INFO - Close client connection: Client-c9b9d1f8-3df1-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,922 - distributed.scheduler - INFO - Close client connection: Client-a0d28054-3df0-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,922 - distributed.scheduler - INFO - Close client connection: Client-303b7ac1-3df3-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,922 - distributed.scheduler - INFO - Close client connection: Client-96ad8c4c-3df4-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,922 - distributed.scheduler - INFO - Close client connection: Client-fd2e81cf-3df5-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,919 - distributed.scheduler - INFO - Remove client Client-e86aea01-3db2-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,919 - distributed.scheduler - INFO - Remove client Client-4eeb8bd1-3db4-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,919 - distributed.scheduler - INFO - Remove client Client-1befbc60-3db7-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,918 - distributed.scheduler - INFO - Remove client Client-23d54afc-3db6-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,918 - distributed.scheduler - INFO - Remove client Client-826ee8e1-3db8-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,918 - distributed.scheduler - INFO - Remove client Client-e8f0a7a6-3db9-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,918 - distributed.scheduler - INFO - Remove client Client-4f7034e6-3dbb-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,917 - distributed.scheduler - INFO - Remove client Client-b5f59eaf-3dbc-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,917 - distributed.scheduler - INFO - Remove client Client-6cf001b2-3dbe-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,917 - distributed.scheduler - INFO - Remove client Client-82ff14d5-3dbf-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,916 - distributed.scheduler - INFO - Remove client Client-e982f36f-3dc0-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,916 - distributed.scheduler - INFO - Remove client Client-5006fa35-3dc2-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,916 - distributed.scheduler - INFO - Remove client Client-b688c753-3dc3-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,915 - distributed.scheduler - INFO - Remove client Client-1d05ea77-3dc5-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,915 - distributed.scheduler - INFO - Remove client Client-c2dc6c6d-3dc6-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,915 - distributed.scheduler - INFO - Remove client Client-ea11c76e-3dc7-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,914 - distributed.scheduler - INFO - Remove client Client-509a6d3b-3dc9-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,914 - distributed.scheduler - INFO - Remove client Client-b71f7b22-3dca-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,914 - distributed.scheduler - INFO - Remove client Client-1d9bf34b-3dcc-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,913 - distributed.scheduler - INFO - Remove client Client-8421e40d-3dcd-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,913 - distributed.scheduler - INFO - Remove client Client-512a71f2-3dd0-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,913 - distributed.scheduler - INFO - Remove client Client-2f7f8dac-3dcf-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,912 - distributed.scheduler - INFO - Remove client Client-b7b7bfe1-3dd1-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,912 - distributed.scheduler - INFO - Remove client Client-1e3ae033-3dd3-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,912 - distributed.scheduler - INFO - Remove client Client-84abab04-3dd4-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,911 - distributed.scheduler - INFO - Remove client Client-eb29dfce-3dd5-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,911 - distributed.scheduler - INFO - Remove client Client-b825f5c9-3dd8-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,910 - distributed.scheduler - INFO - Remove client Client-86337ce5-3dd7-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,910 - distributed.scheduler - INFO - Remove client Client-1eaae4ba-3dda-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,909 - distributed.scheduler - INFO - Remove client Client-8534c685-3ddb-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,909 - distributed.scheduler - INFO - Remove client Client-ebb9c103-3ddc-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,908 - distributed.scheduler - INFO - Remove client Client-523b7cfd-3dde-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,908 - distributed.scheduler - INFO - Remove client Client-1f46465d-3de1-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,908 - distributed.scheduler - INFO - Remove client Client-007883e2-3de0-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,907 - distributed.scheduler - INFO - Remove client Client-85d7ced1-3de2-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,907 - distributed.scheduler - INFO - Remove client Client-ec620886-3de3-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,907 - distributed.scheduler - INFO - Remove client Client-52e00e73-3de5-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,906 - distributed.scheduler - INFO - Remove client Client-b96685b8-3de6-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,906 - distributed.scheduler - INFO - Remove client Client-6299aaf3-3de9-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,905 - distributed.scheduler - INFO - Remove client Client-73482ecd-3de8-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,905 - distributed.scheduler - INFO - Remove client Client-c91812c5-3dea-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,892 - distributed.scheduler - INFO - Remove client Client-2f9ed531-3dec-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,892 - distributed.scheduler - INFO - Remove client Client-9635d8fa-3ded-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,891 - distributed.scheduler - INFO - Remove client Client-fcc13812-3dee-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,890 - distributed.scheduler - INFO - Remove client Client-c9b9d1f8-3df1-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,889 - distributed.scheduler - INFO - Remove client Client-a0d28054-3df0-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,888 - distributed.scheduler - INFO - Remove client Client-303b7ac1-3df3-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,887 - distributed.scheduler - INFO - Remove client Client-96ad8c4c-3df4-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,886 - distributed.scheduler - INFO - Remove client Client-fd2e81cf-3df5-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,763 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:33731
2026-04-22 03:01:02,762 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:33731', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,734 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:34089
2026-04-22 03:01:02,733 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:34089', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,703 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:38125
2026-04-22 03:01:02,702 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:38125', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,689 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:33339
2026-04-22 03:01:02,687 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:33339', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,653 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:42055
2026-04-22 03:01:02,652 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:42055', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,627 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:33303
2026-04-22 03:01:02,626 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:33303', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,604 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:39363
2026-04-22 03:01:02,602 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:39363', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,578 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:34127
2026-04-22 03:01:02,577 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:34127', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,564 - distributed.scheduler - INFO - Close client connection: Client-worker-65f61495-3df7-11f1-8cf1-2eb33e86e1bd
2026-04-22 03:01:02,554 - distributed.scheduler - INFO - Task run-20995ff95188017d71b39bbd7f700e4a marked as failed because 3 workers died while trying to run it
2026-04-22 03:01:02,554 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:40097', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,552 - distributed.scheduler - INFO - Remove client Client-worker-65f61495-3df7-11f1-8cf1-2eb33e86e1bd
2026-04-22 03:01:02,465 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:46491
2026-04-22 03:01:02,464 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:46491', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,429 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:44075
2026-04-22 03:01:02,427 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:44075', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,367 - distributed.scheduler - INFO - Close client connection: Client-63bd6cd4-3df7-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,366 - distributed.scheduler - INFO - Remove client Client-63bd6cd4-3df7-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,365 - distributed.scheduler - INFO - Remove client Client-63bd6cd4-3df7-11f1-8001-daa6852bd5a7
2026-04-22 03:01:02,330 - distributed.scheduler - INFO - Close client connection: Client-worker-32ed46ca-3df3-11f1-8df7-f28f846166bd
2026-04-22 03:01:02,324 - distributed.scheduler - INFO - Task ('from_pandas-6b24fc4d4f63d5531d6c1e7931483f3a', 12) marked as failed because 3 workers died while trying to run it
2026-04-22 03:01:02,323 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:40097', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:02,320 - distributed.scheduler - INFO - Remove client Client-worker-32ed46ca-3df3-11f1-8df7-f28f846166bd
2026-04-22 03:01:01,890 - distributed.scheduler - INFO - Close client connection: Client-worker-32a93baf-3df3-11f1-8c28-0e4c34ff5e38
2026-04-22 03:01:01,884 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:38977', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:01,876 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:45783', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:01,876 - distributed.scheduler - INFO - Remove client Client-worker-32a93baf-3df3-11f1-8c28-0e4c34ff5e38
2026-04-22 03:01:01,848 - distributed.scheduler - INFO - Close client connection: Client-worker-65fe89e0-3df7-11f1-8ce6-fae0fcb74df3
2026-04-22 03:01:01,845 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:43603', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:01,828 - distributed.scheduler - INFO - Remove client Client-worker-65fe89e0-3df7-11f1-8ce6-fae0fcb74df3
2026-04-22 03:01:01,798 - distributed.scheduler - INFO - Close client connection: Client-worker-32ca81ec-3df3-11f1-8c71-7e353d94dc10
2026-04-22 03:01:01,789 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:38539', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:01,788 - distributed.scheduler - INFO - Remove client Client-worker-32ca81ec-3df3-11f1-8c71-7e353d94dc10
2026-04-22 03:01:01,781 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:39013', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:01,739 - distributed.scheduler - INFO - Close client connection: Client-worker-852fd3f1-3db8-11f1-8c78-6ec63a5ab21e
2026-04-22 03:01:01,734 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:46363', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:01,732 - distributed.scheduler - INFO - Remove client Client-worker-852fd3f1-3db8-11f1-8c78-6ec63a5ab21e
2026-04-22 03:01:01,687 - distributed.scheduler - INFO - Close client connection: Client-worker-65dd199b-3df7-11f1-8c42-8288bfee4388
2026-04-22 03:01:01,683 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:44957', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:01,681 - distributed.scheduler - INFO - Remove client Client-worker-65dd199b-3df7-11f1-8c42-8288bfee4388
2026-04-22 03:01:01,615 - distributed.scheduler - INFO - Close client connection: Client-worker-668c9855-3df7-11f1-8cd7-566b4744c386
2026-04-22 03:01:01,611 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:41617', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:01,610 - distributed.scheduler - INFO - Remove client Client-worker-668c9855-3df7-11f1-8cd7-566b4744c386
2026-04-22 03:01:01,571 - distributed.scheduler - INFO - Close client connection: Client-worker-51ba44ca-3def-11f1-8ce0-ee7435aaba1f
2026-04-22 03:01:01,570 - distributed.scheduler - INFO - Close client connection: Client-worker-3283ca25-3df3-11f1-8cc8-b6f02dd406e9
2026-04-22 03:01:01,564 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:46263', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:01,553 - distributed.scheduler - INFO - Remove client Client-worker-51ba44ca-3def-11f1-8ce0-ee7435aaba1f
2026-04-22 03:01:01,548 - distributed.scheduler - INFO - Close client connection: Client-worker-327e921f-3df3-11f1-8d0d-86acc56fd09f
2026-04-22 03:01:01,545 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:45397', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:01,544 - distributed.scheduler - INFO - Remove client Client-worker-3283ca25-3df3-11f1-8cc8-b6f02dd406e9
2026-04-22 03:01:01,523 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:41533', status: closed, memory: 0, processing: 0>
2026-04-22 03:01:01,521 - distributed.scheduler - INFO - Remove client Client-worker-327e921f-3df3-11f1-8d0d-86acc56fd09f
2026-04-22 03:01:01,423 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:41777', status: closed, memory: 0, processing: 0>
2026-04-22 03:00:21,228 - distributed.scheduler - INFO - Receive client connection: Client-worker-668c9855-3df7-11f1-8cd7-566b4744c386
2026-04-22 03:00:20,913 - distributed.scheduler - INFO - Receive client connection: Client-worker-665cd09f-3df7-11f1-8cd6-2e20a4008324
2026-04-22 03:00:20,267 - distributed.scheduler - INFO - Receive client connection: Client-worker-65fe89e0-3df7-11f1-8ce6-fae0fcb74df3
2026-04-22 03:00:20,233 - distributed.scheduler - INFO - Receive client connection: Client-worker-65f61495-3df7-11f1-8cf1-2eb33e86e1bd
2026-04-22 03:00:20,059 - distributed.scheduler - INFO - Receive client connection: Client-worker-65dd199b-3df7-11f1-8c42-8288bfee4388
2026-04-22 03:00:16,504 - distributed.scheduler - INFO - Receive client connection: Client-63bd6cd4-3df7-11f1-8001-daa6852bd5a7
2026-04-22 02:50:14,936 - distributed.scheduler - INFO - Receive client connection: Client-fd2e81cf-3df5-11f1-8001-daa6852bd5a7
2026-04-22 02:40:13,472 - distributed.scheduler - INFO - Receive client connection: Client-96ad8c4c-3df4-11f1-8001-daa6852bd5a7
2026-04-22 02:30:16,613 - distributed.scheduler - INFO - Receive client connection: Client-worker-32ed46ca-3df3-11f1-8df7-f28f846166bd
2026-04-22 02:30:16,404 - distributed.scheduler - INFO - Receive client connection: Client-worker-32ca81ec-3df3-11f1-8c71-7e353d94dc10
2026-04-22 02:30:16,180 - distributed.scheduler - INFO - Receive client connection: Client-worker-32a93baf-3df3-11f1-8c28-0e4c34ff5e38
2026-04-22 02:30:15,936 - distributed.scheduler - INFO - Receive client connection: Client-worker-3283ca25-3df3-11f1-8cc8-b6f02dd406e9
2026-04-22 02:30:15,886 - distributed.scheduler - INFO - Receive client connection: Client-worker-327e921f-3df3-11f1-8d0d-86acc56fd09f
2026-04-22 02:30:12,093 - distributed.scheduler - INFO - Receive client connection: Client-303b7ac1-3df3-11f1-8001-daa6852bd5a7
2026-04-22 02:20:10,626 - distributed.scheduler - INFO - Receive client connection: Client-c9b9d1f8-3df1-11f1-8001-daa6852bd5a7
2026-04-22 02:11:52,498 - distributed.scheduler - INFO - Receive client connection: Client-a0d28054-3df0-11f1-8001-daa6852bd5a7
2026-04-22 02:02:30,304 - distributed.scheduler - INFO - Receive client connection: Client-worker-51ba44ca-3def-11f1-8ce0-ee7435aaba1f
2026-04-22 02:02:27,019 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:46263
2026-04-22 02:02:27,017 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:46263', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:26,711 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:37051
2026-04-22 02:02:26,709 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:37051', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:26,301 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:35933
2026-04-22 02:02:26,300 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:35933', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:25,968 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:39055', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:25,731 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:41531', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:25,552 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:43603
2026-04-22 02:02:25,551 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:43603', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:25,444 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:44957
2026-04-22 02:02:25,442 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:44957', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:25,370 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:45783
2026-04-22 02:02:25,370 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:45783', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:25,368 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:40097
2026-04-22 02:02:25,367 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:40097', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:25,366 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:38977
2026-04-22 02:02:25,349 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:38977', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:25,326 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:40097
2026-04-22 02:02:25,326 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:40097', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:25,324 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:39013
2026-04-22 02:02:25,323 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:39013', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:25,264 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:32847', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:25,153 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:41533
2026-04-22 02:02:25,152 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:41533', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:25,049 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:41617
2026-04-22 02:02:25,048 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:41617', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:24,903 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:41777
2026-04-22 02:02:24,901 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:41777', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:24,863 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:38539
2026-04-22 02:02:24,861 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:38539', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:24,738 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:45397
2026-04-22 02:02:24,736 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:45397', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:24,597 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:36253', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:24,477 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:35937', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:24,468 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:33683', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:24,461 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:45091', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:24,381 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:45145', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:24,350 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:39189', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:24,335 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:36919', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:24,107 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:45667', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:24,042 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:33007', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:23,985 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:34621', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:23,973 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:42373', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:23,900 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:44549', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:12,450 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:35937
2026-04-22 02:02:12,449 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:35937', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:12,348 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:39189
2026-04-22 02:02:12,347 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:39189', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:12,276 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:36253
2026-04-22 02:02:12,275 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:36253', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:12,167 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:45091
2026-04-22 02:02:12,166 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:45091', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:12,127 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:45145
2026-04-22 02:02:12,126 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:45145', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:12,049 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:33683
2026-04-22 02:02:12,048 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:33683', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,980 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:39055
2026-04-22 02:02:11,977 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:39055', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,804 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:34621
2026-04-22 02:02:11,803 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:34621', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,719 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:45667
2026-04-22 02:02:11,718 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:45667', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,661 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:32847
2026-04-22 02:02:11,660 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:32847', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,528 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:41531
2026-04-22 02:02:11,527 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:41531', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,515 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:36919
2026-04-22 02:02:11,514 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:36919', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,423 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:44549
2026-04-22 02:02:11,421 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:44549', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,419 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:42373
2026-04-22 02:02:11,418 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:42373', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,343 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:33007
2026-04-22 02:02:11,342 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:33007', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,339 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:34863', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,310 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:45583', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,062 - distributed.scheduler - INFO - Close client connection: Client-worker-2b73a0b2-3def-11f1-8c91-fae0fcb74df3
2026-04-22 02:02:11,055 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:42157', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,054 - distributed.scheduler - INFO - Remove client Client-worker-2b73a0b2-3def-11f1-8c91-fae0fcb74df3
2026-04-22 02:02:11,048 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:43155', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:11,002 - distributed.scheduler - INFO - Close client connection: Client-worker-2b2f7026-3def-11f1-8dae-f28f846166bd
2026-04-22 02:02:10,998 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:45907', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:10,992 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:37363', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:10,991 - distributed.scheduler - INFO - Remove client Client-worker-2b2f7026-3def-11f1-8dae-f28f846166bd
2026-04-22 02:02:10,971 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:42125', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:10,762 - distributed.scheduler - INFO - Close client connection: Client-worker-177956d7-3def-11f1-8c22-7e353d94dc10
2026-04-22 02:02:10,760 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:41337', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:10,758 - distributed.scheduler - INFO - Remove client Client-worker-177956d7-3def-11f1-8c22-7e353d94dc10
2026-04-22 02:02:10,693 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:39659', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:10,626 - distributed.scheduler - INFO - Close client connection: Client-worker-2afd7c38-3def-11f1-8cbb-86acc56fd09f
2026-04-22 02:02:10,618 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:45769', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:10,617 - distributed.scheduler - INFO - Remove client Client-worker-2afd7c38-3def-11f1-8cbb-86acc56fd09f
2026-04-22 02:02:10,564 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:36409', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:10,519 - distributed.scheduler - INFO - Close client connection: Client-worker-16f895ec-3def-11f1-8c87-566b4744c386
2026-04-22 02:02:10,518 - distributed.scheduler - INFO - Remove client Client-worker-16f895ec-3def-11f1-8c87-566b4744c386
2026-04-22 02:02:10,516 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:40919', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:10,478 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:40997', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:10,425 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:35831', status: closed, memory: 0, processing: 0>
2026-04-22 02:02:10,411 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:34637', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:26,092 - distributed.scheduler - INFO - Receive client connection: Client-worker-2b73a0b2-3def-11f1-8c91-fae0fcb74df3
2026-04-22 02:01:25,651 - distributed.scheduler - INFO - Receive client connection: Client-worker-2b2f7026-3def-11f1-8dae-f28f846166bd
2026-04-22 02:01:25,560 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:36409
2026-04-22 02:01:25,560 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:36409', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:25,315 - distributed.scheduler - INFO - Receive client connection: Client-worker-2afd7c38-3def-11f1-8cbb-86acc56fd09f
2026-04-22 02:01:24,691 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:45583
2026-04-22 02:01:24,690 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:45583', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:24,667 - distributed.scheduler - INFO - Close client connection: Client-worker-16853a66-3def-11f1-8c44-ae00acbfc48e
2026-04-22 02:01:24,657 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:43803', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:24,656 - distributed.scheduler - INFO - Remove client Client-worker-16853a66-3def-11f1-8c44-ae00acbfc48e
2026-04-22 02:01:24,596 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:43155
2026-04-22 02:01:24,595 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:43155', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:24,453 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:40997
2026-04-22 02:01:24,452 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:40997', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:24,282 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:45907
2026-04-22 02:01:24,281 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:45907', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:24,234 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:42157
2026-04-22 02:01:24,233 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:42157', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:24,230 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:34863
2026-04-22 02:01:24,229 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:34863', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:24,027 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:34637
2026-04-22 02:01:24,026 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:34637', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:23,983 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:39659
2026-04-22 02:01:23,981 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:39659', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:23,923 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-9fc64223f5f75369784a481b33457962 NoneType: None
2026-04-22 02:01:23,923 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-c0648734f8f6fb7f02651ebce9187c8c NoneType: None
2026-04-22 02:01:23,923 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-c23d8eded16e0fce101facca828f05da NoneType: None
2026-04-22 02:01:23,922 - distributed.scheduler - ERROR - Couldn't gather keys {'run-c23d8eded16e0fce101facca828f05da': [], 'run-c0648734f8f6fb7f02651ebce9187c8c': [], 'run-9fc64223f5f75369784a481b33457962': []} state: ['processing', 'processing', 'processing'] workers: [] NoneType: None
2026-04-22 02:01:23,738 - distributed.scheduler - INFO - Close client connection: Client-worker-180ceb57-3def-11f1-8c71-2eb33e86e1bd
2026-04-22 02:01:23,736 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:35831 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52890 remote=tcp://172.21.151.150:35831>: Stream is closed
2026-04-22 02:01:23,736 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:35831 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:52888 remote=tcp://172.21.151.150:35831>: Stream is closed
2026-04-22 02:01:23,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:40919 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:32828 remote=tcp://172.21.151.168:40919>: Stream is closed
2026-04-22 02:01:23,733 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:46363 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:35164 remote=tcp://172.21.159.230:46363>: Stream is closed
2026-04-22 02:01:23,728 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:37545', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:23,668 - distributed.scheduler - INFO - Remove client Client-worker-180ceb57-3def-11f1-8c71-2eb33e86e1bd
2026-04-22 02:01:23,657 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:33391', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:23,462 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:33825', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:23,287 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:46675', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:23,270 - distributed.scheduler - INFO - Close client connection: Client-worker-17436fa5-3def-11f1-8bbf-8288bfee4388
2026-04-22 02:01:23,269 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:45203 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:45203 after 30 s
2026-04-22 02:01:23,269 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:45203 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.207:45203 after 30 s
2026-04-22 02:01:23,248 - distributed.scheduler - INFO - Remove client Client-worker-17436fa5-3def-11f1-8bbf-8288bfee4388
2026-04-22 02:01:23,232 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:45203', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:23,171 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:42479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:42479 after 30 s
2026-04-22 02:01:23,171 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:42479 failed: OSError: Timed out during handshake while connecting to tcp://172.21.159.225:42479 after 30 s
2026-04-22 02:01:23,165 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:42479', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:23,104 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:46417 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:46417 after 30 s
2026-04-22 02:01:23,104 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:46417 failed: OSError: Timed out during handshake while connecting to tcp://172.21.151.183:46417 after 30 s
2026-04-22 02:01:23,096 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:46417', status: closed, memory: 0, processing: 0>
2026-04-22 02:01:23,049 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:42425 failed: CommClosedError: Address removed.
2026-04-22 02:01:23,049 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:42425 failed: CommClosedError: Address removed.
2026-04-22 02:01:23,016 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:42425', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:53,546 - distributed.scheduler - INFO - Receive client connection: Client-worker-180ceb57-3def-11f1-8c71-2eb33e86e1bd
2026-04-22 02:00:53,045 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:33825
2026-04-22 02:00:53,044 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:33825', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:52,980 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:46675
2026-04-22 02:00:52,979 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:46675', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:52,721 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:45769
2026-04-22 02:00:52,720 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:45769', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:52,571 - distributed.scheduler - INFO - Receive client connection: Client-worker-177956d7-3def-11f1-8c22-7e353d94dc10
2026-04-22 02:00:52,255 - distributed.scheduler - INFO - Receive client connection: Client-worker-17436fa5-3def-11f1-8bbf-8288bfee4388
2026-04-22 02:00:51,889 - distributed.scheduler - INFO - Close client connection: Client-worker-cb5514b0-3dea-11f1-8ba5-c2349ba99f21
2026-04-22 02:00:51,881 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:44739', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:51,879 - distributed.scheduler - INFO - Remove client Client-worker-cb5514b0-3dea-11f1-8ba5-c2349ba99f21
2026-04-22 02:00:51,833 - distributed.scheduler - INFO - Close client connection: Client-worker-84d0b037-3db8-11f1-8c14-ee7435aaba1f
2026-04-22 02:00:51,829 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:38121', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:51,811 - distributed.scheduler - INFO - Remove client Client-worker-84d0b037-3db8-11f1-8c14-ee7435aaba1f
2026-04-22 02:00:51,750 - distributed.scheduler - INFO - Receive client connection: Client-worker-16f895ec-3def-11f1-8c87-566b4744c386
2026-04-22 02:00:51,629 - distributed.scheduler - INFO - Close client connection: Client-worker-855ea4c4-3db8-11f1-8c70-86acc56fd09f
2026-04-22 02:00:51,621 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:46675', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:51,620 - distributed.scheduler - INFO - Remove client Client-worker-855ea4c4-3db8-11f1-8c70-86acc56fd09f
2026-04-22 02:00:50,960 - distributed.scheduler - INFO - Receive client connection: Client-worker-16853a66-3def-11f1-8c44-ae00acbfc48e
2026-04-22 02:00:50,793 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:42425
2026-04-22 02:00:50,792 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:42425', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:50,652 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:33391
2026-04-22 02:00:50,652 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:33391', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:50,528 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:37545
2026-04-22 02:00:50,528 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:37545', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:50,455 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:45203
2026-04-22 02:00:50,455 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:45203', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:50,441 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:42479
2026-04-22 02:00:50,440 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:42479', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:50,113 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:42125
2026-04-22 02:00:50,112 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:42125', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:50,051 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:41337
2026-04-22 02:00:50,050 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:41337', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:50,047 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:37363
2026-04-22 02:00:50,046 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:37363', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:49,787 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:43803
2026-04-22 02:00:49,787 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:43803', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:49,765 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:33155', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:49,765 - distributed.scheduler - INFO - Close client connection: Client-worker-b7c9500a-3dbc-11f1-8c16-2e20a4008324
2026-04-22 02:00:49,764 - distributed.scheduler - INFO - Remove client Client-worker-b7c9500a-3dbc-11f1-8c16-2e20a4008324
2026-04-22 02:00:49,615 - distributed.scheduler - INFO - Close client connection: Client-worker-84dc71ca-3db8-11f1-8be9-2a51d3dc178e
2026-04-22 02:00:49,599 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:44367', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:49,597 - distributed.scheduler - INFO - Remove client Client-worker-84dc71ca-3db8-11f1-8be9-2a51d3dc178e
2026-04-22 02:00:49,560 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:40919
2026-04-22 02:00:49,559 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:40919', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:49,444 - distributed.scheduler - INFO - Close client connection: Client-worker-21148a86-3dda-11f1-8c24-2eb33e86e1bd
2026-04-22 02:00:49,444 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:35831
2026-04-22 02:00:49,443 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:35831', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:49,435 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:46865', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:49,433 - distributed.scheduler - INFO - Remove client Client-worker-21148a86-3dda-11f1-8c24-2eb33e86e1bd
2026-04-22 02:00:49,380 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:46417
2026-04-22 02:00:49,377 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:46417', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:49,343 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:44385', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:49,338 - distributed.scheduler - INFO - Close client connection: Client-worker-b82d7061-3dbc-11f1-8b82-8288bfee4388
2026-04-22 02:00:49,335 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:38267', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:49,333 - distributed.scheduler - INFO - Remove client Client-worker-b82d7061-3dbc-11f1-8b82-8288bfee4388
2026-04-22 02:00:49,083 - distributed.scheduler - INFO - Close client connection: Client-worker-b7e5c68c-3dbc-11f1-8ba2-0e4c34ff5e38
2026-04-22 02:00:49,076 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:42045', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:49,073 - distributed.scheduler - INFO - Remove client Client-worker-b7e5c68c-3dbc-11f1-8ba2-0e4c34ff5e38
2026-04-22 02:00:49,033 - distributed.scheduler - INFO - Close client connection: Client-worker-854ae10c-3db8-11f1-8d61-f28f846166bd
2026-04-22 02:00:49,031 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:37883', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:49,030 - distributed.scheduler - INFO - Remove client Client-worker-854ae10c-3db8-11f1-8d61-f28f846166bd
2026-04-22 02:00:48,995 - distributed.scheduler - INFO - Close client connection: Client-worker-84b10888-3db8-11f1-8bdc-7e353d94dc10
2026-04-22 02:00:48,976 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:36753', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:48,974 - distributed.scheduler - INFO - Remove client Client-worker-84b10888-3db8-11f1-8bdc-7e353d94dc10
2026-04-22 02:00:48,685 - distributed.scheduler - INFO - Close client connection: Client-worker-2173b8cd-3dda-11f1-8bfe-ae00acbfc48e
2026-04-22 02:00:48,676 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:41513', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:48,675 - distributed.scheduler - INFO - Remove client Client-worker-2173b8cd-3dda-11f1-8bfe-ae00acbfc48e
2026-04-22 02:00:48,657 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:41633', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:48,453 - distributed.scheduler - INFO - Close client connection: Client-worker-b84a1948-3dbc-11f1-8c9a-4ef3162b8ac2
2026-04-22 02:00:48,449 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:44513', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:48,448 - distributed.scheduler - INFO - Remove client Client-worker-b84a1948-3dbc-11f1-8c9a-4ef3162b8ac2
2026-04-22 02:00:48,444 - distributed.scheduler - INFO - Close client connection: Client-worker-50e9d189-3db4-11f1-8c2d-b6f02dd406e9
2026-04-22 02:00:48,428 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:44763', status: closed, memory: 0, processing: 0>
2026-04-22 02:00:48,425 - distributed.scheduler - INFO - Remove client Client-worker-50e9d189-3db4-11f1-8c2d-b6f02dd406e9
2026-04-22 02:00:07,747 - distributed.scheduler - INFO - Receive client connection: Client-fcc13812-3dee-11f1-8001-daa6852bd5a7
2026-04-22 01:50:06,212 - distributed.scheduler - INFO - Receive client connection: Client-9635d8fa-3ded-11f1-8001-daa6852bd5a7
2026-04-22 01:40:04,597 - distributed.scheduler - INFO - Receive client connection: Client-2f9ed531-3dec-11f1-8001-daa6852bd5a7
2026-04-22 01:30:06,831 - distributed.scheduler - INFO - Receive client connection: Client-worker-cb5514b0-3dea-11f1-8ba5-c2349ba99f21
2026-04-22 01:30:03,086 - distributed.scheduler - INFO - Receive client connection: Client-c91812c5-3dea-11f1-8001-daa6852bd5a7
2026-04-22 01:20:01,630 - distributed.scheduler - INFO - Receive client connection: Client-6299aaf3-3de9-11f1-8001-daa6852bd5a7
2026-04-22 01:13:20,129 - distributed.scheduler - INFO - Receive client connection: Client-73482ecd-3de8-11f1-8001-daa6852bd5a7
2026-04-22 01:00:58,775 - distributed.scheduler - INFO - Receive client connection: Client-b96685b8-3de6-11f1-8001-daa6852bd5a7
2026-04-22 00:50:57,264 - distributed.scheduler - INFO - Receive client connection: Client-52e00e73-3de5-11f1-8001-daa6852bd5a7
2026-04-22 00:40:55,815 - distributed.scheduler - INFO - Receive client connection: Client-ec620886-3de3-11f1-8001-daa6852bd5a7
2026-04-22 00:30:54,285 - distributed.scheduler - INFO - Receive client connection: Client-85d7ced1-3de2-11f1-8001-daa6852bd5a7
2026-04-22 00:20:52,708 - distributed.scheduler - INFO - Receive client connection: Client-1f46465d-3de1-11f1-8001-daa6852bd5a7
2026-04-22 00:12:51,525 - distributed.scheduler - INFO - Receive client connection: Client-007883e2-3de0-11f1-8001-daa6852bd5a7
2026-04-22 00:00:49,707 - distributed.scheduler - INFO - Receive client connection: Client-523b7cfd-3dde-11f1-8001-daa6852bd5a7
2026-04-21 23:50:48,234 - distributed.scheduler - INFO - Receive client connection: Client-ebb9c103-3ddc-11f1-8001-daa6852bd5a7
2026-04-21 23:40:46,739 - distributed.scheduler - INFO - Receive client connection: Client-8534c685-3ddb-11f1-8001-daa6852bd5a7
2026-04-21 23:30:49,886 - distributed.scheduler - INFO - Receive client connection: Client-worker-2173b8cd-3dda-11f1-8bfe-ae00acbfc48e
2026-04-21 23:30:49,255 - distributed.scheduler - INFO - Receive client connection: Client-worker-21148a86-3dda-11f1-8c24-2eb33e86e1bd
2026-04-21 23:30:45,213 - distributed.scheduler - INFO - Receive client connection: Client-1eaae4ba-3dda-11f1-8001-daa6852bd5a7
2026-04-21 23:20:43,716 - distributed.scheduler - INFO - Receive client connection: Client-b825f5c9-3dd8-11f1-8001-daa6852bd5a7
2026-04-21 23:12:10,416 - distributed.scheduler - INFO - Receive client connection: Client-86337ce5-3dd7-11f1-8001-daa6852bd5a7
2026-04-21 23:00:40,816 - distributed.scheduler - INFO - Receive client connection: Client-eb29dfce-3dd5-11f1-8001-daa6852bd5a7
2026-04-21 22:50:39,358 - distributed.scheduler - INFO - Receive client connection: Client-84abab04-3dd4-11f1-8001-daa6852bd5a7
2026-04-21 22:40:37,999 - distributed.scheduler - INFO - Receive client connection: Client-1e3ae033-3dd3-11f1-8001-daa6852bd5a7
2026-04-21 22:30:36,513 - distributed.scheduler - INFO - Receive client connection: Client-b7b7bfe1-3dd1-11f1-8001-daa6852bd5a7
2026-04-21 22:20:34,963 - distributed.scheduler - INFO - Receive client connection: Client-512a71f2-3dd0-11f1-8001-daa6852bd5a7
2026-04-21 22:12:28,979 - distributed.scheduler - INFO - Receive client connection: Client-2f7f8dac-3dcf-11f1-8001-daa6852bd5a7
2026-04-21 22:00:31,982 - distributed.scheduler - INFO - Receive client connection: Client-8421e40d-3dcd-11f1-8001-daa6852bd5a7
2026-04-21 21:50:30,478 - distributed.scheduler - INFO - Receive client connection: Client-1d9bf34b-3dcc-11f1-8001-daa6852bd5a7
2026-04-21 21:40:29,045 - distributed.scheduler - INFO - Receive client connection: Client-b71f7b22-3dca-11f1-8001-daa6852bd5a7
2026-04-21 21:30:27,540 - distributed.scheduler - INFO - Receive client connection: Client-509a6d3b-3dc9-11f1-8001-daa6852bd5a7
2026-04-21 21:20:26,053 - distributed.scheduler - INFO - Receive client connection: Client-ea11c76e-3dc7-11f1-8001-daa6852bd5a7
2026-04-21 21:12:10,742 - distributed.scheduler - INFO - Receive client connection: Client-c2dc6c6d-3dc6-11f1-8001-daa6852bd5a7
2026-04-21 21:00:23,018 - distributed.scheduler - INFO - Receive client connection: Client-1d05ea77-3dc5-11f1-8001-daa6852bd5a7
2026-04-21 20:50:21,575 - distributed.scheduler - INFO - Receive client connection: Client-b688c753-3dc3-11f1-8001-daa6852bd5a7
2026-04-21 20:40:20,101 - distributed.scheduler - INFO - Receive client connection: Client-5006fa35-3dc2-11f1-8001-daa6852bd5a7
2026-04-21 20:30:18,609 - distributed.scheduler - INFO - Receive client connection: Client-e982f36f-3dc0-11f1-8001-daa6852bd5a7
2026-04-21 20:20:17,121 - distributed.scheduler - INFO - Receive client connection: Client-82ff14d5-3dbf-11f1-8001-daa6852bd5a7
2026-04-21 20:12:30,613 - distributed.scheduler - INFO - Receive client connection: Client-6cf001b2-3dbe-11f1-8001-daa6852bd5a7
2026-04-21 20:00:18,104 - distributed.scheduler - INFO - Receive client connection: Client-worker-b84a1948-3dbc-11f1-8c9a-4ef3162b8ac2
2026-04-21 20:00:17,846 - distributed.scheduler - INFO - Receive client connection: Client-worker-b82d7061-3dbc-11f1-8b82-8288bfee4388
2026-04-21 20:00:17,378 - distributed.scheduler - INFO - Receive client connection: Client-worker-b7e5c68c-3dbc-11f1-8ba2-0e4c34ff5e38
2026-04-21 20:00:17,189 - distributed.scheduler - INFO - Receive client connection: Client-worker-b7c9500a-3dbc-11f1-8c16-2e20a4008324
2026-04-21 20:00:14,132 - distributed.scheduler - INFO - Receive client connection: Client-b5f59eaf-3dbc-11f1-8001-daa6852bd5a7
2026-04-21 19:50:12,631 - distributed.scheduler - INFO - Receive client connection: Client-4f7034e6-3dbb-11f1-8001-daa6852bd5a7
2026-04-21 19:40:11,176 - distributed.scheduler - INFO - Receive client connection: Client-e8f0a7a6-3db9-11f1-8001-daa6852bd5a7
2026-04-21 19:30:14,647 - distributed.scheduler - INFO - Receive client connection: Client-worker-855ea4c4-3db8-11f1-8c70-86acc56fd09f
2026-04-21 19:30:14,487 - distributed.scheduler - INFO - Receive client connection: Client-worker-854ae10c-3db8-11f1-8d61-f28f846166bd
2026-04-21 19:30:14,350 - distributed.scheduler - INFO - Receive client connection: Client-worker-852fd3f1-3db8-11f1-8c78-6ec63a5ab21e
2026-04-21 19:30:13,803 - distributed.scheduler - INFO - Receive client connection: Client-worker-84dc71ca-3db8-11f1-8be9-2a51d3dc178e
2026-04-21 19:30:13,696 - distributed.scheduler - INFO - Receive client connection: Client-worker-84d0b037-3db8-11f1-8c14-ee7435aaba1f
2026-04-21 19:30:13,481 - distributed.scheduler - INFO - Receive client connection: Client-worker-84b10888-3db8-11f1-8bdc-7e353d94dc10
2026-04-21 19:30:09,702 - distributed.scheduler - INFO - Receive client connection: Client-826ee8e1-3db8-11f1-8001-daa6852bd5a7
2026-04-21 19:20:08,234 - distributed.scheduler - INFO - Receive client connection: Client-1befbc60-3db7-11f1-8001-daa6852bd5a7
2026-04-21 19:13:11,990 - distributed.scheduler - INFO - Receive client connection: Client-23d54afc-3db6-11f1-8001-daa6852bd5a7
2026-04-21 19:01:11,608 - distributed.scheduler - INFO - Receive client connection: Client-76722d87-3db4-11f1-8007-924c5ad31268
2026-04-21 19:01:10,138 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:41513
2026-04-21 19:01:10,136 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:41513', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:10,061 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:46865
2026-04-21 19:01:10,060 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:46865', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:10,043 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:44513
2026-04-21 19:01:10,043 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:44513', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,996 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:46675
2026-04-21 19:01:09,995 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:46675', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,889 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:41633
2026-04-21 19:01:09,888 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:41633', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,865 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:44385
2026-04-21 19:01:09,864 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:44385', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,862 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:33155
2026-04-21 19:01:09,861 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:33155', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,858 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:38121
2026-04-21 19:01:09,857 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:38121', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,787 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:37883
2026-04-21 19:01:09,786 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:37883', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,631 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:42045
2026-04-21 19:01:09,631 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:42045', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,630 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:36753
2026-04-21 19:01:09,629 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:36753', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,613 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:38267
2026-04-21 19:01:09,612 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:38267', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,608 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:46363
2026-04-21 19:01:09,608 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:46363', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,595 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:44367
2026-04-21 19:01:09,594 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:44367', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,541 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:44739
2026-04-21 19:01:09,540 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:44739', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,259 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:46083', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:09,164 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:46717', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,957 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:39443', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,916 - distributed.scheduler - INFO - Close client connection: Client-257b5566-3dac-11f1-8008-924c5ad31268
2026-04-21 19:01:08,916 - distributed.scheduler - INFO - Remove client Client-257b5566-3dac-11f1-8008-924c5ad31268
2026-04-21 19:01:08,915 - distributed.scheduler - INFO - Remove client Client-257b5566-3dac-11f1-8008-924c5ad31268
2026-04-21 19:01:08,911 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:36515', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,909 - distributed.scheduler - INFO - Close client connection: Client-worker-634e3904-3db4-11f1-8c2f-86acc56fd09f
2026-04-21 19:01:08,895 - distributed.scheduler - INFO - Remove client Client-worker-634e3904-3db4-11f1-8c2f-86acc56fd09f
2026-04-21 19:01:08,894 - distributed.scheduler - INFO - Task run-552331c6b8985d8735303d3d126bc3d1 marked as failed because 3 workers died while trying to run it
2026-04-21 19:01:08,893 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:33601', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,811 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:34457', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,803 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:34131', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,745 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:34795', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,713 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:40211', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,698 - distributed.scheduler - INFO - Close client connection: Client-worker-643115c3-3db4-11f1-8b60-0e4c34ff5e38
2026-04-21 19:01:08,690 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:38587', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,685 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:37393', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,683 - distributed.scheduler - INFO - Remove client Client-worker-643115c3-3db4-11f1-8b60-0e4c34ff5e38
2026-04-21 19:01:08,650 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:41133', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,569 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:44183', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,548 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:45705', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,414 - distributed.scheduler - INFO - Close client connection: Client-worker-643bb9b6-3db4-11f1-8b64-c2349ba99f21
2026-04-21 19:01:08,406 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:35115', status: closed, memory: 0, processing: 0>
2026-04-21 19:01:08,403 - distributed.scheduler - INFO - Remove client Client-worker-643bb9b6-3db4-11f1-8b64-c2349ba99f21
2026-04-21 19:00:41,046 - distributed.scheduler - INFO - Receive client connection: Client-worker-643bb9b6-3db4-11f1-8b64-c2349ba99f21
2026-04-21 19:00:40,974 - distributed.scheduler - INFO - Receive client connection: Client-worker-643115c3-3db4-11f1-8b60-0e4c34ff5e38
2026-04-21 19:00:39,489 - distributed.scheduler - INFO - Receive client connection: Client-worker-634e3904-3db4-11f1-8c2f-86acc56fd09f
2026-04-21 19:00:39,346 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:36515
2026-04-21 19:00:39,345 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:36515', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:39,180 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:34457
2026-04-21 19:00:39,179 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:34457', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:39,161 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:39443
2026-04-21 19:00:39,160 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:39443', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:39,157 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:46083
2026-04-21 19:00:39,156 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:46083', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:39,109 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:46717
2026-04-21 19:00:39,107 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:46717', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,934 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:34131
2026-04-21 19:00:38,932 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:34131', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,864 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:34795
2026-04-21 19:00:38,862 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:34795', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,836 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:38587
2026-04-21 19:00:38,835 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:38587', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,777 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:37393
2026-04-21 19:00:38,776 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:37393', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,754 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:33601
2026-04-21 19:00:38,753 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:33601', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,708 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:35115
2026-04-21 19:00:38,708 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:35115', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,675 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:40211
2026-04-21 19:00:38,673 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:40211', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,595 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:45705
2026-04-21 19:00:38,594 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:45705', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,571 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:41133
2026-04-21 19:00:38,570 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:41133', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,532 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:44183
2026-04-21 19:00:38,530 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:44183', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,281 - distributed.scheduler - INFO - Close client connection: Client-worker-517594e1-3db4-11f1-8bb1-fae0fcb74df3
2026-04-21 19:00:38,281 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:42985', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,279 - distributed.scheduler - INFO - Remove client Client-worker-517594e1-3db4-11f1-8bb1-fae0fcb74df3
2026-04-21 19:00:38,259 - distributed.scheduler - INFO - Close client connection: Client-worker-24a8d7c7-3dac-11f1-8b82-ae00acbfc48e
2026-04-21 19:00:38,256 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:41665', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,255 - distributed.scheduler - INFO - Remove client Client-worker-24a8d7c7-3dac-11f1-8b82-ae00acbfc48e
2026-04-21 19:00:38,224 - distributed.scheduler - INFO - Close client connection: Client-worker-24bd6c33-3dac-11f1-8c1d-4ef3162b8ac2
2026-04-21 19:00:38,223 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:38377', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,221 - distributed.scheduler - INFO - Remove client Client-worker-24bd6c33-3dac-11f1-8c1d-4ef3162b8ac2
2026-04-21 19:00:38,168 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:37519', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,114 - distributed.scheduler - INFO - Close client connection: Client-worker-1e0a535f-3db0-11f1-8ba6-2e20a4008324
2026-04-21 19:00:38,111 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:44617', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:38,110 - distributed.scheduler - INFO - Remove client Client-worker-1e0a535f-3db0-11f1-8ba6-2e20a4008324
2026-04-21 19:00:37,946 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:37417', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:37,946 - distributed.scheduler - INFO - Close client connection: Client-worker-5114865c-3db4-11f1-8b72-2a51d3dc178e
2026-04-21 19:00:37,946 - distributed.scheduler - INFO - Remove client Client-worker-5114865c-3db4-11f1-8b72-2a51d3dc178e
2026-04-21 19:00:37,866 - distributed.scheduler - INFO - Close client connection: Client-worker-50ca15e1-3db4-11f1-8ced-f28f846166bd
2026-04-21 19:00:37,866 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:44285', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:37,864 - distributed.scheduler - INFO - Remove client Client-worker-50ca15e1-3db4-11f1-8ced-f28f846166bd
2026-04-21 19:00:37,856 - distributed.scheduler - INFO - Close client connection: Client-worker-1d8b159c-3db0-11f1-8b23-0e4c34ff5e38
2026-04-21 19:00:37,854 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:41001', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:37,853 - distributed.scheduler - INFO - Remove client Client-worker-1d8b159c-3db0-11f1-8b23-0e4c34ff5e38
2026-04-21 19:00:37,828 - distributed.scheduler - INFO - Close client connection: Client-worker-1da43259-3db0-11f1-8bda-566b4744c386
2026-04-21 19:00:37,825 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:39315', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:37,823 - distributed.scheduler - INFO - Remove client Client-worker-1da43259-3db0-11f1-8bda-566b4744c386
2026-04-21 19:00:37,790 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:43749', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:37,735 - distributed.scheduler - INFO - Close client connection: Client-worker-24fbea0b-3dac-11f1-8b24-c2349ba99f21
2026-04-21 19:00:37,730 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:44815', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:37,701 - distributed.scheduler - INFO - Remove client Client-worker-24fbea0b-3dac-11f1-8b24-c2349ba99f21
2026-04-21 19:00:37,659 - distributed.scheduler - INFO - Close client connection: Client-worker-1dae60fb-3db0-11f1-8b07-8288bfee4388
2026-04-21 19:00:37,658 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:46313', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:37,657 - distributed.scheduler - INFO - Remove client Client-worker-1dae60fb-3db0-11f1-8b07-8288bfee4388
2026-04-21 19:00:37,593 - distributed.scheduler - INFO - Close client connection: Client-worker-1e1bf3c5-3db0-11f1-8bfe-6ec63a5ab21e
2026-04-21 19:00:37,585 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:44053', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:37,583 - distributed.scheduler - INFO - Remove client Client-worker-1e1bf3c5-3db0-11f1-8bfe-6ec63a5ab21e
2026-04-21 19:00:37,570 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:34431', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:37,531 - distributed.scheduler - INFO - Close client connection: Client-worker-1de478ad-3db0-11f1-8b9b-ee7435aaba1f
2026-04-21 19:00:37,521 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:39899', status: closed, memory: 0, processing: 0>
2026-04-21 19:00:37,518 - distributed.scheduler - INFO - Remove client Client-worker-1de478ad-3db0-11f1-8b9b-ee7435aaba1f
2026-04-21 19:00:09,552 - distributed.scheduler - INFO - Receive client connection: Client-worker-517594e1-3db4-11f1-8bb1-fae0fcb74df3
2026-04-21 19:00:08,912 - distributed.scheduler - INFO - Receive client connection: Client-worker-5114865c-3db4-11f1-8b72-2a51d3dc178e
2026-04-21 19:00:08,619 - distributed.scheduler - INFO - Receive client connection: Client-worker-50e9d189-3db4-11f1-8c2d-b6f02dd406e9
2026-04-21 19:00:08,422 - distributed.scheduler - INFO - Receive client connection: Client-worker-50ca15e1-3db4-11f1-8ced-f28f846166bd
2026-04-21 19:00:05,307 - distributed.scheduler - INFO - Receive client connection: Client-4eeb8bd1-3db4-11f1-8001-daa6852bd5a7
2026-04-21 18:50:03,829 - distributed.scheduler - INFO - Receive client connection: Client-e86aea01-3db2-11f1-8001-daa6852bd5a7
2026-04-21 18:40:36,593 - distributed.scheduler - INFO - Close client connection: Client-81daf7fa-3db1-11f1-8001-daa6852bd5a7
2026-04-21 18:40:36,592 - distributed.scheduler - INFO - Remove client Client-81daf7fa-3db1-11f1-8001-daa6852bd5a7
2026-04-21 18:40:36,590 - distributed.scheduler - INFO - Remove client Client-81daf7fa-3db1-11f1-8001-daa6852bd5a7
2026-04-21 18:40:02,266 - distributed.scheduler - INFO - Receive client connection: Client-81daf7fa-3db1-11f1-8001-daa6852bd5a7
2026-04-21 18:31:03,475 - distributed.scheduler - INFO - Close client connection: Client-1b5fe2c6-3db0-11f1-8001-daa6852bd5a7
2026-04-21 18:31:03,474 - distributed.scheduler - INFO - Remove client Client-1b5fe2c6-3db0-11f1-8001-daa6852bd5a7
2026-04-21 18:31:03,472 - distributed.scheduler - INFO - Remove client Client-1b5fe2c6-3db0-11f1-8001-daa6852bd5a7
2026-04-21 18:30:05,402 - distributed.scheduler - INFO - Receive client connection: Client-worker-1e1bf3c5-3db0-11f1-8bfe-6ec63a5ab21e
2026-04-21 18:30:05,285 - distributed.scheduler - INFO - Receive client connection: Client-worker-1e0a535f-3db0-11f1-8ba6-2e20a4008324
2026-04-21 18:30:05,081 - distributed.scheduler - INFO - Receive client connection: Client-worker-1de478ad-3db0-11f1-8b9b-ee7435aaba1f
2026-04-21 18:30:04,703 - distributed.scheduler - INFO - Receive client connection: Client-worker-1dae60fb-3db0-11f1-8b07-8288bfee4388
2026-04-21 18:30:04,617 - distributed.scheduler - INFO - Receive client connection: Client-worker-1da43259-3db0-11f1-8bda-566b4744c386
2026-04-21 18:30:04,454 - distributed.scheduler - INFO - Receive client connection: Client-worker-1d8b159c-3db0-11f1-8b23-0e4c34ff5e38
2026-04-21 18:30:00,823 - distributed.scheduler - INFO - Receive client connection: Client-1b5fe2c6-3db0-11f1-8001-daa6852bd5a7
2026-04-21 18:21:35,371 - distributed.scheduler - INFO - Close client connection: Client-d8ba29e0-3dae-11f1-8001-daa6852bd5a7
2026-04-21 18:21:35,370 - distributed.scheduler - INFO - Remove client Client-d8ba29e0-3dae-11f1-8001-daa6852bd5a7
2026-04-21 18:21:35,368 - distributed.scheduler - INFO - Remove client Client-d8ba29e0-3dae-11f1-8001-daa6852bd5a7
2026-04-21 18:20:59,517 - distributed.scheduler - INFO - Receive client connection: Client-d8ba29e0-3dae-11f1-8001-daa6852bd5a7
2026-04-21 18:14:00,807 - distributed.scheduler - INFO - Close client connection: Client-cb5a3329-3dad-11f1-8001-daa6852bd5a7
2026-04-21 18:14:00,805 - distributed.scheduler - INFO - Remove client Client-cb5a3329-3dad-11f1-8001-daa6852bd5a7
2026-04-21 18:14:00,803 - distributed.scheduler - INFO - Remove client Client-cb5a3329-3dad-11f1-8001-daa6852bd5a7
2026-04-21 18:13:27,580 - distributed.scheduler - INFO - Receive client connection: Client-cb5a3329-3dad-11f1-8001-daa6852bd5a7
2026-04-21 18:02:28,870 - distributed.scheduler - INFO - Close client connection: Client-0bb2f39e-3dac-11f1-8001-daa6852bd5a7
2026-04-21 18:02:28,869 - distributed.scheduler - INFO - Remove client Client-0bb2f39e-3dac-11f1-8001-daa6852bd5a7
2026-04-21 18:02:28,867 - distributed.scheduler - INFO - Remove client Client-0bb2f39e-3dac-11f1-8001-daa6852bd5a7
2026-04-21 18:01:39,929 - distributed.scheduler - INFO - Receive client connection: Client-257b5566-3dac-11f1-8008-924c5ad31268
2026-04-21 18:01:38,982 - distributed.scheduler - INFO - Receive client connection: Client-worker-24fbea0b-3dac-11f1-8b24-c2349ba99f21
2026-04-21 18:01:38,583 - distributed.scheduler - INFO - Receive client connection: Client-worker-24bd6c33-3dac-11f1-8c1d-4ef3162b8ac2
2026-04-21 18:01:38,425 - distributed.scheduler - INFO - Receive client connection: Client-worker-24a8d7c7-3dac-11f1-8b82-ae00acbfc48e
2026-04-21 18:01:37,901 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:41001
2026-04-21 18:01:37,900 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:41001', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:37,560 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:44617
2026-04-21 18:01:37,559 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:44617', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:37,336 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:39315
2026-04-21 18:01:37,334 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:39315', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:37,327 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:37519
2026-04-21 18:01:37,326 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:37519', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:37,299 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:39899
2026-04-21 18:01:37,299 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:39899', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:37,127 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:44053
2026-04-21 18:01:37,126 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:44053', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:37,047 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:44285
2026-04-21 18:01:37,046 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:44285', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,991 - distributed.scheduler - INFO - Close client connection: Client-591f13b5-3d97-11f1-8007-924c5ad31268
2026-04-21 18:01:36,990 - distributed.scheduler - INFO - Remove client Client-591f13b5-3d97-11f1-8007-924c5ad31268
2026-04-21 18:01:36,988 - distributed.scheduler - INFO - Remove client Client-591f13b5-3d97-11f1-8007-924c5ad31268
2026-04-21 18:01:36,973 - distributed.scheduler - INFO - Close client connection: Client-worker-9298b462-3d9f-11f1-8ade-0e4c34ff5e38
2026-04-21 18:01:36,969 - distributed.scheduler - INFO - Remove client Client-worker-9298b462-3d9f-11f1-8ade-0e4c34ff5e38
2026-04-21 18:01:36,958 - distributed.scheduler - INFO - Task run-30a11945490277dcdd22527e086895f2 marked as failed because 3 workers died while trying to run it
2026-04-21 18:01:36,957 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:34281', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,949 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:42985
2026-04-21 18:01:36,948 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:42985', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,857 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:43749
2026-04-21 18:01:36,856 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:43749', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,812 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:44763
2026-04-21 18:01:36,806 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:44763', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,749 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:37417
2026-04-21 18:01:36,748 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:37417', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,701 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:34431
2026-04-21 18:01:36,700 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:34431', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,674 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:46313
2026-04-21 18:01:36,673 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:46313', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,597 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:44815
2026-04-21 18:01:36,596 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:44815', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,534 - distributed.scheduler - INFO - Close client connection: Client-worker-9e70c805-3d9f-11f1-8b6d-2e20a4008324
2026-04-21 18:01:36,529 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:43313', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,528 - distributed.scheduler - INFO - Remove client Client-worker-9e70c805-3d9f-11f1-8b6d-2e20a4008324
2026-04-21 18:01:36,518 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:41665
2026-04-21 18:01:36,517 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:41665', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,369 - distributed.scheduler - INFO - Close client connection: Client-worker-935c0e9f-3d9f-11f1-8b67-2eb33e86e1bd
2026-04-21 18:01:36,368 - distributed.scheduler - INFO - Close client connection: Client-worker-925d52fc-3d9f-11f1-8b8a-566b4744c386
2026-04-21 18:01:36,364 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:44467', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,363 - distributed.scheduler - INFO - Remove client Client-worker-935c0e9f-3d9f-11f1-8b67-2eb33e86e1bd
2026-04-21 18:01:36,357 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:36579', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,333 - distributed.scheduler - INFO - Remove client Client-worker-925d52fc-3d9f-11f1-8b8a-566b4744c386
2026-04-21 18:01:36,246 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:38377
2026-04-21 18:01:36,246 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:38377', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,245 - distributed.scheduler - INFO - Close client connection: Client-worker-a78ff0ee-3da3-11f1-8b54-ee7435aaba1f
2026-04-21 18:01:36,239 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:41113', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,215 - distributed.scheduler - INFO - Remove client Client-worker-a78ff0ee-3da3-11f1-8b54-ee7435aaba1f
2026-04-21 18:01:36,150 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:42313', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,149 - distributed.scheduler - INFO - Close client connection: Client-worker-c421b043-3d9f-11f1-8bbe-6ec63a5ab21e
2026-04-21 18:01:36,149 - distributed.scheduler - INFO - Remove client Client-worker-c421b043-3d9f-11f1-8bbe-6ec63a5ab21e
2026-04-21 18:01:36,138 - distributed.scheduler - INFO - Close client connection: Client-worker-91c49a74-3d9f-11f1-8b75-fae0fcb74df3
2026-04-21 18:01:36,121 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:38759', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,120 - distributed.scheduler - INFO - Remove client Client-worker-91c49a74-3d9f-11f1-8b75-fae0fcb74df3
2026-04-21 18:01:36,111 - distributed.scheduler - INFO - Close client connection: Client-worker-daf24bb3-3da7-11f1-8cac-f28f846166bd
2026-04-21 18:01:36,101 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:37575', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:36,099 - distributed.scheduler - INFO - Remove client Client-worker-daf24bb3-3da7-11f1-8cac-f28f846166bd
2026-04-21 18:01:35,915 - distributed.scheduler - INFO - Close client connection: Client-worker-dab400dd-3da7-11f1-8bf0-b6f02dd406e9
2026-04-21 18:01:35,906 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:41203', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:35,904 - distributed.scheduler - INFO - Remove client Client-worker-dab400dd-3da7-11f1-8bf0-b6f02dd406e9
2026-04-21 18:01:35,832 - distributed.scheduler - INFO - Close client connection: Client-worker-a7ca6ff7-3da3-11f1-8bb5-86acc56fd09f
2026-04-21 18:01:35,820 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:44525', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:35,820 - distributed.scheduler - INFO - Remove client Client-worker-a7ca6ff7-3da3-11f1-8bb5-86acc56fd09f
2026-04-21 18:01:35,818 - distributed.scheduler - INFO - Close client connection: Client-worker-b85f605a-3d9f-11f1-8b20-2a51d3dc178e
2026-04-21 18:01:35,810 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:43429', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:35,780 - distributed.scheduler - INFO - Remove client Client-worker-b85f605a-3d9f-11f1-8b20-2a51d3dc178e
2026-04-21 18:01:35,741 - distributed.scheduler - INFO - Close client connection: Client-worker-a73a314b-3da3-11f1-8ac5-8288bfee4388
2026-04-21 18:01:35,732 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:35625', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:35,730 - distributed.scheduler - INFO - Remove client Client-worker-a73a314b-3da3-11f1-8ac5-8288bfee4388
2026-04-21 18:01:35,688 - distributed.scheduler - INFO - Close client connection: Client-worker-db098354-3da7-11f1-8b32-7e353d94dc10
2026-04-21 18:01:35,676 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:42355', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:35,674 - distributed.scheduler - INFO - Remove client Client-worker-db098354-3da7-11f1-8b32-7e353d94dc10
2026-04-21 18:01:35,593 - distributed.scheduler - INFO - Close client connection: Client-worker-0e9c6478-3dac-11f1-8b40-ae00acbfc48e
2026-04-21 18:01:35,581 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:34207', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:35,577 - distributed.scheduler - INFO - Remove client Client-worker-0e9c6478-3dac-11f1-8b40-ae00acbfc48e
2026-04-21 18:01:35,457 - distributed.scheduler - INFO - Close client connection: Client-worker-c016e2f8-3d9f-11f1-8ae5-c2349ba99f21
2026-04-21 18:01:35,430 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:33263', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:35,428 - distributed.scheduler - INFO - Remove client Client-worker-c016e2f8-3d9f-11f1-8ae5-c2349ba99f21
2026-04-21 18:01:35,298 - distributed.scheduler - INFO - Close client connection: Client-worker-9d6d661c-3d9f-11f1-8bd6-4ef3162b8ac2
2026-04-21 18:01:35,293 - distributed.scheduler - INFO - Remove client Client-worker-9d6d661c-3d9f-11f1-8bd6-4ef3162b8ac2
2026-04-21 18:01:35,273 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:45229', status: closed, memory: 0, processing: 0>
2026-04-21 18:01:01,488 - distributed.scheduler - INFO - Receive client connection: Client-worker-0e9c6478-3dac-11f1-8b40-ae00acbfc48e
2026-04-21 18:00:56,548 - distributed.scheduler - INFO - Receive client connection: Client-0bb2f39e-3dac-11f1-8001-daa6852bd5a7
2026-04-21 17:51:35,203 - distributed.scheduler - INFO - Close client connection: Client-a528d14e-3daa-11f1-8001-daa6852bd5a7
2026-04-21 17:51:35,202 - distributed.scheduler - INFO - Remove client Client-a528d14e-3daa-11f1-8001-daa6852bd5a7
2026-04-21 17:51:35,200 - distributed.scheduler - INFO - Remove client Client-a528d14e-3daa-11f1-8001-daa6852bd5a7
2026-04-21 17:50:55,016 - distributed.scheduler - INFO - Receive client connection: Client-a528d14e-3daa-11f1-8001-daa6852bd5a7
2026-04-21 17:41:32,314 - distributed.scheduler - INFO - Close client connection: Client-3e96f089-3da9-11f1-8001-daa6852bd5a7
2026-04-21 17:41:32,313 - distributed.scheduler - INFO - Remove client Client-3e96f089-3da9-11f1-8001-daa6852bd5a7
2026-04-21 17:41:32,312 - distributed.scheduler - INFO - Remove client Client-3e96f089-3da9-11f1-8001-daa6852bd5a7
2026-04-21 17:40:53,436 - distributed.scheduler - INFO - Receive client connection: Client-3e96f089-3da9-11f1-8001-daa6852bd5a7
2026-04-21 17:32:04,239 - distributed.scheduler - INFO - Close client connection: Client-d813651e-3da7-11f1-8001-daa6852bd5a7
2026-04-21 17:32:04,237 - distributed.scheduler - INFO - Remove client Client-d813651e-3da7-11f1-8001-daa6852bd5a7
2026-04-21 17:32:04,236 - distributed.scheduler - INFO - Remove client Client-d813651e-3da7-11f1-8001-daa6852bd5a7
2026-04-21 17:30:56,918 - distributed.scheduler - INFO - Receive client connection: Client-worker-db098354-3da7-11f1-8b32-7e353d94dc10
2026-04-21 17:30:56,764 - distributed.scheduler - INFO - Receive client connection: Client-worker-daf24bb3-3da7-11f1-8cac-f28f846166bd
2026-04-21 17:30:56,342 - distributed.scheduler - INFO - Receive client connection: Client-worker-dab400dd-3da7-11f1-8bf0-b6f02dd406e9
2026-04-21 17:30:51,948 - distributed.scheduler - INFO - Receive client connection: Client-d813651e-3da7-11f1-8001-daa6852bd5a7
2026-04-21 17:21:30,328 - distributed.scheduler - INFO - Close client connection: Client-71944fae-3da6-11f1-8001-daa6852bd5a7
2026-04-21 17:21:30,326 - distributed.scheduler - INFO - Remove client Client-71944fae-3da6-11f1-8001-daa6852bd5a7
2026-04-21 17:21:30,325 - distributed.scheduler - INFO - Remove client Client-71944fae-3da6-11f1-8001-daa6852bd5a7
2026-04-21 17:20:50,491 - distributed.scheduler - INFO - Receive client connection: Client-71944fae-3da6-11f1-8001-daa6852bd5a7
2026-04-21 17:14:25,525 - distributed.scheduler - INFO - Close client connection: Client-77557b33-3da5-11f1-8001-daa6852bd5a7
2026-04-21 17:14:25,524 - distributed.scheduler - INFO - Remove client Client-77557b33-3da5-11f1-8001-daa6852bd5a7
2026-04-21 17:14:25,524 - distributed.scheduler - INFO - Remove client Client-77557b33-3da5-11f1-8001-daa6852bd5a7
2026-04-21 17:13:50,650 - distributed.scheduler - INFO - Receive client connection: Client-77557b33-3da5-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,123 - distributed.scheduler - INFO - Close client connection: Client-d724bb8c-3d99-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,123 - distributed.scheduler - INFO - Close client connection: Client-3da644f3-3d9b-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,122 - distributed.scheduler - INFO - Close client connection: Client-0a957cda-3d9e-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,122 - distributed.scheduler - INFO - Close client connection: Client-d78980d0-3da0-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,122 - distributed.scheduler - INFO - Close client connection: Client-709a7610-3d98-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,122 - distributed.scheduler - INFO - Close client connection: Client-3e0afcef-3da2-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,121 - distributed.scheduler - INFO - Close client connection: Client-71110190-3d9f-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,121 - distributed.scheduler - INFO - Close client connection: Client-0b232ffc-3d9d-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,119 - distributed.scheduler - INFO - Remove client Client-d724bb8c-3d99-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,118 - distributed.scheduler - INFO - Remove client Client-3da644f3-3d9b-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,117 - distributed.scheduler - INFO - Remove client Client-0a957cda-3d9e-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,117 - distributed.scheduler - INFO - Remove client Client-d78980d0-3da0-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,115 - distributed.scheduler - INFO - Remove client Client-709a7610-3d98-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,114 - distributed.scheduler - INFO - Remove client Client-3e0afcef-3da2-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,110 - distributed.scheduler - INFO - Remove client Client-71110190-3d9f-11f1-8001-daa6852bd5a7
2026-04-21 17:02:00,109 - distributed.scheduler - INFO - Remove client Client-0b232ffc-3d9d-11f1-8001-daa6852bd5a7
2026-04-21 17:01:59,461 - distributed.scheduler - INFO - Close client connection: Client-a4931702-3da3-11f1-8001-daa6852bd5a7
2026-04-21 17:01:59,460 - distributed.scheduler - INFO - Remove client Client-a4931702-3da3-11f1-8001-daa6852bd5a7
2026-04-21 17:01:59,458 - distributed.scheduler - INFO - Remove client Client-a4931702-3da3-11f1-8001-daa6852bd5a7
2026-04-21 17:00:53,359 - distributed.scheduler - INFO - Receive client connection: Client-worker-a7ca6ff7-3da3-11f1-8bb5-86acc56fd09f
2026-04-21 17:00:52,566 - distributed.scheduler - INFO - Receive client connection: Client-worker-a78ff0ee-3da3-11f1-8b54-ee7435aaba1f
2026-04-21 17:00:51,995 - distributed.scheduler - INFO - Receive client connection: Client-worker-a73a314b-3da3-11f1-8ac5-8288bfee4388
2026-04-21 17:00:47,546 - distributed.scheduler - INFO - Receive client connection: Client-a4931702-3da3-11f1-8001-daa6852bd5a7
2026-04-21 16:50:46,037 - distributed.scheduler - INFO - Receive client connection: Client-3e0afcef-3da2-11f1-8001-daa6852bd5a7
2026-04-21 16:40:44,553 - distributed.scheduler - INFO - Receive client connection: Client-d78980d0-3da0-11f1-8001-daa6852bd5a7
2026-04-21 16:33:02,498 - distributed.scheduler - INFO - Receive client connection: Client-worker-c421b043-3d9f-11f1-8bbe-6ec63a5ab21e
2026-04-21 16:32:55,718 - distributed.scheduler - INFO - Receive client connection: Client-worker-c016e2f8-3d9f-11f1-8ae5-c2349ba99f21
2026-04-21 16:32:42,772 - distributed.scheduler - INFO - Receive client connection: Client-worker-b85f605a-3d9f-11f1-8b20-2a51d3dc178e
2026-04-21 16:32:25,950 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-563e57812a57c5e2cf60e05b09e36322 NoneType: None
2026-04-21 16:32:25,949 - distributed.scheduler - ERROR - Couldn't gather keys {'run-563e57812a57c5e2cf60e05b09e36322': []} state: ['memory'] workers: [] NoneType: None
2026-04-21 16:31:59,278 - distributed.scheduler - INFO - Receive client connection: Client-worker-9e70c805-3d9f-11f1-8b6d-2e20a4008324
2026-04-21 16:31:58,523 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:41113
2026-04-21 16:31:58,521 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:41113', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:58,340 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:44525
2026-04-21 16:31:58,338 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:44525', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:57,563 - distributed.scheduler - INFO - Receive client connection: Client-worker-9d6d661c-3d9f-11f1-8bd6-4ef3162b8ac2
2026-04-21 16:31:57,439 - distributed.scheduler - INFO - Close client connection: Client-worker-74377a22-3d9f-11f1-8b19-ee7435aaba1f
2026-04-21 16:31:57,415 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:34163', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:57,414 - distributed.scheduler - INFO - Remove client Client-worker-74377a22-3d9f-11f1-8b19-ee7435aaba1f
2026-04-21 16:31:57,291 - distributed.scheduler - INFO - Close client connection: Client-worker-73a84e9c-3d9f-11f1-8b75-86acc56fd09f
2026-04-21 16:31:57,284 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:38553', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:57,282 - distributed.scheduler - INFO - Remove client Client-worker-73a84e9c-3d9f-11f1-8b75-86acc56fd09f
2026-04-21 16:31:40,684 - distributed.scheduler - INFO - Receive client connection: Client-worker-935c0e9f-3d9f-11f1-8b67-2eb33e86e1bd
2026-04-21 16:31:39,874 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:42355
2026-04-21 16:31:39,873 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:42355', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:39,402 - distributed.scheduler - INFO - Receive client connection: Client-worker-9298b462-3d9f-11f1-8ade-0e4c34ff5e38
2026-04-21 16:31:39,070 - distributed.scheduler - INFO - Receive client connection: Client-worker-925d52fc-3d9f-11f1-8b8a-566b4744c386
2026-04-21 16:31:38,977 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:40571', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:38,004 - distributed.scheduler - INFO - Receive client connection: Client-worker-91c49a74-3d9f-11f1-8b75-fae0fcb74df3
2026-04-21 16:31:37,771 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:41203
2026-04-21 16:31:37,770 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:41203', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:37,723 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:43429
2026-04-21 16:31:37,722 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:43429', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:37,684 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:42313
2026-04-21 16:31:37,684 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:42313', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:37,676 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:34207
2026-04-21 16:31:37,676 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:34207', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:37,591 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:37575
2026-04-21 16:31:37,590 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:37575', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:37,583 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:44467
2026-04-21 16:31:37,582 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:44467', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:37,542 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:45229
2026-04-21 16:31:37,540 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:45229', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:37,537 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:33263
2026-04-21 16:31:37,537 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:33263', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:37,526 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:43313
2026-04-21 16:31:37,524 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:43313', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:37,447 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:35625
2026-04-21 16:31:37,447 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:35625', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:37,432 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:38759
2026-04-21 16:31:37,431 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:38759', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:37,354 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:34281
2026-04-21 16:31:37,353 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:34281', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:37,298 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:36579
2026-04-21 16:31:37,297 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:36579', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,735 - distributed.scheduler - INFO - Close client connection: Client-worker-409df992-3d9b-11f1-8b80-6ec63a5ab21e
2026-04-21 16:31:36,733 - distributed.scheduler - INFO - Remove client Client-worker-409df992-3d9b-11f1-8b80-6ec63a5ab21e
2026-04-21 16:31:36,729 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:38791', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,724 - distributed.scheduler - INFO - Close client connection: Client-worker-73ac52f6-3d9f-11f1-8baf-b6f02dd406e9
2026-04-21 16:31:36,720 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:41005', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,718 - distributed.scheduler - INFO - Remove client Client-worker-73ac52f6-3d9f-11f1-8baf-b6f02dd406e9
2026-04-21 16:31:36,692 - distributed.scheduler - INFO - Close client connection: Client-worker-40534805-3d9b-11f1-8adf-2a51d3dc178e
2026-04-21 16:31:36,685 - distributed.scheduler - INFO - Close client connection: Client-worker-738e55b8-3d9f-11f1-8b04-ae00acbfc48e
2026-04-21 16:31:36,682 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:34541', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,681 - distributed.scheduler - INFO - Remove client Client-worker-40534805-3d9b-11f1-8adf-2a51d3dc178e
2026-04-21 16:31:36,671 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:36673', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,669 - distributed.scheduler - INFO - Remove client Client-worker-738e55b8-3d9f-11f1-8b04-ae00acbfc48e
2026-04-21 16:31:36,632 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:36049', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,607 - distributed.scheduler - INFO - Close client connection: Client-worker-40178df6-3d9b-11f1-8c65-f28f846166bd
2026-04-21 16:31:36,597 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:43297', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,596 - distributed.scheduler - INFO - Remove client Client-worker-40178df6-3d9b-11f1-8c65-f28f846166bd
2026-04-21 16:31:36,572 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:42025', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,572 - distributed.scheduler - INFO - Close client connection: Client-worker-4057ddf0-3d9b-11f1-8b93-4ef3162b8ac2
2026-04-21 16:31:36,571 - distributed.scheduler - INFO - Remove client Client-worker-4057ddf0-3d9b-11f1-8b93-4ef3162b8ac2
2026-04-21 16:31:36,560 - distributed.scheduler - INFO - Close client connection: Client-worker-40d39b42-3d9b-11f1-8b33-fae0fcb74df3
2026-04-21 16:31:36,557 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:41249', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,556 - distributed.scheduler - INFO - Remove client Client-worker-40d39b42-3d9b-11f1-8b33-fae0fcb74df3
2026-04-21 16:31:36,504 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:34851', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,481 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:43183', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,427 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:39711', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,425 - distributed.scheduler - INFO - Close client connection: Client-worker-40c78ad6-3d9b-11f1-8b32-2e20a4008324
2026-04-21 16:31:36,418 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:32913', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,416 - distributed.scheduler - INFO - Remove client Client-worker-40c78ad6-3d9b-11f1-8b32-2e20a4008324
2026-04-21 16:31:36,357 - distributed.scheduler - INFO - Close client connection: Client-worker-403d4460-3d9b-11f1-8b46-566b4744c386
2026-04-21 16:31:36,350 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:40435', status: closed, memory: 0, processing: 0>
2026-04-21 16:31:36,321 - distributed.scheduler - INFO - Remove client Client-worker-403d4460-3d9b-11f1-8b46-566b4744c386
2026-04-21 16:30:48,817 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('len-agg-1ada082802f6905050a72421fa9d38e3', 0) NoneType: None
2026-04-21 16:30:48,817 - distributed.scheduler - ERROR - Couldn't gather keys {"('len-agg-1ada082802f6905050a72421fa9d38e3', 0)": []} state: ['waiting'] workers: [] NoneType: None
2026-04-21 16:30:48,485 - distributed.scheduler - INFO - Receive client connection: Client-worker-74377a22-3d9f-11f1-8b19-ee7435aaba1f
2026-04-21 16:30:47,524 - distributed.scheduler - INFO - Receive client connection: Client-worker-73ac52f6-3d9f-11f1-8baf-b6f02dd406e9
2026-04-21 16:30:47,485 - distributed.scheduler - INFO - Receive client connection: Client-worker-73a84e9c-3d9f-11f1-8b75-86acc56fd09f
2026-04-21 16:30:47,310 - distributed.scheduler - INFO - Receive client connection: Client-worker-738e55b8-3d9f-11f1-8b04-ae00acbfc48e
2026-04-21 16:30:43,140 - distributed.scheduler - INFO - Receive client connection: Client-71110190-3d9f-11f1-8001-daa6852bd5a7
2026-04-21 16:20:41,707 - distributed.scheduler - INFO - Receive client connection: Client-0a957cda-3d9e-11f1-8001-daa6852bd5a7
2026-04-21 16:13:33,139 - distributed.scheduler - INFO - Receive client connection: Client-0b232ffc-3d9d-11f1-8001-daa6852bd5a7
2026-04-21 16:00:44,268 - distributed.scheduler - INFO - Receive client connection: Client-worker-40d39b42-3d9b-11f1-8b33-fae0fcb74df3
2026-04-21 16:00:44,147 - distributed.scheduler - INFO - Receive client connection: Client-worker-40c78ad6-3d9b-11f1-8b32-2e20a4008324
2026-04-21 16:00:43,874 - distributed.scheduler - INFO - Receive client connection: Client-worker-409df992-3d9b-11f1-8b80-6ec63a5ab21e
2026-04-21 16:00:43,411 - distributed.scheduler - INFO - Receive client connection: Client-worker-4057ddf0-3d9b-11f1-8b93-4ef3162b8ac2
2026-04-21 16:00:43,374 - distributed.scheduler - INFO - Receive client connection: Client-worker-40534805-3d9b-11f1-8adf-2a51d3dc178e
2026-04-21 16:00:43,234 - distributed.scheduler - INFO - Receive client connection: Client-worker-403d4460-3d9b-11f1-8b46-566b4744c386
2026-04-21 16:00:42,981 - distributed.scheduler - INFO - Receive client connection: Client-worker-40178df6-3d9b-11f1-8c65-f28f846166bd
2026-04-21 16:00:38,893 - distributed.scheduler - INFO - Receive client connection: Client-3da644f3-3d9b-11f1-8001-daa6852bd5a7
2026-04-21 15:50:37,419 - distributed.scheduler - INFO - Receive client connection: Client-d724bb8c-3d99-11f1-8001-daa6852bd5a7
2026-04-21 15:40:35,899 - distributed.scheduler - INFO - Receive client connection: Client-709a7610-3d98-11f1-8001-daa6852bd5a7
2026-04-21 15:32:47,001 - distributed.scheduler - INFO - Receive client connection: Client-591f13b5-3d97-11f1-8007-924c5ad31268
2026-04-21 15:32:45,259 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:39711
2026-04-21 15:32:45,259 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:39711', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:45,239 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:38553
2026-04-21 15:32:45,238 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:38553', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:45,232 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:34163
2026-04-21 15:32:45,231 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:34163', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:45,219 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:32913
2026-04-21 15:32:45,218 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:32913', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:45,170 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:34541
2026-04-21 15:32:45,169 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:34541', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:45,167 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:43297
2026-04-21 15:32:45,166 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:43297', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:45,144 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:34851
2026-04-21 15:32:45,143 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:34851', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:45,125 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:40571
2026-04-21 15:32:45,123 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:40571', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:45,099 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:40435
2026-04-21 15:32:45,098 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:40435', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:45,080 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:38791
2026-04-21 15:32:45,079 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:38791', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:45,066 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:42025
2026-04-21 15:32:45,065 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:42025', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:45,058 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:36673
2026-04-21 15:32:45,056 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:36673', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:45,013 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:41249
2026-04-21 15:32:45,012 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:41249', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,998 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:36049
2026-04-21 15:32:44,997 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:36049', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,958 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:41005
2026-04-21 15:32:44,957 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:41005', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,938 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:43183
2026-04-21 15:32:44,937 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:43183', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,212 - distributed.scheduler - INFO - Close client connection: Client-worker-39fa7c35-3d97-11f1-8af8-2e20a4008324
2026-04-21 15:32:44,212 - distributed.scheduler - INFO - Lost all workers
2026-04-21 15:32:44,212 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:43601', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,211 - distributed.scheduler - INFO - Remove client Client-worker-39fa7c35-3d97-11f1-8af8-2e20a4008324
2026-04-21 15:32:44,187 - distributed.scheduler - INFO - Close client connection: Client-worker-38be11c1-3d97-11f1-8a4f-8288bfee4388
2026-04-21 15:32:44,187 - distributed.scheduler - INFO - Close client connection: Client-26cd73ba-3d82-11f1-8007-924c5ad31268
2026-04-21 15:32:44,184 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:46135', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,182 - distributed.scheduler - INFO - Remove client Client-worker-38be11c1-3d97-11f1-8a4f-8288bfee4388
2026-04-21 15:32:44,181 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:42571', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,174 - distributed.scheduler - INFO - Remove client Client-26cd73ba-3d82-11f1-8007-924c5ad31268
2026-04-21 15:32:44,173 - distributed.scheduler - INFO - Close client connection: Client-worker-3927e977-3d97-11f1-8af3-fae0fcb74df3
2026-04-21 15:32:44,172 - distributed.scheduler - INFO - Remove client Client-26cd73ba-3d82-11f1-8007-924c5ad31268
2026-04-21 15:32:44,170 - distributed.scheduler - INFO - Remove client Client-worker-3927e977-3d97-11f1-8af3-fae0fcb74df3
2026-04-21 15:32:44,169 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:45791', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,167 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:36283', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,165 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:41881', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,147 - distributed.scheduler - INFO - Close client connection: Client-worker-39beb171-3d97-11f1-8b4a-4ef3162b8ac2
2026-04-21 15:32:44,146 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:45609', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,143 - distributed.scheduler - INFO - Close client connection: Client-worker-4b9f67df-3d97-11f1-8aeb-2eb33e86e1bd
2026-04-21 15:32:44,142 - distributed.scheduler - INFO - Task run-da2f0b24d3ded2e858125662d32ce4ea marked as failed because 3 workers died while trying to run it
2026-04-21 15:32:44,142 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:36827', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,126 - distributed.scheduler - INFO - Remove client Client-worker-39beb171-3d97-11f1-8b4a-4ef3162b8ac2
2026-04-21 15:32:44,123 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:36677', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,121 - distributed.scheduler - INFO - Remove client Client-worker-4b9f67df-3d97-11f1-8aeb-2eb33e86e1bd
2026-04-21 15:32:44,110 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:36931', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,107 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:34429', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,094 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:41347', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,080 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:39283', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,078 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:32953', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,071 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:32847', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:44,044 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:45173', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:25,350 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:45609
2026-04-21 15:32:25,349 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:45609', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:24,961 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:34429
2026-04-21 15:32:24,960 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:34429', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:24,909 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:46135
2026-04-21 15:32:24,909 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:46135', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:24,887 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:39283
2026-04-21 15:32:24,887 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:39283', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:24,861 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:36931
2026-04-21 15:32:24,860 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:36931', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:24,714 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:32953
2026-04-21 15:32:24,713 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:32953', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:24,655 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:41347
2026-04-21 15:32:24,654 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:41347', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:24,560 - distributed.scheduler - INFO - Receive client connection: Client-worker-4b9f67df-3d97-11f1-8aeb-2eb33e86e1bd
2026-04-21 15:32:24,406 - distributed.scheduler - INFO - Close client connection: Client-worker-39770a7d-3d97-11f1-8a6c-2a51d3dc178e
2026-04-21 15:32:24,396 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:35331', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:24,395 - distributed.scheduler - INFO - Remove client Client-worker-39770a7d-3d97-11f1-8a6c-2a51d3dc178e
2026-04-21 15:32:23,908 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:37897', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:23,892 - distributed.scheduler - INFO - Close client connection: Client-worker-38ee5b48-3d97-11f1-8b0c-6ec63a5ab21e
2026-04-21 15:32:23,878 - distributed.scheduler - INFO - Remove client Client-worker-38ee5b48-3d97-11f1-8b0c-6ec63a5ab21e
2026-04-21 15:32:23,877 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:37897', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:23,830 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:43919', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:23,801 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:43901', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:23,791 - distributed.scheduler - INFO - Close client connection: Client-worker-39703920-3d97-11f1-8b37-b6f02dd406e9
2026-04-21 15:32:23,787 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:43745', status: closed, memory: 0, processing: 0>
2026-04-21 15:32:23,786 - distributed.scheduler - INFO - Remove client Client-worker-39703920-3d97-11f1-8b37-b6f02dd406e9
2026-04-21 15:32:23,709 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:38771', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:54,778 - distributed.scheduler - INFO - Receive client connection: Client-worker-39fa7c35-3d97-11f1-8af8-2e20a4008324
2026-04-21 15:31:54,348 - distributed.scheduler - INFO - Receive client connection: Client-worker-39beb171-3d97-11f1-8b4a-4ef3162b8ac2
2026-04-21 15:31:53,879 - distributed.scheduler - INFO - Receive client connection: Client-worker-39770a7d-3d97-11f1-8a6c-2a51d3dc178e
2026-04-21 15:31:53,849 - distributed.scheduler - INFO - Receive client connection: Client-worker-39703920-3d97-11f1-8b37-b6f02dd406e9
2026-04-21 15:31:53,363 - distributed.scheduler - INFO - Receive client connection: Client-worker-3927e977-3d97-11f1-8af3-fae0fcb74df3
2026-04-21 15:31:53,041 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:38771
2026-04-21 15:31:53,039 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:38771', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,996 - distributed.scheduler - INFO - Receive client connection: Client-worker-38ee5b48-3d97-11f1-8b0c-6ec63a5ab21e
2026-04-21 15:31:52,832 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:41881
2026-04-21 15:31:52,830 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:41881', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,670 - distributed.scheduler - INFO - Receive client connection: Client-worker-38be11c1-3d97-11f1-8a4f-8288bfee4388
2026-04-21 15:31:52,593 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:43919
2026-04-21 15:31:52,591 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:43919', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,534 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:36283
2026-04-21 15:31:52,533 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:36283', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,519 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:36677
2026-04-21 15:31:52,518 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:36677', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,511 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:32847
2026-04-21 15:31:52,510 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:32847', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,432 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:37897
2026-04-21 15:31:52,431 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:37897', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,403 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:45791
2026-04-21 15:31:52,402 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:45791', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,329 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:35331
2026-04-21 15:31:52,327 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:35331', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,280 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:43901
2026-04-21 15:31:52,278 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:43901', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,212 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:43601
2026-04-21 15:31:52,211 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:43601', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,191 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:45173
2026-04-21 15:31:52,190 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:45173', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,122 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:36827
2026-04-21 15:31:52,121 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:36827', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,006 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:43745
2026-04-21 15:31:52,004 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:43745', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,003 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:36401', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:52,001 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:37897
2026-04-21 15:31:51,999 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:37897', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,910 - distributed.scheduler - INFO - Close client connection: Client-worker-264a98e7-3d97-11f1-8a8f-ae00acbfc48e
2026-04-21 15:31:51,888 - distributed.scheduler - INFO - Remove client Client-worker-264a98e7-3d97-11f1-8a8f-ae00acbfc48e
2026-04-21 15:31:51,887 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:43753', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,592 - distributed.scheduler - INFO - Close client connection: Client-worker-268baeef-3d97-11f1-8a80-7e353d94dc10
2026-04-21 15:31:51,590 - distributed.scheduler - INFO - Task ('hash-join-9df21a6cd9ff2716f57bf2d1d222d80f', 5) marked as failed because 3 workers died while trying to run it
2026-04-21 15:31:51,590 - distributed.scheduler - INFO - Task ('hash-join-9df21a6cd9ff2716f57bf2d1d222d80f', 8) marked as failed because 3 workers died while trying to run it
2026-04-21 15:31:51,590 - distributed.scheduler - INFO - Task ('hash-join-9df21a6cd9ff2716f57bf2d1d222d80f', 0) marked as failed because 3 workers died while trying to run it
2026-04-21 15:31:51,589 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:33259', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,587 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:46563', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,562 - distributed.scheduler - INFO - Remove client Client-worker-268baeef-3d97-11f1-8a80-7e353d94dc10
2026-04-21 15:31:51,531 - distributed.scheduler - INFO - Close client connection: Client-worker-299fc362-3d97-11f1-8aae-fae0fcb74df3
2026-04-21 15:31:51,529 - distributed.scheduler - INFO - Close client connection: Client-worker-26ebc884-3d97-11f1-8a97-566b4744c386
2026-04-21 15:31:51,524 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:40285', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,512 - distributed.scheduler - INFO - Remove client Client-worker-299fc362-3d97-11f1-8aae-fae0fcb74df3
2026-04-21 15:31:51,510 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:35169', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,509 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:44319', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,491 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:33593', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,312 - distributed.scheduler - INFO - Remove client Client-worker-26ebc884-3d97-11f1-8a97-566b4744c386
2026-04-21 15:31:51,270 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:34635', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,251 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:44155', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,172 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:35507', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,135 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:34169', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,110 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:45927', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:51,031 - distributed.scheduler - INFO - Close client connection: Client-worker-2af12e19-3d97-11f1-8af7-b6f02dd406e9
2026-04-21 15:31:51,004 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:38735', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:50,998 - distributed.scheduler - INFO - Remove client Client-worker-2af12e19-3d97-11f1-8af7-b6f02dd406e9
2026-04-21 15:31:50,872 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:38357', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:29,550 - distributed.scheduler - INFO - Receive client connection: Client-worker-2af12e19-3d97-11f1-8af7-b6f02dd406e9
2026-04-21 15:31:28,726 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:33259
2026-04-21 15:31:28,725 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:33259', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:28,525 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:36401
2026-04-21 15:31:28,524 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:36401', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:28,522 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:34169
2026-04-21 15:31:28,521 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:34169', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:28,314 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:45927
2026-04-21 15:31:28,313 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:45927', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:27,633 - distributed.scheduler - INFO - Close client connection: Client-worker-bf977de2-3d8e-11f1-8a8b-86acc56fd09f
2026-04-21 15:31:27,624 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:37885', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:27,623 - distributed.scheduler - INFO - Remove client Client-worker-bf977de2-3d8e-11f1-8a8b-86acc56fd09f
2026-04-21 15:31:27,489 - distributed.scheduler - INFO - Close client connection: Client-worker-bef27256-3d8e-11f1-8a33-ee7435aaba1f
2026-04-21 15:31:27,485 - distributed.scheduler - INFO - Remove client Client-worker-bef27256-3d8e-11f1-8a33-ee7435aaba1f
2026-04-21 15:31:27,484 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:36033', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:27,371 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:43127', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:27,304 - distributed.scheduler - INFO - Receive client connection: Client-worker-299fc362-3d97-11f1-8aae-fae0fcb74df3
2026-04-21 15:31:27,127 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:33185', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:22,792 - distributed.scheduler - INFO - Receive client connection: Client-worker-26ebc884-3d97-11f1-8a97-566b4744c386
2026-04-21 15:31:22,147 - distributed.scheduler - INFO - Receive client connection: Client-worker-268baeef-3d97-11f1-8a80-7e353d94dc10
2026-04-21 15:31:21,739 - distributed.scheduler - INFO - Receive client connection: Client-worker-264a98e7-3d97-11f1-8a8f-ae00acbfc48e
2026-04-21 15:31:21,176 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:40285
2026-04-21 15:31:21,175 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:40285', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:21,108 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:44155
2026-04-21 15:31:21,107 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:44155', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:21,018 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:38735
2026-04-21 15:31:21,017 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:38735', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:21,015 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:34635
2026-04-21 15:31:21,014 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:34635', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:21,009 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:38357
2026-04-21 15:31:21,008 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:38357', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,957 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:35507
2026-04-21 15:31:20,956 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:35507', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,905 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:42571
2026-04-21 15:31:20,904 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:42571', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,842 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:44319
2026-04-21 15:31:20,840 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:44319', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,833 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:35169
2026-04-21 15:31:20,832 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:35169', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,819 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:43753
2026-04-21 15:31:20,818 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:43753', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,813 - distributed.scheduler - INFO - Close client connection: Client-a39e15ee-3d95-11f1-8001-daa6852bd5a7
2026-04-21 15:31:20,813 - distributed.scheduler - INFO - Close client connection: Client-93ad52e3-3d94-11f1-8001-daa6852bd5a7
2026-04-21 15:31:20,812 - distributed.scheduler - INFO - Remove client Client-a39e15ee-3d95-11f1-8001-daa6852bd5a7
2026-04-21 15:31:20,811 - distributed.scheduler - INFO - Remove client Client-93ad52e3-3d94-11f1-8001-daa6852bd5a7
2026-04-21 15:31:20,744 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:46563
2026-04-21 15:31:20,742 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:46563', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,493 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:33593
2026-04-21 15:31:20,490 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:33593', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,173 - distributed.scheduler - INFO - Close client connection: Client-worker-d8f63b0f-3d92-11f1-8a6b-fae0fcb74df3
2026-04-21 15:31:20,169 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:40817', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,153 - distributed.scheduler - INFO - Remove client Client-worker-d8f63b0f-3d92-11f1-8a6b-fae0fcb74df3
2026-04-21 15:31:20,137 - distributed.scheduler - INFO - Close client connection: Client-worker-bfe41da0-3d8e-11f1-89f3-2a51d3dc178e
2026-04-21 15:31:20,133 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:37115', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,131 - distributed.scheduler - INFO - Remove client Client-worker-bfe41da0-3d8e-11f1-89f3-2a51d3dc178e
2026-04-21 15:31:20,088 - distributed.scheduler - INFO - Close client connection: Client-worker-d9a8eb91-3d92-11f1-8a0a-8288bfee4388
2026-04-21 15:31:20,087 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:42435', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,086 - distributed.scheduler - INFO - Remove client Client-worker-d9a8eb91-3d92-11f1-8a0a-8288bfee4388
2026-04-21 15:31:20,084 - distributed.scheduler - INFO - Close client connection: Client-0a279663-3d97-11f1-8001-daa6852bd5a7
2026-04-21 15:31:20,083 - distributed.scheduler - INFO - Remove client Client-0a279663-3d97-11f1-8001-daa6852bd5a7
2026-04-21 15:31:20,082 - distributed.scheduler - INFO - Remove client Client-0a279663-3d97-11f1-8001-daa6852bd5a7
2026-04-21 15:31:20,066 - distributed.scheduler - INFO - Close client connection: Client-worker-d91762a4-3d92-11f1-8a94-6ec63a5ab21e
2026-04-21 15:31:20,053 - distributed.scheduler - INFO - Task ('sjoin-d8d3dff67c9ea1218d13b472f7f06ded', 161) marked as failed because 3 workers died while trying to run it
2026-04-21 15:31:20,052 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:41651', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,052 - distributed.scheduler - INFO - Remove client Client-worker-d91762a4-3d92-11f1-8a94-6ec63a5ab21e
2026-04-21 15:31:20,051 - distributed.scheduler - INFO - Close client connection: Client-worker-befae384-3d8e-11f1-8aaa-b6f02dd406e9
2026-04-21 15:31:20,029 - distributed.scheduler - INFO - Remove client Client-worker-befae384-3d8e-11f1-8aaa-b6f02dd406e9
2026-04-21 15:31:20,023 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:33637', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,013 - distributed.scheduler - INFO - Close client connection: Client-worker-d9ad0785-3d92-11f1-89f1-0e4c34ff5e38
2026-04-21 15:31:20,004 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:41365', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:20,003 - distributed.scheduler - INFO - Remove client Client-worker-d9ad0785-3d92-11f1-89f1-0e4c34ff5e38
2026-04-21 15:31:19,998 - distributed.scheduler - INFO - Close client connection: Client-worker-bfc693fe-3d8e-11f1-8ac4-4ef3162b8ac2
2026-04-21 15:31:19,985 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:44495', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:19,983 - distributed.scheduler - INFO - Remove client Client-worker-bfc693fe-3d8e-11f1-8ac4-4ef3162b8ac2
2026-04-21 15:31:19,920 - distributed.scheduler - INFO - Close client connection: Client-worker-c0054b07-3d8e-11f1-8a51-ae00acbfc48e
2026-04-21 15:31:19,892 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:38091', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:19,889 - distributed.scheduler - INFO - Remove client Client-worker-c0054b07-3d8e-11f1-8a51-ae00acbfc48e
2026-04-21 15:31:19,814 - distributed.scheduler - INFO - Close client connection: Client-worker-d9864899-3d92-11f1-8bb7-f28f846166bd
2026-04-21 15:31:19,813 - distributed.scheduler - INFO - Close client connection: Client-worker-196df55e-3d93-11f1-8a73-2eb33e86e1bd
2026-04-21 15:31:19,812 - distributed.scheduler - INFO - Remove client Client-worker-d9864899-3d92-11f1-8bb7-f28f846166bd
2026-04-21 15:31:19,807 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:35161', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:19,799 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:40683', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:19,798 - distributed.scheduler - INFO - Remove client Client-worker-196df55e-3d93-11f1-8a73-2eb33e86e1bd
2026-04-21 15:31:19,782 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:39671', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:19,600 - distributed.scheduler - INFO - Close client connection: Client-worker-11bbe2a7-3d93-11f1-8a57-566b4744c386
2026-04-21 15:31:19,581 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:34581', status: closed, memory: 0, processing: 0>
2026-04-21 15:31:19,579 - distributed.scheduler - INFO - Remove client Client-worker-11bbe2a7-3d93-11f1-8a57-566b4744c386
2026-04-21 15:30:34,513 - distributed.scheduler - INFO - Receive client connection: Client-0a279663-3d97-11f1-8001-daa6852bd5a7
2026-04-21 15:20:32,984 - distributed.scheduler - INFO - Receive client connection: Client-a39e15ee-3d95-11f1-8001-daa6852bd5a7
2026-04-21 15:12:56,756 - distributed.scheduler - INFO - Receive client connection: Client-93ad52e3-3d94-11f1-8001-daa6852bd5a7
2026-04-21 15:02:22,147 - distributed.scheduler - INFO - Receive client connection: Client-worker-196df55e-3d93-11f1-8a73-2eb33e86e1bd
2026-04-21 15:02:09,234 - distributed.scheduler - INFO - Receive client connection: Client-worker-11bbe2a7-3d93-11f1-8a57-566b4744c386
2026-04-21 15:02:06,131 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:42435 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.253:39653 after 30 s')
2026-04-21 15:02:06,121 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44495 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.253:39653 after 30 s')
2026-04-21 15:02:06,112 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:39653 failed: OSError: Timed out trying to connect to tcp://172.21.159.253:39653 after 30 s
2026-04-21 15:02:06,112 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:46185 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:46185 after 30 s
2026-04-21 15:02:06,089 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:37115 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.253:39653 after 30 s')
2026-04-21 15:02:06,069 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:43127 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.253:39653 after 30 s')
2026-04-21 15:02:06,066 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:34581 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.222:46185 after 30 s')
2026-04-21 15:02:05,987 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:41651 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.253:39653 after 30 s')
2026-04-21 15:02:05,974 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:36033 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.222:46185 after 30 s')
2026-04-21 15:02:05,972 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35161 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.253:39653 after 30 s')
2026-04-21 15:02:05,933 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:41365 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.253:39653 after 30 s')
2026-04-21 15:02:05,931 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:33185 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.222:46185 after 30 s')
2026-04-21 15:02:05,920 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:38091 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.222:46185 after 30 s')
2026-04-21 15:01:45,984 - distributed.scheduler - INFO - Close client connection: Client-d694bbab-3d92-11f1-8001-daa6852bd5a7
2026-04-21 15:01:45,983 - distributed.scheduler - INFO - Remove client Client-d694bbab-3d92-11f1-8001-daa6852bd5a7
2026-04-21 15:01:45,981 - distributed.scheduler - INFO - Remove client Client-d694bbab-3d92-11f1-8001-daa6852bd5a7
2026-04-21 15:01:36,750 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:39671
2026-04-21 15:01:36,748 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:39671', status: closed, memory: 0, processing: 0>
2026-04-21 15:01:36,201 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:40683
2026-04-21 15:01:36,199 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:40683', status: closed, memory: 0, processing: 0>
2026-04-21 15:01:35,786 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:39653', status: closed, memory: 0, processing: 0>
2026-04-21 15:01:35,267 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:46185', status: closed, memory: 0, processing: 0>
2026-04-21 15:00:56,579 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:46185
2026-04-21 15:00:56,578 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:46185', status: closed, memory: 0, processing: 0>
2026-04-21 15:00:56,376 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:39653
2026-04-21 15:00:56,374 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:39653', status: closed, memory: 0, processing: 0>
2026-04-21 15:00:55,568 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:35865', status: closed, memory: 0, processing: 0>
2026-04-21 15:00:55,426 - distributed.scheduler - INFO - Close client connection: Client-worker-bfc9afe9-3d8e-11f1-89d8-7e353d94dc10
2026-04-21 15:00:55,387 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:44261', status: closed, memory: 0, processing: 0>
2026-04-21 15:00:55,386 - distributed.scheduler - INFO - Remove client Client-worker-bfc9afe9-3d8e-11f1-89d8-7e353d94dc10
2026-04-21 15:00:35,250 - distributed.scheduler - INFO - Receive client connection: Client-worker-d9ad0785-3d92-11f1-89f1-0e4c34ff5e38
2026-04-21 15:00:35,237 - distributed.scheduler - INFO - Receive client connection: Client-worker-d9a8eb91-3d92-11f1-8a0a-8288bfee4388
2026-04-21 15:00:34,973 - distributed.scheduler - INFO - Receive client connection: Client-worker-d9864899-3d92-11f1-8bb7-f28f846166bd
2026-04-21 15:00:34,211 - distributed.scheduler - INFO - Receive client connection: Client-worker-d91762a4-3d92-11f1-8a94-6ec63a5ab21e
2026-04-21 15:00:33,989 - distributed.scheduler - INFO - Receive client connection: Client-worker-d8f63b0f-3d92-11f1-8a6b-fae0fcb74df3
2026-04-21 15:00:30,010 - distributed.scheduler - INFO - Receive client connection: Client-d694bbab-3d92-11f1-8001-daa6852bd5a7
2026-04-21 14:51:07,254 - distributed.scheduler - INFO - Close client connection: Client-70193aca-3d91-11f1-8001-daa6852bd5a7
2026-04-21 14:51:07,253 - distributed.scheduler - INFO - Remove client Client-70193aca-3d91-11f1-8001-daa6852bd5a7
2026-04-21 14:51:07,248 - distributed.scheduler - INFO - Remove client Client-70193aca-3d91-11f1-8001-daa6852bd5a7
2026-04-21 14:50:28,571 - distributed.scheduler - INFO - Receive client connection: Client-70193aca-3d91-11f1-8001-daa6852bd5a7
2026-04-21 14:41:06,103 - distributed.scheduler - INFO - Close client connection: Client-09977509-3d90-11f1-8001-daa6852bd5a7
2026-04-21 14:41:06,102 - distributed.scheduler - INFO - Remove client Client-09977509-3d90-11f1-8001-daa6852bd5a7
2026-04-21 14:41:06,100 - distributed.scheduler - INFO - Remove client Client-09977509-3d90-11f1-8001-daa6852bd5a7
2026-04-21 14:40:27,099 - distributed.scheduler - INFO - Receive client connection: Client-09977509-3d90-11f1-8001-daa6852bd5a7
2026-04-21 14:32:08,550 - distributed.scheduler - INFO - Close client connection: Client-a3158b57-3d8e-11f1-8001-daa6852bd5a7
2026-04-21 14:32:08,549 - distributed.scheduler - INFO - Remove client Client-a3158b57-3d8e-11f1-8001-daa6852bd5a7
2026-04-21 14:32:08,547 - distributed.scheduler - INFO - Remove client Client-a3158b57-3d8e-11f1-8001-daa6852bd5a7
2026-04-21 14:31:24,076 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-d3d2c99b710f78fc472266a3d013bcb5 NoneType: None
2026-04-21 14:31:24,076 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-ab1828927890f59f4946db2b776b8744 NoneType: None
2026-04-21 14:31:24,075 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-e4a5c1dd0d4011bbf353cc7cdb586402 NoneType: None
2026-04-21 14:31:24,075 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-5cf3da2adb1499082a6a77452e5b3036 NoneType: None
2026-04-21 14:31:24,075 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-648e6c10af7228154aee2ec89c76c333 NoneType: None
2026-04-21 14:31:24,075 - distributed.scheduler - ERROR - Couldn't gather keys {'run-648e6c10af7228154aee2ec89c76c333': [], 'run-5cf3da2adb1499082a6a77452e5b3036': [], 'run-e4a5c1dd0d4011bbf353cc7cdb586402': [], 'run-ab1828927890f59f4946db2b776b8744': [], 'run-d3d2c99b710f78fc472266a3d013bcb5': []} state: ['processing', 'processing', 'processing', 'processing', 'processing'] workers: [] NoneType: None
2026-04-21 14:31:14,234 - distributed.scheduler - INFO - Receive client connection: Client-worker-c0054b07-3d8e-11f1-8a51-ae00acbfc48e
2026-04-21 14:31:13,947 - distributed.scheduler - INFO - Receive client connection: Client-worker-bfe41da0-3d8e-11f1-89f3-2a51d3dc178e
2026-04-21 14:31:13,775 - distributed.scheduler - INFO - Receive client connection: Client-worker-bfc9afe9-3d8e-11f1-89d8-7e353d94dc10
2026-04-21 14:31:13,763 - distributed.scheduler - INFO - Receive client connection: Client-worker-bfc693fe-3d8e-11f1-8ac4-4ef3162b8ac2
2026-04-21 14:31:13,453 - distributed.scheduler - INFO - Receive client connection: Client-worker-bf977de2-3d8e-11f1-8a8b-86acc56fd09f
2026-04-21 14:31:12,431 - distributed.scheduler - INFO - Receive client connection: Client-worker-befae384-3d8e-11f1-8aaa-b6f02dd406e9
2026-04-21 14:31:12,356 - distributed.scheduler - INFO - Receive client connection: Client-worker-bef27256-3d8e-11f1-8a33-ee7435aaba1f
2026-04-21 14:31:12,324 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:41651
2026-04-21 14:31:12,323 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:41651', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:12,226 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:43127
2026-04-21 14:31:12,225 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:43127', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:12,098 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:34581
2026-04-21 14:31:12,097 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:34581', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:12,075 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:35865
2026-04-21 14:31:12,074 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:35865', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:12,052 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:41365
2026-04-21 14:31:12,052 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:41365', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,837 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:40817
2026-04-21 14:31:11,835 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:40817', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,816 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:33185
2026-04-21 14:31:11,815 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:33185', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,758 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:35161
2026-04-21 14:31:11,757 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:35161', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,628 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:37115
2026-04-21 14:31:11,626 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:37115', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,620 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:38091
2026-04-21 14:31:11,618 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:38091', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,586 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:42435
2026-04-21 14:31:11,585 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:42435', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,564 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:37885
2026-04-21 14:31:11,563 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:37885', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,528 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:44495
2026-04-21 14:31:11,526 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:44495', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,517 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:33637
2026-04-21 14:31:11,516 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:33637', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,502 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:36033
2026-04-21 14:31:11,501 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:36033', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,438 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:44261
2026-04-21 14:31:11,436 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:44261', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,160 - distributed.scheduler - INFO - Close client connection: Client-worker-a63a640d-3d8e-11f1-8a54-6ec63a5ab21e
2026-04-21 14:31:11,157 - distributed.scheduler - INFO - Lost all workers
2026-04-21 14:31:11,154 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:33929', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,154 - distributed.scheduler - INFO - Remove client Client-worker-a63a640d-3d8e-11f1-8a54-6ec63a5ab21e
2026-04-21 14:31:11,117 - distributed.scheduler - INFO - Close client connection: Client-worker-a585d290-3d8e-11f1-8a1c-566b4744c386
2026-04-21 14:31:11,111 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:32803', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,105 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:34777', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:11,104 - distributed.scheduler - INFO - Remove client Client-worker-a585d290-3d8e-11f1-8a1c-566b4744c386
2026-04-21 14:31:10,971 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:33977', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:10,853 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:37569', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:10,777 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:45435', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:10,689 - distributed.scheduler - INFO - Close client connection: Client-worker-a648eccf-3d8e-11f1-8a2e-fae0fcb74df3
2026-04-21 14:31:10,683 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:35453', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:10,682 - distributed.scheduler - INFO - Close client connection: Client-worker-a67022eb-3d8e-11f1-8b75-f28f846166bd
2026-04-21 14:31:10,681 - distributed.scheduler - INFO - Remove client Client-worker-a648eccf-3d8e-11f1-8a2e-fae0fcb74df3
2026-04-21 14:31:10,677 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:37205', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:10,675 - distributed.scheduler - INFO - Remove client Client-worker-a67022eb-3d8e-11f1-8b75-f28f846166bd
2026-04-21 14:31:10,645 - distributed.scheduler - INFO - Close client connection: Client-worker-a5a98665-3d8e-11f1-8a6a-b6f02dd406e9
2026-04-21 14:31:10,644 - distributed.scheduler - INFO - Close client connection: Client-worker-b80412df-3d8a-11f1-8a85-4ef3162b8ac2
2026-04-21 14:31:10,641 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:45315', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:10,640 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:43585', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:10,639 - distributed.scheduler - INFO - Remove client Client-worker-a5a98665-3d8e-11f1-8a6a-b6f02dd406e9
2026-04-21 14:31:10,636 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:37423', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:10,607 - distributed.scheduler - INFO - Remove client Client-worker-b80412df-3d8a-11f1-8a85-4ef3162b8ac2
2026-04-21 14:31:10,579 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:46141', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:10,575 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:38113', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:10,470 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:39291', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:10,325 - distributed.scheduler - INFO - Close client connection: Client-worker-a5d5a6e7-3d8e-11f1-8a46-86acc56fd09f
2026-04-21 14:31:10,323 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:42683', status: closed, memory: 0, processing: 0>
2026-04-21 14:31:10,321 - distributed.scheduler - INFO - Remove client Client-worker-a5d5a6e7-3d8e-11f1-8a46-86acc56fd09f
2026-04-21 14:31:10,314 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:38127', status: closed, memory: 0, processing: 0>
2026-04-21 14:30:31,288 - distributed.scheduler - INFO - Receive client connection: Client-worker-a67022eb-3d8e-11f1-8b75-f28f846166bd
2026-04-21 14:30:31,082 - distributed.scheduler - INFO - Receive client connection: Client-worker-a648eccf-3d8e-11f1-8a2e-fae0fcb74df3
2026-04-21 14:30:30,941 - distributed.scheduler - INFO - Receive client connection: Client-worker-a63a640d-3d8e-11f1-8a54-6ec63a5ab21e
2026-04-21 14:30:30,273 - distributed.scheduler - INFO - Receive client connection: Client-worker-a5d5a6e7-3d8e-11f1-8a46-86acc56fd09f
2026-04-21 14:30:29,931 - distributed.scheduler - INFO - Receive client connection: Client-worker-a5a98665-3d8e-11f1-8a6a-b6f02dd406e9
2026-04-21 14:30:29,701 - distributed.scheduler - INFO - Receive client connection: Client-worker-a585d290-3d8e-11f1-8a1c-566b4744c386
2026-04-21 14:30:25,627 - distributed.scheduler - INFO - Receive client connection: Client-a3158b57-3d8e-11f1-8001-daa6852bd5a7
2026-04-21 14:21:01,203 - distributed.scheduler - INFO - Close client connection: Client-3c8e4ccb-3d8d-11f1-8001-daa6852bd5a7
2026-04-21 14:21:01,201 - distributed.scheduler - INFO - Remove client Client-3c8e4ccb-3d8d-11f1-8001-daa6852bd5a7
2026-04-21 14:21:01,200 - distributed.scheduler - INFO - Remove client Client-3c8e4ccb-3d8d-11f1-8001-daa6852bd5a7
2026-04-21 14:20:24,116 - distributed.scheduler - INFO - Receive client connection: Client-3c8e4ccb-3d8d-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,495 - distributed.scheduler - INFO - Close client connection: Client-3b8a6ee3-3d7f-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,495 - distributed.scheduler - INFO - Close client connection: Client-a20ec83b-3d80-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,494 - distributed.scheduler - INFO - Close client connection: Client-08865efe-3d82-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,494 - distributed.scheduler - INFO - Close client connection: Client-d3a209df-3d83-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,494 - distributed.scheduler - INFO - Close client connection: Client-d58550b2-3d84-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,493 - distributed.scheduler - INFO - Close client connection: Client-3c141559-3d86-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,493 - distributed.scheduler - INFO - Close client connection: Client-a294d217-3d87-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,493 - distributed.scheduler - INFO - Close client connection: Client-090ef11a-3d89-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,492 - distributed.scheduler - INFO - Close client connection: Client-6f8de3c3-3d8a-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,490 - distributed.scheduler - INFO - Remove client Client-3b8a6ee3-3d7f-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,489 - distributed.scheduler - INFO - Remove client Client-a20ec83b-3d80-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,489 - distributed.scheduler - INFO - Remove client Client-08865efe-3d82-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,488 - distributed.scheduler - INFO - Remove client Client-d3a209df-3d83-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,487 - distributed.scheduler - INFO - Remove client Client-d58550b2-3d84-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,486 - distributed.scheduler - INFO - Remove client Client-3c141559-3d86-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,486 - distributed.scheduler - INFO - Remove client Client-a294d217-3d87-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,485 - distributed.scheduler - INFO - Remove client Client-090ef11a-3d89-11f1-8001-daa6852bd5a7
2026-04-21 14:13:23,480 - distributed.scheduler - INFO - Remove client Client-6f8de3c3-3d8a-11f1-8001-daa6852bd5a7
2026-04-21 14:13:22,976 - distributed.scheduler - INFO - Close client connection: Client-2e6db54e-3d8c-11f1-8001-daa6852bd5a7
2026-04-21 14:13:22,976 - distributed.scheduler - INFO - Remove client Client-2e6db54e-3d8c-11f1-8001-daa6852bd5a7
2026-04-21 14:13:22,975 - distributed.scheduler - INFO - Remove client Client-2e6db54e-3d8c-11f1-8001-daa6852bd5a7
2026-04-21 14:12:50,903 - distributed.scheduler - INFO - Receive client connection: Client-2e6db54e-3d8c-11f1-8001-daa6852bd5a7
2026-04-21 14:02:28,131 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:32803
2026-04-21 14:02:28,130 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:32803', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:27,988 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:42683
2026-04-21 14:02:27,987 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:42683', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:27,904 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:45435
2026-04-21 14:02:27,901 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:45435', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:26,995 - distributed.scheduler - INFO - Close client connection: Client-worker-a5b91485-3d8a-11f1-8980-c2349ba99f21
2026-04-21 14:02:26,991 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:38863', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:26,989 - distributed.scheduler - INFO - Remove client Client-worker-a5b91485-3d8a-11f1-8980-c2349ba99f21
2026-04-21 14:02:26,898 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:40803', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:26,895 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:34511', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:22,770 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:45315
2026-04-21 14:02:22,769 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:45315', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:22,767 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:33929
2026-04-21 14:02:22,766 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:33929', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:22,757 - distributed.scheduler - INFO - Receive client connection: Client-worker-b80412df-3d8a-11f1-8a85-4ef3162b8ac2
2026-04-21 14:02:22,743 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:46141
2026-04-21 14:02:22,741 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:46141', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:22,644 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:37205
2026-04-21 14:02:22,642 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:37205', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:22,522 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:38113
2026-04-21 14:02:22,521 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:38113', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:22,507 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:37569
2026-04-21 14:02:22,505 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:37569', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:21,836 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:34409', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:21,763 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:35615', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:21,762 - distributed.scheduler - INFO - Close client connection: Client-worker-3e4fda1f-3d86-11f1-8b2b-f28f846166bd
2026-04-21 14:02:21,761 - distributed.scheduler - INFO - Close client connection: Client-worker-3ebaf4b5-3d86-11f1-8a16-6ec63a5ab21e
2026-04-21 14:02:21,757 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:34665', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:21,756 - distributed.scheduler - INFO - Remove client Client-worker-3e4fda1f-3d86-11f1-8b2b-f28f846166bd
2026-04-21 14:02:21,755 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:39509', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:21,754 - distributed.scheduler - INFO - Remove client Client-worker-3ebaf4b5-3d86-11f1-8a16-6ec63a5ab21e
2026-04-21 14:02:21,657 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:46047', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:21,605 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:33969', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:21,120 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:38127
2026-04-21 14:02:21,119 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:38127', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:20,975 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:35453
2026-04-21 14:02:20,974 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:35453', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:20,945 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:39291
2026-04-21 14:02:20,943 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:39291', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:20,896 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:34777
2026-04-21 14:02:20,896 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:34777', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:20,889 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:43585
2026-04-21 14:02:20,889 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:43585', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:20,888 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:37423
2026-04-21 14:02:20,887 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:37423', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:20,848 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:33977
2026-04-21 14:02:20,847 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:33977', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:20,132 - distributed.scheduler - INFO - Close client connection: Client-worker-25eae3fd-3d82-11f1-89eb-fae0fcb74df3
2026-04-21 14:02:20,130 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:42343', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:20,128 - distributed.scheduler - INFO - Remove client Client-worker-25eae3fd-3d82-11f1-89eb-fae0fcb74df3
2026-04-21 14:02:19,972 - distributed.scheduler - INFO - Close client connection: Client-worker-3e31e422-3d86-11f1-8962-7e353d94dc10
2026-04-21 14:02:19,972 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:33875', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:19,971 - distributed.scheduler - INFO - Remove client Client-worker-3e31e422-3d86-11f1-8962-7e353d94dc10
2026-04-21 14:02:19,950 - distributed.scheduler - INFO - Close client connection: Client-worker-720c43e8-3d8a-11f1-89b8-ee7435aaba1f
2026-04-21 14:02:19,949 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:46217', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:19,947 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:42221', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:19,927 - distributed.scheduler - INFO - Remove client Client-worker-720c43e8-3d8a-11f1-89b8-ee7435aaba1f
2026-04-21 14:02:19,925 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:36177', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:19,913 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:41353', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:19,889 - distributed.scheduler - INFO - Close client connection: Client-worker-232d0962-3d82-11f1-8992-2eb33e86e1bd
2026-04-21 14:02:19,882 - distributed.scheduler - INFO - Remove client Client-worker-232d0962-3d82-11f1-8992-2eb33e86e1bd
2026-04-21 14:02:19,878 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:34811', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:06,074 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:33969
2026-04-21 14:02:06,072 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:33969', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:05,969 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:46217
2026-04-21 14:02:05,967 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:46217', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:05,899 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:41353
2026-04-21 14:02:05,898 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:41353', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:05,696 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:34511
2026-04-21 14:02:05,694 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:34511', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:05,630 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:34409
2026-04-21 14:02:05,627 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:34409', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:05,070 - distributed.scheduler - INFO - Close client connection: Client-worker-3eeae4af-3d86-11f1-8971-0e4c34ff5e38
2026-04-21 14:02:05,064 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:32861', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:05,062 - distributed.scheduler - INFO - Remove client Client-worker-3eeae4af-3d86-11f1-8971-0e4c34ff5e38
2026-04-21 14:02:05,005 - distributed.scheduler - INFO - Close client connection: Client-worker-2436308c-3d82-11f1-8a18-b6f02dd406e9
2026-04-21 14:02:05,002 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:46741', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:05,000 - distributed.scheduler - INFO - Remove client Client-worker-2436308c-3d82-11f1-8a18-b6f02dd406e9
2026-04-21 14:02:04,900 - distributed.scheduler - INFO - Close client connection: Client-worker-71d9b512-3d8a-11f1-8a32-4ef3162b8ac2
2026-04-21 14:02:04,894 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:36641', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:04,892 - distributed.scheduler - INFO - Remove client Client-worker-71d9b512-3d8a-11f1-8a32-4ef3162b8ac2
2026-04-21 14:02:04,781 - distributed.scheduler - INFO - Close client connection: Client-worker-71b1c727-3d8a-11f1-8987-8288bfee4388
2026-04-21 14:02:04,779 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:43461', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:04,776 - distributed.scheduler - INFO - Remove client Client-worker-71b1c727-3d8a-11f1-8987-8288bfee4388
2026-04-21 14:02:04,687 - distributed.scheduler - INFO - Close client connection: Client-worker-3e2f0a1e-3d86-11f1-89f6-2e20a4008324
2026-04-21 14:02:04,683 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:44251', status: closed, memory: 0, processing: 0>
2026-04-21 14:02:04,681 - distributed.scheduler - INFO - Remove client Client-worker-3e2f0a1e-3d86-11f1-89f6-2e20a4008324
2026-04-21 14:01:52,055 - distributed.scheduler - INFO - Receive client connection: Client-worker-a5b91485-3d8a-11f1-8980-c2349ba99f21
2026-04-21 14:01:29,096 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:36167 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:36167 after 30 s
2026-04-21 14:01:29,093 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:32973 failed: OSError: Timed out trying to connect to tcp://172.21.25.29:32973 after 30 s
2026-04-21 14:01:29,093 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:43543 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:43543 after 30 s
2026-04-21 14:01:29,092 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:37251 failed: OSError: Timed out trying to connect to tcp://172.21.159.214:37251 after 30 s
2026-04-21 14:01:28,924 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:35615 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36167 after 30 s')
2026-04-21 14:01:28,052 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34811 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.214:37251 after 30 s')
2026-04-21 14:00:57,841 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:38863
2026-04-21 14:00:57,840 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:38863', status: closed, memory: 0, processing: 0>
2026-04-21 14:00:57,836 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:35615
2026-04-21 14:00:57,835 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:35615', status: closed, memory: 0, processing: 0>
2026-04-21 14:00:57,827 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:40803
2026-04-21 14:00:57,826 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:40803', status: closed, memory: 0, processing: 0>
2026-04-21 14:00:57,713 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:46047
2026-04-21 14:00:57,711 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:46047', status: closed, memory: 0, processing: 0>
2026-04-21 14:00:56,879 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: ['tcp://172.21.25.34:36167'], ('assign-231e12a9cd800ca82000de1f043c39a7', 15) NoneType: None
2026-04-21 14:00:56,874 - distributed.scheduler - ERROR - Couldn't gather keys {"('assign-231e12a9cd800ca82000de1f043c39a7', 15)": ['tcp://172.21.25.34:36167']} state: ['waiting'] workers: ['tcp://172.21.25.34:36167'] NoneType: None
2026-04-21 14:00:56,862 - distributed.scheduler - INFO - Close client connection: Client-worker-72495c93-3d8a-11f1-89cc-86acc56fd09f
2026-04-21 14:00:56,832 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:36167', status: closed, memory: 0, processing: 0>
2026-04-21 14:00:56,826 - distributed.scheduler - INFO - Remove client Client-worker-72495c93-3d8a-11f1-89cc-86acc56fd09f
2026-04-21 14:00:56,793 - distributed.scheduler - INFO - Close client connection: Client-worker-3e0d44f1-3d86-11f1-899e-ae00acbfc48e
2026-04-21 14:00:56,785 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:43543', status: closed, memory: 0, processing: 0>
2026-04-21 14:00:56,770 - distributed.scheduler - INFO - Remove client Client-worker-3e0d44f1-3d86-11f1-899e-ae00acbfc48e
2026-04-21 14:00:56,756 - distributed.scheduler - INFO - Close client connection: Client-worker-3eb9b069-3d86-11f1-893f-2a51d3dc178e
2026-04-21 14:00:56,750 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:37251', status: closed, memory: 0, processing: 0>
2026-04-21 14:00:56,747 - distributed.scheduler - INFO - Remove client Client-worker-3eb9b069-3d86-11f1-893f-2a51d3dc178e
2026-04-21 14:00:56,675 - distributed.scheduler - INFO - Close client connection: Client-worker-7277bd98-3d8a-11f1-8947-c2349ba99f21
2026-04-21 14:00:56,642 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:32973', status: closed, memory: 0, processing: 0>
2026-04-21 14:00:56,637 - distributed.scheduler - INFO - Remove client Client-worker-7277bd98-3d8a-11f1-8947-c2349ba99f21
2026-04-21 14:00:26,080 - distributed.scheduler - INFO - Receive client connection: Client-worker-7277bd98-3d8a-11f1-8947-c2349ba99f21
2026-04-21 14:00:25,765 - distributed.scheduler - INFO - Receive client connection: Client-worker-72495c93-3d8a-11f1-89cc-86acc56fd09f
2026-04-21 14:00:25,360 - distributed.scheduler - INFO - Receive client connection: Client-worker-720c43e8-3d8a-11f1-89b8-ee7435aaba1f
2026-04-21 14:00:25,044 - distributed.scheduler - INFO - Receive client connection: Client-worker-71d9b512-3d8a-11f1-8a32-4ef3162b8ac2
2026-04-21 14:00:24,760 - distributed.scheduler - INFO - Receive client connection: Client-worker-71b1c727-3d8a-11f1-8987-8288bfee4388
2026-04-21 14:00:21,176 - distributed.scheduler - INFO - Receive client connection: Client-6f8de3c3-3d8a-11f1-8001-daa6852bd5a7
2026-04-21 13:50:19,716 - distributed.scheduler - INFO - Receive client connection: Client-090ef11a-3d89-11f1-8001-daa6852bd5a7
2026-04-21 13:40:18,295 - distributed.scheduler - INFO - Receive client connection: Client-a294d217-3d87-11f1-8001-daa6852bd5a7
2026-04-21 13:30:21,623 - distributed.scheduler - INFO - Receive client connection: Client-worker-3eeae4af-3d86-11f1-8971-0e4c34ff5e38
2026-04-21 13:30:21,291 - distributed.scheduler - INFO - Receive client connection: Client-worker-3eb9b069-3d86-11f1-893f-2a51d3dc178e
2026-04-21 13:30:21,287 - distributed.scheduler - INFO - Receive client connection: Client-worker-3ebaf4b5-3d86-11f1-8a16-6ec63a5ab21e
2026-04-21 13:30:20,600 - distributed.scheduler - INFO - Receive client connection: Client-worker-3e4fda1f-3d86-11f1-8b2b-f28f846166bd
2026-04-21 13:30:20,388 - distributed.scheduler - INFO - Receive client connection: Client-worker-3e31e422-3d86-11f1-8962-7e353d94dc10
2026-04-21 13:30:20,361 - distributed.scheduler - INFO - Receive client connection: Client-worker-3e2f0a1e-3d86-11f1-89f6-2e20a4008324
2026-04-21 13:30:20,162 - distributed.scheduler - INFO - Receive client connection: Client-worker-3e0d44f1-3d86-11f1-899e-ae00acbfc48e
2026-04-21 13:30:16,827 - distributed.scheduler - INFO - Receive client connection: Client-3c141559-3d86-11f1-8001-daa6852bd5a7
2026-04-21 13:20:15,266 - distributed.scheduler - INFO - Receive client connection: Client-d58550b2-3d84-11f1-8001-daa6852bd5a7
2026-04-21 13:13:02,602 - distributed.scheduler - INFO - Receive client connection: Client-d3a209df-3d83-11f1-8001-daa6852bd5a7
2026-04-21 13:01:33,666 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:32861
2026-04-21 13:01:33,665 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:32861', status: closed, memory: 0, processing: 0>
2026-04-21 13:01:33,568 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:34665
2026-04-21 13:01:33,566 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:34665', status: closed, memory: 0, processing: 0>
2026-04-21 13:01:32,676 - distributed.scheduler - INFO - Close client connection: Client-worker-25424f81-3d82-11f1-8aed-f28f846166bd
2026-04-21 13:01:32,672 - distributed.scheduler - INFO - Close client connection: Client-worker-235b0524-3d82-11f1-8935-0e4c34ff5e38
2026-04-21 13:01:32,671 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:34717', status: closed, memory: 0, processing: 0>
2026-04-21 13:01:32,671 - distributed.scheduler - INFO - Remove client Client-worker-25424f81-3d82-11f1-8aed-f28f846166bd
2026-04-21 13:01:32,650 - distributed.scheduler - INFO - Remove client Client-worker-235b0524-3d82-11f1-8935-0e4c34ff5e38
2026-04-21 13:01:32,648 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:39941', status: closed, memory: 0, processing: 0>
2026-04-21 13:01:03,152 - distributed.scheduler - INFO - Receive client connection: Client-26cd73ba-3d82-11f1-8007-924c5ad31268
2026-04-21 13:01:01,681 - distributed.scheduler - INFO - Receive client connection: Client-worker-25eae3fd-3d82-11f1-89eb-fae0fcb74df3
2026-04-21 13:01:01,022 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:36167
2026-04-21 13:01:01,021 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:36167', status: closed, memory: 0, processing: 0>
2026-04-21 13:01:00,864 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:32973
2026-04-21 13:01:00,863 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:32973', status: closed, memory: 0, processing: 0>
2026-04-21 13:01:00,565 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:42221
2026-04-21 13:01:00,564 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:42221', status: closed, memory: 0, processing: 0>
2026-04-21 13:01:00,554 - distributed.scheduler - INFO - Receive client connection: Client-worker-25424f81-3d82-11f1-8aed-f28f846166bd
2026-04-21 13:01:00,022 - distributed.scheduler - INFO - Close client connection: Client-1036a330-3d69-11f1-8007-924c5ad31268
2026-04-21 13:01:00,020 - distributed.scheduler - INFO - Remove client Client-1036a330-3d69-11f1-8007-924c5ad31268
2026-04-21 13:01:00,018 - distributed.scheduler - INFO - Remove client Client-1036a330-3d69-11f1-8007-924c5ad31268
2026-04-21 13:00:59,985 - distributed.scheduler - INFO - Task run-93cb3608d73bb3603f4fbb7dabb40b66 marked as failed because 3 workers died while trying to run it
2026-04-21 13:00:59,984 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:36633', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:59,683 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:44523', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:59,348 - distributed.scheduler - INFO - Close client connection: Client-worker-d9074964-3d79-11f1-8976-ee7435aaba1f
2026-04-21 13:00:59,332 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:46087', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:59,330 - distributed.scheduler - INFO - Remove client Client-worker-d9074964-3d79-11f1-8976-ee7435aaba1f
2026-04-21 13:00:58,798 - distributed.scheduler - INFO - Receive client connection: Client-worker-2436308c-3d82-11f1-8a18-b6f02dd406e9
2026-04-21 13:00:57,470 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:37251
2026-04-21 13:00:57,469 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:37251', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:57,358 - distributed.scheduler - INFO - Receive client connection: Client-worker-235b0524-3d82-11f1-8935-0e4c34ff5e38
2026-04-21 13:00:57,254 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:44251
2026-04-21 13:00:57,253 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:44251', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:57,136 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:34717
2026-04-21 13:00:57,134 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:34717', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:57,057 - distributed.scheduler - INFO - Receive client connection: Client-worker-232d0962-3d82-11f1-8992-2eb33e86e1bd
2026-04-21 13:00:56,940 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:36641
2026-04-21 13:00:56,938 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:36641', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:56,878 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:36177
2026-04-21 13:00:56,877 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:36177', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:56,776 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:43461
2026-04-21 13:00:56,775 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:43461', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:56,769 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:39509
2026-04-21 13:00:56,768 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:39509', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:56,743 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:33875
2026-04-21 13:00:56,742 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:33875', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:56,527 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:43543
2026-04-21 13:00:56,526 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:43543', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:56,504 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:42343
2026-04-21 13:00:56,502 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:42343', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:56,497 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:34811
2026-04-21 13:00:56,496 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:34811', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:56,453 - distributed.scheduler - INFO - Close client connection: Client-worker-d81e53d5-3d7d-11f1-88fe-2a51d3dc178e
2026-04-21 13:00:56,449 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:38475', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:56,447 - distributed.scheduler - INFO - Remove client Client-worker-d81e53d5-3d7d-11f1-88fe-2a51d3dc178e
2026-04-21 13:00:56,396 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:46741
2026-04-21 13:00:56,394 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:46741', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:56,364 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:39941
2026-04-21 13:00:56,362 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:39941', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:56,155 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:41607', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:56,136 - distributed.scheduler - INFO - Close client connection: Client-worker-d54e5221-3d79-11f1-89b8-2e20a4008324
2026-04-21 13:00:56,131 - distributed.scheduler - INFO - Remove client Client-worker-d54e5221-3d79-11f1-89b8-2e20a4008324
2026-04-21 13:00:56,125 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:45509', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:55,956 - distributed.scheduler - INFO - Close client connection: Client-worker-d81f3f04-3d7d-11f1-89f4-4ef3162b8ac2
2026-04-21 13:00:55,952 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:39647', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:55,929 - distributed.scheduler - INFO - Remove client Client-worker-d81f3f04-3d7d-11f1-89f4-4ef3162b8ac2
2026-04-21 13:00:55,919 - distributed.scheduler - INFO - Close client connection: Client-worker-d765a1aa-3d7d-11f1-89a9-566b4744c386
2026-04-21 13:00:55,916 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:36355', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:55,915 - distributed.scheduler - INFO - Remove client Client-worker-d765a1aa-3d7d-11f1-89a9-566b4744c386
2026-04-21 13:00:55,801 - distributed.scheduler - INFO - Close client connection: Client-worker-d787af69-3d7d-11f1-893e-8288bfee4388
2026-04-21 13:00:55,800 - distributed.scheduler - INFO - Close client connection: Client-worker-0b2f5e3a-3d82-11f1-89d6-6ec63a5ab21e
2026-04-21 13:00:55,796 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:38315', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:55,796 - distributed.scheduler - INFO - Remove client Client-worker-d787af69-3d7d-11f1-893e-8288bfee4388
2026-04-21 13:00:55,792 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:44153', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:55,770 - distributed.scheduler - INFO - Remove client Client-worker-0b2f5e3a-3d82-11f1-89d6-6ec63a5ab21e
2026-04-21 13:00:55,737 - distributed.scheduler - INFO - Close client connection: Client-worker-0baf6254-3d82-11f1-8920-7e353d94dc10
2026-04-21 13:00:55,730 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:39063', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:55,728 - distributed.scheduler - INFO - Remove client Client-worker-0baf6254-3d82-11f1-8920-7e353d94dc10
2026-04-21 13:00:55,587 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:39201', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:55,512 - distributed.scheduler - INFO - Close client connection: Client-worker-d78d8c58-3d7d-11f1-895c-ae00acbfc48e
2026-04-21 13:00:55,496 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:42527', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:55,495 - distributed.scheduler - INFO - Remove client Client-worker-d78d8c58-3d7d-11f1-895c-ae00acbfc48e
2026-04-21 13:00:55,445 - distributed.scheduler - INFO - Close client connection: Client-worker-d783cff7-3d7d-11f1-8953-2eb33e86e1bd
2026-04-21 13:00:55,443 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:43651', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:55,428 - distributed.scheduler - INFO - Close client connection: Client-worker-0b8f4c67-3d82-11f1-88fa-0e4c34ff5e38
2026-04-21 13:00:55,411 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:38067', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:55,409 - distributed.scheduler - INFO - Remove client Client-worker-d783cff7-3d7d-11f1-8953-2eb33e86e1bd
2026-04-21 13:00:55,393 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:36613', status: closed, memory: 0, processing: 0>
2026-04-21 13:00:55,389 - distributed.scheduler - INFO - Remove client Client-worker-0b8f4c67-3d82-11f1-88fa-0e4c34ff5e38
2026-04-21 13:00:17,687 - distributed.scheduler - INFO - Receive client connection: Client-worker-0baf6254-3d82-11f1-8920-7e353d94dc10
2026-04-21 13:00:17,467 - distributed.scheduler - INFO - Receive client connection: Client-worker-0b8f4c67-3d82-11f1-88fa-0e4c34ff5e38
2026-04-21 13:00:16,814 - distributed.scheduler - INFO - Receive client connection: Client-worker-0b2f5e3a-3d82-11f1-89d6-6ec63a5ab21e
2026-04-21 13:00:12,340 - distributed.scheduler - INFO - Receive client connection: Client-08865efe-3d82-11f1-8001-daa6852bd5a7
2026-04-21 12:50:10,940 - distributed.scheduler - INFO - Receive client connection: Client-a20ec83b-3d80-11f1-8001-daa6852bd5a7
2026-04-21 12:40:09,454 - distributed.scheduler - INFO - Receive client connection: Client-3b8a6ee3-3d7f-11f1-8001-daa6852bd5a7
2026-04-21 12:31:15,468 - distributed.scheduler - INFO - Close client connection: Client-d50d980f-3d7d-11f1-8001-daa6852bd5a7
2026-04-21 12:31:15,467 - distributed.scheduler - INFO - Remove client Client-d50d980f-3d7d-11f1-8001-daa6852bd5a7
2026-04-21 12:31:15,465 - distributed.scheduler - INFO - Remove client Client-d50d980f-3d7d-11f1-8001-daa6852bd5a7
2026-04-21 12:30:13,146 - distributed.scheduler - INFO - Receive client connection: Client-worker-d81f3f04-3d7d-11f1-89f4-4ef3162b8ac2
2026-04-21 12:30:13,137 - distributed.scheduler - INFO - Receive client connection: Client-worker-d81e53d5-3d7d-11f1-88fe-2a51d3dc178e
2026-04-21 12:30:12,193 - distributed.scheduler - INFO - Receive client connection: Client-worker-d78d8c58-3d7d-11f1-895c-ae00acbfc48e
2026-04-21 12:30:12,165 - distributed.scheduler - INFO - Receive client connection: Client-worker-d787af69-3d7d-11f1-893e-8288bfee4388
2026-04-21 12:30:12,126 - distributed.scheduler - INFO - Receive client connection: Client-worker-d783cff7-3d7d-11f1-8953-2eb33e86e1bd
2026-04-21 12:30:11,935 - distributed.scheduler - INFO - Receive client connection: Client-worker-d765a1aa-3d7d-11f1-89a9-566b4744c386
2026-04-21 12:30:08,010 - distributed.scheduler - INFO - Receive client connection: Client-d50d980f-3d7d-11f1-8001-daa6852bd5a7
2026-04-21 12:20:44,091 - distributed.scheduler - INFO - Close client connection: Client-6e871fc6-3d7c-11f1-8001-daa6852bd5a7
2026-04-21 12:20:44,091 - distributed.scheduler - INFO - Remove client Client-6e871fc6-3d7c-11f1-8001-daa6852bd5a7
2026-04-21 12:20:44,090 - distributed.scheduler - INFO - Remove client Client-6e871fc6-3d7c-11f1-8001-daa6852bd5a7
2026-04-21 12:20:06,510 - distributed.scheduler - INFO - Receive client connection: Client-6e871fc6-3d7c-11f1-8001-daa6852bd5a7
2026-04-21 12:13:15,013 - distributed.scheduler - INFO - Close client connection: Client-650afb30-3d7b-11f1-8001-daa6852bd5a7
2026-04-21 12:13:15,012 - distributed.scheduler - INFO - Remove client Client-650afb30-3d7b-11f1-8001-daa6852bd5a7
2026-04-21 12:13:15,010 - distributed.scheduler - INFO - Remove client Client-650afb30-3d7b-11f1-8001-daa6852bd5a7
2026-04-21 12:12:41,096 - distributed.scheduler - INFO - Receive client connection: Client-650afb30-3d7b-11f1-8001-daa6852bd5a7
2026-04-21 12:01:36,683 - distributed.scheduler - INFO - Receive client connection: Client-worker-d9074964-3d79-11f1-8976-ee7435aaba1f
2026-04-21 12:01:31,554 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:44523
2026-04-21 12:01:31,553 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:44523', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:31,054 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:36633
2026-04-21 12:01:31,052 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:36633', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:30,514 - distributed.scheduler - INFO - Close client connection: Client-worker-a50f3ee2-3d79-11f1-88cb-c2349ba99f21
2026-04-21 12:01:30,513 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:35457', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:30,512 - distributed.scheduler - INFO - Remove client Client-worker-a50f3ee2-3d79-11f1-88cb-c2349ba99f21
2026-04-21 12:01:30,441 - distributed.scheduler - INFO - Receive client connection: Client-worker-d54e5221-3d79-11f1-89b8-2e20a4008324
2026-04-21 12:01:30,026 - distributed.scheduler - INFO - Close client connection: Client-worker-a468b9fc-3d79-11f1-8956-86acc56fd09f
2026-04-21 12:01:30,023 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:36319', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:30,021 - distributed.scheduler - INFO - Remove client Client-worker-a468b9fc-3d79-11f1-8956-86acc56fd09f
2026-04-21 12:01:29,622 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:39647
2026-04-21 12:01:29,619 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:39647', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:28,707 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:40217', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:28,064 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:38315
2026-04-21 12:01:28,064 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:38315', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,962 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:46087
2026-04-21 12:01:27,961 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:46087', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,936 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:39063
2026-04-21 12:01:27,935 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:39063', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,903 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:36355
2026-04-21 12:01:27,901 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:36355', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,885 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:36613
2026-04-21 12:01:27,884 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:36613', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,879 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:44153
2026-04-21 12:01:27,878 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:44153', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,834 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:42527
2026-04-21 12:01:27,832 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:42527', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,814 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:45509
2026-04-21 12:01:27,813 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:45509', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,805 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:41607
2026-04-21 12:01:27,804 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:41607', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,790 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:39201
2026-04-21 12:01:27,789 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:39201', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,757 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:38067
2026-04-21 12:01:27,756 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:38067', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,692 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:43651
2026-04-21 12:01:27,691 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:43651', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,672 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:38475
2026-04-21 12:01:27,670 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:38475', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,077 - distributed.scheduler - INFO - Close client connection: Client-worker-a425c461-3d79-11f1-88fd-8288bfee4388
2026-04-21 12:01:27,073 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:37345', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:27,072 - distributed.scheduler - INFO - Remove client Client-worker-a425c461-3d79-11f1-88fd-8288bfee4388
2026-04-21 12:01:26,942 - distributed.scheduler - INFO - Close client connection: Client-worker-a3df015a-3d79-11f1-88e4-7e353d94dc10
2026-04-21 12:01:26,940 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:38155', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:26,940 - distributed.scheduler - INFO - Remove client Client-worker-a3df015a-3d79-11f1-88e4-7e353d94dc10
2026-04-21 12:01:26,934 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:43583', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:26,930 - distributed.scheduler - INFO - Close client connection: Client-worker-a4040951-3d79-11f1-8938-ee7435aaba1f
2026-04-21 12:01:26,927 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:35771', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:26,921 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:44619', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:26,920 - distributed.scheduler - INFO - Remove client Client-worker-a4040951-3d79-11f1-8938-ee7435aaba1f
2026-04-21 12:01:26,897 - distributed.scheduler - INFO - Close client connection: Client-worker-d9e516f3-3d75-11f1-8920-ae00acbfc48e
2026-04-21 12:01:26,896 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:46583', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:26,893 - distributed.scheduler - INFO - Close client connection: Client-worker-a46c8cb5-3d79-11f1-88bc-0e4c34ff5e38
2026-04-21 12:01:26,888 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:36497', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:26,887 - distributed.scheduler - INFO - Remove client Client-worker-d9e516f3-3d75-11f1-8920-ae00acbfc48e
2026-04-21 12:01:26,885 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:33177', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:26,883 - distributed.scheduler - INFO - Remove client Client-worker-a46c8cb5-3d79-11f1-88bc-0e4c34ff5e38
2026-04-21 12:01:26,868 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:46133', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:26,840 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:32829', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:26,779 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:43499', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:26,727 - distributed.scheduler - INFO - Close client connection: Client-worker-de209b3a-3d75-11f1-899c-b6f02dd406e9
2026-04-21 12:01:26,720 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:36419', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:26,719 - distributed.scheduler - INFO - Remove client Client-worker-de209b3a-3d75-11f1-899c-b6f02dd406e9
2026-04-21 12:01:26,672 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:37565', status: closed, memory: 0, processing: 0>
2026-04-21 12:01:19,220 - distributed.scheduler - INFO - Close client connection: Client-a17cdb0a-3d79-11f1-8001-daa6852bd5a7
2026-04-21 12:01:19,218 - distributed.scheduler - INFO - Remove client Client-a17cdb0a-3d79-11f1-8001-daa6852bd5a7
2026-04-21 12:01:19,216 - distributed.scheduler - INFO - Remove client Client-a17cdb0a-3d79-11f1-8001-daa6852bd5a7
2026-04-21 12:00:09,486 - distributed.scheduler - INFO - Receive client connection: Client-worker-a50f3ee2-3d79-11f1-88cb-c2349ba99f21
2026-04-21 12:00:08,436 - distributed.scheduler - INFO - Receive client connection: Client-worker-a46c8cb5-3d79-11f1-88bc-0e4c34ff5e38
2026-04-21 12:00:08,403 - distributed.scheduler - INFO - Receive client connection: Client-worker-a468b9fc-3d79-11f1-8956-86acc56fd09f
2026-04-21 12:00:07,967 - distributed.scheduler - INFO - Receive client connection: Client-worker-a425c461-3d79-11f1-88fd-8288bfee4388
2026-04-21 12:00:07,740 - distributed.scheduler - INFO - Receive client connection: Client-worker-a4040951-3d79-11f1-8938-ee7435aaba1f
2026-04-21 12:00:07,491 - distributed.scheduler - INFO - Receive client connection: Client-worker-a3df015a-3d79-11f1-88e4-7e353d94dc10
2026-04-21 12:00:03,515 - distributed.scheduler - INFO - Receive client connection: Client-a17cdb0a-3d79-11f1-8001-daa6852bd5a7
2026-04-21 11:50:39,717 - distributed.scheduler - INFO - Close client connection: Client-3afcb4fc-3d78-11f1-8001-daa6852bd5a7
2026-04-21 11:50:39,717 - distributed.scheduler - INFO - Remove client Client-3afcb4fc-3d78-11f1-8001-daa6852bd5a7
2026-04-21 11:50:39,716 - distributed.scheduler - INFO - Remove client Client-3afcb4fc-3d78-11f1-8001-daa6852bd5a7
2026-04-21 11:50:02,047 - distributed.scheduler - INFO - Receive client connection: Client-3afcb4fc-3d78-11f1-8001-daa6852bd5a7
2026-04-21 11:40:41,260 - distributed.scheduler - INFO - Close client connection: Client-d475184c-3d76-11f1-8001-daa6852bd5a7
2026-04-21 11:40:41,259 - distributed.scheduler - INFO - Remove client Client-d475184c-3d76-11f1-8001-daa6852bd5a7
2026-04-21 11:40:41,257 - distributed.scheduler - INFO - Remove client Client-d475184c-3d76-11f1-8001-daa6852bd5a7
2026-04-21 11:40:00,531 - distributed.scheduler - INFO - Receive client connection: Client-d475184c-3d76-11f1-8001-daa6852bd5a7
2026-04-21 11:33:07,254 - distributed.scheduler - INFO - Receive client connection: Client-worker-de209b3a-3d75-11f1-899c-b6f02dd406e9
2026-04-21 11:33:01,942 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:35457
2026-04-21 11:33:01,941 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:35457', status: closed, memory: 0, processing: 0>
2026-04-21 11:33:01,717 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:36319
2026-04-21 11:33:01,715 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:36319', status: closed, memory: 0, processing: 0>
2026-04-21 11:33:00,916 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:44423', status: closed, memory: 0, processing: 0>
2026-04-21 11:33:00,706 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:35517', status: closed, memory: 0, processing: 0>
2026-04-21 11:33:00,154 - distributed.scheduler - INFO - Receive client connection: Client-worker-d9e516f3-3d75-11f1-8920-ae00acbfc48e
2026-04-21 11:33:00,123 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:37345
2026-04-21 11:33:00,122 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:37345', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:59,174 - distributed.scheduler - INFO - Close client connection: Client-worker-b5355f12-3d75-11f1-88bd-8288bfee4388
2026-04-21 11:32:59,169 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:45391', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:59,169 - distributed.scheduler - INFO - Remove client Client-worker-b5355f12-3d75-11f1-88bd-8288bfee4388
2026-04-21 11:32:58,374 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:32829
2026-04-21 11:32:58,371 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:32829', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:58,279 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:44619
2026-04-21 11:32:58,278 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:44619', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:58,200 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:36497
2026-04-21 11:32:58,198 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:36497', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:58,128 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:43583
2026-04-21 11:32:58,127 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:43583', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:58,119 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:46133
2026-04-21 11:32:58,118 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:46133', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,999 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:46583
2026-04-21 11:32:57,997 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:46583', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,982 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:40217
2026-04-21 11:32:57,980 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:40217', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,951 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:37565
2026-04-21 11:32:57,950 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:37565', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,916 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:43499
2026-04-21 11:32:57,914 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:43499', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,880 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:33177
2026-04-21 11:32:57,879 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:33177', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,856 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:38155
2026-04-21 11:32:57,855 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:38155', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,793 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:35771
2026-04-21 11:32:57,792 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:35771', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,785 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:36419
2026-04-21 11:32:57,783 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:36419', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,464 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:45779', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,278 - distributed.scheduler - INFO - Close client connection: Client-worker-b213b1a1-3d75-11f1-88f3-ee7435aaba1f
2026-04-21 11:32:57,275 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:38393', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,274 - distributed.scheduler - INFO - Remove client Client-worker-b213b1a1-3d75-11f1-88f3-ee7435aaba1f
2026-04-21 11:32:57,165 - distributed.scheduler - INFO - Close client connection: Client-worker-b32a53d9-3d75-11f1-88dc-ae00acbfc48e
2026-04-21 11:32:57,163 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:45303', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,161 - distributed.scheduler - INFO - Remove client Client-worker-b32a53d9-3d75-11f1-88dc-ae00acbfc48e
2026-04-21 11:32:57,096 - distributed.scheduler - INFO - Close client connection: Client-worker-b50d3052-3d75-11f1-8959-6ec63a5ab21e
2026-04-21 11:32:57,094 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:38011', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,093 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:43161', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,090 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:46103', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:57,061 - distributed.scheduler - INFO - Remove client Client-worker-b50d3052-3d75-11f1-8959-6ec63a5ab21e
2026-04-21 11:32:57,021 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:40327', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:56,988 - distributed.scheduler - INFO - Close client connection: Client-worker-b6055b46-3d75-11f1-8a3f-f28f846166bd
2026-04-21 11:32:56,983 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:34927', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:56,983 - distributed.scheduler - INFO - Remove client Client-worker-b6055b46-3d75-11f1-8a3f-f28f846166bd
2026-04-21 11:32:56,976 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:35269', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:56,963 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:34699', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:56,821 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:40823', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:56,814 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:35529', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:56,794 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:42921', status: closed, memory: 0, processing: 0>
2026-04-21 11:32:07,690 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-ea93a89a44f59423470bd8d52de83dda NoneType: None
2026-04-21 11:32:07,690 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-6f664a88c8d8daa5659e82119d5063d1 NoneType: None
2026-04-21 11:32:07,690 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-f26a8d82a3323670d4510b60c57dd53b NoneType: None
2026-04-21 11:32:07,690 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-6baeaf66aa57f670fdd48cbecb75bc4b NoneType: None
2026-04-21 11:32:07,690 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-8b86f4f0b9ed1270f74c591d2c074e4d NoneType: None
2026-04-21 11:32:07,690 - distributed.scheduler - ERROR - Couldn't gather keys {'run-8b86f4f0b9ed1270f74c591d2c074e4d': [], 'run-6baeaf66aa57f670fdd48cbecb75bc4b': [], 'run-f26a8d82a3323670d4510b60c57dd53b': [], 'run-6f664a88c8d8daa5659e82119d5063d1': [], 'run-ea93a89a44f59423470bd8d52de83dda': []} state: ['processing', 'processing', 'processing', 'processing', 'processing'] workers: [] NoneType: None
2026-04-21 11:31:59,958 - distributed.scheduler - INFO - Receive client connection: Client-worker-b6055b46-3d75-11f1-8a3f-f28f846166bd
2026-04-21 11:31:59,819 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:44423
2026-04-21 11:31:59,818 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:44423', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:59,644 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:35517
2026-04-21 11:31:59,643 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:35517', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:59,187 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:40823
2026-04-21 11:31:59,183 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:40823', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:58,629 - distributed.scheduler - INFO - Close client connection: Client-worker-6155721e-3d71-11f1-8852-c2349ba99f21
2026-04-21 11:31:58,623 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:43657', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:58,622 - distributed.scheduler - INFO - Remove client Client-worker-6155721e-3d71-11f1-8852-c2349ba99f21
2026-04-21 11:31:58,609 - distributed.scheduler - INFO - Receive client connection: Client-worker-b5355f12-3d75-11f1-88bd-8288bfee4388
2026-04-21 11:31:58,423 - distributed.scheduler - INFO - Close client connection: Client-worker-2dab1f01-3d6d-11f1-88d2-86acc56fd09f
2026-04-21 11:31:58,419 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:43369', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:58,417 - distributed.scheduler - INFO - Remove client Client-worker-2dab1f01-3d6d-11f1-88d2-86acc56fd09f
2026-04-21 11:31:58,347 - distributed.scheduler - INFO - Receive client connection: Client-worker-b50d3052-3d75-11f1-8959-6ec63a5ab21e
2026-04-21 11:31:58,003 - distributed.scheduler - INFO - Close client connection: Client-worker-2dadcc88-3d6d-11f1-8908-2e20a4008324
2026-04-21 11:31:57,999 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:36987', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:57,997 - distributed.scheduler - INFO - Remove client Client-worker-2dadcc88-3d6d-11f1-8908-2e20a4008324
2026-04-21 11:31:55,174 - distributed.scheduler - INFO - Receive client connection: Client-worker-b32a53d9-3d75-11f1-88dc-ae00acbfc48e
2026-04-21 11:31:53,393 - distributed.scheduler - INFO - Receive client connection: Client-worker-b213b1a1-3d75-11f1-88f3-ee7435aaba1f
2026-04-21 11:31:51,788 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:45391
2026-04-21 11:31:51,787 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:45391', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:50,822 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:33617', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:50,560 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:45779
2026-04-21 11:31:50,559 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:45779', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:50,171 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:45303
2026-04-21 11:31:50,170 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:45303', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:50,106 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:34927
2026-04-21 11:31:50,104 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:34927', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:50,097 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:34699
2026-04-21 11:31:50,096 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:34699', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:49,963 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:46103
2026-04-21 11:31:49,961 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:46103', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:49,903 - distributed.scheduler - INFO - Close client connection: Client-c3cb3e18-3d64-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,903 - distributed.scheduler - INFO - Close client connection: Client-90d71587-3d67-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,903 - distributed.scheduler - INFO - Close client connection: Client-2a4d0af6-3d66-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,903 - distributed.scheduler - INFO - Close client connection: Client-f75e764c-3d68-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,903 - distributed.scheduler - INFO - Close client connection: Client-b0b2560d-3d6a-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,902 - distributed.scheduler - INFO - Close client connection: Client-c46803f6-3d6b-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,902 - distributed.scheduler - INFO - Close client connection: Client-915cd92c-3d6e-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,902 - distributed.scheduler - INFO - Close client connection: Client-2ad62ec7-3d6d-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,901 - distributed.scheduler - INFO - Close client connection: Client-f7db4b17-3d6f-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,901 - distributed.scheduler - INFO - Close client connection: Client-5e5f9f88-3d71-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,901 - distributed.scheduler - INFO - Close client connection: Client-2b6ea144-3d74-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,901 - distributed.scheduler - INFO - Close client connection: Client-22c10b4b-3d73-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,899 - distributed.scheduler - INFO - Remove client Client-c3cb3e18-3d64-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,899 - distributed.scheduler - INFO - Remove client Client-90d71587-3d67-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,898 - distributed.scheduler - INFO - Remove client Client-2a4d0af6-3d66-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,897 - distributed.scheduler - INFO - Remove client Client-f75e764c-3d68-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,897 - distributed.scheduler - INFO - Remove client Client-b0b2560d-3d6a-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,896 - distributed.scheduler - INFO - Remove client Client-c46803f6-3d6b-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,893 - distributed.scheduler - INFO - Remove client Client-915cd92c-3d6e-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,892 - distributed.scheduler - INFO - Remove client Client-2ad62ec7-3d6d-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,891 - distributed.scheduler - INFO - Remove client Client-f7db4b17-3d6f-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,891 - distributed.scheduler - INFO - Remove client Client-5e5f9f88-3d71-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,890 - distributed.scheduler - INFO - Remove client Client-2b6ea144-3d74-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,889 - distributed.scheduler - INFO - Remove client Client-22c10b4b-3d73-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,803 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:38011
2026-04-21 11:31:49,802 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:38011', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:49,772 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:35269
2026-04-21 11:31:49,770 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:35269', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:49,703 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:43161
2026-04-21 11:31:49,702 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:43161', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:49,642 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:40327
2026-04-21 11:31:49,641 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:40327', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:49,620 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:38393
2026-04-21 11:31:49,619 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:38393', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:49,617 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:42921
2026-04-21 11:31:49,616 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:42921', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:49,426 - distributed.scheduler - INFO - Close client connection: Client-91d5aa31-3d75-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,425 - distributed.scheduler - INFO - Remove client Client-91d5aa31-3d75-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,425 - distributed.scheduler - INFO - Remove client Client-91d5aa31-3d75-11f1-8001-daa6852bd5a7
2026-04-21 11:31:49,409 - distributed.scheduler - INFO - Close client connection: Client-worker-2d3848c2-3d6d-11f1-88fb-fae0fcb74df3
2026-04-21 11:31:49,409 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:35529
2026-04-21 11:31:49,408 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:35529', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:49,384 - distributed.scheduler - INFO - Task ('from_pandas-6eff0ac2d8a5523414337928f5fa4785', 14) marked as failed because 3 workers died while trying to run it
2026-04-21 11:31:49,383 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:36341', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:49,381 - distributed.scheduler - INFO - Remove client Client-worker-2d3848c2-3d6d-11f1-88fb-fae0fcb74df3
2026-04-21 11:31:48,977 - distributed.scheduler - INFO - Close client connection: Client-worker-619431a0-3d71-11f1-8869-7e353d94dc10
2026-04-21 11:31:48,959 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:44547', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:48,958 - distributed.scheduler - INFO - Remove client Client-worker-619431a0-3d71-11f1-8869-7e353d94dc10
2026-04-21 11:31:48,950 - distributed.scheduler - INFO - Close client connection: Client-worker-6140eca5-3d71-11f1-8a03-f28f846166bd
2026-04-21 11:31:48,946 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:38621', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:48,944 - distributed.scheduler - INFO - Remove client Client-worker-6140eca5-3d71-11f1-8a03-f28f846166bd
2026-04-21 11:31:48,898 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:39519', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:48,863 - distributed.scheduler - INFO - Close client connection: Client-worker-2d19d52c-3d6d-11f1-8918-6ec63a5ab21e
2026-04-21 11:31:48,859 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:41269', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:48,857 - distributed.scheduler - INFO - Remove client Client-worker-2d19d52c-3d6d-11f1-8918-6ec63a5ab21e
2026-04-21 11:31:48,771 - distributed.scheduler - INFO - Close client connection: Client-worker-2db7cfd9-3d6d-11f1-8849-2a51d3dc178e
2026-04-21 11:31:48,761 - distributed.scheduler - INFO - Task ('hash-join-ef6b4f39704282dd70aa09b6f45ae113', 1) marked as failed because 3 workers died while trying to run it
2026-04-21 11:31:48,754 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:43817', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:48,752 - distributed.scheduler - INFO - Remove client Client-worker-2db7cfd9-3d6d-11f1-8849-2a51d3dc178e
2026-04-21 11:31:48,722 - distributed.scheduler - INFO - Close client connection: Client-worker-61c1f512-3d71-11f1-8946-4ef3162b8ac2
2026-04-21 11:31:48,718 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:42519', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:48,716 - distributed.scheduler - INFO - Remove client Client-worker-61c1f512-3d71-11f1-8946-4ef3162b8ac2
2026-04-21 11:31:48,682 - distributed.scheduler - INFO - Close client connection: Client-worker-611bdd0a-3d71-11f1-8844-0e4c34ff5e38
2026-04-21 11:31:48,679 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:42567', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:48,677 - distributed.scheduler - INFO - Remove client Client-worker-611bdd0a-3d71-11f1-8844-0e4c34ff5e38
2026-04-21 11:31:48,654 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:32945', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:48,607 - distributed.scheduler - INFO - Close client connection: Client-worker-2d82c5d9-3d6d-11f1-88b4-ee7435aaba1f
2026-04-21 11:31:48,603 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:40723', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:48,602 - distributed.scheduler - INFO - Remove client Client-worker-2d82c5d9-3d6d-11f1-88b4-ee7435aaba1f
2026-04-21 11:31:48,582 - distributed.scheduler - INFO - Close client connection: Client-worker-2d3ba461-3d6d-11f1-88a6-2eb33e86e1bd
2026-04-21 11:31:48,578 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:40361', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:48,576 - distributed.scheduler - INFO - Remove client Client-worker-2d3ba461-3d6d-11f1-88a6-2eb33e86e1bd
2026-04-21 11:31:48,411 - distributed.scheduler - INFO - Close client connection: Client-worker-618b91f7-3d71-11f1-88f0-566b4744c386
2026-04-21 11:31:48,408 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:36475', status: closed, memory: 0, processing: 0>
2026-04-21 11:31:48,407 - distributed.scheduler - INFO - Remove client Client-worker-618b91f7-3d71-11f1-88f0-566b4744c386
2026-04-21 11:30:59,263 - distributed.scheduler - INFO - Receive client connection: Client-91d5aa31-3d75-11f1-8001-daa6852bd5a7
2026-04-21 11:20:57,954 - distributed.scheduler - INFO - Receive client connection: Client-2b6ea144-3d74-11f1-8001-daa6852bd5a7
2026-04-21 11:13:33,896 - distributed.scheduler - INFO - Receive client connection: Client-22c10b4b-3d73-11f1-8001-daa6852bd5a7
2026-04-21 11:01:00,604 - distributed.scheduler - INFO - Receive client connection: Client-worker-61c1f512-3d71-11f1-8946-4ef3162b8ac2
2026-04-21 11:01:00,310 - distributed.scheduler - INFO - Receive client connection: Client-worker-619431a0-3d71-11f1-8869-7e353d94dc10
2026-04-21 11:01:00,253 - distributed.scheduler - INFO - Receive client connection: Client-worker-618b91f7-3d71-11f1-88f0-566b4744c386
2026-04-21 11:00:59,893 - distributed.scheduler - INFO - Receive client connection: Client-worker-6155721e-3d71-11f1-8852-c2349ba99f21
2026-04-21 11:00:59,765 - distributed.scheduler - INFO - Receive client connection: Client-worker-6140eca5-3d71-11f1-8a03-f28f846166bd
2026-04-21 11:00:59,522 - distributed.scheduler - INFO - Receive client connection: Client-worker-611bdd0a-3d71-11f1-8844-0e4c34ff5e38
2026-04-21 11:00:54,934 - distributed.scheduler - INFO - Receive client connection: Client-5e5f9f88-3d71-11f1-8001-daa6852bd5a7
2026-04-21 10:50:53,445 - distributed.scheduler - INFO - Receive client connection: Client-f7db4b17-3d6f-11f1-8001-daa6852bd5a7
2026-04-21 10:40:51,988 - distributed.scheduler - INFO - Receive client connection: Client-915cd92c-3d6e-11f1-8001-daa6852bd5a7
2026-04-21 10:30:55,339 - distributed.scheduler - INFO - Receive client connection: Client-worker-2db7cfd9-3d6d-11f1-8849-2a51d3dc178e
2026-04-21 10:30:55,245 - distributed.scheduler - INFO - Receive client connection: Client-worker-2dadcc88-3d6d-11f1-8908-2e20a4008324
2026-04-21 10:30:55,236 - distributed.scheduler - INFO - Receive client connection: Client-worker-2dab1f01-3d6d-11f1-88d2-86acc56fd09f
2026-04-21 10:30:54,980 - distributed.scheduler - INFO - Receive client connection: Client-worker-2d82c5d9-3d6d-11f1-88b4-ee7435aaba1f
2026-04-21 10:30:54,511 - distributed.scheduler - INFO - Receive client connection: Client-worker-2d3ba461-3d6d-11f1-88a6-2eb33e86e1bd
2026-04-21 10:30:54,468 - distributed.scheduler - INFO - Receive client connection: Client-worker-2d3848c2-3d6d-11f1-88fb-fae0fcb74df3
2026-04-21 10:30:54,284 - distributed.scheduler - INFO - Receive client connection: Client-worker-2d19d52c-3d6d-11f1-8918-6ec63a5ab21e
2026-04-21 10:30:50,482 - distributed.scheduler - INFO - Receive client connection: Client-2ad62ec7-3d6d-11f1-8001-daa6852bd5a7
2026-04-21 10:20:49,140 - distributed.scheduler - INFO - Receive client connection: Client-c46803f6-3d6b-11f1-8001-daa6852bd5a7
2026-04-21 10:13:06,567 - distributed.scheduler - INFO - Receive client connection: Client-b0b2560d-3d6a-11f1-8001-daa6852bd5a7
2026-04-21 10:01:29,241 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('assign-b9270b83db2de78ac6e5cf0bb8a1ef06', 13) NoneType: None
2026-04-21 10:01:29,241 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('assign-b9270b83db2de78ac6e5cf0bb8a1ef06', 8) NoneType: None
2026-04-21 10:01:29,241 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('assign-b9270b83db2de78ac6e5cf0bb8a1ef06', 15) NoneType: None
2026-04-21 10:01:29,241 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('assign-b9270b83db2de78ac6e5cf0bb8a1ef06', 9) NoneType: None
2026-04-21 10:01:29,241 - distributed.scheduler - ERROR - Couldn't gather keys {"('assign-b9270b83db2de78ac6e5cf0bb8a1ef06', 9)": [], "('assign-b9270b83db2de78ac6e5cf0bb8a1ef06', 15)": [], "('assign-b9270b83db2de78ac6e5cf0bb8a1ef06', 8)": [], "('assign-b9270b83db2de78ac6e5cf0bb8a1ef06', 13)": []} state: ['processing', 'waiting', 'memory', 'memory'] workers: [] NoneType: None
2026-04-21 10:01:27,838 - distributed.scheduler - INFO - Receive client connection: Client-1036a330-3d69-11f1-8007-924c5ad31268
2026-04-21 10:01:26,971 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:43817
2026-04-21 10:01:26,970 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:43817', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:26,588 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:36341
2026-04-21 10:01:26,587 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:36341', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:26,518 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:36987
2026-04-21 10:01:26,517 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:36987', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:26,485 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:42567
2026-04-21 10:01:26,484 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:42567', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:26,433 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:41269
2026-04-21 10:01:26,432 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:41269', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:26,314 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:33617
2026-04-21 10:01:26,313 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:33617', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:26,061 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:32945
2026-04-21 10:01:26,059 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:32945', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,967 - distributed.scheduler - INFO - Close client connection: Client-worker-d86dc5ba-3d64-11f1-8805-2a51d3dc178e
2026-04-21 10:01:25,963 - distributed.scheduler - INFO - Remove client Client-worker-d86dc5ba-3d64-11f1-8805-2a51d3dc178e
2026-04-21 10:01:25,959 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:45681', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,949 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:38621
2026-04-21 10:01:25,948 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:38621', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,921 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:39519
2026-04-21 10:01:25,919 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:39519', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,819 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:40723
2026-04-21 10:01:25,818 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:40723', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,810 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:43369
2026-04-21 10:01:25,808 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:43369', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,776 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:40361
2026-04-21 10:01:25,775 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:40361', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,772 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:43657
2026-04-21 10:01:25,771 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:43657', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,741 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:44547
2026-04-21 10:01:25,741 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:44547', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,705 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:42519
2026-04-21 10:01:25,704 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:42519', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,660 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:36475
2026-04-21 10:01:25,658 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:36475', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,553 - distributed.scheduler - INFO - Close client connection: Client-worker-f825c51a-3d64-11f1-88b8-fae0fcb74df3
2026-04-21 10:01:25,552 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:39815', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,533 - distributed.scheduler - INFO - Remove client Client-worker-f825c51a-3d64-11f1-88b8-fae0fcb74df3
2026-04-21 10:01:25,500 - distributed.scheduler - INFO - Close client connection: Client-worker-d7cce037-3d64-11f1-880a-0e4c34ff5e38
2026-04-21 10:01:25,499 - distributed.scheduler - INFO - Close client connection: Client-worker-fa4864fe-3d68-11f1-88c4-2e20a4008324
2026-04-21 10:01:25,497 - distributed.scheduler - INFO - Remove client Client-worker-d7cce037-3d64-11f1-880a-0e4c34ff5e38
2026-04-21 10:01:25,493 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:39709', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,484 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:44539', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,476 - distributed.scheduler - INFO - Remove client Client-worker-fa4864fe-3d68-11f1-88c4-2e20a4008324
2026-04-21 10:01:25,422 - distributed.scheduler - INFO - Close client connection: Client-5b3dcd8b-3d58-11f1-8007-924c5ad31268
2026-04-21 10:01:25,420 - distributed.scheduler - INFO - Remove client Client-5b3dcd8b-3d58-11f1-8007-924c5ad31268
2026-04-21 10:01:25,419 - distributed.scheduler - INFO - Remove client Client-5b3dcd8b-3d58-11f1-8007-924c5ad31268
2026-04-21 10:01:25,407 - distributed.scheduler - INFO - Close client connection: Client-worker-d89a3ae2-3d64-11f1-8847-8288bfee4388
2026-04-21 10:01:25,386 - distributed.scheduler - INFO - Task run-b8ea6ebc5140e5cfabc928500fdd6dad marked as failed because 3 workers died while trying to run it
2026-04-21 10:01:25,385 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:43073', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,384 - distributed.scheduler - INFO - Remove client Client-worker-d89a3ae2-3d64-11f1-8847-8288bfee4388
2026-04-21 10:01:25,376 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:38827', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,049 - distributed.scheduler - INFO - Close client connection: Client-worker-d8c95e81-3d64-11f1-89c1-f28f846166bd
2026-04-21 10:01:25,039 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:40043', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:25,037 - distributed.scheduler - INFO - Remove client Client-worker-d8c95e81-3d64-11f1-89c1-f28f846166bd
2026-04-21 10:01:25,018 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:43613', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:24,879 - distributed.scheduler - INFO - Close client connection: Client-worker-f4574c2e-3d64-11f1-8862-2eb33e86e1bd
2026-04-21 10:01:24,879 - distributed.scheduler - INFO - Close client connection: Client-worker-fa706b50-3d68-11f1-886a-ae00acbfc48e
2026-04-21 10:01:24,875 - distributed.scheduler - INFO - Close client connection: Client-worker-ff6cb806-3d64-11f1-890a-4ef3162b8ac2
2026-04-21 10:01:24,868 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:46439', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:24,865 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:39321', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:24,843 - distributed.scheduler - INFO - Remove client Client-worker-f4574c2e-3d64-11f1-8862-2eb33e86e1bd
2026-04-21 10:01:24,843 - distributed.scheduler - INFO - Remove client Client-worker-fa706b50-3d68-11f1-886a-ae00acbfc48e
2026-04-21 10:01:24,838 - distributed.scheduler - INFO - Close client connection: Client-worker-dda334a2-3d64-11f1-880d-c2349ba99f21
2026-04-21 10:01:24,832 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:34261', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:24,832 - distributed.scheduler - INFO - Remove client Client-worker-ff6cb806-3d64-11f1-890a-4ef3162b8ac2
2026-04-21 10:01:24,827 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:41539', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:24,808 - distributed.scheduler - INFO - Remove client Client-worker-dda334a2-3d64-11f1-880d-c2349ba99f21
2026-04-21 10:01:24,795 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:41235', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:24,783 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:39223', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:24,726 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:40037', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:24,700 - distributed.scheduler - INFO - Close client connection: Client-worker-d7dfdcf5-3d64-11f1-88ae-566b4744c386
2026-04-21 10:01:24,686 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:43413', status: closed, memory: 0, processing: 0>
2026-04-21 10:01:24,684 - distributed.scheduler - INFO - Remove client Client-worker-d7dfdcf5-3d64-11f1-88ae-566b4744c386
2026-04-21 10:00:51,353 - distributed.scheduler - INFO - Receive client connection: Client-worker-fa706b50-3d68-11f1-886a-ae00acbfc48e
2026-04-21 10:00:51,119 - distributed.scheduler - INFO - Receive client connection: Client-worker-fa4864fe-3d68-11f1-88c4-2e20a4008324
2026-04-21 10:00:46,148 - distributed.scheduler - INFO - Receive client connection: Client-f75e764c-3d68-11f1-8001-daa6852bd5a7
2026-04-21 09:50:44,637 - distributed.scheduler - INFO - Receive client connection: Client-90d71587-3d67-11f1-8001-daa6852bd5a7
2026-04-21 09:40:43,106 - distributed.scheduler - INFO - Receive client connection: Client-2a4d0af6-3d66-11f1-8001-daa6852bd5a7
2026-04-21 09:32:21,681 - distributed.scheduler - INFO - Receive client connection: Client-worker-ff6cb806-3d64-11f1-890a-4ef3162b8ac2
2026-04-21 09:32:09,460 - distributed.scheduler - INFO - Receive client connection: Client-worker-f825c51a-3d64-11f1-88b8-fae0fcb74df3
2026-04-21 09:32:03,093 - distributed.scheduler - INFO - Receive client connection: Client-worker-f4574c2e-3d64-11f1-8862-2eb33e86e1bd
2026-04-21 09:31:25,017 - distributed.scheduler - INFO - Receive client connection: Client-worker-dda334a2-3d64-11f1-880d-c2349ba99f21
2026-04-21 09:31:24,755 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:41235
2026-04-21 09:31:24,753 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:41235', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:24,640 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:40037
2026-04-21 09:31:24,639 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:40037', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:23,784 - distributed.scheduler - INFO - Close client connection: Client-worker-5f777c04-3d5c-11f1-883c-ee7435aaba1f
2026-04-21 09:31:23,739 - distributed.scheduler - INFO - Task ('from_pandas-d744eab5f8a0b5d6ecd7c4055734eea7', 3) marked as failed because 3 workers died while trying to run it
2026-04-21 09:31:23,738 - distributed.scheduler - INFO - Task ('from_pandas-d744eab5f8a0b5d6ecd7c4055734eea7', 12) marked as failed because 3 workers died while trying to run it
2026-04-21 09:31:23,738 - distributed.scheduler - INFO - Task ('from_pandas-d744eab5f8a0b5d6ecd7c4055734eea7', 15) marked as failed because 3 workers died while trying to run it
2026-04-21 09:31:23,737 - distributed.scheduler - INFO - Task ('from_pandas-d744eab5f8a0b5d6ecd7c4055734eea7', 7) marked as failed because 3 workers died while trying to run it
2026-04-21 09:31:23,736 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:42591', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:23,735 - distributed.scheduler - INFO - Remove client Client-worker-5f777c04-3d5c-11f1-883c-ee7435aaba1f
2026-04-21 09:31:23,640 - distributed.scheduler - INFO - Close client connection: Client-worker-c6354a88-3d64-11f1-885b-86acc56fd09f
2026-04-21 09:31:23,625 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:34351', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:23,603 - distributed.scheduler - INFO - Remove client Client-worker-c6354a88-3d64-11f1-885b-86acc56fd09f
2026-04-21 09:31:16,846 - distributed.scheduler - INFO - Receive client connection: Client-worker-d8c95e81-3d64-11f1-89c1-f28f846166bd
2026-04-21 09:31:16,548 - distributed.scheduler - INFO - Receive client connection: Client-worker-d89a3ae2-3d64-11f1-8847-8288bfee4388
2026-04-21 09:31:16,263 - distributed.scheduler - INFO - Receive client connection: Client-worker-d86dc5ba-3d64-11f1-8805-2a51d3dc178e
2026-04-21 09:31:15,368 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:34261
2026-04-21 09:31:15,367 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:34261', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:15,358 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:46439
2026-04-21 09:31:15,356 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:46439', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:15,327 - distributed.scheduler - INFO - Receive client connection: Client-worker-d7dfdcf5-3d64-11f1-88ae-566b4744c386
2026-04-21 09:31:15,189 - distributed.scheduler - INFO - Receive client connection: Client-worker-d7cce037-3d64-11f1-880a-0e4c34ff5e38
2026-04-21 09:31:14,722 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:41539
2026-04-21 09:31:14,720 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:41539', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,670 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:44539
2026-04-21 09:31:14,669 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:44539', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,579 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:40043
2026-04-21 09:31:14,578 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:40043', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,538 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:39709
2026-04-21 09:31:14,537 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:39709', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,504 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:43073
2026-04-21 09:31:14,504 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:43073', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,483 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:43613
2026-04-21 09:31:14,482 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:43613', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,436 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:39321
2026-04-21 09:31:14,436 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:39321', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,433 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:39223
2026-04-21 09:31:14,433 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:39223', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,433 - distributed.scheduler - INFO - Close client connection: Client-worker-931e0127-3d60-11f1-88ca-4ef3162b8ac2
2026-04-21 09:31:14,432 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:38827
2026-04-21 09:31:14,432 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:38827', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,431 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:45681
2026-04-21 09:31:14,431 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:45681', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,431 - distributed.scheduler - INFO - Close client connection: Client-worker-c6beb642-3d64-11f1-882f-ae00acbfc48e
2026-04-21 09:31:14,430 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:39815
2026-04-21 09:31:14,429 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:39815', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,428 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:43413
2026-04-21 09:31:14,428 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:43413', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,425 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:36329', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,424 - distributed.scheduler - INFO - Remove client Client-worker-931e0127-3d60-11f1-88ca-4ef3162b8ac2
2026-04-21 09:31:14,420 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:32823', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:14,271 - distributed.scheduler - INFO - Remove client Client-worker-c6beb642-3d64-11f1-882f-ae00acbfc48e
2026-04-21 09:31:13,666 - distributed.scheduler - INFO - Close client connection: Client-worker-5ffce8a8-3d5c-11f1-87c5-c2349ba99f21
2026-04-21 09:31:13,661 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:35193', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:13,648 - distributed.scheduler - INFO - Remove client Client-worker-5ffce8a8-3d5c-11f1-87c5-c2349ba99f21
2026-04-21 09:31:13,611 - distributed.scheduler - INFO - Close client connection: Client-worker-5fc4326b-3d5c-11f1-8824-2eb33e86e1bd
2026-04-21 09:31:13,611 - distributed.scheduler - INFO - Close client connection: Client-worker-5f7088fc-3d5c-11f1-8897-6ec63a5ab21e
2026-04-21 09:31:13,586 - distributed.scheduler - INFO - Task ('from_pandas-f2c9f00a2fdeb85b20c2f0c6ef740c03', 7) marked as failed because 3 workers died while trying to run it
2026-04-21 09:31:13,586 - distributed.scheduler - INFO - Task ('from_pandas-d5a958f3021cdbf5030098754668ac59', 0) marked as failed because 3 workers died while trying to run it
2026-04-21 09:31:13,586 - distributed.scheduler - INFO - Task ('from_pandas-3be5f9432f2fd15c1ed804b25b95b95d', 9) marked as failed because 3 workers died while trying to run it
2026-04-21 09:31:13,585 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:38729', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:13,579 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:46311', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:13,578 - distributed.scheduler - INFO - Remove client Client-worker-5fc4326b-3d5c-11f1-8824-2eb33e86e1bd
2026-04-21 09:31:13,578 - distributed.scheduler - INFO - Remove client Client-worker-5f7088fc-3d5c-11f1-8897-6ec63a5ab21e
2026-04-21 09:31:13,576 - distributed.scheduler - INFO - Close client connection: Client-worker-92fc46e3-3d60-11f1-8982-f28f846166bd
2026-04-21 09:31:13,576 - distributed.scheduler - INFO - Close client connection: Client-worker-5f94a404-3d5c-11f1-87fe-8288bfee4388
2026-04-21 09:31:13,567 - distributed.scheduler - INFO - Close client connection: Client-worker-5aeed9aa-3d58-11f1-8876-2e20a4008324
2026-04-21 09:31:13,567 - distributed.scheduler - INFO - Close client connection: Client-worker-9340d21c-3d60-11f1-8879-fae0fcb74df3
2026-04-21 09:31:13,567 - distributed.scheduler - INFO - Close client connection: Client-worker-5fde1cff-3d5c-11f1-87cc-0e4c34ff5e38
2026-04-21 09:31:13,561 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:33093', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:13,561 - distributed.scheduler - INFO - Remove client Client-worker-92fc46e3-3d60-11f1-8982-f28f846166bd
2026-04-21 09:31:13,554 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:44027', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:13,553 - distributed.scheduler - INFO - Remove client Client-worker-5f94a404-3d5c-11f1-87fe-8288bfee4388
2026-04-21 09:31:13,550 - distributed.scheduler - INFO - Close client connection: Client-worker-5f4ec78c-3d5c-11f1-87c7-2a51d3dc178e
2026-04-21 09:31:13,541 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:45379', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:13,541 - distributed.scheduler - INFO - Remove client Client-worker-5aeed9aa-3d58-11f1-8876-2e20a4008324
2026-04-21 09:31:13,523 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:32797', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:13,521 - distributed.scheduler - INFO - Remove client Client-worker-9340d21c-3d60-11f1-8879-fae0fcb74df3
2026-04-21 09:31:13,513 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:34231', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:13,499 - distributed.scheduler - INFO - Remove client Client-worker-5fde1cff-3d5c-11f1-87cc-0e4c34ff5e38
2026-04-21 09:31:13,478 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:37351', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:13,477 - distributed.scheduler - INFO - Remove client Client-worker-5f4ec78c-3d5c-11f1-87c7-2a51d3dc178e
2026-04-21 09:31:13,436 - distributed.scheduler - INFO - Close client connection: Client-worker-93ad67d9-3d60-11f1-8873-566b4744c386
2026-04-21 09:31:13,430 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:40029', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:13,417 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:42407', status: closed, memory: 0, processing: 0>
2026-04-21 09:31:13,416 - distributed.scheduler - INFO - Remove client Client-worker-93ad67d9-3d60-11f1-8873-566b4744c386
2026-04-21 09:31:13,378 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:33655', status: closed, memory: 0, processing: 0>
2026-04-21 09:30:46,665 - distributed.scheduler - INFO - Receive client connection: Client-worker-c6beb642-3d64-11f1-882f-ae00acbfc48e
2026-04-21 09:30:45,682 - distributed.scheduler - INFO - Receive client connection: Client-worker-c6354a88-3d64-11f1-885b-86acc56fd09f
2026-04-21 09:30:41,631 - distributed.scheduler - INFO - Receive client connection: Client-c3cb3e18-3d64-11f1-8001-daa6852bd5a7
2026-04-21 09:21:16,836 - distributed.scheduler - INFO - Close client connection: Client-5d51f26a-3d63-11f1-8001-daa6852bd5a7
2026-04-21 09:21:16,835 - distributed.scheduler - INFO - Remove client Client-5d51f26a-3d63-11f1-8001-daa6852bd5a7
2026-04-21 09:21:16,834 - distributed.scheduler - INFO - Remove client Client-5d51f26a-3d63-11f1-8001-daa6852bd5a7
2026-04-21 09:20:40,224 - distributed.scheduler - INFO - Receive client connection: Client-5d51f26a-3d63-11f1-8001-daa6852bd5a7
2026-04-21 09:13:56,991 - distributed.scheduler - INFO - Close client connection: Client-59b7d562-3d62-11f1-8001-daa6852bd5a7
2026-04-21 09:13:56,990 - distributed.scheduler - INFO - Remove client Client-59b7d562-3d62-11f1-8001-daa6852bd5a7
2026-04-21 09:13:56,988 - distributed.scheduler - INFO - Remove client Client-59b7d562-3d62-11f1-8001-daa6852bd5a7
2026-04-21 09:13:24,682 - distributed.scheduler - INFO - Receive client connection: Client-59b7d562-3d62-11f1-8001-daa6852bd5a7
2026-04-21 09:01:43,023 - distributed.scheduler - INFO - Close client connection: Client-90612e13-3d60-11f1-8001-daa6852bd5a7
2026-04-21 09:01:43,022 - distributed.scheduler - INFO - Remove client Client-90612e13-3d60-11f1-8001-daa6852bd5a7
2026-04-21 09:01:43,020 - distributed.scheduler - INFO - Remove client Client-90612e13-3d60-11f1-8001-daa6852bd5a7
2026-04-21 09:00:42,915 - distributed.scheduler - INFO - Receive client connection: Client-worker-93ad67d9-3d60-11f1-8873-566b4744c386
2026-04-21 09:00:42,202 - distributed.scheduler - INFO - Receive client connection: Client-worker-9340d21c-3d60-11f1-8879-fae0fcb74df3
2026-04-21 09:00:41,993 - distributed.scheduler - INFO - Receive client connection: Client-worker-931e0127-3d60-11f1-88ca-4ef3162b8ac2
2026-04-21 09:00:41,753 - distributed.scheduler - INFO - Receive client connection: Client-worker-92fc46e3-3d60-11f1-8982-f28f846166bd
2026-04-21 09:00:37,393 - distributed.scheduler - INFO - Receive client connection: Client-90612e13-3d60-11f1-8001-daa6852bd5a7
2026-04-21 08:51:15,101 - distributed.scheduler - INFO - Close client connection: Client-29d9787c-3d5f-11f1-8001-daa6852bd5a7
2026-04-21 08:51:15,101 - distributed.scheduler - INFO - Remove client Client-29d9787c-3d5f-11f1-8001-daa6852bd5a7
2026-04-21 08:51:15,099 - distributed.scheduler - INFO - Remove client Client-29d9787c-3d5f-11f1-8001-daa6852bd5a7
2026-04-21 08:50:35,879 - distributed.scheduler - INFO - Receive client connection: Client-29d9787c-3d5f-11f1-8001-daa6852bd5a7
2026-04-21 08:41:10,586 - distributed.scheduler - INFO - Close client connection: Client-c34e4a24-3d5d-11f1-8001-daa6852bd5a7
2026-04-21 08:41:10,586 - distributed.scheduler - INFO - Remove client Client-c34e4a24-3d5d-11f1-8001-daa6852bd5a7
2026-04-21 08:41:10,585 - distributed.scheduler - INFO - Remove client Client-c34e4a24-3d5d-11f1-8001-daa6852bd5a7
2026-04-21 08:40:34,338 - distributed.scheduler - INFO - Receive client connection: Client-c34e4a24-3d5d-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,304 - distributed.scheduler - INFO - Close client connection: Client-c225f2c4-3d4f-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,304 - distributed.scheduler - INFO - Close client connection: Client-8f2dd663-3d52-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,303 - distributed.scheduler - INFO - Close client connection: Client-86b4c8fe-3d51-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,303 - distributed.scheduler - INFO - Close client connection: Client-f5b6fbdb-3d53-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,303 - distributed.scheduler - INFO - Close client connection: Client-5c3b9d93-3d55-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,303 - distributed.scheduler - INFO - Close client connection: Client-c2be6057-3d56-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,302 - distributed.scheduler - INFO - Close client connection: Client-29444721-3d58-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,300 - distributed.scheduler - INFO - Close client connection: Client-f63c7a84-3d5a-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,300 - distributed.scheduler - INFO - Close client connection: Client-e4394e75-3d59-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,299 - distributed.scheduler - INFO - Remove client Client-c225f2c4-3d4f-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,298 - distributed.scheduler - INFO - Remove client Client-8f2dd663-3d52-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,297 - distributed.scheduler - INFO - Remove client Client-86b4c8fe-3d51-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,296 - distributed.scheduler - INFO - Remove client Client-f5b6fbdb-3d53-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,296 - distributed.scheduler - INFO - Remove client Client-5c3b9d93-3d55-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,295 - distributed.scheduler - INFO - Remove client Client-c2be6057-3d56-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,293 - distributed.scheduler - INFO - Remove client Client-29444721-3d58-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,289 - distributed.scheduler - INFO - Remove client Client-f63c7a84-3d5a-11f1-8001-daa6852bd5a7
2026-04-21 08:31:36,288 - distributed.scheduler - INFO - Remove client Client-e4394e75-3d59-11f1-8001-daa6852bd5a7
2026-04-21 08:31:35,483 - distributed.scheduler - INFO - Close client connection: Client-5ccacce2-3d5c-11f1-8001-daa6852bd5a7
2026-04-21 08:31:35,482 - distributed.scheduler - INFO - Remove client Client-5ccacce2-3d5c-11f1-8001-daa6852bd5a7
2026-04-21 08:31:35,480 - distributed.scheduler - INFO - Remove client Client-5ccacce2-3d5c-11f1-8001-daa6852bd5a7
2026-04-21 08:30:38,207 - distributed.scheduler - INFO - Receive client connection: Client-worker-5ffce8a8-3d5c-11f1-87c5-c2349ba99f21
2026-04-21 08:30:38,043 - distributed.scheduler - INFO - Receive client connection: Client-worker-5fc4326b-3d5c-11f1-8824-2eb33e86e1bd
2026-04-21 08:30:37,995 - distributed.scheduler - INFO - Receive client connection: Client-worker-5fde1cff-3d5c-11f1-87cc-0e4c34ff5e38
2026-04-21 08:30:37,553 - distributed.scheduler - INFO - Receive client connection: Client-worker-5f94a404-3d5c-11f1-87fe-8288bfee4388
2026-04-21 08:30:37,337 - distributed.scheduler - INFO - Receive client connection: Client-worker-5f777c04-3d5c-11f1-883c-ee7435aaba1f
2026-04-21 08:30:37,286 - distributed.scheduler - INFO - Receive client connection: Client-worker-5f7088fc-3d5c-11f1-8897-6ec63a5ab21e
2026-04-21 08:30:37,092 - distributed.scheduler - INFO - Receive client connection: Client-worker-5f4ec78c-3d5c-11f1-87c7-2a51d3dc178e
2026-04-21 08:30:32,851 - distributed.scheduler - INFO - Receive client connection: Client-5ccacce2-3d5c-11f1-8001-daa6852bd5a7
2026-04-21 08:20:31,289 - distributed.scheduler - INFO - Receive client connection: Client-f63c7a84-3d5a-11f1-8001-daa6852bd5a7
2026-04-21 08:12:51,572 - distributed.scheduler - INFO - Receive client connection: Client-e4394e75-3d59-11f1-8001-daa6852bd5a7
2026-04-21 08:01:54,873 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:35193
2026-04-21 08:01:54,873 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:35193', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:54,846 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:34351
2026-04-21 08:01:54,844 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:34351', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:53,813 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:37199', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:53,718 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:43991', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:52,282 - distributed.scheduler - INFO - Receive client connection: Client-5b3dcd8b-3d58-11f1-8007-924c5ad31268
2026-04-21 08:01:51,740 - distributed.scheduler - INFO - Receive client connection: Client-worker-5aeed9aa-3d58-11f1-8876-2e20a4008324
2026-04-21 08:01:50,411 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:37351
2026-04-21 08:01:50,410 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:37351', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:50,257 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:36329
2026-04-21 08:01:50,256 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:36329', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:50,161 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:33655
2026-04-21 08:01:50,160 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:33655', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:50,144 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:46311
2026-04-21 08:01:50,143 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:46311', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,929 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:38729
2026-04-21 08:01:49,928 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:38729', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,926 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:44027
2026-04-21 08:01:49,925 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:44027', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,810 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:42407
2026-04-21 08:01:49,809 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:42407', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,791 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:32797
2026-04-21 08:01:49,790 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:32797', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,679 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:33093
2026-04-21 08:01:49,678 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:33093', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,676 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:34231
2026-04-21 08:01:49,675 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:34231', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,606 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:32823
2026-04-21 08:01:49,606 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:32823', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,525 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:42591
2026-04-21 08:01:49,525 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:42591', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,514 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:40029
2026-04-21 08:01:49,513 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:40029', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,376 - distributed.scheduler - INFO - Close client connection: Client-6f4b96cc-3d43-11f1-8007-924c5ad31268
2026-04-21 08:01:49,376 - distributed.scheduler - INFO - Remove client Client-6f4b96cc-3d43-11f1-8007-924c5ad31268
2026-04-21 08:01:49,375 - distributed.scheduler - INFO - Remove client Client-6f4b96cc-3d43-11f1-8007-924c5ad31268
2026-04-21 08:01:49,360 - distributed.scheduler - INFO - Close client connection: Client-worker-432bd7e1-3d58-11f1-887d-4ef3162b8ac2
2026-04-21 08:01:49,356 - distributed.scheduler - INFO - Task run-ddecefd60677110ce87a981f7319108c marked as failed because 3 workers died while trying to run it
2026-04-21 08:01:49,356 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:42959', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,317 - distributed.scheduler - INFO - Remove client Client-worker-432bd7e1-3d58-11f1-887d-4ef3162b8ac2
2026-04-21 08:01:49,270 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:40323', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,239 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:45379
2026-04-21 08:01:49,237 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:45379', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,169 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:36151', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,081 - distributed.scheduler - INFO - Close client connection: Client-worker-44262dce-3d58-11f1-87bc-7e353d94dc10
2026-04-21 08:01:49,077 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:35053', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:49,076 - distributed.scheduler - INFO - Remove client Client-worker-44262dce-3d58-11f1-87bc-7e353d94dc10
2026-04-21 08:01:48,965 - distributed.scheduler - INFO - Close client connection: Client-worker-42fd04c1-3d58-11f1-87e3-2eb33e86e1bd
2026-04-21 08:01:48,961 - distributed.scheduler - INFO - Task run-a2dbd316f5ed53569c30dea548c0d572 marked as failed because 3 workers died while trying to run it
2026-04-21 08:01:48,961 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:34541', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:48,926 - distributed.scheduler - INFO - Remove client Client-worker-42fd04c1-3d58-11f1-87e3-2eb33e86e1bd
2026-04-21 08:01:48,898 - distributed.scheduler - INFO - Close client connection: Client-worker-42533b15-3d58-11f1-8837-566b4744c386
2026-04-21 08:01:48,893 - distributed.scheduler - INFO - Remove client Client-worker-42533b15-3d58-11f1-8837-566b4744c386
2026-04-21 08:01:48,881 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:44791', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:48,692 - distributed.scheduler - INFO - Close client connection: Client-worker-4354e45f-3d58-11f1-8837-fae0fcb74df3
2026-04-21 08:01:48,688 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:39633', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:48,688 - distributed.scheduler - INFO - Remove client Client-worker-4354e45f-3d58-11f1-8837-fae0fcb74df3
2026-04-21 08:01:48,662 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:34851', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:48,567 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:33993', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:48,515 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:41369', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:48,484 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:37915', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:48,464 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:42501', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:48,368 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:40743', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:48,191 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:43341', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:13,882 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:37199
2026-04-21 08:01:13,881 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:37199', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:13,612 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:43991
2026-04-21 08:01:13,611 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:43991', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:13,543 - distributed.scheduler - INFO - Receive client connection: Client-worker-44262dce-3d58-11f1-87bc-7e353d94dc10
2026-04-21 08:01:12,827 - distributed.scheduler - INFO - Close client connection: Client-worker-2bc5134c-3d58-11f1-874d-c2349ba99f21
2026-04-21 08:01:12,816 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:41311', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:12,815 - distributed.scheduler - INFO - Remove client Client-worker-2bc5134c-3d58-11f1-874d-c2349ba99f21
2026-04-21 08:01:12,516 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:35953', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:12,147 - distributed.scheduler - INFO - Receive client connection: Client-worker-4354e45f-3d58-11f1-8837-fae0fcb74df3
2026-04-21 08:01:11,884 - distributed.scheduler - INFO - Receive client connection: Client-worker-432bd7e1-3d58-11f1-887d-4ef3162b8ac2
2026-04-21 08:01:11,570 - distributed.scheduler - INFO - Receive client connection: Client-worker-42fd04c1-3d58-11f1-87e3-2eb33e86e1bd
2026-04-21 08:01:10,452 - distributed.scheduler - INFO - Receive client connection: Client-worker-42533b15-3d58-11f1-8837-566b4744c386
2026-04-21 08:01:10,433 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:37915
2026-04-21 08:01:10,431 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:37915', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:10,281 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:35053
2026-04-21 08:01:10,280 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:35053', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:10,155 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:43341
2026-04-21 08:01:10,155 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:43341', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:10,128 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:40323
2026-04-21 08:01:10,126 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:40323', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:10,113 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:40743
2026-04-21 08:01:10,112 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:40743', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:10,086 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:33993
2026-04-21 08:01:10,086 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:33993', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:10,072 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:42501
2026-04-21 08:01:10,071 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:42501', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:10,070 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:36151
2026-04-21 08:01:10,069 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:36151', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,973 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:41369
2026-04-21 08:01:09,972 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:41369', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,969 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:34851
2026-04-21 08:01:09,968 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:34851', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,944 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:34541
2026-04-21 08:01:09,943 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:34541', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,941 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:39633
2026-04-21 08:01:09,940 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:39633', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,836 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:44791
2026-04-21 08:01:09,834 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:44791', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,708 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:42959
2026-04-21 08:01:09,707 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:42959', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,264 - distributed.scheduler - INFO - Close client connection: Client-worker-18a7e5ac-3d54-11f1-8779-7e353d94dc10
2026-04-21 08:01:09,254 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:41207', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,253 - distributed.scheduler - INFO - Remove client Client-worker-18a7e5ac-3d54-11f1-8779-7e353d94dc10
2026-04-21 08:01:09,239 - distributed.scheduler - INFO - Close client connection: Client-worker-10266db7-3d54-11f1-8746-2a51d3dc178e
2026-04-21 08:01:09,235 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:37923', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,234 - distributed.scheduler - INFO - Remove client Client-worker-10266db7-3d54-11f1-8746-2a51d3dc178e
2026-04-21 08:01:09,221 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:38739', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,183 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:36617', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,181 - distributed.scheduler - INFO - Close client connection: Client-worker-194758e7-3d54-11f1-890d-f28f846166bd
2026-04-21 08:01:09,172 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:39505', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,171 - distributed.scheduler - INFO - Remove client Client-worker-194758e7-3d54-11f1-890d-f28f846166bd
2026-04-21 08:01:09,092 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:39257', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,079 - distributed.scheduler - INFO - Close client connection: Client-worker-10940339-3d54-11f1-8755-0e4c34ff5e38
2026-04-21 08:01:09,074 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:36989', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,073 - distributed.scheduler - INFO - Remove client Client-worker-10940339-3d54-11f1-8755-0e4c34ff5e38
2026-04-21 08:01:09,046 - distributed.scheduler - INFO - Close client connection: Client-worker-2be6179f-3d58-11f1-87a0-2eb33e86e1bd
2026-04-21 08:01:09,041 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:39641', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,040 - distributed.scheduler - INFO - Close client connection: Client-worker-11364d22-3d54-11f1-8839-b6f02dd406e9
2026-04-21 08:01:09,039 - distributed.scheduler - INFO - Remove client Client-worker-2be6179f-3d58-11f1-87a0-2eb33e86e1bd
2026-04-21 08:01:09,024 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:35351', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:09,023 - distributed.scheduler - INFO - Remove client Client-worker-11364d22-3d54-11f1-8839-b6f02dd406e9
2026-04-21 08:01:08,975 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:43605', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:08,967 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:36563', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:08,902 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:40963', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:08,824 - distributed.scheduler - INFO - Close client connection: Client-worker-0f68acbd-3d54-11f1-87f8-566b4744c386
2026-04-21 08:01:08,804 - distributed.scheduler - INFO - Remove client Client-worker-0f68acbd-3d54-11f1-87f8-566b4744c386
2026-04-21 08:01:08,791 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:42493', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:08,641 - distributed.scheduler - INFO - Close client connection: Client-worker-114eef3b-3d54-11f1-8835-4ef3162b8ac2
2026-04-21 08:01:08,634 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:46749', status: closed, memory: 0, processing: 0>
2026-04-21 08:01:08,607 - distributed.scheduler - INFO - Remove client Client-worker-114eef3b-3d54-11f1-8835-4ef3162b8ac2
2026-04-21 08:00:32,867 - distributed.scheduler - INFO - Receive client connection: Client-worker-2be6179f-3d58-11f1-87a0-2eb33e86e1bd
2026-04-21 08:00:32,612 - distributed.scheduler - INFO - Receive client connection: Client-worker-2bc5134c-3d58-11f1-874d-c2349ba99f21
2026-04-21 08:00:28,415 - distributed.scheduler - INFO - Receive client connection: Client-29444721-3d58-11f1-8001-daa6852bd5a7
2026-04-21 07:50:26,915 - distributed.scheduler - INFO - Receive client connection: Client-c2be6057-3d56-11f1-8001-daa6852bd5a7
2026-04-21 07:40:25,435 - distributed.scheduler - INFO - Receive client connection: Client-5c3b9d93-3d55-11f1-8001-daa6852bd5a7
2026-04-21 07:31:45,538 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-86ec96045ba510dae6b279c07fae229f NoneType: None
2026-04-21 07:31:45,537 - distributed.scheduler - ERROR - Couldn't gather keys {'run-86ec96045ba510dae6b279c07fae229f': []} state: ['processing'] workers: [] NoneType: None
2026-04-21 07:31:23,608 - distributed.scheduler - INFO - Receive client connection: Client-worker-194758e7-3d54-11f1-890d-f28f846166bd
2026-04-21 07:31:22,687 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:40963
2026-04-21 07:31:22,686 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:40963', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:22,671 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:39257
2026-04-21 07:31:22,670 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:39257', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:22,632 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:35953
2026-04-21 07:31:22,631 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:35953', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:22,587 - distributed.scheduler - INFO - Receive client connection: Client-worker-18a7e5ac-3d54-11f1-8779-7e353d94dc10
2026-04-21 07:31:21,686 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:36989 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:45334 remote=tcp://172.21.160.14:36989>: Stream is closed
2026-04-21 07:31:21,638 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:43605 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:60010 remote=tcp://172.21.159.225:43605>: Stream is closed
2026-04-21 07:31:21,633 - distributed.scheduler - INFO - Close client connection: Client-worker-90f56e7b-3d4b-11f1-87a5-86acc56fd09f
2026-04-21 07:31:21,626 - distributed.scheduler - INFO - Close client connection: Client-worker-6e87771d-3d43-11f1-87b1-2e20a4008324
2026-04-21 07:31:21,624 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:32955', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:21,624 - distributed.scheduler - INFO - Remove client Client-worker-90f56e7b-3d4b-11f1-87a5-86acc56fd09f
2026-04-21 07:31:21,623 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:41311 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34418 remote=tcp://172.21.25.29:41311>: Stream is closed
2026-04-21 07:31:21,623 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:39505 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:38622 remote=tcp://172.21.159.197:39505>: Stream is closed
2026-04-21 07:31:21,623 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:41311 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:34434 remote=tcp://172.21.25.29:41311>: Stream is closed
2026-04-21 07:31:21,622 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:39505 failed: CommClosedError: in <TCP (closed) Scheduler Broadcast local=tcp://172.21.151.131:38618 remote=tcp://172.21.159.197:39505>: Stream is closed
2026-04-21 07:31:21,608 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:45817', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:21,607 - distributed.scheduler - INFO - Remove client Client-worker-6e87771d-3d43-11f1-87b1-2e20a4008324
2026-04-21 07:31:21,506 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:38971', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:10,243 - distributed.scheduler - INFO - Receive client connection: Client-worker-114eef3b-3d54-11f1-8835-4ef3162b8ac2
2026-04-21 07:31:10,072 - distributed.scheduler - INFO - Receive client connection: Client-worker-11364d22-3d54-11f1-8839-b6f02dd406e9
2026-04-21 07:31:09,013 - distributed.scheduler - INFO - Receive client connection: Client-worker-10940339-3d54-11f1-8755-0e4c34ff5e38
2026-04-21 07:31:08,288 - distributed.scheduler - INFO - Receive client connection: Client-worker-10266db7-3d54-11f1-8746-2a51d3dc178e
2026-04-21 07:31:07,929 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:41207
2026-04-21 07:31:07,928 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:41207', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:07,344 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:36617
2026-04-21 07:31:07,343 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:36617', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:07,120 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:43605
2026-04-21 07:31:07,119 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:43605', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:07,098 - distributed.scheduler - INFO - Close client connection: Client-worker-5df93e03-3d47-11f1-8735-7e353d94dc10
2026-04-21 07:31:07,095 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:39505
2026-04-21 07:31:07,095 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:39505', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:07,076 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:41255', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:07,075 - distributed.scheduler - INFO - Remove client Client-worker-5df93e03-3d47-11f1-8735-7e353d94dc10
2026-04-21 07:31:07,045 - distributed.scheduler - INFO - Receive client connection: Client-worker-0f68acbd-3d54-11f1-87f8-566b4744c386
2026-04-21 07:31:06,941 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:36563
2026-04-21 07:31:06,940 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:36563', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:06,921 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:38739
2026-04-21 07:31:06,920 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:38739', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:06,358 - distributed.scheduler - INFO - Close client connection: Client-worker-68df5d32-3d43-11f1-87c1-6ec63a5ab21e
2026-04-21 07:31:06,344 - distributed.scheduler - INFO - Remove client Client-worker-68df5d32-3d43-11f1-87c1-6ec63a5ab21e
2026-04-21 07:31:06,333 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:38033', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:06,322 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:39641
2026-04-21 07:31:06,321 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:39641', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:06,287 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:37923
2026-04-21 07:31:06,286 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:37923', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:06,240 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:42493
2026-04-21 07:31:06,239 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:42493', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:06,233 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:36989
2026-04-21 07:31:06,232 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:36989', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:06,162 - distributed.scheduler - INFO - Close client connection: Client-worker-91931f7c-3d4b-11f1-87c5-fae0fcb74df3
2026-04-21 07:31:06,153 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:38971', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:06,151 - distributed.scheduler - INFO - Remove client Client-worker-91931f7c-3d4b-11f1-87c5-fae0fcb74df3
2026-04-21 07:31:06,075 - distributed.scheduler - INFO - Close client connection: Client-worker-6932d549-3d43-11f1-88b3-f28f846166bd
2026-04-21 07:31:06,072 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:45687', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:06,071 - distributed.scheduler - INFO - Remove client Client-worker-6932d549-3d43-11f1-88b3-f28f846166bd
2026-04-21 07:31:06,070 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:35351
2026-04-21 07:31:06,070 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:35351', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:06,044 - distributed.scheduler - INFO - Close client connection: Client-worker-5df4a37a-3d47-11f1-8750-8288bfee4388
2026-04-21 07:31:06,040 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:42465', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:06,039 - distributed.scheduler - INFO - Remove client Client-worker-5df4a37a-3d47-11f1-8750-8288bfee4388
2026-04-21 07:31:06,038 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:41311
2026-04-21 07:31:06,037 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:41311', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:05,938 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:46749
2026-04-21 07:31:05,937 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:46749', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:05,871 - distributed.scheduler - INFO - Close client connection: Client-worker-f8718543-3d53-11f1-877e-ae00acbfc48e
2026-04-21 07:31:05,868 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:44819', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:05,867 - distributed.scheduler - INFO - Remove client Client-worker-f8718543-3d53-11f1-877e-ae00acbfc48e
2026-04-21 07:31:05,331 - distributed.scheduler - INFO - Close client connection: Client-worker-68b523f0-3d43-11f1-871a-0e4c34ff5e38
2026-04-21 07:31:05,330 - distributed.scheduler - INFO - Close client connection: Client-worker-68c52b46-3d43-11f1-87a3-566b4744c386
2026-04-21 07:31:05,329 - distributed.scheduler - INFO - Remove client Client-worker-68b523f0-3d43-11f1-871a-0e4c34ff5e38
2026-04-21 07:31:05,326 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:38257', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:05,318 - distributed.scheduler - INFO - Remove client Client-worker-68c52b46-3d43-11f1-87a3-566b4744c386
2026-04-21 07:31:05,308 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:46103', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:05,252 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:45887', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:05,250 - distributed.scheduler - INFO - Close client connection: Client-worker-91169375-3d4b-11f1-8704-2a51d3dc178e
2026-04-21 07:31:05,239 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:38083', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:05,234 - distributed.scheduler - INFO - Remove client Client-worker-91169375-3d4b-11f1-8704-2a51d3dc178e
2026-04-21 07:31:05,131 - distributed.scheduler - INFO - Close client connection: Client-worker-c4cf14f0-3d4f-11f1-87fd-b6f02dd406e9
2026-04-21 07:31:05,125 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:37493', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:05,122 - distributed.scheduler - INFO - Remove client Client-worker-c4cf14f0-3d4f-11f1-87fd-b6f02dd406e9
2026-04-21 07:31:05,065 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:41321', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:05,034 - distributed.scheduler - INFO - Close client connection: Client-worker-5edeadf1-3d47-11f1-87f3-4ef3162b8ac2
2026-04-21 07:31:05,021 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:35413', status: closed, memory: 0, processing: 0>
2026-04-21 07:31:05,018 - distributed.scheduler - INFO - Remove client Client-worker-5edeadf1-3d47-11f1-87f3-4ef3162b8ac2
2026-04-21 07:30:28,508 - distributed.scheduler - INFO - Receive client connection: Client-worker-f8718543-3d53-11f1-877e-ae00acbfc48e
2026-04-21 07:30:23,942 - distributed.scheduler - INFO - Receive client connection: Client-f5b6fbdb-3d53-11f1-8001-daa6852bd5a7
2026-04-21 07:20:22,414 - distributed.scheduler - INFO - Receive client connection: Client-8f2dd663-3d52-11f1-8001-daa6852bd5a7
2026-04-21 07:12:58,701 - distributed.scheduler - INFO - Receive client connection: Client-86b4c8fe-3d51-11f1-8001-daa6852bd5a7
2026-04-21 07:00:23,900 - distributed.scheduler - INFO - Receive client connection: Client-worker-c4cf14f0-3d4f-11f1-87fd-b6f02dd406e9
2026-04-21 07:00:19,440 - distributed.scheduler - INFO - Receive client connection: Client-c225f2c4-3d4f-11f1-8001-daa6852bd5a7
2026-04-21 06:50:55,278 - distributed.scheduler - INFO - Close client connection: Client-5ba2952a-3d4e-11f1-8001-daa6852bd5a7
2026-04-21 06:50:55,277 - distributed.scheduler - INFO - Remove client Client-5ba2952a-3d4e-11f1-8001-daa6852bd5a7
2026-04-21 06:50:55,276 - distributed.scheduler - INFO - Remove client Client-5ba2952a-3d4e-11f1-8001-daa6852bd5a7
2026-04-21 06:50:17,960 - distributed.scheduler - INFO - Receive client connection: Client-5ba2952a-3d4e-11f1-8001-daa6852bd5a7
2026-04-21 06:40:55,070 - distributed.scheduler - INFO - Close client connection: Client-f5258b67-3d4c-11f1-8001-daa6852bd5a7
2026-04-21 06:40:55,069 - distributed.scheduler - INFO - Remove client Client-f5258b67-3d4c-11f1-8001-daa6852bd5a7
2026-04-21 06:40:55,067 - distributed.scheduler - INFO - Remove client Client-f5258b67-3d4c-11f1-8001-daa6852bd5a7
2026-04-21 06:40:16,526 - distributed.scheduler - INFO - Receive client connection: Client-f5258b67-3d4c-11f1-8001-daa6852bd5a7
2026-04-21 06:31:23,685 - distributed.scheduler - INFO - Close client connection: Client-8ea3a0df-3d4b-11f1-8001-daa6852bd5a7
2026-04-21 06:31:23,684 - distributed.scheduler - INFO - Remove client Client-8ea3a0df-3d4b-11f1-8001-daa6852bd5a7
2026-04-21 06:31:23,682 - distributed.scheduler - INFO - Remove client Client-8ea3a0df-3d4b-11f1-8001-daa6852bd5a7
2026-04-21 06:30:19,968 - distributed.scheduler - INFO - Receive client connection: Client-worker-91931f7c-3d4b-11f1-87c5-fae0fcb74df3
2026-04-21 06:30:19,138 - distributed.scheduler - INFO - Receive client connection: Client-worker-91169375-3d4b-11f1-8704-2a51d3dc178e
2026-04-21 06:30:18,949 - distributed.scheduler - INFO - Receive client connection: Client-worker-90f56e7b-3d4b-11f1-87a5-86acc56fd09f
2026-04-21 06:30:15,041 - distributed.scheduler - INFO - Receive client connection: Client-8ea3a0df-3d4b-11f1-8001-daa6852bd5a7
2026-04-21 06:20:54,809 - distributed.scheduler - INFO - Close client connection: Client-281d8b97-3d4a-11f1-8001-daa6852bd5a7
2026-04-21 06:20:54,807 - distributed.scheduler - INFO - Remove client Client-281d8b97-3d4a-11f1-8001-daa6852bd5a7
2026-04-21 06:20:54,805 - distributed.scheduler - INFO - Remove client Client-281d8b97-3d4a-11f1-8001-daa6852bd5a7
2026-04-21 06:20:13,542 - distributed.scheduler - INFO - Receive client connection: Client-281d8b97-3d4a-11f1-8001-daa6852bd5a7
2026-04-21 06:13:13,404 - distributed.scheduler - INFO - Close client connection: Client-1867a3b6-3d49-11f1-8001-daa6852bd5a7
2026-04-21 06:13:13,403 - distributed.scheduler - INFO - Remove client Client-1867a3b6-3d49-11f1-8001-daa6852bd5a7
2026-04-21 06:13:13,401 - distributed.scheduler - INFO - Remove client Client-1867a3b6-3d49-11f1-8001-daa6852bd5a7
2026-04-21 06:12:37,687 - distributed.scheduler - INFO - Receive client connection: Client-1867a3b6-3d49-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,766 - distributed.scheduler - INFO - Close client connection: Client-8da72b15-3d3d-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,766 - distributed.scheduler - INFO - Close client connection: Client-4b091b09-3d3c-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,766 - distributed.scheduler - INFO - Close client connection: Client-c13bb579-3d41-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,766 - distributed.scheduler - INFO - Close client connection: Client-f42e5127-3d3e-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,765 - distributed.scheduler - INFO - Close client connection: Client-c6f19d56-3d40-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,765 - distributed.scheduler - INFO - Close client connection: Client-27be7142-3d43-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,765 - distributed.scheduler - INFO - Close client connection: Client-8e3a4b99-3d44-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,764 - distributed.scheduler - INFO - Close client connection: Client-f4ab624a-3d45-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,762 - distributed.scheduler - INFO - Remove client Client-8da72b15-3d3d-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,761 - distributed.scheduler - INFO - Remove client Client-4b091b09-3d3c-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,761 - distributed.scheduler - INFO - Remove client Client-c13bb579-3d41-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,760 - distributed.scheduler - INFO - Remove client Client-f42e5127-3d3e-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,759 - distributed.scheduler - INFO - Remove client Client-c6f19d56-3d40-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,758 - distributed.scheduler - INFO - Remove client Client-27be7142-3d43-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,758 - distributed.scheduler - INFO - Remove client Client-8e3a4b99-3d44-11f1-8001-daa6852bd5a7
2026-04-21 06:01:23,756 - distributed.scheduler - INFO - Remove client Client-f4ab624a-3d45-11f1-8001-daa6852bd5a7
2026-04-21 06:01:22,970 - distributed.scheduler - INFO - Close client connection: Client-5b1d4173-3d47-11f1-8001-daa6852bd5a7
2026-04-21 06:01:22,968 - distributed.scheduler - INFO - Remove client Client-5b1d4173-3d47-11f1-8001-daa6852bd5a7
2026-04-21 06:01:22,966 - distributed.scheduler - INFO - Remove client Client-5b1d4173-3d47-11f1-8001-daa6852bd5a7
2026-04-21 06:00:16,907 - distributed.scheduler - INFO - Receive client connection: Client-worker-5edeadf1-3d47-11f1-87f3-4ef3162b8ac2
2026-04-21 06:00:15,393 - distributed.scheduler - INFO - Receive client connection: Client-worker-5df93e03-3d47-11f1-8735-7e353d94dc10
2026-04-21 06:00:15,368 - distributed.scheduler - INFO - Receive client connection: Client-worker-5df4a37a-3d47-11f1-8750-8288bfee4388
2026-04-21 06:00:10,602 - distributed.scheduler - INFO - Receive client connection: Client-5b1d4173-3d47-11f1-8001-daa6852bd5a7
2026-04-21 05:50:09,232 - distributed.scheduler - INFO - Receive client connection: Client-f4ab624a-3d45-11f1-8001-daa6852bd5a7
2026-04-21 05:40:07,864 - distributed.scheduler - INFO - Receive client connection: Client-8e3a4b99-3d44-11f1-8001-daa6852bd5a7
2026-04-21 05:32:06,598 - distributed.scheduler - INFO - Receive client connection: Client-6f4b96cc-3d43-11f1-8007-924c5ad31268
2026-04-21 05:32:05,207 - distributed.scheduler - INFO - Receive client connection: Client-worker-6e87771d-3d43-11f1-87b1-2e20a4008324
2026-04-21 05:32:04,804 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:32955
2026-04-21 05:32:04,803 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:32955', status: closed, memory: 0, processing: 0>
2026-04-21 05:32:04,625 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:41321
2026-04-21 05:32:04,623 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:41321', status: closed, memory: 0, processing: 0>
2026-04-21 05:32:04,036 - distributed.scheduler - INFO - Close client connection: Client-8f8e885b-3d2e-11f1-8007-924c5ad31268
2026-04-21 05:32:04,035 - distributed.scheduler - INFO - Remove client Client-8f8e885b-3d2e-11f1-8007-924c5ad31268
2026-04-21 05:32:04,033 - distributed.scheduler - INFO - Remove client Client-8f8e885b-3d2e-11f1-8007-924c5ad31268
2026-04-21 05:32:03,811 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:44819 failed: Exception: RuntimeError('Worker tcp://172.21.25.29:43731 left during active shuffle 1fee22425ad67e23cfadefcdf29a5b52')
2026-04-21 05:32:03,703 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:37849', status: closed, memory: 0, processing: 0>
2026-04-21 05:32:03,514 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:43731', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:57,051 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:41255
2026-04-21 05:31:57,050 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:41255', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:56,258 - distributed.scheduler - INFO - Receive client connection: Client-worker-6932d549-3d43-11f1-88b3-f28f846166bd
2026-04-21 05:31:56,069 - distributed.scheduler - INFO - Task run-b1e3deb1a63553bc3d23ba43b326c59c marked as failed because 3 workers died while trying to run it
2026-04-21 05:31:56,068 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:38637', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:55,688 - distributed.scheduler - INFO - Receive client connection: Client-worker-68df5d32-3d43-11f1-87c1-6ec63a5ab21e
2026-04-21 05:31:55,526 - distributed.scheduler - INFO - Receive client connection: Client-worker-68c52b46-3d43-11f1-87a3-566b4744c386
2026-04-21 05:31:55,415 - distributed.scheduler - INFO - Receive client connection: Client-worker-68b523f0-3d43-11f1-871a-0e4c34ff5e38
2026-04-21 05:31:55,173 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:38083
2026-04-21 05:31:55,172 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:38083', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:55,114 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:44819
2026-04-21 05:31:55,113 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:44819', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,998 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:38971
2026-04-21 05:31:54,980 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:38971', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,966 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:37493
2026-04-21 05:31:54,965 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:37493', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,919 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:38971
2026-04-21 05:31:54,917 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:38971', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,903 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:42465
2026-04-21 05:31:54,902 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:42465', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,874 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:45817
2026-04-21 05:31:54,873 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:45817', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,839 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:45687
2026-04-21 05:31:54,838 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:45687', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,825 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:35413
2026-04-21 05:31:54,823 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:35413', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,821 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:38257
2026-04-21 05:31:54,820 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:38257', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,797 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:38033
2026-04-21 05:31:54,796 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:38033', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,783 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:45887
2026-04-21 05:31:54,782 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:45887', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,711 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:46103
2026-04-21 05:31:54,709 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:46103', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,089 - distributed.scheduler - INFO - Close client connection: Client-worker-50b75196-3d43-11f1-86bf-2a51d3dc178e
2026-04-21 05:31:54,088 - distributed.scheduler - INFO - Close client connection: Client-worker-497cfe85-3d43-11f1-873d-ae00acbfc48e
2026-04-21 05:31:54,087 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:35007', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,069 - distributed.scheduler - INFO - Remove client Client-worker-50b75196-3d43-11f1-86bf-2a51d3dc178e
2026-04-21 05:31:54,058 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:33917', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:54,056 - distributed.scheduler - INFO - Remove client Client-worker-497cfe85-3d43-11f1-873d-ae00acbfc48e
2026-04-21 05:31:54,011 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:36047', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:53,980 - distributed.scheduler - INFO - Close client connection: Client-worker-49725f5d-3d43-11f1-870c-8288bfee4388
2026-04-21 05:31:53,979 - distributed.scheduler - INFO - Close client connection: Client-worker-5296311a-3d43-11f1-8876-f28f846166bd
2026-04-21 05:31:53,975 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:38487', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:53,975 - distributed.scheduler - INFO - Remove client Client-worker-49725f5d-3d43-11f1-870c-8288bfee4388
2026-04-21 05:31:53,972 - distributed.scheduler - INFO - Remove client Client-worker-5296311a-3d43-11f1-8876-f28f846166bd
2026-04-21 05:31:53,969 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:39537', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:53,963 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:42043', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:53,944 - distributed.scheduler - INFO - Close client connection: Client-worker-50258934-3d43-11f1-8753-ee7435aaba1f
2026-04-21 05:31:53,940 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:34525', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:53,940 - distributed.scheduler - INFO - Remove client Client-worker-50258934-3d43-11f1-8753-ee7435aaba1f
2026-04-21 05:31:53,910 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:34033', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:53,874 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:34527', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:53,788 - distributed.scheduler - INFO - Close client connection: Client-worker-488d93a7-3d43-11f1-8768-566b4744c386
2026-04-21 05:31:53,783 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:45241', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:53,778 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:41501', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:53,777 - distributed.scheduler - INFO - Remove client Client-worker-488d93a7-3d43-11f1-8768-566b4744c386
2026-04-21 05:31:53,771 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:35843', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:53,679 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:33643', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:18,320 - distributed.scheduler - INFO - Receive client connection: Client-worker-5296311a-3d43-11f1-8876-f28f846166bd
2026-04-21 05:31:15,254 - distributed.scheduler - INFO - Receive client connection: Client-worker-50b75196-3d43-11f1-86bf-2a51d3dc178e
2026-04-21 05:31:14,720 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:37849
2026-04-21 05:31:14,719 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:37849', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:14,640 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:43731
2026-04-21 05:31:14,639 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:43731', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:14,331 - distributed.scheduler - INFO - Receive client connection: Client-worker-50258934-3d43-11f1-8753-ee7435aaba1f
2026-04-21 05:31:13,731 - distributed.scheduler - INFO - Close client connection: Client-worker-2a43c771-3d43-11f1-872c-86acc56fd09f
2026-04-21 05:31:13,722 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:44207', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:13,720 - distributed.scheduler - INFO - Remove client Client-worker-2a43c771-3d43-11f1-872c-86acc56fd09f
2026-04-21 05:31:13,625 - distributed.scheduler - INFO - Close client connection: Client-worker-2a1f9c38-3d43-11f1-869f-c2349ba99f21
2026-04-21 05:31:13,621 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:36279', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:13,619 - distributed.scheduler - INFO - Remove client Client-worker-2a1f9c38-3d43-11f1-869f-c2349ba99f21
2026-04-21 05:31:03,074 - distributed.scheduler - INFO - Receive client connection: Client-worker-497cfe85-3d43-11f1-873d-ae00acbfc48e
2026-04-21 05:31:02,979 - distributed.scheduler - INFO - Receive client connection: Client-worker-49725f5d-3d43-11f1-870c-8288bfee4388
2026-04-21 05:31:01,809 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:38637
2026-04-21 05:31:01,808 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:38637', status: closed, memory: 0, processing: 0>
2026-04-21 05:31:01,483 - distributed.scheduler - INFO - Receive client connection: Client-worker-488d93a7-3d43-11f1-8768-566b4744c386
2026-04-21 05:31:00,854 - distributed.scheduler - INFO - Close client connection: Client-worker-cc59820b-3d36-11f1-869e-7e353d94dc10
2026-04-21 05:31:00,845 - distributed.scheduler - INFO - Remove client Client-worker-cc59820b-3d36-11f1-869e-7e353d94dc10
2026-04-21 05:31:00,837 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:45367', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,905 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:33917
2026-04-21 05:30:59,899 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:33917', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,844 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:38487
2026-04-21 05:30:59,843 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:38487', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,623 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:34525
2026-04-21 05:30:59,622 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:34525', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,554 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:36047
2026-04-21 05:30:59,553 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:36047', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,511 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:45241
2026-04-21 05:30:59,510 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:45241', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,495 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:35007
2026-04-21 05:30:59,494 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:35007', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,489 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:34527
2026-04-21 05:30:59,488 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:34527', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,482 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:33643
2026-04-21 05:30:59,481 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:33643', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,465 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:42043
2026-04-21 05:30:59,464 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:42043', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,462 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:39537
2026-04-21 05:30:59,461 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:39537', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,309 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:34033
2026-04-21 05:30:59,307 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:34033', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,256 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:35843
2026-04-21 05:30:59,255 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:35843', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:59,164 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:41501
2026-04-21 05:30:59,162 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:41501', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,878 - distributed.scheduler - INFO - Close client connection: Client-worker-e7b5b1f0-3d3a-11f1-86fc-ae00acbfc48e
2026-04-21 05:30:58,872 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:33257', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,872 - distributed.scheduler - INFO - Remove client Client-worker-e7b5b1f0-3d3a-11f1-86fc-ae00acbfc48e
2026-04-21 05:30:58,869 - distributed.scheduler - INFO - Close client connection: Client-worker-f7a82e29-3d3e-11f1-86ca-8288bfee4388
2026-04-21 05:30:58,852 - distributed.scheduler - INFO - Task ('sjoin-74631e305fd13ebc11330a3a99e262ce', 215) marked as failed because 3 workers died while trying to run it
2026-04-21 05:30:58,851 - distributed.scheduler - INFO - Task ('sjoin-74631e305fd13ebc11330a3a99e262ce', 219) marked as failed because 3 workers died while trying to run it
2026-04-21 05:30:58,850 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:44763', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,846 - distributed.scheduler - INFO - Remove client Client-worker-f7a82e29-3d3e-11f1-86ca-8288bfee4388
2026-04-21 05:30:58,593 - distributed.scheduler - INFO - Close client connection: Client-worker-2a3d8a40-3d43-11f1-8785-b6f02dd406e9
2026-04-21 05:30:58,591 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:37903', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,576 - distributed.scheduler - INFO - Remove client Client-worker-2a3d8a40-3d43-11f1-8785-b6f02dd406e9
2026-04-21 05:30:58,566 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:36381', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,564 - distributed.scheduler - INFO - Close client connection: Client-worker-e84765a9-3d3a-11f1-874b-fae0fcb74df3
2026-04-21 05:30:58,564 - distributed.scheduler - INFO - Remove client Client-worker-e84765a9-3d3a-11f1-874b-fae0fcb74df3
2026-04-21 05:30:58,561 - distributed.scheduler - INFO - Close client connection: Client-worker-e7e63c2f-3d3a-11f1-86e9-2eb33e86e1bd
2026-04-21 05:30:58,557 - distributed.scheduler - INFO - Close client connection: Client-worker-e83e1e91-3d3a-11f1-86a8-0e4c34ff5e38
2026-04-21 05:30:58,554 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:33617', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,553 - distributed.scheduler - INFO - Remove client Client-worker-e7e63c2f-3d3a-11f1-86e9-2eb33e86e1bd
2026-04-21 05:30:58,552 - distributed.scheduler - INFO - Remove client Client-worker-e83e1e91-3d3a-11f1-86a8-0e4c34ff5e38
2026-04-21 05:30:58,547 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:41357', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,545 - distributed.scheduler - INFO - Close client connection: Client-worker-f73eacfd-3d3e-11f1-867d-2a51d3dc178e
2026-04-21 05:30:58,543 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:46347', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,541 - distributed.scheduler - INFO - Remove client Client-worker-f73eacfd-3d3e-11f1-867d-2a51d3dc178e
2026-04-21 05:30:58,487 - distributed.scheduler - INFO - Close client connection: Client-worker-e7b9f297-3d3a-11f1-8835-f28f846166bd
2026-04-21 05:30:58,484 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:45601', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,483 - distributed.scheduler - INFO - Remove client Client-worker-e7b9f297-3d3a-11f1-8835-f28f846166bd
2026-04-21 05:30:58,453 - distributed.scheduler - INFO - Close client connection: Client-worker-f74d5da7-3d3e-11f1-8738-2e20a4008324
2026-04-21 05:30:58,449 - distributed.scheduler - INFO - Remove client Client-worker-f74d5da7-3d3e-11f1-8738-2e20a4008324
2026-04-21 05:30:58,445 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:32819', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,444 - distributed.scheduler - INFO - Close client connection: Client-worker-ce2e822c-3d36-11f1-8713-ee7435aaba1f
2026-04-21 05:30:58,426 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:40449', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,425 - distributed.scheduler - INFO - Remove client Client-worker-ce2e822c-3d36-11f1-8713-ee7435aaba1f
2026-04-21 05:30:58,374 - distributed.scheduler - INFO - Close client connection: Client-worker-cee106f8-3d36-11f1-8774-4ef3162b8ac2
2026-04-21 05:30:58,366 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:33721', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,334 - distributed.scheduler - INFO - Remove client Client-worker-cee106f8-3d36-11f1-8774-4ef3162b8ac2
2026-04-21 05:30:58,228 - distributed.scheduler - INFO - Close client connection: Client-worker-e7994af4-3d3a-11f1-8729-566b4744c386
2026-04-21 05:30:58,221 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:36399', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,218 - distributed.scheduler - INFO - Remove client Client-worker-e7994af4-3d3a-11f1-8729-566b4744c386
2026-04-21 05:30:58,202 - distributed.scheduler - INFO - Close client connection: Client-worker-cebb3a80-3d36-11f1-8747-6ec63a5ab21e
2026-04-21 05:30:58,194 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:34495', status: closed, memory: 0, processing: 0>
2026-04-21 05:30:58,193 - distributed.scheduler - INFO - Remove client Client-worker-cebb3a80-3d36-11f1-8747-6ec63a5ab21e
2026-04-21 05:30:10,656 - distributed.scheduler - INFO - Receive client connection: Client-worker-2a43c771-3d43-11f1-872c-86acc56fd09f
2026-04-21 05:30:10,626 - distributed.scheduler - INFO - Receive client connection: Client-worker-2a3d8a40-3d43-11f1-8785-b6f02dd406e9
2026-04-21 05:30:10,417 - distributed.scheduler - INFO - Receive client connection: Client-worker-2a1f9c38-3d43-11f1-869f-c2349ba99f21
2026-04-21 05:30:06,430 - distributed.scheduler - INFO - Receive client connection: Client-27be7142-3d43-11f1-8001-daa6852bd5a7
2026-04-21 05:20:04,949 - distributed.scheduler - INFO - Receive client connection: Client-c13bb579-3d41-11f1-8001-daa6852bd5a7
2026-04-21 05:13:05,035 - distributed.scheduler - INFO - Receive client connection: Client-c6f19d56-3d40-11f1-8001-daa6852bd5a7
2026-04-21 05:00:08,213 - distributed.scheduler - INFO - Receive client connection: Client-worker-f7a82e29-3d3e-11f1-86ca-8288bfee4388
2026-04-21 05:00:07,173 - distributed.scheduler - INFO - Receive client connection: Client-worker-f74d5da7-3d3e-11f1-8738-2e20a4008324
2026-04-21 05:00:07,069 - distributed.scheduler - INFO - Receive client connection: Client-worker-f73eacfd-3d3e-11f1-867d-2a51d3dc178e
2026-04-21 05:00:01,938 - distributed.scheduler - INFO - Receive client connection: Client-f42e5127-3d3e-11f1-8001-daa6852bd5a7
2026-04-21 04:50:00,427 - distributed.scheduler - INFO - Receive client connection: Client-8da72b15-3d3d-11f1-8001-daa6852bd5a7
2026-04-21 04:40:59,171 - distributed.scheduler - INFO - Receive client connection: Client-4b091b09-3d3c-11f1-8001-daa6852bd5a7
2026-04-21 04:32:00,939 - distributed.scheduler - INFO - Close client connection: Client-e48cb96f-3d3a-11f1-8001-daa6852bd5a7
2026-04-21 04:32:00,937 - distributed.scheduler - INFO - Remove client Client-e48cb96f-3d3a-11f1-8001-daa6852bd5a7
2026-04-21 04:32:00,936 - distributed.scheduler - INFO - Remove client Client-e48cb96f-3d3a-11f1-8001-daa6852bd5a7
2026-04-21 04:31:03,989 - distributed.scheduler - INFO - Receive client connection: Client-worker-e84765a9-3d3a-11f1-874b-fae0fcb74df3
2026-04-21 04:31:03,939 - distributed.scheduler - INFO - Receive client connection: Client-worker-e83e1e91-3d3a-11f1-86a8-0e4c34ff5e38
2026-04-21 04:31:03,355 - distributed.scheduler - INFO - Receive client connection: Client-worker-e7e63c2f-3d3a-11f1-86e9-2eb33e86e1bd
2026-04-21 04:31:03,045 - distributed.scheduler - INFO - Receive client connection: Client-worker-e7b9f297-3d3a-11f1-8835-f28f846166bd
2026-04-21 04:31:03,034 - distributed.scheduler - INFO - Receive client connection: Client-worker-e7b5b1f0-3d3a-11f1-86fc-ae00acbfc48e
2026-04-21 04:31:02,832 - distributed.scheduler - INFO - Receive client connection: Client-worker-e7994af4-3d3a-11f1-8729-566b4744c386
2026-04-21 04:30:57,737 - distributed.scheduler - INFO - Receive client connection: Client-e48cb96f-3d3a-11f1-8001-daa6852bd5a7
2026-04-21 04:21:29,211 - distributed.scheduler - INFO - Close client connection: Client-7e06b876-3d39-11f1-8001-daa6852bd5a7
2026-04-21 04:21:29,210 - distributed.scheduler - INFO - Remove client Client-7e06b876-3d39-11f1-8001-daa6852bd5a7
2026-04-21 04:21:29,209 - distributed.scheduler - INFO - Remove client Client-7e06b876-3d39-11f1-8001-daa6852bd5a7
2026-04-21 04:20:56,252 - distributed.scheduler - INFO - Receive client connection: Client-7e06b876-3d39-11f1-8001-daa6852bd5a7
2026-04-21 04:13:31,529 - distributed.scheduler - INFO - Close client connection: Client-63f087b8-3d38-11f1-8001-daa6852bd5a7
2026-04-21 04:13:31,527 - distributed.scheduler - INFO - Remove client Client-63f087b8-3d38-11f1-8001-daa6852bd5a7
2026-04-21 04:13:31,526 - distributed.scheduler - INFO - Remove client Client-63f087b8-3d38-11f1-8001-daa6852bd5a7
2026-04-21 04:13:02,962 - distributed.scheduler - INFO - Receive client connection: Client-63f087b8-3d38-11f1-8001-daa6852bd5a7
2026-04-21 04:02:32,825 - distributed.scheduler - INFO - Close client connection: Client-b0fc534f-3d36-11f1-8001-daa6852bd5a7
2026-04-21 04:02:32,824 - distributed.scheduler - INFO - Remove client Client-b0fc534f-3d36-11f1-8001-daa6852bd5a7
2026-04-21 04:02:32,823 - distributed.scheduler - INFO - Remove client Client-b0fc534f-3d36-11f1-8001-daa6852bd5a7
2026-04-21 04:02:14,521 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-7d4ee0159f7e227215941086e9ab3bd0 NoneType: None
2026-04-21 04:02:14,521 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-fc1783f23922cc51b048773399525f31 NoneType: None
2026-04-21 04:02:14,520 - distributed.scheduler - ERROR - Couldn't gather keys {'run-fc1783f23922cc51b048773399525f31': [], 'run-7d4ee0159f7e227215941086e9ab3bd0': []} state: ['processing', 'processing'] workers: [] NoneType: None
2026-04-21 04:01:43,717 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:44207
2026-04-21 04:01:43,715 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:44207', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:43,412 - distributed.scheduler - INFO - Receive client connection: Client-worker-cee106f8-3d36-11f1-8774-4ef3162b8ac2
2026-04-21 04:01:43,272 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:36279
2026-04-21 04:01:43,271 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:36279', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:43,133 - distributed.scheduler - INFO - Receive client connection: Client-worker-cebb3a80-3d36-11f1-8747-6ec63a5ab21e
2026-04-21 04:01:42,994 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:32819
2026-04-21 04:01:42,993 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:32819', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:42,640 - distributed.scheduler - INFO - Close client connection: Client-worker-b3754b98-3d36-11f1-86f0-86acc56fd09f
2026-04-21 04:01:42,621 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:44427', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:42,619 - distributed.scheduler - INFO - Remove client Client-worker-b3754b98-3d36-11f1-86f0-86acc56fd09f
2026-04-21 04:01:42,206 - distributed.scheduler - INFO - Receive client connection: Client-worker-ce2e822c-3d36-11f1-8713-ee7435aaba1f
2026-04-21 04:01:42,120 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:36089', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:41,844 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:44869', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:40,127 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:34495
2026-04-21 04:01:40,125 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:34495', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:40,084 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:36381
2026-04-21 04:01:40,082 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:36381', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,954 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:36399
2026-04-21 04:01:39,953 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:36399', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,936 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:40449
2026-04-21 04:01:39,936 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:40449', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,872 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:33721
2026-04-21 04:01:39,870 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:33721', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,849 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:45601
2026-04-21 04:01:39,848 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:45601', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,808 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:41357
2026-04-21 04:01:39,807 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:41357', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,758 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:46347
2026-04-21 04:01:39,757 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:46347', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,747 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:37903
2026-04-21 04:01:39,746 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:37903', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,742 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:33257
2026-04-21 04:01:39,741 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:33257', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,624 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:33617
2026-04-21 04:01:39,622 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:33617', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,519 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:44763
2026-04-21 04:01:39,518 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:44763', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,255 - distributed.scheduler - INFO - Close client connection: Client-worker-b3f09e67-3d36-11f1-870c-fae0fcb74df3
2026-04-21 04:01:39,254 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:37433', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,253 - distributed.scheduler - INFO - Remove client Client-worker-b3f09e67-3d36-11f1-870c-fae0fcb74df3
2026-04-21 04:01:39,124 - distributed.scheduler - INFO - Receive client connection: Client-worker-cc59820b-3d36-11f1-869e-7e353d94dc10
2026-04-21 04:01:39,087 - distributed.scheduler - INFO - Close client connection: Client-worker-c3fc46bb-3d32-11f1-8707-6ec63a5ab21e
2026-04-21 04:01:39,083 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:38015', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:39,059 - distributed.scheduler - INFO - Remove client Client-worker-c3fc46bb-3d32-11f1-8707-6ec63a5ab21e
2026-04-21 04:01:38,961 - distributed.scheduler - INFO - Close client connection: Client-worker-a7da78b1-3d32-11f1-8660-0e4c34ff5e38
2026-04-21 04:01:38,961 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:34581', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:38,952 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:38803', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:38,951 - distributed.scheduler - INFO - Remove client Client-worker-a7da78b1-3d32-11f1-8660-0e4c34ff5e38
2026-04-21 04:01:38,838 - distributed.scheduler - INFO - Close client connection: Client-worker-b442c567-3d36-11f1-86ba-ae00acbfc48e
2026-04-21 04:01:38,835 - distributed.scheduler - INFO - Remove client Client-worker-b442c567-3d36-11f1-86ba-ae00acbfc48e
2026-04-21 04:01:38,832 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:44767', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:38,830 - distributed.scheduler - INFO - Close client connection: Client-worker-ab9faf7d-3d32-11f1-86e6-566b4744c386
2026-04-21 04:01:38,821 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:36119', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:38,820 - distributed.scheduler - INFO - Remove client Client-worker-ab9faf7d-3d32-11f1-86e6-566b4744c386
2026-04-21 04:01:38,767 - distributed.scheduler - INFO - Close client connection: Client-worker-b413f134-3d36-11f1-8747-b6f02dd406e9
2026-04-21 04:01:38,750 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:42787', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:38,746 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:44941', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:38,745 - distributed.scheduler - INFO - Remove client Client-worker-b413f134-3d36-11f1-8747-b6f02dd406e9
2026-04-21 04:01:38,714 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:44009', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:38,674 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:42987', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:38,639 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:41393', status: closed, memory: 0, processing: 0>
2026-04-21 04:01:38,484 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:33489', status: closed, memory: 0, processing: 0>
2026-04-21 04:00:58,757 - distributed.scheduler - INFO - Receive client connection: Client-worker-b442c567-3d36-11f1-86ba-ae00acbfc48e
2026-04-21 04:00:58,712 - distributed.scheduler - INFO - Receive client connection: Client-worker-b413f134-3d36-11f1-8747-b6f02dd406e9
2026-04-21 04:00:58,202 - distributed.scheduler - INFO - Receive client connection: Client-worker-b3f09e67-3d36-11f1-870c-fae0fcb74df3
2026-04-21 04:00:57,367 - distributed.scheduler - INFO - Receive client connection: Client-worker-b3754b98-3d36-11f1-86f0-86acc56fd09f
2026-04-21 04:00:53,236 - distributed.scheduler - INFO - Receive client connection: Client-b0fc534f-3d36-11f1-8001-daa6852bd5a7
2026-04-21 03:51:25,085 - distributed.scheduler - INFO - Close client connection: Client-4a88158b-3d35-11f1-8001-daa6852bd5a7
2026-04-21 03:51:25,085 - distributed.scheduler - INFO - Remove client Client-4a88158b-3d35-11f1-8001-daa6852bd5a7
2026-04-21 03:51:25,084 - distributed.scheduler - INFO - Remove client Client-4a88158b-3d35-11f1-8001-daa6852bd5a7
2026-04-21 03:50:51,900 - distributed.scheduler - INFO - Receive client connection: Client-4a88158b-3d35-11f1-8001-daa6852bd5a7
2026-04-21 03:41:23,930 - distributed.scheduler - INFO - Close client connection: Client-e4007cee-3d33-11f1-8001-daa6852bd5a7
2026-04-21 03:41:23,927 - distributed.scheduler - INFO - Remove client Client-e4007cee-3d33-11f1-8001-daa6852bd5a7
2026-04-21 03:41:23,925 - distributed.scheduler - INFO - Remove client Client-e4007cee-3d33-11f1-8001-daa6852bd5a7
2026-04-21 03:40:50,339 - distributed.scheduler - INFO - Receive client connection: Client-e4007cee-3d33-11f1-8001-daa6852bd5a7
2026-04-21 03:32:47,115 - distributed.scheduler - INFO - Receive client connection: Client-worker-c3fc46bb-3d32-11f1-8707-6ec63a5ab21e
2026-04-21 03:32:42,897 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:42787 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.222:35761 after 30 s')
2026-04-21 03:32:42,888 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:35761 failed: OSError: Timed out trying to connect to tcp://172.21.159.222:35761 after 30 s
2026-04-21 03:32:42,811 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:36119 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.222:35761 after 30 s')
2026-04-21 03:32:42,753 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:38015 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.222:35761 after 30 s')
2026-04-21 03:32:13,759 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:41393
2026-04-21 03:32:13,757 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:41393', status: closed, memory: 0, processing: 0>
2026-04-21 03:32:12,664 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:35761', status: closed, memory: 0, processing: 0>
2026-04-21 03:32:06,245 - distributed.scheduler - INFO - Receive client connection: Client-worker-ab9faf7d-3d32-11f1-86e6-566b4744c386
2026-04-21 03:31:59,933 - distributed.scheduler - INFO - Receive client connection: Client-worker-a7da78b1-3d32-11f1-8660-0e4c34ff5e38
2026-04-21 03:31:59,846 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:44941
2026-04-21 03:31:59,844 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:44941', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:59,006 - distributed.scheduler - INFO - Close client connection: Client-worker-8ef47e11-3d2e-11f1-870c-b6f02dd406e9
2026-04-21 03:31:58,999 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:35181', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:58,998 - distributed.scheduler - INFO - Remove client Client-worker-8ef47e11-3d2e-11f1-870c-b6f02dd406e9
2026-04-21 03:31:58,270 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:36089
2026-04-21 03:31:58,268 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:36089', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:58,140 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:34581
2026-04-21 03:31:58,138 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:34581', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,915 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:38803
2026-04-21 03:31:57,914 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:38803', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,908 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:42987
2026-04-21 03:31:57,907 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:42987', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,894 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:45367
2026-04-21 03:31:57,894 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:45367', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,879 - distributed.scheduler - INFO - Close client connection: Client-cfeca627-3cf7-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,879 - distributed.scheduler - INFO - Close client connection: Client-367a0514-3cf9-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,879 - distributed.scheduler - INFO - Close client connection: Client-9cf92af9-3cfa-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,879 - distributed.scheduler - INFO - Close client connection: Client-038413ad-3cfc-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,879 - distributed.scheduler - INFO - Close client connection: Client-d089691e-3cfe-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,879 - distributed.scheduler - INFO - Close client connection: Client-da6fcb45-3cfd-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,878 - distributed.scheduler - INFO - Close client connection: Client-3717867d-3d00-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,878 - distributed.scheduler - INFO - Close client connection: Client-9da43628-3d01-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,878 - distributed.scheduler - INFO - Close client connection: Client-041edccd-3d03-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,878 - distributed.scheduler - INFO - Close client connection: Client-46c3d138-3d04-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,878 - distributed.scheduler - INFO - Close client connection: Client-13d1b6ef-3d07-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,878 - distributed.scheduler - INFO - Close client connection: Client-f838d112-3d05-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,878 - distributed.scheduler - INFO - Close client connection: Client-7a68bc8e-3d08-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,878 - distributed.scheduler - INFO - Close client connection: Client-e0f1715a-3d09-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,878 - distributed.scheduler - INFO - Close client connection: Client-4771cea8-3d0b-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,878 - distributed.scheduler - INFO - Close client connection: Client-adea0665-3d0c-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,878 - distributed.scheduler - INFO - Close client connection: Client-7adc4d45-3d0f-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,877 - distributed.scheduler - INFO - Close client connection: Client-63d316b8-3d0e-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,877 - distributed.scheduler - INFO - Close client connection: Client-e15b21dc-3d10-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,877 - distributed.scheduler - INFO - Close client connection: Client-47d5f536-3d12-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,877 - distributed.scheduler - INFO - Close client connection: Client-ae6164a1-3d13-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,877 - distributed.scheduler - INFO - Close client connection: Client-7bf92574-3d1d-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,877 - distributed.scheduler - INFO - Close client connection: Client-e1d97cb0-3d17-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,877 - distributed.scheduler - INFO - Close client connection: Client-14dabfca-3d15-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,877 - distributed.scheduler - INFO - Close client connection: Client-c7d0ac1c-3d16-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,877 - distributed.scheduler - INFO - Close client connection: Client-486051ad-3d19-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,876 - distributed.scheduler - INFO - Close client connection: Client-aee6761b-3d1a-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,876 - distributed.scheduler - INFO - Close client connection: Client-15722f30-3d1c-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,876 - distributed.scheduler - INFO - Close client connection: Client-48ec5050-3d20-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,876 - distributed.scheduler - INFO - Close client connection: Client-3fa15130-3d1f-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,876 - distributed.scheduler - INFO - Close client connection: Client-15f52a03-3d23-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,876 - distributed.scheduler - INFO - Close client connection: Client-af6fee64-3d21-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,876 - distributed.scheduler - INFO - Close client connection: Client-7c7e69b0-3d24-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,876 - distributed.scheduler - INFO - Close client connection: Client-e2fffeb5-3d25-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,876 - distributed.scheduler - INFO - Close client connection: Client-aedff465-3d27-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,876 - distributed.scheduler - INFO - Close client connection: Client-affdeb03-3d28-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,875 - distributed.scheduler - INFO - Close client connection: Client-16772c76-3d2a-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,875 - distributed.scheduler - INFO - Close client connection: Client-7cfa2707-3d2b-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,875 - distributed.scheduler - INFO - Close client connection: Client-06e091e9-3d30-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,875 - distributed.scheduler - INFO - Close client connection: Client-17020931-3d31-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,875 - distributed.scheduler - INFO - Close client connection: Client-49f96841-3d2e-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,875 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:44767
2026-04-21 03:31:57,874 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:44767', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,874 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:33489
2026-04-21 03:31:57,873 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:33489', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,872 - distributed.scheduler - INFO - Close client connection: Client-e37b9786-3d2c-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,869 - distributed.scheduler - INFO - Remove client Client-cfeca627-3cf7-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,869 - distributed.scheduler - INFO - Remove client Client-367a0514-3cf9-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,869 - distributed.scheduler - INFO - Remove client Client-9cf92af9-3cfa-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,869 - distributed.scheduler - INFO - Remove client Client-038413ad-3cfc-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,868 - distributed.scheduler - INFO - Remove client Client-d089691e-3cfe-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,868 - distributed.scheduler - INFO - Remove client Client-da6fcb45-3cfd-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,867 - distributed.scheduler - INFO - Remove client Client-3717867d-3d00-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,867 - distributed.scheduler - INFO - Remove client Client-9da43628-3d01-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,867 - distributed.scheduler - INFO - Remove client Client-041edccd-3d03-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,866 - distributed.scheduler - INFO - Remove client Client-46c3d138-3d04-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,866 - distributed.scheduler - INFO - Remove client Client-13d1b6ef-3d07-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,866 - distributed.scheduler - INFO - Remove client Client-f838d112-3d05-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,865 - distributed.scheduler - INFO - Remove client Client-7a68bc8e-3d08-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,865 - distributed.scheduler - INFO - Remove client Client-e0f1715a-3d09-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,865 - distributed.scheduler - INFO - Remove client Client-4771cea8-3d0b-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,864 - distributed.scheduler - INFO - Remove client Client-adea0665-3d0c-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,864 - distributed.scheduler - INFO - Remove client Client-7adc4d45-3d0f-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,863 - distributed.scheduler - INFO - Remove client Client-63d316b8-3d0e-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,863 - distributed.scheduler - INFO - Remove client Client-e15b21dc-3d10-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,863 - distributed.scheduler - INFO - Remove client Client-47d5f536-3d12-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,862 - distributed.scheduler - INFO - Remove client Client-ae6164a1-3d13-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,862 - distributed.scheduler - INFO - Remove client Client-7bf92574-3d1d-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,862 - distributed.scheduler - INFO - Remove client Client-e1d97cb0-3d17-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,861 - distributed.scheduler - INFO - Remove client Client-14dabfca-3d15-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,861 - distributed.scheduler - INFO - Remove client Client-c7d0ac1c-3d16-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,861 - distributed.scheduler - INFO - Remove client Client-486051ad-3d19-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,860 - distributed.scheduler - INFO - Remove client Client-aee6761b-3d1a-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,860 - distributed.scheduler - INFO - Remove client Client-15722f30-3d1c-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,859 - distributed.scheduler - INFO - Remove client Client-48ec5050-3d20-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,859 - distributed.scheduler - INFO - Remove client Client-3fa15130-3d1f-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,859 - distributed.scheduler - INFO - Remove client Client-15f52a03-3d23-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,858 - distributed.scheduler - INFO - Remove client Client-af6fee64-3d21-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,858 - distributed.scheduler - INFO - Remove client Client-7c7e69b0-3d24-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,857 - distributed.scheduler - INFO - Remove client Client-e2fffeb5-3d25-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,857 - distributed.scheduler - INFO - Remove client Client-aedff465-3d27-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,857 - distributed.scheduler - INFO - Remove client Client-affdeb03-3d28-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,856 - distributed.scheduler - INFO - Remove client Client-16772c76-3d2a-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,855 - distributed.scheduler - INFO - Remove client Client-7cfa2707-3d2b-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,855 - distributed.scheduler - INFO - Remove client Client-06e091e9-3d30-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,854 - distributed.scheduler - INFO - Remove client Client-17020931-3d31-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,853 - distributed.scheduler - INFO - Remove client Client-49f96841-3d2e-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,842 - distributed.scheduler - INFO - Remove client Client-e37b9786-3d2c-11f1-8001-daa6852bd5a7
2026-04-21 03:31:57,787 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:44427
2026-04-21 03:31:57,785 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:44427', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,753 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:44009
2026-04-21 03:31:57,752 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:44009', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,743 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:37433
2026-04-21 03:31:57,742 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:37433', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,729 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:42787
2026-04-21 03:31:57,727 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:42787', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,697 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:35761
2026-04-21 03:31:57,695 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:35761', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,681 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:38015
2026-04-21 03:31:57,680 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:38015', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,677 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:44869
2026-04-21 03:31:57,676 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:44869', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,664 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:36119
2026-04-21 03:31:57,662 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:36119', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,259 - distributed.scheduler - INFO - Close client connection: Client-worker-806d610d-3d32-11f1-87bd-f28f846166bd
2026-04-21 03:31:57,256 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:41323', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:57,255 - distributed.scheduler - INFO - Remove client Client-worker-806d610d-3d32-11f1-87bd-f28f846166bd
2026-04-21 03:31:57,209 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:45047', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,930 - distributed.scheduler - INFO - Task ('hash-join-8f31526927863ad5f130de5e921cd6b6', 1) marked as failed because 3 workers died while trying to run it
2026-04-21 03:31:56,930 - distributed.scheduler - INFO - Task ('hash-join-8f31526927863ad5f130de5e921cd6b6', 14) marked as failed because 3 workers died while trying to run it
2026-04-21 03:31:56,929 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:45397', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,912 - distributed.scheduler - INFO - Close client connection: Client-7d803cd0-3d32-11f1-8001-daa6852bd5a7
2026-04-21 03:31:56,911 - distributed.scheduler - INFO - Remove client Client-7d803cd0-3d32-11f1-8001-daa6852bd5a7
2026-04-21 03:31:56,911 - distributed.scheduler - INFO - Remove client Client-7d803cd0-3d32-11f1-8001-daa6852bd5a7
2026-04-21 03:31:56,910 - distributed.scheduler - INFO - Close client connection: Client-worker-7a6f2e4d-3d2e-11f1-85f8-2a51d3dc178e
2026-04-21 03:31:56,910 - distributed.scheduler - INFO - Close client connection: Client-worker-7ff49d48-3d32-11f1-867d-ae00acbfc48e
2026-04-21 03:31:56,886 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:45963', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,886 - distributed.scheduler - INFO - Remove client Client-worker-7a6f2e4d-3d2e-11f1-85f8-2a51d3dc178e
2026-04-21 03:31:56,885 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:39623', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,884 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:45217', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,883 - distributed.scheduler - INFO - Remove client Client-worker-7ff49d48-3d32-11f1-867d-ae00acbfc48e
2026-04-21 03:31:56,880 - distributed.scheduler - INFO - Close client connection: Client-worker-808e4af0-3d32-11f1-865c-7e353d94dc10
2026-04-21 03:31:56,859 - distributed.scheduler - INFO - Task ('from_pandas-c095576b7f753fffed310ea98022566d', 9) marked as failed because 3 workers died while trying to run it
2026-04-21 03:31:56,858 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:35491', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,857 - distributed.scheduler - INFO - Remove client Client-worker-808e4af0-3d32-11f1-865c-7e353d94dc10
2026-04-21 03:31:56,796 - distributed.scheduler - INFO - Close client connection: Client-worker-8e7ca90b-3d2e-11f1-86c4-6ec63a5ab21e
2026-04-21 03:31:56,792 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:36421', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,791 - distributed.scheduler - INFO - Remove client Client-worker-8e7ca90b-3d2e-11f1-86c4-6ec63a5ab21e
2026-04-21 03:31:56,789 - distributed.scheduler - INFO - Close client connection: Client-worker-8ccd4686-3d2e-11f1-86ca-fae0fcb74df3
2026-04-21 03:31:56,785 - distributed.scheduler - INFO - Remove client Client-worker-8ccd4686-3d2e-11f1-86ca-fae0fcb74df3
2026-04-21 03:31:56,780 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:43727', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,730 - distributed.scheduler - INFO - Close client connection: Client-worker-8019024f-3d32-11f1-86fa-4ef3162b8ac2
2026-04-21 03:31:56,728 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:36069', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,725 - distributed.scheduler - INFO - Close client connection: Client-worker-80910dab-3d32-11f1-86b6-86acc56fd09f
2026-04-21 03:31:56,722 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:46489', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,722 - distributed.scheduler - INFO - Remove client Client-worker-8019024f-3d32-11f1-86fa-4ef3162b8ac2
2026-04-21 03:31:56,712 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:44649', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,710 - distributed.scheduler - INFO - Remove client Client-worker-80910dab-3d32-11f1-86b6-86acc56fd09f
2026-04-21 03:31:56,674 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:42875', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,639 - distributed.scheduler - INFO - Close client connection: Client-worker-801a3aa8-3d32-11f1-869d-ee7435aaba1f
2026-04-21 03:31:56,630 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:35675', status: closed, memory: 0, processing: 0>
2026-04-21 03:31:56,629 - distributed.scheduler - INFO - Remove client Client-worker-801a3aa8-3d32-11f1-869d-ee7435aaba1f
2026-04-21 03:31:56,612 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:36831', status: closed, memory: 0, processing: 0>
2026-04-21 03:30:54,023 - distributed.scheduler - INFO - Receive client connection: Client-worker-80910dab-3d32-11f1-86b6-86acc56fd09f
2026-04-21 03:30:53,985 - distributed.scheduler - INFO - Receive client connection: Client-worker-808e4af0-3d32-11f1-865c-7e353d94dc10
2026-04-21 03:30:53,772 - distributed.scheduler - INFO - Receive client connection: Client-worker-806d610d-3d32-11f1-87bd-f28f846166bd
2026-04-21 03:30:53,240 - distributed.scheduler - INFO - Receive client connection: Client-worker-801a3aa8-3d32-11f1-869d-ee7435aaba1f
2026-04-21 03:30:53,212 - distributed.scheduler - INFO - Receive client connection: Client-worker-8019024f-3d32-11f1-86fa-4ef3162b8ac2
2026-04-21 03:30:52,972 - distributed.scheduler - INFO - Receive client connection: Client-worker-7ff49d48-3d32-11f1-867d-ae00acbfc48e
2026-04-21 03:30:48,870 - distributed.scheduler - INFO - Receive client connection: Client-7d803cd0-3d32-11f1-8001-daa6852bd5a7
2026-04-21 03:20:47,413 - distributed.scheduler - INFO - Receive client connection: Client-17020931-3d31-11f1-8001-daa6852bd5a7
2026-04-21 03:13:10,849 - distributed.scheduler - INFO - Receive client connection: Client-06e091e9-3d30-11f1-8001-daa6852bd5a7
2026-04-21 03:02:41,378 - distributed.scheduler - INFO - Receive client connection: Client-8f8e885b-3d2e-11f1-8007-924c5ad31268
2026-04-21 03:02:40,381 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:35675
2026-04-21 03:02:40,380 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:35675', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:40,206 - distributed.scheduler - INFO - Receive client connection: Client-worker-8ef47e11-3d2e-11f1-870c-b6f02dd406e9
2026-04-21 03:02:40,123 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:44649
2026-04-21 03:02:40,122 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:44649', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:39,469 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:45047
2026-04-21 03:02:39,468 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:45047', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:39,373 - distributed.scheduler - INFO - Receive client connection: Client-worker-8e7ca90b-3d2e-11f1-86c4-6ec63a5ab21e
2026-04-21 03:02:39,203 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:35983', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:39,004 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:43525', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:38,351 - distributed.scheduler - INFO - Close client connection: Client-e42ccb9e-3cde-11f1-8007-924c5ad31268
2026-04-21 03:02:38,347 - distributed.scheduler - INFO - Remove client Client-e42ccb9e-3cde-11f1-8007-924c5ad31268
2026-04-21 03:02:38,344 - distributed.scheduler - INFO - Remove client Client-e42ccb9e-3cde-11f1-8007-924c5ad31268
2026-04-21 03:02:38,342 - distributed.scheduler - INFO - Close client connection: Client-worker-79950c55-3d2e-11f1-85f3-c2349ba99f21
2026-04-21 03:02:38,318 - distributed.scheduler - INFO - Remove client Client-worker-79950c55-3d2e-11f1-85f3-c2349ba99f21
2026-04-21 03:02:38,316 - distributed.scheduler - INFO - Task run-39e8934911dffc329054688e116c6ec6 marked as failed because 3 workers died while trying to run it
2026-04-21 03:02:38,315 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:43753', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:37,014 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:42875
2026-04-21 03:02:37,012 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:42875', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:36,805 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:35181
2026-04-21 03:02:36,803 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:35181', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:36,764 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:36069
2026-04-21 03:02:36,763 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:36069', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:36,761 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:46489
2026-04-21 03:02:36,760 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:46489', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:36,688 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:45397
2026-04-21 03:02:36,687 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:45397', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:36,625 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:36831
2026-04-21 03:02:36,624 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:36831', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:36,622 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:36421
2026-04-21 03:02:36,621 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:36421', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:36,589 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:35491
2026-04-21 03:02:36,587 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:35491', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:36,569 - distributed.scheduler - INFO - Receive client connection: Client-worker-8ccd4686-3d2e-11f1-86ca-fae0fcb74df3
2026-04-21 03:02:36,519 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:39623
2026-04-21 03:02:36,517 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:39623', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:36,398 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:41323
2026-04-21 03:02:36,396 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:41323', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:36,094 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:35219', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:35,909 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:44205', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:35,832 - distributed.scheduler - INFO - Close client connection: Client-worker-7a192838-3d2e-11f1-8629-2eb33e86e1bd
2026-04-21 03:02:35,828 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:40429', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:35,827 - distributed.scheduler - INFO - Remove client Client-worker-7a192838-3d2e-11f1-8629-2eb33e86e1bd
2026-04-21 03:02:35,698 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:42359', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:35,677 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:35219', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:35,643 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:45179', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:35,625 - distributed.scheduler - INFO - Close client connection: Client-worker-797eb943-3d2e-11f1-8689-2e20a4008324
2026-04-21 03:02:35,623 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:39223', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:35,618 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:34397', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:35,588 - distributed.scheduler - INFO - Remove client Client-worker-797eb943-3d2e-11f1-8689-2e20a4008324
2026-04-21 03:02:35,484 - distributed.scheduler - INFO - Close client connection: Client-worker-7a09302c-3d2e-11f1-877a-f28f846166bd
2026-04-21 03:02:35,481 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:34227', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:35,457 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:45601', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:35,456 - distributed.scheduler - INFO - Remove client Client-worker-7a09302c-3d2e-11f1-877a-f28f846166bd
2026-04-21 03:02:05,736 - distributed.scheduler - INFO - Receive client connection: Client-worker-7a6f2e4d-3d2e-11f1-85f8-2a51d3dc178e
2026-04-21 03:02:05,160 - distributed.scheduler - INFO - Receive client connection: Client-worker-7a192838-3d2e-11f1-8629-2eb33e86e1bd
2026-04-21 03:02:05,078 - distributed.scheduler - INFO - Receive client connection: Client-worker-7a09302c-3d2e-11f1-877a-f28f846166bd
2026-04-21 03:02:04,674 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:43525
2026-04-21 03:02:04,672 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:43525', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:04,625 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:35983
2026-04-21 03:02:04,623 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:35983', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:04,310 - distributed.scheduler - INFO - Receive client connection: Client-worker-79950c55-3d2e-11f1-85f3-c2349ba99f21
2026-04-21 03:02:04,170 - distributed.scheduler - INFO - Receive client connection: Client-worker-797eb943-3d2e-11f1-8689-2e20a4008324
2026-04-21 03:02:03,597 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:36421', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,501 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:44749', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,424 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:45217
2026-04-21 03:02:03,423 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:45217', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,320 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:35219
2026-04-21 03:02:03,318 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:35219', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,299 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:34227
2026-04-21 03:02:03,298 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:34227', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,271 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:39223
2026-04-21 03:02:03,270 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:39223', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,265 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:35219
2026-04-21 03:02:03,265 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:35219', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,262 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:44205
2026-04-21 03:02:03,261 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:44205', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,257 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:45179
2026-04-21 03:02:03,256 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:45179', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,255 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:42359
2026-04-21 03:02:03,253 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:42359', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,204 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:43727
2026-04-21 03:02:03,203 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:43727', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,203 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:45963
2026-04-21 03:02:03,202 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:45963', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,201 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:45601
2026-04-21 03:02:03,200 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:45601', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,181 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:40429
2026-04-21 03:02:03,179 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:40429', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,127 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:43753
2026-04-21 03:02:03,126 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:43753', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:03,124 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:34397
2026-04-21 03:02:03,122 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:34397', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,501 - distributed.scheduler - INFO - Close client connection: Client-worker-62799edb-3d2e-11f1-85ac-0e4c34ff5e38
2026-04-21 03:02:02,476 - distributed.scheduler - INFO - Remove client Client-worker-62799edb-3d2e-11f1-85ac-0e4c34ff5e38
2026-04-21 03:02:02,468 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:34993', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,449 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:44867', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,370 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:35241', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,326 - distributed.scheduler - INFO - Close client connection: Client-worker-64388fd8-3d2e-11f1-85e8-2eb33e86e1bd
2026-04-21 03:02:02,325 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:44967', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,324 - distributed.scheduler - INFO - Remove client Client-worker-64388fd8-3d2e-11f1-85e8-2eb33e86e1bd
2026-04-21 03:02:02,321 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:41609', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,310 - distributed.scheduler - INFO - Close client connection: Client-worker-64bb218c-3d2e-11f1-864e-6ec63a5ab21e
2026-04-21 03:02:02,310 - distributed.scheduler - INFO - Close client connection: Client-worker-6657fd51-3d2e-11f1-8684-4ef3162b8ac2
2026-04-21 03:02:02,308 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:46397', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,300 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:36417', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,299 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:38903', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,299 - distributed.scheduler - INFO - Remove client Client-worker-64bb218c-3d2e-11f1-864e-6ec63a5ab21e
2026-04-21 03:02:02,293 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:43773', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,292 - distributed.scheduler - INFO - Remove client Client-worker-6657fd51-3d2e-11f1-8684-4ef3162b8ac2
2026-04-21 03:02:02,290 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:44295', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,265 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:43933', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,175 - distributed.scheduler - INFO - Close client connection: Client-worker-63efc71d-3d2e-11f1-8688-fae0fcb74df3
2026-04-21 03:02:02,172 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:35421', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:02,170 - distributed.scheduler - INFO - Remove client Client-worker-63efc71d-3d2e-11f1-8688-fae0fcb74df3
2026-04-21 03:02:02,012 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:44453', status: closed, memory: 0, processing: 0>
2026-04-21 03:02:01,996 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:46307', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:32,123 - distributed.scheduler - INFO - Receive client connection: Client-worker-6657fd51-3d2e-11f1-8684-4ef3162b8ac2
2026-04-21 03:01:31,684 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:44749
2026-04-21 03:01:31,683 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:44749', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:31,515 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:44453
2026-04-21 03:01:31,513 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:44453', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:30,691 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:37139', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:30,415 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:46751', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:29,315 - distributed.scheduler - INFO - Receive client connection: Client-worker-64bb218c-3d2e-11f1-864e-6ec63a5ab21e
2026-04-21 03:01:28,623 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:36421
2026-04-21 03:01:28,621 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:36421', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:28,478 - distributed.scheduler - INFO - Receive client connection: Client-worker-64388fd8-3d2e-11f1-85e8-2eb33e86e1bd
2026-04-21 03:01:28,004 - distributed.scheduler - INFO - Receive client connection: Client-worker-63efc71d-3d2e-11f1-8688-fae0fcb74df3
2026-04-21 03:01:27,525 - distributed.scheduler - INFO - Close client connection: Client-worker-4c6159ae-3d2e-11f1-8605-86acc56fd09f
2026-04-21 03:01:27,518 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:43921', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:27,517 - distributed.scheduler - INFO - Remove client Client-worker-4c6159ae-3d2e-11f1-8605-86acc56fd09f
2026-04-21 03:01:25,535 - distributed.scheduler - INFO - Receive client connection: Client-worker-62799edb-3d2e-11f1-85ac-0e4c34ff5e38
2026-04-21 03:01:25,160 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:38903
2026-04-21 03:01:25,149 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:38903', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:25,074 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:44967
2026-04-21 03:01:25,074 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:44967', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:25,057 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:35241
2026-04-21 03:01:25,056 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:35241', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:25,009 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:36417
2026-04-21 03:01:25,009 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:36417', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,930 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:44295
2026-04-21 03:01:24,928 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:44295', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,901 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:44867
2026-04-21 03:01:24,900 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:44867', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,871 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:43773
2026-04-21 03:01:24,870 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:43773', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,864 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:46397
2026-04-21 03:01:24,862 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:46397', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,857 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:41609
2026-04-21 03:01:24,855 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:41609', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,767 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:46307
2026-04-21 03:01:24,766 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:46307', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,755 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:35421
2026-04-21 03:01:24,753 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:35421', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,704 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:43933
2026-04-21 03:01:24,703 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:43933', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,685 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:34993
2026-04-21 03:01:24,684 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:34993', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,185 - distributed.scheduler - INFO - Close client connection: Client-worker-4c26360f-3d2e-11f1-860d-6ec63a5ab21e
2026-04-21 03:01:24,184 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:44135', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,168 - distributed.scheduler - INFO - Remove client Client-worker-4c26360f-3d2e-11f1-860d-6ec63a5ab21e
2026-04-21 03:01:24,163 - distributed.scheduler - INFO - Close client connection: Client-worker-52066248-3d2a-11f1-85a6-2eb33e86e1bd
2026-04-21 03:01:24,143 - distributed.scheduler - INFO - Remove client Client-worker-52066248-3d2a-11f1-85a6-2eb33e86e1bd
2026-04-21 03:01:24,139 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:33287', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,076 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:41707', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,058 - distributed.scheduler - INFO - Task run-44ea4bd54b93d4a5890f305460f22484 marked as failed because 3 workers died while trying to run it
2026-04-21 03:01:24,058 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:36733', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,047 - distributed.scheduler - INFO - Close client connection: Client-worker-74741ddc-3d2a-11f1-8706-f28f846166bd
2026-04-21 03:01:24,044 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:46393', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:24,042 - distributed.scheduler - INFO - Remove client Client-worker-74741ddc-3d2a-11f1-8706-f28f846166bd
2026-04-21 03:01:23,957 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:33721', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:23,934 - distributed.scheduler - INFO - Close client connection: Client-worker-4cbf02c1-3d2e-11f1-8646-4ef3162b8ac2
2026-04-21 03:01:23,931 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:33667', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:23,929 - distributed.scheduler - INFO - Remove client Client-worker-4cbf02c1-3d2e-11f1-8646-4ef3162b8ac2
2026-04-21 03:01:23,916 - distributed.scheduler - INFO - Close client connection: Client-worker-6b5d1525-3d2a-11f1-85fc-566b4744c386
2026-04-21 03:01:23,909 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:42519', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:23,908 - distributed.scheduler - INFO - Remove client Client-worker-6b5d1525-3d2a-11f1-85fc-566b4744c386
2026-04-21 03:01:23,886 - distributed.scheduler - INFO - Close client connection: Client-worker-4ca7cc87-3d2e-11f1-8646-fae0fcb74df3
2026-04-21 03:01:23,883 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:45227', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:23,873 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:39389', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:23,871 - distributed.scheduler - INFO - Remove client Client-worker-4ca7cc87-3d2e-11f1-8646-fae0fcb74df3
2026-04-21 03:01:23,813 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:38399', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:23,705 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:37155', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:23,701 - distributed.scheduler - INFO - Close client connection: Client-worker-4c0abb7b-3d2e-11f1-856c-0e4c34ff5e38
2026-04-21 03:01:23,678 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:33213', status: closed, memory: 0, processing: 0>
2026-04-21 03:01:23,670 - distributed.scheduler - INFO - Remove client Client-worker-4c0abb7b-3d2e-11f1-856c-0e4c34ff5e38
2026-04-21 03:00:49,121 - distributed.scheduler - INFO - Receive client connection: Client-worker-4cbf02c1-3d2e-11f1-8646-4ef3162b8ac2
2026-04-21 03:00:48,960 - distributed.scheduler - INFO - Receive client connection: Client-worker-4ca7cc87-3d2e-11f1-8646-fae0fcb74df3
2026-04-21 03:00:48,484 - distributed.scheduler - INFO - Receive client connection: Client-worker-4c6159ae-3d2e-11f1-8605-86acc56fd09f
2026-04-21 03:00:48,128 - distributed.scheduler - INFO - Receive client connection: Client-worker-4c26360f-3d2e-11f1-860d-6ec63a5ab21e
2026-04-21 03:00:47,899 - distributed.scheduler - INFO - Receive client connection: Client-worker-4c0abb7b-3d2e-11f1-856c-0e4c34ff5e38
2026-04-21 03:00:44,432 - distributed.scheduler - INFO - Receive client connection: Client-49f96841-3d2e-11f1-8001-daa6852bd5a7
2026-04-21 02:50:42,975 - distributed.scheduler - INFO - Receive client connection: Client-e37b9786-3d2c-11f1-8001-daa6852bd5a7
2026-04-21 02:40:41,510 - distributed.scheduler - INFO - Receive client connection: Client-7cfa2707-3d2b-11f1-8001-daa6852bd5a7
2026-04-21 02:33:17,706 - distributed.scheduler - INFO - Receive client connection: Client-worker-74741ddc-3d2a-11f1-8706-f28f846166bd
2026-04-21 02:33:02,450 - distributed.scheduler - INFO - Receive client connection: Client-worker-6b5d1525-3d2a-11f1-85fc-566b4744c386
2026-04-21 02:32:59,841 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:39041 failed: OSError: Timed out trying to connect to tcp://172.21.159.246:39041 after 30 s
2026-04-21 02:32:59,840 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:34689 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:34689 after 30 s
2026-04-21 02:32:59,840 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:42053 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:42053 after 30 s
2026-04-21 02:32:59,839 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:39731 failed: OSError: Timed out trying to connect to tcp://172.21.159.207:39731 after 30 s
2026-04-21 02:32:59,839 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39551 failed: OSError: Timed out trying to connect to tcp://172.21.160.14:39551 after 30 s
2026-04-21 02:32:59,838 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:37333 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:37333 after 30 s
2026-04-21 02:32:59,604 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:33721 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.246:39041 after 30 s')
2026-04-21 02:32:58,989 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:33213 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.160.14:39551 after 30 s')
2026-04-21 02:32:58,842 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:42519 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42053 after 30 s')
2026-04-21 02:32:29,123 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:38399
2026-04-21 02:32:29,122 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:38399', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:28,745 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:43921
2026-04-21 02:32:28,744 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:43921', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:28,691 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:36733
2026-04-21 02:32:28,690 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:36733', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:28,618 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:39389
2026-04-21 02:32:28,617 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:39389', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:28,499 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:33213
2026-04-21 02:32:28,498 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:33213', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:28,461 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:33721
2026-04-21 02:32:28,459 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:33721', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:28,174 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:37333', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:27,685 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:42053', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:27,672 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:39731', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:27,629 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:39551', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:27,592 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:34689', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:27,571 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:39041', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:19,951 - distributed.scheduler - INFO - Receive client connection: Client-worker-52066248-3d2a-11f1-85a6-2eb33e86e1bd
2026-04-21 02:32:19,864 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:37155
2026-04-21 02:32:19,864 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:37155', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:19,853 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:37139
2026-04-21 02:32:19,852 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:37139', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:18,853 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:46751
2026-04-21 02:32:18,852 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:46751', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:18,852 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:34689
2026-04-21 02:32:18,851 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:34689', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:18,846 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:42553', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,966 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:36653', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,865 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:39731
2026-04-21 02:32:17,863 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:39731', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,705 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:33287
2026-04-21 02:32:17,704 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:33287', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,680 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:39551
2026-04-21 02:32:17,679 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:39551', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,675 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:41707
2026-04-21 02:32:17,673 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:41707', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,657 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:39041
2026-04-21 02:32:17,656 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:39041', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,649 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:45227
2026-04-21 02:32:17,648 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:45227', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,612 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:44135
2026-04-21 02:32:17,611 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:44135', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,586 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:46393
2026-04-21 02:32:17,585 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:46393', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,531 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:42053
2026-04-21 02:32:17,529 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:42053', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,447 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:33667
2026-04-21 02:32:17,446 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:33667', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,434 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:37333
2026-04-21 02:32:17,433 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:37333', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,342 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:42519
2026-04-21 02:32:17,340 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:42519', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,318 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:41335', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:17,221 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:41037', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:16,768 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:33751', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:16,723 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:45249', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:16,723 - distributed.scheduler - INFO - Close client connection: Client-worker-36c26720-3d2a-11f1-8578-7e353d94dc10
2026-04-21 02:32:16,722 - distributed.scheduler - INFO - Remove client Client-worker-36c26720-3d2a-11f1-8578-7e353d94dc10
2026-04-21 02:32:16,719 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:33347', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:16,699 - distributed.scheduler - INFO - Close client connection: Client-worker-35f613cc-3d2a-11f1-8518-0e4c34ff5e38
2026-04-21 02:32:16,697 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:33901', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:16,695 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:40251', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:16,691 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:35947', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:16,667 - distributed.scheduler - INFO - Remove client Client-worker-35f613cc-3d2a-11f1-8518-0e4c34ff5e38
2026-04-21 02:32:16,643 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:32775', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:16,593 - distributed.scheduler - INFO - Close client connection: Client-worker-2d8c14da-3d2a-11f1-86ca-f28f846166bd
2026-04-21 02:32:16,586 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:45599', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:16,578 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:34217', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:16,578 - distributed.scheduler - INFO - Remove client Client-worker-2d8c14da-3d2a-11f1-86ca-f28f846166bd
2026-04-21 02:32:16,535 - distributed.scheduler - INFO - Close client connection: Client-worker-2cd19307-3d2a-11f1-860e-b6f02dd406e9
2026-04-21 02:32:16,530 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:39421', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:16,529 - distributed.scheduler - INFO - Remove client Client-worker-2cd19307-3d2a-11f1-860e-b6f02dd406e9
2026-04-21 02:32:16,520 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:34043', status: closed, memory: 0, processing: 0>
2026-04-21 02:32:16,432 - distributed.scheduler - INFO - Close client connection: Client-worker-2cd88c6b-3d2a-11f1-85b9-566b4744c386
2026-04-21 02:32:16,427 - distributed.scheduler - INFO - Remove client Client-worker-2cd88c6b-3d2a-11f1-85b9-566b4744c386
2026-04-21 02:32:16,423 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:36241', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:34,206 - distributed.scheduler - INFO - Receive client connection: Client-worker-36c26720-3d2a-11f1-8578-7e353d94dc10
2026-04-21 02:31:33,345 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:42553
2026-04-21 02:31:33,344 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:42553', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:33,297 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:41037
2026-04-21 02:31:33,296 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:41037', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:33,012 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:36653
2026-04-21 02:31:33,011 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:36653', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:32,879 - distributed.scheduler - INFO - Receive client connection: Client-worker-35f613cc-3d2a-11f1-8518-0e4c34ff5e38
2026-04-21 02:31:32,289 - distributed.scheduler - INFO - Close client connection: Client-worker-e56f67f3-3d25-11f1-8580-ee7435aaba1f
2026-04-21 02:31:32,287 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:38111', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:32,286 - distributed.scheduler - INFO - Remove client Client-worker-e56f67f3-3d25-11f1-8580-ee7435aaba1f
2026-04-21 02:31:32,219 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:36787', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:31,908 - distributed.scheduler - INFO - Close client connection: Client-worker-6b354357-3cef-11f1-85a4-2e20a4008324
2026-04-21 02:31:31,906 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:42247', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:31,905 - distributed.scheduler - INFO - Remove client Client-worker-6b354357-3cef-11f1-85a4-2e20a4008324
2026-04-21 02:31:26,635 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-75a661d8103b6aef593146bec0e1cfb6 NoneType: None
2026-04-21 02:31:26,635 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-1d69c86d2dda062067e24bce7a881b35 NoneType: None
2026-04-21 02:31:26,635 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-626997a3ac358167faf2972aa8e118fe NoneType: None
2026-04-21 02:31:26,635 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-4cad84048529fc9e382d2a0f9bfc695b NoneType: None
2026-04-21 02:31:26,634 - distributed.scheduler - ERROR - Couldn't gather keys {'run-4cad84048529fc9e382d2a0f9bfc695b': [], 'run-626997a3ac358167faf2972aa8e118fe': [], 'run-1d69c86d2dda062067e24bce7a881b35': [], 'run-75a661d8103b6aef593146bec0e1cfb6': []} state: ['processing', 'processing', 'processing', 'processing'] workers: [] NoneType: None
2026-04-21 02:31:25,849 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:41335
2026-04-21 02:31:25,847 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:41335', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:24,930 - distributed.scheduler - INFO - Close client connection: Client-worker-9ecad52d-3cf3-11f1-850e-c2349ba99f21
2026-04-21 02:31:24,927 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:42077', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:24,925 - distributed.scheduler - INFO - Remove client Client-worker-9ecad52d-3cf3-11f1-850e-c2349ba99f21
2026-04-21 02:31:18,745 - distributed.scheduler - INFO - Receive client connection: Client-worker-2d8c14da-3d2a-11f1-86ca-f28f846166bd
2026-04-21 02:31:17,571 - distributed.scheduler - INFO - Receive client connection: Client-worker-2cd88c6b-3d2a-11f1-85b9-566b4744c386
2026-04-21 02:31:17,524 - distributed.scheduler - INFO - Receive client connection: Client-worker-2cd19307-3d2a-11f1-860e-b6f02dd406e9
2026-04-21 02:31:17,127 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:35947
2026-04-21 02:31:17,125 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:35947', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:17,041 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:45599
2026-04-21 02:31:17,040 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:45599', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:17,003 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:33751
2026-04-21 02:31:17,002 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:33751', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,994 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:32775
2026-04-21 02:31:16,993 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:32775', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,931 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:45249
2026-04-21 02:31:16,930 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:45249', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,928 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:39421
2026-04-21 02:31:16,927 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:39421', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,925 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:33901
2026-04-21 02:31:16,924 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:33901', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,911 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:34043
2026-04-21 02:31:16,910 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:34043', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,903 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:40251
2026-04-21 02:31:16,902 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:40251', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,885 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:33347
2026-04-21 02:31:16,884 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:33347', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,866 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:36241
2026-04-21 02:31:16,865 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:36241', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,797 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:34217
2026-04-21 02:31:16,795 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:34217', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,180 - distributed.scheduler - INFO - Close client connection: Client-worker-e59c3918-3d25-11f1-84da-0e4c34ff5e38
2026-04-21 02:31:16,177 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:41875', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,175 - distributed.scheduler - INFO - Remove client Client-worker-e59c3918-3d25-11f1-84da-0e4c34ff5e38
2026-04-21 02:31:16,085 - distributed.scheduler - INFO - Close client connection: Client-worker-196ebaaa-3d2a-11f1-8531-8288bfee4388
2026-04-21 02:31:16,084 - distributed.scheduler - INFO - Close client connection: Client-worker-9eb0a383-3cf3-11f1-8581-ae00acbfc48e
2026-04-21 02:31:16,079 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:40067', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,070 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:38149', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:16,067 - distributed.scheduler - INFO - Remove client Client-worker-196ebaaa-3d2a-11f1-8531-8288bfee4388
2026-04-21 02:31:16,063 - distributed.scheduler - INFO - Remove client Client-worker-9eb0a383-3cf3-11f1-8581-ae00acbfc48e
2026-04-21 02:31:16,015 - distributed.scheduler - INFO - Close client connection: Client-worker-6bceec1a-3cef-11f1-85b2-fae0fcb74df3
2026-04-21 02:31:15,999 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:40575', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:15,990 - distributed.scheduler - INFO - Remove client Client-worker-6bceec1a-3cef-11f1-85b2-fae0fcb74df3
2026-04-21 02:31:15,989 - distributed.scheduler - INFO - Close client connection: Client-worker-9e73bf28-3cf3-11f1-85ce-b6f02dd406e9
2026-04-21 02:31:15,977 - distributed.scheduler - INFO - Task ('sjoin-d63537210f6bb530c8bf3dbfba0cb6af', 111) marked as failed because 3 workers died while trying to run it
2026-04-21 02:31:15,977 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:36029', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:15,977 - distributed.scheduler - INFO - Remove client Client-worker-9e73bf28-3cf3-11f1-85ce-b6f02dd406e9
2026-04-21 02:31:15,970 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:41831', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:15,969 - distributed.scheduler - INFO - Close client connection: Client-worker-6b62ea36-3cef-11f1-8538-7e353d94dc10
2026-04-21 02:31:15,969 - distributed.scheduler - INFO - Remove client Client-worker-6b62ea36-3cef-11f1-8538-7e353d94dc10
2026-04-21 02:31:15,967 - distributed.scheduler - INFO - Close client connection: Client-worker-74a0f18d-3ceb-11f1-8518-2eb33e86e1bd
2026-04-21 02:31:15,965 - distributed.scheduler - INFO - Remove client Client-worker-74a0f18d-3ceb-11f1-8518-2eb33e86e1bd
2026-04-21 02:31:15,960 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:40371', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:15,956 - distributed.scheduler - INFO - Close client connection: Client-worker-6b5b5893-3cef-11f1-85d2-4ef3162b8ac2
2026-04-21 02:31:15,956 - distributed.scheduler - INFO - Close client connection: Client-worker-e586724b-3d25-11f1-850d-2a51d3dc178e
2026-04-21 02:31:15,955 - distributed.scheduler - INFO - Close client connection: Client-worker-6c02c4a7-3cef-11f1-856b-566b4744c386
2026-04-21 02:31:15,948 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:42505', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:15,948 - distributed.scheduler - INFO - Remove client Client-worker-6b5b5893-3cef-11f1-85d2-4ef3162b8ac2
2026-04-21 02:31:15,945 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:38381', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:15,945 - distributed.scheduler - INFO - Remove client Client-worker-e586724b-3d25-11f1-850d-2a51d3dc178e
2026-04-21 02:31:15,937 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:39701', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:15,915 - distributed.scheduler - INFO - Remove client Client-worker-6c02c4a7-3cef-11f1-856b-566b4744c386
2026-04-21 02:31:15,872 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:44445', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:15,824 - distributed.scheduler - INFO - Close client connection: Client-worker-6b941ef1-3cef-11f1-8680-f28f846166bd
2026-04-21 02:31:15,822 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:33585', status: closed, memory: 0, processing: 0>
2026-04-21 02:31:15,821 - distributed.scheduler - INFO - Remove client Client-worker-6b941ef1-3cef-11f1-8680-f28f846166bd
2026-04-21 02:30:45,067 - distributed.scheduler - INFO - Receive client connection: Client-worker-196ebaaa-3d2a-11f1-8531-8288bfee4388
2026-04-21 02:30:40,027 - distributed.scheduler - INFO - Receive client connection: Client-16772c76-3d2a-11f1-8001-daa6852bd5a7
2026-04-21 02:20:38,608 - distributed.scheduler - INFO - Receive client connection: Client-affdeb03-3d28-11f1-8001-daa6852bd5a7
2026-04-21 02:13:27,236 - distributed.scheduler - INFO - Receive client connection: Client-aedff465-3d27-11f1-8001-daa6852bd5a7
2026-04-21 02:00:40,066 - distributed.scheduler - INFO - Receive client connection: Client-worker-e59c3918-3d25-11f1-84da-0e4c34ff5e38
2026-04-21 02:00:39,933 - distributed.scheduler - INFO - Receive client connection: Client-worker-e586724b-3d25-11f1-850d-2a51d3dc178e
2026-04-21 02:00:39,772 - distributed.scheduler - INFO - Receive client connection: Client-worker-e56f67f3-3d25-11f1-8580-ee7435aaba1f
2026-04-21 02:00:35,693 - distributed.scheduler - INFO - Receive client connection: Client-e2fffeb5-3d25-11f1-8001-daa6852bd5a7
2026-04-21 01:50:34,218 - distributed.scheduler - INFO - Receive client connection: Client-7c7e69b0-3d24-11f1-8001-daa6852bd5a7
2026-04-21 01:40:32,694 - distributed.scheduler - INFO - Receive client connection: Client-15f52a03-3d23-11f1-8001-daa6852bd5a7
2026-04-21 01:30:31,196 - distributed.scheduler - INFO - Receive client connection: Client-af6fee64-3d21-11f1-8001-daa6852bd5a7
2026-04-21 01:20:29,713 - distributed.scheduler - INFO - Receive client connection: Client-48ec5050-3d20-11f1-8001-daa6852bd5a7
2026-04-21 01:13:04,620 - distributed.scheduler - INFO - Receive client connection: Client-3fa15130-3d1f-11f1-8001-daa6852bd5a7
2026-04-21 01:00:26,873 - distributed.scheduler - INFO - Receive client connection: Client-7bf92574-3d1d-11f1-8001-daa6852bd5a7
2026-04-21 00:50:25,360 - distributed.scheduler - INFO - Receive client connection: Client-15722f30-3d1c-11f1-8001-daa6852bd5a7
2026-04-21 00:40:23,820 - distributed.scheduler - INFO - Receive client connection: Client-aee6761b-3d1a-11f1-8001-daa6852bd5a7
2026-04-21 00:30:22,316 - distributed.scheduler - INFO - Receive client connection: Client-486051ad-3d19-11f1-8001-daa6852bd5a7
2026-04-21 00:20:20,808 - distributed.scheduler - INFO - Receive client connection: Client-e1d97cb0-3d17-11f1-8001-daa6852bd5a7
2026-04-21 00:12:27,630 - distributed.scheduler - INFO - Receive client connection: Client-c7d0ac1c-3d16-11f1-8001-daa6852bd5a7
2026-04-21 00:00:17,893 - distributed.scheduler - INFO - Receive client connection: Client-14dabfca-3d15-11f1-8001-daa6852bd5a7
2026-04-20 23:50:16,472 - distributed.scheduler - INFO - Receive client connection: Client-ae6164a1-3d13-11f1-8001-daa6852bd5a7
2026-04-20 23:40:14,936 - distributed.scheduler - INFO - Receive client connection: Client-47d5f536-3d12-11f1-8001-daa6852bd5a7
2026-04-20 23:30:13,505 - distributed.scheduler - INFO - Receive client connection: Client-e15b21dc-3d10-11f1-8001-daa6852bd5a7
2026-04-20 23:20:12,045 - distributed.scheduler - INFO - Receive client connection: Client-7adc4d45-3d0f-11f1-8001-daa6852bd5a7
2026-04-20 23:12:23,900 - distributed.scheduler - INFO - Receive client connection: Client-63d316b8-3d0e-11f1-8001-daa6852bd5a7
2026-04-20 23:00:09,210 - distributed.scheduler - INFO - Receive client connection: Client-adea0665-3d0c-11f1-8001-daa6852bd5a7
2026-04-20 22:50:07,795 - distributed.scheduler - INFO - Receive client connection: Client-4771cea8-3d0b-11f1-8001-daa6852bd5a7
2026-04-20 22:40:06,331 - distributed.scheduler - INFO - Receive client connection: Client-e0f1715a-3d09-11f1-8001-daa6852bd5a7
2026-04-20 22:30:04,815 - distributed.scheduler - INFO - Receive client connection: Client-7a68bc8e-3d08-11f1-8001-daa6852bd5a7
2026-04-20 22:20:03,202 - distributed.scheduler - INFO - Receive client connection: Client-13d1b6ef-3d07-11f1-8001-daa6852bd5a7
2026-04-20 22:12:07,399 - distributed.scheduler - INFO - Receive client connection: Client-f838d112-3d05-11f1-8001-daa6852bd5a7
2026-04-20 22:00:00,182 - distributed.scheduler - INFO - Receive client connection: Client-46c3d138-3d04-11f1-8001-daa6852bd5a7
2026-04-20 21:50:58,879 - distributed.scheduler - INFO - Receive client connection: Client-041edccd-3d03-11f1-8001-daa6852bd5a7
2026-04-20 21:40:57,446 - distributed.scheduler - INFO - Receive client connection: Client-9da43628-3d01-11f1-8001-daa6852bd5a7
2026-04-20 21:30:55,907 - distributed.scheduler - INFO - Receive client connection: Client-3717867d-3d00-11f1-8001-daa6852bd5a7
2026-04-20 21:20:54,338 - distributed.scheduler - INFO - Receive client connection: Client-d089691e-3cfe-11f1-8001-daa6852bd5a7
2026-04-20 21:14:01,452 - distributed.scheduler - INFO - Receive client connection: Client-da6fcb45-3cfd-11f1-8001-daa6852bd5a7
2026-04-20 21:00:51,374 - distributed.scheduler - INFO - Receive client connection: Client-038413ad-3cfc-11f1-8001-daa6852bd5a7
2026-04-20 20:50:49,845 - distributed.scheduler - INFO - Receive client connection: Client-9cf92af9-3cfa-11f1-8001-daa6852bd5a7
2026-04-20 20:40:48,390 - distributed.scheduler - INFO - Receive client connection: Client-367a0514-3cf9-11f1-8001-daa6852bd5a7
2026-04-20 20:30:46,849 - distributed.scheduler - INFO - Receive client connection: Client-cfeca627-3cf7-11f1-8001-daa6852bd5a7
2026-04-20 20:21:10,023 - distributed.scheduler - INFO - Close client connection: Client-69709094-3cf6-11f1-8001-daa6852bd5a7
2026-04-20 20:21:10,022 - distributed.scheduler - INFO - Remove client Client-69709094-3cf6-11f1-8001-daa6852bd5a7
2026-04-20 20:21:10,021 - distributed.scheduler - INFO - Remove client Client-69709094-3cf6-11f1-8001-daa6852bd5a7
2026-04-20 20:20:45,413 - distributed.scheduler - INFO - Receive client connection: Client-69709094-3cf6-11f1-8001-daa6852bd5a7
2026-04-20 20:13:25,369 - distributed.scheduler - INFO - Close client connection: Client-54af5b54-3cf5-11f1-8001-daa6852bd5a7
2026-04-20 20:13:25,368 - distributed.scheduler - INFO - Remove client Client-54af5b54-3cf5-11f1-8001-daa6852bd5a7
2026-04-20 20:13:25,366 - distributed.scheduler - INFO - Remove client Client-54af5b54-3cf5-11f1-8001-daa6852bd5a7
2026-04-20 20:13:01,096 - distributed.scheduler - INFO - Receive client connection: Client-54af5b54-3cf5-11f1-8001-daa6852bd5a7
2026-04-20 20:01:36,976 - distributed.scheduler - INFO - Close client connection: Client-9c70a36f-3cf3-11f1-8001-daa6852bd5a7
2026-04-20 20:01:36,975 - distributed.scheduler - INFO - Remove client Client-9c70a36f-3cf3-11f1-8001-daa6852bd5a7
2026-04-20 20:01:36,973 - distributed.scheduler - INFO - Remove client Client-9c70a36f-3cf3-11f1-8001-daa6852bd5a7
2026-04-20 20:00:46,470 - distributed.scheduler - INFO - Receive client connection: Client-worker-9ecad52d-3cf3-11f1-850e-c2349ba99f21
2026-04-20 20:00:46,241 - distributed.scheduler - INFO - Receive client connection: Client-worker-9eb0a383-3cf3-11f1-8581-ae00acbfc48e
2026-04-20 20:00:45,847 - distributed.scheduler - INFO - Receive client connection: Client-worker-9e73bf28-3cf3-11f1-85ce-b6f02dd406e9
2026-04-20 20:00:42,482 - distributed.scheduler - INFO - Receive client connection: Client-9c70a36f-3cf3-11f1-8001-daa6852bd5a7
2026-04-20 19:51:13,297 - distributed.scheduler - INFO - Close client connection: Client-360220c2-3cf2-11f1-8001-daa6852bd5a7
2026-04-20 19:51:13,296 - distributed.scheduler - INFO - Remove client Client-360220c2-3cf2-11f1-8001-daa6852bd5a7
2026-04-20 19:51:13,294 - distributed.scheduler - INFO - Remove client Client-360220c2-3cf2-11f1-8001-daa6852bd5a7
2026-04-20 19:50:41,138 - distributed.scheduler - INFO - Receive client connection: Client-360220c2-3cf2-11f1-8001-daa6852bd5a7
2026-04-20 19:41:19,598 - distributed.scheduler - INFO - Close client connection: Client-cf82adad-3cf0-11f1-8001-daa6852bd5a7
2026-04-20 19:41:19,596 - distributed.scheduler - INFO - Remove client Client-cf82adad-3cf0-11f1-8001-daa6852bd5a7
2026-04-20 19:41:19,595 - distributed.scheduler - INFO - Remove client Client-cf82adad-3cf0-11f1-8001-daa6852bd5a7
2026-04-20 19:40:39,682 - distributed.scheduler - INFO - Receive client connection: Client-cf82adad-3cf0-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,170 - distributed.scheduler - INFO - Close client connection: Client-672956d1-3cda-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,169 - distributed.scheduler - INFO - Close client connection: Client-342ed6b9-3cdd-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,169 - distributed.scheduler - INFO - Close client connection: Client-1af4e3fb-3cdc-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,169 - distributed.scheduler - INFO - Close client connection: Client-013b747d-3ce0-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,169 - distributed.scheduler - INFO - Close client connection: Client-9ab410fd-3cde-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,169 - distributed.scheduler - INFO - Close client connection: Client-67c2c5f6-3ce1-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,168 - distributed.scheduler - INFO - Close client connection: Client-ce561dbe-3ce2-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,168 - distributed.scheduler - INFO - Close client connection: Client-9b6bfe3b-3ce5-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,168 - distributed.scheduler - INFO - Close client connection: Client-ba8cdd32-3ce4-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,167 - distributed.scheduler - INFO - Close client connection: Client-01f3ea8a-3ce7-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,167 - distributed.scheduler - INFO - Close client connection: Client-6874dcdf-3ce8-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,167 - distributed.scheduler - INFO - Close client connection: Client-cef5b204-3ce9-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,167 - distributed.scheduler - INFO - Close client connection: Client-357defe4-3ceb-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,167 - distributed.scheduler - INFO - Close client connection: Client-0287feb3-3cee-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,166 - distributed.scheduler - INFO - Close client connection: Client-ed632deb-3cec-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,165 - distributed.scheduler - INFO - Remove client Client-672956d1-3cda-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,165 - distributed.scheduler - INFO - Remove client Client-342ed6b9-3cdd-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,164 - distributed.scheduler - INFO - Remove client Client-1af4e3fb-3cdc-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,164 - distributed.scheduler - INFO - Remove client Client-013b747d-3ce0-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,163 - distributed.scheduler - INFO - Remove client Client-9ab410fd-3cde-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,163 - distributed.scheduler - INFO - Remove client Client-67c2c5f6-3ce1-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,159 - distributed.scheduler - INFO - Remove client Client-ce561dbe-3ce2-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,158 - distributed.scheduler - INFO - Remove client Client-9b6bfe3b-3ce5-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,157 - distributed.scheduler - INFO - Remove client Client-ba8cdd32-3ce4-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,156 - distributed.scheduler - INFO - Remove client Client-01f3ea8a-3ce7-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,156 - distributed.scheduler - INFO - Remove client Client-6874dcdf-3ce8-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,155 - distributed.scheduler - INFO - Remove client Client-cef5b204-3ce9-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,154 - distributed.scheduler - INFO - Remove client Client-357defe4-3ceb-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,153 - distributed.scheduler - INFO - Remove client Client-0287feb3-3cee-11f1-8001-daa6852bd5a7
2026-04-20 19:31:41,152 - distributed.scheduler - INFO - Remove client Client-ed632deb-3cec-11f1-8001-daa6852bd5a7
2026-04-20 19:31:40,369 - distributed.scheduler - INFO - Close client connection: Client-68ff1636-3cef-11f1-8001-daa6852bd5a7
2026-04-20 19:31:40,368 - distributed.scheduler - INFO - Remove client Client-68ff1636-3cef-11f1-8001-daa6852bd5a7
2026-04-20 19:31:40,366 - distributed.scheduler - INFO - Remove client Client-68ff1636-3cef-11f1-8001-daa6852bd5a7
2026-04-20 19:30:43,231 - distributed.scheduler - INFO - Receive client connection: Client-worker-6c02c4a7-3cef-11f1-856b-566b4744c386
2026-04-20 19:30:42,885 - distributed.scheduler - INFO - Receive client connection: Client-worker-6bceec1a-3cef-11f1-85b2-fae0fcb74df3
2026-04-20 19:30:42,507 - distributed.scheduler - INFO - Receive client connection: Client-worker-6b941ef1-3cef-11f1-8680-f28f846166bd
2026-04-20 19:30:42,204 - distributed.scheduler - INFO - Receive client connection: Client-worker-6b62ea36-3cef-11f1-8538-7e353d94dc10
2026-04-20 19:30:42,137 - distributed.scheduler - INFO - Receive client connection: Client-worker-6b5b5893-3cef-11f1-85d2-4ef3162b8ac2
2026-04-20 19:30:41,891 - distributed.scheduler - INFO - Receive client connection: Client-worker-6b354357-3cef-11f1-85a4-2e20a4008324
2026-04-20 19:30:38,181 - distributed.scheduler - INFO - Receive client connection: Client-68ff1636-3cef-11f1-8001-daa6852bd5a7
2026-04-20 19:20:36,782 - distributed.scheduler - INFO - Receive client connection: Client-0287feb3-3cee-11f1-8001-daa6852bd5a7
2026-04-20 19:12:51,803 - distributed.scheduler - INFO - Receive client connection: Client-ed632deb-3cec-11f1-8001-daa6852bd5a7
2026-04-20 19:02:19,746 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:38111
2026-04-20 19:02:19,744 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:38111', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:19,700 - distributed.scheduler - INFO - Receive client connection: Client-worker-74a0f18d-3ceb-11f1-8518-2eb33e86e1bd
2026-04-20 19:02:19,074 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:36787
2026-04-20 19:02:19,072 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:36787', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:18,705 - distributed.scheduler - INFO - Close client connection: Client-worker-64998c14-3ceb-11f1-8544-ee7435aaba1f
2026-04-20 19:02:18,702 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:35119', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:18,701 - distributed.scheduler - INFO - Remove client Client-worker-64998c14-3ceb-11f1-8544-ee7435aaba1f
2026-04-20 19:02:18,548 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:42077
2026-04-20 19:02:18,546 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:42077', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:18,020 - distributed.scheduler - INFO - Close client connection: Client-worker-4d902f45-3ceb-11f1-8546-86acc56fd09f
2026-04-20 19:02:18,014 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:40079', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:18,013 - distributed.scheduler - INFO - Remove client Client-worker-4d902f45-3ceb-11f1-8546-86acc56fd09f
2026-04-20 19:02:17,397 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:34465', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,893 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:44445
2026-04-20 19:02:16,891 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:44445', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,854 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:38149
2026-04-20 19:02:16,853 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:38149', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,727 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:41831
2026-04-20 19:02:16,726 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:41831', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,715 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:40371
2026-04-20 19:02:16,714 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:40371', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,705 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:33585
2026-04-20 19:02:16,704 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:33585', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,631 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:42247
2026-04-20 19:02:16,630 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:42247', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,592 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:40067
2026-04-20 19:02:16,591 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:40067', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,555 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:41875
2026-04-20 19:02:16,553 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:41875', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,551 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:40575
2026-04-20 19:02:16,550 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:40575', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,546 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:42505
2026-04-20 19:02:16,545 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:42505', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,534 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:38381
2026-04-20 19:02:16,532 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:38381', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,418 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:36029
2026-04-20 19:02:16,416 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:36029', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:16,346 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:39701
2026-04-20 19:02:16,344 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:39701', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,951 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:38719', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,864 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:37119', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,798 - distributed.scheduler - INFO - Close client connection: Client-worker-4cd17c05-3ceb-11f1-84f7-7e353d94dc10
2026-04-20 19:02:15,795 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:33041', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,766 - distributed.scheduler - INFO - Remove client Client-worker-4cd17c05-3ceb-11f1-84f7-7e353d94dc10
2026-04-20 19:02:15,674 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:42959', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,663 - distributed.scheduler - INFO - Close client connection: Client-worker-4bd934a8-3ceb-11f1-84d8-2eb33e86e1bd
2026-04-20 19:02:15,660 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:34701', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,659 - distributed.scheduler - INFO - Remove client Client-worker-4bd934a8-3ceb-11f1-84d8-2eb33e86e1bd
2026-04-20 19:02:15,653 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:45703', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,634 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:32875', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,607 - distributed.scheduler - INFO - Close client connection: Client-worker-4c446c11-3ceb-11f1-8578-fae0fcb74df3
2026-04-20 19:02:15,591 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:44845', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,590 - distributed.scheduler - INFO - Remove client Client-worker-4c446c11-3ceb-11f1-8578-fae0fcb74df3
2026-04-20 19:02:15,588 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:38671', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,521 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:37009', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,511 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:43139', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,440 - distributed.scheduler - INFO - Close client connection: Client-worker-4c5d1baa-3ceb-11f1-858d-b6f02dd406e9
2026-04-20 19:02:15,438 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:41439', status: closed, memory: 0, processing: 0>
2026-04-20 19:02:15,437 - distributed.scheduler - INFO - Remove client Client-worker-4c5d1baa-3ceb-11f1-858d-b6f02dd406e9
2026-04-20 19:02:15,398 - distributed.scheduler - INFO - Close client connection: Client-worker-4b9bfb41-3ceb-11f1-852d-566b4744c386
2026-04-20 19:02:15,395 - distributed.scheduler - INFO - Remove client Client-worker-4b9bfb41-3ceb-11f1-852d-566b4744c386
2026-04-20 19:02:15,394 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:40985', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:52,814 - distributed.scheduler - INFO - Receive client connection: Client-worker-64998c14-3ceb-11f1-8544-ee7435aaba1f
2026-04-20 19:01:14,648 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:35119
2026-04-20 19:01:14,647 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:35119', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:14,177 - distributed.scheduler - INFO - Receive client connection: Client-worker-4d902f45-3ceb-11f1-8546-86acc56fd09f
2026-04-20 19:01:13,623 - distributed.scheduler - INFO - Close client connection: Client-worker-d1a580a5-3ce2-11f1-8507-ee7435aaba1f
2026-04-20 19:01:13,616 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:43265', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:13,616 - distributed.scheduler - INFO - Remove client Client-worker-d1a580a5-3ce2-11f1-8507-ee7435aaba1f
2026-04-20 19:01:12,926 - distributed.scheduler - INFO - Receive client connection: Client-worker-4cd17c05-3ceb-11f1-84f7-7e353d94dc10
2026-04-20 19:01:12,155 - distributed.scheduler - INFO - Receive client connection: Client-worker-4c5d1baa-3ceb-11f1-858d-b6f02dd406e9
2026-04-20 19:01:11,991 - distributed.scheduler - INFO - Receive client connection: Client-worker-4c446c11-3ceb-11f1-8578-fae0fcb74df3
2026-04-20 19:01:11,295 - distributed.scheduler - INFO - Receive client connection: Client-worker-4bd934a8-3ceb-11f1-84d8-2eb33e86e1bd
2026-04-20 19:01:10,897 - distributed.scheduler - INFO - Receive client connection: Client-worker-4b9bfb41-3ceb-11f1-852d-566b4744c386
2026-04-20 19:01:10,755 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:37119
2026-04-20 19:01:10,754 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:37119', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:10,589 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:34465
2026-04-20 19:01:10,588 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:34465', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:10,532 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:38719
2026-04-20 19:01:10,530 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:38719', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:10,307 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:37009
2026-04-20 19:01:10,306 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:37009', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:10,304 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:40079
2026-04-20 19:01:10,303 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:40079', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:10,235 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:32875
2026-04-20 19:01:10,234 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:32875', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:10,144 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:45703
2026-04-20 19:01:10,142 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:45703', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:10,133 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:42959
2026-04-20 19:01:10,131 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:42959', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:10,083 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:33041
2026-04-20 19:01:10,082 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:33041', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:10,079 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:38671
2026-04-20 19:01:10,078 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:38671', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:10,021 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:43139
2026-04-20 19:01:10,019 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:43139', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,985 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:40985
2026-04-20 19:01:09,984 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:40985', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,973 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:41439
2026-04-20 19:01:09,972 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:41439', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,876 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:34701
2026-04-20 19:01:09,875 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:34701', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,843 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:44845
2026-04-20 19:01:09,841 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:44845', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,760 - distributed.scheduler - INFO - Close client connection: Client-worker-d1784c6b-3ce2-11f1-8506-ae00acbfc48e
2026-04-20 19:01:09,758 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:32805', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,758 - distributed.scheduler - INFO - Remove client Client-worker-d1784c6b-3ce2-11f1-8506-ae00acbfc48e
2026-04-20 19:01:09,690 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:46057', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,416 - distributed.scheduler - INFO - Close client connection: Client-worker-d0e58780-3ce2-11f1-8488-c2349ba99f21
2026-04-20 19:01:09,414 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:35911', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,413 - distributed.scheduler - INFO - Remove client Client-worker-d0e58780-3ce2-11f1-8488-c2349ba99f21
2026-04-20 19:01:09,301 - distributed.scheduler - INFO - Close client connection: Client-worker-04e1cae4-3ce7-11f1-8493-2a51d3dc178e
2026-04-20 19:01:09,298 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:41547', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,261 - distributed.scheduler - INFO - Remove client Client-worker-04e1cae4-3ce7-11f1-8493-2a51d3dc178e
2026-04-20 19:01:09,169 - distributed.scheduler - INFO - Close client connection: Client-worker-d1667d07-3ce2-11f1-8608-f28f846166bd
2026-04-20 19:01:09,166 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:42371', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,165 - distributed.scheduler - INFO - Remove client Client-worker-d1667d07-3ce2-11f1-8608-f28f846166bd
2026-04-20 19:01:09,149 - distributed.scheduler - INFO - Close client connection: Client-worker-ec65ff40-3ce2-11f1-84be-8288bfee4388
2026-04-20 19:01:09,147 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:38991', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,144 - distributed.scheduler - INFO - Remove client Client-worker-ec65ff40-3ce2-11f1-84be-8288bfee4388
2026-04-20 19:01:09,087 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:44171', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,087 - distributed.scheduler - INFO - Close client connection: Client-worker-376f8407-3ceb-11f1-850a-86acc56fd09f
2026-04-20 19:01:09,086 - distributed.scheduler - INFO - Remove client Client-worker-376f8407-3ceb-11f1-850a-86acc56fd09f
2026-04-20 19:01:09,071 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:34955', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,057 - distributed.scheduler - INFO - Close client connection: Client-worker-047a0550-3ce7-11f1-849d-2eb33e86e1bd
2026-04-20 19:01:09,056 - distributed.scheduler - INFO - Close client connection: Client-worker-d0b76ead-3ce2-11f1-852d-2e20a4008324
2026-04-20 19:01:09,055 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:32831', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,054 - distributed.scheduler - INFO - Remove client Client-worker-047a0550-3ce7-11f1-849d-2eb33e86e1bd
2026-04-20 19:01:09,032 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:46409', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:09,031 - distributed.scheduler - INFO - Remove client Client-worker-d0b76ead-3ce2-11f1-852d-2e20a4008324
2026-04-20 19:01:08,991 - distributed.scheduler - INFO - Close client connection: Client-worker-04eedaa4-3ce7-11f1-845e-0e4c34ff5e38
2026-04-20 19:01:08,989 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:40907', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:08,975 - distributed.scheduler - INFO - Remove client Client-worker-04eedaa4-3ce7-11f1-845e-0e4c34ff5e38
2026-04-20 19:01:08,959 - distributed.scheduler - INFO - Close client connection: Client-worker-046cfc63-3ce7-11f1-8533-fae0fcb74df3
2026-04-20 19:01:08,958 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:34227', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:08,958 - distributed.scheduler - INFO - Remove client Client-worker-046cfc63-3ce7-11f1-8533-fae0fcb74df3
2026-04-20 19:01:08,954 - distributed.scheduler - INFO - Close client connection: Client-worker-d0f0264a-3ce2-11f1-8551-b6f02dd406e9
2026-04-20 19:01:08,954 - distributed.scheduler - INFO - Close client connection: Client-worker-d0dd84b6-3ce2-11f1-8553-4ef3162b8ac2
2026-04-20 19:01:08,951 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:39993', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:08,951 - distributed.scheduler - INFO - Remove client Client-worker-d0f0264a-3ce2-11f1-8551-b6f02dd406e9
2026-04-20 19:01:08,941 - distributed.scheduler - INFO - Close client connection: Client-worker-384fbdba-3ceb-11f1-84f3-566b4744c386
2026-04-20 19:01:08,939 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:35803', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:08,938 - distributed.scheduler - INFO - Remove client Client-worker-d0dd84b6-3ce2-11f1-8553-4ef3162b8ac2
2026-04-20 19:01:08,927 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:42535', status: closed, memory: 0, processing: 0>
2026-04-20 19:01:08,925 - distributed.scheduler - INFO - Remove client Client-worker-384fbdba-3ceb-11f1-84f3-566b4744c386
2026-04-20 19:00:38,505 - distributed.scheduler - INFO - Receive client connection: Client-worker-384fbdba-3ceb-11f1-84f3-566b4744c386
2026-04-20 19:00:37,035 - distributed.scheduler - INFO - Receive client connection: Client-worker-376f8407-3ceb-11f1-850a-86acc56fd09f
2026-04-20 19:00:33,785 - distributed.scheduler - INFO - Receive client connection: Client-357defe4-3ceb-11f1-8001-daa6852bd5a7
2026-04-20 18:50:32,267 - distributed.scheduler - INFO - Receive client connection: Client-cef5b204-3ce9-11f1-8001-daa6852bd5a7
2026-04-20 18:40:30,800 - distributed.scheduler - INFO - Receive client connection: Client-6874dcdf-3ce8-11f1-8001-daa6852bd5a7
2026-04-20 18:30:34,333 - distributed.scheduler - INFO - Receive client connection: Client-worker-04eedaa4-3ce7-11f1-845e-0e4c34ff5e38
2026-04-20 18:30:34,265 - distributed.scheduler - INFO - Receive client connection: Client-worker-04e1cae4-3ce7-11f1-8493-2a51d3dc178e
2026-04-20 18:30:33,560 - distributed.scheduler - INFO - Receive client connection: Client-worker-047a0550-3ce7-11f1-849d-2eb33e86e1bd
2026-04-20 18:30:33,485 - distributed.scheduler - INFO - Receive client connection: Client-worker-046cfc63-3ce7-11f1-8533-fae0fcb74df3
2026-04-20 18:30:29,332 - distributed.scheduler - INFO - Receive client connection: Client-01f3ea8a-3ce7-11f1-8001-daa6852bd5a7
2026-04-20 18:20:27,811 - distributed.scheduler - INFO - Receive client connection: Client-9b6bfe3b-3ce5-11f1-8001-daa6852bd5a7
2026-04-20 18:14:10,539 - distributed.scheduler - INFO - Receive client connection: Client-ba8cdd32-3ce4-11f1-8001-daa6852bd5a7
2026-04-20 18:01:15,189 - distributed.scheduler - INFO - Receive client connection: Client-worker-ec65ff40-3ce2-11f1-84be-8288bfee4388
2026-04-20 18:00:30,737 - distributed.scheduler - INFO - Receive client connection: Client-worker-d1a580a5-3ce2-11f1-8507-ee7435aaba1f
2026-04-20 18:00:30,004 - distributed.scheduler - INFO - Receive client connection: Client-worker-d1784c6b-3ce2-11f1-8506-ae00acbfc48e
2026-04-20 18:00:29,891 - distributed.scheduler - INFO - Receive client connection: Client-worker-d1667d07-3ce2-11f1-8608-f28f846166bd
2026-04-20 18:00:29,119 - distributed.scheduler - INFO - Receive client connection: Client-worker-d0f0264a-3ce2-11f1-8551-b6f02dd406e9
2026-04-20 18:00:29,028 - distributed.scheduler - INFO - Receive client connection: Client-worker-d0e58780-3ce2-11f1-8488-c2349ba99f21
2026-04-20 18:00:28,975 - distributed.scheduler - INFO - Receive client connection: Client-worker-d0dd84b6-3ce2-11f1-8553-4ef3162b8ac2
2026-04-20 18:00:28,726 - distributed.scheduler - INFO - Receive client connection: Client-worker-d0b76ead-3ce2-11f1-852d-2e20a4008324
2026-04-20 18:00:24,745 - distributed.scheduler - INFO - Receive client connection: Client-ce561dbe-3ce2-11f1-8001-daa6852bd5a7
2026-04-20 17:50:23,153 - distributed.scheduler - INFO - Receive client connection: Client-67c2c5f6-3ce1-11f1-8001-daa6852bd5a7
2026-04-20 17:40:21,645 - distributed.scheduler - INFO - Receive client connection: Client-013b747d-3ce0-11f1-8001-daa6852bd5a7
2026-04-20 17:32:23,401 - distributed.scheduler - INFO - Receive client connection: Client-e42ccb9e-3cde-11f1-8007-924c5ad31268
2026-04-20 17:32:21,884 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:32805
2026-04-20 17:32:21,883 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:32805', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:21,446 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:41547
2026-04-20 17:32:21,445 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:41547', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:21,423 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:34227
2026-04-20 17:32:21,422 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:34227', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:21,333 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:39993
2026-04-20 17:32:21,332 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:39993', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:21,250 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:42371
2026-04-20 17:32:21,249 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:42371', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:21,247 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:42535
2026-04-20 17:32:21,245 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:42535', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:20,932 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:40907
2026-04-20 17:32:20,930 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:40907', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:20,924 - distributed.scheduler - INFO - Close client connection: Client-2ee4ebf4-37bf-11f1-8007-924c5ad31268
2026-04-20 17:32:20,924 - distributed.scheduler - INFO - Remove client Client-2ee4ebf4-37bf-11f1-8007-924c5ad31268
2026-04-20 17:32:20,923 - distributed.scheduler - INFO - Remove client Client-2ee4ebf4-37bf-11f1-8007-924c5ad31268
2026-04-20 17:32:20,914 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:43265
2026-04-20 17:32:20,913 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:43265', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:20,909 - distributed.scheduler - INFO - Close client connection: Client-worker-da612d25-3cde-11f1-84f2-ae00acbfc48e
2026-04-20 17:32:20,904 - distributed.scheduler - INFO - Remove client Client-worker-da612d25-3cde-11f1-84f2-ae00acbfc48e
2026-04-20 17:32:20,901 - distributed.scheduler - INFO - Task run-5e4b85d5410bc9598538fa90ecb0afd6 marked as failed because 3 workers died while trying to run it
2026-04-20 17:32:20,900 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:39535', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:20,566 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:46447', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:20,524 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:42091', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:20,449 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:44417', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:20,343 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:42381', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:20,178 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:36011', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:20,073 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:34179', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:19,886 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:46067', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,964 - distributed.scheduler - INFO - Receive client connection: Client-worker-da612d25-3cde-11f1-84f2-ae00acbfc48e
2026-04-20 17:32:06,948 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:46409
2026-04-20 17:32:06,947 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:46409', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,936 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:42091
2026-04-20 17:32:06,934 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:42091', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,861 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:35803
2026-04-20 17:32:06,860 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:35803', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,837 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:44171
2026-04-20 17:32:06,837 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:44171', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,707 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:42381
2026-04-20 17:32:06,706 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:42381', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,694 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:35911
2026-04-20 17:32:06,693 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:35911', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,595 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:46057
2026-04-20 17:32:06,594 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:46057', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,593 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:32831
2026-04-20 17:32:06,592 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:32831', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,586 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:46447
2026-04-20 17:32:06,585 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:46447', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,582 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:34179
2026-04-20 17:32:06,581 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:34179', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,498 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:44417
2026-04-20 17:32:06,496 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:44417', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,440 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:34955
2026-04-20 17:32:06,439 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:34955', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,336 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:36011
2026-04-20 17:32:06,335 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:36011', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,199 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:38991
2026-04-20 17:32:06,198 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:38991', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,196 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:46067
2026-04-20 17:32:06,195 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:46067', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:06,187 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:39535
2026-04-20 17:32:06,186 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:39535', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,964 - distributed.scheduler - INFO - Lost all workers
2026-04-20 17:32:05,960 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:40811', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,910 - distributed.scheduler - INFO - Close client connection: Client-worker-b49d4727-3cde-11f1-84ef-2e20a4008324
2026-04-20 17:32:05,907 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:35609', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,905 - distributed.scheduler - INFO - Remove client Client-worker-b49d4727-3cde-11f1-84ef-2e20a4008324
2026-04-20 17:32:05,824 - distributed.scheduler - INFO - Close client connection: Client-worker-b434eece-3cde-11f1-84cb-86acc56fd09f
2026-04-20 17:32:05,818 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:40489', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,814 - distributed.scheduler - INFO - Remove client Client-worker-b434eece-3cde-11f1-84cb-86acc56fd09f
2026-04-20 17:32:05,812 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:39783', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,714 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:43781', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,697 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:44567', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,672 - distributed.scheduler - INFO - Close client connection: Client-worker-b5148fad-3cde-11f1-845b-2eb33e86e1bd
2026-04-20 17:32:05,665 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:44691', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,664 - distributed.scheduler - INFO - Remove client Client-worker-b5148fad-3cde-11f1-845b-2eb33e86e1bd
2026-04-20 17:32:05,582 - distributed.scheduler - INFO - Close client connection: Client-worker-b4109300-3cde-11f1-8443-2a51d3dc178e
2026-04-20 17:32:05,578 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:42413', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,578 - distributed.scheduler - INFO - Remove client Client-worker-b4109300-3cde-11f1-8443-2a51d3dc178e
2026-04-20 17:32:05,573 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:40467', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,571 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:46227', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,560 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:41409', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,543 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:35055', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,379 - distributed.scheduler - INFO - Close client connection: Client-worker-b3956a8c-3cde-11f1-84b1-ae00acbfc48e
2026-04-20 17:32:05,361 - distributed.scheduler - INFO - Remove client Client-worker-b3956a8c-3cde-11f1-84b1-ae00acbfc48e
2026-04-20 17:32:05,358 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:44709', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,357 - distributed.scheduler - INFO - Close client connection: Client-worker-b443b00c-3cde-11f1-85bc-f28f846166bd
2026-04-20 17:32:05,349 - distributed.scheduler - INFO - Remove client Client-worker-b443b00c-3cde-11f1-85bc-f28f846166bd
2026-04-20 17:32:05,345 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:37491', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,256 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:37377', status: closed, memory: 0, processing: 0>
2026-04-20 17:32:05,105 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:40591', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:04,445 - distributed.scheduler - INFO - Receive client connection: Client-worker-b5148fad-3cde-11f1-845b-2eb33e86e1bd
2026-04-20 17:31:03,601 - distributed.scheduler - INFO - Receive client connection: Client-worker-b49d4727-3cde-11f1-84ef-2e20a4008324
2026-04-20 17:31:03,258 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:40591
2026-04-20 17:31:03,257 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:40591', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:03,024 - distributed.scheduler - INFO - Receive client connection: Client-worker-b443b00c-3cde-11f1-85bc-f28f846166bd
2026-04-20 17:31:02,930 - distributed.scheduler - INFO - Receive client connection: Client-worker-b434eece-3cde-11f1-84cb-86acc56fd09f
2026-04-20 17:31:02,684 - distributed.scheduler - INFO - Receive client connection: Client-worker-b4109300-3cde-11f1-8443-2a51d3dc178e
2026-04-20 17:31:02,511 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:44567
2026-04-20 17:31:02,510 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:44567', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:02,498 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:40489
2026-04-20 17:31:02,496 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:40489', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:02,328 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:40467
2026-04-20 17:31:02,326 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:40467', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:02,237 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:40811
2026-04-20 17:31:02,236 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:40811', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:02,221 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:43781
2026-04-20 17:31:02,221 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:43781', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:02,137 - distributed.scheduler - INFO - Close client connection: Client-worker-7e732625-3cda-11f1-8480-ee7435aaba1f
2026-04-20 17:31:02,129 - distributed.scheduler - INFO - Remove client Client-worker-7e732625-3cda-11f1-8480-ee7435aaba1f
2026-04-20 17:31:02,124 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:37003', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:02,094 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:44691
2026-04-20 17:31:02,093 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:44691', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:02,043 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:37491
2026-04-20 17:31:02,043 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:37491', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,992 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:41409
2026-04-20 17:31:01,991 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:41409', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,940 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:35055
2026-04-20 17:31:01,939 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:35055', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,876 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:46227
2026-04-20 17:31:01,875 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:46227', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,865 - distributed.scheduler - INFO - Receive client connection: Client-worker-b3956a8c-3cde-11f1-84b1-ae00acbfc48e
2026-04-20 17:31:01,609 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:39783
2026-04-20 17:31:01,608 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:39783', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,441 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:37377
2026-04-20 17:31:01,440 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:37377', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,428 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:35781', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,424 - distributed.scheduler - INFO - Close client connection: Client-worker-9d5fbc04-3cde-11f1-84da-4ef3162b8ac2
2026-04-20 17:31:01,421 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:43271', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,420 - distributed.scheduler - INFO - Remove client Client-worker-9d5fbc04-3cde-11f1-84da-4ef3162b8ac2
2026-04-20 17:31:01,377 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:35367', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,331 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:35609
2026-04-20 17:31:01,330 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:35609', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,311 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:42413
2026-04-20 17:31:01,310 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:42413', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,287 - distributed.scheduler - INFO - Close client connection: Client-worker-9dcba5da-3cde-11f1-84ac-fae0fcb74df3
2026-04-20 17:31:01,280 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:38839', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,278 - distributed.scheduler - INFO - Remove client Client-worker-9dcba5da-3cde-11f1-84ac-fae0fcb74df3
2026-04-20 17:31:01,276 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:44709
2026-04-20 17:31:01,274 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:44709', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,223 - distributed.scheduler - INFO - Close client connection: Client-worker-9df5f319-3cde-11f1-8472-566b4744c386
2026-04-20 17:31:01,207 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:42941', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,205 - distributed.scheduler - INFO - Remove client Client-worker-9df5f319-3cde-11f1-8472-566b4744c386
2026-04-20 17:31:01,108 - distributed.scheduler - INFO - Close client connection: Client-worker-9d3a9261-3cde-11f1-8580-f28f846166bd
2026-04-20 17:31:01,105 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:43659', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,104 - distributed.scheduler - INFO - Remove client Client-worker-9d3a9261-3cde-11f1-8580-f28f846166bd
2026-04-20 17:31:01,093 - distributed.scheduler - INFO - Task ('sjoin-f9a2051a35f5e80bd3ef025f46b84f53', 158) marked as failed because 3 workers died while trying to run it
2026-04-20 17:31:01,093 - distributed.scheduler - INFO - Task ('sjoin-f9a2051a35f5e80bd3ef025f46b84f53', 154) marked as failed because 3 workers died while trying to run it
2026-04-20 17:31:01,092 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:44117', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,086 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:44459', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,082 - distributed.scheduler - INFO - Close client connection: Client-worker-7fe834a3-3cda-11f1-8449-7e353d94dc10
2026-04-20 17:31:01,074 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:38905', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:01,073 - distributed.scheduler - INFO - Remove client Client-worker-7fe834a3-3cda-11f1-8449-7e353d94dc10
2026-04-20 17:31:00,964 - distributed.scheduler - INFO - Close client connection: Client-worker-801b0330-3cda-11f1-83db-0e4c34ff5e38
2026-04-20 17:31:00,954 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:46369', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:00,936 - distributed.scheduler - INFO - Remove client Client-worker-801b0330-3cda-11f1-83db-0e4c34ff5e38
2026-04-20 17:31:00,581 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:40295', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:00,411 - distributed.scheduler - INFO - Close client connection: Client-worker-7d8b8d7c-3cda-11f1-8442-8288bfee4388
2026-04-20 17:31:00,392 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:42913', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:00,362 - distributed.scheduler - INFO - Remove client Client-worker-7d8b8d7c-3cda-11f1-8442-8288bfee4388
2026-04-20 17:31:00,269 - distributed.scheduler - INFO - Close client connection: Client-worker-804d81c6-3cda-11f1-8401-2a51d3dc178e
2026-04-20 17:31:00,262 - distributed.scheduler - INFO - Remove client Client-worker-804d81c6-3cda-11f1-8401-2a51d3dc178e
2026-04-20 17:31:00,254 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:34981', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:00,187 - distributed.scheduler - INFO - Close client connection: Client-worker-7fee5977-3cda-11f1-8470-ae00acbfc48e
2026-04-20 17:31:00,174 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:37351', status: closed, memory: 0, processing: 0>
2026-04-20 17:31:00,173 - distributed.scheduler - INFO - Remove client Client-worker-7fee5977-3cda-11f1-8470-ae00acbfc48e
2026-04-20 17:31:00,134 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:37975', status: closed, memory: 0, processing: 0>
2026-04-20 17:30:25,628 - distributed.scheduler - INFO - Receive client connection: Client-worker-9df5f319-3cde-11f1-8472-566b4744c386
2026-04-20 17:30:25,368 - distributed.scheduler - INFO - Receive client connection: Client-worker-9dcba5da-3cde-11f1-84ac-fae0fcb74df3
2026-04-20 17:30:24,610 - distributed.scheduler - INFO - Receive client connection: Client-worker-9d5fbc04-3cde-11f1-84da-4ef3162b8ac2
2026-04-20 17:30:24,361 - distributed.scheduler - INFO - Receive client connection: Client-worker-9d3a9261-3cde-11f1-8580-f28f846166bd
2026-04-20 17:30:20,134 - distributed.scheduler - INFO - Receive client connection: Client-9ab410fd-3cde-11f1-8001-daa6852bd5a7
2026-04-20 17:20:18,633 - distributed.scheduler - INFO - Receive client connection: Client-342ed6b9-3cdd-11f1-8001-daa6852bd5a7
2026-04-20 17:12:26,809 - distributed.scheduler - INFO - Receive client connection: Client-1af4e3fb-3cdc-11f1-8001-daa6852bd5a7
2026-04-20 17:00:57,861 - distributed.scheduler - INFO - Receive client connection: Client-worker-804d81c6-3cda-11f1-8401-2a51d3dc178e
2026-04-20 17:00:57,533 - distributed.scheduler - INFO - Receive client connection: Client-worker-801b0330-3cda-11f1-83db-0e4c34ff5e38
2026-04-20 17:00:57,280 - distributed.scheduler - INFO - Receive client connection: Client-worker-7fee5977-3cda-11f1-8470-ae00acbfc48e
2026-04-20 17:00:57,237 - distributed.scheduler - INFO - Receive client connection: Client-worker-7fe834a3-3cda-11f1-8449-7e353d94dc10
2026-04-20 17:00:57,066 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:35781
2026-04-20 17:00:57,064 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:35781', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:57,047 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:40295
2026-04-20 17:00:57,046 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:40295', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:56,743 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:37975
2026-04-20 17:00:56,741 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:37975', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:56,066 - distributed.scheduler - INFO - Close client connection: Client-worker-69d696e4-3cda-11f1-83d7-c2349ba99f21
2026-04-20 17:00:56,040 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:36329', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:56,022 - distributed.scheduler - INFO - Remove client Client-worker-69d696e4-3cda-11f1-83d7-c2349ba99f21
2026-04-20 17:00:55,924 - distributed.scheduler - INFO - Close client connection: Client-worker-362382c0-3cd6-11f1-844c-86acc56fd09f
2026-04-20 17:00:55,896 - distributed.scheduler - INFO - Task ('simple-shuffle-cd00325d172e834747e62e0613d21199', 13) marked as failed because 3 workers died while trying to run it
2026-04-20 17:00:55,895 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:43103', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:55,894 - distributed.scheduler - INFO - Remove client Client-worker-362382c0-3cd6-11f1-844c-86acc56fd09f
2026-04-20 17:00:55,749 - distributed.scheduler - INFO - Close client connection: Client-worker-3601a9fe-3cd6-11f1-8471-2e20a4008324
2026-04-20 17:00:55,730 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:43587', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:55,729 - distributed.scheduler - INFO - Remove client Client-worker-3601a9fe-3cd6-11f1-8471-2e20a4008324
2026-04-20 17:00:54,852 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:46369
2026-04-20 17:00:54,851 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:46369', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:54,735 - distributed.scheduler - INFO - Receive client connection: Client-worker-7e732625-3cda-11f1-8480-ee7435aaba1f
2026-04-20 17:00:54,713 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:44117
2026-04-20 17:00:54,711 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:44117', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:54,620 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:43659
2026-04-20 17:00:54,618 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:43659', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:54,519 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:35367
2026-04-20 17:00:54,517 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:35367', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:54,455 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:38839
2026-04-20 17:00:54,454 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:38839', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:54,384 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:38905
2026-04-20 17:00:54,383 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:38905', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:54,281 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:44459
2026-04-20 17:00:54,280 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:44459', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:54,057 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:37351
2026-04-20 17:00:54,056 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:37351', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:54,043 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:34981
2026-04-20 17:00:54,042 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:34981', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:53,987 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:43271
2026-04-20 17:00:53,986 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:43271', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:53,958 - distributed.scheduler - INFO - Close client connection: Client-worker-3618c5f6-3cd6-11f1-839c-0e4c34ff5e38
2026-04-20 17:00:53,950 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:45245', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:53,949 - distributed.scheduler - INFO - Remove client Client-worker-3618c5f6-3cd6-11f1-839c-0e4c34ff5e38
2026-04-20 17:00:53,886 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:37003
2026-04-20 17:00:53,885 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:37003', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:53,805 - distributed.scheduler - INFO - Close client connection: Client-worker-36c33b86-3cd6-11f1-83df-2eb33e86e1bd
2026-04-20 17:00:53,794 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:34315', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:53,793 - distributed.scheduler - INFO - Remove client Client-worker-36c33b86-3cd6-11f1-83df-2eb33e86e1bd
2026-04-20 17:00:53,791 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:42941
2026-04-20 17:00:53,790 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:42941', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:53,598 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:46381', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:53,592 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:34999', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:53,579 - distributed.scheduler - INFO - Close client connection: Client-worker-6a77a68c-3cda-11f1-8547-f28f846166bd
2026-04-20 17:00:53,572 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:39257', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:53,569 - distributed.scheduler - INFO - Remove client Client-worker-6a77a68c-3cda-11f1-8547-f28f846166bd
2026-04-20 17:00:53,517 - distributed.scheduler - INFO - Close client connection: Client-worker-363cab48-3cd6-11f1-8407-7e353d94dc10
2026-04-20 17:00:53,508 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:34485', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:53,476 - distributed.scheduler - INFO - Remove client Client-worker-363cab48-3cd6-11f1-8407-7e353d94dc10
2026-04-20 17:00:53,281 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:37583', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:53,263 - distributed.scheduler - INFO - Receive client connection: Client-worker-7d8b8d7c-3cda-11f1-8442-8288bfee4388
2026-04-20 17:00:53,074 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:42363', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:53,020 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:42953', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:52,988 - distributed.scheduler - INFO - Close client connection: Client-worker-36b06dc2-3cd6-11f1-83c8-2a51d3dc178e
2026-04-20 17:00:52,984 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:38347', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:52,983 - distributed.scheduler - INFO - Remove client Client-worker-36b06dc2-3cd6-11f1-83c8-2a51d3dc178e
2026-04-20 17:00:52,861 - distributed.scheduler - INFO - Close client connection: Client-worker-417715e5-3cd2-11f1-8432-566b4744c386
2026-04-20 17:00:52,844 - distributed.scheduler - INFO - Remove client Client-worker-417715e5-3cd2-11f1-8432-566b4744c386
2026-04-20 17:00:52,840 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:43191', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:52,835 - distributed.scheduler - INFO - Close client connection: Client-worker-36e57144-3cd6-11f1-8443-ee7435aaba1f
2026-04-20 17:00:52,817 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:39853', status: closed, memory: 0, processing: 0>
2026-04-20 17:00:52,814 - distributed.scheduler - INFO - Remove client Client-worker-36e57144-3cd6-11f1-8443-ee7435aaba1f
2026-04-20 17:00:21,292 - distributed.scheduler - INFO - Receive client connection: Client-worker-6a77a68c-3cda-11f1-8547-f28f846166bd
2026-04-20 17:00:20,166 - distributed.scheduler - INFO - Receive client connection: Client-worker-69d696e4-3cda-11f1-83d7-c2349ba99f21
2026-04-20 17:00:15,683 - distributed.scheduler - INFO - Receive client connection: Client-672956d1-3cda-11f1-8001-daa6852bd5a7
2026-04-20 16:50:50,755 - distributed.scheduler - INFO - Close client connection: Client-009bf330-3cd9-11f1-8001-daa6852bd5a7
2026-04-20 16:50:50,754 - distributed.scheduler - INFO - Remove client Client-009bf330-3cd9-11f1-8001-daa6852bd5a7
2026-04-20 16:50:50,752 - distributed.scheduler - INFO - Remove client Client-009bf330-3cd9-11f1-8001-daa6852bd5a7
2026-04-20 16:50:14,124 - distributed.scheduler - INFO - Receive client connection: Client-009bf330-3cd9-11f1-8001-daa6852bd5a7
2026-04-20 16:40:48,501 - distributed.scheduler - INFO - Close client connection: Client-9a135548-3cd7-11f1-8001-daa6852bd5a7
2026-04-20 16:40:48,500 - distributed.scheduler - INFO - Remove client Client-9a135548-3cd7-11f1-8001-daa6852bd5a7
2026-04-20 16:40:48,498 - distributed.scheduler - INFO - Remove client Client-9a135548-3cd7-11f1-8001-daa6852bd5a7
2026-04-20 16:40:12,610 - distributed.scheduler - INFO - Receive client connection: Client-9a135548-3cd7-11f1-8001-daa6852bd5a7
2026-04-20 16:31:13,580 - distributed.scheduler - INFO - Close client connection: Client-33a2c2c1-3cd6-11f1-8001-daa6852bd5a7
2026-04-20 16:31:13,580 - distributed.scheduler - INFO - Remove client Client-33a2c2c1-3cd6-11f1-8001-daa6852bd5a7
2026-04-20 16:31:13,578 - distributed.scheduler - INFO - Remove client Client-33a2c2c1-3cd6-11f1-8001-daa6852bd5a7
2026-04-20 16:30:16,711 - distributed.scheduler - INFO - Receive client connection: Client-worker-36e57144-3cd6-11f1-8443-ee7435aaba1f
2026-04-20 16:30:16,480 - distributed.scheduler - INFO - Receive client connection: Client-worker-36c33b86-3cd6-11f1-83df-2eb33e86e1bd
2026-04-20 16:30:16,368 - distributed.scheduler - INFO - Receive client connection: Client-worker-36b06dc2-3cd6-11f1-83c8-2a51d3dc178e
2026-04-20 16:30:15,622 - distributed.scheduler - INFO - Receive client connection: Client-worker-363cab48-3cd6-11f1-8407-7e353d94dc10
2026-04-20 16:30:15,462 - distributed.scheduler - INFO - Receive client connection: Client-worker-362382c0-3cd6-11f1-844c-86acc56fd09f
2026-04-20 16:30:15,369 - distributed.scheduler - INFO - Receive client connection: Client-worker-3618c5f6-3cd6-11f1-839c-0e4c34ff5e38
2026-04-20 16:30:15,203 - distributed.scheduler - INFO - Receive client connection: Client-worker-3601a9fe-3cd6-11f1-8471-2e20a4008324
2026-04-20 16:30:11,262 - distributed.scheduler - INFO - Receive client connection: Client-33a2c2c1-3cd6-11f1-8001-daa6852bd5a7
2026-04-20 16:20:45,934 - distributed.scheduler - INFO - Close client connection: Client-cd28fc5a-3cd4-11f1-8001-daa6852bd5a7
2026-04-20 16:20:45,932 - distributed.scheduler - INFO - Remove client Client-cd28fc5a-3cd4-11f1-8001-daa6852bd5a7
2026-04-20 16:20:45,926 - distributed.scheduler - INFO - Remove client Client-cd28fc5a-3cd4-11f1-8001-daa6852bd5a7
2026-04-20 16:20:09,827 - distributed.scheduler - INFO - Receive client connection: Client-cd28fc5a-3cd4-11f1-8001-daa6852bd5a7
2026-04-20 16:12:47,987 - distributed.scheduler - INFO - Close client connection: Client-b0f9262f-3cd3-11f1-8001-daa6852bd5a7
2026-04-20 16:12:47,986 - distributed.scheduler - INFO - Remove client Client-b0f9262f-3cd3-11f1-8001-daa6852bd5a7
2026-04-20 16:12:47,984 - distributed.scheduler - INFO - Remove client Client-b0f9262f-3cd3-11f1-8001-daa6852bd5a7
2026-04-20 16:12:13,039 - distributed.scheduler - INFO - Receive client connection: Client-b0f9262f-3cd3-11f1-8001-daa6852bd5a7
2026-04-20 16:01:57,668 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:39853
2026-04-20 16:01:57,667 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:39853', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:57,483 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:36329
2026-04-20 16:01:57,482 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:36329', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:56,611 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:38191', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:56,453 - distributed.scheduler - INFO - Receive client connection: Client-worker-417715e5-3cd2-11f1-8432-566b4744c386
2026-04-20 16:01:56,379 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:41981', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:56,033 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:43587
2026-04-20 16:01:56,032 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:43587', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,999 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:39257
2026-04-20 16:01:55,998 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:39257', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,985 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:37583
2026-04-20 16:01:55,984 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:37583', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,943 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:34485
2026-04-20 16:01:55,942 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:34485', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,896 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:45245
2026-04-20 16:01:55,895 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:45245', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,891 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:42363
2026-04-20 16:01:55,890 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:42363', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,888 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:34315
2026-04-20 16:01:55,886 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:34315', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,876 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:46381
2026-04-20 16:01:55,874 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:46381', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,858 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:43103
2026-04-20 16:01:55,857 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:43103', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,836 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:42913
2026-04-20 16:01:55,835 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:42913', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,821 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:38347
2026-04-20 16:01:55,820 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:38347', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,775 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:34999
2026-04-20 16:01:55,773 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:34999', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,712 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:42953
2026-04-20 16:01:55,710 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:42953', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,644 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:43191
2026-04-20 16:01:55,642 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:43191', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,050 - distributed.scheduler - INFO - Close client connection: Client-worker-1c94cea6-3cd2-11f1-850b-f28f846166bd
2026-04-20 16:01:55,048 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:32917', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,047 - distributed.scheduler - INFO - Remove client Client-worker-1c94cea6-3cd2-11f1-850b-f28f846166bd
2026-04-20 16:01:55,040 - distributed.scheduler - INFO - Close client connection: Client-worker-1c81b43f-3cd2-11f1-835c-0e4c34ff5e38
2026-04-20 16:01:55,038 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:38339', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:55,037 - distributed.scheduler - INFO - Remove client Client-worker-1c81b43f-3cd2-11f1-835c-0e4c34ff5e38
2026-04-20 16:01:54,980 - distributed.scheduler - INFO - Close client connection: Client-worker-1b4296a5-3cd2-11f1-83c1-7e353d94dc10
2026-04-20 16:01:54,978 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:37089', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:54,978 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:33695', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:54,975 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:34429', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:54,941 - distributed.scheduler - INFO - Remove client Client-worker-1b4296a5-3cd2-11f1-83c1-7e353d94dc10
2026-04-20 16:01:54,889 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:45969', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:54,877 - distributed.scheduler - INFO - Close client connection: Client-worker-1c25eeea-3cd2-11f1-83f7-ae00acbfc48e
2026-04-20 16:01:54,875 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:37439', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:54,873 - distributed.scheduler - INFO - Remove client Client-worker-1c25eeea-3cd2-11f1-83f7-ae00acbfc48e
2026-04-20 16:01:54,866 - distributed.scheduler - INFO - Close client connection: Client-worker-1d811024-3cd2-11f1-8409-8288bfee4388
2026-04-20 16:01:54,861 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:40675', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:54,860 - distributed.scheduler - INFO - Remove client Client-worker-1d811024-3cd2-11f1-8409-8288bfee4388
2026-04-20 16:01:54,850 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:34745', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:54,829 - distributed.scheduler - INFO - Close client connection: Client-worker-1b2c91a6-3cd2-11f1-8459-b6f02dd406e9
2026-04-20 16:01:54,827 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:44365', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:54,826 - distributed.scheduler - INFO - Remove client Client-worker-1b2c91a6-3cd2-11f1-8459-b6f02dd406e9
2026-04-20 16:01:54,804 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:46373', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:54,764 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:37241', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:54,728 - distributed.scheduler - INFO - Close client connection: Client-worker-1a774f7e-3cd2-11f1-8467-4ef3162b8ac2
2026-04-20 16:01:54,722 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:46639', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:54,722 - distributed.scheduler - INFO - Remove client Client-worker-1a774f7e-3cd2-11f1-8467-4ef3162b8ac2
2026-04-20 16:01:54,711 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:37567', status: closed, memory: 0, processing: 0>
2026-04-20 16:01:48,089 - distributed.scheduler - INFO - Close client connection: Client-66346800-3ccc-11f1-8001-daa6852bd5a7
2026-04-20 16:01:48,089 - distributed.scheduler - INFO - Close client connection: Client-488abd01-3ccb-11f1-8001-daa6852bd5a7
2026-04-20 16:01:48,089 - distributed.scheduler - INFO - Close client connection: Client-ccb4bfca-3ccd-11f1-8001-daa6852bd5a7
2026-04-20 16:01:48,088 - distributed.scheduler - INFO - Close client connection: Client-333383c3-3ccf-11f1-8001-daa6852bd5a7
2026-04-20 16:01:48,088 - distributed.scheduler - INFO - Close client connection: Client-99ac6eda-3cd0-11f1-8001-daa6852bd5a7
2026-04-20 16:01:48,087 - distributed.scheduler - INFO - Remove client Client-66346800-3ccc-11f1-8001-daa6852bd5a7
2026-04-20 16:01:48,087 - distributed.scheduler - INFO - Remove client Client-488abd01-3ccb-11f1-8001-daa6852bd5a7
2026-04-20 16:01:48,087 - distributed.scheduler - INFO - Remove client Client-ccb4bfca-3ccd-11f1-8001-daa6852bd5a7
2026-04-20 16:01:48,086 - distributed.scheduler - INFO - Remove client Client-333383c3-3ccf-11f1-8001-daa6852bd5a7
2026-04-20 16:01:48,086 - distributed.scheduler - INFO - Remove client Client-99ac6eda-3cd0-11f1-8001-daa6852bd5a7
2026-04-20 16:01:47,291 - distributed.scheduler - INFO - Close client connection: Client-002a4f25-3cd2-11f1-8001-daa6852bd5a7
2026-04-20 16:01:47,286 - distributed.scheduler - INFO - Remove client Client-002a4f25-3cd2-11f1-8001-daa6852bd5a7
2026-04-20 16:01:47,284 - distributed.scheduler - INFO - Remove client Client-002a4f25-3cd2-11f1-8001-daa6852bd5a7
2026-04-20 16:00:56,258 - distributed.scheduler - INFO - Receive client connection: Client-worker-1d811024-3cd2-11f1-8409-8288bfee4388
2026-04-20 16:00:55,546 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:41981
2026-04-20 16:00:55,545 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:41981', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:55,406 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:37241
2026-04-20 16:00:55,405 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:37241', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:54,676 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('len-agg-ea1961682e7f8c6981ad6830a91f6ceb', 0) NoneType: None
2026-04-20 16:00:54,676 - distributed.scheduler - ERROR - Couldn't gather keys {"('len-agg-ea1961682e7f8c6981ad6830a91f6ceb', 0)": []} state: ['waiting'] workers: [] NoneType: None
2026-04-20 16:00:54,561 - distributed.scheduler - INFO - Receive client connection: Client-worker-1c94cea6-3cd2-11f1-850b-f28f846166bd
2026-04-20 16:00:54,527 - distributed.scheduler - INFO - Close client connection: Client-worker-c035c739-3cc9-11f1-8354-c2349ba99f21
2026-04-20 16:00:54,508 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:46637', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:54,507 - distributed.scheduler - INFO - Remove client Client-worker-c035c739-3cc9-11f1-8354-c2349ba99f21
2026-04-20 16:00:54,462 - distributed.scheduler - INFO - Receive client connection: Client-worker-1c81b43f-3cd2-11f1-835c-0e4c34ff5e38
2026-04-20 16:00:54,281 - distributed.scheduler - INFO - Close client connection: Client-worker-cf7430c9-3ccd-11f1-83cb-86acc56fd09f
2026-04-20 16:00:54,272 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:43229', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:54,270 - distributed.scheduler - INFO - Remove client Client-worker-cf7430c9-3ccd-11f1-83cb-86acc56fd09f
2026-04-20 16:00:53,844 - distributed.scheduler - INFO - Receive client connection: Client-worker-1c25eeea-3cd2-11f1-83f7-ae00acbfc48e
2026-04-20 16:00:52,708 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:33695
2026-04-20 16:00:52,707 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:33695', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:52,563 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:34745
2026-04-20 16:00:52,563 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:34745', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:52,545 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:40675
2026-04-20 16:00:52,543 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:40675', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:52,522 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:38339
2026-04-20 16:00:52,521 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:38339', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:52,357 - distributed.scheduler - INFO - Receive client connection: Client-worker-1b4296a5-3cd2-11f1-83c1-7e353d94dc10
2026-04-20 16:00:52,280 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:32917
2026-04-20 16:00:52,278 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:32917', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:52,208 - distributed.scheduler - INFO - Receive client connection: Client-worker-1b2c91a6-3cd2-11f1-8459-b6f02dd406e9
2026-04-20 16:00:51,791 - distributed.scheduler - INFO - Close client connection: Client-worker-a4945f06-3cc5-11f1-840a-6ec63a5ab21e
2026-04-20 16:00:51,769 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:39767', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:51,768 - distributed.scheduler - INFO - Remove client Client-worker-a4945f06-3cc5-11f1-840a-6ec63a5ab21e
2026-04-20 16:00:51,580 - distributed.scheduler - INFO - Close client connection: Client-worker-cf43d3ac-3ccd-11f1-83c8-8288bfee4388
2026-04-20 16:00:51,578 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:34503', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:51,577 - distributed.scheduler - INFO - Remove client Client-worker-cf43d3ac-3ccd-11f1-83c8-8288bfee4388
2026-04-20 16:00:51,566 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:46745', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:51,562 - distributed.scheduler - INFO - Close client connection: Client-worker-a65d18c8-3cc5-11f1-8350-2a51d3dc178e
2026-04-20 16:00:51,552 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:43995', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:51,552 - distributed.scheduler - INFO - Remove client Client-worker-a65d18c8-3cc5-11f1-8350-2a51d3dc178e
2026-04-20 16:00:51,398 - distributed.scheduler - INFO - Close client connection: Client-worker-a2cfae28-3cc5-11f1-84bb-f28f846166bd
2026-04-20 16:00:51,363 - distributed.scheduler - INFO - Task ('from_pandas-4e5891d616bec63da1296024e3b560d1', 14) marked as failed because 3 workers died while trying to run it
2026-04-20 16:00:51,362 - distributed.scheduler - INFO - Task ('from_pandas-c7290c396563485704814f7826c1fba9', 12) marked as failed because 3 workers died while trying to run it
2026-04-20 16:00:51,362 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:45191', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:51,361 - distributed.scheduler - INFO - Remove client Client-worker-a2cfae28-3cc5-11f1-84bb-f28f846166bd
2026-04-20 16:00:51,070 - distributed.scheduler - INFO - Receive client connection: Client-worker-1a774f7e-3cd2-11f1-8467-4ef3162b8ac2
2026-04-20 16:00:50,992 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:38191
2026-04-20 16:00:50,991 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:38191', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:50,638 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:37439
2026-04-20 16:00:50,637 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:37439', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:50,398 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:46373
2026-04-20 16:00:50,397 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:46373', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:50,344 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:37089
2026-04-20 16:00:50,344 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:37089', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:50,287 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:45969
2026-04-20 16:00:50,286 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:45969', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:50,255 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:37567
2026-04-20 16:00:50,254 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:37567', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:50,230 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:34429
2026-04-20 16:00:50,228 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:34429', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:50,094 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:46639
2026-04-20 16:00:50,093 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:46639', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:49,997 - distributed.scheduler - INFO - Close client connection: Client-worker-cf698e3b-3ccd-11f1-83c1-ee7435aaba1f
2026-04-20 16:00:49,991 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:34365', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:49,988 - distributed.scheduler - INFO - Remove client Client-worker-cf698e3b-3ccd-11f1-83c1-ee7435aaba1f
2026-04-20 16:00:49,933 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:44365
2026-04-20 16:00:49,930 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:44365', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:49,617 - distributed.scheduler - INFO - Close client connection: Client-worker-c03d540e-3cc9-11f1-83bd-ae00acbfc48e
2026-04-20 16:00:49,604 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:34489', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:49,602 - distributed.scheduler - INFO - Remove client Client-worker-c03d540e-3cc9-11f1-83bd-ae00acbfc48e
2026-04-20 16:00:49,390 - distributed.scheduler - INFO - Close client connection: Client-worker-033c18f6-3cd2-11f1-8362-2eb33e86e1bd
2026-04-20 16:00:49,382 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:45709', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:49,373 - distributed.scheduler - INFO - Remove client Client-worker-033c18f6-3cd2-11f1-8362-2eb33e86e1bd
2026-04-20 16:00:49,337 - distributed.scheduler - INFO - Close client connection: Client-worker-a32f9020-3cc5-11f1-83b8-566b4744c386
2026-04-20 16:00:49,331 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:45953', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:49,330 - distributed.scheduler - INFO - Remove client Client-worker-a32f9020-3cc5-11f1-83b8-566b4744c386
2026-04-20 16:00:49,282 - distributed.scheduler - INFO - Close client connection: Client-worker-c02ff873-3cc9-11f1-83fa-2e20a4008324
2026-04-20 16:00:49,278 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:43269', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:49,277 - distributed.scheduler - INFO - Remove client Client-worker-c02ff873-3cc9-11f1-83fa-2e20a4008324
2026-04-20 16:00:49,259 - distributed.scheduler - INFO - Close client connection: Client-worker-a67054e3-3cc5-11f1-83f6-fae0fcb74df3
2026-04-20 16:00:49,248 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:45347', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:49,247 - distributed.scheduler - INFO - Remove client Client-worker-a67054e3-3cc5-11f1-83f6-fae0fcb74df3
2026-04-20 16:00:49,125 - distributed.scheduler - INFO - Close client connection: Client-worker-a348a334-3cc5-11f1-841d-4ef3162b8ac2
2026-04-20 16:00:49,118 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:33689', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:49,093 - distributed.scheduler - INFO - Remove client Client-worker-a348a334-3cc5-11f1-841d-4ef3162b8ac2
2026-04-20 16:00:49,012 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:36229', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:48,992 - distributed.scheduler - INFO - Close client connection: Client-worker-c2266990-3cc5-11f1-837e-7e353d94dc10
2026-04-20 16:00:48,986 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:35485', status: closed, memory: 0, processing: 0>
2026-04-20 16:00:48,984 - distributed.scheduler - INFO - Remove client Client-worker-c2266990-3cc5-11f1-837e-7e353d94dc10
2026-04-20 16:00:12,126 - distributed.scheduler - INFO - Receive client connection: Client-worker-033c18f6-3cd2-11f1-8362-2eb33e86e1bd
2026-04-20 16:00:06,894 - distributed.scheduler - INFO - Receive client connection: Client-002a4f25-3cd2-11f1-8001-daa6852bd5a7
2026-04-20 15:50:05,449 - distributed.scheduler - INFO - Receive client connection: Client-99ac6eda-3cd0-11f1-8001-daa6852bd5a7
2026-04-20 15:40:04,033 - distributed.scheduler - INFO - Receive client connection: Client-333383c3-3ccf-11f1-8001-daa6852bd5a7
2026-04-20 15:30:07,189 - distributed.scheduler - INFO - Receive client connection: Client-worker-cf7430c9-3ccd-11f1-83cb-86acc56fd09f
2026-04-20 15:30:07,117 - distributed.scheduler - INFO - Receive client connection: Client-worker-cf698e3b-3ccd-11f1-83c1-ee7435aaba1f
2026-04-20 15:30:06,867 - distributed.scheduler - INFO - Receive client connection: Client-worker-cf43d3ac-3ccd-11f1-83c8-8288bfee4388
2026-04-20 15:30:02,579 - distributed.scheduler - INFO - Receive client connection: Client-ccb4bfca-3ccd-11f1-8001-daa6852bd5a7
2026-04-20 15:20:01,110 - distributed.scheduler - INFO - Receive client connection: Client-66346800-3ccc-11f1-8001-daa6852bd5a7
2026-04-20 15:12:01,896 - distributed.scheduler - INFO - Receive client connection: Client-488abd01-3ccb-11f1-8001-daa6852bd5a7
2026-04-20 15:02:12,994 - distributed.scheduler - INFO - Close client connection: Client-bd13418a-3cc9-11f1-8001-daa6852bd5a7
2026-04-20 15:02:12,992 - distributed.scheduler - INFO - Remove client Client-bd13418a-3cc9-11f1-8001-daa6852bd5a7
2026-04-20 15:02:12,990 - distributed.scheduler - INFO - Remove client Client-bd13418a-3cc9-11f1-8001-daa6852bd5a7
2026-04-20 15:01:03,730 - distributed.scheduler - INFO - Receive client connection: Client-worker-c03d540e-3cc9-11f1-83bd-ae00acbfc48e
2026-04-20 15:01:03,694 - distributed.scheduler - INFO - Receive client connection: Client-worker-c02ff873-3cc9-11f1-83fa-2e20a4008324
2026-04-20 15:01:03,682 - distributed.scheduler - INFO - Receive client connection: Client-worker-c035c739-3cc9-11f1-8354-c2349ba99f21
2026-04-20 15:00:58,377 - distributed.scheduler - INFO - Receive client connection: Client-bd13418a-3cc9-11f1-8001-daa6852bd5a7
2026-04-20 14:51:33,705 - distributed.scheduler - INFO - Close client connection: Client-5686c04f-3cc8-11f1-8001-daa6852bd5a7
2026-04-20 14:51:33,704 - distributed.scheduler - INFO - Remove client Client-5686c04f-3cc8-11f1-8001-daa6852bd5a7
2026-04-20 14:51:33,703 - distributed.scheduler - INFO - Remove client Client-5686c04f-3cc8-11f1-8001-daa6852bd5a7
2026-04-20 14:50:56,829 - distributed.scheduler - INFO - Receive client connection: Client-5686c04f-3cc8-11f1-8001-daa6852bd5a7
2026-04-20 14:41:32,147 - distributed.scheduler - INFO - Close client connection: Client-f00ac2f3-3cc6-11f1-8001-daa6852bd5a7
2026-04-20 14:41:32,145 - distributed.scheduler - INFO - Remove client Client-f00ac2f3-3cc6-11f1-8001-daa6852bd5a7
2026-04-20 14:41:32,143 - distributed.scheduler - INFO - Remove client Client-f00ac2f3-3cc6-11f1-8001-daa6852bd5a7
2026-04-20 14:40:55,393 - distributed.scheduler - INFO - Receive client connection: Client-f00ac2f3-3cc6-11f1-8001-daa6852bd5a7
2026-04-20 14:32:39,825 - distributed.scheduler - INFO - Close client connection: Client-8986205f-3cc5-11f1-8001-daa6852bd5a7
2026-04-20 14:32:39,824 - distributed.scheduler - INFO - Remove client Client-8986205f-3cc5-11f1-8001-daa6852bd5a7
2026-04-20 14:32:39,823 - distributed.scheduler - INFO - Remove client Client-8986205f-3cc5-11f1-8001-daa6852bd5a7
2026-04-20 14:32:28,922 - distributed.scheduler - INFO - Receive client connection: Client-worker-c2266990-3cc5-11f1-837e-7e353d94dc10
2026-04-20 14:31:48,926 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:46637
2026-04-20 14:31:48,924 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:46637', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:48,507 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:43269
2026-04-20 14:31:48,505 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:43269', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:48,330 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:34365
2026-04-20 14:31:48,328 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:34365', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:47,900 - distributed.scheduler - INFO - Close client connection: Client-worker-24ff9c95-3cbd-11f1-8309-c2349ba99f21
2026-04-20 14:31:47,888 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:38833', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:47,884 - distributed.scheduler - INFO - Remove client Client-worker-24ff9c95-3cbd-11f1-8309-c2349ba99f21
2026-04-20 14:31:47,437 - distributed.scheduler - INFO - Close client connection: Client-worker-258ac937-3cbd-11f1-83b9-2e20a4008324
2026-04-20 14:31:47,414 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:44565', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:47,409 - distributed.scheduler - INFO - Remove client Client-worker-258ac937-3cbd-11f1-83b9-2e20a4008324
2026-04-20 14:31:47,232 - distributed.scheduler - INFO - Close client connection: Client-worker-5869e55e-3cc1-11f1-8386-ee7435aaba1f
2026-04-20 14:31:47,227 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:36419', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:47,224 - distributed.scheduler - INFO - Remove client Client-worker-5869e55e-3cc1-11f1-8386-ee7435aaba1f
2026-04-20 14:31:43,080 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:43229
2026-04-20 14:31:43,078 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:43229', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:42,400 - distributed.scheduler - INFO - Receive client connection: Client-worker-a67054e3-3cc5-11f1-83f6-fae0fcb74df3
2026-04-20 14:31:42,264 - distributed.scheduler - INFO - Receive client connection: Client-worker-a65d18c8-3cc5-11f1-8350-2a51d3dc178e
2026-04-20 14:31:42,069 - distributed.scheduler - INFO - Close client connection: Client-worker-58908cae-3cc1-11f1-8389-86acc56fd09f
2026-04-20 14:31:42,057 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:45447', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:42,036 - distributed.scheduler - INFO - Remove client Client-worker-58908cae-3cc1-11f1-8389-86acc56fd09f
2026-04-20 14:31:39,339 - distributed.scheduler - INFO - Receive client connection: Client-worker-a4945f06-3cc5-11f1-840a-6ec63a5ab21e
2026-04-20 14:31:39,238 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:34503
2026-04-20 14:31:39,235 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:34503', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:38,684 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:46745
2026-04-20 14:31:38,684 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:46745', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:38,667 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:34489
2026-04-20 14:31:38,667 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:34489', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:38,372 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:45709
2026-04-20 14:31:38,371 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:45709', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:38,192 - distributed.scheduler - INFO - Close client connection: Client-worker-097d89bd-3cb9-11f1-8386-8288bfee4388
2026-04-20 14:31:38,188 - distributed.scheduler - INFO - Remove client Client-worker-097d89bd-3cb9-11f1-8386-8288bfee4388
2026-04-20 14:31:38,177 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:44111', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:37,804 - distributed.scheduler - INFO - Close client connection: Client-worker-0c67db54-3cb9-11f1-82d8-0e4c34ff5e38
2026-04-20 14:31:37,774 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:38035', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:37,772 - distributed.scheduler - INFO - Remove client Client-worker-0c67db54-3cb9-11f1-82d8-0e4c34ff5e38
2026-04-20 14:31:37,663 - distributed.scheduler - INFO - Close client connection: Client-worker-8c6e8cd9-3cc5-11f1-8382-ae00acbfc48e
2026-04-20 14:31:37,661 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:39639', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:37,660 - distributed.scheduler - INFO - Remove client Client-worker-8c6e8cd9-3cc5-11f1-8382-ae00acbfc48e
2026-04-20 14:31:37,478 - distributed.scheduler - INFO - Close client connection: Client-worker-0e35db10-3cb9-11f1-8321-2eb33e86e1bd
2026-04-20 14:31:37,476 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:32829', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:37,475 - distributed.scheduler - INFO - Remove client Client-worker-0e35db10-3cb9-11f1-8321-2eb33e86e1bd
2026-04-20 14:31:37,109 - distributed.scheduler - INFO - Receive client connection: Client-worker-a348a334-3cc5-11f1-841d-4ef3162b8ac2
2026-04-20 14:31:36,952 - distributed.scheduler - INFO - Receive client connection: Client-worker-a32f9020-3cc5-11f1-83b8-566b4744c386
2026-04-20 14:31:36,329 - distributed.scheduler - INFO - Receive client connection: Client-worker-a2cfae28-3cc5-11f1-84bb-f28f846166bd
2026-04-20 14:31:35,793 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:39767
2026-04-20 14:31:35,791 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:39767', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:34,975 - distributed.scheduler - INFO - Close client connection: Client-worker-0b5b113b-3cb9-11f1-83c1-6ec63a5ab21e
2026-04-20 14:31:34,964 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:44129', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:34,962 - distributed.scheduler - INFO - Remove client Client-worker-0b5b113b-3cb9-11f1-83c1-6ec63a5ab21e
2026-04-20 14:31:34,374 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:36229
2026-04-20 14:31:34,372 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:36229', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:33,939 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:43995
2026-04-20 14:31:33,938 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:43995', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:33,817 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:45953
2026-04-20 14:31:33,816 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:45953', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:33,807 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:35485
2026-04-20 14:31:33,806 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:35485', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:33,749 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:33689
2026-04-20 14:31:33,748 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:33689', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:33,747 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:45347
2026-04-20 14:31:33,746 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:45347', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:33,680 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:45191
2026-04-20 14:31:33,678 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:45191', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:33,499 - distributed.scheduler - INFO - Close client connection: Client-worker-24da4c1b-3cbd-11f1-83d7-b6f02dd406e9
2026-04-20 14:31:33,488 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:46613', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:33,487 - distributed.scheduler - INFO - Remove client Client-worker-24da4c1b-3cbd-11f1-83d7-b6f02dd406e9
2026-04-20 14:31:32,961 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:34997', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:32,953 - distributed.scheduler - INFO - Close client connection: Client-worker-24fef5f0-3cbd-11f1-8342-7e353d94dc10
2026-04-20 14:31:32,947 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:36769', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:32,944 - distributed.scheduler - INFO - Remove client Client-worker-24fef5f0-3cbd-11f1-8342-7e353d94dc10
2026-04-20 14:31:32,900 - distributed.scheduler - INFO - Close client connection: Client-worker-0905c23a-3cb9-11f1-8372-566b4744c386
2026-04-20 14:31:32,899 - distributed.scheduler - INFO - Close client connection: Client-worker-0bb92c2c-3cb9-11f1-83b1-fae0fcb74df3
2026-04-20 14:31:32,888 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:37265', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:32,887 - distributed.scheduler - INFO - Remove client Client-worker-0905c23a-3cb9-11f1-8372-566b4744c386
2026-04-20 14:31:32,878 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:34549', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:32,877 - distributed.scheduler - INFO - Remove client Client-worker-0bb92c2c-3cb9-11f1-83b1-fae0fcb74df3
2026-04-20 14:31:32,813 - distributed.scheduler - INFO - Close client connection: Client-worker-29b7b536-3cb9-11f1-83e0-4ef3162b8ac2
2026-04-20 14:31:32,807 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:41175', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:32,804 - distributed.scheduler - INFO - Remove client Client-worker-29b7b536-3cb9-11f1-83e0-4ef3162b8ac2
2026-04-20 14:31:32,693 - distributed.scheduler - INFO - Close client connection: Client-worker-58f5af44-3cc1-11f1-8472-f28f846166bd
2026-04-20 14:31:32,678 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:33393', status: closed, memory: 0, processing: 0>
2026-04-20 14:31:32,653 - distributed.scheduler - INFO - Remove client Client-worker-58f5af44-3cc1-11f1-8472-f28f846166bd
2026-04-20 14:30:58,821 - distributed.scheduler - INFO - Receive client connection: Client-worker-8c6e8cd9-3cc5-11f1-8382-ae00acbfc48e
2026-04-20 14:30:53,900 - distributed.scheduler - INFO - Receive client connection: Client-8986205f-3cc5-11f1-8001-daa6852bd5a7
2026-04-20 14:21:28,854 - distributed.scheduler - INFO - Close client connection: Client-23032ed3-3cc4-11f1-8001-daa6852bd5a7
2026-04-20 14:21:28,853 - distributed.scheduler - INFO - Remove client Client-23032ed3-3cc4-11f1-8001-daa6852bd5a7
2026-04-20 14:21:28,851 - distributed.scheduler - INFO - Remove client Client-23032ed3-3cc4-11f1-8001-daa6852bd5a7
2026-04-20 14:20:52,414 - distributed.scheduler - INFO - Receive client connection: Client-23032ed3-3cc4-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,507 - distributed.scheduler - INFO - Close client connection: Client-550c0d55-3cb3-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,506 - distributed.scheduler - INFO - Close client connection: Client-bb9b4b5c-3cb4-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,506 - distributed.scheduler - INFO - Close client connection: Client-22160151-3cb6-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,506 - distributed.scheduler - INFO - Close client connection: Client-88908b06-3cb7-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,506 - distributed.scheduler - INFO - Close client connection: Client-ef0614af-3cb8-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,505 - distributed.scheduler - INFO - Close client connection: Client-9f982688-3cba-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,505 - distributed.scheduler - INFO - Close client connection: Client-2286136b-3cbd-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,505 - distributed.scheduler - INFO - Close client connection: Client-88fe8382-3cbe-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,505 - distributed.scheduler - INFO - Close client connection: Client-ef8a7621-3cbf-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,504 - distributed.scheduler - INFO - Close client connection: Client-bc037321-3cbb-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,503 - distributed.scheduler - INFO - Close client connection: Client-56040e57-3cc1-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,502 - distributed.scheduler - INFO - Remove client Client-550c0d55-3cb3-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,502 - distributed.scheduler - INFO - Remove client Client-bb9b4b5c-3cb4-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,501 - distributed.scheduler - INFO - Remove client Client-22160151-3cb6-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,499 - distributed.scheduler - INFO - Remove client Client-88908b06-3cb7-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,499 - distributed.scheduler - INFO - Remove client Client-ef0614af-3cb8-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,498 - distributed.scheduler - INFO - Remove client Client-9f982688-3cba-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,498 - distributed.scheduler - INFO - Remove client Client-2286136b-3cbd-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,497 - distributed.scheduler - INFO - Remove client Client-88fe8382-3cbe-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,496 - distributed.scheduler - INFO - Remove client Client-ef8a7621-3cbf-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,495 - distributed.scheduler - INFO - Remove client Client-bc037321-3cbb-11f1-8001-daa6852bd5a7
2026-04-20 14:13:16,490 - distributed.scheduler - INFO - Remove client Client-56040e57-3cc1-11f1-8001-daa6852bd5a7
2026-04-20 14:13:15,756 - distributed.scheduler - INFO - Close client connection: Client-00473a5f-3cc3-11f1-8001-daa6852bd5a7
2026-04-20 14:13:15,754 - distributed.scheduler - INFO - Remove client Client-00473a5f-3cc3-11f1-8001-daa6852bd5a7
2026-04-20 14:13:15,752 - distributed.scheduler - INFO - Remove client Client-00473a5f-3cc3-11f1-8001-daa6852bd5a7
2026-04-20 14:12:44,635 - distributed.scheduler - INFO - Receive client connection: Client-00473a5f-3cc3-11f1-8001-daa6852bd5a7
2026-04-20 14:00:54,429 - distributed.scheduler - INFO - Receive client connection: Client-worker-58f5af44-3cc1-11f1-8472-f28f846166bd
2026-04-20 14:00:53,776 - distributed.scheduler - INFO - Receive client connection: Client-worker-58908cae-3cc1-11f1-8389-86acc56fd09f
2026-04-20 14:00:53,514 - distributed.scheduler - INFO - Receive client connection: Client-worker-5869e55e-3cc1-11f1-8386-ee7435aaba1f
2026-04-20 14:00:49,482 - distributed.scheduler - INFO - Receive client connection: Client-56040e57-3cc1-11f1-8001-daa6852bd5a7
2026-04-20 13:50:48,065 - distributed.scheduler - INFO - Receive client connection: Client-ef8a7621-3cbf-11f1-8001-daa6852bd5a7
2026-04-20 13:40:46,525 - distributed.scheduler - INFO - Receive client connection: Client-88fe8382-3cbe-11f1-8001-daa6852bd5a7
2026-04-20 13:30:50,176 - distributed.scheduler - INFO - Receive client connection: Client-worker-258ac937-3cbd-11f1-83b9-2e20a4008324
2026-04-20 13:30:49,260 - distributed.scheduler - INFO - Receive client connection: Client-worker-24ff9c95-3cbd-11f1-8309-c2349ba99f21
2026-04-20 13:30:49,257 - distributed.scheduler - INFO - Receive client connection: Client-worker-24fef5f0-3cbd-11f1-8342-7e353d94dc10
2026-04-20 13:30:49,013 - distributed.scheduler - INFO - Receive client connection: Client-worker-24da4c1b-3cbd-11f1-83d7-b6f02dd406e9
2026-04-20 13:30:45,116 - distributed.scheduler - INFO - Receive client connection: Client-2286136b-3cbd-11f1-8001-daa6852bd5a7
2026-04-20 13:20:43,630 - distributed.scheduler - INFO - Receive client connection: Client-bc037321-3cbb-11f1-8001-daa6852bd5a7
2026-04-20 13:12:46,447 - distributed.scheduler - INFO - Receive client connection: Client-9f982688-3cba-11f1-8001-daa6852bd5a7
2026-04-20 13:02:19,180 - distributed.scheduler - INFO - Receive client connection: Client-worker-29b7b536-3cb9-11f1-83e0-4ef3162b8ac2
2026-04-20 13:02:05,911 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-b543b1370e74193bf009503ef4f3201f NoneType: None
2026-04-20 13:02:05,910 - distributed.scheduler - ERROR - Couldn't gather keys {'run-b543b1370e74193bf009503ef4f3201f': []} state: ['memory'] workers: [] NoneType: None
2026-04-20 13:01:56,397 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('assign-99cab0c081c0e18fb659c99108a5fc6f', 15) NoneType: None
2026-04-20 13:01:56,397 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('assign-99cab0c081c0e18fb659c99108a5fc6f', 4) NoneType: None
2026-04-20 13:01:56,397 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('assign-99cab0c081c0e18fb659c99108a5fc6f', 3) NoneType: None
2026-04-20 13:01:56,397 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('assign-99cab0c081c0e18fb659c99108a5fc6f', 6) NoneType: None
2026-04-20 13:01:56,396 - distributed.scheduler - ERROR - Couldn't gather keys {"('assign-99cab0c081c0e18fb659c99108a5fc6f', 6)": [], "('assign-99cab0c081c0e18fb659c99108a5fc6f', 3)": [], "('assign-99cab0c081c0e18fb659c99108a5fc6f', 4)": [], "('assign-99cab0c081c0e18fb659c99108a5fc6f', 15)": []} state: ['waiting', 'waiting', 'waiting', 'waiting'] workers: [] NoneType: None
2026-04-20 13:01:33,646 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:36419
2026-04-20 13:01:33,645 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:36419', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:33,152 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:44565
2026-04-20 13:01:33,151 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:44565', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:33,075 - distributed.scheduler - INFO - Receive client connection: Client-worker-0e35db10-3cb9-11f1-8321-2eb33e86e1bd
2026-04-20 13:01:33,012 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:45447
2026-04-20 13:01:33,011 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:45447', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:32,978 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.29:38833
2026-04-20 13:01:32,977 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.29:38833', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:32,618 - distributed.scheduler - INFO - Close client connection: Client-worker-8f911b6b-3a8b-11f1-8289-ee7435aaba1f
2026-04-20 13:01:32,608 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:42859', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:32,604 - distributed.scheduler - INFO - Remove client Client-worker-8f911b6b-3a8b-11f1-8289-ee7435aaba1f
2026-04-20 13:01:32,109 - distributed.scheduler - INFO - Close client connection: Client-worker-9b9bc7b3-38d7-11f1-81a9-2e20a4008324
2026-04-20 13:01:32,101 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:38347', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:32,101 - distributed.scheduler - INFO - Remove client Client-worker-9b9bc7b3-38d7-11f1-81a9-2e20a4008324
2026-04-20 13:01:31,999 - distributed.scheduler - INFO - Close client connection: Client-worker-675a9804-38d3-11f1-81ee-86acc56fd09f
2026-04-20 13:01:31,972 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:41657', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:31,972 - distributed.scheduler - INFO - Remove client Client-worker-675a9804-38d3-11f1-81ee-86acc56fd09f
2026-04-20 13:01:31,871 - distributed.scheduler - INFO - Close client connection: Client-worker-6bed2f5c-37cb-11f1-80bb-c2349ba99f21
2026-04-20 13:01:31,859 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.29:37393', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:31,857 - distributed.scheduler - INFO - Remove client Client-worker-6bed2f5c-37cb-11f1-80bb-c2349ba99f21
2026-04-20 13:01:30,008 - distributed.scheduler - INFO - Receive client connection: Client-worker-0c67db54-3cb9-11f1-82d8-0e4c34ff5e38
2026-04-20 13:01:29,127 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.214:34997
2026-04-20 13:01:29,126 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.214:34997', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:28,871 - distributed.scheduler - INFO - Receive client connection: Client-worker-0bb92c2c-3cb9-11f1-83b1-fae0fcb74df3
2026-04-20 13:01:28,676 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.222:32829
2026-04-20 13:01:28,674 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.222:32829', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:28,467 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:39639
2026-04-20 13:01:28,466 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:39639', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:28,245 - distributed.scheduler - INFO - Receive client connection: Client-worker-0b5b113b-3cb9-11f1-83c1-6ec63a5ab21e
2026-04-20 13:01:28,197 - distributed.scheduler - INFO - Close client connection: Client-worker-3865ba20-37c7-11f1-8074-2a51d3dc178e
2026-04-20 13:01:28,190 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.214:46843', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:28,189 - distributed.scheduler - INFO - Remove client Client-worker-3865ba20-37c7-11f1-8074-2a51d3dc178e
2026-04-20 13:01:27,767 - distributed.scheduler - INFO - Close client connection: Client-worker-387de249-37c7-11f1-80af-2eb33e86e1bd
2026-04-20 13:01:27,761 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.222:34465', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:27,759 - distributed.scheduler - INFO - Remove client Client-worker-387de249-37c7-11f1-80af-2eb33e86e1bd
2026-04-20 13:01:27,393 - distributed.scheduler - INFO - Close client connection: Client-worker-8b4f6438-3cb0-11f1-8336-ae00acbfc48e
2026-04-20 13:01:27,380 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:46303', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:27,378 - distributed.scheduler - INFO - Remove client Client-worker-8b4f6438-3cb0-11f1-8336-ae00acbfc48e
2026-04-20 13:01:26,291 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:37265
2026-04-20 13:01:26,289 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:37265', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:25,264 - distributed.scheduler - INFO - Close client connection: Client-worker-ab43bec6-3c86-11f1-8347-fae0fcb74df3
2026-04-20 13:01:25,256 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:44443', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:25,256 - distributed.scheduler - INFO - Remove client Client-worker-ab43bec6-3c86-11f1-8347-fae0fcb74df3
2026-04-20 13:01:25,159 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:44129
2026-04-20 13:01:25,158 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:44129', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:25,143 - distributed.scheduler - INFO - Receive client connection: Client-worker-097d89bd-3cb9-11f1-8386-8288bfee4388
2026-04-20 13:01:24,507 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.253:36769
2026-04-20 13:01:24,506 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.253:36769', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:24,335 - distributed.scheduler - INFO - Receive client connection: Client-worker-0905c23a-3cb9-11f1-8372-566b4744c386
2026-04-20 13:01:24,201 - distributed.scheduler - INFO - Close client connection: Client-worker-10b6e1d4-3c7a-11f1-835f-6ec63a5ab21e
2026-04-20 13:01:24,190 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:43491', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:24,183 - distributed.scheduler - INFO - Remove client Client-worker-10b6e1d4-3c7a-11f1-835f-6ec63a5ab21e
2026-04-20 13:01:24,156 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:33393
2026-04-20 13:01:24,154 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:33393', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:24,122 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.160.14:38035
2026-04-20 13:01:24,121 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.160.14:38035', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:23,603 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:44111
2026-04-20 13:01:23,602 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:44111', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:23,518 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:46613
2026-04-20 13:01:23,517 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:46613', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:23,516 - distributed.scheduler - INFO - Close client connection: Client-worker-3d833088-37bf-11f1-8074-7e353d94dc10
2026-04-20 13:01:23,515 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:41175
2026-04-20 13:01:23,514 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:41175', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:23,505 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.253:38983', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:23,473 - distributed.scheduler - INFO - Remove client Client-worker-3d833088-37bf-11f1-8074-7e353d94dc10
2026-04-20 13:01:23,360 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.168:34549
2026-04-20 13:01:23,359 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.168:34549', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:23,324 - distributed.scheduler - INFO - Close client connection: Client-worker-57b27f2c-3cac-11f1-8431-f28f846166bd
2026-04-20 13:01:23,317 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:41299', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:23,290 - distributed.scheduler - INFO - Remove client Client-worker-57b27f2c-3cac-11f1-8431-f28f846166bd
2026-04-20 13:01:23,123 - distributed.scheduler - INFO - Close client connection: Client-worker-9fa2d682-37cf-11f1-8074-0e4c34ff5e38
2026-04-20 13:01:23,112 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.160.14:39341', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:23,110 - distributed.scheduler - INFO - Remove client Client-worker-9fa2d682-37cf-11f1-8074-0e4c34ff5e38
2026-04-20 13:01:22,632 - distributed.scheduler - INFO - Close client connection: Client-worker-5749af6d-3cac-11f1-8395-b6f02dd406e9
2026-04-20 13:01:22,627 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:43549', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:22,626 - distributed.scheduler - INFO - Remove client Client-worker-5749af6d-3cac-11f1-8395-b6f02dd406e9
2026-04-20 13:01:22,599 - distributed.scheduler - INFO - Close client connection: Client-worker-7b061e2e-3a48-11f1-8237-8288bfee4388
2026-04-20 13:01:22,584 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:41873', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:22,583 - distributed.scheduler - INFO - Remove client Client-worker-7b061e2e-3a48-11f1-8237-8288bfee4388
2026-04-20 13:01:22,557 - distributed.scheduler - INFO - Close client connection: Client-worker-54581693-38a5-11f1-81f0-4ef3162b8ac2
2026-04-20 13:01:22,545 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:44087', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:22,543 - distributed.scheduler - INFO - Remove client Client-worker-54581693-38a5-11f1-81f0-4ef3162b8ac2
2026-04-20 13:01:22,304 - distributed.scheduler - INFO - Close client connection: Client-worker-37ece9c4-37c7-11f1-80ab-566b4744c386
2026-04-20 13:01:22,284 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.168:35221', status: closed, memory: 0, processing: 0>
2026-04-20 13:01:22,283 - distributed.scheduler - INFO - Remove client Client-worker-37ece9c4-37c7-11f1-80ab-566b4744c386
2026-04-20 13:00:40,723 - distributed.scheduler - INFO - Receive client connection: Client-ef0614af-3cb8-11f1-8001-daa6852bd5a7
2026-04-20 12:50:39,327 - distributed.scheduler - INFO - Receive client connection: Client-88908b06-3cb7-11f1-8001-daa6852bd5a7
2026-04-20 12:40:37,899 - distributed.scheduler - INFO - Receive client connection: Client-22160151-3cb6-11f1-8001-daa6852bd5a7
2026-04-20 12:30:36,469 - distributed.scheduler - INFO - Receive client connection: Client-bb9b4b5c-3cb4-11f1-8001-daa6852bd5a7
2026-04-20 12:20:34,919 - distributed.scheduler - INFO - Receive client connection: Client-550c0d55-3cb3-11f1-8001-daa6852bd5a7
2026-04-20 12:12:37,996 - distributed.scheduler - INFO - Close client connection: Client-22233c93-3cb2-11f1-8001-daa6852bd5a7
2026-04-20 12:12:37,995 - distributed.scheduler - INFO - Remove client Client-22233c93-3cb2-11f1-8001-daa6852bd5a7
2026-04-20 12:12:37,995 - distributed.scheduler - INFO - Remove client Client-22233c93-3cb2-11f1-8001-daa6852bd5a7
2026-04-20 12:12:00,006 - distributed.scheduler - INFO - Receive client connection: Client-22233c93-3cb2-11f1-8001-daa6852bd5a7
2026-04-20 12:01:49,055 - distributed.scheduler - INFO - Close client connection: Client-87fdf066-3cb0-11f1-8001-daa6852bd5a7
2026-04-20 12:01:49,054 - distributed.scheduler - INFO - Remove client Client-87fdf066-3cb0-11f1-8001-daa6852bd5a7
2026-04-20 12:01:49,052 - distributed.scheduler - INFO - Remove client Client-87fdf066-3cb0-11f1-8001-daa6852bd5a7
2026-04-20 12:00:37,459 - distributed.scheduler - INFO - Receive client connection: Client-worker-8b4f6438-3cb0-11f1-8336-ae00acbfc48e
2026-04-20 12:00:31,896 - distributed.scheduler - INFO - Receive client connection: Client-87fdf066-3cb0-11f1-8001-daa6852bd5a7
2026-04-20 11:51:13,922 - distributed.scheduler - INFO - Close client connection: Client-21777fd6-3caf-11f1-8001-daa6852bd5a7
2026-04-20 11:51:13,914 - distributed.scheduler - INFO - Remove client Client-21777fd6-3caf-11f1-8001-daa6852bd5a7
2026-04-20 11:51:13,913 - distributed.scheduler - INFO - Remove client Client-21777fd6-3caf-11f1-8001-daa6852bd5a7
2026-04-20 11:50:30,392 - distributed.scheduler - INFO - Receive client connection: Client-21777fd6-3caf-11f1-8001-daa6852bd5a7
2026-04-20 11:41:12,696 - distributed.scheduler - INFO - Close client connection: Client-baff1203-3cad-11f1-8001-daa6852bd5a7
2026-04-20 11:41:12,695 - distributed.scheduler - INFO - Remove client Client-baff1203-3cad-11f1-8001-daa6852bd5a7
2026-04-20 11:41:12,693 - distributed.scheduler - INFO - Remove client Client-baff1203-3cad-11f1-8001-daa6852bd5a7
2026-04-20 11:40:28,975 - distributed.scheduler - INFO - Receive client connection: Client-baff1203-3cad-11f1-8001-daa6852bd5a7
2026-04-20 11:31:44,341 - distributed.scheduler - INFO - Close client connection: Client-547aef6c-3cac-11f1-8001-daa6852bd5a7
2026-04-20 11:31:44,341 - distributed.scheduler - INFO - Remove client Client-547aef6c-3cac-11f1-8001-daa6852bd5a7
2026-04-20 11:31:44,340 - distributed.scheduler - INFO - Remove client Client-547aef6c-3cac-11f1-8001-daa6852bd5a7
2026-04-20 11:30:33,367 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('len-agg-9dde516a5f15a01e891fee64c520ba15', 0) NoneType: None
2026-04-20 11:30:33,367 - distributed.scheduler - ERROR - Couldn't gather keys {"('len-agg-9dde516a5f15a01e891fee64c520ba15', 0)": []} state: ['waiting'] workers: [] NoneType: None
2026-04-20 11:30:32,952 - distributed.scheduler - INFO - Receive client connection: Client-worker-57b27f2c-3cac-11f1-8431-f28f846166bd
2026-04-20 11:30:32,521 - distributed.scheduler - INFO - Receive client connection: Client-worker-5749af6d-3cac-11f1-8395-b6f02dd406e9
2026-04-20 11:30:27,496 - distributed.scheduler - INFO - Receive client connection: Client-547aef6c-3cac-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,390 - distributed.scheduler - INFO - Close client connection: Client-53507f99-3c9e-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,390 - distributed.scheduler - INFO - Close client connection: Client-1fce9d58-3c9a-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,390 - distributed.scheduler - INFO - Close client connection: Client-864ae0b3-3c9b-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,390 - distributed.scheduler - INFO - Close client connection: Client-7bd4b68f-3ca1-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,390 - distributed.scheduler - INFO - Close client connection: Client-eccdf683-3c9c-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,389 - distributed.scheduler - INFO - Close client connection: Client-53edaa6c-3ca5-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,389 - distributed.scheduler - INFO - Close client connection: Client-ed6a7c61-3ca3-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,389 - distributed.scheduler - INFO - Close client connection: Client-86e25b96-3ca2-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,388 - distributed.scheduler - INFO - Close client connection: Client-b9cf85ae-3c9f-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,388 - distributed.scheduler - INFO - Close client connection: Client-b7b66140-3ca9-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,388 - distributed.scheduler - INFO - Close client connection: Client-ba6e2347-3ca6-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,387 - distributed.scheduler - INFO - Close client connection: Client-20ee9df4-3ca8-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,386 - distributed.scheduler - INFO - Remove client Client-53507f99-3c9e-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,385 - distributed.scheduler - INFO - Remove client Client-1fce9d58-3c9a-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,385 - distributed.scheduler - INFO - Remove client Client-864ae0b3-3c9b-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,384 - distributed.scheduler - INFO - Remove client Client-7bd4b68f-3ca1-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,383 - distributed.scheduler - INFO - Remove client Client-eccdf683-3c9c-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,382 - distributed.scheduler - INFO - Remove client Client-53edaa6c-3ca5-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,381 - distributed.scheduler - INFO - Remove client Client-ed6a7c61-3ca3-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,381 - distributed.scheduler - INFO - Remove client Client-86e25b96-3ca2-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,376 - distributed.scheduler - INFO - Remove client Client-b9cf85ae-3c9f-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,375 - distributed.scheduler - INFO - Remove client Client-b7b66140-3ca9-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,374 - distributed.scheduler - INFO - Remove client Client-ba6e2347-3ca6-11f1-8001-daa6852bd5a7
2026-04-20 11:21:12,373 - distributed.scheduler - INFO - Remove client Client-20ee9df4-3ca8-11f1-8001-daa6852bd5a7
2026-04-20 11:21:11,784 - distributed.scheduler - INFO - Close client connection: Client-edfc3370-3caa-11f1-8001-daa6852bd5a7
2026-04-20 11:21:11,782 - distributed.scheduler - INFO - Remove client Client-edfc3370-3caa-11f1-8001-daa6852bd5a7
2026-04-20 11:21:11,780 - distributed.scheduler - INFO - Remove client Client-edfc3370-3caa-11f1-8001-daa6852bd5a7
2026-04-20 11:20:26,043 - distributed.scheduler - INFO - Receive client connection: Client-edfc3370-3caa-11f1-8001-daa6852bd5a7
2026-04-20 11:11:45,465 - distributed.scheduler - INFO - Receive client connection: Client-b7b66140-3ca9-11f1-8001-daa6852bd5a7
2026-04-20 11:02:59,572 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:46303
2026-04-20 11:02:59,570 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:46303', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:59,548 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:41299
2026-04-20 11:02:59,546 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:41299', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:58,994 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:43549
2026-04-20 11:02:58,991 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:43549', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:58,659 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:40201', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:58,602 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:35457', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:58,042 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:43853', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:44,188 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:40201
2026-04-20 11:02:44,186 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:40201', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:44,159 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:35457
2026-04-20 11:02:44,157 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:35457', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:43,451 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:46339', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:43,206 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:35019', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:42,174 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:43853
2026-04-20 11:02:42,172 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:43853', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:41,237 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:46397', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:28,239 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:35019
2026-04-20 11:02:28,238 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:35019', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:28,224 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:46339
2026-04-20 11:02:28,223 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:46339', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:27,483 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:46397
2026-04-20 11:02:27,481 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:46397', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:27,335 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:37501', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:27,330 - distributed.scheduler - INFO - Close client connection: Client-worker-59023086-3ca8-11f1-83f8-f28f846166bd
2026-04-20 11:02:27,325 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:46803', status: closed, memory: 0, processing: 0>
2026-04-20 11:02:27,325 - distributed.scheduler - INFO - Remove client Client-worker-59023086-3ca8-11f1-83f8-f28f846166bd
2026-04-20 11:02:26,633 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:35483', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:57,086 - distributed.scheduler - INFO - Receive client connection: Client-worker-59023086-3ca8-11f1-83f8-f28f846166bd
2026-04-20 11:01:55,359 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:37501
2026-04-20 11:01:55,358 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:37501', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:55,215 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:46803
2026-04-20 11:01:55,214 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:46803', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:54,469 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:35483
2026-04-20 11:01:54,467 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:35483', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:54,362 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:45165', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:54,296 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:45339', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:53,484 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:33017', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:33,895 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:45165
2026-04-20 11:01:33,894 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:45165', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:33,531 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:45339
2026-04-20 11:01:33,528 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:45339', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:32,868 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:46293', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:32,573 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:35195', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:30,841 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:33017
2026-04-20 11:01:30,839 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:33017', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:29,893 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:34891', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:12,913 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.246:46293
2026-04-20 11:01:12,911 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.246:46293', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:11,848 - distributed.scheduler - INFO - Close client connection: Client-worker-03c8416e-37c3-11f1-8074-ae00acbfc48e
2026-04-20 11:01:11,841 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.246:45399', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:11,839 - distributed.scheduler - INFO - Remove client Client-worker-03c8416e-37c3-11f1-8074-ae00acbfc48e
2026-04-20 11:01:08,560 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:35195
2026-04-20 11:01:08,559 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:35195', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:07,622 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:34891
2026-04-20 11:01:07,621 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:34891', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:07,575 - distributed.scheduler - INFO - Close client connection: Client-worker-df302e06-3a26-11f1-8279-f28f846166bd
2026-04-20 11:01:07,569 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:40855', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:07,568 - distributed.scheduler - INFO - Remove client Client-worker-df302e06-3a26-11f1-8279-f28f846166bd
2026-04-20 11:01:06,675 - distributed.scheduler - INFO - Close client connection: Client-worker-4d9e0091-3bfc-11f1-8288-b6f02dd406e9
2026-04-20 11:01:06,653 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:33073', status: closed, memory: 0, processing: 0>
2026-04-20 11:01:06,633 - distributed.scheduler - INFO - Remove client Client-worker-4d9e0091-3bfc-11f1-8288-b6f02dd406e9
2026-04-20 11:00:29,800 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('len-agg-b1250f5a335cba82010d90131b42b93d', 0) NoneType: None
2026-04-20 11:00:29,800 - distributed.scheduler - ERROR - Couldn't gather keys {"('len-agg-b1250f5a335cba82010d90131b42b93d', 0)": []} state: ['waiting'] workers: [] NoneType: None
2026-04-20 11:00:23,013 - distributed.scheduler - INFO - Receive client connection: Client-20ee9df4-3ca8-11f1-8001-daa6852bd5a7
2026-04-20 10:50:21,545 - distributed.scheduler - INFO - Receive client connection: Client-ba6e2347-3ca6-11f1-8001-daa6852bd5a7
2026-04-20 10:40:20,077 - distributed.scheduler - INFO - Receive client connection: Client-53edaa6c-3ca5-11f1-8001-daa6852bd5a7
2026-04-20 10:30:18,595 - distributed.scheduler - INFO - Receive client connection: Client-ed6a7c61-3ca3-11f1-8001-daa6852bd5a7
2026-04-20 10:20:17,075 - distributed.scheduler - INFO - Receive client connection: Client-86e25b96-3ca2-11f1-8001-daa6852bd5a7
2026-04-20 10:12:49,029 - distributed.scheduler - INFO - Receive client connection: Client-7bd4b68f-3ca1-11f1-8001-daa6852bd5a7
2026-04-20 10:00:14,028 - distributed.scheduler - INFO - Receive client connection: Client-b9cf85ae-3c9f-11f1-8001-daa6852bd5a7
2026-04-20 09:50:12,563 - distributed.scheduler - INFO - Receive client connection: Client-53507f99-3c9e-11f1-8001-daa6852bd5a7
2026-04-20 09:40:11,090 - distributed.scheduler - INFO - Receive client connection: Client-eccdf683-3c9c-11f1-8001-daa6852bd5a7
2026-04-20 09:30:09,606 - distributed.scheduler - INFO - Receive client connection: Client-864ae0b3-3c9b-11f1-8001-daa6852bd5a7
2026-04-20 09:20:08,172 - distributed.scheduler - INFO - Receive client connection: Client-1fce9d58-3c9a-11f1-8001-daa6852bd5a7
2026-04-20 09:13:15,821 - distributed.scheduler - INFO - Close client connection: Client-17d2f899-3c99-11f1-8001-daa6852bd5a7
2026-04-20 09:13:15,821 - distributed.scheduler - INFO - Remove client Client-17d2f899-3c99-11f1-8001-daa6852bd5a7
2026-04-20 09:13:15,820 - distributed.scheduler - INFO - Remove client Client-17d2f899-3c99-11f1-8001-daa6852bd5a7
2026-04-20 09:12:45,304 - distributed.scheduler - INFO - Receive client connection: Client-17d2f899-3c99-11f1-8001-daa6852bd5a7
2026-04-20 09:01:13,155 - distributed.scheduler - INFO - Close client connection: Client-52cbf0fb-3c97-11f1-8001-daa6852bd5a7
2026-04-20 09:01:13,154 - distributed.scheduler - INFO - Remove client Client-52cbf0fb-3c97-11f1-8001-daa6852bd5a7
2026-04-20 09:01:13,153 - distributed.scheduler - INFO - Remove client Client-52cbf0fb-3c97-11f1-8001-daa6852bd5a7
2026-04-20 09:00:05,232 - distributed.scheduler - INFO - Receive client connection: Client-52cbf0fb-3c97-11f1-8001-daa6852bd5a7
2026-04-20 08:50:37,227 - distributed.scheduler - INFO - Close client connection: Client-ec45fbd1-3c95-11f1-8001-daa6852bd5a7
2026-04-20 08:50:37,226 - distributed.scheduler - INFO - Remove client Client-ec45fbd1-3c95-11f1-8001-daa6852bd5a7
2026-04-20 08:50:37,225 - distributed.scheduler - INFO - Remove client Client-ec45fbd1-3c95-11f1-8001-daa6852bd5a7
2026-04-20 08:50:03,727 - distributed.scheduler - INFO - Receive client connection: Client-ec45fbd1-3c95-11f1-8001-daa6852bd5a7
2026-04-20 08:40:38,541 - distributed.scheduler - INFO - Close client connection: Client-85b98c29-3c94-11f1-8001-daa6852bd5a7
2026-04-20 08:40:38,540 - distributed.scheduler - INFO - Remove client Client-85b98c29-3c94-11f1-8001-daa6852bd5a7
2026-04-20 08:40:38,539 - distributed.scheduler - INFO - Remove client Client-85b98c29-3c94-11f1-8001-daa6852bd5a7
2026-04-20 08:40:02,185 - distributed.scheduler - INFO - Receive client connection: Client-85b98c29-3c94-11f1-8001-daa6852bd5a7
2026-04-20 08:31:08,581 - distributed.scheduler - INFO - Close client connection: Client-1f34d80a-3c93-11f1-8001-daa6852bd5a7
2026-04-20 08:31:08,579 - distributed.scheduler - INFO - Remove client Client-1f34d80a-3c93-11f1-8001-daa6852bd5a7
2026-04-20 08:31:08,578 - distributed.scheduler - INFO - Remove client Client-1f34d80a-3c93-11f1-8001-daa6852bd5a7
2026-04-20 08:30:00,690 - distributed.scheduler - INFO - Receive client connection: Client-1f34d80a-3c93-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,442 - distributed.scheduler - INFO - Close client connection: Client-a86d77c8-3c86-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,442 - distributed.scheduler - INFO - Close client connection: Client-756eb126-3c89-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,442 - distributed.scheduler - INFO - Close client connection: Client-5d1462bd-3c88-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,442 - distributed.scheduler - INFO - Close client connection: Client-dbfb05d5-3c8a-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,441 - distributed.scheduler - INFO - Close client connection: Client-4283cd03-3c8c-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,441 - distributed.scheduler - INFO - Close client connection: Client-a90f5da0-3c8d-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,441 - distributed.scheduler - INFO - Close client connection: Client-0f94748b-3c8f-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,441 - distributed.scheduler - INFO - Close client connection: Client-dfa066d9-3c90-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,440 - distributed.scheduler - INFO - Remove client Client-a86d77c8-3c86-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,440 - distributed.scheduler - INFO - Remove client Client-756eb126-3c89-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,440 - distributed.scheduler - INFO - Remove client Client-5d1462bd-3c88-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,440 - distributed.scheduler - INFO - Remove client Client-dbfb05d5-3c8a-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,439 - distributed.scheduler - INFO - Remove client Client-4283cd03-3c8c-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,438 - distributed.scheduler - INFO - Remove client Client-a90f5da0-3c8d-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,438 - distributed.scheduler - INFO - Remove client Client-0f94748b-3c8f-11f1-8001-daa6852bd5a7
2026-04-20 08:21:37,438 - distributed.scheduler - INFO - Remove client Client-dfa066d9-3c90-11f1-8001-daa6852bd5a7
2026-04-20 08:21:36,815 - distributed.scheduler - INFO - Close client connection: Client-dc8f7d80-3c91-11f1-8001-daa6852bd5a7
2026-04-20 08:21:36,814 - distributed.scheduler - INFO - Remove client Client-dc8f7d80-3c91-11f1-8001-daa6852bd5a7
2026-04-20 08:21:36,812 - distributed.scheduler - INFO - Remove client Client-dc8f7d80-3c91-11f1-8001-daa6852bd5a7
2026-04-20 08:20:59,371 - distributed.scheduler - INFO - Receive client connection: Client-dc8f7d80-3c91-11f1-8001-daa6852bd5a7
2026-04-20 08:13:55,023 - distributed.scheduler - INFO - Receive client connection: Client-dfa066d9-3c90-11f1-8001-daa6852bd5a7
2026-04-20 08:00:56,480 - distributed.scheduler - INFO - Receive client connection: Client-0f94748b-3c8f-11f1-8001-daa6852bd5a7
2026-04-20 07:50:54,985 - distributed.scheduler - INFO - Receive client connection: Client-a90f5da0-3c8d-11f1-8001-daa6852bd5a7
2026-04-20 07:40:53,439 - distributed.scheduler - INFO - Receive client connection: Client-4283cd03-3c8c-11f1-8001-daa6852bd5a7
2026-04-20 07:30:51,921 - distributed.scheduler - INFO - Receive client connection: Client-dbfb05d5-3c8a-11f1-8001-daa6852bd5a7
2026-04-20 07:20:50,382 - distributed.scheduler - INFO - Receive client connection: Client-756eb126-3c89-11f1-8001-daa6852bd5a7
2026-04-20 07:13:00,022 - distributed.scheduler - INFO - Receive client connection: Client-5d1462bd-3c88-11f1-8001-daa6852bd5a7
2026-04-20 07:00:52,204 - distributed.scheduler - INFO - Receive client connection: Client-worker-ab43bec6-3c86-11f1-8347-fae0fcb74df3
2026-04-20 07:00:47,456 - distributed.scheduler - INFO - Receive client connection: Client-a86d77c8-3c86-11f1-8001-daa6852bd5a7
2026-04-20 06:51:25,327 - distributed.scheduler - INFO - Close client connection: Client-41e78685-3c85-11f1-8001-daa6852bd5a7
2026-04-20 06:51:25,324 - distributed.scheduler - INFO - Remove client Client-41e78685-3c85-11f1-8001-daa6852bd5a7
2026-04-20 06:51:25,322 - distributed.scheduler - INFO - Remove client Client-41e78685-3c85-11f1-8001-daa6852bd5a7
2026-04-20 06:50:45,950 - distributed.scheduler - INFO - Receive client connection: Client-41e78685-3c85-11f1-8001-daa6852bd5a7
2026-04-20 06:41:26,256 - distributed.scheduler - INFO - Close client connection: Client-db5e284c-3c83-11f1-8001-daa6852bd5a7
2026-04-20 06:41:26,255 - distributed.scheduler - INFO - Remove client Client-db5e284c-3c83-11f1-8001-daa6852bd5a7
2026-04-20 06:41:26,253 - distributed.scheduler - INFO - Remove client Client-db5e284c-3c83-11f1-8001-daa6852bd5a7
2026-04-20 06:40:44,429 - distributed.scheduler - INFO - Receive client connection: Client-db5e284c-3c83-11f1-8001-daa6852bd5a7
2026-04-20 06:31:54,092 - distributed.scheduler - INFO - Close client connection: Client-74d98af2-3c82-11f1-8001-daa6852bd5a7
2026-04-20 06:31:54,092 - distributed.scheduler - INFO - Remove client Client-74d98af2-3c82-11f1-8001-daa6852bd5a7
2026-04-20 06:31:54,090 - distributed.scheduler - INFO - Remove client Client-74d98af2-3c82-11f1-8001-daa6852bd5a7
2026-04-20 06:30:42,926 - distributed.scheduler - INFO - Receive client connection: Client-74d98af2-3c82-11f1-8001-daa6852bd5a7
2026-04-20 06:21:22,863 - distributed.scheduler - INFO - Close client connection: Client-0e5d7962-3c81-11f1-8001-daa6852bd5a7
2026-04-20 06:21:22,862 - distributed.scheduler - INFO - Remove client Client-0e5d7962-3c81-11f1-8001-daa6852bd5a7
2026-04-20 06:21:22,860 - distributed.scheduler - INFO - Remove client Client-0e5d7962-3c81-11f1-8001-daa6852bd5a7
2026-04-20 06:20:41,497 - distributed.scheduler - INFO - Receive client connection: Client-0e5d7962-3c81-11f1-8001-daa6852bd5a7
2026-04-20 06:13:36,394 - distributed.scheduler - INFO - Close client connection: Client-facdeed3-3c7f-11f1-8001-daa6852bd5a7
2026-04-20 06:13:36,393 - distributed.scheduler - INFO - Remove client Client-facdeed3-3c7f-11f1-8001-daa6852bd5a7
2026-04-20 06:13:36,392 - distributed.scheduler - INFO - Remove client Client-facdeed3-3c7f-11f1-8001-daa6852bd5a7
2026-04-20 06:12:59,182 - distributed.scheduler - INFO - Receive client connection: Client-facdeed3-3c7f-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,650 - distributed.scheduler - INFO - Close client connection: Client-a6ae1a4f-3c71-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,650 - distributed.scheduler - INFO - Close client connection: Client-0d3895e9-3c73-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,650 - distributed.scheduler - INFO - Close client connection: Client-73c17b87-3c74-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,649 - distributed.scheduler - INFO - Close client connection: Client-da3f9424-3c75-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,649 - distributed.scheduler - INFO - Close client connection: Client-9f6f3dd4-3c77-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,649 - distributed.scheduler - INFO - Close client connection: Client-a74bf439-3c78-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,649 - distributed.scheduler - INFO - Close client connection: Client-0dcd8916-3c7a-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,648 - distributed.scheduler - INFO - Close client connection: Client-7455bdc0-3c7b-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,648 - distributed.scheduler - INFO - Close client connection: Client-dae4eeb1-3c7c-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,647 - distributed.scheduler - INFO - Remove client Client-a6ae1a4f-3c71-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,645 - distributed.scheduler - INFO - Remove client Client-0d3895e9-3c73-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,644 - distributed.scheduler - INFO - Remove client Client-73c17b87-3c74-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,644 - distributed.scheduler - INFO - Remove client Client-da3f9424-3c75-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,643 - distributed.scheduler - INFO - Remove client Client-9f6f3dd4-3c77-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,642 - distributed.scheduler - INFO - Remove client Client-a74bf439-3c78-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,641 - distributed.scheduler - INFO - Remove client Client-0dcd8916-3c7a-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,641 - distributed.scheduler - INFO - Remove client Client-7455bdc0-3c7b-11f1-8001-daa6852bd5a7
2026-04-20 06:01:56,640 - distributed.scheduler - INFO - Remove client Client-dae4eeb1-3c7c-11f1-8001-daa6852bd5a7
2026-04-20 06:01:55,863 - distributed.scheduler - INFO - Close client connection: Client-416101fc-3c7e-11f1-8001-daa6852bd5a7
2026-04-20 06:01:55,862 - distributed.scheduler - INFO - Remove client Client-416101fc-3c7e-11f1-8001-daa6852bd5a7
2026-04-20 06:01:55,860 - distributed.scheduler - INFO - Remove client Client-416101fc-3c7e-11f1-8001-daa6852bd5a7
2026-04-20 06:00:38,583 - distributed.scheduler - INFO - Receive client connection: Client-416101fc-3c7e-11f1-8001-daa6852bd5a7
2026-04-20 05:50:37,146 - distributed.scheduler - INFO - Receive client connection: Client-dae4eeb1-3c7c-11f1-8001-daa6852bd5a7
2026-04-20 05:40:35,585 - distributed.scheduler - INFO - Receive client connection: Client-7455bdc0-3c7b-11f1-8001-daa6852bd5a7
2026-04-20 05:34:24,937 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:44443
2026-04-20 05:34:24,936 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:44443', status: closed, memory: 0, processing: 0>
2026-04-20 05:34:24,052 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:36765', status: closed, memory: 0, processing: 0>
2026-04-20 05:34:11,904 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:42171 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s
2026-04-20 05:34:11,901 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:33073 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:34:11,810 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:34:11,807 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:34:11,806 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:34:11,805 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:42859 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:34:11,805 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:34:11,779 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:34:11,774 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:34:11,773 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:34:11,767 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:40855 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:34:11,767 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:41873 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:34:11,756 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:34:11,755 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:42171 after 30 s')
2026-04-20 05:33:42,644 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:36765
2026-04-20 05:33:42,642 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:36765', status: closed, memory: 0, processing: 0>
2026-04-20 05:33:41,653 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:42171', status: closed, memory: 0, processing: 0>
2026-04-20 05:33:30,913 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:46731 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:46731 after 30 s
2026-04-20 05:33:30,850 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:33073 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:46731 after 30 s')
2026-04-20 05:33:30,838 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:41873 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:46731 after 30 s')
2026-04-20 05:33:30,830 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:46731 after 30 s')
2026-04-20 05:33:30,804 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43491 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:46731 after 30 s')
2026-04-20 05:33:30,801 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:42859 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:46731 after 30 s')
2026-04-20 05:33:30,799 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:46731 after 30 s')
2026-04-20 05:33:30,798 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:46731 after 30 s')
2026-04-20 05:33:30,790 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:46731 after 30 s')
2026-04-20 05:33:30,787 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:46731 after 30 s')
2026-04-20 05:33:30,786 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:40855 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:46731 after 30 s')
2026-04-20 05:33:30,775 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:46731 after 30 s')
2026-04-20 05:33:01,633 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:42171
2026-04-20 05:33:01,631 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:42171', status: closed, memory: 0, processing: 0>
2026-04-20 05:33:00,648 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:46731', status: closed, memory: 0, processing: 0>
2026-04-20 05:32:48,925 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:33073 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:38961 after 30 s')
2026-04-20 05:32:48,922 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:38961 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:38961 after 30 s
2026-04-20 05:32:48,855 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:38961 after 30 s')
2026-04-20 05:32:48,830 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:38961 after 30 s')
2026-04-20 05:32:48,829 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:38961 after 30 s')
2026-04-20 05:32:48,829 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:40855 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:38961 after 30 s')
2026-04-20 05:32:48,829 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:38961 after 30 s')
2026-04-20 05:32:48,796 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:38961 after 30 s')
2026-04-20 05:32:48,789 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43491 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:38961 after 30 s')
2026-04-20 05:32:48,774 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:38961 after 30 s')
2026-04-20 05:32:19,732 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:46731
2026-04-20 05:32:19,730 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:46731', status: closed, memory: 0, processing: 0>
2026-04-20 05:32:18,648 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:38961', status: closed, memory: 0, processing: 0>
2026-04-20 05:32:03,031 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43491 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:33337 after 30 s')
2026-04-20 05:32:02,862 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:33337 failed: OSError: Timed out trying to connect to tcp://172.21.159.225:33337 after 30 s
2026-04-20 05:32:02,813 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:33337 after 30 s')
2026-04-20 05:32:02,735 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:33337 after 30 s')
2026-04-20 05:32:02,731 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:41873 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:33337 after 30 s')
2026-04-20 05:32:02,726 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.225:33337 after 30 s')
2026-04-20 05:31:33,451 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:38961
2026-04-20 05:31:33,450 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:38961', status: closed, memory: 0, processing: 0>
2026-04-20 05:31:32,560 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:33337', status: closed, memory: 0, processing: 0>
2026-04-20 05:31:20,830 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.225:33337
2026-04-20 05:31:20,828 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.225:33337', status: closed, memory: 0, processing: 0>
2026-04-20 05:31:19,896 - distributed.scheduler - INFO - Close client connection: Client-worker-03fdf24c-37c3-11f1-8075-fae0fcb74df3
2026-04-20 05:31:19,891 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.225:32939', status: closed, memory: 0, processing: 0>
2026-04-20 05:31:19,869 - distributed.scheduler - INFO - Remove client Client-worker-03fdf24c-37c3-11f1-8075-fae0fcb74df3
2026-04-20 05:30:38,998 - distributed.scheduler - INFO - Receive client connection: Client-worker-10b6e1d4-3c7a-11f1-835f-6ec63a5ab21e
2026-04-20 05:30:34,061 - distributed.scheduler - INFO - Receive client connection: Client-0dcd8916-3c7a-11f1-8001-daa6852bd5a7
2026-04-20 05:20:32,593 - distributed.scheduler - INFO - Receive client connection: Client-a74bf439-3c78-11f1-8001-daa6852bd5a7
2026-04-20 05:13:09,902 - distributed.scheduler - INFO - Receive client connection: Client-9f6f3dd4-3c77-11f1-8001-daa6852bd5a7
2026-04-20 05:00:29,594 - distributed.scheduler - INFO - Receive client connection: Client-da3f9424-3c75-11f1-8001-daa6852bd5a7
2026-04-20 04:50:28,138 - distributed.scheduler - INFO - Receive client connection: Client-73c17b87-3c74-11f1-8001-daa6852bd5a7
2026-04-20 04:40:26,613 - distributed.scheduler - INFO - Receive client connection: Client-0d3895e9-3c73-11f1-8001-daa6852bd5a7
2026-04-20 04:30:25,092 - distributed.scheduler - INFO - Receive client connection: Client-a6ae1a4f-3c71-11f1-8001-daa6852bd5a7
2026-04-20 04:20:59,330 - distributed.scheduler - INFO - Close client connection: Client-40278056-3c70-11f1-8001-daa6852bd5a7
2026-04-20 04:20:59,328 - distributed.scheduler - INFO - Remove client Client-40278056-3c70-11f1-8001-daa6852bd5a7
2026-04-20 04:20:59,327 - distributed.scheduler - INFO - Remove client Client-40278056-3c70-11f1-8001-daa6852bd5a7
2026-04-20 04:20:23,585 - distributed.scheduler - INFO - Receive client connection: Client-40278056-3c70-11f1-8001-daa6852bd5a7
2026-04-20 04:13:07,478 - distributed.scheduler - INFO - Close client connection: Client-29c1775f-3c6f-11f1-8001-daa6852bd5a7
2026-04-20 04:13:07,477 - distributed.scheduler - INFO - Remove client Client-29c1775f-3c6f-11f1-8001-daa6852bd5a7
2026-04-20 04:13:07,477 - distributed.scheduler - INFO - Remove client Client-29c1775f-3c6f-11f1-8001-daa6852bd5a7
2026-04-20 04:12:36,506 - distributed.scheduler - INFO - Receive client connection: Client-29c1775f-3c6f-11f1-8001-daa6852bd5a7
2026-04-20 04:01:32,820 - distributed.scheduler - INFO - Close client connection: Client-7315b550-3c6d-11f1-8001-daa6852bd5a7
2026-04-20 04:01:32,820 - distributed.scheduler - INFO - Remove client Client-7315b550-3c6d-11f1-8001-daa6852bd5a7
2026-04-20 04:01:32,819 - distributed.scheduler - INFO - Remove client Client-7315b550-3c6d-11f1-8001-daa6852bd5a7
2026-04-20 04:00:20,540 - distributed.scheduler - INFO - Receive client connection: Client-7315b550-3c6d-11f1-8001-daa6852bd5a7
2026-04-20 03:50:54,538 - distributed.scheduler - INFO - Close client connection: Client-0c980568-3c6c-11f1-8001-daa6852bd5a7
2026-04-20 03:50:54,538 - distributed.scheduler - INFO - Remove client Client-0c980568-3c6c-11f1-8001-daa6852bd5a7
2026-04-20 03:50:54,537 - distributed.scheduler - INFO - Remove client Client-0c980568-3c6c-11f1-8001-daa6852bd5a7
2026-04-20 03:50:19,092 - distributed.scheduler - INFO - Receive client connection: Client-0c980568-3c6c-11f1-8001-daa6852bd5a7
2026-04-20 03:40:50,800 - distributed.scheduler - INFO - Close client connection: Client-a60e757e-3c6a-11f1-8001-daa6852bd5a7
2026-04-20 03:40:50,800 - distributed.scheduler - INFO - Remove client Client-a60e757e-3c6a-11f1-8001-daa6852bd5a7
2026-04-20 03:40:50,799 - distributed.scheduler - INFO - Remove client Client-a60e757e-3c6a-11f1-8001-daa6852bd5a7
2026-04-20 03:40:17,566 - distributed.scheduler - INFO - Receive client connection: Client-a60e757e-3c6a-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,633 - distributed.scheduler - INFO - Close client connection: Client-4ddcdfbb-3c1f-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,633 - distributed.scheduler - INFO - Close client connection: Client-b463e5a2-3c20-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,633 - distributed.scheduler - INFO - Close client connection: Client-f70b1bd1-3c21-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,633 - distributed.scheduler - INFO - Close client connection: Client-c4180985-3c24-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,633 - distributed.scheduler - INFO - Close client connection: Client-b2a27180-3c23-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,633 - distributed.scheduler - INFO - Close client connection: Client-2aa1703d-3c26-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,633 - distributed.scheduler - INFO - Close client connection: Client-91244095-3c27-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,632 - distributed.scheduler - INFO - Close client connection: Client-f7ba9bad-3c28-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,632 - distributed.scheduler - INFO - Close client connection: Client-5e34a7af-3c2a-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,632 - distributed.scheduler - INFO - Close client connection: Client-2b2a4feb-3c2d-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,632 - distributed.scheduler - INFO - Close client connection: Client-21617bf6-3c2c-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,632 - distributed.scheduler - INFO - Close client connection: Client-91a8e5d9-3c2e-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,632 - distributed.scheduler - INFO - Close client connection: Client-f82fb729-3c2f-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,632 - distributed.scheduler - INFO - Close client connection: Client-5eb8d3f5-3c31-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,632 - distributed.scheduler - INFO - Close client connection: Client-c5404d35-3c32-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,632 - distributed.scheduler - INFO - Close client connection: Client-9248c8ed-3c35-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,632 - distributed.scheduler - INFO - Close client connection: Client-835c1d45-3c34-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,632 - distributed.scheduler - INFO - Close client connection: Client-f8cf15a9-3c36-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,631 - distributed.scheduler - INFO - Close client connection: Client-5f51839f-3c38-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,631 - distributed.scheduler - INFO - Close client connection: Client-c5d5f532-3c39-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,631 - distributed.scheduler - INFO - Close client connection: Client-2c5a95b6-3c3b-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,631 - distributed.scheduler - INFO - Close client connection: Client-f978d48e-3c3d-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,631 - distributed.scheduler - INFO - Close client connection: Client-f5820b51-3c3c-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,631 - distributed.scheduler - INFO - Close client connection: Client-5ff3eb84-3c3f-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,631 - distributed.scheduler - INFO - Close client connection: Client-c672cadd-3c40-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,631 - distributed.scheduler - INFO - Close client connection: Client-2cfdb96f-3c42-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,631 - distributed.scheduler - INFO - Close client connection: Client-9386c064-3c43-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,631 - distributed.scheduler - INFO - Close client connection: Client-60972ce0-3c46-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,630 - distributed.scheduler - INFO - Close client connection: Client-4f8b0afb-3c45-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,630 - distributed.scheduler - INFO - Close client connection: Client-c7119bc5-3c47-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,630 - distributed.scheduler - INFO - Close client connection: Client-2d8e0892-3c49-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,630 - distributed.scheduler - INFO - Close client connection: Client-940a349f-3c4a-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,630 - distributed.scheduler - INFO - Close client connection: Client-fa95569d-3c4b-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,630 - distributed.scheduler - INFO - Close client connection: Client-c7a481c5-3c4e-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,630 - distributed.scheduler - INFO - Close client connection: Client-9b9447d7-3c4d-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,630 - distributed.scheduler - INFO - Close client connection: Client-2e2a395c-3c50-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,630 - distributed.scheduler - INFO - Close client connection: Client-94af53ba-3c51-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,629 - distributed.scheduler - INFO - Close client connection: Client-fb329f55-3c52-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,629 - distributed.scheduler - INFO - Close client connection: Client-61bf456d-3c54-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,629 - distributed.scheduler - INFO - Close client connection: Client-2ec53fe4-3c57-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,628 - distributed.scheduler - INFO - Close client connection: Client-292fcfc4-3c56-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,628 - distributed.scheduler - INFO - Close client connection: Client-9546c9df-3c58-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,628 - distributed.scheduler - INFO - Close client connection: Client-fbbefd3a-3c59-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,628 - distributed.scheduler - INFO - Close client connection: Client-3e67cc62-3c5b-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,628 - distributed.scheduler - INFO - Close client connection: Client-a4ede8ac-3c5c-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,628 - distributed.scheduler - INFO - Close client connection: Client-71f657e7-3c5f-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,628 - distributed.scheduler - INFO - Close client connection: Client-687bb2b5-3c5e-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,628 - distributed.scheduler - INFO - Close client connection: Client-d8810532-3c60-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,628 - distributed.scheduler - INFO - Close client connection: Client-3f00ff09-3c62-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,628 - distributed.scheduler - INFO - Close client connection: Client-a57c7c0f-3c63-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,627 - distributed.scheduler - INFO - Close client connection: Client-0bfe7d78-3c65-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,627 - distributed.scheduler - INFO - Close client connection: Client-d909ace2-3c67-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,625 - distributed.scheduler - INFO - Close client connection: Client-c7c3cab1-3c66-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,625 - distributed.scheduler - INFO - Remove client Client-4ddcdfbb-3c1f-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,625 - distributed.scheduler - INFO - Remove client Client-b463e5a2-3c20-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,624 - distributed.scheduler - INFO - Remove client Client-f70b1bd1-3c21-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,624 - distributed.scheduler - INFO - Remove client Client-c4180985-3c24-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,624 - distributed.scheduler - INFO - Remove client Client-b2a27180-3c23-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,624 - distributed.scheduler - INFO - Remove client Client-2aa1703d-3c26-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,623 - distributed.scheduler - INFO - Remove client Client-91244095-3c27-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,623 - distributed.scheduler - INFO - Remove client Client-f7ba9bad-3c28-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,623 - distributed.scheduler - INFO - Remove client Client-5e34a7af-3c2a-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,622 - distributed.scheduler - INFO - Remove client Client-2b2a4feb-3c2d-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,622 - distributed.scheduler - INFO - Remove client Client-21617bf6-3c2c-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,622 - distributed.scheduler - INFO - Remove client Client-91a8e5d9-3c2e-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,622 - distributed.scheduler - INFO - Remove client Client-f82fb729-3c2f-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,621 - distributed.scheduler - INFO - Remove client Client-5eb8d3f5-3c31-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,621 - distributed.scheduler - INFO - Remove client Client-c5404d35-3c32-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,621 - distributed.scheduler - INFO - Remove client Client-9248c8ed-3c35-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,620 - distributed.scheduler - INFO - Remove client Client-835c1d45-3c34-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,620 - distributed.scheduler - INFO - Remove client Client-f8cf15a9-3c36-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,620 - distributed.scheduler - INFO - Remove client Client-5f51839f-3c38-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,620 - distributed.scheduler - INFO - Remove client Client-c5d5f532-3c39-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,619 - distributed.scheduler - INFO - Remove client Client-2c5a95b6-3c3b-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,619 - distributed.scheduler - INFO - Remove client Client-f978d48e-3c3d-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,619 - distributed.scheduler - INFO - Remove client Client-f5820b51-3c3c-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,618 - distributed.scheduler - INFO - Remove client Client-5ff3eb84-3c3f-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,618 - distributed.scheduler - INFO - Remove client Client-c672cadd-3c40-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,618 - distributed.scheduler - INFO - Remove client Client-2cfdb96f-3c42-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,617 - distributed.scheduler - INFO - Remove client Client-9386c064-3c43-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,617 - distributed.scheduler - INFO - Remove client Client-60972ce0-3c46-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,617 - distributed.scheduler - INFO - Remove client Client-4f8b0afb-3c45-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,617 - distributed.scheduler - INFO - Remove client Client-c7119bc5-3c47-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,616 - distributed.scheduler - INFO - Remove client Client-2d8e0892-3c49-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,616 - distributed.scheduler - INFO - Remove client Client-940a349f-3c4a-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,616 - distributed.scheduler - INFO - Remove client Client-fa95569d-3c4b-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,615 - distributed.scheduler - INFO - Remove client Client-c7a481c5-3c4e-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,615 - distributed.scheduler - INFO - Remove client Client-9b9447d7-3c4d-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,615 - distributed.scheduler - INFO - Remove client Client-2e2a395c-3c50-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,614 - distributed.scheduler - INFO - Remove client Client-94af53ba-3c51-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,605 - distributed.scheduler - INFO - Remove client Client-fb329f55-3c52-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,604 - distributed.scheduler - INFO - Remove client Client-61bf456d-3c54-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,604 - distributed.scheduler - INFO - Remove client Client-2ec53fe4-3c57-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,603 - distributed.scheduler - INFO - Remove client Client-292fcfc4-3c56-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,602 - distributed.scheduler - INFO - Remove client Client-9546c9df-3c58-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,602 - distributed.scheduler - INFO - Remove client Client-fbbefd3a-3c59-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,601 - distributed.scheduler - INFO - Remove client Client-3e67cc62-3c5b-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,600 - distributed.scheduler - INFO - Remove client Client-a4ede8ac-3c5c-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,600 - distributed.scheduler - INFO - Remove client Client-71f657e7-3c5f-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,599 - distributed.scheduler - INFO - Remove client Client-687bb2b5-3c5e-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,599 - distributed.scheduler - INFO - Remove client Client-d8810532-3c60-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,598 - distributed.scheduler - INFO - Remove client Client-3f00ff09-3c62-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,597 - distributed.scheduler - INFO - Remove client Client-a57c7c0f-3c63-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,597 - distributed.scheduler - INFO - Remove client Client-0bfe7d78-3c65-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,596 - distributed.scheduler - INFO - Remove client Client-d909ace2-3c67-11f1-8001-daa6852bd5a7
2026-04-20 03:31:31,588 - distributed.scheduler - INFO - Remove client Client-c7c3cab1-3c66-11f1-8001-daa6852bd5a7
2026-04-20 03:31:30,906 - distributed.scheduler - INFO - Close client connection: Client-3f8f64d3-3c69-11f1-8001-daa6852bd5a7
2026-04-20 03:31:30,904 - distributed.scheduler - INFO - Remove client Client-3f8f64d3-3c69-11f1-8001-daa6852bd5a7
2026-04-20 03:31:30,903 - distributed.scheduler - INFO - Remove client Client-3f8f64d3-3c69-11f1-8001-daa6852bd5a7
2026-04-20 03:30:16,105 - distributed.scheduler - INFO - Receive client connection: Client-3f8f64d3-3c69-11f1-8001-daa6852bd5a7
2026-04-20 03:20:14,606 - distributed.scheduler - INFO - Receive client connection: Client-d909ace2-3c67-11f1-8001-daa6852bd5a7
2026-04-20 03:12:36,127 - distributed.scheduler - INFO - Receive client connection: Client-c7c3cab1-3c66-11f1-8001-daa6852bd5a7
2026-04-20 03:00:11,607 - distributed.scheduler - INFO - Receive client connection: Client-0bfe7d78-3c65-11f1-8001-daa6852bd5a7
2026-04-20 02:50:10,128 - distributed.scheduler - INFO - Receive client connection: Client-a57c7c0f-3c63-11f1-8001-daa6852bd5a7
2026-04-20 02:40:08,688 - distributed.scheduler - INFO - Receive client connection: Client-3f00ff09-3c62-11f1-8001-daa6852bd5a7
2026-04-20 02:30:07,237 - distributed.scheduler - INFO - Receive client connection: Client-d8810532-3c60-11f1-8001-daa6852bd5a7
2026-04-20 02:20:05,698 - distributed.scheduler - INFO - Receive client connection: Client-71f657e7-3c5f-11f1-8001-daa6852bd5a7
2026-04-20 02:12:40,294 - distributed.scheduler - INFO - Receive client connection: Client-687bb2b5-3c5e-11f1-8001-daa6852bd5a7
2026-04-20 02:03:25,039 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:43491
2026-04-20 02:03:25,037 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:43491', status: closed, memory: 0, processing: 0>
2026-04-20 02:03:24,189 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:42017', status: closed, memory: 0, processing: 0>
2026-04-20 02:03:13,430 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:43643 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:43643 after 30 s
2026-04-20 02:03:13,362 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:43643 after 30 s')
2026-04-20 02:03:13,362 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:43643 after 30 s')
2026-04-20 02:03:13,336 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:42859 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:43643 after 30 s')
2026-04-20 02:03:13,334 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:43643 after 30 s')
2026-04-20 02:03:13,332 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:43643 after 30 s')
2026-04-20 02:03:13,321 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:43643 after 30 s')
2026-04-20 02:03:13,305 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:43643 after 30 s')
2026-04-20 02:02:44,134 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:42017
2026-04-20 02:02:44,132 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:42017', status: closed, memory: 0, processing: 0>
2026-04-20 02:02:43,191 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:43643', status: closed, memory: 0, processing: 0>
2026-04-20 02:02:31,075 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:45665 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:45665 after 30 s
2026-04-20 02:02:31,068 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:33073 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:45665 after 30 s')
2026-04-20 02:02:31,013 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:42859 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:45665 after 30 s')
2026-04-20 02:02:30,994 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:45665 after 30 s')
2026-04-20 02:02:30,960 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:45665 after 30 s')
2026-04-20 02:02:30,941 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:45665 after 30 s')
2026-04-20 02:02:30,932 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:45665 after 30 s')
2026-04-20 02:02:30,930 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:45665 after 30 s')
2026-04-20 02:02:30,929 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:40855 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:45665 after 30 s')
2026-04-20 02:02:30,922 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:45665 after 30 s')
2026-04-20 02:02:01,707 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:43643
2026-04-20 02:02:01,705 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:43643', status: closed, memory: 0, processing: 0>
2026-04-20 02:02:00,787 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:45665', status: closed, memory: 0, processing: 0>
2026-04-20 02:01:50,240 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:34873 failed: OSError: Timed out trying to connect to tcp://172.21.159.230:34873 after 30 s
2026-04-20 02:01:50,101 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:34873 after 30 s')
2026-04-20 02:01:50,100 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:33073 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:34873 after 30 s')
2026-04-20 02:01:49,904 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.230:34873 after 30 s')
2026-04-20 02:01:20,824 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:45665
2026-04-20 02:01:20,823 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:45665', status: closed, memory: 0, processing: 0>
2026-04-20 02:01:19,792 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:34873', status: closed, memory: 0, processing: 0>
2026-04-20 02:00:42,886 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.230:34873
2026-04-20 02:00:42,885 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.230:34873', status: closed, memory: 0, processing: 0>
2026-04-20 02:00:41,892 - distributed.scheduler - INFO - Close client connection: Client-worker-03f99a42-37c3-11f1-8074-6ec63a5ab21e
2026-04-20 02:00:41,840 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.230:44659', status: closed, memory: 0, processing: 0>
2026-04-20 02:00:41,813 - distributed.scheduler - INFO - Remove client Client-worker-03f99a42-37c3-11f1-8074-6ec63a5ab21e
2026-04-20 02:00:02,717 - distributed.scheduler - INFO - Receive client connection: Client-a4ede8ac-3c5c-11f1-8001-daa6852bd5a7
2026-04-20 01:50:01,214 - distributed.scheduler - INFO - Receive client connection: Client-3e67cc62-3c5b-11f1-8001-daa6852bd5a7
2026-04-20 01:40:59,875 - distributed.scheduler - INFO - Receive client connection: Client-fbbefd3a-3c59-11f1-8001-daa6852bd5a7
2026-04-20 01:30:58,470 - distributed.scheduler - INFO - Receive client connection: Client-9546c9df-3c58-11f1-8001-daa6852bd5a7
2026-04-20 01:20:56,995 - distributed.scheduler - INFO - Receive client connection: Client-2ec53fe4-3c57-11f1-8001-daa6852bd5a7
2026-04-20 01:13:38,122 - distributed.scheduler - INFO - Receive client connection: Client-292fcfc4-3c56-11f1-8001-daa6852bd5a7
2026-04-20 01:00:54,049 - distributed.scheduler - INFO - Receive client connection: Client-61bf456d-3c54-11f1-8001-daa6852bd5a7
2026-04-20 00:50:52,482 - distributed.scheduler - INFO - Receive client connection: Client-fb329f55-3c52-11f1-8001-daa6852bd5a7
2026-04-20 00:40:50,996 - distributed.scheduler - INFO - Receive client connection: Client-94af53ba-3c51-11f1-8001-daa6852bd5a7
2026-04-20 00:30:49,494 - distributed.scheduler - INFO - Receive client connection: Client-2e2a395c-3c50-11f1-8001-daa6852bd5a7
2026-04-20 00:20:48,016 - distributed.scheduler - INFO - Receive client connection: Client-c7a481c5-3c4e-11f1-8001-daa6852bd5a7
2026-04-20 00:12:24,572 - distributed.scheduler - INFO - Receive client connection: Client-9b9447d7-3c4d-11f1-8001-daa6852bd5a7
2026-04-20 00:00:44,975 - distributed.scheduler - INFO - Receive client connection: Client-fa95569d-3c4b-11f1-8001-daa6852bd5a7
2026-04-19 23:50:43,438 - distributed.scheduler - INFO - Receive client connection: Client-940a349f-3c4a-11f1-8001-daa6852bd5a7
2026-04-19 23:40:41,993 - distributed.scheduler - INFO - Receive client connection: Client-2d8e0892-3c49-11f1-8001-daa6852bd5a7
2026-04-19 23:30:40,560 - distributed.scheduler - INFO - Receive client connection: Client-c7119bc5-3c47-11f1-8001-daa6852bd5a7
2026-04-19 23:20:39,131 - distributed.scheduler - INFO - Receive client connection: Client-60972ce0-3c46-11f1-8001-daa6852bd5a7
2026-04-19 23:13:01,034 - distributed.scheduler - INFO - Receive client connection: Client-4f8b0afb-3c45-11f1-8001-daa6852bd5a7
2026-04-19 23:00:36,103 - distributed.scheduler - INFO - Receive client connection: Client-9386c064-3c43-11f1-8001-daa6852bd5a7
2026-04-19 22:50:34,575 - distributed.scheduler - INFO - Receive client connection: Client-2cfdb96f-3c42-11f1-8001-daa6852bd5a7
2026-04-19 22:40:33,042 - distributed.scheduler - INFO - Receive client connection: Client-c672cadd-3c40-11f1-8001-daa6852bd5a7
2026-04-19 22:30:31,586 - distributed.scheduler - INFO - Receive client connection: Client-5ff3eb84-3c3f-11f1-8001-daa6852bd5a7
2026-04-19 22:20:30,158 - distributed.scheduler - INFO - Receive client connection: Client-f978d48e-3c3d-11f1-8001-daa6852bd5a7
2026-04-19 22:13:14,004 - distributed.scheduler - INFO - Receive client connection: Client-f5820b51-3c3c-11f1-8001-daa6852bd5a7
2026-04-19 22:00:27,030 - distributed.scheduler - INFO - Receive client connection: Client-2c5a95b6-3c3b-11f1-8001-daa6852bd5a7
2026-04-19 21:50:25,537 - distributed.scheduler - INFO - Receive client connection: Client-c5d5f532-3c39-11f1-8001-daa6852bd5a7
2026-04-19 21:40:24,040 - distributed.scheduler - INFO - Receive client connection: Client-5f51839f-3c38-11f1-8001-daa6852bd5a7
2026-04-19 21:30:22,565 - distributed.scheduler - INFO - Receive client connection: Client-f8cf15a9-3c36-11f1-8001-daa6852bd5a7
2026-04-19 21:20:21,063 - distributed.scheduler - INFO - Receive client connection: Client-9248c8ed-3c35-11f1-8001-daa6852bd5a7
2026-04-19 21:12:46,520 - distributed.scheduler - INFO - Receive client connection: Client-835c1d45-3c34-11f1-8001-daa6852bd5a7
2026-04-19 21:00:18,078 - distributed.scheduler - INFO - Receive client connection: Client-c5404d35-3c32-11f1-8001-daa6852bd5a7
2026-04-19 20:50:16,569 - distributed.scheduler - INFO - Receive client connection: Client-5eb8d3f5-3c31-11f1-8001-daa6852bd5a7
2026-04-19 20:40:15,041 - distributed.scheduler - INFO - Receive client connection: Client-f82fb729-3c2f-11f1-8001-daa6852bd5a7
2026-04-19 20:30:13,537 - distributed.scheduler - INFO - Receive client connection: Client-91a8e5d9-3c2e-11f1-8001-daa6852bd5a7
2026-04-19 20:20:12,080 - distributed.scheduler - INFO - Receive client connection: Client-2b2a4feb-3c2d-11f1-8001-daa6852bd5a7
2026-04-19 20:12:46,165 - distributed.scheduler - INFO - Receive client connection: Client-21617bf6-3c2c-11f1-8001-daa6852bd5a7
2026-04-19 20:00:09,221 - distributed.scheduler - INFO - Receive client connection: Client-5e34a7af-3c2a-11f1-8001-daa6852bd5a7
2026-04-19 19:50:07,798 - distributed.scheduler - INFO - Receive client connection: Client-f7ba9bad-3c28-11f1-8001-daa6852bd5a7
2026-04-19 19:40:06,194 - distributed.scheduler - INFO - Receive client connection: Client-91244095-3c27-11f1-8001-daa6852bd5a7
2026-04-19 19:30:04,710 - distributed.scheduler - INFO - Receive client connection: Client-2aa1703d-3c26-11f1-8001-daa6852bd5a7
2026-04-19 19:20:03,178 - distributed.scheduler - INFO - Receive client connection: Client-c4180985-3c24-11f1-8001-daa6852bd5a7
2026-04-19 19:12:24,391 - distributed.scheduler - INFO - Receive client connection: Client-b2a27180-3c23-11f1-8001-daa6852bd5a7
2026-04-19 19:00:06,174 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('len-agg-6362b17c02d4b5d60d99bbd96cc552dc', 0) NoneType: None
2026-04-19 19:00:06,174 - distributed.scheduler - ERROR - Couldn't gather keys {"('len-agg-6362b17c02d4b5d60d99bbd96cc552dc', 0)": []} state: ['waiting'] workers: [] NoneType: None
2026-04-19 19:00:00,171 - distributed.scheduler - INFO - Receive client connection: Client-f70b1bd1-3c21-11f1-8001-daa6852bd5a7
2026-04-19 18:50:58,843 - distributed.scheduler - INFO - Receive client connection: Client-b463e5a2-3c20-11f1-8001-daa6852bd5a7
2026-04-19 18:40:57,351 - distributed.scheduler - INFO - Receive client connection: Client-4ddcdfbb-3c1f-11f1-8001-daa6852bd5a7
2026-04-19 18:32:03,752 - distributed.scheduler - INFO - Close client connection: Client-e754b1cb-3c1d-11f1-8001-daa6852bd5a7
2026-04-19 18:32:03,752 - distributed.scheduler - INFO - Remove client Client-e754b1cb-3c1d-11f1-8001-daa6852bd5a7
2026-04-19 18:32:03,750 - distributed.scheduler - INFO - Remove client Client-e754b1cb-3c1d-11f1-8001-daa6852bd5a7
2026-04-19 18:30:55,831 - distributed.scheduler - INFO - Receive client connection: Client-e754b1cb-3c1d-11f1-8001-daa6852bd5a7
2026-04-19 18:21:34,268 - distributed.scheduler - INFO - Close client connection: Client-80c8bfb4-3c1c-11f1-8001-daa6852bd5a7
2026-04-19 18:21:34,267 - distributed.scheduler - INFO - Remove client Client-80c8bfb4-3c1c-11f1-8001-daa6852bd5a7
2026-04-19 18:21:34,265 - distributed.scheduler - INFO - Remove client Client-80c8bfb4-3c1c-11f1-8001-daa6852bd5a7
2026-04-19 18:20:54,288 - distributed.scheduler - INFO - Receive client connection: Client-80c8bfb4-3c1c-11f1-8001-daa6852bd5a7
2026-04-19 18:13:49,344 - distributed.scheduler - INFO - Close client connection: Client-6d36f6e7-3c1b-11f1-8001-daa6852bd5a7
2026-04-19 18:13:49,343 - distributed.scheduler - INFO - Remove client Client-6d36f6e7-3c1b-11f1-8001-daa6852bd5a7
2026-04-19 18:13:49,341 - distributed.scheduler - INFO - Remove client Client-6d36f6e7-3c1b-11f1-8001-daa6852bd5a7
2026-04-19 18:13:11,960 - distributed.scheduler - INFO - Receive client connection: Client-6d36f6e7-3c1b-11f1-8001-daa6852bd5a7
2026-04-19 18:02:08,235 - distributed.scheduler - INFO - Close client connection: Client-b3bad52b-3c19-11f1-8001-daa6852bd5a7
2026-04-19 18:02:08,235 - distributed.scheduler - INFO - Remove client Client-b3bad52b-3c19-11f1-8001-daa6852bd5a7
2026-04-19 18:02:08,234 - distributed.scheduler - INFO - Remove client Client-b3bad52b-3c19-11f1-8001-daa6852bd5a7
2026-04-19 18:00:51,291 - distributed.scheduler - INFO - Receive client connection: Client-b3bad52b-3c19-11f1-8001-daa6852bd5a7
2026-04-19 17:51:30,935 - distributed.scheduler - INFO - Close client connection: Client-4d3d917a-3c18-11f1-8001-daa6852bd5a7
2026-04-19 17:51:30,933 - distributed.scheduler - INFO - Remove client Client-4d3d917a-3c18-11f1-8001-daa6852bd5a7
2026-04-19 17:51:30,932 - distributed.scheduler - INFO - Remove client Client-4d3d917a-3c18-11f1-8001-daa6852bd5a7
2026-04-19 17:50:49,828 - distributed.scheduler - INFO - Receive client connection: Client-4d3d917a-3c18-11f1-8001-daa6852bd5a7
2026-04-19 17:41:29,647 - distributed.scheduler - INFO - Close client connection: Client-8043d4d6-3c15-11f1-8001-daa6852bd5a7
2026-04-19 17:41:29,646 - distributed.scheduler - INFO - Remove client Client-8043d4d6-3c15-11f1-8001-daa6852bd5a7
2026-04-19 17:41:28,991 - distributed.scheduler - INFO - Close client connection: Client-e6ce226f-3c16-11f1-8001-daa6852bd5a7
2026-04-19 17:41:28,990 - distributed.scheduler - INFO - Remove client Client-e6ce226f-3c16-11f1-8001-daa6852bd5a7
2026-04-19 17:41:28,988 - distributed.scheduler - INFO - Remove client Client-e6ce226f-3c16-11f1-8001-daa6852bd5a7
2026-04-19 17:40:48,462 - distributed.scheduler - INFO - Receive client connection: Client-e6ce226f-3c16-11f1-8001-daa6852bd5a7
2026-04-19 17:30:46,934 - distributed.scheduler - INFO - Receive client connection: Client-8043d4d6-3c15-11f1-8001-daa6852bd5a7
2026-04-19 17:21:26,035 - distributed.scheduler - INFO - Close client connection: Client-19d0182c-3c14-11f1-8001-daa6852bd5a7
2026-04-19 17:21:26,033 - distributed.scheduler - INFO - Remove client Client-19d0182c-3c14-11f1-8001-daa6852bd5a7
2026-04-19 17:21:26,031 - distributed.scheduler - INFO - Remove client Client-19d0182c-3c14-11f1-8001-daa6852bd5a7
2026-04-19 17:20:45,557 - distributed.scheduler - INFO - Receive client connection: Client-19d0182c-3c14-11f1-8001-daa6852bd5a7
2026-04-19 17:13:41,056 - distributed.scheduler - INFO - Close client connection: Client-08647208-3c13-11f1-8001-daa6852bd5a7
2026-04-19 17:13:41,053 - distributed.scheduler - INFO - Remove client Client-08647208-3c13-11f1-8001-daa6852bd5a7
2026-04-19 17:13:41,051 - distributed.scheduler - INFO - Remove client Client-08647208-3c13-11f1-8001-daa6852bd5a7
2026-04-19 17:13:06,832 - distributed.scheduler - INFO - Receive client connection: Client-08647208-3c13-11f1-8001-daa6852bd5a7
2026-04-19 17:01:57,042 - distributed.scheduler - INFO - Close client connection: Client-4ccd2ca0-3c11-11f1-8001-daa6852bd5a7
2026-04-19 17:01:57,042 - distributed.scheduler - INFO - Remove client Client-4ccd2ca0-3c11-11f1-8001-daa6852bd5a7
2026-04-19 17:01:57,040 - distributed.scheduler - INFO - Remove client Client-4ccd2ca0-3c11-11f1-8001-daa6852bd5a7
2026-04-19 17:00:42,609 - distributed.scheduler - INFO - Receive client connection: Client-4ccd2ca0-3c11-11f1-8001-daa6852bd5a7
2026-04-19 16:51:20,589 - distributed.scheduler - INFO - Close client connection: Client-e64e46d3-3c0f-11f1-8001-daa6852bd5a7
2026-04-19 16:51:20,588 - distributed.scheduler - INFO - Remove client Client-e64e46d3-3c0f-11f1-8001-daa6852bd5a7
2026-04-19 16:51:20,586 - distributed.scheduler - INFO - Remove client Client-e64e46d3-3c0f-11f1-8001-daa6852bd5a7
2026-04-19 16:50:41,159 - distributed.scheduler - INFO - Receive client connection: Client-e64e46d3-3c0f-11f1-8001-daa6852bd5a7
2026-04-19 16:41:18,665 - distributed.scheduler - INFO - Close client connection: Client-7fd4c0a2-3c0e-11f1-8001-daa6852bd5a7
2026-04-19 16:41:18,664 - distributed.scheduler - INFO - Remove client Client-7fd4c0a2-3c0e-11f1-8001-daa6852bd5a7
2026-04-19 16:41:18,663 - distributed.scheduler - INFO - Remove client Client-7fd4c0a2-3c0e-11f1-8001-daa6852bd5a7
2026-04-19 16:40:39,739 - distributed.scheduler - INFO - Receive client connection: Client-7fd4c0a2-3c0e-11f1-8001-daa6852bd5a7
2026-04-19 16:31:51,594 - distributed.scheduler - INFO - Close client connection: Client-b2d4ba9e-3c0b-11f1-8001-daa6852bd5a7
2026-04-19 16:31:51,593 - distributed.scheduler - INFO - Remove client Client-b2d4ba9e-3c0b-11f1-8001-daa6852bd5a7
2026-04-19 16:31:50,964 - distributed.scheduler - INFO - Close client connection: Client-1955a397-3c0d-11f1-8001-daa6852bd5a7
2026-04-19 16:31:50,963 - distributed.scheduler - INFO - Remove client Client-1955a397-3c0d-11f1-8001-daa6852bd5a7
2026-04-19 16:31:50,961 - distributed.scheduler - INFO - Remove client Client-1955a397-3c0d-11f1-8001-daa6852bd5a7
2026-04-19 16:30:38,265 - distributed.scheduler - INFO - Receive client connection: Client-1955a397-3c0d-11f1-8001-daa6852bd5a7
2026-04-19 16:20:36,803 - distributed.scheduler - INFO - Receive client connection: Client-b2d4ba9e-3c0b-11f1-8001-daa6852bd5a7
2026-04-19 16:13:38,382 - distributed.scheduler - INFO - Close client connection: Client-a48ae4d0-3c0a-11f1-8001-daa6852bd5a7
2026-04-19 16:13:38,381 - distributed.scheduler - INFO - Remove client Client-a48ae4d0-3c0a-11f1-8001-daa6852bd5a7
2026-04-19 16:13:38,380 - distributed.scheduler - INFO - Remove client Client-a48ae4d0-3c0a-11f1-8001-daa6852bd5a7
2026-04-19 16:13:03,343 - distributed.scheduler - INFO - Receive client connection: Client-a48ae4d0-3c0a-11f1-8001-daa6852bd5a7
2026-04-19 16:01:49,307 - distributed.scheduler - INFO - Close client connection: Client-e5c847b9-3c08-11f1-8001-daa6852bd5a7
2026-04-19 16:01:49,306 - distributed.scheduler - INFO - Remove client Client-e5c847b9-3c08-11f1-8001-daa6852bd5a7
2026-04-19 16:01:49,304 - distributed.scheduler - INFO - Remove client Client-e5c847b9-3c08-11f1-8001-daa6852bd5a7
2026-04-19 16:00:33,802 - distributed.scheduler - INFO - Receive client connection: Client-e5c847b9-3c08-11f1-8001-daa6852bd5a7
2026-04-19 15:51:13,190 - distributed.scheduler - INFO - Close client connection: Client-7f4107cd-3c07-11f1-8001-daa6852bd5a7
2026-04-19 15:51:13,189 - distributed.scheduler - INFO - Remove client Client-7f4107cd-3c07-11f1-8001-daa6852bd5a7
2026-04-19 15:51:13,187 - distributed.scheduler - INFO - Remove client Client-7f4107cd-3c07-11f1-8001-daa6852bd5a7
2026-04-19 15:50:32,289 - distributed.scheduler - INFO - Receive client connection: Client-7f4107cd-3c07-11f1-8001-daa6852bd5a7
2026-04-19 15:41:09,996 - distributed.scheduler - INFO - Close client connection: Client-18bbda61-3c06-11f1-8001-daa6852bd5a7
2026-04-19 15:41:09,994 - distributed.scheduler - INFO - Remove client Client-18bbda61-3c06-11f1-8001-daa6852bd5a7
2026-04-19 15:41:09,993 - distributed.scheduler - INFO - Remove client Client-18bbda61-3c06-11f1-8001-daa6852bd5a7
2026-04-19 15:40:30,793 - distributed.scheduler - INFO - Receive client connection: Client-18bbda61-3c06-11f1-8001-daa6852bd5a7
2026-04-19 15:31:42,776 - distributed.scheduler - INFO - Close client connection: Client-b23a20f5-3c04-11f1-8001-daa6852bd5a7
2026-04-19 15:31:42,775 - distributed.scheduler - INFO - Remove client Client-b23a20f5-3c04-11f1-8001-daa6852bd5a7
2026-04-19 15:31:42,774 - distributed.scheduler - INFO - Remove client Client-b23a20f5-3c04-11f1-8001-daa6852bd5a7
2026-04-19 15:30:29,322 - distributed.scheduler - INFO - Receive client connection: Client-b23a20f5-3c04-11f1-8001-daa6852bd5a7
2026-04-19 15:21:06,675 - distributed.scheduler - INFO - Close client connection: Client-4bb4c71f-3c03-11f1-8001-daa6852bd5a7
2026-04-19 15:21:06,674 - distributed.scheduler - INFO - Remove client Client-4bb4c71f-3c03-11f1-8001-daa6852bd5a7
2026-04-19 15:21:06,672 - distributed.scheduler - INFO - Remove client Client-4bb4c71f-3c03-11f1-8001-daa6852bd5a7
2026-04-19 15:20:27,812 - distributed.scheduler - INFO - Receive client connection: Client-4bb4c71f-3c03-11f1-8001-daa6852bd5a7
2026-04-19 15:13:12,524 - distributed.scheduler - INFO - Close client connection: Client-32b672d5-3c02-11f1-8001-daa6852bd5a7
2026-04-19 15:13:12,523 - distributed.scheduler - INFO - Remove client Client-32b672d5-3c02-11f1-8001-daa6852bd5a7
2026-04-19 15:13:12,521 - distributed.scheduler - INFO - Remove client Client-32b672d5-3c02-11f1-8001-daa6852bd5a7
2026-04-19 15:12:36,428 - distributed.scheduler - INFO - Receive client connection: Client-32b672d5-3c02-11f1-8001-daa6852bd5a7
2026-04-19 15:01:36,335 - distributed.scheduler - INFO - Close client connection: Client-7e9d44d3-3c00-11f1-8001-daa6852bd5a7
2026-04-19 15:01:36,334 - distributed.scheduler - INFO - Remove client Client-7e9d44d3-3c00-11f1-8001-daa6852bd5a7
2026-04-19 15:01:36,332 - distributed.scheduler - INFO - Remove client Client-7e9d44d3-3c00-11f1-8001-daa6852bd5a7
2026-04-19 15:00:24,741 - distributed.scheduler - INFO - Receive client connection: Client-7e9d44d3-3c00-11f1-8001-daa6852bd5a7
2026-04-19 14:51:03,737 - distributed.scheduler - INFO - Close client connection: Client-18149f37-3bff-11f1-8001-daa6852bd5a7
2026-04-19 14:51:03,735 - distributed.scheduler - INFO - Remove client Client-18149f37-3bff-11f1-8001-daa6852bd5a7
2026-04-19 14:51:03,733 - distributed.scheduler - INFO - Remove client Client-18149f37-3bff-11f1-8001-daa6852bd5a7
2026-04-19 14:50:23,223 - distributed.scheduler - INFO - Receive client connection: Client-18149f37-3bff-11f1-8001-daa6852bd5a7
2026-04-19 14:41:03,837 - distributed.scheduler - INFO - Close client connection: Client-b1894a50-3bfd-11f1-8001-daa6852bd5a7
2026-04-19 14:41:03,835 - distributed.scheduler - INFO - Remove client Client-b1894a50-3bfd-11f1-8001-daa6852bd5a7
2026-04-19 14:41:03,833 - distributed.scheduler - INFO - Remove client Client-b1894a50-3bfd-11f1-8001-daa6852bd5a7
2026-04-19 14:40:21,699 - distributed.scheduler - INFO - Receive client connection: Client-b1894a50-3bfd-11f1-8001-daa6852bd5a7
2026-04-19 14:31:38,855 - distributed.scheduler - INFO - Close client connection: Client-4b02952e-3bfc-11f1-8001-daa6852bd5a7
2026-04-19 14:31:38,855 - distributed.scheduler - INFO - Remove client Client-4b02952e-3bfc-11f1-8001-daa6852bd5a7
2026-04-19 14:31:38,853 - distributed.scheduler - INFO - Remove client Client-4b02952e-3bfc-11f1-8001-daa6852bd5a7
2026-04-19 14:30:24,542 - distributed.scheduler - INFO - Receive client connection: Client-worker-4d9e0091-3bfc-11f1-8288-b6f02dd406e9
2026-04-19 14:30:20,177 - distributed.scheduler - INFO - Receive client connection: Client-4b02952e-3bfc-11f1-8001-daa6852bd5a7
2026-04-19 14:20:55,994 - distributed.scheduler - INFO - Close client connection: Client-e485a50d-3bfa-11f1-8001-daa6852bd5a7
2026-04-19 14:20:55,993 - distributed.scheduler - INFO - Remove client Client-e485a50d-3bfa-11f1-8001-daa6852bd5a7
2026-04-19 14:20:55,991 - distributed.scheduler - INFO - Remove client Client-e485a50d-3bfa-11f1-8001-daa6852bd5a7
2026-04-19 14:20:18,728 - distributed.scheduler - INFO - Receive client connection: Client-e485a50d-3bfa-11f1-8001-daa6852bd5a7
2026-04-19 14:13:00,556 - distributed.scheduler - INFO - Close client connection: Client-cbe87dfd-3bf9-11f1-8001-daa6852bd5a7
2026-04-19 14:13:00,556 - distributed.scheduler - INFO - Remove client Client-cbe87dfd-3bf9-11f1-8001-daa6852bd5a7
2026-04-19 14:13:00,555 - distributed.scheduler - INFO - Remove client Client-cbe87dfd-3bf9-11f1-8001-daa6852bd5a7
2026-04-19 14:12:27,939 - distributed.scheduler - INFO - Receive client connection: Client-cbe87dfd-3bf9-11f1-8001-daa6852bd5a7
2026-04-19 14:04:02,640 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:46347 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:46347 after 30 s
2026-04-19 14:04:02,632 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:46347 after 30 s')
2026-04-19 14:04:02,488 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:46347 after 30 s')
2026-04-19 14:04:02,476 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:41873 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:46347 after 30 s')
2026-04-19 14:04:02,472 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:46347 after 30 s')
2026-04-19 14:04:02,470 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:46347 after 30 s')
2026-04-19 14:04:02,460 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:46347 after 30 s')
2026-04-19 14:04:02,458 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:46347 after 30 s')
2026-04-19 14:04:02,458 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:40855 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:46347 after 30 s')
2026-04-19 14:04:02,457 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:42859 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:46347 after 30 s')
2026-04-19 14:04:02,427 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:46347 after 30 s')
2026-04-19 14:04:02,421 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:46347 after 30 s')
2026-04-19 14:03:33,315 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:33073
2026-04-19 14:03:33,313 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:33073', status: closed, memory: 0, processing: 0>
2026-04-19 14:03:32,316 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:46347', status: closed, memory: 0, processing: 0>
2026-04-19 14:03:23,173 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:46347
2026-04-19 14:03:23,171 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:46347', status: closed, memory: 0, processing: 0>
2026-04-19 14:03:22,216 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:43329', status: closed, memory: 0, processing: 0>
2026-04-19 14:03:10,123 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:36673 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:36673 after 30 s
2026-04-19 14:03:10,011 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:36673 after 30 s')
2026-04-19 14:03:09,960 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:36673 after 30 s')
2026-04-19 14:03:09,918 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:36673 after 30 s')
2026-04-19 14:03:09,908 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:36673 after 30 s')
2026-04-19 14:02:40,813 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:43329
2026-04-19 14:02:40,811 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:43329', status: closed, memory: 0, processing: 0>
2026-04-19 14:02:39,822 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:36673', status: closed, memory: 0, processing: 0>
2026-04-19 14:02:15,013 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:35189 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:35189 after 30 s
2026-04-19 14:02:14,968 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:35189 after 30 s')
2026-04-19 14:02:14,916 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:35189 after 30 s')
2026-04-19 14:02:14,910 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:40855 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:35189 after 30 s')
2026-04-19 14:02:14,909 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:35189 after 30 s')
2026-04-19 14:02:14,887 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:35189 after 30 s')
2026-04-19 14:02:14,877 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:35189 after 30 s')
2026-04-19 14:02:14,874 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:35189 after 30 s')
2026-04-19 14:02:14,874 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:35189 after 30 s')
2026-04-19 14:02:14,832 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:35189 after 30 s')
2026-04-19 14:02:14,821 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:42859 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:35189 after 30 s')
2026-04-19 14:01:45,642 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:36673
2026-04-19 14:01:45,642 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:36673', status: closed, memory: 0, processing: 0>
2026-04-19 14:01:44,708 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:35189', status: closed, memory: 0, processing: 0>
2026-04-19 14:01:37,373 - distributed.scheduler - INFO - Close client connection: Client-177573c6-3bf8-11f1-8001-daa6852bd5a7
2026-04-19 14:01:37,373 - distributed.scheduler - INFO - Remove client Client-177573c6-3bf8-11f1-8001-daa6852bd5a7
2026-04-19 14:01:37,372 - distributed.scheduler - INFO - Remove client Client-177573c6-3bf8-11f1-8001-daa6852bd5a7
2026-04-19 14:01:30,418 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:40179 after 30 s
2026-04-19 14:01:30,361 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:35189 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:40179 after 30 s')
2026-04-19 14:01:30,340 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: OSError: Timed out trying to connect to tcp://172.21.151.150:40179 after 30 s
2026-04-19 14:01:28,561 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:40179 after 30 s')
2026-04-19 14:01:27,796 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:40179 after 30 s')
2026-04-19 14:01:27,729 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:40855 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.150:40179 after 30 s')
2026-04-19 14:00:58,061 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.150:35189
2026-04-19 14:00:58,060 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.150:35189', status: closed, memory: 0, processing: 0>
2026-04-19 14:00:57,055 - distributed.scheduler - INFO - Close client connection: Client-worker-d473561d-37e8-11f1-8074-b6f02dd406e9
2026-04-19 14:00:57,037 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.150:40179', status: closed, memory: 0, processing: 0>
2026-04-19 14:00:57,034 - distributed.scheduler - INFO - Remove client Client-worker-d473561d-37e8-11f1-8074-b6f02dd406e9
2026-04-19 14:00:15,700 - distributed.scheduler - INFO - Receive client connection: Client-177573c6-3bf8-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,510 - distributed.scheduler - INFO - Close client connection: Client-0509febb-3bd1-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,510 - distributed.scheduler - INFO - Close client connection: Client-6b87f937-3bd2-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,509 - distributed.scheduler - INFO - Close client connection: Client-d20a4b04-3bd3-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,509 - distributed.scheduler - INFO - Close client connection: Client-3885d2f9-3bd5-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,509 - distributed.scheduler - INFO - Close client connection: Client-9efe25e9-3bd6-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,509 - distributed.scheduler - INFO - Close client connection: Client-6bedfb5f-3bd9-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,509 - distributed.scheduler - INFO - Close client connection: Client-3dea072e-3bd8-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,509 - distributed.scheduler - INFO - Close client connection: Client-9f8c27bf-3bdd-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,509 - distributed.scheduler - INFO - Close client connection: Client-d2759eb6-3bda-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,509 - distributed.scheduler - INFO - Close client connection: Client-38fd4814-3bdc-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,509 - distributed.scheduler - INFO - Close client connection: Client-399dfa3c-3be3-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,508 - distributed.scheduler - INFO - Close client connection: Client-061d7484-3bdf-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,508 - distributed.scheduler - INFO - Close client connection: Client-d3122a02-3be1-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,508 - distributed.scheduler - INFO - Close client connection: Client-c32627ee-3be0-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,508 - distributed.scheduler - INFO - Close client connection: Client-49e83dc8-3bee-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,508 - distributed.scheduler - INFO - Close client connection: Client-a024b785-3be4-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,507 - distributed.scheduler - INFO - Close client connection: Client-06a8134a-3be6-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,507 - distributed.scheduler - INFO - Close client connection: Client-e358ec99-3bec-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,507 - distributed.scheduler - INFO - Close client connection: Client-7cccb374-3beb-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,507 - distributed.scheduler - INFO - Close client connection: Client-6d2b188c-3be7-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,507 - distributed.scheduler - INFO - Close client connection: Client-16440b7c-3bea-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,507 - distributed.scheduler - INFO - Close client connection: Client-1e1f36b7-3be9-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,507 - distributed.scheduler - INFO - Close client connection: Client-b0733897-3bef-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,506 - distributed.scheduler - INFO - Close client connection: Client-7d61df17-3bf2-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,506 - distributed.scheduler - INFO - Close client connection: Client-774fad50-3bf1-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,506 - distributed.scheduler - INFO - Close client connection: Client-e3f9720f-3bf3-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,506 - distributed.scheduler - INFO - Close client connection: Client-4a704e25-3bf5-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,503 - distributed.scheduler - INFO - Remove client Client-0509febb-3bd1-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,503 - distributed.scheduler - INFO - Remove client Client-6b87f937-3bd2-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,502 - distributed.scheduler - INFO - Remove client Client-d20a4b04-3bd3-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,502 - distributed.scheduler - INFO - Remove client Client-3885d2f9-3bd5-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,501 - distributed.scheduler - INFO - Remove client Client-9efe25e9-3bd6-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,501 - distributed.scheduler - INFO - Remove client Client-6bedfb5f-3bd9-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,501 - distributed.scheduler - INFO - Remove client Client-3dea072e-3bd8-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,500 - distributed.scheduler - INFO - Remove client Client-9f8c27bf-3bdd-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,500 - distributed.scheduler - INFO - Remove client Client-d2759eb6-3bda-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,500 - distributed.scheduler - INFO - Remove client Client-38fd4814-3bdc-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,499 - distributed.scheduler - INFO - Remove client Client-399dfa3c-3be3-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,499 - distributed.scheduler - INFO - Remove client Client-061d7484-3bdf-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,498 - distributed.scheduler - INFO - Remove client Client-d3122a02-3be1-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,493 - distributed.scheduler - INFO - Remove client Client-c32627ee-3be0-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,492 - distributed.scheduler - INFO - Remove client Client-49e83dc8-3bee-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,491 - distributed.scheduler - INFO - Remove client Client-a024b785-3be4-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,491 - distributed.scheduler - INFO - Remove client Client-06a8134a-3be6-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,490 - distributed.scheduler - INFO - Remove client Client-e358ec99-3bec-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,490 - distributed.scheduler - INFO - Remove client Client-7cccb374-3beb-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,489 - distributed.scheduler - INFO - Remove client Client-6d2b188c-3be7-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,488 - distributed.scheduler - INFO - Remove client Client-16440b7c-3bea-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,488 - distributed.scheduler - INFO - Remove client Client-1e1f36b7-3be9-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,487 - distributed.scheduler - INFO - Remove client Client-b0733897-3bef-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,486 - distributed.scheduler - INFO - Remove client Client-7d61df17-3bf2-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,486 - distributed.scheduler - INFO - Remove client Client-774fad50-3bf1-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,485 - distributed.scheduler - INFO - Remove client Client-e3f9720f-3bf3-11f1-8001-daa6852bd5a7
2026-04-19 13:50:56,484 - distributed.scheduler - INFO - Remove client Client-4a704e25-3bf5-11f1-8001-daa6852bd5a7
2026-04-19 13:50:55,582 - distributed.scheduler - INFO - Close client connection: Client-b0ed0007-3bf6-11f1-8001-daa6852bd5a7
2026-04-19 13:50:55,579 - distributed.scheduler - INFO - Remove client Client-b0ed0007-3bf6-11f1-8001-daa6852bd5a7
2026-04-19 13:50:55,577 - distributed.scheduler - INFO - Remove client Client-b0ed0007-3bf6-11f1-8001-daa6852bd5a7
2026-04-19 13:50:14,173 - distributed.scheduler - INFO - Receive client connection: Client-b0ed0007-3bf6-11f1-8001-daa6852bd5a7
2026-04-19 13:40:12,729 - distributed.scheduler - INFO - Receive client connection: Client-4a704e25-3bf5-11f1-8001-daa6852bd5a7
2026-04-19 13:30:11,325 - distributed.scheduler - INFO - Receive client connection: Client-e3f9720f-3bf3-11f1-8001-daa6852bd5a7
2026-04-19 13:20:09,706 - distributed.scheduler - INFO - Receive client connection: Client-7d61df17-3bf2-11f1-8001-daa6852bd5a7
2026-04-19 13:12:50,026 - distributed.scheduler - INFO - Receive client connection: Client-774fad50-3bf1-11f1-8001-daa6852bd5a7
2026-04-19 13:00:06,898 - distributed.scheduler - INFO - Receive client connection: Client-b0733897-3bef-11f1-8001-daa6852bd5a7
2026-04-19 12:50:05,368 - distributed.scheduler - INFO - Receive client connection: Client-49e83dc8-3bee-11f1-8001-daa6852bd5a7
2026-04-19 12:40:03,800 - distributed.scheduler - INFO - Receive client connection: Client-e358ec99-3bec-11f1-8001-daa6852bd5a7
2026-04-19 12:30:02,261 - distributed.scheduler - INFO - Receive client connection: Client-7cccb374-3beb-11f1-8001-daa6852bd5a7
2026-04-19 12:20:00,732 - distributed.scheduler - INFO - Receive client connection: Client-16440b7c-3bea-11f1-8001-daa6852bd5a7
2026-04-19 12:13:04,416 - distributed.scheduler - INFO - Receive client connection: Client-1e1f36b7-3be9-11f1-8001-daa6852bd5a7
2026-04-19 12:00:58,046 - distributed.scheduler - INFO - Receive client connection: Client-6d2b188c-3be7-11f1-8001-daa6852bd5a7
2026-04-19 11:50:56,563 - distributed.scheduler - INFO - Receive client connection: Client-06a8134a-3be6-11f1-8001-daa6852bd5a7
2026-04-19 11:40:55,075 - distributed.scheduler - INFO - Receive client connection: Client-a024b785-3be4-11f1-8001-daa6852bd5a7
2026-04-19 11:30:53,569 - distributed.scheduler - INFO - Receive client connection: Client-399dfa3c-3be3-11f1-8001-daa6852bd5a7
2026-04-19 11:20:52,022 - distributed.scheduler - INFO - Receive client connection: Client-d3122a02-3be1-11f1-8001-daa6852bd5a7
2026-04-19 11:13:15,816 - distributed.scheduler - INFO - Receive client connection: Client-c32627ee-3be0-11f1-8001-daa6852bd5a7
2026-04-19 11:00:49,175 - distributed.scheduler - INFO - Receive client connection: Client-061d7484-3bdf-11f1-8001-daa6852bd5a7
2026-04-19 10:50:47,598 - distributed.scheduler - INFO - Receive client connection: Client-9f8c27bf-3bdd-11f1-8001-daa6852bd5a7
2026-04-19 10:40:46,046 - distributed.scheduler - INFO - Receive client connection: Client-38fd4814-3bdc-11f1-8001-daa6852bd5a7
2026-04-19 10:30:44,525 - distributed.scheduler - INFO - Receive client connection: Client-d2759eb6-3bda-11f1-8001-daa6852bd5a7
2026-04-19 10:20:43,011 - distributed.scheduler - INFO - Receive client connection: Client-6bedfb5f-3bd9-11f1-8001-daa6852bd5a7
2026-04-19 10:12:16,311 - distributed.scheduler - INFO - Receive client connection: Client-3dea072e-3bd8-11f1-8001-daa6852bd5a7
2026-04-19 10:00:40,193 - distributed.scheduler - INFO - Receive client connection: Client-9efe25e9-3bd6-11f1-8001-daa6852bd5a7
2026-04-19 09:50:38,778 - distributed.scheduler - INFO - Receive client connection: Client-3885d2f9-3bd5-11f1-8001-daa6852bd5a7
2026-04-19 09:40:37,342 - distributed.scheduler - INFO - Receive client connection: Client-d20a4b04-3bd3-11f1-8001-daa6852bd5a7
2026-04-19 09:30:35,872 - distributed.scheduler - INFO - Receive client connection: Client-6b87f937-3bd2-11f1-8001-daa6852bd5a7
2026-04-19 09:20:34,430 - distributed.scheduler - INFO - Receive client connection: Client-0509febb-3bd1-11f1-8001-daa6852bd5a7
2026-04-19 09:13:23,815 - distributed.scheduler - INFO - Close client connection: Client-ef468bba-3bcf-11f1-8001-daa6852bd5a7
2026-04-19 09:13:23,814 - distributed.scheduler - INFO - Remove client Client-ef468bba-3bcf-11f1-8001-daa6852bd5a7
2026-04-19 09:13:23,812 - distributed.scheduler - INFO - Remove client Client-ef468bba-3bcf-11f1-8001-daa6852bd5a7
2026-04-19 09:12:48,415 - distributed.scheduler - INFO - Receive client connection: Client-ef468bba-3bcf-11f1-8001-daa6852bd5a7
2026-04-19 09:01:46,713 - distributed.scheduler - INFO - Close client connection: Client-37fd6979-3bce-11f1-8001-daa6852bd5a7
2026-04-19 09:01:46,713 - distributed.scheduler - INFO - Remove client Client-37fd6979-3bce-11f1-8001-daa6852bd5a7
2026-04-19 09:01:46,711 - distributed.scheduler - INFO - Remove client Client-37fd6979-3bce-11f1-8001-daa6852bd5a7
2026-04-19 09:00:31,415 - distributed.scheduler - INFO - Receive client connection: Client-37fd6979-3bce-11f1-8001-daa6852bd5a7
2026-04-19 08:51:10,159 - distributed.scheduler - INFO - Close client connection: Client-d186bb8e-3bcc-11f1-8001-daa6852bd5a7
2026-04-19 08:51:10,158 - distributed.scheduler - INFO - Remove client Client-d186bb8e-3bcc-11f1-8001-daa6852bd5a7
2026-04-19 08:51:10,156 - distributed.scheduler - INFO - Remove client Client-d186bb8e-3bcc-11f1-8001-daa6852bd5a7
2026-04-19 08:50:30,016 - distributed.scheduler - INFO - Receive client connection: Client-d186bb8e-3bcc-11f1-8001-daa6852bd5a7
2026-04-19 08:41:09,738 - distributed.scheduler - INFO - Close client connection: Client-6b116b38-3bcb-11f1-8001-daa6852bd5a7
2026-04-19 08:41:09,738 - distributed.scheduler - INFO - Remove client Client-6b116b38-3bcb-11f1-8001-daa6852bd5a7
2026-04-19 08:41:09,737 - distributed.scheduler - INFO - Remove client Client-6b116b38-3bcb-11f1-8001-daa6852bd5a7
2026-04-19 08:40:28,624 - distributed.scheduler - INFO - Receive client connection: Client-6b116b38-3bcb-11f1-8001-daa6852bd5a7
2026-04-19 08:31:42,679 - distributed.scheduler - INFO - Close client connection: Client-048dfeab-3bca-11f1-8001-daa6852bd5a7
2026-04-19 08:31:42,678 - distributed.scheduler - INFO - Remove client Client-048dfeab-3bca-11f1-8001-daa6852bd5a7
2026-04-19 08:31:42,676 - distributed.scheduler - INFO - Remove client Client-048dfeab-3bca-11f1-8001-daa6852bd5a7
2026-04-19 08:30:27,137 - distributed.scheduler - INFO - Receive client connection: Client-048dfeab-3bca-11f1-8001-daa6852bd5a7
2026-04-19 08:21:07,395 - distributed.scheduler - INFO - Close client connection: Client-9e0d744b-3bc8-11f1-8001-daa6852bd5a7
2026-04-19 08:21:07,395 - distributed.scheduler - INFO - Remove client Client-9e0d744b-3bc8-11f1-8001-daa6852bd5a7
2026-04-19 08:21:07,394 - distributed.scheduler - INFO - Remove client Client-9e0d744b-3bc8-11f1-8001-daa6852bd5a7
2026-04-19 08:20:25,661 - distributed.scheduler - INFO - Receive client connection: Client-9e0d744b-3bc8-11f1-8001-daa6852bd5a7
2026-04-19 08:13:40,363 - distributed.scheduler - INFO - Close client connection: Client-96724e68-3bc7-11f1-8001-daa6852bd5a7
2026-04-19 08:13:40,362 - distributed.scheduler - INFO - Remove client Client-96724e68-3bc7-11f1-8001-daa6852bd5a7
2026-04-19 08:13:40,360 - distributed.scheduler - INFO - Remove client Client-96724e68-3bc7-11f1-8001-daa6852bd5a7
2026-04-19 08:13:03,415 - distributed.scheduler - INFO - Receive client connection: Client-96724e68-3bc7-11f1-8001-daa6852bd5a7
2026-04-19 08:01:41,404 - distributed.scheduler - INFO - Close client connection: Client-d100dfdb-3bc5-11f1-8001-daa6852bd5a7
2026-04-19 08:01:41,401 - distributed.scheduler - INFO - Remove client Client-d100dfdb-3bc5-11f1-8001-daa6852bd5a7
2026-04-19 08:01:41,399 - distributed.scheduler - INFO - Remove client Client-d100dfdb-3bc5-11f1-8001-daa6852bd5a7
2026-04-19 08:00:22,659 - distributed.scheduler - INFO - Receive client connection: Client-d100dfdb-3bc5-11f1-8001-daa6852bd5a7
2026-04-19 07:51:02,356 - distributed.scheduler - INFO - Close client connection: Client-6a74f678-3bc4-11f1-8001-daa6852bd5a7
2026-04-19 07:51:02,354 - distributed.scheduler - INFO - Remove client Client-6a74f678-3bc4-11f1-8001-daa6852bd5a7
2026-04-19 07:51:02,351 - distributed.scheduler - INFO - Remove client Client-6a74f678-3bc4-11f1-8001-daa6852bd5a7
2026-04-19 07:50:21,119 - distributed.scheduler - INFO - Receive client connection: Client-6a74f678-3bc4-11f1-8001-daa6852bd5a7
2026-04-19 07:41:05,544 - distributed.scheduler - INFO - Close client connection: Client-03ef7157-3bc3-11f1-8001-daa6852bd5a7
2026-04-19 07:41:05,543 - distributed.scheduler - INFO - Remove client Client-03ef7157-3bc3-11f1-8001-daa6852bd5a7
2026-04-19 07:41:05,542 - distributed.scheduler - INFO - Remove client Client-03ef7157-3bc3-11f1-8001-daa6852bd5a7
2026-04-19 07:40:19,610 - distributed.scheduler - INFO - Receive client connection: Client-03ef7157-3bc3-11f1-8001-daa6852bd5a7
2026-04-19 07:31:31,713 - distributed.scheduler - INFO - Close client connection: Client-9d75047e-3bc1-11f1-8001-daa6852bd5a7
2026-04-19 07:31:31,713 - distributed.scheduler - INFO - Remove client Client-9d75047e-3bc1-11f1-8001-daa6852bd5a7
2026-04-19 07:31:31,711 - distributed.scheduler - INFO - Remove client Client-9d75047e-3bc1-11f1-8001-daa6852bd5a7
2026-04-19 07:30:18,216 - distributed.scheduler - INFO - Receive client connection: Client-9d75047e-3bc1-11f1-8001-daa6852bd5a7
2026-04-19 07:20:58,488 - distributed.scheduler - INFO - Close client connection: Client-36ee2a47-3bc0-11f1-8001-daa6852bd5a7
2026-04-19 07:20:58,486 - distributed.scheduler - INFO - Remove client Client-36ee2a47-3bc0-11f1-8001-daa6852bd5a7
2026-04-19 07:20:58,485 - distributed.scheduler - INFO - Remove client Client-36ee2a47-3bc0-11f1-8001-daa6852bd5a7
2026-04-19 07:20:16,685 - distributed.scheduler - INFO - Receive client connection: Client-36ee2a47-3bc0-11f1-8001-daa6852bd5a7
2026-04-19 07:13:36,840 - distributed.scheduler - INFO - Close client connection: Client-328df36c-3bbf-11f1-8001-daa6852bd5a7
2026-04-19 07:13:36,839 - distributed.scheduler - INFO - Remove client Client-328df36c-3bbf-11f1-8001-daa6852bd5a7
2026-04-19 07:13:36,839 - distributed.scheduler - INFO - Remove client Client-328df36c-3bbf-11f1-8001-daa6852bd5a7
2026-04-19 07:12:59,845 - distributed.scheduler - INFO - Receive client connection: Client-328df36c-3bbf-11f1-8001-daa6852bd5a7
2026-04-19 07:01:26,566 - distributed.scheduler - INFO - Close client connection: Client-69dd854c-3bbd-11f1-8001-daa6852bd5a7
2026-04-19 07:01:26,565 - distributed.scheduler - INFO - Remove client Client-69dd854c-3bbd-11f1-8001-daa6852bd5a7
2026-04-19 07:01:26,564 - distributed.scheduler - INFO - Remove client Client-69dd854c-3bbd-11f1-8001-daa6852bd5a7
2026-04-19 07:00:18,549 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('len-agg-98a5294f7adf9992c82bc64c96dba084', 0) NoneType: None
2026-04-19 07:00:18,549 - distributed.scheduler - ERROR - Couldn't gather keys {"('len-agg-98a5294f7adf9992c82bc64c96dba084', 0)": []} state: ['waiting'] workers: [] NoneType: None
2026-04-19 07:00:13,651 - distributed.scheduler - INFO - Receive client connection: Client-69dd854c-3bbd-11f1-8001-daa6852bd5a7
2026-04-19 06:50:48,959 - distributed.scheduler - INFO - Close client connection: Client-035fb41f-3bbc-11f1-8001-daa6852bd5a7
2026-04-19 06:50:48,958 - distributed.scheduler - INFO - Remove client Client-035fb41f-3bbc-11f1-8001-daa6852bd5a7
2026-04-19 06:50:48,957 - distributed.scheduler - INFO - Remove client Client-035fb41f-3bbc-11f1-8001-daa6852bd5a7
2026-04-19 06:50:12,198 - distributed.scheduler - INFO - Receive client connection: Client-035fb41f-3bbc-11f1-8001-daa6852bd5a7
2026-04-19 06:40:50,611 - distributed.scheduler - INFO - Close client connection: Client-9ce044b2-3bba-11f1-8001-daa6852bd5a7
2026-04-19 06:40:50,610 - distributed.scheduler - INFO - Remove client Client-9ce044b2-3bba-11f1-8001-daa6852bd5a7
2026-04-19 06:40:50,608 - distributed.scheduler - INFO - Remove client Client-9ce044b2-3bba-11f1-8001-daa6852bd5a7
2026-04-19 06:40:10,740 - distributed.scheduler - INFO - Receive client connection: Client-9ce044b2-3bba-11f1-8001-daa6852bd5a7
2026-04-19 06:31:28,068 - distributed.scheduler - INFO - Close client connection: Client-365f18a5-3bb9-11f1-8001-daa6852bd5a7
2026-04-19 06:31:28,067 - distributed.scheduler - INFO - Remove client Client-365f18a5-3bb9-11f1-8001-daa6852bd5a7
2026-04-19 06:31:28,065 - distributed.scheduler - INFO - Remove client Client-365f18a5-3bb9-11f1-8001-daa6852bd5a7
2026-04-19 06:30:09,267 - distributed.scheduler - INFO - Receive client connection: Client-365f18a5-3bb9-11f1-8001-daa6852bd5a7
2026-04-19 06:20:49,865 - distributed.scheduler - INFO - Close client connection: Client-cfe1e18a-3bb7-11f1-8001-daa6852bd5a7
2026-04-19 06:20:49,864 - distributed.scheduler - INFO - Remove client Client-cfe1e18a-3bb7-11f1-8001-daa6852bd5a7
2026-04-19 06:20:49,863 - distributed.scheduler - INFO - Remove client Client-cfe1e18a-3bb7-11f1-8001-daa6852bd5a7
2026-04-19 06:20:07,822 - distributed.scheduler - INFO - Receive client connection: Client-cfe1e18a-3bb7-11f1-8001-daa6852bd5a7
2026-04-19 06:13:07,497 - distributed.scheduler - INFO - Close client connection: Client-c0f5df0d-3bb6-11f1-8001-daa6852bd5a7
2026-04-19 06:13:07,497 - distributed.scheduler - INFO - Remove client Client-c0f5df0d-3bb6-11f1-8001-daa6852bd5a7
2026-04-19 06:13:07,496 - distributed.scheduler - INFO - Remove client Client-c0f5df0d-3bb6-11f1-8001-daa6852bd5a7
2026-04-19 06:12:33,292 - distributed.scheduler - INFO - Receive client connection: Client-c0f5df0d-3bb6-11f1-8001-daa6852bd5a7
2026-04-19 06:01:19,895 - distributed.scheduler - INFO - Close client connection: Client-02ce3b58-3bb5-11f1-8001-daa6852bd5a7
2026-04-19 06:01:19,894 - distributed.scheduler - INFO - Remove client Client-02ce3b58-3bb5-11f1-8001-daa6852bd5a7
2026-04-19 06:01:19,892 - distributed.scheduler - INFO - Remove client Client-02ce3b58-3bb5-11f1-8001-daa6852bd5a7
2026-04-19 06:00:04,773 - distributed.scheduler - INFO - Receive client connection: Client-02ce3b58-3bb5-11f1-8001-daa6852bd5a7
2026-04-19 05:50:43,669 - distributed.scheduler - INFO - Close client connection: Client-9c40a3fe-3bb3-11f1-8001-daa6852bd5a7
2026-04-19 05:50:43,667 - distributed.scheduler - INFO - Remove client Client-9c40a3fe-3bb3-11f1-8001-daa6852bd5a7
2026-04-19 05:50:43,666 - distributed.scheduler - INFO - Remove client Client-9c40a3fe-3bb3-11f1-8001-daa6852bd5a7
2026-04-19 05:50:03,219 - distributed.scheduler - INFO - Receive client connection: Client-9c40a3fe-3bb3-11f1-8001-daa6852bd5a7
2026-04-19 05:40:43,367 - distributed.scheduler - INFO - Close client connection: Client-35bafe0b-3bb2-11f1-8001-daa6852bd5a7
2026-04-19 05:40:43,365 - distributed.scheduler - INFO - Remove client Client-35bafe0b-3bb2-11f1-8001-daa6852bd5a7
2026-04-19 05:40:43,363 - distributed.scheduler - INFO - Remove client Client-35bafe0b-3bb2-11f1-8001-daa6852bd5a7
2026-04-19 05:40:01,718 - distributed.scheduler - INFO - Receive client connection: Client-35bafe0b-3bb2-11f1-8001-daa6852bd5a7
2026-04-19 05:31:19,117 - distributed.scheduler - INFO - Close client connection: Client-cf345dc7-3bb0-11f1-8001-daa6852bd5a7
2026-04-19 05:31:19,116 - distributed.scheduler - INFO - Remove client Client-cf345dc7-3bb0-11f1-8001-daa6852bd5a7
2026-04-19 05:31:19,114 - distributed.scheduler - INFO - Remove client Client-cf345dc7-3bb0-11f1-8001-daa6852bd5a7
2026-04-19 05:30:00,212 - distributed.scheduler - INFO - Receive client connection: Client-cf345dc7-3bb0-11f1-8001-daa6852bd5a7
2026-04-19 05:21:38,957 - distributed.scheduler - INFO - Close client connection: Client-8c8a3669-3baf-11f1-8001-daa6852bd5a7
2026-04-19 05:21:38,957 - distributed.scheduler - INFO - Remove client Client-8c8a3669-3baf-11f1-8001-daa6852bd5a7
2026-04-19 05:21:38,955 - distributed.scheduler - INFO - Remove client Client-8c8a3669-3baf-11f1-8001-daa6852bd5a7
2026-04-19 05:20:58,870 - distributed.scheduler - INFO - Receive client connection: Client-8c8a3669-3baf-11f1-8001-daa6852bd5a7
2026-04-19 05:13:57,533 - distributed.scheduler - INFO - Close client connection: Client-7b79ff83-3bae-11f1-8001-daa6852bd5a7
2026-04-19 05:13:57,532 - distributed.scheduler - INFO - Remove client Client-7b79ff83-3bae-11f1-8001-daa6852bd5a7
2026-04-19 05:13:57,530 - distributed.scheduler - INFO - Remove client Client-7b79ff83-3bae-11f1-8001-daa6852bd5a7
2026-04-19 05:13:20,746 - distributed.scheduler - INFO - Receive client connection: Client-7b79ff83-3bae-11f1-8001-daa6852bd5a7
2026-04-19 05:02:10,821 - distributed.scheduler - INFO - Close client connection: Client-bf7faca1-3bac-11f1-8001-daa6852bd5a7
2026-04-19 05:02:10,820 - distributed.scheduler - INFO - Remove client Client-bf7faca1-3bac-11f1-8001-daa6852bd5a7
2026-04-19 05:02:10,818 - distributed.scheduler - INFO - Remove client Client-bf7faca1-3bac-11f1-8001-daa6852bd5a7
2026-04-19 05:00:55,875 - distributed.scheduler - INFO - Receive client connection: Client-bf7faca1-3bac-11f1-8001-daa6852bd5a7
2026-04-19 04:51:35,045 - distributed.scheduler - INFO - Close client connection: Client-58eddc0a-3bab-11f1-8001-daa6852bd5a7
2026-04-19 04:51:35,044 - distributed.scheduler - INFO - Remove client Client-58eddc0a-3bab-11f1-8001-daa6852bd5a7
2026-04-19 04:51:35,042 - distributed.scheduler - INFO - Remove client Client-58eddc0a-3bab-11f1-8001-daa6852bd5a7
2026-04-19 04:50:54,298 - distributed.scheduler - INFO - Receive client connection: Client-58eddc0a-3bab-11f1-8001-daa6852bd5a7
2026-04-19 04:41:32,461 - distributed.scheduler - INFO - Close client connection: Client-f25e1ea0-3ba9-11f1-8001-daa6852bd5a7
2026-04-19 04:41:32,461 - distributed.scheduler - INFO - Remove client Client-f25e1ea0-3ba9-11f1-8001-daa6852bd5a7
2026-04-19 04:41:32,459 - distributed.scheduler - INFO - Remove client Client-f25e1ea0-3ba9-11f1-8001-daa6852bd5a7
2026-04-19 04:40:52,728 - distributed.scheduler - INFO - Receive client connection: Client-f25e1ea0-3ba9-11f1-8001-daa6852bd5a7
2026-04-19 04:32:10,218 - distributed.scheduler - INFO - Close client connection: Client-8bd544d1-3ba8-11f1-8001-daa6852bd5a7
2026-04-19 04:32:10,217 - distributed.scheduler - INFO - Remove client Client-8bd544d1-3ba8-11f1-8001-daa6852bd5a7
2026-04-19 04:32:10,215 - distributed.scheduler - INFO - Remove client Client-8bd544d1-3ba8-11f1-8001-daa6852bd5a7
2026-04-19 04:30:51,262 - distributed.scheduler - INFO - Receive client connection: Client-8bd544d1-3ba8-11f1-8001-daa6852bd5a7
2026-04-19 04:21:33,452 - distributed.scheduler - INFO - Close client connection: Client-2551a4f8-3ba7-11f1-8001-daa6852bd5a7
2026-04-19 04:21:33,452 - distributed.scheduler - INFO - Remove client Client-2551a4f8-3ba7-11f1-8001-daa6852bd5a7
2026-04-19 04:21:33,450 - distributed.scheduler - INFO - Remove client Client-2551a4f8-3ba7-11f1-8001-daa6852bd5a7
2026-04-19 04:20:49,713 - distributed.scheduler - INFO - Receive client connection: Client-2551a4f8-3ba7-11f1-8001-daa6852bd5a7
2026-04-19 04:13:50,921 - distributed.scheduler - INFO - Close client connection: Client-166f2606-3ba6-11f1-8001-daa6852bd5a7
2026-04-19 04:13:50,920 - distributed.scheduler - INFO - Remove client Client-166f2606-3ba6-11f1-8001-daa6852bd5a7
2026-04-19 04:13:50,919 - distributed.scheduler - INFO - Remove client Client-166f2606-3ba6-11f1-8001-daa6852bd5a7
2026-04-19 04:13:15,253 - distributed.scheduler - INFO - Receive client connection: Client-166f2606-3ba6-11f1-8001-daa6852bd5a7
2026-04-19 04:02:04,803 - distributed.scheduler - INFO - Close client connection: Client-584c7b52-3ba4-11f1-8001-daa6852bd5a7
2026-04-19 04:02:04,802 - distributed.scheduler - INFO - Remove client Client-584c7b52-3ba4-11f1-8001-daa6852bd5a7
2026-04-19 04:02:04,801 - distributed.scheduler - INFO - Remove client Client-584c7b52-3ba4-11f1-8001-daa6852bd5a7
2026-04-19 04:00:46,765 - distributed.scheduler - INFO - Receive client connection: Client-584c7b52-3ba4-11f1-8001-daa6852bd5a7
2026-04-19 03:51:26,784 - distributed.scheduler - INFO - Close client connection: Client-f1c3d2ab-3ba2-11f1-8001-daa6852bd5a7
2026-04-19 03:51:26,783 - distributed.scheduler - INFO - Remove client Client-f1c3d2ab-3ba2-11f1-8001-daa6852bd5a7
2026-04-19 03:51:26,782 - distributed.scheduler - INFO - Remove client Client-f1c3d2ab-3ba2-11f1-8001-daa6852bd5a7
2026-04-19 03:50:45,237 - distributed.scheduler - INFO - Receive client connection: Client-f1c3d2ab-3ba2-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,167 - distributed.scheduler - INFO - Close client connection: Client-758d6d50-3b57-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,167 - distributed.scheduler - INFO - Close client connection: Client-dc1677d5-3b58-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,167 - distributed.scheduler - INFO - Close client connection: Client-a91e41f4-3b5b-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,167 - distributed.scheduler - INFO - Close client connection: Client-97fa11dc-3b5a-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,167 - distributed.scheduler - INFO - Close client connection: Client-0f9ffd67-3b5d-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,167 - distributed.scheduler - INFO - Close client connection: Client-762979da-3b5e-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,166 - distributed.scheduler - INFO - Close client connection: Client-dcb1ac97-3b5f-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,166 - distributed.scheduler - INFO - Close client connection: Client-433414c6-3b61-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,166 - distributed.scheduler - INFO - Close client connection: Client-03d98490-3b63-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,166 - distributed.scheduler - INFO - Close client connection: Client-10407f69-3b64-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,166 - distributed.scheduler - INFO - Close client connection: Client-76bf9268-3b65-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,166 - distributed.scheduler - INFO - Close client connection: Client-dd3ddadc-3b66-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,166 - distributed.scheduler - INFO - Close client connection: Client-43b77ef5-3b68-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,166 - distributed.scheduler - INFO - Close client connection: Client-aa2de997-3b69-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,166 - distributed.scheduler - INFO - Close client connection: Client-77428374-3b6c-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,166 - distributed.scheduler - INFO - Close client connection: Client-5d0b2847-3b6b-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,165 - distributed.scheduler - INFO - Close client connection: Client-ddcdb14e-3b6d-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,165 - distributed.scheduler - INFO - Close client connection: Client-44519028-3b6f-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,165 - distributed.scheduler - INFO - Close client connection: Client-aae06521-3b70-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,165 - distributed.scheduler - INFO - Close client connection: Client-1155f318-3b72-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,165 - distributed.scheduler - INFO - Close client connection: Client-bd570333-3b73-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,165 - distributed.scheduler - INFO - Close client connection: Client-de6d86b7-3b74-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,165 - distributed.scheduler - INFO - Close client connection: Client-44f39351-3b76-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,165 - distributed.scheduler - INFO - Close client connection: Client-ab7ae5e2-3b77-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,165 - distributed.scheduler - INFO - Close client connection: Client-ee1975e2-3b78-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,165 - distributed.scheduler - INFO - Close client connection: Client-54a2ca2d-3b7a-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,164 - distributed.scheduler - INFO - Close client connection: Client-21bbe1b6-3b7d-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,164 - distributed.scheduler - INFO - Close client connection: Client-eecea654-3b7f-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,164 - distributed.scheduler - INFO - Close client connection: Client-06e7b50a-3b7c-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,164 - distributed.scheduler - INFO - Close client connection: Client-88433ac2-3b7e-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,164 - distributed.scheduler - INFO - Close client connection: Client-554a26c2-3b81-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,164 - distributed.scheduler - INFO - Close client connection: Client-bbcb5fc9-3b82-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,164 - distributed.scheduler - INFO - Close client connection: Client-88c26a06-3b85-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,164 - distributed.scheduler - INFO - Close client connection: Client-611d2409-3b84-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,162 - distributed.scheduler - INFO - Close client connection: Client-ef4afb0b-3b86-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,162 - distributed.scheduler - INFO - Close client connection: Client-55d8553c-3b88-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,162 - distributed.scheduler - INFO - Close client connection: Client-bc5bce20-3b89-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,161 - distributed.scheduler - INFO - Close client connection: Client-22e11dae-3b8b-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,161 - distributed.scheduler - INFO - Close client connection: Client-d23be155-3b8c-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,161 - distributed.scheduler - INFO - Close client connection: Client-efec84a6-3b8d-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,161 - distributed.scheduler - INFO - Close client connection: Client-56792641-3b8f-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,161 - distributed.scheduler - INFO - Close client connection: Client-bd048d51-3b90-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,161 - distributed.scheduler - INFO - Close client connection: Client-8a120ed7-3b93-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,161 - distributed.scheduler - INFO - Close client connection: Client-23911cff-3b92-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,161 - distributed.scheduler - INFO - Close client connection: Client-5715e21c-3b96-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,161 - distributed.scheduler - INFO - Close client connection: Client-2db23202-3b95-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,160 - distributed.scheduler - INFO - Close client connection: Client-bda0714c-3b97-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,160 - distributed.scheduler - INFO - Close client connection: Client-242dd4aa-3b99-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,160 - distributed.scheduler - INFO - Close client connection: Client-8ab7ce21-3b9a-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,160 - distributed.scheduler - INFO - Close client connection: Client-f145a5b4-3b9b-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,160 - distributed.scheduler - INFO - Close client connection: Client-be3132ed-3b9e-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,160 - distributed.scheduler - INFO - Close client connection: Client-b46ea8a5-3b9d-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,160 - distributed.scheduler - INFO - Close client connection: Client-24b4ad21-3ba0-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,156 - distributed.scheduler - INFO - Remove client Client-758d6d50-3b57-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,156 - distributed.scheduler - INFO - Remove client Client-dc1677d5-3b58-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,156 - distributed.scheduler - INFO - Remove client Client-a91e41f4-3b5b-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,156 - distributed.scheduler - INFO - Remove client Client-97fa11dc-3b5a-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,155 - distributed.scheduler - INFO - Remove client Client-0f9ffd67-3b5d-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,155 - distributed.scheduler - INFO - Remove client Client-762979da-3b5e-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,155 - distributed.scheduler - INFO - Remove client Client-dcb1ac97-3b5f-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,155 - distributed.scheduler - INFO - Remove client Client-433414c6-3b61-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,154 - distributed.scheduler - INFO - Remove client Client-03d98490-3b63-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,154 - distributed.scheduler - INFO - Remove client Client-10407f69-3b64-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,154 - distributed.scheduler - INFO - Remove client Client-76bf9268-3b65-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,154 - distributed.scheduler - INFO - Remove client Client-dd3ddadc-3b66-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,153 - distributed.scheduler - INFO - Remove client Client-43b77ef5-3b68-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,153 - distributed.scheduler - INFO - Remove client Client-aa2de997-3b69-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,153 - distributed.scheduler - INFO - Remove client Client-77428374-3b6c-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,153 - distributed.scheduler - INFO - Remove client Client-5d0b2847-3b6b-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,152 - distributed.scheduler - INFO - Remove client Client-ddcdb14e-3b6d-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,152 - distributed.scheduler - INFO - Remove client Client-44519028-3b6f-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,152 - distributed.scheduler - INFO - Remove client Client-aae06521-3b70-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,152 - distributed.scheduler - INFO - Remove client Client-1155f318-3b72-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,151 - distributed.scheduler - INFO - Remove client Client-bd570333-3b73-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,151 - distributed.scheduler - INFO - Remove client Client-de6d86b7-3b74-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,151 - distributed.scheduler - INFO - Remove client Client-44f39351-3b76-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,151 - distributed.scheduler - INFO - Remove client Client-ab7ae5e2-3b77-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,150 - distributed.scheduler - INFO - Remove client Client-ee1975e2-3b78-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,150 - distributed.scheduler - INFO - Remove client Client-54a2ca2d-3b7a-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,149 - distributed.scheduler - INFO - Remove client Client-21bbe1b6-3b7d-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,149 - distributed.scheduler - INFO - Remove client Client-eecea654-3b7f-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,149 - distributed.scheduler - INFO - Remove client Client-06e7b50a-3b7c-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,149 - distributed.scheduler - INFO - Remove client Client-88433ac2-3b7e-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,148 - distributed.scheduler - INFO - Remove client Client-554a26c2-3b81-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,148 - distributed.scheduler - INFO - Remove client Client-bbcb5fc9-3b82-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,148 - distributed.scheduler - INFO - Remove client Client-88c26a06-3b85-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,147 - distributed.scheduler - INFO - Remove client Client-611d2409-3b84-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,139 - distributed.scheduler - INFO - Remove client Client-ef4afb0b-3b86-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,139 - distributed.scheduler - INFO - Remove client Client-55d8553c-3b88-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,138 - distributed.scheduler - INFO - Remove client Client-bc5bce20-3b89-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,138 - distributed.scheduler - INFO - Remove client Client-22e11dae-3b8b-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,138 - distributed.scheduler - INFO - Remove client Client-d23be155-3b8c-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,137 - distributed.scheduler - INFO - Remove client Client-efec84a6-3b8d-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,136 - distributed.scheduler - INFO - Remove client Client-56792641-3b8f-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,136 - distributed.scheduler - INFO - Remove client Client-bd048d51-3b90-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,135 - distributed.scheduler - INFO - Remove client Client-8a120ed7-3b93-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,135 - distributed.scheduler - INFO - Remove client Client-23911cff-3b92-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,134 - distributed.scheduler - INFO - Remove client Client-5715e21c-3b96-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,134 - distributed.scheduler - INFO - Remove client Client-2db23202-3b95-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,133 - distributed.scheduler - INFO - Remove client Client-bda0714c-3b97-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,133 - distributed.scheduler - INFO - Remove client Client-242dd4aa-3b99-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,132 - distributed.scheduler - INFO - Remove client Client-8ab7ce21-3b9a-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,132 - distributed.scheduler - INFO - Remove client Client-f145a5b4-3b9b-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,131 - distributed.scheduler - INFO - Remove client Client-be3132ed-3b9e-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,131 - distributed.scheduler - INFO - Remove client Client-b46ea8a5-3b9d-11f1-8001-daa6852bd5a7
2026-04-19 03:41:24,130 - distributed.scheduler - INFO - Remove client Client-24b4ad21-3ba0-11f1-8001-daa6852bd5a7
2026-04-19 03:41:23,324 - distributed.scheduler - INFO - Close client connection: Client-8b37801c-3ba1-11f1-8001-daa6852bd5a7
2026-04-19 03:41:23,321 - distributed.scheduler - INFO - Remove client Client-8b37801c-3ba1-11f1-8001-daa6852bd5a7
2026-04-19 03:41:23,319 - distributed.scheduler - INFO - Remove client Client-8b37801c-3ba1-11f1-8001-daa6852bd5a7
2026-04-19 03:40:43,692 - distributed.scheduler - INFO - Receive client connection: Client-8b37801c-3ba1-11f1-8001-daa6852bd5a7
2026-04-19 03:30:42,203 - distributed.scheduler - INFO - Receive client connection: Client-24b4ad21-3ba0-11f1-8001-daa6852bd5a7
2026-04-19 03:20:40,722 - distributed.scheduler - INFO - Receive client connection: Client-be3132ed-3b9e-11f1-8001-daa6852bd5a7
2026-04-19 03:13:14,849 - distributed.scheduler - INFO - Receive client connection: Client-b46ea8a5-3b9d-11f1-8001-daa6852bd5a7
2026-04-19 03:00:37,923 - distributed.scheduler - INFO - Receive client connection: Client-f145a5b4-3b9b-11f1-8001-daa6852bd5a7
2026-04-19 02:50:36,376 - distributed.scheduler - INFO - Receive client connection: Client-8ab7ce21-3b9a-11f1-8001-daa6852bd5a7
2026-04-19 02:40:34,858 - distributed.scheduler - INFO - Receive client connection: Client-242dd4aa-3b99-11f1-8001-daa6852bd5a7
2026-04-19 02:30:33,296 - distributed.scheduler - INFO - Receive client connection: Client-bda0714c-3b97-11f1-8001-daa6852bd5a7
2026-04-19 02:20:31,771 - distributed.scheduler - INFO - Receive client connection: Client-5715e21c-3b96-11f1-8001-daa6852bd5a7
2026-04-19 02:12:12,815 - distributed.scheduler - INFO - Receive client connection: Client-2db23202-3b95-11f1-8001-daa6852bd5a7
2026-04-19 02:00:28,809 - distributed.scheduler - INFO - Receive client connection: Client-8a120ed7-3b93-11f1-8001-daa6852bd5a7
2026-04-19 01:50:27,344 - distributed.scheduler - INFO - Receive client connection: Client-23911cff-3b92-11f1-8001-daa6852bd5a7
2026-04-19 01:40:25,801 - distributed.scheduler - INFO - Receive client connection: Client-bd048d51-3b90-11f1-8001-daa6852bd5a7
2026-04-19 01:30:24,263 - distributed.scheduler - INFO - Receive client connection: Client-56792641-3b8f-11f1-8001-daa6852bd5a7
2026-04-19 01:20:22,710 - distributed.scheduler - INFO - Receive client connection: Client-efec84a6-3b8d-11f1-8001-daa6852bd5a7
2026-04-19 01:12:23,400 - distributed.scheduler - INFO - Receive client connection: Client-d23be155-3b8c-11f1-8001-daa6852bd5a7
2026-04-19 01:00:19,708 - distributed.scheduler - INFO - Receive client connection: Client-22e11dae-3b8b-11f1-8001-daa6852bd5a7
2026-04-19 00:50:18,212 - distributed.scheduler - INFO - Receive client connection: Client-bc5bce20-3b89-11f1-8001-daa6852bd5a7
2026-04-19 00:40:16,728 - distributed.scheduler - INFO - Receive client connection: Client-55d8553c-3b88-11f1-8001-daa6852bd5a7
2026-04-19 00:30:15,173 - distributed.scheduler - INFO - Receive client connection: Client-ef4afb0b-3b86-11f1-8001-daa6852bd5a7
2026-04-19 00:20:13,659 - distributed.scheduler - INFO - Receive client connection: Client-88c26a06-3b85-11f1-8001-daa6852bd5a7
2026-04-19 00:11:57,641 - distributed.scheduler - INFO - Receive client connection: Client-611d2409-3b84-11f1-8001-daa6852bd5a7
2026-04-19 00:00:10,789 - distributed.scheduler - INFO - Receive client connection: Client-bbcb5fc9-3b82-11f1-8001-daa6852bd5a7
2026-04-18 23:50:09,318 - distributed.scheduler - INFO - Receive client connection: Client-554a26c2-3b81-11f1-8001-daa6852bd5a7
2026-04-18 23:40:07,880 - distributed.scheduler - INFO - Receive client connection: Client-eecea654-3b7f-11f1-8001-daa6852bd5a7
2026-04-18 23:30:06,348 - distributed.scheduler - INFO - Receive client connection: Client-88433ac2-3b7e-11f1-8001-daa6852bd5a7
2026-04-18 23:20:04,833 - distributed.scheduler - INFO - Receive client connection: Client-21bbe1b6-3b7d-11f1-8001-daa6852bd5a7
2026-04-18 23:12:10,320 - distributed.scheduler - INFO - Receive client connection: Client-06e7b50a-3b7c-11f1-8001-daa6852bd5a7
2026-04-18 23:00:01,744 - distributed.scheduler - INFO - Receive client connection: Client-54a2ca2d-3b7a-11f1-8001-daa6852bd5a7
2026-04-18 22:50:00,214 - distributed.scheduler - INFO - Receive client connection: Client-ee1975e2-3b78-11f1-8001-daa6852bd5a7
2026-04-18 22:40:58,958 - distributed.scheduler - INFO - Receive client connection: Client-ab7ae5e2-3b77-11f1-8001-daa6852bd5a7
2026-04-18 22:30:57,445 - distributed.scheduler - INFO - Receive client connection: Client-44f39351-3b76-11f1-8001-daa6852bd5a7
2026-04-18 22:20:55,939 - distributed.scheduler - INFO - Receive client connection: Client-de6d86b7-3b74-11f1-8001-daa6852bd5a7
2026-04-18 22:12:50,927 - distributed.scheduler - INFO - Receive client connection: Client-bd570333-3b73-11f1-8001-daa6852bd5a7
2026-04-18 22:00:52,862 - distributed.scheduler - INFO - Receive client connection: Client-1155f318-3b72-11f1-8001-daa6852bd5a7
2026-04-18 21:50:51,460 - distributed.scheduler - INFO - Receive client connection: Client-aae06521-3b70-11f1-8001-daa6852bd5a7
2026-04-18 21:40:49,902 - distributed.scheduler - INFO - Receive client connection: Client-44519028-3b6f-11f1-8001-daa6852bd5a7
2026-04-18 21:30:48,413 - distributed.scheduler - INFO - Receive client connection: Client-ddcdb14e-3b6d-11f1-8001-daa6852bd5a7
2026-04-18 21:20:46,875 - distributed.scheduler - INFO - Receive client connection: Client-77428374-3b6c-11f1-8001-daa6852bd5a7
2026-04-18 21:12:53,393 - distributed.scheduler - INFO - Receive client connection: Client-5d0b2847-3b6b-11f1-8001-daa6852bd5a7
2026-04-18 21:00:43,813 - distributed.scheduler - INFO - Receive client connection: Client-aa2de997-3b69-11f1-8001-daa6852bd5a7
2026-04-18 20:50:42,412 - distributed.scheduler - INFO - Receive client connection: Client-43b77ef5-3b68-11f1-8001-daa6852bd5a7
2026-04-18 20:40:40,994 - distributed.scheduler - INFO - Receive client connection: Client-dd3ddadc-3b66-11f1-8001-daa6852bd5a7
2026-04-18 20:30:39,542 - distributed.scheduler - INFO - Receive client connection: Client-76bf9268-3b65-11f1-8001-daa6852bd5a7
2026-04-18 20:20:38,082 - distributed.scheduler - INFO - Receive client connection: Client-10407f69-3b64-11f1-8001-daa6852bd5a7
2026-04-18 20:13:07,773 - distributed.scheduler - INFO - Receive client connection: Client-03d98490-3b63-11f1-8001-daa6852bd5a7
2026-04-18 20:00:35,077 - distributed.scheduler - INFO - Receive client connection: Client-433414c6-3b61-11f1-8001-daa6852bd5a7
2026-04-18 19:50:33,599 - distributed.scheduler - INFO - Receive client connection: Client-dcb1ac97-3b5f-11f1-8001-daa6852bd5a7
2026-04-18 19:40:32,081 - distributed.scheduler - INFO - Receive client connection: Client-762979da-3b5e-11f1-8001-daa6852bd5a7
2026-04-18 19:30:30,556 - distributed.scheduler - INFO - Receive client connection: Client-0f9ffd67-3b5d-11f1-8001-daa6852bd5a7
2026-04-18 19:20:29,079 - distributed.scheduler - INFO - Receive client connection: Client-a91e41f4-3b5b-11f1-8001-daa6852bd5a7
2026-04-18 19:12:50,822 - distributed.scheduler - INFO - Receive client connection: Client-97fa11dc-3b5a-11f1-8001-daa6852bd5a7
2026-04-18 19:00:26,101 - distributed.scheduler - INFO - Receive client connection: Client-dc1677d5-3b58-11f1-8001-daa6852bd5a7
2026-04-18 18:50:24,592 - distributed.scheduler - INFO - Receive client connection: Client-758d6d50-3b57-11f1-8001-daa6852bd5a7
2026-04-18 18:40:59,757 - distributed.scheduler - INFO - Close client connection: Client-0f0caefd-3b56-11f1-8001-daa6852bd5a7
2026-04-18 18:40:59,755 - distributed.scheduler - INFO - Remove client Client-0f0caefd-3b56-11f1-8001-daa6852bd5a7
2026-04-18 18:40:59,754 - distributed.scheduler - INFO - Remove client Client-0f0caefd-3b56-11f1-8001-daa6852bd5a7
2026-04-18 18:40:23,124 - distributed.scheduler - INFO - Receive client connection: Client-0f0caefd-3b56-11f1-8001-daa6852bd5a7
2026-04-18 18:30:53,504 - distributed.scheduler - INFO - Close client connection: Client-a87f316e-3b54-11f1-8001-daa6852bd5a7
2026-04-18 18:30:53,503 - distributed.scheduler - INFO - Remove client Client-a87f316e-3b54-11f1-8001-daa6852bd5a7
2026-04-18 18:30:53,501 - distributed.scheduler - INFO - Remove client Client-a87f316e-3b54-11f1-8001-daa6852bd5a7
2026-04-18 18:30:21,568 - distributed.scheduler - INFO - Receive client connection: Client-a87f316e-3b54-11f1-8001-daa6852bd5a7
2026-04-18 18:20:52,960 - distributed.scheduler - INFO - Close client connection: Client-41f58fc5-3b53-11f1-8001-daa6852bd5a7
2026-04-18 18:20:52,957 - distributed.scheduler - INFO - Remove client Client-41f58fc5-3b53-11f1-8001-daa6852bd5a7
2026-04-18 18:20:52,954 - distributed.scheduler - INFO - Remove client Client-41f58fc5-3b53-11f1-8001-daa6852bd5a7
2026-04-18 18:20:20,055 - distributed.scheduler - INFO - Receive client connection: Client-41f58fc5-3b53-11f1-8001-daa6852bd5a7
2026-04-18 18:13:44,678 - distributed.scheduler - INFO - Close client connection: Client-41076343-3b52-11f1-8001-daa6852bd5a7
2026-04-18 18:13:44,677 - distributed.scheduler - INFO - Remove client Client-41076343-3b52-11f1-8001-daa6852bd5a7
2026-04-18 18:13:44,677 - distributed.scheduler - INFO - Remove client Client-41076343-3b52-11f1-8001-daa6852bd5a7
2026-04-18 18:13:08,984 - distributed.scheduler - INFO - Receive client connection: Client-41076343-3b52-11f1-8001-daa6852bd5a7
2026-04-18 18:01:29,822 - distributed.scheduler - INFO - Close client connection: Client-74f3f3a1-3b50-11f1-8001-daa6852bd5a7
2026-04-18 18:01:29,821 - distributed.scheduler - INFO - Remove client Client-74f3f3a1-3b50-11f1-8001-daa6852bd5a7
2026-04-18 18:01:29,820 - distributed.scheduler - INFO - Remove client Client-74f3f3a1-3b50-11f1-8001-daa6852bd5a7
2026-04-18 18:00:17,110 - distributed.scheduler - INFO - Receive client connection: Client-74f3f3a1-3b50-11f1-8001-daa6852bd5a7
2026-04-18 17:50:58,809 - distributed.scheduler - INFO - Close client connection: Client-0e6ba731-3b4f-11f1-8001-daa6852bd5a7
2026-04-18 17:50:58,808 - distributed.scheduler - INFO - Remove client Client-0e6ba731-3b4f-11f1-8001-daa6852bd5a7
2026-04-18 17:50:58,807 - distributed.scheduler - INFO - Remove client Client-0e6ba731-3b4f-11f1-8001-daa6852bd5a7
2026-04-18 17:50:15,591 - distributed.scheduler - INFO - Receive client connection: Client-0e6ba731-3b4f-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,244 - distributed.scheduler - INFO - Close client connection: Client-963c97a8-3b34-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,244 - distributed.scheduler - INFO - Close client connection: Client-fcc8addc-3b35-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,244 - distributed.scheduler - INFO - Close client connection: Client-634cc8cf-3b37-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,244 - distributed.scheduler - INFO - Close client connection: Client-305c61c5-3b3a-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,244 - distributed.scheduler - INFO - Close client connection: Client-15373260-3b39-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,244 - distributed.scheduler - INFO - Close client connection: Client-96eb4aa6-3b3b-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,244 - distributed.scheduler - INFO - Close client connection: Client-fd7613d0-3b3c-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,243 - distributed.scheduler - INFO - Close client connection: Client-6400493e-3b3e-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,243 - distributed.scheduler - INFO - Close client connection: Client-ca7c069e-3b3f-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,243 - distributed.scheduler - INFO - Close client connection: Client-73a9d8d6-3b42-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,243 - distributed.scheduler - INFO - Close client connection: Client-818efe51-3b41-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,242 - distributed.scheduler - INFO - Close client connection: Client-da375368-3b43-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,242 - distributed.scheduler - INFO - Close client connection: Client-40c10ea6-3b45-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,242 - distributed.scheduler - INFO - Close client connection: Client-a74e001a-3b46-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,242 - distributed.scheduler - INFO - Close client connection: Client-0dd0b41b-3b48-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,242 - distributed.scheduler - INFO - Close client connection: Client-d4b6bcc0-3b49-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,241 - distributed.scheduler - INFO - Close client connection: Client-dada3e79-3b4a-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,239 - distributed.scheduler - INFO - Close client connection: Client-415f8ddf-3b4c-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,237 - distributed.scheduler - INFO - Remove client Client-963c97a8-3b34-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,237 - distributed.scheduler - INFO - Remove client Client-fcc8addc-3b35-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,236 - distributed.scheduler - INFO - Remove client Client-634cc8cf-3b37-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,236 - distributed.scheduler - INFO - Remove client Client-305c61c5-3b3a-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,235 - distributed.scheduler - INFO - Remove client Client-15373260-3b39-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,235 - distributed.scheduler - INFO - Remove client Client-96eb4aa6-3b3b-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,234 - distributed.scheduler - INFO - Remove client Client-fd7613d0-3b3c-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,234 - distributed.scheduler - INFO - Remove client Client-6400493e-3b3e-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,233 - distributed.scheduler - INFO - Remove client Client-ca7c069e-3b3f-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,232 - distributed.scheduler - INFO - Remove client Client-73a9d8d6-3b42-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,231 - distributed.scheduler - INFO - Remove client Client-818efe51-3b41-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,231 - distributed.scheduler - INFO - Remove client Client-da375368-3b43-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,230 - distributed.scheduler - INFO - Remove client Client-40c10ea6-3b45-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,229 - distributed.scheduler - INFO - Remove client Client-a74e001a-3b46-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,228 - distributed.scheduler - INFO - Remove client Client-0dd0b41b-3b48-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,227 - distributed.scheduler - INFO - Remove client Client-d4b6bcc0-3b49-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,226 - distributed.scheduler - INFO - Remove client Client-dada3e79-3b4a-11f1-8001-daa6852bd5a7
2026-04-18 17:40:59,219 - distributed.scheduler - INFO - Remove client Client-415f8ddf-3b4c-11f1-8001-daa6852bd5a7
2026-04-18 17:40:58,496 - distributed.scheduler - INFO - Close client connection: Client-a7dfbadf-3b4d-11f1-8001-daa6852bd5a7
2026-04-18 17:40:58,493 - distributed.scheduler - INFO - Remove client Client-a7dfbadf-3b4d-11f1-8001-daa6852bd5a7
2026-04-18 17:40:58,492 - distributed.scheduler - INFO - Remove client Client-a7dfbadf-3b4d-11f1-8001-daa6852bd5a7
2026-04-18 17:40:14,041 - distributed.scheduler - INFO - Receive client connection: Client-a7dfbadf-3b4d-11f1-8001-daa6852bd5a7
2026-04-18 17:30:12,570 - distributed.scheduler - INFO - Receive client connection: Client-415f8ddf-3b4c-11f1-8001-daa6852bd5a7
2026-04-18 17:20:11,081 - distributed.scheduler - INFO - Receive client connection: Client-dada3e79-3b4a-11f1-8001-daa6852bd5a7
2026-04-18 17:12:51,276 - distributed.scheduler - INFO - Receive client connection: Client-d4b6bcc0-3b49-11f1-8001-daa6852bd5a7
2026-04-18 17:00:08,082 - distributed.scheduler - INFO - Receive client connection: Client-0dd0b41b-3b48-11f1-8001-daa6852bd5a7
2026-04-18 16:50:06,611 - distributed.scheduler - INFO - Receive client connection: Client-a74e001a-3b46-11f1-8001-daa6852bd5a7
2026-04-18 16:40:05,062 - distributed.scheduler - INFO - Receive client connection: Client-40c10ea6-3b45-11f1-8001-daa6852bd5a7
2026-04-18 16:30:03,540 - distributed.scheduler - INFO - Receive client connection: Client-da375368-3b43-11f1-8001-daa6852bd5a7
2026-04-18 16:20:01,979 - distributed.scheduler - INFO - Receive client connection: Client-73a9d8d6-3b42-11f1-8001-daa6852bd5a7
2026-04-18 16:13:15,792 - distributed.scheduler - INFO - Receive client connection: Client-818efe51-3b41-11f1-8001-daa6852bd5a7
2026-04-18 16:00:59,150 - distributed.scheduler - INFO - Receive client connection: Client-ca7c069e-3b3f-11f1-8001-daa6852bd5a7
2026-04-18 15:50:57,718 - distributed.scheduler - INFO - Receive client connection: Client-6400493e-3b3e-11f1-8001-daa6852bd5a7
2026-04-18 15:40:56,187 - distributed.scheduler - INFO - Receive client connection: Client-fd7613d0-3b3c-11f1-8001-daa6852bd5a7
2026-04-18 15:30:54,644 - distributed.scheduler - INFO - Receive client connection: Client-96eb4aa6-3b3b-11f1-8001-daa6852bd5a7
2026-04-18 15:20:53,093 - distributed.scheduler - INFO - Receive client connection: Client-305c61c5-3b3a-11f1-8001-daa6852bd5a7
2026-04-18 15:12:58,042 - distributed.scheduler - INFO - Receive client connection: Client-15373260-3b39-11f1-8001-daa6852bd5a7
2026-04-18 15:00:50,063 - distributed.scheduler - INFO - Receive client connection: Client-634cc8cf-3b37-11f1-8001-daa6852bd5a7
2026-04-18 14:50:48,574 - distributed.scheduler - INFO - Receive client connection: Client-fcc8addc-3b35-11f1-8001-daa6852bd5a7
2026-04-18 14:40:47,040 - distributed.scheduler - INFO - Receive client connection: Client-963c97a8-3b34-11f1-8001-daa6852bd5a7
2026-04-18 14:31:50,507 - distributed.scheduler - INFO - Close client connection: Client-2fa93a9a-3b33-11f1-8001-daa6852bd5a7
2026-04-18 14:31:50,506 - distributed.scheduler - INFO - Remove client Client-2fa93a9a-3b33-11f1-8001-daa6852bd5a7
2026-04-18 14:31:50,504 - distributed.scheduler - INFO - Remove client Client-2fa93a9a-3b33-11f1-8001-daa6852bd5a7
2026-04-18 14:30:45,450 - distributed.scheduler - INFO - Receive client connection: Client-2fa93a9a-3b33-11f1-8001-daa6852bd5a7
2026-04-18 14:21:22,086 - distributed.scheduler - INFO - Close client connection: Client-c9204ddf-3b31-11f1-8001-daa6852bd5a7
2026-04-18 14:21:22,085 - distributed.scheduler - INFO - Remove client Client-c9204ddf-3b31-11f1-8001-daa6852bd5a7
2026-04-18 14:21:22,083 - distributed.scheduler - INFO - Remove client Client-c9204ddf-3b31-11f1-8001-daa6852bd5a7
2026-04-18 14:20:43,931 - distributed.scheduler - INFO - Receive client connection: Client-c9204ddf-3b31-11f1-8001-daa6852bd5a7
2026-04-18 14:13:00,690 - distributed.scheduler - INFO - Close client connection: Client-a265cf5c-3b30-11f1-8001-daa6852bd5a7
2026-04-18 14:13:00,689 - distributed.scheduler - INFO - Remove client Client-a265cf5c-3b30-11f1-8001-daa6852bd5a7
2026-04-18 14:13:00,687 - distributed.scheduler - INFO - Remove client Client-a265cf5c-3b30-11f1-8001-daa6852bd5a7
2026-04-18 14:12:29,451 - distributed.scheduler - INFO - Receive client connection: Client-a265cf5c-3b30-11f1-8001-daa6852bd5a7
2026-04-18 14:01:54,905 - distributed.scheduler - INFO - Close client connection: Client-fc1e54f4-3b2e-11f1-8001-daa6852bd5a7
2026-04-18 14:01:54,904 - distributed.scheduler - INFO - Remove client Client-fc1e54f4-3b2e-11f1-8001-daa6852bd5a7
2026-04-18 14:01:54,902 - distributed.scheduler - INFO - Remove client Client-fc1e54f4-3b2e-11f1-8001-daa6852bd5a7
2026-04-18 14:00:40,991 - distributed.scheduler - INFO - Receive client connection: Client-fc1e54f4-3b2e-11f1-8001-daa6852bd5a7
2026-04-18 13:51:19,157 - distributed.scheduler - INFO - Close client connection: Client-95a1314d-3b2d-11f1-8001-daa6852bd5a7
2026-04-18 13:51:19,156 - distributed.scheduler - INFO - Remove client Client-95a1314d-3b2d-11f1-8001-daa6852bd5a7
2026-04-18 13:51:19,154 - distributed.scheduler - INFO - Remove client Client-95a1314d-3b2d-11f1-8001-daa6852bd5a7
2026-04-18 13:50:39,564 - distributed.scheduler - INFO - Receive client connection: Client-95a1314d-3b2d-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,395 - distributed.scheduler - INFO - Close client connection: Client-8e177a16-3a92-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,395 - distributed.scheduler - INFO - Close client connection: Client-713d7e39-3a91-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,395 - distributed.scheduler - INFO - Close client connection: Client-f4a10425-3a93-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,395 - distributed.scheduler - INFO - Close client connection: Client-5b257c62-3a95-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,395 - distributed.scheduler - INFO - Close client connection: Client-c195acfe-3a96-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,394 - distributed.scheduler - INFO - Close client connection: Client-043b89f9-3a98-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,394 - distributed.scheduler - INFO - Close client connection: Client-d13e8591-3a9a-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,394 - distributed.scheduler - INFO - Close client connection: Client-ad7c479c-3a99-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,394 - distributed.scheduler - INFO - Close client connection: Client-37c83313-3a9c-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,394 - distributed.scheduler - INFO - Close client connection: Client-9e55533a-3a9d-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,394 - distributed.scheduler - INFO - Close client connection: Client-04e73516-3a9f-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,394 - distributed.scheduler - INFO - Close client connection: Client-6b69852a-3aa0-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,394 - distributed.scheduler - INFO - Close client connection: Client-3878384a-3aa3-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,394 - distributed.scheduler - INFO - Close client connection: Client-132577aa-3aa2-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,394 - distributed.scheduler - INFO - Close client connection: Client-9f04014f-3aa4-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,393 - distributed.scheduler - INFO - Close client connection: Client-0592bb93-3aa6-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,393 - distributed.scheduler - INFO - Close client connection: Client-6c196c70-3aa7-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,393 - distributed.scheduler - INFO - Close client connection: Client-d2a3043e-3aa8-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,393 - distributed.scheduler - INFO - Close client connection: Client-9fab26c2-3aab-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,393 - distributed.scheduler - INFO - Close client connection: Client-9b295e42-3aaa-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,393 - distributed.scheduler - INFO - Close client connection: Client-d33f92c6-3aaf-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,393 - distributed.scheduler - INFO - Close client connection: Client-063721b9-3aad-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,393 - distributed.scheduler - INFO - Close client connection: Client-6cbb4aa3-3aae-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,393 - distributed.scheduler - INFO - Close client connection: Client-39bf7c33-3ab1-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,393 - distributed.scheduler - INFO - Close client connection: Client-06b56932-3ab4-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,393 - distributed.scheduler - INFO - Close client connection: Client-ff8c30a3-3ab2-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,393 - distributed.scheduler - INFO - Close client connection: Client-6d38da8c-3ab5-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,392 - distributed.scheduler - INFO - Close client connection: Client-d3bffab8-3ab6-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,392 - distributed.scheduler - INFO - Close client connection: Client-3a4d759b-3ab8-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,392 - distributed.scheduler - INFO - Close client connection: Client-a0d6da7a-3ab9-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,392 - distributed.scheduler - INFO - Close client connection: Client-6dd7d8d2-3abc-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,392 - distributed.scheduler - INFO - Close client connection: Client-3c6618e9-3abb-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,392 - distributed.scheduler - INFO - Close client connection: Client-d44fa103-3abd-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,392 - distributed.scheduler - INFO - Close client connection: Client-3ac71195-3abf-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,392 - distributed.scheduler - INFO - Close client connection: Client-a1416717-3ac0-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,392 - distributed.scheduler - INFO - Close client connection: Client-07c889ba-3ac2-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,392 - distributed.scheduler - INFO - Close client connection: Client-d4de57d3-3ac4-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,392 - distributed.scheduler - INFO - Close client connection: Client-bc81a986-3ac3-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,391 - distributed.scheduler - INFO - Close client connection: Client-3b56f9ca-3ac6-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,391 - distributed.scheduler - INFO - Close client connection: Client-a1eaf98c-3ac7-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,391 - distributed.scheduler - INFO - Close client connection: Client-08753b21-3ac9-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,391 - distributed.scheduler - INFO - Close client connection: Client-6f020e4c-3aca-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,391 - distributed.scheduler - INFO - Close client connection: Client-3c140d90-3acd-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,391 - distributed.scheduler - INFO - Close client connection: Client-11d78e8c-3acc-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,391 - distributed.scheduler - INFO - Close client connection: Client-a29db5c7-3ace-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,391 - distributed.scheduler - INFO - Close client connection: Client-e541b089-3acf-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,391 - distributed.scheduler - INFO - Close client connection: Client-4bc5edd0-3ad1-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,391 - distributed.scheduler - INFO - Close client connection: Client-b24e8ebf-3ad2-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,390 - distributed.scheduler - INFO - Close client connection: Client-7f581f95-3ad5-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,390 - distributed.scheduler - INFO - Close client connection: Client-6f136504-3ad4-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,390 - distributed.scheduler - INFO - Close client connection: Client-4c672e29-3ad8-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,390 - distributed.scheduler - INFO - Close client connection: Client-e5e8b082-3ad6-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,390 - distributed.scheduler - INFO - Close client connection: Client-b2ea5266-3ad9-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,390 - distributed.scheduler - INFO - Close client connection: Client-1968d9cd-3adb-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,390 - distributed.scheduler - INFO - Close client connection: Client-cefa8770-3adc-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,390 - distributed.scheduler - INFO - Close client connection: Client-e6695110-3add-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,390 - distributed.scheduler - INFO - Close client connection: Client-4cf2633c-3adf-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,390 - distributed.scheduler - INFO - Close client connection: Client-b374197d-3ae0-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,390 - distributed.scheduler - INFO - Close client connection: Client-19ed6625-3ae2-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,390 - distributed.scheduler - INFO - Close client connection: Client-8069cdcc-3ae3-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,389 - distributed.scheduler - INFO - Close client connection: Client-4d7b4f15-3ae6-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,389 - distributed.scheduler - INFO - Close client connection: Client-44e235c2-3ae5-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,389 - distributed.scheduler - INFO - Close client connection: Client-b4068512-3ae7-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,389 - distributed.scheduler - INFO - Close client connection: Client-1a8bde04-3ae9-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,389 - distributed.scheduler - INFO - Close client connection: Client-8111cc18-3aea-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,389 - distributed.scheduler - INFO - Close client connection: Client-b49d33c3-3aee-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,389 - distributed.scheduler - INFO - Close client connection: Client-e78f02e6-3aeb-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,389 - distributed.scheduler - INFO - Close client connection: Client-b8ba5739-3aed-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,389 - distributed.scheduler - INFO - Close client connection: Client-1b211ba1-3af0-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,389 - distributed.scheduler - INFO - Close client connection: Client-81aea304-3af1-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,389 - distributed.scheduler - INFO - Close client connection: Client-e830b976-3af2-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,389 - distributed.scheduler - INFO - Close client connection: Client-4eaa0a5d-3af4-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,388 - distributed.scheduler - INFO - Close client connection: Client-07658d2c-3af6-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,388 - distributed.scheduler - INFO - Close client connection: Client-1b99f054-3af7-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,388 - distributed.scheduler - INFO - Close client connection: Client-821f0b03-3af8-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,388 - distributed.scheduler - INFO - Close client connection: Client-4f2bd05f-3afb-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,388 - distributed.scheduler - INFO - Close client connection: Client-e8a4656a-3af9-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,388 - distributed.scheduler - INFO - Close client connection: Client-b5b00ece-3afc-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,388 - distributed.scheduler - INFO - Close client connection: Client-82b03197-3aff-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,388 - distributed.scheduler - INFO - Close client connection: Client-8375feb9-3afe-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,388 - distributed.scheduler - INFO - Close client connection: Client-e939bead-3b00-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,388 - distributed.scheduler - INFO - Close client connection: Client-4fc22650-3b02-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,388 - distributed.scheduler - INFO - Close client connection: Client-b63d117f-3b03-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,387 - distributed.scheduler - INFO - Close client connection: Client-1cc4f1f8-3b05-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,387 - distributed.scheduler - INFO - Close client connection: Client-e9b94919-3b07-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,385 - distributed.scheduler - INFO - Close client connection: Client-dac92f43-3b06-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,385 - distributed.scheduler - INFO - Close client connection: Client-2c608aed-3b09-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,385 - distributed.scheduler - INFO - Close client connection: Client-f962f2c6-3b0b-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,385 - distributed.scheduler - INFO - Close client connection: Client-92e3327d-3b0a-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,385 - distributed.scheduler - INFO - Close client connection: Client-5fe87119-3b0d-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,385 - distributed.scheduler - INFO - Close client connection: Client-2ce36041-3b10-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,385 - distributed.scheduler - INFO - Close client connection: Client-16d86351-3b0f-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,385 - distributed.scheduler - INFO - Close client connection: Client-93656697-3b11-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,385 - distributed.scheduler - INFO - Close client connection: Client-f9db8ebd-3b12-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,385 - distributed.scheduler - INFO - Close client connection: Client-605a27eb-3b14-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,384 - distributed.scheduler - INFO - Close client connection: Client-c6d92539-3b15-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,384 - distributed.scheduler - INFO - Close client connection: Client-a4626c79-3b17-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,384 - distributed.scheduler - INFO - Close client connection: Client-93dc2847-3b18-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,384 - distributed.scheduler - INFO - Close client connection: Client-fa6e78bb-3b19-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,384 - distributed.scheduler - INFO - Close client connection: Client-2debaa13-3b1e-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,384 - distributed.scheduler - INFO - Close client connection: Client-60e7a3e5-3b1b-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,384 - distributed.scheduler - INFO - Close client connection: Client-c769e53e-3b1c-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,384 - distributed.scheduler - INFO - Close client connection: Client-faf6f888-3b20-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,384 - distributed.scheduler - INFO - Close client connection: Client-f069285c-3b1f-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,384 - distributed.scheduler - INFO - Close client connection: Client-c8015b45-3b23-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,384 - distributed.scheduler - INFO - Close client connection: Client-617b9219-3b22-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,383 - distributed.scheduler - INFO - Close client connection: Client-2e80be02-3b25-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,383 - distributed.scheduler - INFO - Close client connection: Client-950bdef0-3b26-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,383 - distributed.scheduler - INFO - Close client connection: Client-4799b8b0-3b28-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,383 - distributed.scheduler - INFO - Close client connection: Client-6221a723-3b29-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,379 - distributed.scheduler - INFO - Close client connection: Client-c8a6cf0c-3b2a-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,379 - distributed.scheduler - INFO - Remove client Client-8e177a16-3a92-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,378 - distributed.scheduler - INFO - Remove client Client-713d7e39-3a91-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,378 - distributed.scheduler - INFO - Remove client Client-f4a10425-3a93-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,378 - distributed.scheduler - INFO - Remove client Client-5b257c62-3a95-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,378 - distributed.scheduler - INFO - Remove client Client-c195acfe-3a96-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,377 - distributed.scheduler - INFO - Remove client Client-043b89f9-3a98-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,377 - distributed.scheduler - INFO - Remove client Client-d13e8591-3a9a-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,377 - distributed.scheduler - INFO - Remove client Client-ad7c479c-3a99-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,377 - distributed.scheduler - INFO - Remove client Client-37c83313-3a9c-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,376 - distributed.scheduler - INFO - Remove client Client-9e55533a-3a9d-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,376 - distributed.scheduler - INFO - Remove client Client-04e73516-3a9f-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,376 - distributed.scheduler - INFO - Remove client Client-6b69852a-3aa0-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,376 - distributed.scheduler - INFO - Remove client Client-3878384a-3aa3-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,376 - distributed.scheduler - INFO - Remove client Client-132577aa-3aa2-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,375 - distributed.scheduler - INFO - Remove client Client-9f04014f-3aa4-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,375 - distributed.scheduler - INFO - Remove client Client-0592bb93-3aa6-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,375 - distributed.scheduler - INFO - Remove client Client-6c196c70-3aa7-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,375 - distributed.scheduler - INFO - Remove client Client-d2a3043e-3aa8-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,374 - distributed.scheduler - INFO - Remove client Client-9fab26c2-3aab-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,374 - distributed.scheduler - INFO - Remove client Client-9b295e42-3aaa-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,374 - distributed.scheduler - INFO - Remove client Client-d33f92c6-3aaf-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,374 - distributed.scheduler - INFO - Remove client Client-063721b9-3aad-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,373 - distributed.scheduler - INFO - Remove client Client-6cbb4aa3-3aae-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,373 - distributed.scheduler - INFO - Remove client Client-39bf7c33-3ab1-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,373 - distributed.scheduler - INFO - Remove client Client-06b56932-3ab4-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,372 - distributed.scheduler - INFO - Remove client Client-ff8c30a3-3ab2-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,372 - distributed.scheduler - INFO - Remove client Client-6d38da8c-3ab5-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,372 - distributed.scheduler - INFO - Remove client Client-d3bffab8-3ab6-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,372 - distributed.scheduler - INFO - Remove client Client-3a4d759b-3ab8-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,372 - distributed.scheduler - INFO - Remove client Client-a0d6da7a-3ab9-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,371 - distributed.scheduler - INFO - Remove client Client-6dd7d8d2-3abc-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,371 - distributed.scheduler - INFO - Remove client Client-3c6618e9-3abb-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,371 - distributed.scheduler - INFO - Remove client Client-d44fa103-3abd-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,371 - distributed.scheduler - INFO - Remove client Client-3ac71195-3abf-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,370 - distributed.scheduler - INFO - Remove client Client-a1416717-3ac0-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,370 - distributed.scheduler - INFO - Remove client Client-07c889ba-3ac2-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,370 - distributed.scheduler - INFO - Remove client Client-d4de57d3-3ac4-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,370 - distributed.scheduler - INFO - Remove client Client-bc81a986-3ac3-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,370 - distributed.scheduler - INFO - Remove client Client-3b56f9ca-3ac6-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,369 - distributed.scheduler - INFO - Remove client Client-a1eaf98c-3ac7-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,369 - distributed.scheduler - INFO - Remove client Client-08753b21-3ac9-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,369 - distributed.scheduler - INFO - Remove client Client-6f020e4c-3aca-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,369 - distributed.scheduler - INFO - Remove client Client-3c140d90-3acd-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,368 - distributed.scheduler - INFO - Remove client Client-11d78e8c-3acc-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,368 - distributed.scheduler - INFO - Remove client Client-a29db5c7-3ace-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,368 - distributed.scheduler - INFO - Remove client Client-e541b089-3acf-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,368 - distributed.scheduler - INFO - Remove client Client-4bc5edd0-3ad1-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,368 - distributed.scheduler - INFO - Remove client Client-b24e8ebf-3ad2-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,367 - distributed.scheduler - INFO - Remove client Client-7f581f95-3ad5-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,367 - distributed.scheduler - INFO - Remove client Client-6f136504-3ad4-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,367 - distributed.scheduler - INFO - Remove client Client-4c672e29-3ad8-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,366 - distributed.scheduler - INFO - Remove client Client-e5e8b082-3ad6-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,366 - distributed.scheduler - INFO - Remove client Client-b2ea5266-3ad9-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,366 - distributed.scheduler - INFO - Remove client Client-1968d9cd-3adb-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,366 - distributed.scheduler - INFO - Remove client Client-cefa8770-3adc-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,366 - distributed.scheduler - INFO - Remove client Client-e6695110-3add-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,365 - distributed.scheduler - INFO - Remove client Client-4cf2633c-3adf-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,365 - distributed.scheduler - INFO - Remove client Client-b374197d-3ae0-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,365 - distributed.scheduler - INFO - Remove client Client-19ed6625-3ae2-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,365 - distributed.scheduler - INFO - Remove client Client-8069cdcc-3ae3-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,364 - distributed.scheduler - INFO - Remove client Client-4d7b4f15-3ae6-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,364 - distributed.scheduler - INFO - Remove client Client-44e235c2-3ae5-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,364 - distributed.scheduler - INFO - Remove client Client-b4068512-3ae7-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,364 - distributed.scheduler - INFO - Remove client Client-1a8bde04-3ae9-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,363 - distributed.scheduler - INFO - Remove client Client-8111cc18-3aea-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,363 - distributed.scheduler - INFO - Remove client Client-b49d33c3-3aee-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,363 - distributed.scheduler - INFO - Remove client Client-e78f02e6-3aeb-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,363 - distributed.scheduler - INFO - Remove client Client-b8ba5739-3aed-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,362 - distributed.scheduler - INFO - Remove client Client-1b211ba1-3af0-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,362 - distributed.scheduler - INFO - Remove client Client-81aea304-3af1-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,362 - distributed.scheduler - INFO - Remove client Client-e830b976-3af2-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,362 - distributed.scheduler - INFO - Remove client Client-4eaa0a5d-3af4-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,362 - distributed.scheduler - INFO - Remove client Client-07658d2c-3af6-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,361 - distributed.scheduler - INFO - Remove client Client-1b99f054-3af7-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,361 - distributed.scheduler - INFO - Remove client Client-821f0b03-3af8-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,361 - distributed.scheduler - INFO - Remove client Client-4f2bd05f-3afb-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,361 - distributed.scheduler - INFO - Remove client Client-e8a4656a-3af9-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,360 - distributed.scheduler - INFO - Remove client Client-b5b00ece-3afc-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,360 - distributed.scheduler - INFO - Remove client Client-82b03197-3aff-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,360 - distributed.scheduler - INFO - Remove client Client-8375feb9-3afe-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,360 - distributed.scheduler - INFO - Remove client Client-e939bead-3b00-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,359 - distributed.scheduler - INFO - Remove client Client-4fc22650-3b02-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,359 - distributed.scheduler - INFO - Remove client Client-b63d117f-3b03-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,359 - distributed.scheduler - INFO - Remove client Client-1cc4f1f8-3b05-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,358 - distributed.scheduler - INFO - Remove client Client-e9b94919-3b07-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,345 - distributed.scheduler - INFO - Remove client Client-dac92f43-3b06-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,345 - distributed.scheduler - INFO - Remove client Client-2c608aed-3b09-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,344 - distributed.scheduler - INFO - Remove client Client-f962f2c6-3b0b-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,344 - distributed.scheduler - INFO - Remove client Client-92e3327d-3b0a-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,344 - distributed.scheduler - INFO - Remove client Client-5fe87119-3b0d-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,343 - distributed.scheduler - INFO - Remove client Client-2ce36041-3b10-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,343 - distributed.scheduler - INFO - Remove client Client-16d86351-3b0f-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,342 - distributed.scheduler - INFO - Remove client Client-93656697-3b11-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,342 - distributed.scheduler - INFO - Remove client Client-f9db8ebd-3b12-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,342 - distributed.scheduler - INFO - Remove client Client-605a27eb-3b14-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,341 - distributed.scheduler - INFO - Remove client Client-c6d92539-3b15-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,341 - distributed.scheduler - INFO - Remove client Client-a4626c79-3b17-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,341 - distributed.scheduler - INFO - Remove client Client-93dc2847-3b18-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,340 - distributed.scheduler - INFO - Remove client Client-fa6e78bb-3b19-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,340 - distributed.scheduler - INFO - Remove client Client-2debaa13-3b1e-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,339 - distributed.scheduler - INFO - Remove client Client-60e7a3e5-3b1b-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,339 - distributed.scheduler - INFO - Remove client Client-c769e53e-3b1c-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,338 - distributed.scheduler - INFO - Remove client Client-faf6f888-3b20-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,337 - distributed.scheduler - INFO - Remove client Client-f069285c-3b1f-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,337 - distributed.scheduler - INFO - Remove client Client-c8015b45-3b23-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,336 - distributed.scheduler - INFO - Remove client Client-617b9219-3b22-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,335 - distributed.scheduler - INFO - Remove client Client-2e80be02-3b25-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,335 - distributed.scheduler - INFO - Remove client Client-950bdef0-3b26-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,334 - distributed.scheduler - INFO - Remove client Client-4799b8b0-3b28-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,334 - distributed.scheduler - INFO - Remove client Client-6221a723-3b29-11f1-8001-daa6852bd5a7
2026-04-18 13:41:16,319 - distributed.scheduler - INFO - Remove client Client-c8a6cf0c-3b2a-11f1-8001-daa6852bd5a7
2026-04-18 13:41:15,423 - distributed.scheduler - INFO - Close client connection: Client-2f2a0acb-3b2c-11f1-8001-daa6852bd5a7
2026-04-18 13:41:15,414 - distributed.scheduler - INFO - Remove client Client-2f2a0acb-3b2c-11f1-8001-daa6852bd5a7
2026-04-18 13:41:15,412 - distributed.scheduler - INFO - Remove client Client-2f2a0acb-3b2c-11f1-8001-daa6852bd5a7
2026-04-18 13:40:38,136 - distributed.scheduler - INFO - Receive client connection: Client-2f2a0acb-3b2c-11f1-8001-daa6852bd5a7
2026-04-18 13:30:36,651 - distributed.scheduler - INFO - Receive client connection: Client-c8a6cf0c-3b2a-11f1-8001-daa6852bd5a7
2026-04-18 13:20:35,166 - distributed.scheduler - INFO - Receive client connection: Client-6221a723-3b29-11f1-8001-daa6852bd5a7
2026-04-18 13:12:41,136 - distributed.scheduler - INFO - Receive client connection: Client-4799b8b0-3b28-11f1-8001-daa6852bd5a7
2026-04-18 13:00:32,075 - distributed.scheduler - INFO - Receive client connection: Client-950bdef0-3b26-11f1-8001-daa6852bd5a7
2026-04-18 12:50:30,550 - distributed.scheduler - INFO - Receive client connection: Client-2e80be02-3b25-11f1-8001-daa6852bd5a7
2026-04-18 12:40:29,089 - distributed.scheduler - INFO - Receive client connection: Client-c8015b45-3b23-11f1-8001-daa6852bd5a7
2026-04-18 12:30:27,581 - distributed.scheduler - INFO - Receive client connection: Client-617b9219-3b22-11f1-8001-daa6852bd5a7
2026-04-18 12:20:26,084 - distributed.scheduler - INFO - Receive client connection: Client-faf6f888-3b20-11f1-8001-daa6852bd5a7
2026-04-18 12:12:58,887 - distributed.scheduler - INFO - Receive client connection: Client-f069285c-3b1f-11f1-8001-daa6852bd5a7
2026-04-18 12:00:23,098 - distributed.scheduler - INFO - Receive client connection: Client-2debaa13-3b1e-11f1-8001-daa6852bd5a7
2026-04-18 11:50:21,760 - distributed.scheduler - INFO - Receive client connection: Client-c769e53e-3b1c-11f1-8001-daa6852bd5a7
2026-04-18 11:40:20,140 - distributed.scheduler - INFO - Receive client connection: Client-60e7a3e5-3b1b-11f1-8001-daa6852bd5a7
2026-04-18 11:30:18,721 - distributed.scheduler - INFO - Receive client connection: Client-fa6e78bb-3b19-11f1-8001-daa6852bd5a7
2026-04-18 11:20:17,143 - distributed.scheduler - INFO - Receive client connection: Client-93dc2847-3b18-11f1-8001-daa6852bd5a7
2026-04-18 11:13:35,360 - distributed.scheduler - INFO - Receive client connection: Client-a4626c79-3b17-11f1-8001-daa6852bd5a7
2026-04-18 11:00:14,196 - distributed.scheduler - INFO - Receive client connection: Client-c6d92539-3b15-11f1-8001-daa6852bd5a7
2026-04-18 10:50:12,751 - distributed.scheduler - INFO - Receive client connection: Client-605a27eb-3b14-11f1-8001-daa6852bd5a7
2026-04-18 10:40:11,279 - distributed.scheduler - INFO - Receive client connection: Client-f9db8ebd-3b12-11f1-8001-daa6852bd5a7
2026-04-18 10:30:09,880 - distributed.scheduler - INFO - Receive client connection: Client-93656697-3b11-11f1-8001-daa6852bd5a7
2026-04-18 10:20:08,413 - distributed.scheduler - INFO - Receive client connection: Client-2ce36041-3b10-11f1-8001-daa6852bd5a7
2026-04-18 10:12:21,922 - distributed.scheduler - INFO - Receive client connection: Client-16d86351-3b0f-11f1-8001-daa6852bd5a7
2026-04-18 10:00:05,522 - distributed.scheduler - INFO - Receive client connection: Client-5fe87119-3b0d-11f1-8001-daa6852bd5a7
2026-04-18 09:50:04,013 - distributed.scheduler - INFO - Receive client connection: Client-f962f2c6-3b0b-11f1-8001-daa6852bd5a7
2026-04-18 09:40:02,552 - distributed.scheduler - INFO - Receive client connection: Client-92e3327d-3b0a-11f1-8001-daa6852bd5a7
2026-04-18 09:30:01,072 - distributed.scheduler - INFO - Receive client connection: Client-2c608aed-3b09-11f1-8001-daa6852bd5a7
2026-04-18 09:20:59,758 - distributed.scheduler - INFO - Receive client connection: Client-e9b94919-3b07-11f1-8001-daa6852bd5a7
2026-04-18 09:13:25,228 - distributed.scheduler - INFO - Receive client connection: Client-dac92f43-3b06-11f1-8001-daa6852bd5a7
2026-04-18 09:00:56,900 - distributed.scheduler - INFO - Receive client connection: Client-1cc4f1f8-3b05-11f1-8001-daa6852bd5a7
2026-04-18 08:50:55,385 - distributed.scheduler - INFO - Receive client connection: Client-b63d117f-3b03-11f1-8001-daa6852bd5a7
2026-04-18 08:40:53,955 - distributed.scheduler - INFO - Receive client connection: Client-4fc22650-3b02-11f1-8001-daa6852bd5a7
2026-04-18 08:30:52,443 - distributed.scheduler - INFO - Receive client connection: Client-e939bead-3b00-11f1-8001-daa6852bd5a7
2026-04-18 08:20:50,906 - distributed.scheduler - INFO - Receive client connection: Client-82b03197-3aff-11f1-8001-daa6852bd5a7
2026-04-18 08:13:42,704 - distributed.scheduler - INFO - Receive client connection: Client-8375feb9-3afe-11f1-8001-daa6852bd5a7
2026-04-18 08:00:47,980 - distributed.scheduler - INFO - Receive client connection: Client-b5b00ece-3afc-11f1-8001-daa6852bd5a7
2026-04-18 07:50:46,485 - distributed.scheduler - INFO - Receive client connection: Client-4f2bd05f-3afb-11f1-8001-daa6852bd5a7
2026-04-18 07:40:44,976 - distributed.scheduler - INFO - Receive client connection: Client-e8a4656a-3af9-11f1-8001-daa6852bd5a7
2026-04-18 07:30:43,482 - distributed.scheduler - INFO - Receive client connection: Client-821f0b03-3af8-11f1-8001-daa6852bd5a7
2026-04-18 07:20:41,981 - distributed.scheduler - INFO - Receive client connection: Client-1b99f054-3af7-11f1-8001-daa6852bd5a7
2026-04-18 07:12:58,589 - distributed.scheduler - INFO - Receive client connection: Client-07658d2c-3af6-11f1-8001-daa6852bd5a7
2026-04-18 07:00:39,165 - distributed.scheduler - INFO - Receive client connection: Client-4eaa0a5d-3af4-11f1-8001-daa6852bd5a7
2026-04-18 06:50:37,741 - distributed.scheduler - INFO - Receive client connection: Client-e830b976-3af2-11f1-8001-daa6852bd5a7
2026-04-18 06:40:36,282 - distributed.scheduler - INFO - Receive client connection: Client-81aea304-3af1-11f1-8001-daa6852bd5a7
2026-04-18 06:30:34,720 - distributed.scheduler - INFO - Receive client connection: Client-1b211ba1-3af0-11f1-8001-daa6852bd5a7
2026-04-18 06:20:33,227 - distributed.scheduler - INFO - Receive client connection: Client-b49d33c3-3aee-11f1-8001-daa6852bd5a7
2026-04-18 06:13:30,630 - distributed.scheduler - INFO - Receive client connection: Client-b8ba5739-3aed-11f1-8001-daa6852bd5a7
2026-04-18 06:00:30,207 - distributed.scheduler - INFO - Receive client connection: Client-e78f02e6-3aeb-11f1-8001-daa6852bd5a7
2026-04-18 05:50:28,765 - distributed.scheduler - INFO - Receive client connection: Client-8111cc18-3aea-11f1-8001-daa6852bd5a7
2026-04-18 05:40:27,269 - distributed.scheduler - INFO - Receive client connection: Client-1a8bde04-3ae9-11f1-8001-daa6852bd5a7
2026-04-18 05:30:25,758 - distributed.scheduler - INFO - Receive client connection: Client-b4068512-3ae7-11f1-8001-daa6852bd5a7
2026-04-18 05:20:24,221 - distributed.scheduler - INFO - Receive client connection: Client-4d7b4f15-3ae6-11f1-8001-daa6852bd5a7
2026-04-18 05:13:00,297 - distributed.scheduler - INFO - Receive client connection: Client-44e235c2-3ae5-11f1-8001-daa6852bd5a7
2026-04-18 05:00:21,174 - distributed.scheduler - INFO - Receive client connection: Client-8069cdcc-3ae3-11f1-8001-daa6852bd5a7
2026-04-18 04:50:19,737 - distributed.scheduler - INFO - Receive client connection: Client-19ed6625-3ae2-11f1-8001-daa6852bd5a7
2026-04-18 04:40:18,320 - distributed.scheduler - INFO - Receive client connection: Client-b374197d-3ae0-11f1-8001-daa6852bd5a7
2026-04-18 04:30:16,847 - distributed.scheduler - INFO - Receive client connection: Client-4cf2633c-3adf-11f1-8001-daa6852bd5a7
2026-04-18 04:20:15,333 - distributed.scheduler - INFO - Receive client connection: Client-e6695110-3add-11f1-8001-daa6852bd5a7
2026-04-18 04:12:26,518 - distributed.scheduler - INFO - Receive client connection: Client-cefa8770-3adc-11f1-8001-daa6852bd5a7
2026-04-18 04:00:12,404 - distributed.scheduler - INFO - Receive client connection: Client-1968d9cd-3adb-11f1-8001-daa6852bd5a7
2026-04-18 03:50:10,949 - distributed.scheduler - INFO - Receive client connection: Client-b2ea5266-3ad9-11f1-8001-daa6852bd5a7
2026-04-18 03:40:09,461 - distributed.scheduler - INFO - Receive client connection: Client-4c672e29-3ad8-11f1-8001-daa6852bd5a7
2026-04-18 03:30:08,007 - distributed.scheduler - INFO - Receive client connection: Client-e5e8b082-3ad6-11f1-8001-daa6852bd5a7
2026-04-18 03:20:06,434 - distributed.scheduler - INFO - Receive client connection: Client-7f581f95-3ad5-11f1-8001-daa6852bd5a7
2026-04-18 03:12:29,643 - distributed.scheduler - INFO - Receive client connection: Client-6f136504-3ad4-11f1-8001-daa6852bd5a7
2026-04-18 03:00:03,443 - distributed.scheduler - INFO - Receive client connection: Client-b24e8ebf-3ad2-11f1-8001-daa6852bd5a7
2026-04-18 02:50:01,927 - distributed.scheduler - INFO - Receive client connection: Client-4bc5edd0-3ad1-11f1-8001-daa6852bd5a7
2026-04-18 02:40:00,437 - distributed.scheduler - INFO - Receive client connection: Client-e541b089-3acf-11f1-8001-daa6852bd5a7
2026-04-18 02:30:59,133 - distributed.scheduler - INFO - Receive client connection: Client-a29db5c7-3ace-11f1-8001-daa6852bd5a7
2026-04-18 02:20:57,605 - distributed.scheduler - INFO - Receive client connection: Client-3c140d90-3acd-11f1-8001-daa6852bd5a7
2026-04-18 02:12:37,243 - distributed.scheduler - INFO - Receive client connection: Client-11d78e8c-3acc-11f1-8001-daa6852bd5a7
2026-04-18 02:00:54,560 - distributed.scheduler - INFO - Receive client connection: Client-6f020e4c-3aca-11f1-8001-daa6852bd5a7
2026-04-18 01:50:53,017 - distributed.scheduler - INFO - Receive client connection: Client-08753b21-3ac9-11f1-8001-daa6852bd5a7
2026-04-18 01:40:51,483 - distributed.scheduler - INFO - Receive client connection: Client-a1eaf98c-3ac7-11f1-8001-daa6852bd5a7
2026-04-18 01:30:49,892 - distributed.scheduler - INFO - Receive client connection: Client-3b56f9ca-3ac6-11f1-8001-daa6852bd5a7
2026-04-18 01:20:48,477 - distributed.scheduler - INFO - Receive client connection: Client-d4de57d3-3ac4-11f1-8001-daa6852bd5a7
2026-04-18 01:12:58,101 - distributed.scheduler - INFO - Receive client connection: Client-bc81a986-3ac3-11f1-8001-daa6852bd5a7
2026-04-18 01:00:45,410 - distributed.scheduler - INFO - Receive client connection: Client-07c889ba-3ac2-11f1-8001-daa6852bd5a7
2026-04-18 00:50:43,898 - distributed.scheduler - INFO - Receive client connection: Client-a1416717-3ac0-11f1-8001-daa6852bd5a7
2026-04-18 00:40:42,469 - distributed.scheduler - INFO - Receive client connection: Client-3ac71195-3abf-11f1-8001-daa6852bd5a7
2026-04-18 00:30:41,063 - distributed.scheduler - INFO - Receive client connection: Client-d44fa103-3abd-11f1-8001-daa6852bd5a7
2026-04-18 00:20:39,657 - distributed.scheduler - INFO - Receive client connection: Client-6dd7d8d2-3abc-11f1-8001-daa6852bd5a7
2026-04-18 00:12:07,199 - distributed.scheduler - INFO - Receive client connection: Client-3c6618e9-3abb-11f1-8001-daa6852bd5a7
2026-04-18 00:00:36,720 - distributed.scheduler - INFO - Receive client connection: Client-a0d6da7a-3ab9-11f1-8001-daa6852bd5a7
2026-04-17 23:50:35,195 - distributed.scheduler - INFO - Receive client connection: Client-3a4d759b-3ab8-11f1-8001-daa6852bd5a7
2026-04-17 23:40:33,652 - distributed.scheduler - INFO - Receive client connection: Client-d3bffab8-3ab6-11f1-8001-daa6852bd5a7
2026-04-17 23:30:32,135 - distributed.scheduler - INFO - Receive client connection: Client-6d38da8c-3ab5-11f1-8001-daa6852bd5a7
2026-04-17 23:20:30,653 - distributed.scheduler - INFO - Receive client connection: Client-06b56932-3ab4-11f1-8001-daa6852bd5a7
2026-04-17 23:13:09,130 - distributed.scheduler - INFO - Receive client connection: Client-ff8c30a3-3ab2-11f1-8001-daa6852bd5a7
2026-04-17 23:00:27,791 - distributed.scheduler - INFO - Receive client connection: Client-39bf7c33-3ab1-11f1-8001-daa6852bd5a7
2026-04-17 22:50:26,327 - distributed.scheduler - INFO - Receive client connection: Client-d33f92c6-3aaf-11f1-8001-daa6852bd5a7
2026-04-17 22:40:24,839 - distributed.scheduler - INFO - Receive client connection: Client-6cbb4aa3-3aae-11f1-8001-daa6852bd5a7
2026-04-17 22:30:23,344 - distributed.scheduler - INFO - Receive client connection: Client-063721b9-3aad-11f1-8001-daa6852bd5a7
2026-04-17 22:20:21,798 - distributed.scheduler - INFO - Receive client connection: Client-9fab26c2-3aab-11f1-8001-daa6852bd5a7
2026-04-17 22:13:04,747 - distributed.scheduler - INFO - Receive client connection: Client-9b295e42-3aaa-11f1-8001-daa6852bd5a7
2026-04-17 22:00:18,864 - distributed.scheduler - INFO - Receive client connection: Client-d2a3043e-3aa8-11f1-8001-daa6852bd5a7
2026-04-17 21:50:17,300 - distributed.scheduler - INFO - Receive client connection: Client-6c196c70-3aa7-11f1-8001-daa6852bd5a7
2026-04-17 21:40:15,790 - distributed.scheduler - INFO - Receive client connection: Client-0592bb93-3aa6-11f1-8001-daa6852bd5a7
2026-04-17 21:30:14,233 - distributed.scheduler - INFO - Receive client connection: Client-9f04014f-3aa4-11f1-8001-daa6852bd5a7
2026-04-17 21:20:12,687 - distributed.scheduler - INFO - Receive client connection: Client-3878384a-3aa3-11f1-8001-daa6852bd5a7
2026-04-17 21:12:00,571 - distributed.scheduler - INFO - Receive client connection: Client-132577aa-3aa2-11f1-8001-daa6852bd5a7
2026-04-17 21:00:09,661 - distributed.scheduler - INFO - Receive client connection: Client-6b69852a-3aa0-11f1-8001-daa6852bd5a7
2026-04-17 20:50:08,179 - distributed.scheduler - INFO - Receive client connection: Client-04e73516-3a9f-11f1-8001-daa6852bd5a7
2026-04-17 20:40:06,609 - distributed.scheduler - INFO - Receive client connection: Client-9e55533a-3a9d-11f1-8001-daa6852bd5a7
2026-04-17 20:30:05,058 - distributed.scheduler - INFO - Receive client connection: Client-37c83313-3a9c-11f1-8001-daa6852bd5a7
2026-04-17 20:20:03,535 - distributed.scheduler - INFO - Receive client connection: Client-d13e8591-3a9a-11f1-8001-daa6852bd5a7
2026-04-17 20:11:54,040 - distributed.scheduler - INFO - Receive client connection: Client-ad7c479c-3a99-11f1-8001-daa6852bd5a7
2026-04-17 20:00:00,587 - distributed.scheduler - INFO - Receive client connection: Client-043b89f9-3a98-11f1-8001-daa6852bd5a7
2026-04-17 19:50:59,273 - distributed.scheduler - INFO - Receive client connection: Client-c195acfe-3a96-11f1-8001-daa6852bd5a7
2026-04-17 19:40:57,915 - distributed.scheduler - INFO - Receive client connection: Client-5b257c62-3a95-11f1-8001-daa6852bd5a7
2026-04-17 19:30:56,424 - distributed.scheduler - INFO - Receive client connection: Client-f4a10425-3a93-11f1-8001-daa6852bd5a7
2026-04-17 19:20:54,895 - distributed.scheduler - INFO - Receive client connection: Client-8e177a16-3a92-11f1-8001-daa6852bd5a7
2026-04-17 19:12:57,003 - distributed.scheduler - INFO - Receive client connection: Client-713d7e39-3a91-11f1-8001-daa6852bd5a7
2026-04-17 19:01:56,806 - distributed.scheduler - INFO - Close client connection: Client-c0f9723a-3a8f-11f1-8001-daa6852bd5a7
2026-04-17 19:01:56,805 - distributed.scheduler - INFO - Remove client Client-c0f9723a-3a8f-11f1-8001-daa6852bd5a7
2026-04-17 19:01:56,804 - distributed.scheduler - INFO - Remove client Client-c0f9723a-3a8f-11f1-8001-daa6852bd5a7
2026-04-17 19:00:51,783 - distributed.scheduler - INFO - Receive client connection: Client-c0f9723a-3a8f-11f1-8001-daa6852bd5a7
2026-04-17 18:51:24,700 - distributed.scheduler - INFO - Close client connection: Client-5a6f8e5a-3a8e-11f1-8001-daa6852bd5a7
2026-04-17 18:51:24,698 - distributed.scheduler - INFO - Remove client Client-5a6f8e5a-3a8e-11f1-8001-daa6852bd5a7
2026-04-17 18:51:24,696 - distributed.scheduler - INFO - Remove client Client-5a6f8e5a-3a8e-11f1-8001-daa6852bd5a7
2026-04-17 18:50:50,289 - distributed.scheduler - INFO - Receive client connection: Client-5a6f8e5a-3a8e-11f1-8001-daa6852bd5a7
2026-04-17 18:41:20,188 - distributed.scheduler - INFO - Close client connection: Client-f3f7509b-3a8c-11f1-8001-daa6852bd5a7
2026-04-17 18:41:20,186 - distributed.scheduler - INFO - Remove client Client-f3f7509b-3a8c-11f1-8001-daa6852bd5a7
2026-04-17 18:41:20,185 - distributed.scheduler - INFO - Remove client Client-f3f7509b-3a8c-11f1-8001-daa6852bd5a7
2026-04-17 18:40:48,876 - distributed.scheduler - INFO - Receive client connection: Client-f3f7509b-3a8c-11f1-8001-daa6852bd5a7
2026-04-17 18:31:51,222 - distributed.scheduler - INFO - Close client connection: Client-8d739882-3a8b-11f1-8001-daa6852bd5a7
2026-04-17 18:31:51,221 - distributed.scheduler - INFO - Remove client Client-8d739882-3a8b-11f1-8001-daa6852bd5a7
2026-04-17 18:31:51,219 - distributed.scheduler - INFO - Remove client Client-8d739882-3a8b-11f1-8001-daa6852bd5a7
2026-04-17 18:30:50,880 - distributed.scheduler - INFO - Receive client connection: Client-worker-8f911b6b-3a8b-11f1-8289-ee7435aaba1f
2026-04-17 18:30:47,354 - distributed.scheduler - INFO - Receive client connection: Client-8d739882-3a8b-11f1-8001-daa6852bd5a7
2026-04-17 18:21:21,008 - distributed.scheduler - INFO - Close client connection: Client-26efbac6-3a8a-11f1-8001-daa6852bd5a7
2026-04-17 18:21:21,007 - distributed.scheduler - INFO - Remove client Client-26efbac6-3a8a-11f1-8001-daa6852bd5a7
2026-04-17 18:21:21,005 - distributed.scheduler - INFO - Remove client Client-26efbac6-3a8a-11f1-8001-daa6852bd5a7
2026-04-17 18:20:45,862 - distributed.scheduler - INFO - Receive client connection: Client-26efbac6-3a8a-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,748 - distributed.scheduler - INFO - Close client connection: Client-9833e079-3a17-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,748 - distributed.scheduler - INFO - Close client connection: Client-febd0f1b-3a18-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,748 - distributed.scheduler - INFO - Close client connection: Client-654b01a3-3a1a-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,748 - distributed.scheduler - INFO - Close client connection: Client-326b089b-3a1d-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,748 - distributed.scheduler - INFO - Close client connection: Client-14096434-3a1c-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,748 - distributed.scheduler - INFO - Close client connection: Client-98f9acbd-3a1e-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,748 - distributed.scheduler - INFO - Close client connection: Client-ff8540af-3a1f-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,748 - distributed.scheduler - INFO - Close client connection: Client-66150d5b-3a21-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,747 - distributed.scheduler - INFO - Close client connection: Client-cca5d602-3a22-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,747 - distributed.scheduler - INFO - Close client connection: Client-99b35b40-3a25-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,747 - distributed.scheduler - INFO - Close client connection: Client-97f83615-3a24-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,747 - distributed.scheduler - INFO - Close client connection: Client-dc5c2ddd-3a26-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,747 - distributed.scheduler - INFO - Close client connection: Client-42dc5a38-3a28-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,747 - distributed.scheduler - INFO - Close client connection: Client-a95ff7cd-3a29-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,747 - distributed.scheduler - INFO - Close client connection: Client-0fe1f6ef-3a2b-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,747 - distributed.scheduler - INFO - Close client connection: Client-b77c38c1-3a2c-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,747 - distributed.scheduler - INFO - Close client connection: Client-dcf18510-3a2d-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,747 - distributed.scheduler - INFO - Close client connection: Client-4374bd0a-3a2f-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-a9f48a9a-3a30-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-106cd986-3a32-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-76edc57e-3a33-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-43f8a2d7-3a36-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-14ea2494-3a35-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-aa7bb3da-3a37-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-10f65f4b-3a39-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-7780890c-3a3a-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-de06c0c9-3a3b-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-ab16f231-3a3e-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-8c9dcd60-3a3d-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-119bfcac-3a40-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,746 - distributed.scheduler - INFO - Close client connection: Client-7818e1bc-3a41-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-de9333d8-3a42-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-451c1c9f-3a44-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-122811bb-3a47-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-0111194a-3a46-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-78b25567-3a48-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-df36be52-3a49-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-45b18e6e-3a4b-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-ac208288-3a4c-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-655535a5-3a4e-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-7915770a-3a4f-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-df9dae66-3a50-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-4626e1dc-3a52-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,745 - distributed.scheduler - INFO - Close client connection: Client-acb139cb-3a53-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,744 - distributed.scheduler - INFO - Close client connection: Client-132f58a9-3a55-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,744 - distributed.scheduler - INFO - Close client connection: Client-e02c9e1c-3a57-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,744 - distributed.scheduler - INFO - Close client connection: Client-d41ee198-3a56-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,744 - distributed.scheduler - INFO - Close client connection: Client-46b5145b-3a59-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,744 - distributed.scheduler - INFO - Close client connection: Client-ad3f8b7e-3a5a-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,744 - distributed.scheduler - INFO - Close client connection: Client-13c84de2-3a5c-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,744 - distributed.scheduler - INFO - Close client connection: Client-7a47efc6-3a5d-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,744 - distributed.scheduler - INFO - Close client connection: Client-23745f8f-3a60-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,744 - distributed.scheduler - INFO - Close client connection: Client-3ed13202-3a5f-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,744 - distributed.scheduler - INFO - Close client connection: Client-8a00b3b1-3a61-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,744 - distributed.scheduler - INFO - Close client connection: Client-f084a689-3a62-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,743 - distributed.scheduler - INFO - Close client connection: Client-57121fd6-3a64-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,743 - distributed.scheduler - INFO - Close client connection: Client-bd9b99ec-3a65-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,743 - distributed.scheduler - INFO - Close client connection: Client-8a9ebfc1-3a68-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,743 - distributed.scheduler - INFO - Close client connection: Client-f125d34e-3a69-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,743 - distributed.scheduler - INFO - Close client connection: Client-862ef942-3a67-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,743 - distributed.scheduler - INFO - Close client connection: Client-57a5f31b-3a6b-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,743 - distributed.scheduler - INFO - Close client connection: Client-24a119ff-3a6e-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,743 - distributed.scheduler - INFO - Close client connection: Client-be275524-3a6c-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,743 - distributed.scheduler - INFO - Close client connection: Client-f4d9b9bb-3a6f-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,743 - distributed.scheduler - INFO - Close client connection: Client-f1ae897c-3a70-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,742 - distributed.scheduler - INFO - Close client connection: Client-58279aa4-3a72-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,741 - distributed.scheduler - INFO - Close client connection: Client-bead4dd9-3a73-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,741 - distributed.scheduler - INFO - Close client connection: Client-25369329-3a75-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,740 - distributed.scheduler - INFO - Close client connection: Client-8bbc1041-3a76-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,740 - distributed.scheduler - INFO - Close client connection: Client-58bf9718-3a79-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,740 - distributed.scheduler - INFO - Close client connection: Client-4c88d375-3a78-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,740 - distributed.scheduler - INFO - Close client connection: Client-bf3adf4c-3a7a-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,740 - distributed.scheduler - INFO - Close client connection: Client-25bfda7d-3a7c-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,740 - distributed.scheduler - INFO - Close client connection: Client-8c5264a4-3a7d-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,740 - distributed.scheduler - INFO - Close client connection: Client-f2db26e9-3a7e-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,740 - distributed.scheduler - INFO - Close client connection: Client-b434b51a-3a80-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,740 - distributed.scheduler - INFO - Close client connection: Client-bfd9859f-3a81-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,740 - distributed.scheduler - INFO - Close client connection: Client-f364311a-3a85-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,739 - distributed.scheduler - INFO - Close client connection: Client-2656aa46-3a83-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,739 - distributed.scheduler - INFO - Close client connection: Client-8cde2698-3a84-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,734 - distributed.scheduler - INFO - Close client connection: Client-59ddb68b-3a87-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,733 - distributed.scheduler - INFO - Remove client Client-9833e079-3a17-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,733 - distributed.scheduler - INFO - Remove client Client-febd0f1b-3a18-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,733 - distributed.scheduler - INFO - Remove client Client-654b01a3-3a1a-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,733 - distributed.scheduler - INFO - Remove client Client-326b089b-3a1d-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,732 - distributed.scheduler - INFO - Remove client Client-14096434-3a1c-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,732 - distributed.scheduler - INFO - Remove client Client-98f9acbd-3a1e-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,732 - distributed.scheduler - INFO - Remove client Client-ff8540af-3a1f-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,732 - distributed.scheduler - INFO - Remove client Client-66150d5b-3a21-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,731 - distributed.scheduler - INFO - Remove client Client-cca5d602-3a22-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,731 - distributed.scheduler - INFO - Remove client Client-99b35b40-3a25-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,731 - distributed.scheduler - INFO - Remove client Client-97f83615-3a24-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,731 - distributed.scheduler - INFO - Remove client Client-dc5c2ddd-3a26-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,730 - distributed.scheduler - INFO - Remove client Client-42dc5a38-3a28-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,730 - distributed.scheduler - INFO - Remove client Client-a95ff7cd-3a29-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,730 - distributed.scheduler - INFO - Remove client Client-0fe1f6ef-3a2b-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,730 - distributed.scheduler - INFO - Remove client Client-b77c38c1-3a2c-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,729 - distributed.scheduler - INFO - Remove client Client-dcf18510-3a2d-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,729 - distributed.scheduler - INFO - Remove client Client-4374bd0a-3a2f-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,729 - distributed.scheduler - INFO - Remove client Client-a9f48a9a-3a30-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,728 - distributed.scheduler - INFO - Remove client Client-106cd986-3a32-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,728 - distributed.scheduler - INFO - Remove client Client-76edc57e-3a33-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,728 - distributed.scheduler - INFO - Remove client Client-43f8a2d7-3a36-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,728 - distributed.scheduler - INFO - Remove client Client-14ea2494-3a35-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,727 - distributed.scheduler - INFO - Remove client Client-aa7bb3da-3a37-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,727 - distributed.scheduler - INFO - Remove client Client-10f65f4b-3a39-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,727 - distributed.scheduler - INFO - Remove client Client-7780890c-3a3a-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,727 - distributed.scheduler - INFO - Remove client Client-de06c0c9-3a3b-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,726 - distributed.scheduler - INFO - Remove client Client-ab16f231-3a3e-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,726 - distributed.scheduler - INFO - Remove client Client-8c9dcd60-3a3d-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,726 - distributed.scheduler - INFO - Remove client Client-119bfcac-3a40-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,726 - distributed.scheduler - INFO - Remove client Client-7818e1bc-3a41-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,725 - distributed.scheduler - INFO - Remove client Client-de9333d8-3a42-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,725 - distributed.scheduler - INFO - Remove client Client-451c1c9f-3a44-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,725 - distributed.scheduler - INFO - Remove client Client-122811bb-3a47-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,724 - distributed.scheduler - INFO - Remove client Client-0111194a-3a46-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,724 - distributed.scheduler - INFO - Remove client Client-78b25567-3a48-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,724 - distributed.scheduler - INFO - Remove client Client-df36be52-3a49-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,723 - distributed.scheduler - INFO - Remove client Client-45b18e6e-3a4b-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,723 - distributed.scheduler - INFO - Remove client Client-ac208288-3a4c-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,723 - distributed.scheduler - INFO - Remove client Client-655535a5-3a4e-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,723 - distributed.scheduler - INFO - Remove client Client-7915770a-3a4f-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,722 - distributed.scheduler - INFO - Remove client Client-df9dae66-3a50-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,722 - distributed.scheduler - INFO - Remove client Client-4626e1dc-3a52-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,722 - distributed.scheduler - INFO - Remove client Client-acb139cb-3a53-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,721 - distributed.scheduler - INFO - Remove client Client-132f58a9-3a55-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,721 - distributed.scheduler - INFO - Remove client Client-e02c9e1c-3a57-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,721 - distributed.scheduler - INFO - Remove client Client-d41ee198-3a56-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,721 - distributed.scheduler - INFO - Remove client Client-46b5145b-3a59-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,720 - distributed.scheduler - INFO - Remove client Client-ad3f8b7e-3a5a-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,720 - distributed.scheduler - INFO - Remove client Client-13c84de2-3a5c-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,720 - distributed.scheduler - INFO - Remove client Client-7a47efc6-3a5d-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,720 - distributed.scheduler - INFO - Remove client Client-23745f8f-3a60-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,719 - distributed.scheduler - INFO - Remove client Client-3ed13202-3a5f-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,719 - distributed.scheduler - INFO - Remove client Client-8a00b3b1-3a61-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,715 - distributed.scheduler - INFO - Remove client Client-f084a689-3a62-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,714 - distributed.scheduler - INFO - Remove client Client-57121fd6-3a64-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,714 - distributed.scheduler - INFO - Remove client Client-bd9b99ec-3a65-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,714 - distributed.scheduler - INFO - Remove client Client-8a9ebfc1-3a68-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,713 - distributed.scheduler - INFO - Remove client Client-f125d34e-3a69-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,713 - distributed.scheduler - INFO - Remove client Client-862ef942-3a67-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,713 - distributed.scheduler - INFO - Remove client Client-57a5f31b-3a6b-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,713 - distributed.scheduler - INFO - Remove client Client-24a119ff-3a6e-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,712 - distributed.scheduler - INFO - Remove client Client-be275524-3a6c-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,712 - distributed.scheduler - INFO - Remove client Client-f4d9b9bb-3a6f-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,712 - distributed.scheduler - INFO - Remove client Client-f1ae897c-3a70-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,711 - distributed.scheduler - INFO - Remove client Client-58279aa4-3a72-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,698 - distributed.scheduler - INFO - Remove client Client-bead4dd9-3a73-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,697 - distributed.scheduler - INFO - Remove client Client-25369329-3a75-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,697 - distributed.scheduler - INFO - Remove client Client-8bbc1041-3a76-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,696 - distributed.scheduler - INFO - Remove client Client-58bf9718-3a79-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,696 - distributed.scheduler - INFO - Remove client Client-4c88d375-3a78-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,695 - distributed.scheduler - INFO - Remove client Client-bf3adf4c-3a7a-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,694 - distributed.scheduler - INFO - Remove client Client-25bfda7d-3a7c-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,694 - distributed.scheduler - INFO - Remove client Client-8c5264a4-3a7d-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,693 - distributed.scheduler - INFO - Remove client Client-f2db26e9-3a7e-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,692 - distributed.scheduler - INFO - Remove client Client-b434b51a-3a80-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,692 - distributed.scheduler - INFO - Remove client Client-bfd9859f-3a81-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,691 - distributed.scheduler - INFO - Remove client Client-f364311a-3a85-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,690 - distributed.scheduler - INFO - Remove client Client-2656aa46-3a83-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,690 - distributed.scheduler - INFO - Remove client Client-8cde2698-3a84-11f1-8001-daa6852bd5a7
2026-04-17 18:13:40,683 - distributed.scheduler - INFO - Remove client Client-59ddb68b-3a87-11f1-8001-daa6852bd5a7
2026-04-17 18:13:39,795 - distributed.scheduler - INFO - Close client connection: Client-14368f0e-3a89-11f1-8001-daa6852bd5a7
2026-04-17 18:13:39,794 - distributed.scheduler - INFO - Remove client Client-14368f0e-3a89-11f1-8001-daa6852bd5a7
2026-04-17 18:13:39,792 - distributed.scheduler - INFO - Remove client Client-14368f0e-3a89-11f1-8001-daa6852bd5a7
2026-04-17 18:13:04,948 - distributed.scheduler - INFO - Receive client connection: Client-14368f0e-3a89-11f1-8001-daa6852bd5a7
2026-04-17 18:00:42,806 - distributed.scheduler - INFO - Receive client connection: Client-59ddb68b-3a87-11f1-8001-daa6852bd5a7
2026-04-17 17:50:41,387 - distributed.scheduler - INFO - Receive client connection: Client-f364311a-3a85-11f1-8001-daa6852bd5a7
2026-04-17 17:40:39,889 - distributed.scheduler - INFO - Receive client connection: Client-8cde2698-3a84-11f1-8001-daa6852bd5a7
2026-04-17 17:30:38,379 - distributed.scheduler - INFO - Receive client connection: Client-2656aa46-3a83-11f1-8001-daa6852bd5a7
2026-04-17 17:20:36,944 - distributed.scheduler - INFO - Receive client connection: Client-bfd9859f-3a81-11f1-8001-daa6852bd5a7
2026-04-17 17:13:07,893 - distributed.scheduler - INFO - Receive client connection: Client-b434b51a-3a80-11f1-8001-daa6852bd5a7
2026-04-17 17:05:11,510 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:40835 after 30 s')
2026-04-17 17:05:11,508 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:40835 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:40835 after 30 s
2026-04-17 17:05:11,417 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:40855 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:40835 after 30 s')
2026-04-17 17:05:11,413 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:40835 after 30 s')
2026-04-17 17:05:11,379 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:40835 after 30 s')
2026-04-17 17:05:11,355 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:40835 after 30 s')
2026-04-17 17:05:11,354 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:40835 after 30 s')
2026-04-17 17:05:11,353 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:40835 after 30 s')
2026-04-17 17:05:11,350 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:40835 after 30 s')
2026-04-17 17:05:11,348 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:40835 after 30 s')
2026-04-17 17:05:11,347 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:40835 after 30 s')
2026-04-17 17:05:11,341 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:40835 after 30 s')
2026-04-17 17:04:42,386 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:42859
2026-04-17 17:04:42,384 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:42859', status: closed, memory: 0, processing: 0>
2026-04-17 17:04:41,214 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:40835', status: closed, memory: 0, processing: 0>
2026-04-17 17:04:30,778 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:38849 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:38849 after 30 s
2026-04-17 17:04:29,955 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:38849 after 30 s')
2026-04-17 17:04:29,896 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:40855 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:38849 after 30 s')
2026-04-17 17:04:29,893 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:38849 after 30 s')
2026-04-17 17:04:29,891 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:38849 after 30 s')
2026-04-17 17:04:29,881 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:41873 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:38849 after 30 s')
2026-04-17 17:04:29,877 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:38849 after 30 s')
2026-04-17 17:04:29,871 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:38849 after 30 s')
2026-04-17 17:04:29,868 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:38849 after 30 s')
2026-04-17 17:04:29,830 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:38849 after 30 s')
2026-04-17 17:04:29,826 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:38849 after 30 s')
2026-04-17 17:04:29,812 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:38849 after 30 s')
2026-04-17 17:04:00,865 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:40835
2026-04-17 17:04:00,862 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:40835', status: closed, memory: 0, processing: 0>
2026-04-17 17:03:59,719 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:38849', status: closed, memory: 0, processing: 0>
2026-04-17 17:03:42,350 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:44837 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:44837 after 30 s
2026-04-17 17:03:42,308 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:44837 after 30 s')
2026-04-17 17:03:42,287 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:44837 after 30 s')
2026-04-17 17:03:42,277 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:44837 after 30 s')
2026-04-17 17:03:42,276 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:44837 after 30 s')
2026-04-17 17:03:42,275 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:44837 after 30 s')
2026-04-17 17:03:42,275 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:44837 after 30 s')
2026-04-17 17:03:42,274 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:44837 after 30 s')
2026-04-17 17:03:42,274 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:44837 after 30 s')
2026-04-17 17:03:42,245 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:44837 after 30 s')
2026-04-17 17:03:42,233 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:44837 after 30 s')
2026-04-17 17:03:13,176 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:38849
2026-04-17 17:03:13,173 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:38849', status: closed, memory: 0, processing: 0>
2026-04-17 17:03:12,129 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:44837', status: closed, memory: 0, processing: 0>
2026-04-17 17:02:47,516 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:35813 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:35813 after 30 s
2026-04-17 17:02:47,482 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:35813 after 30 s')
2026-04-17 17:02:47,407 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:35813 after 30 s')
2026-04-17 17:02:47,370 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:35813 after 30 s')
2026-04-17 17:02:47,357 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:35813 after 30 s')
2026-04-17 17:02:18,299 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:44837
2026-04-17 17:02:18,297 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:44837', status: closed, memory: 0, processing: 0>
2026-04-17 17:02:17,222 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:35813', status: closed, memory: 0, processing: 0>
2026-04-17 17:01:54,482 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s
2026-04-17 17:01:54,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s
2026-04-17 17:01:54,356 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:54,355 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:54,335 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:54,276 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:54,257 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:54,219 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:40855 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:54,208 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:40855 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:54,097 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:54,080 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: OSError: Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s
2026-04-17 17:01:54,053 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:54,032 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:54,024 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:54,021 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:41873 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:53,996 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:53,976 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:53,965 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:53,946 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:53,908 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:53,892 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:53,863 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:53,845 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.54:37809 after 30 s')
2026-04-17 17:01:24,849 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.54:35813
2026-04-17 17:01:24,848 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.54:35813', status: closed, memory: 0, processing: 0>
2026-04-17 17:01:23,658 - distributed.scheduler - INFO - Close client connection: Client-worker-5e940006-37bf-11f1-8074-ee7435aaba1f
2026-04-17 17:01:23,641 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.54:37809', status: closed, memory: 0, processing: 0>
2026-04-17 17:01:23,639 - distributed.scheduler - INFO - Remove client Client-worker-5e940006-37bf-11f1-8074-ee7435aaba1f
2026-04-17 17:00:40,708 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('len-agg-21ea8031340281d8b83f945b72278c8c', 0) NoneType: None
2026-04-17 17:00:40,707 - distributed.scheduler - ERROR - Couldn't gather keys {"('len-agg-21ea8031340281d8b83f945b72278c8c', 0)": []} state: ['waiting'] workers: [] NoneType: None
2026-04-17 17:00:34,016 - distributed.scheduler - INFO - Receive client connection: Client-f2db26e9-3a7e-11f1-8001-daa6852bd5a7
2026-04-17 16:50:32,498 - distributed.scheduler - INFO - Receive client connection: Client-8c5264a4-3a7d-11f1-8001-daa6852bd5a7
2026-04-17 16:40:30,912 - distributed.scheduler - INFO - Receive client connection: Client-25bfda7d-3a7c-11f1-8001-daa6852bd5a7
2026-04-17 16:30:29,421 - distributed.scheduler - INFO - Receive client connection: Client-bf3adf4c-3a7a-11f1-8001-daa6852bd5a7
2026-04-17 16:20:27,981 - distributed.scheduler - INFO - Receive client connection: Client-58bf9718-3a79-11f1-8001-daa6852bd5a7
2026-04-17 16:12:57,992 - distributed.scheduler - INFO - Receive client connection: Client-4c88d375-3a78-11f1-8001-daa6852bd5a7
2026-04-17 16:00:25,031 - distributed.scheduler - INFO - Receive client connection: Client-8bbc1041-3a76-11f1-8001-daa6852bd5a7
2026-04-17 15:50:23,544 - distributed.scheduler - INFO - Receive client connection: Client-25369329-3a75-11f1-8001-daa6852bd5a7
2026-04-17 15:40:22,014 - distributed.scheduler - INFO - Receive client connection: Client-bead4dd9-3a73-11f1-8001-daa6852bd5a7
2026-04-17 15:30:20,510 - distributed.scheduler - INFO - Receive client connection: Client-58279aa4-3a72-11f1-8001-daa6852bd5a7
2026-04-17 15:20:19,094 - distributed.scheduler - INFO - Receive client connection: Client-f1ae897c-3a70-11f1-8001-daa6852bd5a7
2026-04-17 15:13:14,908 - distributed.scheduler - INFO - Receive client connection: Client-f4d9b9bb-3a6f-11f1-8001-daa6852bd5a7
2026-04-17 15:00:16,076 - distributed.scheduler - INFO - Receive client connection: Client-24a119ff-3a6e-11f1-8001-daa6852bd5a7
2026-04-17 14:50:14,660 - distributed.scheduler - INFO - Receive client connection: Client-be275524-3a6c-11f1-8001-daa6852bd5a7
2026-04-17 14:40:13,189 - distributed.scheduler - INFO - Receive client connection: Client-57a5f31b-3a6b-11f1-8001-daa6852bd5a7
2026-04-17 14:30:11,715 - distributed.scheduler - INFO - Receive client connection: Client-f125d34e-3a69-11f1-8001-daa6852bd5a7
2026-04-17 14:20:10,213 - distributed.scheduler - INFO - Receive client connection: Client-8a9ebfc1-3a68-11f1-8001-daa6852bd5a7
2026-04-17 14:12:53,266 - distributed.scheduler - INFO - Receive client connection: Client-862ef942-3a67-11f1-8001-daa6852bd5a7
2026-04-17 14:00:07,262 - distributed.scheduler - INFO - Receive client connection: Client-bd9b99ec-3a65-11f1-8001-daa6852bd5a7
2026-04-17 13:50:05,744 - distributed.scheduler - INFO - Receive client connection: Client-57121fd6-3a64-11f1-8001-daa6852bd5a7
2026-04-17 13:40:04,184 - distributed.scheduler - INFO - Receive client connection: Client-f084a689-3a62-11f1-8001-daa6852bd5a7
2026-04-17 13:30:02,695 - distributed.scheduler - INFO - Receive client connection: Client-8a00b3b1-3a61-11f1-8001-daa6852bd5a7
2026-04-17 13:20:01,153 - distributed.scheduler - INFO - Receive client connection: Client-23745f8f-3a60-11f1-8001-daa6852bd5a7
2026-04-17 13:13:37,556 - distributed.scheduler - INFO - Receive client connection: Client-3ed13202-3a5f-11f1-8001-daa6852bd5a7
2026-04-17 13:00:58,338 - distributed.scheduler - INFO - Receive client connection: Client-7a47efc6-3a5d-11f1-8001-daa6852bd5a7
2026-04-17 12:50:56,878 - distributed.scheduler - INFO - Receive client connection: Client-13c84de2-3a5c-11f1-8001-daa6852bd5a7
2026-04-17 12:40:55,352 - distributed.scheduler - INFO - Receive client connection: Client-ad3f8b7e-3a5a-11f1-8001-daa6852bd5a7
2026-04-17 12:30:53,812 - distributed.scheduler - INFO - Receive client connection: Client-46b5145b-3a59-11f1-8001-daa6852bd5a7
2026-04-17 12:20:52,302 - distributed.scheduler - INFO - Receive client connection: Client-e02c9e1c-3a57-11f1-8001-daa6852bd5a7
2026-04-17 12:13:22,578 - distributed.scheduler - INFO - Receive client connection: Client-d41ee198-3a56-11f1-8001-daa6852bd5a7
2026-04-17 12:00:49,390 - distributed.scheduler - INFO - Receive client connection: Client-132f58a9-3a55-11f1-8001-daa6852bd5a7
2026-04-17 11:50:47,939 - distributed.scheduler - INFO - Receive client connection: Client-acb139cb-3a53-11f1-8001-daa6852bd5a7
2026-04-17 11:40:46,413 - distributed.scheduler - INFO - Receive client connection: Client-4626e1dc-3a52-11f1-8001-daa6852bd5a7
2026-04-17 11:30:44,885 - distributed.scheduler - INFO - Receive client connection: Client-df9dae66-3a50-11f1-8001-daa6852bd5a7
2026-04-17 11:20:43,366 - distributed.scheduler - INFO - Receive client connection: Client-7915770a-3a4f-11f1-8001-daa6852bd5a7
2026-04-17 11:13:00,735 - distributed.scheduler - INFO - Receive client connection: Client-655535a5-3a4e-11f1-8001-daa6852bd5a7
2026-04-17 11:00:40,513 - distributed.scheduler - INFO - Receive client connection: Client-ac208288-3a4c-11f1-8001-daa6852bd5a7
2026-04-17 10:50:39,163 - distributed.scheduler - INFO - Receive client connection: Client-45b18e6e-3a4b-11f1-8001-daa6852bd5a7
2026-04-17 10:40:37,735 - distributed.scheduler - INFO - Receive client connection: Client-df36be52-3a49-11f1-8001-daa6852bd5a7
2026-04-17 10:30:40,159 - distributed.scheduler - INFO - Receive client connection: Client-worker-7b061e2e-3a48-11f1-8237-8288bfee4388
2026-04-17 10:30:36,290 - distributed.scheduler - INFO - Receive client connection: Client-78b25567-3a48-11f1-8001-daa6852bd5a7
2026-04-17 10:20:34,715 - distributed.scheduler - INFO - Receive client connection: Client-122811bb-3a47-11f1-8001-daa6852bd5a7
2026-04-17 10:12:56,540 - distributed.scheduler - INFO - Receive client connection: Client-0111194a-3a46-11f1-8001-daa6852bd5a7
2026-04-17 10:00:38,950 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('len-agg-33b4affde99f8efc2f3be41ef0e84b9a', 0) NoneType: None
2026-04-17 10:00:38,949 - distributed.scheduler - ERROR - Couldn't gather keys {"('len-agg-33b4affde99f8efc2f3be41ef0e84b9a', 0)": []} state: ['waiting'] workers: [] NoneType: None
2026-04-17 10:00:31,709 - distributed.scheduler - INFO - Receive client connection: Client-451c1c9f-3a44-11f1-8001-daa6852bd5a7
2026-04-17 09:50:30,189 - distributed.scheduler - INFO - Receive client connection: Client-de9333d8-3a42-11f1-8001-daa6852bd5a7
2026-04-17 09:40:28,764 - distributed.scheduler - INFO - Receive client connection: Client-7818e1bc-3a41-11f1-8001-daa6852bd5a7
2026-04-17 09:32:38,946 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:41873
2026-04-17 09:32:38,943 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:41873', status: closed, memory: 0, processing: 0>
2026-04-17 09:32:38,064 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:41265', status: closed, memory: 0, processing: 0>
2026-04-17 09:32:27,245 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:41265
2026-04-17 09:32:27,243 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:41265', status: closed, memory: 0, processing: 0>
2026-04-17 09:32:26,464 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:46059', status: closed, memory: 0, processing: 0>
2026-04-17 09:32:15,444 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:46059
2026-04-17 09:32:15,441 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:46059', status: closed, memory: 0, processing: 0>
2026-04-17 09:32:14,469 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:43723', status: closed, memory: 0, processing: 0>
2026-04-17 09:31:47,892 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:43723
2026-04-17 09:31:47,890 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:43723', status: closed, memory: 0, processing: 0>
2026-04-17 09:31:46,929 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:39495', status: closed, memory: 0, processing: 0>
2026-04-17 09:31:05,296 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.207:39495
2026-04-17 09:31:05,294 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.207:39495', status: closed, memory: 0, processing: 0>
2026-04-17 09:31:04,140 - distributed.scheduler - INFO - Close client connection: Client-worker-37b4ab77-37c7-11f1-80b5-8288bfee4388
2026-04-17 09:31:04,116 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.207:46541', status: closed, memory: 0, processing: 0>
2026-04-17 09:31:04,089 - distributed.scheduler - INFO - Remove client Client-worker-37b4ab77-37c7-11f1-80b5-8288bfee4388
2026-04-17 09:30:27,317 - distributed.scheduler - INFO - Receive client connection: Client-119bfcac-3a40-11f1-8001-daa6852bd5a7
2026-04-17 09:20:25,825 - distributed.scheduler - INFO - Receive client connection: Client-ab16f231-3a3e-11f1-8001-daa6852bd5a7
2026-04-17 09:12:25,193 - distributed.scheduler - INFO - Receive client connection: Client-8c9dcd60-3a3d-11f1-8001-daa6852bd5a7
2026-04-17 09:00:22,787 - distributed.scheduler - INFO - Receive client connection: Client-de06c0c9-3a3b-11f1-8001-daa6852bd5a7
2026-04-17 08:50:21,282 - distributed.scheduler - INFO - Receive client connection: Client-7780890c-3a3a-11f1-8001-daa6852bd5a7
2026-04-17 08:40:19,750 - distributed.scheduler - INFO - Receive client connection: Client-10f65f4b-3a39-11f1-8001-daa6852bd5a7
2026-04-17 08:30:18,323 - distributed.scheduler - INFO - Receive client connection: Client-aa7bb3da-3a37-11f1-8001-daa6852bd5a7
2026-04-17 08:20:16,843 - distributed.scheduler - INFO - Receive client connection: Client-43f8a2d7-3a36-11f1-8001-daa6852bd5a7
2026-04-17 08:11:48,395 - distributed.scheduler - INFO - Receive client connection: Client-14ea2494-3a35-11f1-8001-daa6852bd5a7
2026-04-17 08:00:13,842 - distributed.scheduler - INFO - Receive client connection: Client-76edc57e-3a33-11f1-8001-daa6852bd5a7
2026-04-17 07:50:12,372 - distributed.scheduler - INFO - Receive client connection: Client-106cd986-3a32-11f1-8001-daa6852bd5a7
2026-04-17 07:40:10,963 - distributed.scheduler - INFO - Receive client connection: Client-a9f48a9a-3a30-11f1-8001-daa6852bd5a7
2026-04-17 07:30:09,504 - distributed.scheduler - INFO - Receive client connection: Client-4374bd0a-3a2f-11f1-8001-daa6852bd5a7
2026-04-17 07:20:08,013 - distributed.scheduler - INFO - Receive client connection: Client-dcf18510-3a2d-11f1-8001-daa6852bd5a7
2026-04-17 07:11:55,671 - distributed.scheduler - INFO - Receive client connection: Client-b77c38c1-3a2c-11f1-8001-daa6852bd5a7
2026-04-17 07:00:04,990 - distributed.scheduler - INFO - Receive client connection: Client-0fe1f6ef-3a2b-11f1-8001-daa6852bd5a7
2026-04-17 06:50:03,514 - distributed.scheduler - INFO - Receive client connection: Client-a95ff7cd-3a29-11f1-8001-daa6852bd5a7
2026-04-17 06:40:02,027 - distributed.scheduler - INFO - Receive client connection: Client-42dc5a38-3a28-11f1-8001-daa6852bd5a7
2026-04-17 06:30:05,315 - distributed.scheduler - INFO - Receive client connection: Client-worker-df302e06-3a26-11f1-8279-f28f846166bd
2026-04-17 06:30:00,560 - distributed.scheduler - INFO - Receive client connection: Client-dc5c2ddd-3a26-11f1-8001-daa6852bd5a7
2026-04-17 06:20:59,230 - distributed.scheduler - INFO - Receive client connection: Client-99b35b40-3a25-11f1-8001-daa6852bd5a7
2026-04-17 06:13:46,822 - distributed.scheduler - INFO - Receive client connection: Client-97f83615-3a24-11f1-8001-daa6852bd5a7
2026-04-17 06:05:03,977 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:05:03,209 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:43827 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s
2026-04-17 06:05:03,192 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:05:03,175 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:05:03,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:05:03,165 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:05:03,164 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:05:03,126 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:46541 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:05:03,126 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:05:03,126 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:05:03,125 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:05:03,098 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:05:03,096 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:05:03,086 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:43827 after 30 s')
2026-04-17 06:04:33,920 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:40855
2026-04-17 06:04:33,919 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:40855', status: closed, memory: 0, processing: 0>
2026-04-17 06:04:32,955 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:43827', status: closed, memory: 0, processing: 0>
2026-04-17 06:04:20,557 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:41295 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s
2026-04-17 06:04:20,541 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:04:20,411 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:04:20,410 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:04:20,409 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:04:20,407 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:04:20,387 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:04:20,382 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:04:20,381 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:04:20,380 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:04:20,379 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:46541 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:04:20,378 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:04:20,378 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:04:20,366 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41295 after 30 s')
2026-04-17 06:03:51,342 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:43827
2026-04-17 06:03:51,340 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:43827', status: closed, memory: 0, processing: 0>
2026-04-17 06:03:50,255 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:41295', status: closed, memory: 0, processing: 0>
2026-04-17 06:03:39,213 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:41445 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s
2026-04-17 06:03:39,185 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,179 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,178 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,178 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:46541 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,144 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,142 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,141 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,129 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,128 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,111 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,109 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,108 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,104 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,104 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:39,091 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41445 after 30 s')
2026-04-17 06:03:09,788 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:41295
2026-04-17 06:03:09,786 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:41295', status: closed, memory: 0, processing: 0>
2026-04-17 06:03:08,961 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:41445', status: closed, memory: 0, processing: 0>
2026-04-17 06:02:45,553 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:39171 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:39171 after 30 s
2026-04-17 06:02:45,553 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:39171 after 30 s')
2026-04-17 06:02:44,146 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:39171 after 30 s')
2026-04-17 06:02:14,842 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:41445
2026-04-17 06:02:14,840 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:41445', status: closed, memory: 0, processing: 0>
2026-04-17 06:02:13,876 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:39171', status: closed, memory: 0, processing: 0>
2026-04-17 06:01:38,820 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:39171
2026-04-17 06:01:38,819 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:39171', status: closed, memory: 0, processing: 0>
2026-04-17 06:01:37,857 - distributed.scheduler - INFO - Close client connection: Client-worker-0bb24f2c-39a5-11f1-81ca-f28f846166bd
2026-04-17 06:01:37,849 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:34163', status: closed, memory: 0, processing: 0>
2026-04-17 06:01:37,848 - distributed.scheduler - INFO - Remove client Client-worker-0bb24f2c-39a5-11f1-81ca-f28f846166bd
2026-04-17 06:00:56,237 - distributed.scheduler - INFO - Receive client connection: Client-cca5d602-3a22-11f1-8001-daa6852bd5a7
2026-04-17 05:50:54,644 - distributed.scheduler - INFO - Receive client connection: Client-66150d5b-3a21-11f1-8001-daa6852bd5a7
2026-04-17 05:40:53,072 - distributed.scheduler - INFO - Receive client connection: Client-ff8540af-3a1f-11f1-8001-daa6852bd5a7
2026-04-17 05:30:51,534 - distributed.scheduler - INFO - Receive client connection: Client-98f9acbd-3a1e-11f1-8001-daa6852bd5a7
2026-04-17 05:20:49,970 - distributed.scheduler - INFO - Receive client connection: Client-326b089b-3a1d-11f1-8001-daa6852bd5a7
2026-04-17 05:12:49,495 - distributed.scheduler - INFO - Receive client connection: Client-14096434-3a1c-11f1-8001-daa6852bd5a7
2026-04-17 05:00:46,835 - distributed.scheduler - INFO - Receive client connection: Client-654b01a3-3a1a-11f1-8001-daa6852bd5a7
2026-04-17 04:50:45,305 - distributed.scheduler - INFO - Receive client connection: Client-febd0f1b-3a18-11f1-8001-daa6852bd5a7
2026-04-17 04:40:43,767 - distributed.scheduler - INFO - Receive client connection: Client-9833e079-3a17-11f1-8001-daa6852bd5a7
2026-04-17 04:32:05,518 - distributed.scheduler - INFO - Close client connection: Client-31b258f2-3a16-11f1-8001-daa6852bd5a7
2026-04-17 04:32:05,517 - distributed.scheduler - INFO - Remove client Client-31b258f2-3a16-11f1-8001-daa6852bd5a7
2026-04-17 04:32:05,515 - distributed.scheduler - INFO - Remove client Client-31b258f2-3a16-11f1-8001-daa6852bd5a7
2026-04-17 04:30:42,298 - distributed.scheduler - INFO - Receive client connection: Client-31b258f2-3a16-11f1-8001-daa6852bd5a7
2026-04-17 04:21:18,441 - distributed.scheduler - INFO - Close client connection: Client-cb33ff31-3a14-11f1-8001-daa6852bd5a7
2026-04-17 04:21:18,439 - distributed.scheduler - INFO - Remove client Client-cb33ff31-3a14-11f1-8001-daa6852bd5a7
2026-04-17 04:21:18,438 - distributed.scheduler - INFO - Remove client Client-cb33ff31-3a14-11f1-8001-daa6852bd5a7
2026-04-17 04:20:40,845 - distributed.scheduler - INFO - Receive client connection: Client-cb33ff31-3a14-11f1-8001-daa6852bd5a7
2026-04-17 04:13:55,598 - distributed.scheduler - INFO - Close client connection: Client-c6a2b0d5-3a13-11f1-8001-daa6852bd5a7
2026-04-17 04:13:55,598 - distributed.scheduler - INFO - Remove client Client-c6a2b0d5-3a13-11f1-8001-daa6852bd5a7
2026-04-17 04:13:55,597 - distributed.scheduler - INFO - Remove client Client-c6a2b0d5-3a13-11f1-8001-daa6852bd5a7
2026-04-17 04:13:23,683 - distributed.scheduler - INFO - Receive client connection: Client-c6a2b0d5-3a13-11f1-8001-daa6852bd5a7
2026-04-17 04:01:50,426 - distributed.scheduler - INFO - Close client connection: Client-fe38d6b5-3a11-11f1-8001-daa6852bd5a7
2026-04-17 04:01:50,424 - distributed.scheduler - INFO - Remove client Client-fe38d6b5-3a11-11f1-8001-daa6852bd5a7
2026-04-17 04:01:50,423 - distributed.scheduler - INFO - Remove client Client-fe38d6b5-3a11-11f1-8001-daa6852bd5a7
2026-04-17 04:00:37,949 - distributed.scheduler - INFO - Receive client connection: Client-fe38d6b5-3a11-11f1-8001-daa6852bd5a7
2026-04-17 03:51:12,646 - distributed.scheduler - INFO - Close client connection: Client-97b2937e-3a10-11f1-8001-daa6852bd5a7
2026-04-17 03:51:12,646 - distributed.scheduler - INFO - Remove client Client-97b2937e-3a10-11f1-8001-daa6852bd5a7
2026-04-17 03:51:12,645 - distributed.scheduler - INFO - Remove client Client-97b2937e-3a10-11f1-8001-daa6852bd5a7
2026-04-17 03:50:36,443 - distributed.scheduler - INFO - Receive client connection: Client-97b2937e-3a10-11f1-8001-daa6852bd5a7
2026-04-17 03:41:11,645 - distributed.scheduler - INFO - Close client connection: Client-3133b589-3a0f-11f1-8001-daa6852bd5a7
2026-04-17 03:41:11,644 - distributed.scheduler - INFO - Remove client Client-3133b589-3a0f-11f1-8001-daa6852bd5a7
2026-04-17 03:41:11,643 - distributed.scheduler - INFO - Remove client Client-3133b589-3a0f-11f1-8001-daa6852bd5a7
2026-04-17 03:40:34,992 - distributed.scheduler - INFO - Receive client connection: Client-3133b589-3a0f-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,387 - distributed.scheduler - INFO - Close client connection: Client-e886ac60-39ce-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,387 - distributed.scheduler - INFO - Close client connection: Client-b59215f3-39d1-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,387 - distributed.scheduler - INFO - Close client connection: Client-b231584f-39d0-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,387 - distributed.scheduler - INFO - Close client connection: Client-1c1bcc66-39d3-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,387 - distributed.scheduler - INFO - Close client connection: Client-e932068c-39d5-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,386 - distributed.scheduler - INFO - Close client connection: Client-82ad206d-39d4-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,386 - distributed.scheduler - INFO - Close client connection: Client-4fc761c5-39d7-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,386 - distributed.scheduler - INFO - Close client connection: Client-14f79586-39d9-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,386 - distributed.scheduler - INFO - Close client connection: Client-1ce77474-39da-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,386 - distributed.scheduler - INFO - Close client connection: Client-8369a7aa-39db-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,386 - distributed.scheduler - INFO - Close client connection: Client-e9ea6f8e-39dc-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,386 - distributed.scheduler - INFO - Close client connection: Client-50729acf-39de-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,386 - distributed.scheduler - INFO - Close client connection: Client-b6ee1bb1-39df-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,386 - distributed.scheduler - INFO - Close client connection: Client-8409f309-39e2-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,386 - distributed.scheduler - INFO - Close client connection: Client-6cd12e2f-39e1-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,385 - distributed.scheduler - INFO - Close client connection: Client-ea935d71-39e3-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,385 - distributed.scheduler - INFO - Close client connection: Client-b7a93a3c-39e6-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,385 - distributed.scheduler - INFO - Close client connection: Client-5119f3a4-39e5-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,385 - distributed.scheduler - INFO - Close client connection: Client-1e306dea-39e8-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,385 - distributed.scheduler - INFO - Close client connection: Client-d94aa679-39e9-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,385 - distributed.scheduler - INFO - Close client connection: Client-eb400e71-39ea-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,385 - distributed.scheduler - INFO - Close client connection: Client-51ccbf5b-39ec-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,385 - distributed.scheduler - INFO - Close client connection: Client-b856e205-39ed-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,385 - distributed.scheduler - INFO - Close client connection: Client-fafbbef9-39ee-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,385 - distributed.scheduler - INFO - Close client connection: Client-618802d9-39f0-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,384 - distributed.scheduler - INFO - Close client connection: Client-13649087-39f2-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,384 - distributed.scheduler - INFO - Close client connection: Client-2e94dcb8-39f3-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,384 - distributed.scheduler - INFO - Close client connection: Client-95210bb3-39f4-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,384 - distributed.scheduler - INFO - Close client connection: Client-fbb06080-39f5-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,384 - distributed.scheduler - INFO - Close client connection: Client-622b3788-39f7-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,384 - distributed.scheduler - INFO - Close client connection: Client-c8a9a815-39f8-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,384 - distributed.scheduler - INFO - Close client connection: Client-95a8e7bc-39fb-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,384 - distributed.scheduler - INFO - Close client connection: Client-76f786c6-39fa-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,383 - distributed.scheduler - INFO - Close client connection: Client-fc3b45b8-39fc-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,383 - distributed.scheduler - INFO - Close client connection: Client-62c27a2e-39fe-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,383 - distributed.scheduler - INFO - Close client connection: Client-c944fa4f-39ff-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,382 - distributed.scheduler - INFO - Close client connection: Client-2fce82b1-3a01-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,382 - distributed.scheduler - INFO - Close client connection: Client-fce8e62b-3a03-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,382 - distributed.scheduler - INFO - Close client connection: Client-f2dd93d9-3a02-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,382 - distributed.scheduler - INFO - Close client connection: Client-636f43a4-3a05-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,382 - distributed.scheduler - INFO - Close client connection: Client-c9f9e831-3a06-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,382 - distributed.scheduler - INFO - Close client connection: Client-3087d61e-3a08-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,382 - distributed.scheduler - INFO - Close client connection: Client-97153d53-3a09-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,381 - distributed.scheduler - INFO - Close client connection: Client-64177984-3a0c-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,379 - distributed.scheduler - INFO - Close client connection: Client-60b1107f-3a0b-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,379 - distributed.scheduler - INFO - Remove client Client-e886ac60-39ce-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,379 - distributed.scheduler - INFO - Remove client Client-b59215f3-39d1-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,379 - distributed.scheduler - INFO - Remove client Client-b231584f-39d0-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,378 - distributed.scheduler - INFO - Remove client Client-1c1bcc66-39d3-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,378 - distributed.scheduler - INFO - Remove client Client-e932068c-39d5-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,378 - distributed.scheduler - INFO - Remove client Client-82ad206d-39d4-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,378 - distributed.scheduler - INFO - Remove client Client-4fc761c5-39d7-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,377 - distributed.scheduler - INFO - Remove client Client-14f79586-39d9-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,377 - distributed.scheduler - INFO - Remove client Client-1ce77474-39da-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,377 - distributed.scheduler - INFO - Remove client Client-8369a7aa-39db-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,377 - distributed.scheduler - INFO - Remove client Client-e9ea6f8e-39dc-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,376 - distributed.scheduler - INFO - Remove client Client-50729acf-39de-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,376 - distributed.scheduler - INFO - Remove client Client-b6ee1bb1-39df-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,376 - distributed.scheduler - INFO - Remove client Client-8409f309-39e2-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,376 - distributed.scheduler - INFO - Remove client Client-6cd12e2f-39e1-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,375 - distributed.scheduler - INFO - Remove client Client-ea935d71-39e3-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,375 - distributed.scheduler - INFO - Remove client Client-b7a93a3c-39e6-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,375 - distributed.scheduler - INFO - Remove client Client-5119f3a4-39e5-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,375 - distributed.scheduler - INFO - Remove client Client-1e306dea-39e8-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,374 - distributed.scheduler - INFO - Remove client Client-d94aa679-39e9-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,374 - distributed.scheduler - INFO - Remove client Client-eb400e71-39ea-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,374 - distributed.scheduler - INFO - Remove client Client-51ccbf5b-39ec-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,374 - distributed.scheduler - INFO - Remove client Client-b856e205-39ed-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,374 - distributed.scheduler - INFO - Remove client Client-fafbbef9-39ee-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,373 - distributed.scheduler - INFO - Remove client Client-618802d9-39f0-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,373 - distributed.scheduler - INFO - Remove client Client-13649087-39f2-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,373 - distributed.scheduler - INFO - Remove client Client-2e94dcb8-39f3-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,372 - distributed.scheduler - INFO - Remove client Client-95210bb3-39f4-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,372 - distributed.scheduler - INFO - Remove client Client-fbb06080-39f5-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,372 - distributed.scheduler - INFO - Remove client Client-622b3788-39f7-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,371 - distributed.scheduler - INFO - Remove client Client-c8a9a815-39f8-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,371 - distributed.scheduler - INFO - Remove client Client-95a8e7bc-39fb-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,371 - distributed.scheduler - INFO - Remove client Client-76f786c6-39fa-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,362 - distributed.scheduler - INFO - Remove client Client-fc3b45b8-39fc-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,361 - distributed.scheduler - INFO - Remove client Client-62c27a2e-39fe-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,361 - distributed.scheduler - INFO - Remove client Client-c944fa4f-39ff-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,360 - distributed.scheduler - INFO - Remove client Client-2fce82b1-3a01-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,359 - distributed.scheduler - INFO - Remove client Client-fce8e62b-3a03-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,358 - distributed.scheduler - INFO - Remove client Client-f2dd93d9-3a02-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,351 - distributed.scheduler - INFO - Remove client Client-636f43a4-3a05-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,350 - distributed.scheduler - INFO - Remove client Client-c9f9e831-3a06-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,349 - distributed.scheduler - INFO - Remove client Client-3087d61e-3a08-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,348 - distributed.scheduler - INFO - Remove client Client-97153d53-3a09-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,347 - distributed.scheduler - INFO - Remove client Client-64177984-3a0c-11f1-8001-daa6852bd5a7
2026-04-17 03:31:45,341 - distributed.scheduler - INFO - Remove client Client-60b1107f-3a0b-11f1-8001-daa6852bd5a7
2026-04-17 03:31:44,602 - distributed.scheduler - INFO - Close client connection: Client-caa40799-3a0d-11f1-8001-daa6852bd5a7
2026-04-17 03:31:44,600 - distributed.scheduler - INFO - Remove client Client-caa40799-3a0d-11f1-8001-daa6852bd5a7
2026-04-17 03:31:44,599 - distributed.scheduler - INFO - Remove client Client-caa40799-3a0d-11f1-8001-daa6852bd5a7
2026-04-17 03:30:33,422 - distributed.scheduler - INFO - Receive client connection: Client-caa40799-3a0d-11f1-8001-daa6852bd5a7
2026-04-17 03:20:31,872 - distributed.scheduler - INFO - Receive client connection: Client-64177984-3a0c-11f1-8001-daa6852bd5a7
2026-04-17 03:13:16,667 - distributed.scheduler - INFO - Receive client connection: Client-60b1107f-3a0b-11f1-8001-daa6852bd5a7
2026-04-17 03:00:28,929 - distributed.scheduler - INFO - Receive client connection: Client-97153d53-3a09-11f1-8001-daa6852bd5a7
2026-04-17 02:50:27,383 - distributed.scheduler - INFO - Receive client connection: Client-3087d61e-3a08-11f1-8001-daa6852bd5a7
2026-04-17 02:40:25,821 - distributed.scheduler - INFO - Receive client connection: Client-c9f9e831-3a06-11f1-8001-daa6852bd5a7
2026-04-17 02:30:24,297 - distributed.scheduler - INFO - Receive client connection: Client-636f43a4-3a05-11f1-8001-daa6852bd5a7
2026-04-17 02:20:22,786 - distributed.scheduler - INFO - Receive client connection: Client-fce8e62b-3a03-11f1-8001-daa6852bd5a7
2026-04-17 02:12:56,433 - distributed.scheduler - INFO - Receive client connection: Client-f2dd93d9-3a02-11f1-8001-daa6852bd5a7
2026-04-17 02:00:19,680 - distributed.scheduler - INFO - Receive client connection: Client-2fce82b1-3a01-11f1-8001-daa6852bd5a7
2026-04-17 01:50:18,166 - distributed.scheduler - INFO - Receive client connection: Client-c944fa4f-39ff-11f1-8001-daa6852bd5a7
2026-04-17 01:40:16,680 - distributed.scheduler - INFO - Receive client connection: Client-62c27a2e-39fe-11f1-8001-daa6852bd5a7
2026-04-17 01:30:15,168 - distributed.scheduler - INFO - Receive client connection: Client-fc3b45b8-39fc-11f1-8001-daa6852bd5a7
2026-04-17 01:20:13,587 - distributed.scheduler - INFO - Receive client connection: Client-95a8e7bc-39fb-11f1-8001-daa6852bd5a7
2026-04-17 01:12:12,591 - distributed.scheduler - INFO - Receive client connection: Client-76f786c6-39fa-11f1-8001-daa6852bd5a7
2026-04-17 01:00:10,672 - distributed.scheduler - INFO - Receive client connection: Client-c8a9a815-39f8-11f1-8001-daa6852bd5a7
2026-04-17 00:50:09,208 - distributed.scheduler - INFO - Receive client connection: Client-622b3788-39f7-11f1-8001-daa6852bd5a7
2026-04-17 00:40:07,783 - distributed.scheduler - INFO - Receive client connection: Client-fbb06080-39f5-11f1-8001-daa6852bd5a7
2026-04-17 00:30:06,276 - distributed.scheduler - INFO - Receive client connection: Client-95210bb3-39f4-11f1-8001-daa6852bd5a7
2026-04-17 00:20:04,675 - distributed.scheduler - INFO - Receive client connection: Client-2e94dcb8-39f3-11f1-8001-daa6852bd5a7
2026-04-17 00:12:09,584 - distributed.scheduler - INFO - Receive client connection: Client-13649087-39f2-11f1-8001-daa6852bd5a7
2026-04-17 00:00:01,667 - distributed.scheduler - INFO - Receive client connection: Client-618802d9-39f0-11f1-8001-daa6852bd5a7
2026-04-16 23:50:00,121 - distributed.scheduler - INFO - Receive client connection: Client-fafbbef9-39ee-11f1-8001-daa6852bd5a7
2026-04-16 23:40:58,816 - distributed.scheduler - INFO - Receive client connection: Client-b856e205-39ed-11f1-8001-daa6852bd5a7
2026-04-16 23:30:57,285 - distributed.scheduler - INFO - Receive client connection: Client-51ccbf5b-39ec-11f1-8001-daa6852bd5a7
2026-04-16 23:20:55,738 - distributed.scheduler - INFO - Receive client connection: Client-eb400e71-39ea-11f1-8001-daa6852bd5a7
2026-04-16 23:13:16,103 - distributed.scheduler - INFO - Receive client connection: Client-d94aa679-39e9-11f1-8001-daa6852bd5a7
2026-04-16 23:00:52,712 - distributed.scheduler - INFO - Receive client connection: Client-1e306dea-39e8-11f1-8001-daa6852bd5a7
2026-04-16 22:50:51,202 - distributed.scheduler - INFO - Receive client connection: Client-b7a93a3c-39e6-11f1-8001-daa6852bd5a7
2026-04-16 22:40:49,632 - distributed.scheduler - INFO - Receive client connection: Client-5119f3a4-39e5-11f1-8001-daa6852bd5a7
2026-04-16 22:30:48,130 - distributed.scheduler - INFO - Receive client connection: Client-ea935d71-39e3-11f1-8001-daa6852bd5a7
2026-04-16 22:20:46,606 - distributed.scheduler - INFO - Receive client connection: Client-8409f309-39e2-11f1-8001-daa6852bd5a7
2026-04-16 22:12:58,144 - distributed.scheduler - INFO - Receive client connection: Client-6cd12e2f-39e1-11f1-8001-daa6852bd5a7
2026-04-16 22:00:43,576 - distributed.scheduler - INFO - Receive client connection: Client-b6ee1bb1-39df-11f1-8001-daa6852bd5a7
2026-04-16 21:50:42,061 - distributed.scheduler - INFO - Receive client connection: Client-50729acf-39de-11f1-8001-daa6852bd5a7
2026-04-16 21:40:40,550 - distributed.scheduler - INFO - Receive client connection: Client-e9ea6f8e-39dc-11f1-8001-daa6852bd5a7
2026-04-16 21:30:39,079 - distributed.scheduler - INFO - Receive client connection: Client-8369a7aa-39db-11f1-8001-daa6852bd5a7
2026-04-16 21:20:37,594 - distributed.scheduler - INFO - Receive client connection: Client-1ce77474-39da-11f1-8001-daa6852bd5a7
2026-04-16 21:13:14,782 - distributed.scheduler - INFO - Receive client connection: Client-14f79586-39d9-11f1-8001-daa6852bd5a7
2026-04-16 21:00:34,463 - distributed.scheduler - INFO - Receive client connection: Client-4fc761c5-39d7-11f1-8001-daa6852bd5a7
2026-04-16 20:50:32,858 - distributed.scheduler - INFO - Receive client connection: Client-e932068c-39d5-11f1-8001-daa6852bd5a7
2026-04-16 20:40:31,364 - distributed.scheduler - INFO - Receive client connection: Client-82ad206d-39d4-11f1-8001-daa6852bd5a7
2026-04-16 20:30:29,784 - distributed.scheduler - INFO - Receive client connection: Client-1c1bcc66-39d3-11f1-8001-daa6852bd5a7
2026-04-16 20:20:28,261 - distributed.scheduler - INFO - Receive client connection: Client-b59215f3-39d1-11f1-8001-daa6852bd5a7
2026-04-16 20:13:13,088 - distributed.scheduler - INFO - Receive client connection: Client-b231584f-39d0-11f1-8001-daa6852bd5a7
2026-04-16 20:00:25,269 - distributed.scheduler - INFO - Receive client connection: Client-e886ac60-39ce-11f1-8001-daa6852bd5a7
2026-04-16 19:50:59,456 - distributed.scheduler - INFO - Close client connection: Client-81fd0487-39cd-11f1-8001-daa6852bd5a7
2026-04-16 19:50:59,454 - distributed.scheduler - INFO - Remove client Client-81fd0487-39cd-11f1-8001-daa6852bd5a7
2026-04-16 19:50:59,452 - distributed.scheduler - INFO - Remove client Client-81fd0487-39cd-11f1-8001-daa6852bd5a7
2026-04-16 19:50:23,742 - distributed.scheduler - INFO - Receive client connection: Client-81fd0487-39cd-11f1-8001-daa6852bd5a7
2026-04-16 19:40:58,929 - distributed.scheduler - INFO - Close client connection: Client-1b6ea191-39cc-11f1-8001-daa6852bd5a7
2026-04-16 19:40:58,927 - distributed.scheduler - INFO - Remove client Client-1b6ea191-39cc-11f1-8001-daa6852bd5a7
2026-04-16 19:40:58,925 - distributed.scheduler - INFO - Remove client Client-1b6ea191-39cc-11f1-8001-daa6852bd5a7
2026-04-16 19:40:22,186 - distributed.scheduler - INFO - Receive client connection: Client-1b6ea191-39cc-11f1-8001-daa6852bd5a7
2026-04-16 19:31:33,942 - distributed.scheduler - INFO - Close client connection: Client-b4e8b701-39ca-11f1-8001-daa6852bd5a7
2026-04-16 19:31:33,941 - distributed.scheduler - INFO - Remove client Client-b4e8b701-39ca-11f1-8001-daa6852bd5a7
2026-04-16 19:31:33,940 - distributed.scheduler - INFO - Remove client Client-b4e8b701-39ca-11f1-8001-daa6852bd5a7
2026-04-16 19:30:20,684 - distributed.scheduler - INFO - Receive client connection: Client-b4e8b701-39ca-11f1-8001-daa6852bd5a7
2026-04-16 19:21:01,775 - distributed.scheduler - INFO - Close client connection: Client-4e6680d9-39c9-11f1-8001-daa6852bd5a7
2026-04-16 19:21:01,774 - distributed.scheduler - INFO - Remove client Client-4e6680d9-39c9-11f1-8001-daa6852bd5a7
2026-04-16 19:21:01,772 - distributed.scheduler - INFO - Remove client Client-4e6680d9-39c9-11f1-8001-daa6852bd5a7
2026-04-16 19:20:19,203 - distributed.scheduler - INFO - Receive client connection: Client-4e6680d9-39c9-11f1-8001-daa6852bd5a7
2026-04-16 19:13:22,387 - distributed.scheduler - INFO - Close client connection: Client-41c79004-39c8-11f1-8001-daa6852bd5a7
2026-04-16 19:13:22,386 - distributed.scheduler - INFO - Remove client Client-41c79004-39c8-11f1-8001-daa6852bd5a7
2026-04-16 19:13:22,385 - distributed.scheduler - INFO - Remove client Client-41c79004-39c8-11f1-8001-daa6852bd5a7
2026-04-16 19:12:48,536 - distributed.scheduler - INFO - Receive client connection: Client-41c79004-39c8-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,911 - distributed.scheduler - INFO - Close client connection: Client-3c203d67-39a9-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,911 - distributed.scheduler - INFO - Close client connection: Client-a2ad9545-39aa-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,911 - distributed.scheduler - INFO - Close client connection: Client-09321bbb-39ac-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,911 - distributed.scheduler - INFO - Close client connection: Client-6fc0758f-39ad-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,911 - distributed.scheduler - INFO - Close client connection: Client-3d206df6-39af-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,911 - distributed.scheduler - INFO - Close client connection: Client-3cbb9cf7-39b0-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,911 - distributed.scheduler - INFO - Close client connection: Client-a33f7853-39b1-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,911 - distributed.scheduler - INFO - Close client connection: Client-09c49051-39b3-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,910 - distributed.scheduler - INFO - Close client connection: Client-704b503e-39b4-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,910 - distributed.scheduler - INFO - Close client connection: Client-d6dad548-39b5-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,910 - distributed.scheduler - INFO - Close client connection: Client-7ffa26d9-39b8-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,910 - distributed.scheduler - INFO - Close client connection: Client-ac8ebaa4-39b7-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,910 - distributed.scheduler - INFO - Close client connection: Client-e68804fd-39b9-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,909 - distributed.scheduler - INFO - Close client connection: Client-4d1459d3-39bb-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,909 - distributed.scheduler - INFO - Close client connection: Client-b3a1be0c-39bc-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,909 - distributed.scheduler - INFO - Close client connection: Client-1a2f8a6b-39be-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,909 - distributed.scheduler - INFO - Close client connection: Client-e732e0cf-39c0-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,908 - distributed.scheduler - INFO - Close client connection: Client-eab9a0ea-39bf-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,908 - distributed.scheduler - INFO - Close client connection: Client-4db64e19-39c2-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,908 - distributed.scheduler - INFO - Close client connection: Client-b4367f40-39c3-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,908 - distributed.scheduler - INFO - Close client connection: Client-1abf1463-39c5-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,906 - distributed.scheduler - INFO - Remove client Client-3c203d67-39a9-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,905 - distributed.scheduler - INFO - Remove client Client-a2ad9545-39aa-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,905 - distributed.scheduler - INFO - Remove client Client-09321bbb-39ac-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,904 - distributed.scheduler - INFO - Remove client Client-6fc0758f-39ad-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,904 - distributed.scheduler - INFO - Remove client Client-3d206df6-39af-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,903 - distributed.scheduler - INFO - Remove client Client-3cbb9cf7-39b0-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,903 - distributed.scheduler - INFO - Remove client Client-a33f7853-39b1-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,902 - distributed.scheduler - INFO - Remove client Client-09c49051-39b3-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,902 - distributed.scheduler - INFO - Remove client Client-704b503e-39b4-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,901 - distributed.scheduler - INFO - Remove client Client-d6dad548-39b5-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,901 - distributed.scheduler - INFO - Remove client Client-7ffa26d9-39b8-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,900 - distributed.scheduler - INFO - Remove client Client-ac8ebaa4-39b7-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,900 - distributed.scheduler - INFO - Remove client Client-e68804fd-39b9-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,893 - distributed.scheduler - INFO - Remove client Client-4d1459d3-39bb-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,893 - distributed.scheduler - INFO - Remove client Client-b3a1be0c-39bc-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,892 - distributed.scheduler - INFO - Remove client Client-1a2f8a6b-39be-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,891 - distributed.scheduler - INFO - Remove client Client-e732e0cf-39c0-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,890 - distributed.scheduler - INFO - Remove client Client-eab9a0ea-39bf-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,890 - distributed.scheduler - INFO - Remove client Client-4db64e19-39c2-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,889 - distributed.scheduler - INFO - Remove client Client-b4367f40-39c3-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,888 - distributed.scheduler - INFO - Remove client Client-1abf1463-39c5-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,303 - distributed.scheduler - INFO - Close client connection: Client-814e17b8-39c6-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,302 - distributed.scheduler - INFO - Remove client Client-814e17b8-39c6-11f1-8001-daa6852bd5a7
2026-04-16 19:01:30,300 - distributed.scheduler - INFO - Remove client Client-814e17b8-39c6-11f1-8001-daa6852bd5a7
2026-04-16 19:00:16,108 - distributed.scheduler - INFO - Receive client connection: Client-814e17b8-39c6-11f1-8001-daa6852bd5a7
2026-04-16 18:50:14,547 - distributed.scheduler - INFO - Receive client connection: Client-1abf1463-39c5-11f1-8001-daa6852bd5a7
2026-04-16 18:40:13,027 - distributed.scheduler - INFO - Receive client connection: Client-b4367f40-39c3-11f1-8001-daa6852bd5a7
2026-04-16 18:30:11,564 - distributed.scheduler - INFO - Receive client connection: Client-4db64e19-39c2-11f1-8001-daa6852bd5a7
2026-04-16 18:20:10,078 - distributed.scheduler - INFO - Receive client connection: Client-e732e0cf-39c0-11f1-8001-daa6852bd5a7
2026-04-16 18:13:06,493 - distributed.scheduler - INFO - Receive client connection: Client-eab9a0ea-39bf-11f1-8001-daa6852bd5a7
2026-04-16 18:00:07,133 - distributed.scheduler - INFO - Receive client connection: Client-1a2f8a6b-39be-11f1-8001-daa6852bd5a7
2026-04-16 17:50:05,576 - distributed.scheduler - INFO - Receive client connection: Client-b3a1be0c-39bc-11f1-8001-daa6852bd5a7
2026-04-16 17:40:04,027 - distributed.scheduler - INFO - Receive client connection: Client-4d1459d3-39bb-11f1-8001-daa6852bd5a7
2026-04-16 17:30:02,482 - distributed.scheduler - INFO - Receive client connection: Client-e68804fd-39b9-11f1-8001-daa6852bd5a7
2026-04-16 17:20:00,929 - distributed.scheduler - INFO - Receive client connection: Client-7ffa26d9-39b8-11f1-8001-daa6852bd5a7
2026-04-16 17:14:06,265 - distributed.scheduler - INFO - Receive client connection: Client-ac8ebaa4-39b7-11f1-8001-daa6852bd5a7
2026-04-16 17:00:58,193 - distributed.scheduler - INFO - Receive client connection: Client-d6dad548-39b5-11f1-8001-daa6852bd5a7
2026-04-16 16:50:56,628 - distributed.scheduler - INFO - Receive client connection: Client-704b503e-39b4-11f1-8001-daa6852bd5a7
2026-04-16 16:40:55,116 - distributed.scheduler - INFO - Receive client connection: Client-09c49051-39b3-11f1-8001-daa6852bd5a7
2026-04-16 16:30:53,626 - distributed.scheduler - INFO - Receive client connection: Client-a33f7853-39b1-11f1-8001-daa6852bd5a7
2026-04-16 16:20:52,140 - distributed.scheduler - INFO - Receive client connection: Client-3cbb9cf7-39b0-11f1-8001-daa6852bd5a7
2026-04-16 16:13:43,294 - distributed.scheduler - INFO - Receive client connection: Client-3d206df6-39af-11f1-8001-daa6852bd5a7
2026-04-16 16:00:49,243 - distributed.scheduler - INFO - Receive client connection: Client-6fc0758f-39ad-11f1-8001-daa6852bd5a7
2026-04-16 15:50:47,692 - distributed.scheduler - INFO - Receive client connection: Client-09321bbb-39ac-11f1-8001-daa6852bd5a7
2026-04-16 15:40:46,194 - distributed.scheduler - INFO - Receive client connection: Client-a2ad9545-39aa-11f1-8001-daa6852bd5a7
2026-04-16 15:30:44,651 - distributed.scheduler - INFO - Receive client connection: Client-3c203d67-39a9-11f1-8001-daa6852bd5a7
2026-04-16 15:21:23,303 - distributed.scheduler - INFO - Close client connection: Client-d599ae52-39a7-11f1-8001-daa6852bd5a7
2026-04-16 15:21:23,302 - distributed.scheduler - INFO - Remove client Client-d599ae52-39a7-11f1-8001-daa6852bd5a7
2026-04-16 15:21:23,300 - distributed.scheduler - INFO - Remove client Client-d599ae52-39a7-11f1-8001-daa6852bd5a7
2026-04-16 15:20:43,145 - distributed.scheduler - INFO - Receive client connection: Client-d599ae52-39a7-11f1-8001-daa6852bd5a7
2026-04-16 15:13:11,234 - distributed.scheduler - INFO - Close client connection: Client-b295d8d9-39a6-11f1-8001-daa6852bd5a7
2026-04-16 15:13:11,233 - distributed.scheduler - INFO - Remove client Client-b295d8d9-39a6-11f1-8001-daa6852bd5a7
2026-04-16 15:13:11,232 - distributed.scheduler - INFO - Remove client Client-b295d8d9-39a6-11f1-8001-daa6852bd5a7
2026-04-16 15:12:34,900 - distributed.scheduler - INFO - Receive client connection: Client-b295d8d9-39a6-11f1-8001-daa6852bd5a7
2026-04-16 15:01:58,710 - distributed.scheduler - INFO - Close client connection: Client-08a26d65-39a5-11f1-8001-daa6852bd5a7
2026-04-16 15:01:58,709 - distributed.scheduler - INFO - Remove client Client-08a26d65-39a5-11f1-8001-daa6852bd5a7
2026-04-16 15:01:58,707 - distributed.scheduler - INFO - Remove client Client-08a26d65-39a5-11f1-8001-daa6852bd5a7
2026-04-16 15:00:45,434 - distributed.scheduler - INFO - Receive client connection: Client-worker-0bb24f2c-39a5-11f1-81ca-f28f846166bd
2026-04-16 15:00:40,297 - distributed.scheduler - INFO - Receive client connection: Client-08a26d65-39a5-11f1-8001-daa6852bd5a7
2026-04-16 14:51:13,036 - distributed.scheduler - INFO - Close client connection: Client-a22e60e8-39a3-11f1-8001-daa6852bd5a7
2026-04-16 14:51:13,036 - distributed.scheduler - INFO - Remove client Client-a22e60e8-39a3-11f1-8001-daa6852bd5a7
2026-04-16 14:51:13,035 - distributed.scheduler - INFO - Remove client Client-a22e60e8-39a3-11f1-8001-daa6852bd5a7
2026-04-16 14:50:38,892 - distributed.scheduler - INFO - Receive client connection: Client-a22e60e8-39a3-11f1-8001-daa6852bd5a7
2026-04-16 14:41:12,704 - distributed.scheduler - INFO - Close client connection: Client-3bb2f50e-39a2-11f1-8001-daa6852bd5a7
2026-04-16 14:41:12,702 - distributed.scheduler - INFO - Remove client Client-3bb2f50e-39a2-11f1-8001-daa6852bd5a7
2026-04-16 14:41:12,701 - distributed.scheduler - INFO - Remove client Client-3bb2f50e-39a2-11f1-8001-daa6852bd5a7
2026-04-16 14:40:37,457 - distributed.scheduler - INFO - Receive client connection: Client-3bb2f50e-39a2-11f1-8001-daa6852bd5a7
2026-04-16 14:31:41,085 - distributed.scheduler - INFO - Close client connection: Client-d52bcdeb-39a0-11f1-8001-daa6852bd5a7
2026-04-16 14:31:41,082 - distributed.scheduler - INFO - Remove client Client-d52bcdeb-39a0-11f1-8001-daa6852bd5a7
2026-04-16 14:31:41,081 - distributed.scheduler - INFO - Remove client Client-d52bcdeb-39a0-11f1-8001-daa6852bd5a7
2026-04-16 14:30:35,950 - distributed.scheduler - INFO - Receive client connection: Client-d52bcdeb-39a0-11f1-8001-daa6852bd5a7
2026-04-16 14:21:08,640 - distributed.scheduler - INFO - Close client connection: Client-6ea5e2a7-399f-11f1-8001-daa6852bd5a7
2026-04-16 14:21:08,639 - distributed.scheduler - INFO - Remove client Client-6ea5e2a7-399f-11f1-8001-daa6852bd5a7
2026-04-16 14:21:08,638 - distributed.scheduler - INFO - Remove client Client-6ea5e2a7-399f-11f1-8001-daa6852bd5a7
2026-04-16 14:20:34,445 - distributed.scheduler - INFO - Receive client connection: Client-6ea5e2a7-399f-11f1-8001-daa6852bd5a7
2026-04-16 14:13:00,036 - distributed.scheduler - INFO - Close client connection: Client-4e2d4317-399e-11f1-8001-daa6852bd5a7
2026-04-16 14:13:00,036 - distributed.scheduler - INFO - Remove client Client-4e2d4317-399e-11f1-8001-daa6852bd5a7
2026-04-16 14:13:00,035 - distributed.scheduler - INFO - Remove client Client-4e2d4317-399e-11f1-8001-daa6852bd5a7
2026-04-16 14:12:30,468 - distributed.scheduler - INFO - Receive client connection: Client-4e2d4317-399e-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,550 - distributed.scheduler - INFO - Close client connection: Client-8f203fff-3975-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,550 - distributed.scheduler - INFO - Close client connection: Client-7beeae4a-3974-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,550 - distributed.scheduler - INFO - Close client connection: Client-f5a4c551-3976-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,550 - distributed.scheduler - INFO - Close client connection: Client-f634403f-397d-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,549 - distributed.scheduler - INFO - Close client connection: Client-5c249391-3978-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,549 - distributed.scheduler - INFO - Close client connection: Client-c2a6c757-3979-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,549 - distributed.scheduler - INFO - Close client connection: Client-2926793f-397b-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,549 - distributed.scheduler - INFO - Close client connection: Client-d5b27488-397c-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,549 - distributed.scheduler - INFO - Close client connection: Client-38d28a38-397f-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,549 - distributed.scheduler - INFO - Close client connection: Client-9fd36f13-3987-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,549 - distributed.scheduler - INFO - Close client connection: Client-9f566bcb-3980-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,549 - distributed.scheduler - INFO - Close client connection: Client-05d44d98-3982-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,548 - distributed.scheduler - INFO - Close client connection: Client-6c5ba8f2-3983-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,548 - distributed.scheduler - INFO - Close client connection: Client-237b41bf-3985-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,548 - distributed.scheduler - INFO - Close client connection: Client-3957318f-3986-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,548 - distributed.scheduler - INFO - Close client connection: Client-065280d0-3989-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,547 - distributed.scheduler - INFO - Close client connection: Client-6ccd12d1-398a-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,547 - distributed.scheduler - INFO - Close client connection: Client-d353ccc5-398b-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,547 - distributed.scheduler - INFO - Close client connection: Client-6d6781cc-3991-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,547 - distributed.scheduler - INFO - Close client connection: Client-a05b5c33-398e-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,547 - distributed.scheduler - INFO - Close client connection: Client-91a0a94d-398d-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,547 - distributed.scheduler - INFO - Close client connection: Client-06deded7-3990-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,546 - distributed.scheduler - INFO - Close client connection: Client-d3ec666d-3992-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,546 - distributed.scheduler - INFO - Close client connection: Client-3a707a96-3994-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,546 - distributed.scheduler - INFO - Close client connection: Client-6e0a4bd3-3998-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,546 - distributed.scheduler - INFO - Close client connection: Client-07878b20-3997-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,546 - distributed.scheduler - INFO - Close client connection: Client-e49f5a58-3995-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,546 - distributed.scheduler - INFO - Close client connection: Client-d4961389-3999-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,546 - distributed.scheduler - INFO - Close client connection: Client-3b19901a-399b-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,543 - distributed.scheduler - INFO - Remove client Client-8f203fff-3975-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,543 - distributed.scheduler - INFO - Remove client Client-7beeae4a-3974-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,543 - distributed.scheduler - INFO - Remove client Client-f5a4c551-3976-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,542 - distributed.scheduler - INFO - Remove client Client-f634403f-397d-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,542 - distributed.scheduler - INFO - Remove client Client-5c249391-3978-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,542 - distributed.scheduler - INFO - Remove client Client-c2a6c757-3979-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,542 - distributed.scheduler - INFO - Remove client Client-2926793f-397b-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,541 - distributed.scheduler - INFO - Remove client Client-d5b27488-397c-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,541 - distributed.scheduler - INFO - Remove client Client-38d28a38-397f-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,541 - distributed.scheduler - INFO - Remove client Client-9fd36f13-3987-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,540 - distributed.scheduler - INFO - Remove client Client-9f566bcb-3980-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,540 - distributed.scheduler - INFO - Remove client Client-05d44d98-3982-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,535 - distributed.scheduler - INFO - Remove client Client-6c5ba8f2-3983-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,535 - distributed.scheduler - INFO - Remove client Client-237b41bf-3985-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,534 - distributed.scheduler - INFO - Remove client Client-3957318f-3986-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,533 - distributed.scheduler - INFO - Remove client Client-065280d0-3989-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,533 - distributed.scheduler - INFO - Remove client Client-6ccd12d1-398a-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,533 - distributed.scheduler - INFO - Remove client Client-d353ccc5-398b-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,532 - distributed.scheduler - INFO - Remove client Client-6d6781cc-3991-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,531 - distributed.scheduler - INFO - Remove client Client-a05b5c33-398e-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,531 - distributed.scheduler - INFO - Remove client Client-91a0a94d-398d-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,530 - distributed.scheduler - INFO - Remove client Client-06deded7-3990-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,530 - distributed.scheduler - INFO - Remove client Client-d3ec666d-3992-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,529 - distributed.scheduler - INFO - Remove client Client-3a707a96-3994-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,528 - distributed.scheduler - INFO - Remove client Client-6e0a4bd3-3998-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,527 - distributed.scheduler - INFO - Remove client Client-07878b20-3997-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,526 - distributed.scheduler - INFO - Remove client Client-e49f5a58-3995-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,526 - distributed.scheduler - INFO - Remove client Client-d4961389-3999-11f1-8001-daa6852bd5a7
2026-04-16 14:01:40,525 - distributed.scheduler - INFO - Remove client Client-3b19901a-399b-11f1-8001-daa6852bd5a7
2026-04-16 14:01:39,642 - distributed.scheduler - INFO - Close client connection: Client-a19a9f12-399c-11f1-8001-daa6852bd5a7
2026-04-16 14:01:39,641 - distributed.scheduler - INFO - Remove client Client-a19a9f12-399c-11f1-8001-daa6852bd5a7
2026-04-16 14:01:39,640 - distributed.scheduler - INFO - Remove client Client-a19a9f12-399c-11f1-8001-daa6852bd5a7
2026-04-16 14:00:31,440 - distributed.scheduler - INFO - Receive client connection: Client-a19a9f12-399c-11f1-8001-daa6852bd5a7
2026-04-16 13:50:29,961 - distributed.scheduler - INFO - Receive client connection: Client-3b19901a-399b-11f1-8001-daa6852bd5a7
2026-04-16 13:40:28,488 - distributed.scheduler - INFO - Receive client connection: Client-d4961389-3999-11f1-8001-daa6852bd5a7
2026-04-16 13:30:26,943 - distributed.scheduler - INFO - Receive client connection: Client-6e0a4bd3-3998-11f1-8001-daa6852bd5a7
2026-04-16 13:20:25,466 - distributed.scheduler - INFO - Receive client connection: Client-07878b20-3997-11f1-8001-daa6852bd5a7
2026-04-16 13:12:17,393 - distributed.scheduler - INFO - Receive client connection: Client-e49f5a58-3995-11f1-8001-daa6852bd5a7
2026-04-16 13:00:22,380 - distributed.scheduler - INFO - Receive client connection: Client-3a707a96-3994-11f1-8001-daa6852bd5a7
2026-04-16 12:50:20,885 - distributed.scheduler - INFO - Receive client connection: Client-d3ec666d-3992-11f1-8001-daa6852bd5a7
2026-04-16 12:40:19,394 - distributed.scheduler - INFO - Receive client connection: Client-6d6781cc-3991-11f1-8001-daa6852bd5a7
2026-04-16 12:33:09,064 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:34163
2026-04-16 12:33:09,061 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:34163', status: closed, memory: 0, processing: 0>
2026-04-16 12:33:08,222 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:39803', status: closed, memory: 0, processing: 0>
2026-04-16 12:32:57,914 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:42485 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s
2026-04-16 12:32:57,858 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:46541 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:57,837 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:57,823 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:57,809 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:57,809 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38347 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:57,801 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:57,801 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:41657 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:57,800 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:57,800 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:57,800 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:57,782 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:57,781 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:57,776 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:42485 after 30 s')
2026-04-16 12:32:28,586 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:39803
2026-04-16 12:32:28,584 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:39803', status: closed, memory: 0, processing: 0>
2026-04-16 12:32:27,677 - distributed.scheduler - INFO - Close client connection: Client-worker-4d6028a2-3990-11f1-81a8-f28f846166bd
2026-04-16 12:32:27,654 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:42485', status: closed, memory: 0, processing: 0>
2026-04-16 12:32:27,652 - distributed.scheduler - INFO - Remove client Client-worker-4d6028a2-3990-11f1-81a8-f28f846166bd
2026-04-16 12:32:16,160 - distributed.scheduler - INFO - Receive client connection: Client-worker-4d6028a2-3990-11f1-81a8-f28f846166bd
2026-04-16 12:32:13,073 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:41835 failed: OSError: Timed out trying to connect to tcp://172.21.159.197:41835 after 30 s
2026-04-16 12:32:13,063 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.159.197:41835 after 30 s')
2026-04-16 12:31:43,393 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:42485
2026-04-16 12:31:43,392 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:42485', status: closed, memory: 0, processing: 0>
2026-04-16 12:31:42,258 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:41835', status: closed, memory: 0, processing: 0>
2026-04-16 12:30:57,766 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], run-b46339f6a94a548c1c22b461402d13ca NoneType: None
2026-04-16 12:30:57,765 - distributed.scheduler - ERROR - Couldn't gather keys {'run-b46339f6a94a548c1c22b461402d13ca': []} state: ['processing'] workers: [] NoneType: None
2026-04-16 12:30:57,050 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.159.197:41835
2026-04-16 12:30:57,049 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.159.197:41835', status: closed, memory: 0, processing: 0>
2026-04-16 12:30:56,118 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.159.197:35361', status: closed, memory: 0, processing: 0>
2026-04-16 12:30:56,117 - distributed.scheduler - INFO - Close client connection: Client-worker-04426882-37c3-11f1-8074-f28f846166bd
2026-04-16 12:30:56,115 - distributed.scheduler - INFO - Remove client Client-worker-04426882-37c3-11f1-8074-f28f846166bd
2026-04-16 12:30:17,877 - distributed.scheduler - INFO - Receive client connection: Client-06deded7-3990-11f1-8001-daa6852bd5a7
2026-04-16 12:20:16,433 - distributed.scheduler - INFO - Receive client connection: Client-a05b5c33-398e-11f1-8001-daa6852bd5a7
2026-04-16 12:12:42,177 - distributed.scheduler - INFO - Receive client connection: Client-91a0a94d-398d-11f1-8001-daa6852bd5a7
2026-04-16 12:00:13,412 - distributed.scheduler - INFO - Receive client connection: Client-d353ccc5-398b-11f1-8001-daa6852bd5a7
2026-04-16 11:50:11,901 - distributed.scheduler - INFO - Receive client connection: Client-6ccd12d1-398a-11f1-8001-daa6852bd5a7
2026-04-16 11:40:10,479 - distributed.scheduler - INFO - Receive client connection: Client-065280d0-3989-11f1-8001-daa6852bd5a7
2026-04-16 11:30:09,018 - distributed.scheduler - INFO - Receive client connection: Client-9fd36f13-3987-11f1-8001-daa6852bd5a7
2026-04-16 11:20:07,585 - distributed.scheduler - INFO - Receive client connection: Client-3957318f-3986-11f1-8001-daa6852bd5a7
2026-04-16 11:12:21,408 - distributed.scheduler - INFO - Receive client connection: Client-237b41bf-3985-11f1-8001-daa6852bd5a7
2026-04-16 11:00:04,686 - distributed.scheduler - INFO - Receive client connection: Client-6c5ba8f2-3983-11f1-8001-daa6852bd5a7
2026-04-16 10:50:03,172 - distributed.scheduler - INFO - Receive client connection: Client-05d44d98-3982-11f1-8001-daa6852bd5a7
2026-04-16 10:40:01,725 - distributed.scheduler - INFO - Receive client connection: Client-9f566bcb-3980-11f1-8001-daa6852bd5a7
2026-04-16 10:30:00,241 - distributed.scheduler - INFO - Receive client connection: Client-38d28a38-397f-11f1-8001-daa6852bd5a7
2026-04-16 10:20:58,969 - distributed.scheduler - INFO - Receive client connection: Client-f634403f-397d-11f1-8001-daa6852bd5a7
2026-04-16 10:12:54,935 - distributed.scheduler - INFO - Receive client connection: Client-d5b27488-397c-11f1-8001-daa6852bd5a7
2026-04-16 10:00:55,955 - distributed.scheduler - INFO - Receive client connection: Client-2926793f-397b-11f1-8001-daa6852bd5a7
2026-04-16 09:50:54,493 - distributed.scheduler - INFO - Receive client connection: Client-c2a6c757-3979-11f1-8001-daa6852bd5a7
2026-04-16 09:40:53,017 - distributed.scheduler - INFO - Receive client connection: Client-5c249391-3978-11f1-8001-daa6852bd5a7
2026-04-16 09:30:51,554 - distributed.scheduler - INFO - Receive client connection: Client-f5a4c551-3976-11f1-8001-daa6852bd5a7
2026-04-16 09:20:50,060 - distributed.scheduler - INFO - Receive client connection: Client-8f203fff-3975-11f1-8001-daa6852bd5a7
2026-04-16 09:13:08,373 - distributed.scheduler - INFO - Receive client connection: Client-7beeae4a-3974-11f1-8001-daa6852bd5a7
2026-04-16 09:02:06,351 - distributed.scheduler - INFO - Close client connection: Client-c22979e4-3972-11f1-8001-daa6852bd5a7
2026-04-16 09:02:06,351 - distributed.scheduler - INFO - Remove client Client-c22979e4-3972-11f1-8001-daa6852bd5a7
2026-04-16 09:02:06,349 - distributed.scheduler - INFO - Remove client Client-c22979e4-3972-11f1-8001-daa6852bd5a7
2026-04-16 09:00:47,200 - distributed.scheduler - INFO - Receive client connection: Client-c22979e4-3972-11f1-8001-daa6852bd5a7
2026-04-16 08:51:27,571 - distributed.scheduler - INFO - Close client connection: Client-5ba2fe20-3971-11f1-8001-daa6852bd5a7
2026-04-16 08:51:27,569 - distributed.scheduler - INFO - Remove client Client-5ba2fe20-3971-11f1-8001-daa6852bd5a7
2026-04-16 08:51:27,568 - distributed.scheduler - INFO - Remove client Client-5ba2fe20-3971-11f1-8001-daa6852bd5a7
2026-04-16 08:50:45,696 - distributed.scheduler - INFO - Receive client connection: Client-5ba2fe20-3971-11f1-8001-daa6852bd5a7
2026-04-16 08:41:25,838 - distributed.scheduler - INFO - Close client connection: Client-f5185282-396f-11f1-8001-daa6852bd5a7
2026-04-16 08:41:25,836 - distributed.scheduler - INFO - Remove client Client-f5185282-396f-11f1-8001-daa6852bd5a7
2026-04-16 08:41:25,835 - distributed.scheduler - INFO - Remove client Client-f5185282-396f-11f1-8001-daa6852bd5a7
2026-04-16 08:40:44,166 - distributed.scheduler - INFO - Receive client connection: Client-f5185282-396f-11f1-8001-daa6852bd5a7
2026-04-16 08:31:57,303 - distributed.scheduler - INFO - Close client connection: Client-8e8ee533-396e-11f1-8001-daa6852bd5a7
2026-04-16 08:31:57,303 - distributed.scheduler - INFO - Remove client Client-8e8ee533-396e-11f1-8001-daa6852bd5a7
2026-04-16 08:31:57,301 - distributed.scheduler - INFO - Remove client Client-8e8ee533-396e-11f1-8001-daa6852bd5a7
2026-04-16 08:30:42,640 - distributed.scheduler - INFO - Receive client connection: Client-8e8ee533-396e-11f1-8001-daa6852bd5a7
2026-04-16 08:21:21,144 - distributed.scheduler - INFO - Close client connection: Client-2811d00b-396d-11f1-8001-daa6852bd5a7
2026-04-16 08:21:21,143 - distributed.scheduler - INFO - Remove client Client-2811d00b-396d-11f1-8001-daa6852bd5a7
2026-04-16 08:21:21,141 - distributed.scheduler - INFO - Remove client Client-2811d00b-396d-11f1-8001-daa6852bd5a7
2026-04-16 08:20:41,194 - distributed.scheduler - INFO - Receive client connection: Client-2811d00b-396d-11f1-8001-daa6852bd5a7
2026-04-16 08:13:20,732 - distributed.scheduler - INFO - Close client connection: Client-f48e6f5e-3968-11f1-8001-daa6852bd5a7
2026-04-16 08:13:20,732 - distributed.scheduler - INFO - Close client connection: Client-5b11e8d3-396a-11f1-8001-daa6852bd5a7
2026-04-16 08:13:20,731 - distributed.scheduler - INFO - Remove client Client-f48e6f5e-3968-11f1-8001-daa6852bd5a7
2026-04-16 08:13:20,731 - distributed.scheduler - INFO - Remove client Client-5b11e8d3-396a-11f1-8001-daa6852bd5a7
2026-04-16 08:13:20,039 - distributed.scheduler - INFO - Close client connection: Client-0bacbc14-396c-11f1-8001-daa6852bd5a7
2026-04-16 08:13:20,037 - distributed.scheduler - INFO - Remove client Client-0bacbc14-396c-11f1-8001-daa6852bd5a7
2026-04-16 08:13:20,035 - distributed.scheduler - INFO - Remove client Client-0bacbc14-396c-11f1-8001-daa6852bd5a7
2026-04-16 08:12:44,049 - distributed.scheduler - INFO - Receive client connection: Client-0bacbc14-396c-11f1-8001-daa6852bd5a7
2026-04-16 08:00:38,259 - distributed.scheduler - INFO - Receive client connection: Client-5b11e8d3-396a-11f1-8001-daa6852bd5a7
2026-04-16 07:50:36,783 - distributed.scheduler - INFO - Receive client connection: Client-f48e6f5e-3968-11f1-8001-daa6852bd5a7
2026-04-16 07:41:14,253 - distributed.scheduler - INFO - Close client connection: Client-8e098f76-3967-11f1-8001-daa6852bd5a7
2026-04-16 07:41:14,246 - distributed.scheduler - INFO - Remove client Client-8e098f76-3967-11f1-8001-daa6852bd5a7
2026-04-16 07:41:14,244 - distributed.scheduler - INFO - Remove client Client-8e098f76-3967-11f1-8001-daa6852bd5a7
2026-04-16 07:40:35,292 - distributed.scheduler - INFO - Receive client connection: Client-8e098f76-3967-11f1-8001-daa6852bd5a7
2026-04-16 07:31:45,922 - distributed.scheduler - INFO - Close client connection: Client-27867605-3966-11f1-8001-daa6852bd5a7
2026-04-16 07:31:45,920 - distributed.scheduler - INFO - Remove client Client-27867605-3966-11f1-8001-daa6852bd5a7
2026-04-16 07:31:45,917 - distributed.scheduler - INFO - Remove client Client-27867605-3966-11f1-8001-daa6852bd5a7
2026-04-16 07:30:33,807 - distributed.scheduler - INFO - Receive client connection: Client-27867605-3966-11f1-8001-daa6852bd5a7
2026-04-16 07:21:14,936 - distributed.scheduler - INFO - Close client connection: Client-c105f446-3964-11f1-8001-daa6852bd5a7
2026-04-16 07:21:14,935 - distributed.scheduler - INFO - Remove client Client-c105f446-3964-11f1-8001-daa6852bd5a7
2026-04-16 07:21:14,933 - distributed.scheduler - INFO - Remove client Client-c105f446-3964-11f1-8001-daa6852bd5a7
2026-04-16 07:20:32,345 - distributed.scheduler - INFO - Receive client connection: Client-c105f446-3964-11f1-8001-daa6852bd5a7
2026-04-16 07:13:15,755 - distributed.scheduler - INFO - Close client connection: Client-a6dacdfe-3963-11f1-8001-daa6852bd5a7
2026-04-16 07:13:15,748 - distributed.scheduler - INFO - Remove client Client-a6dacdfe-3963-11f1-8001-daa6852bd5a7
2026-04-16 07:13:15,746 - distributed.scheduler - INFO - Remove client Client-a6dacdfe-3963-11f1-8001-daa6852bd5a7
2026-04-16 07:12:38,935 - distributed.scheduler - INFO - Receive client connection: Client-a6dacdfe-3963-11f1-8001-daa6852bd5a7
2026-04-16 07:01:48,093 - distributed.scheduler - INFO - Close client connection: Client-f3feefc7-3961-11f1-8001-daa6852bd5a7
2026-04-16 07:01:48,091 - distributed.scheduler - INFO - Remove client Client-f3feefc7-3961-11f1-8001-daa6852bd5a7
2026-04-16 07:01:48,089 - distributed.scheduler - INFO - Remove client Client-f3feefc7-3961-11f1-8001-daa6852bd5a7
2026-04-16 07:00:29,368 - distributed.scheduler - INFO - Receive client connection: Client-f3feefc7-3961-11f1-8001-daa6852bd5a7
2026-04-16 06:51:08,201 - distributed.scheduler - INFO - Close client connection: Client-8d826ed9-3960-11f1-8001-daa6852bd5a7
2026-04-16 06:51:08,201 - distributed.scheduler - INFO - Remove client Client-8d826ed9-3960-11f1-8001-daa6852bd5a7
2026-04-16 06:51:08,200 - distributed.scheduler - INFO - Remove client Client-8d826ed9-3960-11f1-8001-daa6852bd5a7
2026-04-16 06:50:27,926 - distributed.scheduler - INFO - Receive client connection: Client-8d826ed9-3960-11f1-8001-daa6852bd5a7
2026-04-16 06:41:07,759 - distributed.scheduler - INFO - Close client connection: Client-26fca818-395f-11f1-8001-daa6852bd5a7
2026-04-16 06:41:07,759 - distributed.scheduler - INFO - Remove client Client-26fca818-395f-11f1-8001-daa6852bd5a7
2026-04-16 06:41:07,757 - distributed.scheduler - INFO - Remove client Client-26fca818-395f-11f1-8001-daa6852bd5a7
2026-04-16 06:40:26,432 - distributed.scheduler - INFO - Receive client connection: Client-26fca818-395f-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,648 - distributed.scheduler - INFO - Close client connection: Client-f2e63e2b-3953-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,647 - distributed.scheduler - INFO - Close client connection: Client-bc940c9f-3952-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,647 - distributed.scheduler - INFO - Close client connection: Client-596f27f4-3955-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,647 - distributed.scheduler - INFO - Close client connection: Client-bff40d12-3956-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,647 - distributed.scheduler - INFO - Close client connection: Client-2aa4ea64-395b-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,647 - distributed.scheduler - INFO - Close client connection: Client-2677582d-3958-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,646 - distributed.scheduler - INFO - Close client connection: Client-8cfabf9e-3959-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,646 - distributed.scheduler - INFO - Close client connection: Client-59f587db-395c-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,645 - distributed.scheduler - INFO - Remove client Client-f2e63e2b-3953-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,645 - distributed.scheduler - INFO - Remove client Client-bc940c9f-3952-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,644 - distributed.scheduler - INFO - Remove client Client-596f27f4-3955-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,644 - distributed.scheduler - INFO - Remove client Client-bff40d12-3956-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,644 - distributed.scheduler - INFO - Remove client Client-2aa4ea64-395b-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,643 - distributed.scheduler - INFO - Remove client Client-2677582d-3958-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,643 - distributed.scheduler - INFO - Remove client Client-8cfabf9e-3959-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,641 - distributed.scheduler - INFO - Remove client Client-59f587db-395c-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,146 - distributed.scheduler - INFO - Close client connection: Client-c07ca63d-395d-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,145 - distributed.scheduler - INFO - Remove client Client-c07ca63d-395d-11f1-8001-daa6852bd5a7
2026-04-16 06:31:38,145 - distributed.scheduler - INFO - Remove client Client-c07ca63d-395d-11f1-8001-daa6852bd5a7
2026-04-16 06:30:24,951 - distributed.scheduler - INFO - Receive client connection: Client-c07ca63d-395d-11f1-8001-daa6852bd5a7
2026-04-16 06:20:23,445 - distributed.scheduler - INFO - Receive client connection: Client-59f587db-395c-11f1-8001-daa6852bd5a7
2026-04-16 06:11:54,565 - distributed.scheduler - INFO - Receive client connection: Client-2aa4ea64-395b-11f1-8001-daa6852bd5a7
2026-04-16 06:00:20,553 - distributed.scheduler - INFO - Receive client connection: Client-8cfabf9e-3959-11f1-8001-daa6852bd5a7
2026-04-16 05:50:19,064 - distributed.scheduler - INFO - Receive client connection: Client-2677582d-3958-11f1-8001-daa6852bd5a7
2026-04-16 05:40:17,581 - distributed.scheduler - INFO - Receive client connection: Client-bff40d12-3956-11f1-8001-daa6852bd5a7
2026-04-16 05:30:16,087 - distributed.scheduler - INFO - Receive client connection: Client-596f27f4-3955-11f1-8001-daa6852bd5a7
2026-04-16 05:20:14,563 - distributed.scheduler - INFO - Receive client connection: Client-f2e63e2b-3953-11f1-8001-daa6852bd5a7
2026-04-16 05:11:33,938 - distributed.scheduler - INFO - Receive client connection: Client-bc940c9f-3952-11f1-8001-daa6852bd5a7
2026-04-16 05:01:21,753 - distributed.scheduler - INFO - Close client connection: Client-25f0afef-3951-11f1-8001-daa6852bd5a7
2026-04-16 05:01:21,747 - distributed.scheduler - INFO - Remove client Client-25f0afef-3951-11f1-8001-daa6852bd5a7
2026-04-16 05:01:21,745 - distributed.scheduler - INFO - Remove client Client-25f0afef-3951-11f1-8001-daa6852bd5a7
2026-04-16 05:00:11,715 - distributed.scheduler - INFO - Receive client connection: Client-25f0afef-3951-11f1-8001-daa6852bd5a7
2026-04-16 04:50:58,294 - distributed.scheduler - INFO - Close client connection: Client-bf70526f-394f-11f1-8001-daa6852bd5a7
2026-04-16 04:50:58,293 - distributed.scheduler - INFO - Remove client Client-bf70526f-394f-11f1-8001-daa6852bd5a7
2026-04-16 04:50:58,292 - distributed.scheduler - INFO - Remove client Client-bf70526f-394f-11f1-8001-daa6852bd5a7
2026-04-16 04:50:10,267 - distributed.scheduler - INFO - Receive client connection: Client-bf70526f-394f-11f1-8001-daa6852bd5a7
2026-04-16 04:40:50,924 - distributed.scheduler - INFO - Close client connection: Client-58fc269d-394e-11f1-8001-daa6852bd5a7
2026-04-16 04:40:50,923 - distributed.scheduler - INFO - Remove client Client-58fc269d-394e-11f1-8001-daa6852bd5a7
2026-04-16 04:40:50,921 - distributed.scheduler - INFO - Remove client Client-58fc269d-394e-11f1-8001-daa6852bd5a7
2026-04-16 04:40:08,864 - distributed.scheduler - INFO - Receive client connection: Client-58fc269d-394e-11f1-8001-daa6852bd5a7
2026-04-16 04:31:21,982 - distributed.scheduler - INFO - Close client connection: Client-f285b142-394c-11f1-8001-daa6852bd5a7
2026-04-16 04:31:21,982 - distributed.scheduler - INFO - Remove client Client-f285b142-394c-11f1-8001-daa6852bd5a7
2026-04-16 04:31:21,980 - distributed.scheduler - INFO - Remove client Client-f285b142-394c-11f1-8001-daa6852bd5a7
2026-04-16 04:30:14,081 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('len-agg-c1b9eca4e1bb93d6e5855f679451849a', 0) NoneType: None
2026-04-16 04:30:14,081 - distributed.scheduler - ERROR - Couldn't gather keys {"('len-agg-c1b9eca4e1bb93d6e5855f679451849a', 0)": []} state: ['waiting'] workers: [] NoneType: None
2026-04-16 04:30:07,464 - distributed.scheduler - INFO - Receive client connection: Client-f285b142-394c-11f1-8001-daa6852bd5a7
2026-04-16 04:20:41,857 - distributed.scheduler - INFO - Close client connection: Client-8c176acb-394b-11f1-8001-daa6852bd5a7
2026-04-16 04:20:41,857 - distributed.scheduler - INFO - Remove client Client-8c176acb-394b-11f1-8001-daa6852bd5a7
2026-04-16 04:20:41,856 - distributed.scheduler - INFO - Remove client Client-8c176acb-394b-11f1-8001-daa6852bd5a7
2026-04-16 04:20:06,114 - distributed.scheduler - INFO - Receive client connection: Client-8c176acb-394b-11f1-8001-daa6852bd5a7
2026-04-16 04:12:23,134 - distributed.scheduler - INFO - Close client connection: Client-6538560b-394a-11f1-8001-daa6852bd5a7
2026-04-16 04:12:23,133 - distributed.scheduler - INFO - Remove client Client-6538560b-394a-11f1-8001-daa6852bd5a7
2026-04-16 04:12:23,132 - distributed.scheduler - INFO - Remove client Client-6538560b-394a-11f1-8001-daa6852bd5a7
2026-04-16 04:11:51,404 - distributed.scheduler - INFO - Receive client connection: Client-6538560b-394a-11f1-8001-daa6852bd5a7
2026-04-16 04:01:11,155 - distributed.scheduler - INFO - Close client connection: Client-bf15cdc5-3948-11f1-8001-daa6852bd5a7
2026-04-16 04:01:11,154 - distributed.scheduler - INFO - Remove client Client-bf15cdc5-3948-11f1-8001-daa6852bd5a7
2026-04-16 04:01:11,153 - distributed.scheduler - INFO - Remove client Client-bf15cdc5-3948-11f1-8001-daa6852bd5a7
2026-04-16 04:00:03,179 - distributed.scheduler - INFO - Receive client connection: Client-bf15cdc5-3948-11f1-8001-daa6852bd5a7
2026-04-16 03:50:37,046 - distributed.scheduler - INFO - Close client connection: Client-588feed0-3947-11f1-8001-daa6852bd5a7
2026-04-16 03:50:37,046 - distributed.scheduler - INFO - Remove client Client-588feed0-3947-11f1-8001-daa6852bd5a7
2026-04-16 03:50:37,045 - distributed.scheduler - INFO - Remove client Client-588feed0-3947-11f1-8001-daa6852bd5a7
2026-04-16 03:50:01,706 - distributed.scheduler - INFO - Receive client connection: Client-588feed0-3947-11f1-8001-daa6852bd5a7
2026-04-16 03:40:36,885 - distributed.scheduler - INFO - Close client connection: Client-f20d3258-3945-11f1-8001-daa6852bd5a7
2026-04-16 03:40:36,884 - distributed.scheduler - INFO - Remove client Client-f20d3258-3945-11f1-8001-daa6852bd5a7
2026-04-16 03:40:36,883 - distributed.scheduler - INFO - Remove client Client-f20d3258-3945-11f1-8001-daa6852bd5a7
2026-04-16 03:40:00,197 - distributed.scheduler - INFO - Receive client connection: Client-f20d3258-3945-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,549 - distributed.scheduler - INFO - Close client connection: Client-ceb231d7-3905-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,549 - distributed.scheduler - INFO - Close client connection: Client-9bc36257-3908-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,549 - distributed.scheduler - INFO - Close client connection: Client-752c0104-3907-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,549 - distributed.scheduler - INFO - Close client connection: Client-0249808c-390a-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,549 - distributed.scheduler - INFO - Close client connection: Client-68cf02ac-390b-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,549 - distributed.scheduler - INFO - Close client connection: Client-ab7597b8-390c-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,549 - distributed.scheduler - INFO - Close client connection: Client-11f358e4-390e-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,549 - distributed.scheduler - INFO - Close client connection: Client-aea49c2c-390f-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,549 - distributed.scheduler - INFO - Close client connection: Client-df085d5c-3910-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,549 - distributed.scheduler - INFO - Close client connection: Client-458ec4fa-3912-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,548 - distributed.scheduler - INFO - Close client connection: Client-ac0dffe4-3913-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,548 - distributed.scheduler - INFO - Close client connection: Client-128821a7-3915-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,548 - distributed.scheduler - INFO - Close client connection: Client-45fa97b4-3919-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,548 - distributed.scheduler - INFO - Close client connection: Client-78f9a6ab-3916-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,548 - distributed.scheduler - INFO - Close client connection: Client-14928b47-3918-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,548 - distributed.scheduler - INFO - Close client connection: Client-ac7f9628-391a-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,548 - distributed.scheduler - INFO - Close client connection: Client-1308fe4f-391c-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-7982df79-391d-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-e0009d8a-391e-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-8eb9a9a4-3920-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-acffc45e-3921-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-46ff9823-3927-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-1381b056-3923-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-79f67558-3924-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-e07f4fb8-3925-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-dc517b60-3928-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-13fa6e4c-392a-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-7a731a5a-392b-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-e0f1af6f-392c-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,547 - distributed.scheduler - INFO - Close client connection: Client-47744766-392e-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,546 - distributed.scheduler - INFO - Close client connection: Client-ae001735-392f-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,546 - distributed.scheduler - INFO - Close client connection: Client-7af943a3-3932-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,546 - distributed.scheduler - INFO - Close client connection: Client-58fded7b-3931-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,546 - distributed.scheduler - INFO - Close client connection: Client-e17e9720-3933-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,546 - distributed.scheduler - INFO - Close client connection: Client-47eccbe9-3935-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,546 - distributed.scheduler - INFO - Close client connection: Client-ae661fb2-3936-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,546 - distributed.scheduler - INFO - Close client connection: Client-14ee7098-3938-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,546 - distributed.scheduler - INFO - Close client connection: Client-ba282bf6-3939-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,546 - distributed.scheduler - INFO - Close client connection: Client-e1f6a28d-393a-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,546 - distributed.scheduler - INFO - Close client connection: Client-4873295a-393c-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,546 - distributed.scheduler - INFO - Close client connection: Client-aef2b7da-393d-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,546 - distributed.scheduler - INFO - Close client connection: Client-1577644c-393f-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,545 - distributed.scheduler - INFO - Close client connection: Client-7bf637a2-3940-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,543 - distributed.scheduler - INFO - Close client connection: Client-48f0682b-3943-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,543 - distributed.scheduler - INFO - Close client connection: Client-28ed55f9-3942-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,542 - distributed.scheduler - INFO - Remove client Client-ceb231d7-3905-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,542 - distributed.scheduler - INFO - Remove client Client-9bc36257-3908-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,541 - distributed.scheduler - INFO - Remove client Client-752c0104-3907-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,541 - distributed.scheduler - INFO - Remove client Client-0249808c-390a-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,541 - distributed.scheduler - INFO - Remove client Client-68cf02ac-390b-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,540 - distributed.scheduler - INFO - Remove client Client-ab7597b8-390c-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,540 - distributed.scheduler - INFO - Remove client Client-11f358e4-390e-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,540 - distributed.scheduler - INFO - Remove client Client-aea49c2c-390f-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,539 - distributed.scheduler - INFO - Remove client Client-df085d5c-3910-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,539 - distributed.scheduler - INFO - Remove client Client-458ec4fa-3912-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,539 - distributed.scheduler - INFO - Remove client Client-ac0dffe4-3913-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,539 - distributed.scheduler - INFO - Remove client Client-128821a7-3915-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,538 - distributed.scheduler - INFO - Remove client Client-45fa97b4-3919-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,538 - distributed.scheduler - INFO - Remove client Client-78f9a6ab-3916-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,533 - distributed.scheduler - INFO - Remove client Client-14928b47-3918-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,533 - distributed.scheduler - INFO - Remove client Client-ac7f9628-391a-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,533 - distributed.scheduler - INFO - Remove client Client-1308fe4f-391c-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,532 - distributed.scheduler - INFO - Remove client Client-7982df79-391d-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,532 - distributed.scheduler - INFO - Remove client Client-e0009d8a-391e-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,531 - distributed.scheduler - INFO - Remove client Client-8eb9a9a4-3920-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,531 - distributed.scheduler - INFO - Remove client Client-acffc45e-3921-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,530 - distributed.scheduler - INFO - Remove client Client-46ff9823-3927-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,530 - distributed.scheduler - INFO - Remove client Client-1381b056-3923-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,530 - distributed.scheduler - INFO - Remove client Client-79f67558-3924-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,529 - distributed.scheduler - INFO - Remove client Client-e07f4fb8-3925-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,529 - distributed.scheduler - INFO - Remove client Client-dc517b60-3928-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,528 - distributed.scheduler - INFO - Remove client Client-13fa6e4c-392a-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,528 - distributed.scheduler - INFO - Remove client Client-7a731a5a-392b-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,527 - distributed.scheduler - INFO - Remove client Client-e0f1af6f-392c-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,527 - distributed.scheduler - INFO - Remove client Client-47744766-392e-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,527 - distributed.scheduler - INFO - Remove client Client-ae001735-392f-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,526 - distributed.scheduler - INFO - Remove client Client-7af943a3-3932-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,526 - distributed.scheduler - INFO - Remove client Client-58fded7b-3931-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,525 - distributed.scheduler - INFO - Remove client Client-e17e9720-3933-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,525 - distributed.scheduler - INFO - Remove client Client-47eccbe9-3935-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,525 - distributed.scheduler - INFO - Remove client Client-ae661fb2-3936-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,524 - distributed.scheduler - INFO - Remove client Client-14ee7098-3938-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,524 - distributed.scheduler - INFO - Remove client Client-ba282bf6-3939-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,523 - distributed.scheduler - INFO - Remove client Client-e1f6a28d-393a-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,522 - distributed.scheduler - INFO - Remove client Client-4873295a-393c-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,521 - distributed.scheduler - INFO - Remove client Client-aef2b7da-393d-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,521 - distributed.scheduler - INFO - Remove client Client-1577644c-393f-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,520 - distributed.scheduler - INFO - Remove client Client-7bf637a2-3940-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,509 - distributed.scheduler - INFO - Remove client Client-48f0682b-3943-11f1-8001-daa6852bd5a7
2026-04-16 03:32:11,508 - distributed.scheduler - INFO - Remove client Client-28ed55f9-3942-11f1-8001-daa6852bd5a7
2026-04-16 03:32:10,577 - distributed.scheduler - INFO - Close client connection: Client-af6ca4f5-3944-11f1-8001-daa6852bd5a7
2026-04-16 03:32:10,575 - distributed.scheduler - INFO - Remove client Client-af6ca4f5-3944-11f1-8001-daa6852bd5a7
2026-04-16 03:32:10,574 - distributed.scheduler - INFO - Remove client Client-af6ca4f5-3944-11f1-8001-daa6852bd5a7
2026-04-16 03:30:58,909 - distributed.scheduler - INFO - Receive client connection: Client-af6ca4f5-3944-11f1-8001-daa6852bd5a7
2026-04-16 03:20:57,471 - distributed.scheduler - INFO - Receive client connection: Client-48f0682b-3943-11f1-8001-daa6852bd5a7
2026-04-16 03:12:54,266 - distributed.scheduler - INFO - Receive client connection: Client-28ed55f9-3942-11f1-8001-daa6852bd5a7
2026-04-16 03:00:54,585 - distributed.scheduler - INFO - Receive client connection: Client-7bf637a2-3940-11f1-8001-daa6852bd5a7
2026-04-16 02:50:53,136 - distributed.scheduler - INFO - Receive client connection: Client-1577644c-393f-11f1-8001-daa6852bd5a7
2026-04-16 02:40:51,639 - distributed.scheduler - INFO - Receive client connection: Client-aef2b7da-393d-11f1-8001-daa6852bd5a7
2026-04-16 02:30:56,940 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('len-agg-d3e2971c85ed26029071b7f49b2d80e3', 0) NoneType: None
2026-04-16 02:30:56,940 - distributed.scheduler - ERROR - Couldn't gather keys {"('len-agg-d3e2971c85ed26029071b7f49b2d80e3', 0)": []} state: ['waiting'] workers: [] NoneType: None
2026-04-16 02:30:50,174 - distributed.scheduler - INFO - Receive client connection: Client-4873295a-393c-11f1-8001-daa6852bd5a7
2026-04-16 02:20:48,735 - distributed.scheduler - INFO - Receive client connection: Client-e1f6a28d-393a-11f1-8001-daa6852bd5a7
2026-04-16 02:12:32,447 - distributed.scheduler - INFO - Receive client connection: Client-ba282bf6-3939-11f1-8001-daa6852bd5a7
2026-04-16 02:00:45,764 - distributed.scheduler - INFO - Receive client connection: Client-14ee7098-3938-11f1-8001-daa6852bd5a7
2026-04-16 01:50:44,241 - distributed.scheduler - INFO - Receive client connection: Client-ae661fb2-3936-11f1-8001-daa6852bd5a7
2026-04-16 01:40:42,808 - distributed.scheduler - INFO - Receive client connection: Client-47eccbe9-3935-11f1-8001-daa6852bd5a7
2026-04-16 01:30:41,473 - distributed.scheduler - INFO - Receive client connection: Client-e17e9720-3933-11f1-8001-daa6852bd5a7
2026-04-16 01:20:39,974 - distributed.scheduler - INFO - Receive client connection: Client-7af943a3-3932-11f1-8001-daa6852bd5a7
2026-04-16 01:12:33,458 - distributed.scheduler - INFO - Receive client connection: Client-58fded7b-3931-11f1-8001-daa6852bd5a7
2026-04-16 01:00:37,091 - distributed.scheduler - INFO - Receive client connection: Client-ae001735-392f-11f1-8001-daa6852bd5a7
2026-04-16 00:50:35,546 - distributed.scheduler - INFO - Receive client connection: Client-47744766-392e-11f1-8001-daa6852bd5a7
2026-04-16 00:40:34,070 - distributed.scheduler - INFO - Receive client connection: Client-e0f1af6f-392c-11f1-8001-daa6852bd5a7
2026-04-16 00:30:32,616 - distributed.scheduler - INFO - Receive client connection: Client-7a731a5a-392b-11f1-8001-daa6852bd5a7
2026-04-16 00:20:31,198 - distributed.scheduler - INFO - Receive client connection: Client-13fa6e4c-392a-11f1-8001-daa6852bd5a7
2026-04-16 00:11:48,320 - distributed.scheduler - INFO - Receive client connection: Client-dc517b60-3928-11f1-8001-daa6852bd5a7
2026-04-16 00:00:28,306 - distributed.scheduler - INFO - Receive client connection: Client-46ff9823-3927-11f1-8001-daa6852bd5a7
2026-04-15 23:50:26,840 - distributed.scheduler - INFO - Receive client connection: Client-e07f4fb8-3925-11f1-8001-daa6852bd5a7
2026-04-15 23:40:25,320 - distributed.scheduler - INFO - Receive client connection: Client-79f67558-3924-11f1-8001-daa6852bd5a7
2026-04-15 23:30:23,934 - distributed.scheduler - INFO - Receive client connection: Client-1381b056-3923-11f1-8001-daa6852bd5a7
2026-04-15 23:20:22,456 - distributed.scheduler - INFO - Receive client connection: Client-acffc45e-3921-11f1-8001-daa6852bd5a7
2026-04-15 23:12:22,163 - distributed.scheduler - INFO - Receive client connection: Client-8eb9a9a4-3920-11f1-8001-daa6852bd5a7
2026-04-15 23:00:19,539 - distributed.scheduler - INFO - Receive client connection: Client-e0009d8a-391e-11f1-8001-daa6852bd5a7
2026-04-15 22:50:18,085 - distributed.scheduler - INFO - Receive client connection: Client-7982df79-391d-11f1-8001-daa6852bd5a7
2026-04-15 22:40:16,658 - distributed.scheduler - INFO - Receive client connection: Client-1308fe4f-391c-11f1-8001-daa6852bd5a7
2026-04-15 22:30:15,140 - distributed.scheduler - INFO - Receive client connection: Client-ac7f9628-391a-11f1-8001-daa6852bd5a7
2026-04-15 22:20:13,645 - distributed.scheduler - INFO - Receive client connection: Client-45fa97b4-3919-11f1-8001-daa6852bd5a7
2026-04-15 22:11:41,256 - distributed.scheduler - INFO - Receive client connection: Client-14928b47-3918-11f1-8001-daa6852bd5a7
2026-04-15 22:00:10,714 - distributed.scheduler - INFO - Receive client connection: Client-78f9a6ab-3916-11f1-8001-daa6852bd5a7
2026-04-15 21:50:09,339 - distributed.scheduler - INFO - Receive client connection: Client-128821a7-3915-11f1-8001-daa6852bd5a7
2026-04-15 21:40:07,917 - distributed.scheduler - INFO - Receive client connection: Client-ac0dffe4-3913-11f1-8001-daa6852bd5a7
2026-04-15 21:30:06,461 - distributed.scheduler - INFO - Receive client connection: Client-458ec4fa-3912-11f1-8001-daa6852bd5a7
2026-04-15 21:20:04,953 - distributed.scheduler - INFO - Receive client connection: Client-df085d5c-3910-11f1-8001-daa6852bd5a7
2026-04-15 21:11:34,270 - distributed.scheduler - INFO - Receive client connection: Client-aea49c2c-390f-11f1-8001-daa6852bd5a7
2026-04-15 21:00:01,891 - distributed.scheduler - INFO - Receive client connection: Client-11f358e4-390e-11f1-8001-daa6852bd5a7
2026-04-15 20:50:00,434 - distributed.scheduler - INFO - Receive client connection: Client-ab7597b8-390c-11f1-8001-daa6852bd5a7
2026-04-15 20:40:59,125 - distributed.scheduler - INFO - Receive client connection: Client-68cf02ac-390b-11f1-8001-daa6852bd5a7
2026-04-15 20:30:57,630 - distributed.scheduler - INFO - Receive client connection: Client-0249808c-390a-11f1-8001-daa6852bd5a7
2026-04-15 20:20:56,120 - distributed.scheduler - INFO - Receive client connection: Client-9bc36257-3908-11f1-8001-daa6852bd5a7
2026-04-15 20:12:41,872 - distributed.scheduler - INFO - Receive client connection: Client-752c0104-3907-11f1-8001-daa6852bd5a7
2026-04-15 20:00:53,099 - distributed.scheduler - INFO - Receive client connection: Client-ceb231d7-3905-11f1-8001-daa6852bd5a7
2026-04-15 19:51:22,669 - distributed.scheduler - INFO - Close client connection: Client-683589cd-3904-11f1-8001-daa6852bd5a7
2026-04-15 19:51:22,667 - distributed.scheduler - INFO - Remove client Client-683589cd-3904-11f1-8001-daa6852bd5a7
2026-04-15 19:51:22,666 - distributed.scheduler - INFO - Remove client Client-683589cd-3904-11f1-8001-daa6852bd5a7
2026-04-15 19:50:51,651 - distributed.scheduler - INFO - Receive client connection: Client-683589cd-3904-11f1-8001-daa6852bd5a7
2026-04-15 19:41:23,011 - distributed.scheduler - INFO - Close client connection: Client-01af71c6-3903-11f1-8001-daa6852bd5a7
2026-04-15 19:41:23,010 - distributed.scheduler - INFO - Remove client Client-01af71c6-3903-11f1-8001-daa6852bd5a7
2026-04-15 19:41:23,008 - distributed.scheduler - INFO - Remove client Client-01af71c6-3903-11f1-8001-daa6852bd5a7
2026-04-15 19:40:50,149 - distributed.scheduler - INFO - Receive client connection: Client-01af71c6-3903-11f1-8001-daa6852bd5a7
2026-04-15 19:31:49,147 - distributed.scheduler - INFO - Close client connection: Client-9b2dbc01-3901-11f1-8001-daa6852bd5a7
2026-04-15 19:31:49,146 - distributed.scheduler - INFO - Remove client Client-9b2dbc01-3901-11f1-8001-daa6852bd5a7
2026-04-15 19:31:49,144 - distributed.scheduler - INFO - Remove client Client-9b2dbc01-3901-11f1-8001-daa6852bd5a7
2026-04-15 19:30:48,674 - distributed.scheduler - INFO - Receive client connection: Client-9b2dbc01-3901-11f1-8001-daa6852bd5a7
2026-04-15 19:21:20,155 - distributed.scheduler - INFO - Close client connection: Client-34b396a0-3900-11f1-8001-daa6852bd5a7
2026-04-15 19:21:20,154 - distributed.scheduler - INFO - Remove client Client-34b396a0-3900-11f1-8001-daa6852bd5a7
2026-04-15 19:21:20,152 - distributed.scheduler - INFO - Remove client Client-34b396a0-3900-11f1-8001-daa6852bd5a7
2026-04-15 19:20:47,253 - distributed.scheduler - INFO - Receive client connection: Client-34b396a0-3900-11f1-8001-daa6852bd5a7
2026-04-15 19:13:29,528 - distributed.scheduler - INFO - Close client connection: Client-1d40949c-38ff-11f1-8001-daa6852bd5a7
2026-04-15 19:13:29,525 - distributed.scheduler - INFO - Remove client Client-1d40949c-38ff-11f1-8001-daa6852bd5a7
2026-04-15 19:13:29,522 - distributed.scheduler - INFO - Remove client Client-1d40949c-38ff-11f1-8001-daa6852bd5a7
2026-04-15 19:12:58,403 - distributed.scheduler - INFO - Receive client connection: Client-1d40949c-38ff-11f1-8001-daa6852bd5a7
2026-04-15 19:01:53,444 - distributed.scheduler - INFO - Close client connection: Client-0148da4e-38fc-11f1-8001-daa6852bd5a7
2026-04-15 19:01:53,443 - distributed.scheduler - INFO - Remove client Client-0148da4e-38fc-11f1-8001-daa6852bd5a7
2026-04-15 19:01:52,792 - distributed.scheduler - INFO - Close client connection: Client-67c2ceec-38fd-11f1-8001-daa6852bd5a7
2026-04-15 19:01:52,790 - distributed.scheduler - INFO - Remove client Client-67c2ceec-38fd-11f1-8001-daa6852bd5a7
2026-04-15 19:01:52,789 - distributed.scheduler - INFO - Remove client Client-67c2ceec-38fd-11f1-8001-daa6852bd5a7
2026-04-15 19:00:44,411 - distributed.scheduler - INFO - Receive client connection: Client-67c2ceec-38fd-11f1-8001-daa6852bd5a7
2026-04-15 18:50:42,998 - distributed.scheduler - INFO - Receive client connection: Client-0148da4e-38fc-11f1-8001-daa6852bd5a7
2026-04-15 18:41:20,244 - distributed.scheduler - INFO - Close client connection: Client-9acf6546-38fa-11f1-8001-daa6852bd5a7
2026-04-15 18:41:20,242 - distributed.scheduler - INFO - Remove client Client-9acf6546-38fa-11f1-8001-daa6852bd5a7
2026-04-15 18:41:20,241 - distributed.scheduler - INFO - Remove client Client-9acf6546-38fa-11f1-8001-daa6852bd5a7
2026-04-15 18:40:41,570 - distributed.scheduler - INFO - Receive client connection: Client-9acf6546-38fa-11f1-8001-daa6852bd5a7
2026-04-15 18:31:37,298 - distributed.scheduler - INFO - Close client connection: Client-345601fa-38f9-11f1-8001-daa6852bd5a7
2026-04-15 18:31:37,297 - distributed.scheduler - INFO - Remove client Client-345601fa-38f9-11f1-8001-daa6852bd5a7
2026-04-15 18:31:37,295 - distributed.scheduler - INFO - Remove client Client-345601fa-38f9-11f1-8001-daa6852bd5a7
2026-04-15 18:30:40,156 - distributed.scheduler - INFO - Receive client connection: Client-345601fa-38f9-11f1-8001-daa6852bd5a7
2026-04-15 18:21:12,193 - distributed.scheduler - INFO - Close client connection: Client-cdd9315b-38f7-11f1-8001-daa6852bd5a7
2026-04-15 18:21:12,192 - distributed.scheduler - INFO - Remove client Client-cdd9315b-38f7-11f1-8001-daa6852bd5a7
2026-04-15 18:21:12,190 - distributed.scheduler - INFO - Remove client Client-cdd9315b-38f7-11f1-8001-daa6852bd5a7
2026-04-15 18:20:38,718 - distributed.scheduler - INFO - Receive client connection: Client-cdd9315b-38f7-11f1-8001-daa6852bd5a7
2026-04-15 18:13:07,426 - distributed.scheduler - INFO - Close client connection: Client-acc52b5d-38f6-11f1-8001-daa6852bd5a7
2026-04-15 18:13:07,425 - distributed.scheduler - INFO - Remove client Client-acc52b5d-38f6-11f1-8001-daa6852bd5a7
2026-04-15 18:13:07,424 - distributed.scheduler - INFO - Remove client Client-acc52b5d-38f6-11f1-8001-daa6852bd5a7
2026-04-15 18:12:33,722 - distributed.scheduler - INFO - Receive client connection: Client-acc52b5d-38f6-11f1-8001-daa6852bd5a7
2026-04-15 18:01:45,959 - distributed.scheduler - INFO - Close client connection: Client-00dd19ee-38f5-11f1-8001-daa6852bd5a7
2026-04-15 18:01:45,959 - distributed.scheduler - INFO - Remove client Client-00dd19ee-38f5-11f1-8001-daa6852bd5a7
2026-04-15 18:01:45,957 - distributed.scheduler - INFO - Remove client Client-00dd19ee-38f5-11f1-8001-daa6852bd5a7
2026-04-15 18:00:35,816 - distributed.scheduler - INFO - Receive client connection: Client-00dd19ee-38f5-11f1-8001-daa6852bd5a7
2026-04-15 17:51:15,541 - distributed.scheduler - INFO - Close client connection: Client-33cddebb-38f2-11f1-8001-daa6852bd5a7
2026-04-15 17:51:15,539 - distributed.scheduler - INFO - Remove client Client-33cddebb-38f2-11f1-8001-daa6852bd5a7
2026-04-15 17:51:15,081 - distributed.scheduler - INFO - Close client connection: Client-9a5a57df-38f3-11f1-8001-daa6852bd5a7
2026-04-15 17:51:15,080 - distributed.scheduler - INFO - Remove client Client-9a5a57df-38f3-11f1-8001-daa6852bd5a7
2026-04-15 17:51:15,078 - distributed.scheduler - INFO - Remove client Client-9a5a57df-38f3-11f1-8001-daa6852bd5a7
2026-04-15 17:50:34,323 - distributed.scheduler - INFO - Receive client connection: Client-9a5a57df-38f3-11f1-8001-daa6852bd5a7
2026-04-15 17:40:32,789 - distributed.scheduler - INFO - Receive client connection: Client-33cddebb-38f2-11f1-8001-daa6852bd5a7
2026-04-15 17:31:44,789 - distributed.scheduler - INFO - Close client connection: Client-cd46d370-38f0-11f1-8001-daa6852bd5a7
2026-04-15 17:31:44,789 - distributed.scheduler - INFO - Remove client Client-cd46d370-38f0-11f1-8001-daa6852bd5a7
2026-04-15 17:31:44,787 - distributed.scheduler - INFO - Remove client Client-cd46d370-38f0-11f1-8001-daa6852bd5a7
2026-04-15 17:30:31,278 - distributed.scheduler - INFO - Receive client connection: Client-cd46d370-38f0-11f1-8001-daa6852bd5a7
2026-04-15 17:21:10,203 - distributed.scheduler - INFO - Close client connection: Client-66c71d5c-38ef-11f1-8001-daa6852bd5a7
2026-04-15 17:21:10,202 - distributed.scheduler - INFO - Remove client Client-66c71d5c-38ef-11f1-8001-daa6852bd5a7
2026-04-15 17:21:10,200 - distributed.scheduler - INFO - Remove client Client-66c71d5c-38ef-11f1-8001-daa6852bd5a7
2026-04-15 17:20:29,818 - distributed.scheduler - INFO - Receive client connection: Client-66c71d5c-38ef-11f1-8001-daa6852bd5a7
2026-04-15 17:12:50,046 - distributed.scheduler - INFO - Close client connection: Client-3f4930e2-38ee-11f1-8001-daa6852bd5a7
2026-04-15 17:12:50,046 - distributed.scheduler - INFO - Remove client Client-3f4930e2-38ee-11f1-8001-daa6852bd5a7
2026-04-15 17:12:50,044 - distributed.scheduler - INFO - Remove client Client-3f4930e2-38ee-11f1-8001-daa6852bd5a7
2026-04-15 17:12:14,061 - distributed.scheduler - INFO - Receive client connection: Client-3f4930e2-38ee-11f1-8001-daa6852bd5a7
2026-04-15 17:01:40,445 - distributed.scheduler - INFO - Close client connection: Client-99d45266-38ec-11f1-8001-daa6852bd5a7
2026-04-15 17:01:40,444 - distributed.scheduler - INFO - Remove client Client-99d45266-38ec-11f1-8001-daa6852bd5a7
2026-04-15 17:01:40,443 - distributed.scheduler - INFO - Remove client Client-99d45266-38ec-11f1-8001-daa6852bd5a7
2026-04-15 17:00:26,979 - distributed.scheduler - INFO - Receive client connection: Client-99d45266-38ec-11f1-8001-daa6852bd5a7
2026-04-15 16:51:06,975 - distributed.scheduler - INFO - Close client connection: Client-33602bce-38eb-11f1-8001-daa6852bd5a7
2026-04-15 16:51:06,974 - distributed.scheduler - INFO - Remove client Client-33602bce-38eb-11f1-8001-daa6852bd5a7
2026-04-15 16:51:06,972 - distributed.scheduler - INFO - Remove client Client-33602bce-38eb-11f1-8001-daa6852bd5a7
2026-04-15 16:50:25,590 - distributed.scheduler - INFO - Receive client connection: Client-33602bce-38eb-11f1-8001-daa6852bd5a7
2026-04-15 16:41:06,827 - distributed.scheduler - INFO - Close client connection: Client-cce2e4a3-38e9-11f1-8001-daa6852bd5a7
2026-04-15 16:41:06,825 - distributed.scheduler - INFO - Remove client Client-cce2e4a3-38e9-11f1-8001-daa6852bd5a7
2026-04-15 16:41:06,823 - distributed.scheduler - INFO - Remove client Client-cce2e4a3-38e9-11f1-8001-daa6852bd5a7
2026-04-15 16:40:24,157 - distributed.scheduler - INFO - Receive client connection: Client-cce2e4a3-38e9-11f1-8001-daa6852bd5a7
2026-04-15 16:31:39,806 - distributed.scheduler - INFO - Close client connection: Client-66609109-38e8-11f1-8001-daa6852bd5a7
2026-04-15 16:31:39,804 - distributed.scheduler - INFO - Remove client Client-66609109-38e8-11f1-8001-daa6852bd5a7
2026-04-15 16:31:39,802 - distributed.scheduler - INFO - Remove client Client-66609109-38e8-11f1-8001-daa6852bd5a7
2026-04-15 16:30:22,668 - distributed.scheduler - INFO - Receive client connection: Client-66609109-38e8-11f1-8001-daa6852bd5a7
2026-04-15 16:20:59,864 - distributed.scheduler - INFO - Close client connection: Client-ffe1cdda-38e6-11f1-8001-daa6852bd5a7
2026-04-15 16:20:59,863 - distributed.scheduler - INFO - Remove client Client-ffe1cdda-38e6-11f1-8001-daa6852bd5a7
2026-04-15 16:20:59,861 - distributed.scheduler - INFO - Remove client Client-ffe1cdda-38e6-11f1-8001-daa6852bd5a7
2026-04-15 16:20:21,233 - distributed.scheduler - INFO - Receive client connection: Client-ffe1cdda-38e6-11f1-8001-daa6852bd5a7
2026-04-15 16:12:24,883 - distributed.scheduler - INFO - Close client connection: Client-cf3aaac8-38e5-11f1-8001-daa6852bd5a7
2026-04-15 16:12:24,882 - distributed.scheduler - INFO - Remove client Client-cf3aaac8-38e5-11f1-8001-daa6852bd5a7
2026-04-15 16:12:24,881 - distributed.scheduler - INFO - Remove client Client-cf3aaac8-38e5-11f1-8001-daa6852bd5a7
2026-04-15 16:11:50,089 - distributed.scheduler - INFO - Receive client connection: Client-cf3aaac8-38e5-11f1-8001-daa6852bd5a7
2026-04-15 16:01:39,815 - distributed.scheduler - INFO - Close client connection: Client-32e61740-38e4-11f1-8001-daa6852bd5a7
2026-04-15 16:01:39,814 - distributed.scheduler - INFO - Remove client Client-32e61740-38e4-11f1-8001-daa6852bd5a7
2026-04-15 16:01:39,812 - distributed.scheduler - INFO - Remove client Client-32e61740-38e4-11f1-8001-daa6852bd5a7
2026-04-15 16:00:18,310 - distributed.scheduler - INFO - Receive client connection: Client-32e61740-38e4-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,643 - distributed.scheduler - INFO - Close client connection: Client-87328d27-38be-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,643 - distributed.scheduler - INFO - Close client connection: Client-edafca0a-38bf-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,643 - distributed.scheduler - INFO - Close client connection: Client-54298aa2-38c1-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,643 - distributed.scheduler - INFO - Close client connection: Client-ba9a771c-38c2-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,642 - distributed.scheduler - INFO - Close client connection: Client-87a7a610-38c5-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,642 - distributed.scheduler - INFO - Close client connection: Client-79e53276-38c4-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,642 - distributed.scheduler - INFO - Close client connection: Client-ee39eaab-38c6-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,642 - distributed.scheduler - INFO - Close client connection: Client-54baeefd-38c8-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,642 - distributed.scheduler - INFO - Close client connection: Client-bb30695b-38c9-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,642 - distributed.scheduler - INFO - Close client connection: Client-21afde34-38cb-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,642 - distributed.scheduler - INFO - Close client connection: Client-eebd3530-38cd-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,641 - distributed.scheduler - INFO - Close client connection: Client-d0d94014-38cc-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,641 - distributed.scheduler - INFO - Close client connection: Client-553b4cf7-38cf-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,641 - distributed.scheduler - INFO - Close client connection: Client-bbce836d-38d0-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,641 - distributed.scheduler - INFO - Close client connection: Client-2246ab07-38d2-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,641 - distributed.scheduler - INFO - Close client connection: Client-64e7b571-38d3-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,641 - distributed.scheduler - INFO - Close client connection: Client-31e6818d-38d6-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,641 - distributed.scheduler - INFO - Close client connection: Client-15198627-38d5-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,640 - distributed.scheduler - INFO - Close client connection: Client-9867a53b-38d7-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,640 - distributed.scheduler - INFO - Close client connection: Client-fef0bc4f-38d8-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,640 - distributed.scheduler - INFO - Close client connection: Client-65770233-38da-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,640 - distributed.scheduler - INFO - Close client connection: Client-cbeeefc2-38db-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,640 - distributed.scheduler - INFO - Close client connection: Client-98de08c8-38de-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,640 - distributed.scheduler - INFO - Close client connection: Client-7818d551-38dd-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,638 - distributed.scheduler - INFO - Close client connection: Client-ff59cbbc-38df-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,638 - distributed.scheduler - INFO - Close client connection: Client-65def7ce-38e1-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,637 - distributed.scheduler - INFO - Remove client Client-87328d27-38be-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,637 - distributed.scheduler - INFO - Remove client Client-edafca0a-38bf-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,637 - distributed.scheduler - INFO - Remove client Client-54298aa2-38c1-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,635 - distributed.scheduler - INFO - Remove client Client-ba9a771c-38c2-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,634 - distributed.scheduler - INFO - Remove client Client-87a7a610-38c5-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,634 - distributed.scheduler - INFO - Remove client Client-79e53276-38c4-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,633 - distributed.scheduler - INFO - Remove client Client-ee39eaab-38c6-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,633 - distributed.scheduler - INFO - Remove client Client-54baeefd-38c8-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,632 - distributed.scheduler - INFO - Remove client Client-bb30695b-38c9-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,631 - distributed.scheduler - INFO - Remove client Client-21afde34-38cb-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,630 - distributed.scheduler - INFO - Remove client Client-eebd3530-38cd-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,630 - distributed.scheduler - INFO - Remove client Client-d0d94014-38cc-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,629 - distributed.scheduler - INFO - Remove client Client-553b4cf7-38cf-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,629 - distributed.scheduler - INFO - Remove client Client-bbce836d-38d0-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,628 - distributed.scheduler - INFO - Remove client Client-2246ab07-38d2-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,628 - distributed.scheduler - INFO - Remove client Client-64e7b571-38d3-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,627 - distributed.scheduler - INFO - Remove client Client-31e6818d-38d6-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,627 - distributed.scheduler - INFO - Remove client Client-15198627-38d5-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,626 - distributed.scheduler - INFO - Remove client Client-9867a53b-38d7-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,626 - distributed.scheduler - INFO - Remove client Client-fef0bc4f-38d8-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,625 - distributed.scheduler - INFO - Remove client Client-65770233-38da-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,625 - distributed.scheduler - INFO - Remove client Client-cbeeefc2-38db-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,624 - distributed.scheduler - INFO - Remove client Client-98de08c8-38de-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,624 - distributed.scheduler - INFO - Remove client Client-7818d551-38dd-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,614 - distributed.scheduler - INFO - Remove client Client-ff59cbbc-38df-11f1-8001-daa6852bd5a7
2026-04-15 15:51:01,613 - distributed.scheduler - INFO - Remove client Client-65def7ce-38e1-11f1-8001-daa6852bd5a7
2026-04-15 15:51:00,739 - distributed.scheduler - INFO - Close client connection: Client-cc61b004-38e2-11f1-8001-daa6852bd5a7
2026-04-15 15:51:00,738 - distributed.scheduler - INFO - Remove client Client-cc61b004-38e2-11f1-8001-daa6852bd5a7
2026-04-15 15:51:00,736 - distributed.scheduler - INFO - Remove client Client-cc61b004-38e2-11f1-8001-daa6852bd5a7
2026-04-15 15:50:16,812 - distributed.scheduler - INFO - Receive client connection: Client-cc61b004-38e2-11f1-8001-daa6852bd5a7
2026-04-15 15:40:15,337 - distributed.scheduler - INFO - Receive client connection: Client-65def7ce-38e1-11f1-8001-daa6852bd5a7
2026-04-15 15:30:13,834 - distributed.scheduler - INFO - Receive client connection: Client-ff59cbbc-38df-11f1-8001-daa6852bd5a7
2026-04-15 15:20:12,401 - distributed.scheduler - INFO - Receive client connection: Client-98de08c8-38de-11f1-8001-daa6852bd5a7
2026-04-15 15:12:07,916 - distributed.scheduler - INFO - Receive client connection: Client-7818d551-38dd-11f1-8001-daa6852bd5a7
2026-04-15 15:00:09,610 - distributed.scheduler - INFO - Receive client connection: Client-cbeeefc2-38db-11f1-8001-daa6852bd5a7
2026-04-15 14:50:08,170 - distributed.scheduler - INFO - Receive client connection: Client-65770233-38da-11f1-8001-daa6852bd5a7
2026-04-15 14:40:06,663 - distributed.scheduler - INFO - Receive client connection: Client-fef0bc4f-38d8-11f1-8001-daa6852bd5a7
2026-04-15 14:30:10,551 - distributed.scheduler - INFO - Receive client connection: Client-worker-9b9bc7b3-38d7-11f1-81a9-2e20a4008324
2026-04-15 14:30:05,143 - distributed.scheduler - INFO - Receive client connection: Client-9867a53b-38d7-11f1-8001-daa6852bd5a7
2026-04-15 14:20:03,679 - distributed.scheduler - INFO - Receive client connection: Client-31e6818d-38d6-11f1-8001-daa6852bd5a7
2026-04-15 14:12:05,851 - distributed.scheduler - INFO - Receive client connection: Client-15198627-38d5-11f1-8001-daa6852bd5a7
2026-04-15 14:00:04,860 - distributed.scheduler - INFO - Receive client connection: Client-worker-675a9804-38d3-11f1-81ee-86acc56fd09f
2026-04-15 14:00:00,757 - distributed.scheduler - INFO - Receive client connection: Client-64e7b571-38d3-11f1-8001-daa6852bd5a7
2026-04-15 13:50:59,471 - distributed.scheduler - INFO - Receive client connection: Client-2246ab07-38d2-11f1-8001-daa6852bd5a7
2026-04-15 13:40:58,066 - distributed.scheduler - INFO - Receive client connection: Client-bbce836d-38d0-11f1-8001-daa6852bd5a7
2026-04-15 13:34:32,403 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:41657
2026-04-15 13:34:32,402 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:41657', status: closed, memory: 0, processing: 0>
2026-04-15 13:34:32,276 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:38347
2026-04-15 13:34:32,274 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:38347', status: closed, memory: 0, processing: 0>
2026-04-15 13:34:31,484 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:37573', status: closed, memory: 0, processing: 0>
2026-04-15 13:34:31,291 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:43645', status: closed, memory: 0, processing: 0>
2026-04-15 13:34:19,484 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:37399 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:37399 after 30 s
2026-04-15 13:34:19,484 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:36611 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:36611 after 30 s
2026-04-15 13:34:19,383 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:36611 after 30 s')
2026-04-15 13:34:19,374 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:36611 after 30 s')
2026-04-15 13:34:19,368 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:36611 after 30 s')
2026-04-15 13:34:19,366 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:36611 after 30 s')
2026-04-15 13:34:19,360 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:37399 after 30 s')
2026-04-15 13:34:19,353 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:37399 after 30 s')
2026-04-15 13:34:19,351 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:46541 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:37399 after 30 s')
2026-04-15 13:34:19,349 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:36611 after 30 s')
2026-04-15 13:34:19,339 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:36611 after 30 s')
2026-04-15 13:34:19,339 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:37399 after 30 s')
2026-04-15 13:34:19,336 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:36611 after 30 s')
2026-04-15 13:33:50,209 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:43645
2026-04-15 13:33:50,208 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:43645', status: closed, memory: 0, processing: 0>
2026-04-15 13:33:50,195 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:37573
2026-04-15 13:33:50,193 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:37573', status: closed, memory: 0, processing: 0>
2026-04-15 13:33:49,192 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:36611', status: closed, memory: 0, processing: 0>
2026-04-15 13:33:49,133 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:37399', status: closed, memory: 0, processing: 0>
2026-04-15 13:33:38,643 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:37399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:33227 after 30 s')
2026-04-15 13:33:38,555 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:33227 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:33227 after 30 s
2026-04-15 13:33:38,555 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:34765 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:34765 after 30 s
2026-04-15 13:33:37,911 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:33227 after 30 s')
2026-04-15 13:33:37,911 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:34765 after 30 s')
2026-04-15 13:33:37,879 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:33227 after 30 s')
2026-04-15 13:33:37,878 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:33227 after 30 s')
2026-04-15 13:33:37,869 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35361 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:33227 after 30 s')
2026-04-15 13:33:37,864 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:33227 after 30 s')
2026-04-15 13:33:37,863 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:46541 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:33227 after 30 s')
2026-04-15 13:33:37,861 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:34765 after 30 s')
2026-04-15 13:33:37,860 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:34765 after 30 s')
2026-04-15 13:33:37,859 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:33227 after 30 s')
2026-04-15 13:33:37,859 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:34765 after 30 s')
2026-04-15 13:33:37,821 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:33227 after 30 s')
2026-04-15 13:33:08,649 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:36611
2026-04-15 13:33:08,648 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:36611', status: closed, memory: 0, processing: 0>
2026-04-15 13:33:07,695 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:33227', status: closed, memory: 0, processing: 0>
2026-04-15 13:33:03,847 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:37399
2026-04-15 13:33:03,844 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:37399', status: closed, memory: 0, processing: 0>
2026-04-15 13:33:02,834 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:34765', status: closed, memory: 0, processing: 0>
2026-04-15 13:32:52,012 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:39607 after 30 s')
2026-04-15 13:32:51,892 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:39607 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:39607 after 30 s
2026-04-15 13:32:51,892 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:44831 failed: OSError: Timed out trying to connect to tcp://172.21.25.32:44831 after 30 s
2026-04-15 13:32:51,656 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.32:44831 after 30 s')
2026-04-15 13:32:21,873 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:34765
2026-04-15 13:32:21,872 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:34765', status: closed, memory: 0, processing: 0>
2026-04-15 13:32:21,869 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:33227
2026-04-15 13:32:21,866 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:33227', status: closed, memory: 0, processing: 0>
2026-04-15 13:32:20,834 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:39607', status: closed, memory: 0, processing: 0>
2026-04-15 13:32:20,701 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:44831', status: closed, memory: 0, processing: 0>
2026-04-15 13:31:43,918 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.32:44831
2026-04-15 13:31:43,909 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.32:44831', status: closed, memory: 0, processing: 0>
2026-04-15 13:31:42,858 - distributed.scheduler - INFO - Close client connection: Client-worker-3cc38c46-37bf-11f1-8074-2e20a4008324
2026-04-15 13:31:42,849 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.32:38115', status: closed, memory: 0, processing: 0>
2026-04-15 13:31:42,849 - distributed.scheduler - INFO - Remove client Client-worker-3cc38c46-37bf-11f1-8074-2e20a4008324
2026-04-15 13:31:40,095 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:39607
2026-04-15 13:31:40,092 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:39607', status: closed, memory: 0, processing: 0>
2026-04-15 13:31:39,031 - distributed.scheduler - INFO - Close client connection: Client-worker-2444365d-38cb-11f1-8171-86acc56fd09f
2026-04-15 13:31:39,027 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:44849', status: closed, memory: 0, processing: 0>
2026-04-15 13:31:39,025 - distributed.scheduler - INFO - Remove client Client-worker-2444365d-38cb-11f1-8171-86acc56fd09f
2026-04-15 13:30:56,475 - distributed.scheduler - INFO - Receive client connection: Client-553b4cf7-38cf-11f1-8001-daa6852bd5a7
2026-04-15 13:20:55,024 - distributed.scheduler - INFO - Receive client connection: Client-eebd3530-38cd-11f1-8001-daa6852bd5a7
2026-04-15 13:12:55,373 - distributed.scheduler - INFO - Receive client connection: Client-d0d94014-38cc-11f1-8001-daa6852bd5a7
2026-04-15 13:00:56,337 - distributed.scheduler - INFO - Receive client connection: Client-worker-2444365d-38cb-11f1-8171-86acc56fd09f
2026-04-15 13:00:52,009 - distributed.scheduler - INFO - Receive client connection: Client-21afde34-38cb-11f1-8001-daa6852bd5a7
2026-04-15 12:50:50,547 - distributed.scheduler - INFO - Receive client connection: Client-bb30695b-38c9-11f1-8001-daa6852bd5a7
2026-04-15 12:40:49,156 - distributed.scheduler - INFO - Receive client connection: Client-54baeefd-38c8-11f1-8001-daa6852bd5a7
2026-04-15 12:34:43,112 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45987 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45987 after 30 s
2026-04-15 12:34:43,083 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:45987 after 30 s')
2026-04-15 12:34:43,010 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38115 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:45987 after 30 s')
2026-04-15 12:34:42,982 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:45987 after 30 s')
2026-04-15 12:34:42,978 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:45987 after 30 s')
2026-04-15 12:34:42,976 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:46541 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:45987 after 30 s')
2026-04-15 12:34:42,976 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:45987 after 30 s')
2026-04-15 12:34:42,973 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:45987 after 30 s')
2026-04-15 12:34:42,926 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:45987 after 30 s')
2026-04-15 12:34:13,850 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:44849
2026-04-15 12:34:13,848 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:44849', status: closed, memory: 0, processing: 0>
2026-04-15 12:34:12,869 - distributed.scheduler - INFO - Close client connection: Client-worker-5ec72102-38c7-11f1-8157-86acc56fd09f
2026-04-15 12:34:12,845 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:45987', status: closed, memory: 0, processing: 0>
2026-04-15 12:34:12,843 - distributed.scheduler - INFO - Remove client Client-worker-5ec72102-38c7-11f1-8157-86acc56fd09f
2026-04-15 12:33:56,513 - distributed.scheduler - INFO - Receive client connection: Client-worker-5ec72102-38c7-11f1-8157-86acc56fd09f
2026-04-15 12:33:52,639 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:36155 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s
2026-04-15 12:33:52,540 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s')
2026-04-15 12:33:52,509 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s')
2026-04-15 12:33:52,505 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:46541 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s')
2026-04-15 12:33:52,502 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s')
2026-04-15 12:33:52,502 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s')
2026-04-15 12:33:52,500 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s')
2026-04-15 12:33:52,492 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s')
2026-04-15 12:33:52,491 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s')
2026-04-15 12:33:52,481 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s')
2026-04-15 12:33:52,481 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s')
2026-04-15 12:33:52,478 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s')
2026-04-15 12:33:52,432 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38115 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:36155 after 30 s')
2026-04-15 12:33:23,308 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:45987
2026-04-15 12:33:23,305 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:45987', status: closed, memory: 0, processing: 0>
2026-04-15 12:33:22,336 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:36155', status: closed, memory: 0, processing: 0>
2026-04-15 12:33:09,511 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:42811 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s
2026-04-15 12:33:09,468 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s')
2026-04-15 12:33:09,439 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s')
2026-04-15 12:33:09,400 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s')
2026-04-15 12:33:09,392 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s')
2026-04-15 12:33:09,391 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38115 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s')
2026-04-15 12:33:09,388 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35361 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s')
2026-04-15 12:33:09,385 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:44087 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s')
2026-04-15 12:33:09,383 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s')
2026-04-15 12:33:09,383 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s')
2026-04-15 12:33:09,383 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s')
2026-04-15 12:33:09,358 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s')
2026-04-15 12:33:09,330 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:42811 after 30 s')
2026-04-15 12:32:40,471 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:36155
2026-04-15 12:32:40,468 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:36155', status: closed, memory: 0, processing: 0>
2026-04-15 12:32:39,237 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:42811', status: closed, memory: 0, processing: 0>
2026-04-15 12:32:24,213 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45643 failed: OSError: Timed out trying to connect to tcp://172.21.25.34:45643 after 30 s
2026-04-15 12:32:23,985 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.25.34:45643 after 30 s')
2026-04-15 12:31:54,018 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.25.34:42811
2026-04-15 12:31:54,017 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.25.34:42811', status: closed, memory: 0, processing: 0>
2026-04-15 12:31:52,963 - distributed.scheduler - INFO - Close client connection: Client-worker-332ee1af-37c3-11f1-8074-86acc56fd09f
2026-04-15 12:31:52,943 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.25.34:45643', status: closed, memory: 0, processing: 0>
2026-04-15 12:31:52,942 - distributed.scheduler - INFO - Remove client Client-worker-332ee1af-37c3-11f1-8074-86acc56fd09f
2026-04-15 12:30:47,688 - distributed.scheduler - INFO - Receive client connection: Client-ee39eaab-38c6-11f1-8001-daa6852bd5a7
2026-04-15 12:20:46,104 - distributed.scheduler - INFO - Receive client connection: Client-87a7a610-38c5-11f1-8001-daa6852bd5a7
2026-04-15 12:13:13,520 - distributed.scheduler - INFO - Receive client connection: Client-79e53276-38c4-11f1-8001-daa6852bd5a7
2026-04-15 12:00:43,089 - distributed.scheduler - INFO - Receive client connection: Client-ba9a771c-38c2-11f1-8001-daa6852bd5a7
2026-04-15 11:50:41,718 - distributed.scheduler - INFO - Receive client connection: Client-54298aa2-38c1-11f1-8001-daa6852bd5a7
2026-04-15 11:40:40,305 - distributed.scheduler - INFO - Receive client connection: Client-edafca0a-38bf-11f1-8001-daa6852bd5a7
2026-04-15 11:30:38,856 - distributed.scheduler - INFO - Receive client connection: Client-87328d27-38be-11f1-8001-daa6852bd5a7
2026-04-15 11:21:11,688 - distributed.scheduler - INFO - Close client connection: Client-20b7145b-38bd-11f1-8001-daa6852bd5a7
2026-04-15 11:21:11,687 - distributed.scheduler - INFO - Remove client Client-20b7145b-38bd-11f1-8001-daa6852bd5a7
2026-04-15 11:21:11,686 - distributed.scheduler - INFO - Remove client Client-20b7145b-38bd-11f1-8001-daa6852bd5a7
2026-04-15 11:20:37,435 - distributed.scheduler - INFO - Receive client connection: Client-20b7145b-38bd-11f1-8001-daa6852bd5a7
2026-04-15 11:13:24,009 - distributed.scheduler - INFO - Close client connection: Client-0d9e86d0-38bc-11f1-8001-daa6852bd5a7
2026-04-15 11:13:24,008 - distributed.scheduler - INFO - Remove client Client-0d9e86d0-38bc-11f1-8001-daa6852bd5a7
2026-04-15 11:13:24,006 - distributed.scheduler - INFO - Remove client Client-0d9e86d0-38bc-11f1-8001-daa6852bd5a7
2026-04-15 11:12:55,894 - distributed.scheduler - INFO - Receive client connection: Client-0d9e86d0-38bc-11f1-8001-daa6852bd5a7
2026-04-15 11:01:40,465 - distributed.scheduler - INFO - Close client connection: Client-53b2e1ec-38ba-11f1-8001-daa6852bd5a7
2026-04-15 11:01:40,464 - distributed.scheduler - INFO - Remove client Client-53b2e1ec-38ba-11f1-8001-daa6852bd5a7
2026-04-15 11:01:40,463 - distributed.scheduler - INFO - Remove client Client-53b2e1ec-38ba-11f1-8001-daa6852bd5a7
2026-04-15 11:00:34,482 - distributed.scheduler - INFO - Receive client connection: Client-53b2e1ec-38ba-11f1-8001-daa6852bd5a7
2026-04-15 10:51:04,862 - distributed.scheduler - INFO - Close client connection: Client-ed3c5a3f-38b8-11f1-8001-daa6852bd5a7
2026-04-15 10:51:04,862 - distributed.scheduler - INFO - Remove client Client-ed3c5a3f-38b8-11f1-8001-daa6852bd5a7
2026-04-15 10:51:04,861 - distributed.scheduler - INFO - Remove client Client-ed3c5a3f-38b8-11f1-8001-daa6852bd5a7
2026-04-15 10:50:33,082 - distributed.scheduler - INFO - Receive client connection: Client-ed3c5a3f-38b8-11f1-8001-daa6852bd5a7
2026-04-15 10:41:04,990 - distributed.scheduler - INFO - Close client connection: Client-86b89f6d-38b7-11f1-8001-daa6852bd5a7
2026-04-15 10:41:04,988 - distributed.scheduler - INFO - Remove client Client-86b89f6d-38b7-11f1-8001-daa6852bd5a7
2026-04-15 10:41:04,987 - distributed.scheduler - INFO - Remove client Client-86b89f6d-38b7-11f1-8001-daa6852bd5a7
2026-04-15 10:40:31,591 - distributed.scheduler - INFO - Receive client connection: Client-86b89f6d-38b7-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,222 - distributed.scheduler - INFO - Close client connection: Client-c28ec97f-382b-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,222 - distributed.scheduler - INFO - Close client connection: Client-8fa51cf1-382e-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,222 - distributed.scheduler - INFO - Close client connection: Client-6939a497-382d-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,221 - distributed.scheduler - INFO - Close client connection: Client-f632470f-382f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,221 - distributed.scheduler - INFO - Close client connection: Client-5cb2172f-3831-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,221 - distributed.scheduler - INFO - Close client connection: Client-c329a259-3832-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,221 - distributed.scheduler - INFO - Close client connection: Client-29a56898-3834-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,221 - distributed.scheduler - INFO - Close client connection: Client-f6b3bec3-3836-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,221 - distributed.scheduler - INFO - Close client connection: Client-ce34b908-3835-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,221 - distributed.scheduler - INFO - Close client connection: Client-5d4a4003-3838-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,221 - distributed.scheduler - INFO - Close client connection: Client-c3d8728b-3839-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,221 - distributed.scheduler - INFO - Close client connection: Client-2a5dcb57-383b-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,221 - distributed.scheduler - INFO - Close client connection: Client-90ddab17-383c-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,221 - distributed.scheduler - INFO - Close client connection: Client-5def6098-383f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,220 - distributed.scheduler - INFO - Close client connection: Client-439c65af-383e-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,220 - distributed.scheduler - INFO - Close client connection: Client-c4788420-3840-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,220 - distributed.scheduler - INFO - Close client connection: Client-2afd3657-3842-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,220 - distributed.scheduler - INFO - Close client connection: Client-918223d1-3843-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,220 - distributed.scheduler - INFO - Close client connection: Client-f80392b1-3844-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,220 - distributed.scheduler - INFO - Close client connection: Client-c513e133-3847-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,220 - distributed.scheduler - INFO - Close client connection: Client-ab033046-3846-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,220 - distributed.scheduler - INFO - Close client connection: Client-2b9b6495-3849-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,220 - distributed.scheduler - INFO - Close client connection: Client-9225b54e-384a-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,220 - distributed.scheduler - INFO - Close client connection: Client-f8af0f95-384b-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,220 - distributed.scheduler - INFO - Close client connection: Client-5f351503-384d-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,220 - distributed.scheduler - INFO - Close client connection: Client-2c391284-3850-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,219 - distributed.scheduler - INFO - Close client connection: Client-02322f3a-384f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,219 - distributed.scheduler - INFO - Close client connection: Client-92b8cf85-3851-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,219 - distributed.scheduler - INFO - Close client connection: Client-f94d4b32-3852-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,219 - distributed.scheduler - INFO - Close client connection: Client-5fd5f50c-3854-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,219 - distributed.scheduler - INFO - Close client connection: Client-c6570749-3855-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,219 - distributed.scheduler - INFO - Close client connection: Client-7a318f9e-3857-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,219 - distributed.scheduler - INFO - Close client connection: Client-9358d19c-3858-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,219 - distributed.scheduler - INFO - Close client connection: Client-f9dc41a3-3859-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,219 - distributed.scheduler - INFO - Close client connection: Client-605a4821-385b-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,219 - distributed.scheduler - INFO - Close client connection: Client-c6ec5411-385c-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,219 - distributed.scheduler - INFO - Close client connection: Client-2d793d4a-385e-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-fa786c8f-3860-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-c23415d6-385f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-3d1ce715-3862-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-a3a5a31b-3863-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-0a2c705d-3865-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-70b201d5-3866-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-3db38583-3869-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-1363fc7c-3868-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-a431e1c4-386a-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-0ab56c8d-386c-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-71397551-386d-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-d7bec258-386e-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,218 - distributed.scheduler - INFO - Close client connection: Client-97cdb65b-3870-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,217 - distributed.scheduler - INFO - Close client connection: Client-a4e6f003-3871-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,217 - distributed.scheduler - INFO - Close client connection: Client-0b722fc7-3873-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,217 - distributed.scheduler - INFO - Close client connection: Client-71f55b38-3874-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,217 - distributed.scheduler - INFO - Close client connection: Client-d8824ad1-3875-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,217 - distributed.scheduler - INFO - Close client connection: Client-3f0fbac9-3877-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,217 - distributed.scheduler - INFO - Close client connection: Client-0c1e83c7-387a-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,217 - distributed.scheduler - INFO - Close client connection: Client-04a8d8e4-3879-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,217 - distributed.scheduler - INFO - Close client connection: Client-72a1a4df-387b-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,217 - distributed.scheduler - INFO - Close client connection: Client-d91e49d6-387c-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,217 - distributed.scheduler - INFO - Close client connection: Client-3fa56a30-387e-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,217 - distributed.scheduler - INFO - Close client connection: Client-a6289561-387f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,216 - distributed.scheduler - INFO - Close client connection: Client-733d049f-3882-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,216 - distributed.scheduler - INFO - Close client connection: Client-5991845f-3881-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,216 - distributed.scheduler - INFO - Close client connection: Client-d9c61164-3883-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,216 - distributed.scheduler - INFO - Close client connection: Client-403c5ad1-3885-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,216 - distributed.scheduler - INFO - Close client connection: Client-a6b54bb8-3886-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,216 - distributed.scheduler - INFO - Close client connection: Client-0d33c9c2-3888-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,216 - distributed.scheduler - INFO - Close client connection: Client-da213a3d-388a-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,216 - distributed.scheduler - INFO - Close client connection: Client-db714b3d-3889-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,216 - distributed.scheduler - INFO - Close client connection: Client-40a1ae1d-388c-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,216 - distributed.scheduler - INFO - Close client connection: Client-a7259f84-388d-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,216 - distributed.scheduler - INFO - Close client connection: Client-0dabcd4a-388f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,215 - distributed.scheduler - INFO - Close client connection: Client-742c9d28-3890-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,215 - distributed.scheduler - INFO - Close client connection: Client-3020d18e-3892-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,215 - distributed.scheduler - INFO - Close client connection: Client-4132bff2-3893-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,215 - distributed.scheduler - INFO - Close client connection: Client-a7bf29a6-3894-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,215 - distributed.scheduler - INFO - Close client connection: Client-0e3d1e58-3896-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,215 - distributed.scheduler - INFO - Close client connection: Client-74c013f2-3897-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,215 - distributed.scheduler - INFO - Close client connection: Client-db3f28b9-3898-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,215 - distributed.scheduler - INFO - Close client connection: Client-92ad5bc6-389a-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,215 - distributed.scheduler - INFO - Close client connection: Client-8461c735-389b-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,215 - distributed.scheduler - INFO - Close client connection: Client-eaee4a1d-389c-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,215 - distributed.scheduler - INFO - Close client connection: Client-51726c7d-389e-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,214 - distributed.scheduler - INFO - Close client connection: Client-b7fa6fe3-389f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,214 - distributed.scheduler - INFO - Close client connection: Client-1e81f26e-38a1-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,214 - distributed.scheduler - INFO - Close client connection: Client-f21862d4-38a2-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,214 - distributed.scheduler - INFO - Close client connection: Client-51fe0d90-38a5-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,214 - distributed.scheduler - INFO - Close client connection: Client-eb787e3d-38a3-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,214 - distributed.scheduler - INFO - Close client connection: Client-b87a5813-38a6-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,214 - distributed.scheduler - INFO - Close client connection: Client-85912867-38a9-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,214 - distributed.scheduler - INFO - Close client connection: Client-1f04e4e7-38a8-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,211 - distributed.scheduler - INFO - Close client connection: Client-529411b4-38ac-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,211 - distributed.scheduler - INFO - Close client connection: Client-4fe8afc1-38ab-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,211 - distributed.scheduler - INFO - Close client connection: Client-b91af5f7-38ad-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,210 - distributed.scheduler - INFO - Close client connection: Client-1faf71ad-38af-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,210 - distributed.scheduler - INFO - Close client connection: Client-862eef83-38b0-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,210 - distributed.scheduler - INFO - Close client connection: Client-b9b7e8da-38b4-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,210 - distributed.scheduler - INFO - Close client connection: Client-ecb25406-38b1-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,210 - distributed.scheduler - INFO - Close client connection: Client-a63dd4a1-38b3-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,205 - distributed.scheduler - INFO - Remove client Client-c28ec97f-382b-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,205 - distributed.scheduler - INFO - Remove client Client-8fa51cf1-382e-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,205 - distributed.scheduler - INFO - Remove client Client-6939a497-382d-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,204 - distributed.scheduler - INFO - Remove client Client-f632470f-382f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,204 - distributed.scheduler - INFO - Remove client Client-5cb2172f-3831-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,204 - distributed.scheduler - INFO - Remove client Client-c329a259-3832-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,204 - distributed.scheduler - INFO - Remove client Client-29a56898-3834-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,204 - distributed.scheduler - INFO - Remove client Client-f6b3bec3-3836-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,203 - distributed.scheduler - INFO - Remove client Client-ce34b908-3835-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,203 - distributed.scheduler - INFO - Remove client Client-5d4a4003-3838-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,203 - distributed.scheduler - INFO - Remove client Client-c3d8728b-3839-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,203 - distributed.scheduler - INFO - Remove client Client-2a5dcb57-383b-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,202 - distributed.scheduler - INFO - Remove client Client-90ddab17-383c-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,202 - distributed.scheduler - INFO - Remove client Client-5def6098-383f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,202 - distributed.scheduler - INFO - Remove client Client-439c65af-383e-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,202 - distributed.scheduler - INFO - Remove client Client-c4788420-3840-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,202 - distributed.scheduler - INFO - Remove client Client-2afd3657-3842-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,201 - distributed.scheduler - INFO - Remove client Client-918223d1-3843-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,201 - distributed.scheduler - INFO - Remove client Client-f80392b1-3844-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,201 - distributed.scheduler - INFO - Remove client Client-c513e133-3847-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,201 - distributed.scheduler - INFO - Remove client Client-ab033046-3846-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,200 - distributed.scheduler - INFO - Remove client Client-2b9b6495-3849-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,200 - distributed.scheduler - INFO - Remove client Client-9225b54e-384a-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,200 - distributed.scheduler - INFO - Remove client Client-f8af0f95-384b-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,200 - distributed.scheduler - INFO - Remove client Client-5f351503-384d-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,200 - distributed.scheduler - INFO - Remove client Client-2c391284-3850-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,199 - distributed.scheduler - INFO - Remove client Client-02322f3a-384f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,199 - distributed.scheduler - INFO - Remove client Client-92b8cf85-3851-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,199 - distributed.scheduler - INFO - Remove client Client-f94d4b32-3852-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,199 - distributed.scheduler - INFO - Remove client Client-5fd5f50c-3854-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,198 - distributed.scheduler - INFO - Remove client Client-c6570749-3855-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,198 - distributed.scheduler - INFO - Remove client Client-7a318f9e-3857-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,198 - distributed.scheduler - INFO - Remove client Client-9358d19c-3858-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,198 - distributed.scheduler - INFO - Remove client Client-f9dc41a3-3859-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,197 - distributed.scheduler - INFO - Remove client Client-605a4821-385b-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,197 - distributed.scheduler - INFO - Remove client Client-c6ec5411-385c-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,197 - distributed.scheduler - INFO - Remove client Client-2d793d4a-385e-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,197 - distributed.scheduler - INFO - Remove client Client-fa786c8f-3860-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,196 - distributed.scheduler - INFO - Remove client Client-c23415d6-385f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,196 - distributed.scheduler - INFO - Remove client Client-3d1ce715-3862-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,196 - distributed.scheduler - INFO - Remove client Client-a3a5a31b-3863-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,196 - distributed.scheduler - INFO - Remove client Client-0a2c705d-3865-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,196 - distributed.scheduler - INFO - Remove client Client-70b201d5-3866-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,195 - distributed.scheduler - INFO - Remove client Client-3db38583-3869-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,195 - distributed.scheduler - INFO - Remove client Client-1363fc7c-3868-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,195 - distributed.scheduler - INFO - Remove client Client-a431e1c4-386a-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,195 - distributed.scheduler - INFO - Remove client Client-0ab56c8d-386c-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,194 - distributed.scheduler - INFO - Remove client Client-71397551-386d-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,194 - distributed.scheduler - INFO - Remove client Client-d7bec258-386e-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,194 - distributed.scheduler - INFO - Remove client Client-97cdb65b-3870-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,194 - distributed.scheduler - INFO - Remove client Client-a4e6f003-3871-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,193 - distributed.scheduler - INFO - Remove client Client-0b722fc7-3873-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,193 - distributed.scheduler - INFO - Remove client Client-71f55b38-3874-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,193 - distributed.scheduler - INFO - Remove client Client-d8824ad1-3875-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,193 - distributed.scheduler - INFO - Remove client Client-3f0fbac9-3877-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,192 - distributed.scheduler - INFO - Remove client Client-0c1e83c7-387a-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,192 - distributed.scheduler - INFO - Remove client Client-04a8d8e4-3879-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,192 - distributed.scheduler - INFO - Remove client Client-72a1a4df-387b-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,192 - distributed.scheduler - INFO - Remove client Client-d91e49d6-387c-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,192 - distributed.scheduler - INFO - Remove client Client-3fa56a30-387e-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,191 - distributed.scheduler - INFO - Remove client Client-a6289561-387f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,191 - distributed.scheduler - INFO - Remove client Client-733d049f-3882-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,191 - distributed.scheduler - INFO - Remove client Client-5991845f-3881-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,190 - distributed.scheduler - INFO - Remove client Client-d9c61164-3883-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,190 - distributed.scheduler - INFO - Remove client Client-403c5ad1-3885-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,190 - distributed.scheduler - INFO - Remove client Client-a6b54bb8-3886-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,190 - distributed.scheduler - INFO - Remove client Client-0d33c9c2-3888-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,189 - distributed.scheduler - INFO - Remove client Client-da213a3d-388a-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,189 - distributed.scheduler - INFO - Remove client Client-db714b3d-3889-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,189 - distributed.scheduler - INFO - Remove client Client-40a1ae1d-388c-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,189 - distributed.scheduler - INFO - Remove client Client-a7259f84-388d-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,188 - distributed.scheduler - INFO - Remove client Client-0dabcd4a-388f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,188 - distributed.scheduler - INFO - Remove client Client-742c9d28-3890-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,187 - distributed.scheduler - INFO - Remove client Client-3020d18e-3892-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,187 - distributed.scheduler - INFO - Remove client Client-4132bff2-3893-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,187 - distributed.scheduler - INFO - Remove client Client-a7bf29a6-3894-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,186 - distributed.scheduler - INFO - Remove client Client-0e3d1e58-3896-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,186 - distributed.scheduler - INFO - Remove client Client-74c013f2-3897-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,186 - distributed.scheduler - INFO - Remove client Client-db3f28b9-3898-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,185 - distributed.scheduler - INFO - Remove client Client-92ad5bc6-389a-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,185 - distributed.scheduler - INFO - Remove client Client-8461c735-389b-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,184 - distributed.scheduler - INFO - Remove client Client-eaee4a1d-389c-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,184 - distributed.scheduler - INFO - Remove client Client-51726c7d-389e-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,184 - distributed.scheduler - INFO - Remove client Client-b7fa6fe3-389f-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,184 - distributed.scheduler - INFO - Remove client Client-1e81f26e-38a1-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,183 - distributed.scheduler - INFO - Remove client Client-f21862d4-38a2-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,183 - distributed.scheduler - INFO - Remove client Client-51fe0d90-38a5-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,183 - distributed.scheduler - INFO - Remove client Client-eb787e3d-38a3-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,183 - distributed.scheduler - INFO - Remove client Client-b87a5813-38a6-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,182 - distributed.scheduler - INFO - Remove client Client-85912867-38a9-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,182 - distributed.scheduler - INFO - Remove client Client-1f04e4e7-38a8-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,164 - distributed.scheduler - INFO - Remove client Client-529411b4-38ac-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,163 - distributed.scheduler - INFO - Remove client Client-4fe8afc1-38ab-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,163 - distributed.scheduler - INFO - Remove client Client-b91af5f7-38ad-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,162 - distributed.scheduler - INFO - Remove client Client-1faf71ad-38af-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,161 - distributed.scheduler - INFO - Remove client Client-862eef83-38b0-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,161 - distributed.scheduler - INFO - Remove client Client-b9b7e8da-38b4-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,161 - distributed.scheduler - INFO - Remove client Client-ecb25406-38b1-11f1-8001-daa6852bd5a7
2026-04-15 10:31:34,159 - distributed.scheduler - INFO - Remove client Client-a63dd4a1-38b3-11f1-8001-daa6852bd5a7
2026-04-15 10:31:33,170 - distributed.scheduler - INFO - Close client connection: Client-2033a1cd-38b6-11f1-8001-daa6852bd5a7
2026-04-15 10:31:33,168 - distributed.scheduler - INFO - Remove client Client-2033a1cd-38b6-11f1-8001-daa6852bd5a7
2026-04-15 10:31:33,167 - distributed.scheduler - INFO - Remove client Client-2033a1cd-38b6-11f1-8001-daa6852bd5a7
2026-04-15 10:30:30,102 - distributed.scheduler - INFO - Receive client connection: Client-2033a1cd-38b6-11f1-8001-daa6852bd5a7
2026-04-15 10:20:28,660 - distributed.scheduler - INFO - Receive client connection: Client-b9b7e8da-38b4-11f1-8001-daa6852bd5a7
2026-04-15 10:12:46,475 - distributed.scheduler - INFO - Receive client connection: Client-a63dd4a1-38b3-11f1-8001-daa6852bd5a7
2026-04-15 10:00:25,695 - distributed.scheduler - INFO - Receive client connection: Client-ecb25406-38b1-11f1-8001-daa6852bd5a7
2026-04-15 09:50:24,206 - distributed.scheduler - INFO - Receive client connection: Client-862eef83-38b0-11f1-8001-daa6852bd5a7
2026-04-15 09:40:22,745 - distributed.scheduler - INFO - Receive client connection: Client-1faf71ad-38af-11f1-8001-daa6852bd5a7
2026-04-15 09:30:21,151 - distributed.scheduler - INFO - Receive client connection: Client-b91af5f7-38ad-11f1-8001-daa6852bd5a7
2026-04-15 09:20:19,641 - distributed.scheduler - INFO - Receive client connection: Client-529411b4-38ac-11f1-8001-daa6852bd5a7
2026-04-15 09:13:05,665 - distributed.scheduler - INFO - Receive client connection: Client-4fe8afc1-38ab-11f1-8001-daa6852bd5a7
2026-04-15 09:00:16,696 - distributed.scheduler - INFO - Receive client connection: Client-85912867-38a9-11f1-8001-daa6852bd5a7
2026-04-15 08:50:15,154 - distributed.scheduler - INFO - Receive client connection: Client-1f04e4e7-38a8-11f1-8001-daa6852bd5a7
2026-04-15 08:40:13,616 - distributed.scheduler - INFO - Receive client connection: Client-b87a5813-38a6-11f1-8001-daa6852bd5a7
2026-04-15 08:30:16,117 - distributed.scheduler - INFO - Receive client connection: Client-worker-54581693-38a5-11f1-81f0-4ef3162b8ac2
2026-04-15 08:30:12,184 - distributed.scheduler - INFO - Receive client connection: Client-51fe0d90-38a5-11f1-8001-daa6852bd5a7
2026-04-15 08:20:10,675 - distributed.scheduler - INFO - Receive client connection: Client-eb787e3d-38a3-11f1-8001-daa6852bd5a7
2026-04-15 08:13:12,291 - distributed.scheduler - INFO - Receive client connection: Client-f21862d4-38a2-11f1-8001-daa6852bd5a7
2026-04-15 08:04:08,954 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:41455 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:41455 after 30 s
2026-04-15 08:04:08,917 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41455 after 30 s')
2026-04-15 08:04:08,915 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41455 after 30 s')
2026-04-15 08:04:08,910 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35361 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41455 after 30 s')
2026-04-15 08:04:08,909 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41455 after 30 s')
2026-04-15 08:04:08,908 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41455 after 30 s')
2026-04-15 08:04:08,906 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41455 after 30 s')
2026-04-15 08:04:08,882 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41455 after 30 s')
2026-04-15 08:04:08,878 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41455 after 30 s')
2026-04-15 08:04:08,877 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41455 after 30 s')
2026-04-15 08:04:08,865 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41455 after 30 s')
2026-04-15 08:03:39,974 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:44087
2026-04-15 08:03:39,972 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:44087', status: closed, memory: 0, processing: 0>
2026-04-15 08:03:38,728 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:41455', status: closed, memory: 0, processing: 0>
2026-04-15 08:03:24,100 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:45743 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:45743 after 30 s
2026-04-15 08:03:24,038 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45743 after 30 s')
2026-04-15 08:03:24,003 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45743 after 30 s')
2026-04-15 08:03:24,002 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45743 after 30 s')
2026-04-15 08:03:24,001 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45743 after 30 s')
2026-04-15 08:03:24,001 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:46541 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45743 after 30 s')
2026-04-15 08:03:24,001 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45743 after 30 s')
2026-04-15 08:03:24,000 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45743 after 30 s')
2026-04-15 08:03:23,970 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45743 after 30 s')
2026-04-15 08:03:23,965 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38115 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45743 after 30 s')
2026-04-15 08:03:23,957 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45743 after 30 s')
2026-04-15 08:03:23,947 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45743 after 30 s')
2026-04-15 08:02:54,829 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:41455
2026-04-15 08:02:54,827 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:41455', status: closed, memory: 0, processing: 0>
2026-04-15 08:02:53,831 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:45743', status: closed, memory: 0, processing: 0>
2026-04-15 08:02:37,562 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:41569 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:41569 after 30 s
2026-04-15 08:02:37,489 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:46541 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41569 after 30 s')
2026-04-15 08:02:37,488 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35361 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41569 after 30 s')
2026-04-15 08:02:37,486 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38115 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41569 after 30 s')
2026-04-15 08:02:37,477 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41569 after 30 s')
2026-04-15 08:02:37,476 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41569 after 30 s')
2026-04-15 08:02:37,454 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41569 after 30 s')
2026-04-15 08:02:37,448 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45643 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:41569 after 30 s')
2026-04-15 08:02:08,507 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:45743
2026-04-15 08:02:08,504 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:45743', status: closed, memory: 0, processing: 0>
2026-04-15 08:02:07,332 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:41569', status: closed, memory: 0, processing: 0>
2026-04-15 08:01:52,699 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39327 after 30 s')
2026-04-15 08:01:52,689 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:39327 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:39327 after 30 s
2026-04-15 08:01:52,481 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39327 after 30 s')
2026-04-15 08:01:22,396 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:41569
2026-04-15 08:01:22,394 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:41569', status: closed, memory: 0, processing: 0>
2026-04-15 08:01:21,520 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:39327', status: closed, memory: 0, processing: 0>
2026-04-15 08:00:47,136 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:39327
2026-04-15 08:00:47,135 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:39327', status: closed, memory: 0, processing: 0>
2026-04-15 08:00:46,114 - distributed.scheduler - INFO - Close client connection: Client-worker-b28f89aa-37fd-11f1-8119-4ef3162b8ac2
2026-04-15 08:00:46,111 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:39877', status: closed, memory: 0, processing: 0>
2026-04-15 08:00:46,110 - distributed.scheduler - INFO - Remove client Client-worker-b28f89aa-37fd-11f1-8119-4ef3162b8ac2
2026-04-15 08:00:07,812 - distributed.scheduler - INFO - Receive client connection: Client-1e81f26e-38a1-11f1-8001-daa6852bd5a7
2026-04-15 07:50:06,299 - distributed.scheduler - INFO - Receive client connection: Client-b7fa6fe3-389f-11f1-8001-daa6852bd5a7
2026-04-15 07:40:04,789 - distributed.scheduler - INFO - Receive client connection: Client-51726c7d-389e-11f1-8001-daa6852bd5a7
2026-04-15 07:30:03,296 - distributed.scheduler - INFO - Receive client connection: Client-eaee4a1d-389c-11f1-8001-daa6852bd5a7
2026-04-15 07:20:01,757 - distributed.scheduler - INFO - Receive client connection: Client-8461c735-389b-11f1-8001-daa6852bd5a7
2026-04-15 07:13:16,260 - distributed.scheduler - INFO - Receive client connection: Client-92ad5bc6-389a-11f1-8001-daa6852bd5a7
2026-04-15 07:00:58,997 - distributed.scheduler - INFO - Receive client connection: Client-db3f28b9-3898-11f1-8001-daa6852bd5a7
2026-04-15 06:50:57,544 - distributed.scheduler - INFO - Receive client connection: Client-74c013f2-3897-11f1-8001-daa6852bd5a7
2026-04-15 06:40:56,063 - distributed.scheduler - INFO - Receive client connection: Client-0e3d1e58-3896-11f1-8001-daa6852bd5a7
2026-04-15 06:30:54,613 - distributed.scheduler - INFO - Receive client connection: Client-a7bf29a6-3894-11f1-8001-daa6852bd5a7
2026-04-15 06:20:53,066 - distributed.scheduler - INFO - Receive client connection: Client-4132bff2-3893-11f1-8001-daa6852bd5a7
2026-04-15 06:13:14,925 - distributed.scheduler - INFO - Receive client connection: Client-3020d18e-3892-11f1-8001-daa6852bd5a7
2026-04-15 06:00:50,108 - distributed.scheduler - INFO - Receive client connection: Client-742c9d28-3890-11f1-8001-daa6852bd5a7
2026-04-15 05:50:48,626 - distributed.scheduler - INFO - Receive client connection: Client-0dabcd4a-388f-11f1-8001-daa6852bd5a7
2026-04-15 05:40:47,127 - distributed.scheduler - INFO - Receive client connection: Client-a7259f84-388d-11f1-8001-daa6852bd5a7
2026-04-15 05:30:45,641 - distributed.scheduler - INFO - Receive client connection: Client-40a1ae1d-388c-11f1-8001-daa6852bd5a7
2026-04-15 05:20:44,164 - distributed.scheduler - INFO - Receive client connection: Client-da213a3d-388a-11f1-8001-daa6852bd5a7
2026-04-15 05:13:36,869 - distributed.scheduler - INFO - Receive client connection: Client-db714b3d-3889-11f1-8001-daa6852bd5a7
2026-04-15 05:00:41,362 - distributed.scheduler - INFO - Receive client connection: Client-0d33c9c2-3888-11f1-8001-daa6852bd5a7
2026-04-15 04:50:39,914 - distributed.scheduler - INFO - Receive client connection: Client-a6b54bb8-3886-11f1-8001-daa6852bd5a7
2026-04-15 04:40:38,490 - distributed.scheduler - INFO - Receive client connection: Client-403c5ad1-3885-11f1-8001-daa6852bd5a7
2026-04-15 04:30:37,098 - distributed.scheduler - INFO - Receive client connection: Client-d9c61164-3883-11f1-8001-daa6852bd5a7
2026-04-15 04:20:35,577 - distributed.scheduler - INFO - Receive client connection: Client-733d049f-3882-11f1-8001-daa6852bd5a7
2026-04-15 04:12:43,003 - distributed.scheduler - INFO - Receive client connection: Client-5991845f-3881-11f1-8001-daa6852bd5a7
2026-04-15 04:00:32,513 - distributed.scheduler - INFO - Receive client connection: Client-a6289561-387f-11f1-8001-daa6852bd5a7
2026-04-15 03:50:31,073 - distributed.scheduler - INFO - Receive client connection: Client-3fa56a30-387e-11f1-8001-daa6852bd5a7
2026-04-15 03:40:29,518 - distributed.scheduler - INFO - Receive client connection: Client-d91e49d6-387c-11f1-8001-daa6852bd5a7
2026-04-15 03:30:28,074 - distributed.scheduler - INFO - Receive client connection: Client-72a1a4df-387b-11f1-8001-daa6852bd5a7
2026-04-15 03:20:26,600 - distributed.scheduler - INFO - Receive client connection: Client-0c1e83c7-387a-11f1-8001-daa6852bd5a7
2026-04-15 03:13:04,574 - distributed.scheduler - INFO - Receive client connection: Client-04a8d8e4-3879-11f1-8001-daa6852bd5a7
2026-04-15 03:00:23,576 - distributed.scheduler - INFO - Receive client connection: Client-3f0fbac9-3877-11f1-8001-daa6852bd5a7
2026-04-15 02:50:22,022 - distributed.scheduler - INFO - Receive client connection: Client-d8824ad1-3875-11f1-8001-daa6852bd5a7
2026-04-15 02:40:20,471 - distributed.scheduler - INFO - Receive client connection: Client-71f55b38-3874-11f1-8001-daa6852bd5a7
2026-04-15 02:30:18,981 - distributed.scheduler - INFO - Receive client connection: Client-0b722fc7-3873-11f1-8001-daa6852bd5a7
2026-04-15 02:20:17,441 - distributed.scheduler - INFO - Receive client connection: Client-a4e6f003-3871-11f1-8001-daa6852bd5a7
2026-04-15 02:12:45,972 - distributed.scheduler - INFO - Receive client connection: Client-97cdb65b-3870-11f1-8001-daa6852bd5a7
2026-04-15 02:00:14,264 - distributed.scheduler - INFO - Receive client connection: Client-d7bec258-386e-11f1-8001-daa6852bd5a7
2026-04-15 01:50:12,756 - distributed.scheduler - INFO - Receive client connection: Client-71397551-386d-11f1-8001-daa6852bd5a7
2026-04-15 01:40:11,271 - distributed.scheduler - INFO - Receive client connection: Client-0ab56c8d-386c-11f1-8001-daa6852bd5a7
2026-04-15 01:30:09,782 - distributed.scheduler - INFO - Receive client connection: Client-a431e1c4-386a-11f1-8001-daa6852bd5a7
2026-04-15 01:20:08,331 - distributed.scheduler - INFO - Receive client connection: Client-3db38583-3869-11f1-8001-daa6852bd5a7
2026-04-15 01:11:47,843 - distributed.scheduler - INFO - Receive client connection: Client-1363fc7c-3868-11f1-8001-daa6852bd5a7
2026-04-15 01:00:05,399 - distributed.scheduler - INFO - Receive client connection: Client-70b201d5-3866-11f1-8001-daa6852bd5a7
2026-04-15 00:50:03,893 - distributed.scheduler - INFO - Receive client connection: Client-0a2c705d-3865-11f1-8001-daa6852bd5a7
2026-04-15 00:40:02,389 - distributed.scheduler - INFO - Receive client connection: Client-a3a5a31b-3863-11f1-8001-daa6852bd5a7
2026-04-15 00:30:00,865 - distributed.scheduler - INFO - Receive client connection: Client-3d1ce715-3862-11f1-8001-daa6852bd5a7
2026-04-15 00:20:59,611 - distributed.scheduler - INFO - Receive client connection: Client-fa786c8f-3860-11f1-8001-daa6852bd5a7
2026-04-15 00:12:15,660 - distributed.scheduler - INFO - Receive client connection: Client-c23415d6-385f-11f1-8001-daa6852bd5a7
2026-04-15 00:00:56,642 - distributed.scheduler - INFO - Receive client connection: Client-2d793d4a-385e-11f1-8001-daa6852bd5a7
2026-04-14 23:50:55,097 - distributed.scheduler - INFO - Receive client connection: Client-c6ec5411-385c-11f1-8001-daa6852bd5a7
2026-04-14 23:40:53,515 - distributed.scheduler - INFO - Receive client connection: Client-605a4821-385b-11f1-8001-daa6852bd5a7
2026-04-14 23:30:52,065 - distributed.scheduler - INFO - Receive client connection: Client-f9dc41a3-3859-11f1-8001-daa6852bd5a7
2026-04-14 23:20:50,578 - distributed.scheduler - INFO - Receive client connection: Client-9358d19c-3858-11f1-8001-daa6852bd5a7
2026-04-14 23:12:58,877 - distributed.scheduler - INFO - Receive client connection: Client-7a318f9e-3857-11f1-8001-daa6852bd5a7
2026-04-14 23:00:47,643 - distributed.scheduler - INFO - Receive client connection: Client-c6570749-3855-11f1-8001-daa6852bd5a7
2026-04-14 22:50:46,161 - distributed.scheduler - INFO - Receive client connection: Client-5fd5f50c-3854-11f1-8001-daa6852bd5a7
2026-04-14 22:40:44,654 - distributed.scheduler - INFO - Receive client connection: Client-f94d4b32-3852-11f1-8001-daa6852bd5a7
2026-04-14 22:30:43,053 - distributed.scheduler - INFO - Receive client connection: Client-92b8cf85-3851-11f1-8001-daa6852bd5a7
2026-04-14 22:20:41,590 - distributed.scheduler - INFO - Receive client connection: Client-2c391284-3850-11f1-8001-daa6852bd5a7
2026-04-14 22:12:21,584 - distributed.scheduler - INFO - Receive client connection: Client-02322f3a-384f-11f1-8001-daa6852bd5a7
2026-04-14 22:00:38,641 - distributed.scheduler - INFO - Receive client connection: Client-5f351503-384d-11f1-8001-daa6852bd5a7
2026-04-14 21:50:37,136 - distributed.scheduler - INFO - Receive client connection: Client-f8af0f95-384b-11f1-8001-daa6852bd5a7
2026-04-14 21:40:35,609 - distributed.scheduler - INFO - Receive client connection: Client-9225b54e-384a-11f1-8001-daa6852bd5a7
2026-04-14 21:30:34,083 - distributed.scheduler - INFO - Receive client connection: Client-2b9b6495-3849-11f1-8001-daa6852bd5a7
2026-04-14 21:20:32,574 - distributed.scheduler - INFO - Receive client connection: Client-c513e133-3847-11f1-8001-daa6852bd5a7
2026-04-14 21:12:39,335 - distributed.scheduler - INFO - Receive client connection: Client-ab033046-3846-11f1-8001-daa6852bd5a7
2026-04-14 21:00:36,251 - distributed.scheduler - ERROR - Shut down workers that don't have promised key: [], ('len-agg-7587135aabed7bd00f8659d913314f1e', 0) NoneType: None
2026-04-14 21:00:36,251 - distributed.scheduler - ERROR - Couldn't gather keys {"('len-agg-7587135aabed7bd00f8659d913314f1e', 0)": []} state: ['waiting'] workers: [] NoneType: None
2026-04-14 21:00:29,535 - distributed.scheduler - INFO - Receive client connection: Client-f80392b1-3844-11f1-8001-daa6852bd5a7
2026-04-14 20:50:28,063 - distributed.scheduler - INFO - Receive client connection: Client-918223d1-3843-11f1-8001-daa6852bd5a7
2026-04-14 20:40:26,567 - distributed.scheduler - INFO - Receive client connection: Client-2afd3657-3842-11f1-8001-daa6852bd5a7
2026-04-14 20:30:25,078 - distributed.scheduler - INFO - Receive client connection: Client-c4788420-3840-11f1-8001-daa6852bd5a7
2026-04-14 20:20:23,551 - distributed.scheduler - INFO - Receive client connection: Client-5def6098-383f-11f1-8001-daa6852bd5a7
2026-04-14 20:12:29,883 - distributed.scheduler - INFO - Receive client connection: Client-439c65af-383e-11f1-8001-daa6852bd5a7
2026-04-14 20:00:20,505 - distributed.scheduler - INFO - Receive client connection: Client-90ddab17-383c-11f1-8001-daa6852bd5a7
2026-04-14 19:50:19,039 - distributed.scheduler - INFO - Receive client connection: Client-2a5dcb57-383b-11f1-8001-daa6852bd5a7
2026-04-14 19:40:17,546 - distributed.scheduler - INFO - Receive client connection: Client-c3d8728b-3839-11f1-8001-daa6852bd5a7
2026-04-14 19:30:15,990 - distributed.scheduler - INFO - Receive client connection: Client-5d4a4003-3838-11f1-8001-daa6852bd5a7
2026-04-14 19:20:14,379 - distributed.scheduler - INFO - Receive client connection: Client-f6b3bec3-3836-11f1-8001-daa6852bd5a7
2026-04-14 19:11:56,935 - distributed.scheduler - INFO - Receive client connection: Client-ce34b908-3835-11f1-8001-daa6852bd5a7
2026-04-14 19:00:11,356 - distributed.scheduler - INFO - Receive client connection: Client-29a56898-3834-11f1-8001-daa6852bd5a7
2026-04-14 18:50:09,921 - distributed.scheduler - INFO - Receive client connection: Client-c329a259-3832-11f1-8001-daa6852bd5a7
2026-04-14 18:40:08,512 - distributed.scheduler - INFO - Receive client connection: Client-5cb2172f-3831-11f1-8001-daa6852bd5a7
2026-04-14 18:30:07,053 - distributed.scheduler - INFO - Receive client connection: Client-f632470f-382f-11f1-8001-daa6852bd5a7
2026-04-14 18:20:05,504 - distributed.scheduler - INFO - Receive client connection: Client-8fa51cf1-382e-11f1-8001-daa6852bd5a7
2026-04-14 18:11:51,545 - distributed.scheduler - INFO - Receive client connection: Client-6939a497-382d-11f1-8001-daa6852bd5a7
2026-04-14 18:00:02,440 - distributed.scheduler - INFO - Receive client connection: Client-c28ec97f-382b-11f1-8001-daa6852bd5a7
2026-04-14 17:50:39,615 - distributed.scheduler - INFO - Close client connection: Client-5c0a6497-382a-11f1-8001-daa6852bd5a7
2026-04-14 17:50:39,613 - distributed.scheduler - INFO - Remove client Client-5c0a6497-382a-11f1-8001-daa6852bd5a7
2026-04-14 17:50:39,612 - distributed.scheduler - INFO - Remove client Client-5c0a6497-382a-11f1-8001-daa6852bd5a7
2026-04-14 17:50:00,947 - distributed.scheduler - INFO - Receive client connection: Client-5c0a6497-382a-11f1-8001-daa6852bd5a7
2026-04-14 17:41:36,823 - distributed.scheduler - INFO - Close client connection: Client-1964e1a3-3829-11f1-8001-daa6852bd5a7
2026-04-14 17:41:36,822 - distributed.scheduler - INFO - Remove client Client-1964e1a3-3829-11f1-8001-daa6852bd5a7
2026-04-14 17:41:36,821 - distributed.scheduler - INFO - Remove client Client-1964e1a3-3829-11f1-8001-daa6852bd5a7
2026-04-14 17:40:59,639 - distributed.scheduler - INFO - Receive client connection: Client-1964e1a3-3829-11f1-8001-daa6852bd5a7
2026-04-14 17:32:08,772 - distributed.scheduler - INFO - Close client connection: Client-b2ec7181-3827-11f1-8001-daa6852bd5a7
2026-04-14 17:32:08,772 - distributed.scheduler - INFO - Remove client Client-b2ec7181-3827-11f1-8001-daa6852bd5a7
2026-04-14 17:32:08,771 - distributed.scheduler - INFO - Remove client Client-b2ec7181-3827-11f1-8001-daa6852bd5a7
2026-04-14 17:30:58,220 - distributed.scheduler - INFO - Receive client connection: Client-b2ec7181-3827-11f1-8001-daa6852bd5a7
2026-04-14 17:21:35,313 - distributed.scheduler - INFO - Close client connection: Client-4c65f980-3826-11f1-8001-daa6852bd5a7
2026-04-14 17:21:35,312 - distributed.scheduler - INFO - Remove client Client-4c65f980-3826-11f1-8001-daa6852bd5a7
2026-04-14 17:21:35,310 - distributed.scheduler - INFO - Remove client Client-4c65f980-3826-11f1-8001-daa6852bd5a7
2026-04-14 17:20:56,717 - distributed.scheduler - INFO - Receive client connection: Client-4c65f980-3826-11f1-8001-daa6852bd5a7
2026-04-14 17:12:49,999 - distributed.scheduler - INFO - Close client connection: Client-15cada7d-3825-11f1-8001-daa6852bd5a7
2026-04-14 17:12:49,998 - distributed.scheduler - INFO - Remove client Client-15cada7d-3825-11f1-8001-daa6852bd5a7
2026-04-14 17:12:49,996 - distributed.scheduler - INFO - Remove client Client-15cada7d-3825-11f1-8001-daa6852bd5a7
2026-04-14 17:12:15,609 - distributed.scheduler - INFO - Receive client connection: Client-15cada7d-3825-11f1-8001-daa6852bd5a7
2026-04-14 17:02:10,818 - distributed.scheduler - INFO - Close client connection: Client-7f559ec6-3823-11f1-8001-daa6852bd5a7
2026-04-14 17:02:10,817 - distributed.scheduler - INFO - Remove client Client-7f559ec6-3823-11f1-8001-daa6852bd5a7
2026-04-14 17:02:10,815 - distributed.scheduler - INFO - Remove client Client-7f559ec6-3823-11f1-8001-daa6852bd5a7
2026-04-14 17:00:53,686 - distributed.scheduler - INFO - Receive client connection: Client-7f559ec6-3823-11f1-8001-daa6852bd5a7
2026-04-14 16:51:32,382 - distributed.scheduler - INFO - Close client connection: Client-18d5422e-3822-11f1-8001-daa6852bd5a7
2026-04-14 16:51:32,381 - distributed.scheduler - INFO - Remove client Client-18d5422e-3822-11f1-8001-daa6852bd5a7
2026-04-14 16:51:32,380 - distributed.scheduler - INFO - Remove client Client-18d5422e-3822-11f1-8001-daa6852bd5a7
2026-04-14 16:50:52,217 - distributed.scheduler - INFO - Receive client connection: Client-18d5422e-3822-11f1-8001-daa6852bd5a7
2026-04-14 16:41:29,304 - distributed.scheduler - INFO - Close client connection: Client-b25bba70-3820-11f1-8001-daa6852bd5a7
2026-04-14 16:41:29,303 - distributed.scheduler - INFO - Remove client Client-b25bba70-3820-11f1-8001-daa6852bd5a7
2026-04-14 16:41:29,302 - distributed.scheduler - INFO - Remove client Client-b25bba70-3820-11f1-8001-daa6852bd5a7
2026-04-14 16:40:50,794 - distributed.scheduler - INFO - Receive client connection: Client-b25bba70-3820-11f1-8001-daa6852bd5a7
2026-04-14 16:32:01,145 - distributed.scheduler - INFO - Close client connection: Client-4bddae5a-381f-11f1-8001-daa6852bd5a7
2026-04-14 16:32:01,144 - distributed.scheduler - INFO - Remove client Client-4bddae5a-381f-11f1-8001-daa6852bd5a7
2026-04-14 16:32:01,142 - distributed.scheduler - INFO - Remove client Client-4bddae5a-381f-11f1-8001-daa6852bd5a7
2026-04-14 16:30:49,337 - distributed.scheduler - INFO - Receive client connection: Client-4bddae5a-381f-11f1-8001-daa6852bd5a7
2026-04-14 16:21:26,699 - distributed.scheduler - INFO - Close client connection: Client-e55873f1-381d-11f1-8001-daa6852bd5a7
2026-04-14 16:21:26,697 - distributed.scheduler - INFO - Remove client Client-e55873f1-381d-11f1-8001-daa6852bd5a7
2026-04-14 16:21:26,695 - distributed.scheduler - INFO - Remove client Client-e55873f1-381d-11f1-8001-daa6852bd5a7
2026-04-14 16:20:47,848 - distributed.scheduler - INFO - Receive client connection: Client-e55873f1-381d-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,466 - distributed.scheduler - INFO - Close client connection: Client-7cb25b36-3800-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,466 - distributed.scheduler - INFO - Close client connection: Client-e3347bcf-3801-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,466 - distributed.scheduler - INFO - Close client connection: Client-6ff589a7-3803-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,466 - distributed.scheduler - INFO - Close client connection: Client-b036f31a-3804-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,466 - distributed.scheduler - INFO - Close client connection: Client-16c5261b-3806-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,465 - distributed.scheduler - INFO - Close client connection: Client-7d4a184d-3807-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,465 - distributed.scheduler - INFO - Close client connection: Client-e3d232a5-3808-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,465 - distributed.scheduler - INFO - Close client connection: Client-4a4a6b2d-380a-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,465 - distributed.scheduler - INFO - Close client connection: Client-e31a1677-380b-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,465 - distributed.scheduler - INFO - Close client connection: Client-174e03c6-380d-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,465 - distributed.scheduler - INFO - Close client connection: Client-4ad008f2-3811-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,464 - distributed.scheduler - INFO - Close client connection: Client-7dc65990-380e-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,464 - distributed.scheduler - INFO - Close client connection: Client-e447101f-380f-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,464 - distributed.scheduler - INFO - Close client connection: Client-b159ab04-3812-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,463 - distributed.scheduler - INFO - Close client connection: Client-7e5e1d1f-3815-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,463 - distributed.scheduler - INFO - Close client connection: Client-32c0ced2-3814-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,463 - distributed.scheduler - INFO - Close client connection: Client-e4d49e60-3816-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,463 - distributed.scheduler - INFO - Close client connection: Client-4b560ba5-3818-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,463 - distributed.scheduler - INFO - Close client connection: Client-b1d55690-3819-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,462 - distributed.scheduler - INFO - Close client connection: Client-1854da6c-381b-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,461 - distributed.scheduler - INFO - Remove client Client-7cb25b36-3800-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,460 - distributed.scheduler - INFO - Remove client Client-e3347bcf-3801-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,460 - distributed.scheduler - INFO - Remove client Client-6ff589a7-3803-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,460 - distributed.scheduler - INFO - Remove client Client-b036f31a-3804-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,459 - distributed.scheduler - INFO - Remove client Client-16c5261b-3806-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,458 - distributed.scheduler - INFO - Remove client Client-7d4a184d-3807-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,458 - distributed.scheduler - INFO - Remove client Client-e3d232a5-3808-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,458 - distributed.scheduler - INFO - Remove client Client-4a4a6b2d-380a-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,457 - distributed.scheduler - INFO - Remove client Client-e31a1677-380b-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,456 - distributed.scheduler - INFO - Remove client Client-174e03c6-380d-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,456 - distributed.scheduler - INFO - Remove client Client-4ad008f2-3811-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,455 - distributed.scheduler - INFO - Remove client Client-7dc65990-380e-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,454 - distributed.scheduler - INFO - Remove client Client-e447101f-380f-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,453 - distributed.scheduler - INFO - Remove client Client-b159ab04-3812-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,446 - distributed.scheduler - INFO - Remove client Client-7e5e1d1f-3815-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,446 - distributed.scheduler - INFO - Remove client Client-32c0ced2-3814-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,445 - distributed.scheduler - INFO - Remove client Client-e4d49e60-3816-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,444 - distributed.scheduler - INFO - Remove client Client-4b560ba5-3818-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,443 - distributed.scheduler - INFO - Remove client Client-b1d55690-3819-11f1-8001-daa6852bd5a7
2026-04-14 16:12:22,443 - distributed.scheduler - INFO - Remove client Client-1854da6c-381b-11f1-8001-daa6852bd5a7
2026-04-14 16:12:21,754 - distributed.scheduler - INFO - Close client connection: Client-a08541f8-381c-11f1-8001-daa6852bd5a7
2026-04-14 16:12:21,748 - distributed.scheduler - INFO - Remove client Client-a08541f8-381c-11f1-8001-daa6852bd5a7
2026-04-14 16:12:21,746 - distributed.scheduler - INFO - Remove client Client-a08541f8-381c-11f1-8001-daa6852bd5a7
2026-04-14 16:11:42,870 - distributed.scheduler - INFO - Receive client connection: Client-a08541f8-381c-11f1-8001-daa6852bd5a7
2026-04-14 16:00:44,894 - distributed.scheduler - INFO - Receive client connection: Client-1854da6c-381b-11f1-8001-daa6852bd5a7
2026-04-14 15:50:43,430 - distributed.scheduler - INFO - Receive client connection: Client-b1d55690-3819-11f1-8001-daa6852bd5a7
2026-04-14 15:40:41,967 - distributed.scheduler - INFO - Receive client connection: Client-4b560ba5-3818-11f1-8001-daa6852bd5a7
2026-04-14 15:30:40,493 - distributed.scheduler - INFO - Receive client connection: Client-e4d49e60-3816-11f1-8001-daa6852bd5a7
2026-04-14 15:20:39,096 - distributed.scheduler - INFO - Receive client connection: Client-7e5e1d1f-3815-11f1-8001-daa6852bd5a7
2026-04-14 15:11:22,737 - distributed.scheduler - INFO - Receive client connection: Client-32c0ced2-3814-11f1-8001-daa6852bd5a7
2026-04-14 15:00:36,146 - distributed.scheduler - INFO - Receive client connection: Client-b159ab04-3812-11f1-8001-daa6852bd5a7
2026-04-14 14:50:34,616 - distributed.scheduler - INFO - Receive client connection: Client-4ad008f2-3811-11f1-8001-daa6852bd5a7
2026-04-14 14:40:33,095 - distributed.scheduler - INFO - Receive client connection: Client-e447101f-380f-11f1-8001-daa6852bd5a7
2026-04-14 14:30:31,627 - distributed.scheduler - INFO - Receive client connection: Client-7dc65990-380e-11f1-8001-daa6852bd5a7
2026-04-14 14:20:30,212 - distributed.scheduler - INFO - Receive client connection: Client-174e03c6-380d-11f1-8001-daa6852bd5a7
2026-04-14 14:11:53,133 - distributed.scheduler - INFO - Receive client connection: Client-e31a1677-380b-11f1-8001-daa6852bd5a7
2026-04-14 14:00:27,266 - distributed.scheduler - INFO - Receive client connection: Client-4a4a6b2d-380a-11f1-8001-daa6852bd5a7
2026-04-14 13:50:25,851 - distributed.scheduler - INFO - Receive client connection: Client-e3d232a5-3808-11f1-8001-daa6852bd5a7
2026-04-14 13:40:24,335 - distributed.scheduler - INFO - Receive client connection: Client-7d4a184d-3807-11f1-8001-daa6852bd5a7
2026-04-14 13:30:22,840 - distributed.scheduler - INFO - Receive client connection: Client-16c5261b-3806-11f1-8001-daa6852bd5a7
2026-04-14 13:20:21,283 - distributed.scheduler - INFO - Receive client connection: Client-b036f31a-3804-11f1-8001-daa6852bd5a7
2026-04-14 13:11:23,981 - distributed.scheduler - INFO - Receive client connection: Client-6ff589a7-3803-11f1-8001-daa6852bd5a7
2026-04-14 13:00:18,342 - distributed.scheduler - INFO - Receive client connection: Client-e3347bcf-3801-11f1-8001-daa6852bd5a7
2026-04-14 12:50:16,865 - distributed.scheduler - INFO - Receive client connection: Client-7cb25b36-3800-11f1-8001-daa6852bd5a7
2026-04-14 12:40:50,213 - distributed.scheduler - INFO - Close client connection: Client-1633449f-37ff-11f1-8001-daa6852bd5a7
2026-04-14 12:40:50,211 - distributed.scheduler - INFO - Remove client Client-1633449f-37ff-11f1-8001-daa6852bd5a7
2026-04-14 12:40:50,210 - distributed.scheduler - INFO - Remove client Client-1633449f-37ff-11f1-8001-daa6852bd5a7
2026-04-14 12:40:15,415 - distributed.scheduler - INFO - Receive client connection: Client-1633449f-37ff-11f1-8001-daa6852bd5a7
2026-04-14 12:31:22,410 - distributed.scheduler - INFO - Close client connection: Client-afb042ed-37fd-11f1-8001-daa6852bd5a7
2026-04-14 12:31:22,409 - distributed.scheduler - INFO - Remove client Client-afb042ed-37fd-11f1-8001-daa6852bd5a7
2026-04-14 12:31:22,408 - distributed.scheduler - INFO - Remove client Client-afb042ed-37fd-11f1-8001-daa6852bd5a7
2026-04-14 12:30:18,794 - distributed.scheduler - INFO - Receive client connection: Client-worker-b28f89aa-37fd-11f1-8119-4ef3162b8ac2
2026-04-14 12:30:13,936 - distributed.scheduler - INFO - Receive client connection: Client-afb042ed-37fd-11f1-8001-daa6852bd5a7
2026-04-14 12:20:49,144 - distributed.scheduler - INFO - Close client connection: Client-4929bf0e-37fc-11f1-8001-daa6852bd5a7
2026-04-14 12:20:49,142 - distributed.scheduler - INFO - Remove client Client-4929bf0e-37fc-11f1-8001-daa6852bd5a7
2026-04-14 12:20:49,142 - distributed.scheduler - INFO - Remove client Client-4929bf0e-37fc-11f1-8001-daa6852bd5a7
2026-04-14 12:20:12,426 - distributed.scheduler - INFO - Receive client connection: Client-4929bf0e-37fc-11f1-8001-daa6852bd5a7
2026-04-14 12:11:52,647 - distributed.scheduler - INFO - Close client connection: Client-0deb7951-37fb-11f1-8001-daa6852bd5a7
2026-04-14 12:11:52,646 - distributed.scheduler - INFO - Remove client Client-0deb7951-37fb-11f1-8001-daa6852bd5a7
2026-04-14 12:11:52,645 - distributed.scheduler - INFO - Remove client Client-0deb7951-37fb-11f1-8001-daa6852bd5a7
2026-04-14 12:11:23,535 - distributed.scheduler - INFO - Receive client connection: Client-0deb7951-37fb-11f1-8001-daa6852bd5a7
2026-04-14 12:01:15,787 - distributed.scheduler - INFO - Close client connection: Client-7c1ff876-37f9-11f1-8001-daa6852bd5a7
2026-04-14 12:01:15,784 - distributed.scheduler - INFO - Remove client Client-7c1ff876-37f9-11f1-8001-daa6852bd5a7
2026-04-14 12:01:15,783 - distributed.scheduler - INFO - Remove client Client-7c1ff876-37f9-11f1-8001-daa6852bd5a7
2026-04-14 12:00:09,438 - distributed.scheduler - INFO - Receive client connection: Client-7c1ff876-37f9-11f1-8001-daa6852bd5a7
2026-04-14 11:50:41,493 - distributed.scheduler - INFO - Close client connection: Client-15a27275-37f8-11f1-8001-daa6852bd5a7
2026-04-14 11:50:41,493 - distributed.scheduler - INFO - Remove client Client-15a27275-37f8-11f1-8001-daa6852bd5a7
2026-04-14 11:50:41,492 - distributed.scheduler - INFO - Remove client Client-15a27275-37f8-11f1-8001-daa6852bd5a7
2026-04-14 11:50:07,987 - distributed.scheduler - INFO - Receive client connection: Client-15a27275-37f8-11f1-8001-daa6852bd5a7
2026-04-14 11:40:39,270 - distributed.scheduler - INFO - Close client connection: Client-af1f40ee-37f6-11f1-8001-daa6852bd5a7
2026-04-14 11:40:39,266 - distributed.scheduler - INFO - Remove client Client-af1f40ee-37f6-11f1-8001-daa6852bd5a7
2026-04-14 11:40:39,265 - distributed.scheduler - INFO - Remove client Client-af1f40ee-37f6-11f1-8001-daa6852bd5a7
2026-04-14 11:40:06,507 - distributed.scheduler - INFO - Receive client connection: Client-af1f40ee-37f6-11f1-8001-daa6852bd5a7
2026-04-14 11:34:15,256 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:35679 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:35679 after 30 s
2026-04-14 11:34:15,163 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:35679 after 30 s')
2026-04-14 11:34:15,162 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:35679 after 30 s')
2026-04-14 11:34:15,157 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:35679 after 30 s')
2026-04-14 11:34:15,155 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:35679 after 30 s')
2026-04-14 11:34:15,155 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:35679 after 30 s')
2026-04-14 11:34:15,153 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:35679 after 30 s')
2026-04-14 11:34:15,153 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:35679 after 30 s')
2026-04-14 11:34:15,151 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35361 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:35679 after 30 s')
2026-04-14 11:34:15,140 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38115 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:35679 after 30 s')
2026-04-14 11:33:46,000 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:39877
2026-04-14 11:33:45,998 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:39877', status: closed, memory: 0, processing: 0>
2026-04-14 11:33:44,990 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:35679', status: closed, memory: 0, processing: 0>
2026-04-14 11:33:34,639 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:39445 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:39445 after 30 s
2026-04-14 11:33:34,573 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39445 after 30 s')
2026-04-14 11:33:34,569 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39445 after 30 s')
2026-04-14 11:33:34,568 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39445 after 30 s')
2026-04-14 11:33:34,568 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39445 after 30 s')
2026-04-14 11:33:34,565 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39445 after 30 s')
2026-04-14 11:33:34,564 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.222:34465 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39445 after 30 s')
2026-04-14 11:33:34,564 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:46541 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39445 after 30 s')
2026-04-14 11:33:34,564 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35361 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39445 after 30 s')
2026-04-14 11:33:34,544 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38115 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39445 after 30 s')
2026-04-14 11:33:34,542 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45643 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39445 after 30 s')
2026-04-14 11:33:34,537 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:39445 after 30 s')
2026-04-14 11:33:05,332 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:35679
2026-04-14 11:33:05,330 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:35679', status: closed, memory: 0, processing: 0>
2026-04-14 11:33:04,392 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:39445', status: closed, memory: 0, processing: 0>
2026-04-14 11:32:54,071 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:46007 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s
2026-04-14 11:32:54,055 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,993 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.197:35361 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,991 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.32:38115 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,990 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,990 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.246:45399 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,960 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.230:44659 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,948 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.253:38983 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,947 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,941 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.214:46843 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,940 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.207:46541 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,930 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.34:45643 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,929 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.29:37393 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,928 - distributed.scheduler - ERROR - broadcast to tcp://172.21.159.225:32939 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:53,918 - distributed.scheduler - ERROR - broadcast to tcp://172.21.25.54:37809 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:46007 after 30 s')
2026-04-14 11:32:24,694 - distributed.scheduler - INFO - Starting worker compute stream, tcp://172.21.151.183:39445
2026-04-14 11:32:24,692 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://172.21.151.183:39445', status: closed, memory: 0, processing: 0>
2026-04-14 11:32:23,801 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://172.21.151.183:46007', status: closed, memory: 0, processing: 0>
2026-04-14 11:32:00,065 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.183:45361 failed: OSError: Timed out trying to connect to tcp://172.21.151.183:45361 after 30 s
2026-04-14 11:32:00,059 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.150:40179 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45361 after 30 s')
2026-04-14 11:31:59,985 - distributed.scheduler - ERROR - broadcast to tcp://172.21.160.14:39341 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45361 after 30 s')
2026-04-14 11:31:59,984 - distributed.scheduler - ERROR - broadcast to tcp://172.21.151.168:35221 failed: Exception: OSError('Timed out trying to connect to tcp://172.21.151.183:45361 after 30 s')