2026-01-19 15:02:38,669 - distributed.worker - WARNING - Compute Failed Key: ('shuffle-transfer-dd0235e2b443afff522d4ae5c408aeda', 8) Function: shuffle_transfer args: ( schema_version detector_id ... heading_diff_cur_and_next _partitions 271 1 2794848503 ... -0.3 6 271 1 2794848459 ... 0.3 4 271 1 2794848489 ... -0.1 3 271 1 2794848489 ... 0.3 10 271 1 2794848483 ... 0.2 10 .. ... ... ... ... ... 304 1 2794848489 ... 0.0 10 304 1 2794848516 ... 0.0 3 304 1 2794848417 ... -0.1 0 304 1 2794848449 ... -0.1 7 304 1 2794848516 ... 0.0 0 [544 rows x 21 columns], 'dd0235e2b443afff522d4ae5c408aeda', 8, 16, '_part kwargs: {} Exception: "RuntimeError('shuffle_transfer failed during shuffle dd0235e2b443afff522d4ae5c408aeda')"

2026-01-19 15:02:23,049 - distributed.worker - WARNING - Compute Failed Key: ('shuffle-transfer-6f75e65e733bc182a533b90edd3d68d0', 12) Function: shuffle_transfer args: ( source_id timestamp ... date_diff_ms _partitions 0 2794848523 1768834310820 ... 200.0 7 1 2794848523 1768834313052 ... 195.0 7 2 2794848614 1768834314864 ... 827.0 6 3 2794848614 1768834315272 ... 408.0 6 4 2794848523 1768834315539 ... 248.0 7 ... ... ... ... ... ... 2071205 2794848523 1768833618055 ... 243.0 7 2071206 2794848523 1768833618831 ... 254.0 7 2071207 2794848523 1768833620630 ... 305.0 7 2071208 2794848523 1768833628768 ... 307.0 7 2071209 2794848523 1768833635487 ... 249.0 7 [2071210 rows x 5 columns], '6f75e65e733bc182a533b90edd3d68d0', 12, 16, '_partitions') kwargs: {} Exception: "RuntimeError('shuffle_transfer failed during shuffle 6f75e65e733bc182a533b90edd3d68d0')"

2026-01-19 15:02:02,395 - distributed.worker - WARNING - Compute Failed Key: ('shuffle-transfer-6bc3d86cb004340f35e7e3f39bdfd805', 8) Function: shuffle_transfer args: ( source_id timestamp ... date_diff_ms _partitions 0 2794848524 1768833348290 ... 570.0 7 1 2794848524 1768833350783 ... 508.0 7 2 2794848523 1768833352218 ... 204.0 7 3 2794848524 1768833353544 ... 515.0 7 4 2794848524 1768833357776 ... 462.0 7 ... ... ... ... ... ... 1686557 2794848523 1768833866289 ... 155.0 7 1686558 2794848523 1768833866443 ... 154.0 7 1686559 2794848523 1768833866993 ... 121.0 7 1686560 2794848523 1768833872847 ... 196.0 7 1686561 2794848523 1768833878790 ... 409.0 7 [1686562 rows x 5 columns], '6bc3d86cb004340f35e7e3f39bdfd805', 8, 16, '_partitions') kwargs: {} Exception: "RuntimeError('shuffle_transfer failed during shuffle 6bc3d86cb004340f35e7e3f39bdfd805')"

2026-01-19 03:40:14,593 - distributed.worker - WARNING - Compute Failed Key: ('assign-4d092cfcbcdc03d5275e9c9561b8f02b', 12) Function: subgraph_callable-0c6590eb-9d13-4a73-9cf1-d23d7186 args: ( source_id geometry 968 2794848613 POINT Z (446251.476 5897838.915 0.000)) kwargs: {} Exception: "GEOSException('IllegalArgumentException: point array must contain 0 or >1 elements\\n')"

2026-01-18 10:30:59,288 - distributed.worker - ERROR - Worker stream died during communication: tcp://172.21.25.68:32987 Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/tornado/iostream.py", line 869, in _read_to_buffer bytes_read = self.read_from_fd(buf) File "/opt/conda/lib/python3.10/site-packages/tornado/iostream.py", line 1138, in read_from_fd return self.socket.recv_into(buf, len(buf)) ConnectionResetError: [Errno 104] Connection reset by peer The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2066, in gather_dep response = await get_data_from_worker( File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2892, in get_data_from_worker response = await send_recv( File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1024, in send_recv response = await comm.read(deserializers=deserializers) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read convert_stream_closed_error(self, e) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 142, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://172.21.25.92:45492 remote=tcp://172.21.25.68:32987>: ConnectionResetError: [Errno 104] Connection reset by peer

2026-01-16 03:32:35,322 - distributed.worker - INFO - -------------------------------------------------

2026-01-16 03:32:35,321 - distributed.worker - INFO - Registered to: tcp://dask-scheduler:8786

2026-01-16 03:32:34,891 - distributed.worker - INFO - -------------------------------------------------

2026-01-16 03:32:34,891 - distributed.worker - INFO - Local Directory: /tmp/dask-worker-space/worker-ldc6y6ug

2026-01-16 03:32:34,891 - distributed.worker - INFO - Memory: 3.73 GiB

2026-01-16 03:32:34,890 - distributed.worker - INFO - Threads: 1

2026-01-16 03:32:34,890 - distributed.worker - INFO - -------------------------------------------------

2026-01-16 03:32:34,890 - distributed.worker - INFO - Waiting to connect to: tcp://dask-scheduler:8786

2026-01-16 03:32:34,890 - distributed.worker - INFO - dashboard at: 172.21.25.92:8790

2026-01-16 03:32:34,890 - distributed.worker - INFO - Listening to: tcp://172.21.25.92:33649

2026-01-16 03:32:34,890 - distributed.worker - INFO - Start worker at: tcp://172.21.25.92:33649