Logs: tcp://172.21.12.242:41381

2026-01-19 15:02:22,277 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-transfer-f8e59ce3dd3848ae2f76070607d62973', 0) Function: merge_transfer args: ( schema_version detector_id ... date_diff_ms __hash_partition 0 1 2794848514 ... 29.0 0 1 1 2794848456 ... 624.0 0 2 1 2794848514 ... 30.0 0 3 1 2794848469 ... 137.0 0 4 1 2794848469 ... 203.0 0 ... ... ... ... ... ... 1745561 1 2794848422 ... 231.0 0 1745562 1 2794848489 ... 906.0 0 1745563 1 2794848422 ... 206.0 0 1745564 1 2794848449 ... 57.0 0 1745565 1 2794848422 ... 512.0 0 [1745566 rows x 28 columns], 'f8e59ce3dd3848ae2f76070607d62973', 0, 16) kwargs: {} Exception: "RuntimeError('shuffle_transfer failed during shuffle f8e59ce3dd3848ae2f76070607d62973')"

2026-01-19 05:32:56,078 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-54b9c2a97c6027c584d9cbbe910240a6 Function: shuffle_barrier args: ('54b9c2a97c6027c584d9cbbe910240a6', [6076, 6076, 6076, 6076, 6076, 6076, 6076, 6076, 6076, 6076, 6076, 6076, 6076, 6076, 6076, 6076]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle 54b9c2a97c6027c584d9cbbe910240a6')"

2026-01-19 05:32:05,005 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-75ac88329e3fce621aa4985cad5595f2 Function: shuffle_barrier args: ('75ac88329e3fce621aa4985cad5595f2', [6049, 6049, 6049, 6049, 6049, 6049, 6049, 6049, 6049, 6049, 6049, 6049, 6049, 6049, 6049, 6049]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle 75ac88329e3fce621aa4985cad5595f2')"

2026-01-19 03:30:27,801 - distributed.worker - WARNING - Compute Failed Key: ('assign-16d26042a00fc441423ad71accc306e5', 7) Function: subgraph_callable-ab9cbdf4-8720-4d3f-9b55-86cf1165 args: ( source_id geometry 205 2794848524 POINT Z (447964.634 5899731.769 0.000)) kwargs: {} Exception: "GEOSException('IllegalArgumentException: point array must contain 0 or >1 elements\\n')"

2026-01-18 10:31:48,781 - distributed.worker - WARNING - Compute Failed Key: ('shuffle-transfer-c98cebc2438525934e901a68c83f36f4', 13) Function: shuffle_transfer args: ( schema_version detector_id ... heading_diff_cur_and_next _partitions 276 1 2794848416 ... 0.0 8 276 1 2794848471 ... -0.2 13 276 1 2794848467 ... 0.2 1 276 1 2794848472 ... 0.0 2 276 1 2794848516 ... 0.0 14 .. ... ... ... ... ... 297 1 2794848467 ... 0.0 1 297 1 2794848483 ... 0.0 9 297 1 2794848500 ... 0.3 7 297 1 2794848467 ... -0.2 3 297 1 2794848461 ... -0.3 2 [352 rows x 21 columns], 'c98cebc2438525934e901a68c83f36f4', 13, 16, '_par kwargs: {} Exception: "RuntimeError('shuffle_transfer failed during shuffle c98cebc2438525934e901a68c83f36f4')"

2026-01-18 03:03:39,993 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-5ea19e8ed0835e1767f3d466556b87eb Function: shuffle_barrier args: ('5ea19e8ed0835e1767f3d466556b87eb', [4701, 4701, 4701, 4701, 4701, 4701, 4701, 4701, 4701, 4701, 4701, 4701, 4701, 4701, 4701, 4701]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle 5ea19e8ed0835e1767f3d466556b87eb')"

2026-01-18 03:03:00,197 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-e4d9e92a361ec89a00d53a070b22c840 Function: shuffle_barrier args: ('e4d9e92a361ec89a00d53a070b22c840', [4686, 4686, 4686, 4686, 4686, 4686, 4686, 4686, 4686, 4686, 4686, 4686, 4686, 4686, 4686, 4686]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle e4d9e92a361ec89a00d53a070b22c840')"

2026-01-18 03:02:02,847 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-23ec8e5079772a5917b0d27dac7535a5 Function: shuffle_barrier args: ('23ec8e5079772a5917b0d27dac7535a5', [4645, 4645, 4645, 4645, 4645, 4645, 4645, 4645, 4645, 4645, 4645, 4645, 4645, 4645, 4645, 4645]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle 23ec8e5079772a5917b0d27dac7535a5')"

2026-01-17 07:04:10,533 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-d7aeb8c66a738d8bbce53053563cebbb Function: shuffle_barrier args: ('d7aeb8c66a738d8bbce53053563cebbb', [3782, 3782, 3782, 3782, 3782, 3782, 3782, 3782, 3782, 3782, 3782, 3782, 3782, 3782, 3782, 3782]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle d7aeb8c66a738d8bbce53053563cebbb')"

2026-01-17 07:03:30,369 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-55f01372aa03c81debc1170522181d2f Function: shuffle_barrier args: ('55f01372aa03c81debc1170522181d2f', [3767, 3767, 3767, 3767, 3767, 3767, 3767, 3767, 3767, 3767, 3767, 3767, 3767, 3767, 3767, 3767]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle 55f01372aa03c81debc1170522181d2f')"

2026-01-17 07:02:51,854 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-fdfe932b91c54d95899e6ec92d671def Function: shuffle_barrier args: ('fdfe932b91c54d95899e6ec92d671def', [3752, 3752, 3752, 3752, 3752, 3752, 3752, 3752, 3752, 3752, 3752, 3752, 3752, 3752, 3752, 3752]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle fdfe932b91c54d95899e6ec92d671def')"

2026-01-17 07:02:04,953 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-b0ecc915a1a8910e42b90cce4a505ac4 Function: shuffle_barrier args: ('b0ecc915a1a8910e42b90cce4a505ac4', [3725, 3725, 3725, 3725, 3725, 3725, 3725, 3725, 3725, 3725, 3725, 3725, 3725, 3725, 3725, 3725]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle b0ecc915a1a8910e42b90cce4a505ac4')"

2026-01-17 03:30:06,585 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-6ecf3f6b6bbf2dcd44736aa9f15af56e', 0) Function: merge_unpack args: ('0daf590db3ff54d36d8b96524d4704eb', '32bc94ad0f277ef7c0ea52efd9f2b324', 0, 3466, 3468, 'inner', 'hashed_source_and_date', 'hashed_source_and_date', <distributed.protocol.serialize.Serialized object at 0x7f6175ac7ca0>, ['_x', '_y']) kwargs: {} Exception: "KeyError('32bc94ad0f277ef7c0ea52efd9f2b324')"

2026-01-17 03:04:02,874 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-fe06b586dfce930cd1b658299dc67b10 Function: shuffle_barrier args: ('fe06b586dfce930cd1b658299dc67b10', [3465, 3465, 3465, 3465, 3465, 3465, 3465, 3465, 3465, 3465, 3465, 3465, 3465, 3465, 3465, 3465]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle fe06b586dfce930cd1b658299dc67b10')"

2026-01-17 03:03:22,982 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-8f462bddb5ec7e824950eb2e1ac2b8c1 Function: shuffle_barrier args: ('8f462bddb5ec7e824950eb2e1ac2b8c1', [3450, 3450, 3450, 3450, 3450, 3450, 3450, 3450, 3450, 3450, 3450, 3450, 3450, 3450, 3450, 3450]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle 8f462bddb5ec7e824950eb2e1ac2b8c1')"

2026-01-17 03:02:33,591 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-4d586856617ee5ce730c1546cd326eb5 Function: shuffle_barrier args: ('4d586856617ee5ce730c1546cd326eb5', [3421, 3421, 3421, 3421, 3421, 3421, 3421, 3421, 3421, 3421, 3421, 3421, 3421, 3421, 3421, 3421]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle 4d586856617ee5ce730c1546cd326eb5')"

2026-01-17 03:01:22,563 - distributed.worker - ERROR - Worker stream died during communication: tcp://172.21.151.165:39925 Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/tornado/iostream.py", line 869, in _read_to_buffer bytes_read = self.read_from_fd(buf) File "/opt/conda/lib/python3.10/site-packages/tornado/iostream.py", line 1138, in read_from_fd return self.socket.recv_into(buf, len(buf)) ConnectionResetError: [Errno 104] Connection reset by peer The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2066, in gather_dep response = await get_data_from_worker( File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2892, in get_data_from_worker response = await send_recv( File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1024, in send_recv response = await comm.read(deserializers=deserializers) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read convert_stream_closed_error(self, e) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 142, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://172.21.12.242:53308 remote=tcp://172.21.151.165:39925>: ConnectionResetError: [Errno 104] Connection reset by peer

2026-01-16 17:00:57,668 - distributed.worker - INFO - -------------------------------------------------

2026-01-16 17:00:57,668 - distributed.worker - INFO - Registered to: tcp://dask-scheduler:8786

2026-01-16 17:00:57,185 - distributed.worker - INFO - -------------------------------------------------

2026-01-16 17:00:57,185 - distributed.worker - INFO - Local Directory: /tmp/dask-worker-space/worker-_4zpklud

2026-01-16 17:00:57,185 - distributed.worker - INFO - Memory: 3.73 GiB

2026-01-16 17:00:57,185 - distributed.worker - INFO - Threads: 1

2026-01-16 17:00:57,185 - distributed.worker - INFO - -------------------------------------------------

2026-01-16 17:00:57,185 - distributed.worker - INFO - Waiting to connect to: tcp://dask-scheduler:8786

2026-01-16 17:00:57,185 - distributed.worker - INFO - dashboard at: 172.21.12.242:8790

2026-01-16 17:00:57,185 - distributed.worker - INFO - Listening to: tcp://172.21.12.242:41381

2026-01-16 17:00:57,185 - distributed.worker - INFO - Start worker at: tcp://172.21.12.242:41381