2025-12-06 01:00:50,934 - distributed.worker - WARNING - Compute Failed Key: ('shuffle-p2p-86bd619d06d55badca637dddcb6987c5', 4) Function: shuffle_unpack args: ('d4f2bcb88bd283c797b0c8a2e03a2799', 4, 21324) kwargs: {} Exception: "RuntimeError('shuffle_unpack failed during shuffle d4f2bcb88bd283c797b0c8a2e03a2799')"

2025-12-06 01:00:37,490 - distributed.worker - ERROR - Exception during execution of task ('getitem-0d51c5c3085c0ba9edb097d16d739d58', 4). Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2382, in _prepare_args_for_execution data[k] = self.data[k] File "/opt/conda/lib/python3.10/site-packages/distributed/spill.py", line 226, in __getitem__ return super().__getitem__(key) File "/opt/conda/lib/python3.10/site-packages/zict/buffer.py", line 108, in __getitem__ raise KeyError(key) KeyError: "('sort_values-bdda57bd97c8c52389757ffcf83e58e9', 4)" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2259, in execute args2, kwargs2 = self._prepare_args_for_execution(ts, args, kwargs) File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2386, in _prepare_args_for_execution data[k] = Actor(type(self.state.actors[k]), self.address, k, self) KeyError: "('sort_values-bdda57bd97c8c52389757ffcf83e58e9', 4)"

2025-12-06 01:00:17,477 - distributed.worker - WARNING - Compute Failed Key: ('shuffle-p2p-178fa069fdc4e10d687c9f8c1b4055d8', 4) Function: shuffle_unpack args: ('b643354404bbe7799c8c94f4036bf1af', 4, 21294) kwargs: {} Exception: "RuntimeError('shuffle_unpack failed during shuffle b643354404bbe7799c8c94f4036bf1af')"

2025-12-05 10:21:04,245 - distributed.worker - WARNING - Compute Failed Key: ('assign-fbbc74300fd18669aa11b97fc58d7a55', 7) Function: subgraph_callable-43bf9304-3d0a-4a1b-ae59-1e15e575 args: ( source_id geometry 0 2794848523 POINT Z (440886.485 5927288.700 0.000)) kwargs: {} Exception: "GEOSException('IllegalArgumentException: point array must contain 0 or >1 elements\\n')"

2025-12-05 08:01:04,425 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-ccaa8528e53722d0f9b96e687eadf4c4', 4) Function: merge_unpack args: ('b878824e3ae7b7c0995e6bece22028bf', '17a1cf67b44c34916e03ab4aac157de9', 4, 20487, 20488, 'inner', 'hashed_source_and_date', 'hashed_source_and_date', <distributed.protocol.serialize.Serialized object at 0x7f6657c0dcf0>, ['_x', '_y']) kwargs: {} Exception: "ArrowInvalid('Schema at index 3 was different: \\nschema_version: int64\\ndetector_id: int64\\nsource_id: int64\\ntimestamp: int64\\ngdt: int64\\nsrt: int64\\nheading: double\\nspeed: double\\nevent_type: string\\nexterior_lights: string\\nlongitude: double\\nlatitude: double\\naltitude: int64\\ngeometry_type: string\\nis_day: bool\\nhashed_source_and_date: string\\nindex: int64\\nheading_diff_prev_and_cur: double\\nheading_diff_cur_and_next: double\\n__hash_partition: uint64\\ndate: timestamp[ns]\\nvs\\nschema_version: int64\\ndetector_id: int64\\nsource_id: int64\\ntimestamp: int64\\ngdt: int64\\nsrt: int64\\nheading: double\\nspeed: double\\nevent_type: string\\nexterior_lights: null\\nlongitude: double\\nlatitude: double\\naltitude: int64\\ngeometry_type: string\\nis_day: bool\\nhashed_source_and_date: string\\nindex: int64\\nheading_diff_prev_and_cur: double\\nheading_diff_cur_and_next: double\\n__hash_partition: uint64\\ndate: timestamp[ns]')"

2025-12-03 14:31:27,945 - distributed.worker - ERROR - Worker stream died during communication: tcp://172.21.25.101:34871 Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/tornado/iostream.py", line 869, in _read_to_buffer bytes_read = self.read_from_fd(buf) File "/opt/conda/lib/python3.10/site-packages/tornado/iostream.py", line 1138, in read_from_fd return self.socket.recv_into(buf, len(buf)) ConnectionResetError: [Errno 104] Connection reset by peer The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2066, in gather_dep response = await get_data_from_worker( File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2892, in get_data_from_worker response = await send_recv( File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1024, in send_recv response = await comm.read(deserializers=deserializers) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read convert_stream_closed_error(self, e) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 142, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://172.21.12.199:46230 remote=tcp://172.21.25.101:34871>: ConnectionResetError: [Errno 104] Connection reset by peer

2025-12-03 07:03:08,111 - distributed.worker - INFO - -------------------------------------------------

2025-12-03 07:03:08,111 - distributed.worker - INFO - Registered to: tcp://dask-scheduler:8786

2025-12-03 07:03:07,678 - distributed.worker - INFO - -------------------------------------------------

2025-12-03 07:03:07,678 - distributed.worker - INFO - Local Directory: /tmp/dask-worker-space/worker-tbryr0ej

2025-12-03 07:03:07,677 - distributed.worker - INFO - Memory: 3.73 GiB

2025-12-03 07:03:07,677 - distributed.worker - INFO - Threads: 1

2025-12-03 07:03:07,677 - distributed.worker - INFO - -------------------------------------------------

2025-12-03 07:03:07,677 - distributed.worker - INFO - Waiting to connect to: tcp://dask-scheduler:8786

2025-12-03 07:03:07,677 - distributed.worker - INFO - dashboard at: 172.21.12.199:8790

2025-12-03 07:03:07,677 - distributed.worker - INFO - Listening to: tcp://172.21.12.199:39713

2025-12-03 07:03:07,677 - distributed.worker - INFO - Start worker at: tcp://172.21.12.199:39713