2026-03-23 11:31:25,255 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-523c17806364629e4005d95dadabd641', 7) Function: merge_unpack args: ('20a86657348ea505f9ff13adb95582a3', 'cf91ce8cb01a95207611e5637aa67b8e', 7, 393, 398, 'inner', 'hashed_source_and_date', 'hashed_source_and_date', <distributed.protocol.serialize.Serialized object at 0x7fcb29489990>, ['_x', '_y']) kwargs: {} Exception: "RuntimeError('Worker tcp://172.21.159.225:39121 left during active shuffle 20a86657348ea505f9ff13adb95582a3')"

2026-03-23 11:31:25,009 - distributed.worker - ERROR - Exception during execution of task ('sjoin-e710b2e19a162a9694119cb5bbda9bd1', 206). Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2382, in _prepare_args_for_execution data[k] = self.data[k] File "/opt/conda/lib/python3.10/site-packages/distributed/spill.py", line 226, in __getitem__ return super().__getitem__(key) File "/opt/conda/lib/python3.10/site-packages/zict/buffer.py", line 108, in __getitem__ raise KeyError(key) KeyError: "('set_geometry-510df2030ddffbb9fcf4aa508e455c3b', 14)" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2259, in execute args2, kwargs2 = self._prepare_args_for_execution(ts, args, kwargs) File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2386, in _prepare_args_for_execution data[k] = Actor(type(self.state.actors[k]), self.address, k, self) KeyError: "('set_geometry-510df2030ddffbb9fcf4aa508e455c3b', 14)"

2026-03-23 09:01:03,896 - distributed.worker - ERROR - Exception during execution of task ('group-simple-shuffle-a0dc125aa293aa5a2196e6915a874ee4', 6). Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2382, in _prepare_args_for_execution data[k] = self.data[k] File "/opt/conda/lib/python3.10/site-packages/distributed/spill.py", line 226, in __getitem__ return super().__getitem__(key) File "/opt/conda/lib/python3.10/site-packages/zict/buffer.py", line 108, in __getitem__ raise KeyError(key) KeyError: "('lambda-9b04adac80a864f9d4abc6bdf0f425e8', 6)" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2259, in execute args2, kwargs2 = self._prepare_args_for_execution(ts, args, kwargs) File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2386, in _prepare_args_for_execution data[k] = Actor(type(self.state.actors[k]), self.address, k, self) KeyError: "('lambda-9b04adac80a864f9d4abc6bdf0f425e8', 6)"

2026-03-23 07:15:50,240 - distributed.worker - INFO - -------------------------------------------------

2026-03-23 07:15:50,240 - distributed.worker - INFO - Registered to: tcp://dask-scheduler:8786

2026-03-23 07:15:48,859 - distributed.worker - INFO - -------------------------------------------------

2026-03-23 07:15:48,859 - distributed.worker - INFO - Local Directory: /tmp/dask-worker-space/worker-_5aw2rl9

2026-03-23 07:15:48,859 - distributed.worker - INFO - Memory: 3.73 GiB

2026-03-23 07:15:48,859 - distributed.worker - INFO - Threads: 1

2026-03-23 07:15:48,859 - distributed.worker - INFO - -------------------------------------------------

2026-03-23 07:15:48,859 - distributed.worker - INFO - Waiting to connect to: tcp://dask-scheduler:8786

2026-03-23 07:15:48,858 - distributed.worker - INFO - dashboard at: 172.21.12.133:8790

2026-03-23 07:15:48,858 - distributed.worker - INFO - Listening to: tcp://172.21.12.133:35307

2026-03-23 07:15:48,858 - distributed.worker - INFO - Start worker at: tcp://172.21.12.133:35307