2026-04-22 11:01:34,572 - distributed.worker - ERROR - Worker stream died during communication: tcp://172.21.159.225:46131 ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/comm/core.py", line 292, in connect comm = await wait_for( File "/opt/conda/lib/python3.10/site-packages/distributed/utils.py", line 1849, in wait_for return await asyncio.wait_for(fut, timeout) File "/opt/conda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 511, in connect convert_stream_closed_error(self, e) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 142, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc distributed.comm.core.CommClosedError: in <distributed.comm.tcp.TCPConnector object at 0x7faa3597b6a0>: ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2066, in gather_dep response = await get_data_from_worker( File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2889, in get_data_from_worker comm = await rpc.connect(worker) File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1506, in connect return await connect_attempt File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1427, in _connect comm = await connect( File "/opt/conda/lib/python3.10/site-packages/distributed/comm/core.py", line 318, in connect raise OSError( OSError: Timed out trying to connect to tcp://172.21.159.225:46131 after 30 s
2026-04-22 11:01:04,770 - distributed.worker - ERROR - Worker stream died during communication: tcp://172.21.159.214:40849 Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1427, in _connect comm = await connect( File "/opt/conda/lib/python3.10/site-packages/distributed/comm/core.py", line 292, in connect comm = await wait_for( File "/opt/conda/lib/python3.10/site-packages/distributed/utils.py", line 1849, in wait_for return await asyncio.wait_for(fut, timeout) File "/opt/conda/lib/python3.10/asyncio/tasks.py", line 432, in wait_for await waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2066, in gather_dep response = await get_data_from_worker( File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2889, in get_data_from_worker comm = await rpc.connect(worker) File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1506, in connect return await connect_attempt File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1450, in _connect raise CommClosedError(reason) distributed.comm.core.CommClosedError: Address removed.
2026-04-22 11:01:04,747 - distributed.worker - ERROR - Worker stream died during communication: tcp://172.21.25.34:37891 Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1427, in _connect comm = await connect( File "/opt/conda/lib/python3.10/site-packages/distributed/comm/core.py", line 292, in connect comm = await wait_for( File "/opt/conda/lib/python3.10/site-packages/distributed/utils.py", line 1849, in wait_for return await asyncio.wait_for(fut, timeout) File "/opt/conda/lib/python3.10/asyncio/tasks.py", line 432, in wait_for await waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2066, in gather_dep response = await get_data_from_worker( File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2889, in get_data_from_worker comm = await rpc.connect(worker) File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1506, in connect return await connect_attempt File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1450, in _connect raise CommClosedError(reason) distributed.comm.core.CommClosedError: Address removed.
2026-04-22 11:01:04,573 - distributed.worker - ERROR - Worker stream died during communication: tcp://172.21.159.222:35927 Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 317, in write raise StreamClosedError() tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2066, in gather_dep response = await get_data_from_worker( File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2892, in get_data_from_worker response = await send_recv( File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1022, in send_recv await comm.write(msg, serializers=serializers, on_error="raise") File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 328, in write convert_stream_closed_error(self, e) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://172.21.159.246:54010 remote=tcp://172.21.159.222:35927>: Stream is closed
2026-04-22 11:01:04,518 - distributed.worker - ERROR - Worker stream died during communication: tcp://172.21.151.168:39731 Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/tornado/iostream.py", line 869, in _read_to_buffer bytes_read = self.read_from_fd(buf) File "/opt/conda/lib/python3.10/site-packages/tornado/iostream.py", line 1138, in read_from_fd return self.socket.recv_into(buf, len(buf)) ConnectionResetError: [Errno 104] Connection reset by peer The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2066, in gather_dep response = await get_data_from_worker( File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2892, in get_data_from_worker response = await send_recv( File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1024, in send_recv response = await comm.read(deserializers=deserializers) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read convert_stream_closed_error(self, e) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 142, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://172.21.159.246:35644 remote=tcp://172.21.151.168:39731>: ConnectionResetError: [Errno 104] Connection reset by peer
2026-04-22 11:01:04,463 - distributed.worker - ERROR - Worker stream died during communication: tcp://172.21.159.207:32835 Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/tornado/iostream.py", line 869, in _read_to_buffer bytes_read = self.read_from_fd(buf) File "/opt/conda/lib/python3.10/site-packages/tornado/iostream.py", line 1138, in read_from_fd return self.socket.recv_into(buf, len(buf)) ConnectionResetError: [Errno 104] Connection reset by peer The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2066, in gather_dep response = await get_data_from_worker( File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2892, in get_data_from_worker response = await send_recv( File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1024, in send_recv response = await comm.read(deserializers=deserializers) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read convert_stream_closed_error(self, e) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 142, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://172.21.159.246:58960 remote=tcp://172.21.159.207:32835>: ConnectionResetError: [Errno 104] Connection reset by peer
2026-04-22 11:01:03,805 - distributed.worker - ERROR - Worker stream died during communication: tcp://172.21.25.32:41105 Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/tornado/iostream.py", line 869, in _read_to_buffer bytes_read = self.read_from_fd(buf) File "/opt/conda/lib/python3.10/site-packages/tornado/iostream.py", line 1138, in read_from_fd return self.socket.recv_into(buf, len(buf)) ConnectionResetError: [Errno 104] Connection reset by peer The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2066, in gather_dep response = await get_data_from_worker( File "/opt/conda/lib/python3.10/site-packages/distributed/worker.py", line 2892, in get_data_from_worker response = await send_recv( File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1024, in send_recv response = await comm.read(deserializers=deserializers) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read convert_stream_closed_error(self, e) File "/opt/conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 142, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://172.21.159.246:41168 remote=tcp://172.21.25.32:41105>: ConnectionResetError: [Errno 104] Connection reset by peer
2026-04-22 11:01:03,773 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-4b4b25fcaf703c1e99e652bac3539b7f', 3) Function: merge_unpack args: ('3d4472ea35c0f53629acee4ad7bde540', '0eb3db6a98954db9b0424aa70e32eb2f', 3, 11112, 11114, 'inner', 'hashed_source_and_date', 'hashed_source_and_date', <distributed.protocol.serialize.Serialized object at 0x7faa358a45b0>, ['_x', '_y']) kwargs: {} Exception: "RuntimeError('Worker tcp://172.21.25.29:34733 left during active shuffle 3d4472ea35c0f53629acee4ad7bde540')"
2026-04-22 11:01:03,676 - distributed.worker - WARNING - Compute Failed Key: ('shuffle-p2p-210aea3b503452ad401d37bd774444eb', 15) Function: shuffle_unpack args: ('416a4a8a9303bd1b6a928557fd2c4409', 15, 11117) kwargs: {} Exception: "RuntimeError('shuffle_unpack failed during shuffle 416a4a8a9303bd1b6a928557fd2c4409')"
2026-04-22 11:01:03,650 - distributed.worker - WARNING - Compute Failed Key: ('shuffle-p2p-210aea3b503452ad401d37bd774444eb', 11) Function: shuffle_unpack args: ('416a4a8a9303bd1b6a928557fd2c4409', 11, 11117) kwargs: {} Exception: "RuntimeError('shuffle_unpack failed during shuffle 416a4a8a9303bd1b6a928557fd2c4409')"
2026-04-22 11:01:03,643 - distributed.worker - WARNING - Compute Failed Key: ('shuffle-p2p-210aea3b503452ad401d37bd774444eb', 0) Function: shuffle_unpack args: ('416a4a8a9303bd1b6a928557fd2c4409', 0, 11117) kwargs: {} Exception: "RuntimeError('shuffle_unpack failed during shuffle 416a4a8a9303bd1b6a928557fd2c4409')"
2026-04-22 11:01:03,618 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-transfer-2f4fca06bc9c5475b26c12cf93722f4f', 0) Function: merge_transfer args: ( schema_version detector_id ... daytime_running_lights __hash_partition 0 1 2794848470 ... False 5 1 1 2794848470 ... False 5 2 1 2794848470 ... False 5 3 1 2794848470 ... False 11 4 1 2794848470 ... False 11 .. ... ... ... ... ... 247 1 2794848516 ... False 9 248 1 2794848516 ... False 11 249 1 2794848516 ... False 5 250 1 2794848516 ... False 13 251 1 2794848516 ... False 3 [204 rows x 20 columns], '2f4fca06bc9c5475b26c12cf kwargs: {} Exception: "RuntimeError('shuffle_transfer failed during shuffle 2f4fca06bc9c5475b26c12cf93722f4f')"
2026-04-22 10:31:00,565 - distributed.worker - INFO - -------------------------------------------------
2026-04-22 10:31:00,565 - distributed.worker - INFO - Registered to: tcp://dask-scheduler:8786
2026-04-22 10:31:00,133 - distributed.worker - INFO - -------------------------------------------------
2026-04-22 10:31:00,133 - distributed.worker - INFO - Local Directory: /tmp/dask-worker-space/worker-s9qkdttb
2026-04-22 10:31:00,133 - distributed.worker - INFO - Memory: 3.73 GiB
2026-04-22 10:31:00,133 - distributed.worker - INFO - Threads: 1
2026-04-22 10:31:00,133 - distributed.worker - INFO - -------------------------------------------------
2026-04-22 10:31:00,133 - distributed.worker - INFO - Waiting to connect to: tcp://dask-scheduler:8786
2026-04-22 10:31:00,133 - distributed.worker - INFO - dashboard at: 172.21.159.246:8790
2026-04-22 10:31:00,133 - distributed.worker - INFO - Listening to: tcp://172.21.159.246:40119
2026-04-22 10:31:00,132 - distributed.worker - INFO - Start worker at: tcp://172.21.159.246:40119