2026-03-23 11:32:50,519 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-0063f5a7fec3f4c8ac95666f367d067d', 7) Function: merge_unpack args: ('f865bdd6706fd113507ec8389890fea1', '157d2672c645499692e689a267ca6dc4', 7, 466, 468, 'inner', 'hashed_source_and_date', 'hashed_source_and_date', <distributed.protocol.serialize.Serialized object at 0x7f83c8c96ec0>, ['_x', '_y']) kwargs: {} Exception: "RuntimeError('Worker tcp://172.21.159.225:42875 left during active shuffle f865bdd6706fd113507ec8389890fea1')"

2026-03-23 11:32:36,872 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-7f9bb0065575eb6df469736115d215c2', 7) Function: merge_unpack args: ('45483f26f18d7cc8df1ef22ce81b70c1', '9e494aa26fe5b3aa0dce84f37895d2ea', 7, 452, 454, 'inner', 'hashed_source_and_date', 'hashed_source_and_date', <distributed.protocol.serialize.Serialized object at 0x7f83c8b14340>, ['_x', '_y']) kwargs: {} Exception: "RuntimeError('Worker tcp://172.21.159.225:43761 left during active shuffle 45483f26f18d7cc8df1ef22ce81b70c1')"

2026-03-23 11:32:22,046 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-73c3280c5d20cbae7b4cd083d74b76ab', 7) Function: merge_unpack args: ('27a5994a60881feb37c917e0932d60e5', 'f56a0c03fc00b3310bb39bfec49ccf16', 7, 438, 440, 'inner', 'hashed_source_and_date', 'hashed_source_and_date', <distributed.protocol.serialize.Serialized object at 0x7f83e3688340>, ['_x', '_y']) kwargs: {} Exception: "RuntimeError('Worker tcp://172.21.159.225:37639 left during active shuffle 27a5994a60881feb37c917e0932d60e5')"

2026-03-23 11:32:07,169 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-980a3cfe6c27d0aa464e93ca131ba39b', 7) Function: merge_unpack args: ('a2b1515ccb6c1f92bda46a137c1ffc90', '59e536186bab4bebef6b562a8dc24429', 7, 424, 426, 'inner', 'hashed_source_and_date', 'hashed_source_and_date', <distributed.protocol.serialize.Serialized object at 0x7f83cf7f04c0>, ['_x', '_y']) kwargs: {} Exception: "RuntimeError('Worker tcp://172.21.159.225:34119 left during active shuffle a2b1515ccb6c1f92bda46a137c1ffc90')"

2026-03-23 11:31:49,076 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-42a8ffd7ef4e87e1f4b6bc5e8f9cb9d3', 7) Function: merge_unpack args: ('57e741dd40a5ad934281471595060e60', '9eb1eb5714fd3ee401b83ddd7e22df10', 7, 410, 412, 'inner', 'hashed_source_and_date', 'hashed_source_and_date', <distributed.protocol.serialize.Serialized object at 0x7f83bc6c2170>, ['_x', '_y']) kwargs: {} Exception: "RuntimeError('Worker tcp://172.21.159.225:37747 left during active shuffle 57e741dd40a5ad934281471595060e60')"

2026-03-23 10:34:17,797 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-85249e3977285cb89993ff679494a817 Function: shuffle_barrier args: ('85249e3977285cb89993ff679494a817', [336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle 85249e3977285cb89993ff679494a817')"

2026-03-23 10:33:36,795 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-4723f3baf1ca1f7700c5182f9e94c514 Function: shuffle_barrier args: ('4723f3baf1ca1f7700c5182f9e94c514', [321, 321, 321, 321, 321, 321, 321, 321, 321, 321, 321, 321, 321, 321, 321, 321]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle 4723f3baf1ca1f7700c5182f9e94c514')"

2026-03-23 10:32:56,055 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-b84652d98aac3e22c3537f3e06a5d150 Function: shuffle_barrier args: ('b84652d98aac3e22c3537f3e06a5d150', [306, 306, 306, 306, 306, 306, 306, 306, 306, 306, 306, 306, 306, 306, 306, 306]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle b84652d98aac3e22c3537f3e06a5d150')"

2026-03-23 09:34:14,922 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-fe5805e84f68c5be6e2ba7bd12899007 Function: shuffle_barrier args: ('fe5805e84f68c5be6e2ba7bd12899007', [204, 204, 204, 204, 204, 204, 204, 204, 204, 204, 204, 204, 204, 204, 204, 204]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle fe5805e84f68c5be6e2ba7bd12899007')"

2026-03-23 09:33:23,485 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-094cfd07bb12f709ea5c2d286d97dd90 Function: shuffle_barrier args: ('094cfd07bb12f709ea5c2d286d97dd90', [175, 175, 175, 175, 175, 175, 175, 175, 175, 175, 175, 175, 175, 175, 175, 175]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle 094cfd07bb12f709ea5c2d286d97dd90')"

2026-03-23 09:32:30,891 - distributed.worker - WARNING - Compute Failed Key: shuffle-barrier-51426940ecc0f663db5fb31cc6664d76 Function: shuffle_barrier args: ('51426940ecc0f663db5fb31cc6664d76', [148, 148, 148, 148, 148, 148, 148, 148, 148, 148, 148, 148, 148, 148, 148, 148]) kwargs: {} Exception: "RuntimeError('shuffle_barrier failed during shuffle 51426940ecc0f663db5fb31cc6664d76')"

2026-03-23 07:13:50,266 - distributed.worker - INFO - -------------------------------------------------

2026-03-23 07:13:50,265 - distributed.worker - INFO - Registered to: tcp://dask-scheduler:8786

2026-03-23 07:13:48,639 - distributed.worker - INFO - -------------------------------------------------

2026-03-23 07:13:48,638 - distributed.worker - INFO - Local Directory: /tmp/dask-worker-space/worker-wbx88fx0

2026-03-23 07:13:48,638 - distributed.worker - INFO - Memory: 3.73 GiB

2026-03-23 07:13:48,638 - distributed.worker - INFO - Threads: 1

2026-03-23 07:13:48,638 - distributed.worker - INFO - -------------------------------------------------

2026-03-23 07:13:48,638 - distributed.worker - INFO - Waiting to connect to: tcp://dask-scheduler:8786

2026-03-23 07:13:48,638 - distributed.worker - INFO - dashboard at: 172.21.12.129:8790

2026-03-23 07:13:48,638 - distributed.worker - INFO - Listening to: tcp://172.21.12.129:44431

2026-03-23 07:13:48,638 - distributed.worker - INFO - Start worker at: tcp://172.21.12.129:44431