2026-01-19 15:02:57,027 - distributed.worker.memory - WARNING - Worker is at 24% memory usage. Resuming worker. Process memory: 0.91 GiB -- Worker memory limit: 3.73 GiB
2026-01-19 15:02:56,912 - distributed.worker.memory - WARNING - Worker is at 83% memory usage. Pausing worker. Process memory: 3.10 GiB -- Worker memory limit: 3.73 GiB
2026-01-19 15:02:56,316 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-transfer-9d367886730c1a6f221942d84c750289', 7) Function: merge_transfer args: ( schema_version detector_id ... date_diff_ms __hash_partition 0 1 2794848470 ... 209.0 7 1 1 2794848459 ... 204.0 7 2 1 2794848470 ... 403.0 7 3 1 2794848459 ... 193.0 7 4 1 2794848459 ... 194.0 7 ... ... ... ... ... ... 1680003 1 2794848509 ... 203.0 7 1680004 1 2794848509 ... 203.0 7 1680005 1 2794848509 ... 206.0 7 1680006 1 2794848509 ... 82.0 7 1680007 1 2794848509 ... 193.0 7 [1680008 rows x 28 columns], '9d367886730c1a6f221942d84c750289', 7, 16) kwargs: {} Exception: "RuntimeError('shuffle_transfer failed during shuffle 9d367886730c1a6f221942d84c750289')"
2026-01-19 15:02:50,968 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 2.28 GiB -- Worker memory limit: 3.73 GiB
2026-01-19 15:02:40,255 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 2.64 GiB -- Worker memory limit: 3.73 GiB
2026-01-19 15:02:40,037 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-transfer-63d43e8244a911d4eb6d656d9bff791f', 7) Function: merge_transfer args: ( schema_version detector_id ... date_diff_ms __hash_partition 0 1 2794848489 ... 375.0 7 1 1 2794848489 ... 631.0 7 2 1 2794848489 ... 4628.0 7 3 1 2794848489 ... 434.0 7 4 1 2794848504 ... 404.0 7 ... ... ... ... ... ... 1680003 1 2794848459 ... 193.0 7 1680004 1 2794848459 ... 194.0 7 1680005 1 2794848459 ... 204.0 7 1680006 1 2794848470 ... 529.0 7 1680007 1 2794848516 ... 468.0 7 [1680008 rows x 28 columns], '63d43e8244a911d4eb6d656d9bff791f', 7, 16) kwargs: {} Exception: "RuntimeError('shuffle_transfer failed during shuffle 63d43e8244a911d4eb6d656d9bff791f')"
2026-01-19 15:02:24,110 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-transfer-f8e59ce3dd3848ae2f76070607d62973', 7) Function: merge_transfer args: ( schema_version detector_id ... date_diff_ms __hash_partition 0 1 2794848514 ... 9.0 7 1 1 2794848514 ... 102.0 7 2 1 2794848469 ... 204.0 7 3 1 2794848469 ... 204.0 7 4 1 2794848469 ... 203.0 7 ... ... ... ... ... ... 1680003 1 2794848422 ... 284.0 7 1680004 1 2794848422 ... 306.0 7 1680005 1 2794848422 ... 314.0 7 1680006 1 2794848422 ... 306.0 7 1680007 1 2794848422 ... 308.0 7 [1680008 rows x 28 columns], 'f8e59ce3dd3848ae2f76070607d62973', 7, 16) kwargs: {} Exception: "RuntimeError('shuffle_transfer failed during shuffle f8e59ce3dd3848ae2f76070607d62973')"
2026-01-19 15:02:02,821 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-transfer-83ebb5dd97ec23d66c72d5526c6a4609', 7) Function: merge_transfer args: ( schema_version detector_id ... date_diff_ms __hash_partition 0 1 2794848417 ... 305.0 7 1 1 2794848516 ... 23.0 7 2 1 2794848443 ... 9.0 7 3 1 2794848516 ... 2976.0 7 4 1 2794848417 ... 292.0 7 ... ... ... ... ... ... 1680003 1 2794848509 ... 203.0 7 1680004 1 2794848509 ... 203.0 7 1680005 1 2794848509 ... 206.0 7 1680006 1 2794848509 ... 82.0 7 1680007 1 2794848509 ... 193.0 7 [1680008 rows x 28 columns], '83ebb5dd97ec23d66c72d5526c6a4609', 7, 16) kwargs: {} Exception: "RuntimeError('shuffle_transfer failed during shuffle 83ebb5dd97ec23d66c72d5526c6a4609')"
2026-01-19 15:01:31,997 - distributed.worker - WARNING - Compute Failed Key: ('hash-join-transfer-b94b140c6de8b361dc40112d22e1f016', 7) Function: merge_transfer args: ( schema_version detector_id ... date_diff_ms __hash_partition 0 1 2794848516 ... 572.0 7 1 1 2794848503 ... 266.0 7 2 1 2794848503 ... 231.0 7 3 1 2794848503 ... 307.0 7 4 1 2794848503 ... 306.0 7 ... ... ... ... ... ... 1680003 1 2794848509 ... 184.0 7 1680004 1 2794848470 ... 209.0 7 1680005 1 2794848459 ... 204.0 7 1680006 1 2794848470 ... 403.0 7 1680007 1 2794848459 ... 193.0 7 [1680008 rows x 28 columns], 'b94b140c6de8b361dc40112d22e1f016', 7, 16) kwargs: {} Exception: "RuntimeError('shuffle_transfer failed during shuffle b94b140c6de8b361dc40112d22e1f016')"
2026-01-19 12:31:06,611 - distributed.worker - INFO - -------------------------------------------------
2026-01-19 12:31:06,611 - distributed.worker - INFO - Registered to: tcp://dask-scheduler:8786
2026-01-19 12:31:06,233 - distributed.worker - INFO - -------------------------------------------------
2026-01-19 12:31:06,233 - distributed.worker - INFO - Local Directory: /tmp/dask-worker-space/worker-9wlq47h9
2026-01-19 12:31:06,233 - distributed.worker - INFO - Memory: 3.73 GiB
2026-01-19 12:31:06,233 - distributed.worker - INFO - Threads: 1
2026-01-19 12:31:06,233 - distributed.worker - INFO - -------------------------------------------------
2026-01-19 12:31:06,233 - distributed.worker - INFO - Waiting to connect to: tcp://dask-scheduler:8786
2026-01-19 12:31:06,233 - distributed.worker - INFO - dashboard at: 172.21.151.188:8790
2026-01-19 12:31:06,233 - distributed.worker - INFO - Listening to: tcp://172.21.151.188:41715
2026-01-19 12:31:06,232 - distributed.worker - INFO - Start worker at: tcp://172.21.151.188:41715