Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(notebook): trim train items before DDP sharding

lab120750b398f88981
diff --git a/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb b/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb
index 33ee17d8..56485dba 100644
--- a/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb
+++ b/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb
@@ -261,7 +261,7 @@
" n_calib = min(CALIB_MAX, len(all_items) // 10)\n",
" calib_items = [all_items[i] for i in sorted(order[:n_calib])]\n",
" train_items = [all_items[i] for i in sorted(order[n_calib:])]\n",
- " my_items = train_items[rank::world_size]\n",
+ " # Trim before sharding so every rank gets the same count (drops at most\n # world_size-1 items). Otherwise ranks can run different numbers of\n # micro-batches and DDP deadlocks on the unmatched all-reduce.\n # See https://github.com/NandhaKishorM/laya/issues/678\n train_items = train_items[:len(train_items) // world_size * world_size]\n my_items = train_items[rank::world_size]\n",
" \n",
" EPOCHS = 4\n",
" MICRO_BATCH = 8 # 8 sequences per forward pass per GPU\n",