PLEASE could you tell us what modified RSME you are using? In stage-3, the answer depends on the loss function. I am using softL1, L1, L2, and combinations. I am getting better out-of-fold but not better LB. Best result LB for me is with [ |true_phi - pred_phi|/(0.01 + abs(true_phi)) ]^2, but I am just guessing.