Gradient Clipping Parallel
Located in Gradien.GradClip. Used to prevent exploding gradients, especially in RNNs or deep networks.
Functions
.clipValue
Clips gradient values element-wise to be within [-clip, clip].
lua
(params: {Tensor}, clip: number) -> ().clipNorm
Scales gradients so that their total norm does not exceed maxNorm. This preserves the direction of the gradient.
lua
(params: {Tensor}, maxNorm: number) -> ()