Gradient clipping in RNNs
Transfer and task-feature learning in recurrent networks
Gradient Clipping, Transfer, and Task-Feature Learning in Recurrent Networks Schaffer Lab, Icahn School of Medicine at Mount Sinai. Visiting Research Scientist, May 2026 – present.
Gradient clipping is near-universal in recurrent network training, yet it is usually treated as an implementation detail rather than something that shapes which solutions a network finds. This project treats clipping as a first-class object in the nonconvex training dynamics.
Questions
- How does gradient clipping shape nonconvex RNN training dynamics, particularly for transfer and task-feature learning?
- How do clipping thresholds and model parameters jointly govern the learned solutions and the complexity of training?
Motivation
Beyond the optimization question, characterizing how clipping alters what a recurrent network learns offers a handle on altered learning in neurological and psychiatric disorders, where the relationship between learning dynamics and task representation is the object of interest.