When a primary residence ceases to be primary
You don't have to refinance, however if the rates are much better then what your current rate is, then it might make sense to refinance while it is still your primary residence... if you have to move in the future, at that point your primary becomes a rental. (depending on the type of loan, FHA, VA,...