Which is exactly what I just said... 
Recoil impulse is directly related to escaping hot gases.
Not "escaping".
the combustion gasses expand evenly in all directions .
All the forces generated , are generated at the same time.
As the gasses expand and push the bullet forward they start pushing the rifle backward with the same force at the same time.
They also expand sideways but since there is no give to the sides of the barrel they generate "chamber pressure" and are redirected to give more forward push to the bullet and rearward push to the rifle.
When the hot gasses escape the muzzle is the end of the recoil impulse because they are no longer confined.