Avoiding Loss of Quality while in Pursuit of a Tiny Model