Transformer-based models have gained widespread popularity in varying fields. Deploying these large models for real-world applications inevitably requires fine-tuning on task-specific data, followed by quantization for efficient inference in real-world scenarios. However, QAT (Quantization-Aware Training) is time-consuming and computationally expensive, while PTQ (Post-Training Quantization) often leads to substantial accuracy loss. Since fine-tuning is inevitable, one question would arise: can we just make the model quantization-friendly during fine-tuning? The answer is affirmed. We propose QuantTune, a quantization-friendly fine-tuning method based on restricting the dynamic range amplification effect of outliers across Transformer-based models using the proposed outlier-driven loss. Importantly, QuantTune seamlessly integrates into existing fine-tuning workflows without increasing training time or requiring extra inference complexity. Our approach achieves significant improvements across Transformer-based models, including ViT, BERT-base, and OPT, using naive PTQ only. QuantTune reduces accuracy drops by 12.09% at 8-bit quantization compared to top calibration methods, outperforming state-of-the-art solutions by over 18.84% across ViT models, also demonstrating that the proposed QuantTune achieve the best trade-off in the real-world scenarios.