Continual learning methods based on pre-trained models (PTMs) have recently gained attention which adapt to successive downstream tasks without catastrophic forgetting. It typically refrains from updating the pre-trained parameters and employs additional adapters, prompts, and classifiers. However, these methods introduce additional complexity and fail to exploit the inherent properties of PTMs fully. In this paper, we propose Sparse Orthogonal Parameters TUning (SoTU), a novel parameter-efficient tuning method that leverages sparse delta parameters with orthogonality constraints to enhance continual learning. Through extensive empirical analysis, we observe that merging high-sparsity delta parameters across tasks can significantly mitigate forgetting, whereas low sparsity leads to severe parameter conflicts. We theoretically analyzed that high-sparsity delta parameters can preserve task-specific knowledge while maintaining performance akin to pre-training. Guided by this insight, SoTU enforces sparsity and approximate orthogonality in parameter updates, ensuring that knowledge from different tasks remains mutually independent yet synergistic. Furthermore, we propose a lightweight merging strategy to integrate delta parameters efficiently without requiring additional classifiers or complex architectures. Comprehensive evaluations on diverse CL benchmarks demonstrate that SoTU significantly outperforms existing CL methods, achieving optimal feature representation for streaming data without necessitating complex classifier designs. Notably, SoTU enables parameter-efficient continual adaptation while preserving PTMs integrity, making it a scalable and effective solution for real-world CL applications.