Deep network architectures struggle to continually learn new tasks without\nforgetting the previous tasks. A recent trend indicates that dynamic\narchitectures based on an expansion of the parameters can reduce catastrophic\nforgetting efficiently in continual learning. However, existing approaches\noften require a task identifier at test-time, need complex tuning to balance\nthe growing number of parameters, and barely share any information across\ntasks. As a result, they struggle to scale to a large number of tasks without\nsignificant overhead. In this paper, we propose a transformer architecture\nbased on a dedicated encoder/decoder framework. Critically, the encoder and\ndecoder are shared among all tasks. Through a dynamic expansion of special\ntokens, we specialize each forward of our decoder network on a task\ndistribution. Our strategy scales to a large number of tasks while having\nnegligible memory and time overheads due to strict control of the parameters\nexpansion. Moreover, this efficient strategy doesn't need any hyperparameter\ntuning to control the network's expansion. Our model reaches excellent results\non CIFAR100 and state-of-the-art performances on the large-scale ImageNet100\nand ImageNet1000 while having less parameters than concurrent dynamic\nframeworks.\n