Incremental Learning (IL) is useful when artificial systems need to deal with\nstreams of data and do not have access to all data at all times. The most\nchallenging setting requires a constant complexity of the deep model and an\nincremental model update without access to a bounded memory of past data. Then,\nthe representations of past classes are strongly affected by catastrophic\nforgetting. To mitigate its negative effect, an adapted fine tuning which\nincludes knowledge distillation is usually deployed. We propose a different\napproach based on a vanilla fine tuning backbone. It leverages initial\nclassifier weights which provide a strong representation of past classes\nbecause they are trained with all class data. However, the magnitude of\nclassifiers learned in different states varies and normalization is needed for\na fair handling of all classes. Normalization is performed by standardizing the\ninitial classifier weights, which are assumed to be normally distributed. In\naddition, a calibration of prediction scores is done by using state level\nstatistics to further improve classification fairness. We conduct a thorough\nevaluation with four public datasets in a memoryless incremental learning\nsetting. Results show that our method outperforms existing techniques by a\nlarge margin for large-scale datasets.\n