The optimization of performance of a neural network is a time taking and tedious process, this iterative and continuous process has no definite solution that works well for every possible use case. To tackle this problem we propose an architecture of neural networks called "Deep Genetic Network", which can help in automatic selection of hyper parameter values based on fitness measures during training of the network. The algorithm is a confluence of deep neural networks and genetic algorithm. The problem of optimizing a neural network can be classified into - Architecture and Hyperparameter optimization. A variety of algorithms have been proposed to solve this issue. Our approach uses concepts of mutation and mating (from genetic algorithms) for helping the neural net in finding the optimal set of hyperparameter values during training without requiring any manually setting the values in an iterative trial and error approach. The architecture that we propose here works well in optimization of hyperparameter values in convolutional, recurrent and affine layers. The usage of genetic algorithms for resolving this issue has worked well given adequate training time and computational resources.