State-of-the-art parameter-efficient fine-tuning methods rely on introducing\nadapter modules between the layers of a pretrained language model. However,\nsuch modules are trained separately for each task and thus do not enable\nsharing information across tasks. In this paper, we show that we can learn\nadapter parameters for all layers and tasks by generating them using shared\nhypernetworks, which condition on task, adapter position, and layer id in a\ntransformer model. This parameter-efficient multi-task learning framework\nallows us to achieve the best of both worlds by sharing knowledge across tasks\nvia hypernetworks while enabling the model to adapt to each individual task\nthrough task-specific adapters. Experiments on the well-known GLUE benchmark\nshow improved performance in multi-task learning while adding only 0.29%\nparameters per task. We additionally demonstrate substantial performance\nimprovements in few-shot domain generalization across a variety of tasks. Our\ncode is publicly available in https://github.com/rabeehk/hyperformer.\n