This work develops a fully decentralized multi-agent algorithm for policy\nevaluation. The proposed scheme can be applied to two distinct scenarios. In\nthe first scenario, a collection of agents have distinct datasets gathered\nfollowing different behavior policies (none of which is required to explore the\nfull state space) in different instances of the same environment and they all\ncollaborate to evaluate a common target policy. The network approach allows for\nefficient exploration of the state space and allows all agents to converge to\nthe optimal solution even in situations where neither agent can converge on its\nown without cooperation. The second scenario is that of multi-agent games, in\nwhich the state is global and rewards are local. In this scenario, agents\ncollaborate to estimate the value function of a target team policy. The\nproposed algorithm combines off-policy learning, eligibility traces and linear\nfunction approximation. The proposed algorithm is of the variance-reduced kind\nand achieves linear convergence with $O(1)$ memory requirements. The linear\nconvergence of the algorithm is established analytically, and simulations are\nused to illustrate the effectiveness of the method.\n