Evaluating a policy by deploying it in the real world can be risky and\ncostly. Off-policy policy evaluation (OPE) algorithms use historical data\ncollected from running a previous policy to evaluate a new policy, which\nprovides a means for evaluating a policy without requiring it to ever be\ndeployed. Importance sampling is a popular OPE method because it is robust to\npartial observability and works with continuous states and actions. However,\nthe amount of historical data required by importance sampling can scale\nexponentially with the horizon of the problem: the number of sequential\ndecisions that are made. We propose using policies over temporally extended\nactions, called options, and show that combining these policies with importance\nsampling can significantly improve performance for long-horizon problems. In\naddition, we can take advantage of special cases that arise due to\noptions-based policies to further improve the performance of importance\nsampling. We further generalize these special cases to a general covariance\ntesting rule that can be used to decide which weights to drop in an IS\nestimate, and derive a new IS algorithm called Incremental Importance Sampling\nthat can provide significantly more accurate estimates for a broad class of\ndomains.\n