With data-driven development now widely adopted in the software industry, online A/B testing is an established method for measuring the effects of new technologies. However, deploying online experiments demands considerable resources for experimental design, implementation, and production deployment, and may negatively impact users (e.g., unsafe or unethical outcomes) while requiring multiple weeks of data collection. To tackle these issues, a growing research area known as off-policy evaluation (OPE) (offline A/B testing) has gained significant attention. Its goal is to assess new technologies offline by leveraging previously collected logged data. OPE is also a fundamental problem in reinforcement learning (RL) and is important in areas where online A/B testing is expensive or risky, such as healthcare, recommender systems, education, dialogue systems, and robotics. Despite recent advances in code-generation large language models (LLMs) and agentic workflows, little is known about whether and how LLMs and LLM-based agents can be leveraged to automatically optimize OPE implementations. In this work, we propose GrowthHacker, a benchmark that evaluates baseline LLMs and LLM-based agents on large-scale public datasets. GrowthHacker autonomously and iteratively modifies code, runs OPE, and uses the resulting metrics to guide subsequent rounds of optimization. We evaluate methods on Open Bandit Pipeline (OBP) [33] and Scope-RL [22], and we develop a two_agent framework that addresses limitations of existing agent frameworks while reducing system complexity. Across both libraries, two_agent demonstrates the highest reliability (98.1%–100% success rate) and the highest positive-outcome rate (78%), with a median improvement of 4.4% among positive outcomes; CrewAI achieves the highest average improvement (37.9%) and is the only framework with zero extreme-value failures. AutoGen and Default each reach 65% positive-outcome rates. These results establish the feasibility of using LLM-based agents as automated “growth hackers” to continuously improve OPE systems, with implications for scaling data-driven decision-making in production environments where manual optimization is expensive. Our replication package is available at https://doi.org/10.5281/zenodo.17496869 and https://github.com/jie-jw-wu/ope-agent.