Neural networks increasingly run on hardware outside the user's control (cloud GPUs, inference marketplaces, edge specialized accelerators) for both training and inference. Yet ML-as-a-Service reveals little about what actually ran or whether returned outputs faithfully reflect the intended inputs and models. Users lack recourse against service downgrades such as model swaps, quantization, graph rewrites, or discrepancies like altered advertisement embeddings. Verifying outputs is especially difficult because floating-point execution on heterogeneous accelerators is inherently non-deterministic. Existing approaches like zkML, deterministic replay, TEEs, and replication are either impractical for real floating-point neural networks or reintroduce the need to trust the vendor. We present TAO: a Tolerance-Aware Optimistic verification protocol for floating-point neural networks that accepts outputs within principled operator-level acceptance regions rather than requiring bitwise equality. TAO combines two complementary error models: (i) sound per-operator IEEE-754 worst-case bounds and (ii) tight empirical percentile profiles calibrated across hardware types. Discrepancies are resolved via a Merkle-anchored, threshold-guided interactive dispute game that recursively partitions the traced computation graph until one operator remains; at the leaf, adjudication reduces to either a lightweight theoretical-bound check or a small honest-majority vote against empirical thresholds. Unchallenged results finalize after a challenge window, without requiring trusted hardware or deterministic kernels. We implement TAO as a PyTorch-compatible runtime and a coordinator contract layer; in our prototype, the coordinator is instantiated as an Ethereum smart-contract deployment (Holesky testnet) to quantify coordination cost. The runtime instruments graphs, computes bounds for each operator in real time, and runs unmodified vendor kernels in FP32 with negligible overhead (0.3% additional latency on Qwen3-8B). Our evaluation spans CNNs, Transformers, LLMs, and diffusion models across multiple GPUs (A100, H100, RTX 6000, RTX 4090). Results show that empirical thresholds are 102-103× tighter than theoretical bounds, and bound-aware adversarial attacks achieve 0% success under empirical checks. Together, TAO reconciles scalability with verifiability for real-world heterogeneous ML compute.