Domain-specific neural network accelerators have seen growing interest in\nrecent years due to their improved energy efficiency and inference performance\ncompared to CPUs and GPUs. In this paper, we propose a novel cross-layer\noptimized neural network accelerator called CrossLight that leverages silicon\nphotonics. CrossLight includes device-level engineering for resilience to\nprocess variations and thermal crosstalk, circuit-level tuning enhancements for\ninference latency reduction, and architecture-level optimization to enable\nhigher resolution, better energy-efficiency, and improved throughput. On\naverage, CrossLight offers 9.5x lower energy-per-bit and 15.9x higher\nperformance-per-watt at 16-bit resolution than state-of-the-art photonic deep\nlearning accelerators.\n