Response to Reviewer 1bmQ
We sincerely thank the reviewer for specific comments. We have provided a detailed response to each comment below. We hope our answers can properly address your concerns.
## Weakness 1
> There is some nonrigorous math in the paper.
### Response
We truly thank the reviewer for pointing out these issues! We have carefully revised the paper to fix the nonrigorous math equations.
The revised Equation 4 in our paper is shown here.
$$
\begin{align}
\mathcal{L}\_{\text{plan}} = D\_{KL}(Q(z)||P\_\theta(z|q)) &= D\_{KL}(Q(z|a, q, \mathcal{G})||P\_\theta(z|q)),\tag{1} \\\\
&\simeq - \frac{1}{|\mathcal{Z^*}|}\sum\_{z\in\mathcal{Z^*}} \log P\_\theta(z|q),\tag{2}
\end{align}
$$
where $\mathcal{Z}^*\subseteq\mathcal{Z}$ denotes the set of shortest paths between $e_q$ and $e_a$ in KGs, and $\mathcal{Z}$ denotes all valid relation paths in KGs.
Due to the limitation of space, we put the detailed derivation in Appendix A.1, which is also shown as follows.
The posterior distribution $Q(z)$ can be approximated by $Q(z|a, q, \mathcal{G})$, given as
$$
\begin{equation}
\tag{3}
Q(z) \simeq Q(z|a,q,\mathcal{G}) = \begin{cases}
& \frac{1}{|\mathcal{Z}|}, \exists w_z(e_q,e_a)\in \mathcal{G},\\\\
& 0, \text{else},
\end{cases}
\end{equation}
$$
where we assume a uniform distribution over all valid relation paths $\mathcal{Z}$ [1].
Thus, the KL divergence can be calculated as
$$
\begin{align}
\mathcal{L}\_{\text{plan}} = D\_{KL}(Q(z)||P\_\theta(z|q)) &= D\_{KL}(Q(z|a, q, \mathcal{G})||P\_\theta(z|q)), \tag{4}\\\\
&=\mathbb{E}\_{z\sim Q(z|a, q, \mathcal{G})}[\log Q(z|a, q, \mathcal{G})-\log P\_\theta(z|q)], \tag{5}\\\\
&= - \mathbb{E}\_{z\sim Q(z|a, q, \mathcal{G})}\log P\_\theta(z|q) + \text{CONST}, \tag{6}
\end{align}
$$
where the expectations cannot be computed exactly because of the large number of valid relation paths $\mathcal{Z}$, so we approximate it by using the set of shortest paths $\mathcal{Z}^*\subseteq\mathcal{Z}$ between $e_q$ and $e_a$ in KGs. This can be formulated as
$$
\begin{align}
\mathcal{L}\_{\text{plan}} = - \sum\_{z\in\mathcal{Z^*}} Q(z|a, q, \mathcal{G}) \log P\_\theta(z|q) + \text{CONST}. \tag{7}
\end{align}
$$
By assuming a uniform distribution over the set of shortest paths $\mathcal{Z}^*$, we can rewrite Eq. (7) as
$$
\begin{align}
\mathcal{L}\_{\text{plan}} & = - \frac{1}{|\mathcal{Z^*}|}\sum\_{z\in\mathcal{Z^*}} \log P\_\theta(z|q) + \text{CONST}.\tag{8}
\end{align}
$$
We keep the CONST in the equation. However, we omit it in the final objective since it makes no contributions to the optimization.
The revised Equation 6 in our paper is presented here.
$$
\begin{align}
\mathcal{L}\_{\text{reason}} &= \mathbb{E}\_{z\sim Q(z|a, q, \mathcal{G})}[\log P\_\theta(a|q,z,\mathcal{G})],\tag{9}\\\\
&= \sum\_{z\in\mathcal{Z}^*\_K} \log P\_\theta(a|q,z,\mathcal{G}), \tag{10}\\\\
&= \log P\_\theta(a|q,\mathcal{Z}^*\_{K},\mathcal{G}). \tag{11}
\end{align}
$$
The expectation is approximated by sampling $K$ plans from the $\mathcal{Z^*}$, denoted as $\mathcal{Z}^*_K\subseteq \mathcal{Z}^*$. The conversion from Eq. (9) to Eq. (10) is based on the FiD framework [1], where we can simultaneously use multiple plans for reasoning. The FiD framework is introduced in Equation 5 of our paper, which is also shown here.
The FiD framework assumes each $z\in \mathcal{Z}$ contributes independently. The probability $P_\theta(a|q,\mathcal{Z},\mathcal{G})$ can be approximated as the product of each $P_\theta(a|q,z,\mathcal{G})$, which is formulated as
$$
\begin{equation}
P\_\theta(a|q,\mathcal{Z},\mathcal{G}) = \prod\_{z\in\mathcal{Z}} P\_\theta(a|q,z,\mathcal{G}). \tag{12}
\end{equation}
$$
We also want to clarify the marginalization in Equation 10 of our paper is reasonable based on the FiD framework shown in Eq. (12), which can be derived as
$$
\begin{align*}
&\text{Because:} \\\\
& P\_\theta(a|q,\mathcal{Z},\mathcal{G}) = \prod\_{z\in\mathcal{Z}} P\_\theta(a|q,z,\mathcal{G}), \\\\
&\Rightarrow \\\\
&\log P\_\theta(a|q,\mathcal{Z}^*\_K, \mathcal{G}) = \log \sum\_{z\in\mathcal{Z}^*\_K} P\_\theta(a|q,z,\mathcal{G}), \\\\
&\Rightarrow \\\\
&\underset{\theta}{\operatorname{arg max}}\log P\_\theta(a|q,\mathcal{Z}^*\_K, \mathcal{G}) = \underset{\theta}{\operatorname{arg max}} \log \sum\_{z\in\mathcal{Z}^*\_K} P\_\theta(a|q,z,\mathcal{G}), \\\\
&= \underset{\theta}{\operatorname{arg max}} \log \sum\_{z\in\mathcal{Z}^*\_K}\sum\_{w\_z\in\mathcal{W}\_z}\prod\_{t=1}^{|a|}P\_\theta(t_i|t_{<i}, q, w_z), \\\\
&\text{Q.E.D.}
\end{align*}
$$
[1] Singh, D., Reddy, S., Hamilton, W., Dyer, C., & Yogatama, D. (2021). End-to-end training of multi-document reader and retriever for open-domain question answering. Advances in Neural Information Processing Systems, 34, 25968-25981.