Dear reviewer:
Thank you for your attention to our research. We greatly respect your perspective on our research domain, which we acknowledge is somewhat narrow. However, we cannot construct a meaningful rebuttal without understanding your specific concerns. We would appreciate clarification on any technical flaws you've identified, such as issues with our experimental design or methodology. We are eager to address these concerns or any confusion you might have. We also welcome any suggestions you might offer.
We are happy to provide a more detailed explanation of our methods and to re-explain our work and its importance using clear, accessible language. We will explain our research step-by-step, from background to technical details, to enhance your understanding.
* Approximately 1 in 10 people suffer from speech disorders such as aphasia or dyslexia.
* When people suffer from speech disorders, they typically seek treatment at hospitals. However, the cost is often prohibitive, usually exceeding $1,000 for a single diagnosis, which can take over 5 hours. Additionally, there is a shortage of doctors and speech-language pathologists (SLPs). Consequently, such costs are often unaffordable for low-income families. Even in countries where the government covers these fees, treatment is time-consuming, and many people lack access due to the scarcity of specialists.
* This raises the question: Can AI automate this process? AI has the potential to provide an efficient and more affordable solution.
For AI to automate speech disorder diagnosis, it needs to perform deep transcription of speech, including dysfluencies such as insertions, omissions, replacements, stutters, prolongations, and phonetic errors. Accurate timestamping is also crucial. SLPs have informed us that these are the official diagnostic procedures.
* Here's an example of what an AI model should output: Given an impaired speech sample, we expect our system to produce: "Y-You [block] wi-[stutter]-sh [missing] know all [prolongation] about my grandfather". In contrast, state-of-the-art models like Whisper, ChatGPT, or Salmonn typically produce a perfect transcription: "You wish to know all about my grandfather", which is not useful for our purposes. This demonstrates that current large-scale Automatic Speech Recognition (ASR) models show limited performance on these tasks.
* To underscore the importance of this research: Recall that a single treatment session costs about \\$1,000 and takes approximately 5 hours. Our AI solution could potentially save both patients and healthcare providers (or governments) \\$1,000 and 5 hours per session. Given that 1 in 10 people have disordered speech, the market size is substantial. To our knowledge, AI exploration in this area is limited. Therefore, our work has broad impact, including both clinical and business implications.
Now, let's discuss the technical aspects of our work. We will introduce these step-by-step, and please don't hesitate to ask for clarification.
Our pipeline has two inputs: dysfluent speech and either golden truth text or a prompt.
* We obtain speech representations from dysfluent speech.
* 2.1. While we could use traditional speech representations such as HuBERT, we found that our gestural scores are more scalable for this task, as demonstrated in Table 1. Therefore, we employ gestural scores instead of traditional SSL units.
* 2.2. To obtain gestural scores, we first need articulatory kinematics data. We use Acoustic to Articulatory Inversion (AAI) to derive articulatory data X from speech. We then developed a VAE framework to generate gestural scores H from X. Gestural scores are sparse and duration-aware, so we developed a duration predictor, intensity predictor, and sparse sampler. We're happy to discuss any of these steps in more detail if you have questions.
* Connectionist Subsequence Alignment (CSA): We discovered that the Longest Common Subsequence (LCS) algorithm naturally captures speech dysfluencies, as shown in Figure 3. For the ground truth phoneme "AH", it aligns to the speech "AH AH ER AH", indicating a repetition of "AH" and an insertion of "ER". Based on this discovery, we developed a stochastic and differentiable LCS, named CSA, based on the vanilla CTC algorithm to provide dysfluency-aware alignments.
* CSA takes gestural scores H and text as input and outputs an alignment. This alignment is then input into LLaMA with pre-designed instructions to achieve end-to-end learning.
* Due to the lack of large-scale data, we developed a large-scale, TTS-based simulation dataset called Libridys, comprising about 4,000 hours of speech. We trained our entire model on Libridys. We then performed evaluations on Libridys, VCTK++, and real disordered speech data (nfvPPA).
We look forward to your feedback and are happy to provide further explanations or clarifications as needed. Please let us what step might confuse you.