Using FCOS and an Encoder-Decoder Model to Detect and Recognize Visual Mathematical Equations

Abstract

Many data sources on the Internet contain math information within them, and math is used throughout daily life while being important for avenues of study and industry. Understanding math enables better problem solving, pattern comprehension, quantifying relationships, and making predictions of the future. Unfortunately, less people have proficiency in math in recent times. To make the situation worse, it is difficult to locate sources of relevant math information, particularly when the searcher has little familiarity with the subject area. Having Math Information Retrieval (IR) systems would help facilitate searches for math information and assist learners with understanding math concepts. Sadly, extracting mathematical notation in graphical representations into a standardized text-based format is a non-trivial task, since it is required to detect unique symbols and spatial arrangements of mathematical characters, as well as formula positioning in documents. Failure to correctly detecting and recognizing visual math formulas and their notation produce errors that alter the entire meaning of the resulting formulas, or simply do not have the speed needed for a real-time Math IR system. To address these problems, we have developed a combined FCOS and Image2Latex framework to detect and extract math formulas from images and translate them accurately into LaTeX in a reasonable time frame.