Concurrent Scanning through Adaptive Task Distribution for Simultaneous Lexing on Multi-Core Platforms

The integration of advanced multi-core architectures is swiftly becoming customary in the realm of digital technology. This can be credited to their remarkable parallel computational capabilities, leading to substantial enhancements in performance. To fully leverage the potential of multi-core systems, a redesign of system software, including compilers, is imperative to accommodate parallel processing. Over time, notable progress has been made in enhancing the effectiveness of lexing through utilizing the inherent concurrent computational capabilities present in computer systems with multiple cores. This current advancement in implementation highlights the superiority of a concurrent lexical analyzer in contrast to its conventional sequential equivalent, especially during the execution of lexing tasks. The research explores the strategy of incorporating parallelism during the source code scanning phase within lexing process.The primary objective of this work is to elucidate the techniques employed to achieve parallel lexical analysis. Within the realm of processors with multiple cores, various instances of the lexer application can be executed simultaneously. This enables token identification by scanning many lines in the input stream at the same time. Tasks are assigned on a per-line basis to idle cores, thus enabling efficient parallel processing. Both theoretical analysis and empirical observations confirm that the introduced technique offers a significant advantage over the sequential approach regarding tokenization performance. Significantly, it diminishes the duration needed for tokenization process in the compilation. The results conclusively demonstrate that the efficiency of the parallel process in lexical analyzer is expected to exhibit a linear relationship with the cores employed in process. It is clear that the acceleration will increase with a higher count of CPU cores, potentially further expediting the compilation method.