OCSL: An Online Compression Scheme for Streaming Semi-Structured Logs

The rapid growth of log data poses a serious challenge to storage cost. Some works have realized the coding replacement of log data by constructing template, which effectively improves the compression rate of log data. However, their compression effect is not satisfactory in the real-time compression scenario of semi-structured logs. In this paper, we propose OCSL, an Online Compression scheme for Streaming semi-structured Log data. OCSL first performs online parsing and word frequency statistics on the log data, adds the words that may be frequently accessed in the future to the Token dictionary, and then encodes and compresses the recurring patterns via the global shared template and TokenList dictionary, and finally sends them to the general compression tool for block compression. Speciallly, OCSL introduces two indicators, dictionary growth rate and utilization rate, to balance encoding income and dictionary growth speed, and adopts “one write, multiple reads” dictionary structure to reduce the impact of compression on system performance. The real industrial data of Tencent was used to carry out the experiment. The results show that OCSL's compression ratio can reach 83.75, which is 253% higher than that of LogReducer. Meanwhile, OCSL can compress up to 230MB/s, which is 8.8x faster than that of LogReducer.