Merging Differential Updates in In-Memory Column Store

To meet the performance requirements of enterprise application for both, transactional application as well as analyt-ical scenarios, data storage of in-memory databases are split into two parts: One optimized for reading and a write-optimized differential buffer. The read-optimized main storage together with the differential buffer for inserts provide the current state of the database. In regular intervals the differential buffer is merged with the main database to maintain compression and query performance. This merge process runs asynchronous to minimize the impact on query performance. However, simple duplication of the data structures prior to the merge process lead to a main memory consumption of at least twice the size of the database. In this paper we propose a differential merge update based on single columns. In typical enterprise application data environments this leads to a significant reduction of memory consumption as this type of applications tend to store transactional data in very large single tables. The Single Column Merge has been implemented in HYRISE and proved in a test scenario based on real enterprise data.