Insert-only for Time Travel

Insert-only or append-only describes how data is managed when inserting new data. The principle idea of insert-only is that changes to existing data are handled by appending new tuples to the data storage. In other words, the database does not allow applications to perform updates or deletions on physically stored tuples of data.

This design approach allows the introduction of a specific write-optimized data store for fast writing and a read-optimized data store for fast reading.Traditional database systems support four operations for data manipulations, i.e. inserting new data, selecting data, delete data, and updating data. The latter two are considered as destructive since original data is no longer available after its execution. In other words, it is neither possible to detect nor to reconstruct all values for a certain attribute; only the latest value is available. Insert-only enables storing the complete history of value changes and the latest value for a certain attribute. For instance, this is also a foundation of all bookkeeping systems to guarantee transparency. For the history-based access control, insert-only builds the basis to store the entire history of queries for access decision. In addition, insert-only enables tracing of access decision, which can be used to perform incident analysis.

Please also see our podcast on this technology concept.

High Performance In-Memory Cloud Services

Cloud Services in combination with high performance in-memory computing will change how enterprises work. Currently, most of the data is stored in silos of slow disk-based row-oriented database systems. Besides, transactional data is not stored in the same database as analytical data, but in separate data warehouses and gets replicated in batch jobs. Consequently, instant real time analytics are not possible and company leaders often have to make decisions in a very short time frame based on insufficient information.

This is about to change. In the last decade, hardware architectures have evolved dramatically. Multi core architectures and the availability of large amounts of main memory at low costs are about to set new breakthroughs in the software industry. It has become possible to store data sets of whole companies entirely in main memory, which offers a performance that is orders of magnitudes faster than disk. Traditional disks are one of the remaining mechanical devices in a world of silicon and are about to become the new tape, only necessary for backup. With in-memory computing and hybrid databases using row or column stores where appropriate, transactional and analytic data storages will be unified.

Cloud Computing will provide the elasticity, flexibility, and possibility to access all information of a company from everywhere whereas in-memory computing in combination with hybrid databases will enable systems to provide all information in real-time. With clouds, the IT infrastructure and data storage can automatically grow with the demand of the company. Even in case of an unpredictable higher workload, tomorrow’s systems will automatically request the necessary amount of computing power and will, therefore, be able to guarantee higher service availability at reduced costs. Therefore, current software management and monitoring systems have to be adapted and modified for the new in-memory computing paradigm.

Clouds will be totally stored in RAM and, in consequence, durable storage such as disks will only be used for backup purposes. Updates will not be stored synchronously on disk but will be replicated synchronously to further servers in order to guarantee data consistency even in case of hardware failures. For enterprise environments, clouds will not base on commodity hardware, but on high-end servers, which offer a much higher reliability and higher performance. By providing information from everywhere in real-time at high availability but low costs, such new technologies will offer enterprises entirely new opportunities how businesses are run and operated.

At the chair of Prof. Dr. Hasso Plattner and Dr. Alexander Zeier at the Hasso-Plattner Institute, we are conducting research projects how enterprises can access information in real-time. We built an in-memory hybrid database that unifies the advantages of column- and row oriented database systems. In corporation projects with SAP, we could show that with in-memory column-oriented databases the time for business transactions, like dunning, can be reduced from 20 min to 1s. We are also augmenting Available-to-Promise applications with real time analytics and flexible order fulfillment. We are conducting research how multi-tenant analytics can be provided through a cloud-based infrastructure. Besides, we are analyzing architectures for clouds that are totally stored in main memory and how software management systems can be adapted to provide high service availability at low costs.

Author: Jan Schaffner

New Approach for a Cloud Based OLTP System

In this blog entry, I want to summarize some emerging ideas about, how OLTP can be performed on distributed systems by weakening consistency.
Scaling out servers is a common approach to achieve a higher performance or higher throughput. If systems are scaled out, more servers are used to handle workload. Using more servers often involves the usage of distributed transactions and partitioning. However, distributed transactions according to ACID which can be achieved using two phase commits are expensive, as they increase the latency of a transaction and weaken the availability or the resilience to network partitions.

CAP Theorem and ACID 2.0

According to the CAP theorem, consistency, availability, and partitioning cannot be achieved at the same time. Since availability is a crucial business factor and a system should also be tolerate network partitions, it is common in key value stores to offer reduced consistency guarantees in order to achieve full availability.
According to Helland (Helland 2007), a transaction should only involve single entities that are stored on the same instance. If a transaction should involve more than one entity, the business logic developer has to deal with potential inconsistencies. Between groups of entities the system could support store-and-forward messaging.
Helland defines a new kind of consistency that is called ACID 2.0 that stands for Associative, Commutative, Idempotent, and Distributed (Helland 2009). Main goal for ACID 2.0 is to succeed if the pieces of work happen: at least once, anywhere in the system, and in any order.

Fault Tolerance

Another issue is fault resilience. A common approach to increase fault resilience and availability is to use replication, e.g. store multiple copies of the data. However, storing two copies of data synchronously leads to the issue of increasing latency. Helland proposes to asynchronously replicate updates of data to a second server and let the business application developer cope with the probability that data may be lost in case of errors. For transactions involving low business values, it might work to live with inconsistencies or lost data. In contrast, for transactions with high business values, the system should provide guaranteed consistency and durability of a transaction.
For providing fault tolerance, single transactions need to be idempotent, which means that executing a single transaction twice should not lead to a substantive change of an entity, whereas the application shall define what substantive really means. To achieve idempotent transactions, each transaction has to be versioned or hash values over the request have to be stored, to reject transactions that have already been executed. For that purpose, Helland proposes to store all activities of related entities, e.g. the received messages.
The ideas proposed by Helland are very interesting. However, further research has to show whether it is really feasible and viable to let the business application developer cope with data inconsistencies.

References

(Helland 2007) Pat Helland: Life beyond Distributed Transactions: an Apostate's Opinion. CIDR 2007: 132-141
(Helland 2009) Pat Helland, David Campbell: Building on Quicksand. CIDR 2009

Author: Benjamin Eckart