preview

Benchmarking Lmdb And Leveldb For Deep Learning

Better Essays
Open Document

Benchmarking LMDB and LevelDB for deep learning
Weiyue Wang
ABSTRACT
Deep learning is a new emerging area of machine learning research, which has been shown to produce state-of-the-art results on various tasks. A high performance database management in deep learning framework will help increase learning efficiency. This work compares the performance of two key value data storage, Lightning Memory-Mapped Database (LMDB) and google LevelDB, for deep learning framework. Our key findings are followings. 1.Introduction
Deep Learning (DL) has been shown to outperform most traditional machine learning methods in fields like computer vision, natural language processing, and bioinformatics. DL seeks to model high-level abstractions of data by constructing multiple layers with complex structures, which compose of hundreds millions of parameters to be tuned. For example, a deep learning structure for processing visual and other two-dimensional data, convolutional neural network (CNN) [1], which consists of three convolutional layers and three pooling layers, has more than 130 millions of parameters if the input has 28x28 pixels. While these large neural networks are powerful, we need high amount of training data. DL tasks need considerable data storage and memory bandwidth.
Key-value stores provide users simple yet powerful interface to data storage, which are often used in complicated systems. [2] LMDB is a framework that provides high-performance key-value storage

Get Access
Related
Get Access