The Berkley Database

Introduction to Berkeley DB

Berkeley DB is an open source library which provides a high performance embedded data management with various programming languages. This library provides a simple function-call API for data access and management. Berkeley DB is originated at the University of California, Berkeley and from the beginning it is distributed under open source license agreement for that the complete source code for is freely available for download and use.

Berkeley DB provides an embedded database management because it directly links into the application itself and runs in the same address space. This eliminates the need to inter-process communications in the internal machine and between external machines over the network. For that the cost of maintaining a communication between processes is replaced with the cost of making function calls since the latter one is much less costly. Once Berkeley DB is linked to the application the end user generally does not know that there's a database present at all.

Berkeley DB provides a simple function-call API for the programming languages like C, C++, Java, Perl, Tcl, Python, and PHP. For that any system built in these language can easily reference to Berkeley DB library and create a database management. This library handles all the database operations inside, even the low-level services like locking, transaction logging, shared buffer management, memory management. This enables multiple processes or threads to use the database at the same time.

Data Management with Berkeley DB

Berkeley DB provides a relatively simple data management when compared to the commonly used modern database management software products.

All the records in Berkeley DB is as treated as key-value pairs in which value part is simply the payload of the key. Berkeley DB only operates on the key part of the record. There are only a few logical record-operations can be performed in Berkeley DB. These are:

   * Inserting a record
   * Deleting a record
   * Finding a record by its key
   * Updating a record

There is no specific record format in Berkeley DB. Key-value pairs are byte strings that can be either in fixed or variable length. Database developers can put data structures into the database before converting them to any record format. However, in order to perform storage and retrieval operations applications should know what the structure of key and value is. That is because Berkeley DB does not holds this information and should always be fed by the application. Even if Berkeley DB has this disadvantage of not providing the programmer the information on the contents or the structure of the values, it literally is limitless on the data types that can be stored in a Berkeley DB database. Berkeley is able to work on any data type determined by the programmer no matter how complex it is.

In Berkeley DB the size of the keys and values can be up to four gigabytes for that a single record can store images, audio/video streams or other large data types. Management of this larger values requires no specific management. They are simply broken into page-sized chunks, and reassembled on demand when needed.

What Berkeley DB is not

Berkeley DB is not a relational database it does not support SQL queries. Data access can only be managed by the Berkeley DB API function calls.

In relational databases, users simply can write queries in a high level language to reach the data stored in the database since database knows everything about the content and the structure of the data. This makes database management simpler and eliminates to need of programming. However, if the programmer can be supplied with enough information about how an application will access data, writing a program to manage the database considerably fastens the operations, since the overhead of query parsing, optimization, and execution is removed. This puts the workload on the programmer however when the program is written, application performs much faster.

Berkeley DB does not holds a schema in the way that relational databases do. In relational databases, schema provides data about the tables and the relationships between tables. However in Berkeley DB, there is no such a storage since every single table is treated as a database and relations between tables should be maintained by the programmer.

Berkeley DB does not know about the structure of the value part of the record for that cannot divide the value part into its constituent parts. In order to use the data stored in different parts of the value an application, which knows the data structures, should be provided. Unlike relational databases, Berkeley DB does not support indexing on the tables or any automatic management is not provided. If the programmer needs indexing s/he should implement the routines responsible for index management.

Relational databases high-level database access on the other hand Berkeley DB is a high-performance, transactional library for data storage. It is possible for the programmer to built a relational system on the top of Berkeley after creating routines managing all the relations between different record types.

Berkeley DB is not a standalone database server, it is a library running in the same address space of the application that it is called by. However, this does not prevents different applications using the same database at the same time since the library itself handles all the threads and coordination among different applications. Berkeley DB guarantees that different applications linked with the same database do not interfere each others work.

Berkeley DB can be used to build a data management server application. For example, many Lightweight Directory Access Protocol (LDAP) servers uses Berkeley DB for record storage. When LDAP clients connect to these servers to ask for records servers make Berkeley DB API calls to find records and return them to the clients. However, again it is programmer to implement the code to perform these server operations.

Access Methods in Berkeley DB

Berkeley DB provides four access methods which are Btree, Hash, Queue and Recno.

* Btree: This is a method of sorted, balances tree structure. All the operations in the tree take O(log base_b N) time where base_b is the average number of keys per page, and N is the total number of keys stored.

* Hash: This access method is an implementation of Extended Linear Hashing.

* Queue: This access method stores fixed-length records where the logical record numbers are the keys.

* Recno: This access method stores both fixed and variable-length records where the logical record numbers are the keys.

Selecting a proper access method is an important part of database management in Berkeley DB since different access methods may have different efficiency results among different applications.

Most applications using Berkeley DB chooses between Btree or Hash access methods or between Queue and Recno. These two pairs mostly provides similar functionality.

The Btree and Hash access methods should be used when logical record numbers are not the primary key used for data access. Btrees store keys in sorted manner for that there is a relationship determined by that sort order. For that, the Btree access method should be used when there is any local relation among keys.

The performance of the Hash and Btree access methods are mostly similar on small data sets, However when a data set becomes larger, the Hash access method can perform better since it contains less metadata pages than Btree databases. In Btree the metadata pages can begin to dominate the cache.

The Queue or Recno access methods are prefered when logical record numbers are the primary key used for data access. Queue access method provides record level locking and so that it supports considerably higher levels of concurrency the Recno. On the other, hand Recno access method supports variable record length and databases having a permanent flat text file storage.

Berkeley DB Databases

In Berkeley DB a database is a collection of records(key-value) pairs. Unlike high level database management systems Berkeley DB handles each collection of records as a single database. For that, when we say database we are referring to a single table of records. A database is like a table having two columns one for key one for value. Both of these columns are are managed by DBT structures. These structures are used to putting, getting and deleting records from the database they are assigned.

Opening and Closing Databases

In order to open a database, first a DB handle should be initialized with the function db_create(). After that you can open the database by calling the method open(). DB does not creates databases if they do not already exist in order to specify this behavior you should use the flag DB_CREATE flag on the open() method.

Below is a an example code on opening a Berkeley DB database:

    DB *dbp;			/* Database structure handle */
    u_int32_t flags;		/* Database open flags */
    int ret			/* Return value to check error */

    /*Initialize the DB */
    ret = db_create(&dbp, NULL, 0);
    if(ret != 0)
    { 
       /* if creating db handle failed print an error message */    
    } 

    /* if database does not exist create it */
    flags = DB_CREATE;
    /* Open the database */    
    ret = dbp->open(dbp,     	/* DB structure pointer */
                       NULL,           /* Transaction pointer */
                       DBDBCELLMBR,	/* File that holds the database. */
                       NULL,           /* Optional logical database name */
                       DBTREE,         /* Database access method */
                       flags,          /* Open flags */
                       0);             /* Default file mode (using defaults) */
       if(ret != 0)
       {
           printf(MSG05);       /* if opening database failed print an error message */
           return 5;
       }

In order to close a database the DB handle close method should be called:

       ret = dbp->close(dbp, 0);
       if (ret != 0)
       {
         printf(MSG01);	/* if closing dbp failed print an error message */
         return 1;			/* Return error value of the message*/
       }

Database Open Flags

Below is a list of some of the flags that you may want to use at database open:

   * DB_CREATE: If the database does not currently exists, create it.
   * DB_EXCL: Create database exclusively. If the database the database already exists open fails. This flag is only meaningful when used with DB_CREATE.
   * DB_RDONLY: Opens the database only for read operations. Any database write attempt fails.
   * DB_TRUNCATE: Empty the on-disk file that contains the database. All databases physically contained in that file are deleted.

Databases in Environments

Environments are widely used in many Berkeley DB applications. Since all the tables are treated as different databases in Berkeley DB a different structure is needed to encapsulate multiple databases. When the databases are linked with an environment they are created in the environment's home directory. It is important to note that an environment directory must be created before any open attempts. Below is an example of a database management in environments:

       DB_ENV *db_env;        /* Environment structure handle */
       DB *dbp;               /* Database structure handle */
       u_int32_t flags;       /* Database open flags */
       u_int32_t env_flags    /* Env open flags */

       /* Create environment */
       ret = db_env_create(&db_env, 0);
       if(ret != 0)
       {
           printf(MSG02);	/* If creating environment failed print an error message */
           return 2;		/* Return error value of the message*/
       }  

       /* set environment flags */
       env_flags = DB_CREATE |				/* If the environment does not exist, create it. */
                   DB_INIT_MPOOL;			/* Initialize the in-memory cache. */

       /* Open environment */
       ret = db_env->open(db_env,	/* DB_ENV ptr */
                          DBNOMENV,	/* env home directory */
                          env_flags,	/* Open flags */
                          0);          /* File mode (default) */
       if((ret != 0)
       {
           printf(MSG03);      /* If opening environment failed print an error message */
           return 3;
       }
       /* Create database handler */
       ret = db_create(&dbp, db_env, 0);
       if(ret != 0)
       {
         printf(MSG04);	/* if creating db handle failed print an error message */
         return 4;				/* Return error value of the message*/
       }
       
       flags = DB_CREATE;		/* if database does not exist create it*/
       ret = dbp->open(dbp,     	/* DB structure pointer */
                       NULL,           /* Transaction pointer */
                       DBDBCELLMBR,	/* File that holds the database. */
                       NULL,           /* Optional logical database name */
                       DBTREE,         /* Database access method */
                       flags,          /* Open flags */
                       0);             /* Default file mode (using defaults) */
       if(ret != 0)
       {
           printf(MSG05);       /* if opening database failed print an error message */
           return 5;
       }

Berkeley DB Database Records

In Berkeley DB each record is composed of key and value pairs. Both of these pairs requires DBT structures to enable the reach to data. Certain operations -using DBT structures- on database records are detailed in the following sections.