.. _chapter_moldatabase:

********************************************************************************
Molecular Database Handling
********************************************************************************

.. index:: OEMolDatabase, file formats

The :ref:`OEMolDatabase <OEChem_OEMolDatabase>` class provides a fundamentally
different abstraction over a molecule file than the combination of
:ref:`oemolistream <OEChem_oemolistream>` and
:oe:func:`OEChem::OEReadMolecule`. The central underlying principle
utilized by this class is that many operations can be performed on a
molecular file without requiring the overhead of fully parsing the
molecule record into an :ref:`OEMolBase <OEChem_OEMolBase>` object. Instead, we
can think of a molecular file as a database that can be manipulated
with much cheaper operations than :oe:func:`OEChem::OEReadMolecule` and
:oe:func:`OEChem::OEWriteMolecule`.

.. _section_open_and_read:

Opening and Reading
================================================================================

:ref:`OEMolDatabase <OEChem_OEMolDatabase>` objects provide the ability to access
any molecule in a molecular database file in constant time,
``O(1)``. This is accomplished by paying the overhead of scanning the
file during the :oe:meth:`OEChem::OEMolDatabase::Open` call. However,
:oe:meth:`OEChem::OEMolDatabase::Open` is designed to operate extremely
fast on any molecule file format **OEChem TK**
supports. :oe:meth:`OEChem::OEMolDatabase::Open` is usually limited by
disk bandwidth instead of parsing and perception like
:oe:func:`OEChem::OEReadMolecule` and
:oe:func:`OEChem::OEWriteMolecule`.

After a database file is opened, the memory overhead of
:ref:`OEMolDatabase <OEChem_OEMolDatabase>` is minimal since no molecules are
stored in memory. Instead, the :ref:`OEMolDatabase <OEChem_OEMolDatabase>` only
stores a 8 byte file offset for each molecule record in the
file. :oe:src:`Listing 1<SNIPPET-NthMolecule>` demonstrates how to utilize this
feature to retrieve the "Nth" molecule from a molecule file using the
:oe:meth:`OEChem::OEMolDatabase::GetMolecule` method.

.. oe:source:: NthMolecule.cpp
   :language: c++
   :refnum:   1
   :caption:  Retrieving the Nth molecule in a file
   :frag:     SNIPPET

.. oe:source:: NthMolecule.py
   :language: python
   :refnum:   1
   :caption:  Retrieving the Nth molecule in a file
   :frag:     SNIPPET

.. oe:language:: python

	:ref:`Full listing. <listing-NthMolecule.py>`



.. oe:source:: NthMolecule.java
   :language: java
   :refnum:   1
   :caption:  Retrieving the Nth molecule in a file
   :frag:     SNIPPET

.. oe:source:: NthMolecule.cs
   :language: csharp
   :refnum:   1
   :caption:  Retrieving the Nth molecule in a file
   :frag:     SNIPPET

:oe:src:`Listing 1<SNIPPET-NthMolecule>` checks the return value of
:oe:meth:`OEChem::OEMolDatabase::GetMolecule` for false, indicating the
molecule record at that position in the file does not contain a valid
molecule. For example, molecules without any atoms are valid records
in :file:`.sdf` files.

.. note::

   :oe:meth:`OEChem::OEMolDatabase::Open` is still a ``O(N)`` operation
   in order to know the position of each molecule record in the
   file. However, this method is significantly cheaper than using
   :oe:func:`OEChem::OEReadMolecule`, instead being limited by hard
   disk bandwidth instead of processing speed. The
   :oe:meth:`OEChem::OEMolDatabase::Open` method can also be sped up by
   creating an associated :file:`.idx` file as described by the
   :ref:`section_index_files` section.

.. _section_direct_data_access:

Direct Data Access
================================================================================

:ref:`OEMolDatabase <OEChem_OEMolDatabase>` achieves much of its speed by treating
molecules as chunks of bytes instead of :ref:`OEMolBase <OEChem_OEMolBase>`
objects. This abstraction is leaked a little bit by providing users
the ability to access the raw bytes of the molecule record as well
through the :oe:meth:`OEChem::OEMolDatabase::GetMolecule` method that
takes a :ref:`oemolostream <OEChem_oemolostream>`. For example, the user could
pass this method a :ref:`oemolostream <OEChem_oemolostream>` that has been opened
with :oe:meth:`OEChem::oemolostream::openstring` in order to dump the
desired bytes to an in memory buffer. :oe:src:`Listing
2<SNIPPET-DatabaseSubset>` demonstrates how to use this feature to retrieve a
subset of molecules from a database file similar to how the ``LIMIT``
and ``OFFSET`` keywords work for an SQL query.

.. oe:source:: DatabaseSubset.cpp
   :language: c++
   :refnum:   2
   :caption:  Retrieving a subset of a file
   :frag:     SNIPPET

.. oe:source:: DatabaseSubset.py
   :language: python
   :refnum:   2
   :caption:  Retrieving a subset of a file
   :frag:     SNIPPET

.. oe:language:: python

	:ref:`Full listing. <listing-DatabaseSubset.py>`



.. oe:source:: DatabaseSubset.java
   :language: java
   :refnum:   2
   :caption:  Retrieving a subset of a file
   :frag:     SNIPPET

.. oe:source:: DatabaseSubset.cs
   :language: csharp
   :refnum:   2
   :caption:  Retrieving a subset of a file
   :frag:     SNIPPET


.. note::

   The :ref:`oemolostream <OEChem_oemolostream>` must be set up to write output to
   the exact same file format that the :ref:`OEMolDatabase <OEChem_OEMolDatabase>`
   is opened on. If file format conversion is also desired during the
   read operation, the user should user
   :oe:meth:`OEChem::OEMolDatabase::GetMolecule` to read the molecule
   into an :ref:`OEMolBase <OEChem_OEMolBase>` and then use
   :oe:func:`OEChem::OEWriteMolecule`.


.. _section_title_access:

Title Access
================================================================================

Molecule meta-data is often useful for manipulating databases
regardless of the molecule connection table. For this reason the
:ref:`OEMolDatabase <OEChem_OEMolDatabase>` provides access to the molecule title
through the :oe:meth:`OEChem::OEMolDatabase::GetTitle` method. The
:oe:meth:`OEChem::OEMolDatabase::GetTitle` returns the same string that
would be returned by :oe:meth:`OEChem::OEMolBase::GetTitle` if the
molecule was read in with :oe:func:`OEChem::OEReadMolecule`. The
difference is that the :oe:meth:`OEChem::OEMolDatabase::GetTitle`
method is more efficient because it will only parse the title from the
molecule record, and skip the rest of the bytes in the molecule
record. :oe:src:`Listing 3<SNIPPET-DatabaseExtract>` demonstrates how to use
:oe:meth:`OEChem::OEMolDatabase::GetTitle` to implement a more
efficient version of the :oe:src:`molextract` example.


.. oe:source:: DatabaseExtract.cpp
   :language: c++
   :refnum:   3
   :caption:  Extract molecules by title
   :frag:     SNIPPET

.. oe:source:: DatabaseExtract.py
   :language: python
   :refnum:   3
   :caption:  Extract molecules by title
   :frag:     SNIPPET

.. oe:language:: python

	:ref:`Full listing. <listing-DatabaseExtract.py>`



.. oe:source:: DatabaseExtract.java
   :language: java
   :refnum:   3
   :caption:  Extract molecules by title
   :frag:     SNIPPET

.. oe:source:: DatabaseExtract.cs
   :language: csharp
   :refnum:   3
   :caption:  Extract molecules by title
   :frag:     SNIPPET

.. note::

   Multi-conformer :file:`.oeb` files can have multiple titles per
   molecule record. The top-level :ref:`OEMCMolBase <OEChem_OEMCMolBase>` can
   have a title, as well as each :ref:`OEConfBase <OEChem_OEConfBase>` can have
   a title. :oe:meth:`OEChem::OEMolDatabase::GetTitle` will only return
   the title of the top-level :ref:`OEMCMolBase <OEChem_OEMCMolBase>` object and
   make no attempt to search for a title among the conformer data. In
   practice, this is fine since *OMEGA* will leave the
   :ref:`OEMCMolBase <OEChem_OEMCMolBase>` title the same as the input file, and
   append warts to the individual conformer titles.

.. _section_index_files:

Index Files
================================================================================

The speed of :oe:meth:`OEChem::OEMolDatabase::Open` is limited by how
fast data can be read from disk. For this reason, file position
offsets can be precomputed and stored in a parallel :file:`.idx`
file. :oe:meth:`OEChem::OEMolDatabase::Open` will automatically detect
the presence of this file based upon the file name of the file being
opened and use those file offsets instead. For example, if the
database file is called, "my_corporate_conformers.oeb",
:oe:meth:`OEChem::OEMolDatabase::Open` will look for a file named
"my_corporate_conformers.oeb.idx" to open as an index file. If an index
file can not be located, a full file scan will occur instead. For
files written once, and read many times, it can be highly beneficial
to create a parallel index file with the
:oe:func:`OEChem::OECreateMolDatabaseIdx` function.

.. note::

   :oe:meth:`OEChem::OEMolDatabase::Save` will automatically create a
   :file:`.idx` file parallel to the file being saved. This behavior
   can be modified by the
   :oe:meth:`OEChem::OEMolDatabaseSaveOptions::SetWriteIdx` method on
   the :ref:`OEMolDatabaseSaveOptions <OEChem_OEMolDatabaseSaveOptions>` options class.

.. seealso::

   * :oe:func:`OEChem::OEGetMolDatabaseIdxFileName` for the default way
     index file names are created.

.. _section_database_generic_data:

Database Generic Data
================================================================================

:ref:`OEMolDatabase <OEChem_OEMolDatabase>` inherits from
:ref:`OEBase <OESystem_OEBase>`, allowing it to contain and round-trip
generic data as described in the :ref:`chapter_genericdata`
chapter. This data is only written and read from :file:`.oeb`
files. It is stored in the :file:`.oeb` file as a
:ref:`OEHeader <OESystem_OEHeader>` record at the beginning of the
file. :oe:meth:`OEChem::OEMolDatabase::Save` will write this record
back out the :file:`.oeb` file so that it be read by a subsequent
:oe:meth:`OEChem::OEMolDatabase::Open` operation.

.. _section_database_caveats:

Caveats
================================================================================

.. warning::

   :ref:`OEMolDatabase <OEChem_OEMolDatabase>` will make an uncompressed copy of a
   molecular database file when opened on a :file:`.gz`, GZipped
   file. The temporary file will be deleted upon the destruction of
   the :ref:`OEMolDatabase <OEChem_OEMolDatabase>` object, however, it is still
   recommended to not use :file:`.gz` files with
   :ref:`OEMolDatabase <OEChem_OEMolDatabase>` if it can be avoided. If a
   different directory is desired for the uncompressed file, alter the
   environment variables used by the
   :oe:func:`OEPlatform::OEFileTempPath` function.

The largest caveat when working with :ref:`OEMolDatabase <OEChem_OEMolDatabase>`
objects is that they require a file on disk to provide storage. This
allows the object to have a very small in-memory footprint at the
expense of higher latency access to individual molecule records. The
:ref:`OEMolDatabase <OEChem_OEMolDatabase>` object makes no attempt to cache
molecules that may be frequently read, instead leaving this up to the
user at a higher level, or up to the operating system to cache frequently
accessed disk pages.

The above reason is why :file:`.gz` files are not well supported. The
:ref:`OEMolDatabase <OEChem_OEMolDatabase>` needs to be able to read a molecule
record by seeking to a particular location in the file. Though the
result is that multi-threaded applications can efficiently call
``const`` :ref:`OEMolDatabase <OEChem_OEMolDatabase>` methods like
:oe:meth:`OEChem::OEMolDatabase::GetMolecule` without any
synchronization overhead.
