.. _chapter_multithreading:

********************************************************************************
Multi-Threading
********************************************************************************

**OpenEye** deems only the following toolkits "thread-safe":

 - OEChem TK (including oeplatform, oesystem, oechem, and oebio)
 - OEDepict TK
 - OEDocking TK
 - Grapheme TK
 - GraphSim TK
 - Lexichem TK
 - Omega TK

.. warning::

   Absolutely no guarantee is made about the thread-safety of any
   other library not on the preceding list.

The most common reason to multi-thread a program is for
performance. However, threading has certain overhead costs associated
with it. The user should profile their application to determine the
best possible way to parallelize it. There is no guarantee
that multi-threading a program will make it run faster, in fact, it
can make it run dramatically more slowly. Some of the examples presented in
this chapter will perform more slowly than their single-threaded counterparts. 
They were chosen for the simplicity of demonstrating certain
concepts. The user should perform timing analysis on the code in this
chapter on their own systems to determine whether the methods are
appropriate for them.

.. oe:language:: c++ python

   .. warning::

      By default, it is not safe to pass instances of the following
      **OpenEye Toolkit** objects between threads:

      - Any **OpenEye Toolkit** :ref:`iterator object
        <section_atombondtraversal_iter>`, i.e., something derived
        from :ref:`OEIterBase <OESystem_OEIterBase>`.
      - Any :ref:`OEGetNbrAtom <OEChem_OEGetNbrAtom>` functor. This is used by
        some implementations of
        :oe:meth:`OEChem::OEAtomBase::GetAtoms` that returns an
        iterator as well, see previous.
      - The :ref:`OEMatch <OEChem_OEMatch>` class. Used by almost any API
        that operates with an instance of
        :ref:`OEMatchBase <OEChem_OEMatchBase>`.
      - :oe:const:`OEChem::OEMolBaseType::OEMiniMol` instances of
        :ref:`OEGraphMol <OEChem_OEGraphMol>`.
      
      Doing so will likely cause a crash due to **OpenEye Toolkits**
      use of thread-local memory caches that can not be shared across
      threads. If a multi-threaded **OpenEye Toolkit** program
      crashes, a good first solution to try is inserting the following
      at the beginning of ``main``.

      .. oe:language:: c++

         .. code-block:: c++

            OESetMemPoolMode(OEMemPoolMode::System);

      .. oe:language:: python

         .. code-block:: python

            OESetMemPoolMode(OEMemPoolMode_System)

      However, there is a performance cost to turning off all
      **OpenEye Toolkits** memory caches using the above. It may be
      possible to gain that performance back by using an allocator
      better tuned to the specific application. For multi-threaded
      applications, the following allocators are known to be very
      performant:

       - `TCMalloc <http://goog-perftools.sourceforge.net/doc/tcmalloc.html>`_
       - `Hoard <http://www.hoard.org>`_

      .. note:: 

         Prior to the 2015.Oct release, molecule and substructure
         search objects could not be shared between threads. This
         could easily cause crashes to occur whenever using the
         following classes inside an :ref:`OEOnce <OEPlatform_OEOnce>`
         construct:
         
         - :ref:`OEGraphMol <OEChem_OEGraphMol>` (note:
           :oe:const:`OEChem::OEMolBaseType::OEMiniMol` is still not
           safe)
         - :ref:`OEMol <OEChem_OEMol>`
         - :ref:`OEQMol <OEChem_OEQMol>`
         - :ref:`OESubSearch <OEChem_OESubSearch>`

         The above objects can now be safely passed between threads
         and used in :ref:`OEOnce <OEPlatform_OEOnce>` function local
         statics. As of 2015.Oct, the above objects now manage their
         own memory pools on a per-object basis. This can possibly
         lead to more memory waste, as each molecule has enough space
         to store 64 atoms and bonds by default, even if it is
         empty. This is why the
         :oe:const:`OEChem::OEMolBaseType::OEMiniMol` molecule
         implementation uses the older memory pooling, as that is
         designed for memory efficiency.


.. _section_inputandoutputthreads:

Input and Output Threads
================================================================================

**OEChem TK** provides a rather advanced mechanism for spawning a thread to
handle I/O asynchronously. Parallelizing I/O operations can yield
significant performance gains when dealing with  bulky file
formats like SDF and MOL2, or when dealing with files on network
mounted directories. Naturally, performance gains may vary based upon
the system configuration in question.

The input and output threads are encapsulated inside the
:oe:cls:`oemolthread<OEChem::oemolthreadbase>`
abstraction. :oe:cls:`oemolthreads<OEChem::oemolthreadbase>` are
designed to behave as similarly as possible to
:oe:cls:`oemolstreams<OEChem::oemolstreambase>`. This should allow easy
drop in replacement to existing code by simply changing the type of a
variable from :ref:`oemolistream <OEChem_oemolistream>` to
:ref:`oemolithread <OEChem_oemolithread>` for input, and
:ref:`oemolostream <OEChem_oemolostream>` to :ref:`oemolothread <OEChem_oemolothread>` for
output.

For example, the :oe:src:`MolFind<MolFind>` program described in the
:ref:`section_patternmatch_sss` section can be modified to the nearly
identical program in :oe:src:`Listing 1<SNIPPET-ThreadedMolFind>`.

.. oe:source:: ThreadedMolFind.cpp
   :language: c++
   :refnum:   1
   :caption:  Using Threaded Input and Output
   :frag:     SNIPPET

.. oe:source:: ThreadedMolFind.py
   :language: python
   :refnum:   1
   :caption:  Using Threaded Input and Output
   :frag:     SNIPPET

.. oe:language:: python

	:ref:`Full listing. <listing-ThreadedMolFind.py>`



.. oe:source:: ThreadedMolFind.java
   :language: java
   :refnum:   1
   :caption:  Using Threaded Input and Output
   :frag:     SNIPPET

.. oe:source:: ThreadedMolFind.cs
   :language: csharp
   :refnum:   1
   :caption:  Using Threaded Input and Output
   :frag:     SNIPPET

.. note:: 

   Both :oe:cls:`oemolstreams<OEChem::oemolstreambase>` and
   :oe:cls:`oemolthreads<OEChem::oemolthreadbase>` should be evaluated
   for performance. :oe:cls:`oemolthreads<OEChem::oemolthreadbase>`
   increase user time CPU usage to save on potentially expensive
   system calls.

The difference in :oe:src:`Listing 1<SNIPPET-oemolthread-ThreadedMolFind>` may be a little
difficult to spot for the experienced **OEChem TK** programmer used to
seeing :oe:cls:`oemolstream<OEChem::oemolstreambase>` declarations,
however, the following two declarations are not
:oe:cls:`oemolstreams<OEChem::oemolstreambase>`. They are
:oe:cls:`oemolthreads<OEChem::oemolthreadbase>` meant to behave as
closely as possible to :oe:cls:`oemolstreams<OEChem::oemolstreambase>`.

.. oe:source:: ThreadedMolFind.cpp
   :language: c++
   :frag:     SNIPPET-oemolthread

.. oe:source:: ThreadedMolFind.py
   :language: python
   :frag:     SNIPPET-oemolthread

.. oe:source:: ThreadedMolFind.java
   :language: java
   :frag:     SNIPPET-oemolthread

.. oe:source:: ThreadedMolFind.cs
   :language: csharp
   :frag:     SNIPPET-oemolthread

However, :oe:cls:`oemolthreads<OEChem::oemolthreadbase>` could not be
made to behave *exactly* like
:oe:cls:`oemolstreams<OEChem::oemolstreambase>`. The following key
differences between :oe:cls:`oemolthreads<OEChem::oemolthreadbase>` and
:oe:cls:`oemolstreams<OEChem::oemolstreambase>` should be taken into
account:

:oe:cls:`oemolthreads<OEChem::oemolthreadbase>` do not support stream
methods: :oe:meth:`read<OEChem::oemolistream::read>`,
:oe:meth:`seek<OEChem::oemolistream::seek>`,
:oe:meth:`size<OEChem::oemolistream::size>`, and
:oe:meth:`tell<OEChem::oemolistream::tell>`.

   These do not make much sense in a multi-threaded environment where
   the I/O thread can be at any arbitrary location in the file. Using
   the methods would be prone to race conditions.

The :ref:`oemolithread <OEChem_oemolithread>` default
:oe:meth:`constructor<OEChem::oemolithread::Constructors>` does not
default to ``stdin``.

   The default constructor of :ref:`oemolistream <OEChem_oemolistream>` will open
   the stream on ``stdin``. However, read operations are handled by a
   separate opaque I/O thread inside the
   :ref:`oemolithread <OEChem_oemolithread>`. The I/O thread uses blocking reads
   that do not return unless the user closes ``stdin`` of the
   process. :oe:cls:`oemolithreads<OEChem::oemolithread>` can still
   read from ``stdin`` using the same syntactic trick described in the
   :ref:`section_molreadwrite_commandlineformatcontrol` section
   supplied to :oe:meth:`open<OEChem::oemolstreambase::open>` or the
   :oe:meth:`constructor<OEChem::oemolithread::Constructors>` which
   takes a ``string``.
  
The :ref:`oemolithread <OEChem_oemolithread>` does not support the
:oe:meth:`SetConfTest<OEChem::oemolistream::SetConfTest>` interface.

   As will be described later, the I/O threads inside
   :oe:cls:`oemolthreads<OEChem::oemolthreadbase>` do a minimal amount
   of work to maximize scalability in parallel applications. The work
   described in the :ref:`section_oemol_inputandoutput` section to
   recover multi-conformer molecules from single-conformer file formats
   can be quite expensive. Note, this does not preclude
   :oe:cls:`oemolithreads<OEChem::oemolithread>` from reading
   multi-conformer molecules from file formats that support them
   inherently, *e.g.*, multi-conformer molecules can be read out of
   multi-conformer OEB files through
   :oe:cls:`oemolithreads<OEChem::oemolithread>`.

With all the preceding considerations taken into account the real gain
of :oe:cls:`oemolthreads<OEChem::oemolthreadbase>` is that
:oe:func:`OEChem::OEReadMolecule` and :oe:func:`OEChem::OEWriteMolecule`
can be safely called from multiple threads onto the same
:oe:cls:`oemolthread<OEChem::oemolthreadbase>`. They are guaranteed to
be free of race conditions when applied to
:oe:cls:`oemolthreads<OEChem::oemolthreadbase>`. Not only that, the
expensive parsing (:oe:func:`OEChem::OEReadMolecule`) and serialization
(:oe:func:`OEChem::OEWriteMolecule`) occurs in the thread that calls
the function, not in the :oe:cls:`oemolthread<OEChem::oemolthreadbase>`
itself. The :oe:cls:`oemolthread<OEChem::oemolthreadbase>` takes care
of I/O and compression if reading or writing to a ``.gz`` file.

.. _figure_oemolithread:

.. figure:: images/oemolithread.png
   :align:  center
   :scale:  100

   **oemolithread**

   *Diagram of data flow inside an oemolithread*

.. _figure_oemolothread:

.. figure:: images/oemolothread.png
   :align:  center
   :scale:  100

   **oemolothread**

   *Diagram of data flow inside an oemolothread*

:ref:`Figure: oemolithread<figure_oemolithread>` and 
:ref:`Figure: oemolothread<figure_oemolothread>` shows how
data flows between an arbitrary number of worker threads and the I/O
thread. Molecule data passes through a protected bounded buffer. It is
"protected" against race conditions when multiple threads operate on
the buffer. It is "bounded" in size to prevent increasing the required
amount of memory by inserting many items into the buffer faster than
can be consumed on the other side. The I/O threads will be put to
sleep, allowing other threads to use the CPU, when there is not a
sufficient amount of work for them to do. This means the number of
worker threads should typically be the number of available CPUs. The
I/O threads don't require much CPU time, therefore, they don't need
their own dedicated CPU. This could change as the number of worker
threads increases, or if the I/O thread is expected to handle data
compression or uncompression.

:oe:src:`Listing 2<SNIPPET-MultiCoreMolFind>` demonstrates how to scale the
:ref:`molecule searching <section_patternmatch_sss>` program to
multiple CPUs. A worker thread is dedicated for each CPU. There are
also I/O threads to handle the input and output operations. The data
parsing (:oe:func:`OEChem::OEReadMolecule`), substructure
searching(:oe:meth:`OEChem::OESubSearch::SingleMatch`), and data
serialization (:oe:func:`OEChem::OEWriteMolecule`) is all done in
parallel by the worker threads. The costly synchronization point of
reading and writing data is handled by the separate I/O threads.

.. oe:source:: MultiCoreMolFind.cpp
   :language: c++
   :refnum:   2
   :caption:  Using multiple threads to read and write to molthreads
   :frag:     SNIPPET

.. oe:source:: MultiCoreMolFind.py
   :language: python
   :refnum:   2
   :caption:  Using multiple threads to read and write to molthreads
   :frag:     SNIPPET

.. oe:language:: python

	:ref:`Full listing. <listing-MultiCoreMolFind.py>`



.. oe:source:: MultiCoreMolFind.java
   :language: java
   :refnum:   2
   :caption:  Using multiple threads to read and write to molthreads
   :frag:     SNIPPET

.. oe:source:: MultiCoreMolFind.cs
   :language: csharp
   :refnum:   2
   :caption:  Using multiple threads to read and write to molthreads
   :frag:     SNIPPET

.. oe:language:: c++

   .. note:: 

      Threading is automatically handled by `OpenMP
      <http://openmp.org>`_ pragmas.

.. oe:language:: python

   .. warning:: 

      The `global interpreter lock
      <http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock>`_
      can eliminate any chance of scaling across multiple
      processors. It depends how much of the run time is in C++ and
      how much is in Python. 

:oe:src:`Listing 2<SNIPPET-MultiCoreMolFind>` will generate the same set of
molecules in the output file as :oe:src:`Listing
1<ThreadedMolFind>`. However, there is big difference between the
output files, the molecule order is non-deterministic. For most users
this does not matter, database files are treated as arbitrarily
ordered sets of molecules. If the order of the molecule database is
important do not use the program in :oe:src:`Listing
2<SNIPPET-MultiCoreMolFind>`.

.. note:: 
   
   Database order is preserved by the program in :oe:src:`Listing
   1<SNIPPET-ThreadedMolFind>` because there is only one worker thread. This
   allows users with database order restrictions to still take
   advantage of threaded asynchronous I/O.

.. _section_threadsafety:

Thread Safety
================================================================================

**OEChem TK** is considered to be a "thread-safe" library. It makes the
same guarantees the C++ Standard Template Library (STL) makes in
concern of threads:

::

   Unless otherwise specified, standard library classes may safely be
   instantiated from multiple threads and standard library functions
   are reentrant, but non-const use of objects of standard library
   types is not safe if shared between threads. Use of non-constness
   to determine thread-safety requirements ensures consistent
   thread-safety specifications without having to add additional
   wording to each and every standard library type.

.. seealso::

   * `Thread-Safety in the Standard Library
     <http://www.open-std.org/JTC1/sc22/wg21/docs/papers/2008/n2669.htm>`_


Classes
----------

Unless otherwise stated in the API documentation, :ref:`OEChem objects
<OEChemClasses>` can be instantiated and used in multiple
threads. For example, in :oe:src:`Listing 2<SNIPPET-MultiCoreMolFind>`, special
care was taken to make the molecule object local to each thread.

.. oe:language:: c++
   
   This was accomplished through the private clause to the `OpenMP
   <http://openmp.org>`_ parallel pragma.

.. oe:language:: python

   This was accomplished by calling the
   ``OEChem::oemolithread::GetOEGraphMols``
   method in each thread. An :ref:`OEGraphMol <OEChem_OEGraphMol>` is
   constructed local to each
   ``OEChem::oemolithread::GetOEGraphMols``
   generator invocation.
   
.. oe:language:: java

   This was accomplished by making an :ref:`OEGraphMol <OEChem_OEGraphMol>`
   instance local to the thread's ``run`` method. Therefore, ensuring
   each thread had a separate :ref:`OEGraphMol <OEChem_OEGraphMol>` to read
   molecules into.

.. oe:language:: csharp

   This was accomplished by making an :ref:`OEGraphMol <OEChem_OEGraphMol>`
   instance local to the thread's ``run`` method. Therefore, ensuring
   each thread had a separate :ref:`OEGraphMol <OEChem_OEGraphMol>` to read
   molecules into.

.. oe:source:: MultiCoreMolFind.cpp
   :noindent:
   :language: c++
   :frag:     SNIPPET-privatemol

.. oe:source:: MultiCoreMolFind.py
   :noindent:
   :language: python
   :frag:     SNIPPET-privatemol

.. oe:source:: MultiCoreMolFind.java
   :noindent:
   :language: java
   :frag:     SNIPPET-privatemol

.. oe:source:: MultiCoreMolFind.cs
   :noindent:
   :language: csharp
   :frag:     SNIPPET-privatemol

The same object can be accessed from multiple threads if all access is
through a ``const`` method. A ``const`` method is any method that does
not change data internal to the class. In C++, a method can be
explicitly marked ``const``, allowing the compiler to cause the
code to not compile if this guarantee is not kept.

For example, the :oe:meth:`OEChem::OESubSearch::SingleMatch` method is
a ``const`` method. Therefore, the same :ref:`OESubSearch <OEChem_OESubSearch>`
object can be used from multiple threads without the need to make
local copies of the :ref:`OESubSearch <OEChem_OESubSearch>` object in
:oe:src:`Listing 2<SNIPPET-MultiCoreMolFind>`. However, notice that the
molecule argument passed to :oe:meth:`OEChem::OESubSearch::SingleMatch`
is local to each thread. Otherwise, one thread could be reading data
into the molecule object while another thread is trying to perform the
substructure search, leading to a race condition and undefined
behavior.

.. oe:source:: MultiCoreMolFind.cpp
   :noindent:
   :language: c++
   :frag:     SNIPPET-constmethod

.. oe:source:: MultiCoreMolFind.py
   :noindent:
   :language: python
   :frag:     SNIPPET-constmethod

.. oe:source:: MultiCoreMolFind.java
   :noindent:
   :language: java
   :frag:     SNIPPET-constmethod

.. oe:source:: MultiCoreMolFind.cs
   :noindent:
   :language: csharp
   :frag:     SNIPPET-constmethod

.. warning::
   
   Care must be taken when reusing the same objects in multiple
   threads. Only one invocation of a non-``const`` method
   (*e.g.* :oe:meth:`OEChem::OESubSearch::AddConstraint`) concurrently
   could crash the program. Concurrent execution of non-``const``
   methods must to be protected by :oe:cls:`mutual
   exclusion<OEPlatform::OEMutex>`.


Functions
----------

:ref:`OEChem functions <chapter_oechemfunctions>`, unless otherwise
stated in the API documentation, can be safely called from multiple
threads. For example, :oe:func:`OESystem::OEGetTag` needs to access a
global table to make ``string`` to ``unsigned integer`` associations
global to the program. :oe:func:`OESystem::OEGetTag` ensures its own
thread safety so that it is safe to call from multiple threads.

However, mutual exclusion can be very expensive. This is why **OEChem TK**
(and the STL) make no effort to make concurrent object access
safe. So when mutual exclusion can not be avoided, as in the case of
:oe:func:`OESystem::OEGetTag`, faster alternatives should be
sought. For example, the return of :oe:func:`OESystem::OEGetTag` can
be cached to a local variable.

.. oe:language:: c++ 

   .. code-block:: c++

      void func()
      {
         static unsigned int tag = OEGetTag("foo");
         ...
      }

   or reuse that tag,

   .. code-block:: c++

      unsigned int MyFooTag()
      {
          static unsigned int tag = OEGetTag("foo");
          return tag;
      }

It is important to realize that even though the function may be
thread-safe, the function may be altering the state of its arguments,
which could be shared across threads. Again, the "const-ness" of the
arguments must be taken into account. For example, it is not safe to
call :oe:func:`OEChem::OEWriteMolecule` on the same molecule from
separate threads (even on different
:oe:cls:`oemolostreams<OEChem::oemolostream>`) because the molecule
argument to :oe:func:`OEChem::OEWriteMolecule` is non-``const``. The
molecule argument is non-``const`` for performance, the molecule may
need to be changed to conform to the file format being written
to. **OEChem TK** provides :oe:func:`OEChem::OEWriteConstMolecule` that will
make a local copy of the molecule before making necessary changes for
the file format being requested.

Exceptions to the ``const`` argument rule must be explicitly stated in
the API documentation. For example, the
:oe:func:`OEChem::OEWriteMolecule` function applied to the
non-``const`` argument of :ref:`oemolothread <OEChem_oemolothread>` is thread-safe
only by virtue of design. 

.. oe:source:: MultiCoreMolFind.cpp
   :noindent:
   :language: c++
   :frag:     SNIPPET-writemolecule

.. oe:source:: MultiCoreMolFind.py
   :noindent:
   :language: python
   :frag:     SNIPPET-writemolecule

.. oe:source:: MultiCoreMolFind.java
   :noindent:
   :language: java
   :frag:     SNIPPET-writemolecule

.. oe:source:: MultiCoreMolFind.cs
   :noindent:
   :language: csharp
   :frag:     SNIPPET-writemolecule


.. _section_memorymanagement:

Memory Management
================================================================================

Memory allocation is often a bottleneck in any program. This is
especially the case in multi-threaded programs because memory
allocation must protect a globally shared resource, the memory space
of the process. Most ``malloc`` implementations use mutual exclusion
to make themselves thread-safe. However, mutual exclusion is
expensive, and cuts down on the scalability of the multi-threaded
program.  **OEChem TK** attempts to alleviate the problem by implementing a
memory cache for the following small objects: 

- Any **OpenEye Toolkit** :ref:`iterator object
  <section_atombondtraversal_iter>`, i.e., something derived
  from :ref:`OEIterBase <OESystem_OEIterBase>`.
- Any :ref:`OEGetNbrAtom <OEChem_OEGetNbrAtom>` functor. This is used by
  some implementations of
  :oe:meth:`OEChem::OEAtomBase::GetAtoms` that returns an
  iterator as well, see previous.
- The :ref:`OEMatch <OEChem_OEMatch>` class. Used by almost any API
  that operates with an instance of
  :ref:`OEMatchBase <OEChem_OEMatchBase>`.
- :oe:const:`OEChem::OEMolBaseType::OEMiniMol` instances of
  :ref:`OEGraphMol <OEChem_OEGraphMol>`.

The cache is totally opaque to the normal **OEChem TK** user. However, it
can cause problems in multi-threaded situations.

The small object cache implementation is controlled through the
:oe:func:`OESystem::OESetMemPoolMode` function. The parameter passed to
this function determines what sort of cache implementation to use, the
possible options are a set of values bit-wise or'd from the
:oe:constns:`OESystem::OEMemPoolMode` constant namespace. The possible
combinations are as follows:

.. oe:language:: java

   .. warning:: 

      :oe:func:`OESystem::OESetMemPoolMode` is not available in
      Java. The reason is explained :ref:`later
      <multithreading_javamempoolmode>`.

.. |SingleThreaded| replace:: Makes no attempt at thread safety. 

.. |Mutexed| replace:: Protects a single global cache with a mutex.

.. |ThreadLocal| replace:: Uses a separate cache for each thread.

.. |BoundedCache| replace:: Will only cache allocations up to a
   certain size, then further allocations will be sent to the global
   ``operator new(size_t)`` function.

.. |UnboundedCache| replace:: Will cache all allocations up to any
   size, never returning memory to the operating system after it has
   been requested.

:SingleThreaded|BoundedCache: 

   |SingleThreaded| |BoundedCache|

:SingleThreaded|UnboundedCache:

   |SingleThreaded| |UnboundedCache|

:Mutexed|BoundedCache:

   |Mutexed| |BoundedCache|

:Mutexed|UnboundedCache:

   |Mutexed| |UnboundedCache|

:ThreadLocal|BoundedCache:

   |ThreadLocal| |BoundedCache|

:ThreadLocal|UnboundedCache:

   |ThreadLocal| |UnboundedCache|

:System:

   Forwards all allocations to the global ``operator new(size_t)``
   function. This allows the user to circumvent the entire caching
   system in preference to their own ``operator new(size_t)`` function
   allowing other third party memory allocators to be linked against.

.. oe:language:: c++ python

   The default implementation is ``ThreadLocal|UnboundedCache`` for
   performance in single threaded programs and thread safety in the
   most common multi-threaded programs (*e.g.* :oe:src:`Listing
   2<SNIPPET-MultiCoreMolFind>`).

.. oe:language:: java

   The default implementation in Java is ``Mutexed|UnboundedCache``.
   :oe:const:`Unbounded<OESystem::OEMemPoolMode::UnboundedCache>` for
   performance, however,
   :oe:const:`Mutexed<OESystem::OEMemPoolMode::Mutexed>` since the Java
   garbage collector runs in a separate thread.

.. oe:language:: csharp

   The default implementation in C# is ``Mutexed|UnboundedCache``.
   :oe:const:`Unbounded<OESystem::OEMemPoolMode::UnboundedCache>` for
   performance, however,
   :oe:const:`Mutexed<OESystem::OEMemPoolMode::Mutexed>` since the C#
   garbage collector runs in a separate thread.


Thread Safety Options
-----------------------

.. _multithreading_javamempoolmode:

.. oe:language:: java

   .. warning::

      The small object cache in Java **OEChem TK** must always be mutexed
      since the garbage collector runs in another thread. This is why
      :oe:func:`OESystem::OESetMemPoolMode` is not exposed in the Java
      interface.

.. oe:language:: c++ python

   :oe:func:`OESystem::OESetMemPoolMode` offers the following options
   for deciding the thread-safety of the cache implementation:
   :oe:const:`SingleThreaded<OESystem::OEMemPoolMode::SingleThreaded>`,
   :oe:const:`Mutexed<OESystem::OEMemPoolMode::Mutexed>`, 
   :oe:const:`ThreadLocal<OESystem::OEMemPoolMode::ThreadLocal>`, and
   :oe:const:`System<OESystem::OEMemPoolMode::System>`.

   If the program is known to only have a single thread there is a
   potential performance benefit in using the
   :oe:const:`OESystem::OEMemPoolMode::SingleThreaded` option. This
   flag tells the cache implementation it does not have to worry about
   being thread-safe.

.. oe:source:: SingleThreadedMolFind.cpp
   :language: c++
   :caption:  Setting single threaded memory pool mode
   :frag:     SNIPPET

.. oe:source:: SingleThreadedMolFind.py
   :language: python
   :caption:  Setting single threaded memory pool mode
   :frag:     SNIPPET

.. oe:language:: c++ python

   .. note::
   
      :oe:cls:`oemolthreads<OEChem::oemolthreadbase>` can still be used
      since they do not use the small object cache.

   The preceding program demonstrates how to use
   :oe:func:`OESystem::OESetMemPoolMode` to tell **OEChem TK** the program
   only contains a single thread using
   :oe:const:`OESystem::OEMemPoolMode::SingleThreaded`.
   :oe:const:`OESystem::OEMemPoolMode::UnboundedCache` is also used to get
   the maximum amount of performance out of **OEChem TK**. The options for
   the cache implementation are bitwise or'd together as shown in the
   following code snippet:

.. oe:source:: SingleThreadedMolFind.cpp
   :noindent:
   :language: c++
   :frag:     SNIPPET-SetMemPoolMode

.. oe:source:: SingleThreadedMolFind.py
   :noindent:
   :language: python
   :frag:     SNIPPET-SetMemPoolMode

.. oe:language:: c++ python

   The default thread safety level in **OEChem TK** is to use
   :oe:const:`OESystem::OEMemPoolMode::ThreadLocal` since it can offer
   performance nearly as good as
   :oe:const:`OESystem::OEMemPoolMode::SingleThreaded`. It also means
   the vast majority of multi-threaded **OEChem TK** programs will be
   thread-safe. This is because each thread gets its own dedicated
   small object cache.

   However, this introduces a memory leak when certain cached objects
   are created in one thread and then passed to another thread to be
   deleted.

If database order is a necessity it may be possible to gain
performance from threading using "pipeline"
parallelism. :oe:src:`Listing 3<SNIPPET-PipelineMolFind>` demonstrates how the
molecule searching problem can be broken down into 5 stages, each
running in a separate thread. In order, the 5 stages are:

 #. Reading data from disk - :ref:`oemolithread <OEChem_oemolithread>`
 #. Parsing the data into a molecule - ``ParseThread``
 #. Performing the substructure search on the molecule - ``SearchThread``
 #. Serializing the molecule - ``main thread``
 #. Writing data to disk - :ref:`oemolothread <OEChem_oemolothread>`

.. oe:source:: PipelineMolFind.cpp
   :language: c++
   :refnum:   3
   :caption:  Setting mutexed memory pool mode
   :frag:     SNIPPET

.. oe:source:: PipelineMolFind.py
   :language: python
   :refnum:   3
   :caption:  Setting mutexed memory pool mode
   :frag:     SNIPPET

.. oe:language:: python

	:ref:`Full listing. <listing-PipelineMolFind.py>`



.. oe:source:: PipelineMolFind.java
   :language: java
   :refnum:   3
   :caption:  Using pipeline parallelism in Java OEChem TK
   :frag:     SNIPPET

.. oe:source:: PipelineMolFind.cs
   :language: csharp
   :refnum:   3
   :caption:  Using pipeline parallelism in Java OEChem TK
   :frag:     SNIPPET

.. oe:language:: c++ python

   :oe:src:`Listing 3<SNIPPET-PipelineMolFind>` shows how a molecule can be
   passed down a pipeline of threads. 

.. oe:language:: c++ 
   
   .. note:: 
      
      The molecule factory, :oe:func:`OEChem::OENewMolBase`, is used so
      that pointers to molecules can be passed around. If
      :oe:cls:`OEGraphMols<OEChem::OEGraphMol>` were passed through the
      :ref:`OEProtectedBuffer <OESystem_OEProtectedBuffer>` it would require costly
      molecule copy operations to be performed in the
      :oe:meth:`Put<OESystem::OEProtectedBuffer::Put>` and
      :oe:meth:`Get<OESystem::OEProtectedBuffer::Get>`
      methods. Pointers are small and quick to copy, though care must
      be taken to ``delete`` the molecule in ``SearchThread`` or the
      ``main thread``.

Overriding the allocator
------------------------

.. oe:language:: csharp

   **OpenEye Toolkits** are hardcoded to use
   :oe:const:`OESystem::OEMemPoolMode::System` for C# and can not be
   changed at this time. Please contact support@eyesopen.com if you
   think this option may help you. 

.. oe:language:: java

   Memory pooling is even more important in Java because all **OpenEye
   Toolkit** memory allocation actually allocates memory from the JVM
   heap. Java maintains strict control over the amount of heap memory
   the JVM should use through the -Xmx flag. In order for the
   **OpenEye Toolkits** to adhere to these limits and put pressure on
   the garbage collector, all **OpenEye Toolkit** memory is allocated
   through the JNI as byte array objects. This makes small object
   memory caching even more important for Java as there is a higher
   overhead in both space and time for each memory allocation. 

   **OpenEye** testing has shown that the current default of
   ``Mutexed|UnboundedCache`` to be a good balance of performance and
   memory usage. However, it is not particularly suited to heavily
   multi-threaded Java programs due to the use of a mutex. While
   :oe:func:`OESystem::OESetMemPoolMode` is not available in Java
   because it is very race condition prone inside the JVM, there are
   the following environment variables that can be set to experiment
   with a better memory strategy:

   - ``OEJAVA_BOUNDED_CACHE``
   - ``OEJAVA_MEM_POOL_MODE``

   ``OEJAVA_BOUNDED_CACHE`` specifies that the memory pool mode should
   be ``Mutexed|UnboundedCache`` to be more aggressive about returning
   memory to the operating system.
   
   ``OEJAVA_MEM_POOL_MODE`` can take any integral value that
   :oe:func:`OESystem::OESetMemPoolMode` will accept. The user will
   have to go investigate the ``oesystem/threadsafe.h`` to figure out
   the right integer. The most common one is to use the integer ``16``
   for :oe:const:`OESystem::OEMemPoolMode::System` to bypass all small
   object caching. 

   .. warning::

      The use of these variable is intentionally difficult. This is an
      area of active research at **OpenEye** and we hope to remove
      them entirely in the future with a better default mode that
      works well for both multi-threaded and single threaded Java
      programs. Encouraging results have been seen by tighter
      integration with `TCMalloc
      <http://goog-perftools.sourceforge.net/doc/tcmalloc.html>`_, and
      these variable are only here to encourage such experimentation.

.. oe:language:: c++ python

   Passing :oe:const:`OESystem::OEMemPoolMode::System` to
   :oe:func:`OESystem::OESetMemPoolMode` tells the **OpenEye
   Toolkits** to using the system defined ``operator
   new(size_t)``. The implementation is thread-safe on all modern
   systems and may even be faster than using a mutexed cache. It is up
   to the user to figure out which is best for their system. The
   following code snippet will bypass the cache entirely in favor of
   calling the systems memory allocation routines:

   .. oe:language:: c++

      .. code-block:: c++

         OESetMemPoolMode(OEMemPoolMode::System);

   .. oe:language:: python

      .. code-block:: python

         OESetMemPoolMode(OEMemPoolMode_System)


   The real benefit of specifying
   :oe:const:`OESystem::OEMemPoolMode::System` is the ability to plug
   in third party memory allocators. The following memory allocators
   are recommended for multi-threaded programs:

    - `TCMalloc <http://goog-perftools.sourceforge.net/doc/tcmalloc.html>`_
    - `Hoard <http://www.hoard.org>`_
    

(Un)bounded Options
----------------------

Two types of small object caches can be set with
:oe:func:`OESystem::OESetMemPoolMode`:
:oe:const:`Bounded<OESystem::OEMemPoolMode::BoundedCache>` and
:oe:const:`Unbounded<OESystem::OEMemPoolMode::UnboundedCache>`. The default
is an :oe:const:`Unbounded<OESystem::OEMemPoolMode::UnboundedCache>` cache
for performance. The assumption is most **OEChem TK** programs will have a
constant memory usage. Even if the program has a steadily rising
memory footprint the program will usually exit after it has maxed out
its memory requirements.

This may not be the case for long running programs, for example, a web
service. These programs may need to create a temporarily large memory
footprint and then release it back to the system for some other
program to use. An
:oe:const:`Unbounded<OESystem::OEMemPoolMode::UnboundedCache>` cache
will hold onto all memory it sees, never returning any to the
operating system.

A :oe:const:`Bounded<OESystem::OEMemPoolMode::BoundedCache>` cache will
only cache allocations up to a limit. After that limit has been
reached all further allocations will be sent to the system memory
allocator. This allows the deallocations later in the program to
return the memory to the operating system. This allows certain objects
to still use the fast cache while the other objects that are bloating
the memory footprint will incur the cost of asking the operating
system for the memory they require.

.. oe:language:: java

   .. note:: 

      Since :oe:func:`OESystem::OESetMemPoolMode` is not available in
      Java the bounded cache is selected by turning on the
      ``OEJAVA_BOUNDED_CACHE`` environment variable.

   
 
