.. oe:namespace:: OEChem



.. meta::
   :keywords: OETokenizerBase, TokenizerBase


.. _OEChem_OETokenizerBase:

.. oe:class:: OETokenizerBase

   .. oe:api::


      class OETokenizerBase

   This is an abstract base class used to represent an object that can
   efficiently split up a molecular file into a stream of tokens, each
   token representing one molecule in the stream. Implementations of
   this class are returned by the :oe:func:`OEChem::OEGetTokenizer`
   function. This class is expected to do the minimal amount of string
   parsing possible to determine the next chunk of bytes representing
   a molecule.

   .. oe:method:: GetNextToken

      .. oe:language:: c++ java csharp

         .. oe:api::

            std::string* GetNextToken(OEPlatform::oeistream& ifs)=0

      .. oe:language:: python

         .. oe:api:: python

            GetNextToken(ifs: oeistream) -> str

      Returns a chunk of bytes representing the next molecule in the
      stream ``ifs``. The stream of bytes is expected to be in the
      same file format as was passed to the
      :oe:func:`OEChem::OEGetTokenizer` factory function used to
      construct this object.

      .. oe:language:: c++

         This method returns a ``new std::string``, ownership is
         passed to the caller, that is expected to call ``delete``.

   .. oe:method:: ParseTitle

      .. oe:language:: c++ java csharp

         .. oe:api::

            std::string ParseTitle(const std::string& data) const = 0

      .. oe:language:: python

         .. oe:api:: python

            ParseTitle(data: str) -> str

      Return the string that :oe:meth:`OEChem::OEMolBase::GetTitle`
      would return for the molecule record data passed in. The
      implementations of this method are designed to do the minimal
      amount of parsing required to retrieve the title. 
