Refactor for cx_Oracle version 6

Drops support for cx_Oracle prior to version 5.x, reworks
numeric and binary support.

Fixes: #4064

Change-Id: Ib9ae9aba430c15cd2a6eeb4e5e3fd8e97b5fe480

doc/build/changelog/migration_12.rst

@@ -1358,6 +1358,72 @@ The above will render::
 Dialect Improvements and Changes - Oracle
 =========================================
 
+.. _change_cxoracle_12:
+
+Major Refactor to cx_Oracle Dialect, Typing System
+--------------------------------------------------
+
+With the introduction of the 6.x series of the cx_Oracle DBAPI, SQLAlchemy's
+cx_Oracle dialect has been reworked and simplified to take advantage of recent
+improvements in cx_Oracle as well as dropping support for patterns that were
+more relevant before the 5.x series of cx_Oracle.
+
+* The minimum cx_Oracle version supported is now 5.1.3; 5.3 or the most recent
+  6.x series are recommended.
+
+* The handling of datatypes has been refactored.  The ``cursor.setinputsizes()``
+  method is no longer used for any datatype except LOB types, per advice from
+  cx_Oracle's developers. As a result, the parameters ``auto_setinputsizes``
+  and ``exclude_setinputsizes`` are deprecated and no longer have any effect.
+
+* The ``coerce_to_decimal`` flag, when set to False to indicate that coercion
+  of numeric types with precision and scale to ``Decimal`` should not occur,
+  only impacts untyped (e.g. plain string with no :class:`.TypeEngine` objects)
+  statements. A Core expression that includes a :class:`.Numeric` type or
+  subtype will now follow the decimal coercion rules of that type.
+
+* The "two phase" transaction support in the dialect, already dropped for the
+  6.x series of cx_Oracle, has now been removed entirely as this feature has
+  never worked correctly and is unlikely to have been in production use.
+  As a result, the ``allow_twophase`` dialect flag is deprecated and also has no
+  effect.
+
+* Fixed a bug involving the column keys present with RETURNING.  Given
+  a statement as follows::
+
+    result = conn.execute(table.insert().values(x=5).returning(table.c.a, table.c.b))
+
+  Previously, the keys in each row of the result would be ``ret_0`` and ``ret_1``,
+  which are identifiers internal to the cx_Oracle RETURNING implementation.
+  The keys will now be ``a`` and ``b`` as is expected for other dialects.
+
+* cx_Oracle's LOB datatype represents return values as a ``cx_Oracle.LOB``
+  object, which is a cursor-associated proxy that returns the ultimate data
+  value via a ``.read()`` method.  Historically, if more rows were read before
+  these LOB objects were consumed (specifically, more rows than the value of
+  cursor.arraysize which causes a new batch of rows to be read), these LOB
+  objects would raise the error "LOB variable no longer valid after subsequent
+  fetch". SQLAlchemy worked around this by both automatically calling
+  ``.read()`` upon these LOBs within its typing system, as well as using a
+  special ``BufferedColumnResultSet`` which would ensure this data was buffered
+  in case a call like ``cursor.fetchmany()`` or ``cursor.fetchall()`` were
+  used.
+
+  The dialect now makes use of a cx_Oracle outpttypehandler to handle these
+  ``.read()`` calls, so that they are always called up front regardless of how
+  many rows are being fetched, so that this error can no longer occur.  As a
+  result, the use of the ``BufferedColumnResultSet``, as well as some other
+  internals to the Core ``ResultSet`` that were specific to this use case,
+  have been removed.   The type objects are also simplified as they no longer
+  need to process a binary column result.
+
+  Additionally, cx_Oracle 6.x has removed the conditions under which this error
+  occurs in any case, so the error is no longer possible.   The error
+  can occur on SQLAlchemy in the case that the seldom (if ever) used
+  ``auto_convert_lobs=False`` option is in use, in conjunction with the
+  previous 5.x series of cx_Oracle, and more rows are read before the LOB
+  objects can be consumed.  Upgrading to cx_Oracle 6.x will resolve that issue.
+
 .. _change_4003:
 
 Oracle Unique, Check constraints now reflected

doc/build/changelog/unreleased_12/oracle_refactor.rst

@@ -0,0 +1,61 @@
+.. change::
+    :tags: bug, oracle
+    :tickets: 4064
+
+    Partial support for persisting and retrieving the Oracle value
+    "infinity" is implemented with cx_Oracle, using Python float values
+    only, e.g. ``float("inf")``.  Decimal support is not yet fulfilled by
+    the cx_Oracle DBAPI driver.
+
+.. change::
+    :tags: bug, oracle
+
+    The cx_Oracle dialect has been reworked and modernized to take advantage of
+    new patterns that weren't present in the old 4.x series of cx_Oracle. This
+    includes that the minimum cx_Oracle version is the 5.x series and that
+    cx_Oracle 6.x is now fully tested. The most significant change involves
+    type conversions, primarily regarding the numeric / floating point and LOB
+    datatypes, making more effective use of cx_Oracle type handling hooks to
+    simplify how bind parameter and result data is processed.
+
+    .. seealso::
+
+        :ref:`change_cxoracle_12`
+
+.. change::
+    :tags: bug, oracle
+    :tickets: 3997
+
+    two phase support for cx_Oracle has been completely removed for all
+    versions of cx_Oracle, whereas in 1.2.0b1 this change only took effect for
+    the 6.x series of cx_Oracle.  This feature never worked correctly
+    in any version of cx_Oracle and in cx_Oracle 6.x, the API which SQLAlchemy
+    relied upon was removed.
+
+    .. seealso::
+
+        :ref:`change_cxoracle_12`
+
+.. change::
+    :tags: bug, oracle
+
+    The column keys present in a result set when using :meth:`.Insert.returning`
+    with the cx_Oracle backend now use the correct column / label names
+    like that of all other dialects.  Previously, these came out as
+    ``ret_nnn``.
+
+    .. seealso::
+
+        :ref:`change_cxoracle_12`
+
+.. change::
+    :tags: bug, oracle
+
+    Several parameters to the cx_Oracle dialect are now deprecated and will
+    have no effect: ``auto_setinputsizes``, ``exclude_setinputsizes``,
+    ``allow_twophase``.
+
+    .. seealso::
+
+        :ref:`change_cxoracle_12`
+

lib/sqlalchemy/dialects/oracle/__init__.py

@@ -19,6 +19,7 @@
 __all__ = (
     'VARCHAR', 'NVARCHAR', 'CHAR', 'DATE', 'NUMBER',
     'BLOB', 'BFILE', 'CLOB', 'NCLOB', 'TIMESTAMP', 'RAW',
-    'FLOAT', 'DOUBLE_PRECISION', 'LONG', 'dialect', 'INTERVAL',
+    'FLOAT', 'DOUBLE_PRECISION', 'BINARY_DOUBLE' 'BINARY_FLOAT',
+    'LONG', 'dialect', 'INTERVAL',
     'VARCHAR2', 'NVARCHAR2', 'ROWID'
 )

lib/sqlalchemy/dialects/oracle/base.py

@@ -422,6 +422,28 @@ def __init__(self, precision=None, scale=None, asdecimal=None):
             precision=precision, scale=scale, asdecimal=asdecimal)
 
 
+class BINARY_DOUBLE(sqltypes.Numeric):
+    __visit_name__ = 'BINARY_DOUBLE'
+
+    def __init__(self, precision=None, scale=None, asdecimal=None):
+        if asdecimal is None:
+            asdecimal = False
+
+        super(BINARY_DOUBLE, self).__init__(
+            precision=precision, scale=scale, asdecimal=asdecimal)
+
+
+class BINARY_FLOAT(sqltypes.Numeric):
+    __visit_name__ = 'BINARY_FLOAT'
+
+    def __init__(self, precision=None, scale=None, asdecimal=None):
+        if asdecimal is None:
+            asdecimal = False
+
+        super(BINARY_FLOAT, self).__init__(
+            precision=precision, scale=scale, asdecimal=asdecimal)
+
+
 class BFILE(sqltypes.LargeBinary):
     __visit_name__ = 'BFILE'
 
@@ -557,6 +579,12 @@ def visit_TIMESTAMP(self, type_, **kw):
     def visit_DOUBLE_PRECISION(self, type_, **kw):
         return self._generate_numeric(type_, "DOUBLE PRECISION", **kw)
 
+    def visit_BINARY_DOUBLE(self, type_, **kw):
+        return self._generate_numeric(type_, "BINARY_DOUBLE", **kw)
+
+    def visit_BINARY_FLOAT(self, type_, **kw):
+        return self._generate_numeric(type_, "BINARY_FLOAT", **kw)
+
     def visit_NUMBER(self, type_, **kw):
         return self._generate_numeric(type_, "NUMBER", **kw)
 
@@ -746,12 +774,14 @@ def get_render_as_alias_suffix(self, alias_name_text):
     def returning_clause(self, stmt, returning_cols):
         columns = []
         binds = []
+
         for i, column in enumerate(
                 expression._select_iterables(returning_cols)):
             if column.type._has_column_expression:
                 col_expr = column.type.column_expression(column)
             else:
                 col_expr = column
+
             outparam = sql.outparam("ret_%d" % i, type_=column.type)
             self.binds[outparam.key] = outparam
             binds.append(
@@ -760,7 +790,8 @@ def returning_clause(self, stmt, returning_cols):
                 self.process(col_expr, within_columns_clause=False))
 
             self._add_to_result_map(
-                outparam.key, outparam.key,
+                getattr(col_expr, 'name', col_expr.anon_label),
+                getattr(col_expr, 'name', col_expr.anon_label),
                 (column, getattr(column, 'name', None),
                  getattr(column, 'key', None)),
                 column.type