BUG: df.where(series, axis=1) always results in TypeError

[wjandrea]

Pandas version checks

• I have checked that this issue has not already been reported.
  • BUG: df.where(series, axis=1) always results in TypeError #58144 was self-closed once they realized the cause, but the bug still exists
• I have confirmed this bug exists on the latest version of pandas.
• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Based on this Stack Overflow question:

A = pd.DataFrame(
    [[0.0, 0.5, 0.0],
     [0.1, 0.0, 0.2],
     [0.2, 0.0, 0.0]])
B = pd.Series([1.0, 1.0, np.nan])

Then:

A.where(B.notna(), axis=1)

Issue Description

Result: TypeError: 'float' object is not subscriptable

It seems to be trying to index other, which is NaN.

Traceback

Traceback (most recent call last):

  Cell In[11], line 2
    A.where(B.notna(), axis=1)

  File ~/Programs/micromamba/envs/pandas_pre/lib/python3.12/site-packages/pandas/core/generic.py:9897 in where
    return self._where(cond, other, inplace=inplace, axis=axis, level=level)

  File ~/Programs/micromamba/envs/pandas_pre/lib/python3.12/site-packages/pandas/core/generic.py:9686 in _where
    new_data = self._mgr.where(

  File ~/Programs/micromamba/envs/pandas_pre/lib/python3.12/site-packages/pandas/core/internals/managers.py:468 in where
    return self.apply(

  File ~/Programs/micromamba/envs/pandas_pre/lib/python3.12/site-packages/pandas/core/internals/managers.py:432 in apply
    kwargs[k] = obj[b.mgr_locs.indexer]

TypeError: 'float' object is not subscriptable

Expected Behavior

B.notna() is broadcast to A along the columns.

     0    1    2
0  0.0  0.5  NaN
1  0.1  0.0  NaN
2  0.2  0.0  NaN

Cf. A.mul(B, axis=1):

     0    1   2
0  0.0  0.5 NaN
1  0.1  0.0 NaN
2  0.2  0.0 NaN

Installed Versions

INSTALLED VERSIONS

commit : b8a4691
python : 3.12.0.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-174-generic
Version : #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : fr_CA.UTF-8
LOCALE : fr_CA.UTF-8

pandas : 3.0.0.dev0+680.gb8a4691647
numpy : 2.1.0.dev0+git20240403.e59c074
pytz : 2024.1
dateutil : 2.8.2
setuptools : 68.2.2
pip : 24.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.22.2
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[wjandrea]

As a workaround I tried:

A.where(B.notna().values[None, :])

But got ValueError: Array conditional must be same shape as self

Cf.

np.where(B.notna().values[None, :], A, np.nan)

[[0.  0.5 nan]
 [0.1 0.  nan]
 [0.2 0.  nan]]

In the end, this works:

A.where(np.broadcast_to(B.notna().values[None, :], A.shape))

     0    1   2
0  0.0  0.5 NaN
1  0.1  0.0 NaN
2  0.2  0.0 NaN

[aabhinavg]

The earlier code is failing because the where method was trying to access the mgr_locs attribute of the Series object B, which is not present in the Series object. This resulted in a TypeError because the code was attempting to subscript a float object.

The corrected code converts the boolean Series notna_result to a numpy array before indexing. By using notna_result.values[:, None], we ensure that the boolean mask has the appropriate shape for broadcasting with the DataFrame A. This prevents the TypeError from occurring and results in the desired output.
Here is the backtrace

File "/Users/compile_sanatan/project_to_contribute/pandas/issue/58190/test.py", line 10, in <module>
result = A.where(B.notna(), axis=1)
File "/Users/compile_sanatan/project_to_contribute/pandas/pandas/core/generic.py", line 9897, in where
return self._where(cond, other, inplace=inplace, axis=axis, level=level)
File "/Users/compile_sanatan/project_to_contribute/pandas/pandas/core/generic.py", line 9686, in _where
new_data = self._mgr.where(
File "/Users/compile_sanatan/project_to_contribute/pandas/pandas/core/internals/managers.py", line 468, in where
return self.apply(
File "/Users/compile_sanatan/project_to_contribute/pandas/pandas/core/internals/managers.py", line 432, in apply
kwargs[k] = obj[b.mgr_locs.indexer]
TypeError: 'float' object is not subscriptable

The corrected code is as follows:

import pandas as pd
import numpy as np

# Define DataFrame A and Series B
A = pd.DataFrame(
    [[0.0, 0.5, 0.0],
     [0.1, 0.0, 0.2],
     [0.2, 0.0, 0.0]])
B = pd.Series([1.0, 1.0, np.nan])

# Check the Result of B.notna()
notna_result = B.notna()
print("B.notna() result:")
print(notna_result)

# Check the values of B
print("Values of B:")
print(B.values)

# Convert the boolean Series to a numpy array before indexing
result = pd.DataFrame(np.where(notna_result.values[:, None], A.values, np.nan),
                      index=A.index, columns=A.columns)
print("Desired Output:")
print(result)

[aabhinavg]

Hi @wjandrea one more I found to fix the subscriptable for the float.

Fix for pandas where function issue

If you encounter the error TypeError: 'float' object is not subscriptable when using the where function in pandas, it's likely due to an issue with the types of the objects being used. This error usually occurs when you're trying to access an element of a non-subscriptable object, such as a float.

Solution

To resolve this issue, you can use a workaround that involves transposing the DataFrame, applying the where function, and then transposing it back to its original shape. Here's the modified code:

import pandas as pd
import numpy as np

A = pd.DataFrame(
    [[0.0, 0.5, 0.0],
     [0.1, 0.0, 0.2],
     [0.2, 0.0, 0.0]])
B = pd.Series([1.0, 1.0, np.nan])

# Transpose the DataFrame, apply the where function, and then transpose it back
result = A.T.where(B.notna()).T

print(result)

```output
+ /Users/compile_sanatan/miniforge3/envs/pandas-dev/bin/ninja
[1/1] Generating write_version_file with a custom command
     0    1   2
0  0.0  0.5 NaN
1  0.1  0.0 NaN
2  0.2  0.0 NaN

Note

Please note that this workaround is specific to the situation where you want to apply the condition along axis 1.

What if Axis value change

    Axis 0 (default behavior):
          No transposition of the DataFrame is needed.
         Apply the where function directly.
         Return the result.
   Axis n (where n > 0):
         Transpose the DataFrame n times.
         Apply the where function.
         Transpose the result back n times to its original shape.
         Return the result.

import pandas as pd
import numpy as np

def fix_where_issue(df, series, axis):
    if axis == 1:
        df = df.T
    elif axis == 2:
        df = df.T.T
    result = df.where(series.notna())
    if axis == 1:
        result = result.T
    elif axis == 2:
        result = result.T.T
    return result

# Test cases
def test_fix_where_issue():
    # Test case 1: Axis 0
    A1 = pd.DataFrame([[0.0, 0.5, 0.0], [0.1, 0.0, 0.2], [0.2, 0.0, 0.0]])
    B1 = pd.Series([1.0, 1.0, np.nan])
    expected_output1 = A1.where(B1.notna())
    assert fix_where_issue(A1, B1, axis=0).equals(expected_output1)

    # Test case 2: Axis 1
    A2 = pd.DataFrame([[0.0, 0.5, 0.0], [0.1, 0.0, 0.2], [0.2, 0.0, 0.0]])
    B2 = pd.Series([1.0, 1.0, np.nan])
    expected_output2 = A2.T.where(B2.notna()).T
    assert fix_where_issue(A2, B2, axis=1).equals(expected_output2)

    # Test case 3: Axis 2
    A3 = pd.DataFrame([[0.0, 0.5, 0.0], [0.1, 0.0, 0.2], [0.2, 0.0, 0.0]])
    B3 = pd.Series([1.0, 1.0, np.nan])
    expected_output3 = A3.T.T.where(B3.notna())
    assert fix_where_issue(A3, B3, axis=2).equals(expected_output3)

    print("All test cases passed successfully.")

# Run the test cases
test_fix_where_issue()