error during pip install of cylp, os environ COIN_INSTALL_DIR not set with apt install of coin

[matt-telstra]

I installed COIN using sudo apt-get install coinor-cbc on an ubuntu machine. (Which is the suggested method of installation on the official coin page).

Now when I try to install cylp using pip, I get the following error:

Collecting cylp
  Using cached cylp-0.2.3.6.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-setUG3/cylp/setup.py", line 44, in <module>
        'to the location of the COIN installation')
    Exception: Please set the environment variable COIN_INSTALL_DIR to the location of the COIN installation

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-setUG3/cylp/

What should COIN_INSTALL_DIR be set to if I installed coin through apt? I have no idea where the location of my COIN installation is. How do I find out?

Ideally this error message should say something like:

Please set the environment variable COIN_INSTALL_DIR to the location of the COIN installation. If you installed COIN through your package manager, this is probably /whateverTheDirIs

I don't know why this environment variable wasn't permanantly set upon install of COIN. I did logout and log back in to refresh the environment. That didn't fix it.

[tkralphs]

The setup.py script was implemented before cbc was easily installable with apt. It assumes cbc was installed from source, in which case you would know exactly where it is installed. The easiest path to getting CyLP working is just to install cbc from source according to the instructions on the wiki.

If you want to get CyLP working with the officially packaged version of cbc, then you first probably need to install the package coinor-libcbc-dev to get the header files. Even then, the setup.py would need to be modified to work with libraries and header that are installed in system-standard locations (or better yet, to use pkg-config to find everything). I can probably produce such a script pretty easily so I'll give it a try this weekend if I can find some time. I've been meaning to get back to CyLP for quite some time and do some badly needed maintenance.

[matt-telstra]

I tried installing from source too, but couldn't get that working. (There were mysterious compilation errors, and the automatic installation of dependencies didn't work correctly)

Then I gave up and tried apt again, then source again etc.

After several hours of trying different things, pip install cylp finally worked with apt-get install coinor-cbc.
Unfortunately I was so confused, and had tried so many different things, I don't know what actually made it work.

For the record, if I enter echo $COIN_INSTALL_DIR into the terminal, it echos an empty string.
So I don't know what's going on there.

Then after pip install worked, I had runtime issues because the shared libraries weren't installed correctly. I had a co-worker fix that for me. I don't really know exactly what he did, just that he added soft links to a bunch of libraries, because cylp was expecting version 0 of some shared libraries, but I had only later versions installed.

Sorry I can't be more help with the solution. The good news is that I do have the system up and running now, so there's no need to rush to fix anything for my benefit. Although the installation process is extremely unpleasant. I believe that making it work with apt-get install coinor-cbc and pip install cylp and nothing else should be a high priority.

[matt-telstra]

I just submitted issue #48

I'm not sure whether that bug that I'm experiencing is due to a bad installation. Probably not, but I just thought I'd mention the possibility in this thread.

[tkralphs]

I had forgotten that setup.py now includes a fallback that set COIN_INSTALL_DIR automatically by looking for the location of clp, which should be in your path. Anyway, no worries on not being able to report the solution to your problem. I know how to replicate the issues you're seeing and what the possible gotchas are, just need to find some time to look into this and document procedures.

[matt-telstra]

Now I'm trying to install it on another machine and I'm having troubles again. It took a while, but now I've got something that can reproducibly compile cylp into a .whl and then install that into a virtual env.

I think this script should replace the instructions on the README. Those instructions don't explicitly state all steps.

deactivate # in case we're currently in a virtual env
set -e # halt if error on single line
set -x # display commands as they are run

# Step 1
rm -r Cbc-2.8.5 -f # remove stuff from previous attempt
wget https://www.coin-or.org/download/source/Cbc/Cbc-2.8.5.zip # no later than 2.8.5
unzip Cbc-2.8.5.zip
cd Cbc-2.8.5
./configure
make
make install

# Step 2
export COIN_INSTALL_DIR=$(pwd) # add to .bash_rc?

cd ../

# suitesparse
rm -r -f SuiteSparse-4.5.3  # remove stuff from previous attempt
wget http://faculty.cse.tamu.edu/davis/SuiteSparse/SuiteSparse-4.5.3.tar.gz
tar -xf SuiteSparse-4.5.3.tar.gz
export CVXOPT_SUITESPARSE_SRC_DIR=$(pwd)/SuiteSparse

# Step 3
rm -r -f CyLP  # remove stuff from previous attempt
git clone https://github.com/coin-or/CyLP.git

cd CyLP
python setup.py bdist_wheel

cd ../


# Step 4
export LD_LIBRARY_PATH="$(pwd)/Cbc-2.8.5/lib:$LD_LIBRARY_PATH"

# install bdist into worker env
rm -rf worker-env  # remove stuff from previous attempt
virtualenv -p /usr/bin/python2.7 ./worker-env
TARGET=$(pwd)/CyLP/dist/cylp-0.7.4-cp27-cp27mu-linux_x86_64.whl
./worker-env/bin/pip install $TARGET # install whl into virtual env
. ./worker-env/bin/activate

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)/worker-env/lib/python2.7/site-packages/lib/


# run MWE
python mwe.py

mwe.py:

import numpy as np
from cylp.cy import CyClpSimplex
from cylp.py.modeling.CyLPModel import CyLPArray

s = CyClpSimplex()

# Add variables
x = s.addVariable('x', 3)
y = s.addVariable('y', 2)

# Create coefficients and bounds
A = np.matrix([[1., 2., 0],[1., 0, 1.]])
B = np.matrix([[1., 0, 0], [0, 0, 1.]])
D = np.matrix([[1., 2.],[0, 1]])
a = CyLPArray([5, 2.5])
b = CyLPArray([4.2, 3])
x_u= CyLPArray([2., 3.5])

# Add constraints
s += A * x <= a
s += 2 <= B * x + D * y <= b
s += y >= 0
s += 1.1 <= x[1:3] <= x_u

# Set the objective function
c = CyLPArray([1., -2., 3.])
s.objective = c * x + 2 * y.sum()

# Solve using primal Simplex
s.primal()
print s.primalVariableSolution['x']

print('it worked!')

[BrannonKing]

Any update on supporting the apt install of cbc?

[matt-telstra]

I don't know anything about the backend of apt. Perhaps we can use my scripts from #57 to make this installable with apt?

[tkralphs]

I guess that most of what is going on in the scripts from #57 would be unnecessary in Linux, since most of the Python dependencies would either be installed by default or you would want to also install them with apt (maybe someone who has done this would like to comment). Installing Cbc from source should work out of the box in Linux and I believe this would be the easiest path to getting CyLP working in Linux until we have an improved installation script.

[matt-telstra]

Installing Cbc from source does not work out of the box. (That's why I created this issue.)

It's not the case that python dependencies are installed by default, since the specific version of numpy matters during compilation.

• The version of numpy which works for compiling Cbc is not the latest version. Then to use cylp you need to use an even older version
• There are multiple different versions of numpy for some numerical versions (e.g. there are at least two different numpy 1.12.1 libraries)

So really you need to do the compilation inside a virtual env with the right dependencies.

[Saelyos]

In case some people have trouble installing CyLP on Debian, this is the script I used :

#!/usr/bin/bash

set -e
# set -x

# Install dependencies
sudo apt-get install -y git pkg-config coinor-libcbc-dev coinor-libosi-dev coinor-libcoinutils-dev coinor-libcgl-dev

# This file is required by the Cylp installer (not present in standard coinor-libcbc-dev package of all debian version)
CBC_ADDLIBS="/usr/share/coin/doc/Cbc/cbc_addlibs.txt"

# If it does not exist,
if [[ ! (-f "$CBC_ADDLIBS") ]]; then

    # Clone the Cbc repository
    if [[ ! (-d "Cbc/") ]]; then
        git clone https://github.com/coin-or/Cbc.git Cbc/
    fi

    # Go into the directory 
    cd Cbc/

    # Go to the same version as the one in the official repositories
    git checkout releases/`pkg-config --modversion cbc`

    # Configure and install (in the current directory) the library
    ./configure -C
    make
    make install

    # Create a symlink in the default system folder
    sudo ln -s "$(pwd)/share/coin/doc/Cbc/cbc_addlibs.txt" "$CBC_ADDLIBS"
fi


echo
echo "----------------------------------"
echo "In order to install CyLP, please use the following command in you Python virtual environment:"
echo
echo "COIN_INSTALL_DIR=/usr/ pip install --pre cylp"
echo
echo "----------------------------------"
echo

exit 0

I've used it on Debian 9 and haven't tested on any other distribution, I'm just posting this here in case it can help other people.

[tkralphs]

Thanks, it should be possible to make it so that one does not have to build Cbc from source, as you did. I still plan to do this.

[tkralphs]

This should be fixed now in the latest version of the setup script (see ddff1a3). Documentation will be updated shortly.

[ir1979]

I used sudo apt-get install -y git pkg-config coinor-libcbc-dev coinor-libosi-dev coinor-libcoinutils-dev coinor-libcgl-dev, then export COIN_INSTALL_DIR=/usr/ and finally pip3 install --pre cylp.

[mdavis-xyz]

I'm not able to install cylp.

What's the current status of this?

Can someone just show me a Dockerfile? Maybe that's the easiest way to test and document how to install something with such a messy installation process.

e.g. here's a Dockerfile based on @ir1979 's steps, which don't work for me.

FROM python:3-buster
RUN apt-get update
RUN apt-get install -y pkg-config coinor-libcbc-dev coinor-libosi-dev coinor-libcoinutils-dev coinor-libcgl-dev
ENV COIN_INSTALL_DIR /usr/
RUN pip3 install numpy
RUN pip3 install --pre cylp

@tkralphs has the documentation been updated yet?

[mdavis-xyz]

Here's another attempt, trying to compile CBC from source because @tkralphs said it's more likely to succeed.

FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y python3 python3-pip
RUN apt-get install -y wget
RUN apt-get install -y unzip
RUN wget https://www.coin-or.org/download/source/Cbc/Cbc-2.10.3.zip
RUN unzip Cbc-2.10.3.zip
WORKDIR Cbc-2.10.3
RUN ./configure
RUN make
RUN make install
ENV COIN_INSTALL_DIR /usr
RUN pip3 install numpy
RUN pip3 install --pre cylp

This one gives the error:

    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-r65cjqq2/cylp/setup.py", line 87, in <module>
        libs = get_libs()
      File "/tmp/pip-build-r65cjqq2/cylp/setup.py", line 58, in get_libs
        'doc', 'Cbc', 'cbc_addlibs.txt')) as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/coin/doc/Cbc/cbc_addlibs.txt'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-r65cjqq2/cylp/
The command '/bin/sh -c pip3 install --pre cylp' returned a non-zero code: 1

Note that even though CBC has been compiled and installed, no file exists on my hard drive called cbc_addlibs.txt.

[tkralphs]

Sorry, no , I did not get back to updating the documentation, but @ir1979's recipe should work. What went wrong exactly? There was no error message in your output.

For the installation from source, the problem is that you didn't install Cbc in /usr (by default, it installs in the source directory). If you do something like

RUN ./configure --prefix=/usr

then I think it should work.

[tkralphs]

There was a small issue with setup.py under Python 3.6, which is the version installed on the latest Ubuntu Docker image. I fixed that (hopefully not breaking the build on any other platforms) and now I can install CyLP successfully in Docker. See https://github.com/tkralphs/cylp-docker and https://hub.docker.com/r/tkralphs/cylp/. Note that this version fetches source from Github and uses the coinbrew build script, which is the recommended way to build. It is set up to build with Cbc master. If you want to get release 2.10.3, then change the fetch command to

./coinbrew fetch Cbc:2.10.3 --no-prompt --no-third-party

[nkaimcaudle]

I was also having issues installing CyLP on Debian. But rather than missing file errors I was encountering compiler errors. Using Debian Bullseye, Python 3.7 installed via miniconda.

cylp/cy/CyCbcModel.cpp:10544:22: error: ▒~@~XPyThreadState▒~@~Y {aka ▒~@~Xstruct _ts▒~@~Y} has no member named ▒~@~Xexc_traceback▒~@~Y; did you mean ▒~@~Xcurexc_traceback▒~@~Y?

Eventually I got it to work using essentially the same code as @ir1979 but at the end installing CyLP from the github master branch. But I was attempting lots of different approaches, including installing from source, so I cannot guarantee that just these steps will work for others.
pip install git+https://github.com/coin-or/CyLP