MacOS vs Anaconda vs Jupyter vs Vim vs YouCompleteMe

Today, I took some time off of my projects and set out to try Jupyter. I had never installed and used it before, so I thought of trying it out to see how I can use it for my needs.

Before I proceed with the rest of the story, let me make it clear that this article is not at all about Jupyter, which of course is an easy to use, wonderful tool for scientific computing.

This article is about the adventures a modern developer may find themselves involved in, without having the slightest idea how they got there.

Jupyter

Anyway, lets get back to the story. The installation instructions could not be simpler. Either install Anaconda, or install it through pip (pip install jupyter), provided that you have python installed of course.

By no means, do I want to doubt the simplicity of the installation. Besides, whatever I faced next, had nothing to do with Jupyter or any of the packages involved in the story alone. Ah, I forgot to mention, that everything is taking place on MacOS X El Capitan and this does have to do with it 🙂

The first thing I tried, contrary to the authors’ advice, was to try pip … and it failed. Why? Because Apple ships python as a natively installed package and has also added something called “System Integrity Protection“, which more or less prevents you from writing in system folders, even with a sudo command. Again, I am passing no judgments, it is indeed for security purposes and does not look like a bad thing. However, since python is natively installed, it resides on a location protected by System Integrity Protection and pip cannot write there (I wonder if this renders pip totally useless. Please comment if you have an opinion on the matter, I did not do any more digging into that).

There is a lot of advice out there, to bypass System Integrity Protection but this looks like a nasty work around that may open the door to future problems. In general, it is my advice to avoid work arounds and spend some time to find a better solution.

So, I tried the second option, Anaconda installation. That worked like a charm.

But, I was not over yet.

Python

Anaconda installs its own python. So, so far I had the native python, a Homebrew installed python I happened to install in the past and the Anaconda python. This already starts to look like a big mess. So I decided to remove the brew python (brew uninstall python is enough to do the trick).

Vim

Much better now. Let’ s try out Jupyter starting with a test python script. I am using vim. Or rather, I was using vim until that moment. Vim started to complain and the main idea was that it could not execute python. Why? After a lot of searching, I found out that the problem was that vim was installed with the brew python I had previously removed, because it was installed through Homebrew. To figure this out, you can do

$ otool -L `which vim`

and search in the linked libraries, or

$ otool -L `which vim` |grep -i python

That was the case for me and the only remedy was to remove vim and build it from sources, with the Anaconda python.

$ brew uninstall vim

Checkout the source code

$ mkdir vim_build
$ git clone https://github.com/vim/vim.git
$ cd vim
$ git pull

 Configure it to use python

$ ./configure --enable-pythoninterp

 Build it and install it

YouCompleteme

So, this problem was fixed, but vim was still not working, as it could not find YouCompleteMe symbols. I had YouCompleteMe installed because it is an excellent vim code assistance tool. Here is the error:

YouCompleteMe unavailable: dlopen(/Users/me/anaconda2/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: _PyByteArray_Type
  Referenced from: /Users/me/anaconda2/lib/python2.7/lib-dynload/_io.so
  Expected in: flat namespace
 in /Users/me/anaconda2/lib/python2.7/lib-dynload/_io.so

This looks like vim is using anaconda python and that perhaps YouCompleteMe was built with another one (Homebrew or system). We can figure this out by checking the main YouCompleteMe library, ycm_core.so, usually located in the bundle installation folder 

$ otool -L ~/.vim/bundle/YouCompleteMe/third_party/ycmd/ycm_core.so
	@rpath/ycm_core.so (compatibility version 0.0.0, current version 0.0.0)
	/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

It is using the native python. To fix this we must rebuild vim and point it to the anaconda python with the PYTHON_LIBRARY and PYTHON_INCLUDE_DIR build options. For me the anaconda installation folder was ~/anaconda2.

$ mkdir ycm_build
$ cd ycm_build
$ make -G "Unix Makefiles" -DPYTHON_LIBRARY=~/anaconda2/lib/libpython2.7.dylib -DPYTHON_INCLUDE_DIR=~/anaconda2/include/. ~/.vim/bundle/YouCompleteMe/third_party/ycmd/cpp/
$ cmake --build . --target ycm_core

That did change the python used in YouCompleteMe

$ otool -L ~/.vim/bundle/YouCompleteMe/third_party/ycmd/ycm_core.so |grep -i python
	@rpath/libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)

To see where @rpath points to

$ otool -l YouCompleteMe/third_party/ycmd/ycm_core.so |grep path
         name @rpath/ycm_core.so (offset 24)
         name @rpath/libpython2.7.dylib (offset 24)
         path /Users/me/anaconda2/lib (offset 12)

So it is linked to the anaconda python.

But to my disappointment, the error was still there. The same error as before. Where does this leave us? It is some kind of clash between vim and YCM or something else? At least they both seem to be using the same python, and python is the key in this situation. So what is the problem? 

By looking at the manually built vim again I noticed that the one I built and the one installed in the /usr/local/bin are not the same files, even though the second one ought to be a copy of the first one, installed there with the “sudo make install” command. And most important, the manually built one, works like a charm without any errors!

$ diff vim /usr/local/bin/vim
Binary files vim and /usr/local/bin/vim differ

So perhaps, installation failed. I repeated it and again the same error. That means that the installation script is doing something nasty. It did not take much time to figure it out. It was the strip command. The installation script had this

### Program to run on installed binary.  Use the second one to disable strip.
#STRIP = strip<
#STRIP = /bin/true

MacOS El Capitan was about to be defeated, but with its last breath it posed one more problem. “True” command is not in /bin folder but in the /usr/bin. So I changed the Makefile like this

### Program to run on installed binary.  Use the second one to disable strip.
#STRIP = strip
#STRIP = /bin/true
STRIP = /usr/bin/true

By that time everything was back in place and working beautifully. But wait a minute, where did all this start from … . Ah, yes, Jupyter. I was about to give it a try. I will do that, after I take some time to recover from all that first!

Take care and thanks for reading.