Integrating the Jupyter Notebook with ESE
During my time at NV5 (formerly L3Harris/Harris), I have had the opportunity to work with some cutting edge technologies. One of these cool technologies is the addition of the IDL-Python bridge with IDL 8.5. With the IDL-Python bridge you can easily extend IDL to consume open-source code for tasks that IDL may not have pre-written algorithms for. Another great use of the IDL-Python bridge is access to the Jupyter Notebook, a web based solution for programming with IDL.
This made me wonder if it was possible to have a Jupyter Notebook server be a rest endpoint (a web address) for ENVI Services Engine (ESE). It turns out it is possible and a great web-based solution that allows for an interactive use of IDL and the ENVI API on ESE instance. This will not run tasks through the ESE server, but instead will run IDL through Jupyter notebooks on the ESE server and is a great way to make the most of ENVI + IDL for web based solutions.
This blog is going to step through everything you need to do from set-up of the IDL-Python bridge, modifying the Jupyter configuration files, and creating a rest endpoint for the Jupyter server. Don't be scared by the length of the blog, it is actually pretty easy to get the Jupyter Notebook set up and running with ESE. If you have the IDL-Python bridge set up, then you can skip to step two after installing Jupyter for Python.
1) Installing Python and getting the IDL-Python bridge set up
First you will need to get Anaconda (with Python 2.7, not Python 3.4/3.5) installed and a generic google search will take you to the download location. Make sure that during installation make sure you DO NOT specify the options to have Anaconda registered as the system Python or add Anaconda to the PATH environment variable.
Once you have Anaconda installed, you will need to add some environment variables and open up a command prompt. After installation, here is what I use in a batch file for setting up my environment variables. This uses the Anaconda installation location of "C:\python_installs\Anaconda" (just replace these paths with your installation location which will probably be different):
@echo off
REM Change directory so that Jupyter Notebooks will be made in one place
cd %PROGRAMDATA%\ipython\notebooks
REM set our paths for setting up the bridge
REM with these we can call python or IDL from the command line
set PATH=C:\python_installs\Anaconda;^
C:\Program Files\Exelis\IDL85\bin\bin.x86_64;^
%PATH%
set PYTHONPATH=C:\Program Files\Exelis\IDL85\bin\bin.x86_64;^
C:\Program Files\Exelis\IDL85\lib\bridges;
set PYTHONHOME=C:\python_installs\Anaconda
Once you have a batch file with this for the contents, open up a command prompt and then just drag and drop the .bat file into the command prompt window and press enter. Then, type "conda install Jupyter" and wait for Python to install the Jupyter Notebook kernel. You may need to press 'y' to accept the download/update of some python packages.
The next step is to add the IDL-Kernel to ipython's notebooks (ipython is installed with Jupyter). Copy the file C:\Program Files\Exelis\IDL85\lib\bridges\IDL\kernel.json into the directory C:\ProgramData\ipython\kernels\IDL. Make the directories if they don't exist already.
2) Modify the configuration of the Jupyter Notebook server.
Because we have to make the Jupyter Notebook a public server so that we can access it via the host IP address, it is important to introduce some security so no one on your network can access the server without permission. The easiest way to do this is to add a password. There is also the option to set up SSL encryption, but I'm not going to go through that here. If you want to set up SSL, then visit:
https://jupyter-notebook.readthedocs.io/en/4.x/public_server.html
Before creating the password, we have to set up some configuration files for Jupyter. To do this, type the following in the same command prompt from the step above:
python -m Jupyter Notebook --generate-config
Once this has finished, we will now generate a password and set up a public Jupyter server. To generate a password type the following in the same command prompt:
python
from notebook.auth import passwd
passwd()
Once you enter your password, you should see an output which looks like 'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'. At this point, you can then add the hashed password to your jupyter_notebook_config.py file. The default location for this file is in the .jupyter folder in your home directory, C:\Uers\YourUsername\.jupyter. Once you have the text file open, add the following line to the bottom. Make sure that you replace the text in quotes with the output from the hashed password above, otherwise this will not work for you.
c.NotebookApp.password = u'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'
Next, we have to add a few more items to the jupyter_notebook_config.py file so that the server is public which means you can access it from your machine's IP address (add these lines below the line for the password):
# Set ip to '*' to bind on all interfaces (ips) for the public server
c.NotebookApp.ip = '*'
c.NotebookApp.password = u'sha1:bcd259ccf...<your hashed password here>'
c.NotebookApp.open_browser = False
# It is a good idea to set a known, fixed port for server access
c.NotebookApp.port = 9999
Lastly we need to create an exception in the Jupyter Notebook security that will allow the Jupyter web page to be embedded in an HTML iframe. Embedding the notebook in an iframe allows us to create a rest endpoint in ESE that you can go to for directly opening a Jupyter Notebook. For the same configuration file above, "jupyter_notebook_config.py", add the following lines. You should replace yourhostname in the URL with the hostname of your machine. If you don't know this, you can type "HOSTNAME" (without the quotes) into a command prompt. Make all the letters lower case before adding them to the URL where "yourhostname" is found.
c.NotebookApp.tornado_settings = {
'headers': {
'Content-Security-Policy': "frame-ancestors 'http://yourhostname:9191/Jupyter/' 'self' "
}
}
3) Creating the rest endpoint and HTML for the Jupyter Notebook
With ESE 6.0, it is really easy to create new rest endpoints (URLs) which will go to different web pages. The URL that I decided to create for the Jupyter Notebook is http://your.IP.address:9191/jupyter. There are three small text files that you need to create to get this web page set up correctly. First, we have to modify a configuration file for ESE which makes ESE aware of this new URL. To do this, open up the text file C:\ENVIServer60\config.json and add "./requestHandlers/JupyterRequestHandler" (with the quotes) in the request handlers section. The json for requestHandlers should then look something like the following:
"requestHandlers": [
"./requestHandlers/InfoRequestHandler",
"./requestHandlers/HelpRequestHandler",
"./requestHandlers/ESEClassicRequestHandler",
"./requestHandlers/ClusterRequestHandler",
"./requestHandlers/AdminConsoleRequestHandler",
"./requestHandlers/JupyterRequestHandler"
]
Next, we have to make a directory to hold some javascript, html, and json information for this rest endpoint. Make the directory JupyterRequestHandler in the directory "C:\ENVIServer60\requestHandlers" and then open up a text editor. Note that the next three text files should be placed inside the directory C:\ENVIServer60\requestHandlers\JupyterRequestHandler The first text file is going to be called handler.js and should contain:
/**
* This module exposes a /jupyter endpoint that can be used to access a jupyter notebook hosted on the
* ESE server
* @module JupyterRequestHandler
*/
var config = require('../../utils/configLoader').config;
var express = require('express');
var log = require('Logger').logger;
/**
* @class
* @implements {RequestHandler}
*/
function JupyterRequestHandler() {
// Load help pages for all modules
// Inherits docs from RequestHandler interface
this.init = function(app) {
var root = __dirname + '/../..';
app.use('/jupyter', express.static(__dirname));
};
}
module.exports = JupyterRequestHandler;
The second text file will be called package.json and will contain:
{
"name": "JupyterRequestHandler",
"version": "1.0.0",
"description": "Custom request handler that provides access to the Jupyter IDL notebook",
"main": "handler.js",
"dependencies": {},
"devDependencies": {},
"scripts": {}
}
The last file will be called index.html and should contain the following. Make sure you replace "your.IP.address" with the IP address of your machine:
<!DOCTYPE html>
<head>
<style type="text/css">
iframe {position:fixed;
border:0;
top:0;
bottom:0;
left:0;
right:0;
width:100%;
height:100%
}
</style>
</head>
<html>
<iframe src="http://your.IP.address:9999/" ></iframe>
</html>
With that, the set up is complete. You just need to restart ESE and start the Jupyter Notebook server. To start Jupyter, just type the following in the command prompt that has the IDL-Python bridge set up:
python -m jupyter notebook
Once the server has started you should then be able to type http://your.IP.address:9191/jupyter in Internet Explorer and the log in screen for the Jupyter Notebook should pop up. Have fun!