Unleash the power of Jupyter Notebooks

This can be handy for ensuring that errors and warnings about things like invalid inputs or parameterisations are hard to miss for anyone who might be using your notebooks.

An easy, customisable way to hook into this is via the standard Python logging module.

(Note: Just for this section, we’ll use some screenshots so that we can see how these errors look in a real notebook.

)The logging output is displayed separately from print statements or standard cell output, appearing above all of this.

This actually works because Jupyter notebooks listen to both standard output streams, stdout and stderr, but handle each differently; print statements and cell output route to stdout and by default logging has been configured to stream over stderr.

This means we can configure logging to display other kinds of messages over stderr too.

We can customise the format of these messages like so:Note that every time you run a cell that adds a new stream handler via logger.

addHandler(handler), you will receive an additional line of output each time for each message logged.

We could place all the logging config in its own cell near the top of our notebook and leave it be or, as we have done here, brute force replace all existing handlers on the logger.

We had to do that in this case anyway to remove the default handler.

It’s also easy to log to an external file, which might come in handy if you’re executing your notebooks from the command line as discussed later.

Just use a FileHandler instead of a StreamHandler:handler = logging.

FileHandler(filename=’important_log.

log’, mode=’a’)A final thing to note is that the logging described here is not to be confused with using the %config magic to change the application's logging level via %config Application.

log_level="INFO", as this determines what Jupyter outputs to the terminal while it runs.

Working with DatabasesDatabases are a data scientist’s bread and butter, so smoothing the interface between your databases and notebooks is going to be a real boon.

Catherine Devlin’s IPython SQL magic extension let’s you write SQL queries directly into code cells with minimal boilerplate as well as read the results straight into pandas DataFrames.

First, go ahead and:pip install ipython-sqlWith the package installed, we start things off by executing the following magic in a code cell:%load_ext sqlThis loads the ipython-sql extension we just installed into our notebook.

Let's connect to a database!%sql sqlite://'Connected: @None'Here, we just connected to a temporary in-memory database for the convenience of this example, but you’ll probably want to specify details appropriate to your database.

Connection strings follow the SQLAlchemy standard:dialect+driver://username:password@host:port/databaseNote that if you leave the connection string empty, the extension will try to use the DATABASE_URL environment variable; read more about how to customise this in the Scripted Execution section above.

Next, let’s quickly populate our database from the tips dataset from Seaborn we used earlier.

tips = sns.

load_dataset(“tips”)%sql PERSIST tips* sqlite:// 'Persisted tips'We can now execute queries on our database.

Note that we can use a multiline cell magic %% for multiline SQL.

%%sqlSELECT *FROM tipsLIMIT 3* sqlite:// Done.

The complexity of our queries is not limited by the extension, so we can easily write more expressive queries such as finding all the results with a total bill greater than the mean.

result = %sql SELECT * FROM tips WHERE total_bill > (SELECT AVG(total_bill) FROM tips)larger_bills = result.

DataFrame()larger_bills.

head(3)* sqlite:// Done.

And as you can see, converting to a pandas DataFrame was easy too, which makes plotting results from our queries a piece of cake.

The ipython-sql extension also integrates with Matplotlib to let you call .

plot(), .

pie(), and .

bar() straight on your query result, and can dump results direct to a CSV file via .

csv(filename='my-file.

csv').

Using different languages inside single notebookIf you’re missing those much, using other computational kernels:%%python2%%python3%%bash%%Ris possible, but obviously you’ll need to setup the corresponding kernel first.

%%bashecho 'Hi, this is bash.

'Hi, this is bash.

Writing functions in other languagesSometimes the speed of numpy is not enough and I need to write some fast code.

In principle, you can compile function in the dynamic library and write python wrappers…You can write functions in cython or fortran and use those directly from python code.

!pip install cython fortran-magic%load_ext Cython%%cythondef myltiply_by_2(float x): return 2.

0 * xmyltiply_by_2(23.

)46.

0Try fortran, which is very convenient for writing number-crunching functions.

%load_ext fortranmagic/Users/axelr/.

venvs/rep/lib/python2.

7/site-packages/IPython/utils/path.

py:265: UserWarning: get_ipython_cache_dir has moved to the IPython.

paths module warn("get_ipython_cache_dir has moved to the IPython.

paths module")%%fortransubroutine compute_fortran(x, y, z) real, intent(in) :: x(:), y(:) real, intent(out) :: z(size(x, 1))z = sin(x + y)end subroutine compute_fortrancompute_fortran([1, 2, 3], [4, 5, 6])array([-0.

95892429, 0.

65698659, 0.

41211849], dtype=float32)I also should mention that there are different jitter systems which can speed up your python code.

Multiple cursorsJupyer has multicursor support.

Alt + mouse selection for multiline selection and Ctrl + mouse clicks for multicursors.

http://swanintelligence.

com/multi-cursor-in-jupyter.

htmlJupyter-contrib extensionsRun the following in a command prompt:pip install jupyter_contrib_nbextensions && jupyter contrib nbextension installStart up a Jupyter Notebook and navigate to the new Nbextensions tab:Jupyter Notebook Extensions TabEnable the extensions you want and enjoy the productivity benefits.

(If you don’t see a tab, open a notebook and click Edit > nbextensions config)1.

Table of Contents: For easier navigation.

Once you start getting dozens of cells in one Jupyter Notebook, it can be difficult to keep track of them all.

The Table of Contents solves that problem by adding a linked TOC that can be positioned anywhere on the page.

Table of ContentsYou can also use the extension to add a linked table of contents at the top of the notebook.

This even shows which cell is selected and which is running.

2.

Autopep8: For a neat code in one click.

When you’re done writing that amazing code, this extension allows you to simply click the gavel and automatically format your messy code.

Autopep8 your code.

3.

Variable inspector: To keep track of your workspace.

The variable inspector shows the names of all variables you’ve created in the notebook, along with their type, size, shape, and value.

Variable inspector4.

ExecuteTime: To show when and how long cells ran.

To figure out how long a cell took to run or when was it last ran a notebook that has been open for days .

PS : Cooler than %%timeExecuteTime extension output5.

Hide Code input: To hide the work show the results.

The Hide input all extension allows you to instantly hide all the code in the notebook while keeping outputs.

Hide all code6.

Gist-it: To turn a Jupyter Notebook into a gist.

Generate a personal access token.

To allow Jupyter Notebooks access to a personal GitHub API, go to the Github token generator page, enter your password, and click “Generate new token”.

On the next page create a token description, select scope = gist, and click “Generate token”.

Enter access token.

Return to the Jupyter Notebook nbextensions tab and click on the text of “Gist-it”, so that “Gist-it” becomes highlighted in blue.

Scroll down to Gist-it and click “Disable” so that Enable is now blue.

Paste in your Github token, tic the box that says “Gists default to public”, and click “Enable”.

Gist a Jupyter Notebook.

From a Jupyter Notebook, click on the GitHub icon, the symbol 6th from right in the toolbar.

The window shown in figure below should pop up.

Leave the Gist id blank.

You can use the same name as your file name or edit it here.

Click “Gist it!”.

RISE: presentations with notebookExtension by Damian Avila makes it possible to show notebooks as demonstrations.

Find it here.

It is very useful when you teach others e.

g.

to use some library.

Jupyter outputsNotebooks are displayed as HTML and the cell output can be HTML, so you can return virtually anything: video/audio/images.

In this example I scan the folder with images in my repository and show the first five of them:import osfrom IPython.

display import display, Imagenames = [f for f in os.

listdir('.

/images/ml_demonstrations/') if f.

endswith('.

png')]for name in names[:5]: display(Image('.

/images/examples/' + name, width=300))I could take the same list with a bash command because magics and bash calls return python variables:names = !ls .

/images/ml_demonstrations/*.

pngnames[:5]['.

/images/ml_demonstrations/colah_embeddings.

png', '.

/images/ml_demonstrations/convnetjs.

png', '.

/images/ml_demonstrations/decision_tree.

png', '.

/images/ml_demonstrations/decision_tree_in_course.

png', '.

/images/ml_demonstrations/dream_mnist.

png']Wrapping Things UpAlthough you’ll learning an IDE(VS Code) is most necessary if you are writing production code, Jupyter Notebooks are still an integral part of a data science workflow.

If you are going to be using this, you might as well get the most from your tools.

Install these Jupyter Notebook extensions, spend some time figuring out which ones are useful to you, and improve your productivity.

While none of these is life-changing, they all add just enough benefit to be worthwhile, cumulatively saving you hours of valuable development time.

If you feel like writing an extension yourself, take a look at this.

TAh TAh!!.

. More details

Leave a Reply