Prerequisites

1.2. Prerequisites#

part of

MSE672: Introduction to Transmission Electron Microscopy

Spring 2024

Gerd Duscher	Khalid Hattar
Microscopy Facilities	Tennessee Ion Beam Materials Laboratory
Materials Science & Engineering	Nuclear Engineering
Institute of Advanced Materials & Manufacturing
The University of Tennessee, Knoxville

Background and methods to analysis and quantification of data acquired with transmission electron microscopes.

1.2.1. Language#

The notebooks are all in python 3.

At this point the common version is python 3.9

cd

1.2.2. Packages#

The idea behind any python program is to make use of the highly efficient libraries that already exist.

I use anaconda3 (not miniconda) which is available for all major operating systems.

We use a few modules that come with every python installation, like:

math
sys
os

We use mostly the common packages for scientific computing in python (all included in anaconda3) The most important ones are:

Numpy - the numerical library
Scipy the scientific library
Matplotlib the interactive plotting library

These packages are expected to be installed on your computer to run the notebooks of this book. These packages are already installed in Google colab.

For specialist applications we do not reinvent the wheel and use those on an as needed basis.

Example is the library to register a stack of images:

SimpleITK integration of AI in image analysis and for high performing computer algorithms
pyNSID the atomistic simulation program is used for crystallographic data
ase together with a symmetry package
spglib

For dialogs, we use the capabilities of Jupyter Widgets provided by:

ipywidgets

All routines that are introduced in the notebooks are also available (for analysis) in the provided package

pyTEMlib

If you install pyTEMlib with the code cell below all packages you need for this book will be installed.

1.2.3. Basic Installation of Python Environment#

I recommend installing the free anaconda3 from this link.

If you have an old version of anaconda, please reinstall the new version. The lecture is based on fairly new packages of scipy and numpy, dask, and h5py, which will create problems in old anaconda versions.

Once you have installed anaconda type

pip install pyTEMlib

Alternatively you can just run the code cell below. Be aware that the installation process may take a while.

import sys
import importlib.metadata
def test_package(package_name):
    """Test if package exists and returns version or -1"""
    try:
        version = importlib.metadata.version(package_name)
    except importlib.metadata.PackageNotFoundError:
        version = '-1'
    return version

# pyTEMlib setup ------------------
if test_package('pyTEMlib') < '0.2023.9.1':
    print('installing pyTEMlib')
    !{sys.executable} -m pip install --upgrade git+https://github.com/pycroscopy/pyTEMlib.git@main -q --upgrade

if 'google.colab' in sys.modules:
    !{sys.executable} -m pip install numpy==1.24.4
# ------------------------------
print('done')

done

1.2.4. Data Format#

All data in this course are stored in the data format of

pyNSID

which is based on

HDF5

1.2.5. Notebook preamble#

As a minimum Any notebook in this course has to have the following libraries loaded :

# import matplotlib and numpy 
# Define iteractivity with this **magic** command
#                       use "inline" instead of "widget" for non-interactive plots
%matplotlib widget

import matplotlib.pylab as plt
import numpy as np

1.2.6. Numpy#

The calculations depend on Numpy and an installation of that package that is compiled to include BLAS and LAPACK libraries will be much faster than the standalone version. For example the numpy installed on ubuntu with > sudo apt-get install python3-numpy or at windows you can install the numpy package from Gohlke’s webpage which compiled against the MKL library of Intel. If you used anaconda3, everything is already optimized.

The command below lets you see what you have

## What is numpy compiled against
np.__config__.show()
print('numpy version: ',np.version.version)
import scipy as sp
print('scipy version: ',sp.__version__)

Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /c/opt/64/include
    lib directory: /c/opt/64/lib
    name: openblas64
    openblas configuration: USE_64BITINT=1 DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= SKYLAKEX MAX_THREADS=2
    pc file directory: C:/opt/64/lib/pkgconfig
    version: 0.3.23.dev
  lapack:
    detection method: internal
    found: true
    include directory: unknown
    lib directory: unknown
    name: dep2628220032720
    openblas configuration: unknown
    pc file directory: unknown
    version: 1.26.3
Compilers:
  c:
    commands: cl
    linker: link
    name: msvc
    version: 19.29.30153
  c++:
    commands: cl
    linker: link
    name: msvc
    version: 19.29.30153
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.0.7
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
Python Information:
  path: C:\Users\runneradmin\AppData\Local\Temp\cibw-run-frm2g0yv\cp311-win_amd64\build\venv\Scripts\python.exe
  version: '3.11'
SIMD Extensions:
  baseline:
  - SSE
  - SSE2
  - SSE3
  found:
  - SSSE3
  - SSE41
  - POPCNT
  - SSE42
  - AVX
  - F16C
  - FMA3
  - AVX2
  - AVX512F
  - AVX512CD
  - AVX512_SKX
  - AVX512_CLX
  - AVX512_CNL
  - AVX512_ICL

numpy version:  1.26.3
scipy version:  1.11.4

1.2.7. TEM Library#

You will have to run the code cell below at least once to install the library with the programs needed for the analysis of data.

The code cell below will install pyTEMlib directly from pypi

import sys
import importlib.metadata
def test_package(package_name):
    """Test if package exists and returns version or -1"""
    try:
        version = importlib.metadata.version(package_name)
    except importlib.metadata.PackageNotFoundError:
        version = -1
    return version

# pyTEMlib setup ------------------
if test_package('pyTEMlib') < '0.2023.9.1':
    print('installing pyTEMlib')
    !{sys.executable} -m pip install --upgrade git+https://github.com/pycroscopy/pyTEMlib.git@main -q --upgrade

if 'google.colab' in sys.modules:
    !{sys.executable} -m pip install numpy==1.24.4
# ------------------------------
print('done')

installing pyTEMlib
done

Now we load pyTEMlib to make it available for this notebook.

import pyTEMlib
print(f'pyTEM version: {pyTEMlib.__version__}')

pyTEM version: 0.2023.1.0

1.2.8. Test#

Let’s test if the installation was successful and plot a unit cell. You can rotate the plot around, zoom and select. Try it!

import pyTEMlib.kinematic_scattering as ks # Import kinematic scattering library from pyTEMlib

# make a structure dictionary
atoms = ks.structure_by_name('Graphite')

atoms

You don't have igor2 installed.     If you wish to open igor files, you will need to install it     (pip install igor2) before attempting.
You don't have gwyfile installed.     If you wish to open .gwy files, you will need to      install it (pip install gwyfile) before attempting.
Symmetry functions of spglib enabled
Using kinematic_scattering library version {_version_ }  by G.Duscher

Atoms(symbols='C4', pbc=False, cell=[[2.46772414, 0.0, 0.0], [-1.2338620699999996, 2.1371117947721068, 0.0], [0.0, 0.0, 6.711]])

from ase.visualize import view
view(atoms*(4,4,1))

<Popen: returncode: None args: ['C:\\Users\\gduscher\\AppData\\Local\\anacon...>

Get used to changing parameters and type silicon instead of Graphite in the code cell above.

1.2.9. Summary#

We now have tested all tools to load data and save our analysis.

We are ready to go

1.2.11. Appendix#

I am using some extensions to jupyter notebooks which can be installed with the following cells

I like mostly the table of content extension that shows where in the notebook I am and lets me jump to different parts easily.

# Install a pip package in the current Jupyter kernel
import sys
!{sys.executable} -m jupyter nbextension list

Known nbextensions:
  config dir: C:\Users\gdusc\anaconda3\etc\jupyter\nbconfig
    notebook section
      jupyter-jsmol/extension enabled 
      jupyterlab-plotly/extension enabled 
      jupyter-js-widgets/extension enabled 

      - Validating: ok
      - Validating: ok
      - Validating: ok

Use one of the following (uncomment pip and comment out the conda command line, if you are not in an anaconda environment)

import sys

#!{sys.executable} -m pip install jupyter_contrib_nbextensions
!conda install --yes --prefix {sys.prefix} jupyter_contrib_nbextensions

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - jupyter_contrib_nbextensions

Current channels:

  - https://repo.anaconda.com/pkgs/main/win-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/win-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-64
  - https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

!{sys.executable} -m  jupyter nbextension enable toc2

Enabling notebook extension toc2...
      - Validating: problems found:
        - require?  X toc2

%%javascript
$('<div id="toc"></div>').css({position: 'fixed', top: '120px', left: 0}).appendTo(document.body);
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js');