Skip to article frontmatterSkip to article content

Open DM3 Images, Spectra, Spectrum-Images and Image-Stacks with pyNSID

Chapter 1: Introduction

Open DM3 Images, Spectra, Spectrum-Images and Image-Stacks with pyNSID

Download

OpenInColab

part of

MSE672: Introduction to Transmission Electron Microscopy

Spring 2025
by Gerd Duscher

Microscopy Facilities
Institute of Advanced Materials & Manufacturing
Materials Science & Engineering
The University of Tennessee, Knoxville

Background and methods to analysis and quantification of data acquired with transmission electron microscopes.

Reading a dm file and translating the data in a pyNSID style hf5py file to be compatible with the pycroscopy package.

Because, many other packages and programs for TEM data manipulation are based on the hdf5 file-formats it is relatively easy to convert back and forward between them.

Import packages for figures and

Check Installed Packages

import sys
import importlib.metadata
def test_package(package_name):
    """Test if package exists and returns version or -1"""
    try:
        version = importlib.metadata.version(package_name)
    except importlib.metadata.PackageNotFoundError:
        version = '-1'
    return version

# pyTEMlib setup ------------------
if test_package('pyTEMlib') < '0.2024.9.0':
    print('installing pyTEMlib')
    !{sys.executable} -m pip install --upgrade git+https://github.com/pycroscopy/pyTEMlib.git@main -q --upgrade
# ------------------------------
print('done')
done

Load the plotting and figure packages

%matplotlib widget
import matplotlib.pylab as plt
import numpy as np
import sys

import pyTEMlib
import pyTEMlib.file_tools  as ft     # File input/ output library

import sidpy
import pyNSID
import h5py

if 'google.colab' in sys.modules:
    from google.colab import output
    output.enable_custom_widget_manager()
    from google.colab import drive
    drive.mount("/content/drive")

# For archiving reasons it is a good idea to print the version numbers out at this point
print('pyTEM version: ',pyTEMlib.__version__)
__notebook__='CH1_04-Reading_File'
__notebook_version__='2024_01_09'
You don't have igor2 installed.     If you wish to open igor files, you will need to install it     (pip install igor2) before attempting.
You don't have gwyfile installed.     If you wish to open .gwy files, you will need to      install it (pip install gwyfile) before attempting.
Symmetry functions of spglib enabled
pyTEM version:  0.2024.09.0

Open a file

This function opens a hfd5 file in the pyNSID style which enables you to keep track of your data analysis.

Please see the Installation notebook for installation.

We want to consolidate files into one dataset that belongs together. For example a spectrum image dataset consists of:

  • Survey image,

  • EELS spectra

  • Z-contrast image acquired simultaneously with the spectra.

So load the top dataset first in the above example the survey image.

Please note that the plotting routine of matplotlib was introduced in Matplotlib and Numpy for Micrographs notebook.

Use the file p1-3hr.dm3 from TEM_data directory for a practice run

# ------ Input ------- #
load_example = True
# -------------------- #

# Open file widget and select file which will be opened in code cell below
if not load_example:
    drive_directory = ft.get_last_path()
    file_widget = ft.FileWidget(drive_directory)
if load_example:
    file_name = '../example_data/p1-3-hr3.dm3'
    datasets = ft.open_file(file_name)
    main_dataset = datasets[list(datasets.keys())[0]]
else:
    main_dataset = file_widget.selected_dataset
    datasets = file_widget.datasets

view = main_dataset.plot()
Loading...

Data Structure

The data themselves reside in a sidpy dataset which we name current_dataset.

The current_dataset has additional information stored as attributes which can be accessed through their name.

print(main_dataset)
main_dataset
Loading...
print(f'size of current dataset is {main_dataset.shape}')
size of current dataset is (2048, 2048)

The current_dataset has additional information stored as attributes which can be accessed through their name.

There are two dictionaries within that attributes:

  • metadata

  • original_metadata

which contain additional information about the data

print('title: ', main_dataset.title)
print('data type: ', main_dataset.data_type)

for key in datasets:
    print(key)
    print(datasets[key].original_metadata.keys())
    
main_dataset.metadata  
title:  p1-3-hr3
data type:  DataType.IMAGE
Channel_000
dict_keys(['ImageData', 'ImageTags', 'Name', 'UniqueID', 'DM', 'original_filename', 'ApplicationBounds', 'DocumentObjectList', 'DocumentTags', 'HasWindowPosition', 'Image Behavior', 'ImageSourceList', 'InImageMode', 'MinVersionList', 'NextDocumentObjectID', 'Page Behavior', 'PageSetup', 'SentinelList', 'Thumbnails', 'WindowPosition'])
{'experiment': {'exposure_time': 1.0, 'microscope': 'Libra 200 MC', 'acceleration_voltage': 199990.28125}}

Data Structure

The datasets variable is a dictionary (like a directory in a file system) which containes contains datasets.

Below I show how to access one of those datasets with a pull down menu.

chooser = ft.ChooseDataset(datasets)
Loading...
current_dataset = chooser.dataset
view = current_dataset.plot()
Loading...

An important attribute in current_dataset is the original_metadata group, where all the original metadata of your file reside in the attributes. This is usually a long list for dm3 files.

current_dataset.original_metadata.keys()
dict_keys(['ImageData', 'ImageTags', 'Name', 'UniqueID', 'DM', 'original_filename', 'ApplicationBounds', 'DocumentObjectList', 'DocumentTags', 'HasWindowPosition', 'Image Behavior', 'ImageSourceList', 'InImageMode', 'MinVersionList', 'NextDocumentObjectID', 'Page Behavior', 'SentinelList', 'Thumbnails', 'WindowPosition', 'original_title'])

The original_metadata attribute has all information stored from the orginal file.

No information will get lost

for key,value in current_dataset.original_metadata.items():
    print(key, value)
print(current_dataset.h5_dataset)    
ImageData {'Calibrations': {'Brightness': {'Origin': 0.0, 'Scale': 1.0, 'Units': ''}, 'Dimension': {'0': {'Origin': 0.0, 'Scale': 0.03894666209816933, 'Units': 'nm'}, '1': {'Origin': 0.0, 'Scale': 0.03894666209816933, 'Units': 'nm'}}, 'DisplayCalibratedUnits': 1}, 'Data': 'read', 'DataType': 7, 'Dimensions': {'0': 2048, '1': 2048}, 'PixelDepth': 4}
ImageTags {'Acquisition': {'Device': {'Active Size (pixels)': [2048, 2048], 'Camera Number': 0, 'CCD': {'Pixel Size (um)': [14.0, 14.0]}, 'Configuration': {'Transpose': {'Diagonal Flip': 0, 'Horizontal Flip': 1, 'Vertical Flip': 0}}, 'Name': 'US1000XP 1', 'Source': 'US1000XP 1'}, 'Frame': {'Area': {'Transform': {'Class Name': 'cm_acquisitiontransform_list', 'Transform List': {'0': {'Binning': [1, 1], 'Class Name': 'cm_acquisitiontransform', 'Sub Area Adjust': [0, 0, 0, 0], 'Transpose': {'Diagonal Flip': 0, 'Horizontal Flip': 1, 'Vertical Flip': 0}}}}}, 'CCD': {'Pixel Size (um)': [14.0, 14.0]}, 'Intensity': {'Range': {'Bias (counts)': 250.0, 'Dark Current (counts/s)': 0.0, 'Dark Level (counts)': 250.0, 'Maximum Value (counts)': 65535.0, 'Minimum Value (counts)': 0.0, 'Saturation Level (counts)': 65785.0}, 'Transform': {'Class Name': 'cm_valuetransform_list', 'Transform List': {'0': {'Class Name': 'cm_valuetransform_affine', 'Offset': 250.0, 'Scale': 1.0}, '1': {'ADC Max': 65535.0, 'ADC Min': 0.0, 'Class Name': 'cm_valuetransform_adc'}}}}, 'Reference Images': {'Dark': {'Mean (counts)': 251.31182837486267, 'Standard Deviation (counts)': 3.28830754005297}}, 'Sequence': {'Acquisition Start Time (epoch)': 1528991109318.0, 'Exposure Start (ns)': 130828979.37529667, 'Exposure Time (ns)': 1000000000.0, 'Frame Index': 0, 'Frame Start (ns)': 130828979.37529667, 'Frame Time (ns)': 3482273464.1519103, 'Readout Time (ns)': 3482273464.1519103}}, 'Parameters': {'Acquisition Write Flags': 4294967295, 'Base Detector': {'Class Name': 'cm_namedcameradetectorparameterset', 'Name': 'default'}, 'Detector': {'continuous': 0, 'exposure (s)': 1.0, 'hbin': 1, 'height': 2048, 'left': 0, 'top': 0, 'vbin': 1, 'width': 2048}, 'Environment': {'Mode Name': 'Imaging'}, 'High Level': {'Acquisition Buffer Size': 0, 'Antiblooming': 0, 'Binning': [1, 1], 'CCD Read Area': [0, 0, 2048, 2048], 'CCD Read Ports': 1, 'Choose Number Of Frame Shutters Automatically': 1, 'Class Name': 'cm_camera_highlevelparameters', 'Continuous Readout': 0, 'Corrections': 817, 'Corrections Mask': 817, 'Exposure (s)': 1.0, 'Number Of Frame Shutters': 1, 'Processing': 'Gain Normalized', 'Quality Level': 1, 'Read Frame Style': 0, 'Read Mode': 0, 'Secondary Shutter Post Exposure Compensation (s)': 0.0, 'Secondary Shutter Pre Exposure Compensation (s)': 0.0, 'Shutter': {'Primary Shutter States': 0, 'Primary Shutter States Mask': 0, 'Secondary Shutter States': 0, 'Secondary Shutter States Mask': 0, 'Shutter Exposure': 0, 'Shutter Index': 0}, 'Shutter Post Exposure Compensation (s)': 0.0, 'Shutter Pre Exposure Compensation (s)': 0.0, 'Transform': {'Diagonal Flip': 0, 'Horizontal Flip': 0, 'Vertical Flip': 0}}, 'Objects': {'0': {'Class Name': 'cm_autoexpose_acquireimage', 'Do Auto Expose': 0, 'Maximum Exposure (s)': 5.0, 'Minimum Exposure (s)': 0.1, 'Target Intensity (%)': 50.0, 'Test Exposure (s)': 0.1}, '1': {'Class Name': 'cm_imgproc_finalcombine', 'Frame Combine Style': 'Copy', 'Parameter 1': 1.0}, '2': {'Class Name': 'cm_stdviewerimagedisplayer', 'Do Auto Zoom': 0, 'Screen Relative Position': [1.0, 0.0], 'Version': 33947648, 'View Name': 'Frame', 'Viewer Class': 'acquire', 'Window Relative Position': [1.0, 0.0], 'Zoom': 1.0}, '3': {'Class Name': 'cm_imgproc_histogram'}}, 'Parameter Set Name': 'Record', 'Parameter Set Tag Path': 'Imaging:Acquire:Record', 'Version': 33947648}}, 'DataBar': {'Acquisition Date': '6/14/2018', 'Acquisition Time': '11:45:13 AM', 'Acquisition Time (OS)': 1.3173464713128774e+17, 'Binning': 1, 'Custom elements': {}, 'Device Name': 'US1000XP 1', 'Exposure Number': 13083663, 'Exposure Time (s)': 1.0}, 'Microscope Info': {'Actual Magnification': 359465.9784890286, 'Cs(mm)': 2.2, 'Emission Current (A)': 230.0, 'Formatted Indicated Mag': '315kx', 'Formatted Voltage': '200.0kV', 'HT Extrapolated': 0, 'Illumination Mode': 'TEM', 'Imaging Mode': 'Image Mag', 'Indicated Magnification': 315000.0, 'Items': {'0': {'Data Type': 20, 'Label': 'Specimen', 'Tag path': 'Microscope Info:Specimen', 'Value': 'Fe-9Cr(0.3Y)-3E10(17)-475C'}, '1': {'Data Type': 20, 'Label': 'Operator', 'Tag path': 'Microscope Info:Operator', 'Value': 'Tengfei Yang'}, '2': {'Data Type': 20, 'Label': 'Microscope', 'Tag path': 'Microscope Info:Microscope', 'Value': 'Libra 200 MC'}}, 'Magnification Interpolated': 0, 'Microscope': 'Libra 200 MC', 'Name': 'Libra COM', 'Operation Mode': 'IMAGING', 'Operator': 'Tengfei Yang', 'Probe Current (nA)': 0.0, 'Probe Size (nm)': 0.0, 'Specimen': 'Fe-9Cr(0.3Y)-3E10(17)-475C', 'STEM Camera Length': 94.49999779462814, 'Voltage': 199990.28125}}
Name p1-3-hr3
UniqueID {'0': 3677084, '1': 1068567676, '2': 1927893732, '3': 2094808766}
DM {'dm_version': 3, 'file_size': 17382688, 'full_file_name': '../example_data/p1-3-hr3.dm3'}
original_filename ../example_data/p1-3-hr3.dm3
ApplicationBounds [0, 0, 1465, 2236]
DocumentObjectList {'0': {'AnnotationGroupList': {'0': {'AnnotationType': 31, 'BackgroundColor': [0, 0, 0], 'BackgroundMode': 1, 'FillMode': 1, 'Font': {'Attributes': 7, 'FamilyName': 'Arial Narrow', 'Size': 85}, 'ForegroundColor': [-1, -1, -1], 'HasBackground': 1, 'IsMoveable': 1, 'IsResizable': 1, 'IsSelectable': 1, 'IsTranslatable': 1, 'IsVisible': 1, 'ObjectTags': {}, 'Rectangle': [1768.0, 128.0, 1920.0, 1088.0], 'TextOffsetH': 1.0, 'TextOffsetV': 1.0, 'TextWidth': 195.18429565429688, 'UniqueID': 9}}, 'AnnotationType': 20, 'BackgroundColor': [-1, -1, -1], 'BackgroundMode': 2, 'FillMode': 1, 'ForegroundColor': [-1, 0, -32640], 'HasBackground': 0, 'ImageDisplayInfo': {'BrightColor': [-1, -1, -1], 'Brightness': 0.5, 'CaptionOn': 0, 'CaptionSize': 10, 'CLUT': [[0, 0, 0], [257, 257, 257], [514, 514, 514], [771, 771, 771], [1028, 1028, 1028], [1285, 1285, 1285], [1542, 1542, 1542], [1799, 1799, 1799], [2056, 2056, 2056], [2313, 2313, 2313], [2570, 2570, 2570], [2827, 2827, 2827], [3084, 3084, 3084], [3341, 3341, 3341], [3598, 3598, 3598], [3855, 3855, 3855], [4112, 4112, 4112], [4369, 4369, 4369], [4626, 4626, 4626], [4883, 4883, 4883], [5140, 5140, 5140], [5397, 5397, 5397], [5654, 5654, 5654], [5911, 5911, 5911], [6168, 6168, 6168], [6425, 6425, 6425], [6682, 6682, 6682], [6939, 6939, 6939], [7196, 7196, 7196], [7453, 7453, 7453], [7710, 7710, 7710], [7967, 7967, 7967], [8224, 8224, 8224], [8481, 8481, 8481], [8738, 8738, 8738], [8995, 8995, 8995], [9252, 9252, 9252], [9509, 9509, 9509], [9766, 9766, 9766], [10023, 10023, 10023], [10280, 10280, 10280], [10537, 10537, 10537], [10794, 10794, 10794], [11051, 11051, 11051], [11308, 11308, 11308], [11565, 11565, 11565], [11822, 11822, 11822], [12079, 12079, 12079], [12336, 12336, 12336], [12593, 12593, 12593], [12850, 12850, 12850], [13107, 13107, 13107], [13364, 13364, 13364], [13621, 13621, 13621], [13878, 13878, 13878], [14135, 14135, 14135], [14392, 14392, 14392], [14649, 14649, 14649], [14906, 14906, 14906], [15163, 15163, 15163], [15420, 15420, 15420], [15677, 15677, 15677], [15934, 15934, 15934], [16191, 16191, 16191], [16448, 16448, 16448], [16705, 16705, 16705], [16962, 16962, 16962], [17219, 17219, 17219], [17476, 17476, 17476], [17733, 17733, 17733], [17990, 17990, 17990], [18247, 18247, 18247], [18504, 18504, 18504], [18761, 18761, 18761], [19018, 19018, 19018], [19275, 19275, 19275], [19532, 19532, 19532], [19789, 19789, 19789], [20046, 20046, 20046], [20303, 20303, 20303], [20560, 20560, 20560], [20817, 20817, 20817], [21074, 21074, 21074], [21331, 21331, 21331], [21588, 21588, 21588], [21845, 21845, 21845], [22102, 22102, 22102], [22359, 22359, 22359], [22616, 22616, 22616], [22873, 22873, 22873], [23130, 23130, 23130], [23387, 23387, 23387], [23644, 23644, 23644], [23901, 23901, 23901], [24158, 24158, 24158], [24415, 24415, 24415], [24672, 24672, 24672], [24929, 24929, 24929], [25186, 25186, 25186], [25443, 25443, 25443], [25700, 25700, 25700], [25957, 25957, 25957], [26214, 26214, 26214], [26471, 26471, 26471], [26728, 26728, 26728], [26985, 26985, 26985], [27242, 27242, 27242], [27499, 27499, 27499], [27756, 27756, 27756], [28013, 28013, 28013], [28270, 28270, 28270], [28527, 28527, 28527], [28784, 28784, 28784], [29041, 29041, 29041], [29298, 29298, 29298], [29555, 29555, 29555], [29812, 29812, 29812], [30069, 30069, 30069], [30326, 30326, 30326], [30583, 30583, 30583], [30840, 30840, 30840], [31097, 31097, 31097], [31354, 31354, 31354], [31611, 31611, 31611], [31868, 31868, 31868], [32125, 32125, 32125], [32382, 32382, 32382], [32639, 32639, 32639], [-32640, -32640, -32640], [-32383, -32383, -32383], [-32126, -32126, -32126], [-31869, -31869, -31869], [-31612, -31612, -31612], [-31355, -31355, -31355], [-31098, -31098, -31098], [-30841, -30841, -30841], [-30584, -30584, -30584], [-30327, -30327, -30327], [-30070, -30070, -30070], [-29813, -29813, -29813], [-29556, -29556, -29556], [-29299, -29299, -29299], [-29042, -29042, -29042], [-28785, -28785, -28785], [-28528, -28528, -28528], [-28271, -28271, -28271], [-28014, -28014, -28014], [-27757, -27757, -27757], [-27500, -27500, -27500], [-27243, -27243, -27243], [-26986, -26986, -26986], [-26729, -26729, -26729], [-26472, -26472, -26472], [-26215, -26215, -26215], [-25958, -25958, -25958], [-25701, -25701, -25701], [-25444, -25444, -25444], [-25187, -25187, -25187], [-24930, -24930, -24930], [-24673, -24673, -24673], [-24416, -24416, -24416], [-24159, -24159, -24159], [-23902, -23902, -23902], [-23645, -23645, -23645], [-23388, -23388, -23388], [-23131, -23131, -23131], [-22874, -22874, -22874], [-22617, -22617, -22617], [-22360, -22360, -22360], [-22103, -22103, -22103], [-21846, -21846, -21846], [-21589, -21589, -21589], [-21332, -21332, -21332], [-21075, -21075, -21075], [-20818, -20818, -20818], [-20561, -20561, -20561], [-20304, -20304, -20304], [-20047, -20047, -20047], [-19790, -19790, -19790], [-19533, -19533, -19533], [-19276, -19276, -19276], [-19019, -19019, -19019], [-18762, -18762, -18762], [-18505, -18505, -18505], [-18248, -18248, -18248], [-17991, -17991, -17991], [-17734, -17734, -17734], [-17477, -17477, -17477], [-17220, -17220, -17220], [-16963, -16963, -16963], [-16706, -16706, -16706], [-16449, -16449, -16449], [-16192, -16192, -16192], [-15935, -15935, -15935], [-15678, -15678, -15678], [-15421, -15421, -15421], [-15164, -15164, -15164], [-14907, -14907, -14907], [-14650, -14650, -14650], [-14393, -14393, -14393], [-14136, -14136, -14136], [-13879, -13879, -13879], [-13622, -13622, -13622], [-13365, -13365, -13365], [-13108, -13108, -13108], [-12851, -12851, -12851], [-12594, -12594, -12594], [-12337, -12337, -12337], [-12080, -12080, -12080], [-11823, -11823, -11823], [-11566, -11566, -11566], [-11309, -11309, -11309], [-11052, -11052, -11052], [-10795, -10795, -10795], [-10538, -10538, -10538], [-10281, -10281, -10281], [-10024, -10024, -10024], [-9767, -9767, -9767], [-9510, -9510, -9510], [-9253, -9253, -9253], [-8996, -8996, -8996], [-8739, -8739, -8739], [-8482, -8482, -8482], [-8225, -8225, -8225], [-7968, -7968, -7968], [-7711, -7711, -7711], [-7454, -7454, -7454], [-7197, -7197, -7197], [-6940, -6940, -6940], [-6683, -6683, -6683], [-6426, -6426, -6426], [-6169, -6169, -6169], [-5912, -5912, -5912], [-5655, -5655, -5655], [-5398, -5398, -5398], [-5141, -5141, -5141], [-4884, -4884, -4884], [-4627, -4627, -4627], [-4370, -4370, -4370], [-4113, -4113, -4113], [-3856, -3856, -3856], [-3599, -3599, -3599], [-3342, -3342, -3342], [-3085, -3085, -3085], [-2828, -2828, -2828], [-2571, -2571, -2571], [-2314, -2314, -2314], [-2057, -2057, -2057], [-1800, -1800, -1800], [-1543, -1543, -1543], [-1286, -1286, -1286], [-1029, -1029, -1029], [-772, -772, -772], [-515, -515, -515], [-258, -258, -258], [-1, -1, -1]], 'CLUTName': 'Greyscale', 'ComplexMode': 4, 'ComplexRange': 1000.0, 'Contrast': 0.5, 'ContrastMode': 1, 'DimensionLabels': {'0': ''}, 'DoAutoSurvey': 1, 'EstimatedMax': 656.0, 'EstimatedMaxTrimPercentage': 0.0010000000474974513, 'EstimatedMin': 2354.0, 'EstimatedMinTrimPercentage': 0.0010000000474974513, 'Gamma': 0.5, 'HighLimit': 2107.03125, 'HiLimitContrastDeltaTriggerPercentage': 0.0, 'IsInverted': 0, 'LowLimit': 718.9080810546875, 'LowLimitContrastDeltaTriggerPercentage': 0.0, 'MainSliceId': {'0': 0}, 'MinimumContrast': 0.0, 'RangeAdjust': 1.0, 'SparseSurvey_GridSize': 16, 'SparseSurvey_NumberPixels': 32, 'SparseSurvey_UseNumberPixels': 1, 'SurveyTechique': 2}, 'ImageDisplayType': 1, 'ImageSource': 0, 'IsMoveable': 1, 'IsResizable': 1, 'IsSelectable': 1, 'IsTranslatable': 1, 'IsVisible': 1, 'ObjectTags': {}, 'Rectangle': [0.0, 0.0, 1427.0, 1427.0], 'UniqueID': 8}}
DocumentTags {}
HasWindowPosition 1
Image Behavior {'DoIntegralZoom': 0, 'ImageDisplayBounds': [0.0, 0.0, 1427.0, 1427.0], 'IsZoomedToWindow': 1, 'UnscaledTransform': {'Offset': [0.0, 0.0], 'Scale': [1.0, 1.0]}, 'ViewDisplayID': 8, 'WindowRect': [0.0, 0.0, 1427.0, 1427.0], 'ZoomAndMoveTransform': {'Offset': [0.0, 0.0], 'Scale': [1.0, 1.0]}}
ImageSourceList {'0': {'ClassName': 'ImageSource:Simple', 'Id': {'0': 0}, 'ImageRef': 1}}
InImageMode 1
MinVersionList {'0': {'RequiredVersion': 50659328}}
NextDocumentObjectID 10
Page Behavior {'DoIntegralZoom': 0, 'DrawMargins': 1, 'DrawPaper': 1, 'IsFixedInPageMode': 0, 'IsZoomedToWindow': 1, 'LayedOut': 0, 'PageTransform': {'Offset': [0.0, 0.0], 'Scale': [1.0, 1.0]}, 'RestoreImageDisplayBounds': [0.0, 0.0, 2048.0, 2048.0], 'RestoreImageDisplayID': 8, 'TargetDisplayID': 4294967295}
PageSetup {'General': [1, 1000, 8500, 11000, 1000, 1000, -1000, -1000], 'Win32': b'\x06\x00\x00\x004!\x00\x00\xf8*\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe8\x03\x00\x00\xe8\x03\x00\x00\xe8\x03\x00\x00\xe8\x03\x00\x00\x00\x00\x00\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x1b\x10', 'Win32_DevModeW': b'S\x00e\x00n\x00d\x00 \x00T\x00o\x00 \x00O\x00n\x00e\x00N\x00o\x00t\x00e\x00 \x002\x000\x001\x000\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x04\x00\x06\xdc\x00\x0c\x03\x03\xff\x81\x03\x01\x00\x01\x00\xea\no\x08d\x00\x01\x00\x07\x00\xfd\xff\x02\x00\x01\x00X\x02\x01\x00\x00\x00L\x00e\x00t\x00t\x00e\x00r\x00 \x008\x00.\x005\x00"\x00x\x001\x001\x00"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00\xff\xff\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00DINU"\x00\xd0\x00\x0c\x03\x00\x00\xc2\xac\x90Q\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x00\x00\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xd0\x00\x00\x00SMTJ\x00\x00\x00\x00\x10\x00\xc0\x00S\x00e\x00n\x00d\x00 \x00T\x00o\x00 \x00M\x00i\x00c\x00r\x00o\x00s\x00o\x00f\x00t\x00 \x00O\x00n\x00e\x00N\x00o\x00t\x00e\x00 \x002\x000\x001\x000\x00 \x00D\x00r\x00i\x00v\x00e\x00r\x00\x00\x00RESDLL\x00UniresDLL\x00PaperSize\x00LETTER\x00Orientation\x00PORTRAIT\x00Resolution\x00DPI600\x00ColorMode\x0024bpp\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'Win32_DevNamesW': b'\x04\x00*\x00?\x00\x00\x00S\x00e\x00n\x00d\x00 \x00T\x00o\x00 \x00M\x00i\x00c\x00r\x00o\x00s\x00o\x00f\x00t\x00 \x00O\x00n\x00e\x00N\x00o\x00t\x00e\x00 \x002\x000\x001\x000\x00 \x00D\x00r\x00i\x00v\x00e\x00r\x00\x00\x00S\x00e\x00n\x00d\x00 \x00T\x00o\x00 \x00O\x00n\x00e\x00N\x00o\x00t\x00e\x00 \x002\x000\x001\x000\x00\x00\x00n\x00u\x00l\x00:\x00\x00\x00'}
SentinelList {}
Thumbnails {'0': {'ImageIndex': 0, 'SourceSize_Pixels': [1427, 1427]}}
WindowPosition [30, 801, 1457, 2228]
None

Any python object will provide a help.

help(current_dataset)
Fetching long content....

All attributes of a python object can be viewed with the * dir* command.

As above: too much information for normal use, but it is there if needed.

dir(current_dataset)
['A', 'T', '_Array__chunks', '_Array__name', '_Dataset__protected', '_Dataset__rearrange_axes', '_Dataset__reduce_dimensions', '_Dataset__validate_dim', '__abs__', '__add__', '__and__', '__array__', '__array_function__', '__array_priority__', '__array_ufunc__', '__await__', '__bool__', '__class__', '__complex__', '__dask_graph__', '__dask_keys__', '__dask_layers__', '__dask_optimize__', '__dask_postcompute__', '__dask_postpersist__', '__dask_scheduler__', '__dask_tokenize__', '__deepcopy__', '__delattr__', '__dict__', '__dir__', '__div__', '__divmod__', '__doc__', '__eq__', '__float__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getstate__', '__gt__', '__hash__', '__index__', '__init__', '__init_subclass__', '__int__', '__invert__', '__iter__', '__le__', '__len__', '__long__', '__lshift__', '__lt__', '__matmul__', '__mod__', '__module__', '__mul__', '__ne__', '__neg__', '__new__', '__nonzero__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdiv__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmatmul__', '__rmod__', '__rmul__', '__ror__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__setitem__', '__sizeof__', '__slots__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__weakref__', '__xor__', '_axes', '_cached_keys', '_chunks', '_data_type', '_elemwise', '_griddata_transform', '_h5_dataset', '_key_array', '_meta', '_metadata', '_min_dist', '_modality', '_name', '_original_metadata', '_quantity', '_rebuild', '_repr_html_', '_reset_cache', '_scalarfunc', '_source', '_structures', '_title', '_units', '_vindex', 'abs', 'add_structure', 'adjust_axis', 'all', 'angle', 'any', 'argmax', 'argmin', 'argtopk', 'astype', 'blocks', 'choose', 'chunks', 'chunksize', 'clip', 'compute', 'compute_chunk_sizes', 'conj', 'copy', 'cumprod', 'cumsum', 'dask', 'data_descriptor', 'data_type', 'del_dimension', 'dim_0', 'dim_1', 'dot', 'dtype', 'fft', 'filename', 'flatten', 'flatten_complex', 'fold', 'from_array', 'get_dimension_by_number', 'get_dimensions_by_type', 'get_extent', 'get_image_dims', 'get_spectrum_dims', 'h5_dataset', 'hdf_close', 'imag', 'itemsize', 'labels', 'like_data', 'map_blocks', 'map_overlap', 'max', 'mean', 'metadata', 'min', 'modality', 'moment', 'name', 'nbytes', 'ndim', 'nonzero', 'npartitions', 'numblocks', 'original_metadata', 'partitions', 'persist', 'plot', 'prod', 'quantity', 'ravel', 'real', 'rechunk', 'reduce_dims', 'rename_dimension', 'repeat', 'reshape', 'round', 'set_dimension', 'set_thumbnail', 'shape', 'size', 'source', 'squeeze', 'std', 'store', 'structures', 'sum', 'swapaxes', 'title', 'to_backend', 'to_dask_dataframe', 'to_delayed', 'to_hdf5', 'to_svg', 'to_tiledb', 'to_zarr', 'topk', 'trace', 'transpose', 'unfold', 'units', 'var', 'view', 'view_metadata', 'view_original_metadata', 'vindex', 'visualize', 'x', 'y']

Adding Data

To add another dataset that belongs to this measurement we will use the h5_add_channel from file_tools in the pyTEMlib package.

Here is how we add a channel there.

We can also add a new measurement group (add_measurement in pyTEMlib) for similar datasets.

This is equivalent to making a new directory in a file structure on your computer.

datasets['Copied_of_Channel_000'] = current_dataset.copy()

We use above functions to add the content of a (random) data-file to the current file.

This is important if you for example want to add a Z-contrast or survey-image to a spectrum image.

Therefore, these functions enable you to collect the data from different files that belong together.

datasets.keys()
dict_keys(['Channel_000', 'Copied_of_Channel_000'])

Adding additional information

Similarly, we can add a whole new measurement group or a structure group.

This function will be contained in the KinsCat package of pyTEMlib.

If you loaded the example image, with graphite and ZnO both are viewed in the [1,1,1] zone axis.

import pyTEMlib.kinematic_scattering as ks         # kinematic scattering Library
                             # with Atomic form factors from Kirkland's book
import ase

                                                                                 
graphite = ks.structure_by_name('Graphite')
print(graphite)
Using kinematic_scattering library version {_version_ }  by G.Duscher
Atoms(symbols='C4', pbc=False, cell=[[2.46772414, 0.0, 0.0], [-1.2338620699999996, 2.1371117947721068, 0.0], [0.0, 0.0, 6.711]])

current_dataset.structures['Crystal_000'] = graphite
                                                            
zinc_oxide = ks.structure_by_name('ZnO')
current_dataset.structures['ZnO'] =zinc_oxide

Keeping Track of Analysis and Results

A notebook is notorious for getting confusing, especially if one uses different notebooks for different task, but store them in the same file.

If you like a result of your calculation, log it.

Use the datasets dictionary to add a analysed and/or modified dataset. Make sure the metadata contain all the necessary information, so that you will know later what you did.

The convention in this class will be to call the dataset Log_000.

new_dataset = current_dataset.T
new_dataset.metadata = {'analysis': 'Nothing', 'name': 'Nothing'}
datasets['Log_000'] = new_dataset

An example for a log

We log the Fourier Transform of the image we loaded

First we perform the calculation

fft_image = current_dataset.fft().abs()
fft_image = np.log(60+fft_image)

view = fft_image.plot()
Loading...

Now that we like this we log it.

Please note that just saving the fourier transform would not be good as we also need the scale and such.

fft_image.title = 'FFT Gamma corrected'
fft_image.metadata = {'analysis': 'fft'}
datasets['Log_001'] = fft_image

view = fft_image.plot()
Loading...

We added quite a few datasets to our dictionary.

Let’s have a look

chooser = ft.ChooseDataset(datasets)
Loading...
view = chooser.dataset.plot()
Loading...

Save Datasets to hf5_file

Write all datasets to one h5_file, which we then close immediatedly

h5_group = ft.save_dataset(datasets, filename='./nix.hf5')

Cannot overwrite file. Using:  nix-1.hf5

C:\Users\gduscher\AppData\Local\anaconda3\envs\pyTEMlib\Lib\site-packages\pyNSID\io\hdf_io.py:111: UserWarning: main_data_name should not contain the "-" character. Reformatted name from:p1-3-hr3 to p1_3_hr3
  warn('main_data_name should not contain the "-" character. Reformatted'
C:\Users\gduscher\AppData\Local\anaconda3\envs\pyTEMlib\Lib\site-packages\pyNSID\io\hdf_utils.py:376: FutureWarning: validate_h5_dimension may be removed in a future version
  warn('validate_h5_dimension may be removed in a future version',
C:\Users\gduscher\AppData\Local\anaconda3\envs\pyTEMlib\Lib\site-packages\pyNSID\io\hdf_io.py:111: UserWarning: main_data_name should not contain the "-" character. Reformatted name from:p1-3-hr3 to p1_3_hr3
  warn('main_data_name should not contain the "-" character. Reformatted'
C:\Users\gduscher\AppData\Local\anaconda3\envs\pyTEMlib\Lib\site-packages\pyNSID\io\hdf_utils.py:376: FutureWarning: validate_h5_dimension may be removed in a future version
  warn('validate_h5_dimension may be removed in a future version',
C:\Users\gduscher\AppData\Local\anaconda3\envs\pyTEMlib\Lib\site-packages\pyNSID\io\hdf_io.py:111: UserWarning: main_data_name should not contain the "-" character. Reformatted name from:Transposed_p1-3-hr3 to Transposed_p1_3_hr3
  warn('main_data_name should not contain the "-" character. Reformatted'
C:\Users\gduscher\AppData\Local\anaconda3\envs\pyTEMlib\Lib\site-packages\pyNSID\io\hdf_utils.py:376: FutureWarning: validate_h5_dimension may be removed in a future version
  warn('validate_h5_dimension may be removed in a future version',
C:\Users\gduscher\AppData\Local\anaconda3\envs\pyTEMlib\Lib\site-packages\pyNSID\io\hdf_utils.py:376: FutureWarning: validate_h5_dimension may be removed in a future version
  warn('validate_h5_dimension may be removed in a future version',

Close the file

h5_group.file.close()

Open h5_file

Open the h5_file that we just created

datasets2= ft.open_file(filename='./nix.hf5')

chooser = ft.ChooseDataset(datasets2)
Loading...

Short check if we got the data right

we print the tree and we plot the data

view = chooser.dataset.plot()
Loading...
Foreword (ReadME)
Matplotlib and Numpy for Micrographs
Foreword (ReadME)
Course Organization