Data formats

In-memory data

Basic Concept

The main format we’re using within plottr is the DataDict. While most of the actual numeric data will typically live in numpy arrays (or lists, or similar), they don’t typically capture easily arbitrary metadata and relationships between arrays. Say, for example, we have some data z that depends on two other variables, x and y. This information has be stored somewhere, and numpy doesn’t offer readily a solution here. There are various extensions, for example xarray or the MetaArray class. Those however typically have a grid format in mind, which we do not want to impose. Instead, we use a wrapper around the python dictionary that contains all the required meta information to infer the relevant relationships, and that uses numpy arrays internally to store the numeric data. Additionally we can store any other arbitrary meta data.

A DataDict container (a dataset) can contain multiple data fields (or variables), that have values and can contain their own meta information. Importantly, we distinct between independent fields (the axes) and dependent fields (the data).

Despite the naming, axes is not meant to imply that the data have to have a certain shape (but the degree to which this is true depends on the class used). A list of classes for different shapes of data can be found below.

The basic structure of data conceptually looks like this (we inherit from dict):

{
    'data_1' : {
        'axes' : ['ax1', 'ax2'],
        'unit' : 'some unit',
        'values' : [ ... ],
        '__meta__' : 'This is very important data',
        ...
    },
    'ax1' : {
        'axes' : [],
        'unit' : 'some other unit',
        'values' : [ ... ],
        ...,
    },
    'ax2' : {
        'axes' : [],
        'unit' : 'a third unit',
        'values' : [ ... ],
        ...,
    },
    '__globalmeta__' : 'some information about this data set',
    '__moremeta__' : 1234,
    ...
}

In this case we have one dependent variable, data_1, that depends on two axes, ax1 and ax2. This concept is restricted only in the following way:

• A dependent can depend on any number of independents.

• An independent cannot depend on other fields itself.

• Any field that does not depend on another, is treated as an axis.

Note that meta information is contained in entries whose keys start and end with double underscores. Both the DataDict itself, as well as each field can contain meta information.

In the most basic implementation, the only restriction on the data values is that they need to be contained in a sequence (typically as list, or numpy array), and that the length of all values in the data set (the number of records) must be equal. Note that this does not preclude nested sequences!

Relevant data classes

DataDictBase

The main base class. Only checks for correct dependencies. Any requirements on data structure is left to the inheriting classes. The class contains methods for easy access to data and metadata.

DataDict

The only requirement for valid data is that the number of records is the same for all data fields. Contains some tools for expansion of data.

MeshgridDataDict

For data that lives on a grid (not necessarily regular).

DataDict

Note

Because DataDicts are python dictionaries , we highly recommend becoming familiar with them before utilizing DataDicts.

Basic Use

We can start by creating an empty DataDict like any other python object:

>>> data_dict = DataDict()
>>> data_dict
{}

We can create the structure of the data_dict by creating dictionary items and populating them like a normal python dictionary:

>>> data_dict['x'] = dict(unit='m')
>>> data_dict
{'x': {'unit': 'm'}}

We can also start by creating a DataDict that has the structure of the data we are going to record:

>>> data_dict = DataDict(x=dict(unit='m'), y = dict(unit='m'), z = dict(axes=['x', 'y']))
>>> data_dict
{'x': {'unit': 'm'}, 'y': {'unit': 'm'}, 'z': {'axes': ['x', 'y']}}

The DataDict that we just created contains no data yet, only the structure and relationship of the data fields. We have also specified the unit of x and y and which variables are independent variables (x, y), or how we will call them from now on, axes and dependent variables (z), or, dependents.

Structure

From the basic and empty DataDict we can already start to inspect its structure. To see the entire structure of a DataDict we can use the structure method:

>>> data_dict = DataDict(x=dict(unit='m'), y = dict(unit='m'), z = dict(axes=['x', 'y']))
>>> data_dict.structure()
{'x': {'unit': 'm', 'axes': [], 'label': ''},
 'y': {'unit': 'm', 'axes': [], 'label': ''},
 'z': {'axes': ['x', 'y'], 'unit': '', 'label': ''}}

We can check for specific things inside the DataDict. We can look at the axes:

>>> data_dict.axes()
['x', 'y']

We can look at all the dependents:

>>> data_dict.dependents()
['z']

We can also see the shape of a DataDict by using the shapes method:

>>> data_dict.shapes()
{'x': (0,), 'y': (0,), 'z': (0,)}

Populating the DataDict

One of the only “restrictions” that DataDict implements is that every data field must have the same number of records (items). However, restrictions is in quotes because there is nothing that is stopping you from having different data fields have different number of records, this will only make the DataDict invalid. We will explore what his means later.

There are 2 different ways of safely populating a DataDict, adding data to it or appending 2 different DataDict to each other.

Note

You can always manually update the item values any data field like any other item of a python dictionary, however, populating the DataDict this way can result in an invalid DataDict if you are not being careful. Both population methods presented below contains checks to make sure that the new data being added will not create an invalid DataDict.

We can add data to an existing DataDict with the add_data method:

>>> data_dict = DataDict(x=dict(unit='m'), y = dict(unit='m'), z = dict(axes=['x', 'y']))
>>> data_dict.add_data(x=[0,1,2], y=[0,1,2], z=[0,1,4])
>>> data_dict
{'x': {'unit': 'm', 'axes': [], 'label': '', 'values': array([0, 1, 2])},
 'y': {'unit': 'm', 'axes': [], 'label': '', 'values': array([0, 1, 2])},
 'z': {'axes': ['x', 'y'],  'unit': '',  'label': '',  'values': array([0, 1, 4])}}

We now have a populated DataDict. It is important to notice that this method will also add any of the missing special keys that a data field doesn’t have (values, axes, unit, and label). Populating the DataDict with this method will also ensure that every item has the same number of records and the correct shape, either by adding nan to the other data fields or by nesting the data arrays so that the outer most dimension of every data field has the same number of records.

We can see this in action if we add a single record to a data field with items but no the rest:

>>> data_dict.add_data(x=[9])
>>> data_dict
{'x': {'unit': 'm', 'axes': [], 'label': '', 'values': array([0, 1, 2, 9])},
 'y': {'unit': 'm', 'axes': [], 'label': '', 'values': array([ 0.,  1.,  2., nan])},
 'z': {'axes': ['x', 'y'], 'unit': '', 'label': '', 'values': array([ 0.,  1.,  4., nan])}}

As we can see, both y and z have an extra nan record in them. We can observe the change of dimension if we do not add the same number of records to all data fields:

>>> data_dict = DataDict(x=dict(unit='m'), y = dict(unit='m'), z = dict(axes=['x', 'y']))
>>> data_dict.add_data(x=[0,1,2], y=[0,1,2],z=[0])
>>> data_dict
{'x': {'unit': 'm', 'axes': [], 'label': '', 'values': array([[0, 1, 2]])},
 'y': {'unit': 'm', 'axes': [], 'label': '', 'values': array([[0, 1, 2]])},
 'z': {'axes': ['x', 'y'], 'unit': '', 'label': '', 'values': array([0])}}

If we want to expand our DataDict by appending another one, we need to make sure that both of our DataDicts have the same inner structure. We can check that by utilizing the static method same_structure:

>>> data_dict_1 = DataDict(x=dict(unit='m'), y=dict(unit='m'), z=dict(axes=['x','y']))
>>> data_dict_2 = DataDict(x=dict(unit='m'), y=dict(unit='m'), z=dict(axes=['x','y']))
>>> data_dict_1.add_data(x=[0,1,2], y=[0,1,2], z=[0,1,4])
>>> data_dict_2.add_data(x=[3,4], y=[3,4], z=[9,16])
>>> DataDict.same_structure(data_dict_1, data_dict_2)
True

Note

Make sure that both DataDicts have the exact same structure. This means that every item of every data field that appears when using the method same_structure (unit, axes, and label) are identical to one another, except for values. Any slight difference will make this method fail due to conflicting structures.

The append method will do this check before appending the 2 DataDict, and will only append them if the check returns True. Once we know that the structure is the same we can append them:

>>> data_dict_1.append(data_dict_2)
>>> data_dict_1
{'x': {'unit': 'm', 'axes': [], 'label': '', 'values': array([0, 1, 2, 3, 4])},
 'y': {'unit': 'm', 'axes': [], 'label': '', 'values': array([0, 1, 2, 3, 4])},
 'z': {'axes': ['x', 'y'], 'unit': '', 'label': '', 'values': array([ 0,  1,  4,  9, 16])}}

Meta Data

One of the advantages DataDicts have over regular python dictionaries is their ability to contain meta data. Meta data can be added to the entire DataDict or to individual data fields. Any object inside a DataDict whose key starts and ends with 2 underscores is considered meta data.

We can simply add meta data manually by adding an item with the proper notation:

>>> data_dict['__metadata__'] = 'important meta data'

Or we can use the add_meta method:

>>> data_dict.add_meta('sample_temperature', '10mK')
>>> data_dict
{'x': {'unit': 'm', 'axes': [], 'label': '', 'values': array([0, 1, 2])},
 'y': {'unit': 'm', 'axes': [], 'label': '', 'values': array([0, 1, 2])},
 'z': {'axes': ['x', 'y'], 'unit': '', 'label': '', 'values': array([0, 1, 4])},
 '__metadata__': 'important meta data',
 '__sample_temperature__': '10mK'}

We can also add meta data to a specific data field by passing its name as the last argument:

>>> data_dict.add_meta('extra_metadata', 'important meta data', 'x')
>>> data_dict
{'x': {'unit': 'm', 'axes': [], 'label': '', 'values': array([0, 1, 2]), '__extra_metadata__': 'important meta data'},
 'y': {'unit': 'm', 'axes': [], 'label': '', 'values': array([0, 1, 2])},
 'z': {'axes': ['x', 'y'], 'unit': '', 'label': '', 'values': array([0, 1, 4])},
 '__metadata__': 'important meta data',
 '__sample_temperature__': '10mK'}

We can check if a certain meta field exists with the method has_meta:

>>> data_dict.has_meta('sample_temperature')
True

We can retrieve the meta data with the meta_val method:

>>> data_dict.meta_val('sample_temperature')
'10mK'

We can also ask for a meta value from a specific data field by passing the data field as the second argument:

>>> data_dict.meta_val('extra_metadata','x')
'important meta data'

We can delete a specific meta field by using the delete_meta method:

>>> data_dict.delete_meta('metadata')
>>> data_dict.has_meta('metadata')
False

This also work for meta data in data fields by passing the data field as the last argument:

>>> data_dict.delete_meta('extra_metadata', 'x')
>>> data_dict['x']
{'unit': 'm', 'axes': [], 'label': '', 'values': array([0, 1, 2])}

We can delete all the meta data present in the DataDict with the clear_meta method:

>>> data_dict.add_meta('metadata', 'important meta data')
>>> data_dict.add_meta('extra_metadata', 'important meta data', 'x')
>>> data_dict.clear_meta()
>>> data_dict
{'x': {'unit': 'm', 'axes': [], 'label': '', 'values': array([0, 1, 2])},
 'y': {'unit': 'm', 'axes': [], 'label': '', 'values': array([0, 1, 2])},
 'z': {'axes': ['x', 'y'], 'unit': '', 'label': '', 'values': array([0, 1, 4])}}

Note

There are 3 helper functions in the datadict module that help converting from meta data name to key. These are: is_meta_key(), meta_key_to_name() , and meta_name_to_key().

Meshgrid DataDict

A dataset where the axes form a grid on which the dependent values reside.

This is a more special case than DataDict, but a very common scenario. To support flexible grids, this class requires that all axes specify values for each datapoint, rather than a single row/column/dimension.

For example, if we want to specify a 3-dimensional grid with axes x, y, z, the values of x, y, z all need to be 3-dimensional arrays; the same goes for all dependents that live on that grid. Then, say, x[i,j,k] is the x-coordinate of point i,j,k of the grid.

This implies that a MeshgridDataDict can only have a single shape, i.e., all data values share the exact same nesting structure.

For grids where the axes do not depend on each other, the correct values for the axes can be obtained from np.meshgrid (hence the name of the class).

Example: a simple uniform 3x2 grid might look like this; x and y are the coordinates of the grid, and z is a function of the two:

x = [[0, 0],
     [1, 1],
     [2, 2]]

y = [[0, 1],
     [0, 1],
     [0, 1]]

z = x * y =
    [[0, 0],
     [0, 1],
     [0, 2]]

Note

Internally we will typically assume that the nested axes are ordered from slow to fast, i.e., dimension 1 is the most outer axis, and dimension N of an N-dimensional array the most inner (i.e., the fastest changing one). This guarantees, for example, that the default implementation of np.reshape has the expected outcome. If, for some reason, the specified axes are not in that order (e.g., we might have z with axes = ['x', 'y'], but x is the fast axis in the data). In such a case, the guideline is that at creation of the meshgrid, the data should be transposed such that it conforms correctly to the order as given in the axis = [...] specification of the data. The function datadict_to_meshgrid provides options for that.

This implementation of DataDictBase consists only of 3 extra methods:

• MeshgridDataDict.shape

• MeshgridDataDict.validate

• MeshgridDataDict.reorder_axis

So the only way of populating it is by manually modifying the values object of each data field since the tools for populating the DataDict are specific to the DataDict implementation.

DataDict Storage

The datadict_storage.py module offers tools to help with saving DataDicts into disk by storing them in DDH5 files (HDF5 files that contains DataDicts inside).

Description of the HDF5 storage format

We use a simple mapping from DataDict to the HDF5 file. Within the file, a single DataDict is stored in a (top-level) group of the file. The data fields are datasets within that group.

Global meta data of the DataDict are attributes of the group; field meta data are attributes of the dataset (incl., the unit and axes values). The meta data keys are given exactly like in the DataDict, i.e., includes the double underscore pre- and suffix.

For more specific information on how HDF5 works please read the following documentation

Working with DDH5 files

When we are working with data, the first thing we usually want to do is to save it in disk. We can directly save an already existing DataDict into disk by calling the function datadict_to_hdf5.

>>> data_dict = DataDict(x=dict(values=np.array([0,1,2]), axes=[], __unit__='cm'), y=dict(values=np.array([3,4,5]), axes=['x']))
>>> data_dict
{'x': {'values': array([0, 1, 2]), 'axes': [], '__unit__': 'cm'},
 'y': {'values': array([3, 4, 5]), 'axes': ['x']}}
>>> datadict_to_hdf5(data_dict, 'folder\data.ddh5')

datadict_to_hdf5 will save data_dict in a file named ‘data.ddh5’ in whatever directory is passed to it, creating new folders if they don’t already exists. The file will contain all of the data fields as well as all the metadata, with some more metadata generated to specify when the DataDict was created.

Note

Meta data is only written during initial writing of the dataset. If we’re appending to existing datasets, we’re not setting meta data anymore.

Warning

For this method to properly work the objects that are being saved in the values key of a data field must by a numpy array, or numpy array like.

Data saved on disk is useless however if we do not have a way of accessing it. To do this we use the datadict_from_hdf5:

>>> loaded_data_dict = datadict_from_hdf5('folder\data.ddh5')
>>> loaded_data_dict
{'__creation_time_sec__': 1651159636.0,
 '__creation_time_str__': '2022-04-28 10:27:16',
 'x': {'values': array([0, 1, 2]),
  'axes': [],
  '__shape__': (3,),
  '__creation_time_sec__': 1651159636.0,
  '__creation_time_str__': '2022-04-28 10:27:16',
  '__unit__': 'cm',
  'unit': '',
  'label': ''},
 'y': {'values': array([3, 4, 5]),
  'axes': ['x'],
  '__shape__': (3,),
  '__creation_time_sec__': 1651159636.0,
  '__creation_time_str__': '2022-04-28 10:27:16',
  'unit': '',
  'label': ''}}

We can see that the DataDict is the same one we saved earlier with the added metadata that indicates the time it was created.

By default both datadict_to_hdf5 and datadict_from_hdf5 save and load the datadict in the ‘data’ group of the DDH5. Both of these can by changed by passing another group to the argument ‘groupname’. We can see this if we manually create a second group and save a new DataDict there:

>>> data_dict2 = DataDict(a=dict(values=np.array([0,1,2]), axes=[], __unit__='cm'), b=dict(values=np.array([3,4,5]), axes=['a']))
>>> with h5py.File('folder\data.ddh5', 'a') as file:
>>>    file.create_group('other_data')
>>> datadict_to_hdf5(data_dict2, 'folder\data.ddh5', groupname='other_data')

If we then load the DDH5 file like before we only see the first DataDict:

>>> loaded_data_dict = datadict_from_hdf5('folder\data.ddh5', 'data')
>>> loaded_data_dict
{'__creation_time_sec__': 1651159636.0,
 '__creation_time_str__': '2022-04-28 10:27:16',
 'x': {'values': array([0, 1, 2]),
  'axes': [],
  '__shape__': (3,),
  '__creation_time_sec__': 1651159636.0,
  '__creation_time_str__': '2022-04-28 10:27:16',
  '__unit__': 'cm',
  'unit': '',
  'label': ''},
 'y': {'values': array([3, 4, 5]),
  'axes': ['x'],
  '__shape__': (3,),
  '__creation_time_sec__': 1651159636.0,
  '__creation_time_str__': '2022-04-28 10:27:16',
  'unit': '',
  'label': ''}}

To see the other DataDict we can specify the group in the argument ‘groupname’:

>>> loaded_data_dict = datadict_from_hdf5('folder\data.ddh5', 'other_data')
>>> loaded_data_dict
{'a': {'values': array([0, 1, 2]),
  'axes': [],
  '__shape__': (3,),
  '__creation_time_sec__': 1651159636.0,
  '__creation_time_str__': '2022-04-28 10:27:16',
  '__unit__': 'cm',
  'unit': '',
  'label': ''},
 'b': {'values': array([3, 4, 5]),
  'axes': ['a'],
  '__shape__': (3,),
  '__creation_time_sec__': 1651159636.0,
  '__creation_time_str__': '2022-04-28 10:27:16',
  'unit': '',
  'label': ''}}

We can also use all_datadicts_from_hdf5 to get a dictionary with all DataDicts in every group inside:

>>> all_datadicts = all_datadicts_from_hdf5('folder\data.ddh5')
>>> all_datadicts
{'data': {'__creation_time_sec__': 1651159636.0,
  '__creation_time_str__': '2022-04-28 10:27:16',
  'x': {'values': array([0, 1, 2]),
   'axes': [],
   '__shape__': (3,),
   '__creation_time_sec__': 1651159636.0,
   '__creation_time_str__': '2022-04-28 10:27:16',
   '__unit__': 'cm',
   'unit': '',
   'label': ''},
  'y': {'values': array([3, 4, 5]),
   'axes': ['x'],
   '__shape__': (3,),
   '__creation_time_sec__': 1651159636.0,
   '__creation_time_str__': '2022-04-28 10:27:16',
   'unit': '',
   'label': ''}},
 'other_data': {'a': {'values': array([0, 1, 2]),
   'axes': [],
   '__shape__': (3,),
   '__creation_time_sec__': 1651159636.0,
   '__creation_time_str__': '2022-04-28 10:27:16',
   '__unit__': 'cm',
   'unit': '',
   'label': ''},
  'b': {'values': array([3, 4, 5]),
   'axes': ['a'],
   '__shape__': (3,),
   '__creation_time_sec__': 1651159636.0,
   '__creation_time_str__': '2022-04-28 10:27:16',
   'unit': '',
   'label': ''}}}

DDH5 Writer

Most times we want to be saving data to disk as soon as it is generated by an experiment (or iteration), instead of waiting to have a complete DataDict. To do this, Datadict_storage also offers a context manager with which we can safely save our incoming data.

To use it we first need to create an empty DataDict that contains the structure of how the data is going to look like:

>>> data_dict = DataDict(
>>> x = dict(unit='x_unit'),
>>> y = dict(unit='y_unit', axes=['x']))

With our created DataDict, we can start the DDH5Writer context manager and add data to our DataDict utilizing the add_data

>>> with DDH5Writer(datadict=data_dict, basedir='./data/', name='Test') as writer:
>>>    for x in range(10):
>>>        writer.add_data(x=x, y=x**2)
Data location:  data\2022-04-27\2022-04-27T145308_a986867c-Test\data.ddh5

The writer created the folder ‘data’ (because it did not exist before) and inside that folder, created another new folder for the current day and another new folder inside of it day folder for the the DataDict that we saved with the naming structure of YYYY-mm-dd_THHMMSS_<ID>-<name>/<filename>.ddh5, where name is the name parameter passed to the writer. The writer creates this structure such that when we run the writer again with new data, it will create another folder following the naming structure inside the current date folder. This way each new DataDict will be saved in the date it was generated with a time stamp in the name of the folder containing it.

Changing File Extension and Time Format

Finally, datadict_storage contains 2 module variables, ‘DATAFILEXT’ and ‘TIMESTRFORMAT’.

‘DATAFILEXT’ by default is ‘ddh5’, and it is used to specify the extension file of all of the module saving functions. Change this variable if you want your HDF5 to have a different extension by default, instead of passing it everytime.

‘TIMESTRFORMAT’ specifies how the time is formated in the new metadata created when saving a DataDict. The default is: "%Y-%m-%d %H:%M:%S", and it follows the structure of strftime.

Reference

DataDict

DataDictBase

The following is the base class from which both DataDict and MeshgridDataDict are inheriting.

class plottr.data.datadict.DataDictBase(**kw: Any)

Simple data storage class that is based on a regular dictionary.

This base class does not make assumptions about the structure of the values. This is implemented in inheriting classes.

static to_records(**data: Any) → Dict[str, ndarray]

Convert data to records that can be added to the DataDict. All data is converted to np.array, and reshaped such that the first dimension of all resulting arrays have the same length (chosen to be the smallest possible number that does not alter any shapes beyond adding a length-1 dimension as first dimension, if necessary).

If a data field is given as None, it will be converted to numpy.array([numpy.nan]).

Parameters

data – keyword arguments for each data field followed by data.

Returns

Dictionary with properly shaped data.

data_items() → Iterator[Tuple[str, Dict[str, Any]]]

Generator for data field items.

Like dict.items(), but ignores meta data.

Returns

Generator yielding first the key of the data field and second its value.

meta_items(data: Optional[str] = None, clean_keys: bool = True) → Iterator[Tuple[str, Dict[str, Any]]]

Generator for meta items.

Like dict.items(), but yields only meta entries. The keys returned do not contain the underscores used internally.

Parameters

• data – If None iterate over global meta data. If it’s the name of a data field, iterate over the meta information of that field.

• clean_keys – If True, remove the underscore pre/suffix.

Returns

Generator yielding first the key of the data field and second its value.

data_vals(key: str) → ndarray

Return the data values of field key.

Equivalent to DataDict['key'].values.

Parameters

key – Name of the data field.

Returns

Values of the data field.

has_meta(key: str) → bool

Check whether meta field exists in the dataset.

Returns

True if it exists, False if it doesn’t.

meta_val(key: str, data: Optional[str] = None) → Any

Return the value of meta field key (given without underscore).

Parameters

• key – Name of the meta field.

• data – None for global meta; name of data field for data meta.

Returns

The value of the meta information.

add_meta(key: str, value: Any, data: Optional[str] = None) → None

Add meta info to the dataset.

If the key already exists, meta info will be overwritten.

Parameters

• key – Name of the meta field (without underscores).

• value – Value of the meta information.

• data – If None, meta will be global; otherwise assigned to data field data.

delete_meta(key: str, data: Optional[str] = None) → None

Deletes specific meta data.

Parameters

• key – Name of the meta field to remove.

• data – If None, this affects global meta; otherwise remove from data field data.

clear_meta(data: Optional[str] = None) → None

Deletes all meta data.

Parameters

data – If not None, delete all meta only from specified data field data. Else, deletes all top-level meta, as well as meta for all data fields.

extract(data: List[str], include_meta: bool = True, copy: bool = True, sanitize: bool = True) → T

Extract data from a dataset.

Return a new datadict with all fields specified in data included. Will also take any axes fields along that have not been explicitly specified. Will return empty if data consists of only axes fields.

Parameters

• data – Data field or list of data fields to be extracted.

• include_meta – If True, include the global meta data. data meta will always be included.

• copy – If True, data fields will be deep copies of the original.

• sanitize – If True, will run DataDictBase.sanitize before returning.

Returns

New DataDictBase containing only requested fields.

static same_structure(*data: T, check_shape: bool = False) → bool

Check if all supplied DataDicts share the same data structure (i.e., dependents and axes).

Ignores meta data and values. Checks also for matching shapes if check_shape is True.

Parameters

• data – The data sets to compare.

• check_shape – Whether to include shape check in the comparison.

Returns

True if the structure matches for all, else False.

structure(add_shape: bool = False, include_meta: bool = True, same_type: bool = False) → Optional[T]

Get the structure of the DataDict.

Return the datadict without values (value omitted in the dict).

Parameters

• add_shape – Deprecated – ignored.

• include_meta – If True, include the meta information in the returned dict.

• same_type – If True, return type will be the one of the object this is called on. Else, DataDictBase.

Returns

The DataDict containing the structure only. The exact type is the same as the type of self.

label(name: str) → Optional[str]

Get the label for a data field. If no label is present returns the name of the data field as the label. If a unit is present, it will be appended at the end in brackets: “label (unit)”.

Parameters

name – Name of the data field.

Returns

Labelled name.

axes_are_compatible() → bool

Check if all dependent data fields have the same axes.

This includes axes order.

Returns

True or False.

axes(data: Optional[Union[Sequence[str], str]] = None) → List[str]

Return a list of axes.

Parameters

data – if None, return all axes present in the dataset, otherwise only the axes of the dependent data.

Returns

The list of axes.

dependents() → List[str]

Get all dependents in the dataset.

Returns

A list of the names of dependents.

shapes() → Dict[str, Tuple[int, ...]]

Get the shapes of all data fields.

Returns

A dictionary of the form {key : shape}, where shape is the np.shape-tuple of the data with name key.

validate() → bool

Check the validity of the dataset.

Checks performed:

• All axes specified with dependents must exist as data fields.

Other tasks performed:

• unit keys are created if omitted.

• label keys are created if omitted.

• shape meta information is updated with the correct values (only if present already).

Returns

True if valid, False if invalid.

Raises

ValueError if invalid.

remove_unused_axes() → T

Removes axes not associated with dependents.

Returns

Cleaned dataset.

sanitize() → T

Clean-up tasks:

• Removes unused axes.

Returns

Sanitized dataset.

reorder_axes_indices(name: str, **pos: int) → Tuple[Tuple[int, ...], List[str]]

Get the indices that can reorder axes in a given way.

Parameters

• name – Name of the data field of which we want to reorder axes.

• pos – New axes position in the form axis_name = new_position. Non-specified axes positions are adjusted automatically.

Returns

The tuple of new indices, and the list of axes names in the new order.

reorder_axes(data_names: Optional[Union[Sequence[str], str]] = None, **pos: int) → T

Reorder data axes.

Parameters

• data_names – Data name(s) for which to reorder the axes. If None, apply to all dependents.

• pos – New axes position in the form axis_name = new_position. Non-specified axes positions are adjusted automatically.

Returns

Dataset with re-ordered axes.

copy() → T

Make a copy of the dataset.

Returns

A copy of the dataset.

astype(dtype: dtype) → T

Convert all data values to given dtype.

Parameters

dtype – np dtype.

Returns

Copy of the dataset, with values as given type.

mask_invalid() → T

Mask all invalid data in all values. :return: Copy of the dataset with invalid entries (nan/None) masked.

DataDict

class plottr.data.datadict.DataDict(**kw: Any)

The most basic implementation of the DataDict class.

It only enforces that the number of records per data field must be equal for all fields. This refers to the most outer dimension in case of nested arrays.

The class further implements simple appending of datadicts through the DataDict.append method, as well as allowing addition of DataDict instances.

append(newdata: DataDict) → None

Append a datadict to this one by appending data values.

Parameters

newdata – DataDict to append.

Raises

ValueError, if the structures are incompatible.

add_data(**kw: Any) → None

Add data to all values. new data must be valid in itself.

This method is useful to easily add data without needing to specify meta data or dependencies, etc.

Parameters

kw – one array per data field (none can be omitted).

nrecords() → Optional[int]

Gets the number of records in the dataset.

Returns

The number of records in the dataset.

is_expanded() → bool

Determine if the DataDict is expanded.

Returns

True if expanded. False if not.

is_expandable() → bool

Determine if the DataDict can be expanded.

Expansion flattens all nested data values to a 1D array. For doing so, we require that all data fields that have nested/inner dimensions (i.e, inside the records level) shape the inner shape. In other words, all data fields must be of shape (N,) or (N, (shape)), where shape is common to all that have a shape not equal to (N,).

Returns

True if expandable. False otherwise.

expand() → DataDict

Expand nested values in the data fields.

Flattens all value arrays. If nested dimensions are present, all data with non-nested dims will be repeated accordingly – each record is repeated to match the size of the nested dims.

Returns

The flattened dataset.

Raises

ValueError if data is not expandable.

validate() → bool

Check dataset validity.

Beyond the checks performed in the base class DataDictBase, check whether the number of records is the same for all data fields.

Returns

True if valid.

Raises

ValueError if invalid.

sanitize() → DataDict

Clean-up.

Beyond the tasks of the base class DataDictBase:

• remove invalid entries as far as reasonable.

Returns

sanitized DataDict.

remove_invalid_entries() → DataDict

Remove all rows that are None or np.nan in all dependents.

Returns

The cleaned DataDict.

Meshgrid DataDict

class plottr.data.datadict.MeshgridDataDict(**kw: Any)

Implementation of DataDictBase meant to be used for when the axes form a grid on which the dependent values reside.

It enforces that all dependents have the same axes and all shapes need to be identical.

shape() → Union[None, Tuple[int, ...]]

Return the shape of the meshgrid.

Returns

The shape as tuple. None if no data in the set.

validate() → bool

Validation of the dataset.

Performs the following checks: * All dependents must have the same axes. * All shapes need to be identical.

Returns

True if valid.

Raises

ValueError if invalid.

reorder_axes(data_names: Optional[Union[Sequence[str], str]] = None, **pos: int) → MeshgridDataDict

Reorder the axes for all data.

This includes transposing the data, since we’re on a grid.

Parameters

pos – New axes position in the form axis_name = new_position. non-specified axes positions are adjusted automatically.

Returns

Dataset with re-ordered axes.

Extra Module Functions

datadict.py :

Data classes we use throughout the plottr package, and tools to work on them.

plottr.data.datadict.is_meta_key(key: str) → bool

Checks if key is meta information.

Parameters

key – The key we are checking.

Returns

True if it is, False if it isn’t.

plottr.data.datadict.meta_key_to_name(key: str) → str

Converts a meta data key to just the name. E.g: for key: “__meta__” returns “meta”

Parameters

key – The key that is being converted

Returns

The name of the key.

Raises

ValueError if the key is not a meta key.

plottr.data.datadict.meta_name_to_key(name: str) → str

Converts name into a meta data key. E.g: “meta” gets converted to “__meta__”

Parameters

name – The name that is being converted.

Returns

The meta data key based on name.

plottr.data.datadict.guess_shape_from_datadict(data: DataDict) → Dict[str, Union[None, Tuple[List[str], Tuple[int, ...]]]]

Try to guess the shape of the datadict dependents from the axes values.

Parameters

data – Dataset to examine.

Returns

A dictionary with the dependents as keys, and inferred shapes as values. Value is None, if the shape could not be inferred.

plottr.data.datadict.datadict_to_meshgrid(data: DataDict, target_shape: Optional[Tuple[int, ...]] = None, inner_axis_order: Union[None, Sequence[str]] = None, use_existing_shape: bool = False) → MeshgridDataDict

Try to make a meshgrid from a dataset.

Parameters

• data – Input DataDict.

• target_shape – Target shape. If None we use guess_shape_from_datadict to infer.

• inner_axis_order –

  If axes of the datadict are not specified in the ‘C’ order (1st the slowest, last the fastest axis) then the ‘true’ inner order can be specified as a list of axes names, which has to match the specified axes in all but order. The data is then transposed to conform to the specified order.

  Note

  If this is given, then target_shape needs to be given in in the order of this inner_axis_order. The output data will keep the axis ordering specified in the axes property.

• use_existing_shape – if True, simply use the shape that the data already has. For numpy-array data, this might already be present. If False, flatten and reshape.

Raises

GriddingError (subclass of ValueError) if the data cannot be gridded.

Returns

The generated MeshgridDataDict.

plottr.data.datadict.meshgrid_to_datadict(data: MeshgridDataDict) → DataDict

Make a DataDict from a MeshgridDataDict by reshaping the data.

Parameters

data – Input MeshgridDataDict.

Returns

Flattened DataDict.

plottr.data.datadict.combine_datadicts(*dicts: DataDict) → Union[DataDictBase, DataDict]

Try to make one datadict out of multiple.

Basic rules:

• We try to maintain the input type.

• Return type is ‘downgraded’ to DataDictBase if the contents are not compatible (i.e., different numbers of records in the inputs).

Returns

Combined data.

plottr.data.datadict.datastructure_from_string(description: str) → DataDict

Construct a DataDict from a string description.

Examples:

• "data[mV](x, y)" results in a datadict with one dependent data with unit mV and two independents, x and y, that do not have units.

• "data_1[mV](x, y); data_2[mA](x); x[mV]; y[nT]" results in two dependents, one of them depening on x and y, the other only on x. Note that x and y have units. We can (but do not have to) omit them when specifying the dependencies.

• "data_1[mV](x[mV], y[nT]); data_2[mA](x[mV])". Same result as the previous example.

Rules:

We recognize descriptions of the form field1[unit1](ax1, ax2, ...); field1[unit2](...); ....

• Field names (like field1 and field2 above) have to start with a letter, and may contain word characters.

• Field descriptors consist of the name, optional unit (presence signified by square brackets), and optional dependencies (presence signified by round brackets).

• Dependencies (axes) are implicitly recognized as fields (and thus have the same naming restrictions as field names).

• Axes are separated by commas.

• Axes may have a unit when specified as dependency, but besides the name, square brackets, and commas no other characters are recognized within the round brackets that specify the dependency.

• In addition to being specified as dependency for a field, axes may be specified also as additional field without dependency, for instance to specify the unit (may simplify the string). For example, z1[x, y]; z2[x, y]; x[V]; y[V].

• Units may only consist of word characters.

• Use of unexpected characters will result in the ignoring the part that contains the symbol.

• The regular expression used to find field descriptors is: ((?<=\A)|(?<=\;))[a-zA-Z]+\w*(\[\w*\])?(\(([a-zA-Z]+\w*(\[\w*\])?\,?)*\))?

plottr.data.datadict.str2dd(description: str) → DataDict

shortcut to datastructure_from_string().

plottr.data.datadict.datasets_are_equal(a: DataDictBase, b: DataDictBase, ignore_meta: bool = False) → bool

Check whether two datasets are equal.

Compares type, structure, and content of all fields.

Parameters

• a – First dataset.

• b – Second dataset.

• ignore_meta – If True, do not verify if metadata matches.

Returns

True or False.

DataDict Storage

plottr.data.datadict_storage

Provides file-storage tools for the DataDict class.

Note

Any function in this module that interacts with a ddh5 file, will create a lock file while it is using the file. The lock file has the following format: ~<file_name>.lock. The file lock will get deleted even if the program crashes. If the process is suddenly stopped however, we cannot guarantee that the file lock will be deleted.

class plottr.data.datadict_storage.AppendMode(value)

How/Whether to append data to existing data.

new = 0

Data that is additional compared to already existing data is appended.

all = 1

All data is appended to existing data.

none = 2

Data is overwritten.

plottr.data.datadict_storage.h5ify(obj: Any) → Any

Convert an object into something that we can assign to an HDF5 attribute.

Performs the following conversions: - list/array of strings -> numpy chararray of unicode type

Parameters

obj – Input object.

Returns

Object, converted if necessary.

plottr.data.datadict_storage.deh5ify(obj: Any) → Any

Convert slightly mangled types back to more handy ones.

Parameters

obj – Input object.

Returns

Object

plottr.data.datadict_storage.set_attr(h5obj: Any, name: str, val: Any) → None

Set attribute name of object h5obj to val

Use h5ify() to convert the object, then try to set the attribute to the returned value. If that does not succeed due to a HDF5 typing restriction, set the attribute to the string representation of the value.

plottr.data.datadict_storage.add_cur_time_attr(h5obj: Any, name: str = 'creation', prefix: str = '__', suffix: str = '__') → None

Add current time information to the given HDF5 object, following the format of: <prefix><name>_time_sec<suffix>.

Parameters

• h5obj – The HDF5 object.

• name – The name of the attribute.

• prefix – Prefix of the attribute.

• suffix – Suffix of the attribute.

plottr.data.datadict_storage.datadict_to_hdf5(datadict: DataDict, path: Union[str, Path], groupname: str = 'data', append_mode: AppendMode = AppendMode.new, file_timeout: Optional[float] = None) → None

Write a DataDict to DDH5

Note: Meta data is only written during initial writing of the dataset. If we’re appending to existing datasets, we’re not setting meta data anymore.

Parameters

• datadict – Datadict to write to disk.

• path – Path of the file (extension may be omitted).

• groupname – Name of the top level group to store the data in.

• append_mode –

  • AppendMode.none : Delete and re-create group.

  • AppendMode.new : Append rows in the datadict that exceed the number of existing rows in the dataset already stored. Note: we’re not checking for content, only length!

  • AppendMode.all : Append all data in datadict to file data sets.

• file_timeout – How long the function will wait for the ddh5 file to unlock. Only relevant if you are writing to a file that already exists and some other program is trying to read it at the same time. If none uses the default value from the FileOpener.

plottr.data.datadict_storage.datadict_from_hdf5(path: Union[str, Path], groupname: str = 'data', startidx: Optional[int] = None, stopidx: Optional[int] = None, structure_only: bool = False, ignore_unequal_lengths: bool = True, file_timeout: Optional[float] = None) → DataDict

Load a DataDict from file.

Parameters

• path – Full filepath without the file extension.

• groupname – Name of hdf5 group.

• startidx – Start row.

• stopidx – End row + 1.

• structure_only – If True, don’t load the data values.

• ignore_unequal_lengths – If True, don’t fail when the rows have unequal length; will return the longest consistent DataDict possible.

• file_timeout – How long the function will wait for the ddh5 file to unlock. If none uses the default value from the FileOpener.

Returns

Validated DataDict.

plottr.data.datadict_storage.all_datadicts_from_hdf5(path: Union[str, Path], file_timeout: Optional[float] = None, **kwargs: Any) → Dict[str, Any]

Loads all the DataDicts contained on a single HDF5 file. Returns a dictionary with the group names as keys and the DataDicts as the values of that key.

Parameters

• path – The path of the HDF5 file.

• file_timeout – How long the function will wait for the ddh5 file to unlock. If none uses the default value from the FileOpener.

Returns

Dictionary with group names as key, and the DataDicts inside them as values.

class plottr.data.datadict_storage.FileOpener(path: Union[Path, str], mode: str = 'r', timeout: Optional[float] = None, test_delay: float = 0.1)

Context manager for opening files, creates its own file lock to indicate other programs that the file is being used. The lock file follows the following structure: “~<file_name>.lock”.

Parameters

• path – The file path.

• mode – The opening file mode. Only the following modes are supported: ‘r’, ‘w’, ‘w-’, ‘a’. Defaults to ‘r’.

• timeout – Time, in seconds, the context manager waits for the file to unlock. Defaults to 30.

• test_delay – Length of time in between checks. I.e. how long the FileOpener waits to see if a file got unlocked again

class plottr.data.datadict_storage.DDH5Loader(name: str)

nodeName = 'DDH5Loader'

Name of the node. used in the flowchart node library.

uiClass

alias of DDH5LoaderWidget

useUi = True

Whether or not to automatically set up a UI widget.

process(dataIn: Optional[DataDictBase] = None) → Optional[Dict[str, Any]]

Process data through this node. This method is called any time the flowchart wants the node to process data. It will be called with one keyword argument corresponding to each input terminal, and must return a dict mapping the name of each output terminal to its new value.

This method is also called with a ‘display’ keyword argument, which indicates whether the node should update its display (if it implements any) while processing this data. This is primarily used to disable expensive display operations during batch processing.

class plottr.data.datadict_storage.DDH5Writer(datadict: DataDict, basedir: Union[str, Path] = '.', groupname: str = 'data', name: Optional[str] = None, filename: str = 'data', filepath: Optional[Union[str, Path]] = None, file_timeout: Optional[float] = None)

Context manager for writing data to DDH5. Based on typical needs in taking data in an experimental physics lab.

Creates lock file when writing data.

Parameters

• basedir – The root directory in which data is stored. create_file_structure() is creating the structure inside this root and determines the file name of the data. The default structure implemented here is <root>/YYYY-MM-DD/YYYY-mm-dd_THHMMSS_<ID>-<name>/<filename>.ddh5, where <ID> is a short identifier string and <name> is the value of parameter name. To change this, re-implement data_folder() and/or create_file_structure().

• datadict – Initial data object. Must contain at least the structure of the data to be able to use add_data() to add data.

• groupname – Name of the top-level group in the file container. An existing group of that name will be deleted.

• name – Name of this dataset. Used in path/file creation and added as meta data.

• filename – Filename to use. Defaults to ‘data.ddh5’.

• file_timeout – How long the function will wait for the ddh5 file to unlock. If none uses the default value from the FileOpener.

data_folder() → Path

Return the folder, relative to the data root path, in which data will be saved.

Default format: <basedir>/YYYY-MM-DD/YYYY-mm-ddTHHMMSS_<ID>-<name>. In this implementation we use the first 8 characters of a UUID as ID.

Returns

The folder path.

data_file_path() → Path

Determine the filepath of the data file.

Returns

The filepath of the data file.

add_data(**kwargs: Any) → None

Add data to the file (and the internal DataDict).

Requires one keyword argument per data field in the DataDict, with the key being the name, and value the data to add. It is required that all added data has the same number of ‘rows’, i.e., the most outer dimension has to match for data to be inserted faithfully. If some data is scalar and others are not, then the data should be reshaped to (1, ) for the scalar data, and (1, …) for the others; in other words, an outer dimension with length 1 is added for all.