tools.helpers.data_manager package¶

Submodules¶

tools.helpers.data_manager.data_loader module¶

Functions to load CSV and Excel files to dataframes.

tools.helpers.data_manager.data_loader.read_data_file(path=None, config_dict=None, config_key=None, date_columns=None, sheet_name=0, check_path=False, ask_header=False, behaviour_on_error='error', open_file_kwargs=None, read_kwargs=None, to_datetime_kwargs=None)¶

Returns a dataframe with the content of the selected CSV or Excel file.

Parameters

path – file path. Must be CSV or Excel file (to be checked before with ‘open_file’ function).
config_dict – dictionary-like object containing the configuration
config_key – key of config_dict to get path (replaces path argument)
date_columns – column or list of columns to format as datetime
sheet_name – name or index of the Excel sheet (not used for CSV)
check_path – if True, use open_file to check path
ask_header – if True and key ‘header’ is not in read_kwargs, ask whether the file has a header or not and update read_kwargs[‘header’] to 0 or None.
behaviour_on_error – ‘error’ (default): raise an error, ‘ignore’: return None
open_file_kwargs – kwargs for open_file function ONLY.
read_kwargs – kwargs for read functions pd.read_excel ONLY. # can evolve in the future
to_datetime_kwargs – kwargs for pd.to_datetime function ONLY.

Returns

tools.helpers.data_manager.data_writer module¶

Functions to export data files to Excel or images to PowerPoint.

tools.helpers.data_manager.data_writer.export_img_to_powerpoint(img_paths, output_path=None)¶

Export images to a PowerPoint presentation.

Parameters

img_paths – list of images to save in PowerPoint slides
output_path – path to save output file

Returns

path

tools.helpers.data_manager.data_writer.save_excel_file(path, dataframes, sheet_names=None, extension='.xlsx', check_path=False, to_excel_kwargs=None, **options)¶

Write dataframes to disk.

Parameters

path – Excel file path (to be checked before with ‘save_file’ function).
dataframes – dataframe or list of dataframes
sheet_names – sheet name or ordered list of sheet names
extension – Excel file extension
check_path – check path with save_file
to_excel_kwargs – keyword arguments for _write_excel function
options – options for save_file function

Returns

path

tools.helpers.data_manager.file_utils module¶

Utils linked to files.

tools.helpers.data_manager.file_utils.add_file_extension(path: str, extension=None, replace=False, keep_existing=False, force_add=False) → tools.helpers.models.path_models.Path¶

Add a specific extension to ‘path’.

Parameters

path – path
extension – extension. None is considered as ‘’ extension
replace – replace instead of append extension
keep_existing – add extension only if it doesn’t already exist
force_add – add extension even if the correct extension already exists (not recommended)

:return Path object

tools.helpers.data_manager.file_utils.check_extension(path: Union[tools.helpers.models.path_models.Path, list, tuple, set], extension=None, filetype=None, check_ext='ignore') → bool¶: Check extension of path or collection of paths which must arg ‘extension’ or extension linked to ‘filetype’, unless they are None

tools.helpers.data_manager.file_utils.check_path_arguments(path: str, config_dict: Dict, config_key: str) → str¶: User should use either ‘path’ or ‘config_dict’ and ‘config_key’ arguments to indicate a path.

tools.helpers.data_manager.file_utils.choose_filedialog(dialog_type: str, multiple_paths: bool = False, return_on_cancellation: str = None, behavior_on_cancellation: str = 'ignore', initialdir: str = None, filetypes: list = None, title: str = None, **kwargs) → Union[tuple, tools.helpers.models.path_models.Path]¶: Open a filedialog window.

tools.helpers.data_manager.filepath_manager module¶

Functions to select and validate file or directory paths.

tools.helpers.data_manager.filepath_manager.handle_bad_extension_error(err, func, path, args=None, kwargs=None, pos_path=0, key_path=None, change_path_func=<function save_file>, title='', msg='', extension=None)¶: extension: default extension to add to the path. If None, nothing is changed

tools.helpers.data_manager.filepath_manager.handle_file_error(err, func, path, args=None, kwargs=None, pos_path=0, key_path=None, change_path_func=<function save_file>, title='', msg='', return_if_ignore=None)¶

If PermissionError when opening/saving file, propose to retry, change file path or cancel

Parameters

err – exception
func – function to execute if the user wants to retry
path – file path
args – args to pass to func
kwargs – kwargs to pass to func
pos_path – position of the positional argument path in func (only if key_path is None)
key_path – name of the keyword argument path in func (if None, positional argument is used)
change_path_func – function to get a new path, with no positional argument and ‘initialdir’ keyword argument
title – title of the error
msg – message of the error
return_if_ignore – return if Ignore option is selected

Returns

tools.helpers.data_manager.filepath_manager.handle_file_not_found_error(err, func, path, args=None, kwargs=None, pos_path=0, key_path=None, change_path_func=<function save_file>, title='', msg='')¶

tools.helpers.data_manager.filepath_manager.handle_permission_error(err, func, path, args=None, kwargs=None, pos_path=0, key_path=None, change_path_func=<function save_file>, title='', msg='', handle_read_only_error=False)¶: handle_read_only_error: True to handle the case of a read-only file: the function tries to make the file writable.

tools.helpers.data_manager.filepath_manager.open_dir(*args, **kwargs) → Union[tools.helpers.models.path_models.Path, tools.helpers.models.path_models.PathCollection]¶

Check path and open file dialog(s) to select directory(ies) if needed.

>>> dir1 = os.path.dirname(__file__)
>>> open_dir(dir1).filename
Path: data_manager

>>> open_dir(None, ask_path=False)
Path: None

>>> open_dir([dir1, dir1, None], multiple_paths=True, ask_path=False).filename
[Path: data_manager, Path: data_manager, Path: ]

tools.helpers.data_manager.filepath_manager.open_file(*args, **kwargs) → Union[tools.helpers.models.path_models.Path, tools.helpers.models.path_models.PathCollection]¶

Check path and open file dialog(s) to select file(s) if needed.

>>> open_file(__file__).filename
Path: filepath_manager.py

>>> open_file(None, ask_path=False)
Path: None

>>> open_file([__file__, __file__, None], multiple_paths=True, ask_path=False).filename
[Path: filepath_manager.py, Path: filepath_manager.py, Path: ]

tools.helpers.data_manager.filepath_manager.open_file_or_dir(path: Union[str, list, tuple, set] = None, config_dict=None, config_key=None, multiple_paths=False, return_on_cancellation: str = None, behavior_on_cancellation='warning', filetype=None, extension=None, check_ext='ignore', ask_path=True, path_type='file', **kwargs) → Union[tools.helpers.models.path_models.Path, tools.helpers.models.path_models.PathCollection]¶

Check path and open file dialogs if needed.

Parameters

path – path to check. If path is None, a file dialog is opened to choose path, unless ‘ask_path’ is False
config_dict – instead of path, use config_dict and config_key. path will be set to config_dict[config_key]
config_key – instead of path, use config_dict and config_key. path will be set to config_dict[config_key]
multiple_paths – if True, multiple files are allowed and returned as PathCollection (instead of Path)
return_on_cancellation – return on user cancellation. Default: Path(None) (highly recommended)
behavior_on_cancellation – error flag used to raise anomaly ‘raise_no_file_selected_anomaly’ on cancellation . Must be ‘ask’, ‘ignore’, ‘warning’, or ‘error’.
filetype – filetype in FILETYPE_TO_FILEDIALOG_FILETYPES keys. If filetype is None, extension is not checked.
extension – final extension to check. If extension is None, extension is not checked.
check_ext – error flag used to raise anomaly ‘raise_bad_extension_anomaly’
ask_path – if True and path is None, ask path to the user
path_type – ‘file’ or ‘dir’. ‘file by default
kwargs – keywords arguments for filedialog methods, such as ‘title’, ‘message’, ‘filetypes’, etc.

Returns

path or collection of paths

tools.helpers.data_manager.filepath_manager.save_file(path: str = None, config_dict=None, config_key=None, return_on_cancellation=Path: None, behavior_on_cancellation='warning', auto_mkdir=True, extension=None, replace_ext=False, filetype=None, overwrite='ask', backup=False, **kwargs) → tools.helpers.models.path_models.Path¶

Check the path to save a file.

param path

path or None

param config_dict

dictionary-like object containing the configuration

param config_key

key of config_dict to get path (replaces path argument)

Parameters

return_on_cancellation –

return on user cancellation. Default: Path(None) (highly recommended) :param behavior_on_cancellation: error flag used to raise anomaly ‘raise_no_file_selected_anomaly’ on cancellation .

Must be ‘ask’, ‘ignore’, ‘warning’, or ‘error’.

param auto_mkdir: if True, try to create output directories if they don’t exist
param filetype: filetype in FILETYPE_TO_FILEDIALOG_FILETYPES keys. If filetype is None, extension is not checked.
param extension: final extension to check. If extension is None, extension is not checked.
param replace_ext: if True, replaces extension instead of adding one
param overwrite: bool or flag ‘ask’, ‘rename’, ‘ignore’, ‘overwrite’, used in case file already exists
param backup: bool, used in case file already exists
param kwargs: for filedialog
return: path

tools.helpers.data_manager.plots module¶

Functions to save graphs.

tools.helpers.data_manager.plots.save_error_graph(plot_path, err_msg=None, **save_file_kwargs)¶

tools.helpers.data_manager.plots.save_plot(plot_path, overwrite='overwrite', **save_file_kwargs)¶

Module contents¶

Data manager

Provides high level function to check paths, open and read data files

file_utils: various functions
filepath_manager: functions to check paths, using file dialogs
data_loader: functions to open files: CSV or Excel
date_writer: functions to write files: Excel or PowerPoint
plots: functions to create simple plots

Imports: logger, utils, models, interface ——————————————–

| | |

v v v v

file_utils –> filepath_manager –> data_loader
|-> data_writer

tools.helpers.data_manager.open_file(*args, **kwargs) → Union[tools.helpers.models.path_models.Path, tools.helpers.models.path_models.PathCollection]¶

Check path and open file dialog(s) to select file(s) if needed.

>>> open_file(__file__).filename
Path: filepath_manager.py

>>> open_file(None, ask_path=False)
Path: None

>>> open_file([__file__, __file__, None], multiple_paths=True, ask_path=False).filename
[Path: filepath_manager.py, Path: filepath_manager.py, Path: ]

tools.helpers.data_manager.open_dir(*args, **kwargs) → Union[tools.helpers.models.path_models.Path, tools.helpers.models.path_models.PathCollection]¶

Check path and open file dialog(s) to select directory(ies) if needed.

>>> dir1 = os.path.dirname(__file__)
>>> open_dir(dir1).filename
Path: data_manager

>>> open_dir(None, ask_path=False)
Path: None

>>> open_dir([dir1, dir1, None], multiple_paths=True, ask_path=False).filename
[Path: data_manager, Path: data_manager, Path: ]

tools.helpers.data_manager.save_file(path: str = None, config_dict=None, config_key=None, return_on_cancellation=Path: None, behavior_on_cancellation='warning', auto_mkdir=True, extension=None, replace_ext=False, filetype=None, overwrite='ask', backup=False, **kwargs) → tools.helpers.models.path_models.Path¶

Check the path to save a file.

param path

path or None

param config_dict

dictionary-like object containing the configuration

param config_key

key of config_dict to get path (replaces path argument)

Parameters

return_on_cancellation –

return on user cancellation. Default: Path(None) (highly recommended) :param behavior_on_cancellation: error flag used to raise anomaly ‘raise_no_file_selected_anomaly’ on cancellation .

Must be ‘ask’, ‘ignore’, ‘warning’, or ‘error’.

param auto_mkdir: if True, try to create output directories if they don’t exist
param filetype: filetype in FILETYPE_TO_FILEDIALOG_FILETYPES keys. If filetype is None, extension is not checked.
param extension: final extension to check. If extension is None, extension is not checked.
param replace_ext: if True, replaces extension instead of adding one
param overwrite: bool or flag ‘ask’, ‘rename’, ‘ignore’, ‘overwrite’, used in case file already exists
param backup: bool, used in case file already exists
param kwargs: for filedialog
return: path

tools.helpers.data_manager.read_data_file(path=None, config_dict=None, config_key=None, date_columns=None, sheet_name=0, check_path=False, ask_header=False, behaviour_on_error='error', open_file_kwargs=None, read_kwargs=None, to_datetime_kwargs=None)¶

Returns a dataframe with the content of the selected CSV or Excel file.

Parameters

path – file path. Must be CSV or Excel file (to be checked before with ‘open_file’ function).
config_dict – dictionary-like object containing the configuration
config_key – key of config_dict to get path (replaces path argument)
date_columns – column or list of columns to format as datetime
sheet_name – name or index of the Excel sheet (not used for CSV)
check_path – if True, use open_file to check path
ask_header – if True and key ‘header’ is not in read_kwargs, ask whether the file has a header or not and update read_kwargs[‘header’] to 0 or None.
behaviour_on_error – ‘error’ (default): raise an error, ‘ignore’: return None
open_file_kwargs – kwargs for open_file function ONLY.
read_kwargs – kwargs for read functions pd.read_excel ONLY. # can evolve in the future
to_datetime_kwargs – kwargs for pd.to_datetime function ONLY.

Returns

tools.helpers.data_manager.save_excel_file(path, dataframes, sheet_names=None, extension='.xlsx', check_path=False, to_excel_kwargs=None, **options)¶

Write dataframes to disk.

Parameters

path – Excel file path (to be checked before with ‘save_file’ function).
dataframes – dataframe or list of dataframes
sheet_names – sheet name or ordered list of sheet names
extension – Excel file extension
check_path – check path with save_file
to_excel_kwargs – keyword arguments for _write_excel function
options – options for save_file function

Returns

path

tools.helpers.data_manager.export_img_to_powerpoint(img_paths, output_path=None)¶

Export images to a PowerPoint presentation.

Parameters

img_paths – list of images to save in PowerPoint slides
output_path – path to save output file

Returns

path

tools.helpers.data_manager.save_plot(plot_path, overwrite='overwrite', **save_file_kwargs)¶

tools.helpers.data_manager.save_error_graph(plot_path, err_msg=None, **save_file_kwargs)¶

tools.helpers.data_manager package¶

Submodules¶

tools.helpers.data_manager.data_loader module¶

tools.helpers.data_manager.data_writer module¶

tools.helpers.data_manager.file_utils module¶

tools.helpers.data_manager.filepath_manager module¶

tools.helpers.data_manager.plots module¶

Module contents¶

Table of Contents

Previous topic

Next topic

This Page