pat2vec.util.post_processing_utils

Functions

copy_files_and_dirs(source_root, ...[, ...])

Copies specified directories and files from a source location to a new destination.

count_files(path)

Recursively counts the number of files in a directory.

filter_and_update_csv(target_directory, ...)

Filters and updates CSV files in a target directory based on patient IPW records.

process_chunk(args)

Processes a chunk of CSV files, concatenating their data into a dictionary.
pat2vec.util.post_processing_utils.count_files(path)[source]

Recursively counts the number of files in a directory.

Return type:

int

Parameters:

path (str)

pat2vec.util.post_processing_utils.process_chunk(args)[source]

Processes a chunk of CSV files, concatenating their data into a dictionary.

This helper function is designed for multiprocessing. It reads a specified range of files, extracts data for a given set of unique columns, and returns a dictionary where keys are column names and values are lists of data from those columns.

Parameters:

args (tuple) – A tuple containing (part_chunk, all_files, part_size, unique_columns).

Return type:

Dict[str, List[str]]

Returns:

A dictionary with concatenated data for the specified unique columns.

pat2vec.util.post_processing_utils.copy_files_and_dirs(source_root, source_name, destination, items_to_copy=None, loose_files=None)[source]

Copies specified directories and files from a source location to a new destination.

Return type:

None

Parameters:

  • source_root (str)

  • source_name (str)

  • destination (str)

  • items_to_copy (List[str] | None)

  • loose_files (List[str] | None)

pat2vec.util.post_processing_utils.filter_and_update_csv(target_directory, ipw_dataframe, filter_type='after', verbosity=False)[source]

Filters and updates CSV files in a target directory based on patient IPW records.

This function iterates through each patient record in the ipw_dataframe, finds corresponding CSV files in the target_directory (and its subdirectories), and filters the rows in those CSV files based on a timestamp column and a filter date.

Parameters:

  • target_directory (str) – The root directory containing the CSV files to be filtered.

  • ipw_dataframe (pd.DataFrame) – A DataFrame containing patient IPW records, including ‘client_idcode’ and a timestamp column (e.g., ‘updatetime’).

  • filter_type (str, optional) – The type of filtering to apply: “after” (keep records after filter_date) or “before” (keep records before filter_date). Defaults to “after”.

  • verbosity (bool, optional) – If True, print verbose messages during processing.

Return type:

None