pat2vec.util.post_processing_annotations

Functions

`check_list_presence`(df, column, lst[, ...])	Checks if any string in a list is present in a specified DataFrame column, optionally after applying annotation filters.
`extract_types_from_csv`(directory)	Extracts all unique 'types' from CSV files within a given directory and its subdirectories.
`filter_and_select_rows`(dataframe, filter_list)	Filter a dataframe based on a filter_column and filter_list, and return either the earliest or latest rows.
`filter_annot_dataframe2`(dataframe, filter_args)	Filter a DataFrame based on specified filter arguments.
`filter_dataframe_by_cui`(dataframe, filter_list)	Filter an annotation DataFrame based on a list of CUI codes and a specified mode.
`filter_dataframe_n_lists`(df, column_name, ...)	Filters a DataFrame to include rows where the value in a specified column is present in all of the provided lists.
`get_all_target_annots`(all_pat_list, n_lists)	Retrieves and filters target annotations for a list of patients.
`join_icd10_OPC4S_codes_to_annot`(df[, inner])	Joins ICD-10 and OPCS-4 codes to an annotation DataFrame.
`join_icd10_codes_to_annot`(df[, inner])	Joins ICD-10 codes to an annotation DataFrame.
`produce_filtered_annotation_dataframe`([...])	Filter annotation dataframe based on specified criteria.
`remove_file_from_paths`(current_pat_idcode[, ...])	Removes patient-specific data from various predefined project paths or database tables.
`retrieve_pat_annots_mct_epr`(client_idcode, ...)	Retrieves and merges annotation data for a single patient from multiple sources (files or database).

pat2vec.util.post_processing_annotations.filter_annot_dataframe2(dataframe, filter_args)[source]

Filter a DataFrame based on specified filter arguments.

Parameters:

dataframe (DataFrame) – The DataFrame to filter.
filter_args (Dict[str, Any]) – A dictionary containing filter arguments. Keys are column names, and values are the filter criteria. Special handling for ‘types’, ‘Time_Value’, ‘Presence_Value’, ‘Subject_Value’, ‘Time_Confidence’, ‘Presence_Confidence’, ‘Subject_Confidence’, and ‘acc’.

Return type:

DataFrame

Returns:

The filtered DataFrame.

pat2vec.util.post_processing_annotations.produce_filtered_annotation_dataframe(cui_filter=False, meta_annot_filter=False, pat_list=None, config_obj=None, filter_custom_args=None, cui_code_list=None, mct=False)[source]

Filter annotation dataframe based on specified criteria.

Parameters:

cui_filter (bool) – Whether to filter by CUI codes.
meta_annot_filter (bool) – Whether to apply meta annotation filtering.
pat_list (Optional[List[str]]) – List of patient identifiers. If None, uses config_obj.all_patient_list.
config_obj (Optional[Any]) – Configuration object containing necessary parameters.
filter_custom_args (Optional[Dict[str, Any]]) – Custom filter arguments. If None, uses config_obj.filter_arguments.
cui_code_list (Optional[List[int]]) – List of CUI codes for filtering.
mct (bool) – If True, processes MCT annotation batches; otherwise, processes EPR.

Returns:

Filtered annotation dataframe.

Return type:

pd.DataFrame

pat2vec.util.post_processing_annotations.extract_types_from_csv(directory)[source]

Extracts all unique ‘types’ from CSV files within a given directory and its subdirectories.

Parameters:: directory (str) – The path to the directory to search for CSV files.
Return type:: List[str]
Returns:: A list of all unique ‘types’ found in the ‘types’ column of the CSV files.

pat2vec.util.post_processing_annotations.join_icd10_codes_to_annot(df, inner=False)[source]

Joins ICD-10 codes to an annotation DataFrame.

This function merges the input DataFrame df with a predefined ICD-10 mapping DataFrame based on the ‘cui’ column in df and ‘referencedComponentId’ in the mapping.

Parameters:

df (DataFrame) – The annotation DataFrame.
inner (bool) – If True, performs an inner merge; otherwise, performs a left merge.

Return type:

DataFrame

Returns:

The DataFrame with ICD-10 codes joined.

pat2vec.util.post_processing_annotations.join_icd10_OPC4S_codes_to_annot(df, inner=False)[source]

Joins ICD-10 and OPCS-4 codes to an annotation DataFrame.

This function merges the input DataFrame df with a predefined ICD-10/OPCS-4 mapping DataFrame based on the ‘cui’ column in df and ‘conceptId’ in the mapping.

Parameters:

df (DataFrame) – The annotation DataFrame.
inner (bool) – If True, performs an inner merge; otherwise, performs a left merge.

Return type:

DataFrame

Returns:

The DataFrame with ICD-10 and OPCS-4 codes joined.

pat2vec.util.post_processing_annotations.filter_and_select_rows(dataframe, filter_list, verbosity=0, time_column='updatetime', filter_column='cui', mode='earliest', n_rows=1)[source]

Filter a dataframe based on a filter_column and filter_list, and return either the earliest or latest rows.

Parameters:

dataframe (DataFrame) – Input dataframe.
filter_list (List[Any]) – List of values to filter the dataframe.
verbosity (int) – If > 0, print additional information during execution.
time_column (str) – Column representing time, used for sorting if specified.
filter_column (str) – Column used for filtering based on filter_list.
mode (str) – Either ‘earliest’ or ‘latest’ to specify the rows to return.
n_rows (int) – Number of rows to return if they exist.

Returns:

Filtered and selected rows from the input dataframe.

Return type:

pd.DataFrame

pat2vec.util.post_processing_annotations.filter_dataframe_by_cui(dataframe, filter_list, filter_column='cui', mode='earliest', temporal='before', verbosity=0, time_column='updatetime')[source]

Filter an annotation DataFrame based on a list of CUI codes and a specified mode.

Parameters:

dataframe (DataFrame) – The input DataFrame.
filter_list (List[int]) – List of CUI codes to filter the DataFrame.
filter_column (str) – The column containing filter.
mode (str) – Specifies whether to consider the earliest or latest entry for each filter.
temporal (str) – Specifies whether to retain entries before or after the selected mode entry.
verbosity (int) – Verbosity level. 0 for no debug statements, higher values for more verbosity.
time_column (str) – The column containing time information.

Returns:

Filtered DataFrame based on the specified criteria.

Return type:

pd.DataFrame

pat2vec.util.post_processing_annotations.check_list_presence(df, column, lst, annot_filter_arguments=None)[source]

Checks if any string in a list is present in a specified DataFrame column, optionally after applying annotation filters.

Parameters:

df (pd.DataFrame) – The input DataFrame.
column (str) – The name of the column to check for string presence.
lst (list) – A list of strings to search for.
annot_filter_arguments (dict, optional) – Arguments to filter the DataFrame before checking for list presence. Defaults to None.

Returns:

True if any string from lst is found in column (case-insensitive), False otherwise.

Return type:

bool

pat2vec.util.post_processing_annotations.filter_dataframe_n_lists(df, column_name, n_lists)[source]

Filters a DataFrame to include rows where the value in a specified column is present in all of the provided lists.

Parameters:

df (DataFrame) – The input DataFrame.
column_name (str) – The name of the column to filter.
n_lists (List[List[Any]]) – A list of lists. A row is kept only if the value in column_name is present in every sublist within n_lists.

Return type:

DataFrame

Returns:

The filtered DataFrame.

pat2vec.util.post_processing_annotations.get_all_target_annots(all_pat_list, n_lists, config_obj=None, annot_filter_arguments=None)[source]

Retrieves and filters target annotations for a list of patients.

This function iterates through a list of patient IDs, retrieves their annotations, applies optional annotation filters, and then filters the annotations to include only those where the ‘cui’ (Concept Unique Identifier) is present in all of the provided n_lists. The results are concatenated into a single DataFrame and saved.

Parameters:

all_pat_list (List[str]) – A list of patient IDs to process.
n_lists (List[List[int]]) – A list of lists of CUI codes. Annotations are kept if their CUI is in all sublists.
config_obj (Optional[Any]) – A configuration object.
annot_filter_arguments (Optional[Dict[str, Any]]) – Arguments to filter annotations.

Returns:

A DataFrame containing all target annotations.

Return type:

pd.DataFrame

pat2vec.util.post_processing_annotations.retrieve_pat_annots_mct_epr(client_idcode, config_obj, columns_epr=None, columns_mct=None, columns_to=None, columns_report=None, merge_columns=True)[source]

Retrieves and merges annotation data for a single patient from multiple sources (files or database).

This function reads annotation data for a specified patient from four potential sources: EPR annotations, MCT annotations, textual observations annotations, and reports annotations. It loads the corresponding CSV files, optionally selecting specific columns, and concatenates them into a single DataFrame. It can also merge related columns (e.g., timestamps, content) to create a more unified dataset.

Parameters:

client_idcode (str) – The unique identifier for the patient.
config_obj (Any) – A configuration object containing paths to the various annotation batch files.
columns_epr (Optional[List[str]]) – A list of columns to load from the EPR annotations CSV.
columns_mct (Optional[List[str]]) – A list of columns to load from the MCT annotations CSV.
columns_to (Optional[List[str]]) – A list of columns to load from the textual observations annotations CSV.
columns_report (Optional[List[str]]) – A list of columns to load from the reports annotations CSV.
merge_columns (bool, optional) – If True, attempts to merge corresponding columns (e.g., timestamps, content) from the different sources into a unified set of columns. Defaults to True.

Returns:

A DataFrame containing the concatenated and optionally: merged annotation data for the patient. Returns an empty DataFrame if no data is found for the patient in any of the sources.

Return type:

pd.DataFrame

pat2vec.util.post_processing_annotations.remove_file_from_paths(current_pat_idcode, project_name='new_project', verbosity=0, config_obj=None)[source]

Removes patient-specific data from various predefined project paths or database tables.

If storage_backend is ‘database’, it removes records for the patient from relevant tables. Otherwise, it removes CSV files.

Parameters:

current_pat_idcode (str) – The unique identifier of the patient whose files are to be removed.
project_name (str) – The name of the project. Used if config_obj is None.
verbosity (int) – Verbosity level for printing messages.
config_obj (Optional[Any]) – A configuration object containing project paths. If provided, project_name is overridden by config_obj.proj_name. Defaults to None.

Return type:

None