Panel

class neer_match_utilities.panel.GenerateID(df_panel, panel_var, time_var, model, similarity_map=None, prediction_threshold=<object object>, subgroups=None, relation='m:m', blocking=None)

A class to generate and harmonize unique IDs across time periods for panel data.

group_by_subgroups():

Group the panel data into subgroups.

generate_suggestions(df_slice):

Generate ID suggestions for consecutive time periods.

harmonize_ids(suggestions, periods, original_df):

Harmonize IDs across time periods.

assign_ids(id_mapping):

Assign unique IDs to the harmonized IDs.

execute():

Execute the full ID generation and harmonization process.

__init__(df_panel, panel_var, time_var, model, similarity_map=None, prediction_threshold=<object object>, subgroups=None, relation='m:m', blocking=None)

Initialize the GenerateID class.

Parameters:

• df_panel (DataFrame) – The panel dataset.

• panel_var (str) – The panel identifier variable that is supposed to be created.

• time_var (str) – The time period variable.

• subgroups (List[str]) – List of subgroup variables for slicing. Defaults to None.

• model (object) – A model object with a suggest method.

• similarity_map (Union[dict, SimilarityMap, None]) – A dictionary of similarity functions for columns.

• prediction_threshold (float | None | object) – Probability from which a suggested pair counts as a match. By default the threshold tuned and saved with the model (best_threshold_, as TrainingPipe and BaselineTrainingPipe’s gradient boosting tune it), else 0.9. None takes the tuned threshold too, else 0.9 for a similarity map given as a SimilarityMap and 0.5 otherwise.

• relation (str) – Relationship between observations in cross sectional data. Default is ‘m:m’

• blocking (Blocking, function or None, optional) – A rule of neer_match_utilities.blocking: in each pair of consecutive periods, the model scores only the candidate pairs. It needs a model whose suggest takes blocking, as the baseline models (Logit, Probit, Gradient Boosting) do. subgroups works as before: it splits the panel into groups first and compares the consecutive periods present in each group, so a group that skips a period compares the periods around the gap. Blocking(exact=...) on the same columns compares the panel’s consecutive periods instead. Both can be combined.

assign_ids(id_mapping)

Assign unique IDs to the harmonized IDs.

Parameters:

id_mapping (DataFrame) – The harmonized ID mapping dataframe.

Returns:

Dataframe with assigned unique IDs.

Return type:

DataFrame

execute()

Execute the full ID generation and harmonization process.

Returns:

The final ID mapping.

Return type:

DataFrame

generate_suggestions(df_slice)

Generate ID suggestions for consecutive time periods.

Parameters:

df_slice (DataFrame) – A dataframe slice containing data to process.

Returns:

A tuple containing: - pd.DataFrame: A concatenated dataframe of suggestions. - list of int: A list of periods.

Return type:

Tuple[DataFrame, List[int]]

group_by_subgroups()

Group the panel data into subgroups.

Returns:

Grouped dataframe by subgroups.

Return type:

pd.core.groupby.generic.DataFrameGroupBy

harmonize_ids(suggestions, periods, original_df)

Harmonize IDs across time periods.

Parameters:

• suggestions (DataFrame) – The dataframe with suggestions.

• periods (List[int]) – List of periods.

• original_df (DataFrame) – The original dataframe.

Returns:

Harmonized ID mapping.

Return type:

DataFrame

relations_left_right(df, relation=None)

Apply validation rules to enforce relationships between matched observations.

Parameters:

• df (DataFrame) – DataFrame containing ‘left’, ‘right’, and ‘prediction’ columns.

• relation (str) – Validation mode for relationships. If None, defaults to self.relation. Options: - ‘m:m’ : Many-to-many, no duplicates removed. - ‘1:m’ : Unique ‘left’ values. - ‘m:1’ : Unique ‘right’ values. - ‘1:1’ : Unique ‘left’ and ‘right’ values.

Returns:

A reduced DataFrame with relationships enforced based on relation.

Return type:

DataFrame

Raises:

ValueError – If relation is not one of [‘m:m’, ‘1:m’, ‘m:1’, ‘1:1’].

class neer_match_utilities.panel.SetupData(matches=None)

A class for processing and preparing data with overlapping matches and panel relationships.

matches

A list of tuples representing matches.

Type:

list

__init__(matches=None)

Initialize the SetupData class.

Parameters:

matches (list) – A list of tuples representing matches. Defaults to an empty list.

adjust_overlap(dfm)

Adjusts the overlap in the matches DataFrame by generating additional ordered pairs from connected components in the match graph.

This function takes a DataFrame containing match pairs in the ‘left’ and ‘right’ columns. It constructs a full connection graph from these matches, computes connected components using depth-first search (DFS), and then, for each connected component with more than one element, generates all ordered pairs (i.e. permutations) of distinct IDs. These new pairs are appended to the original DataFrame.

Parameters:

dfm (DataFrame) – A DataFrame with columns ‘left’ and ‘right’ representing the matching pairs, where each entry is a unique identifier.

Returns:

A combined DataFrame that includes both the original match pairs and the newly generated pairs from connected components. Note that pairs of the form (A, A) are not generated, and for any distinct IDs A and B, both (A, B) and (B, A) may appear. Duplicates are not removed.

Return type:

DataFrame

Notes

• Connected components are determined by treating the match pairs as edges in an undirected graph.

• The function uses permutations of length 2 to generate ordered pairs, ensuring that only pairs of distinct IDs are created.

• The new pairs are simply appended to the original DataFrame without dropping duplicates.

create_connected_groups(df_dict, matches)

Create a list of lists where sublists contain connected values as one group.

Parameters:

• df_dict (Dict) – A dictionary where keys are integers and values are lists of integers.

• matches (List[Tuple[int, int]]) – A list of tuples representing connections between values.

Returns:

A list of lists with connected values grouped together.

Return type:

List[List[int]]

data_preparation_panel(df_panel, unique_id, panel_id=None)

Prepare data by handling overlaps, panel combinations, and duplicates.

This function converts the unique identifier column to a numeric type (if possible), then prepares match pairs by adjusting overlaps and dropping duplicate pairs. It then extracts the subset of the panel data corresponding to the matched IDs, and finally sorts the left and right DataFrames according to the unique identifier.

The returned matches hold every pair of left x right whose two records belong to the same group: a record that sits in both left and right is paired with itself, and a pair whose records both sit in left and in right is listed in both orientations (since 1.2.11). When groups have more than two records, split them with split_test_train(…, keep_components=True).

Parameters:

• df_panel (DataFrame) – Panel DataFrame containing IDs and panel information.

• unique_id (str) – Column name of unique identifiers in df_panel.

• panel_id (str) – Column name of panel identifiers in df_panel.

Returns:

A tuple of three DataFrames: left, right, and the final matches.

Return type:

Tuple[DataFrame, DataFrame, DataFrame]

static drop_repetitions(df)

Remove duplicate pairs in the DataFrame irrespective of the order of elements.

This function treats each pair in the ‘left’ and ‘right’ columns as unordered. It creates a temporary column ‘sorted_pair’ that contains a sorted tuple of the ‘left’ and ‘right’ values for each row. For example, the pairs (A, B) and (B, A) will both be transformed into (A, B), and only one instance will be retained. The function then drops duplicate rows based on this sorted pair and removes the temporary column before returning the result.

Parameters:

df (DataFrame) – A DataFrame containing at least the columns ‘left’ and ‘right’, which represent paired elements.

Returns:

A DataFrame in which duplicate pairs (ignoring order) have been removed.

Return type:

DataFrame

panel_preparation(dfm, df_panel, unique_id, panel_id)

Generate combinations of IDs for each panel and append them to the DataFrame.

Parameters:

• dfm (DataFrame) – DataFrame to append combinations to.

• df_panel (DataFrame) – Panel DataFrame containing IDs and panel information.

• unique_id (str) – Column name of unique identifiers in df_panel.

• panel_id (str) – Column name of panel identifiers in df_panel.

Returns:

Updated DataFrame with appended combinations.

Return type:

DataFrame

• Index

• Module Index