U 9hݗ@sdZddlZddlmZddlZddlmZmZddl m Z m Z ddl m Z ddlmZmZmZdd lmZmZmZmZmZmZmZd$d d Zd%ddZd&ddZd'ddZddZddZddZ d(ddZ!d)ddZ"ddZ#d d!Z$d"d#Z%dS)*aflat.py Provides alternative functions to hdbscan.HDBSCAN and others to 1. Allow prediction on a flat clustering by specifying 'n_clusters'. This is done by choosing the best cluster_selection_epsilon that produces the required number of clusters without adding unnecessary outliers. 2. Makes approximate_predict, membership_vector, and all_points_membership_vectors consistent with cluster_selection_epsilon Provides the following functions: ================================== HDBSCAN_flat: trained HDBSCAN instance with 'n_clusters' clusters The attributes (labels, probabilities, prediction_data) are tuned to produce 'n_clusters' clusters. approximate_predict_flat: labels and probabilities for novel points Allows selecting n_clusters for novel points, or using the original clustering (potentially specified using cluster_selection_epsilon) membership_vector_flat: Soft-clustering probabilities for novel points Similar to approximate_predict_flat, but for soft-clustering. **Use with caution** all_points_membership_vectors_flat: Soft-clustering probabilities Similar to membership_vector_flat, but for points in training set **Use with caution** N)warn)compute_stabilityget_cluster_tree_leaves)HDBSCAN_tree_to_labels)_bfs_from_cluster_tree)PredictionData_find_cluster_and_probability_find_neighbor_and_lambda)get_tree_row_with_childdist_membership_vectoroutlier_membership_vectorprob_in_some_cluster!all_points_dist_membership_vector$all_points_outlier_membership_vectorall_points_prob_in_some_clusterFc Ks|dkrJ|dkrJt|tr|s2d|d<tf|}n |}d|_|||S|dk rn|dkrntd|dd}t|tsd|d<tf|}||n*|r|}n t|}d|_d|_|||jdkrt|j }|dk r|jdkr||krtd|d d |_t |j |}n|}t ||_ |j} | } |j} |j} |j} d } td| | | | | |d }|\|_|_|_|_|_t|j|j |d |S) a Train a HDBSCAN clusterer by specifying n_clusters. Or, modify a trained clusterer to return specific n_clusters. Parameters ---------- X: array-like Data to be passed to HDBSCAN for training. n_clusters: int, default=None Number of clusters to produce. If None, revert to default HDBSCAN cluster_selection_epsilon: float, default=0. core-distance below which to stop splitting clusters. This can indirectly impose n_clusters. This argument is ignored if n_clusters is supplied. clusterer: HDBSCAN, default=None If supplied, modify this clusterer to produce n_clusters clusters. inplace: bool, default=False If 'clusterer' parameter is supplied, and inplace is True, modify the previous clusterer inplace. If False, return a modified copy of the previous clusterer. **kwargs: keyword arguments All init arguments for HDBSCAN Returns ------- new_clusterer: HDBSCAN New HDBSCAN instance; returned irrespective of inplace=True or False Usage ----- # Extract flat clustering from HDBSCAN's hierarchy for 7 clusters clusterer = HDBSCAN_flat(X_train, n_clusters=7, min_cluster_size=12, min_samples=8) labels = clusterer.labels_ proba = clusterer.probabilities_ # Use a previously initialized/trained HDBSCAN old_clusterer = HDBSCAN(min_cluster_size=12, min_samples=8) clusterer = HDBSCAN_flat(X_train, n_clusters=7, clusterer=old_clusterer, inplace=True) labels = clusterer.labels_ proba = clusterer.probabilities_ See Also --------- :py:func:`hdbscan.HDBSCAN` :py:func:`re_init` NrTprediction_dataz'cluster_selection_epsilon' (=z+) is ignored when 'n_clusters' is supplied.eomCannot predict more than B with cluster selection method 'eom'. Changing to method 'leaf'...leafFcluster_selection_epsilon) isinstancerrfitrcopydeepcopycluster_selection_methodlencondensed_tree__select_clustersselect_epsilonfloatrZsingle_linkage_tree_Zto_numpymin_cluster_sizeallow_single_clusterrlabels_Zprobabilities_Zcluster_persistence_Z_condensed_treeZ_single_linkage_treere_initprediction_data_)X n_clustersr clustererZinplacekwargsZ new_clustererZmax_eom_clustersepsilonZsingle_linkage_treer%rr&match_reference_implementationoutputr1_/var/www/html/CrowdFlow/HYROX/ble_analysis_env_py38/lib/python3.8/site-packages/hdbscan/flat.py HDBSCAN_flat0sx:            r3c Cstt|jdk}|dk r&t|}|j}|dkrR|dksD||krR|dkrR|j}t|ts|jdkrnt d|j}t |}|j dkr|dk r||krt d|dd|_ t||||dt|}|jd |jjd krt d |jjddkrHt d d tj|jdtjd }tj|jdtjd } |r@|| |fS|| fStj|jdtjd }tj|jdtjd } |jp|j} |jj|d| d\} } t|jdD]B} t||j| | | | |j|j |j!| \}}}||| <|| | <q|r|| |fS|| fSdS)a Predict the cluster label of new points at a particular flat clustering, specified by n_clusters. This is a modified version of hdbscan.approximate_predict to allow selection of n_clusters. Parameters ---------- clusterer : HDBSCAN A clustering object that has been fit to the data and either had ``prediction_data=True`` set, or called the ``generate_prediction_data`` method after the fact. points_to_predict : array, or array-like (n_samples, n_features) The new data points to predict cluster labels for. They should have the same dimensionality as the original dataset over which clusterer was fit. n_clusters: int, default=None The number of clusters to have in the flat clustering (over the training data, not points_to_predict) Ignored when prediction_data is supplied. cluster_selection_epsilon: float, default=None core-distance below which to stop splitting clusters. This can indirectly impose n_clusters. This argument is ignored if n_clusters is supplied. prediction_data: PredictionData, default=None If supplied, use this to predict clusters for points_to_predict. This allows predicting on multiple datasets without corrupting prediction data associated with clusterer. If neither n_clusters, nor prediction_data are supplied, then the prediction_data associated with clusterer is used. return_prediction_data: bool, default=False If True, return prediction_data along with labels and proba. Returns ------- labels : array (n_samples,) The predicted labels of the ``points_to_predict`` probabilities : array (n_samples,) The soft cluster scores for each of the ``points_to_predict`` prediction_data: PredictionData, optional prediction_data used to predict. Returned if return_prediciton_data is set to True. Usage ----- # From a fitted HDBSCAN model, predict for n_clusters=5 labels, proba = approximate_predict_flat( clusterer, X_predict, n_clusters=5) # Store prediciton data for later use. labels, proba, pred_data = approximate_predict_flat( clusterer, X_predict, n_clusters=5, return_prediction_data=True) # and use this prediction data to predict on new points labels1, proba1 = approximate_predict_flat( clusterer, X_pred1, prediction_data=pred_data) See Also --------- :py:func:`hdbscan.prediction.approximate_predict` rNzClusterer does not have prediction data! Try fitting with prediction_data=True set, or run generate_prediction_data on the clustererrrrr)r+rrz-New points dimension does not match fit data!zfPrediction data does not have any defined clusters, new data will be automatically predicted as noise.Zdtypek)"npsumuniquer'intr!r)rr ValueErrorrrrrr(Zasarrayshaperaw_data cluster_treeZonesZint32zerosZfloat32emptyfloat64 min_samplesr%treequeryranger core_distances cluster_map max_lambdas)r,points_to_predictr+rrZreturn_prediction_dataZn_clusters_fitcondensed_treelabelsZ probabilitiesrDneighbor_distancesneighbor_indicesilabelZprobZ neighborsr1r1r2approximate_predict_flats~M          rRc Cs|tj}|j}|dkr8|dkr8|dkr8|j}|j}nt|dkr|dk rzt||}Wqtk rtd|dd|_ t||}YqXn|}t |j}t |||dt |}tj|jd|jdftjd}|jp|j} |jj|d | d \} } t|jdD]} t| | | | |j| \} }t|j| }|d |krH|d }t|| |j|j}t| |||j|j|j}|d |d || <|| || <|| t | |||j|j|j9<q|}|S)a  (Adaptation of hdbscan's membership_vector for n_clusters, epsilon) Predict soft cluster membership probabilities; a vector for each point in ``points_to_predict`` that gives a probability that the given point is a member of a cluster for each of the selected clusters of the ``clusterer``. Parameters ---------- clusterer: HDBSCAN A clustering object that has been fit to the data and either had ``prediction_data=True`` set, or called the ``generate_prediction_data`` method after the fact. points_to_predict: array, or array-like (n_samples, n_features) The new data points to predict cluster labels for. They should have the same dimensionality as the original dataset over which clusterer was fit. prediction_data: PredictionData, default=None Prediction data associated with HDBSCAN for some flat clustering n_clusters: int, default=None Number of clusters over which to compute membership probabilities. These clusters are obtained as a flat clustering at some cluster_selection_epsilon. cluster_selection_epsilon: float, default=0. core-distance below which to stop splitting clusters. This can indirectly impose n_clusters. This argument is ignored if n_clusters is supplied. Note: If neither n_clusters nor cluster_selection_epsilon are supplied, the clusterer's original clustering is used. Returns ------- membership_vectors : array (n_samples, n_clusters) The probability that point ``i`` is a member of cluster ``j`` is in ``membership_vectors[i, j]``. See Also -------- :py:func:`hdbscan.predict.membership_vector` :py:func:`hdbscan.predict.all_points_membership_vectors` NrFailed to predict E clusters with cluster selection method 'eom'. Switching to 'leaf'...rrrr5r6r7 lambda_valg?g@)!astyper9rCr!rr)r#AssertionErrorrrrrr(clusters_from_prediction_datarBr>rDr%rErFrGr rHr _raw_treer exemplars dist_metricrleaf_max_lambdasr@r:r)r,rKrr+rrLr.clustersresultrDrNrOrPZnearest_neighborZlambda_Zneighbor_tree_rowZ distance_vecZ outlier_vecmembership_vectorsr1r1r2membership_vector_flatjs2       r`cCs@|j}|dkr$|dkr$|j}|j}nt|dkr|dk rzzt||}Wq~tk rvtd|dd|_t||}Yq~Xn|}t|j}t |||dt |}|j }|j dkrt |jdSt||j|j}t||j|j|j} t||j|j|j} || } | jdd} | | ddt jf} | | ddt jf9} | } | S) a (Adaptation of hdbscan's all_points_membership_vector for n_clusters, epsilon) Predict soft cluster membership vectors for all points in the original dataset the clusterer was trained on. This function is more efficient by making use of the fact that all points are already in the condensed tree, and processing in bulk. Parameters ---------- clusterer : HDBSCAN A clustering object that has been fit to the data and either had ``prediction_data=True`` set, or called the ``generate_prediction_data`` method after the fact. This method does not work if the clusterer was trained with ``metric='precomputed'``. prediction_data: PredictionData, default=None Prediction data associated with HDBSCAN for some flat clustering n_clusters: int, optional, default=None Number of clusters over which to compute membership probabilities. These clusters are obtained as a flat clustering at some cluster_selection_epsilon. cluster_selection_epsilon: float, optional, default=None core-distance below which to stop splitting clusters. This can indirectly impose n_clusters. This argument is ignored if n_clusters is supplied. Note: If neither n_clusters nor cluster_selection_epsilon are supplied, the clusterer's original clustering is used. Returns ------- membership_vectors : array (n_samples, n_clusters) The probability that point ``i`` of the original dataset is a member of cluster ``j`` is in ``membership_vectors[i, j]``. See Also -------- :py:func:`hdbscan.prediction.all_points_membership_vectors` :py:func:`hdbscan.prediction.membership_vector` NrSrTrrrr)Zaxis)r!rr)r#rWrrrrr(rXr?sizer9rAr>rrZr[rrYr\r@rr:Znewaxis)r,rr+rrLr.r]Z all_pointsZ distance_vecsZ outlier_vecsZin_cluster_probsr^Zrow_sumsr_r1r1r2"all_points_membership_vectors_flatsX/   rbcCs6|j}|dkrt||S|dkr*t||StddS)z Pick optimal epsilon from condensed tree based on n_clusters, calls functions specific to 'eom' or 'leaf' selection methods rrEInvalid Cluster Selection Method: %s Should be one of: "eom", "leaf" N)rselect_epsilon_eomselect_epsilon_leafr=)rLr+rr1r1r2r#zs   r#c Cs|}t|}||ks,td|dd|j}|d|ddk}dt|d}t|d d d }|D]}t||}t||krpqqptd |S) z Select epsilon so that persistence-based clustering, after truncating the tree at the above epsilon, has exactly 'n_clusters' clusters zCannot produce more than z with method 'eom'. z5Use method 'leaf' instead to extract flat clustering.rU child_sizer?-q=Nr4zCould not find epsilon) r"r rWrYr9r;sort_new_select_clusters RuntimeError) rLr+Zeom_base_clustersZ max_clustersrEZcluster_lambdasZcandidate_epsilonsr.Z sel_clustersr1r1r2rds"    rdcCs|jd}|jd}|t}||dk}t|t}|t|dkr|tdt|ddt|ddt|d}d||d}|d S) z Select epsilon so that the leaves of condensed tree, after truncating at the above epsilon, has exactly 'n_clusters' clusters rUrfrzHDBSCAN can only compute z! clusters. Setting n_clusters to z...rgr6rh)rYrVr<r9r;r$r r)rLr+ZlambdasZ child_sizesr.r1r1r2res    & rec Cs4|dk rt||}t||}|j}ddttt|D|_dd|jD|_i|_ g|_ |D]}|d|d|k |j |<| |D]$}|j||j|<|j ||j |<qt jgt jd}||D]J} |d|d| k } |d|d| k|d| k@} t || g}q|j |j|qddS) ap Modify PredictionData of HDBSCAN to account for epsilon. epsilon is the cluster_selection_epsilon that controls granularity of clusters; Large epsilon => More clusters Parameters ---------- predData: PredictionData Contains data to use for predicting novel points. Defined in the HDBSCAN module condensed_tree: CondensedTree Tree structure that contains hierarchical clustering. Defined in the HDBSCAN module n_clusters: int, optional, default=None If specified, use this to obtain cluster_selection_epsilon from CondensedTree; Overrides cluster_selection_epsilon parameter cluster_selection_epsilon: float, default=0. In cluster tree, nodes are not split further beyond (>=) this value. epsilon is the inverse of core distance. Returns ------- None NcSsi|]\}}t||qSr1r<).0ncr1r1r2 szre_init..cSsi|]\}}||qSr1r1)rmrornr1r1r2rpsrUparentr5child)r#rjrY enumeratesortedlistrIitemsreverse_cluster_maprJrZmaxZ_clusters_belowr9arrayZint64Z_recurse_leaf_dfsZhstackappendr?) ZpredDatarLr+rselected_clustersZraw_condensed_treeclusterZ sub_clusterZcluster_exemplarsrZleaf_max_lambdaZpointsr1r1r2r(sF     r(cs|j}|j}t||r*tdd}ntdddd}||ddk}dd|D|d kr6|D]n}|d |k} tfd d |d | D} | |krd|<| |<qnt||D]} | |krd| <qqn|dkrtfdd D} t | |||} D]"}|| kr(d|<nd|<qn|dkrtt |}t |dkr~D]}d|<q^d|d  <|dkrt ||||} n|} D]"}|| krd|<nd|<qnt dtfdd D}|S)z Adaptation of get_clusters from hdbscan._hdbscan_tree. Avoids the label and proba computation at the end, and returns only the selected clusters instead. T)reverseNr4rfrcSsi|] }|dqS)Tr1)rmr|r1r1r2rp:sz(_new_select_clusters..rrqcsg|] }|qSr1r1)rmrr) stabilityr1r2 ?sz(_new_select_clusters..rrFrcsg|]}|r|qSr1r1rmro is_clusterr1r2rKsrrrccsg|]}|rt|qSr1rlrrr1r2rls)rYrrrtkeysr9r:rsetepsilon_searchrr minr=)rLrr&r/rErZ node_listr@nodeZchild_selectionZsubtree_stabilitysub_nodeZ eom_clustersr{roleavesr]r1)rr~r2rj&sb                rjc Csg}g}|D]}d|d|d|kd}||kr||krt||||}t|dr\|d}||t||D]} | |krp|| qpq ||q t|S)NrrUrrr__len__)traverse_upwardshasattrrzrr) rr@rr&r{ processedrepsZ epsilon_childrr1r1r2rps,   rcCsj|d}||d|kd}||kr4|r0|S|Sd||d|kd}||krX|St||||SdS)NrqrrrrU)rr)r@rrr&rootrqZ parent_epsr1r1r2rs rcCs ttt|jtjS)zA Extract selected clusters from PredictionData instance. )r9ryrtrurwvaluesrVZintp)rr1r1r2rXs rX)NrNF)NNNF)NNr)NNN)Nr)FF)&__doc__rwarningsrnumpyr9Z _hdbscan_treerrZhdbscan_rrZplotsrZ predictionr r r Z_prediction_utilsr r rrrrrr3rRr`rbr#rdrer(rjrrrXr1r1r1r2sR  $   (  w"# \ J