U 9hid@sddlZddlmZmZddlmZddlmZddl m Z m Z m Z m Z mZmZmZddlmZGdd d eZd d Zd d ZddZdddZddZddZddZdS)N)KDTreeBallTree)DistanceMetric)recurse_leaf_dfs)get_tree_row_with_childdist_membership_vectoroutlier_membership_vectorprob_in_some_cluster!all_points_dist_membership_vector$all_points_outlier_membership_vectorall_points_prob_in_some_cluster)warnc@s4eZdZdZeedZddZddZd d d Z d S) PredictionDataa Extra data that allows for faster prediction if cached. Parameters ---------- data : array (n_samples, n_features) The original data set that was clustered condensed_tree : CondensedTree The condensed tree object created by a clustering min_samples : int The min_samples value used in clustering tree_type : string, optional Which type of space tree to use for core distance computation. One of: * ``kdtree`` * ``balltree`` metric : string, optional The metric used to determine distance for the clustering. This is the metric that will be used for the space tree to determine core distances etc. **kwargs : Any further arguments to the metric. Attributes ---------- raw_data : array (n_samples, n_features) The original data set that was clustered tree : KDTree or BallTree A space partitioning tree that can be queried for nearest neighbors. core_distances : array (n_samples,) The core distances for every point in the original data set. cluster_map : dict A dictionary mapping cluster numbers in the condensed tree to labels in the final selected clustering. cluster_tree : structured array A version of the condensed tree that only contains clusters, not individual points. max_lambdas : dict A dictionary mapping cluster numbers in the condensed tree to the maximum lambda value seen in that cluster. )kdtreeZballtreecCsBg}|g}|r>|||jdt|jd|}|}q |S)Nchildparent)extend cluster_treenpisintolist)selfclusterresultZ to_processre/var/www/html/CrowdFlow/HYROX/ble_analysis_env_py38/lib/python3.8/site-packages/hdbscan/prediction.py_clusters_belowMs  zPredictionData._clusters_belowcsFjjd|kd}t|dkr*|gStfdd|DgSdS)Nrrrcsg|]}tj|qSr)rr).0rrrr `sz4PredictionData._recurse_leaf_dfs..)rlensum)rZ current_nodechildrenrrr_recurse_leaf_dfsZs z PredictionData._recurse_leaf_dfsr euclideancKs|tj|_|j||jfd|i||_|jj||dddddf|_tj |f||_ t | }|j }ddtt t|D|_dd|jD|_||dd k|_i|_i|_g|_tt|jd |jd g} | D]"} |d |d | k|j| <q|D]} |d |d | k|j| <|| D]&} |j| |j| <|j| |j| <qnsz+PredictionData.__init__..cSsi|]\}}||qSrr)rr+r*rrrr,osZ child_sizerrr lambda_valZdtype)astyperfloat64raw_data_tree_type_maptreequerycore_distancesrZ get_metric dist_metricsorted_select_clusters _raw_tree enumeratelist cluster_mapitemsZreverse_cluster_mapr max_lambdasleaf_max_lambdas exemplarssetZhstackmaxrarrayZint64r$append)rdataZcondensed_tree min_samplesZ tree_typer&kwargsZselected_clustersZraw_condensed_treeZ all_clustersrZ sub_clusterZcluster_exemplarsleafZleaf_max_lambdaZpointsrrr__init__bsR"     zPredictionData.__init__N)rr%) __name__ __module__ __qualname____doc__rrr2rr$rIrrrrrs5   rc Csx||}||t|jd}t|||fjdd}|}||}||dkrbd||} nttjj} || fS)ax Find the nearest mutual reachability neighbor of a point, and compute the associated lambda value for the point, given the mutual reachability distance to a nearest neighbor. Parameters ---------- neighbor_indices : array (2 * min_samples, ) An array of raw distance based nearest neighbor indices. neighbor_distances : array (2 * min_samples, ) An array of raw distances to the nearest neighbors. core_distances : array (n_samples, ) An array of core distances for all points min_samples : int The min_samples value used to generate core distances. Returns ------- neighbor : int The index into the full raw data set of the nearest mutual reachability distance neighbor of the point. lambda_ : float The lambda value at which this point joins/merges with `neighbor`. rZaxis?)ronesshapeZvstackrBZargminZfinfodouble) neighbor_indicesneighbor_distancesr5rFZneighbor_core_distancesZpoint_core_distancesZ mr_distancesZnn_indexnearest_neighborlambda_rrr_find_neighbor_and_lambdas"  rXc Cs|d}t||||\}}t||}|d} |d|krN| dd|df} nF| |kr||d| kd|kr|d|d| kd} qN| dd|f} t|| S)a Create a new condensed tree with an additional point added, allowing for computations as if this point had been part of the original tree. Note that this makes as little change to the tree as possible, with no re-optimizing/re-condensing so that the selected clusters remain effectively unchanged. Parameters ---------- tree : structured array The raw format condensed tree to update. neighbor_indices : array (2 * min_samples, ) An array of raw distance based nearest neighbor indices. neighbor_distances : array (2 * min_samples, ) An array of raw distances to the nearest neighbors. core_distances : array (n_samples, ) An array of core distances for all points min_samples : int The min_samples value used to generate core distances. Returns ------- new_tree : structured array The original tree with an extra row providing the parent cluster and lambda information for a new point given index -1. rr-r)rrr)minrXrrrD) r3rTrUr5rF tree_rootrVrWneighbor_tree_rowpotential_clusterZ new_tree_rowrrr_extend_condensed_trees0    r]cCs|j}|d} t||||\} } t|| } | d} | d| kr|| | kr||d|d| k| kr||d|d| kd} qB| |kr|| }nd}|dkr|| }|dkrt|| } | |}qd}nd}||| fS)aY Return the cluster label (of the original clustering) and membership probability of a new data point. Parameters ---------- tree : CondensedTree The condensed tree associated with the clustering. cluster_tree : structured_array The raw form of the condensed tree with only cluster information (no data on individual points). This is significantly more compact. neighbor_indices : array (2 * min_samples, ) An array of raw distance based nearest neighbor indices. neighbor_distances : array (2 * min_samples, ) An array of raw distances to the nearest neighbors. core_distances : array (n_samples, ) An array of core distances for all points cluster_map : dict A dictionary mapping cluster numbers in the condensed tree to labels in the final selected clustering. max_lambdas : dict A dictionary mapping cluster numbers in the condensed tree to the maximum lambda value seen in that cluster. min_samples : int The min_samples value used to generate core distances. rr-rrr)rOrP)r9rYrXr)r3rrTrUr5r<r>rFZraw_treerZrVrWr[r\Z cluster_label max_lambdaprobrrr_find_cluster_and_probabilitysB%        r`Fc Cs|jdkrtdt|}|jd|jjjdkr) clustererpoints_to_predictZreturn_connecting_pointslabelsZ probabilitiesZ neighborsrFrUrTilabelr_Zneighborrrrapproximate_predictKsT0        rrcCsz |jWntk r&tdYnXt|}|jd|jjjdkrRtd|jjjddkrtdtj |jdtj d}|Stj |jdtj d}|j p|j}|jjj|d|d\}}|jj}|d }|}i} t|d D] } ||d | kd | | <qt|D]H} |d | } | |kr8qb|| } | | | | kr| | | | <qt|jdD]|} t|| || |jj|\}}t||}|d }|| dkr|d }| |}|d kr||||| <nd || <qp|S) aPredict the outlier score of new points. The returned scores will be based on the original clustering found by ``clusterer``, and therefore are not (necessarily) the outlier scores that would be found by clustering the original data combined with ``points_to_predict``, hence the 'approximate' label. If you simply wish to calculate the outlier scores for new points in the 'best' way possible, this is the function to use. If you want to predict the outlier score of ``points_to_predict`` with the original data under HDBSCAN the most efficient existing approach is to simply recluster with the new point(s) added to the original dataset. Parameters ---------- clusterer : HDBSCAN A clustering object that has been fit to the data and either had ``prediction_data=True`` set, or called the ``generate_prediction_data`` method after the fact. points_to_predict : array, or array-like (n_samples, n_features) The new data points to predict cluster labels for. They should have the same dimensionality as the original dataset over which clusterer was fit. Returns ------- scores : array (n_samples,) The predicted scores of the ``points_to_predict`` See Also -------- :py:func:`hdbscan.predict.membership_vector` :py:func:`hdbscan.predict.all_points_membership_vectors` rarrbrzcClusterer does not have any defined clusters, new data will be automatically predicted as outliers.r.rcr'rr-rrO)rdAttributeErrorrerrfrRr1rrrQrgrir0rFrjr3r4rlr9rYuniquerBZargsortrkrXr5r)rmrnZscoresrFrUrTr3Z parent_arrayrZr>rr*rrpZneighrWr[r\r^rrrapproximate_predict_scoress`$          ruc CsV|tj}ttt|jtj}tj |j d|j dftjd}|j pV|j }|j jj|d|d\}}t|j dD]}t|||||j j|\}} t|jj|} | d| kr| d} t|||j j|j j} t|| ||jj|j j|j j} | d| d||<||||<||t|| ||jj|j j|j j9<q|S)aPredict soft cluster membership. The result produces a vector for each point in ``points_to_predict`` that gives a probability that the given point is a member of a cluster for each of the selected clusters of the ``clusterer``. Parameters ---------- clusterer : HDBSCAN A clustering object that has been fit to the data and either had ``prediction_data=True`` set, or called the ``generate_prediction_data`` method after the fact. points_to_predict : array, or array-like (n_samples, n_features) The new data points to predict cluster labels for. They should have the same dimensionality as the original dataset over which clusterer was fit. Returns ------- membership_vectors : array (n_samples, n_clusters) The probability that point ``i`` is a member of cluster ``j`` is in ``membership_vectors[i, j]``. See Also -------- :py:func:`hdbscan.predict.predict` :py:func:`hdbscan.predict.all_points_membership_vectors` rr.rcr'r-g?g@)r/rr0rCr7r;rlr8intprirRrFrjrdr3r4rkrXr5rr9rr@r6r r?rr"r ) rmrnclustersrrFrUrTrprVrWr[Z distance_vecZ outlier_vecrrrmembership_vectorsh      rxcCsttt|jtj}|jj }|j dkrBt |j dSt ||jj|jj}t||jj|jj|jj}t||jj|jj|jj}||}|jdd}||ddtjf}||ddtjf9}|S)aPredict soft cluster membership vectors for all points in the original dataset the clusterer was trained on. This function is more efficient by making use of the fact that all points are already in the condensed tree, and processing in bulk. Parameters ---------- clusterer : HDBSCAN A clustering object that has been fit to the data and either had ``prediction_data=True`` set, or called the ``generate_prediction_data`` method after the fact. This method does not work if the clusterer was trained with ``metric='precomputed'``. Returns ------- membership_vectors : array (n_samples, n_clusters) The probability that point ``i`` of the original dataset is a member of cluster ``j`` is in ``membership_vectors[i, j]``. See Also -------- :py:func:`hdbscan.predict.predict` :py:func:`hdbscan.predict.all_points_membership_vectors` rrrNN)rrCr7r;rlr8r/rvrdr1sizerhrRr r@r6r r9r?rr r"Znewaxis)rmrwZ all_pointsZ distance_vecsZ outlier_vecsZin_cluster_probsrZrow_sumsrrrall_points_membership_vectorsms4   rz)F)numpyrZsklearn.neighborsrrZ dist_metricsrZ _hdbscan_treerZ_prediction_utilsrrr r r r r warningsrobjectrrXr]r`rrrurxrzrrrrs  $ }2;L diU