In view of the advantages of deep learning model in the extraction of abstract concept, a new text clustering algorithm is designed based on Deep Boltzmann Machines. Based on Replicate Softmax Model and new Deep Boltzmann Machine, energy function of this model is proposed and the detail learning algorithm is introduced. The learning can be made more efficient by using a layer-by-layer “pre-training” phase that allows variation inference to be initialized with a single bottom up pass. The values of the latent variables in the deepest layer are easy to infer and give a much better representation of each document than low learning. The 20-newsgroups document sets experiment results illustrated that the novel algorithm learn good generative models, get the better competence of a shallow modelReplicate Softmax Model in handling with an extract abstract concept and has good feasibility in large scale text clustering analysis. Introduction Text Clustering is mainly based on the famous retrieval hypothesis: the same kind of document similarity greater without the same kind of document similarity small [1]. As an unsupervised machine learning methods, retrieval does not require advance documentation manual annotation categories because of no training process, so it has a higher degree of flexibility and automation capabilities, text messaging has become effectively an important means of organization, summary and navigation, as more and more researchers are concerned [2]. Deep Learning networks are a new type of neural network that can discovers more important object features [3]. These networks determine features adept at learning high level abstractions about their datasets without supervision. It has been successfully applied in various application domains [4]. Given the development of deep learning model, based on a number of distribution and use of text analysis model -RSM (Replicated Softmax Model, RSM) [5], this paper designs based on its depth Boltzmann machine (Deep Boltzmann Machine, DBM) [6] for text feature extraction, combined with K-mean clustering algorithm to text clustering. Background: DBMs and K-means Deep Boltzmann Machines. A deep Boltzmann machine (DBM) is a probabilistic model consisting of many layers of random variables, most of which are latent binary units and also can be seen as a network of symmetrically coupled stochastic binary units. There are connections only between hidden units in adjacent layers. For example, we design a DBM in two hidden layers that contains a set of visible units { } 0,1 D ∈ v , and a sequence of layer of hidden units ( ) { } 1 1 0,1 F ∈ h , ( ) { } 2 2 0,1 F ∈ h , as shown in Fig.2. The energy of the joint configuration { } , v h is defined as (ignoring bias terms): ( ) ( ) ( ) ( ) ( ) ( ) ; E θ = 1 1 1 T 2 2 T v, h -v W h h W h (1) Where ( ) ( ) { } 1 2 , = h h h are the set of hidden units, and ( ) ( ) { } 1 2 , θ = W W are the model parameters, representing visible-to-hidden and hidden-to-hidden symmetric interaction terms. The probability International Symposium on Computers & Informatics (ISCI 2015) © 2015. The authors Published by Atlantis Press 1328 that the model assigns to a visible vector v is: ( ) ( ) ( ) ( ) ( ) ( ) * 1 2 ; 1 ; exp , , ; P P E Z Z θ θ θ θ θ = = − ∑ h v v v h h (2) Where ( ) ( ) ( ) exp ; Z E θ θ = − ∑ 1 2 v,h v, h , h represents the partition function. We use a greedy layer-wise pre-training strategy to initialize the model parameters to good value and employ Stochastic Approximation to fine-tune the parameters after the previous stage. Replicated Softmax Model. The Replicated Softmax Model is useful for modeling word count vectors in a document. Let K ∈ v � be a vector of visible units where k v is the number of times word k occurs in the document with the vocabulary of size K and { } 0,1 F ∈ h seemed as be binary stochastic hidden topic features. The energy of the state { } , v h is defined as follows ( ) 1 1 1 1 , ; K F K F k kj j k k j j k j k j E v h v W h b v M a h θ = = = = = − − − ∑∑ ∑ ∑ (3) Where { } , , θ = a b W are the model parameters and k k M v =∑ is the total number of words in a document. K-means. K-means clustering [7, 8] is the most famous division clustering algorithm and popular for cluster analysis in data mining. Given a set of data points and the required number of clusters K, this algorithm aims to partition into K clusters in which each observation belongs to the nearest mean. We first select K objects as the initial cluster centers randomly, then calculate the distance of each object and each seed cluster centers, assign each object to its distance from the nearest cluster center. Once all objects have been assigned, the center of each cluster will be recalculated based on the existing cluster objects. This process is repeated until a termination condition is satisfied. The termination condition may be any of the following: (1) No (or minimal number) of an object to be re-assigned to different clusters. (2) There is no (or minimal number) of the cluster centers vary. (3) Local minimum squared error. Algorithm Design Through the introduction of K-means and mechanisms for DBM structure analysis, we exchange the lowest unit for RSM visible cells so to constitute Deep Boltzmann machine model based on Replicated Softmax Model, the principle mechanism for this model are as follows, we make the underlying characteristics as the top RBM input, to enhance the ability of the DBM feature extraction by fine-tuning parameters and weights biased to achieve automatic feature extraction purposes. In a given sample{ } 1 2 , ,..., K v v v , the probability of the entire model, such as follows ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 2 1 1 2 , 1 1 , , ; P P P θ = ∑ ∑
Paper
Full text
Text Retrieval analysis based on Deep Learning
Semantic Scholar · Computer Science · 2015
Abstract
In view of the advantages of deep learning model in the extraction of abstract concept, a new text clustering algorithm is designed based on Deep Boltzmann Machines. Based on Replicate Softmax Model and new Deep Boltzmann Machine, energy function of this model is proposed and the detail learning algorithm is introduced. The learning can be made more efficient by using a layer-by-layer “pre-training” phase that allows variation inference to be initialized with a single bottom up pass. The values of the latent variables in the deepest layer are easy to infer and give a much better representation of each document than low learning. The 20-newsgroups document sets experiment results illustrated that the novel algorithm learn good generative models, get the better competence of a shallow modelReplicate Softmax Model in handling with an extract abstract concept and has good feasibility in large scale text clustering analysis. Introduction Text Clustering is mainly based on the famous retrieval hypothesis: the same kind of document similarity greater without the same kind of document similarity small [1]. As an unsupervised machine learning methods, retrieval does not require advance documentation manual annotation categories because of no training process, so it has a higher degree of flexibility and automation capabilities, text messaging has become effectively an important means of organization, summary and navigation, as more and more researchers are concerned [2]. Deep Learning networks are a new type of neural network that can discovers more important object features [3]. These networks determine features adept at learning high level abstractions about their datasets without supervision. It has been successfully applied in various application domains [4]. Given the development of deep learning model, based on a number of distribution and use of text analysis model -RSM (Replicated Softmax Model, RSM) [5], this paper designs based on its depth Boltzmann machine (Deep Boltzmann Machine, DBM) [6] for text feature extraction, combined with K-mean clustering algorithm to text clustering. Background: DBMs and K-means Deep Boltzmann Machines. A deep Boltzmann machine (DBM) is a probabilistic model consisting of many layers of random variables, most of which are latent binary units and also can be seen as a network of symmetrically coupled stochastic binary units. There are connections only between hidden units in adjacent layers. For example, we design a DBM in two hidden layers that contains a set of visible units { } 0,1 D ∈ v , and a sequence of layer of hidden units ( ) { } 1 1 0,1 F ∈ h , ( ) { } 2 2 0,1 F ∈ h , as shown in Fig.2. The energy of the joint configuration { } , v h is defined as (ignoring bias terms): ( ) ( ) ( ) ( ) ( ) ( ) ; E θ = 1 1 1 T 2 2 T v, h -v W h h W h (1) Where ( ) ( ) { } 1 2 , = h h h are the set of hidden units, and ( ) ( ) { } 1 2 , θ = W W are the model parameters, representing visible-to-hidden and hidden-to-hidden symmetric interaction terms. The probability International Symposium on Computers & Informatics (ISCI 2015) © 2015. The authors Published by Atlantis Press 1328 that the model assigns to a visible vector v is: ( ) ( ) ( ) ( ) ( ) ( ) * 1 2 ; 1 ; exp , , ; P P E Z Z θ θ θ θ θ = = − ∑ h v v v h h (2) Where ( ) ( ) ( ) exp ; Z E θ θ = − ∑ 1 2 v,h v, h , h represents the partition function. We use a greedy layer-wise pre-training strategy to initialize the model parameters to good value and employ Stochastic Approximation to fine-tune the parameters after the previous stage. Replicated Softmax Model. The Replicated Softmax Model is useful for modeling word count vectors in a document. Let K ∈ v � be a vector of visible units where k v is the number of times word k occurs in the document with the vocabulary of size K and { } 0,1 F ∈ h seemed as be binary stochastic hidden topic features. The energy of the state { } , v h is defined as follows ( ) 1 1 1 1 , ; K F K F k kj j k k j j k j k j E v h v W h b v M a h θ = = = = = − − − ∑∑ ∑ ∑ (3) Where { } , , θ = a b W are the model parameters and k k M v =∑ is the total number of words in a document. K-means. K-means clustering [7, 8] is the most famous division clustering algorithm and popular for cluster analysis in data mining. Given a set of data points and the required number of clusters K, this algorithm aims to partition into K clusters in which each observation belongs to the nearest mean. We first select K objects as the initial cluster centers randomly, then calculate the distance of each object and each seed cluster centers, assign each object to its distance from the nearest cluster center. Once all objects have been assigned, the center of each cluster will be recalculated based on the existing cluster objects. This process is repeated until a termination condition is satisfied. The termination condition may be any of the following: (1) No (or minimal number) of an object to be re-assigned to different clusters. (2) There is no (or minimal number) of the cluster centers vary. (3) Local minimum squared error. Algorithm Design Through the introduction of K-means and mechanisms for DBM structure analysis, we exchange the lowest unit for RSM visible cells so to constitute Deep Boltzmann machine model based on Replicated Softmax Model, the principle mechanism for this model are as follows, we make the underlying characteristics as the top RBM input, to enhance the ability of the DBM feature extraction by fine-tuning parameters and weights biased to achieve automatic feature extraction purposes. In a given sample{ } 1 2 , ,..., K v v v , the probability of the entire model, such as follows ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 2 1 1 2 , 1 1 , , ; P P P θ = ∑ ∑