CC BY
13МЕЖДУНАРОДНАЯ НАУЧНО-ПРАКТИЧЕСКАЯ КОНФЕРЕНЦИЯ «НАУКА И ТЕХНИКА В XXI ВЕКЕ»
УДК 004.912
Mukhanova M.B.
2nd year master's student of the Department of Artificial Intelligence and BigData
Al-Farabi Kazakh National University (Kazakhstan, Almaty)
LITERATURE SURVEY OF DEEP LEARNING IN NATURAL LANGUAGE PROCESSING
Abstract: This paper underlines the necessity to incorporate Deep learning and Neural networking in language models under scrutiny for Natural Language Processing. The paper describes various statistical models proposed and the limitations incurred in the same due to limited intelligence of a machine. We have discussed different neural networks highlighting the importance of Convolutional Neural Networking. We have discussed open source software TensorFlow that works on Deep learning and the edge it has over the conventional models.
Keywords: natural language processing, Neural Networks, deep learning, TensorFlow.
Introduction
Natural Language Processing (NLP) is one of the dominant fields in data mining. With the increasing importance of Big Data Analytics today, NLP plays a major role in acquiring relevant information of importance to business and intelligence. Millions of items are uploaded on the Web everyday, with relevant as well as irrelevant data. Information retrieval and extraction from reviews, comments, social media etc by customers is a complex task since most of the information is in semistructured and unstructured form. Ambiguity of large corpora on the Web underlines the need for decent and efficient data mining techniques.
The branch of NLP predominantly works to analyze, summarize or retrieve pertinent information from the large pool of data available. Exploration in this field
311
dates back to 1950 when Turing's article on 'Computing Machinery and Intelligence' was published [1] and Message Understanding Conferences in '90s. NLP requires a combination of linguistics and computational knowledge. It can be done for various languages. For English, various problems incurred during information extraction include paraphrasing, idioms, rhetoric, metaphors etc. [2]
Deep Learning and Neural networks are gaining importance in the field of NLP with hidden states between the input and output and extensive networking to provide best results [3].
In Recursive Neural Network, semantics are isolated via tree structures. Since textual tree construction can be time consuming for long sentences, it is inefficient. Recurrent Neural Networks can extract contextual information by utilizing stored previous text in the form of fixed sized hidden layers. The problem with the same is its bias towards the end of the document. Hence keywords in the other parts of the document will be ignored.
One of the best alternatives in neural networks is Convolutional Neural Network is an unbiased model that uses convolutional kernels as a part of its deep learning architecture. 3 layers of CNN are [4]:
1. Convolution layer
2. Pooling layer
3. Activation layer (fully-connected)
Deep learning in CNN is achieved with convolving filters of variable widths and feature map. Pooling is responsible for downsampling of the matrix from filters whereas a Fully-connected layer computes class score [5]. Deep neural networks open source software TensorFlow has been proposed for application in [Paul] Youtube Recommendation with the help of matrix factorization approach in minimizing cross entropy loss.
In November 2015, Google open sourced TensorFlow, which is one of the projects under Googlebrain. TensorFlow is an open source software library for machine learning which is used by Google for many of Google products, such as speech
recognition, Gmail, Google Photos etc. TensorFlow is now being widely used for research purposes, creating a number of useful applications. It runs on multiple CPUs and GPUs (with optional CUDA extensions for general-purpose computing on graphics processing units).It can work on different platforms like Linux, Windows and Mac OSX .It also works on Android and Apple's iOS [6]. To understand how it works we must first understand what "Tensor'' is. So, first, we recall matrix multiplication,
which is given as {
v[x]^-vector is a simple array of one dimension m[x][y][z]^matrix (is a 2 Or 3 dimensional) t[x][y][z][?][?]..^ tensor (is arbitrary large number of dimension) } TensorFlow [7] is based on Deep Learning of Neural networks such that the input is given as a tensor and then that tensor flows through nodes in the neural network adding some weight to it and the softmax function in the final layer of the neural networks. TensorFlow [8,9] library can easily be downloaded and installed in your system and coding in tensor flow is done in python .So TensorFlow works with the python API (compatible with python or python3).It is loaded up with many different packages like speech recognition and image recognition etc.
In conclusion, neural networks and deep learning resolve most of the problems incurred in NLP. The hidden states between input word and output vector form an intensive network for thorough and efficient learning. This technology can be used as the backbone of Artificial Intelligence. Future works to be done in this field include Cross Language IR and machine-human dialog.
REFERENCES:
Jona, "Natural Language Processing",
https://en. wikipedia. org/wiki/Natural_language_processing.
Paul Shoebotttom, "Syntax- English sentence structure", esl.fis. edu/learners/advice/syntax. htm.
Siwei Lai, Liheng Xu, Kang Liu, Jun Zhao, " Recurrent Convolutional Neural Networks for Text Classification, Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence", pp 2267- 2273,2015
Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, "Recent Advances in Convolutional Neural Networks" , arXiv: 1512-07108v5 ,pp 1-37,2017.
Karpathy, "Convolutional Neural Networks (CNNs / ConvNets)" , css\n. github.io/convolutional-networks/
Ancheta Wis, "TensorFlow", https://en. wikipedia.org/wiki/TensorFlo w#TensorFlow Wolfram MathWorld, "Tensor", http://mathworld.wolfram.com/Tensor.html Mart'm Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado et al., "TensorFlow: LargeScale Machine Learning on Heterogeneous Distributed Systems", Preliminary White Paper, pp 1-19, 2015. TensorFlow,"An open source software library for Machine Intelligence", https://www.tensorflow.org/
