Skip to content

Commit f23190d

Browse files
kk7nckk7nc
authored
Update README.rst
1 parent 5ede6a6 commit f23190d

1 file changed

Lines changed: 279 additions & 0 deletions

File tree

‎README.rst‎

Lines changed: 279 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1009,8 +1009,150 @@ Deep Learning
10091009
Deep Neural Networks
10101010
-----------------------------------------
10111011

1012+
Deep Neural Networks' architecture is designed to learn by multi connection of layers that each single layer only receives connection from previous and provides connections only to the next layer in hidden part. The input is a connection of feature space (As discussed in Section Feature_extraction with first hidden layer. For Deep Neural Networks (DNN), input layer could be tf-ifd, word embedding, or etc. as shown in standard DNN in Figure~\ref{fig:DNN}. The output layer is number of classes for multi-class classification and only one output for binary classification. But our main contribution of this paper is that we have many training DNN for different purposes. In our techniques, we have multi-classes DNNs which each learning models is generated randomly (number of nodes in each layer and also number of layers are completely random assigned). Our implementation of Deep Neural Networks (DNN) is discriminative trained model that uses standard back-propagation algorithm using sigmoid or ReLU as activation function. The output layer for multi-class classification, should use Softmax.
1013+
1014+
10121015
.. image:: docs/pic/DNN.png
10131016

1017+
import packages:
1018+
1019+
.. code:: python
1020+
1021+
from sklearn.datasets import fetch_20newsgroups
1022+
from keras.layers import Dropout, Dense
1023+
from keras.models import Sequential
1024+
from sklearn.feature_extraction.text import TfidfVectorizer
1025+
import numpy as np
1026+
from sklearn import metrics
1027+
1028+
1029+
convert text to TF-IDF:
1030+
1031+
.. code:: python
1032+
1033+
def TFIDF(X_train, X_test,MAX_NB_WORDS=75000):
1034+
vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS)
1035+
X_train = vectorizer_x.fit_transform(X_train).toarray()
1036+
X_test = vectorizer_x.transform(X_test).toarray()
1037+
print("tf-idf with",str(np.array(X_train).shape[1]),"features")
1038+
return (X_train,X_test)
1039+
1040+
1041+
Build a DNN Model for Text:
1042+
1043+
.. code:: python
1044+
1045+
def Build_Model_DNN_Text(shape, nClasses, dropout=0.5):
1046+
"""
1047+
buildModel_DNN_Tex(shape, nClasses,dropout)
1048+
Build Deep neural networks Model for text classification
1049+
Shape is input feature space
1050+
nClasses is number of classes
1051+
"""
1052+
model = Sequential()
1053+
node = 512 # number of nodes
1054+
nLayers = 4 # number of hidden layer
1055+
1056+
model.add(Dense(node,input_dim=shape,activation='relu'))
1057+
model.add(Dropout(dropout))
1058+
for i in range(0,nLayers):
1059+
model.add(Dense(node,input_dim=node,activation='relu'))
1060+
model.add(Dropout(dropout))
1061+
model.add(Dense(nClasses, activation='softmax'))
1062+
1063+
model.compile(loss='sparse_categorical_crossentropy',
1064+
optimizer='adam',
1065+
metrics=['accuracy'])
1066+
1067+
return model
1068+
1069+
1070+
1071+
Load text dataset (20newsgroups):
1072+
1073+
.. code:: python
1074+
1075+
newsgroups_train = fetch_20newsgroups(subset='train')
1076+
newsgroups_test = fetch_20newsgroups(subset='test')
1077+
X_train = newsgroups_train.data
1078+
X_test = newsgroups_test.data
1079+
y_train = newsgroups_train.target
1080+
y_test = newsgroups_test.target
1081+
1082+
1083+
1084+
run DNN and see our result:
1085+
1086+
1087+
.. code:: python
1088+
1089+
X_train_tfidf,X_test_tfidf = TFIDF(X_train,X_test)
1090+
model_DNN = Build_Model_DNN_Text(X_train_tfidf.shape[1], 20)
1091+
model_DNN.fit(X_train_tfidf, y_train,
1092+
validation_data=(X_test_tfidf, y_test),
1093+
epochs=10,
1094+
batch_size=128,
1095+
verbose=2)
1096+
1097+
predicted = model_DNN.predict(X_test_tfidf)
1098+
1099+
print(metrics.classification_report(y_test, predicted))
1100+
1101+
1102+
1103+
Output:
1104+
1105+
::
1106+
1107+
Train on 11314 samples, validate on 7532 samples
1108+
Epoch 1/10
1109+
- 16s - loss: 2.7553 - acc: 0.1090 - val_loss: 1.9330 - val_acc: 0.3184
1110+
Epoch 2/10
1111+
- 15s - loss: 1.5330 - acc: 0.4222 - val_loss: 1.1546 - val_acc: 0.6204
1112+
Epoch 3/10
1113+
- 15s - loss: 0.7438 - acc: 0.7257 - val_loss: 0.8405 - val_acc: 0.7499
1114+
Epoch 4/10
1115+
- 15s - loss: 0.2967 - acc: 0.9020 - val_loss: 0.9214 - val_acc: 0.7767
1116+
Epoch 5/10
1117+
- 15s - loss: 0.1557 - acc: 0.9543 - val_loss: 0.8965 - val_acc: 0.7917
1118+
Epoch 6/10
1119+
- 15s - loss: 0.1015 - acc: 0.9705 - val_loss: 0.9427 - val_acc: 0.7949
1120+
Epoch 7/10
1121+
- 15s - loss: 0.0595 - acc: 0.9835 - val_loss: 0.9893 - val_acc: 0.7995
1122+
Epoch 8/10
1123+
- 15s - loss: 0.0495 - acc: 0.9866 - val_loss: 0.9512 - val_acc: 0.8079
1124+
Epoch 9/10
1125+
- 15s - loss: 0.0437 - acc: 0.9867 - val_loss: 0.9690 - val_acc: 0.8117
1126+
Epoch 10/10
1127+
- 15s - loss: 0.0443 - acc: 0.9880 - val_loss: 1.0004 - val_acc: 0.8070
1128+
1129+
1130+
precision recall f1-score support
1131+
1132+
0 0.76 0.78 0.77 319
1133+
1 0.67 0.80 0.73 389
1134+
2 0.82 0.63 0.71 394
1135+
3 0.76 0.69 0.72 392
1136+
4 0.65 0.86 0.74 385
1137+
5 0.84 0.75 0.79 395
1138+
6 0.82 0.87 0.84 390
1139+
7 0.86 0.90 0.88 396
1140+
8 0.95 0.91 0.93 398
1141+
9 0.91 0.92 0.92 397
1142+
10 0.98 0.92 0.95 399
1143+
11 0.96 0.85 0.90 396
1144+
12 0.71 0.69 0.70 393
1145+
13 0.95 0.70 0.81 396
1146+
14 0.86 0.91 0.88 394
1147+
15 0.85 0.90 0.87 398
1148+
16 0.79 0.84 0.81 364
1149+
17 0.99 0.77 0.87 376
1150+
18 0.58 0.75 0.65 310
1151+
19 0.52 0.60 0.55 251
1152+
1153+
avg / total 0.82 0.81 0.81 7532
1154+
1155+
10141156
-----------------------------------------
10151157
Recurrent Neural Networks (RNN)
10161158
-----------------------------------------
@@ -1019,6 +1161,143 @@ Recurrent Neural Networks (RNN)
10191161

10201162
.. image:: docs/pic/LSTM.png
10211163

1164+
1165+
import packages:
1166+
1167+
.. code:: python
1168+
1169+
1170+
from keras.layers import Dropout, Dense, GRU, Embedding
1171+
from keras.models import Sequential
1172+
from sklearn.feature_extraction.text import TfidfVectorizer
1173+
import numpy as np
1174+
from sklearn import metrics
1175+
from keras.preprocessing.text import Tokenizer
1176+
from keras.preprocessing.sequence import pad_sequences
1177+
from sklearn.datasets import fetch_20newsgroups
1178+
1179+
convert text to word embedding (Using GloVe):
1180+
1181+
.. code:: python
1182+
1183+
def loadData_Tokenizer(X_train, X_test,MAX_NB_WORDS=75000,MAX_SEQUENCE_LENGTH=500):
1184+
np.random.seed(7)
1185+
text = np.concatenate((X_train, X_test), axis=0)
1186+
text = np.array(text)
1187+
tokenizer = Tokenizer(num_words=MAX_NB_WORDS)
1188+
tokenizer.fit_on_texts(text)
1189+
sequences = tokenizer.texts_to_sequences(text)
1190+
word_index = tokenizer.word_index
1191+
text = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)
1192+
print('Found %s unique tokens.' % len(word_index))
1193+
indices = np.arange(text.shape[0])
1194+
# np.random.shuffle(indices)
1195+
text = text[indices]
1196+
print(text.shape)
1197+
X_train = text[0:len(X_train), ]
1198+
X_test = text[len(X_train):, ]
1199+
embeddings_index = {}
1200+
f = open("C:\\Users\\kamran\\Documents\\GitHub\\RMDL\\Examples\\Glove\\glove.6B.50d.txt", encoding="utf8")
1201+
for line in f:
1202+
1203+
values = line.split()
1204+
word = values[0]
1205+
try:
1206+
coefs = np.asarray(values[1:], dtype='float32')
1207+
except:
1208+
pass
1209+
embeddings_index[word] = coefs
1210+
f.close()
1211+
print('Total %s word vectors.' % len(embeddings_index))
1212+
return (X_train, X_test, word_index,embeddings_index)
1213+
1214+
Build a RNN Model for Text:
1215+
1216+
.. code:: python
1217+
1218+
1219+
def Build_Model_RNN_Text(word_index, embeddings_index, nclasses, MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50, dropout=0.5):
1220+
"""
1221+
def buildModel_RNN(word_index, embeddings_index, nclasses, MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50, dropout=0.5):
1222+
word_index in word index ,
1223+
embeddings_index is embeddings index, look at data_helper.py
1224+
nClasses is number of classes,
1225+
MAX_SEQUENCE_LENGTH is maximum lenght of text sequences
1226+
"""
1227+
1228+
model = Sequential()
1229+
hidden_layer = 3
1230+
gru_node = 32
1231+
1232+
embedding_matrix = np.random.random((len(word_index) + 1, EMBEDDING_DIM))
1233+
for word, i in word_index.items():
1234+
embedding_vector = embeddings_index.get(word)
1235+
if embedding_vector is not None:
1236+
# words not found in embedding index will be all-zeros.
1237+
if len(embedding_matrix[i]) != len(embedding_vector):
1238+
print("could not broadcast input array from shape", str(len(embedding_matrix[i])),
1239+
"into shape", str(len(embedding_vector)), " Please make sure your"
1240+
" EMBEDDING_DIM is equal to embedding_vector file ,GloVe,")
1241+
exit(1)
1242+
embedding_matrix[i] = embedding_vector
1243+
model.add(Embedding(len(word_index) + 1,
1244+
EMBEDDING_DIM,
1245+
weights=[embedding_matrix],
1246+
input_length=MAX_SEQUENCE_LENGTH,
1247+
trainable=True))
1248+
1249+
1250+
print(gru_node)
1251+
for i in range(0,hidden_layer):
1252+
model.add(GRU(gru_node,return_sequences=True, recurrent_dropout=0.2))
1253+
model.add(Dropout(dropout))
1254+
model.add(GRU(gru_node, recurrent_dropout=0.2))
1255+
model.add(Dropout(dropout))
1256+
model.add(Dense(256, activation='relu'))
1257+
model.add(Dense(nclasses, activation='softmax'))
1258+
1259+
1260+
model.compile(loss='sparse_categorical_crossentropy',
1261+
optimizer='adam',
1262+
metrics=['accuracy'])
1263+
return model
1264+
1265+
1266+
1267+
1268+
run RNN and see our result:
1269+
1270+
1271+
.. code:: python
1272+
1273+
newsgroups_train = fetch_20newsgroups(subset='train')
1274+
newsgroups_test = fetch_20newsgroups(subset='test')
1275+
X_train = newsgroups_train.data
1276+
X_test = newsgroups_test.data
1277+
y_train = newsgroups_train.target
1278+
y_test = newsgroups_test.target
1279+
1280+
X_train_Glove,X_test_Glove, word_index,embeddings_index = loadData_Tokenizer(X_train,X_test)
1281+
1282+
1283+
model_RNN = Build_Model_RNN_Text(word_index,embeddings_index, 20)
1284+
1285+
model_RNN.fit(X_train_Glove, y_train,
1286+
validation_data=(X_test_Glove, y_test),
1287+
epochs=10,
1288+
batch_size=128,
1289+
verbose=2)
1290+
1291+
predicted = Build_Model_RNN_Text.predict_classes(X_test_Glove)
1292+
1293+
print(metrics.classification_report(y_test, predicted))
1294+
1295+
Output:
1296+
1297+
::
1298+
1299+
1300+
10221301
-----------------------------------------
10231302
Convolutional Neural Networks (CNN)
10241303
-----------------------------------------

0 commit comments

Comments
 (0)