Compare Feature Extraction Techniques

| Model | Advantages | Limitation |

| Weighted Words | * Easy to compute | * It does not capture the position in the text (syntactic) |

| | | |

| | * Easy to compute the similarity between 2 documents using it | * It does not capture meaning in the text (semantics) |

| | | |

| | * Basic metric to extract the most descriptive terms in a document | |

| | | * Common words effect on the results (e.g., “am”, “is”, etc.) |

| | * Works with an unknown word (e.g., New words in languages) | |

| TF-IDF | * Easy to compute | * It does not capture the position in the text (syntactic) |

| | | |

| | | |

| | * Easy to compute the similarity between 2 documents using it | * It does not capture meaning in the text (semantics) |

| | | |

| | | |

| | * Basic metric to extract the most descriptive terms in a document | |

| | | |

| | | |

| | * Common words do not affect the results due to IDF (e.g., “am”, “is”, etc.) | |

| Word2Vec | * It captures the position of the words in the text (syntactic) | * It cannot capture the meaning of the word from the text (fails to capture polysemy) |

| | | |

| | * It captures meaning in the words (semantics) | * It cannot capture out-of-vocabulary words from corpus |

| GloVe (Pre-Trained) | * It captures the position of the words in the text (syntactic) | * It cannot capture the meaning of the word from the text (fails to capture polysemy) |

| | | |

| | * It captures meaning in the words (semantics) | |

| | | * Memory consumption for storage |

| | * Trained on huge corpus | |

| | | |

| | | * It cannot capture out-of-vocabulary words from corpus |

| GloVe (Trained) | * It is very straightforward, e.g., to enforce the word vectors to capture sub-linear relationships in the vector space (performs better than Word2vec) | * Memory consumption for storage |

| | | |

| | * Lower weight for highly frequent word pairs such as stop words like “am”, “is”, etc. Will not dominate training progress | * Needs huge corpus to learn |

| | | |

| | | * It cannot capture out-of-vocabulary words from the corpus |

| | | |

| | | * It cannot capture the meaning of the word from the text (fails to capture polysemy) |

| FastText | * Works for rare words (rare in their character n-grams which are still shared with other words | * It cannot capture the meaning of the word from the text (fails to capture polysemy) |

| | | |

| | | * Memory consumption for storage |

| | * Solves out of vocabulary words with n-gram in character level | |

| | | * Computationally is more expensive in comparing with GloVe and Word2Vec |

| Contextualized Word Representations | * It captures the meaning of the word from the text (incorporates context, handling polysemy) | * Memory consumption for storage |

| | | |

| | | * Improves performance notably on downstream tasks. Computationally is more expensive in comparison to others |

| | | |

| | | * Needs another word embedding for all LSTM and feedforward layers |

| | | |

| | | * It cannot capture out-of-vocabulary words from a corpus |

| | | |

| | | |

| | | * Works only sentence and document level (it cannot work for individual word level) |