# 1. Language Processing and Python

# 1   Computing with Language: Texts and Words

## 1.1   Getting Started with Python

```plaintext
# print hello world
print('hello world')
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699076954357/4d1a921b-1484-4de4-939a-adac5ce32f13.png align="center")

## 1.2   Getting Started with NLTK

```plaintext
# import nltk
import nltk
nltk
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699077090748/5804ac43-19c2-452c-bfff-3bc2b4ef9e6d.png align="center")

```plaintext
# import gutenberg plaintext corpus reader
from nltk.corpus import gutenberg
gutenberg
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699077234899/af65b93e-a65b-43bf-af1e-fb13b9c53d87.png align="center")

```plaintext
# download gutenberg files
nltk.download('gutenberg')
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699073956449/963e063b-7ac7-4057-9134-09f4f7643382.png align="left")

```plaintext
# get gutenberg file ids
gutenberg.fileids()
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699077578760/d3080b0d-bedd-4c82-9036-004422ac7b3b.png align="center")

```plaintext
# test open using regular approach
f = open("/root/nltk_data/corpora/gutenberg/melville-moby_dick.txt", "r")
print(f.read()[0:300])
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699077716134/cc811518-b7b1-458a-afc7-ebcd188de052.png align="center")

```plaintext
# create a corpus for moby dick
corpus1 = gutenberg.words('melville-moby_dick.txt')
text1 = nltk.Text(corpus1)
text1
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699078408767/5131f44c-e8cd-4c38-b12c-71f735055a7e.png align="center")

```plaintext
# create a corpus for sense and sensibility
corpus2 = gutenberg.words('austen-sense.txt')
text2 = nltk.Text(corpus2)
text2
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699078475135/3adb5a14-e67e-498e-b493-d04894466b16.png align="center")

## 1.3   Searching Text

```plaintext
# get concordance for a sample word monstrous
text1.concordance("monstrous")
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699078785802/6bbce799-38a8-4663-b931-96bb6b03eae5.png align="center")

```plaintext
# compare between two corpus
text1.similar("monstrous")
text2.similar("monstrous")
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699078882804/f037f052-b8bd-483b-9bd5-3431f024ea9c.png align="center")

```plaintext
# examine just the contexts 
# that are shared by two or more words, 
# such as monstrous and very
text2.common_contexts(["monstrous", "very"])
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699079189953/855453ee-119e-4ef4-bc59-e82126c7c9b7.png align="center")

```plaintext
# determine the location of words ie dispersion
text1.dispersion_plot(["citizens", "democracy", "freedom", "duties", "America"])
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699079418766/decc5e05-62a6-4770-a372-b0270084ce5a.png align="center")

```plaintext
import nltk
nltk.download('punkt')
# just for fun, generate text from corpus
text2.generate()
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699079549242/aa57cfd5-2bf6-4fff-8cd7-c0f04b8b9fa4.png align="center")

## 1.4   Counting Vocabulary

```plaintext
# get length of corpus text
len(text2)
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699079884846/fa1a748f-e3b5-4414-a0dd-132ea218e3d8.png align="center")

```plaintext
# create unique set of corpus text and sort in ascending order
sorted(set(text2))
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699079952117/5c08a1fc-ea19-408c-a07b-b4cfbc698463.png align="center")

```plaintext
# measure lexical richness of text
len(set(text2)) / len(text2)
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699080925312/b426f62f-fa45-4f4e-8556-7b69c613a9c3.png align="center")

Note: the number of distinct words is just 4% of the total number of words i.e 96% of the corpus text contains repeated words.

```plaintext
# count how often a word occurs in a text, and 
# compute what percentage of the text is taken up by a specific word
print(text1.count("whale"))
print(100 * text1.count('whale') / len(text1))
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699081260870/68b354d4-ef42-4d23-87a0-4b72e61c7152.png align="center")

Using functions for repetitive tasks.

Define functions:

```plaintext

# define a function, lexical_diversity
def lexical_diversity(text):
  return len(set(text)) / len(text)
  
# define a function, percentage
def percentage(count, total):
  return 100 * count / total
```

Call functions:

```plaintext
# call lexical_diversity
print(  lexical_diversity(text2)  )

# call percentage
print(  percentage (text1.count('whale') , len(text1) )  )
```

# 2   A Closer Look at Python: Texts as Lists of Words

## 2.1   Lists

```plaintext
# declare a list of words
sent1 = ['Call', 'me', 'Ishmael', '.']
# print the list content
print ( sent1 )
['Call', 'me', 'Ishmael', '.']
# compute the length of sent1
print ( len(sent1) )
# get the lexical diversity of sent1
print ( lexical_diversity(sent1) )
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699081879667/8445cbce-7d81-48d2-b4a8-363d00961723.png align="center")

```plaintext
# append new word to an existing list
sent1.append("Some")
print( sent1 )
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699081967505/0ba24dd5-660d-493e-a749-59ba9347d5c7.png align="center")

## 2.2   Indexing Lists

```plaintext
# get a word in index no, 173 of text1 list
print( text1[1] )
'Moby'
# get an index no. given the word 'awaken' 
print( text1.index('Moby') )
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699082286412/275a1b46-ee05-412d-bd2b-ceaa76a059cf.png align="center")

```plaintext
# print words in the index range that starts at index 0 and end before index 6
print( text1[0:6] )
# print words in the index range that starts at index 4 and end before index 6
print( text1[4:6] )
# print words in the index range that starts at second last index and ends at the last index
print( text1[-2:] )
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699082678840/c50c00f1-5c51-493a-b9d7-3b3687ba01ec.png align="center")

# 3   Computing with Language: Simple Statistics

## 3.1   Frequency Distributions

```plaintext
from nltk import FreqDist
# calculate frequency distribution
fdist1 = FreqDist([w.lower() for w in text1])
print(fdist1)
# get top 50 common words
fdist1.most_common(50)
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699082997976/3a5dc10b-69f0-4cf0-86fd-ad9a27685928.png align="center")

## 3.2   Fine-grained Selection of Words

```plaintext
#Fine-grained Selection of Words
V = set(text1)
long_words = [w for w in V if len(w) > 15]
sorted(long_words)
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699083156998/18c8250d-c2e7-42f7-b898-926db9a6bc6d.png align="center")

```plaintext
# select specific length of words
fdist2 = FreqDist(text2)
sorted(w for w in set(text2) if len(w) > 7 and fdist2[w] > 7)
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699083239950/a0ad0e5e-98c4-4f83-b17b-cb2c660a55aa.png align="center")

## 3.3   Collocations and Bigrams

```plaintext
from nltk import bigrams

list(bigrams(['more', 'is', 'said', 'than', 'done']))
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699083348465/e1760cc6-037a-4771-b2d4-c3799f45b69f.png align="center")

## 3.4 Frequency Distribution Functions

Functions Defined for NLTK's Frequency Distributions

| Example | Description |
| --- | --- |
| fdist = FreqDist(samples) | create a frequency distribution containing the given samples |
| fdist\[sample\] += 1 | increment the count for this sample |
| fdist\['monstrous'\] | count of the number of times a given sample occurred |
| fdist.freq('monstrous') | frequency of a given sample |
| fdist.N() | total number of samples |
| fdist.most\_common(n) | the n most common samples and their frequencies |
| for sample in fdist: | iterate over the samples |
| fdist.max() | sample with the greatest count |
| fdist.tabulate() | tabulate the frequency distribution |
| fdist.plot() | graphical plot of the frequency distribution |
| fdist.plot(cumulative=True) | cumulative plot of the frequency distribution |
| fdist1 | \= fdist2 |
| fdist1 < fdist2 | test if samples in fdist1 occur less frequently than in fdist2 |

```plaintext
fdist = FreqDist(text1	)

print(type(fdist))

print(len(fdist))

print(list(fdist))

print(fdist.N())

print(fdist.most_common(5))

fdist['and']
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699084612049/606f5409-5d59-487c-9bc6-bdb768d8a3ad.png align="center")

```plaintext
# plot word frequency distribution
fdist.plot()
```

output:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699084550770/7dadb72d-d45f-4ea3-8c72-913e03f1e446.png align="center")

## 4 Back to Python: Making Decisions and Taking Control

### 4.1   Conditionals

### 4.2 Operating on Every Element

### 4.3 Nested Code Blocks

### 4.4 Looping with Conditions

# Colab Notebook:

[https://colab.research.google.com/drive/1K9RVe42Kp79RyOlHC749wIdXVQR0eysI?usp=sharing](https://colab.research.google.com/drive/1K9RVe42Kp79RyOlHC749wIdXVQR0eysI?usp=sharing)

# Source:

[https://www.nltk.org/book/ch01.html](https://www.nltk.org/book/ch01.html)