Text Analysis / Source notebook
A vocabulary in songs
Explore the words on the page: repetitions, rare forms and the changing texture of each song version.
69 retained song versions · 3,959 lyric lines. These describe this corpus, not a person's language ability.
Corpus · all song versions
Word tokens
35,088
All observed forms, including stopwords.
Distinct forms
5,266
Casefold forms; not an estimate of personal vocabulary.
Type–token ratio
0.150
Distinct forms ÷ tokens; sensitive to text length.
MATTR · 50
0.817
Mean over fixed 50-token windows, never crossing songs.
Hapax forms
2,781
Forms appearing exactly once in the selected scope.
31,707 within-song MATTR windows.
Explore word frequencies
Search and stopword filters affect only the chart and table. Cards use all word forms in the selected song or corpus. Song spread counts retained versions, not independent compositions.
Interactive filters require the local analysis data and JavaScript.
Top 20 word forms · by occurrence
Frequency index
| Rank | Stopword | |||
|---|---|---|---|---|
| 1 | i | 1,137 | 68 | Yes |
| 2 | the | 1,055 | 69 | Yes |
| 3 | a | 787 | 69 | Yes |
| 4 | to | 753 | 69 | Yes |
| 5 | you | 668 | 65 | Yes |
| 6 | and | 565 | 69 | Yes |
| 7 | my | 559 | 67 | Yes |
| 8 | it | 486 | 67 | Yes |
| 9 | me | 474 | 65 | Yes |
| 10 | of | 394 | 62 | Yes |
| 11 | in | 388 | 65 | Yes |
| 12 | is | 373 | 65 | Yes |
| 13 | with | 357 | 68 | Yes |
| 14 | but | 282 | 62 | Yes |
| 15 | im | 278 | 40 | No |
| 16 | that | 274 | 64 | Yes |
| 17 | for | 272 | 64 | Yes |
| 18 | so | 272 | 61 | Yes |
| 19 | i'm | 239 | 50 | Yes |
| 20 | don't | 236 | 64 | Yes |
| 21 | this | 232 | 61 | Yes |
| 22 | all | 226 | 61 | Yes |
| 23 | we | 217 | 52 | Yes |
| 24 | just | 203 | 58 | Yes |
| 25 | be | 198 | 55 | Yes |
| 26 | what | 196 | 50 | Yes |
| 27 | on | 190 | 59 | Yes |
| 28 | no | 181 | 56 | Yes |
| 29 | if | 169 | 52 | Yes |
| 30 | shit | 164 | 45 | No |
| 31 | up | 164 | 50 | Yes |
| 32 | fuck | 162 | 42 | No |
| 33 | your | 160 | 45 | Yes |
| 34 | its | 149 | 52 | Yes |
| 35 | like | 142 | 47 | No |
| 36 | now | 136 | 50 | No |
| 37 | see | 135 | 52 | No |
| 38 | was | 132 | 43 | Yes |
| 39 | can | 128 | 49 | Yes |
| 40 | ain't | 126 | 50 | Yes |
Method, provenance & limits
Source repository · Revision:
5285918631d545c0ef78bb30cc68203ddeeca277.
The input is the local working tree. A recorded HEAD commit is provenance, not proof of a clean checkout; linked source may differ from the analyzed text.
69 lyric files → 69 retained song versions; 0 normalized duplicate files are aliases, not extra observations. All versions with differing text are kept. Repeated verses remain counted.
Read the exact analysis rules
Local UTF-8 text working tree; .git, ._ metadata, symlinks/junctions and non-.txt files excluded. No source is executed or modified.
NFC/casefold letter forms with internal normalized apostrophes; digits, punctuation, underscores and hyphens delimit. Timestamps and standalone section headers excluded. No stemming; stopwords flagged, never removed.
One record per nonblank physical line except standalone headers/timestamps; original text and 1-based line number preserved. No sentence segmentation, even for long paragraphs.
Exact equality after NFC/apostrophe normalization, timestamp/header/blank-line removal and per-line whitespace collapse. Case, punctuation and lyric-line boundaries retained. First sorted path is canonical; aliases excluded from all corpus metrics.
Nonidentical versions, repeated verses and shared phrases remain, so revisions/repetition and collection sizes bias frequencies. Counts are not independent songs or representative language samples.
Observed casefold word forms only, not total personal vocabulary, intelligence/IQ or writing quality. TTR is distinct forms / tokens; hapaxCount counts forms appearing once. Empty TTR is null.
Mean distinct-form fraction across every overlapping 50-token window, never resized. Null below 50 tokens. Corpus pools within-song windows weighted by window count, never crossing songs; null if no song has a window.