chore: reqs, ignore, readme

.gitignore

@@ -1,3 +1,5 @@
+# ignore knn index
+gpt4all/index/**/
 .DS_Store
 *.pkl
 ckpts*

gpt4all/index/README.md

@@ -0,0 +1,27 @@
+# How to Tokenize and Embed
+
+Split text into chunks
+```
+python tokenize_texts.py
+```
+
+Embbed Texts
+
+```
+torchrun --master_port=29085 --nproc-per-node 8  embed_texts.py --ds_path=tokenized --batch_size=2048
+```
+
+
+Combine Embeddings and Build Index
+```
+python build_index.py --ds_path=wiki_sample_tokenized --embed_folder=wiki_sample_embedded
+```
+
+To use the Index
+
+```
+import hnswlib
+
+index = hnswlib.Index(space='l2', dim=384)
+index.load_index(<path to index>)
+```

requirements.txt

@@ -12,4 +12,7 @@ sentencepiece
 jsonlines
 nomic
 scikit-learn
-matplotlib
+matplotlib
+apache_beam 
+mwparserfromhell
+hnswlib