Hold on…. hear me out… we were inspired to start this project because LLMs were interesting…
We started with a flow chart and discussed about how the A.I would work. We wanted to talk to the A.I using our voices and it should be able to respond back to our query (pointing us to Speech to Text Programs),
Making the logical flow chart was easy as we used online references to figure out parts that we had no knowledge about i.e. Quantization. The chart went something like this:
To explain what is going on the above flowchart, we would write python code to record the microphones on our PCs and save it as .WAV which would then be converted to a .TXT file. the text file would then be sent over to the LLM which would be locally hosted on our PCs, generating a response to our query. This response would be saved in a .TXT file which would be then sent over to the Text-to-Speech Program generating audio of the response paired with VTube Studio, a very popular Virtual Youtuber Avatar Studio used by many. We imported a pre-made model into it and using a Virtual Cable, this model was taking inputs from the python application hence we programmed and rigged the model to move on the virtual cable’s frequency leading the mouth to move.
We quickly learnt that APIs are very expensive and the club budget could not handle it so rather we went the difficult route, coding everything in python using libraries 🙂
As for the Language Model, we researched for a viable one because paying money to open.ai was our last option. Utilizing the huge catalogs of LLMs on Hugging Face we settled on Llama 2 from meta. The type of quantization our model is using is 4 bit (we had to compromise a bit because of hardware resources and availability. Our Model: Llama-2-7b-chat-hf-GGUF-Q4_K_M.
These are some of the libraries we used: Open AI (not for the LLM but for the references in our code because hooked the LLM to a local port and needed to use it), Pyaudio, Wave, Keyboard, Speech Recognition.
GitHub Link: Coming Soon !
P.S. her name is hana… come say hi at the Club Drive Booth !!
0 Comments