**Theremin Hero** Introduction and Overview =============================================================================== Introduction ------------------------------------------------------------------------------- The Theremin is an electronic instrument invented in the late 1930s with the peculiar characteristic of being controlled with no physical contact, only variations in proximity. A traditional theremin consists of two antennas (one for volume and one for pitch) that form open-air capacitors with a player’s nearby hands that affect a circuit to produce audible frequencies. Although objectively awesome, the Theremin has yet to gain mainstream recognition as the best musical instrument of all time but that’s all about to change. With the introduction of Theremin Hero comes the dawn of a new musical era, bringing this obscure instrument to every household in America and igniting the competitive theremin gaming scene. Overview ------------------------------------------------------------------------------- Using time of flight sensing, the Theremin Hero measures the proximity of a user’s hand along a marked fret board and translates this proximity into matching frequencies from C3 to C4. Adjusting the volume with a slider, users can play all their favorite songs and share their own creations with family and friends by uploading named songs to the server where they can be displayed and played back in the browser. Not just this, but users can try their hand at playing uploaded songs and compete for high scores for each song. For a less thrilling experience, users can jam out in free play mode or sit back and browse songs on the Theremin Hero or online. How to Play! ------------------------------------------------------------------------------- When a user first turns on their Theremin Hero, they will be asked to input a username, which will be used later for saving scores and new songs. After inputting a name, the user will see a menu of three different modes: * Free Play Mode * Record Song * Get Song and Compete In **Free Play Mode**, the user can simply play around with different notes and volumes. It’s chill; the user does not have to worry about competing with anything and can listen to the pretty sounds alongside watching the pretty purple lights. The further away the user’s hand is from the sensor, the higher pitched the sound. There is also a fretboard which indicates where each note is relative to the sensor. For confirmation of where a user’s hand is, LEDs will light up on the fretboard showing exactly what note is being played. Additionally, the note the user is playing is also displayed on the LCD screen which is mounted on top of the Theremin Hero. In order to have full control of the volume, there is a slider which changes the volume of the Theremin Hero. In **Record Song**, a user who is proud of a particular song that they have played can record up to 15 seconds of the song and upload it to the internet. While a user is recording, they will see the same features as in **Free Play Mode** except the LCD screen will give a countdown of how much time there is left to record a song. The songs that are uploaded are also added to the song database used in the **Get Song and Compete** mode. In **Get Song and Compete**, a user can retrieve songs and try to play it the most accurately. Before the user jumps straight into competing, they can listen to the song and see the notes that are supposed to be played. On the LCD screen, there are falling blocks which show where the user’s hand needs to be at a certain time and in the near future. Additionally, the LED light strip shows where a note is and a range of where a user’s hand needs to be in order to get points. This mode is best for practicing as the user can hear and see what needs to be played. When the user feels confident in their abilities, they can actually compete and try to play the song correctly for points. The LED light strip will still show where the user’s hand should be as well as what the user is currently playing. The LCD screen shows the same thing as the practice mode. After competing, the user’ score is sent online so that other people can see and compare. In addition to the physical Theremin Hero, there are plenty of things to do [online at this link](https://608dev.net/sandbox/sc/kgarner/project/html/simple-form.html) Here, the user has the option to **View songs** and **Upload a song**. In **View songs**, the user sees a list of all the songs recorded or previously uploaded. The user can click on a particular song to listen to what the song sounds like and to view the scoreboard of everyone who has competed against this particular song. If a song seems particularly enticing and a user wants to keep it on their computer, they can also download the audio of a song. In **Upload a song**, a user can type out the durations of notes of a song and the song will be added to the song database. The user will be able to play their generated in **View songs** and in the **Get Song and Compete** mode on the Theremin Hero. That should be everything you need to know to play! Rock on! System Overview =============================================================================== Hardware ------------------------------------------------------------------------------- The final product should look similar to a box with a ruler extending out from one side. Inside the box is all the hardware necessary to run the code and connect parts together, while outside are all the components the user is supposed to interact with. Before gluing anything down, let’s begin with the breadboard set up. Hardware list: * TOF * Amplifier * Potentiometer * Speaker * LCD Screen * LEDs * Buttons * Battery Board * IMU Use the following circuit diagrams to wire the Theremin hero. ### ESP32 The ESP32 is the microcontroller used in this project. We can upload .ino files onto the board and the ESP32 board will run the program. ### Time of Flight Sensor In this project, a note is supposed to be played based on the position of someone’s hand relative to a starting location. In order to accomplish this, we can use a Time of Flight sensor (TOF), which measures the distance between a sensor and an object based on the time difference between the emission of a signal and its return to the sensor, after being reflected by an object. We can use readings from the TOF to get the position of someone’s hand in millimeters and convert the reading into sound which can be played on a speaker. ![Figure [esp to tof]: Wiring the TOF to the ESP32.](./images/esp-tof.png width="500px" border="1") ### LCD Screen and Buttons The main user interface for the Theremin Hero is the LCD screen with buttons. It allows the user to interact with scroll through menus and flip through the different modes. ![Figure [esp to lcd]: Wiring the LCD screen and buttons to the ESP32.](./images/esp-lcd.png width="500px" border="1") ### IMU In order to create a quick way to input text for song names or for usernames, we attached an IMU to our board. The IMU contains three types of sensors: an accelerometer, a gyroscope and a magnetometer. The sensor that is important for this project is the accelerometer. By measuring readings from the accelerometer, we can control elements in our project by tilting the sensor. ![Figure [esp to imu]: Wiring the IMU to the ESP32..](./images/esp-imu.png width="500px" border="1") ### Audio Amplifier, Potentiometer and Speaker Pitch can be taken as the measure of sound frequency expressed in terms of Hertz. The higher the frequency, the higher the pitch. In order to create a specific pitch from the ESP32 to a speaker, we must create the correct frequency via Pulse Width Modulation, or PWM. Luckily for us, the function *ledcWriteTone(channel, frequency)* allows us to vary the frequency of the PWM signal, which in turn, allows us to vary the pitch of the sound produced through the speaker. Since we are using PWM, a form of digital control, we end up creating a square wave as the signal is switched between off and on, or in other words, between 0V and 3.3V. Great! Now we have a signal, but we have no way of hearing the signal. In order to produce sound so that everyone can hear, we will use an audio amplifier, potentiometer and a speaker. The audio amplifier converts the small input voltage from the ESP32 into a much larger output voltage which the speaker can use to actually produce sound. In order to control the volume, we need to control the voltage of the square wave. The higher the voltage, the louder the sound. In the same way, the lower the voltage, the quieter the sound. The slide potentiometer that we are using in this project is a variable resistor, and, when placed in series with the input of the audio amplifier, controls the voltage. In turn, this controls the intensity of the sound. ![Figure [esp to audio]: Wiring the amplifier, potentiometer, and speaker to the ESP32..](./images/esp-audio.png width="500px" border="1") ### LED Strip The LED strip is used to show roughly where a player’s hand is while they are playing with the theremin. ![Figure [esp to led]: Wiring the LED to the ESP32.](./images/esp-led.png width="500px" border="1") ### Battery Board So that we can make sure our Theremin Hero is portable, we can attach a battery to the setup, using the following schematic ![Figure [board with pictures]: Wiring the board to battery and converter (pictures)](./images/board-1.png width="500px" border="1") ![Figure [board miniman diagram]: Wiring the board to battery and converter (diagram)](./images/board-2.png width="500px" border="1") ### Casing In order to make our theremin look sleek and cool, we laser cut out a neat box. The most important part of the box is the ruler which tells the user where their hand should be in order to play a specific note. Also, the lights attach onto the ruler to visually show what note the user is currently playing. Our specific ruler lines up so that every two leds on the light strip matches a letter. See [this link](https://drive.google.com/open?id=1-3Nppd0PISgEtYm3FIpB9hFlXenwfGjT) for our dxf files. Block Diagram =============================================================================== ![Figure [block-diagram]: A Diagram of user interaction](./images/block-diagram.png width="500px" border="1") State Machine =============================================================================== ![Figure [state machine]: The state machine for our divice](./images/state-machine.png width="500px" border="1") Turning on the device ------------------------------------------------------------------------------- Free-play and recording modes ------------------------------------------------------------------------------- Play-back and competitive modes ------------------------------------------------------------------------------- Challenges =============================================================================== **Server-Side Audio Generation** * We formatted music strings as pairs of frequency and interval length values. We used python’s *numpy* and *soundfile* libraries to generate sine waves and concatenate them into a single .OGG sound file. * However, the dicontinuities in the concatenated sine wave produced clikcing sounds whenever the tone changed in any music file. * Therefore, we saved the last amplitude value in each sine wave subinterval representing a single frequency, from which we calculated the amplitude value at which the next different-frequency sine wave would start. **Distance Sensing** * Ultrasonic Sensor * The original design of the Theremin Hero relied on pulsing ultrasonic sensors to provide information on the distance of the player’s hands. * However, these sensors were prone to noise, required perfectly flat surfaces, and were quite unreliable even with the help of averaging filters and other software fixes. * Infrared Sensor * Similar to the ultrasonic sensors, the IR sensor did not produce very consistent results and did not work very well with rough surfaces * TOF (Time of Flight) Sensor * The TOF sensor was ultimately the best fit for this project. Although it does not have as large a range as the other sensors, accuracy was the greatest concern for this project, not range. * The pulsating light beam works well with complex surfaces like a hand and can even detect figures such as fingers reasonably well. * Unlike with ultrasound, the environment does not have much noise that would interfere with the TOF sensor. Additionally, the light pulses are well isolated in a region in front of the sensor and are roughly the size of a hand (viewed through UV/IR laser **Volume Control** * Ultrasonic Sensor * Although the original plan was for an additional ultrasonic sensor to measure distance for the volume, the interference of one ultrasonic sensor rendered readings inaccurate, let alone another polluting the environment with ultrasonic propagations. * TOF Sensor * A secondary TOF sensor was another option for the volume control but since they would share the same I2C bus and, being the same part, have the same address it would likely cause problems * Digital Inputs * Initially believing it would be possible to control the volume with PWM/Duty cycles it became clear that would not yield appropriate results. * Considered using a digi-pot to change the voltage entering the buzzer but would be limited to discrete values. * Analog Potentiometer * A sliding potentiometer became the volume control for the final iteration of the project. Trouble with digitally controlling the volume suggested an analog fix would be ideal and allow for more continuous volume adjustments. * Originally using a screw-top potentiometer, many different resistors were attempted including photocells and flex sensors but a sliding potentiometer became the most reliable and functional solution. **Visualization** * Screen visualization * When implementing the “game” portion of the project, a difficult design choice was determining how to transmit to the player what notes to play in an intuitive way. * Like many other beat/rhythm games it became obvious the best structure was some sort of moving or falling note system. * After some thought, it seemed integral that the player have a visual reminiscent of the instrument’s fretboard. This allows the player, while in a competitive mode, to be able to focus on the screen and the prompted notes rather than the fretboard. * Fretboard/LEDs * Besides the screen a physical representation for the player seemed ideal which intuitively came in the form of a fretboard. * For the player to get a better idea of their positioning, LEDs were added and programmed not only light up the current note but also to act as a guide for what notes to play in competitive mode. **Audio Generation** * Piezo Buzzer * Figuring out how to emit audio of reasonable quality was difficult for much of the design process. * A simple piezo buzzer sounded subpar with square waves and would have many issues with audio cut outs, overtones, and popping. However, the buzzer was useful for initial testing of the system. * Amplifier/Speaker * The final audio setup consists of an amplifier chip and speaker which greatly improved audio quality and made volume control much more reasonable. * Software * After playing with the idea of MIDI files or using an SD card and MP3s it seemed the best to just go with arduinos built in tone producing functions using PWM at an analog input. This allowed us to easily produce and change the tones without any other files. * Choosing when to write and update tones was crucial to removing periodic buzzing and audio cut-outs so ensuring not to rewrite notes every time interval helped smooth out sound * The PWM used for screen brightness seemed to receive some interference from the PWM for audio output. A simple and unexpected fix was to create greater distance between PWM channels. (Initially using 0,1 and moving to use 0,14). **User Inputs** * IMU * A small but crucial design decision was implementing the IMU for scrolling movement through the alphabet for song names and usernames. * A significant improvement from button scrolling. **Note Mapping** * In the beginning we employed a linear scaling to map notes to frequencies. However, we found this to be difficult to map (and sometimes imprecise). * To more properly reflect how notes worked, we found a new mapping function based off of powers/logs Parts List =============================================================================== * **Time of Flight sensor** * VL6180X Time-of-Flight Distance Sensor Carrier with Voltage Regulator * **Slider Potentiometer** * Robotdyn Analog Slide Potentiometer 10K ohm * **Arduino Stereo Audio Amplifier** * PAM8403 5V Two-channel Stereo Mini Class-D 3W+3W Audio Amplifier * **Speaker** * 1.5" 4Ohm 3W Full Range Audio Speaker Stereo Woofer Loudspeaker * ~1ft of side emitting RBG LEDs * 1/4th Inch Black and Clear Acrylic * Additional small breadboard Code Documetation =============================================================================== Visit [this link (MIT certs required)](https://github.mit.edu/kgarner/608-s19-final-project) for the full code. Hardware ------------------------------------------------------------------------------- ### ui_tools --- **`ui_tools.ino`** ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C const std::vector NOTES = {"C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B", "C" }; ... const int notes_freq[] = {262, 277, 294, 311, 330, 349, 370, 392, 415, 440, 466, 494, 523}; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ While we have decided to play the notes based on position continuously, we have a `NOTES` string to separate the ranges of frequencies into discretized notes. This makes the visualization much easier and less cluttered for the player especially since it can now be done in a Guitar Hero-like style familiar to many people. This also makes scoring much easier since the conversion from distance to frequency is logarithmic. So rather than scoring based on a set frequency or distance off from the ideal which does not scale properly, we can score based on whether the same note is hit or not. This also logically makes sense due to the visualization separating the notes and gives a proper margin of error based on the note hit which is what would really matter in the from the musical point of view. To get the correct note to display, we grab the distance in millimeters from the time-of-flight sensor, and convert that to a frequency to play. Once we have the frequency, we find the closest frequency from the `notes_freq` list above and get the index which matches up with the note. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C int find_closest_idx(float freq) { int semitone = round(log2(freq / 440.0) * 12.0); semitone += 9; if (semitone < 0) { return 0; } else if (semitone >= NOTES.size()) { return NOTES.size() - 1; } else { return semitone; } } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The function above returns the index in the `notes_freq` list. It takes in the frequency converted from the distance and the equation `semitone = round(log2(freq / 440.0) * 12.0);` finds the amount of semitones away from the base note which is set to 440 Hz or "A". To get frequencies for different notes, the equation $f_n = f_0*a^n$ is used where a is $2^(1/12)$, $f_0$ is the base note's frequency, and $n$ is the amount of semitones away from the base note. This equation is reversed to get us the offset in the function above. Since "A" is the 9th index, we convert the offset to the index by adding 9. Then the function bounds the index if it is too large or small and then returns the index in the `NOTES` and `notes_freq` list. This function is constantly used to determine what note is being played on the continous scale to determine points and visualization of the notes. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C class Note { private: ... public: Note(float freq, int inp_y = 15) { this->y = inp_y; int idx = find_closest_idx(freq); this->x = 12 * idx; this->freq = notes_freq[idx]; } void draw(uint16_t inp_color = TFT_RED) { if (y > 15) { tft.fillRect(x + 1, y -19, 10, 18, TFT_BLACK); } if (y < 115) { tft.fillRect(x + 1, y + 1, 10, 18, inp_color); } y += 20; } int get_y() { return y; } int frequency() { return freq; } }; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The `Note` class creates a note instance for every note in a song. It stores the x and y values to draw on screen in addition to the frequency of the note. The constructor stores the input y value and determines the x value based on the note from the input frequency and the corresponding frequency from that note. The class also has a get_y and frequency method to safely retrieve the y and frequency attribute. The draw method draws a rectangle based on the x and y values and increments the y value by 20 each time it is called so that the note moves down the screen when visualized. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C std::vector parse_song(char* input_song) { std::vector output; char * token = strtok(input_song, delim); while (token != NULL) { float freq = atof(token); token = strtok(NULL, delim); if (token == NULL) { break; } // fix a weird issue of premature null int duration = atof(token); token = strtok(NULL, delim); // appends duration * LOOP_COUNT frequencies for (int i = 0; i < duration * LOOP_COUNT; i++) { output.push_back(freq); } } return output; } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The parse_song function takes in a char_array in the form "freq,duration;freq,duration;" and returns an output vector of frequencies to play. The `input_song` is tokenized into frequencies and durations and frequencies are appended duration*LOOP_COUNT times into the output array. When the songs are pulled from the server, the songs are are not in the proper format to display or play. However, the format they are in saves a lot of space so this function is used to parse the songs into the proper, more memory consuming format as needed. Thus, this function is called to play or listen to a recorded song. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C struct SongEntry { std::string id; std::string text; }; ... std::vector getEntries(char *entries) { std::vector output; char *token = strtok(entries, ENTRY_DELIM); while(token != NULL) { std::string id(token); token = strtok(NULL, ENTRY_DELIM); if (token == NULL) { break; } std::string text(token); output.push_back(SongEntry{id, text}); token = strtok(NULL, ENTRY_DELIM); } return output; } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The SongEntry structure is used to store the song name and id. The getEntries function uses the SongEntry structure to receive a list of song names and ids from the entries char array. The entries array is tokenized and turned into a list of songs to display and select from by making SongEntry and adding it to the output list. The response body from the server returns a char array and this function parses that into titles to match the songs with. This is used to display the songs to choose from when playing. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void get_angle(float* x, float* y) { imu.readAccelData(imu.accelCount); *x = imu.accelCount[0] * imu.aRes; *y = imu.accelCount[1] * imu.aRes; } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This function gets the data from the imu and stores it in the x and y variables which is used to calculate the angle for scrolling through the alphabet array in the NameGetter class. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C class NameGetter { private: char alphabet[50] = " ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"; char entered_name[50] = {0}; ... public: NameGetter(const char* nameType) { // nameType: either "Song" or "User" ... } void get_name(char* output) { // writes the name entered through the imu in output sprintf(output, entered_name); memset(entered_name, 0, sizeof(entered_name)); } void update(float angle, int button, char* output) { ... } }; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The `NameGetter` class was created to be able to create a username to track scores when playing songs or to name new songs after recording them. The `alphabet` array has all the valid characters to use in a username or song name and allows us to use the imu to quickly scroll through it. The `entered_name` array tracks the previous characters in the username/song name. The class has the constructor to initialize a new instance of `NameGetter`, the methods get_name, and update. The get_name method writes whatever is stored in the `entered_name` array to the `output` array and clears the `entered_name array`. The update method uses a state machine with three cases. The first case uses a long press to start entering the name. The second case calls upon the previous get_angle function to scroll through the `alphabet` array by tilting and writing characters to the `entered_name` array through short button presses. The third case then saves what was written using a long button press. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void clearStrip() { for (int i = 0; i < PixelCount; i++) { strip.SetPixelColor(i, RgbColor(0, 0, 0)); } strip.Show(); } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This function goes through each LED on the strip and turns it off to clear the strip. After each loop of visualization, this clears the previous loops LEDs and is especially important as the LED draws a significant amount of power and can cause a brownout if too many LEDs are on. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void initializeWifi() { ... } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This helper function is used to connect to the wifi using the network info on setup. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void setup() { ledcSetup(10, 60, 12); ledcAttachPin(13, 10); ledcWrite(10, 4095); ... char request[100]; sprintf(request, "GET https://608dev.net/sandbox/sc/kgarner/project/server.py HTTP/1.1\r\n"); strcat(request, "Host: 608dev.net\r\n\r\n"); do_http_request(host, request, response_buffer, OUT_BUFFER_SIZE, RESPONSE_TIMEOUT, true); menu = getEntries(response_buffer); songEnd = min(ENTRIES_PER_SCREEN, menu.size()); ... } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Upon setup, the ESP32 clears the screen, sets up the buttons, connects to the wifi, clears the LEDs, starts the imu, TOF sensor, and amplifier along with enabling the PWM for the monitor to control brightness and save power. It also sends a GET request to the server and pulls all the possible songs to choose from. This is later parsed and shown when picking songs to play. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void playNote(int inp_freq = 0) { if (listen_song == false){ mm = sensor.readRange(); older_mm = old_mm; old_mm = mm; // get an input from TOF and wrap it in the range C3 to C4 // for all the semitones avg_mm = ((mm+old_mm+older_mm)/3); avg_mm = map(avg_mm,0,200,-9,3); // scale the range if (avg_mm < -9) { avg_mm = -9; } else if (avg_mm > 3) { avg_mm = 3; } clearStrip(); int pixelIdx = (avg_mm + 9) * 2 + 1; strip.SetPixelColor(pixelIdx,RgbColor(255, 0, 255) ); strip.SetPixelColor(pixelIdx + 1,RgbColor(255, 0, 255) ); strip.Show(); // scale to frequency using power and play song avg_mm = 440 * pow(2, avg_mm / 12.0); } else{ avg_mm = inp_freq; } if (old_freq != avg_mm) { ledcWriteTone(0, avg_mm); ledcWrite(0,100); old_freq = avg_mm; } } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The playNote function is the heart and soul of the theremin and is heavily used in all three modes to play whatever frequencies are needed. When listening to a song (listen_song == True), the function takes in a frequency and writes that to the amplifier. To deal with the clicking when writing a frequency to the amplifier, a frequecy will only be written if it is different from one currently being played which minimizes the amount of times the amplifier has to play a sound and reduces extraneous noise. When playing something from the theremin, the function gets a distance reading from the TOF sensor in millimeters this is then averaged to account for any spikes due to errors in the sensor. The distance is then mapped and scaled to a note based on the distance. The note is represented in numbers and the LEDs around the note light up. The note is then converted to frequency using the previous equation $f_n = f_0*a^n$ with the base note being "A" at 440 Hz. This is then written to the amplifier as before. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C Note note_play = Note(avg_mm, 95); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This variable is used to keep track of what note the theremin is currently playing and displays it to the player. The takes in the frequency from the sensor and writes it to the bottom row of notes over the upcoming if there are any as a blue box. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void handleNotes() { note_idx++; // add Notes while we have not gone through the song if (note_idx < freqs.size()) { notes.push_back(Note(freqs[note_idx])); } // draw all of the notes for (Note ¬e : notes) { note.draw(); if (note.get_y() == 95 && listen_song == true){ // play and display note } } // while we have notes if (notes.size() > 0) { // have we started playing a note? if so, play the note if (note_idx > 4 && notes.size() > 1 && listen_song == false) { playNote(); // score the current note with what is being played int expected_note = find_closest_idx(notes.front().frequency()); int actual_note = find_closest_idx(avg_mm); int lowPixel = expected_note * 2; int highPixel = lowPixel + 3; strip.SetPixelColor(lowPixel,RgbColor(0, 255, 255) ); strip.SetPixelColor(highPixel,RgbColor(0, 255, 255) ); strip.Show(); Serial.printf("Expected: (%d, %d), Actual: (%d, %d)\n", expected_note, notes.front().frequency(), avg_mm, actual_note); if (expected_note == actual_note) { score += 10; itoa(score, score_str, 10); } } // remove Notes that have expired if (notes.front().get_y() > 115) { notes.pop_front(); } } if (listen_song == false){ // move the note to the current location we are playing ... } // if all the notes are done, go to show score mode if (note_idx >= freqs.size() && notes.size() == 0) { tft.fillScreen(TFT_BLACK); state = SHOW_SCORE; note_idx = 0; listen_song = false; ledcWrite(0,0); clearStrip(); } } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The handleNotes function deals with the bulk of the visualization of the notes along with anything else involving the notes in the play or listen mode such as audio or scoring. There is a running note_idx variable to track where in the song we are. The `notes` list tracks what notes need to be drawn and if there more notes left in the song it adds it to `notes` and removes what notes no longer need to be drawn. The function loops through all the notes in `notes` calling the Note draw method and if we are listening to the song it will display the bottom note on the LED and play it. While there are notes left, the function calls playNote and plays the note based on the TOF and finds the index of the note to turn on the corresponding LEDs. It then compares the note from the TOF to the note from the song and if they match, increments the score. If we are playing the song, it also shows what notes we are playing on the tft and LED. Once the song is done, the appropriate variables and parts are reset or cleared to show us the score. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void showScore() { tft.setCursor(0, 0); tft.printf("Your score: %d, %s", score, songId); // button pressed if (bottomPin.update() != 0 || topPin.update() != 0) { selectedOption = 0; state = MENU; tft.fillScreen(TFT_BLACK); // post the score to the server ... do_http_request(host,request,response_buffer,OUT_BUFFER_SIZE, RESPONSE_TIMEOUT,true); state = MENU; } } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This function takes the gets the global score variable which holds the score from the last song played and displays it on the tft. Once a button is pressed, the score is uploaded to the server in a POST request using the do_http_request helper function and sends us back to the main menu. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void handleMainMenu() { tft.setCursor(0, 0); for (int i = 0; i < 3; i++) { tft.printf(selectedOption == i ? "-> ": " "); tft.printf("%s\n", selectionScreenOptions[i].c_str()); } int top = topPin.update(); int bot = bottomPin.update(); if (top == 2 || bot == 2) { // toggle powersave mode ... tft.fillScreen(TFT_BLACK); } else if (top != 0) { // toggle selected option selectedOption = (selectedOption + 1) % 3; } else if (bot!= 0) { // select the mode switch (selectedOption) { ... } } } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This function draws the main menu and deals with functionality for it. A long button press here dims the tft and LED to save power. Hitting the top button chooses the mode and the bottom button selects it, changing the state. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void drawGameScreen() { ... } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This function draws the different semitones and sets up the game screen to display notes. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void drawFreePlayScreen() { ... } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This function is repeatedly called when in the freeplay mode. It draws the free play screen which includes where the current note is on the tft and LED and rest of the semitones while also playing the note based off of the TOF. A button press sends us back to the main menu. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void drawRecordScreen() { ... //are we done yet??? if (recordCount < 150) { recordCount++; tft.setCursor(0, 0); float timeLeft = (150 - recordCount) / 10.0; tft.printf("Time remaining: %f seconds\n", timeLeft); // same note previously if (old == avg_mm){ count += 1; old = avg_mm; } else { // add a new note char output[15]; sprintf(output, "%d,%d;", old, count); strcat(song, output); old = avg_mm; count = 1; } } // when we are done, post if (recordCount == 150) { state = SONG_NAME_ENTRY; ledcWrite(0,0); clearStrip(); } } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This function is extremely similar to the previous function. However, it also tracks what notes are being played in the output variable in the form "freq,duration;". A button press won't send this back to the menu but hitting the time limit will and leaving the state uploads the song in output to the server. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void drawPlayOrReselectScreen() { ... } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Once a song is chosen the player can choose to play it (short top button press), listen to it (long top button press), or choose another song (bottom button) which can be done here. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void handleSongNameEntry() { float x, y; get_angle(&x, &y); //get angle values int bv = topPin.update(); //get button value songNameGetter.update(-y, bv, songEntryResponse); if (strcmp(songEntryResponse, oldSongEntryResponse) != 0) { ... tft.println(songEntryResponse); } memset(oldSongEntryResponse, 0, sizeof(oldSongEntryResponse)); strcat(oldSongEntryResponse, songEntryResponse); if (saved) { saved = 0; songNameGetter.get_name(songName); ... do_http_request(host,request,response_buffer,OUT_BUFFER_SIZE, RESPONSE_TIMEOUT,true); tft.fillScreen(TFT_BLACK); state = MENU; } } ... void handleUserNameEntry() { ... } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This function handleSongNameEntry uses the imu to calcuate the tilt of the theremin and using an instance of `NameGetter` creates a new song name to upload to the server using a POST request. handleUserNameEntry is nearly identical but uses a different `NameGetter` and EntryResponse instance. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void handleSongMenu() { ... if (bottomPin.update() != 0) { sprintf(songId, menu[selectedSong].id.c_str()); ... do_http_request(host, request, response_buffer, OUT_BUFFER_SIZE, RESPONSE_TIMEOUT, true); freqs = parse_song(response_buffer); state = PLAY_OR_RESELECT; tft.fillScreen(TFT_BLACK); } else if (topPin.update() != 0) { selectedSong++; if (selectedSong == menu.size()) { ... } else if (selectedSong % 16 == 0) { ... } } } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The function displays all the songs pulled from the server on startup. Clicking the top button switches between songs and clicking the bottom button chooses the song and pulls the song from the server using a GET request and parses the response into the proper song format. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void handleGameState() { switch (state) { case USERNAME_ENTRY: handleUserNameEntry(); break; case MENU: handleMainMenu(); break; ... } } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This function is called every loop and is a massive state machine. Each state calls a corresponding function to deal with everything in that state. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void loop() { imu.readAccelData(imu.accelCount); float x = imu.accelCount[0]*imu.aRes; primary_timer = millis(); handleGameState(); while(millis() - primary_timer < LOOP_PERIOD); } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The loop calls handleGameState which has cases to deal with all the different states in the game. It also updates the data from the imu and repeats after a delay. **`Button.h`** ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C class Button { ... public: Button(int p); void read(); int update(); }; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The Button.h file holds the declaration of the button class used to differentiate between long and short presses and debounce the button. It holds a constructor, read, and update function defined in Button.cpp. **`Button.cpp`** ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C Button::Button(int p) { flag = 0; state = 0; pin = p; t_of_state_2 = millis(); //init t_of_button_change = millis(); //init debounce_time = 10; long_press_time = 1000; button_pressed = 0; } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is the button constructor used to initialize objects of the button class seen in Button.h. In the code, this is used to create the top button and bottom button. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void Button::read() { uint8_t button_state = digitalRead(pin); button_pressed = !button_state; } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The read method reads the button's value to update the value of the button being pressed. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C int Button::update() { read(); flag = 0; switch (state) { ... } return flag; } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The update method returns the flag int which tells us if the button has been pressed or not and whether that was a short or long press. The method uses a state machine, implemented uses a series of cases and timers tracking the amount of time the button was held down to return the type of button press on release. The timers also do not register the button press if it is less than the debounce time. The flag returns 0 for no button press, 1 for a short button press, and 2 for a long button press. **`support_functions.ino`** ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C uint8_t char_append(char* buff, char c, uint16_t buff_size) { int len = strlen(buff); if (len>buff_size) return false; buff[len] = c; buff[len+1] = '\0'; return true; } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If possible, the char append function appends the what is in the char array c to the buffer and returns true, otherwise it returns false and which means that the buffer is full. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C void do_http_request(char* host, char* request, char* response, uint16_t response_size, uint16_t response_timeout, uint8_t serial){ ... } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This helper function takes care of the http get and post requests. While connected to the host, it sends the specified request and reads out the response. Server ------------------------------------------------------------------------------- ### Songs * We used an SQLite database to save songs recorded through the ESP32 and POSTed to the server or directly uploaded through the project’s website. * Each of the song table entries consists of a song ID generated through the database, a song name chosen by the uploader of the song, a text of the form `freq,cycles;freq,cycles;...` representing the song itself, and a timestamp representing when the song was uploaded. * POST format involves 2 arguments: `songName` representing the name of the uploaded song and `musicString` representing the body of the song. * GET requests are used to return the IDs and the names of all the songs in the database if no arguments are passed, or the name of a single song if a song ID argument is passed in. ### Scores * We created a separate table in the same database to save scores POSTed through the ESP32. * Each of the score table entries consists of a score value, an ID referencing a specific song in the song table, a username value representing the player with that score, a timestamp representing when the player got that score for that particular song. * POST requests involve the following arguments: `userName`, `songId`, and `score`. * The leaderboard of each song in the song table can be viewed on the front-end website in a descending order by score value. GET request format to view the leaderboard for a song involves a `songId` argument. * Adding a `userName` argument to a GET request returns all the scores by the player represented by `userName` for the song represented by `songId`. Website ------------------------------------------------------------------------------- Our [front-end website](https://608dev.net/sandbox/sc/kgarner/project/html/simple-form.html) is a simple single-page app using the Bootstrap framework. The website has the following capabilities: * **View songs**: You can see a list of all the songs that have been uploaded to server. This is updated whenever the user visits the page. * **View a specific song**: : You can listen to a single song (in OGG format), as well as see any scores associated with it. You can also filter to see the high score of a single person. * **Upload a song**: You can upload a song using our song-file format. Our format is the following: `SONG_FILE ::= (frequency “,” duration “;”)*` where frequency is a float corresponding to a note frequency, and duration is how long the note should be played (measured in 100 ms blocks) Energy management =============================================================================== To conserve power when using Theremin Hero, we have a special power-saving mode which can be toggled by long pressing in the main menu. In power-saving mode, the display brightness is reduced to 1/8 (from 4095 to 511 PWM), and the brightnes of the LEDS is reduced (from 255 to 8). | State | Regular usage | Power saving usage |-----------|---------------|------------------- | Menu | ~120 mA | ~110 mA | Freeplay | ~230 mA | ~140 mA | Challenge | ~230 mA | ~140 mA | Record | 180-230 mA | 110-150 mA Assuming that, on average, a user spends 10% of their time in menus and 90% of their time playing, and the battery capacity is 1500 mAh, we can calculate the usage time saved. ### Regular usage **Average total input current** = $230 mA \times 0.9 + 140 \times 0.1 = 221 mA$ **Maximum usage time** = $\frac{1500 mAh}{221 mA} = 6.79$ hours ### Power saving usage **Average total input current** = $140 mA \times 0.9 + 110 \times 0.1 = 137 mA$ **Maximum usage time** = $\frac{1500 mAh}{137 mA} = 10.95$ hours Comparing the maximum usage time for regular and power saving, we see a 60% improvement.