**Laser Harp Guitar Hero** Julia Gonik, Tiffany Huang, Hyunji Kim, and Joie Le
Team 77 Overview ============================================================== One of our favorite childhood games is Guitar Hero. One day, whilst reminiscing about the carefree joys of youth, we decided that there was no better way to pay homage to a classic video game like Guitar Hero than to recreate it with an EECS embedded systems twist. Thus, Laser Harp Guitar Hero was born. Instead of pressing buttons and strumming on a contrived, cliche guitar-shaped game controller, users break lasers in order to "pluck" the laser harp and play notes. Players choose from a plethora of songs, including classic Guitar Hero tunes in addition to some of our present-day favorites. The user earns a score based on how well they pluck the notes at the appropriate times, and the system keeps track of the user's name and scores in a server-side database that contains leaderboards for each song. Challenge your friends and determine the true laser harp master! Below is a video of our group demonstrating the game. How fun!

Laser Harp Guitar Hero from Julia Gonik on Vimeo.

Watch Julia struggle!
System Design ============================================================== Below is an overview of the design of our system. The vast majority of our project code was written for the ESP32, as this is where the majority of the game logic is handled. The Python server code was mostly used for storing persistent data in databases, including note information for songs and users' scores for the leaderboards. In addition, we wrote helper functions in Python that allowed us to easily create new note mappings for songs using our computers' keyboards instead of manually figuring out note timings, which would be tedious. You can find all of our code here. Software -------------------------------------------------------------- **Server-Side and Helper Code** ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ leaderboards.py ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This code lives on the server and handles get requests for obtaining leaderboard information and post requests for adding new information to the leaderboards database. This is all handled in the appropriately named `request_handler` function. First, we create a connection to our leaderboards database. Then, if the `scores_table` does not already exist, we create it. This table has three columns: score, song name , and username. We take the values from the body of the post request and insert them into their respective columns. In particular, the body of the post request should have three keys: 'score', 'song', and 'user'. We then close our connection to the database, and the post request is complete. If a get request is made, we follow a similar process. First, we extract the name of the song that we want the leaderboards of. This name should come from the query argument 'song'. Then, we get all of the rows in the database with that song name and order them in descending order by score. This is so that it will be easier to display the top scores in order on the ESP32 display. In addition, we only want the top ten scores, so we limit our results to ten rows. Then, we create a string called `outs` that will hold the output that we want to send back to the ESP32. After that, we iterate over all the rows that we retrieved from the database and add them to the `outs` string with the following format: `"[score,username]"`. Thus, if your username is joules and you got a score of 18, the format would be `"[18,joules]"`. Multiple scores are represented as follows: `"[55,user1] [17,user3] [3,user2]."` Finally, we close our connection to the database and return this string to the ESP32 for parsing. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ get_song.py ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This code is also on the server. It handles get requests that are sent when the user has selected a song and the ESP32 now needs to retrieve the note mapping for that song. This functionality occurs in the `request_handler` function. First, we check that the request is a get request, since this code is only designed to return values from our database rather than post any new values to the server. After that, we connect to our songs database and retrieve the name of the song from the request. In particular, the name of the song is found in a query argument called 'song'. Next, we get all of the values from the table with the same name as the song in our database. This database contains information about all of the notes that are played during gameplay of a song. It stores which of the four notes is played, the time it started playing, and its duration. See `song_timing.py` for more information about how we obtain and store this information. Then, we close the connection to the database and return our results as a string to the ESP32 for parsing. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ get_songlist.py ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This file lives on the server and works to handle get requests for information about all of the songs that we have created notes for. Particularly, when a get request is made to this file, it returns a list containing the song name, artist name, and song duration for each song in the database. All of this logic is handled in the `request_handler` function. First, we establish a connection to our database. Then, we select all of the values from the table called songlist. This table has 3 columns: a column for the song name, a column for the artist name, and a column for the song duration. After fetching all of the rows from this table, we close our connection to the database and return the table information to the ESP32 as a string for further parsing and use. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ song_timing.py ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This file is the basis for the note-mapping. On a high level, when this file is executed, a chosen song is played locally on the user's computer, and the developer will hit the `a`, `s`, `d`, or `f` keys to indicate that this note (representing a corresponding laser) should be broken at this time during the Laser Harp Guitar Hero game. These keys can also be held to represent a prolonged note during the game. For this kind of note, the player must hold their hand in the laser for the duration of the note in order to gain points. Pressing the space bar will end the song. In-depth discussion of each function is below: - Note that a note keypress of `a` represents the leftmost laser on the laser harp, `s` is the second laser, `d` is the third, and `f` is the final, rightmost laser. - `def map_notes_to_song(song, artist)` - Takes in the name of the song and artist. Plays the song through the `playsound` Python library. - Uses the song name to create a table in the database. We replace spaces with `_` and remove quotation marks and commas from the song name to make interactions between the ESP and server easier. To delete any previous iteration of the song's beatmap, any existing table for the song is dropped from the database before a new fresh one is created. - The table has the columns `(key text, start real, duration real)`, where key is one of the four notes (`a`, `s`, `d`, or `f`), start is the start time of the note as a float, and duration is how long the note is held for. - To keep track of notes, our code detects when `a`, `s`, `d`, or `f` is pressed and then released using the `pygame` and `time` libraries. It keeps track of these events in a list called `all_notes`. After the song is over, we iterate through this list and add the appropriate information for each note to the `beatmaps.db` database. This database is then stored on the server where it can be accessed via get requests from the ESP32. - The beatmap generation stops when the spacebar is pressed, terminating the code. The song, artist, and song duration (based on when the spacebar is pressed) are recorded as a new row in a table called `songlist`, which holds all songs with available beatmaps for our game. Specifically, `song_list` has three columns: a column for the song name, a column for the artist name, and a column for the duration of the song in seconds. Each row represents a particular song. If the song already exists in the table, its duration is simply updated in accordance with the new input. - `def delete_song()` - An additional function which is able to delete songs from the `songlist` table in the `beatmaps.db` database. It is used when we want to redo the beatmap for a certain song or remove a song altogether. **ESP32 Libraries and Classes** We wanted to modularize our code in order to make it neater and easier to work with. This also eliminates the need for repetitive code that is both unnecessary and hard to debug. Thus, we split the main functionalities of our game into distinct classes and libraries that each represent one of the game's core features. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ laser_harp_hero.ino ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This file contains the main driver code for our project. This code is uploaded to the ESP32 and contains the necessary `setup()` and `loop()` functions for complete operation. It sets up the WiFi, the TFT display, and the MP3 player, as well as intializes a `Game` object. In `loop()`, this file makes repeated calls to the `gamePlay()` function (see below) of the `Game` object, updating the state of the game appropriately with each pass. It also performs timing measurements and markings necessary for proper functionality of the game as well as carries out HTTP requests when necessitated by the game's mechanisms. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Game Library ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This library controls the state machine and making updates to the state of the game. Combining all the various software components of our game, it makes calls to different classes and functions and integrates the libraries to form a more fully-fledged user experience from start screen to gameplay to leaderboard display. It's also responsible for parsing data about beatmaps. - `Game(Adafruit_RA8875* input_tft, DFRobotDFPlayerMini* input_mp3_player)` - Constructor for Game objects - Takes in a pointer to the screen and MP3 player so that they may be used within the class - Initializes all four LaserString objects for our four notes (green, red, yellow, blue), initializes arrays of RectNotes to later hold beatmaps, sets score to 0 (start of a game), and state to go into the starting state - `void setUpLED()` - Readies each of the four LaserString's LEDs to be able to toggle on and off - `void gamePlay(int elapsed, char* request_buffer, char* reponse_buffer)` - Controls the state machine that underlies the game, called repeatedly by the `loop()` function of the ESP32 in order to constantly update game state - Takes in `elapsed`, or the length of the song that has been played once gameplay has actually begun, `request_buffer` and `response_buffer` to be able to interact with HTTP requests and responses - Transitions between states, described in the state diagram below--each function of each state is typically described by a class (see libraries below) - When in the game play state itself, `gamePlay()` updates each RectNote object so that it appears as if the notes are moving down the screen, as well as calls functions in LaserString so that LEDs are lit up appropriately and user input and scores are accounted for - `int getState()` - Returns the current state of the score - Important for calculations of `elapsed` in `laser_harp_hero.ino` (since we are focused on the gameplay state with this parameter) - `int getScore()` - Returns the current score that user has obtained in gameplay mode - `void getSongData(char* request_buffer)` (private) - Constructs the HTTP request buffer which is then sent up to the server by `laser_harp_hero.ino` in order to retrieve the beatmap of the selected song - `void getSongList(char* request_buffer)` (private) - Constructs the HTTP request buffer which is then sent up to the server by `laser_harp_hero.ino` to obtain the full list of song options that can be played - `void parseSongData(char* response_buffer, char* note_arr, float* note_time_arr, float* duration_arr)` (private) - Takes in the response buffer, which holds the beatmap in a string which needs to be parsed for data, that is returned by the HTTP request constructed in `getSongData()` - Parses string to isolate each individual note in the beatmap - These notes are separated into specific laser to hit, time to hit, and duration to continue breaking the laser, which are stored in corresponding entries in the `note_arr`, `note_time_arr`, and `duration_arr`, respectively, that are passed in - `void extractTimes(char* note_arr, float* note_time_arr, float* duration_arr)` (private) - Takes the three arrays, now filled with beatmap data from `parseSongData()`, and initializes a RectNote object to represent each note in the beatmap - RectNote objects will match in terms of length (representing duration of the note) and color (LED color for the corresponding LaserString) - RectNote objects are stored in the appropriate array of the four based on which specific laser should be broken (there are four RectNote arrays, one per laser string) - `void same_song()` (private) - Readies the game so that the player can replay the same song - Resets the game (score, tracking of which RectNotes to display, etc.), but maintains the username of the player and the song - `void new_round_same_user()` (private) - Readies the game so that the player can choose a new song to play - Resets the game (score, tracking of which RectNotes to display, song, etc.), but maintains the username of the player - `void reset()` (private) - Resets all aspects of the game, bringing it back to the start state, leaving all attributes as if the Game object were just initialized - Lets a new user enter their name and play ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ LaserString Library ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This library wraps up all components of a "laser string" on our harp into one class. It contains functions necessary to time the toggling of the LEDs and to read user input ("plucking of the string"/breaking the laser) as well as scores any actions made by the player. - `LaserString(int LED_pin, int analog_pin)` - Initializes the pin of the LED and the pin of the phototransistor that will be used to read voltage - Also initializes other attributes important to the functionality of the class, such as for taking in user input and scoring actions - `void beginLights()` - Sets up the pin mode (output) of the LEDs and turns them off to begin - `void LEDControl(bool on)` - Toggles the LED on and off based on whether the boolean `on` is true or false, respectively - `bool broken()` - Calculates the voltage in decimals based on what the ESP reads from the phototransistor - If this voltage is above a threshold of 1.5V, we return that the laser has been broken; otherwise we return false - `void setRefTime(int time_ms)` - Sets a correct reference time for when the laser should be broken (stored in `ref_hand_in_time`) - Used so we can check user actions for accuracy - Updated each time it's indicated by a RectNote that we should hit this specific LaserString - `void toScoreNote()` - Sets a instance variable `scored` to false - Tells us that the most recent user action has not been scored - `void userAction()` - Marks the time in `actual_hand_in_time` whenever the user puts in their hand to break a laser - This is only updated if the user has taken their hand out, then put it back in (time recorded on the transition from unbroken to broken) - `int scoreAction(int time_ms)` - Compares the time difference between `ref_hand_in_time` and `actual_hand_in_time` to determine how accurate the user was - A difference of 10 ms is "Perfect", adding 5 to the score - A difference of 25 ms is "Great", adding 3 to the score - A difference of 50 ms is "Good", adding 2 to the score - A difference of 100 ms is "OK", adding 1 to the score - Anything else is a "Miss", and doesn't impact score - `void displayFeedback(int time_ms, Adafruit_RA8875* tft, int text_loc)` - Displays the feedback (the level of accuracy of the score--different possibilities described in `scoreAction` description) - Written below where the RectNote object hits the white bar on the screen - The feedback lingers for half a second before disappear, offering players the time to read what's written - `void reset()` - Resets the class to how it is when initialized--no reference times or actual times for hand-in, no feedback, etc. - Useful for when we want to restart the game to a state before gameplay has begun ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ RectNote Library ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This library creates a rectangular note that displays on the TFT screen for every note in the beatmap and controls the movement of these rectangles down the screen. These rectangles signal a player when to hit a certain laser. - `RectNote()` - An empty constructor for when we want to declare an array of RectNotes without initializing values yet - `RectNote(float start_time, float end_time, int dur, int wid, int x, uint16_t color)` - Constructor which initializes various attributes of a RectNote object - Start time and end time as well as duration `dur` are passed in at the time of creation depending on the object's corresponding note in the beatmap - Width `wid` and starting `x` position, as well as color (should match LED) are all predetermined by which laser the RectNote corresponds to - Widths are always 150 pixels - The possible `x` positions in order are 50, 250, 450, 650 - `void update(int screen_bottom, Adafruit_RA8875* tft)` - Calls the functions `drawRect`, `updateLength`, and `updateYCoord` below to shift the RectNote forward one timestep so that it looks like it's moving down the screen - Covers up old location, then redraws the rectangle in its updated location and with its updated dimensions - Also determines whether a rectangle has passed out of bounds (i.e. the note it represents is out of bounds) - `bool toPress()` - Returns whether the RectNote's corresponding laser should be broken - Determined by whether the RectNote has hit the white boundary at the bottom - `bool passed()` - If the rectangle is out of bounds, then we return true - Otherwise return false - `float getStart()` - Gives the `start_time` of the rectangle's corresponding note in the beatmap, noted when creating the RectNote - `float getEnd()` - Gives the `end_time` of the rectangle's corresponding note in the beatmap, noted when creating the RectNote - `void updateLength(int screen_bottom)` (private) - Used when the RectNote has just started appearing on the screen and isn't fully displayed yet - Also used when RectNote is at the bottom of the screen and is disappearing - Appropriately grows/shrinks the length of the rectangle by 3 pixels to make it appear as if the rectangle is sliding in or out of the view of the screen - `void updateYCoord(int screen_bottom)` (private) - Used when the rectangle of RectNote is fully on the screen - Move the entire rectangle down by 3 pixels by adding 3 to its y-coordinate - `void drawRect(Adafruit_RA8875* tft, bool old)` (private) - Draws the rectangle using TFT method `fillRect` - If `old` is set to true, we cover up the old locations rectangle by redrawing in black to match the background color - Otherwise, we are drawing the rectangle in its new location ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ StartScreen Library ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This simple library displays a welcome message to the user, acting as a home page for the game. This display is visited when the user starts / restarts the game. - `StartScreen::StartScreen()` - An empty constructor - `void StartScreen::display_entry(Adafruit_RA8875* tft)` - Passes in the display screen to be written on and writes the following text: - "Laser Harp Guitar Hero" - "Hit the leftmost laser to start" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ UsernameGetter Library ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The UsernameGetter library implements the functionality for choosing a username by breaking lasers. The following functions describe the implementation: - `UsernameGetter::UsernameGetter()` - The UsernameGetter class constructor has no parameters, but initializes several private variables. `alphabet` is a constant char array of the letters in the alphabet plus a blank space in the front. `query_string` holds the text that the user is building. 'char_index' refers to the position in `alphabet` that the player is currently in. `scrolling_timer` keeps track of how long it has been since the user last scrolled. The constant `scrolling_threshold` is set to 500 ms. to prevent users from scrolling too quickly. `choosing_threshold` keeps track of how long it has been since the user has chosen a character. The constant `choosing_threshold` is set to 750ms to prevent users from choosing characters too quickly. - `void UsernameGetter::update_name(int input, char* output)` - A public function that updates `output`, the text shown to the user, based off of `input`. An `input` value of 1 corresponds to scrolling left, and `input` value of 2 scrolls right, an `input` value of 3 chooses a letter. Each of these numbers matches up with the laser (i.e. leftmost laser is one and so on). The fourth laser confirms the username and sets the state to the next state (Song Selection) within the `Game` class. - Within each case, the user can only scroll left/right at a minimum of every 500 ms. The same holds true for choosing a letter but at 750 ms. This improves the usability of our interface because it prevents users from scrolling too quickly and thus not being able to see the choices clearly. - Once the user chooses a letter, `char_index` is set back to 0 so that we start at the beginning of the alphabet for the next time a user needs to choose a name. - `void UsernameGetter::set_char_index(int num)` - A public function that will set the `char_index` variable to the input `num`. - When a user wants to restart the game, we reset the index back to the starting point. - `void UsernameGetter::clear_query()` - A public function that will clear out `query_string` using the function `memset`. - Once a user wants to restart the game, we want to make sure the username displayed on the screen is not the prior user's. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ SongSelection Library ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The `SongSelection` library is responsible for getting data about songs from the server, parsing that data, and displaying it to the TFT display so that users can select the song that they want to play. It does so using the various functions described below: - `SongSelection::SongSelection()` - The SongSelection class constructor. It takes no arguments, but it has several private field variables: an array of strings called `songs` that represents the available songs that the user can choose from, an array of artists called `artists` that represents the singers or bands that perform each song, an array of durations called, you guessed it, `durations` that stores the lengths of the songs, an integer called `array_size` that stores the length of the aforementioned arrays, an integer called `scrolling_timer` that keeps track of how long it has been since the user last scrolled, an integer called `scrolling_threshold` that is set to 750 and represents the time a user must wait before scrolling again, a string called `old_song` that keeps track of the previous song, and finally a string called `selected_song` that keeps track of the current song. - Initially, `curr_index` is set to zero, `array_size` is set to zero, and `scrolling_timer` is set to `millis()`. - `void SongSelection::get_song_selection(char* request_buffer)` - This function builds a request buffer that will be used to get the song list from the server. It takes in a single argument: a pointer to a `char` array `request_buffer` that will be modified to represent our get request. - The function concatenates the appropriate URL to the request buffer. Note that nothing is returned because `request_buffer` is directly modified. - `void SongSelection::parse_song_selection(std::string str)` - This function parses the response from the get request in the previous method so that it is stored in the array fields discussed in the constructor. It takes in a single parameter: `str`, which is a string version of the response buffer from the aforementioned get request. - We initialize `array_index` to zero. This variable represents the current index in the array that we are modifying. - First, we will continue parsing while the length of the string is greater than zero. We find the first apostrophe because it represents the start of the name. Similarly, we find the next apostrophe because it represents the end of the name. We then take the substring of the response representing the song name and add it to the next open index in the `songs` array. - Similarly, we find the next apostrophe, which signifies the end of the artist's name. We then take the substring of the response that represents the name and add it to the next open spot in the `artists` array. - Next, we find the index of the next closing parentheses, which signifies the end of the duration. We take the substring representing the duration, convert it to a float, and then add it to our `durations` array at the next open position. - Finally, we increment the `array_index` by one for the next iteration. We continue until the length of the string is zero. - The `array_size` field is set to the last value of `array_index`. - `void SongSelection::update_song_index(bool forward)` - This function modifies the `curr_index` field variable. It takes in a single parameter: `forward`, which is a boolean. It is `true` if we should scroll forward in the songs list, and `false` if we should scroll backwards. - First, we increment `curr_index` if `forward` is true, and otherwise decrement it. - Then, we make sure that `curr_index` is not out of bounds. If it is greater than or equal to `array_size`, we set it to zero because the function scrolls circularly. Similarly, if it is less than 0, we set it to the last valid index in the array. - `void SongSelection::display_song_selection(Adafruit_RA8875* tft)` -This function displays information about the current song on the TFT display. It takes in a pointer to the tft object. -It accesses the `songs` array, `artists` array, and `durations` array at `curr_index` in order to extract the current values. It then uses the tft object's methods to display these values to the screen. - `std::string SongSelection::get_curr_song()` - A simple getter function that returns the name of the current song. It does so by returning the value of the `songs` array at `curr_index`. - `void update_screen(int input, Adafruit_RA8875* tft)` - This function updates the TFT screen when the user breaks a laser to scroll through song options. It takes an integer `input`, which represents the number of the laser that was broken, in addition to a pointer to a tft object. - Firstly, if it has been at least `scrolling_threshold` milliseconds from the last time the user scrolled, the function sets `old_song` to the value of `selected_song`. It then calls `update_song_index(false)` to scroll backwards and updates the value of `selected_song` using `get_curr_song()`. Then, if the old and new songs aren't the same, it calls `display_song_selection(tft)` to display the new song information to the screen. It then resets `scrolling_timer` to `millis()` - A similar process is followed if the second laser is broken in order to scroll forward through the songs. The only difference is that this time, 'update_song_index` is called with `true` in order to go forwards instead of backwards. - If `input` is three, the third laser has broken, which means that the user selected a song. `selected_song` is set to the current song, and then the function displays confirmation that the user selected this song to the TFT screen. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Leaderboard Library ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This library is responsible for getting the leaderboard data from the server, parsing the server's response, and displaying the leaderboard information to the TFT screen. It does so using several functions: - `Leaderboard::Leaderboard()` - The Leaderboard class constructor. It takes in no arguments, but it has several private field variables: an array of scores represented as integers called `scores`, an array of usernames represented as strings called `users`, a string representing the song's name called `song_name`, and an integer `array_size` representing the size of the `scores` and `users` arrays. `song_name` is first initialized to an empty string and `array_size` is first set to zero. - `void Leaderboard::setSongName(std::string song_name)` - A simple function that sets the `song_name` private field variable to the parameter that is passed in. - `void Leaderboard::postToLeaderboard(char* request_buffer, int score, std::string user)` - Formats the request buffer for a post request to add a new score for the song represented by `song_name` to the server-side database. It takes in a pointer to a `char` array `request_buffer`, an integer `score`, and a string `user`. - The function first creates the body of the post request using the parameters in addition to the `song_name` field variable. There are three keys that need to be included in the body of the post request: score, user, and song. - Then, it builds the request_buffer of the post request in the same way that we did during labs and exercises. - Note that the function modifies the request_buffer `char` array and thus does not need to return anything. - `void Leaderboard::getLeaderboard(char* request_buffer)` - Formats the request buffer to make a get request for the leaderboard that is represented by the `song_name` field variable. It takes in a single parameter: a pointer to a `char` array called `request_buffer`. The function modifies this `request_buffer` by adding `song_name` to the appropriate URL. It does not return anything since `request_buffer` is directly modified. - `void Leaderboard::parseLeaderboard(char* response_buffer)` - Parses the `response_buffer` from the get request performed using the `request_buffer` from the previous method. The results are stored in the `users` array and the `score` array, which are private fields of the class. - First, the function converts the `char` array `response_buffer` to a C++ string `response` in order to make it easier to parse. - We also initialize an integer `array_index` that keeps track of the current index in the `scores` and `users` array that we are modifying. - Then, we begin to parse the string and continue to do so while the size of the string is greater than zero. First, we look for the index of the first comma, which signifies the end of the score. We then take the substring of `response` that represents this score, convert it to an integer, and then add it to the next open position in the `scores` array. Next, the string is spliced to remove the score that we just added. - After that, we find the next closing square bracket ("]"), which signifies the end of the user. We then take the substring of `response` that represents the user and add it to the next open position in the `users` array. After that, we splice the string to remove the user that we just added and increment `array_index` by one. The process continues while the string is not empty. - Finally, the `array_size` field variable is set to the final value of `array_index` after the while loop terminates. - `int Leaderboard::displayScore(Adafruit_RA8875* tft, LaserString* string_1, LaserString* string_2, LaserString* string_3, LaserString* string_4, int score)` - This function displays the user's score immediately after the game is over. It takes in a pointer to the TFT screen object, four pointers to `LaserString` objects that represent each of the lasers, and an integer `score`. - First, it displays the user's score using the TFT's built-in display methods. Note that since we are using a different, larger screen than we did for the labs, these methods are slightly different than the ones we used for labs and exercises. - Next, it displays instructions to the user on how to navigate away from this screen. For instance, the user can press the first laser to replay the song, the second laser to choose a new song, the third laser to view the leaderboards for the song, and the fourth laser to restart the game with a new username. - The function then checks if any of the lasers were broken using methods of the `LaserString` objects in order to handle the appropriate state transitions. See the description of the `LaserString` library for more details regarding these functions. - The function returns the number of the laser that was broken, or zero if none of the lasers were broken yet. - `int Leaderboard::displayLeaderboard(Adafruit_RA8875* tft, LaserString* string_1, LaserString* string_2, LaserString* string_3)` - This function is responsible for displaying the leaderboard contents on the TFT screen. It takes in a pointer to a tft screen object in addition to pointers to `LaserString` objects representing the first three lasers. - It first displays the title of the screen, which consists of "Leaderboards: " followed by the song name. - Then, it iterates through the values stored in the `scores` and `users` arrays and displays them on the screen, making sure to put each one on a new line. Note that the values are already stored in the proper order of first place to last place because of the way that scores are stored in and returned from the database. - After that, the function displays instructions for the user. If they break the first laser, they replay the song, if they break the second laser, they choose a new song, and if they break the third laser, they go back to the home page and start again with a new username. - The function then checks if each of these lasers is broken using functions from the `LaserString` class. See the description of this class for more details. - The function returns the number of the laser that is broken as an integer, or zero if none of the lasers has been broken yet. Hardware -------------------------------------------------------------- The hardware of our system consists of four main components: the ESP32 microcontroller, the frame of the laser harp including all of the lasers, LEDs, and phototransistors, the TFT screen, and the MP3 player attached to a speaker. We highlight here the role of each part in relation to the rest of the Laser Harp Guitar Hero game system. Greater detail about specific construction and wiring of these hardware components is reserved for discussion below. - **ESP32:** Handles all the microcontroller software described above, as well as coordinates the other hardware components so that they correctly work together with the software, resulting in our integrated game. - **Laser harp frame:** We focus on the lasers, LEDs, and the phototransistors. The LEDs (one per laser string) flash in time with when the strings should be broken. This info is conveyed to the LED through the ESP32 via software that interprets our pre-made beatmaps. The lasers and phototransistors together form the laser string, as the laser points directly into the phototransistor. Thus, when we break the beam, the voltage of the phototransistor decreases sharply, which is sensed by the ESP32 and used by the software to detect and score user actions during gameplay. The functionality of the LEDs and phototransistors is wrapped up into the `LaserString` class, which is described above in the software section. - **TFT screen:** Displays all of the instructions for proceeding between states of the game, from user selection to song selection to the game itself. It also shows the notes running down the screen, indicating when the player should break a laser. It also gives feedback during the game (by displaying the user's score on the bottom of the screen as it updates and displaying feedback for each note, such as "Perfect", "Great", "Good", "Okay", or "Miss") and after the game in the form of a score display and leaderboards. The main state machine described in the 'Game' classes controls what is displayed to screen at a particular time in the flow of gameplay. - **MP3 player:** Plays songs only during gameplay itself to accompany the notes displayed on the screen. The timing of when to begin and end the song is carefully aligned with the start and end of the beatmap through thoughtful state transitions in the code. Parts List ============================================================== - 1 ESP32 Microcontroller - 1 Breadboard - 1 Micro USB Cable - 1 Adafruit 5.0" 40-pin TFT Display: 800x480 pixels - 1 RA8875 Driver Board for 40-pin TFT Touch Displays - 1 Adafruit 3" Diameter Speaker (4 Ohm 3 Watt) - 1 DFPlayer Mini MP3 Player - 1 Micro SD Card - 4 LEDs (1 green, 1 red, 1 yellow, and 1 blue) - 4 Lasers - 4 Phototransistors - 8 4.7 kΩ Resistors - 4 12" blocks of wood - 1 12" x 12" piece of acrylic - Wires of various lengths - Screws of various lengths - Nuts of various sizes - Electrical tape - Hot glue - Shrink wrap - Zip ties - Laser cutter (for cutting the acrylic for the mount) The 5.0" 40-pin TFT Display and the driver board display the whole graphic user interface for our game. The speaker, MP3 player, and micro SD card are required to play music. Lasers and phototransistors serve as the strings of the harp. The LEDs help signal a user to break a particular laser at the appropriate time. In addition, both the phototransistors and LEDs require resistors for proper wiring. Finally, we used the wood, acrylic, screws, and nuts to make the laser harp's frame. Wiring Schematic ============================================================== The ESP32 is connected to the following components: screen, MP3 player, lasers, phototransistors, and LEDs. With so many different pins and wires being used, the wiring was initially very messy. It looked like our harp was having a bad hair day with all the wires sticking out in seemingly random directions. Thus, when problems arose, it was hard to pinpoint whether the bug was in the code or in the wiring itself, leading to much frustration. To fix this issue, for the phototransistors, lasers, and LEDs, we intertwined two wires (one indicating LED color and power, the other ground) with a drill. We also snaked these wires down the side of the frame. In addition, we glued the breadboard to the back of the acrylic backing to keep it out of the way. The screen was connected to the breadboard using male-to-female wire adapters that connect from the screen's driver board to the breadboard in order to accommodate the positioning of the screen in the mount. The wiring of each component will be described separately below. **Images of the actual wiring:** ![Figure [Wiring]: The back of our laser harp in all its glory.](./images/back_of_board.JPG width="600px") ![Figure [Wiring]: A closer look at the breadboard](./images/breadboard_close_up.JPG width="600px") ![Figure [Wiring]: The ESP32 steals the show. Note that the red-orange wires in the front of the image connect the speaker to the MP3 player](./images/esp_close_up.JPG width="600px") ![Figure [Wiring]: The wiring for a small but mighty MP3 player.](./images/mp3_player.JPG width="600px") ![Figure [Wiring]: The screen wiring featuring some handy male-to-female adapters.](./images/screen_wiring.JPG width="600px") ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Adafruit_RA8875 TFT Board ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After soldering the header pins onto the driver board, the pins in the image were connected by following a guide here. Some pins, such as Y+, Y-, X+, and X- are not used because we are not employing the touch screen. RST and INT can be connected to any MCU pin, and we chose to use IO21 and IO16. In addition, we connected our screen to 3.3V rather than 5V. ![Figure [TFT Driver Board]: Wiring schematic for driver board to TFT screen](./images/screen_driver_board_connect.png width="400px") ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ DFPlayer Mini MP3 Player ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We followed the schematic provided on the 6.08 website in order to hook up the MP3 player. However, it's important to note that we ultimately ended up connecting the module to 5V rather than 3.3V, as indicated in the schematics, since 3.3V often didn't provide enough power for the MP3 player when all the other loads were pulling power as well. ![Figure [MP3 Player]: Wiring schematic for MP3 player to ESP32.](./images/mp3_hookup.png width="400px") ![Figure [MP3 Player]: Pinout diagram for MP3 player.](./images/mp3_wiring.jpg width="400px") ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Lasers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Black and white wires were soldered to the ends of the laser. Black represents ground and white represents power. The lasers were connected directly to 3.3V and ground. The black and white wires were twisted together using a drill in order to make the wiring neater and easier to follow. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Phototransistors ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The long leg of the phototransistor connects to ground, while the short leg connects to a 4.7 kΩ resistor. The resistor then connects to power (3.3V). A voltage divider in between this resistor and the phototransistor is used to read the analog value of voltage. The reading was in terms of discrete "bins," which we converted back to a decimal voltage using the methods described and practiced in class. From left to right laser string, the voltage dividers for each were connected to analog pins A7, A6, A3, and A0, respectively, on the ESP32. In addition, white wires were soldered to the long leg of the phototransistor and colored wires were soldered to the short leg. Each colored wire corresponded to the color of the note and LED that a particular phototransistor was connected to. These wires were twisted together to make the wiring neater and easier to follow. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ LEDs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ On the harp, the order of the LEDs is green, red, yellow, then blue. In this order, pins 13, 12, 14, and 26 are occupied. Each LED is also connected to a 4.7 kΩ resistor. We soldered colored wires to the long legs of the LEDs and black wires to the short legs of the LEDs. Each colored wire corresponds to the color of the LED and the note that the LED represents. The resistor connects to the 3.3V power rail of the breadboard and the colored wire of the LED, while the black wire connects directly to ground. The resistors were necessary to make sure that we did not blow out the LEDs by giving them too much power. Harp Frame ============================================================== To create a rectangular frame, 4 blocks of wood were used. Using around 3-inch long wood screws, the blocks of wood were joined together to form a box, roughly of dimensions 12 in. by 12 in. On the top and bottom pieces of wood, 4 pairs of holes were drilled with even spacing such that the holes on the bottom piece of wood would align vertically with the holes on the top piece of wood. The lasers were placed into the top holes and photransistors were put in on the bottom, and the wires soldered onto these components were pulled through the other side of the hole. Alignment was integral to calibrating the harp so that the laser beam would directly hit the phototransistors. LED holes were drilled perpendicular to the holes made for lasers on the top piece of wood such that once the LEDs are put in, a user sitting facing the front of the harp will be able to see them flashing and indicating which notes to press. In order to allow the user to have a convenient view of the screen and clear access to the speaker, as well as to neaten the general design of the hardware of our game, a piece of acrylic was used as a backing mount. From a rectangular piece of acrylic (12 in. by 12 in.), a rectangle big enough to show the screen and a circle to fit the speaker were cut out using a laser cutter. Holes were also made so that we could attach the parts to the acrylic with screws and hex nuts. Additionally, other holes on the border were made to screw the backing to the wood frame. The design used is shown as below: ![Figure [Frame Backboard Design]: Design for the backboard. Link to design file can be found here.](./images/backboard.png width="400px") A 3D-printed screen holder was used to secure the screen to acrylic and provide a secure mount for the screen. The design of the screen holder is as shown below (the slit on the bottom allows the circuitry of the screen to slide through and connect to the breadboard): ![Figure [Screen Holder Design]: Design for the screen holder. Link to design file can be found here.](./images/screen_holder.png width="400px") After adding all the various parts to our frame, our laser harp looked as follows: ![Figure [Laser Harp]: Our finished laser harp frame. Truly stunning.](./images/harp.png width="400px") Functional Block Diagram ============================================================== ![Figure [Functional Block Diagram]: Block diagram of our game.](./images/functional_block_diagram.png width="800px") State Machine Block Diagram ============================================================== ![Figure [state machine]: State machine diagram of our game](./images/state_machine.png width="800px") **1. Start:** A simple state that just provides a title screen for the game. When the user breaks the first (leftmost) laser, we move onto the "Enter Username" state. **2. Enter Username:** Users select their username using the first three lasers. The first laser scrolls backwards through the alphabet while the second laser scrolls forwards through the alphabet. The third laser is used to select a character. The user breaks the fourth (rightmost) laser to confirm their name and move to the "Choose Song" state. **3. Choose Song:** Users select the song they want to play using the first two lasers. The first laser scrolls backwards through the song list, while the second laser scrolls forward through the song list. Information about each song is displayed, including the name of the song, the song's artist, and its duration in seconds. The user breaks the third laser to select a song and moves onto the "Play" state. **4. Play:** This is the state where the actual game is played. First, the notes for the selected song are retrieved from the server via a get request. Next, the notes are displayed on the screen as colored rectangles, and the user must break the appropriate laser when each rectangular note reaches the bottom of the screen. The user earns points based on how accurately they break the lasers, and feedback about this is displayed for the user to see. Once all of the song's notes have been played, the game moves to the "Show Score" state. **5. Show Score:** The user's score is sent to the leaderboards database via a post request. In addition, the score is printed to the screen along with further instructions. When the first laser is broken, the song is replayed and we move to the "Play Again" state. When the second laser is broken, the user can select another song to play with the same username, and we move to the "New Song" state. When the third laser is broken, the user can view leaderboards for that particular song, and we move to the "Leaderboard" state. When the fourth laser is broken, the user starts a new game with a new username, and we move to the "Start" state. **6. Play Again:** We keep the same song and user info and automatically move to the "Play" state. **7. New Song:** We keep the same username but clear the song information and automatically move to the "Choose Song" state. **8. Leaderboard:** We retrieve the leaderboard information for the specified song via a get request to our server-side database and display it on the screen. When the first laser is broken, the user plays the same song again and we move to the "Play Again" state. When the second laser is broken, the user selects another song to play with the same username, and we move to the "New Song" state. Finally, when the third laser is broken, the user starts a new game with a new username, and we move to the "Start" state. Decision Making and Challenges ============================================================== Alignment of Lasers and Phototransistors -------------------------------------------------------------- Our game relies heavily on being able to detect whether or not a laser is broken. For instance, the player must break the laser at the correct time in order to gain points while playing the song. We also use broken lasers as a means of navigating from one page to the other and even selecting a username. Thus, it is crucial that the detection of broken lasers is very accurate. In order to achieve this accuracy, we had to make sure that each laser is pointed exactly at its corresponding phototransistor and that it stayed that way even when we moved the laser harp or the harp was bumped or jostled. We first considered this issue when building the laser harp frame. Firstly, when drilling the holes in the wood for where the lasers and phototransistors would be placed, we tried our very best to make sure that the holes were aligned vertically on top of each other. This was a bit challenging since none of us had much experience with woodwork or power tools, but Tiffany and her Maker Lodge training were key. Next, we tried to drill the holes so their diameters were as close as possible to the diameters of the lasers and phototransistors in order to ensure a snug fit. This was also quite difficult because drillbits only come in predetermined sizes, so the holes were a bit too large. In order to attempt to stabilize the lasers and phototransistors, we first tried simply wrapping them in electrical tape in order to increase their diameters so that they fit more snugly inside the holes. However, this didn't work very well because the lasers would still shift over time and could easily be jostled out of position. Thus, we decided to use a combination of this taping method in addition to hot glue. Once the lasers were taped and placed in a position that we liked, we hot glued the top of the wire that stuck out of the hole onto the wooden frame. This still wasn't completely perfect, and some of the lasers had to be adjusted at times, but it worked reliably for our purposes. For the phototransistors, we decided to forgo the electrical tape and solely use hot glue to secure them since their diameters were already pretty small, so you would have to use a lot of tape to make them fit securely in the holes. We first tested to make sure that the phototransistors' measured voltage still changed reliably even when hot glue was placed on top of them. We found that the hot glue had no significant effect on the phototransistors' performance, so we went ahead with our plan. First, we turned on the lasers so we could see where the phototransistors should be placed. Then, we placed hot glue on each phototransistor and secured them to our liking. We also glued the wires that stuck out of the wooden frame into place as well. Thus, our lasers and phototransistors were beautifully aligned and secured! The Wiring -------------------------------------------------------------- Since we had to connect so many wires to the breadboard for the LEDs, lasers, phototransistors, speaker, and screen, the wiring often became very messy very quickly. It was difficult to examine the wiring while debugging and determine if the wiring was done incorrectly or if the problem was somewhere in our code. In addition, due to the sheer number of wires and the way they initially stuck out at odd angles, wires inevitably popped out of the board on occasion, leaving us wondering why the code was suddenly not working, only to realize that it was actually a missing wire connection after some fruitless debugging. In addition, the wires provided in the labkits often broke after extended use, and some baby chunks of wires even got lost somewhere in the depths of our breadboards, never to be seen again. Hyunji's breadboard was the only one that did not claim any chunks of wires as victims. In order to solve these challenges, we color-coded our wires and twisted the pairs of wires that belong to each laser, phototransistor, and LED. Wires for the same type of hardware were then grouped together using tape and zip ties. For lasers, white wires went to power and black wires went to ground. For each LED, a wire that is the same color (green, red, yellow, or blue) as the LED went to a resistor connected to power via an input pin on the ESP32, and a black wire went to ground. For the phototransistors, white wires go to ground and wires with the same color as the phototransistor's corresponding note and LED went to a resistor connected to power via a voltage divider. In addition, we started using the spools of wire available in the lab to create the wiring for our breadboard instead of using the wires from the labkit. As a result, we were able to create wires that were exactly the right length we needed and significantly neaten the wiring on our breadboard. We also were able to stop worrying about the wires breaking and getting lost in our breadboards. MP3 Player -------------------------------------------------------------- We initially faced many struggles with configuring the MP3 Player. Sometimes the module could not be found by our program, the speaker would stop playing midway through a song, or the entire system would constantly reboot. We were able to get the MP3 module to work some of the time, but not consistently. At this stage, the wiring was still all over the place, so it is likely that the issue was bad wiring that was hard to debug due to the messiness of our system. After we cleaned up the wiring, the system was still not working perfectly. The MP3 player would start playing a song and then cut out abruptly. We realized that it was most likely an issue with power; the 3.3V output on the ESP32 was likely delivering less than 3.3V due to the power use from other modules, such as the large display. Thus, when we connected the MP3 player to 5V, it began behaving reliably. We also intended to use 2 MP3 Players, so that when a player broke a laser they were not supposed to, an error sound would play. We scrapped this idea and instead displayed a `MISS` message as feedback on the screen. This wouldn't ruin the flow of the song and was easily integrated into our score feedback display system. While adding more modern songs to our repertoire, we noticed that different songs would be played on the DFPlayer Mini MP3 Player than we had intended. For example, although we labeled each MP3 file name with a number, playing `3.mp3` would play what we had labeled as `7.mp3` instead of the third song. We then realized that the MP3 files were numbered based off the order they were added to the SD card, not what we labeled them. Screen and Mount -------------------------------------------------------------- We faced many struggles when trying to get our screen to work. After soldering the driver board and connecting the wires, we were confused because the screen and MP3 player would work perfectly separately, but when we uploaded a program that used both the screen and MP3 player, one of the modules would stop working. We realized that this was likely a problem with power, since the ESP32 would occassionally give us a brownout warning. Thus, we decided to turn off the Wifi during gameplay after making a get request for song information. This decreased the power draw during gameplay significantly and allowed both the screen and the MP3 player to work simultaneously. Another struggle we faced was that our screen stopped working even though we hadn't changed any of the wiring. Even a simple test script that just drew circles on the screen would not work; the screen would either just flicker or not turn on at all. We initially tried to completely rewire just in case some wires broke within the breadboard, which happened fairly frequently. We also noticed that the driver board was getting fairly hot to the touch while powering the screen. No matter how we tested, we could not find the cause of the issue. Ultimately, we had to obtain a new driver board and screen. Considering that the new driver board and screen fixed the issue, it is probable that the screen had somehow broken. In addition, we didn't want to leave the screen dangling on the side of the breadboard because that would risk the screen being damaged. This would also not be very user friendly because the user would have to crane their neck to the side to see the notes they have to hit, which would be very difficult while trying to accurately break lasers. In order to solve this problem, we initially created a prototype mount to hold the screen in place by manually cutting a frame from styrofoam. We then 3D printed this frame for the screen. We initially did not know where we wanted to place the mount for the screen and the speaker. We either wanted to place it on top of the harp frame or behind it, in the middle of the hollow wooden rectangle. There were concerns about placing the acrylic mount behind the laser harp because we thought that there might not be enough room for users to break the lasers as they play the game. However, placing the mount on top of the harp would be unstable and difficult to design, so we ultimately chose to place the backing behind the harp. There was still enough room for the user to hit the lasers because the backing sits behind the frame, so there is still hollow space between the backing and the front of the harp. In addition, this made it easier for us to attach both our breadboard and speaker to the acrylic backing. We simply cut out a rectangle for the screen and a circle for the harp. Small holes for screws were also cut out. We could screw the speaker directly to the harp, and the holder we made for the screen also had holes so that it could be screwed in. In addition, we made sure that the hole for the screen was slightly smaller than the size of the screen itself so that there was no risk of the screen falling out and becoming damaged. Energy Management ============================================================== We initially faced challenges with power because our screen and MP3 player would not work at the same time. They would work perfectly when they ran separately, but when we ran code that used both modules, the ESP32 would constantly reboot, or one of the modules would not work correctly. We realized that this was likely an issue with power because the ESP32 would occasionally issue an error message that said that the brownout detector was triggered. Thus, we decided to conserve energy by turning off the ESP32's Wifi module during actual gameplay after making a get request for note information. The Wifi Module is not needed during gameplay, so this saved a considerable amount of energy and allowed the MP3 player and screen to work in harmony. Time (seconds)| Current (A) ---------------|------------ 0 | 0.52 5 | 0.60 10 | 0.53 15 | 0.57 20 | 0.52 25 | 0.52 30 | 0.53 35 | 0.52 40 | 0.52 45 | 0.50 50 | 0.57 55 | 0.52 60 | 0.52 65 | 0.52 70 | 0.52 75 | 0.52 80 | 0.52 85 | 0.52 90 | 0.53 95 | 0.52 100 | 0.52 105 | 0.54 110 | 0.52 115 | 0.52 Across this two minute reading, the average current draw is 12.7 / 24 = 0.5291666667 A. Across these two minutes, the user selected username, song, and played the game. After choosing a song, the WiFi is turned off to conserve power, but as we can see, the current remains relatively stable. The LEDs and phototransistors are now in use. With a battery capacity of 1500 mAH, our system would last for 1500 mAH / 529.1666667 mA = 2.83464567 hours. Luckily, most of our players thus far have either become tired, frustrated, or defeated after 10 minutes. Rock on! ~ Team 77 Shenanigans ============================================================== ![Figure [Shenanigans]: Joie is a mood.](./images/joie1.jpeg width = "400px") ![Figure [Shenanigans]: Look how much fun she's having!](./images/joie2.jpeg width="400px") ![Figure [Shenanigans]: Wild humans amazed by a functioning laser harp](./images/julia_and_tiff.jpg width="400px")
Here's a video of the first time our project worked!! **Foul langugage & cringiness warning**

Laser Harp Guitar Hero - 1st Trial from Joie Le on Vimeo.

"I can never top this; this is the peak of my career" - Tiffany