Vegetable Assassin Overview

Component Documentation

Documentation for each component is linked below.

• Vegetable Assassin Overview
• Microcontroller Docs
• Websockets Docs
• Statistics Docs

System Summary

Vegetable Assassin is a web-based game where players slice virtual vegetables with an internet-connected foam sword. The sword communicates orientation information to the game session using a persistent WebSocket connection to a server, the game is generated and rendered using a 3D client JavaScript library, and game statistics are saved into a server database.

Game Details

The game draws heavy inspiration from the game Fruit Ninja by Halfbrick Studios. Vegetable Assassin is played by launching the game website from a web browser, which displays various fruit vegetables as they are thrown into the air. Players are tasked with cutting all of them with a foam sword that is tracked by an onboard micro-controller and inertial measurement unit (IMU). But beware! Some vegetables are not vegetables at all, but fruits trying to disguise themselves! Players gain points for cutting vegetables, get a strike if they miss a vegetable, and lose if they accidentally cut a fruit.

System Design and Implementation

High-Level System Hierarchy

Our game implements a server-client model where data from the client sword is sent via a server connection to the client browser. The server facilitates communication between all components through multiple WebSocket connections, which enables low-overhead and low-latency TCP communication between the sword and the browser for smooth and responsive game-play.

Single-Player Game

We first describe the layout of a system during a single-player Vegetable Assassin game. There are four components of the system: the sword, the browser, the WebSocket server (hereafter named the WS server), and the statistics server. Each of these components is described in greater detail in later sections. While the WS server and statistics server are disjoint applications, they do not necessarily have to be run on different hardware. In fact, the public implementation of these two services run on port 80 and port 8080, respectively, on a single server.

Communications within Vegetable Assassin follow a hub-and-spoke system layout. The WS server is responsible for serving as the hub, and the sword and browser are both connected to the same channel on the server using a persistent WebSocket connection. Each channel is identified by a session ID, where all messages sent to the channel by a client are emitted to every other client on the same channel. This behavior is analogous to a web chat room.

The statistics server manages the allocation of unique session IDs and stores persistent game data, such as high scores, for later viewing.

When a player first turns on the sword, they are greeted with a welcome screen, menu options, and settings. When the 'play' menu option is selected, the sword communicates with the statistics server to obtain a unique session ID that will allow the sword and browser to communicate with each other. The sword displays the session ID to the user, connects to the WS server using the session ID, and waits for further instructions from the WebSockets connection.

The player then navigates to http://vegetableassass.in/ using their browser and clicks on the button to start a single player game. The browser prompts the user for the session ID displayed on the sword, as well as a username to identify the user on the high score. The browser then connects to the channel on the WS server corresponding to that session ID. Once the game starts, the browser walks through steps to calibrate the sword, which involves moving the sword up, down, left, and right within the play area. The game then counts down to the start of the game.

As soon as the calibration is finished, the sword begins sending coordinate information to the server using the already established WebSocket connection. This information is generated using the Magwick algorithm, which is a 9-degrees-of-freedom algorithm that ingests 3-axis data from the gyroscope, accelerometer, and magnetometer of the sword's IMU. The algorithm outputs the absolute orientation of the sword, which is then converted into coordinate information relating the sword's orientation with the position of the tip on the playing field.

The sword sends this coordinate information through the WebSocket connection with a frequency of 10Hz. This data is then relayed to the client's browser through the WS server, which depicts the location of the sword, generates and displays the game to the player, keeps track of game score and game strikes, and transmits the final game statistics to the statistics server.

When the game concludes, the web browser sends the final game statistics to the WebSocket server, which forwards the message to the statistics server through WebSocket. This is then saved into a database to be displayed on the game leaderboard.

Multiplayer Game

A multiplayer game follows a similar system layout as a single-player game, with a few notable differences. Initially, there are two swords that communicate with the statistics server and obtain unique session IDs. The player then navigates to the multiplayer page on the Vegetable Assassin website, enters both session IDs, and enters a team name.

The web browser will then initiate two concurrent WebSocket connections to the WS server, one per sword. Calibration follows a similar series of steps, but calibration messages are sent to each sword individually: sword 1 is calibrated first, followed by sword 2. Once calibrated, both swords communicate over their respective WebSocket channels. The game is played with two swords on the playing field, one red and one green. Both players work together to maximize their game score while avoiding game hazards. Once the game is over, the browser sends the final game statistics to the WS server, which is forwarded to the statistics server to be stored in a multiplayer statistics database.

MQTT vs. WebSocket

We considered using MQTT, which also leverages a persistent TCP connection (like WebSocket) but has lower overhead and has better support with the ESP32 micro-controller, but we decided to implement WebSocket, because communication between the browser client and the server must be implemented through WS to maximize browser support and minimize additional complexity (such as transmitting MQTT from sword to server and WS from server to browser client). We considered the additional overhead by WS to be dwarfed by the large send rate required for Vegetable Assassin, and try to minimize overhead in other ways, such as through purposefully not securing WS connections using TLS.

System Components

In this section, we describe the different system components that make Vegetable Assassin work, how they are designed, and how each component communicates with the others.

Sword

The sword is a Nerf foam sword that has an attached micro-controller used to track the sword, communicate orientation data, and display information to the player. These are the components that make up the sword:

• Nerf foam sword
• ESP32 Micro-controller
• TFT display
• MPU-9255 Inertial Measurement Unit
• Vibration Motor for collision haptic feedback
• 18650 Battery with charging board and voltage step-up converter.
• Buttons for user interaction

The ESP32 micro-controller uses built-in Wi-Fi capabilities to communicate with the other components using HTTP requests and WebSocket connections. The TFT display allows the user to view game connection data, sword status, and change settings. Motion tracking of the sword is enabled by the MPU-9255, which communicates gyroscope, accelerometer, and magnetometer data with the ESP32. When the sword hits a fruit or vegetable in the game, the vibration motor provides haptic feedback to the user (which can be disabled in the settings). The battery allows for truly-wireless usage of the sword, giving the player freedom to move around with the sword.

As mentioned previously, the micro-controller calculates the absolute orientation of the sword by using the Magwick algorithm, which ingests 9DOF IMU data and outputs the orientation of the sword as a quaternion. This is then converted to game coordinates using basic trigonometric computation. More detailed documentation is provided in the micro-controller README.

Browser

The browser is integral to displaying the game, processing inputs, and implementing game flow and logic. The browser code is written in a combination of HTML, CSS, and JavaScript, with the majority of computation occurring in JavaScript.

Using a template provided by HTML5Up, the website http://vegetableassass.in/ contains information regarding how to play the game and buttons that direct the player to the actual game.

Upon starting a single- or multi-player game, the user is brought to a page that requests the session ID of the sword(s) along with a username or team name to identify them on the leaderboard. This form sends a GET request (yes a GET request) to the actual game file, which retrieves the query parameters from the URL and processes them in JavaScript.

The game consists of multiple JavaScript files that perform different tasks. They are all located in the /assets/js/vegetableassassin3d/ directory.

• game.js initializes the game, loads the Three.js WebGL library, creates the scene and camera, calls supporting game functions, and loads the game into the browser's animation frame.
• websocket.js handles the browser-server communication through one (or two, if multiplayer) WebSocket connections. It parses coordinate data from the sword and updates the sword display, and performs calibration routines.
• welcome.js is responsible for the initial overlay at the start of the game. It is set up as a finite state machine that walks through the steps of the game based on sword data and timing information.
• generator.js procedurally generates which vegetables (and fruits) are launched in the game, at what initial velocity and position, and at what time.
• loadModels.js loads the 3D models of each fruit and vegetable into the 3D scene, and creates corresponding physics objects that can be tracked for collisions
• tracking.js is responsible for updating the sword position based on new information passed from websocket.js and rendering the spline that trails the sword.
• physics.js updates the 3D models of each object in the scene based on the physics world provided by Cannon.js, a physics engine that calculates object physics using a global gravity.
• ui.js renders the UI elements at the top of the screen and updates these elements based on game events.

WebSocket Server

The WebSocket server is responsible for facilitating WebSocket communications between the server and clients. It is implemented in less than 50 lines in Node.js. Our team initially implemented WebSockets functionality on the statistics server, but limitations of the Django Channels framework we used prevented high-performance WebSocket communications that was necessitated by our use case.

Read more about the WebSocket server in the README.

Statistics Server

The statistics server is responsible for allocating unique session IDs for WebSocket communications as well as store high score data for player viewing.

The statistics server is built on Django Channels, originally because of the support for WebSockets, but our team determined that Django Channels was not capable of the high-performance and concurrency required by our implementation of Vegetable Assassin.

Read more about the Statistics server in the README.

Communications Standard

Communications between the components occurs through two different application-level protocols: HTTP and WebSockets. Both protocols use TCP as the underlying transport layer, but HTTP is used for stateless operations with no latency concerns, while WebSockets is used for stateful communication between the sword, server, and browser that requires low-latency for the best user experience. We explain how each protocol is used in Vegetable Assassin in the following sections.

HTTP

There are five different HTTP endpoints:

• http://game.vegetableassass.in:8080/getid/ is an endpoint that allows swords to obtain a unique session ID that is never reused.
• http://game.vegetableassass.in:8080/leaderboard/ is an endpoint that serves JSON of the top ten players in the single-player mode. This is used for the leaderboard display on the main Vegetable Assassin website.
• • http://game.vegetableassass.in:8080/leaderboard/{username}/ is an endpoint that retrieves JSON data of the single-player game statistics of the username specified.
• http://game.vegetableassass.in:8080/multileaderboard/ is an endpoint that serves JSON of the top ten players in the multi-player mode. This is also used for the leaderboard display.
• http://game.vegetableassass.in:8080/multileaderboard/{teamname}/ is an endpoint that retrieves JSON data of the multi-player game statistics of the team name specified.

WebSocket

WebSocket connections are initiated using the WS server at ws://game.vegetableassass.in/{sid} where {sid} is the session ID of the WS channel.

Messages are mostly in JSON format to facilitate easy parsing by the server and browser, but two types of messages are in plaintext for implementation ease by the micro-controller.

• {"msg": "sword", "x": 1.3, "y": -2.8} is an example JSON message that is sent by the sword to the server to communicate coordinate data. The keys x and y take signed-float values where -8 < x < 8 and -4.5 < y < 4.5.
• {"msg": "c1r"} is sent from the browser to the sword to indicate that Calibration step 1 is Requested to begin. This will trigger the sword's calibration in the vertical axis. Similarly, {"msg": "c2r"} requests the second calibration step for the horizontal axis.
• {"msg": "c1c"} is sent from the sword to the browser to indicate that Calibration step 1 is Complete. This triggers the second calibration instructions in the browser. Similarly, {"msg": "c2c"} indicates that the second calibration step is complete and the sword is ready for play.
• {"msg": "eg", "username": "max", "score": "15"} is sent by the browser to the WebSocket server (to be forwarded to the statistics server) to indicate that the End Game condition has been reached, and the username has scored a certain amount of points. For multiplayer games, {"msg": "egm", "score": "120", name: "threeAmigos"} indicates that the end game Multiplayer condition has been reached for the given team name and score.

Installation

The files in this directory are static files to be served from a web server. The microcontroller folder contains information about the installation steps required to use the ESP32 code for Vegetable Assassin. The server folder contains instructions on installing both the WS server and the statistics server.

Design Challenges and Decision Rationale

• Absolute Position and Calibration
  • A key challenge of this project was figuring out a way to convert the IMU's sensor data to an absolute coordinate position that is able to be represented on a screen. We started out by only using accelerometer data to detect changes in the z-acceleration based on how the IMU was tilted. This essentially allowed us to map changes in gravitational acceleration detected by the accelerometer to absolute y-position. This approach proved effective, except it did not allow us to apply a similar strategy to calculate x-position.
  • Upon further research, we discovered the Madgwick algorithm, which feeds gyroscopic, accelerometer, and magnetometer data to a filter to produce a rotational output. This algorithm allowed us to have rotational data about the x, y, and z axes, which was essential to measuring a user's slicing motions. Specifically, we wanted to measure the Roll (up/down) and the Heading (left/right) of the IMU, which change as a user swings the sword around.
  • Now came the challenge of mapping the rotational data to absolute x and y coordinates. We started out by having an initial calibration step where the user defines a set region to play in by moving their sword up, down, left, and right. This created boundaries corresponding to the edge coordinates. While this approach was effective, we decided to enhance the user experience by predefining the playing region. The calibration step now only requires users to point the sword at the center of the screen. From there, the playing region is +/- 45 degrees of rotation in either direction.
  • This was now optimal, though it provided one final challenge. Because the left/right rotation is measured in degrees from 0 to 360, we wanted to support gameplay regardless of which direction the user starts out pointing. As a result, we had to account for cases based on the user's starting heading. Upon doing so, we had produced a means of using the IMU to smoothly detect slashes and successfully represent them on screen.
• Sword
  • WebSocket connection: One of the more unique aspects of our project is the use of WebSockets to transmit our data over a persistent connection to the server. We struggled a bit to find information on what we were trying to do because WebSocket library documentation for Arduino was a bit hard to find. Especially with inconsistencies with the server and Django channels it could be challenging to find the root of our problems. However, once we found an example to work off of it was much easier to model or code after to send and receive code over the WebSocket connection.
• Network latency
  • We initially suspected that calculating sword positions in real time, communicating with the web interface, and state transitions in real time would prove to be time intensive and that network latency would be a factor that could prove detrimental to the user experience.
• Server-side code
  • Static files: After trying to serve both dynamic channels and static interfaces on the same host, we realized that Django is not optimized to support static files in production (such as the game.html rendering for our front end interface displays). Taking a closer look at the Django documentation on Deploying static files, we learned that "Most larger Django sites use a separate Web server – i.e., one that’s not also running Django – for serving static files. This server often runs a different type of web server – faster but less full-featured" and thus switched to Apache to host static files on vegetableassass.in and channels on game.vegetableassass.in.
  • Django vs Node.js: We first decided to use Django channels via WebSockets to facilitate real time communication between the web client and the sword with minimal latency. After implementing the entire server with Python using Django channels, we ran into some last minute problems. The server could not handle high volumes of single player game traffic or long periods of multiplayer traffic. Django is synchronous at the core, meaning that like Python (before version 3), it runs lines of code in sequential order. A WebSocket supports "connect", "accept/reject", "receive/send" and "disconnect". Everything is translated to a message and sent to a channel where a WebSocket server accepts WebSocket connections on one end and sends out messages to the other end. This poses limitations to cases that are dependent on the order of incoming messages, such as multiplayer mode. While specific channels can be configured to maintain ordering, most solution would compromise speed and efficiency, which is integral to game experience. We thus decided to migrate to node.js in order to better support asyncronous functions.

Energy Management

The ESP32 turns off Wi-Fi disconnecting WebSocket connection when returning to the title screen. Also have a mode that turns off the vibrational disk to save battery life. Players might often leave the EWP32 on when they are done playing or get distracted. The ESP32 will then go to the title screen from any menu state after 30 seconds of inactivity and turn off the Wi-Fi thereby disconnecting WebSockets. By disconnecting the Wi-Fi on the title screen the battery life nearly doubles from what it would be otherwise.

System current with Wi-Fi disconnected on title screen: 0.10A Battery life: 34 hours

System current with Wi-Fi connected not playing: 0.19A Battery life: 17.9 hours

System current with Wi-Fi connected sending websocket data: 0.22A Battery life: 15.45 hours

Collaborators

By last name:

• Bradley Albright
• Johnny Bui
• Eesam Hourani
• Ting Li