Software Setup

You are not logged in.
If you are a current student, please Log In for full access to the web site.
Note that this link will take you to an external site (https://shimmer.mit.edu) to authenticate, and then you will be redirected back to this page.

1) Overview

This page covers software installation for the ESP32 microcontroller that we use in this class.

If you are using a USB-C adapter, we strongly encourage you to use the "simple" one that we included in your kit. More complicated breakouts sometimes throttle power to their distribution ports which can make programming the microcontrollers a bit more stochastic. Some of the breakouts have their own drivers, so then you have drivers on top of drivers; it can get weird. The simple ones route via metal the necessary pins for USB 2.0 compatibility. The power breakouts/ones with adapters often may have circuits/repeaters in between which have their own specifications we need to worry about. You can try your own adapter, but don't spend six hours trying to get it to work.

2) Installation Instructions

The installation of the software that we need is heavily dependent on your Operating System type and version. Please follow the instructions for your particular situation below:

2.1) Windows Users

First install Arduino (even if you've already got it, install it again to get version 1.8.13). The Arduino IDE can be found here. Download the appropriate distribution for your laptop's operating system. Note: Windows Users do NOT install the Windows App. Install only the regular Windows distribution!
Next install the ESP32 core. Open up Arduino and do the following:
- Go to File>Preferences
- Under the Additional Boards Manager URL field paste in: https://dl.espressif.com/dl/package_esp32_index.json and then Click OK
- Now go to Tools>Boards>Boards Manager
- In the window that pops up, search for esp32. One option should appear. Install it.
- Now to prepare your Arduino environment to work with the ESP32, go to Tools>Boards>ESP32 Dev Module
- You should be good to go now with software. We'll test it in a little bit.
You will also need to install a cable driver for your laptop. Go to here and install. Your operating system must be somewhat recent for this to be valid so if you get bugs, please talk to a staff member. In the zip file, there should be a README that will have somewhat clear instructions.
- click "download VCP" under the appropriate version of Windows
- Unzip the folder, run Installer_x64.exe if 64 bit or Installer_x86.exe if 32 bit

If you've done everything above and no errors came through, it means you've installed most of the correct software. Move onto to the test section below.

2.2) Mac OSX Users

The steps to install the software are based on whether you have a Mac running OSX Catalina and before or if you've already updated to Big Sur. Pre-Catalina Mac's have been relatively stable in our experience. The Catalina transition caused a mess of bugs with hardware and drivers that were mostly figured out by mid 2020, however Mac's release of Big Sur caused a whole other mess of issues. You will need to use a development version of the ESP32 software, rather than the stable one if you are on Big Sur.

2.2.1) Catalina and Before

First install Arduino (even if you've already got it, install it again to get version 1.8.13). The Arduino IDE can be found here. Download the appropriate distribution for your laptop's operating system.
Next install the ESP32 core. Open up Arduino and do the following:
- Go to Arduino>Preferences
- Under the Additional Boards Manager URL field paste in: https://dl.espressif.com/dl/package_esp32_index.json and then Click OK
- Now go to Tools>Boards>Boards Manager
- In the window that pops up, search for esp32. One option should appear. Install the most up to date version (1.04 I believe, though do 1.05 if it is available)
- Now to prepare your Arduino environment to work with the ESP32, go to Tools>Boards>ESP32 Arduino >ESP32 Dev Module
- You should be good to go now with software. We'll test it in a little bit.
You'll also need to install a cable driver for your laptop. Go to here and install. Your operating system must be somewhat recent for this to be valid so if you get bugs, please talk to a staff member. **Mac Users: After installing, go to System Preferences > Security and click on the message about enabling Silicon Labs Drivers on your machine! In software setup for Mac, you can also get a message in “Security & Privacy” in System Preferences, saying that the Driver is blocked from loading. So, students can just click “allow” to let it complete loading. **

Mac users should check the Security&Privacy tab in System Preferences, even if it looks as though the Cable Driver was downloaded and the Mac never informed the user that they need to override the security preferences in order to download the software. go to the General tab within Security and Privacy and if there is an issue, it'll say something akin to "Some system software was blocked from loading." It may also prompt you to do this during the installation so keep an eye out!

If you've done everything above and no errors came through, it means you've installed most of the correct software. Move onto to the test section below.

2.2.2) Big Sur (current OSX)

First install Arduino (even if you've already got it, install it again to get version 1.8.13). The Arduino IDE can be found here. Download the appropriate distribution for your laptop's operating system.
Next install the ESP32 core. Open up Arduino and do the following:
- Go to Arduino>Preferences
- Under the Additional Boards Manager URL field paste in: https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json and then Click OK
- Now go to Tools>Boards>Boards Manager
- In the window that pops up, search for esp32. One option should appear. Install the most up to date version (1.0.5-rc7)
- Now to prepare your Arduino environment to work with the ESP32, go to Tools>Boards>ESP32 Arduino >ESP32 Dev Module
- You should be good to go now with software. We'll test it in a little bit.
You'll also need to install a cable driver for your laptop. Go to here and install. Your operating system must be somewhat recent for this to be valid so if you get bugs, please talk to a staff member. **Mac Users: After installing, go to System Preferences > Security and click on the message about enabling Silicon Labs Drivers on your machine! In software setup for Mac, you can also get a message in “Security & Privacy” in System Preferences, saying that the Driver is blocked from loading. So, students can just click “allow” to let it complete loading. **

If you've done everything above and no errors came through, it means you've installed most of the correct software. Move onto to the test section below.

2.3) Most *Nix Users

Below are instructions for relatively tame (Ubuntu-ish/Debian) family distributions. Note some nightly build of a Arch might have separate issues (see section below) that we'll do our best to help with, but can't guarantee it'll work.

Because of some install issues, we strongly recommend installing from the Development Repo. This is different and a bit more involved than the Windows and Mac installations above. The Arduino Installer from the Boards Manager does a poor job setting the appropriate permissions in the broad array of Linux environments.

First install Arduino (even if you've already got it, install it again to get the most up-to-date distribution, 1.8.13). The Arduino IDE can be found here. Download the appropriate distribution for your laptop's operating system.
Next install the ESP32 core. It is found here.Follow the appropriate instructions for installing in Linux (it will involve install git). Follow the instructions for Using Arduino IDE with the development repository.
Pay close attention to what the output of copy-pasting that script says. If there is an error or some text about unicode not existing you need to change the word python to python2.7 in that command string.
Make sure to install everything/run the scripts using Python2, not Python3! You should have this on your machine regardless, but some of you have renamed it from other classes.
You'll also need to install a cable driver for your laptop. Depending on what you want, there are two versions (try the version 3/4 one first):
- Linux_3.x.x_4.x.x_VCP_Driver_Source.zip
- Linux_2.6.x_VCP_Driver_Source.zip

When you've installed everything you should be able to open up Arduino and in the top right you should be able to see in the menu Tools>Board>ESP32 Dev Module. If you see that, it means you've installed most of the correct software.

The only other issue that may appear, dear *Nix user is that you don't have permission to program your ESP32. This can be fixed temporarily (after the device has been plugged in by running either):

sudo chmod 666

on whatever port your ESP32 shows up as (for example on my Ubuntu box, it'll be /dev/ttyUSB1 or /dev/ttyUSB0). This is only a temporary fix, however, and will need to be re-run every time you plug in the device.

The other (longer term) option is to add yourself to the dialout group. Do this by running the following command:

sudo usermod -a -G dialout YOUR_USERNAME_HERE

where you replace your username in. Once this is done, you will need to restart your computer for this change to take permanent affect.

Make sure to disable Secure Boot! That can prevent things from working!

If you've done everything above and no errors came through, it means you've installed most of the correct software. Move onto to the test section below.

2.4) Manjaro/Arch

If you're too cool for school and are running Arch or something, a helpful LA wrote this up.

Install the latest Arduino IDE (I think sudo pacman -S arduino should get 1.8.13).

Then the following should get the esp32 specific stuff:

sudo pacman -S python2 python2-pip && \
sudo pip2 install pyserial && \
sudo gpasswd -a $USER uucp && \
mkdir -p ~/Arduino/hardware/espressif && \
cd ~/Arduino/hardware/espressif && \
git clone https://github.com/espressif/arduino-esp32.git esp32 && \
cd esp32 && \
git submodule update --init --recursive && \
cd tools && \
python2 get.py

The LA also thinks on new versions of Arch/Manjaro (5+) installing the cp210x driver is not necessary.

3) Test the Software

OK hopefully things have worked. To test the whole setup let's try to run a small script that will attempt to program the ESP32, then have it try to connect to the internet.

Open Up Arduino.
Create a new file and replace what is the with the following code (feel free to copy paste!) Read the comment at the top of the code file for instructions, mostly involved in setting the WiFi network and password.

#include <WiFi.h> //Connect to WiFi Network
#include <string.h>  //used for some string handling and processing.

/*#########################################
  This is a  simple Test of the ESP32.
  Don't worry too much about the code...we'll deal with that in class
  INSTRUCTIONS:

  Below change the network and password variables to a 2.4 GHz Wifi network and password.
  If the network of interest is open, leave the password blank.  For example:

  Network is  MY_WIFI and the password is SUPER_SECRET then you'd do:

  char network[] = "MY_WIFI";
  char password[] = "SUPER_SECRET";

  If the network is inSecureSite and there is no password, you'd do:

  char network[] = "inSecureSite";
  char password[] = "";

  If you are on campus, avoid using MIT_SECURE

  Once you've changed that,  you should be ready to upload.
  Save the code, make sure the ESP32 board is selected under Tools>Ports,
  And press the upload button!

  Once it uploads, open the Serial monitor (under Tools) and watch to see what gets printed.


*/

char network[] = "some_network";
char password[] = "some_password";


const int RESPONSE_TIMEOUT = 6000; //ms to wait for response from host
const int GETTING_PERIOD = 5000; //periodicity of getting a number fact.
const uint16_t IN_BUFFER_SIZE = 1000; //size of buffer to hold HTTP request
const uint16_t OUT_BUFFER_SIZE = 1000; //size of buffer to hold HTTP response
char request_buffer[IN_BUFFER_SIZE]; //char array buffer to hold HTTP request
char response_buffer[OUT_BUFFER_SIZE]; //char array buffer to hold HTTP response

uint32_t loop_controller; //used for timing
uint32_t last_time; //used for timing


void setup() {
  Serial.begin(115200); //begin serial
  delay(100); //wait a bit (100 ms)
  // WiFi.scanNetworks will return the number of networks found
  int n = WiFi.scanNetworks();
  Serial.println("scan done");
  if (n == 0) {
    Serial.println("no networks found");
  } else {
    Serial.print(n);
    Serial.println(" networks found");
    for (int i = 0; i < n; ++i) {
      Serial.printf("%d: %s, Ch:%d (%ddBm) %s ", i + 1, WiFi.SSID(i).c_str(), WiFi.channel(i), WiFi.RSSI(i), WiFi.encryptionType(i) == WIFI_AUTH_OPEN ? "open" : "");
      Serial.println("");
    }
  }
  Serial.println("");

  WiFi.begin(network, password); //attempt to connect to wifi
  uint8_t count = 0; //count used for Wifi check times
  Serial.print("Attempting to connect to ");
  Serial.println(network);
  while (WiFi.status() != WL_CONNECTED && count < 6) { //can change this to more attempts
    delay(500);
    Serial.print(".");
    count++;
  }
  delay(2000);  //acceptable since it is in the setup function.
  if (WiFi.isConnected()) { //if we connected then print our IP, Mac, and SSID we're on
    Serial.println("CONNECTED!");
    Serial.printf("%d:%d:%d:%d (%s) (%s)\n", WiFi.localIP()[3], WiFi.localIP()[2],
                  WiFi.localIP()[1], WiFi.localIP()[0],
                  WiFi.macAddress().c_str() , WiFi.SSID().c_str());
    delay(500);
  } else { //if we failed to connect just Try again.
    Serial.println("Failed to Connect :/  Going to restart");
    Serial.println(WiFi.status());
    ESP.restart(); // restart the ESP (proper way)
  }
  randomSeed(analogRead(A0)); //"seed" random number generator
}


void loop() {
  if ((millis() - last_time) > GETTING_PERIOD) { // GETTING_PERIOD since last lookup? Look up again
    //formulate GET request...first line:
    sprintf(request_buffer, "GET http://numbersapi.com/%d/trivia HTTP/1.1\r\n", random(200));
    strcat(request_buffer, "Host: numbersapi.com\r\n"); //add more to the end
    strcat(request_buffer, "\r\n"); //add blank line!
    //submit to function that performs GET.  It will return output using response_buffer char array
    do_http_GET("numbersapi.com", request_buffer, response_buffer, OUT_BUFFER_SIZE, RESPONSE_TIMEOUT, true);
    //Serial.println(response_buffer); //print to serial monitor
    last_time = millis();//remember when this happened so we perform next lookup in GETTING_PERIOD milliseconds
  }
}


uint8_t char_append(char* buff, char c, uint16_t buff_size) {
  int len = strlen(buff);
  if (len > buff_size) return false;
  buff[len] = c;
  buff[len + 1] = '\0';
  return true;
}


void do_http_GET(char* host, char* request, char* response, uint16_t response_size, uint16_t response_timeout, uint8_t serial) {
  WiFiClient client; //instantiate a client object
  if (client.connect(host, 80)) { //try to connect to host on port 80
    if (serial) Serial.print(request);//Can do one-line if statements in C without curly braces
    client.print(request);
    memset(response, 0, response_size); //Null out (0 is the value of the null terminator '\0') entire buffer
    uint32_t count = millis();
    while (client.connected()) { //while we remain connected read out data coming back
      client.readBytesUntil('\n', response, response_size);
      if (serial) Serial.println(response);
      if (strcmp(response, "\r") == 0) { //found a blank line! (end of response header)
        break;
      }
      memset(response, 0, response_size);
      if (millis() - count > response_timeout) break;
    }
    memset(response, 0, response_size);  //empty in prep to store body
    count = millis();
    while (client.available()) { //read out remaining text (body of response)
      char_append(response, client.read(), OUT_BUFFER_SIZE);
    }
    if (serial) Serial.println(response);
    client.stop();
    if (serial) Serial.println("-----------");
  } else {
    if (serial) Serial.println("connection failed :/");
    if (serial) Serial.println("wait 0.5 sec...");
    client.stop();
  }
}

Under Tools>Board>ESP32 Arduino select ESP32 Dev Module
Now take a peek at Tools>port in your Arduino environment. You should see maybe one or a couple things listed. Take note of what is there. Now take your Micro-USB-to-USB cable, connect it to your ESP32 board and then to the computer, and take a fresh peek at Tools>port. You should hopefully see one more thing. On Windows this will often be something like COM3. On Mac/Unix it will probably be USB_to_UART.... Some Macs and Linux systems might give you two ports. For Macs pick the USB_to_UART one. Click on it to select that port. If it appeared, then you are 99% good. On Windows, sometimes the Port field is greyed out until something is actually inserted, so if that's the case, just plug in your device and see what port it shows up as.

Does your ESP32 appear under Tools>port when you connect it to your computer? If not, ask for help from a staff member.

Now we're ready to try and upload. In the top left corner press the Upload button (the right facing arrow). The code should compile. If it compiles successfully, then you should soon see some numbers counting up (this is the progress of programming the board). When the upload is successfully completed it should say something similar to the following:

Hash of data verified.
Compressed 669008 bytes to 405691...
Writing at 0x00010000... (4 %)
Writing at 0x00014000... (8 %)
Writing at 0x00018000... (12 %)
Writing at 0x0001c000... (16 %)
Writing at 0x00020000... (20 %)
Writing at 0x00024000... (24 %)
Writing at 0x00028000... (28 %)
Writing at 0x0002c000... (32 %)
Writing at 0x00030000... (36 %)
Writing at 0x00034000... (40 %)
Writing at 0x00038000... (44 %)
Writing at 0x0003c000... (48 %)
Writing at 0x00040000... (52 %)
Writing at 0x00044000... (56 %)
Writing at 0x00048000... (60 %)
Writing at 0x0004c000... (64 %)
Writing at 0x00050000... (68 %)
Writing at 0x00054000... (72 %)
Writing at 0x00058000... (76 %)
Writing at 0x0005c000... (80 %)
Writing at 0x00060000... (84 %)
Writing at 0x00064000... (88 %)
Writing at 0x00068000... (92 %)
Writing at 0x0006c000... (96 %)
Writing at 0x00070000... (100 %)
Wrote 669008 bytes (405691 compressed) at 0x00010000 in 6.6 seconds (effective 807.6 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 128...
Writing at 0x00008000... (100 %)
Wrote 3072 bytes (128 compressed) at 0x00008000 in 0.0 seconds (effective 2343.7 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

If you got this, then go up to Tools>Serial Monitor. Another window/tab will appear. Some text might already be appearing in it. If it is illegible, make sure to set the one field in the lower right to 115200 baud (sometimes it defaults to 9600 baud). Then observe what gets printed to the serial monitor. At first you should see a dump of nearby networks detected by the ESP32. After that you should hopefully see some text about trying to connect to your WiFi access point you specified. Assuming it connects (it may need to restart a few times), you should then see a bunch of text fly by every two seconds and at the bottom will be a fact about a number...this should happen every two seconds or so. If you see this, you're good to go!

An example of my ESP32 running is shown below.

scan done
22 networks found
1: Cat_316_EXT, Ch:1 (-39dBm)  
2: Lou103, Ch:1 (-64dBm)  
3: Cat_316, Ch:1 (-66dBm)  
4: maytagsmartfridge, Ch:3 (-66dBm)  
5: Beverly, Ch:11 (-73dBm)  
6: MOTOFAEC, Ch:6 (-74dBm)  
7: DIRECT-38-HP OfficeJet Pro 6970, Ch:11 (-81dBm)  
8: ESP_D395F2, Ch:1 (-83dBm) open 
9: 31_Saint_Gregory, Ch:6 (-83dBm)  
10: Wineisgood2.4, Ch:6 (-84dBm)  
11: StripperDogs, Ch:11 (-84dBm)  
12: [range] Samsung, Ch:1 (-85dBm)  
13: DIRECT-D3-HP OfficeJet 3830, Ch:6 (-85dBm)  
14: HOME-70ED-2.4, Ch:6 (-86dBm)  
15: williamsden7, Ch:6 (-86dBm)  
16: Northpoint, Ch:11 (-86dBm)  
17: WiFi_OBDII, Ch:1 (-90dBm) open 
18: SEXYTERRY, Ch:1 (-90dBm)  
19: NETGEAR85, Ch:1 (-90dBm)  
20: vuwifi, Ch:11 (-90dBm)  
21: HP-Print-2F-Officejet 6800, Ch:6 (-92dBm)  
22: StripperDogs-5G-2G, Ch:11 (-93dBm)  

Attempting to connect to Cat_316
......CONNECTED!
246:1:168:192 (24:0A:C4:E8:07:74) (Cat_316)
GET http://numbersapi.com/96/trivia HTTP/1.1
Host: numbersapi.com

HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Wed, 10 Feb 2021 22:54:11 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 45
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: 
X-Numbers-API-Number: 96
X-Numbers-API-Type: trivia
Pragma: no-cache
Cache-Control: no-cache
Expires: 0
ETag: W/"2d-OsgM9EqZqM8JqFYt1U2nXa5+lzI"
Last-Modified: 1612997651
Cache-Control: no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0

96 is the rating of Skyrim on metacritic.com.
-----------

If you got this all working, if you could please enter the word "GOOD" below. This has no influence on your grade, We are just using it to get statistics on who has working software.

Enter 'GOOD' if you are getting number facts back above.

But what if you didn't get this? Well the next section has some debugging ideas.

4) Debugging

If you see it printing stuff in the serial monitor, but it is saying it is failing to connect that is still sorta good. That means you can program the board (yay), but aren't connecting to the internet (saaaaad). Why aren't you connecting to the internet? Here's some things to check:

Are your WiFi credentials correct? Did you mistype them? It is case sensitive.
Is the WiFi router you're trying to connect seen by the ESP32? At the start of the program it dumps all the networks on the 2.4 GHz WiFi band. Is the one you care about appearing? What is the signal strength of it? You can usually connect to anything at the ~-70dBm level or better (smaller numbers are better...so -35 dBM is better than -40dBM). Anything in the 80's or 90's might be tough to connect to. Can you move closer to the access point?
The ESP32 can only connect to 2.4GHz Wifi. Most new routers out there are dual band (meaning 2.4 GHz and 5.0GHz) and most older routers are 2.4 GHz only, so it is pretty rare to have a 5GHZ-ONLY router, but still double check this.
Some routers do actually need a minute or two of connection retries to finally let the ESP on, and after that it is relatively quick. Make sure you give it a minute.