Lab 01b

HTTP GETs and the ESP32

The questions below are due on Thursday February 07, 2019; 10:00:00 PM.
 

Partners: You have not yet been assigned a partner for this lab.
You are not logged in.

If you are a current student, please Log In for full access to the web site.
Note that this link will take you to an external site (https://oidc.mit.edu) to authenticate, and then you will be redirected back to this page.

Music for this Lab


Goals: In this lab we'll get more practice with state machines, as well as start to perform rudimentary HTTP GET requests, as an initial foray into the "I" part of IOT. We'll then merge these two topics to implement a basic number fact reporter. In particular we'll:
  • First, explore pulling some information from the web using HTTP GET requests with our ESP32 and a bare-bones public API
  • Build a more complicated state-machine to allow us to request information from an API based on user input.
  • There's no need to wire up new hardware in this lab so you can keep your wire kits in your bag, unless you like how they look.

Unless otherwise stated, all checkoffs in this lab are due. Also we expect groups to work together on labs so during the checkoff we expect to engage with both partners in discussion.

1) The GET Method

Download and extract the code found in lab01b.zip. Open the file up, compile it, and then upload it to your ESP32 (the code as-written will only work in the lab on the 6S08 access point. If you are doing this lab elsewhere, you'll likely need to change some parameters, which we'll learn about below). Your LCD should flash some text, and if you monitor the output on the Serial monitor you should see the a bunch of text flying by and after a little bit, interspersed factoids about numbers. Your little microcontroller is talking to the web!

1.1) HTTP

At the core of any communication protocol is an agreed-upon syntax which enables both parties to effectively convey information. This can be generalized up to things like human language, and obsessed over at the level of individual bits, and EECS as a field covers all of that. The internet is no exception, with a vast array of entities using standardized protocols. Initially those entities were supposed to be just standard "computers" (in the 1980s sense), but in recent years as computation has become more ubiquitous, the limits (size, cost, availability) of what exactly is needed to join that great system known as the Internet have been greatly relaxed. This is really where the term "IOT" comes from...it is a new period in the history of the internet where arbitrary "things" can be on the internet in addition to a Dell Latitude laptop.

What exactly comprises a "web-communication"? Well, a lot of web communication is built on top of the Hypertext Transfer Protocol (HTTP) which is itself generally built on top of another protocol (Transmission Control Protocol (TCP)) which is itself built on other "layers" (take 6.033 if this stuff interests you...it should since it is pretty cool).

HTTP is far from the only Application Layer that is available. Things like SSH, FTP, IMAP, and MQTT are all other acronyms you might encounter in your day-to-day millenial lives, which refer to particular communication protocols that have other intentions. We'll talk a bit about a select few of these throughout the semester.

1.2) The HTTP GET

One of the simplest things you can do over HTTP is a "GET". We actually already know how to perform an HTTP GET using a web browser. You do that simply by visiting a website, believe it or not. But what actually happens behind the scenes?

First the client (your laptop or phone) must establish a connection to the server of interest (the host). The host can either be an IP address or a domain name like google.com. When we connect we are a client of the service.

If the connection is successful (and a lot goes on behind the scenes to ensure that this happens), we next need to send the actual HTTP GET request. When you visit a site in a browser, this is almost always a GET request behind the scenes and actually involves sending a text message that looks similar to the following:.

GET /resource HTTP/1.1
Host: server

Line-by-line this corresponds to:

GET /resource HTTP/1.1

which is the HTTP operation that we're doing and at what Uniform Resource Identifier (of which URL, Uniform Resource Locator, is a subtype) on the server we're interested in. We also specify which version of the HTTP protocol we want (just use 1.1).

The next line:

Host: server

lists the server we're talking with/interested in.

Finally we must always end a GET request with a single blank line which we accomplish like so


We should make a note on two special characters at this point that have proven invisible so far: \r and \n. Both of these characters should be thought of as individual characters even though we express them using two typed-characters. They are special since they're usually rendered as invisible when we show a string featuring them but you can think of them the same as the character "a" or "T" since they still influence how things get rendered. Those two symbols are:

  • \r Originally called a "carriage return", some OS's use it to indicate the end of a line.
  • \n Often called a "new-line" or "line-feed", many other OS's use this as a standard end of line/new line indicator (*Nix, Mac). In C and Python, if you want to include a new line in a string you are creating, this is also how you do it

On the internet (for various historical reasons), the way to terminate a line is the two character sequence: \r\n. That means the above GET request can be (and is) expressed as a single sequence of characters in the following way:

GET /resource HTTP/1.1\r\nHost: server\r\n\r\n

We will come back to that a bit later in the lab when we start doing this on the ESP32.

1.3) Specifying Resources

Now when you go into a browser like Chrome and visit google.com, what your browser is really doing behind the scenes is establishing a connection with the google.com host, and then sending a GET request of the following:

GET  HTTP/1.1
Host: google.com

Note there is nothing after the GET in this case since we are just going to the root of the host (google.com)

If we wanted to visit a more specific URI within the google.com, for example Google's "about" page (yes they have one) in the browser field you'd type in: https://google.com/about/ while behind the scenes it is performing the following GET:

GET /about/ HTTP/1.1
Host: google.com

We'll get tons of experience with this, but what's really neat is if we wanted to perform a search (even though we usually only use GUI buttons and our mouse to do it for us) we can in the browser search for "cat" by manually typing: https://google.com/search?q=cat, and when you press enter you'll show up on Google's search results page for 'cat'. Behind the scenes that looked like the following:

GET /search?q=cat HTTP/1.1
Host: google.com

The ? part indicates a query and the q=cat is the key-value pair of the argument we provide.

2) Doing this on the ESP32

The ESP32 lacks a browser, but once we realize that a web browser is really just sugar coating on top of the client/user side of a robust method of requesting and receiving information over HTTP, we can start to figure out how to do this on our microcontroller. To do this on our ESP32 there's a few ways. First and foremost we need to use Wifi to establish a connection to the greater network in the world (the so-called "world-wide-web" everyone's been talking about lately). We'll do that by including a library that allows the ESP32 to connect to a WiFi network. Rather appropriately that library is called WiFi. We include that in our code with the following syntax:

#include <WiFi.h>

Next when the code first runs (therefore in the setup function) we need to provide credentials to connect to our network of choice (just the same as when you connect your laptop). The example below will connect to the class router in the 6.08 room (yes there is an 's' in the SSID and password).

WiFi.begin("6s08","iesc6s08");

As an aside if you want to connect to an open network, you can leave the second argument off (the password). So for example, a common open network in Building 38/36 is EECS-MTL-RLE, so you could join it by doing WiFi.begin("EECS-MTL-RLE");.

Be careful joining open networks on MIT's campus since often times there are many access points labeled "MIT" or "EECS-MTL-RLE" and if you tell the ESP32 to just join one of them it may pick the weakest one and have a repeatedly hard time connecting. There are ways around this by using the unique identifying numbers of each access point (the MAC address), but we'll not go into that this week.

Next, (also in setup) we'll need to pause the code until we establish a connection. This can be done in tons of ways, but a short convenient one that gives a bit of debugging information is shown below - while we are NOT connected, wait for about 3 seconds or until we get connected (whichever comes first).

uint8_t count = 0;
Serial.print("Attempting to connect to ");
Serial.println(network);
while (WiFi.status() != WL_CONNECTED && count<6) {
  delay(500);
  Serial.print(".");
  count++;
}

Then afterwards, let's check to see if we are connected. If we are, print out over serial the credentials assigned to our device by the network for our own records, but if not, restart the device (ESP.restart(), which will proceed to run the setup function again, and try to connect again. (Note for right now, if you try to connect to a non-existent AP, the system will just repeatedly run this try/fail/restart loop, so finding the strongest WiFi network is good here.) The piece of code that implements this connection check, credential printing, and restart looks like:

if (WiFi.isConnected()) { //if we connected then print our IP, Mac, and SSID we're on
  Serial.println("CONNECTED!");
  Serial.println(WiFi.localIP().toString() + " (" + WiFi.macAddress() + ") (" + WiFi.SSID() + ")");
  delay(500);
} else { //if we failed to connect just Try again.
  Serial.println("Failed to Connect :/  Going to restart");
  Serial.println(WiFi.status());
  ESP.restart(); // restart the ESP (proper way)
}

Once we're through that we are connected to a network and into the loop function.

The loop function is relatively simple. It runs on a timer (which we'll discuss in detail below). In the short term, every GETTING_PERIOD milliseconds, a GET request is formulated to the Numbers API asking about a random number from 0 to 200. It carries this out using the following chunk:

  //formulate GET request...first line:
  sprintf(request_buffer,"GET http://numbersapi.com/%d/trivia HTTP/1.1\r\n",random(200));
  strcat(request_buffer,"Host: numbersapi.com\r\n"); //add more to the end
  strcat(request_buffer,"\r\n"); //add blank line!
The syntax of these lines is covered in The Intro Char-Array exercise for this week.

This request is then held within the char-array request_buffer, which is then handed to the do_http_GET function defined in your file at the bottom. The function is written to generalize most HTTP actions by performing three actions:

  • Connect to the specified host
  • Send the GET request handed to it through its request argument
  • Wait for a response to come back and then parse it and store it in the char array pointed to by the response argument

Ignoring the generalized structure of the function, we can instead walk through most of what it is doing given our use case today.

In order to connect to a host, we must create a client object. A client as we've mentioned earlier is the name for the entity connecting to a host (a server). In most of our daily techno-consumerist existence we are clients, and entities like Google and Facebook are the hosts.

WiFiClient client;

We then instruct the client to try and connect to the host we're interested in. The first value can either be an IP address or a domain name (like google.com), and the second argument is a port number. For today and at least a little while into the future, we'll be using port 80 which is a particular communication channel reserved for HTTP traffic. (Later we'll use port 443 for HTTPS).

For today's lab we're going to connect to a host called numbersapi.com which provides a free Application Programming Interface (API) for interesting information about numbers. Yes that might seem silly, but you are at MIT, and trivia about numbers should appeal to you.

So to connect:

client.connect("numbersapi.com", 80)

This member function connect will return a true or false. You can use this to do some error handling if you'd like or try to reconnect. Assuming it does connect, however, it means a channel of communication exists from client to host. In order to perform a GET request we need to simply use the member function print and/or println which work very similarly to the Serial.print and Serial.println except that instead of printing to the Serial monitor they are sent over the socket connecting our client to the host in interest. Specifically print prints only the string provided while println appends a \r\n to the line. A three-line example of sending a GET request is (or you could put it all in one char array and send it in one go with a single client.println call)

client.println("GET http://numbersapi.com/51/trivia HTTP/1.1");
client.println("Host: numbersapi.com");
client.print("\r\n");

which sends up the GET request to the host of the following form (which should look very familiar to what we saw earlier):

GET http://numbersapi.com/51/trivia HTTP/1.1
Host: numbersapi.com

which when written in one line is:

GET http://numbersapi.com/51/trivia HTTP/1.1\r\nHost: numbersapi.com\r\n\r\n

In our particular code, because of how we wrap some functionality into modules, we first build up a char array called request_buffer like so (again we'll be covering this in the the Intro Char-Array exercise this week):

sprintf(request_buffer,"GET http://numbersapi.com/%d/trivia HTTP/1.1\r\n",random(200));
strcat(request_buffer,"Host: numbersapi.com\r\n");
strcat(request_buffer,"\r\n");

And that array is then printed to client (inside of the do_http_GET function)

Check Yourself:

Is the char array thing concerning you? That's ok. We do go over it in exercises this week, but feel free to ask about it in your checkoffs!

Note also that this HTTP GET request:

GET http://numbersapi.com/51/trivia HTTP/1.1
Host: numbersapi.com

...is equivalent and redundant with (sometimes you'll see the host specified in the URI and other times you won't):

GET /51/trivia HTTP/1.1
Host: numbersapi.com

After receipt of the properly formatted request, the host (numbersapi.com) parses it and formulates a response In future weeks we'll work on performing the type of operations that go on behind the scenes there, but let's not worry about that now. Instead, let's treat it as a black box that takes in an input and generates a response.

An example response that will come back from the numbersapi.com server looks like the following:

HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Sun, 04 Feb 2018 14:48:06 GMT
Content-Type: text/plain; charset="UTF-8"; charset=utf-8
Content-Length: 36
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: X-Requested-With
X-Numbers-API-Number: 51
X-Numbers-API-Type: trivia
Pragma: no-cache
Cache-Control: no-cache
Expires: 0

51 is the atomic number of antimony.

Now a lot of the information present in this response from the host is usually hidden from us in our web browsing experience. The first line for example is the HTTP response status code. 200 means essentially "all good". We'll rarely, if ever, see a 200 in browser, because if we get that it means things have been returned correctly and the browser instead just returns the content. More often we'll see something like a 404 appear which means that while the client could talk to the server/host, the host was unable to locate the appropriate resource. There's a whole list of HTTP status codes which can be used to dictate how to behave in regards to an HTTP response. We won't worry too much about that here and instead just focus on the response body.

How do we determine the response body?

The end of the header is indicated by an extra blank line in the response. If we look for a line that is nothing more than a \r\n, we'll be able to know where our actual content is. We can collect and parse the host response using a set of client member functions:

  • client.connected(): returns true if the client is still connected to the host (server). This will be true when data is being transferred to the host and back from the host as well as sometimes afterwards as well. Otherwise returns false.
  • client.available(): returns true if there are characters to read from the data buffer of the client object, otherwise false.
  • client.read(): Reads out one character from the data buffer of the client object.
  • client.readBytesUntil('\n',response,response_size): Reads out characters from the data buffer of the client object until a \n is encountered and places them into the response buffer! This is convenient since individual pieces of information are usually separated by known characters, in this case we can use the newline character "\n".

When placed all together the following bit of code will wait for up to six seconds while the client is connected and at first print out over Serial (visible in the Serial monitor) incoming lines of data until it encounters a line containing only a "blank" (a \r). After that it will build up a String object that it will use to contain the body of the response.

memset(response, 0, response_size); //clear out string buffer.
uint32_t count = millis();  //mark the time 
while (client.connected()) { //while we remain connected read out data coming back
  client.readBytesUntil('\n',response,response_size); //writes bytes in client buffer and puts them in response buffer
  if (strcmp(response,"\r")==0) { //found a blank line! (end of response header)
    break;
  }
  memset(response, 0, response_size); //reblank the buffer
  if (millis()-count>response_timeout) break; //been too long!
}
memset(response, 0, response_size);  //re-empty string buffer
while (client.available()) { //read out remaining text (body of response)
  char_append(response,client.read(),OUT_BUFFER_SIZE); //add each char of the response into a buffer
  //char_append adds individual characters to response char buffer
}
client.stop(); //all done.

At the end of this code the character array response holds the body of the host's response.

In review, the transfer of data during a simple HTTP GET request is illustrated below:

client and host messages

Illustrated example of HTTP communication between client (in this case an ESP32) and the host (in this case a server called numbersapi.com) where we perform an HTTP GET request for trivia about the number about the numer 31 and the host responds back with trivia about the number 31.

Checkoff 1:
Describe the HTTP response listening code. Briefly verify and discuss with a staff member the code from above. How are the header and body of the HTTP response separated in code? In the code running on your micrcontroller, where do the numbers it is requesting come from?

3) Some Extra Concepts

During the upcoming design experience in this lab, the following concepts might prove helpful for this assignment:

3.1) Timeouts

Sometimes we only want to do something for a little while. Maybe that means check an input, or Wifi connectivity, or something else. This usually means we'll run the task we want with a timeout that corresponds to some check for the amount of time that has passed since a given starting point. In C++ on our microcontroller we can implement a timeout as shown below:

const int TIMEOUT = 400;
unsigned long timeo = millis(); //
while (millis()-timeo < TIMEOUT){
  //do things in here while waiting (e.g. check inputs, etc...)
  //you may also just do nothing depending on the situation
}

The millis() function is an internal counter in the microcontroller and is useful for timing events. The code above will do an action for timeout milliseconds, assuming that the time to execute the code within the while loop is short (<<timeout), which it usually is.

Another way to implement a timeout which can work better in the context of our loop function (since it allows loop to keep running) is:

const int TIMEOUT = 400;
uint32_t timeo = millis(); //

void setup(){
  timeo = millis();
}
void loop(){
  if (millis()-timeo < TIMEOUT){
    //do things in here while waiting (e.g. check inputs, etc...)
    //you may also just do nothing depending on the situation
  }else{
    //do other things after time is up
  }
}

We'll talk about the pro's and con's of both of these in lecture next week!

3.2) Switch Statements

Sometimes as you deal with lots of situations or things to check (such as when you get more states in your state machine) it will get annoying to do nested if-else-if things. Instead in C/C++ you can use a switch statement:

int var1;
//var1 gets assigned to be something.

switch(var1){
  case 2: 
    //stuff you'd like to do when var1==2;
    break;
  case 3:
    //stuff you'd like to do when var1==3;
    break;
  default:
    //if individual values of var1 are not covered do this
    break;
}

When written like shown, a switch statement can provide a slightly more readable way to perform a one-of-many decision in your code.

Do not forget the break;! in your switch statement's individual cases! Failure to do so will result in fall-through where the following cases will be checked as well.

3.3) Definitions

Finally, sometimes as we start to introduce more and more states it gets annoying to have to remember what each state's number means. We can instead start to use variables or definitions for this (these two things have very different implications).

One common convention is to use a definition:

#define IDLE 0
#define PUSHONE 1
#define RELEASE 2

Then in usage you can do something like the following:

//state starts as some value!
loop(){
  switch(state){
    case IDLE:
      //do your idle state stuff
      break;
    case PUSHONE:
      //do you stuff while waiting for state
      break;
    case RELEASE:
      //do stuff here
      break;
  }
}

During code compilation, the compiler will replace each of the keywords (IDLE, etc.) with the relevant number (0, etc.). Using DEFINE statements has the benefit of making analyzing your flow-control (if-else, switch-statments, etc.) much more readable, i.e. you won't be wondering what "4" means anymore if you give it a nice name.

3.4) The Design

Use lab01b_assignment.zip as a starting point for your design.

Ok, so we've got a button, the LCD, and a really powerful microcontroller with internet access. We've also seen that there's a simple API we can talk to where we can give it a number and receive trivia about that number. Right now those numbers are randomly generated. Your assignment for this lab is to build a number-fact looker-upper. The number requested will be based on the number of button presses a user provides as input in quick succession. To be more specific:

  1. Your system will start at rest.
  2. Upon pressing and releasing your button, the system will wait for up to one second for another press and release. If the press-release sequence occurs, the system will restart its timeout and wait a second for another press-release, all the while tallying the number of times the button has been pressed and then released.
  3. Following at least one press-release sequence, if the button remains unpressed for more than one second, the system is to look up trivia on the number of presses that occured and display it on the LCD, before returning to rest.

A high-definition video of the system working is shown below. This is the functionality we're looking for:

Design your system as a state machine (draw out a diagram)

As a first step in your design, draw your finite state machine diagram, sketch some pseudo-code1, and if needed discuss with a staff member how you're implementing it. Not sure where to start? Take advantage of the nicely-wrapped Wifi handler function to keep the number of lines in your primary loop function to a minimum for the sake of clarity and readability! Please ask for help if you get confused!

Use lab01b_assignment.zip as a starting point for your design. It runs by repeatedly calling the number_fsm function giving it the measured value of your button switch as an input. You should build your state machine inside there.

Create a block diagram and when ready, ask for the checkoff below to discuss your design prior to implementation:

Checkoff 2:
Show your state machine diagram and design idea to a staff member and discuss your design.

Finally, when ready, implement your system in code and then show it working. You'll notice the text just gets written right off the screen (we'll fix that in a future homework). For demo purposes, the output should be printed to Serial monitor as well so you can actually see the numerical facts which we are harvesting.

Since this will be the first time debugging on an embedded system for many of you (depending on how far you got in lab 01a), here are a few tips to get started:

  • When you click the right arrow to compile and upload your code, you may end up with an error. The error messages will show up in the bottom part of the Arduino IDE window (the black region). Error messages will show up red (same color as the messages telling you the code was succesfully sent to the ESP). What you'll want to do is to increase the size of the black bottom region, and if you get an error, scroll up to the first error you see. That's the one to fix. Then try again to recompile, until no more errors show up and success is upon you.

  • When debugging, you'll often want to print out the values of certain variables at different points of execution. Serial.println and Serial.print are great ways to do this. Use them and the Serial monitor just like you'd use print in Python!!!

You can also print to the LCD, though that takes a tiny bit more work.

Checkoff 3:
Show your working system to a staff member.

 
Footnotes

1By pseudo-code we mean don't obsess over syntactic correctness but get the general idea and flow down. (click to return to text)



This page was last updated on Thursday February 07, 2019 at 08:41:25 AM (revision 1c9bf28).
 
Course Site powered by CAT-SOOP 14.0.4.dev5.
CAT-SOOP is free/libre software, available under the terms
of the GNU Affero General Public License, version 3.
(Download Source Code)
CSS/stryling from the Outboxcraft library Beauter, licensed under MIT
Copyright 2017