Memory model and pointers

When programming in Python, we typically ignore what is happening "behind the scenes" in terms of where data is stored and how it is manipulated. This is, indeed, one of the benefits of Python! For example in Python when we do this:

x = 5

we mostly think it is the same as when we do the following in C/C++:

int x = 5;

In reality even these two very similar lines are doing very different things underneath, and we'll spend the next few sections in this week's assignment investigating these differences and the implications of them.

1) The C Approach

1.1) Memory

We can think of the memory of a device like our ESP32's code-running part as a giant list of single-byte cells, where a byte is comprised of 8 bits, and where a bit is just a 1 or a 0. As a result, each cell can hold one of 2^8 unique 1-0 patterns or "values": 0 through 255 in decimal or 00000000 through 11111111 in binary or 00 to FF in hexadecimal (which we usually denote by prepending the "0x" tag in front of the value (so 0x00 to 0xFF). The meaning of this binary information is dependent upon how the microcontroller labels the cell (discussed below). Each byte has its own address associated with it, and the microcontroller uses these addresses to deposit and access information. Let's say we're going to deal with a particular part of the memory that starts with byte address 30000 and goes to 30018. At startup, the individual memory cells will most likely be filled with random 1s and 0s. The system can access these cells and their 1/0 combinations, but critically, it doesn't know the meaning of them.

When you create a variable in C, let's say int founding_of_mit = 1861;, what happens behind the scenes is that a portion of memory (four bytes worth in the case of a normal integer declaration) is given the label founding_of_mit and inside of those four bytes the number 1861 is stored. In the case of a 4 byte (32 bit) integer, the decimal number "1861" is stored in binary as: 00000000 00000000 00000111 01000101. We oftentimes will write these values out using hexadecimal for shorthand¹. So the value of 1861 can be represented as: 0x00000745 where the 0x is notation for a hexadecimal number. After declaration and assignment of our variable the memory looks like the following (assuming the memory orders the least-significant to most-significant bytes of the int in increasing address order...which is called little-endian. There is a big-endian format too that's the opposite.):

Four bytes are taken up because that is how integers are specified in C/C++ on the ESP32. Different data types take up different amounts of memory. The particulars of how big different data types are can vary from system to system (or even compiler to compiler), within limits. The basic data types in C/++ that we have been using on our 32-bit ESP32 are:

• char: 1 byte, can represent all major ASCII characters
• int: 4 bytes, can represent integers from -2^{31} to 2^{31}-1. The values are represented in two's-complement representation.
• float: 4 bytes, short for floating point number, which is a way to represent a wide range of decimal numbers with limited precision.
• double: 8 bytes, a double gets its name from being double the "precision" of a float.
• long: 4 bytes, and which on our ESP32 is essentially the same as an integer (not necessarily true on other systems)
• unsigned long: 4 bytes, and which is used to represent unsigned (positive only) integers.

As mentioned on the previous exercise, there are other data types as well with more explicit sizing in their name, which we'll also sometimes use.

The fact that each of these data types takes up a well-defined amount of memory is a main reason behind why the data type of a variable needs to be specified. The C compiler uses this to dedicate the correct amount of memory for that given data type. When you want to print a value, the system looks up the value in the corresponding memory block using the specifications of the data type associated with those cells. So when you do Serial.println(founding_of_mit); the microcontroller will know to pull (in the correct order) the data stored in memory cells 30000 through 30003 (inclusive).

While all the data in every memory address is nothing but 1's and 0's, the associated type instructs the microcontroller on how to handle the values, since a certain string of 1's and 0's will mean different things whether you are viewing it through the lens of an int, the lens of a char, the lens of a float, etc.

If later on in the program, we were to have a line: char cold_out = 'Y';, the memory could² look like this then:

Note the value stored is 0x59 which is the hexadecimal encoding of the ASCII encoding of the character 'Y'.

If we then wrote float age_of_universe = 13.82e9; we'd have:

and so on...

Floating-Point numbers (type float) and Double-Precision Floating-Point Numbers (type double) are really interesting to work with and the details of how you convert a signed exponential to binary representation is a bit of an aside from our goal with these notes, but there are some good explanations on the Internet about how they are encoded/decoded.

2) Pointers

Besides the major types of variables described above, C has another type of variable, which instead of storing an integer, or an ASCII char, holds an address, typically of another variable. This new data type is called a pointer, since it is used to point to the address of another variable.

If you want to create a variable that is a pointer, you need to specify it as a pointer when declared. This can be done by using the format of type* name;. So if you'd like to create a pointer to an int, you'd write: int* int_ptr; or for a pointer to a float you'd write: float* floatPtr;, etc... The * operator (pronounced 'star' in this context) denotes that the variables int_ptr and floatPtr are pointers, that is, they hold addresses rather than other data types.

To store something useful in a pointer, we need to actually find the address of a variable of interest. To do that, we can use the & operator, aka the "address of" operator. For example, &var can be read as "the address of the variable var".

Let's combine these commands to store an address in a pointer variable. In the following, the first line declares a variable x to be an int and assigns it a value of 5. Next we declare ptr_to_x to be a pointer to an int. Finally, the last line looks up the address of x using &x and stores that address in ptr_to_x.

int x = 5;
int* ptr_to_x;
ptr_to_x = &x;

We could also do this a bit more succinctly in two lines:

int x = 5;
int* ptr_to_x = &x;

In our memory map above, for example, we could create a pointer to the founding date by doing the following (assuming founding_of_mit was already declared):

int* ptr_to_founding;
ptr_to_founding = &founding_of_mit;

So again what we did above is create an integer pointer called ptr_to_founding and then assigned it a value of the address in memory of the variable founding_of_mit (30000 which is 0x00007530 in four byte hexadecimal). Note that the pointer points to the first byte in a block of memory holding the data type of interest.

If we were to print the pointer ptr_to_founding nobody would stop you...you could just call Serial.println(ptr_to_founding)...and what would pop out would be the memory address it holds. Alternatively, if we'd like to print the value in the memory block to which ptr_to_founding points, we do what is called dereferencing and type: Serial.println(*ptr_to_founding);

2.1) Two Meanings for *

It is very, very important that in regards to pointers, the * has two different meanings. When declaring a pointer, such as float* ptr_to_some_float; it is used to create a pointer. We can then point that pointer variable to an address of an already existing float-type variable with something like ptr_to_some_float = &already_existing_float;. When used on an already existing pointer variable, the * symbol is the dereference operator, which means "the value pointed to", and is how we can operate on the actual value in memory that is being pointed to. So if we do

float x = 5+ *ptr_to_some_float;

we are assigning the value x to be whatever the value located in the memory slot pointed to by ptr_to_some_float plus 5.

2.2) Practice

For the sake of learning, let's say that we then create another integer called start_of_us_civil_war and since our US history tells us that it shares the same value as the founding of MIT, we do the following in our code:

int start_of_us_civil_war = founding_of_mit;

What happens in the memory map is the following:

Nothing really special...the spot in memory dedicated to founding_of_mit contains the same integer value of as the spot dedicated to start_of_us_civil_war. Now what if we do the following: we change the value of MIT's founding because you it was actually 1862 for some reason³? So you write:

founding_of_mit = 1862;

If we were to then write Serial.println(*ptr_to_founding); what would we see?

What if we were to write Serial.println(start_of_us_civil_war); what would we see?

Because a pointer points to a specific spot in memory, when we dereference and print out its value, we should see whatever is currently in the founding_of_mit spot in memory. But because variable assignment is what is termed by value, when we write int start_of_us_civil_war = founding_of_mit; we were making a copy of the value in the founding_of_mit variable and placing it into the start_of_us_civil_war's memory location. The value stored in ptr_to_founding, however is not the value contained in the founding_of_mit memory location, but rather that variable's address. So if you change what is in the founding_of_mit's memory location, whenever you look up what is there via the pointer ptr_to_founding, it will see the exact same thing as when you access the founding_of_mit variable directly.

In other words, *ptr_to_founding is synonymous with founding_of_mit, and can be used any place you'd write founding_of_mit. In fact, we could have also changed the value of a memory location via the pointer. For example, *ptr_to_founding = 1862; would have resulted in the exact same behavior as above.

Because start_of_us_civil_war was declared before the change in value of founding_of_mit, its copy is unaffected and unchanged. The resulting memory map looks like the following:

3) Arrays and Pointers in C/C++

So we've just seen a bit about what variables are in C/C++, and what pointers are. Do things stay the same when we get to arrays? Well yes and no. Consider the following piece of C/C++ code where we create an int array that is three elements long:

int x[] = {2,4,6};//create a three-element-long array of ints

In memory it will look like the following diagram.

Let's say we then create an integer pointer like so: int *ptr_to_x;. What happens if we do the following?

int x[] = {2,4,6};
int *ptr_to_x;
void setup() {
  ptr_to_x = x;
  Serial.begin(115200);
}

void loop() {
  Serial.println(*ptr_to_x);
  ptr_to_x++;
}

When your run this and monitor the serial output you get something like the following (every person's will be different every time it is run. The transcript below is just a result of me running it on my machine):

2
4
6
75699904
33620224
259
5505816
6619237
7536750
6553721
6881397
7274606
4391433
-1073741566
264498
33685760
604307457
83955712
16843044
100803588
402437
-2113599743
1073745923

So what is going on? First, the statement:ptr_to_x = x; is saying that our pointer is pointing to the address of the array x. Why didn't we need to use an & operator, however? You'll remember that when dealing with individual integers or floats or chars, when you want to access their address to store in a pointer you need to use the "address of" operator (&, also known as the reference operator) so that ptr_to_founding = &founding_of_mit;, int* ptr_to_founding = &founding_of_mit;, etc... Why don't we need to do that with an array? The answer is that an array's "name" is the equivalent to a pointer to its 0th element. So when we say that ptr_to_x = x;, the assignment works because x is actually a pointer itself.

When accessing an individual element in an array, however, the result is not a pointer. So x[0] returns the value of the integer stored in the 0th element, but x returns the address in memory of the zeroth element. This implies that ptr_to_x = x; is the same as ptr_to_x = &x[0], and indeed you can do both.

Now what is happening in the code up above? Well, first you assign the pointer to the address of the initial element of the x array. Then during each iteration of the loop function you dereference and print out the value the pointer is pointing to, and then increment the pointer just like you do with any other normal variable. The result of incrementing the pointer is that it now points to the next element in memory, which is the next element in the array. Because it is an int pointer it knows to skip four spots in memory (since the size of an int is 4 bytes), and the result is that the next time through it is pointing to the next integer element of the array. The net result is shown below:

So this starts out making sense in the transcript above...we see 2, then 4, then 6, but then what happens? The next number is 75699904. Where did that come from? Well, that is the value sitting in the next four bytes of memory when interpreted as an int. The pointer just keeps right on incrementing through the memory, jumping by four cells each time it goes through loop. This is one of the dangers with pointers and pointer arithmetic (the ++ or -- or any similar operation on the pointer). You can easily start pointing at something outside of what you meant to if you're not careful. The result is usually what are termed "garbage values"...that is, other binary data in memory being interpreted as an int.

Try the code above on your ESP32 and experiment with various methods of creating pointers with the array.

There are many resources about pointers on the web, such as:

• 5-minute-guide-to-c-pointers.
• Pointers in C++
• c-pointer overview
• And others

If you find others that you think are helpful for learning, please share on Piazza!

¹hexadecimal is a base 16 numbering system, using the symbols 0 through F. In comparison to base 10, which we know and love so much, in base 16, when you roll past 9 in counting up, you go to A rather than "10" as we do in base 10, and continue until F, at which time you then roll over to 10 again (which means 16 in base 10). The Internet is loaded with binary-to-decimal-to-hexadecimal converters so it isn't a bad idea to be at least somewhat familiar with them if you're looking into EECS. (click to return to text)

²where exactly in memory different variables get stored is generally up to the compiler. (click to return to text)

³Maybe a time traveler messed something up. (click to return to text)