The Server Side

Spring 2019

The questions below are due on Sunday February 24, 2019; 11:59:00 PM.
 
You are not logged in.

If you are a current student, please Log In for full access to the web site.
Note that this link will take you to an external site (https://oidc.mit.edu) to authenticate, and then you will be redirected back to this page.
A Python Error Occurred:

Error on line 8 of python tag (line 8 of file /S19/ex03/server_side):
    kerberos = cs_user_info['username']

KeyError: 'username'

Back to Exercise 03

Serverside

1) Overview

So far in 6.08 we've been working solely on our microcontroller. In Lab 02B, we started to work a bit with our class server, but again, it was only from the perspective of our microcontroller. Today that changes. We'll take a look at how we can deploy scripts on the 6.08 class server and in future weeks use this quite a bit.

2) Python on the Server

We have a separate class server from the one we host the course website on, called 608dev.net. We used this in Lab 02B, you might remember. We've set up a sandbox on this server where you can place Python files to handle incoming HTTP requests. The way this will work is that you'll create a Python file containing a definition for a specific function request_handler, which will take in one argument called request, a Python dictionary which will contain information about the specific HTTP request. The job of your request_handler function is to respond as needed to each request—and this will vary from exercise to exercise and project to project.

For example, if you'd like to simply echo back the structure of the request dictionary provided you could create the following function:

def request_handler(request):
    return str(request)

This function needs to be placed inside a Python file and placed onto the server. We'll get into how to do that below, but assuming you do place this file onto the server in the correct way, it will now be accessible to the outside world—and when you visit it, the server will use request_handler as you defined it to process the incoming HTTP request.

For example, if you placed the above function definition in a file called test.py, saved it onto the server, and then went and visited the appropriate URL in your web browser, you'd get the following data returned to you:

{'method': 'GET', 'values': {}, 'args': []}

If instead you visited it in the browser and appended some query arguments to the URL such as ?x=5&y=4 you'd get instead:

{'values': {'y': '4', 'x': '5'}, 'method': 'GET', 'args': ['y', 'x']}

If you were to POST to the site, other different keys would be available in the request dictionary for your use. We'll get more practice with the exact structure of this request dictionary in coming exercises and labs. Now, though, we want to focus on how we can place files onto the server.

3) The Sandbox

Each student in 6.08 has had an account on 608dev.net made for them.

  • Your username on 608dev.net is: <div style='color:red;'><b>A Python Error Occurred:</b><p>
    Error on line 1 of python tag (line 10001 of file UNKNOWN):
        print('%s' % (kerberos,))
    
    NameError: name 'kerberos' is not defined
    
    </p></div>
  • Your password on 608dev.net is: <div style='color:red;'><b>A Python Error Occurred:</b><p>
    Error on line 1 of python tag (line 10001 of file UNKNOWN):
        print('%s' % (cs_user_logos[kerberos],))
    
    NameError: name 'kerberos' is not defined
    
    </p></div>

When you log in to the machine, it will put you into your personal home directory. From here you are capable of carrying out a limited number of operations—such as building and editing the file structure, creating and moving files, and other things. There is no GUI to do this however, so you'll need to do this through the terminal. The course staff feels getting some exposure to working in a *Nix-type shell is a useful skill and so we'll expect you to gain some fluency in this in order to carry out homeworks and labs. If you've never done this before, it isn't too bad. We're really only going to be using around four or five commands total and we'll introduce them all today (for the most part).

This server is a sandbox where you, the students, can set up and mess with some server-side Python scripts. While we've put some protections in place so that you don't accidentally bomb the machine or negatively impact other students on the server, we make no guarantees as to the overall security of information on this machine. Consequently, do not put critical personal information on the server. Place Python files, static resources, and databases as needed for this class, but do not put personal information on it like credit card numbers, nuclear launch codes, Equifax administrator login information, etc...

3.1) Getting a Terminal Working

For Mac and Linux folks, you'll be using your terminal. It should already exist. Either control-alt-T or search for it in the Task bar to bring it up. (For Mac users, you can search for 'Terminal' in spotlight (command-space).

Windows users, unless you already have an alternative you like to use, we are going to recommend that you download and install Cygwin with openssh support. It provides a *nix like interface that will enable you to conveniently sftp and/or ssh into our class server as needed. In general you can follow the instructions on the site, but we'll need to make sure you install openssh support, so:

  1. Download the setup.exe for your machine as it says on the site (linked above).
  2. Follow the instructions, choosing "Install from Internet" and place in your "C:\cygwin" directory, allowing all users to have access and support for Linux defaults
  3. Select a download mirror (I usually just pick the first one and it works at least when on MIT's campus)
  4. When it prompts you for additional packages, type in "openssh" into the search field of the package selection window. On the left side of the things that come up, click on the "skip" icon to change the two openssh packages to checkboxes. Then press Next.
  5. Wait for installation to finish and then you should be good!

Once it is installed, proceed to the following section. If you have installation issues, please post on Piazza and we'll help you figure it out.

3.2) How to Think About the Terminal

If you're comfortable working in a terminal, feel free to skip this section. For those of you who are not (and that is totally fine, 6.08 is a judgement-free zone), we're going to talk about the basics.

First when you're in a terminal, just remind yourself that it is simply a textual representation of the GUI file-managers we've all seen on Windows, Mac, and *Nix. When you open up a terminal, it will often place you into you home directory1. To see where you are in the terminal you can always type the following command (followed by hitting enter), which asks for Present Working Directory:

pwd

On my Mac, for example, it spits back the following: /Users/jodalyst. The response you get will probably be slightly different. To see what files and folders are present in your current directory you can type the following (again, followed by hitting enter):

ls

Depending on how clean/organized and/or full your current directory is, either a little or a lot of stuff might fly by. These are all the files and folders currently present in the directory you're in. This just showed you in text what you'd see in your File Manager/Documents/My Computer GUI window.

Windows users: If you've just installed Cygwin, then chances are you'll have an empty "home" directory, meaning running ls will return a blank line. That's ok. You'll fill it up with files soon.

To change into a directory you will use the Change Directory command or cd followed by the directory you want to move into. This is the same as when you double-click on a folder in a graphical file manager.

cd directory_to_move_into

My Documents folder is within my home directory, and within my Documents folder is my Arduino folder. I could therefore do the following:

cd Documents/Arduino

Following that if I then type pwd I'll get back /Users/jodalyst/Documents/Arduino.

If you want to go up one directory level you can type:

cd ..

If you want to go up two levels, you can type:

cd ../..

If you want to move back to the previous directory you were in (whatever that was), you can type:

cd -

If you just want to go/return to your home directory, you can type:

cd 

Or more explicitely (using the ~ symbol which means "home")

cd ~

If you're currently in /Users/jodalyst/Documents/Arduino and you want to move to a completely different directory you can type:

cd ~/other_folder/other_nested_folder

Please feel free to Google or ask on Piazza and/or in office hours for help with this. It can seem very weird at first.

3.3) Remote Server

Ok, now we've got a little idea about how to move around using the command line. So far we were doing it on our own local machine. Now we want to do it on a remote machine. There's two general ways to do this. The first is with sftp and the second is with ssh and scp. You can also use these three programs in parallel depending on your needs.

  • ssh stands for Secure SHell, and is a way for you to get into the terminal on a remote machine. You can do anything you can do on your local machine on this remote machine through this (in theory...in practice we greatly limit you in 6.08 for the greater good). When logged in over ssh, you can make directories, cp files and even edit them (have to use a text editor, and your options are nano or vim)
  • scp stands for SSH COPY and is a means of transfering files to and from one machine to another.
  • sftp stands for SSH File Transfer Protocol where SSH stands for Secure SHell. It is a program that allows you to move files between your local machine (your laptop) and a remote machine (in this case the server 608dev.net) as well as create and manipulate them on the remote machine.

Let's first look at how to work with sftp and then we'll look at how to use ssh and scp. It would be good to gain some fluency in all of these.

4) SFTP

To log in to your home on the server, you need to run the following command from your terminal on your computer:

sftp 
A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

@608dev.net

It will prompt you for your password. Your specific password is: <div style='color:red;'><b>A Python Error Occurred:</b><p>

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (cs_user_logos[kerberos],))

NameError: name 'kerberos' is not defined
</p></div>. It will not display any characters or visual feedback while you are typing so be aware of that. If you get an error like ssh does not exist and you're on Windows, it means you didn't install openssh when you were installing cygwin up above so you'll need to go back and redo that (don't worry, the Cygwin package reinstalls pretty easily). If it doesn't like your password, email me (jodalyst@mit.edu)

If you successfully log in, you'll get the following message:

Connected to 608dev.net
sftp>

You are now in the server. You can exit at any time from this login session using the following command, which will return you back to your home computer.

exit

Within the sftp environment on the server, all the commands we previously talked about will work similarly. For example, typing pwd right when you log in will spit back: Remote working directory: /home/

A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

. Similarly, you could type ls, although as right now your remote home directory is empty, the command will return nothing.

For practice, let's make a directory by using another command, mkdir (Make Directory). Go ahead and run the command mkdir testdir. This will make a directory called "testdir" inside of where you are currently located in your file structure (your home directory on 608dev.net). Then, if you run ls, you'll see it show up. You've just made a directory! You have some power. Now let's move into it using the cd command....so do cd testdir. Then verify where you're at by running pwd (you should see Remote working directory: /home/

A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

/testdir).

4.1) Transfering Files in SFTP

OK, almost done. We now know how to move around on our remote server and make directories. This will be critical to setting up our file structure for integrating with the web sandbox. Next, we'll need the ability to move files onto this remote directory, such as Python files.

On your local computer, create a Python file called testo.py (use your favorite Python editor, we have no biases in 6.08) and inside of it create the following request_handler function definition:

def request_handler(request):
    return "Request Dictionary: " + str(request)

Save testo.py to your home directory ON YOUR LOCAL COMPUTER. Windows-Cygwin users, you'll need to figure out where your *Nix-like home directory is. On my Windows machine it is under C:\cygwin64\home\jodalyst since my name is jodalyst, but that's me and your's is probably different.

Back in the sftp shell, run the following command:

put testo.py 
It will respond with:
Uploading testo.py to /home/
A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

/testdir/testo.py testo.py

What this has just done is move the file testo.py from its specified location on your home computer (~/testo.py) onto where you're currently located on the server.

If you go to the following site in your web browser: A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

/testdir/testo.py, you should get the following response:
Request Dictionary: {'method': 'GET', 'values': {}, 'args': []}

This is showing up because of code that you placed on the server!

If you added some query arguments at the end like so:
A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

/testdir/testo.py?hey=5&yo=11 you'll get:
Request Dictionary: {'method': 'GET', 'args': ['yo', 'hey'], 'values': {'yo': '11', 'hey': '5'}}

If you see this, then congratulations, you've successfully placed a Python script on our system. This is basically how we'll deploy Python on the server in this course.

There are lots of other commands used in sftp that may be useful, but really pwd, ls, cd, mkdir, put, and possibly get are the big ones we'll be using. Feel free to ask more questions on Piazza about them.

A helpful table is provided below2

Command Function
cd (and lots of other things) Change the directory on the remote host.
chmod Change the permissions of files on the remote host.
chown Change the owner of files on the remote host.
exit (or quit) Close the connection to the remote host, and exit SFTP.
get Copy a file from the remote host to the local computer.
help (or ?) Get help on the use of SFTP commands.
lcd Change the directory on the local computer.
lls List the contents of the current directory on the local computer.
lmkdir Create a directory on the local computer.
ln (or symlink) Create a symbolic link for a file on the remote host.
lpwd Show the present working directory on the local computer.
ls (or dir) List the contents of the current directory on the remote host.
lumask Change the local umask value.
mkdir Create a directory on the remote host.
put Copy a file from the local computer to the remote host.
pwd Show the present working directory on the remote host.
rename Rename a file on the remote host.
rm Delete a file on the remote host.
rmdir Remove a directory on the remote host (the directory usually has to be empty).
version Display the SFTP version.

5) SSH and SCP

Another way to work with the server remotely is using the programs ssh and scp. In contrast to sftp, you can think of ssh as a fully-fledged way of working directly on the server (as if it was your own computer), but it lacks the ability to transfer files between machines like you can in sftp. We can use scp to fill in that gap however. It acts like the standard cp command (which stands for "copy"), but can act between machines.

When using ssh and scp at the same time, I usually have two terminal windows open. One logged in using ssh to the remote machine, and one active in my local machine. With that in mind in one terminal window, let's log into the server using ssh. To do that you'd do enter the following:

ssh 
A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

@608dev.net

It will prompt you for your password. Type it in...(again for reference for you it is: <div style='color:red;'><b>A Python Error Occurred:</b><p>

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (cs_user_logos[kerberos],))

NameError: name 'kerberos' is not defined
</p></div>). As a reminder, it will not show any visual feedback while typing. This is for security reasons so don't freak out.
When into the terminal you can basically do anything on the server that you'd do locally in your own shell. For a simple task let's make a second folder called testdir2 in your home directory. This can be done with:

mkdir ~/testdir2

copy the file you already put on previously via sftp into testdir2. Do this with the following:

cp ~/testdir/testo.py ~/testdir2/testo.py

Cool! If you go to the following site in your web browser: A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

/testdir2/testo.py, you should get the following response:
Request Dictionary: {'method': 'GET', 'values': {}, 'args': []}

This is showing up because of code that you placed on the server!

If you added some query arguments at the end like so:
A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

/testdir2/testo.py?hey=5&yo=11 you'll get:
Request Dictionary: {'method': 'GET', 'args': ['yo', 'hey'], 'values': {'yo': '11', 'hey': '5'}}

What if you wanted to transfer a file onto the server while working in ssh? You can't...but usually what you can do is with a second terminal use scp. Let's give that a try. On your local computer create a python file called nice_message.py that has the following in it:

def request_handler(request):
    return "Live your best life!"

Now in the terminal that is active in your local computer navigate to where that file is located (use some cd calls and verify with pwd and ls calls.). Once there do the following:

scp nice_message.py 
A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

@608dev.net:~/

This will prompt for you password. Enter it, and then you'll see it reporting the transfer.

Now back in the other terminal, move to your home directory on the server (just a simple call cd), and then see what files are there (ls)...you should see among the other files nice_message.py appears. Huzzah!

If you then went to A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

/nice_message.py in a browser you'll see your nice message appear!

If you want it to look differently, we could add some in-line css for styling. Edit the local file so that it contains the following:

def request_handler(request):
    return """<b style="color:green;">Live your best life!</b>"""

Re-copy it via scp to the server and revisit the page (careful you'll overwrite...it doesn't matter here, but be careful in the future). Now when you revisit the URL in the browser, you should see some nice green text, encouraging you3.

If you end up modifying some stuff on the server and want to get it back onto your local computer you can also use scp. To do that in your local computer's terminal do the following:

scp 
A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

@608dev.net:~/nice_message.py ./nice_message_bup.py

What this will do is copy from the specified directory on the server the file nice_message.py and name it nice_message_bup.py in your local directory on your computer!

So there you have it! A quick summary of some things you can do with ssh is shown below.

Command Function
cd (and lots of other things) Change the directory on the remote host.
chmod Change the permissions of files on the remote host.
chown Change the owner of files on the remote host.
exit (or quit) Close the connection to the remote host, and exit SFTP.
help (or ?) Get help on the use of SFTP commands.
ls (or dir) List the contents of the current directory on the remote host.
mkdir Create a directory on the remote host.
pwd Show the present working directory on the remote host.
rename Rename a file on the remote host.
rm Delete a file on the remote host.
rmdir Remove a directory on the remote host (the directory usually has to be empty).
nano An easy to use text editor here's a quick intro to how to use it
vim A more capable text editor4 vim is also a pain to get started in, so don't stress if it is confusing make sense. We put it there if you know it and for staff debugging, but don't feel obligated to use it.

And remember in order to use scp, when on a terminal in a local computer's environment to transfer from the local computer to a remote one you do:

scp file_location/local_file_name username@remote_host:remote_file_location/remote_file_name

and to transfer from a remote computer/server to your local machine you do:

scp username@remote_host:remote_file_location/remote_file_name file_location/local_file_name

6) File Structure on 608dev.net

Regardless of how you manipulate files, in general, any Python file you place on the server will be served from its location relative to your home directory. For example, if you place a file right in your home directory, called fun.py, you can access it at http://608dev.net/sandbox/sc/<div style='color:red;'><b>A Python Error Occurred:</b><p>

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined
</p></div>/fun.py.

And if you made a file structure such that there were a series of nested folders like so:

home > project > integration > messaging

and you put a file called response.py in it, then you could access it at: http://608dev.net/sandbox/sc/<div style='color:red;'><b>A Python Error Occurred:</b><p>

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined
</p></div>/project/integration/messaging/response.py

In future weeks we'll discuss how to set up databases and serve static files, but for today, we'll just worry about placing Python files that handle HTTP request dictionaries in a stateless manner.

7) Writing a Request Handler Function

To finish this exercise off, you're going to need to write a request_handler function and then place it in the correct location on 608dev.net.

7.1) The Spec

The file should be placed under a directory called ex03 in your home directory and be called initial_test.py. The particular implementation of request_handler you need to implement should take in two arguments and carry out some math operations on them. In particular determine and repor the Least Common Multiple and Greatest Common Divide. Specifically:

  • The function should expect two query arguments, x and y
  • If an x and y argument are not provided the function must return the following: "You must enter an x and y value."
  • If either x or y is not a number, the function must return "Both x and y must be valid numbers." If you're curious about how to try do that in Python consider using Try-Except logic. Check out the Python 3 documentation on how to handle errors.5
  • The system will only work with integers. Treat non-integers the same as non numbers above.
  • If a POST request happens, the function must return "POST requests not allowed."
  • If everything is good, return the least common multiple and the greatest common divisor of the two numbers in the following format: "The LCM of {} and {} is {}. The GCD of {} and {} is {}." where the {} will be replaced with actual appropriate values.

When put all together, if your function is correct and you place it correctly (in the right location) on the site and you visit the following URL with specified query arguments like so: A Python Error Occurred:

Error on line 1 of python tag (line 10001 of file UNKNOWN):
    print('%s' % (kerberos,))

NameError: name 'kerberos' is not defined

/ex03/initial_test.py?x=20&y=30, you should get back:
The LCM of 20 and 30 is 60. The GCD of 20 and 30 is 10.

Use the code checker below to get your request_handler function working. Feel free to use the test cases provided for local debugging (don't just debug in the code checker here).

All code on the server is limited to a run time of a few seconds. If your code doesn't execute by a certain point it will be terminated and you will get an error message. You shouldn't need to very long to do things on the server.

def request_handler(request): pass

Finally, we must make sure that you've placed your code at the proper spot on the server. It isn't that we don't trust you, but we just want to make sure you're doing things right. Run the check below which throws some HTTP requests at the location on 608dev.net that corresponds to where you should have placed your file. You can ignore the text input-it doesn't get used.

If it returns a 0.00% that means you do not have the working function from above in a file called initial_test.py within a directory called ex03 in your home directory on the server.

Back to Exercise 03


 
Footnotes

1though not a guarantee (click to return to text)

2From this helpful site at Indiana University (click to return to text)

3While a lot of what we put on the server is intended for consumption by another machine, and therefore doesn't require pretty-looking fonts in different colors, you can directly create html-text for rendering in a browser as needed here (click to return to text)

4Also the best text editor in all of humanity. JK obviously, you use what you like best. No emacs is not installed on the sandboxed environment. (click to return to text)

5In Python it is often easier to ask for forgiveness rather permission...by this we mean try to do something and catch the failure if it doesn't work out. (click to return to text)



This page was last updated on Sunday March 10, 2019 at 09:32:39 PM (revision b2109a2).
 
Course Site powered by CAT-SOOP 14.0.4.dev5.
CAT-SOOP is free/libre software, available under the terms
of the GNU Affero General Public License, version 3.
(Download Source Code)
CSS/stryling from the Outboxcraft library Beauter, licensed under MIT
Copyright 2017