Partially-Observable Markov Decision Processess

On Wednesday's lecture we discussed POMDPs. Here is a quick review:

Value Iteration in NetLogo: Finding Optimal Policies for Markov Decision Processes

On Wednesday I will talk about policies and the Value Iteration algorithm.

In the video below you can watch me implement the value iteration algorithm in NetLogo.


If that is too long to watch (it is!) then the one below gives you an overview of the finished implementation.

HW1: Single File

The first homework is a fun simple challenge to get you started learning NetLogo. You will implement a model where the turtles all form a single line (one after the other) that is constantly marching. That is, you will:

1. Set the world to be a torus.
2. Make your 'setup' button create 'num-turtles' (sliders) all in randomly chosen locations.
3. Each turtle has to behave autonomously and can only change its heading or move forward 1.

I am looking for a single line, a queue, with even spacing between each turtle. You will find that your first efforts lead to some turtles moving on top of each other. Try to spread them out.

Also, try not to solve this centrally (by having the observer calculate the exact position each turtle should be, based on their 'who' number). Instead, imagine each turtle is a person, but no one can talk to anyone else. How would people form a single line?

Email me your final .nlogo file, making sure to put your name and email in the 'Info' tab, by Monday, January 28 @10am.

Agent-Based Modeling Tools

I chose to use NetLogo for this class because it lets us build models very quickly without having to worry about all the graphical, GUI, or plotting code. There are many agent-based modeling libraries and platforms out there. NetLogo is the best at getting a novice from knowing nothing to having fun with his custom-built model.

The Mason Java library is good for large-scale simulations, of the type you would run on a 'supercomputer'. The Repast Java library and IDE is another popular library that integrates with eclipse. Obviously, you have to know Java to use either one of these.

NetLogo Tutorials

On Wednesday's lecture I will give a quick introduction to NetLogo. You should install it and start learning how to use it. The best way is to go to "Help->User Manual" and read the whole thing. I also have some video tutorials which you might find useful, starting with the ones below.

All of the homeworks consist of you writing a NetLogo program that implements a given algorithm or problem, starting with the first one, so, you will want to get proficient fast.










The NetLogo models page has the models that implement the algorithms in the book. These are good for you to play with (change paramaters/code) and get a better understanding of the algorithm and emergent behavior of the system.

Utility Theory

Our first meeting is on Monday. On that day we I will be talking about the class, the history of multiagent systems, and getting started on the first chapter of our textbook.

The first chapter talks about utility functions:


Markov Decision Processes


and Value Iteration

Game Theory

If you want to get an early start on this class then checkout the coursera game theory class. It is taught in part by Yoav Shoham, professor at Stanford, who wrote that other Multiagent Systems book. This class, like theirs will start by covering Game Theory concepts.

I might do some of my own videos.

We will then read some papers and try to build some fun agent-based models, using NetLogo.

We are on for Spring 2013

I will be teaching this class again in the Spring 2013, so, please sign up!

The only requirements are a bit of mathematical sophistication and a desire to implement agent-based models, using NetLogo.