Project 5: Second Learning Project

Overview video

Project 5 tasks and ideas

As with project 4, this project is somewhat open-ended but with constraints.  Your task is to implement either RL or a genetic algorithms method to improve the behavior of your agent.  Some ideas are listed below.  You are welcome to use any of these ideas but you must propose your plan in part 1 (just as with project 4).  We want to help make sure your ideas are feasible!

Some ideas to get you thinking:

• Use genetic algorithms to decide the best breakpoints for each decision in a heuristic agent.  For example, instead of always going to beacons if energy is less than a constant, learn what the different constants should be for the different choices.  This would probably be a better choice for the cooperative environment rather than the CTF competitive environment.
• Use GAs or RL to create effective behavior (this would work in either environment, just needs to handle multiple agents in CTF).  For example, you create a set of high-level behaviors such as “go to flag” “go to nearest asteroid” “go to home base” etc and then you learn when to choose each action to maximize your score.
• Use RL to train a policy for a specific action such as shooting an opponent (best in CTF/competitive) or for navigating without hitting obstacles (either environment)
• Use RL or GAs to train a moveToObject action that performs efficiently (and does not suffer from the extreme slow-downs that PD controllers do).

A few notes to help make this doable:

• RL is easiest if you have a discrete state and action space.  To make this much easier, you should propose an approach to discretizing the state and action space in your proposal.  This approach can be very simple (gridding the variables you need for the task you chose) or it could rely on other methods such as clustering.
• GAs can handle a continuous state space, depending on your problem definition.  If you are choosing to the idea of GAs to learn the best behavior cutoffs, that is definitely best done with a continuous state space!  If you are choosing to do the state -> action idea (learning a behavior policy), that is best done with a discrete state space but I would also say much better with RL than GAs.  Each has their advantages!

CS 5013 students need to implement two learning methods but one of these can be from the traditional methods in project 4 (e.g. if you can use RL or GAs AND make use of your previous learning, that works, or you could use clustering to create a state space, or some other way to combine two).  You could also implement two RL or GA tasks.

Part 2: Due Dec 1 11:59 PM

1. Update your code from the last project. 
2. Write your learning code as described above
3. Build and test your code using the ant compilation system 
4. Submit your project on spacesettlers.cs.nor.ou.edu using the submit script as described below.  You can submit as many times as you want and we will only grade the last submission.
   • Submit ONLY the writeup to the correct Project 5 on canvas:
     • Project 5 for CS 4013
     • Project 5 for CS 5013
   • Submit your file using one of the following commands (be sure your java files come last). You can submit to only ONE ladder. If you submit to both, small green monsters will track you down and deal with you appropriately.

/home/spacewar/bin/submit --config_file spacesettlersinit.xml \
--project project5_coop \
--java_files *.java

/home/spacewar/bin/submit --config_file spacesettlersinit.xml \
--project project5_compete \
--java_files *.java

• • After the project deadline, the above command will not accept submissions. If you want to turn in your project late, use:

/home/spacewar/bin/submit --config_file spacesettlersinit.xml \
--project project5_coop_late \
--java_files *.java

/home/spacewar/bin/submit --config_file spacesettlersinit.xml \
--project project5_compete_late \
--java_files *.java