- Project proposals are due on March 8. (deadline moved)
- Each project should have 2-3 contributors. For <2 or >3 contributors, please speak to the instructor during office hours.
- Provide Brief Description of the project at most 2-page, and least 1 page. Summarize the project, provide a reading list, and directions to be explored.
- Online Submission Only, link will be provided soon
- A progress report is due on April 5.
- A poster session will be organized (May 6) where you can present your results. For survey papers, I will try to schedule presentations in class, depending on number of such projects proposed.
- You may use another week to improve the reports, and submit final report by May 10.
You can pick any topic for the project you wish so long as there is some direct connection to reinforcement learning. For your project, you can implement an RL solution, or you can think about a theoretical problem or algorithm, or you can do a blend of both. You can work individually or with a partner. Working as a pair is very much encouraged -- you are likely to learn more, have more fun, and accomplish more if working with a classmate.
Here are examples of possible types of projects:
- Study and survey one of the many related topics not covered in this course. Examples include: inverse RL, multiagent RL, imitation learning, use of RL in application domains like robotics, self-driving cars, natural language processing, vision, etc., To get ideas, browse assigned or optional readings, or the chapters of books that we did not cover, or explore recent survey articles.
- Design algorithms and run experiments on some simulation environments like advanced environments on openAI gym, Mujoco simulator http://www.mujoco.org/, OpenSim http://opensim.stanford.edu/. Compare to existing work.
- Find an open theory problem, and attempt to make progress on it. Besides the discussions in published papers, a great place to look for these is the proceedings of recent COLT conferences (Conference on Learning Theory) which every year publishes short open-problem papers (some of which have monetary awards for solving). (Obviously, since these were published a while ago, it is possible that some of these will have already been solved. But that doesn't mean they can't be the starting point for a project.)
- This page from a class on reinforcement learning provides some very good project ideas.
- Build or apply an algorithm tailored to a particular application coming from some other field such as medicine, HCI, natural language processing, chatbox, robotics.
- Come up with your own idea! Be original and creative.
The end result of your project should be a written report clearly and concisely describing what you did, what results you got and what the results mean. Your report should be 5-6 pages long. If necessary, you can include further details or plots/figures in at most 5 page appendix. The report should use 11pt font, 1-inch margins, and single spacing. Papers that vary from these guidelines risk receiving a grade deduction and/or some sections not being read.
Your report should follow the general format of a scholarly paper in this area. You should write your report as clearly as possible in a manner that would be understandable to a fellow student of this course. In other words, you should not assume that the reader has background beyond what has been covered in class (as well as the prerequisites of the course).
Your report should begin by describing the problem you are studying, some background (what has been done before) and the motivation for the problem, i.e., why it is worth studying. Previous work and outside sources should be cited throughout your report in a scholarly fashion following the style of academic papers in this area.
Next, you should clearly explain what you did, both from a high level, and then in more detail. For an experimental paper, you should explain the experiments in enough detail that there is a reasonable possibility that a motivated reader would be able to replicate them. You also should outline some of the theory underlying the algorithms you are studying. State your results clearly, and think about graphical tools you can use to make your results clearer (a table of numbers is usually less compelling than a graphical representation of the same data). Look through published papers for ideas. For a theoretical paper, the model and other mathematical details should be explained well enough for the results to be stated with mathematical precision and clarity.
In every case, be sure to explain the meaning of your results. Don't just give a table of results or a dry mathematical formula. Explain what the results mean, and what conclusions can be drawn from them. Again, do all this in a way that would be understandable and interesting to a fellow student of this course. What did you expect to find? What did you find instead? What are the implications? If you found something surprising, can you think of how it might be explained?
What you will be graded on: Projects will be graded along the following dimensions:
- originality and creativity
- background material
- theoretical component and/or experimental design and execution
- discussion and interpretation of results
- writing of the final report, including clarity, completeness and conciseness
- overall effort
(contents of this page were derived from this excellent guidance from Rob Schapire on choosing projects for his course at Princeton.)