Class Review


The lectures presented in this class were enjoyable and fun to go to, especially the early ones. However, as class went on, to me, things traveled into harder and harder material, that involved a lot of higher level math. The problem there for me was a lot of the math was skimmed over, so the focus was put on the wrong spots. I know I took Calc 3 and Statistics, and I may be an abnormality here, but I took my last math class outside of differential equations about 2 years ago at tech. So the stuff isn’t entirely fresh in my mind. (This doesn’t mean that I am not familiar with the subjects however, it just means that I need a reminder every once and a while).

So, two ways I could address this is:
1) Studying on my own ahead of time, if I knew what portion I needed a refresher on.

2) Having that basic level re-introduced in class.

Now, normally, there wouldn’t be much issue with this, and I am perfectly capable of taking notes and figuring it out outside of class, but the in-class breaks detracted from this, because usually it involved either asking Robert, whom isn’t the best of help when it comes to explaining things, or Sterling, whom is just as lost as me sometimes, as he is another one of those people who have taken the math years ago.

So the in class breaks? For the first few weeks, I felt it was very productive, but that also may be because I was able to help those around me understand the concepts.

I would also suggest maybe a refresher on matrix math, or at least jacovians, as that can get complicated, and much of the class is based around it, as it favors matrix approaches to problems instead of what I am going to call algorithmic, or equation based solutions.


Polls are terrible.

Firstly, I believe this because to be due to the inconsistency of the posting time.  I know Piazza was sending me emails, but I have no clue where it was sending them , as I could never find them, and I have no clue where to change my email in that system. So that is kinda an issue on Piazza’s side.

Secondly, the polls were posted inconsistently, so I couldn’t develop much of a pattern of going onto piazza to answer them.  So I would have enjoyed a bit of regularity as to when to get on to answer them. Also, on that note, Piazza didn’t really put the polls into one place, And I can think of at least one occasion where I got on to attempt to answer a poll, and didn’t because I thought it wasn’t posted, but it turned out it was buried under a pile of other peoples questions.

As far as poll content, the polls were not really posted predictably enough to get a group of people together on a regular basis to do them.  And the content was mostly only useful as far as I personally felt as to attempt to figure out what would be on the next quiz. Though this even changed throughout the semester, as this became a bit less reliable. So over all, I am meh on the subject of polls, but in my opinion, the best two things that could be done to improve them would be:

1)      Have them set up to be sent out on a regular timer, like for example an hour after class is finished. This way, groups can be formed for this. Also this way, it turns them into regular focus questions. My biggest issue is the regularity as it is anyways. Currently, I just make an attempt to answer them a few hours before class just to make sure I do it.

2)      Have them be a different color on Piazza or something to separate them from the regular comments. That way they are identifiable.


As far as the assignments go, I feel they where enjoyable, with my biggest issue with them being a lack of practical implementation experience for the robot ones, (The Rovio one didn’t count in my mind. The Rovio is a whole another issue, and it would be a great project on another platform.)

The best project was the first one. This one had the implementation and the practical of using our code on a real robot. The worst project was most likely the Rovio one.

I would also say the last project was funky, and didn’t quite follow the pattern set forth by the other projects. Mostly, because I wish I had a bit more guidance on what to pick as far as stuff. I feel like getting pointed to a website and told pick two isn’t the best way to go about it. At least not with that timeframe given for that project. It felt rushed to me.

And the Kinematics one was special. Besides sitting in a hotel puking as I was attempting to do it, It comes back to the sudden switch into hard math.


The quizzes shifted for me as the semester went on. The beginning I thought they were great, as they followed the polls online, and to me where at a good level of difficulty. Then they shifted and tangent a bit, and became a hair harder to follow and guess what we should study for them.


I think the portfolio is a great concept, as I was attempting to start this on my own already.  I’m sorry I didn’t put as much time as I should have when I was working on it, but It became a bit of an afterthought. What I think would be cool, and more effective then the portfolio and poll system as it stands is merge the polls and the portfolio assignments, having the class talk about the class weekly. This way, it becomes a genuine reflection on the class, forces us to focus on it, and It most importantly gives you feedback as to what we are thinking, way more so then a poll question that can be guessed. You can also then better judge our individual interests and what we focus on during the class. Once again, another positive.

So yes, I think more portfolio would be a good thing. One other class I was in took this concept soo far as to have a tumbler account set up for the class, and had us post something about the papers for the class, could be anything, before the class started. This gave him both insight into our thoughts, and forced us to be prepared for class.

So yes to portfolios, and yes to more.

Final Comments

Andrew is awesome. And I loved the AR Drone. Lets do more with it in the future.  Also, I loved the class, it was one of the more enjoyable ones I’ve been in. I just wish that it stayed with robots for the last project.  What I really wanted to do for the last project was implement vision mapping for the AR Drone. Problem was I didn’t find anything on my quickish search that would allow me to do that with SLAM. Mostly, I suppose I wasn’t looking hard enough, but I still think that would have been an awesome last assignment. Even if we implemented (or attempted) to implement our own slam algorithm. Actually, that would have been 100 times better than running two others in my book.

SLAM Algorthms

SLAMfinal <- PDF link to the paper we wrote.

First and foremost about these SLAM algorithms  They, in general, seem a hair platform specific, and need to be calibrated to the application. Which means applying them to an arbitrary dataset can be a bit of a challenge. That, or I suck at picking SLAM algorithms to run experiments on. Either is a valid possibility. Continue reading “SLAM Algorthms”


This thing is one sexy, sexy beast of a robot. At least when your coming from a Rovio. For one, it is accurate, secondly, it is responsive. ( Did I mention the senors are accurate too?

But, onto the meat of this encounter…

For this little tidbit, we had to take this thing with ROS, and get it to “fixate” on three color dots that we printed out. The greatest challenge for this was convincing this bot that Blue was Blue due to shifting lights in the room. (It kept attempting to turn blue green). But all in all, once we got the camera calibrated, it worked like magic

It worked very, very well too, and I was very happy with this assignment. As an experiment, I took a hold of the aft of the cowling on the robot, and pulled it back a hair, and it has… amazing stickiness to that colored dot. And by that, I mean it sprung right pack to where it should be. We could also lead it about my moving the dots about (ie, carrying them). All in all, this was a fun bot to work with.



First and foremost, the Rovio, as a robotics platform sucks. This is of course because the sensors in it are terry inaccurate.

The second reason for this is that the same commands, given repeatedly, give vastly different results. Another sign of the horribleness of the platform.

I also am a bit disappointed by this assignment, because it feels like we didn’t actually do anything autonomous with the Rovio platform. We just gave it dead reckoning commands, told it to run them, then analysed what it did. It just doesn’t feel fun or interesting to me. But that may be just me on this too.

So heres to hoping that we get to have more fun with the next robot!


Q Learning


The implementation for section two was pretty straight forward. We had to implement the value itteration aspect, and the value computation aspect

Value Iteration:
This was the most complicated aspect of project two. The core of it is defined by

Q(s,a) = R(s,a) +gV(d(s,a))

This says, that the QValue for action on every square on the map is defined by
the sum of the reward value for performing that action on that square, plus the
value of going to that square in the first place.

So to start with, we use a standard itterative for loop to access all the squares
on the map.

Then we go ahead and for each square, we populate the appropiate value for that
action on that square. Of course, we also set for the fringe cases where the
action would cause us to move off of the map. For these cases, we set the Q value
to zero.

Then, we just follow this standard:

Q[x][y][i] = R[x][y][i] * .25 + V[x1][y+(i-1)] * .5

of course, this is modified to be generic. A slight variation in wich the 3
possible moves are hardcoded are used in the code.

I chose the values of .25 as the weight for the reward and .5 for the weight for
the value because they work, and they meet the rule where the sum of the weights
needs to be less than 1.

Value Computation:
This portion is trivial. We compare all the possible actions in Q for a square,
and set the value to be the largest one. So just a max() function.

How Value Itteration Works:

Value itteration works itteritively… Every time the function is called to learn, it
goes ahead and has each square recalculate its q_value.

Each square, in turn, looks ahead only ~one~ move, and sets its q_value based upon
the reward of going into the target square, plus the biggest residual q_value from
prior learning calls.

This allows the q_values to grow over time, and eventually point the way to the
greatest reward. The q_values also make sense because the q_values are the value
function more or less. The value function just represents the biggest q_value.

So basically, Q_values provide for a giant, creeping map of values that give a weight
to a square based upon the optimal path from all the squares it can get to to a
reward square.


My implementation of Q_learning closely resembles the prior sections q_learning,
except that it only performs Q_learning on the last two nodes visited, rather than
updating the whole grid with every learning section.

And it can be summed up in about one line of code:

Q[s.x][s.y][a] = Q[t.x][t.y][a]*(1-alpha-.5) + alpha *(reward + V[t.x][t.y] * .5)

So we set the Q value of the square we came from to be equal to sum of the Q value of
the square we just entered with respect to direction plus the greatest Q_value in the
square we just came from, and we add the reward value to that for good mesure.

This way, if we enter trash, the Q_Value of the square we came from goes down.

If we enter a good square, the Q_Value goes up.

Then we recycle the calculate_values from the last section (Mostly because I didnt
feel like writing it again, and it has no functional effect on how the code operates)
and things start to work

The downside is, this type of Q_Learning is slow, and takes many, many, many

It behaves very similarly to Value itteration, bieng a subset of the problem where
we only focus on the relationship of two squares, rather then the relationship of
every square on the map with its neighbors.

Exploration is useful as a way to speed up Q_Learning, and, if it somehow gets
itself into a working path, to get it out of the path it is stuck in.

How I set my values, however, tends to have the agent doing exploration on its
own, without needing me to specify the need for exploration. This doesnt mean
that I couldnt turn on exploration to speed things up, or to hit points it isnt
focusing on. It just means that for most cases, I dont have to.

3Pi Robot, A Reflection:

These past few weeks, Ive been playing with the 3pi robot for CS3630. Fun experience, but when it came down to testing, didn’t quite work out. And of course, there’s allays a few things we like to say about the experience.

First off, the build design was done without really testing the robot in person, mostly because I didnt have the maze-following code finished in time to be used during the competition.

The design was to create a robot that used DFS to find the shortest path through the maze. It did this by creating a representation of the maze in memory (All 2kB of it), holding only visited, left, right, straight and parent information. It also did this relying ~very~ heavily upon sensors. It was not reliant on timing loops by doing so.

Downfalls of course are we dont record x-y coords of nodes, and memory concerns. ( 256 total nodes in a perfect heap. And theres no way Im getting all 2kB for these elements in the heap ). Also, it required writing a C++ Library for the bot, which attempted to mirror the Java library used for the simulator. (The rest of course just somehow magically working because of processing for the most part). And unfortunately  despite how close the two languages are, they do not translate 1 to 1, even if I avoid using libraries within them.

The DFS itself ran on the preference L/R/S, which of course is a bit strange, but made no difference in production.

The downfall of the bot was our speed. Because I spent all the time coding, I didnt get time to test things, (Seeing as we had very limited access to the bot itself), and didnt get to verify that settings that worked in the simulator also worked in Real Life Conditions. And because we turned all our speeds way down in order to give the senors more time to acquire and process data as we moved, it turns out the bot outright didn’t move when it the code was brought into the real world. And even after a quick fix, our turns where off, and for somer reason, instead of turning on a dime, it made arcs. (and arcs are subpar for what we where doing).

so, short story short, what works in the simulator doesnt work in the lab.

Link to the Codebase: GITHUB