So let's really think about what we're doing when we're executing the active TD learning algorithm.

If we tried to go up from this state because that's what the policy said but, stochastically, we slipped over to this state then we wouldn't do something else, according to the policy and so we'd get a little bit of exploration, but we'd only vary from the chosen path because of that variation and we wouldn't intentionally explore enough of the space.

Now, one possibility is, certainly, random exploration.

And it's converging to much better results.

First, we're keeping track of the optimal policy we've found so far and that gets updated as we go, and replaced with new policies.

And we can see for example, this looks like a mistake here.

I'm closing up the wall over here, opening the wall down here, run it.

What does that mean? so that's the difference between the policy that the agent has learned and the optimal policy.

A policy was not met.

Policies

Policy

Now what that suggests is the design for an exploration agent that will be more proactive about exploring the world when it's uncertain, and will fall back to exploiting the optimal policy or whatever policy it has as close to optimal when it becomes more certain about the world.

That's not our policy, sir.

So we'll start at a start state, we'll apply the policy and let's say the policy tells us to move in this direction.

I have mortgage and I have young child and I want to make sure that if anything were to happen to me over the next...

I've also graphed, with a dotted line, the policy loss.

Java policy

JavaScript policy

New Java Policy

New JavaScript Policy

And if we did end up here, then we'd immediately head back towards the road, so we'd aim back down in this direction.

As you can see, each grid cell now has a label.

I'll serve his purpose.

So we wish to have a planning method that provides an answer no matter where we are and that's called a policy.

Default Policy

Duplicate Policy

Default Policy

Send policy

Offline Cache Policy

So that policy does, in fact, work randomly making moves with some probability but it tends to be slow to converge.

Use setting from global policy.

Realize that even states I'm not likely to every reach,

The immortal leader of our race. To the order for which he stands.

They happen to be 1000 for the two obstacles.

This is the way of backing up values, and once values have been backed up, this is the way to find the optimal thing to do.

Domain Specific Policies

Bad policy

but the insurance company aren't silly people. They're very ughh scrutinize probabilities.

I pledge myself... to your teachings.

While I haven't reached the goal state quite yet as indicated by checking for the star in my policy table.

Now to solve an MDP, we're trying to find a policy pi of S that's going to be our answer.

And they all stem from the same cause, from the fact that passive learning stubbornly sticks to the same policy throughout.

After conversions for this grid, I want the value function to look just like this.

I'll act on your advice.

I will follow your advice.