That is, here's one thing we can say.

Early in this class I had already mentioned a few applications.

Let's build a probabilistic word model of segmentation.

We clearly don't want to have to enumerate them all.

But it turns out that this row here is at least 1,000 times better than any of the other segmentations.

And notice that this is independent.

So this model, simplified though it is, coming up with this naive Bayes assumption, gets this one right, and it does about 99 of the segmentations accurately.

And let's answer that not just for 12 characters, but for n characters in general.

So if we had a Markov model where the probability of insignificant depended on the probability of small, then we could catch that, and we could get this segmentation correct.

We can say that the best segmentation is equal to the argmax over all possible segmentations of the string into a first word and the rest of the words of the probability of that first word times the probability of the best segmentation of the rest of the words.

Now, 2 n is a lot.

Again, that depends on exactly what type of smoothing you choose to do.

In this case, it just seems somewhat like bad luck that that was the right answer, but both segmentations seem like good segmentations.

Then if we take the rest and multiply out this whole term, the best segmentation of the rest times the probability of the first that comes from this column, then that column will give us about 10 to 19 for the segmentation that starts with n, 10 to 13 for the one that starts with no, 10 t the 10 for the one that starts with now, and 10 to 18 for the one starts with nowi.

Now this is going to be a little unwieldy to deal with, so we can make an approximation.

For a given string say we have this string with 12 characters how many possible segmentations are there?

Either you look at config.nice file, in the source tree of your current PHP installation or, if this is not available, you simply run a

By definition, the best segmentation, which we'll call S , is equal to the one which maximizes the joint probability of the segmentation.

So we're going to segment the text into a sequence of words word 1 through word n and find that segmentation into words that maximize the joint probability.

Number two thing that Howard did is he made us realize it's another very critical point he made us realize the importance of what he likes to call horizontal segmentation.

Here it seems like it really has erred that small and insignificant seems like a much better segmentation than the one my program came up with.

Here I've trained it on a corpus of 4 billion words not just the Shakespeare corpus but a larger corpus, and then I give it some test cases to try to find the best segmentation.