![\"\"](\"http:\/\/bskog.com\/ai\/wp-content\/uploads\/2017\/05\/Ska\u0308rmavbild-2017-05-23-kl.-21.32.13.png\")
<\/p>\nWhen you have to deal with non-linear decision making, you can use decision trees to transform it into a linear decision surface.<\/p>\n
Let say we have a buddy that goes wakeboarding\u00a0if the weather is sunny but\u00a0not windy. Whenever he sees the sun is up he considers going wakeboarding, but if it is too windy there will be too much waves on the lake and it is not as fun as when the surface is still. This data is not linearly separable, as shown in the following image: We can’t separate this with one line.<\/p>\n
<\/p>\n
Decision trees\u00a0enables you to\u00a0make several linearly separable decisions one after another. In this case when we look at the data we can clearly see that it follows a pattern. First we can see that for instance if it is windy, he will not go wakeboarding regardless of if it is sunny or not. So by asking ourselves, is it windy? we can get a definite answer if it is windy, then we will not go wakeboarding.<\/p>\n
![\"\"](\"http:\/\/bskog.com\/ai\/wp-content\/uploads\/2017\/05\/Ska\u0308rmavbild-2017-05-23-kl.-21.32.28.png\")
If the answer was yes, we had an answer to our question if we shall go wakeboarding, and if it is not windy, we will ask another question: is it sunny? And if not, we won’t go wakeboarding, but if it is sunny, we’ll head to the lake.<\/p>\n
![\"\"](\"http:\/\/bskog.com\/ai\/wp-content\/uploads\/2017\/05\/Ska\u0308rmavbild-2017-05-23-kl.-21.42.40.png\")
<\/p>\n
This way, we could make a non linearly separable data set into a linear one by stepping through a decision tree.<\/p>\n
<\/p>\n
If you then look at a little bit more complex data. You can see that for\u00a0after a certain treshold on x, the data behaves differently. Here the data is not linearly separable.<\/p>\n
![\"\"](\"http:\/\/bskog.com\/ai\/wp-content\/uploads\/2017\/05\/Ska\u0308rmavbild-2017-05-23-kl.-21.54.00.png\")
<\/p>\n
This can be made into linear decisions using a decision tree. ![\"\"](\"http:\/\/bskog.com\/ai\/wp-content\/uploads\/2017\/05\/Ska\u0308rmavbild-2017-05-23-kl.-21.53.42.png\")
<\/p>\n
Decision trees are easy to understand and interpret since they can be visualized as a tree structure. The data need not be prepared much whereas other methods may require normalisation. As we saw earlier that naive Base was good for classifying text, a decision tree is good when it comes to numerical and categorical data. They are, however, prone to overfitting. If some classes dominate in the training data, the generated tree may become biased. In that case the testing data needs to be balanced\u00a0before training.<\/p>\n
You want to find split points and variables that can separate the dataset into as pure subsets as possible, meaning that there\u00a0the classes\u00a0are preferably of only one type in the subset of the\u00a0data. This is then done recursively with the generated subsets until you have classified the data.<\/p>\n
Entropy (a measure of impurity) is what a\u00a0decision tree uses to determine where to split the data when constructing the tree. Entropy is the opposite of purity, if all examples of a sample are of the same class, then the entropy is 0. If all examples are evenly<\/span> split between all the available classes, then\u00a0the entropy is 1.<\/p>\n Decision Trees use the entropy of a node to calculate what the split shall be.<\/p>\n <\/p>\n Pi<\/sub> is a fraction\u00a0of examples in the\u00a0class i.<\/p>\n <\/p>\n The way a decision tree affects its boundaries (choosing which features to make a split on) using entropy is by maximizing\u00a0something called information gain.<\/p>\n Information gain = entropy(parent) – [weighted average]entropy(children)<\/p><\/blockquote>\n <\/p>\n See\u00a0the example found in this video <\/a>for an explanation about using information gain to choose which feature to use to split data in a decision tree.<\/p>\n <\/p>\n <\/p>\n Here you can find more information about decision trees:\u00a0http:\/\/scikit-learn.org\/stable\/modules\/tree<\/a><\/p>\n![\"\"](\"http:\/\/bskog.com\/ai\/wp-content\/uploads\/2017\/05\/entropy.png\")
<\/p>\n