Let's learn about cross entropy

Cross-entropy can be used as a loss function for classification models like logistic regression or artificial neural networks.

It basically describes the loss between two probability distributions.

The "true" distribution (the one that your machine learning algorithm is trying to match) is expressed in terms of a one-hot encoding distribution.

Pr -> Probability of that label

Consider a specific training instance, with the true label as B

One hot encoding will be:

Pr(Class A) Pr(Class B) Pr(Class C) 0.0 1.0 0.0

Suppose your model predicts:

Pr(Class A) Pr(Class B) Pr(Class C) 0.228 0.619 0.153

Loss will be:

L = - ( 0.0 ln(0.228) + 1.0 ln(0.619) + 0.0 * ln(0.153) ) L = 0.479

That's how "wrong" or "far away" your prediction is from the true distribution.

Basically cross-entropy describes the loss between two probability distributions and it is among one of many possible loss functions.

Hope now you have understanding of what is Cross Entropy

Don't forget to like and share it.