How to Train Your Network

Just like training a dragon, training a network to do your bidding gets tricky at times and requires patience and understanding. After several rounds of training and retraining, more new grey hair than we care to count and many hours of obsessing about how to best train a network to detect defective units of a consumer product mass produced to the tune of millions per day we have learned a few lessons about convolutional networks and their proper training:

• Convolutional networks have a natural error rate. They are only as good as the eye of the human who trains them. If you don’t see it and train your network accordingly, the network will not recognize it.
• Human biases and perceptions get easily baked into a algorithm during training. If the “trainer” is red-green blind and trains the algo accordingly, the network won’t be able to differentiate between red and green.
• Beware of category shifts that get introduced when human tagging of the training set is inconsistent, which it is bound to be after, say, tagging brown vs. red-furred cats for hours on end. Is that kitty still a red-head or already a brunette? If I call it a brunette, did I tag a similar borderline case in the same way three hours ago or have my categories shifted as I get tired and annoyed by looking at thousands of cats?
• Getting somebody else to help with tagging will likely make the problem worse. Is their idea of red-furred cat the same as mine? Probably not, so each additional tagger will introduce their own bias and inconsistency further “confusing” the network.
• The way to go in borderline cases is to get several experts to agree on the category and then consistently apply that category to these borderline cases across the entire training set. This might sound straight-forward but can be difficult in real life.
• If their is no human consensus the network will not be able to make the perfect call, just like that dragon when you give it two conflicting commands.

Once the network has been trained and an acceptable error rate has been agreed upon, we have established an objective measure of quality. Manufacturing output can now be tracked on an objective basis with statistical certainty based on the accuracy rate. Unlike a human inspector, the network will give you a consistently accurate assessment of quality.

One last word of caution: just like your dragon, your network will need retraining on a regular basis so performance will improve over time and with exposure to more training data.