Proposal Critique

You're working for one of the world's largest financial institutions (GloboBank). They're building a system to monitor salespeople's electronic communications with the company's customers. The goal is to help reduce bad behavior among the company's salespeople, such as overpromising, understating risks, and so on. The company is unhappy with their current surveillance system, because it creates tons of false alarms, which wastes the time of the analysts, and also they know that it misses a lot of important types of cases. Below is a proposal they have received from a vendor, for a better system, that will address their issues. Specifically, it will monitor each outgoing email from a salesperson and flag those that are suspicious. The flagged-suspicious emails would be examined by an analyst, who would decide which ones ought to be escalated for further investigation. 

Assess the proposal and provide constructive criticism: identify what you assess to be the three most important potential flaws and suggest ways to fix each of them.

Proposal from Yellow Fin Consulting

We will use machine learning techniques to build an AI system to classify emails as "suspicious" or not. Those classified as suspicious will be "flagged"; our surveillance system will rank emails and provide the most suspicious ones to the analysts. The system will maximize the lift at the top of the ranking, and minimize the number of missed cases (false negatives). 

The flagging model will take as input a feature-vector representation of the email, where each word is a feature and the feature value represents whether the word is present in the email; more sophisticated representations will be added later. We will leverage the existing system to provide training labels. Specifically, if the existing system flags the emails as being suspicious, we will give them a label of yes. Otherwise we will give a label of no. As we archive all salesperson emails specifically for compliance purposes, this will allow us almost unlimited training data.

We will evaluate the system based on its generalization accuracy and the area under the ROC curve (AUC) on holdout data. The system should be able to achieve accuracies greater than 90% as well as high AUCs. We also will show the flagged emails to compliance experts for domain knowledge validation.