Turning data into insights doesn’t happen magically. You must first understand your data and use it to create reports that drive actions. If your competitors are using machine learning and artificial intelligence to automatically drive actions and you aren’t, you are at a disadvantage.

Getting your data ready for ML and AI involves combining structured and semistructured data sets in order to clean and standardize data into a format ready for machine learning or integration with BI and data visualization tools. When you prepare your data correctly, you benefit from insights that can be processed quickly and easily, resulting in faster time to value.

Data transformation and standardization help you build powerful models, reporting, and ad hoc analysis that all share a single source of truth. In fact, not only does data prep help you with AI models, you can use AI in your ETL process to prepare data for the data warehouse itself. For example, you can use AI to extract valuable sentiment data from customer comments without having to read them all. Either way, at the beginning of a data journey, a company’s problem is not analytics or model-fitting, it is data ingestion and transformation.

Based on our customers’ experiences, there are common data transformations required before data is ready for use in machine learning models.

Remove unused and repeated columns: Handpicking the data that you specifically need will improve the speed at which your model trains and unclutters your analysis.

Change data types: Using the correct data types reduces memory resources. It can also be a requirement — for example, making numerical data an integer in order to perform calculations or to enable a model to recognize what algorithms are best suited to the data.

Handle missing data: At some point you’ll come across incomplete data. Tactics for resolving the problem can vary depending on the data set. For example, if the missing value doesn’t render its associated data useless, you may want to consider imputation — the process of replacing the missing value with a simple placeholder or another value, based on an assumption. Otherwise, if your data set is large enough, it is likely that you can remove the data without incurring substantial loss to your statistical power. Proceed with caution. On the one hand, you may inadvertently create a bias in your model; on the other hand, not dealing with the missing data can skew your results.

Remove string formatting and non-alphanumeric characters: You will want to remove characters such as line breaks, carriage returns, and white spaces at the beginning and the end of values, currency symbols, and other characters. You may also want to consider word-stemming as part of this process. Although removing formatting and other characters makes the sentence less readable for humans, this approach helps the algorithm better digest the data.

Convert categorical data to numerical: Although not always necessary, many machine learning models require categorical data to be in a numerical format. This means converting values such as yes and no into 1 and 0. However, be cautious not to accidentally create order to unordered categories, for example, converting Mr., Miss, and Mrs. into 1, 2, and 3.

Convert timestamps: You may encounter timestamps in all types of formats. It’s a good idea to define a specific date/time format and consistently convert all timestamps to this format. It’s often useful to “explode” a timestamp (using a data warehouse date dimension) into its constituent parts — separate year, month, day-of-week, and hour-of-day fields all have more predictive power than milliseconds since 1960.

Getting Started

This list is not exhaustive and is offered as a simple guideline to get you started. There are other factors you may want to consider such as how to handle outliers. You may want to remove them from your data set depending on the training model you use. Retaining outliers may skew your training results, or you might need to include outlier data for an anomaly detection algorithm.

To get the most from data analytics and visualization tools, have your data ready and available for analytics by bringing all the relevant data together in a clean and standardized format to ensure that the data is high-quality and can be trusted. Preparing this as a pipeline of operations within a cloud ETL tool means that when you need to bring more data up to date, potentially from many different external sources, you can just press “Run” again and all data is refreshed.

The post Preparing Data for Machine Learning appeared first on Artificial Intelligence.

As someone who often finds himself explaining machine learning to non-experts, I offer the following list as a public service announcement.

1. 1. Machine learning means learning from data; AI is a buzzword. Machine learning lives up to the hype: there are an incredible number of problems that you can solve by providing the right training data to the right learning algorithms. Call it AI if that helps you sell it, but know that AI is a buzzword that can mean whatever people want it to mean.

1. 1. Machine learning is about data and algorithms, but mostly data. There’s a lot of excitement about advances in machine learning algorithms, and particularly about deep learning. But data is the key ingredient that makes machine learning possible. You can have machine learning without sophisticated algorithms, but not without good data.
   2. Unless you have a lot of data, you should stick to simple models. Machine learning trains a model from patterns in your data, exploring a space of possible models defined by parameters. If your parameter space is too big, you’ll overfit to your training data and train a model that doesn’t generalize beyond it. A detailed explanation requires more math, but as a rule you should keep your models as simple as possible.

1. Machine learning can only be as good as the data you use to train it. The phrase “garbage in, garbage out” predates machine learning, but it aptly characterizes a key limitation of machine learning. Machine learning can only discover patterns that are present in your training data. For supervised machine learning tasks like classification, you’ll need a robust collection of correctly labeled, richly featured training data.
2. Machine learning only works if your training data is representative. Just as a fund prospectus warns that “past performance is no guarantee of future results”, machine learning should warn that it’s only guaranteed to work for data generated by the same distribution that generated its training data. Be vigilant of skews between training data and production data, and retrain your models frequently so they don’t become stale.
3. Most of the hard work for machine learning is data transformation. From reading the hype about new machine learning techniques, you might think that machine learning is mostly about selecting and tuning algorithms. The reality is more prosaic: most of your time and effort goes into data cleansing and feature engineering — that is, transforming raw features into features that better represent the signal in your data.
4. Deep learning is a revolutionary advance, but it isn’t a magic bullet. Deep learning has earned its hype by delivering advances across a broad range of machine learning application areas. Moreover, deep learning automates some of the work traditionally performed through feature engineering, especially for image and video data. But deep learning isn’t a silver bullet. You can’t just use it out of the box, and you’ll still need to invest significant effort in data cleansing and transformation.
5. Machine learning systems are highly vulnerable to operator error. With apologies to the NRA, “Machine learning algorithms don’t kill people; people kill people.” When machine learning systems fail, it’s rarely because of problems with the machine learning algorithm. More likely, you’ve introduced human error into the training data, creating bias or some other systematic error. Always be skeptical, and approach machine learning with the discipline you apply to software engineering.
6. Machine learning can inadvertently create a self-fulfilling prophecy. In many applications of machine learning, the decisions you make today affect the training data you collect tomorrow. Once your machine learning system embeds biases into its model, it can continue generating new training data that reinforces those biases. And some biases can ruin people’s lives. Be responsible: don’t create self-fulfilling prophecies.
7. AI is not going to become self-aware, rise up, and destroy humanity. A surprising number of people (cough) seem to be getting their ideas about artificial intelligence from science fiction movies. We should be inspired by science fiction, but not so credulous that we mistake it for reality. There are enough real and present dangers to worry about, from consciously evil human beings to unconsciously biased machine learning models. So you can stop worrying about SkyNet and “superintelligence”.

There’s far more to machine learning than I can explain in a top-10 list. But hopefully this serves as a useful introduction for non-experts.

The post Ten Things Everyone Should Know About Machine Learning appeared first on Artificial Intelligence.