Business intelligence techniques help understand trends and identify patterns from big data

In the digital world, modern businesses generate big data on daily basis. The recent advancement in technology has opened the door for companies to effectively store and process big data to unleash data-driven decisions and insights. Unfortunately, there is a void between data storage and usage. Many companies, starting from small to big, collect huge data but only use very little of it to make business decisions. In order to mitigate this data gap, business intelligence is being deployed. With the rise in the need for real-time data processing, business intelligence techniques have exploded, making data and analytics accessible for more than just analysts. While business intelligence technology helps decision-makers to analyze data and make informed decisions, top business intelligence techniques drive the initiatives. They help analysts understand trends and aid them to identify patterns in the mountains of big data that businesses build up. The need for more disruption in decision-making and the growing demand for business intelligence has opened the door for a surplus amount of business intelligence techniques. In this Article, Analytics Insight has listed top business intelligence techniques that help companies to get the maximum out of big data.

• TOP BUSINESS INTELLIGENCE ROLES AND SALARIES ONE SHOULD KNOW ABOUT IN 2021
• 5 BUSINESS INTELLIGENCE TOOLS ONE MUST ACQUIRE IN 2021
• BUSINESS INTELLIGENCE IMPACT ON ONLINE CASINO INDUSTRY

Top Business Intelligence Techniques

OLAP

Online Analytical Processing (OLAP) is an important business intelligence technique, that is used to solve analytical problems with different dimensions. A major benefit of using OLAP is that its multi-dimensional nature provides leniency for users to look at data issues from different views. By doing so, they can even identify hidden problems in the process. OLAP is mainly used to complete tasks like budgeting, CRM data analysis, and financial forecasting.

Data Visualization

Data is often stored in form of numbers that are put together as a matrix. But interpreting the matrix to make business decisions is a critical task. A commoner, or even an analyst, can find the progress of data when it is stored as a set. To untangle the knot, data visualization is used. Data visualizations help professionals look at data from more than one dimension and help them make informed decisions. Therefore, visualization of data in charts is an easy and convenient way to understand the stance.

Data Mining

Data mining is the process of analyzing large quantities of data to discover meaningful patterns and rules by automatic or semi-automatic means. In a corporate data warehouse, the amount of data stored is very huge. Finding the actual data that could drive business decisions is quite critical. Therefore, analysts use data mining techniques to unravel the hidden patterns and relationships in data. Knowledge discovery in databases is the whole process of using the database along with any required selection, processing, sub-sampling, choosing the proper way for data transformation.

Reporting

Reporting in business intelligence represents the whole process of designing, scheduling, generating the performance, sales, reconciliation, and saving the content. It helps companies to effectively gather and present information to stand by the management, planning, and decision-making process. Business leaders get to view the reports at daily, weekly, or monthly intervals as per their needs.

Analytics

Analytics in Business Intelligence defines the study of data to extract effective decisions and figure out the trends. Analytics is famous among business companies as it lets analysts and business leaders deeply understand the data they have and drive value from it. Many business perspectives, from marketing to call centers to use analytics in different forms. For example, call centers leverage speech analytics to monitor customer sentiments and improve the way answers are presented.

Multi-Cloud

Following the outbreak of the pandemic and the lockdown that came to effect, companies across the globe started moving their routine working into cloud modes. The rise of cloud technology has greatly impacted many businesses. However, even after the restrictions are lifted, companies still prefer to work over the cloud because of its lenient accessibility and easy-to-use attributes. Moving a step forward, even Research & Development initiatives are being moved to the cloud, thanks to its cost-saving and easy-to-use nature.

ETL

Extraction-Transaction-Loading (ETL) is a unique business intelligence technique that takes care of the overall data processing routine. It extracts data from storage, transforms it into the processor, and loads it into the business intelligence system. They are mainly used as a transaction tool that transforms data from various sources to data warehouses. ETL also moderates the data to address the need of the company. It improves the quality level by loading it into the end targets such as databases or data warehouses.

Statistical Analysis

Statistical analysis uses mathematical techniques to create the significance and reliability of observed relations. It also grasps the change of behavior in people that are visible in data with its distribution analysis and confidence intervals. Post data mining, analysts carry out statistical analysis to devise and get effective answers.

The post TOP BUSINESS INTELLIGENCE TECHNIQUES TO STREAMLINE DATA PROCESSING appeared first on Artificial Intelligence.

What Is Data Science?

The terminology came into the picture when the amount of data had started expanding in the starting years of the 21st century. As the data increased, there was a newly emerged need to select only the data that is required for a specific task. The primary function of data science is to extract knowledge and insights from all kinds of data. While data mining is a task that involves finding patterns and relations in large data sets, data science is a broader concept of finding, analyzing, and providing insights as an outcome.

In short, data science is the parent category of computational studies, dealing with machine learning, and big data.

Data science is closely related to Statistics. But as opposed to statistics, it goes way beyond the concepts of mathematics. Statistics is the collection, interpretation of quantitative data where there is accountability for assumptions ( like any other pure science field). Data science is an applied branch of statistics dealing with huge databases which require a background in computer science. And, because they are dealing with such an incomprehensible amount of data, there is no need to consider assumptions. In-depth knowledge of mathematics, programming languages, ML, graphic designing, and the domain of the business is essential to become a successful data scientist.

How Does It Work?

Several practical applications provide personalized solutions for business problems. The goals and working of data science depend on the requirements of a business. The companies expect prediction from the extracted data; to predict or estimate a value based on the inputs. Via prediction graphs and forecasting, companies can retrieve actionable insights. There’s also a need for classifying the data, especially to recognize whether or not the given data is spam. Classification helps in work reduction in further cases. A similar algorithm is to detect patterns and group them so that the searching process becomes more convenient.

Commonly Used Techniques In The Market

Data Science is a vast field; it is very difficult to name uses of all the types and algorithms used by data scientists today. Those techniques are generally categorized according to their functions as follows:

Classification – The act of putting data into classes on both structured and unstructured data (unstructured data is not easy to process, at times distorted, and requires more storage).

Further in this category, there are 7 commonly followed algorithms arranged in ascending order of efficiency. Each one has its pros and cons, so you have to use it according to your need.

Logistic Regression is based on binary probability, most suitable for a larger sample. The bigger the size of the data, the better it functions. Even though it is a type of regression, it is used as a classifier.

The Naïve Bayes algorithm works best on a small amount of data and relatively easy work such as document classification and spam filtering. Many don’t use it for bigger data because the algorithm turns out to be a bad estimator.

Stochastic Gradient Descent is the algorithm that keeps updating itself after every change or addition for minimal error, in simple words. But a huge problem is that the gradient changes drastically even with a small input.

K-Nearest Neighbours is typically common to deal with large data and acts as the first step before further acting on the unstructured data. It does not generate a separate model for classification, just shows the data nearest to the K. The main work here lies in determining the K so that you get the best graph of the data.

The Decision Tree provides simple visualized data but can be very unstable as the whole tree can change with a small variation. After giving attributes and classes, it provides a sequence of rules for classifying the data.

Random forest is the most used technique for classification. It is a step ahead of the decision tree, by applying the concept of the latter to various subsets within the data. Owing to its complicated algorithm, the real-time analysis gets slower and is difficult to implement.

Support Vector Machine(SVM) is the representation of training data in space, separated with as much space as possible. It’s very effective in high dimensional spaces, and very memory efficient. But for the direct probability estimations, companies have to use an expensive five-fold cross-validation.

Feature Selection – Finding the best set of features to build a model

Filtering defines the properties of a feature via univariate statistics, which proves to be cheaper in high-dimensional data. Chi-square test, fisher score, and correlation coefficient are some of the algorithms of this technique.

Wrapper methods search all the space for all possible subsets of features against the criterion you introduce. It is more effective than filtering but costs a lot more

Embedding maintains a cost-effective computation by using a mix of filtering and wrapping. It identifies the features that contribute the most to a dataset.

The hybrid method uses any of the above alternatively in an algorithm. This assures minimum cost and the least number of errors possible.

The post What Is Data Science And What Techniques Do The Data Scientists Use? appeared first on Artificial Intelligence.

Data science has taken hold at many enterprises, and data scientist is quickly becoming one of the most sought-after roles for data-centric organizations. Data science applications utilize technologies such as machine learning and the power of big data to develop deep insights and new capabilities, from predictive analytics to image and object recognition, conversational AI systems and beyond.

Indeed, organizations that aren’t adequately investing in data science likely will soon be left in the dust by competitors that are gaining significant competitive advantages by doing so.

What exactly are data scientists doing that provides such transformative business benefits? The field of data science is a collection of a few key components: statistical and mathematical approaches for accurately extracting quantifiable data; technical and algorithmic approaches that facilitate working with large data sets, using advanced analytics techniques and methodologies that tackle data analysis from a scientific perspective; and engineering tools and methods that can help wrangle large amounts of data into the formats needed to derive high-quality insights.

In this article, we’ll dive deeper into common statistical and analytical techniques that data scientists use. Some of these data science techniques are rooted in centuries of mathematics and statistics work, while others are relatively new ones that take advantage of the latest research in machine learning, deep learning and other forms of advanced analytics.

How data science finds relationships between data

When trying to identify information needles in data haystacks, data scientists, first, need to discern how different data elements correlate with or relate to each other. For example, if you have a bunch of data points plotted on a graph, how do you know if there’s any meaning in them?

Perhaps the data represents a relationship between two or more variables and the job is to plot some sort of line or multidimensional plane that best describes the relationship. Or perhaps it represents clustered groups that have some affinity. Other data could represent different categories. By finding these relationships, we give meaning to the otherwise randomness of the data, which can then be analyzed and visualized to provide information that organizations can use to make decisions or plan strategies.

Now, let’s look closer at the various data science techniques and methods that are available to perform the analysis.

Classification techniques

The primary question data scientists are looking to answer in classification problems is, “What category does this data belong to?” There are many reasons for classifying data into categories. Perhaps the data is an image of handwriting and you want to know what letter or number the image represents. Or perhaps the data represents loan applications and you want to know if it should be in the “approved” or “declined” category. Other classifications could be focused on determining patient treatments or whether an email message is spam.

The algorithms and methods that data scientists use to filter data into categories include the following, among others:

• Decision trees. These are a branching logic structure that uses machine-generated trees of parameters and values to classify data into defined categories.
• Naïve Bayes classifiers. Using the power of probability, Bayes classifiers can help put data into simple categories.
• Support vector machines. SVMs aim to draw a line or plane with a wide margin to separate data into different categories.
• K-nearest neighbor. This technique uses a simple “lazy decision” method to identify what category a data point should belong to based on the categories of its nearest neighbors in a data set.
• Logistic regression. A classification technique despite its name, it uses the idea of fitting data to a line to distinguish between different categories on each side. The line is shaped such that data is shifted to one category or another rather than allowing more fluid correlations.
• Neural networks. This approach uses trained artificial neural networks, especially deep learning ones with multiple hidden layers. Neural nets have shown profound capabilities for classification with extremely large sets of training data.

Regression techniques

What if instead of trying to find out which category the data falls into, you’d like to know the relationship between different data points? The main idea of regression is to answer the question, “What is the predicted value for this data?” A simple concept that comes from the statistical idea of “regression to the mean,” it can either be a straightforward regression between one independent and one dependent variable or a multidimensional one that tries to find the relationship between multiple variables.

Some classification techniques, such as decision trees, SVMs and neural networks, can also be used to do regressions. In addition, the regression techniques available to data scientists include the following:

• Linear regression. One of the most widely used data science methods, this approach tries to find the line that best fits the data being analyzed based on the correlation between two variables.
• Lasso regression. Lasso, short for “least absolute shrinkage and selection operator,” is a technique that improves upon the prediction accuracy of linear regression models by using a subset of data in a final model.
• Multivariate regression. This involves different ways to find lines or planes that fit multiple dimensions of data potentially containing many variables.

Clustering and association analysis techniques

Another set of data science techniques focuses on answering the question, “How does this data form into groups, and which groups do different data points belong to?” Data scientists can discover clusters of related data points that share various characteristics in common, which can yield useful information in analytics applications.

The methods available for clustering uses include the following:

• K-means clustering. A k-means algorithm determines a certain number of clusters in a data set and finds the “centroids” that identify where different clusters are located, with data points assigned to the closest one.
• Mean-shift clustering. Another centroid-based clustering technique, it can be used separately or to improve on k-means clustering by shifting the designated centroids.
• DBSCAN. Short for “Density-Based Spatial Clustering of Applications with Noise,” DBSCAN is another technique for discovering clusters that uses a more advanced method of identifying cluster densities.
• Gaussian mixture models. GMMs help find clusters by using a Gaussian distribution to group data together rather than treating the data as singular points.
• Hierarchical clustering. Similar to a decision tree, this technique uses a hierarchical, branching approach to find clusters.

Association analysis is a related, but separate, technique. The main idea behind it is to find association rules that describe the commonality between different data points. Similar to clustering, we’re looking to find groups that data belongs to. However, in this case, we’re trying to determine when data points will occur together, rather than just identify clusters of them. In clustering, the goal is to segregate a large data set into identifiable groups, whereas with association analysis, we’re measuring the degree of association between data points.

Data science application examples

The above methods and techniques in the data science tool belt need to be applied appropriately to specific analytics problems or questions and the data that’s available to address them. Good data scientists must be able to understand the nature of the problem at hand — is it clustering, classification or regression? — and the best algorithmic approach that can yield the desired answers given the characteristics of the data. This is why data science is, in fact, a scientific process, rather than one that has hard and fast rules and allows you to just program your way to a solution.

Using these techniques, data scientists can tackle a wide range of applications, many of which are commonly seen across different types of industries and organizations. Here are a few examples.

Anomaly detection. If you can find the pattern for expected or “normal” data, then you can also find those data points that don’t fit the pattern. Companies in industries as diverse as financial services, healthcare, retail and manufacturing regularly employ a variety of data science methods to identify anomalies in their data for uses such as fraud detection, customer analytics, cybersecurity and IT systems monitoring. Anomaly detection can also be used to eliminate outlier values from data sets for better analytics accuracy.

Binary and multiclass classification. One primary application of classification techniques is to determine if something is or is not in a particular category. This is known as binary classification, because we could ask something like, “Is there a cat in the picture, or not?” A practical business application is to identify contracts or invoices among piles of documents using image recognition. In multiclass classification, we have many different categories in a data set and we’re trying to find the best fit for data points. For example, the U.S. Bureau of Labor Statistics does automated classification of workplace injuries.

Personalization. Organizations looking to personalize interactions with people or recommend products and services to customers first need to group them into data buckets with shared characteristics. Effective data science work enables websites, marketing offers and more to be tailored to the specific needs and preferences of individuals, using technologies such as recommendation engines and hyper-personalization systems that are driven by matching the data in detailed profiles of people.

That’s just a sample of useful data science applications. By understanding the various techniques, methods, tools and analytical approaches, data scientists can help the organizations that employ them achieve the strategic and competitive benefits that many business rivals are already enjoying.

The post 15 common data science techniques to know and use appeared first on Artificial Intelligence.

Future telescopes like the James Webb Space Telescope (JWST) and the Atmospheric Remote-sensing Infrared Exoplanet Large-survey (ARIEL) are designed to sample the chemistry of exoplanet atmospheres. Ten years from now, spectra of alien skies will be coming in by the hundreds, and the data will be of a higher quality than is currently possible.

Astronomers agree that new analysis techniques, including machine learning algorithms, will be needed to keep up with the flow of data and have been testing options in advance. An upcoming study in Monthly Notices of the Royal Astronomical Society trialed one such algorithm against the current gold standard method for decoding exoplanet atmospheres to see whether the algorithm could tackle this future big-data problem.

“We got really good agreement between [the answers from] our machine learning method and the traditional Bayesian method that most people are using,” said Matthew Nixon. Nixon is the lead researcher on the project and an astronomy doctoral student at the University of Cambridge in the United Kingdom.

However, “as we increased the parameter space, the computational efficiency of our method drops…. As we started to add more parameters, we started to get hit by the curse of dimensionality.”

A Random Forest Breathing in Exotic Air

Astronomers measure the spectrum of an exoplanet’s atmosphere when starlight shines through it or when heat from inside the planet lights it up from within. In either scenario, the atmosphere imprints its chemical signature on the light and is detected by our telescopes.

The current front-runner for best deciphering a planet’s spectrum is called atmospheric retrieval. It uses statistical inference to calculate the likelihood that given an observed spectrum, an exoplanet’s atmosphere has a certain composition, temperature, level of cloud cover, and heat flow. The technique has so far proven very reliable but can be computationally expensive.

“The more detailed the data, the more detailed the model needs to be,” said Ingo Waldmann, an astrophysicist at University College London in the United Kingdom who was not involved with this study. “Perhaps unsurprisingly, the more detailed the model, the longer it takes to compute its results. Today we are rapidly reaching a stage where our traditional techniques become too slow to compute these increasingly complex models.”

Nixon and his advisor and coauthor, Nikku Madhusudhan, also at the University of Cambridge, tested a type of supervised machine learning algorithm called a random forest, which is made up of thousands of decision trees. Each decision tree makes its prediction for a likely combination of atmospheric properties, and then the algorithm generates an artificial spectrum that has those properties. The algorithm compares each artificial spectrum with the real one and chooses the closest match.

The researchers tested their algorithm on two exoplanets with exceptionally well studied atmospheres and found that the random forest’s solution matched the one from atmospheric retrieval. Moreover, “the authors achieve a much faster interpretation of the data than otherwise possible with traditional techniques,” Waldmann said.

However, the two exoplanets in question, WASP-12b and HD 209458b, are both very hot Jupiter-sized planets. The algorithm could easily simplify its decision because each planet’s atmosphere consists mostly of hydrogen and helium, Nixon said.

“Generally speaking,” Madhusudhan explained, “it is going to be slightly harder to retrieve atmospheric properties of cooler and smaller planets,” for example, super-Earths or Earths. “This is because the spectral signatures are expected to be smaller for such planets, which makes it harder to extract the same amount of information as we have for hot Jupiters currently.” For planets with faint signals and those whose base constituents are unknown—an ocean world, super-Earth, or temperate-zone planet—the random forest would lose its computational edge.

A Balanced Approach for the Way Forward

This study adds to a growing effort by exoplanet scientists to find an efficient way to handle the upcoming deluge of atmospheric data. “It is great to see a growing group in the community using machine learning methods and cross-checking each other’s results and claims,” said Daniel Angerhausen, an astrophysicist at ETH Zürich in Switzerland who was not involved with this research.

Missions like JWST and ARIEL are first at bat, but Angerhausen is also thinking about missions that will come after those. Astronomers will need to strategize the most efficient ways to observe interesting targets. “This problem is predestined for a [machine learning] approach,” Angerhausen said. A random forest approach is just the “tip of the iceberg” for algorithms to try.

Nixon agreed and said that “going forward, looking at different machine learning algorithms is definitely a positive [step], and also looking at how we can combine these machine learning approaches into hybrid methods to really boost these retrievals to the next level.”

As exoplanet atmosphere research moves into the big-data era, machine learning will become an increasingly important research tool scientists should be trained to use, Madhusudhan said. Some graduate programs are already integrating more data science learning into students’ training. (Nixon’s doctorate work is supported by one such program in the United Kingdom.)

“On the other hand,” Madhusudhan added, “it also needs to be recognized that while machine learning is a great research tool in various areas, there are also important areas of research where other numerical, statistical, and analytic approaches are more suitable for some important problems. Therefore, I believe the right balance needs to be met while integrating machine learning into graduate programs in the right research areas.”

“Machine learning may never replace an atmospheric expert,” Waldmann said, “but I’m certain that artificial intelligence will certainly play a role as a helping hand.”

The post Machine Learning Can Help Decode Alien Skies—Up to a Point appeared first on Artificial Intelligence.

As per a recent report, the global process automation market size was somewhat around $138 billion in 2016 and is expected to grow at a CAGR of 6.6% to $178 billion in 2020. Now, with the current pandemic, it’s quite debatable whether that’s achievable.

However, startups have been strongly rooting for leveraging AI in their daily operations. Siddhartha S, founder, IN-D tells us why is that of paramount importance.

Have you learnt any critical lessons during the pandemic? Does technology play a major role in easing things out?

One critical lesson reinforced in last few months is the need for organisations to be able to run a location-agnostic seamless operation. KYC and customer onboarding, insurance claims administration, processing invoices, onboarding employees, even audits – an organisation needs to leverage the power of Artificial Intelligence (AI) to be able to run these functions remotely but seamlessly. Therefore, what was earlier a need for which people were exploring solutions has become a necessity now, and in some cases critical to surviving.

How is IN-D leveraging AI to the maximum possible extent?

The problem with wider adoption of companies providing AI-based platforms is that they have been treated more like services companies than product companies. So, you bear the expense of training of their models in terms of your time and money. IN-D is an AI platform that helps organizations do all this without the time or cost to train and deploy AI models. Because IN-D comes with a suite of pre-trained, ready to deploy solutions, the user organization thus feel like a product for her specific problem. For instance, IN-D KYC is like any other KYC product, but lower in cost and more intelligent because of which it is adaptable to new IDs or new countries’ requirements. Same is the case with IN-D HR which can help companies’ onboard employees but also help colleges complete the admission process remotely for the coming season by reading and understanding degree certificates and marksheets, company offer letters and relieving letters, etc.

Could you give us an insight into the execution of technology at IN-D?

As a company, we are completely focused on getting data from documents, images and videos, followed by synthesising it into actionable information. All our products have this mechanism at their core. To execute this, we use various machine learning and deep learning techniques. IN-D’s genesis is at the AI Lab of Intain, therefore, we retain the same R&D focus and rigor in areas of computer vision, NLP etc.

How critical do you think is it for organisations to be fully tech-enabled? Would that be the best option?

As mentioned earlier, this is now a necessity. It is critical for organisations adopt AI in their operations and important for companies like ours to make this adoption easier. As far as follies are concerned, there are two things to bear in mind – (1) AI is about decisions and hence probabilistic, so it will never be like a rule-based software. The threshold has to ensure whether it is better, faster, and cheaper than the current operating model. (2) Ethics in AI is a wide area of debate. An easy example here is that if we fed all the data to an AI engine and trained it based on past outcomes of loans, it might be able to map even buildings and neighborhoods, which in turn may correspond to communities. It will create the best model to decide on a loan, but the question will be whether that is the most ethical model?

Where does IN-D see itself 3 years down the line?

IN-D now has ready to deploy solutions for six different industry specific or enterprise processes. In some of these like KYC, we believe that the industry has got a raw deal till now and we will work towards market leadership. In other areas, like Operational Risk, we hold a unique position where our job is to create the market for AI enabled automation.

Our immediate focus is India, SE Asia and Middle East. Over the long term, we will expand to Europe and the US, and also add to our ready to deploy solutions portfolio – like legal contracts and equities research.

Takeaway for the wannabes?

Each venture is different. However, something common that I see between the journeys of Intain, a US and Europe focused capital markets platform using a blockchain at the core, and IN-D, an AI enabled automation platform,  are frugality and focus. For example, to stick the core R&D spirit of IN-D, we refuse too much routine process automation work that does not involve AI and if critical to a process, work with other RPA players and system integrators. This means that we have to be good enough that these large multi-billion-dollar entities partner with us to complement their offering. Frugality is a painful lesson many startups are learning through this crisis, so I think no one will advising on this anymore!

The post ‘Frugality is a painful lesson many startups are learning through this crisis appeared first on Artificial Intelligence.

When training a machine learning or AI model, typically the main goal is to make the most accurate prediction possible. Data scientists and machine learning engineers will transform their data in myriad ways and tweak algorithms in any way possible to bring that accuracy score as close to 100 percent as possible, which can unintentionally lead to a model that is difficult to interpret or creates ethical quandaries.

Considering the increasing awareness and consequences of faulty AI, explainable AI is going to be “one of the seminal issues that’s going to be facing data science over the next ten years,” Josh Poduska, Chief Data Scientist at Domino Data Lab noted during his talk at the recent virtual Open Data Science Conference (ODSC) East.

What is Explainable AI?

Explainable AI, or xAI, is the concept of understanding what is happening “under the hood” of AI models and not just taking the most accurate model and blindly trusting its results.

It is important because machine learning models, and in particular neural networks, have a reputation for being “black boxes,” where we do not really know how the algorithm came up with its prediction. All we know is how well it performed.

Models that are not easily explainable or interpretable can lead to some of the following problems:

• Models that are not understood by the end user could be used inappropriately or, in fact, could be wrong altogether.
• Ethical issues that arise in models that have some bias towards or against certain groups of people.
• Customers may require models that are interpretable, otherwise they may not end up using them at all.

Furthermore, there are recent regulations, and potentially new ones in the future, that may require models, at least in certain contexts, to be explainable. As Poduska explains, GDPR gives customers the right to understand why a model gave a certain outcome. For example, if a banking customer’s loan application was rejected, that customer has a right to know what contributed to this model result.

So, how do we address these issues and create AI models that are more easily interpretable? The first issue is to understand how one wants to apply the model. Poduska explains that there is a balance between “global” versus “local” explainability.

Global interpretability refers to understanding generally the resulting predictions from different examples that you feed your model. In other words, if an online store is trying to predict who will buy a certain item, a model may find that people within a certain age range who have bought a similar item in the past will purchase that item.

In the case of local interpretability, one is trying to understand how the model came up with its result for one particular input example. In other words, how much does age versus purchase history affect the prediction of one person’s future buying habits?

Techniques for Understanding AI Reasoning

One standard option that has been around for a while is the concept of feature importance, which is often examined in training decision tree models, such as a random forest. However, there are issues with this method.

A more sophisticated option is called SHAP (SHapley Additive exPlanations). The basic idea behind this option is to hold one input feature of the model constant and randomize the other features, in order to estimate how that feature contributes to the prediction. The downside here is that this method can be very computationally expensive, especially for models with a large number of input features.

For understanding a model on a local level, LIME (Local Interpretable Model-agnostic Explanations) builds a simpler, linear model around each prediction of the original model in order to understand an individual prediction. This method is much faster, computationally, than SHAP, but is focused on local interpretability.

Going even further than the above solutions, some designers of machine learning algorithms are starting to reconstruct the underlying mathematics of these algorithms in order to give better interpretability and high accuracy simultaneously. One such algorithm is AddTree.

When training an AddTree model, one of the hyperparameters of the model is how interpretable the model should be. Depending on how this hyperparameter is set, the AddTree algorithm will train a decision tree model that is either weighted toward better explainability or toward higher accuracy.

For deep neural networks, two options are TCAV and Interpretable CNNs. TCAV (Testing with Concept Activation Vectors) is focused on global interpretability, in particular showing how important different everyday concepts are for making different predictions. For example, how important is color in predicting whether an image is a cat or not?

Interpretable CNNs is a modification of Convolutional Neural Networks where the algorithm automatically forces each filter to represent a distinct part of an object in an image. For example, when training on images of a cat, a standard CNN may have a layer that includes different parts of a cat, whereas the Interpretable CNN has a layer that identifies just a cat’s head.

If your goal is to be able to better understand and explain an existing model, techniques like SHAP and LIME are good options. However, as the demands for more explainable AI continue to increase, even more models will be built in the coming years that have interpretability baked into the algorithm itself, Poduska predicts.

Poduska has a preview of some of these techniques here. These new algorithms will make it easier for all machine learning practitioners to produce explainable models that will hopefully make businesses, customers, and governments more comfortable with the ever-increasing reach of AI.

The post Peeking Inside the Black Box: Techniques for Making AI Models More Easily Interpretable appeared first on Artificial Intelligence.

A real-time deep-learning model is proposed to classify the volume of cuttings from a shale shaker on an offshore drilling rig by analyzing the real-time monitoring video stream. As opposed to the traditional, time-consuming video-analytics method, the proposed model can implement a real-time classification and achieve remarkable accuracy. The approach is composed of three modules. Compared with results manually labeled by engineers, the model can achieve highly accurate results in real time without dropping frames.

Introduction

A complete work flow already exists to guide the maintenance and cleaning of the borehole for many oil and gas companies. A well-formulated work flow helps support well integrity and reduce drilling risks and costs. One traditional method needs human observation of cuttings at the shale shaker and a hydraulic and torque-and-drag model; the operation includes a number of cleanup cycles. This continuous manual monitoring of the cuttings volume at the shale shaker becomes the bottleneck of the traditional work flow and is unable to provide a consistent evaluation of the hole-cleaning condition because the human labor cannot be available consistently, and the torque-and-drag operation is discrete, containing a break between two cycles.

Most of the previous work used image-analysis techniques to perform quantitative analyses on the cuttings volume. The traditional image-processing approach requires significant work on feature engineering. Because the raw data are usually noisy with missing components, preprocessing and augmenting the data play an important role in making the learning model more efficient and productive. The deep-learning framework, on the other hand, automatically discovers the representations needed for feature detection or classification from raw data. It can help overcome the difficulties in setting up and monitoring devices in a harsh environment, and the data-acquisition requirement for a cuttings-volume-monitoring system at the offshore rig might be relaxed.

The objective of this study is to verify the feasibility of building a real-time, automatic cuttings-volume-monitoring system on a remote site with a limited data-transmission bandwidth. The minimum data-acquisition hardware requirement includes the following:

• Single uncalibrated charged-coupled-device camera
• Inconsistent lighting sources
• Low-bit-rate transmission
• Image-processing unit without graphics-processing-unit support (e.g., a laptop)

A deep neural network (DNN) is adopted to perform the image processing and classification on cuttings volumes from a shale shaker at a remote rig site. Specifically, the convolutional neural networks are implemented as feature extractors and classifiers in the described model. The main contributions of this study can be summarized as follows:

• A deep-learning framework that can classify the volume of cuttings in real time
• A real-time video analysis system that requires minimum hardware setup efforts, capable of processing low-resolution images
• An object-detection work flow to detect automatically the region covered by cuttings
• A multithread video encoder/decoder implemented to improve real-time video-streaming processing

Overview of the Real-Time Cuttings-Volume Monitoring System

The work flow mainly consists of the following child processes: real-time video processing (decoding and encoding), region of interest (ROI) proposal, and the data preprocessing and deep-learning classification. During the drilling process, cuttings with mud are transported through the vibrating shale shaker. An intelligent video-processing engine has been developed by the authors to analyze videos captured when the cuttings are transported to the shaker. The analysis results will be transported and presented on a monitor in the office in real time, which is convenient for the drilling engineer to obtain the information of the cuttings volume promptly. The continuous and real-time inference (classification) results can be used as the histogram for further analysis.

The real-time video-processing module is designed for adapting to the dynamic drilling environment. Monitoring the cuttings volume at the shale shaker in real time is an important approach to overall drilling-risk management.

Methodologies

Video Frame Extraction. A two-threads mechanism is used for reading and writing the source stream in real time. The decoding process should be conducted in an adaptive manner because the server is pushing the video stream continuously. If the decoding process fails to catch up with the speed of the video stream, a chance exists that the synchronization and drop frames could be lost. To overcome this obstacle, a fast thread-safe circular buffer is implemented.

ROI Proposal. To guarantee steady inference results, the users (engineers or developers) need to provide the ROI to indicate the area in which cuttings are flowing on the shaker. The described learning model will pay attention to this ROI and obtain input data with much less variety. The camera will not change its position or angle after the ROI is settled. The ROI will filter out many noises interfering with the classifier. A manual or an automatic approach can be used to facilitate ROI selection. Before the decoding of the video stream begins, the interactive graphical user interface (GUI) will present one frame to the user indicating the position of the shaker. The user can highlight the ROI simply by selecting four corner points from the first frame demonstrated by the GUI.

However, manual region selection requires repeated labor. For a certain shaker, the camera angle might be changed slightly during the drilling operation purposely or accidentally by the workers. For different shakers, the preset camera angle might be different.

To automate this procedure, a faster region-based convolutional-neural-network (faster R-CNN) ROI detection method can detect the region that contains the cuttings flow. The raw video frame is used as input and is labeled manually with the ROI by using a bounding box. Every raw frame is fed into a feature extractor, which produces a feature map. The feature map is fed into a much smaller convolutional neural network that takes the feature map as inputs and outputs in the region proposals. Those proposals are fed into a classifier that classifies those proposals to the background class or ROI class. If a region proposal is classified into the ROI class, its coordinate, width, and height will be adjusted further by a region regressor. Backpropagation is used to train the model.

The authors used 50 video frames for training and four images for testing. When considering the results of the training of the cuttings-area detection, the loss of classification decreases and converges after approximately 1,800 training steps. The loss of localization undergoes growth at the beginning of the training but gradually decreases and converges at approximately 2,000 training steps.

Fig. 1 illustrates the results of ROI detection. The bounding boxes contain the predicted region that covers the flow of cuttings. The machine predicts the correct region with high confidence. The success of implementing ROI detection brings the following benefits to the project:

• Automation of the attention mechanism
• Adaptation to different camera angles and distances

Randomized Subsampling Inside ROI. An ROI is selected either manually by the user at the beginning of the video stream or automatically by the cuttings-region detector on the basis of the faster R-CNN framework. However, the vibration or the wind might nudge the camera’s position and angle, which will compromise the classification performance if the system is trained without proper motion compensations. In this study, a randomized subsampling strategy is proposed by using a stack of small image patches to overcome this problem. Image patches are densely sampled from the ROI. Instead of using the entire ROI as the input to the DNN, a stack of image patches is fed to the DNN.

Principal-Component Analysis (PCA) Whitening Transformation. The PCA whitening transformation is applied to video frames immediately before they are fed into the DNN. The goal is to make the input less redundant. The PCA whitening transformation removes the underlying correlations among adjacent frames and potentially improves the convergence of the model.

Experiment and Performance Evaluation

To evaluate performance, the proposed method was tested on a live-­video stream and the real-time classification results are compared with the manual annotation. On the basis of criteria used by rig engineers monitoring the return cuttings flow in real time, the cuttings volume was classified into four discrete levels: extra heavy, heavy, light, and none. Each video was labeled by four experts (the ground-truth labeling represents consensus among the experts). The testing results show that the system can handle the live-stream video without dropping frames. The proposed DNN successfully classifies all classes. The result shows that the proposed model achieves a significant performance boost compared with the performance of traditional networks.

The post Deep-Learning Techniques Classify Cuttings Volume of Shale Shakers appeared first on Artificial Intelligence.

In the ever-increasing data science landscape, learning and retaining concepts have become difficult as one fails to dig deep and assimilate various data science approaches to its fullest. This is because data scientists do not evaluate their understanding of different concepts. However, there are numerous ways one can put their knowledge to the test, and one such method is teaching. To assimilate several best practices in data science, every week, we interview data science leaders who are an inspiration to aspirants and help learners make effective career decisions.

In this edition, Analytics India Magazine got in touch with Anand S, CEO at Gramener for our weekly column My Journey In Data Science. Anand has over 24 years of experience at some of the most prominent companies, such as IBM Global Services, The Boston Consulting Group, and Infosys Consulting, at various roles — ranging from consultant to chief data scientist and CEO.

The Onset

In 1996, Anand completed his B.Tech in chemical engineering from IIT Madras. In the same year, he joined IBM Global Services as a programmer and worked there till 1999. Following this, he completed his MBA from IIM Bangalore in 2001 and joined The Boston Consulting Group as a consultant. However, Anand said it was only during his MBA days that he was introduced to analytics and visualisation while doing his second-year project in 2000 on behavioural finance using bonds data, which was 700 MB in size.

“In this project, both my statistics and programming skills came in handy. I learned statistics in college and had started programming when I was in seventh grade,” says Anand. On gaining insights into the historical data, Anand was fascinated by how he could analyse human behaviour with it. He figured out that younger analysts were less accurate than older ones. But another staggering insight was that the former were less biased than the latter. Besides, he also revealed that financial analysts used to manipulate the rating by lowering the estimates of a company, and when the financial results were declared by companies, it used to be higher than the expected value, resulting in the surge of the stock price. 

Analytics In Anand’s Early Professional Career

Since Anand joined a consulting firm post-MBA, his work involved the use of analysis and visualisation techniques. He used to explore the simulation of employees, staffing level, cost optimisation, among others. However, he described a project that he was a part of in 2004, which required them to do text analysis of bank statements.

“We had to analyse every single customer’s data for the nationalised bank. Getting the data was tough and doing text analysis was even tougher. But, we were able to identify customers who were also banking with the competitors,” explains Anand.

Consulting can be done either by having a strong understanding of the domain or a strong understanding of the data. Anand, in spite of a lower domain understanding, was able to collaborate with the team in the decision-making process with his analytical skills. After four years at The Boston Consulting Group, Anand joined Infosys in 2005, where he worked for more than six years. However, it was completely technical consulting and had a little to do with data. But, he continued to learn analytics and visualisation as a hobby. 

It was around 2009 when he felt that he should fully commit to data analytics. Anand was passionate about the potential of data even before the word ‘data science’ was coined, which happened in 2011.

Strategy To Make A Data Science Career

Anand was always ahead of the curve when it came to learning and getting into data science. However, he knew that no company would hire him for analytics and allow him to follow his passion the way he wanted since data science was not a thing in 2009. Consequently, his strategy was to start a company and get it onto the course of data science.

“It was an unusual strategy, but even today, it is a viable option for aspirants,” believes Anand. “And not just a viable strategy, but also something aspirants should think pretty hard about,” he adds.

Ardent about starting a company, Anand along with his former IBM colleagues, laid the cornerstone of Gramener in 2011. However, the company’s core was not data science; it was focused on energy management and rural BPO. It took Anand another year to convince his co-founders that data is the next big thing. And by 2011, Gramener became a data science company. 

Staying Abreast Of The Data Science Landscape

Anand, in his college, used to teach statistics to other students. Teaching had helped him not only when he was a student but also as a professional. “Teaching others allows you to learn more. When I used to teach, students used to ask difficult questions for which I had to do research and find answers, thereby helping me gain more knowledge,” says Anand. “Even in my professional career, interacting with clients is vital as they demand various results, which, in turn, motivates me to find answers through data science techniques. All of my learning always comes from doing,” he adds.

Since Anand was one of the pioneers in the data science landscape, he, along with his team, devised a data visualisation course in collaboration with an IIIT institute in 2013. However, Anand never had the opportunity to learn from online courses when he started. Whenever he got stuck or needed a solution, he used to read research papers and at times, create his own algorithms like group names and more.

Work Experience At Gramener 

Talking about one of the successful projects, Anand said how they associated with other organisations to analyse the 2014 Lok Sabha election. The initiative was more focused towards pre, live, and post-analysis of the election. And his worst experience revolves around people asking inane questions about data science. For instance, one of the banks from the Middle East reached out to him for carrying out big data analysis and forecast the sales with just three data points.

Besides, while hiring data scientists, Anand evaluates applicants foundation skills and prefers aspirants who have done certifications but do not look for PhDs candidates. He said, PhD applicants have a research mindset; however, we need an application mindset because we rely on quick results. PhDs are focused on depth to improve the technique approach, take multiple routes to get there. For time-bound problems, they are not the ideal candidates. We do not seek people who can create algorithms, want someone who can use these algorithms to quickly solve the problems.

Advice To Aspirants

Anand had a couple of advice for aspirants and professionals who struggle to find success in the competitive landscape. He suggests one should primarily focus on building a foundation and creating a portfolio using whatever data they have. However, just finding insights does not work; one must go a step further and say what should be done to improve the outcome. 

In addition, he stressed the importance of teaching others. “If you can communicate the learnings to others that means you properly understand the approach or techniques, which brings confidence,” concludes Anand.

The post TEACHING IS THE BEST WAY TO LEARN DATA SCIENCE, SAYS THIS DATA SCIENTIST appeared first on Artificial Intelligence.

A new collaboration has demonstrated fully-autonomous Scanning Probe Microscopy (SPM) operation, applying artificial intelligence (AI) and deep learning to remove the need for constant human supervision.

According to the researchers, the new system, dubbed DeepSPM, bridges the gap between nanoscience, automation and AI, firmly establishing the use of machine learning for experimental scientific research.

“Optimising SPM data acquisition can be very tedious. This optimisation process is usually performed by the human experimentalist and is rarely reported,” said Future Low-Energy Electronics Technologies (FLEET) Chief Investigator Dr Agustin Schiffrin, at Monash University, Australia. “Our new AI-driven system can operate and acquire optimal SPM data autonomously, for multiple straight days and without any human supervision.”

The advance brings advanced SPM methodologies such as atomically-precise nanofabrication and high-throughput data acquisition closer to a fully automated turnkey application, say the researchers. 

The new deep learning approach can also be generalised to other SPM techniques. The researchers have made the entire framework publicly available online as open source, creating an important resource for the nanoscience research community.

“Crucial to the success of DeepSPM is the use of a self-learning agent, as the correct control inputs are not known beforehand,” said Dr Cornelius Krull, project co-leader. “Learning from experience, our agent adapts to changing experimental conditions and finds a strategy to maintain the system stable.” 

The AI-driven system begins with an algorithmic search of the best sample regions and proceeds with autonomous data acquisition. It then uses a convolutional neural network to assess the quality of the data. If the quality of the data is poor, DeepSPM uses a reinforcement learning agent to improve the condition of the probe.

The system can run for several days, acquiring and processing data continuously, while managing SPM parameters in response to varying experimental conditions, without any supervision, highlight the researchers. 

The post Autonomous Scanning Probe Microscopy technique developed using AI appeared first on Artificial Intelligence.

Bottom Line: Phishing is the leading cause of all breaches, succeeding because impersonation, redirection, and social engineering methods are always improving. And, phishing is only one way emails are used in fraud. Businesses need to understand if an email address can be trusted before moving forward with a transaction.

Microsoft thwarts billions of phishing attempts a year on Office365 alone by relying on heuristics, detonation, and machine learning, strengthened by Microsoft Threat Protection Services. In 2018 Microsoft blocked 5 billion phish emails in Office 365 and detonated 11 billion unique items by ATP sandboxing. Microsoft is succeeding with its cybersecurity partners in defeating phishing attacks. Phishers are going to extraordinary lengths to discover new techniques to evade detection and successfully carry out phishing attempts. By analyzing Office 365 ATP signals, Microsoft sees phishers attempt to abuse many legitimate cloud services, including Amazon, Google, Microsoft Office365, Microsoft Azure, and others. Microsoft is creating processes that identify and destroy phishing attempts without impacting legitimate applications’ performance.

Phishers’ Favorite Trojan Horse Is Office365 Followed By Cybersecurity Companies  

Phishers are hiding malicious links, scripts and, in some cases, mutated software code behind legitimate Microsoft files and code to evade detection. Using legitimate code and links as a Trojan Horse to successfully launch a phishing campaign became very popular in 2019 and continues today. Cybercriminals and state-sponsored hackers have been mutating legitimate code and applications for years attempting to exfiltrate priceless data from enterprises and governments globally. Office365 is the phisher’s Trojan Horse of choice, closely followed dozens of cybersecurity companies that have seen hackers attempt to impersonate their products. Cybersecurity companies targeted include Citrix, Comodo, Imperva, Kaspersky, LastPass, Microsoft, BitDefender, CyberRoam, and others.

Using Trojan Horses To Hijack Search Results

In 2019 Microsoft discovered a sophisticated phishing attack that combined impersonation, redirection, and social engineering methods. The phishing attack relied on using links to Google search results as a Trojan Horse to deliver URLs that were poisoned so that they pointed to an attacker-controlled page, which eventually redirected to the phishing page. Microsoft discovered that a traffic generator ensured that the redirector page was the top result for specific keywords. The following graphic explains how the phishing attack was used to poison search results:

Using this workflow, phishers attempted to send phishing emails that relied on legitimate URLs as their Trojan Horses from legitimate domains to take advantage of the recipient’s trust. Knowing which e-mails to trust or not is becoming foundational to stopping fraud and phishing attacks.

How Kount Is Battling Sophisticated Attacks 

Meanwhile, email addresses can be a valuable source of information for businesses looking to prevent digital fraud. Misplaced trust can lead to chargebacks, manual reviews, and other undesirable outcomes. But, Kount’s Real-Time Identity Trust Network calculates Identity Trust Levels in milliseconds, reducing friction, blocking fraud, and delivering improved user experiences. Kount discovered that e-mail age is one of the most reliable identity trust signals there are for identifying and stopping automated fraudulent activity.

Based on their research and product development, Kount announced Email First Seen capabilities as part of its AI-powered Identity Trust Global Network. Email First Seen applies throughout the customer journey, from payments to account login to account creation. The Identity Trust Global Network consists of fraud and trust signals from over half a billion email addresses. It also spans 32 billion annual interactions and 17.5 billion devices across 75 business sectors and 50-plus payment providers and card networks. The network is linked by Kount’s next-generation artificial intelligence (AI) and works to establish real-time trust for each identity behind a payment transaction, log in or account creation

Email Age Is Proving To Be A Reliable Indicator Of Trust

A favorite tactic of cybercriminals is to create as many new e-mail aliases as they need to deceive online businesses and defraud them of merchandise and payments. Kount is finding that when businesses can identify the age of an e-mail address, they can more accurately determine identity trust. Kount’s expertise is in fraud prevention effectiveness, relying on a combination of fraud and risk signals to generate a complete picture of authentication details. The following graphic illustrates what a Kount customer using Email First Seen will see in every e-mail they receive.

Kount’s Identity Trust Global Network relies on AI-based algorythms that can analyze all available identifiers or data points to establish real-time links between identity elements, and return identity trust decisions in real-time. Kount’s unique approach to using AI to improve customer experiences by reducing friction while blocking fraud reflects the future of fraud detection. In addition, Kount’s AI can discern if additional authentication is needed to verify the identity behind the transaction and relies on half a billion email addresses that are integral to AI-based analysis and risk scoring algorithms. Kount is making Email First Seen available to all existing customers for no charge. It’s been designed to be native on the Kount platform, allowing the information to be accessible in real-time to inform fraud and trust decisions.

Conclusion

In 2020 phishing attempts will increasingly rely on legitimate code, links, and executables as Trojan Horses to evade detection and launch phishing attacks at specific targets. Microsoft’s research and continued monitoring of phishing attempts uncovered architecturally sophisticated approaches to misdirecting victims through impersonation and social engineering.

The post How To Know If An E-Mail Is Trustworthy appeared first on Artificial Intelligence.