Automated predictive analysis is making way for the greatest transformations in the industries!

Automated predictive analysis, or predictive analytics, uses historical data to predict future events. Throughout history, humans are obsessed with predicting the future. The fear of the unknown has led several scientific researchers and professors to develop technologies that can determine the future so that necessary steps can be taken to avoid drastic losses.

Predictive analytics has received a lot of attention in recent years due to its advances in supporting various technologies, particularly in the area of big data and artificial intelligence.

With the increased competition, businesses seek power over their competitors in bringing products and services to crowded markets. Data-driven predictive models can bring companies solutions to long-standing problems in terms of business operations. This technology provides a trove of information from which analytics tools and applications draw insights and predict the upcoming opportunities, suitable investments, and dangers in the market.

Businesses use tools like Hadoop and Spark to extract information from big data. These data sources might consist of transactional databases, equipment log files, images, videos, audios, sensors, and other types of data.

With all this data, tools are necessary to extract insights and trends. Predictive analytics finds patterns in data to build models that predict future outcomes. Other varieties of machine learning techniques are also available, including linear and nonlinear regression, neural networks, support vector machines, decision trees, and other algorithms.

For decades, automated predictive analysis has been used by meteorologists to predict weather and climate forecasts. With time, this concept has been used to study consumer behavior, forecast supply and demand in economic statistics, and related purposes.

• How Can Artificial Intelligence Drive Predictive Analytics To New Heights
• How Predictive Analytics Will Impact Human Resources
• What Is Predictive Analytics And Can It Help You Achieve Business Objectives

Automated predictive technology: Today and beyond 

Data is the core of predictive analytics. Earlier, when there were no computers, businesses used other creative ways to understand what the customers want and predict market conditions. These ways did not involve technological tools or applications.

Currently, one of the most vital industrial applications of predictive models includes energy load forecasting to predict energy demand in the future. Energy producers, grid operators, and traders need accurate predictions of energy load to make decisions for managing tasks in electric grids. Grid operators use data to draw actionable insights.

Artificial intelligence and predictive technology, have revolutionized the way advertisers and marketers work. Targeted advertising uses data like previously purchased products, location, and age to serve the target audience. Today, consumer profiles are much more advanced, and enterprises can gather information from various sources.

Predictive analytics is also used to measure vehicle and pedestrian traffic to coordinate traffic lights, public transportation, and even pedestrian crosswalks to facilitate convenience and efficiency in community design. This also boosts the safety of the public and allocates emergency services more efficiently by predicting the number of officers needed on a task and reassigns posts accordingly.

Automated predictive technology, has played a crucial role in facilitating better medical resources. This technology helps improve the patients’ health outcomes. Rather than completely relying on the patient’s medical history, predictive systems can generate data from a broad spectrum of symptoms, data of other patients, and the treatments used to cure the disease.

AI and machine learning have provided us with various ways through which we can predict the future. With the growing technological evolution in automation and data analysis, our lives will be changed forever and for the better.

The post HOW HAS AUTOMATED PREDICTIVE ANALYSIS DEVELOPED OVER THE YEARS appeared first on Artificial Intelligence.

Overfitting is a common modeling error all enterprises who deploy machine and deep learning will encounter. When machine learning models allow noise, random data or natural fluctuations in data to dictate target functions, the model suffers in its ability to generalize new data beyond its training set. 

Data scientists must build their models carefully, train them effectively, improve their machine learning literacy and instill incentive systems to combat overfitting.

Defining an overfitted model

Training machine learning and deep learning models is rife with potential failure — a major issue being overfitting. Generally, overfitting is when a model has trained so accurately on a specific dataset that it has only become useful at finding data points within that training set and struggles to adapt to a new set.

In overfitting, the model has memorized what patterns to look for in the training set, rather than learned what to look for in general data. To a data scientist, the model appears to be trained effectively, but when brought against other datasets it is unable to repeat this success.

“We want the model to generalize well to unseen data,” Benyamin Ghojogh, a graduate research assistant of machine learning and PhD candidate at the University of Waterloo, said.

For Ghojogh, avoiding overfitting requires a delicate balance of giving the right amount of details for the model to look for and train on, without giving too little information that the model is underfit.

“Something in the middle is good,” Ghojogh said. “And that’s a perfect fit, which can generalize to the new data and seen data.”

But this is a difficult point to get your models to and one that becomes even more of an issue as your models become more complex.

“If you make the model complex but you don’t have sufficient data to train it, the model will definitely get overfit and that’s why neural networks are prone to it,” Ghojogh said.

Ghojogh pointed out that all models must tackle the problem with overfitting, especially if the organization trains the model too much.

Rooting out overfitting in enterprise models

While getting ahead of the overfitting problem is one step in avoiding this common issue, enterprise data science teams also need to identify and avoid models that have become overfitted. But just as overfitting becomes more common as models grow in complexity, it also becomes more difficult to identify. Organizations who deploy neural networks and deep learning especially struggle with this step.

Forrester analyst Kjell Carlsson said that it has never been easier for overfitting to slip through the cracks. With the layers involved in deep learning, it’s difficult to point out when and where a model has become over-trained.

With high demand on data and analytics in the enterprise, there’s a time crunch on data science teams to deploy effective models. But as important as timeliness is, organizations have to implement processes and a culture that prioritizes machine learning effectiveness.

“You have to have very disciplined processes in your data science organization to go in and make sure that [you] are adhering to best practices,” Carlsson said.

When it comes to model deployment, organizations need to work with their data science team. If a model isn’t working or if the results aren’t as promising as previously thought, Carlsson said, this is where communication becomes important. The relationship between data scientists and their organization can help determine if the model is functioning poorly due to overfitting, or the end-user application, over-hyped projections or another non-data-related cause.

The cost of overfitting and how to avoid it

Getting ahead of overfitting is crucial for model deployment. An overfitted model, when deployed against real-world data, will provide either no useable gains or false insights. Both cost an enterprise significant amounts of time, resources and buy-in from executives; and this cost grows even greater if the overfitted models’ false insights go unnoticed and trusted.

As Gartner analyst Alex Linden put it, an overfitted model can render the insights from software or AI analysis useless. In the case of predictive maintenance, the system is not able to predict where or when the machine is going to fail. In predicting sales, the model will either be unable to predict whether somebody is going to react favorably to a selling proposition or will provide false positives.

Linden’s prescription for this is testing. Running the model through numerous trials with different data sets is the best way to prove its capabilities to generalize rather than memorize. For Linden it is the number one way of avoiding overfitting and there are machine learning technologies that can ease this process of trial and error.

“Mostly these days you can actually avoid overfitting by brute force, and the brute force capabilities of automated machine learning,” Linden said.

Instead of having your data science team members run these trials manually and deploying numerous models to find the correct one to use, automated machine learning can run hundreds of data sets in a short amount of time.

Another way to avoid overfitting models is building in a forgetting function, especially with deep neural networks. Having your data science teams encode a forget function allows for the model to organize, create data rules and create a space to ignore noise.

“They try to only memorize the generalities of things and try to forget the exact nature of the things,” Linden said.

Another way to prevent overfitting in machine and deep learning models is ensuring that you have a holdout set of data to test your model on. If your model can generalize well enough then it should do well against this test data.

Building a core knowledge of machine learning and AI

Training a model often and with variety coupled with formatting forgetting functions and separate test data sets are all effective measures against overfitting. On top of this, organizations need to ensure there is a basic level of competency about common machine learning model failures throughout the company.

This entails investing and improving your organizational machine learning and data literacy levels. Though data scientists and members of the data science team are the first line of defense against overfitting and model problems, organizations require even more oversight to ensure successful application of machine learning.

“You’ll need the line-of-business stakeholder to be aware of overfitting, aware of these pitfalls and be on the lookout for them,” Carlsson said.

Having another part of the enterprise team understand common problems with model applications and what to look for adds another layer of safety. While there are numerous courses about machine learning and deep learning, finding the right level of basic AI literacy among employees can be a challenge. Most of these are tailored towards data scientists and provide information that a business owner or team member won’t realistically need to know.

Targeting the correct training to the right business professionals can decrease the chances of overfitting and prevent poor applications, Carlsson said.

The post How to avoid overfitting in machine learning models appeared first on Artificial Intelligence.

UNIVERSITY PARK, Pa. — Just as people need to protect their sensitive data, such as social security numbers, manufacturing companies need to protect their sensitive corporate data. There are currently fewer protections for proprietary manufacturing information, making it a ripe environment for corporate data theft of such things as design models.

A particular approach known as differential privacy may be able to better preserve a manufacturer’s business, sensitive design details and overall company reputation, a team of Penn State researchers and graduate students report in the Journal of Smart and Sustainable Manufacturing of the American Society for Testing and Materials.

“Cyberattacks are increasingly seen in manufacturing,” said Hui Yang, professor of industrial engineering. “This brings unexpected disruptions to routine operations and causes the loss of billions of dollars. For example, adversaries often attempt to infer samples included in the training dataset used to create an analytical model or use the released model to infer sensitivity of a target when other background information about this target is available. As manufacturing systems are the backbone of a nation’s critical infrastructure for economic growth, there is an urgent need to protect privacy information of manufacturing enterprises and minimize the risk of model inversion attacks.”

Companies often data mine large datasets to understand patterns that could increase profits, lower costs, reduce risks and more. Data mining can inadvertently expose private data, posing significant security threats to manufacturers because confidential data such as customers’ identities, production specifications and confidential business information may be compromised.

Differential privacy is an emerging approach to safeguard data from any attempt that may reveal any sensitive data within a system. Differential privacy can fix this problem by creating a scheme that forces the system to create “noise” around the data that needs most protection and by optimizing the privacy parameters for these different kinds of data.

“The idea of preserving privacy was already present, but it gets much more attention now,” said Soundar Kumara, the Allen E. Pearce and Allen M. Pearce Professor of Industrial Engineering. “Differential privacy methods are able to put measurements on how much privacy is needed in various scenarios, which is greatly useful for companies. Some information simply isn’t as sensitive, like a pet’s name versus credit card information. There are applications aimed at differential privacy for smart manufacturing and data mining, and our proposed methodology shows great potential to be applicable for data-enabled, smart and sustainable manufacturing.”

The researchers carefully calibrated a model with noise for specific, more sensitive kinds of raw data. The curated, regulated noise contains numerical values that sit among the real information to create distractions, or randomness, within the system to blur what an attacker may see.

The group used test data to evaluate and validate the proposed privacy-preserving data mining framework. They specifically focused on power consumption modeling in computer numerical control (CNC) turning processes.

According to the team, the CNC turning is a precise and intricate manufacturing process in which a rotating workpiece is held in place while a cutter shapes the material. This kind of information can be critical for a manufacturing company, because it may be for their specific product in a competitive market.

“A simple example is a hospital with 500 patients where medical treatments are guided by data mining models trained with their genotype and demographic background,” said Qianyu Hu, an industrial engineering doctoral candidate. “If someone outside of the system wants to know specific attributes on patients, for example, their genetic markers, they will attack the model. With normal data, unprotected by noise, an attacker with some background information is able to gain knowledge of the genomic attributes of patients. This knowledge can be adversely used against them in various ways. In this example, adding noise to the data mining process, based on our model, can lower the risk of privacy leakage.”

The team noted that in their future research, they plan to continue testing the proposed data mining framework to a network of collaborative manufacturers.

Ruimin Chen, industrial engineering doctoral candidate, also contributed to this work.

The post New method enables automated protections for sensitive data appeared first on Artificial Intelligence.

Hypatos, a Germany and Poland based automation startup, raised around $11.8 million in a seed funding round. Many companies like Blackfin Tech, UVC Partners, Grazia Equity, and Plug Ventures participated in the round. The company will use these funds to widen its scope of documents for automated document processing. Hypatos will also focus on further expansion of the company across Europe, North America, and Asia.

Hypatos started due to the need for an AI solution for an accounting startup called Smacc to apply deep learning algorithms to automate a more comprehensive range of back-office operations. The primary focus was on the financial and insurance sectors with heavy financial document processing needs. 

Manual document processing is costly and time-consuming. To solve this problem, Hypatos uses language processing AI and computer vision technology to automate complex document processing tasks such as invoices, travel and expense management, tax compliance checks, loan application validation, and insurance claims in different businesses. It claims to cut down 90 percent of the cost used in traditional manual document processing. Hypatos uses its deep learning algorithms for in-depth document processing and document classification, information monitoring, auto accounting, content validation, etc. It uses Cognitive Process automation (CPA), a combination of deep learning technology and traditional rule-based software robots for document processing giving it an edge from other automated document processing solutions. It can process many document types and also offers custom models for different automation needs. It also provides APIs for software providers using its machine learning technology for their applications.

In the COVID-19 pandemic, Hypatos claims to have seen an uplift as it is providing its services to more than a dozen Fortune 500 companies. This uplift’s primary reason is the lockdown, less budget, and less workforce in many companies. Many companies are now looking for automated solutions because they’re cost-effective and time-saving.

The post Hypatos, deep learning based document processing startup, raises $11.8M appeared first on Artificial Intelligence.

Data scientists are at present particularly popular. In any case, there is simply a question regarding whether they can automate themselves out of their positions. Can artificial intelligence replace data scientists? Assuming this is the case, how much can their tasks be automated? Gartner as of late stated that 40% of data science tasks will be automated by 2020. So what sort of aptitudes can be effectively dealt with via automation? This theory adds fuel to the progressing ‘Man versus Machine’ banter.

Data scientists are costly to recruit and there is a lack of this expertise in the business as it’s a generally new field. Numerous organizations try to search for alternative arrangements. A few AI algorithms have now been created, which can analyse data and give experiences like a data scientist. The algorithm needs to give the information yield and make exact forecasts, which should be possible by utilizing Natural Language Processing (NLP).

Numerous people who are all in with the possibility that data scientists will be automated and jobless soon underestimate the complexities of the data preparation process. To automate anything, you have to take care of smart information” to the machine. By smart, it implies that this data should be by one way or another structured and gathered with a plan in mind in the first place.

You have to execute a predictive solution for commercial loan evaluation. As a data scientist, you should investigate the intricate details of the business. Also, at exactly that point you will concoct a type of plan on the best way to gather and analyze data that will be utilized to implement the solution.

While you may contend that banks will give engineers all the information they need, this can’t be farther from reality. In reality, it is data scientists who are liable for scanning all the data for the model. They have to make sense of significant factors, find patterns, and analyze key indicators to decide a decent versus the awful business loan.

Indeed, even the most brilliant machine learning frameworks work with what you tell it to work with. In like manner, it will improve information utilizing a representative training data set that you have prepared.

Automation in data science will crush some physical work out of the work process rather than totally supplanting the data scientists. Low-level capacities can be productively dealt with by AI systems. There are numerous technologies to do this.

Automation has its own set of limitations, nonetheless. It can just go up until this point. Artificial intelligence can automate data engineering and machine learning processes yet AI can’t automate itself. Data wrangling comprises physically changing over raw data to an effectively consumable structure.

The cycle despite everything requires human judgment to transform raw data into insights that bode well for a company, and consider all of a company’s complexities. Indeed, even unsupervised learning isn’t totally automated. Data scientists despite everything prepare sets, clean them, indicate which algorithms to utilize, and decipher the insights. Data visualisation, more often than not, needs a human as the discoveries to be introduced to laymen must be exceptionally customised, contingent upon the technical knowledge of the audience. A machine can’t in any way, trained for that.

Low-level visualisations can be automated, yet human insight would be needed to decipher and clarify the data. It will likewise be expected to compose AI algorithms that can deal with mundane visualisation tasks. Besides, intangibles like human curiosity, intuition or the desire to create/validate experiments can’t be reproduced by AI. This part of data science presumably won’t be ever dealt with by AI soon as the technology hasn’t developed to that level.

There is an unmistakable point of reference in history to propose data science won’t be automated away. There is another field where exceptionally trained people are making code to cause computers to perform astonishing accomplishments. These people are paid a noteworthy premium over other people who are not trained in this field and there are education programs specializing in training this skill. The subsequent financial strain to automate this field is similarly, if not more, intense. Data science field is software engineering.

Moreover, as software engineering has gotten simpler, the demand for software engineers has only increased. This paradox, that automation increases efficiency, driving down costs and at last driving up demand isn’t new. We’ve seen it over and over in fields running from software engineering to financial analysis to accounting. Data science is no exemption and automation will probably drive up demand for this range of abilities, not down.

Automation will act as a supplementary tool that will boost data science tasks and make them more efficient. Bots can take care of lower-level tasks, whereas data scientists can take care of problem-solving tasks. This combination of human problem-solving and automation will, moreover empower data scientists, rather than threatening their jobs. There will be more technological advancements coming up in the future. However, it is important to understand that data scientists possess a very important skill – intuition, which is very difficult to be emulated by advanced artificial intelligence.

The post WILL AUTOMATION PUT AN END TO DATA SCIENCE JOBS? appeared first on Artificial Intelligence.

Companies operating in the Consumer Packaged Goods (CPG) segment work under a unique ecosystem. They have to strike a balance between maintaining top-line revenue growth and managing sustainable profit margins. The reason why the task becomes difficult is because they have to do so while managing a dynamic set of operations. Moreover, a shift in consumer preferences, advances in data & analytics, channel shifts, and pandemic-led disruptions have created new challenges for retail and CPG companies.

However, in the current market scenario, they also have the opportunity to upgrade RGM, thereby creating equilibrium between growth and efficiency. They can increase the use of complex and action-oriented analytics across the product catalogue, optimize value capture approaches, use automated technology, and partner with retailers for shared value creation.

Why upgrading RGM is important for CPG companies?

Competitiveness has considerably intensified within the CPG industry. Today, there are limited avenues in terms of customer touchpoints vis-a-vis the year-ago period. The demand pockets have also changed dramatically and so has the New Normal’s supply chain management. CPG businesses today need to unlock as much efficiency from as many avenues as possible.

For those who cannot determine whether they should upgrade their RGM or not, there are a few questions that will help them overcome this quandary. Is your inflation getting outpaced by net revenue realization? Are your capabilities increasing faster than your competitors and retailers? Is the RGM beyond silo and addresses online business as well? Has it been integrated to your overall business strategy and at scale? Are you sufficiently leveraging data and analytics to grow revenue?

If the answer to a majority of these questions is ‘Yes’, then you don’t need to upgrade your RGM. If it is ‘No’, then it is time to think otherwise. Depending on the answers, companies must commit one or more paths towards RGM differentiation today to help their business triumph in the market. Leading data analytics solution providers help you to lay out and execute RGM roadmap for both strategic RGM and tactical implementations seamlessly.

Role of artificial intelligence in revenue growth

Over the last decade, a lot of CPG companies have implemented & tried to bring in new data and technologies for revenue growth management. This has enabled them to gain knowledge on what, how, and why shoppers buy and consume. Organisations are further bringing in MACRO data and other studies like segmentation into the toolkit. While RGM is important to operate effectively in the market, companies no longer have a competitive edge.

For sustainable revenues and growth, CPG companies must adopt AI. Its importance in terms of advanced capabilities in pricing, promotions, assortments, and trade investment will only increase as the competition intensifies within the CPG industry.

Artificial intelligence and ML provide companies with the scalable capability to utilize the power of data and navigate complexity. AI lets CPGs and retailers access customer insights and predict future actions based on the past behaviours. AI uses predictive analysis to help understand the desires, motivations, and actions across physical and digital channels. It allows retailers and suppliers to improve functions such as executing hyper-personalized campaigns and trade promotions efforts.

Artificial intelligence brings in key aspects such as

• The ability to add in a variety of data sources.
• Quick feedback loop. It creates a learning mechanism to update the model/recommendations on the basis of the ever-changing market dynamics (such as consumer preferences)
• Speed to market. For instance, one of the modules within RGM can help identify & recommend price tiers, better trade investments, and so forth. It can also predict future out-of-stock incidences more accurately, thereby helping to optimize supply chains.

Such solutions provide swift and actionable insights that lead to better conversion/engagement rates with customers. It further leverages predictive algorithms for guided decision making, scenario planning, and simulation to drive prepared outcomes.

How artificial intelligence supports RGM:

The most critical function of AI in RGM is that it converts plain data into the famous ‘So What’ or the relevant implication/suggestion. It helps in shifting the output from ‘Insights’ to Recommendations.

These are some of its other advantages:

• Unified Data View: AI helps leaders understand the consumer and drive efficacy with a unified data view. It reveals how actions impact Key Performance Indicators (KPIs) across the business, and not only within each function. The algorithmic recommendations enable one to look beyond interim improvements and suggest actions that achieve end-goals.

AI unifies data scattered across multiple channels and sources (both structured and unstructured). It can detect and classify relevant information on consumers relating to individual household information, scan cart-level data at point-of-sale, social sentiment, purchase behaviour across channels, travel patterns and dwell time in various venues to gain deep insights of the consumer path to purchase. AI technology enables teams to comb through massive data, analyse, and decode customer shopping behaviour on micro parameters. This also helps CPG companies create a 360-degree customer view.

• Granular Predictive Models: Predictive AI models built on unified data are very granular and micro-segmented models. This makes them capable of large-scale analysis with tailored objectives and limitations at all levels. They can learn from history and can also predict probable future outcomes. Such models can also estimate baseline and raise forecasts combining a diverse set of influencing factors. They leverage deep learning to recognize shopping patterns and complex interactions. For instance, switching between brands within a category, or switching between channels, or even between shopping occasions. They can identify interrelated patternsbetween trade and other actions in the market.

AI provides minutest insights to understand each customer by sifting through massive structured and unstructured datasets from the first- and third-party sources. It helps spot micro-segments and emerging demand spaces (eg. new occasions, sub-segments, servicing opportunities, etc.) to build new business models, optimize the product, pricing, promotion, and marketing activities.

• Forestall & Recommend Actions: AI assistants can scan all channels, markets, competitor actions, retailer actions, and self-assess to find opportunities and threats. Next, predictive models can evaluate complex interactions to explore numerous possibilities and suggest the best action to people based on their roles, owing to each market and the account relationship.

AI also helps to redefine businesses by automating manual, repetitive, and high-volume processes. Its learning capabilities allow it to self-optimize over time and reduce the work volume of employees. The technology’s deployment not only boosts employee productivity but also unlocks greater ROI for businesses.

• Growth Hacking Through Quick Test And Learn: AI systems evolve themselves by learning from experiences. Event analysis in RGM takes a futuristic approach towards learning about consumer behaviour. AI models bridge the gap between plan, execution, and results with the process of continuous learning using ‘recommend’, ‘act’, ‘measure’, and ‘learn’ methodology. RGM or related teams can conduct well-designed experiments in choicest markets, analyse the outcome, and roll out smart strategies across the business.
• Ongoing Feedback:It creates a loop mechanism for continuous learning model and recommendation improvements. The feedback loop ensures that the model keeps on updating itself without much human intervention.

It also helps in creating hyper-curated experiences. Because AI analyzes massive unstructured data such as photos, audio, video, etc., this helps in creating most relevant and personalized messaging and offers, and value-added services. This is while basing on consumer preferences in real-time.

AI-led RGM fosters sustainable success

Leveraging AI to thoroughly understand the consumer and reinvent relevance, CPG companies can develop a powerful capability. It can help them retain and expand their user-base, reduce costs, stand-out competitively, and drive new opportunities. Also, AI can improve their ever-evolving standards of performance by optimizing interactions and transactions – paving the way for never-ending growth.

Pricing and trade spend within Revenue Growth Management are some of the most powerful yet complex functions. If done well, they can help organizations win over not just their customers, but the market as well. So, prefer a solution provider that has a proven track record of RGM toolkit deployment and its subsequent scaling.

The post How artificial intelligence fuels revenue growth management appeared first on Artificial Intelligence.

Big Data Exchange (BDX) has officially opened a new data centre in the Paya Lebar area of Singapore, more than three months after acquiring the facility from Telstra.

Branded as SIN1, the new facility houses 1,800 racks with a 7.3MW power capacity, holding certificates such as UpTime Tier 3+ Design, SS564 GreenMark Gold Plus, TVRA, ISO27001 and PCI-DSS.

Going forward, BDX plans to upgrade the facility to increase power usage effectiveness and drive higher efficiency, with the city-state housing the highest megawatt per capita across the world.

“The BDX expansion into Singapore is an important addition to the BDX portfolio,” said Braham Singh, CEO of BDX. “After a successful acquisition and months of preparation, we are thrilled to officially launch the SIN1 facility.”

According to Singh, customers will have the ability to “easily integrate” both physical and virtual infrastructures via the BDX Single Pane into a managed hybrid ecosystem, connecting the new Singapore location through a highly automated cluster of data centres globally via BDX SoftConnect.

“The strategic proximity within the Asia Pacific region offers great geographical and connectivity advantages to meet the increasing internet and cloud exchange needs of our customers,” he added.

BDX operates data centres across Hong Kong, mainland China and Singapore, backed by the provision of hybrid cloud, connectivity and colocation solutions as a carrier-neutral provider.

The acquisition in April came two months after BDX announced the construction of a new data centre in Nanjing, China, which launched in June. Collectively, BDX now operates eight data centres in cities across two continents, including Nanjing, Guangzhou, Hong Kong, Singapore and London.

The post Big Data Exchange opens data centre doors in Singapore, takes over Telstra facility appeared first on Artificial Intelligence.

Automated machine learning (ML) sounds like the stuff of business leaders’ dreams.

Take the question, ‘which customers should we focus our marketing budget on this year?’ ML can now deliver robust answers to these types of business questions even faster than before, through greater use of automation.

Data in at one end, seriously useful business insight out of the other.

That is partly why Forbes predicts businesses will be investing $125bn a year on Artificial Intelligence and Machine Learning by 2025.

But even though numerous vendors boast of “AutoML” capabilities, the reality is that the act of developing ML models is still very much driven by humans and requires an awful lot of manual trial and error, performed by (expensive) experts.

Whilst the human element will never completely disappear, new automation techniques will help to reduce the vast amount of labor intensive work required. Not only will this reduce the overall cost and effort, it should reduce the levels of skill and experience required to build reliable ML models.

By today’s standards, it is certainly an unfortunate fact that manual effort still accounts for 80 percent of the machine learning development process. The most important part of this manual effort is the feature engineering process, where different data elements are combined and enriched to generate the most potent formula for predicting future events.

In the case of working out which customers might churn in the next year, for example, the data may include the size of their last discount. But prediction accuracy would improve if further features were engineered such as the time since the last discount, the average time between discounts and how the discount compares to those offered to other customers.

The challenge here is that nobody knows for sure whether these feature combinations will work until they have been developed, tested and fully assessed together as part of an ML model.

Specialist knowledge has been essential in these endeavors: you can’t produce a good algorithm without a subject matter expert knowing something about which features may be the most significant, or without experienced data scientists with deep knowledge of the ML process.

This need for expensive experts is one of the factors that have limited the application of ML to the organizations with the skills, patience and deep pockets to indulge lengthy developments, and to low-risk use cases with the clear potential for high levels of return on investment. But this is now starting to change.

One area of data science development that offers the potential to transform this endeavor is automated feature engineering: Using a computer to shortcut one of the most manually-intensive aspects of ML development.

The challenge of bringing automation to every stage of the ML workflow is one that my company, Peak Indicators, has been exploring for years. From this work, we created Tallinn ML, a platform providing all of the components required to build and deploy predictive models automatically, significantly reducing the reliance on highly-skilled data scientists.

Tallinn ML includes a unique feature-engineering engine that drastically cuts the time taken to develop new predictive algorithms by generating and testing thousands of different metrics as part of the data engineering, a process of trial and error that can take human months or even years to deliver.

Earlier this year we applied it on Kaggle – Google’s online home for the world’s data scientists and machine-learning experts, a kind of Premier League of ML. Kaggle set an unusual challenge. Can you develop an algorithm to predict which people were most likely to survive the world’s most infamous shipwreck – the Titanic?

Competitors were given a set of features, such as passenger age or gender, and asked to develop the most powerful algorithm to predict who would survive. Among Kaggle’s 1 million users are some of the world’s best-known researchers and data science teams. Peak’s Tallinn ML algorithm reached the top 5 percent for accuracy.

While other world-class competitors developed their models through manual means, our model was produced automatically. It involved no coding and no manual trial and error. It proved that machine learning has now reached a new level of automation.

• How automation can provide a foundation for digital transformation

The business impact of automated ML

So what difference does this make to business? Well, potentially a huge one.

The insights provided by predictive analytics and machine learning have been seen for some time as potentially revolutionary for business. Suddenly firms are far better able to answer crucial questions like:

• What are the impacts of a particular marketing campaign likely to be for specific target customers?
• Which of our employees are likely to leave in the next year?
• Which transactions in an account are most likely to be fraudulent?
• What seems to be causing a particular business problem?
• Harnessing the power of automation

Those questions are just the start. Answering them reliably means resources can be put where they are most needed. Inefficiently-used time and money can be reallocated to more productive tasks. Robust new insights into what is needed next appear magically.

But making that promise a reality is difficult. As Gartner highlighted just last year, “doing predictive analytics is tough. Your team needs to possess the right skills, understand business priorities and deal with data accuracy”.

That meant that any business, according to Gartner’s research, had previously to ask an important question: “What’s the likelihood you’ll sink under the weight of your organization’s data or swim to successful results?”.

Now that question is no longer so pressing. An automated solution makes it far more likely an organization will swim, because it will eliminate a considerable amount of time and effort in ML projects, and significantly reduce the need for very high-level expertise. The chances of an organization sinking – or treading water – in a sea of data become far smaller.

Problems that took months to solve previously can now be addressed in a matter of hours and days, and it has become economically viable to use ML to solve a much more extensive range of problems. We expect to see more experimentation and innovation using ML across all areas of business, including use-cases that didn’t justify the cost of data science projects lasting several months before.

Trials of Tallinn ML at a global retail and consumer-goods company produced a predictive model in two hours that was 18 percent more accurate, and delivered 19 times fewer false positives, than one developed over a three-month period by a team of experienced data scientists.

Another at a global financial-services organization showed that Tallinn ML’s automated feature engineering improved the accuracy of employee-churn predictions by 51 percent.

Beyond these improvements in pace and accuracy, this new approach promises to bring the benefits of ML to a much wider range of organizations. Automating the entire ML workflow democratizes data science to the point that any organization with an IT manager and big data sets to explore can start to derive value from it.

ML and the ability for algorithms to improve automatically through experience has long been recognized for its potential to bring greater intelligence and automation to the world of business. But to date, it has relied on expert humans to set up the machines to do what they do best.

Fully automating the development of ML models means that, for the first time, ML can deliver on its full promise. Efficiency. Productivity. Speed. Precision in prediction. Seriously useful business insight. Genuinely letting the machine take the strain, and freeing up humans to do what they do best.

The post Can automated feature engineering produce machine learning that finally lives up to its name? appeared first on Artificial Intelligence.

A deep-learning system developed at Google outperformed general pathologists for Gleason grading of prostate cancer biopsies, company researchers reported in JAMA Oncology online July 23.

The research comes from a deep-learning/artificial intelligence (AI) research team at Google Health in Palo Alto, CA. For the study, performance of the deep-learning system and general pathologists were compared to assessment by a panel of urologic subspecialist pathologists for the evaluation of prostate cancer needle core biopsy formalin-fixed paraffin-embedded (FFPE) specimens.

In a validation set of 498 specimens that were positive for cancer, the deep-learning system came up with the same result as the panel of sub-specialists in 71.7% of cases, versus 58% for general pathologists, a statistically significant result.

“The [deep-learning system] warrants evaluation as an assistive tool for improving prostate cancer diagnosis and treatment decisions, especially where subspecialist expertise is unavailable,” wrote Dr. Craig Mermel, PhD, product lead for pathology at Google Brain, and colleagues.

Gleason grading an ‘imperfect diagnostic tool’

Deep learning has potential to play a role in an area of pathologic evaluation of prostate biopsies that needs improvement — Gleason grading, which is an imperfect diagnostic tool, the authors wrote. Grading of prostate biopsy specimens to indicate clinical risk is based on subjective assessment of various patterns, such as pattern 5, which represents poorly differentiated cells, they noted.

“Consequently, it is common for different pathologists to assign a different [Gleason grade group] to the same biopsy (30%- 50% discordances),” Mermel et al wrote. “In general, pathologists with urologic subspeciality training show higher rates of interobserver agreement than general pathologists, and reviews by experts lead to more accurate risk stratification than reviews by less experienced pathologists.”

In recent years, Google has been actively developing AI algorithms for a variety of healthcare applications, such as detecting breast cancer on mammograms, assessing cancer risk on computed tomography (CT) lung cancer screening exams, finding diabetic retinopathy on retinal scans, and predicting medical events through analysis of electronic medical record software.

Company researchers reported results for their deep-learning system in analyzing pathology images and predicting survival of patients with 10 cancer types in PLOS One in June.

False positives trade-off

In the latest study, subspecialists made their assessments using three histological sections plus an immunohistochemical stained section for each specimen. A majority opinion was determined. Performance was compared to the deep-learning system and evaluations from a panel of board-certified general pathologists. General pathologists and the algorithm worked without immunohistochemical stained sections, simulating routine workflow.

In addition to Gleason grading, the study evaluated performance of subspecialists, general pathologists, and the deep-learning system for differentiating specimens with and without cancer. A total of 752 specimens were assessed for this part of the study. And in this scenario, the performance of the general pathologists and the deep-learning system were on par; that is, in agreement with the subspecialist findings in 94.3% and 94.7% of cases, respectively. Compared with general pathologists, the deep-learning system caught more cancers, but also flagged more false positives.

“This trade-off suggests that the [deep-learning system] could help alert pathologists to tumors that may otherwise be missed, while relying on pathologist judgment to overrule false-positive categorizations on small tissue regions,” the authors advised.

Overall, the results suggest that an automated system could bring performance closer to the level of experts and boost the value of prostate biopsies, in the authors’ view.

“Future research is necessary to evaluate the potential utility of using the [deep-learning system] as a decision support tool in clinical workflows and to improve the quality of prostate cancer grading for therapy decisions,” Mermel et al concluded.

The post Google’s deep learning outperforms general pathologists in prostate biopsies appeared first on Artificial Intelligence.

A machine-learned AI system used to assess recidivism risks in Broward County, Fla., often gave higher risk scores to African Americans than to whites, even when the latter had criminal records. The popular sentence-completion facility in Google Mail was caught assuming that an “investor” must be a male.

A celebrated natural language generator called GPT, with an uncanny ability to write polished-looking essays for any prompt, produced seemingly racist and sexist completions when given prompts about minorities. Amazon found, to its consternation, that an automated AI-based hiring system it built didn’t seem to like female candidates.

Commercial gender-recognition systems put out by industrial heavy-weights, including Amazon, IBM and Microsoft, have been shown to suffer from high misrecognition rates for people of color. Another commercial face-recognition technology that Amazon tried to sell to government agencies has been shown to have significantly higher error rates for minorities. And a popular selfie lens by Snapchat appears to “whiten” people’s faces, apparently to make them more attractive.

These are not just academic curiosities. Broward County’s recidivism system, while supposedly only one of several factors judges were to consider, was shown to have a substantial impact. Just recently, we learned of the first false arrest of an African American based largely on a facial-recognition system.

Even as these embedded biases are discovered, new ones come up.

Perhaps the most egregious are what may be called “mugshot AI,” which claims to unearth useful patterns from physiognomic characteristics. From phrenology to palmistry, pseudosciences that claim to tell personality and mental states from physical characteristics are nothing new. AI’s newfound ability to process, recognize or find patterns from large-scale physiognomic data has, however, given a dubious new lease to these dubious undertakings. Various companies claim to discern personality characteristics, including criminality, from mugshots or to speed up recruitment by analyzing job candidates from online video interviews. Indeed, there is a tremendous temptation to look for some arbitrary correlational mapping from one high-dimensional object — a person’s face, voice, posture — to another critical decision variable, given enough data.

Of course, bias existed before the advent of AI systems. Human decision-makers, from law enforcement to employment agencies, have been known to act on internal biases. One saving grace is that there is variance in individual human biases, which works to reduce their macro harm; not all humans have the same difficulty in distinguishing between nonwhite faces, for example.

Yet, bias internalized in a widely deployed AI system can be insidious — precisely because a single set of biases become institutionalized with little variance.

The situation is further exacerbated by our well-known automation bias, which makes us subconsciously give greater weight to machine decisions. 

Reining in inadvertent amplification of societal biases thus has become one of the most urgent tasks in managing the risks of data-driven AI technology.

So why do AI systems exhibit racist or sexist biases? Are people in commercial AI labs deliberately writing biased algorithms or training systems on deliberately biased data? Turns out that the offending behavior is most often learned than designed, and most of these systems have been trained on readily available public data, often gleaned from the web.

A critical catalyst for the recent successes of AI has been the automatically captured digital footprints of our lives and quotidian interactions. This allowed image-recognition systems to be trained on troves of pictures (often with labels) that we collectively upload onto the web, and natural language systems to be trained on the enormous body of language captured on the web — from Reddit to Wikipedia — through our daily interactions.

Indeed, the web and internet have become a repository of our Jungian collective subconscious — and a convenient way to train AI systems. A problem with the collective subconscious is that it is often raw, unwashed and rife with prejudices; an AI system trained on it, not surprisingly, winds up learning these and, when deployed at scale, can unwittingly exacerbate existing biases.

In other words, although it is no longer socially acceptable to admit to racist or sexist views, such views — and their consequences — often are still implicit in our collective behavior and captured in our digital footprints. Modern data-driven AI systems can unwittingly learn these biases, even if we didn’t quite intend them to.

AI systems trained on such biased data not only are used in predictive decision-making (policing, job interviews, etc.), but also to generate or finish incompletely specified data (e.g., to improve a low-resolution picture by upsampling it). This generation phase can itself be a vehicle for further propagation of biases. It shouldn’t come as a surprise, for example, that a system trained on images of engineering faculty members will more readily imagine a male face than a female one. The fact that machine-learning systems are limited to capturing correlational patterns in the data — and that some correlations may result from ingrained inequitable societal structures — means societal biases can seep in, despite well-intentioned design.

Increasingly, designers are combating societal biases in AI systems. First and foremost is curating the training data. Unlike traditional disciplines, like statistics, that paid significant attention to data collection strategies, progress in AI came mostly from exploiting the copious data available on the web. Unfortunately, that readily available data often is asymmetric and doesn’t have sufficient diversity — Western societies, for example, tend to have a larger digital footprint than others. Such asymmetries, in turn, lead to the kinds of asymmetric failures observed in gender-detection systems. Some obvious ideas for curation, such as “blinding” the learning system to certain sensitive attributes such as gender and race, have been shown to be of limited effectiveness.

The other issue of using readily available data is that it is often rife with hidden biases. For example, as much as there is temptation to train large-scale language generation systems on the profusion of text on the web, it is not surprising that a lot of this user-generated text on forums that allow anonymous postings can be rife with biases. This explains to a large extent the types of biased text completions observed in some state-of-the-art language-generation systems. There is a growing understanding that training data must be carefully vetted. Such steps may increase the costs of — and severely reduce — the data available for training. Nevertheless, given the insidious societal costs of using uncurated data, we must be ready to bear those costs.

Some also have advocated explicitly “de-biasing” the data (e.g., by balancing the classes in the training samples). While a tempting solution, such steps in essence correspond to a form of social engineering — in this case, of data. If there is social engineering to be done, it seems much better for society to do it at the societal level, rather than just by AI developers.

Part of the challenge in controlling harmful societal biases in today’s AI systems is that most of them are largely data-driven, and typically do not take any explicit knowledge as input. Given that explicit knowledge is often the most natural way to state societal norms and mores, there are efforts to effectively infuse explicit knowledge into data-driven predictive systems.

Another proactive step is looking more carefully at what is being optimized by learning systems. Most systems focus on optimizing the accuracy of a predictive system. It is, however, possible for a system that had high overall accuracy to still have bad performance on certain minority classes. More generally, there is increasing recognition that the degree of egregiousness in misclassifications must be considered — after all, confusing apples with oranges is less egregious than confusing humans with animals. The prohibitive costs of false positives in some applications (e.g., face recognition in predictive policing) might caution a civilized society that, in some cases, predictive systems based on correlational patterns should be avoided, despite their seemingly high accuracy.

As AI technology matures and becomes widely deployed, there is increased awareness — in the research community, companies, governments and society — of the importance of considering its impacts. There are now premier academic conferences devoted to scholarly understanding of the impact of AI technology in exacerbating societal biases; increasingly, AI publications ask for an explicit discussion of the broader impacts of technical work. Alerted by ongoing research, companies such as IBM, Amazon and Microsoft are declaring moratoriums on the sale of technologies such as face recognition, pending greater understanding of their impacts. Several U.S. cities have banned or suspended facial-recognition technology in policing.

There is, of course, no magic bullet for removing societal bias from AI systems. The only way to make sure fair learning can happen from the digital traces of our lives is to actually lead fair lives, however tall an order that might be.

But we also should acknowledge that these systems, rightly used, can hold a mirror up to society. Just as television brought racial injustices into our living rooms during the 1960s’ civil rights movement and helped change us for the better, AI systems based on our digital footprints can help show us ourselves and, thus, be a force for our betterment.

Subbarao Kambhampati, PhD, is a professor of computer science at Arizona State University and chief AI officer for AI Foundation, which focuses on the responsible development of AI technologies. He was president of the Association for the Advancement of Artificial Intelligence and helped start the Conference on AI, Ethics and Society. He was also a founding board member of Partnership on AI. 

The post Why are Artificial Intelligence systems biased? appeared first on Artificial Intelligence.