Have you ever noticed how accurate Netflix’s recommendations are to your taste? And how is Google Maps so confident I’m going home, that it will suggest directions to my house? Even my iPhone suggests what time I should set my alarm clock right before I go to bed. This means it knows when I’m going to bed and tells me the optimal time to wake up, down to the minute, based on my sleep patterns. Amazing, right?

So why do most organizations continue to use their data the same way they would have used it 10, 15 or even 20 years ago? Wouldn’t it make sense for businesses to use this technology to create better customer experiences and operational improvements?

What if you could use this technology to help your customers improve their lives? The good news is that the same machine learning technologies utilized within these large organizations are offered to the general public by various providers, such as AWS, Google Cloud Platform and Azure — for a fee, of course.

At AWS re:Invent 2019, there was a clear, overarching theme: machine learning. And not just machine learning, but the big push for using technology that can make smarter decisions for your business in real time, with minimal involvement from your organization (once it’s set up, of course).

And what exactly makes machine learning and artificial intelligence work? Data — and lots of it. To make this work, you need to experiment with machine learning and data lakes and create an open data culture.

Experiment with machine learning and data lakes.

Let’s say you’re a successful organization looking to uncover your most profitable customer cohort. Knowing this information will give you better insights into what products are working best for which customers. This means you can make more informed marketing and operational decisions based on consumer activity.

You could go into your financial reporting systems and see what products are selling the most, but that only tells you which products are selling, not who buys these products. So you dig further in your separate, disconnected-from-the-financials CRM system to understand the types of customers buying those products. There, with some manual or development effort, you can uncover this information.

And now, what if you wanted to know what marketing campaigns are driving sales to these specific customers?

Well, that’s when it gets interesting. This is possible through the integration of data silos, mostly through manual effort in copying and pasting, formula-driven Excel spreadsheets or specific development effort. This is a lot of work, but still possible in a data silo.

Let’s take it a step further.

What if you want your marketing system to automate personalized marketing campaigns based on purchasing behavior, incident tickets, ratings of products, and your customers’ activity on social media and browsing behavior on your website. Now, life becomes much harder, if not impossible, when all of your data reporting platforms are siloed.

This is why many organizations and cloud vendors are pushing organizations to data lakes. A data lake, unlike a data silo, is natively suited for artificial intelligence and machine learning analysis, as well as building predictive models from disparate, disconnected data sources.

In our example, if all of the data is within a data lake, you can not only uncover insights you were never able to before, but you can build models in real time to send out personalized marketing campaigns at the right time.

Consider moving your data to data lakes to see the impact for yourself.

Create an open data culture.

If Jeff Bezos didn’t mandate that every single department open up access to its data through APIs, then AWS actually wouldn’t exist as we know it today. The culture of departments working together and sharing data between each other was the start of the Amazon S3 storage system.

Traditionally, we build our data warehouses around a type of data, in a data silo. Most data reporting systems are siloed and only accessible by their respective departments. The data is also nicely organized like a typical relational database and is easy to understand.

But, as I mentioned, the data isn’t connected, which leaves an organization with many blind spots. Instead of finance, sales, marketing and operations living in their own worlds, wouldn’t it make sense for teams to be able to make decisions based on connected data systems?

Companies, specifically department leads, should consider changing their mindset and open up their data to departments within the organization. By moving away from a data silo world to a data lake environment, you’re giving your business an extra edge to compete. You’re giving the data scientists permission to find new business opportunities that would never be possible by partitioning your department’s data.

Machine learning is here to stay. It’s accessible, and the technology has improved to where any organization, not just Google, Microsoft and Apple, can utilize these advancements to make a positive impact with their customers and the world. It’s not too late. In fact, it’s just the beginning.

The post How Traditional Companies Can Utilize AI And Machine Learning To Build Better Products appeared first on Artificial Intelligence.

I have written on several occasions about Netflix’s use of data analysis and how this strategy has given it a clear advantage over traditional approaches based on studio executives’ “gut feeling”. We now have tangible evidence of the success of this strategy: in December Netflix dominated the Golden Globes nominations, and at the upcoming Oscars, has 22, way ahead of its rivals.

Some people in the film industry will probably not be too happy with Neflix’s strategy, which has already made it an industry leader. In the meantime, its competitors are now openly trying to imitate that strategy, which is based on two fundamental methods: the use of big data, and playing a long game.

Coincidence? Not really. Netflix’s use of big data has already been commented on in countless analyses: it is based on the channel’s supremacy, something that the company began to verify when it used to send DVDs by mail, but of which it was completely aware when it began to replace this with streaming. An approach that provides superior data and instantaneous feedback, as well as setting it apart from the competition. Box office feedback is useful, but no matter what we do, it can’t offer us the level of detail the internet provides, with users voting in real time with mouse clicks or their remote control. Steven Spielberg may insist Netflix is “minor league” because it uses television rather than the big screen, but its goal is not to dominate television content, but all content. Period. Wherever it is consumed.

That said, big data is not magic. Netflix says that it is always secondary to creativity, and that signing opportunities, contracts or development and creative ideas related to the production of a film or series make up 70% of the decision, with analysis of data the remainder, generally related to the volume of resources that will be put on the table. According to Ted Sarandos, chief content officer for Netflix, “choosing content and working with the creative community is a very human function, data doesn’t help you in that process, it helps you evaluate the investment.”

How much would change in companies if we had decision-making systems based on a proper analytical data architecture, instead of relying on management’s overrated “intuition”? In the end, the experience that guides intuition shows that all managers are wrong by roughly the same percentage, and that some are just lucky in that their biggest mistakes coincide with decisions that had little impact. Netflix’s success proves this: if we consistently resort to data analysis, a greater percentage of our decisions will be better made, the risks we take will be more balanced, and the results will be better.

Many managers would dismiss this out of hand and say I have no idea about the value of experience, the role of intuition or the importance of intangibles in decision making. This is not the only part of Netflix’s strategy that prompts such a reaction: the other is the company’s long-term orientation. For traditional managers, looking beyond the quarter that determines, in many cases, the size of their bonus is difficult and unjustifiable. They live with the pressure of the quarterly results, with the anguish of not letting down the analysts who determine the value of their stock option package, and therefore, when they see companies like Netflix, which has been losing significant amounts of money since its origins, and a manager, Reed Hastings, who proudly states that those losses will continue for many years, dismiss it as a some sort of scam or a pyramid scheme, and file it under “the exception that proves the rule” and predicting that it cannot last.

A few weeks ago, that “exception to the rule” called Netflix beat all the traditional studios and positioned itself as the current industry leader. In fact, it has been doing so for several years now at different award ceremonies, interrupted only by major events such as the last season of Game of Thrones, confirming beyond doubt that its model is superior. The same could be said of other companies that play a long game without blinking, such as Tesla or Amazon, again scandalizing traditional managers. If you are also scandalized take it into account: today’s leadership is based on big data and a long game. Your analysis and decision models are probably outdated, and at some point, these flaws might get exposed. Don’t say you weren’t warned.

The post Netflix: Big Data And Playing A Long Game Is Proving A Winning Strategy appeared first on Artificial Intelligence.

Whether you noticed or not, you are receiving and creating countless data in your everyday life, sometimes merely by sending messages and browsing items on a shopping site. Many fields, such as medicine and entertainment are data-rich, which drives researchers to find new ways to capture and analyze this rapidly increasing information.

Carnegie Mellon University’s Yuejie Chi is one of these researchers.

“There’re lots of interesting questions about how you can model such data and how you can extract information from these data,” said Chi, an associate professor of electrical and computer engineering. “They allow me to apply the type of tools I know to some practical problems that domain experts might be interested in.”

For her research, Chi earned a Presidential Early Career Award for Scientists and Engineers (PECASE). Established in 1996, the PECASE is the highest honor bestowed by the United States Government to outstanding scientists and engineers who have begun their independent research careers and have shown exceptional promise for advancing their fields.

Chi’s research focuses on representing data efficiently to reduce complexity and improve decision making.

“We can obtain plenty of information from big data, but the data we observe and collect every day can be highly redundant, messy, and incomplete,” Chi said. “Take movie sites such as Netflix as an example; the users may only review a small number of films even though there are thousands of films out there.”

How, then, can people extract useful information from these raw data? Though overwhelming at first glance, the entries in big data matrices can be correlated. There may be millions of users in a movie site, but they have many similarities such as age, country of origin and educational background. Likewise, movies can have the same genres, directors and lead actors. By studying entries by their correlations, researchers can obtain their hidden features. By focusing on these latent variables, movie sites can predict the missing entries and what movies the users might like. In this way, they can design algorithms to build an effective recommendation system.

“You don’t directly just think about the data itself; you’re trying to get some structures,” Chi said. “Once you get a good model of the latent structure, you can think about solving an inverse problem where you try to recover those latent structures using optimization. So we’re studying how to design algorithms to recover these structures.”

Aside from recommendation systems, Chi also uses latent representations to examine problems associated with imaging modalities. Biologists build devices, such as single-molecule super-resolution microscopy, to look at structures within cells, but the images they collect often lack the desirable resolution due to limitations of the device. By studying latent structures, Chi’s team has developed a new algorithm that significantly enhances the image resolutions; it uses the same available data but fewer computational resources.

Recently, Chi has been developing algorithms for distributed optimization. Nowadays, people often distribute data to different machines, as the data sets are too massive to fit onto a single device. Once they establish a distributed setting, however, communication issues may arise among individual machines. There may be adversarial events, and some entities may not want to share data with the central location for privacy reasons. Thus, Chi aims to design algorithms that are communication-efficient and resilient to outlier events.

“Once you know how to represent your data, you can leverage the structures in your algorithm design and achieve the goal more efficiently,” Chi said.

The post Chi Explores Essence of Big Data appeared first on Artificial Intelligence.