If alien anthropologists wanted to learn about human behavior, they would likely examine our literary works. Among the embarrassing flotsam, they would also discover Thomas Paine’s The Age of Reason, Toni Morrison’s Song of Solomon, or the lyrics to Bob Dylan’s “Blowin’ in the Wind.” The aliens might conclude that we are a troubled species, plagued by a mix of arrogance and ignorance, but with an overall trajectory that is progressive and promising.

The point is that there’s a treasure trove of information about our nature that can be extracted from the collective works of humanity. According to Google, there are approximately 130 million books in the world, and the company intends to scan all of them. So far, they’ve scanned about 30 million, and this impressive database is already being mined by scientists who are seeking answers to questions about our historical behavior.

Thirty million books contain an extraordinary amount of information, thus it qualifies as “big data.” As our computational ability to manage and probe large datasets increases, researchers are poised to answer queries that we couldn’t dream of addressing just a few years ago. Today, big data is routinely used in science laboratories—for example, when geneticists compare DNA sequences between tens of thousands of individuals to find correlations between gene variants and behavior (these are referred to as GWAS, or genome-wide association studies).

In much the same way geneticists can analyze millions of genes to learn about human physiology, scientists can scan millions of books to learn about human culture. And like “genomics,” this new science has been dubbed “culturomics” by its pioneers, Erez Aiden and Jean-Baptiste Michel. In their 2013 book, Uncharted: Big Data as a Lens on Human Culture, they mine the Google Book database to answer intriguing questions about our past behavior. It’s like asking questions of someone who has memorized the content of 30 million books spanning the existence of human civilization.

Remember learning about irregular verbs in grade school? For most verbs, we simply add an -ed suffix to denote the past tense: one dances today, but danced yesterday. This rule emerged a long time ago, between 500 and 250 BCE. So why do we say fell instead of falled in 2019? Drove instead of drived? Ate instead of eated? These pesky exceptions to the -ed rule are one of the things that make the English language so peculiar. Aiden and Michel used their big data approach to show that the adaptation of -ed correlated with the popularity of the word in question. In other words, today’s irregular verbs were used so commonly in the past that they resisted the -ed rule. Time does seem to catch up to these verbs, slowly but surely, and they estimate that of the 177 irregular verbs that remain, only 83 will still be irregular in the year 2500. Unfortunately, we will have drawed our last breath long before then, and won’t be around to verify this prediction.

In addition to our language quirks, culturomics can shine a spotlight on fame. By probing humanity’s books for someone’s name, one can get a sense of how famous that person was over time (although unique names like Mark Twain are less confounding than John Smith). Fame is a strange feature of human society—some people become popular because of their intelligence, heroism, or athletic prowess. Others because they have been naughty. And still others for dumb luck. Through clever probing of their dataset, Aiden and Michel not only identified the most famous people born each year between 1800 and 1949 but also which occupations were most likely to generate famous people. The top four included politicians, authors, actors, and scientists. As an author and a scientist, perhaps my odds are pretty good?

Culturomics can also be used to discern when a phrase became famous. Today’s ubiquitous holiday greeting, “Merry Christmas,” was practically absent from the literature until the year 1843. Why? That’s the year Charles Dickens published A Christmas Carol. Dickens should be acknowledged for popularizing “Merry Christmas” as much as “Bah…humbug!” Interesting facts about our inventions also emerge from analyzing big data. Some inventions, like the iPod, took off like wildfire. There’s another invention that is wildly popular today but took over a century to catch on: blue jeans.

As always, learning about our past vaccinates us against cultural ailments that caused great suffering among our forbearers. By surveying the dark periods of human history, culturomics teaches a haunting lesson. The data reveal a striking paucity of non-Nazi artists and authors in German literature during World War II, creating an unsettling gap on a graph of the country’s history. Lest we get too comfortable that such censorship never happens in the home of the free, a similar gap appears on a graph of blacklisted actors during the era of McCarthyism.

If you would like to probe the big data yourself, you can do so here. But be warned: this can be habit-forming! Here is a sample graph I generated that analyzes food trends over the centuries.

History can indeed be (re)written by the victors, making the preservation of big data all the more vital. The words humanity leaves behind are like the threads of a story quilt, and the lessons learned from big data may prevent it from becoming a burial shroud.

The post Culturomics: Using Big Data to Study Human Behavior appeared first on Artificial Intelligence.

We humans generally like to pride ourselves on our ability to be unpredictable, to do things for no other reason than because we want to. The truth is, we are complicated, but with the emergence of deep learning, we may become more predictable.

The theory goes something like this: In the past, people’s behavior was generally considered to be unpredictable because we had no way of tracking and analyzing everything that goes into our decision-making processes. Now, with sophisticated computer systems, deep learning and the ability to store vast amounts of data, we can begin to parse that information to find patterns in the ways people operate.

The question that every marketer asks themselves daily is: How can I find more people that want to buy my products? For a decade, the answer has been data. Most marketers, however, have only seen modest gains with data collection because there is too much to make sense of.

Think about all the decision making that goes into choosing where to eat for lunch. Some people go to the same place every day. But if they’re like me, they make a game-time decision on what they’ll eat for lunch. Living and working in New York City, I can choose between nearly every cuisine (Indian, Chinese, pizza, etc.) and every type of restaurant (fast, slow, fancy, casual). Needless to say, it’s much more difficult to accurately predict my lunch than the other person’s.

Let’s say that someone somewhere has been keeping tabs on where I’ve eaten for the last six months. That person might be able to predict my next meal, but chances are, it won’t be all that accurate. After all, there are so many factors that go into how I choose my meal — from what I ate for breakfast to whether or not I saw an ad for fried chicken that morning on my phone (in which case, Popeye’s might be getting some business from me). Even assuming that the all-seeing overseer of my lunch habits has data on what I eat for breakfast and dinner, when and how often I go to the gym, whether I was out late last night, my sleeping habits, the weather, what ads I saw on my phone and so on, there are so many variables that it would take ages for that person to find a pattern — if one even exists.

And that’s just my data. If a marketer really wanted to understand people’s eating habits on a larger scale, they’d need to gather all of this information from millions of people. This is the kind of problem that has kept most marketing executives up at night. But with deep learning, they can finally rest easy. We are now in an era where gargantuan amounts of data can be coupled with the power of deep learning to finally make a marketer’s data-driven dreams come true.

So, now we have eating and daily habit data on tons of people. Along comes Jeremy’s Hot Dog Stand: A huge success on its little corner of midtown New York, Jeremy’s is now rolling out carts all around the city. In order to attract new customers, we use our existing customers to build an algorithm that predicts which other people will want our hot dogs.

The thousand customers that ate our hot dogs last month are a diverse group, and we plug all of their data from the last six months into our deep learning platform. Out comes a deep neural network algorithm that is good at recognizing someone who likes our hot dogs. Now, we can start entering data on the eating habits of the millions of other people who are not yet customers so that our neural network can tell us how likely each person is to be a future customer. With that information, I can create a list of the people who are most likely to enjoy our hot dogs and target them with ads. That way, I don’t have to waste my money advertising to people unlikely to convert.

Even more impressively, I can dynamically bid different amounts for every single one of the millions I’ve scored. So even if someone has a low probability score from my neural network, I can decide to spend a little on them to see if my model can learn something. Within six months, Jeremy’s Hot Dog Stand has become a national success!

Of course, the power of deep learning can do more than just power a hot dog stand to success. Retailers can use deep learning to target those most likely to visit their stores. Financial institutions may be able to employ deep learning in order to improve future performance. And advertisers across all fields can harness deep learning to find the people who are likely to buy what they’re selling. This is why deep learning is exponentially valuable to the typical business.

While it might seem daunting to build a neural network or deep-learning platform from scratch, marketers should keep in mind that there are numerous tools out there, from established players such as IBM and Amazon to small startups, that can help them harness the insights of big data and deep learning.

That is essentially what marketing is about — anticipating people’s needs and future behavior. Thanks to the rise of big data and the development of advanced machine-learning technologies, including deep learning, predicting human behavior more accurately has finally become possible. Instead of relying on generalized observations meant to apply to as many people as possible, marketers can build their own deep-learning platforms or rely on tools that will allow them to personalize marketing approaches in order to best appeal to all the individuals, no matter how diverse, that will buy their products. Instead of choosing a specific demographic (males, 18-49, with income over $50,000), they can just choose “People who will buy my product.”

The post How Deep Learning Can Help Predict Human Behavior appeared first on Artificial Intelligence.

UNIVERSITY PARK, Pa. — Twitter may have started out as a way to connect to other people and share news quickly, but the social media platform is also a powerful tool, with the data generated representing the largest publicly accessible archive of human behavior in existence.

Guangqing Chi, associate professor of rural sociology and demography and public health sciences in Penn State’s Department of Agricultural Economics, Sociology, and Education and director of the Computational and Spatial Analysis (CSA) Core in the Social Science Research Institute, and his team have collected over 30 terabytes of geo-tagged tweets over the last four years.

“Our work has the potential to change the landscape of population research,” said Chi. “It could open the door for demographers to take advantage of rich geo-tagged Twitter data and strengthen research in many other disciplines that use demographic data.”

Geo-tagged tweets are tagged with real-world geographic location information which are derived from location-based-service-enabled devices such as smartphones and tablets via GPS and Wi-Fi positioning. “Each geo-located tweet is essentially a digital trace of the Twitter user, including information such as location, time, and the content of the message,” Chi said. “Twitter data can provide a significant amount of individual social, behavioral and emotional information for researchers of many disciplines.”

Junjun Yin, CSA research associate on the project, and Chi have built an infrastructure to collect, manage, and analyze the data. “We’re storing the data in a high-performance computing cluster with large amounts of storage capacity and memory,” Yin explained. “In addition, a distributed computing environment with integrated machine learning and data-mining packages and toolsets is up and running to provide efficient parallel data processing, which includes data extraction, calculation and analysis. We’ve also developed data processing programs so the data can be useful to researchers from many disciplines.”

According to Chi, although this digital trace is not a complete trajectory tracking every movement of a user over space and time, nor is the whole data collection a representative sample of the whole population, the geo-located Twitter data can offer certain unique qualities for potential interdisciplinary research.

“Geographically annotated social media is extremely valuable for modern information retrieval. The data offers large spatial coverage and multiple years of a large sample of the population, making it helpful in determining geographical uses of space, such as urban mobility and understanding functions of urban regions,” Chi explained. “The data can also be used to explore quality of life issues, such as health, education and income. Other uses include analysis of social ties and dissemination pattern of news and events, as well as enriching existing survey data.”

In one project, Chi and his team are developing a set of methods to accurately predict demographics in real time. Knowing the demographics of a group is usually the first step in population research. Previously, Twitter data was limited to only a few demographics of Twitter users, and the Twitter user demographics and language use changed frequently, making prediction methods inaccurate.

Chi and his team are also developing algorithm models to predict the composition of a group of twitter users. “Our goal is to find a way to predict Twitter user demographics, so that we will know each Twitter user represents how many people with similar characteristics. When we can do that, we can develop weights and make the data representative.”

The approach is based on the premise that it is difficult to make predictions about an individual but is much easier to make predictions about large groups of individuals. The researchers compare their findings to U.S. Census data to determine how effective their models are.

CSA plans to offer workshops starting this fall to promote the use of Big Data for social science research and packaging the Twitter data and capacity into a product for collaboration with Penn State researchers.

The work is being supported by a National Science Foundation grant and the Social Science Research Institute and the Population Research Institute at Penn State.

Additional researchers participating in this project include Daniel Kifer, associate professor of computer science; Jennifer Van Hook, professor of sociology and demography; Lee Giles, professor of information sciences and technology and director of the Intelligent Information Systems Research Laboratory, all at Penn State; as well as Xiaopeng Li, assistant professor of civil engineering at the University of South Florida; and Tse-Chuan Yang, associate professor of sociology at the State University of New York at Albany.

The post Twitter data changing future of population research appeared first on Artificial Intelligence.