Zig, or go Zags? Favor new blood or blue blood? Dance with Cinderella or a stepsister?

Every March Madness bracket is a bet (often literally) on one of 9.2 quintillion possible permutations of winners and losers, front-runners and dark horses, drowsy blowouts and rousing upsets.

While out walking his dog, the University of Nebraska–Lincoln’s Matt Waite realized that the annual rite of spring and college basketball was also an ideal opportunity to apply some lessons he was teaching in Sports Media and Communication 460: Advanced Sports Data Analysis. Some of the 19 undergrads in the class might be unfamiliar with, if not openly wary of, the quantitative realm, but most were already planning to fill out brackets. So he decided to turn the ritual exercise into a class exercise.

“To tell you the truth, it wasn’t on the syllabus when I started the class,” said Waite, professor of practice of journalism and mass communications. “Between the way that the course schedule was working out, the progression that the students were making, and the timing of the tournament, it just sort of all came together.

“That’s something that I really, really try to do in my sports data classes, is make examples of the moment.”

To Waite’s mind, March Madness is especially suited to teaching the fundamentals of machine learning — in the simplest terms, feeding data into a computer algorithm for the sake of training it to predict future outcomes. Analytically inclined college basketball fans and bettors have increasingly looked to machine learning for an assist when filling out their brackets. Waite has even built his own models on the foundations of books like “Basketball on Paper” and other sacred texts of analytics.

“I wanted to have sports communicators dip their toes into the waters of machine learning and predictive analytics — where the tools of doing this have become easy enough to use, but understanding what’s going into the algorithm, and what’s coming out of it, takes some work,” he said. “But once you have some key concepts, you can communicate with it. You can tell stories with the output.”

Waite began by giving his SPMC 460 students access to the box scores of every men’s college basketball game going back to the 2014-15 season. (He tried to do the same for the women’s tournament, but despite his ongoing efforts, a lack of available data made it unworkable. “There is sexism in sports data, just as there is in sports in general. Game-level statistics for women’s basketball are vastly more difficult to get your hands on than men’s,” Waite said.)

Those box scores were stuffed with the raw statistics used to calculate more advanced metrics that have historically proven predictive of successful teams: average margin of victory, points scored per possession, shooting percentages, turnovers, offensive rebounding rates, and so on. But it was up to each student to decide which statistics they would feed into an algorithm, and which of three algorithms would consume those stats.

“Machine learning is not magic, and the algorithms are doing a very specific thing: using input that you give them and coming up with answers,” Waite said. “And you, as a human being, need to be able to evaluate those.”

With those fateful decisions made, the students tested their inputs and algorithms by asking the latter to predict the winners of games that had already been played but whose outcomes were a mystery to the machine. After some fine-tuning, the students were ready to run their newly trained algorithms through the bracket-busting gauntlet of March Madness, picking all 63 games (not including the so-called First Four) ahead of time.

“My goal was to let them run wild, see where they got, and then talk about where it went wrong after it happened,” Waite said.

Or, in the case of a few students, where it’s gone especially right.

“I’ve got a handful of folks who are just absolute basketball maniacs and were skeptical that some computer was going to tell them better than they knew,” Waite said. “I have a handful who have absolutely no interest in basketball whatsoever. I had to literally explain the rules of basketball to them, and what these statistics are, for them to even be able to function with this. And the irony is (that) two of those folks are in the top five of the class.”

Thomas Baker, a junior who leads the pack with a bracket in the 99th percentile of those submitted to ESPN.com, is an “absolute hoop-head” who can “rattle off names and their season narratives” at the drop of a basketball, Waite said. Baker put that Bilas-esque knowledge to use by occasionally disregarding an algorithm-based prediction. But he also chose a relatively sophisticated algorithm: a so-called random forest that, true to its name, consists of many decision-tree analyses that proceed in a random fashion to limit the possibility of statistical bias.

“The decision tree learns where to make splits based on the amount of similarity in data,” Waite said. “So you might take all of the teams that shoot better than 40% from the 3-point line and put them over in this group. The teams that shoot worse than that, we’re going to put them over in that group. Then those groups get split by something. And then those (subsequent) groups get split by something (else). So on and so forth, until you get to the end, where if you have a team that matches all of these particular parameters, the model says there’s a 58% chance that they’re going to win the game.”

Kaitlynn Johnson, a senior in fourth place and the 96th percentile, could hardly be more different — a total college basketball novice who built “maybe the most simplistic model,” input only some basic shooting stats, and dutifully followed every prediction. Still, Waite said, anyone who’s spent as much time as he has with brackets might have predicted the seemingly unpredictable success of a rookie predictor.

“Before this even got going, I honestly predicted that somebody like that was going to be near the top,” Waite said. “Because it happens in every bracket pool. If you’ve ever filled out a bracket in an office, you know there’s somebody in there who’s like, ‘I don’t know anything about basketball, but those uniforms are cool! Let’s pick those.’ Or, ‘I like Wildcats more than Blue Devils, so I’ll take them.’ And they always seem to do really well. So I saw her coming a mile away.”

As for Waite himself? He’s just glad to no longer be bringing up the rear, where he spent about half of the tournament. Riding a hot streak that began in the Sweet 16, he’s ascended to a respectable 11th place and breached the 60th percentile on ESPN.com. If nothing else, he said, his marginal March should at least help him illustrate an important point to the class: that while the machine needs a properly educated ghost to guide it, that education goes only so far — and even the best-informed ghosts can be busted.

“There is a certain amount of humility and, I would even say, naivete that needs to go into this, where there is such a thing as the curse of knowledge,” Waite said. “I read the canonical basketball analysis book and tried, as close as I could, to implement the analysis steps into a model. I spent hours and hours on mine, used the fanciest algorithms that I could — and immediately just got my head kicked in. Meanwhile, somebody who didn’t know what a field goal was three weeks ago came up with a very simple and, truthfully, elegant model, and is crushing it.”

And if, in the process of tracking their brackets and retracing their missteps and claiming bragging rights for the rest of the semester, the future media professionals forget or even begin losing some of their lingering aversion to numbers? So much the better, Waite said.

“The students I’ve got are not computer scientists; they’re not statistics majors,” he said. “They’ve (often) avoided math as much as possible. So, for me, the trick is trying to make this as relevant as possible, and draw them in that way. You know, it’s sort of the spoonful of sugar.

“We’re using the tournament to introduce some pretty complex topics in an environment that is easy to understand, in a way that’s accessible, using something that they’re doing anyway. If you can bring those things together, I think you’re in good territory.”

The post Model behavior: Waite teaching machine learning via March Madness appeared first on Artificial Intelligence.

A pair of newly published studies are demonstrating how passive smartphone data can be used to effectively predict relapse episodes in schizophrenia patients. The research used machine learning to analyze behavioral data and predict schizophrenic relapses up to one month before they occurred.

The data used in both new papers was gathered from a cohort of 60 subjects with schizophrenia. Passive smartphone data, such as accelerometer readings and phone call metadata (such as frequency of calls and durations) was captured for the entire cohort. Eighteen of the subjects suffered a schizophrenic relapse during the course of the study.

A type of machine learning, dubbed encoder-decoder neural networks, was then used to analyzed the mass of data looking for anomalous behavioral patterns within 30 days of a major relapse. The results revealed an 108 percent increase in behavior anomalies could be detected in the month leading up to a relapse, suggesting this kind of system may be useful for detecting and treating patients before a major schizophrenic episode arises.

“We tried to create an approach where we could tell a clinician: not only is this participant experiencing unusual behavior, these are the specific things that are different in this particular patient,” says Dan Adler, a researcher from Cornell Tech working on the project. “If we can predict when someone’s symptoms are going to change before relapse, we can get them early treatment and possibly prevent an inpatient visit.”

As well as predicting relapses ahead of time, the system could effectively predict patients’ self-assessments of their conditions. And a more granular analysis of the data revealed fine-grained symptom changes could also be predicted.

Different kinds of behavioral patterns, as tracked through passive smartphone data, could be associated with specific symptom characteristics. One of the papers, published in the journal Scientific Reports, strikingly presents a hypothetical scenario whereby the system itself could conceivably intervene in real-time to help guide subjects toward behavioral patterns that prevent a looming relapse.

“For example, if there is an unusual change in the ultradian rhythm of environment noise for a couple of hours, the system can prompt the patient to move to an environment that has a lower and more stable level of ambient noise to prevent the noise from affecting the patients’ cognitive performance,” the researchers write. “If the system notices that the patient’s phone usage in certain periods, for example in evening, has a very different pattern than in other periods (morning and afternoon), the system can intervene to change the patient’s phone usage pattern, delaying the arrival of phone notifications for instance, to avoid an increase in stress.”

anzeem Choudhury, from Cornell Tech and co-author on both of the new papers, suggests the system developed could be appropriated for many mental health conditions. Even major depressive episodes, he suggests, could be predicted ahead of time by passively tracking extreme behavioral changes.

“By focusing on changes in behavioral routines and misalignment with underlying biological rhythms, we expect our approach to generate clinically actionable insights that generalize across a diverse demographic of users,” says Choudhury.

The post Machine learning predicts schizophrenia relapses using smartphone data appeared first on Artificial Intelligence.

Why Understanding the Key Differences Between Data Science  and Software Development Matters

As Data Science  becomes a critical value driver for organizations of all sizes, business leaders who depend on both Data Science  and Software Development teams need to know how the two differ and how they should work together. Although there are lots of similarities across Software Development  and Data Science , they also have three main differences: processes, tooling and behavior. In practice, IT teams are typically responsible for enabling Data Science teams with infrastructure and tools. Because Data Science  looks similar to Software Development (they both involve writing code, right?), many IT leaders with the best intentions approach this problem with misguided assumptions, and ultimately undermine the Data Science teams they are trying to support.

Data Science  != Software Engineering

I. Process

Software engineering has well established methodologies for tracking progress such as agile points and burndown charts. Thus, managers can predict and control the process by using clearly defined metrics. Data Science  is different as research is more exploratory in nature. Data Science projects have goals such as building a model that predicts something, but like a research process, the desired end state isn’t known up front. This means Data Science  projects do not progress linearly through a lifecycle. There isn’t an agreed upon lifecycle definition for Data Science work and each organization uses its own. It would be hard for a research lab to predict the timing of a breakthrough drug discovery. In the same way, the inherent uncertainty of research makes it hard to track progress and predict the completion of Data Science  projects.

The second unique aspect of Data Science  work process is the concept of hit rate, which is the percentage of models actually being deployed and used by the business. Models created by Data Scientists are similar to leads in a sales funnel in the sense that only a portion of them will materialize. A team with 100 percent reliability is probably being too conservative and not taking on enough audacious projects. Alternatively, an unreliable team will rarely have meaningful impact from their projects. Even when a model didn’t get used by the business, it doesn’t mean it’s a waste of work or the model is bad. Like a good research team, Data Science  teams learn from their mistakes and document insights in searchable knowledge management systems. This is very different from Software Development where the intention is to put all the development to use in specific projects.

The third key difference in the model development process is the level of integration with other parts of the organization. Engineering is usually able to operate somewhat independently from other parts of the business. Engineering’s priorities are certainly aligned with other departments, but they generally don’t need to interact with marketing, finance or HR on a daily basis. In fact, the entire discipline of product management exists to help facilitate these conversations and translate needs and requirements. In contrast, a Data Science  team is most effective when it works closely with the business units who will use their models or analyses. Thus, Data Science team needs to organize themselves effectively to enable seamless, frequent cross-organization communication to iterate on model effectiveness. For example, to help business stakeholders collaborate on in-flight Data Science projects, it’s critical that Data Scientists have easy ways of sharing results with business users.

II. Tools and Infrastructure

There is a tremendous amount of innovation in the Data Science  open source ecosystem, including vibrant communities around R and Python, commercial packages like H2O and SAS, and rapidly advancing deep learning tools like TensorFlow that leverage powerful GPUs. Data Scientists should be able to easily test new packages and techniques, without IT bottlenecks or risking destabilizing the systems that their colleagues rely on. They need easy access to different languages so they can choose the right tool for the job. And they shouldn’t have to use different environments or silos when they switch languages. Although it is preferable to allow greater tool flexibility at the experimentation stage, once the project goes into deployment stage, higher technical validation bars and joint efforts with IT become key to success.

On the infrastructure front, Data Scientists should be able to access large machines, specialized hardware for running experiments or doing exploratory analysis. They need to be able to easily use burst/elastic compute on demand, with minimal DevOps help. The infrastructure demands of Data Science  teams are also very different from those of engineering teams. For a data scientist, memory and CPU can be a bottleneck on their progress because much of their work involves computationally intensive experiments. For example, it can take 30 minutes to write code for an experiment that would take 8 hours to run on a laptop. Furthermore, compute capacity needs aren’t constant over the course of a Data Science  project, with burst compute consumption being the norm rather than the exception. Many Data Science techniques utilize large machines by parallelizing work across cores or loading more data into the memory.

III. Behavior 

With software, there is a notion of a correct answer and prescribed functionality, which means it’s possible to write tests that verify the intended behavior. This doesn’t hold for Data Science  work, because there is no “right” answer, only better or worse ones. Oftentimes, we’ll hear Data Scientists discuss how they are responsible for building a model as a product or making a slew of models that build on each other that impact business strategy. Unlike statistical models which assume that the distribution of data will remain the same, the distribution of data in machine learning are probabilistic, not deterministic. As a result, they drift and need constant feedback from end users. Data Science  managers often act as a bridge to the business lines and are focused on the quality and pace of the output. Evaluating the model and detecting distribution drift enables people to identify when to retrain the model. Rather than writing unit tests like software engineers, Data Scientists inspect outputs, then obtain feedback from business stakeholders to gauge the performance of their models. Effective models need to be constantly retrained to stay relevant as opposed to a “set it and forget it” workflow.

Final Thoughts

In general, there are several good practices for Data Scientists to learn from Software Development , but there are also some key differences to keep top of mind. The rigor and discipline that modern Software Development  has created is great and should be emulated where appropriate, but we must also realize that what Data Scientists build is fundamentally different from software engineers. Software Development and Data Science processes often intersect as software captures much of the data used by Data Scientists as well as serving as the “delivery vehicle” for many models. So the two disciplines, while distinct, should work alongside each other to ultimately drive business value. Understanding the fundamental nature of Data Science  work can set a solid foundation for companies to build value-added Data Science  teams with the support of senior leadership and IT team.

The post WHAT IS THE REAL DIFFERENCE BETWEEN DATA SCIENCE AND SOFTWARE ENGINEERING TEAMS? appeared first on Artificial Intelligence.