It’s 2020 and the world has changed remarkably, including in how companies screen data science candidates. While many things have changed, there is one change that stands out above the rest. At The Data Incubator, we run a data science fellowship and are responsible for hundreds of data science hires each year. We have observed these hires go from a rare practice to being standard for over 80% of hiring companies. Many of the holdouts tend to be the largest (and traditionally most cautious) enterprises. At this point, they are at a serious competitive disadvantage in hiring.

Historically, data science hiring practices evolved from software engineering. A hallmark of software engineering interviewing is the dreaded brain teaser, puzzles like “How many golf balls would fit inside a Boeing 747?” or “Implement the quick-sort algorithm on the whiteboard.” Candidates will study for weeks or months for these and the hiring website Glassdoor  has an entire section devoted to them. In data science, the traditional coding brain teaser has been supplemented with statistics ones as well — “What is the probability that the sum of two dice rolls is divisible by three?” Over the years, companies are starting to realize that these brain teasers are not terribly effective and have started cutting down their usage.

In their place, firms are focusing on project-based data assessments. These ask data science candidates to analyze real-world data provided by the company. Rather than having a single correct answer, project-based assessments are often more open-ended, encouraging exploration. Interviewees typically submit code and a write-up of their results. These have a number of advantages, both in terms of form and substance.

First, the environment for data assessments is far more realistic. Brain teasers unnecessarily put candidates on the spot or compel them to awkwardly code on a whiteboard. Because answers to brain teasers are readily Google-able, internet resources are off-limits. On the job, it is unlikely that you’ll be asked to code on a whiteboard or perform mental math with someone peering over your shoulder. It is incomprehensible that you’ll be denied internet access during work hours. Data assessments also allow the applicants to complete the assessment at a more realistic pace, using their favorite IDE or coding environment.

“Take-home challenges give you a chance to simulate how the candidate will perform on the job more realistically than with puzzle interview questions,” said Sean Gerrish, an engineering manager and author of “How Smart Machines Think.”

Second, the substance of data assessments is also more realistic. By design, brainteasers are tricky or test knowledge of well-known algorithms. In real life, one would never write these algorithms by hand (you would use one of the dozens of solutions freely available on the internet) and the problems encountered on the job are rarely tricky in the same way. By giving candidates real data they might work with and structuring the deliverable in line with how results are actually shared at the company, data projects are more closely aligned with actual job skills.

Jesse Anderson, an industry veteran and author of “Data Teams,” is a big fan of data assessments: “It’s a mutually beneficial setup. Interviewees are given a fighting chance that mimics the real-world. Managers get closer to an on-the-job look at a candidate’s work and abilities.” Project-based assessments have the added benefit of assessing written communication strength, an increasingly important skill in the work-from-home world of COVID-19.

Finally, written technical project work can help avoid bias by de-emphasizing traditional but prejudicially fraught aspects of the hiring process. Resumes with Hispanic and African American names receive fewer callbacks than the same resume with white names. In response, minority candidates deliberately “whiten” their resumes to compensate. In-person interviews often rely on similarly problematic gut feel. By emphasizing an assessment closely tied to job performance, interviewers can focus their energies on actual qualifications, rather than relying on potentially biased “instincts.” Companies looking to embrace #BLM and #MeToo beyond hashtagging may consider how tweaking their hiring processes can lead to greater equality.

The exact form of data assessments vary. At The Data Incubator, we found that over 60% of firms provide take-home data assessments. These best simulate the actual work environment, allowing the candidate to work from home (typically) over the course of a few days. Another roughly 20% require interview data projects, where candidates analyze data as a part of the interview process. While candidates face more time pressure from these, they also do not feel the pressure to ceaselessly work on the assessment. “Take-home challenges take a lot of time,” explains Field Cady, an experienced data scientist and author of “The Data Science Handbook.” “This is a big chore for candidates and can be unfair (for example) to people with family commitments who can’t afford to spend many evening hours on the challenge.”

To reduce the number of custom data projects, smart candidates are preemptively building their own portfolio projects to showcase their skills and companies are increasingly accepting these in lieu of custom work.

Companies relying on old-fashioned brainteasers are a vanishing breed. Of the recalcitrant 20% of employers still sticking with brainteasers, most are the larger, more established enterprises that are usually slower to adapt to change. They need to realize that the antiquated hiring process doesn’t just look quaint, it’s actively driving candidates away. At a recent virtual conference, one of my fellow panelists was a data science new hire who explained that he had turned down opportunities based on the firm’s poor screening process.

How strong can the team be if the hiring process is so outmoded? This sentiment is also widely shared by the Ph.D.s completing The Data Incubator’s data science fellowship. Companies that fail to embrace the new reality are losing the battle for top talent.

The post What’s different about hiring data scientists in 2020? appeared first on Artificial Intelligence.

Data scientists are now part of the gig culture movement, but should you hire a freelancer instead of a full-time data scientist? If you lack data science talent or your existing data science team needs expertise it lacks, such as computer vision or natural language processing, perhaps you should consider a contract hire. But freelance data scientists aren’t always the answer to an organization’s needs.

Overall, companies are more inclined to hire a full-time data scientist than an individual contractor, if employment sites are any indication. On March 5, 2020, Indeed had 11,297 full-time data scientist positions listed and only 283 contract jobs. CareerBuilder had 3,122 full-time positions listed and 451 contract jobs.

Of course, there are other options. Employers could hire a part-time data scientist or use a consulting firm. If enterprises have a particularly compelling problem, such as curing a form of cancer, another option is to host a data science competition using a platform such as Kaggle.

However, before hiring data science talent, it’s best to understand what data scientists do because there are some nuances specific to this type of freelancer that hiring managers are wise to understand.

What is a data scientist?

A data scientist is a data expert who often holds an advanced degree in mathematics or statistics and probably knows how to code in R or Python. The most sought-after data scientists also have relevant business domain expertise.

While skill sets vary among individuals, a data scientist’s job is to help their employer solve difficult problems often involving discovery, optimization and/or prediction. The role may be considered part of IT or it may be specific to a departmental function. Of all possible data-related roles, data scientists tend to be the most sophisticated type of talent.

There are many myths surrounding data scientists, which can be counterproductive to hiring for the role.

The most common myth is the “unicorn” many organizations look for. This fictional character knows everything there is to know about data and is a coding superhero and a mathematical or statistical genius. Just point this individual at data and magic will happen.

This false belief results in unrealistic job requirements and unrealistic expectations of what data scientists and data science can do.

Why hire a freelance data scientist?

Matt Johnson, COO of data science consultancy Data Mettle, said there are three reasons clients tend to turn to freelance data scientists versus hiring full-time help: They aren’t sure they need a data scientist, they lack the expertise to understand what skills they need to hire or they just want to do a stand-alone project.

“Often, if they have some data and they think they can do something interesting or of value with it — rather than hiring a data scientist — it makes more sense to bring in someone for a few weeks or a month to explore the data, understand the business challenges and opportunities and what’s feasible,” Johnson said.

If a company doesn’t understand data science at all, it’s hard to hire for certain skills because hiring managers are unable to articulate what they need and why they need it.

“If they just want to do a stand-alone project, for example, they want a tool that optimizes scheduling for their workforce [which will take] a month or two of work to build the tool, then they won’t have much of a need for a full-time data scientist after that,” Johnson said.

A freelance data scientist can help decision-makers understand some of the basics, including what a data scientist does, what a data scientist needs to be successful and what data science can and cannot accomplish given the available data and other important factors that should be considered.

What can go wrong with contract help

If a company hires a full-time data scientist, most likely no one will expect that person to produce results on day one. Before a data scientist can share any valuable insights, that individual must first understand what the business hopes to achieve, what data is available, what data isn’t available, etc.

“The success of data science is completely predicated upon the data and if your data is insufficient, incomplete or inaccurate, you’re not going to get results — or good results — and the data scientist can’t fix that because the data you have is the data you have,” said Brandon Purcell, principal analyst at Forrester Research.

Nevertheless, unlike a new full-time data scientist, organizations often expect a freelance data scientist to be productive immediately just as with other types of contractors, and they struggle with getting results as quickly as desired.

“Even the most experienced data scientists face this problem as every company’s data can be extremely different,” said Robert O’Callaghan, director of data science at relationship commerce platform provider Ordergroove. O’Callaghan is also a former freelance data scientist.

“Unfortunately, that happens a lot of the time,” Purcell said. “A data scientist will come in and do their best, and they may be very talented but any model they create is only as good as a coin toss.”

Another misconception is that a freelance data scientist’s project is complete once the analysis is finished, when implementation and maintenance are also necessary for the company to extract business value from the data. For example, as new data comes in, a model must be tuned or it will drift, becoming less accurate.

“I have seen multiple brilliantly analyzed — and expensive — projects fail to deliver value due to businesses believing that a project was complete before the back-end work was in place,” O’Callaghan said. “[That’s] an issue that does not occur with full-time data scientists.”

It’s also important to understand what should happen after the contract concludes.

“In an ideal world you’d 100% plan ahead to say this data science freelancer will do this piece of work, and at the end of that I will have this insight and then I can do X, Y or Z,” O’Callaghan said. “You can never really 100% anticipate your results, so you will need to be more flexible in understanding what the next step is once the work is complete.”

Fundamentally, companies are not scoping freelance data science projects appropriately. And, they may underestimate the impact these insights will have on business operations.

“You’re going to be using that analysis to change the way you interact with customers, perform your operations or the way your human resources behave,” Purcell said. “That’s going to take longer than building a model. [If the analysis doesn’t result in] process changes [or] operational changes, there’s a good chance the model is going to end up being this shiny science project that never gets adopted.”

Bottom line

If you don’t already have a data science function, a freelance data scientist may help you better understand the opportunities and pitfalls. Freelancers are also a good choice for project work whether a data science function exists or not.

Be wary of making assumptions about what data science and data scientists can and can’t do if you don’t have the benefit of expert insight, however. Otherwise, your data science efforts and their results may fall short of expectations or fail entirely.

The post Should your company hire a freelance data scientist? appeared first on Artificial Intelligence.

With the demand for data scientists still booming, many are flocking to the career, and they bring various skills with them to the field of data science. Data science can also be applied in many different fields, so often those without a title related to data science use these skills in their everyday work.

Skills gained in varying areas, from mathematics to cancer research to astrophysics, can be valuable in a data science career.

Where do data scientists come from?

Especially since colleges and universities are only just beginning to offer data science majors and master’s degrees, few people entering the field have a specific data science background.

One common background for data scientists is mathematics. Speaking at the Women in Data Science conference in Cambridge, Mass. on March 2, Moon Duchin, associate professor of mathematics at Tufts University, described her background, which mainly involves using data science in voter redistricting.

“It’s all math skills that you build [and those skills] help you better when they improve,” she said when asked how her background in mathematics helps her data projects.

Duchin uses math and data science for the purpose of civil rights. She currently works with city and state officials to redraw voting districts so groups that are currently underrepresented can have their votes counted more accurately.

Building expertise in mathematics is crucial in data science, said Duchin. Mathematics helps “push the frontiers in machine learning — ask why and how” models work.

What brings people to data science?

But it’s not only a math background that prepares a person for a career in data science. Katie Montgomery, who attended the conference, is new to the data science community. With a background in communications, Montgomery is a staff assistant in the capital giving team for the Faculty of Arts & Science at Harvard University. Data plays a major role in increasing fundraising capital. Her team works with data to determine where to focus fundraising efforts.

“There must be a system other than humans to pull that information,” Montgomery said. “I’m here to see that there are other ways to get there without a background in data science.”

With the rise of embedded BI platforms and the increase in citizen data scientists, it’s increasingly common to have more people in similar situations.

Vasudha Shivamoggi, senior data scientist at Rapid7, didn’t come from a true data science background. She earned a PhD in physics and later moved into a career in data science.

“Long story short, I needed a job, and I moved into data science,” she said in an interview at the conference about how she decided to make the switch from physics. “I think physics has a lot in common with data science already.”

To Shivamoggi, there’s almost a straight line between a quantitative science like physics and data science.

“A lot of the focus is on modeling so it’s quantitative, it’s mathematical,” she said, “but also we know what we produce inherently is an approximation. And that’s true in physics as well. We don’t ever get the full picture, but we’re trying to figure out what’s the simplest model that will still answer our questions.”

Data science: Just like research?

Jess Stauth, managing director at Fidelity Labs, said in a presentation at the conference that a history of scientific research could help data scientists excel in their roles.

Stauth pointed out that theoretical or research-based sciences and applied sciences, such as data science, are often viewed as two entirely separate things.

“These are two different things, but I think it’s a disservice to separate them so strongly,” Stauth said. She added that there’s a strong value in teams with diverse skills working together. A person with a more general science background can bring valuable skills to this kind of team.

Stauth has worked in data science teams in startup companies. She said that in a startup atmosphere data scientists are asked to wear many different hats and sometimes step into several roles. That’s why having a diverse data science background can be an advantage for people pursuing the discipline as a career.

“A data scientist can also be an analyst or a business user — those don’t have to be separate,” Stauth said. “They’re two different hats that you have to wear.”

The post Varied experience builds strong data science background appeared first on Artificial Intelligence.

It has always been hard for a human to evolve with rising trends yet it’s in their nature. With the commencement of the fourth industrial revolution, we noticed that people are becoming more and more agile to embrace new technologies instead of fearing the rigorous changes they would have to go through. Such a welcoming gesture and enhance acceptability has made large scale innovations possible. In the case of automation and data-technologies, businesses and their humanly assets have been sport to incorporate and excel with such advancements. Automation and data science has always been complementary to each other. The former is a must for driving data-enabled decisions and culture as well. Automation unlocks new edges of data that were not explored before.

However, not all businesses are keen to understand the potentials of automation in delivering more value to data science projects. Nick Elprin, CEO and Co-Founder of Domino Data Lab, said, “Sixty percent of companies plan to double the size of their data science teams in 2018. Ninety percent believe data science contributes to business innovation. However, less than 9% can actually quantify the business impact of all their models, and only 11% can claim more than 50 predictive models working in production.”

Ryohei Fujimaki, Ph.D., founder, and CEO of dotData said, “We’ve seen studies that report only 4% of companies successfully implement business intelligence (BI) and artificial intelligence (AI). dotData is a company that focuses on data science automation for enterprises. “It naturally makes you wonder what the other 96% are doing,” he added.

“There is a great deal of business interest in this,” said Fujimaki. “Data science is key to business growth if you can unlock its potential. You can predict new products and costs, and even customer churn. The insights that data science can generate cuts across all industries, whether it is pharma, aerospace, manufacturing, retail, finance, or other.”

However, the problem lies with the work is that companies take an average of two to three months to complete a single data science project.

“Data science is difficult for enterprises because it requires an interdisciplinary team to be successful,” said Fujimaki. “First, you have company ‘domain experts’ who know particular areas of the business and can assist in defining important business use cases. Data science talent is also difficult to hire. Then, you have to collect, clean, and prepare data, which can consume more than 80% of the project time. You then must define different data models, algorithms and visualizations and try them out in an iterative mode, knowing that not all of them will work. Finally, when you get a strong project that meets a business case, you have to migrate the project into production. This often impacts business processes.”

As the entire process becomes too time-consuming, to achieve a successful AI project, a number of companies are migrating towards adding machine learning to get even more out of the initial AI work. However, adding machine learning can take another 20 to 30 percent of project time. “Again, you must continually test and retest, to ensure that data is accurate and that you are realizing your business case objectives,” said Fujimaki. And this where automation enters the picture.

In order to go beyond the AI and ML techniques with a fast pace, companies can further automate ML processes. “With this capability, you still need business domain experts, data scientists, and engineers, but you can automate many of the statistical and mathematical operations of data science,” said Fujimaki. “This makes data science more sustainable in organizations, and it enables companies to cover more ground because they can provide data science products faster.

He further continued, “There are many elements in this process, but data science automation can help. In addition to enabling your enterprise to complete more data science projects and get products to market sooner without having to do all of the data science operations yourself, a kind of ‘democratization’ of data science begins to occur in organizations. Now, many people who might be business domain specialists can also use automation without having to become full-blown data scientists.”

The post ACHIEVING SUCCESSFUL DATA SCIENCE PROJECTS WITH AUTOMATION appeared first on Artificial Intelligence.

There is a £2 million opportunity for new health data science partnership and projects to improve patient care as organisations across the UK are invited to contribute to a national programme that aims to enable better patient care through responsible use of health data used in health and care decisions.

Health Data Research UK – the national institute for health data science – is launching the opportunity that aims to help patients across the UK to benefit from health and care decisions that are informed by analysing large scale data. This work includes up to four projects and a brand-new partnership that will integrate with Health Data Research UK’s existing research activities.

Health Data Research UK is seeking a Better Care Partnership to lead a new data science initiative, which could receive up to £1.2 million. Initially funded for a period of three years, the Better Care Partnership will use continuous improvement methods to integrate clinical practice, large scale health data and advanced analytics to deliver insights for improving care for patients across the UK.

In addition, Health Data Research UK will also be supporting one year Catalyst Projects to demonstrate examples of how patient care can be improved through data-driven health and care decisions. Health Data Research UK is teaming up with the Health Foundation to jointly support these 12-month projects, which could each receive up to £200,000.

Organisations putting forward new partnership and project opportunities will be expected to demonstrate how they plan to listen to patients and understand their wishes about how their health data will be used. They will also be expected to show how patients will be involved at all stages of the project or partnership. The aim is to use health data responsibly and ethically with a clear focus on improving patient care.

Professor Simon Ball, Medical Director at University Hospitals Birmingham NHS Foundation Trust, and National Lead for Health Data Research UK’s Better Care priority, said: “As healthcare professionals we make hundreds of decisions a week with our patients. In doing so we aim to decide what will work best for each individual. Electronic healthcare records offer the opportunity to combine patients’ data with information on best practice, so that we can reliably deliver high quality care in complex settings and pressured environments. Beyond that we can use the resulting data on patients’ outcomes and experience, to continuously learn from, and improve on, everyday practice in ways that are applicable across the NHS.”

Every day doctors, nurses and patients in GP practices and hospitals across the UK are making countless decisions that impact on people’s health and care. These decisions are primarily based on the knowledge of the individual clinician and patient, relevant medical research and the data they can access. The UK has vast and rich data about people’s health and care, however this is often not available quickly for clinicians or patients to access to support their decision making. This causes delays and, in some cases, prevents the data from being analysed to deliver better care and improve people’s health. Both the Catalyst Projects and the Better Care Partnership, which will start in May 2020, aim to help address some of these issues.

The closing date for applying for the Catalyst and Better Care Partnership is 11 Mar 2020 at 4pm.

The Health Data Research UK’s website has more information about the Better Care Programme and details of this opportunity.

The post Opportunity for new data science partnership and projects appeared first on Artificial Intelligence.