What’s the best way to help your city thrive after a global pandemic? For Mark Gannon, director of business change and information solutions at Sheffield City Council, the answer might lie in big data.

“We want to give the organisation and the city the greatest opportunity to bounce back from COVID. Part of that focus is around using digital and data,” he says.

Gannon aims to create an organisation that’s driven by data. Working alongside counterparts in other public sector organisations, he wants to combine public data sources and create new insight into some of Sheffield’s most-pressing health and wellbeing issues, helping the city’s councillors make timely and cost-effective decisions in a post-COVID age.

“So when we make policy decisions, they’re based on intelligence rather than gut instinct. We’re a political organisation, so obviously there are going to be politicians that will always have views. But I think if we can inform those views with data and intelligence, that will be good,” he says.

The council already has a small business intelligence (BI) team that Gannon says is “doing some really clever stuff”, such as helping social workers to maximise their engagement with citizens to understand the complexity of cases. The aim right now is to think about how big data can create further insight-led improvements for people across the city.

A crucial element will be the establishment of an Office of Data Analytics (ODA) for Sheffield, a cross-city initiative that aims to draw in information from a range of institutions, including health and care organisations, the police, and the city’s two higher education establishments, the University of Sheffield and Sheffield Hallam University.

“We want to use all the enthusiasm, data and intelligence that we have to create a gearing effect from that combination to get more than the sum of its parts. So that’s a really exciting conversation and it means we’ve got an opportunity to do something in our response to COVID that could actually move the city forward,” he says.

The ODA will be research-led: while a physical office might follow at some stage, the project will operate – like so many other workplaces right now – as a virtual office.

Gannon is helping to develop the terms of reference for the ODA, while his team puts together some candidate projects. The aim is to identify these projects through workshops that can help to create benefits for the city and its citizens.

He gives the example of the Urban Flows Observatory, which is a massive data-gathering project at the University of a Sheffield that draws information from sensors across the city. The initiative collects information, such as data on energy use, climate change and pollution, that relate to the physical process that take place within the city.

Gannon says the council is talking to the university about how it might combine this data with its own information on social concerns, such as inequality and health and wellbeing. The aim is to create new insight on the urban area of Sheffield.

“So there’s various conversations taking place and, because of COVID, the need to have quick access to data and intelligence is really driving a conversation where people get to see why this insight so important,” he says.

Gannon and his public sector peers in Sheffield are keen to make the ODA work on its own terms – and that means open data and application programming interfaces (APIs). While the council has had offers from external providers that are keen to sell their data products, Gannon says the aim is to not become reliant on proprietary tools.

Rather than create data lakes of stored information, the ODA – and the council’s other data initiatives – will aim to collect and analyse information once its data scientists understand the questions that need to be answered. Gannon hopes this open approach will create projects that deliver data-led benefits that people can believe in.

“I think there’s a massive trust issue with this stuff,” he says. “People tend to not trust government with their data, so we’re going to spend quite a bit of time upfront on the public trust element. The aim is to do everything in the open, so we publish all of our plans, working-out, algorithms; everything we use, we’ll make it public so people can see what we’re doing.”

Sheffield’s desire to make the most of its information should be welcomed. Data science is still at a nascent stage in UK local government, according to University of Oxford researchers. They suggest that there is enormous potential for the use of big data to be expanded and to help with the delivery of better services to citizens.

More efficient forms of service delivery will be critical in a post-COVID age, with estimates suggesting UK councils will face significant funding cuts due to a £1.2bn hole in their finances from the coronavirus pandemic. Gannon recognises the scale of the challenge ahead and says big data can help the council to understand the value of its interventions.

“The issue for local government and public sector is going to be budgets – they’re going to be significantly reduced,” he says. “So we’re going to have to think about how we make sure that when we do intervene, that that’s going to have the impact on key areas like health and social care.”

Gannon gives the example of identifying and then tracking vulnerable adults through the health and social-care system. Information is often held in stove pipes by different public sector organisations, which means tracing an individual’s journey – and the impact of interventions – is difficult in normal times, never mind during a global pandemic.

“The bit that’s often missing is about understanding the impact of your work on vulnerable adults. So we’ve got a prevention strategy, but what we don’t have is the data to demonstrate that the prevention is having an impact. We want to track the interventions that we’ve made, and then tweak the interventions as required,” he says.

The post Big data has a trust issue. This city wants to take a smarter approach appeared first on Artificial Intelligence.

In this special guest feature, Matthew Mahowald, Lead Data Scientist and Software Engineer for Open Data Group, shares his perspectives on how the speed at which tech and tools have been developed, has caused problems with the way analytic deployment is made possible. Matthew holds a Ph.D. in Mathematics from Northwestern University, with a focus on the geometry of string theory and topological field theory. At Open Data Group, Matthew focuses on developing and deploying machine learning models with FastScore.

The history of predictive analytics might be said to begin with Bayes’ famous theorem relating the conditional probabilities of two events.  Even today, the importance of foundational work like Bayes’ theorem cannot be overstated: it is both the basis for most significance tests across the experimental sciences, as well as a useful tool in its own right for assessing correlation.

P(A|B)=P(B|A)P(A)/P(B)

In recent years, as the sultans of Silicon Valley have pressed both computation speeds and data storage capacities to dizzying heights, researchers and analysts working at the intersection of statistics and computer science have leveraged new tools to chase increasingly sophisticated modeling techniques. This dramatic expansion in both software tools and, especially, the quantity and quality of data available led to the emergence of data science as a discipline, and most important the assets created by a data science teams: predictive analytic models.

However historically, when it was time to deploy a new predictive analytic model into production, the burden of deployment on IT and the production pipeline was fairly minor. Long lead times meant that each model could be manually restructured (and sometimes even translated into another programming language). Moreover, the comparative simplicity of the models themselves meant that this recoding was not unreasonably labor-intensive.

The proliferation of tools and techniques in data science have not changed the fundamental deployment problem. However, the complexity of the models strains the feasibly of traditional deployment methodology. There are now more than 10,000 open-source packages on CRAN (the global R package repository). With open-source projects like Scikit-Learn and Pandas, Python offers similarly comprehensive support. Today’s vast data science environment has the ability to construct a wider variety of models faster, at lower cost, and leveraging more data than ever before.

The trend has seeped into the speed at which analytic models are being built.  What used to be a leisurely build, with a small number of fairly simplistic rules-based or linear-regression models each year, has turned into the creation of dozens of complex models leveraging the latest and greatest gradient boosting machine or convolutional neural net toolsets. As a result, the traditional model deployment process becomes simply unsustainable.

So, what’s the solution? It’s imperative that everyone – from IT professionals to data scientists – understand and address the challenges of analytic deployment in the modern era. One way to ensure that an enterprise is making analytic deployment a core competency is with an analytic deployment engine. To find success, such an engine would have properties like:

• Ensuring it’s a software component that sits in the production data pipeline, where it receives and executes models.
• It provides native support (without recoding) for any modeling language or package, that is, the engine is language agnostic.
• It can connect to any data source or sink used in the production data pipeline.

This engine should be simultaneously easy enough to use that the data science team can validate and deploy models without requiring IT involvement, and sufficiently robust and scalable that it can be used with confidence in the production pipeline.

Finally (and most importantly), an analytic deployment engine should be future-proof: new libraries and packages in R and Python shouldn’t require upgrading the engine, nor should the emergence of other new techniques and tools.

As organizations continue to gather massive data sets and develop more advanced analytic models to extract value, the number of barriers that are being encountered continue to pile up. By having the right set of data science tools that focus on analytic deployment technology, the IT and Analytics teams can find that sweet spot of success to drive ROI for their businesses.

The post The Proliferation of Data Science Tools & Technology appeared first on Artificial Intelligence.

In between meeting with customers, crowdchatting with our communities and hosting theCUBE, the research team at Wikibon, owned by the same company as SiliconANGLE, finds time to meet and discuss trends and topics regarding digital business transformation and technology markets. We look at things from the standpoints of business, the Internet of Things, big data, application, cloud and infrastructure modernization. We use the results of our research meetings to explore new research topics, further current research projects and share insights. This is the fifth summary of findings from these regular meetings, which we plan to publish every week. This week’s meeting included Dave Vellante and Jim Kobielus offering insights on serverless computing.

Premise: Serverless computing is coming fast and furious to the cloud world, bringing many advantages. In particular, with serverless, developers don’t have to manage complex infrastructure typically associated with containers, virtual machines and other underlying infrastructure. With serverless, developers build applications using functional programming languages and tools which we believe will largely be a complement to, not a replacement for, traditional programming models. The latter, in our opinion, will remain in vogue for stateful enterprise applications while serverless models will increasingly address stateless apps.

What exactly is serverless computing?

Serverless computing is a cloud-oriented operating model that dynamically manages underlying infrastructure resources.  Serverless is typically deployed as a functional microservices architecture that allows developers to invoke functions as they’re needed and pay for resources based on what’s consumed by an application versus paying for fixed units of capacity.

Serverless still requires hardware and the name is somewhat misleading, but the management of the infrastructure resource is essentially “invisible” to application developers. Specifically, in serverless environments, developers don’t have to define the attributes of the servers. The infrastructure that supports invoked services is managed by the cloud provider and developers don’t need to know what’s sitting behind the functions. Serverless can be thought of as completely preconfigured functions-as-a-service where pricing for the functions is utilitylike and paid for by consumption at some interval of granularity, for example hours, minutes or seconds.

What are the benefits?

Serverless architectures are much simpler for application developers to manage. Serverless virtually eliminates the responsibility to maintain software, microcode, operating system levels and the like, and developers need only worry about developing and testing a function-based offering. As such, serverless architectures are highly scalable and potentially much less expensive platforms on which to develop and maintain applications. As a result, the compute fabrics that support serverless can be exceedingly efficient and cost effective.

What are the main use cases?

The main use cases for serverless are stateless applications and functional programming models. Examples include application programming interface publishing, query response, face recognition and voice recognition; these are typical for stateless apps using functional programming models.

Edge-oriented environments are another emerging use case for serverless ,omputing. As edge devices capture data on certain events — for example, an Internet of Things device emitting some data over time — the device platform can call functions or a model or logic service to perform some real-time analysis and make an on-the-fly adjustments, such as increasing or decreasing flow. Notably, we believe the serverless model will be used extensively for edge applications, even those that are end-to-end, as long as these applications are stateless. Stateful applications are likely to use more traditional models for some time.

We also view certain data analytic workloads such as business intelligence and high-performance computing use cases — for example, climate modeling, genomics and basic scientific research — as potentially good candidates for serverless.

Where did serverless come from?

Serverless is a relatively immature space. Amazon Web Services Inc. announced Lambda in 2014 as the industry’s first serverless offering. Other clouds vendors have followed suit, including Google Inc. with Cloud Functions, Microsoft Corp. with Azure Functions and IBM Corp. with Bluemix OpenWhisk.

What are the key caveats for developers?

Serverless environments today run in a shared cloud environment, so this means there will be peaks, valleys and competition for resources. As such, developers must be manage unexpected situations as they arise, especially those related to latency and error recovery. Users of serverless computing models must do rigorous testing in this new environment and focus on recovery, for example how to deal with timeouts. As well, practitioners should expect that service level agreements from cloud providers will be less rigorous with serverless than with stateful apps, at least for now.

Also, by deploying multiple serverless cloud offerings, organizations can be exposed to “serverless creep.” Just as spinning up virtual machines and using containers extensively has created challenges for organizations, development managers must be sensitive to an explosion of serverless apps. In our view, customers must be wary of getting to a point where they lose track of what’s being developed within the application portfolio, a probability precisely because of the lack of state. The risks here include compliance and audit challenges, duplicative work products and cost overruns. Moreover, different clouds will support different functional languages, such as Javascript vs. Python, and serverless apps may not be very portable to other clouds. This brings up a potential issue of diluting skill sets across an organization where the cloud choice wags the skills dog, versus a more deliberate and well-thought-out people and process strategy.

Where do containers and platform-as-a-service fit?

Serverless computing leverages containers as the underlying infrastructure. Serverless allows developers to essentially abstract away the core container complexity. Platform-as-a-service is a microservices environment by its very nature. Containerized microservices require management by developers, whereas the functional microservices associated with serverless abstract that complexity — assuming the cloud provider is doing its job.

Action Item: Serverless is an emerging and highly useful concept for developers of cloud-based services, and Wikibon believes that it’s a fundamental operating model that’s here to stay. Developers should begin using serverless and start with simple use cases. In particular, we advise embracing stateless functions such as Web content publishing, API notification and alerts, and other event-driven applications. However, developers must be careful to consider recovery plans in these new environments. As always, hope for the best, plan for the worst.

The post Serverless computing: It’s all about functional stateless microservices appeared first on Artificial Intelligence.