As the volume of machine-generated data continues to grow, organizations require effective tools to collect, process, and analyze this data in real-time. Logstash is a powerful open-source data collection and processing tool that serves as a core component of the Elastic Stack. It enables organizations to ingest, parse, and transform data from a variety of sources, making it a vital tool for log management, analytics, and observability.

Logstash plays a crucial role in modern IT operations, security analytics, and business intelligence. By acting as a pipeline that collects, enriches, and routes data, Logstash ensures that organizations can make better use of their data, improving decision-making and operational efficiency.

What is Logstash?

Logstash is an open-source data processing pipeline designed to collect, process, and forward data to various storage and analysis tools, such as Elasticsearch, Amazon S3, or other databases. It allows users to ingest data from diverse sources, transform the data into a usable format, and export it to a destination for further analysis or visualization.

Logstash is highly extensible, with a rich library of plugins that enable integration with multiple input sources, data processing filters, and output destinations. Its flexibility makes it a preferred choice for handling logs, metrics, events, and other types of data from servers, applications, network devices, and more.

Top 10 Use Cases of Logstash

1. Centralized Log Management
   Collect and process logs from multiple systems, applications, and devices into a central repository for easier analysis.
2. Application Performance Monitoring (APM)
   Track application logs and metrics to monitor performance, identify bottlenecks, and optimize user experience.
3. Security Information and Event Management (SIEM)
   Enrich and forward logs to security tools to detect, analyze, and respond to security incidents.
4. Infrastructure Monitoring
   Gather metrics from servers, network devices, and containers to monitor system health and performance.
5. IoT Data Processing
   Ingest and process data from IoT devices, enabling real-time analytics and operational insights.
6. Data Enrichment
   Enhance raw log data with additional context, such as geolocation or user agent parsing, for better insights.
7. Event Correlation
   Aggregate logs from distributed systems to identify patterns and correlations that point to root causes of issues.
8. Cloud Monitoring
   Process logs and metrics from cloud platforms like AWS, Azure, and Google Cloud to ensure optimal performance and cost efficiency.
9. Compliance Reporting
   Collect and normalize logs to meet regulatory compliance requirements, such as GDPR, HIPAA, and PCI DSS.
10. Business Analytics
    Ingest and transform data from sales, marketing, and customer engagement platforms for actionable business insights.

What Are the Features of Logstash?

1. Wide Input Source Support
   Logstash supports numerous input sources, including Syslog, Beats, HTTP, TCP, Kafka, and databases.
2. Flexible Data Processing
   Use filters to parse, enrich, and transform data, such as grok patterns for log parsing or GeoIP for geolocation enrichment.
3. Extensive Plugin Ecosystem
   Choose from over 200 plugins to customize input, filter, and output stages for specific use cases.
4. Real-Time Data Processing
   Process and forward data in real time, ensuring up-to-date insights for monitoring and analytics.
5. Integration with Elastic Stack
   Seamlessly integrate with Elasticsearch and Kibana for storage, search, and visualization.
6. Scalability and High Performance
   Handle large volumes of data efficiently, scaling horizontally by deploying multiple Logstash instances.
7. Rich Event Metadata
   Include metadata such as timestamps, source information, and pipeline stages for better event context.
8. Error Handling
   Handle failed data processing gracefully by using dead letter queues or routing problematic events for further inspection.
9. Support for Structured and Unstructured Data
   Process JSON, XML, CSV, and unstructured text data, making it versatile for different use cases.
10. Open-Source and Extensible
    Customize and extend Logstash’s functionality using community plugins or custom code.

How Logstash Works and Architecture

How It Works:
Logstash operates as a pipeline with three main stages: Input, Filter, and Output. Data flows through these stages, where it is collected, processed, and sent to the desired destination.

Architecture Overview:

1. Input Stage:
   Collect data from various sources such as log files, databases, or message queues. Inputs define where the data originates and how it enters Logstash.
2. Filter Stage:
   Transform and enrich data using filters like grok (pattern matching), mutate (data modification), and GeoIP (geolocation enrichment).
3. Output Stage:
   Send processed data to destinations like Elasticsearch, S3, or other storage and analysis systems.
4. Plugins:
   Logstash uses plugins for inputs, filters, and outputs, making it flexible to handle diverse data pipelines.
5. Pipeline Management:
   Define multiple pipelines for different use cases, enabling parallel processing of diverse data streams.

How to Install Logstash

Steps to Install Logstash on Linux:

1. Update Your System:

sudo apt update
sudo apt upgrade

2. Install Java:
Logstash requires Java. Install it using:

sudo apt install openjdk-11-jdk

3. Add the Elastic Repository:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update

4. Install Logstash:

sudo apt install logstash

5. Configure Logstash:

• Edit the pipeline configuration file:

sudo nano /etc/logstash/conf.d/logstash.conf

• Example configuration:

input {
  beats {
    port => 5044
  }
}
filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
  }
}

6. Start Logstash:

sudo systemctl start logstash
sudo systemctl enable logstash

7. Test Logstash:

• Send sample data to the configured input and check Elasticsearch or other output destinations for processed logs.

Basic Tutorials of Logstash: Getting Started

1. Creating a Simple Pipeline:

• Define an input (e.g., reading logs from a file), apply a filter (e.g., parsing logs with grok), and set an output (e.g., sending logs to Elasticsearch).

2. Using the Grok Filter:

• Use grok patterns to extract meaningful data from log entries:

filter {
  grok {
    match => { "message" => "%{COMMONAPACHELOG}" }
  }
}

3. Testing Pipelines:

• Test pipelines locally using:

echo '{"message": "Test log entry"}' | /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf

4. Handling Multiple Pipelines:

• Configure multiple pipelines in /etc/logstash/pipelines.yml for processing different data streams.

5. Integrating with Beats:

• Use Filebeat to ship logs to Logstash:

filebeat.inputs:
  - type: log
    paths:
      - /var/log/*.log
output.logstash:
  hosts: ["localhost:5044"]

6. Monitoring Logstash:

• Enable monitoring features to track pipeline performance and troubleshoot bottlenecks.

The post What is Logstash and Its Use Cases? appeared first on Artificial Intelligence.

Managing and analyzing data efficiently is vital in today’s data-driven environment, where logs, metrics, and events from systems and applications are constantly generated. The Elastic Stack, formerly known as the ELK Stack, is a suite of open-source tools designed to help organizations collect, process, store, analyze, and visualize large volumes of data in real-time. It is built around four core components: Elasticsearch, Logstash, Kibana, and Beats, each serving a specific role in the data pipeline.

Elastic Stack provides an end-to-end solution for observability, search, and analytics. It is widely used for log management, infrastructure monitoring, application performance tracking, and security analytics. Its scalability and flexibility make it an indispensable tool for DevOps, IT operations, and data engineering teams, empowering them to gain actionable insights from their data.

What is Elastic Stack?

The Elastic Stack is an open-source platform that enables you to collect data from diverse sources, transform it into structured formats, and analyze it for actionable insights. It is comprised of the following tools:

• Elasticsearch: A distributed search and analytics engine designed for fast and scalable indexing, querying, and analysis.
• Logstash: A data pipeline that ingests, processes, and transforms raw data before sending it to Elasticsearch.
• Kibana: A visualization and analytics platform that provides dashboards, charts, and reports for analyzing data stored in Elasticsearch.
• Beats: Lightweight data shippers that send data from edge devices to Logstash or Elasticsearch.

Elastic Stack allows organizations to monitor their infrastructure, analyze logs, secure systems, and extract business intelligence from their data. Its ability to handle petabytes of data in real-time makes it a preferred choice for enterprises and startups alike.

Top 10 Use Cases of Elastic Stack

1. Centralized Log Management
   Elastic Stack excels at aggregating logs from multiple systems and applications into a central repository, making it easy to search, analyze, and troubleshoot issues.
2. Application Performance Monitoring (APM)
   Monitor application performance metrics, such as response times, error rates, and transaction volumes, using Elastic APM integrated with Elastic Stack.
3. Security Analytics
   Use Elastic Security to detect, investigate, and respond to security threats, such as unauthorized access and data breaches.
4. Infrastructure Monitoring
   Gain visibility into your servers, networks, and containers by collecting metrics and events from your infrastructure.
5. Real-Time Anomaly Detection
   Leverage machine learning capabilities to identify anomalies in system behavior, helping to predict and prevent potential issues.
6. Business Intelligence
   Analyze business metrics, such as sales trends or customer interactions, by visualizing data in custom dashboards.
7. DevOps Observability
   Track system performance and application health across CI/CD pipelines, Kubernetes clusters, and microservices architectures.
8. E-Commerce Search Optimization
   Power search functionality for e-commerce platforms by indexing product catalogs in Elasticsearch and providing fast, relevant results.
9. IoT Data Analysis
   Collect, process, and analyze data from IoT devices for insights into device health, usage patterns, and operational efficiency.
10. Compliance and Audit Logging
    Store and analyze logs for regulatory compliance, ensuring that your systems adhere to industry standards and guidelines.

What Are the Features of Elastic Stack?

1. High-Performance Search and Indexing
   Elasticsearch provides lightning-fast search and indexing capabilities, even for massive datasets.
2. Real-Time Data Ingestion
   Logstash and Beats enable the ingestion of data from various sources in real time, ensuring that insights are always up to date.
3. Customizable Dashboards
   Kibana allows users to create interactive dashboards and visualizations tailored to their specific needs.
4. Scalable Architecture
   Elastic Stack is built for scalability, allowing organizations to handle growing datasets by adding more nodes to the cluster.
5. Multi-Source Data Collection
   Beats can collect data from logs, metrics, network packets, and other sources, providing a comprehensive view of system performance.
6. Machine Learning
   Built-in machine learning features allow for anomaly detection, forecasting, and predictive analytics.
7. Security Features
   Elastic Security offers role-based access control (RBAC), encryption, and intrusion detection to secure your data.
8. Integration Ecosystem
   Seamlessly integrates with third-party tools like Grafana, Prometheus, and Kubernetes for extended observability.
9. Role-Based Access and Control
   Define access permissions for different users and teams to secure sensitive data.
10. Rich Query Language
    Elasticsearch supports complex queries, including full-text search, filtering, and aggregation, to retrieve meaningful insights from data.

How Elastic Stack Works and Architecture

Elastic Stack follows a modular architecture where each component plays a distinct role in the data lifecycle:

1. Beats (Data Collection):
   Beats are lightweight agents that collect data from edge devices, such as logs, metrics, and network packets, and ship them to Logstash or Elasticsearch.
2. Logstash (Data Processing):
   Logstash acts as a data pipeline, ingesting raw data, transforming it into structured formats, and forwarding it to Elasticsearch.
3. Elasticsearch (Data Storage and Search):
   Elasticsearch indexes and stores the data, enabling efficient search, analysis, and querying.
4. Kibana (Visualization and Analysis):
   Kibana provides a user-friendly interface for visualizing data through dashboards, charts, and graphs, as well as managing alerts and machine learning models.
5. Security and Observability Layers:
   Elastic Security and observability features add an additional layer of monitoring and protection, ensuring data integrity and compliance.

How to Install Elastic Stack

Steps to Install Elastic Stack on Linux:

1.Install Elasticsearch:

• Download Elasticsearch from the official website.
• Install it using:

sudo apt update
sudo apt install elasticsearch

• Start the Elasticsearch service:

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

2.Install Logstash:

• Download and install Logstash:

sudo apt install logstash

• Configure Logstash by creating a pipeline configuration file (/etc/logstash/conf.d/logstash.conf).

3. Install Kibana:

• Install Kibana for data visualization:

sudo apt install kibana

• Start the Kibana service:

sudo systemctl start kibana
sudo systemctl enable kibana

4. Install Beats (Optional):

• Install Filebeat for log collection:

sudo apt install filebeat

• Configure Filebeat to send data to Logstash or Elasticsearch.

5. Access Kibana Dashboard:

• Open your browser and navigate to http://<your_server_ip>:5601 to access the Kibana interface.

6. Test the Setup:

• Generate sample data and verify that it flows from Beats to Elasticsearch and is visualized in Kibana.

Basic Tutorials of Elastic Stack: Getting Started

1. Configuring Filebeat for Log Collection:

• Define the log paths in filebeat.yml and test the configuration:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/*.log

• Start Filebeat:

2. Creating a Logstash Pipeline:
Define input, filter, and output in the pipeline configuration file:

input {
  beats {
    port => 5044
  }
}
filter {
  grok {
    match => { "message" => "%{COMMONAPACHELOG}" }
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

3. Exploring Data in Kibana:

• Create an index pattern to visualize data stored in Elasticsearch.
• Build custom dashboards to monitor logs, metrics, or application traces.

4. Enabling Machine Learning:
Use Kibana’s machine-learning capabilities to set up anomaly detection for your data streams.

The post What is Elastic Stack and Its Use Cases? appeared first on Artificial Intelligence.

In today’s complex IT environments, managing and resolving incidents quickly is critical for maintaining service reliability and reducing downtime. Moogsoft is a leading AIOps (Artificial Intelligence for IT Operations) platform that helps organizations streamline incident management, reduce alert noise, and ensure proactive issue resolution. By leveraging AI and machine learning, Moogsoft provides actionable insights and automates key operational tasks to enhance efficiency and performance.

Moogsoft integrates seamlessly with monitoring tools, cloud platforms, and ITSM systems, making it a cornerstone of modern IT operations. With its real-time anomaly detection, event correlation, and intelligent noise reduction capabilities, Moogsoft empowers IT teams to manage complex infrastructures effortlessly.

What is Moogsoft?

Moogsoft is an AIOps and observability platform designed to simplify IT operations by automating incident management and improving system reliability. It uses AI and machine learning to correlate events, identify root causes, and provide actionable insights, enabling teams to resolve issues faster and more efficiently.

Built to handle the complexities of modern IT environments, Moogsoft reduces alert fatigue by filtering noise from monitoring systems, helping teams focus on what truly matters. Whether you’re managing cloud-native applications, hybrid infrastructures, or on-premises systems, Moogsoft ensures proactive, data-driven operations.

Top 10 Use Cases of Moogsoft

1. Noise Reduction
   Moogsoft uses machine learning to filter and deduplicate alerts, reducing noise and ensuring teams focus only on actionable incidents.
2. Event Correlation
   Automatically correlates related events across tools and systems to provide a unified view of incidents.
3. Anomaly Detection
   Identifies unusual patterns or behaviors in your systems before they escalate into critical issues.
4. Root Cause Analysis
   Pinpoints the root cause of incidents quickly, minimizing downtime and improving mean time to resolution (MTTR).
5. Cloud Monitoring
   Provides end-to-end visibility for cloud-native environments, ensuring optimal performance of cloud services.
6. Multi-Tool Integration
   Integrates with monitoring tools, ITSM platforms, and DevOps solutions, consolidating data for better insights.
7. Proactive Issue Resolution
   Detects and resolves issues proactively to prevent service disruptions.
8. Dynamic Topology Mapping
   Maps dependencies between services and infrastructure components in real time, aiding in incident analysis.
9. IT Operations Automation
   Automates workflows for ticket creation, escalation, and resolution, streamlining IT operations.
10. Service Reliability Improvement
    Enhances system reliability by providing actionable insights, enabling continuous improvement in IT operations.

What Are the Features of Moogsoft?

1. AI-Powered Noise Reduction
   Automatically filters out irrelevant alerts, ensuring only critical incidents are escalated to teams.
2. Real-Time Event Correlation
   Links related events across different systems to provide a comprehensive view of incidents.
3. Advanced Anomaly Detection
   Uses machine learning to detect anomalies and unusual patterns in system behavior.
4. Intelligent Automation
   Automates repetitive tasks such as alert routing, ticket creation, and escalation processes.
5. Unified Observability
   Provides a centralized platform for monitoring and managing alerts from multiple sources.
6. Root Cause Analysis
   Leverages AI to identify the primary cause of incidents quickly and accurately.
7. Integration Ecosystem
   Connects with popular monitoring tools like Datadog, Splunk, and Nagios, as well as ITSM platforms like ServiceNow.
8. Dynamic Dashboards
   Offers customizable dashboards with real-time insights into system performance and incident trends.
9. Collaboration Tools
   Facilitates collaboration with built-in tools for incident resolution and knowledge sharing.
10. Scalability
    Designed to handle large-scale, complex IT environments, making it suitable for enterprises.

How Moogsoft Works and Architecture

How It Works:
Moogsoft processes data from monitoring and observability tools, applying AI algorithms to correlate events, reduce noise, and detect anomalies. It provides actionable insights through dashboards and automated alerts, enabling teams to resolve issues faster and more efficiently.

Architecture Overview:

1. Data Ingestion Layer: Collects data from monitoring tools, logs, and metrics.
2. AI Engine: Applies machine learning to filter noise, correlate events, and detect anomalies.
3. Event Correlation Engine: Identifies relationships between events to streamline incident analysis.
4. User Interface: Dashboards and collaboration tools provide actionable insights and facilitate resolution.
5. Integration Layer: Connects with ITSM platforms, monitoring tools, and cloud services.

How to Install Moogsoft

Steps to Install Moogsoft:

1. Sign Up: Create an account on the Moogsoft platform or contact their sales team for enterprise plans.
2. Deploy Moogsoft: Choose between cloud-based or on-premises deployment options.
3. Integrate Monitoring Tools: Connect Moogsoft with tools like Datadog, Nagios, or Splunk using pre-built integrations.
4. Configure Alert Rules: Set up alert thresholds, routing rules, and automation workflows.
5. Test the System: Simulate alerts and monitor how Moogsoft processes and escalates incidents.

Basic Tutorials of Moogsoft: Getting Started

1. Setting Up Integrations
   Connect Moogsoft with your existing monitoring tools and ITSM platforms using its integration library.
2. Creating Alert Rules
   Define rules for alert prioritization, routing, and escalation based on your organizational needs.
3. Using Dashboards
   Customize dashboards to monitor key performance indicators, trends, and incident statuses in real-time.
4. Analyzing Incidents
   Use Moogsoft’s AI-powered insights to perform root cause analysis and resolve issues efficiently.
5. Automating Workflows
   Set up automated workflows for repetitive tasks such as ticket creation and incident escalation.

The post What is Moogsoft and Its Use Cases? appeared first on Artificial Intelligence.

We examine how monitoring and observability help development teams keep a distributed architecture from coming unraveled by individual failures and performance bottlenecks.

Failure is rarely predictable, and detecting the exact cause of complex application errors post-deployment is excruciatingly difficult. Even the most experienced development teams struggle to prepare for all the possible scenarios that could bring down their applications and put data at risk.

For this reason, the ability to detect problems in real time and address them quickly is essential. This is where observability and monitoring come into play, and architects who approach these two tasks diligently will reap the rewards of a more resilient software architecture. Let’s explore more about the specifics of observability and monitoring, including how they differ and the fundamental practices that each one dictates.

What is observability?

Observability in microservices largely revolves around making sure development teams have access to the data they need to identify problems and detect failures. For example, an observable system can help developers understand why a specific service call failed, or determine the source of bottlenecks in a particular application workflow.

With the surge in microservices adoption, it is imperative that a system is observable for effective debugging and diagnostics. Since services can span across multiple systems and run operations independently, tracing the source of a failure is a grueling and time-consuming task — if even possible.

Observability consists of three fundamental components:

• Logs are timestamped records that provide comprehensive information about an application’s behavior as it executes functions and communications. These logs are particularly useful when things go wrong in a microservices architecture, because architects can use this information to better identify specific defects and debug code.
• Metrics are numeric records of an application’s resource use, performance and stability. For example, metrics will show the number of requests a service can handle per second, or the total amount of resources an activity consumes.
• Traces keep track of IDs, names and other values and help architects monitor application transactions that cross multiple systems. This makes tracing particularly useful for microservices-based, serverless and containerized applications that rely on multitudes of integrations and asynchronous communication.

What is monitoring?

Monitoring is a process that tracks performance and identifies problems and anomalies. Overall, it describes the health, performance, efficiency and other essential features relative to the internal state.

Much like observability, monitoring can help detect and identify failures, but it does so with a focus on qualitative information. For example, you might want to monitor an application for issues such as excessive data consumption, service messaging failures or breaking changes. To use monitoring effectively, architects must determine core sets of metrics that provide a benchmark for the overall health of the system, such as acceptable latency times and call failure rates.

When monitoring microservices-based applications, architects must gain a comprehensive understanding of the various calls an application and its related services make. Don’t forget to monitor APIs and containerized services, and map monitoring processes and responsibilities based on team structure. Everyone should know who owns what service, and who needs to address a certain failure.

Microservices monitoring and observability tools

Some organizations try to adopt a manual, do-it-yourself approach to observability and monitoring by stringing homegrown monitoring solutions into their architecture. However, this takes up a lot of time, and is not likely to meet the needs of large, distributed systems.

Before attempting to do it yourself, you might want to look into existing tools designed to provide the essential aspects of monitoring and observability in microservices. Here are a few notable tools and platforms worth consideration.

Sentry

Sentry is an open source monitoring system designed with a focus on real-time, code-level error tracking that pinpoints failures and allows developers to address issues quickly. Part of Sentry’s appeal rests in its ability to analyze the scope of a failure, allowing developers to easily prioritize errors based on severity. It also features ready-made integrations with most popular development languages and frameworks, such as JavaScript, Python, Objective-C and iOS, as well as services like GitHub and Splunk.

Sensu

Sensu is another open source observability and monitoring tool that excels at monitoring applications, services, servers and containers deployed across large software ecosystems and cloud environments. Some of Sensu’s spotlight features include role-based service identification, its alignment with publish-subscribe messaging patterns and an interface that provides quick visuals of code workflows.

Sumo Logic

Thanks to this platform’s notable proficiency in data aggregation and analysis, Sumo Logic is a very useful tool for gleaning continuous metrics from application logs in real time and quickly spotting performance and stability issues in service workflows. Sumo Logic boasts a number of microservices-specific observability features, such as distributed tracing for services, transactions and application data.

The post The basics of monitoring and observability in microservices appeared first on Artificial Intelligence.

But the Oracle of this decade (and recent history) is more specifically all about data and information observability, autonomous control, analytics, diagnostics and we might just pepper in a double observability for good measure i.e. this is a company on a mission to cut through the risk of fragmented observance across what are increasingly complex multi-cloud environments.

Larry ‘did I tell you about my sailing career’ Ellison’s words are still ringing in our ears from the last time we attended Oracle Open World (Ed – ah yes, 2019)… during which the Oracle CTO and co-founder described how his firm had engineered a substantial proportion of autonomous intelligence into its core database offering (to perform everything from system patches to upgrades & maintenance and onwards to data preparation) and platform.

So what of Oracle 2020 in a year when (of course) the organisation’s main customer/user event has had to switch to virtual?

The company’s news this month sees it talk about the Oracle Cloud Observability and Management Platform.

Quite a mouthful, but one designed to explain the ‘bringing together’ of a set of management, diagnostic and analytics services to help eliminate the complexity, risk, and cost associated with today’s fragmented approach for managing multi-cloud work zones, spanning public, hybrid and on-premises environments as they do.

A unified view, just for you

Part of the Oracle Cloud Infrastructure (OCI) suite of services, OCI itself claims to provide a unified view across an entire software stack. 

“It enables easy diagnostics of cloud-native and traditional technologies deployed in the cloud or on-premises. With built-in machine learning, it automatically detects anomalies and enables quick remediation in near-real time. The platform has adopted an open, standards-based approach that is vendor-agnostic, supporting ecosystem interoperability out-of-the-box with Slack, Grafana, Twilio, PagerDuty and others,” notes Oracle, in a press statement.

Ellison and co contend that cloud environment complexity has grown, but monitoring and management tooling has generally not kept pace with that same complexity curve.

Wait for the payoff – yep – Oracle OCI with the new function actually DOES keep up, right?

Yes, that’s pretty much what the firm is saying, especially in light of the rise of a growing number of emerging technologies such as Kubernetes, containers, converged databases and microservices. 

Instead of a collection of siloed and fragmented tools, the Oracle Cloud Observability and Management Platform promises to provide a connected solution comprised of related services. 

What’s in the box?

This includes the newly announced Logging, Logging Analytics, Database Management, Application Performance Monitoring, Operations Insights and Service Connector Hub services, as well as existing services such as Monitoring, Notifications, Events, Functions, Streaming and OS Management. 

“Oracle has deep domain expertise in operating the largest portfolio of SaaS and enterprise application environments. We also manage the largest and most critical datasets for our customers, and we develop and operate on-premises infrastructure, unlike other cloud providers,” said Clay Magouyrk, executive vice president, Oracle Cloud Infrastructure.

Magouyrk insists that Oracle is eliminating the complexity and reducing the risks and costs associated with today’s multi-tool approach to make the overall management process more intuitive and cost-effective.

The integrated platform aggregates all observability data for holistic analysis and applies operations-optimised ML algorithms that can identify anomalous system behaviour, rapidly isolate and remediate performance problems and prevent outages by providing accurate forecasting of impending issues. 

This information is delivered in out-of-the-box and customer-designed dashboards with cross-tier views that provide complete visibility across applications, databases, infrastructure and cloud environments. 

One-click instrumentation & observability is now a thing, get used to the term.

The post Oracle Cloud: One-click instrumentation & observability is now a thing appeared first on Artificial Intelligence.

Australia Post’s adoption of microservices and cloud via a digital transformation allowed it to move faster, but also created a complex environment of interdependencies and a need to establish “observability” across that.

When Post kicked off its digital transformation in 2013, the overarching goal was speed.

“Our time to market was incredibly slow at the time,” recalls head of platform engineering Andrew Nette.

“It was taking up to 50 days for code to reach production, so environments were really slow in spinning up.”

The organisation set up a digital delivery centre internally to help internal application owners transform. 

The centre was a predecessor to platform engineering, where 30 engineers now assist Post’s delivery teams to get products to market quickly, and then support the application once in production.

Nette told last month’s New Relic FutureStack19 conference that Post re-architected applications into arrays of cloud-hosted microservices.

That structure worked insofar as it reduced the time needed to get code into production.

“We were quite successful,” Nette said.

“We were able to get things into production in about 12 minutes, so our time to market has significantly improved.

“[But] we were using a microservices architecture, so our number of things in production scaled out as well, which means our environment got a lot more complex, and we had to think about the way we monitored those applications differently.”

Establishing visibility

The problems appeared “after 2013”, not long into the transformation.

“Microservices were proliferating, and it was really difficult for us to keep up and keep the focus on the number of microservices that we had,” Nette said.

Post’s challenge quickly became establishing “observability” over the transformed environment, and Nette said the organisation had spent “a lot of time” getting to a point of “100 percent visibility.”

“Observability is more than just monitoring – it’s the ability to understand what’s happening inside your application or inside your system through all its dependencies,” Nette said.

“Being able to understand if there’s an issue in the network layer, infrastructure layer, application layer, and even out to third party services that you’re utilising.

“If you can do that, then you have a truly observable system, and if you can do it all in one place then … you have your single pane of glass where you can see all of your issues.”

Nette continued: “If I think about the way Post’s observability platform or our monitoring ecosystem developed … we’ve spent a lot of time trying to develop our ecosystem so that we have 100 percent visibility and no gaps.”

Tool-wise, Australia Post uses a mix of “New Relic APM [application performance monitoring], synthetics, Sumo Logic event [management], even Bash scripts if that was what was required to get the visibility.”

Nette described finding the right mix of tools as Post’s “Goldilocks zone” – “not too many, not too few, just the right number.” 

“The tools [also] need to provide value and not add more toil,” he said.

The ‘Vanilla Ice rule’

Post worked closely with delivery teams to instrument all parts of the environment.

“Collaboration was really important,” Nette said.

“We follow the Vanilla Ice rule: ‘stop, collaborate and listen’.

“It was a two-way conversation with delivery teams. They needed to understand why we were trying to do the things we were doing, and we needed to appreciate that they had other work, they were delivering features.”

One of the main things to monitor were Post’s customer-facing APIs, which large enterprise customers use to directly integrate with Post’s parcel delivery systems.

Post built all its APIs using a standard pattern that “had API health checks built in”, Nette said.

“It was really important that we worked with the delivery teams and the developers to make sure that those health checks were instrumented correctly, that they were calling their dependencies and that the dependencies were showing the correct states,” Nette said.

“Once we had a screen where all of our APIs were calling all of their dependencies, we could very quickly identify when there was an issue, and once we had that, we significantly reduced our mean time to identify issues and also our mean time to resolve.

“It was a big win, and it showed effective collaboration was really helpful.”

Australia Post used an unspecified set of open source tools to create that “API health check dashboard”, and then other “observability” dashboards useful to platform engineering.

It then started creating dashboards for individual delivery teams that showed them only the alerts they needed to see for the code and services they maintained.

In part, these dashboards helped the delivery teams to “evolve their DevOps capabilities”, Nette said, instead of fully relying on operations for alerting and support.

Platform engineering at Australia Post now offers “hybrid” support options to delivery teams.

“There’s full end-to-end support that we provide with delivery teams providing a third level [of] escalation, all the way up to the delivery teams providing full DevOps and doing all of the support [themselves], and [us just] providing advice where required,” Nette said.

The post Australia Post tackles ‘observability’ after digital transformation appeared first on Artificial Intelligence.