In Apache NiFi, there are multiple ways to monitor the different statistics of the system like errors, memory usage, CPU usage, Data Flow statistics, etc. We will discuss the most popular ones in this tutorial.

In built Monitoring

In this section, we will learn more about in-built monitoring in Apache NiFi.

Bulletin Board

The bulletin board shows the latest ERROR and WARNING getting generated by NiFi processors in real time. To access the bulletin board, a user will have to go the right-hand drop-down menu and select the Bulletin Board option. It refreshes automatically and a user can disable it also. A user can also navigate to the actual processor by double-clicking the error. A user can also filter the bulletins by working out with the subsequent −

by message
by name
by id
by group id

Data provenance UI

To monitor the Events occurring on any specific processor or throughout NiFi, a user can access the Data provenance from the same menu as the bulletin board. A user can also filter the events in data provenance repository by working out with the subsequent fields −

by component name
by component type
by type

NiFi Summary UI

Apache NiFi summary also can be accessed from the same menu as the bulletin board. This UI contains information about all the components of that particular NiFi instance or cluster. They can be filtered by name, by type or by URI. There are different tabs for different component types. Subsequent are the components, which can be monitored within the NiFi summary UI −

Processors
Input ports
Output ports
Remote process groups
Connections
Process groups

In this UI, there is a link at the bottom right hand side named system diagnostics to check the JVM statistics.

Reporting Tasks

Apache NiFi provides multiple reporting tasks to support external monitoring systems like Ambari, Grafana, etc. A developer can create a custom reporting task or can configure the inbuilt ones to send the metrics of NiFi to the externals monitoring systems. The subsequent table lists down the reporting tasks offered by NiFi 1.7.1.

S.No.	Reporting Task Name	Description
1	AmbariReportingTask	To setup Ambari Metrics Service for NiFi.
2	ControllerStatusReportingTask	To report the information from the NiFi summary UI for the last 5 minutes.
3	MonitorDiskUsage	To report and warn about the disk usage of a specific directory.
4	MonitorMemory	To monitor the amount of Java Heap employed in a Java Memory pool of JVM.
5	SiteToSiteBulletinReportingTask	To report the errors and warning in bulletins using Site to Site protocol.
6	SiteToSiteProvenanceReportingTask	To report the NiFi Data Provenance events using Site to Site protocol.

NiFi API

There is an API named system diagnostics, which can be employed to monitor the NiFI stats in any custom developed application. Let us check the API in postman.

Request

http://localhost:8080/nifi-api/system-diagnostics

Response

{

   "systemDiagnostics": {

      "aggregateSnapshot": {

         "totalNonHeap": "183.89 MB",

         "totalNonHeapBytes": 192819200,

         "employedNonHeap": "173.47 MB",

         "employedNonHeapBytes": 181894560,

         "freeNonHeap": "10.42 MB",

         "freeNonHeapBytes": 10924640,

         "maxNonHeap": "-1 bytes",

         "maxNonHeapBytes": -1,

         "totalHeap": "512 MB",

         "totalHeapBytes": 536870912,

         "employedHeap": "273.37 MB",

         "employedHeapBytes": 286652264,

         "freeHeap": "238.63 MB",

         "freeHeapBytes": 250218648,

         "maxHeap": "512 MB",

         "maxHeapBytes": 536870912,

         "heapUtilization": "53.0%",

         "availableProcessors": 4,

         "processorLoadAverage": -1,

         "totalThreads": 71,

         "daemonThreads": 31,

         "uptime": "17:30:35.277",

         "flowFileRepositoryStorageUsage": {

            "freeSpace": "286.93 GB",

            "totalSpace": "464.78 GB",

            "employedSpace": "177.85 GB",

            "freeSpaceBytes": 308090789888,

            "totalSpaceBytes": 499057160192,

            "employedSpaceBytes": 190966370304,

            "utilization": "38.0%"

         },

         "contentRepositoryStorageUsage": [

            {

               "identifier": "default",

               "freeSpace": "286.93 GB",

               "totalSpace": "464.78 GB",

               "employedSpace": "177.85 GB",

               "freeSpaceBytes": 308090789888,

               "totalSpaceBytes": 499057160192,

               "employedSpaceBytes": 190966370304,

               "utilization": "38.0%"

            }

         ],

         "provenanceRepositoryStorageUsage": [

            {

               "identifier": "default",

               "freeSpace": "286.93 GB",

               "totalSpace": "464.78 GB",

               "employedSpace": "177.85 GB",

               "freeSpaceBytes": 308090789888,

               "totalSpaceBytes": 499057160192,

               "employedSpaceBytes": 190966370304,

               "utilization": "38.0%"

            }

         ],

         "garbageCollection": [

            {

               "name": "G1 Young Generation",

               "collectionCount": 344,

               "collectionTime": "00:00:06.239",

               "collectionMillis": 6239

            },

            {

               "name": "G1 Old Generation",

               "collectionCount": 0,

               "collectionTime": "00:00:00.000",

               "collectionMillis": 0

            }

         ],

         "statsLastRefreshed": "09:30:20 SGT",

         "versionInfo": {

            "niFiVersion": "1.7.1",

            "javaVendor": "Oracle Corporation",

            "javaVersion": "1.8.0_151",

            "osName": "Windows 7",

            "osVersion": "6.1",

            "osArchitecture": "amd64",

            "buildTag": "nifi-1.7.1-RC1",

            "buildTimestamp": "07/12/2018 12:54:43 SGT"

         }

      }

   }

}

So, this brings us to the end of Deep Dive into Apache NiFi User Interface blog. This Tecklearn ‘How to Monitor System statistics using Apache NiFi’ helps you with commonly asked questions if you are looking out for a job in Apache NiFi and Big Data Domain.

If you wish to learn Apache NiFi and build a career in Apache NiFi or Big Data domain, then check out our interactive, Apace NiFi Training, that comes with 24*7 support to guide you throughout your learning period. Please find the link for course details:

https://www.tecklearn.com/course/apache-nifi-training/

Apache NiFi Training

About the Course

Tecklearn Apache NiFi Training makes you an expert in Cluster integration and the challenges associated, Usefulness of Automation, Apache NiFi configuration challenges and etc. Apache NiFi that helps you master various aspects of automating dataflow, managing flow of information between systems, streaming analytics, the concepts of data lake and constructs, various methods of data ingestion and real-world Apache NiFi projects .Transforming the databases is becoming a challenge for many organizations and thus they often look for those who have certification in Apache NiFi to help them in automating the flow of data between the systems.

Why Should you take Apache NiFi Training?

The Average Salary for Apache NiFi Developers is $96,578 per year. – paysa.com
Micron, Macquarie Telecom Group , Dovestech, Payoff, Flexilogix , Hashmap Inc. & many other MNC’s worldwide use Ansible across industries.
Apache NiFi is an open source software for automating and managing the flow of data between systems. It is a powerful and reliable system to process and distribute data. It provides a web-based User Interface for creating, monitoring, & controlling data flows.

What you will Learn in this Course?

Overview of Apache NiFi and its capabilities

Understanding the Apache NiFi
Apache NiFi most interesting features and capabilities

High Level Overview of Key Apache NiFi Features

Key features categories: Flow management, Ease of use, Security, Extensible architecture and Flexible scaling model

Advantages of Apache NiFi over other traditional ETL tools

Features of NiFi which make it different form traditional ETL tool and gives NiFi an edge over them

Apache NiFi as a Data Ingestion Tool

Introduction to Apache NiFi for data ingestion
Apache NiFi Processor : Data ingestion tools available for transferring , importing , loading and processing of data

Data Lake Concepts and Constructs (Big Data & Hadoop Environment)

Concept of data lake and its attributes
Support for colocation of data in various formats and overcoming the problem of data silos

Apache NiFi capabilities in Big Data and Hadoop Environment

Introduction to NiFi processors which sync with data lake and Hadoop ecosystem
An overview of the various components of the Hadoop ecosystem and data lake

Installation Requirements and Cluster Integration

Apache NiFi installation requirements and cluster integration
Successfully running Apache NiFi and addition of processor to NiFi
Working with attributes and Process of scaling up and down
Hands On

Apache NiFi Core Concepts

Apache NiFi fundamental concepts
Overview of FlowFile, Flow Controller ,FlowFile Processor, and their attributes
Functions in dataflow

Architecture of Apache NiFi

Architecture of Apache NiFi
Various components including FlowFile Repository, Content Repository, Provenance Repository and web-based user interface
Hands On

Performance Expectation and Characteristics of NiFi

How to utilize maximization of resources is particularly strong with respect to CPU and disk
Understand the best practices and configuration tips

Queuing and Buffering Data

Buffering of Data in Apache NiFi
Concept of queuing, recovery and latency
Working with controller services and directed graphs
Data transformation and routing
Processor connection, addition and configuration
Hands On

Database Connection with Apache NiFi

Apache NiFi Connection with database
Data Splitting, Transforming and Aggregation
Monitoring of NiFi and process of data egress
Reporting and Data lineage
Expression language and Administration of Apache NiFi
Hands On

Apache NiFi Configuration Best Practices

Apache NiFi configuration Best Practices
ZooKeeper access, properties, custom properties and encryption
Guidelines for developers
Security of Data in Hadoop and NiFi Kerberos interface
Hands On

Apache NiFi Project

Apache NiFi Installation
Configuration and Deployment of toolbar
Building a dataflow using NiFi
Creating, importing and exporting various templates to construct a dataflow
Deploying Real-time ingestion and Batch ingestion in NiFi
Hands On

Got a question for us? Please mention it in the comments section and we will get back to you.

496