Apache NiFi offers shared services, which may be shared by processors and reporting task is named controller settings. These are like Database connection pool, which may be employed by processors accessing equivalent database.

To access the controller settings, use the drop-down menu at the right top corner of NiFi UI as shown within the below image.

8.1

There are many controller settings offered by Apache NiFi, we’ll discuss a commonly employed one and how we set it up in NiFi.

DBCPConnectionPool

Add the plus sign within the Nifi Settings page after clicking the Controller settings option. Then select the DBCPConnectionPool from the list of controller settings. DBCPConnectionPool will be added within the main NiFi settings page as shown within the below image.

It contains the subsequent information about the controller setting:Name

Type
Bundle
State
Scope
Configure and delete icon

Click on the configure icon and fill the required fields. The fields are listed down within the table below −

S.No.	Field Name	Default value	description
1	Database Connection URL	empty	To specify the connection URL to database.
2	Database Driver Class Name	empty	To specify the driver class name for database like com.mysql.jdbc.Driver for mysql.
3	Max Wait Time	500 millis	To specify time to wait for the data from a connection to database.
4	Max Total Connections	8	To specify the maximum number of allocated connections in database connection pool.

To stop or configure a controller setting, first all the attached NiFi components should be stopped. NiFi also adds scope in controller settings to manage the configuration of it. Therefore, only the ones which shared the equivalent settings will not get impacted and will use the equivalent controller settings.

Apache NiFi – Reporting Task

Apache NiFi reporting tasks are similar to the controller services, which run within the background and send or log the statistics of NiFi instance. NiFi reporting task can also be accessed from the equivalent page as controller settings, but in a different tab.

8.3

To add a reporting task, a developer needs to click on the plus button present at the top right hand side of the reporting tasks page. These reporting tasks are mainly employed for monitoring the activities of a NiFi instance, in either the bulletins or the provenance. Mainly these reporting tasks uses Site-to-Site to transport the NiFi statistics data to other node or external system.

Let us now add a configured reporting task for more understanding.

MonitorMemory

This reporting task is employed to generate bulletins, when a memory pool crosses specified percentage. Follow these steps to configure the MonitorMemory reporting task −

Add within the plus sign and search for MonitorMemory within the list.
Select MonitorMemory and click on ADD.
Once it is added within the main page of reporting tasks main page, click on the configure icon.
Within the properties tab, select the memory pool, which you want to monitor.
Select the percentage after which you want bulletins to alert the users.
Start the reporting task.

Apache NiFi – Custom Processor

Apache NiFi is an open source platform and gives developers the options to add their custom processor within the NiFi library. Follow these steps to create a custom processor.

Download Maven latest version from the link given below.

https://maven.apache.org/download.cgi

Add an environment variable named M2_HOME and set value as the installation directory of maven.
Download Eclipse IDE from the below link.

https://www.eclipse.org/downloads/download.php

Open command prompt and execute Maven Archetype command.

> mvn archetype:generate

Search for the nifi type within the archetype projects.
Select org.apache.nifi:nifi-processor-bundle-archetype project.
Then from the list of versions select the latest version i.e. 1.7.1 for this tutorial.
Enter the groupId, artifactId, version, package, and artifactBaseName etc.
Then a maven project will be created having to directories.
- nifi-<artifactBaseName>-processors
- nifi-<artifactBaseName>-nar
Run the below command in nifi-<artifactBaseName>-processors directory to add the project within the eclipse.

mvn install eclipse:eclipse

Open eclipse and select import from the file menu.
Then select “Existing Projects into workspace” and add the project from nifi-<artifactBaseName>-processors directory in eclipse.
Add your code in public void onTrigger(ProcessContext context, ProcessSession session) function, which runs when ever a processor is scheduled to run.
Then package the code to a NAR file by running the below mentioned command.

mvn clean install

A NAR file will be created at nifi–nar/target directory.
Copy the NAR file to the lib folder of Apache NiFi and restart the NiFi.
After successful restart of NiFi, check the processor list for the new custom processor.
For any errors, check ./logs/nifi.log file.

Apache NiFi – Custom Controllers Service

Apache NiFi is an open source platform and gives developers the options to add their custom controllers service in Apache NiFi. The steps and tools are almost the equivalent as employed to create a custom processor.

Open command prompt and execute Maven Archetype command.

> mvn archetype:generate

Search for the nifi type within the archetype projects.
Select org.apache.nifi:nifi-service-bundle-archetype project.
Then from the list of versions, select the latest version – 1.7.1 for this tutorial.
Enter the groupId, artifactId, version, package, and artifactBaseName, etc.
A maven project will be created having directories.
- nifi-<artifactBaseName>
- nifi-<artifactBaseName>-nar
- nifi-<artifactBaseName>-api
- nifi-<artifactBaseName>-api-nar
Run the below command in nifi-<artifactBaseName> and nifi-<artifactBaseName>-api directories to add these two projects within the eclipse.
- mvn install eclipse:eclipse
Open eclipse and select import from the file menu.
Then select “Existing Projects into workspace” and add the project from nifi-<artifactBaseName> and nifi-<artifactBaseName>-api directories in eclipse.
Add your code within the source files.
Then package the code to a NAR file by running the below mentioned command.
- mvn clean install
Two NAR files are going to be created in each nifi-<artifactBaseName>/target and nifi-<artifactBaseName>-api/target directory.
Copy these NAR files to the lib folder of Apache NiFi and restart the NiFi.
After successful restart of NiFi, check the processor list for the new custom processor.
For any errors, check ./logs/nifi.log file.

So, this brings us to the end of Deep Dive into Apache NiFi User Interface blog. This Tecklearn ‘Understand the concepts of Apache NiFi Controller , Reporting Task , Add Custom Processor and Custom Controllers service’ helps you with commonly asked questions if you are looking out for a job in Apache NiFi and Big Data Domain.

If you wish to learn Apache NiFi and build a career in Apache NiFi or Big Data domain, then check out our interactive, Apace NiFi Training, that comes with 24*7 support to guide you throughout your learning period. Please find the link for course details:

https://www.tecklearn.com/course/apache-nifi-training/

Apache NiFi Training

About the Course

Tecklearn Apache NiFi Training makes you an expert in Cluster integration and the challenges associated, Usefulness of Automation, Apache NiFi configuration challenges and etc. Apache NiFi that helps you master various aspects of automating dataflow, managing flow of information between systems, streaming analytics, the concepts of data lake and constructs, various methods of data ingestion and real-world Apache NiFi projects .Transforming the databases is becoming a challenge for many organizations and thus they often look for those who have certification in Apache NiFi to help them in automating the flow of data between the systems.

Why Should you take Apache NiFi Training?

The Average Salary for Apache NiFi Developers is $96,578 per year. – paysa.com
Micron, Macquarie Telecom Group , Dovestech, Payoff, Flexilogix , Hashmap Inc. & many other MNC’s worldwide use Ansible across industries.
Apache NiFi is an open source software for automating and managing the flow of data between systems. It is a powerful and reliable system to process and distribute data. It provides a web-based User Interface for creating, monitoring, & controlling data flows.

What you will Learn in this Course?

Overview of Apache NiFi and its capabilities

Understanding the Apache NiFi
Apache NiFi most interesting features and capabilities

High Level Overview of Key Apache NiFi Features

Key features categories: Flow management, Ease of use, Security, Extensible architecture and Flexible scaling model

Advantages of Apache NiFi over other traditional ETL tools

Features of NiFi which make it different form traditional ETL tool and gives NiFi an edge over them

Apache NiFi as a Data Ingestion Tool

Introduction to Apache NiFi for data ingestion
Apache NiFi Processor : Data ingestion tools available for transferring , importing , loading and processing of data

Data Lake Concepts and Constructs (Big Data & Hadoop Environment)

Concept of data lake and its attributes
Support for colocation of data in various formats and overcoming the problem of data silos

Apache NiFi capabilities in Big Data and Hadoop Environment

Introduction to NiFi processors which sync with data lake and Hadoop ecosystem
An overview of the various components of the Hadoop ecosystem and data lake

Installation Requirements and Cluster Integration

Apache NiFi installation requirements and cluster integration
Successfully running Apache NiFi and addition of processor to NiFi
Working with attributes and Process of scaling up and down
Hands On

Apache NiFi Core Concepts

Apache NiFi fundamental concepts
Overview of FlowFile, Flow Controller ,FlowFile Processor, and their attributes
Functions in dataflow

Architecture of Apache NiFi

Architecture of Apache NiFi
Various components including FlowFile Repository, Content Repository, Provenance Repository and web-based user interface
Hands On

Performance Expectation and Characteristics of NiFi

How to utilize maximization of resources is particularly strong with respect to CPU and disk
Understand the best practices and configuration tips

Queuing and Buffering Data

Buffering of Data in Apache NiFi
Concept of queuing, recovery and latency
Working with controller services and directed graphs
Data transformation and routing
Processor connection, addition and configuration
Hands On

Database Connection with Apache NiFi

Apache NiFi Connection with database
Data Splitting, Transforming and Aggregation
Monitoring of NiFi and process of data egress
Reporting and Data lineage
Expression language and Administration of Apache NiFi
Hands On

Apache NiFi Configuration Best Practices

Apache NiFi configuration Best Practices
ZooKeeper access, properties, custom properties and encryption
Guidelines for developers
Security of Data in Hadoop and NiFi Kerberos interface
Hands On

Apache NiFi Project

Apache NiFi Installation
Configuration and Deployment of toolbar
Building a dataflow using NiFi
Creating, importing and exporting various templates to construct a dataflow
Deploying Real-time ingestion and Batch ingestion in NiFi
Hands On

Got a question for us? Please mention it in the comments section and we will get back to you.

450