AWS Data Pipeline is a web service, designed to make it easier for users to integrate data spread across multiple AWS services and analyze it from a single location.

Using AWS Data Pipeline, data can be accessed from the source, processed, and then the results can be efficiently transferred to the respective AWS services.

Following are the steps to set up data pipeline −

Step 1 − Create the Pipeline using the following steps.

Sign-in to AWS account.
Use this link to Open AWS Data Pipeline console −https://console.aws.amazon.com/datapipeline/
Select the region in the navigation bar.
Click the Create New Pipeline button.
Fill the required details in the respective fields.
In the Source field, choose Build using a template and then select this template − Getting Started using ShellCommandActivity.

1 11

The Parameters section opens only when the template is selected. Leave the S3 input folder and Shell command to run with their default values. Click the folder icon next to S3 output folder, and select the buckets.
In Schedule, leave the values as default.
In Pipeline Configuration, leave the logging as enabled. Click the folder icon under S3 location for logs and select the buckets.
In Security/Access, leave IAM roles values as default.
Click the Activate button.

How to Delete a Pipeline?

Deleting the pipeline will also delete all associated objects.

Step 1 − Select the pipeline from the pipelines list.
Step 2 − Click the Actions button and then choose Delete.

2 10

Step 3 − A confirmation prompt window opens. Click Delete.

Features of AWS Data Pipeline

Simple and cost-efficient − Its drag-and-drop features makes it easy to create a pipeline on console. Its visual pipeline creator provides a library of pipeline templates. These templates make it easier to create pipelines for tasks like processing log files, archiving data to Amazon S3, etc.

Reliable − Its infrastructure is designed for fault tolerant execution activities. If failures occur in the activity logic or data sources, then AWS Data Pipeline automatically retries the activity. If the failure continues, then it will send a failure notification. We can even configure these notification alerts for situations like successful runs, failure, delays in activities, etc.

Flexible − AWS Data Pipeline provides various features like scheduling, tracking, error handling, etc. It can be configured to take actions like run Amazon EMR jobs, execute SQL queries directly against databases, execute custom applications running on Amazon EC2, etc.

So, this brings us to the end of blog. This Tecklearn ‘How to SetUp AWS Data Pipeline ’ helps you with commonly asked questions if you are looking out for a job in AWS and Cloud Computing. If you wish to learn AWS and build a career in Cloud Computing domain, then check out our interactive, AWS Solutions Architect Training, that comes with 24*7 support to guide you throughout your learning period. Please find the link for course details:

https://www.tecklearn.com/course/aws-solutions-architect-certification-training/

AWS Solutions Architect Certification Training

About the Course

Tecklearn’s AWS Architect Certification Training is curated by industry professionals as per the industry requirements and demands. The entire AWS training course is in line with the AWS Certified Solutions Architect exam. You will learn various aspects of AWS like Elastic Cloud Compute, Simple Storage Service, Virtual Private Cloud, Aurora database service, Load Balancing, Auto Scaling and more by working on hands-on projects and case studies. You will master AWS architectural principles and services such as IAM, VPC, EC2, EBS and elevate your career to the cloud, and beyond with this AWS solutions architect course.

Why Should you take AWS Architect Certification Training?

The Average salary of an AWS Certified Solutions Architect is $129k per annum – Indeed.com
AWS market is expected to reach $236 Billion by 2020 at a CAGR of 22% – Forrester
Netflix, Twitter, LinkedIn, Facebook, BBC, Baidu, ESPN & other MNCs worldwide use Amazon AWS Cloud

What you will Learn in this Course?

Overview of Cloud Computing and AWS

What is Cloud Computing
Definition of Cloud Computing
On Premises Vs Service Models
Advantages and Disadvantages of Cloud Computing
Cloud Computing Providers
Why AWS
What is AWS
AWS Benefits
AWS Services
Traditional Vs AWS Components
AWS Global Infrastructure
AWS Availability Zone
AWS Edge Locations
How to Access the AWS Services
AWS architecture
AWS Management Console
AWS offerings Listing (EC2, VPC, AMI, EBS, ELB, Backup)

Amazon Elastic Compute Cloud (EC2)

Overview of EC2
Elastic IP Vs Public IP
Launching of AWS EC2 instance demo
How to access EC2
EC2 Purchasing Options
Amazon Machine Images (AMI)
EC2 Storage for the Root Device
EC2 Creating AMI
EC2 Instance Types
Auto Scaling
Cost of EC2
Best Practices of EC2
EC2 Resizing
Placement Groups
Amazon Backup and various Concepts
EC2 Demo
Hands On

Networking and Monitoring Services: Amazon Virtual Public Cloud

Virtual Private Cloud (VPC) and its benefits
Default and Non-Default VPC
IP Address
CIDR – Classless Inter-domain Routing
Subnet: Subnet Mask and Subnet Mask Classes
Private and Public Subnet
IPv4 v/s IPv6 – As in AWS Infrastructure
Internet Gateway and Route Tables
Security Group with VPC
Access Control List, NACL and Security Group
NAT Devices: NAT Gateway and NAT Instance
Flow Logs
VPC Peering and its working
VPN and Direct Connect
VPC Limitations
Need for Monitoring Services
AWS CloudWatch and it’s working
AWS Command Line Interface
Use Cases
Hands On

Amazon Storage Services: Elastic Block Storage

What is Storage Services
What is Elastic Block Storage (EBS)
Persistent Storage
EBC Features
EBS Benefits
EBS Types
EBS Pricing
EBS Life Cycle
EBS Snapshot
EBS General Purposed SSD
EBS Provisioned IOPS SSD
EBS Throughput Optimized HDD
EBS Cold HDD
EBS Comparison
EBS Previous Generation Volumes
EBS How Incremental Snapshots Work
EBS Deleting an Amazon EBS Snapshot
EBS Summary
Hands On

Amazon Storage Services: Simple Storage Services (S3)

What is Amazon AWS S3
Simple Storage Services (S3) Advantages
S3 Buckets, Objects, Keys and Endpoints
S3 Data Consistency Model
S3 Transfer Acceleration
S3 Storage Types
S3 Versioning
S3 Life Cycle Management
S3 Data Protection
S3 Cross-Region Replication
S3 Hosting a Static Website
Hands On

Amazon Storage Services

Amazon Glacier Storage
Amazon Storage Gateway
Amazon Snowball (Data Import /Export)
Billing with Amazon CloudWatch
Hands On

AWS Database Services: Relational Database Service (RDS)

Overview of Databases and Relational Database Service (RDS)
What is Amazon RDS
AWS RDS Components
AWS RDS: Interface
AWS RDS: Charges
AWS RDS Multi-AZ: Benefits
AWS RDS Multi-AZ: Failover Process
NoSQL Database: Amazon DynamoDB
Overview of DynamoDB
DynamoDB Benefits
Hands On

AWS Database Services Continued

Data Warehouse: Amazon Redshift
Overview of Amazon Redshift
Redshift Architecture
Amazon Redshift features
In Memory Cache: Amazon ElasticCache
Redis Vs MemCache
Amazon ElasticCache Cluster
Database Migration: AWS Database Migration Service

Load Balancing in AWS

What is Fault Tolerant System
Features of Elastic Load Balancing
What is AWS ELB (Elastic Load Balancer)
Types of Load Balancer: Classic, Application and Network
Classic Load Balancer: Features, Health Check Configuration, Cross-Zone, Connection Draining, Sticky Sessions, Access Logs, Limitation
Application Load Balancer: Features, Application Flow, Limitation
Network Load Balancer
Access Elastic Load Balancing: AWS Management Console, AWS CLI, AWS SDKs, HTTPS Query API

Amazon Route 53

What is Amazon Route 53
Domain Name Registration
Routing Internet Traffic to Resources
Automated check of the health of Resources + Data Pipeline

AWS Identity and Access Management (IAM) – Control user access

Authentication (Who can use) and Authorization (Level of Access)
IAM Policies – JSON Structure
Users, Groups and their Roles
AWS IAM Features
User Sign-in to Account
Switch Role
Role to EC2 Instance
Password Policy
How to Access AWS
Multi-Factor Authentication (MFA)
Permissions and Permission Types
Policies Structure
User Based Policies
Resource Based Policies
Resource Based Permission
Policies Types
Request Flow
Limitations
IAM HTTPS API
Logging IAM Events with AWS CloudTail
Hands On

Amazon CloudWatch

What is Amazon CloudWatch
Features and Benefits
CloudWatch Architecture
Hands On

AWS Auto Scaling

What is AWS Auto Scaling
Auto Scaling Components
Auto Scaling Group
Auto Scaling Launch Configuration
Auto Scaling Benefits
Auto Scaling Lifecycle
Auto Scaling Plans
Manual Scaling
Schedule Scaling
Dynamic Scaling
Auto Scaling Step Adjustment
Auto Scaling Termination Policy
Default Termination Policy
Health Check
Hands On

Amazon Application Services

Elastic BeanStalk
Simple Email Services (SES)
Simple Queue Service (SQS)
Simple Notification Services (SNS)
AWS Lambda
Introduction to Elastic OpWorks
Hands On

About AWS Solution Architect Associate Exam

Got a question for us? Please mention it in the comments section and we will get back to you.

507