As learned earlier, Keras layers are the primary building block of Keras models. Each layer receives input information, do some computation and finally output the transformed information. The output of one layer will flow into the next layer as its input. Let us learn complete details about layers in this blog.

Introduction

A Keras layer requires shape of the input (input_shape) to understand the structure of the input data, initializer to set the weight for each input and finally activators to transform the output to make it non-linear. In between, constraints restrict and specify the range in which the weight of input data to be generated and regularizer will try to optimize the layer (and the model) by dynamically applying the penalties on the weights during optimization process.

To summarise, Keras layer requires below minimum details to create a complete layer.

Shape of the input data
Number of neurons / units in the layer
Initializers
Regularizers
Constraints
Activations

Let us understand the basic concept in the next chapter. Before understanding the basic concept, let us create a simple Keras layer using Sequential model API to get the idea of how Keras model and layer works.

from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import initializers
from keras import regularizers
from keras import constraints
model = Sequential()
model.add(Dense(32, input_shape=(16,), kernel_initializer = 'he_uniform',
kernel_regularizer = None, kernel_constraint = 'MaxNorm', activation = 'relu'))
model.add(Dense(16, activation = 'relu'))
model.add(Dense(8))

where,

Line 1-5 imports the necessary modules.
Line 7 creates a new model using Sequential API.
Line 9 creates a new Dense layer and add it into the model. Dense is an entry level layer provided by Keras, which accepts the number of neurons or units (32) as its required parameter. If the layer is first layer, then we need to provide Input Shape, (16,) as well. Otherwise, the output of the previous layer will be used as input of the next layer. All other parameters are optional.
- First parameter represents the number of units (neurons).
- input_shape represent the shape of input data.
- kernel_initializer represent initializer to be used. he_uniform function is set as value.
- kernel_regularizer represent regularizer to be used. None is set as value.
- kernel_constraint represent constraint to be used. MaxNorm function is set as value.
- activation represent activation to be used. relu function is set as value.
Line 10 creates second Dense layer with 16 units and set relu as the activation function.
Line 11 creates final Dense layer with 8 units.

Basic Concept of Layers

Let us understand the basic concept of layer as well as how Keras supports each concept.

Input shape

In machine learning, all type of input data like text, images or videos will be first converted into array of numbers and then feed into the algorithm. Input numbers may be single dimensional array, two dimensional array (matrix) or multi-dimensional array. We can specify the dimensional information using shape, a tuple of integers. For example, (4,2) represent matrix with four rows and two columns.

>>> import numpy as np
>>> shape = (4, 2)
>>> input = np.zeros(shape)
>>> print(input)
[
[0. 0.]
[0. 0.]
[0. 0.]
[0. 0.]
]
>>>
Similarly, (3,4,2) three-dimensional matrix having three collections of 4x2 matrix (two rows and four columns).
>>> import numpy as np
>>> shape = (3, 4, 2)
>>> input = np.zeros(shape)
>>> print(input)
[
[[0. 0.] [0. 0.] [0. 0.] [0. 0.]]
[[0. 0.] [0. 0.] [0. 0.] [0. 0.]]
[[0. 0.] [0. 0.] [0. 0.] [0. 0.]]
]
>>>

To create the first layer of the model (or input layer of the model), shape of the input data should be specified.

Initializers

In Machine Learning, weight will be assigned to all input data. Initializers module provides different functions to set these initial weights. Some of the Keras Initializer function are as follows −

Zeros

Generates 0 for all input data.

from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import initializers
my_init = initializers.Zeros()
model = Sequential()
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_initializer = my_init))

Where, kernel_initializer represent the initializer for kernel of the model.

Ones

Generates 1 for all input data.

from keras.models import Sequential

from keras.layers import Activation, Dense

from keras import initializers

my_init = initializers.Ones()

model.add(Dense(512, activation = ‘relu’, input_shape = (784,),

kernel_initializer = my_init))

Constant

Generates a constant value (say, 5) specified by the user for all input data.

from keras.models import Sequential

from keras.layers import Activation, Dense

from keras import initializers

my_init = initializers.Constant(value = 0) model.add(

Dense(512, activation = ‘relu’, input_shape = (784,), kernel_initializer = my_init)

)

where, value represent the constant value

RandomNormal

Generates value using normal distribution of input data.

from keras.models import Sequential

from keras.layers import Activation, Dense

from keras import initializers

my_init = initializers.RandomNormal(mean=0.0,

stddev = 0.05, seed = None)

model.add(Dense(512, activation = ‘relu’, input_shape = (784,),

kernel_initializer = my_init))

where,

mean represent the mean of the random values to generate
stddev represent the standard deviation of the random values to generate
seed represent the values to generate random number

RandomUniform

Generates value using uniform distribution of input data.

from keras import initializers

my_init = initializers.RandomUniform(minval = -0.05, maxval = 0.05, seed = None)

model.add(Dense(512, activation = ‘relu’, input_shape = (784,),

kernel_initializer = my_init))

where,

minval represent the lower bound of the random values to generate
maxval represent the upper bound of the random values to generate

TruncatedNormal

Generates value using truncated normal distribution of input data.

from keras.models import Sequential

from keras.layers import Activation, Dense

from keras import initializers

my_init = initializers.TruncatedNormal(mean = 0.0, stddev = 0.05, seed = None

model.add(Dense(512, activation = ‘relu’, input_shape = (784,),

kernel_initializer = my_init))

VarianceScaling

Generates value based on the input shape and output shape of the layer along with the specified scale.

from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import initializers
my_init = initializers.VarianceScaling(
scale = 1.0, mode = ‘fan_in’, distribution = ‘normal’, seed = None)
model.add(Dense(512, activation = ‘relu’, input_shape = (784,),
skernel_initializer = my_init))

where,

scale represent the scaling factor
mode represent any one of fan_in, fan_out and fan_avg values
distribution represent either of normal or uniform

VarianceScaling

It finds the stddev value for normal distribution using below formula and then find the weights using normal distribution,

stddev = sqrt(scale / n)

where n represent,

number of input units for mode = fan_in
number of out units for mode = fan_out
average number of input and output units for mode = fan_avg

Similarly, it finds the limit for uniform distribution using below formula and then find the weights using uniform distribution,

limit = sqrt(3 * scale / n)

lecun_normal

Generates value using lecun normal distribution of input data.

from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import initializers
my_init = initializers.RandomUniform(minval = -0.05, maxval = 0.05, seed = None)
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_initializer = my_init))
It finds the stddev using the below formula and then apply normal distribution
stddev = sqrt(1 / fan_in)
where, fan_in represent the number of input units.
lecun_uniform
Generates value using lecun uniform distribution of input data.
from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import initializers
my_init = initializers.lecun_uniform(seed = None)
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_initializer = my_init))
It finds the limit using the below formula and then apply uniform distribution
limit = sqrt(3 / fan_in)

where,

fan_in represents the number of input units
fan_out represents the number of output units

glorot_normal

Generates value using glorot normal distribution of input data.

from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import initializers
my_init = initializers.glorot_normal(seed=None) model.add(
Dense(512, activation = 'relu', input_shape = (784,), kernel_initializer = my_init)
)
It finds the stddev using the below formula and then apply normal distribution
stddev = sqrt(2 / (fan_in + fan_out))

where,

fan_in represents the number of input units
fan_out represents the number of output units

glorot_uniform

Generates value using glorot uniform distribution of input data.

from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import initializers
my_init = initializers.glorot_uniform(seed = None)
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_initializer = my_init))
It finds the limit using the below formula and then apply uniform distribution
limit = sqrt(6 / (fan_in + fan_out))

where,

fan_in represent the number of input units.
fan_out represents the number of output units

he_normal

Generates value using he normal distribution of input data.
from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import initializers
my_init = initializers.RandomUniform(minval = -0.05, maxval = 0.05, seed = None)
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_initializer = my_init))
It finds the stddev using the below formula and then apply normal distribution.
stddev = sqrt(2 / fan_in)
where, fan_in represent the number of input units.
he_uniform
Generates value using he uniform distribution of input data.
from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import initializers
my_init = initializers.he_normal(seed = None)
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_initializer = my_init))
It finds the limit using the below formula and then apply uniform distribution.
limit = sqrt(6 / fan_in)
where, fan_in represent the number of input units.
Orthogonal
Generates a random orthogonal matrix.
from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import initializers
my_init = initializers.Orthogonal(gain = 1.0, seed = None)
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_initializer = my_init))
where, gain represent the multiplication factor of the matrix.
Identity
Generates identity matrix.
from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import initializers
my_init = initializers.Identity(gain = 1.0) model.add(
Dense(512, activation = 'relu', input_shape = (784,), kernel_initializer = my_init)
)

Constraints

In machine learning, a constraint will be set on the parameter (weight) during optimization phase. <>Constraints module provides different functions to set the constraint on the layer. Some of the constraint functions are as follows.

NonNeg

Constrains weights to be non-negative.

from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import initializers
my_init = initializers.Identity(gain = 1.0) model.add(
Dense(512, activation = 'relu', input_shape = (784,),
kernel_initializer = my_init)
)
where, kernel_constraint represent the constraint to be used in the layer.
UnitNorm
Constrains weights to be unit norm.
from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import constraints
my_constrain = constraints.UnitNorm(axis = 0)
model = Sequential()
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_constraint = my_constrain))
MaxNorm
Constrains weight to norm less than or equals to the given value.
from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import constraints
my_constrain = constraints.MaxNorm(max_value = 2, axis = 0)
model = Sequential()
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_constraint = my_constrain))
where,
max_value represent the upper bound
axis represent the dimension in which the constraint to be applied. e.g. in Shape (2,3,4) axis 0 denotes first dimension, 1 denotes second dimension and 2 denotes third dimension
MinMaxNorm
Constrains weights to be norm between specified minimum and maximum values.
from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import constraints
my_constrain = constraints.MinMaxNorm(min_value = 0.0, max_value = 1.0, rate = 1.0, axis = 0)
model = Sequential()
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_constraint = my_constrain))
where, rate represent the rate at which the weight constrain is applied.

Regularizers

In machine learning, regularizers are used in the optimization phase. It applies some penalties on the layer parameter during optimization. Keras regularization module provides below functions to set penalties on the layer. Regularization applies per-layer basis only.

L1 Regularizer

It provides L1 based regularization.

from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import regularizers
my_regularizer = regularizers.l1(0.)
model = Sequential()
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_regularizer = my_regularizer))
where, kernel_regularizer represent the rate at which the weight constrain is applied.
L2 Regularizer
It provides L2 based regularization.
from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import regularizers
my_regularizer = regularizers.l2(0.)
model = Sequential()
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_regularizer = my_regularizer))
L1 and L2 Regularizer
It provides both L1 and L2 based regularization.
from keras.models import Sequential
from keras.layers import Activation, Dense
from keras import regularizers
my_regularizer = regularizers.l2(0.)
model = Sequential()
model.add(Dense(512, activation = 'relu', input_shape = (784,),
kernel_regularizer = my_regularizer))

Activations

In machine learning, activation function is a special function used to find whether a specific neuron is activated or not. Basically, the activation function does a nonlinear transformation of the input data and thus enable the neurons to learn better. Output of a neuron depends on the activation function.

As you recall the concept of single perception, the output of a perceptron (neuron) is simply the result of the activation function, which accepts the summation of all input multiplied with its corresponding weight plus overall bias, if any available.

result = Activation(SUMOF(input * weight) + bias)

So, activation function plays an important role in the successful learning of the model. Keras provides a lot of activation function in the activations module. Let us learn all the activations available in the module.

linear

Applies Linear function. Does nothing.

from keras.models import Sequential
from keras.layers import Activation, Dense
model = Sequential()
model.add(Dense(512, activation = 'linear', input_shape = (784,)))
Where, activation refers the activation function of the layer. It can be specified simply by the name of the function and the layer will use corresponding activators.
elu
Applies Exponential linear unit.
from keras.models import Sequential
from keras.layers import Activation, Dense
model = Sequential()
model.add(Dense(512, activation = 'elu', input_shape = (784,)))
selu
Applies Scaled exponential linear unit.
from keras.models import Sequential
from keras.layers import Activation, Dense
model = Sequential()
model.add(Dense(512, activation = 'selu', input_shape = (784,)))
relu
Applies Rectified Linear Unit.
from keras.models import Sequential
from keras.layers import Activation, Dense
model = Sequential()
model.add(Dense(512, activation = 'relu', input_shape = (784,)))
softmax
Applies Softmax function.
from keras.models import Sequential
from keras.layers import Activation, Dense
model = Sequential()
model.add(Dense(512, activation = 'softmax', input_shape = (784,)))
softplus
Applies Softplus function.
from keras.models import Sequential
from keras.layers import Activation, Dense
model = Sequential()
model.add(Dense(512, activation = 'softplus', input_shape = (784,)))
softsign
Applies Softsign function.
from keras.models import Sequential
from keras.layers import Activation, Dense
model = Sequential()
model.add(Dense(512, activation = 'softsign', input_shape = (784,)))
tanh
Applies Hyperbolic tangent function.
from keras.models import Sequential
from keras.layers import Activation, Dense
model = Sequential()
model.add(Dense(512, activation = 'tanh', input_shape = (784,)))
sigmoid
Applies Sigmoid function.
from keras.models import Sequential
from keras.layers import Activation, Dense
model = Sequential()
model.add(Dense(512, activation = 'sigmoid', input_shape = (784,)))
hard_sigmoid
Applies Hard Sigmoid function.
from keras.models import Sequential
from keras.layers import Activation, Dense
model = Sequential()
model.add(Dense(512, activation = 'hard_sigmoid', input_shape = (784,)))
exponential
Applies exponential function.
from keras.models import Sequential
from keras.layers import Activation, Dense
model = Sequential()
model.add(Dense(512, activation = 'exponential', input_shape = (784,)))

Sr.No	Layers & Description
1	Dense Layer Dense layer is the regular deeply connected neural network layer.
2	Dropout Layers *Dropout* is one of the important concepts in the machine learning.
3	Flatten Layers Flatten is used to flatten the input.
4	Reshape Layers *Reshape* is used to change the shape of the input.
5	Permute Layers Permute is also used to change the shape of the input using pattern.
6	RepeatVector Layers *RepeatVector* is used to repeat the input for set number, n of times.
7	Lambda Layers *Lambda* is used to transform the input data using an expression or function.
8	Convolution Layers Keras contains a lot of layers for creating Convolution based ANN, popularly called as Convolution Neural Network (CNN).
9	Pooling Layer It is used to perform max pooling operations on temporal data.
10	Locally connected layer Locally connected layers are similar to Conv1D layer but the difference is Conv1D layer weights are shared but here weights are unshared.
11	Merge Layer It is used to merge a list of inputs.
12	Embedding Layer It performs embedding operations in input layer.

So, this brings us to the end of blog. This Tecklearn ‘Detail understanding of Keras Layers’ blog helps you with commonly asked questions if you are looking out for a job in Artificial Intelligence. If you wish to learn Artificial Intelligence and build a career in AI or Machine Learning domain, then check out our interactive, Artificial Intelligence and Deep Learning with TensorFlow Training, that comes with 24*7 support to guide you throughout your learning period. Please find the link for course details:

https://www.tecklearn.com/course/artificial-intelligence-and-deep-learning-with-tensorflow/

Artificial Intelligence and Deep Learning with TensorFlow Training

About the Course

Tecklearn’s Artificial Intelligence and Deep Learning with Tensor Flow course is curated by industry professionals as per the industry requirements & demands and aligned with the latest best practices. You’ll master convolutional neural networks (CNN), TensorFlow, TensorFlow code, transfer learning, graph visualization, recurrent neural networks (RNN), Deep Learning libraries, GPU in Deep Learning, Keras and TFLearn APIs, backpropagation, and hyperparameters via hands-on projects. The trainee will learn AI by mastering natural language processing, deep neural networks, predictive analytics, reinforcement learning, and more programming languages needed to shine in this field.

Why Should you take Artificial Intelligence and Deep Learning with Tensor Flow Training?

According to Paysa.com, an Artificial Intelligence Engineer earns an average of $171,715, ranging from $124,542 at the 25th percentile to $201,853 at the 75th percentile, with top earners earning more than $257,530.
Worldwide Spending on Artificial Intelligence Systems Will Be Nearly $98 Billion in 2023, According to New IDC Spending Guide at a GACR of 28.5%.
IBM, Amazon, Apple, Google, Facebook, Microsoft, Oracle and almost all the leading companies are working on Artificial Intelligence to innovate future technologies.

What you will Learn in this Course?

Introduction to Deep Learning and AI

What is Deep Learning?
Advantage of Deep Learning over Machine learning
Real-Life use cases of Deep Learning
Review of Machine Learning: Regression, Classification, Clustering, Reinforcement Learning, Underfitting and Overfitting, Optimization
Pre-requisites for AI & DL
Python Programming Language
Installation & IDE

Environment Set Up and Essentials

Installation
Python – NumPy
Python for Data Science and AI
Python Language Essentials
Python Libraries – Numpy and Pandas
Numpy for Mathematical Computing

More Prerequisites for Deep Learning and AI

Pandas for Data Analysis
Machine Learning Basic Concepts
Normalization
Data Set
Machine Learning Concepts
Regression
Logistic Regression
SVM – Support Vector Machines
Decision Trees
Python Libraries for Data Science and AI

Introduction to Neural Networks

Creating Module
Neural Network Equation
Sigmoid Function
Multi-layered perception
Weights, Biases
Activation Functions
Gradient Decent or Error function
Epoch, Forward & backword propagation
What is TensorFlow?
TensorFlow code-basics
Graph Visualization
Constants, Placeholders, Variables

Multi-layered Neural Networks

Error Back propagation issues
Drop outs

Regularization techniques in Deep Learning

Deep Learning Libraries

Tensorflow
Keras
OpenCV
SkImage
PIL

Building of Simple Neural Network from Scratch from Simple Equation

Training the model

Dual Equation Neural Network

TensorFlow
Predicting Algorithm

Introduction to Keras API

Define Keras
How to compose Models in Keras
Sequential Composition
Functional Composition
Predefined Neural Network Layers
What is Batch Normalization
Saving and loading a model with Keras
Customizing the Training Process
Using TensorBoard with Keras
Use-Case Implementation with Keras

GPU in Deep Learning

Introduction to GPUs and how they differ from CPUs
Importance of GPUs in training Deep Learning Networks
The GPU constituent with simpler core and concurrent hardware
Keras Model Saving and Reusing
Deploying Keras with TensorBoard

Keras Cat Vs Dog Modelling

Activation Functions in Neural Network

Optimization Techniques

Some Examples for Neural Network

Convolutional Neural Networks (CNN)

Introduction to CNNs
CNNs Application
Architecture of a CNN
Convolution and Pooling layers in a CNN
Understanding and Visualizing a CNN

RNN: Recurrent Neural Networks

Introduction to RNN Model
Application use cases of RNN
Modelling sequences
Training RNNs with Backpropagation
Long Short-Term memory (LSTM)
Recursive Neural Tensor Network Theory
Recurrent Neural Network Model

Application of Deep Learning in image recognition, NLP and more

Real world projects in recommender systems and others

Got a question for us? Please mention it in the comments section and we will get back to you.

632