Missing data is usually a drag in real world scenarios. Areas like machine learning and data processing face severe issues within the accuracy of their model predictions due to poor quality of data caused by missing values. In these areas, missing value treatment may be a major point of focus to form their models more accurate and valid.

When and Why Is Data Missed?

Let us consider a web survey for a product. many a times, people don’t share all the data associated with them. Few people share their experience, but not how long they’re using the merchandise; few people share how long they’re using the product, their experience but not their contact information. Thus, in some or the opposite way a neighbourhood of data is usually missing, and this is often quite common in real time.

Let us now see how we will handle missing values (say NA or NaN) using Pandas.

# import the pandas library

import pandas as pd

import numpy as np

df = pd.DataFrame(np.random.randn(5, 3), index=['a', 'c', 'e', 'f',

'h'],columns=['one', 'two', 'three'])
df = df.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])

print df
Its output is as follows −
 one two three

a 0.077988 0.476149 0.965836

b NaN NaN NaN

c -0.390208 -0.551605 -2.301950

d NaN NaN NaN

e -2.000303 -0.788201 1.510072

f -0.930230 -0.670473 1.146615

g NaN NaN NaN

h 0.085100 0.532791 0.887415

Using reindexing, we've created a DataFrame with missing values. within the output, NaN means Not variety .

Check for Missing Values

To make detecting missing values easier (and across different array dtypes), Pandas provides the isnull() and notnull() functions, which also are methods on Series and DataFrame objects −

Example

import pandas as pd

import numpy as np

df = pd.DataFrame(np.random.randn(5, 3), index=['a', 'c', 'e', 'f',

'h'],columns=['one', 'two', 'three'])

df = df.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])

print df['one'].isnull()

Its output is as follows −

a False

b True

c False

d True

e False

f False

g True

h False

Name: one, dtype: bool

Cleaning / Filling Missing Data

Pandas provides various methods for cleaning the missing values. The fillna function can “fill in” NA values with non-null data during a few ways, which we’ve illustrated within the following sections.

Replace NaN with a Scalar Value

The following program shows how you’ll replace “NaN” with “0”.

import pandas as pd

import numpy as np

df = pd.DataFrame(np.random.randn(3, 3), index=['a', 'c', 'e'],columns=['one',

'two', 'three'])

df = df.reindex(['a', 'b', 'c'])

print df

print ("NaN replaced with '0':")

print df.fillna(0)

Its output is as follows −

 one two three

a -0.576991 -0.741695 0.553172

b NaN NaN NaN

c 0.744328 -1.735166 1.749580


NaN replaced with '0':

 one two three

a -0.576991 -0.741695 0.553172

b 0.000000 0.000000 0.000000

c 0.744328 -1.735166 1.749580

Here, we are filling with value zero; instead we will also fill with the other value.

Fill NA Forward and Backward

Using the concepts of filling discussed within the ReIndexing Chapter we’ll fill the missing values.

Method Action

pad/fill Fill methods Forward

bfill/backfill Fill methods Backward

Example

import pandas as pd

import numpy as np

df = pd.DataFrame(np.random.randn(5, 3), index=['a', 'c', 'e', 'f',

'h'],columns=['one', 'two', 'three'])

df = df.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])


print df.fillna(method='pad')

Its output is as follows −

one two three

a 0.077988 0.476149 0.965836

b 0.077988 0.476149 0.965836

c -0.390208 -0.551605 -2.301950

d -0.390208 -0.551605 -2.301950

e -2.000303 -0.788201 1.510072

f -0.930230 -0.670473 1.146615

g -0.930230 -0.670473 1.146615

h 0.085100 0.532791 0.887415

Drop Missing Values

If you would like to easily exclude the missing values, then use the dropna function along side the axis argument. By default, axis=0, i.e., along row, which suggests that if any value within a row is NA then the entire row is excluded.

Example

import pandas as pd

import numpy as np

df = pd.DataFrame(np.random.randn(5, 3), index=['a', 'c', 'e', 'f',

'h'],columns=['one', 'two', 'three'])


df = df.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])

print df.dropna()

Its output is as follows −

one two three

a 0.077988 0.476149 0.965836

c -0.390208 -0.551605 -2.301950

e -2.000303 -0.788201 1.510072

f -0.930230 -0.670473 1.146615

h 0.085100 0.532791 0.887415

Replace Missing (or) Generic Values

Many times, we’ve to exchange a generic value with some specific value. we will achieve this by applying the replace method.

Replacing NA with a scalar value is equivalent behavior of the fillna() function.

Example

import pandas as pd

import numpy as np

df = pd.DataFrame({'one':[10,20,30,40,50,2000],

'two':[1000,0,30,40,50,60]})

print df.replace({1000:10,2000:60})

Its output is as follows −

So, this brings us to the end of blog. This Tecklearn ‘Perform Data Cleansing in Python’ blog helps you with commonly asked questions if you are looking out for a job in Python Programming. If you wish to learn Python and build a career in Data Science domain, then check out our interactive, Python with Data Science Training, that comes with 24*7 support to guide you throughout your learning period. Please find the link for course details:

https://www.tecklearn.com/course/python-with-data-science/

Python with Data Science Training

About the Course

Python with Data Science training lets you master the concepts of the widely used and powerful programming language, Python. This Python Course will also help you master important Python programming concepts such as data operations, file operations, object-oriented programming and various Python libraries such as Pandas, NumPy, Matplotlib which are essential for Data Science. You will work on real-world projects in the domain of Python and apply it for various domains of Big Data, Data Science and Machine Learning.

Why Should you take Python with Data Science Training?

Python is the preferred language for new technologies such as Data Science and Machine Learning.
Average salary of Python Certified Developer is $123,656 per annum – Indeed.com
Python is by far the most popular language for data science. Python held 65.6% of the data science market.

What you will Learn in this Course?

Introduction to Python

Define Python
Understand the need for Programming
Know why to choose Python over other languages
Setup Python environment
Understand Various Python concepts – Variables, Data Types Operators, Conditional Statements and Loops
Illustrate String formatting
Understand Command Line Parameters and Flow control

Python Environment Setup and Essentials

Python installation
Windows, Mac & Linux distribution for Anaconda Python
Deploying Python IDE
Basic Python commands, data types, variables, keywords and more

Python language Basic Constructs

Looping in Python
Data Structures: List, Tuple, Dictionary, Set
First Python program
Write a Python Function (with and without parameters)
Create a member function and a variable
Tuple
Dictionary
Set and Frozen Set
Lambda function

OOP (Object Oriented Programming) in Python

Object-Oriented Concepts

Working with Modules, Handling Exceptions and File Handling

Standard Libraries
Modules Used in Python (OS, Sys, Date and Time etc.)
The Import statements
Module search path
Package installation ways
Errors and Exception Handling
Handling multiple exceptions

Introduction to NumPy

Introduction to arrays and matrices
Indexing of array, datatypes, broadcasting of array math
Standard deviation, Conditional probability
Correlation and covariance
NumPy Exercise Solution

Introduction to Pandas

Pandas for data analysis and machine learning
Pandas for data analysis and machine learning Continued
Time series analysis
Linear regression
Logistic Regression
ROC Curve
Neural Network Implementation
K Means Clustering Method

Data Visualisation

Matplotlib library
Grids, axes, plots
Markers, colours, fonts and styling
Types of plots – bar graphs, pie charts, histograms
Contour plots

Data Manipulation

Perform function manipulations on Data objects
Perform Concatenation, Merging and Joining on DataFrames
Iterate through DataFrames
Explore Datasets and extract insights from it

Scikit-Learn for Natural Language Processing

What is natural language processing, working with NLP on text data
Scikit-Learn for Natural Language Processing
The Scikit-Learn machine learning algorithms
Sentimental Analysis – Twitter

Introduction to Python for Hadoop

Deploying Python coding for MapReduce jobs on Hadoop framework.
Python for Apache Spark coding
Deploying Spark code with Python
Machine learning library of Spark MLlib
Deploying Spark MLlib for Classification, Clustering and Regression

Got a question for us? Please mention it in the comments section and we will get back to you.

635