15.4 Exploratory data analysis

Exploratory Data Analysis (EDA) is a crucial step in understanding datasets. It involves examining data structure, uncovering patterns, and generating insights. EDA helps identify data quality issues and guides further analysis.

In Python, EDA techniques include data retrieval, filtering, and handling missing values. These methods allow you to select, subset, and clean data effectively. Visualization and statistical analysis further enhance your understanding of the dataset's characteristics and relationships.

Introduction to Exploratory Data Analysis (EDA)

Purpose of exploratory data analysis

Top images from around the web for Purpose of exploratory data analysis

• 

  Exploratory Data Analysis View original

  Is this image relevant?

• 

  Exploratory Data Analysis View original

  Is this image relevant?

• 

  Exploratory Data Analysis View original

  Is this image relevant?

• 

  Exploratory Data Analysis View original

  Is this image relevant?

• 

  Exploratory Data Analysis View original

  Is this image relevant?

1 of 3

Top images from around the web for Purpose of exploratory data analysis

• 

  Exploratory Data Analysis View original

  Is this image relevant?

• 

  Exploratory Data Analysis View original

  Is this image relevant?

• 

  Exploratory Data Analysis View original

  Is this image relevant?

• 

  Exploratory Data Analysis View original

  Is this image relevant?

• 

  Exploratory Data Analysis View original

  Is this image relevant?

1 of 3

• Gain deep understanding of dataset by examining its structure, dimensions, and data types
• Uncover hidden patterns, relationships, and anomalies within the data (correlations, trends)
• Generate valuable insights and hypotheses to guide further analysis and decision-making
• Identify data quality issues and preprocess data for subsequent modeling or analysis

Data retrieval with indexing

• Select single column using square brackets and column name

  df['age']

• Select multiple columns using list of column names

  df[['name', 'age', 'city']]

• Access single column as DataFrame attribute using dot notation

  df.age

• Use

  [loc[]](https://www.fiveableKeyTerm:loc[])

  for label-based indexing to select rows and columns by their labels

  df.loc[2:5, 'name':'age']

• Use

  [iloc[]](https://www.fiveableKeyTerm:iloc[])

  for integer-based indexing to select rows and columns by their integer positions

  df.iloc[1:4, 2:5]

Filtering and slicing for subsets

• Create boolean mask based on condition and use it to filter rows

  filtered_df = df[df['age'] > 18]

• Select range of rows by integer positions using slicing

  df[2:6]

• Select range of rows by labels using

  loc[]

  df.loc['2022-01-01':'2022-01-07']

• Filter rows using

  query()

  method with boolean expression as string

  filtered_df = df.query('age > 18 & city == "New York"')

• Combine multiple conditions using boolean operators

  &

  for AND and

  |

  for OR

  filtered_df = df[(df['age'] > 18) & (df['city'] == 'New York')]

Detection of missing values

• Use

  [isnull()](https://www.fiveableKeyTerm:isnull())

  or

  [isna()](https://www.fiveableKeyTerm:isna())

  to create boolean mask indicating missing values

  df.isnull()

• Count missing values in each column using

  sum()

  df.isnull().sum()

• Missing values reduce sample size, lead to biased or inaccurate results, and pose challenges for machine learning
• Identify type of missingness:
  1. Missing completely at random (MCAR) values are independent of other variables
  2. Missing at random (MAR) values depend on observed variables
  3. Missing not at random (MNAR) values depend on unobserved variables

Strategies for handling null data

• Remove rows with missing values using

  [dropna()](https://www.fiveableKeyTerm:dropna())

  df.dropna()

• Remove columns with missing values using

  dropna()

  with

  axis=1

  df.dropna(axis=1)

• Fill missing values with specific value using

  [fillna()](https://www.fiveableKeyTerm:fillna())

  df.fillna(0)

• Fill missing values with mean or median of column:
  • Mean imputation

    df.fillna(df.mean())

  • Median imputation

    df.fillna(df.median())

• Forward or backward fill missing values using

  [ffill()](https://www.fiveableKeyTerm:ffill())

  or

  [bfill()](https://www.fiveableKeyTerm:bfill())

  df.fillna(method='ffill')

• Impute missing values using advanced techniques (k-Nearest Neighbors, regression models)
• Data cleaning techniques help address missing values and improve data quality

Exploratory Analysis Techniques

• Descriptive statistics provide summary measures of central tendency, dispersion, and shape of data distribution
• Data visualization techniques help identify patterns, trends, and outliers in the data
• Univariate analysis examines the distribution of a single variable
• Bivariate analysis explores relationships between two variables
• Multivariate analysis investigates interactions among three or more variables
• Exploratory graphs (e.g., histograms, scatter plots, box plots) offer visual insights into data characteristics