# Detect and fix data issues

IntelliJ IDEA can automatically analyze datasets loaded into a pandas DataFrame and displayed in a Jupyter Notebook cell. The IDE highlights potential problems such as missing values, outliers, duplicate rows, or correlated columns — making it easier to clean your data before modeling.

IntelliJ IDEA automatically analyzes any DataFrame that meets the following conditions:

* Contains not more than 100,000 rows.

* Contains not more than 200 columns.

* Is displayed using `df`, `df.head()`, `df[x:y]`, `df.loc(.. .)`, or `df.iloc(...)`.

If at least one issue is found, the ![](https://resources.jetbrains.com.cn/help/img/idea/2026.2/app.general.inspectionsWarningIcon.svg) Dataset Issues button appears on the DataFrame toolbar.

## Fix data issues with AI

Procedure:

1. [Install and enable the
AI Assistant plugin](https://www.jetbrains.com.cn/en-us/help/ai-assistant/installation-guide-ai-assistant.html).

2. Click the ![](https://resources.jetbrains.com.cn/help/img/idea/2026.2/app.general.inspectionsWarningIcon.svg) Dataset Issues button in the DataFrame toolbar. The Dataset Quality Issues dialog opens.

![Issues button](https://resources.jetbrains.com.cn/help/img/idea/2026.2/py_data_issues_button.png)

3. Select the issues that you want to fix and click the Fix with AI button.

![Dataset Quality Issues dialog](https://resources.jetbrains.com.cn/help/img/idea/2026.2/py_data_issues_dialog.png)

A new code cell will be created below the DataFrame with the suggested changes.

> **Note:**
> This cell is not executed automatically. You can review and run it manually.

## Common issues and how to handle them

Procedure: Missing values

Some cells in your dataset may have no value, represented as `NaN`, `None`, or an empty string. This often happens during data collection, merging, or file import. If ignored, missing values can introduce bias into your analysis. Certain machine learning algorithms also cannot run on data that contain them.

For example, six empty cells were found in the `salary` column:

![Missing Values](https://resources.jetbrains.com.cn/help/img/idea/2026.2/py_data_issues_missing_values.png)

> **Note:**
> You can check which columns contain missing data, using `df.isna().sum()`.

Typical fixes to handle this issue include:

* Removing rows or columns that have too many unknown values: ```PYTHON df = df.dropna() ```

* Replacing unknown values with a constant, such as `0` or `Unknown`: ```PYTHON df['Name'] = df['Name'].fillna('Unknown') ```

* Replacing unknown values with the mean, median, or mode of the column: ```PYTHON df = df.fillna(df.mean()) ```

* Predicting and replacing the missing values using a machine learning model, such as linear regression or k-nearest neighbors: ```PYTHON from sklearn.impute import KNNImputer imputer = KNNImputer(n_neighbors=5) df_imputed = pd.DataFrame(imputer.fit_transform(df), columns=df.columns) ```

> **Tip:**
> It is important to understand the reason for missing values and how it may affect your data. For more details, refer to [this article](https://blog.jetbrains.com/pycharm/2025/01/data-cleaning-in-data-science/#missing-values).
>
>
>
> Each replacement method also has its own advantages and disadvantages and should be chosen in the context of your dataset.

Procedure: Duplicate rows

Some rows in your dataset may be identical, for example, due to repeated imports or logging errors. If ignored, they can create bias and make statistical results unreliable.

For example, two duplicate rows were found in the dataset:

![Duplicate Rows](https://resources.jetbrains.com.cn/help/img/idea/2026.2/py_data_issues_duplicate_rows.png)

> **Note:**
> You can identify the duplicates, using `df[df.duplicated()]`.

Typical fixes to handle this issue include:

* Removing all duplicate rows: ```PYTHON df = df.drop_duplicates() ```

* Removing all duplicates, except the first occurrence: ```PYTHON df = df.drop_duplicates(keep='first') ```

> **Tip:**
> If duplicates represent valid repeated events (for example, identical purchases), consider keeping them instead of removing.

Procedure: Outliers

Outliers are data points that differ significantly from most other values in the dataset. They can distort statistical results and affect machine learning models.

You may define an outlier as a value that exceeds an absolute threshold or falls outside a statistical boundary.

It is generally recommended to only remove outliers if you believe they belong to a different population from the one you’re attempting to target. See [this article](https://blog.jetbrains.com/pycharm/2025/01/data-cleaning-in-data-science/#addressing-outliers) for more details.

For example, outliers were found in the `salary` and `age` columns:

![Outliers](https://resources.jetbrains.com.cn/help/img/idea/2026.2/py_data_issues_outliers.png)

Typical fixes to handle this issue include:

* Discovering extreme values using a statistical method, such as mean absolute deviation: ```PYTHON from scipy.stats import median_abs_deviation median = data['colname'].median() mean_absolute_deviation = median_abs_deviation(data['colname'], scale='normal') data['modified_z_score'] = (data['colname'] - median) / mean_absolute_deviation data['is_outlier'] = (data['modified_z_score'] > 3) | (data['modified_z_score'] < -3) ```

* Removing rows with extreme values: ```PYTHON df = df[df['salary'] < 20000] ```

* Capping extreme values to a threshold: ```PYTHON df['salary'] = np.where(df['salary'] > 20000, 20000, df['salary']) ```

* Transforming values to reduce the effect of outliers (for example, by applying a logarithmic transformation): ```PYTHON df['salary_log'] = np.log1p(df['salary']) ```

> **Tip:**
> Visualize data distributions, using `df.boxplot()` or `df['column'].hist()` before and after removing or transforming outliers to confirm that the the distributions of your data look suitable for your planned analysis or model.

Procedure: Constant columns

Some columns in your dataset may contain the same value in all the rows. Such constant columns often do not add any useful information to your analysis or models, because they have no variability.

For example, the column `country` with the constant value `USA` was found:

![Constant Columns](https://resources.jetbrains.com.cn/help/img/idea/2026.2/py_data_issues_constant_columns.png)

> **Note:**
> You can identify single-value columns by checking how many unique values each column contains:
>
>
>
>
> ```PYTHON
> single_value_cols = [col for col in df.columns if df[col].nunique() == 1]
> print(single_value_cols)
> ```

Typical fix to handle this issue is:

* Removing all single-value columns: ```PYTHON df = df.drop(columns=single_value_cols) ```

> **Tip:**
> Consider keeping the constant columns, if they provide meaningful context.