# Spark DataFrame coding assistance

The Spark plugin provides coding assistance for Apache Spark DataFrames in your Scala and Python code.

> **Note:**
> The examples below are in Python, but the same actions are available in Scala.

## Completion for available columns

If you create a DataFrame or read it from a file, IntelliJ IDEA will assist you in accessing the DataFrame columns, for example, while selecting or filtering DataFrames.

![Column completion in PySpark](https://resources.jetbrains.com.cn/help/img/idea/2026.2/spark_dataframe_column_completion_pyspark.png)

## Detecting unresolved columns

If you refer to a column that doesn't exist in the DataFrame, IntelliJ IDEA highlights it and suggests replacing it with one of the available column names.

You can enable and disable this inspection in the IDE settings (`Ctrl+Alt+S` (Windows), `⌘ Comma` (macOS), `⌘ Comma` (IntelliJ IDEA Classic (macOS)), `⌘ Comma` (macOS System Shortcuts), `Ctrl+Alt+S` (XWin), `Ctrl+Alt+S` (GNOME), `Ctrl+Alt+S` (KDE), `Ctrl+Alt+S` (Emacs), `Ctrl+Alt+S` (Sublime Text), `⌘ Comma` (Sublime Text (macOS)), `Ctrl+Alt+S` (NetBeans), `Ctrl+Alt+S` (Visual Studio), `⌘ Comma` (Visual Studio (macOS)), `Ctrl+Alt+S` (Eclipse), `⌘ Comma` (Eclipse (macOS))), under `Editor | Inspections | Spark | Unresolved columns`.

![Column completion in PySpark](https://resources.jetbrains.com.cn/help/img/idea/2026.2/spark_dataframe_column_completion_pyspark.animated.gif)

## Getting a schema

Completion of column names and the corresponding inspection are available if IntelliJ IDEA can access the DataFrame schema. The schema can be specified in multiple ways:

* Columns and their types are specified directly in the `read` method: ```PYTHON df = (spark.read .schema("name STRING, value BIGINT, planet STRING") .parquet("aliens.parquet")) .parquet("aliens.parquet")) ```

* The schema is specified as a separate variable and then used in the `read` method: ```PYTHON schema = StructType([ StructField("name", StringType(), False), StructField("value", LongType(), False), StructField("planet", StringType(), False), ]) df = spark.read.schema(schema).parquet("aliens.parquet") ```

If you have not specified schema in either of these ways, you can use the dedicated inlay hint to infer the schema from a Parquet file. The file can be located locally or on a remote storage.

Procedure: Infer schema from a file

1. Use the `read.parquet()` method in your Spark code, for example:

```PYTHON
df = spark.read.parquet("/myfilepath")
```

2. Click the `Choose schema` inlay hint.

![Choose schema for dataframe](https://resources.jetbrains.com.cn/help/img/idea/2026.2/spark_choose_schema_for_dataframe.png)

3. In the window that opens, select a file from which the schema can be inferred.

> **Tip:**
> You can connect to any of the storages [supported by
> the Remote File Systems plugin](big-data-tools-remote-file-systems.html). If you want to specify a file located on your computer, [configure a local connection](big-data-tools-local-file-system.html).

The schema inferred from the selected file will be displayed as an inlay hint next to the method. You can hover over it to preview the available columns and their types. And you can click it to insert the schema using the `schema` method or to select another one.

![DataFrame Schema](https://resources.jetbrains.com.cn/help/img/idea/2026.2/spark_dataframe_schema_as_inlay.png)

You can enable and disable this inlay hint in the IDE settings (`Ctrl+Alt+S` (Windows), `⌘ Comma` (macOS), `⌘ Comma` (IntelliJ IDEA Classic (macOS)), `⌘ Comma` (macOS System Shortcuts), `Ctrl+Alt+S` (XWin), `Ctrl+Alt+S` (GNOME), `Ctrl+Alt+S` (KDE), `Ctrl+Alt+S` (Emacs), `Ctrl+Alt+S` (Sublime Text), `⌘ Comma` (Sublime Text (macOS)), `Ctrl+Alt+S` (NetBeans), `Ctrl+Alt+S` (Visual Studio), `⌘ Comma` (Visual Studio (macOS)), `Ctrl+Alt+S` (Eclipse), `⌘ Comma` (Eclipse (macOS))), under `Editor | Inlay Hints | Other | Python | DataFrame analysis`  for Python and `Editor | Inlay Hints | Other | Scala | DataFrame analysis` for Scala.