abraxaslabs.tech  ·  github.com/abraxas  ·  @abraxas_null  ·  [email protected]  ·  spark-python-ds-static

spark-python-ds-static

Class: Privilege leftover Reach: Remote

Apache Spark 4.2.0 (32f72996) - Apache Software Foundation

DataSourceManager.dataSourceBuilders is a JVM-global first-wins map. SPARK-45917 auto-registers pyspark_* Python Data Sources from the lookup worker's sys.path. Connect session isolation keeps artifacts under SparkFiles/<sessionUUID>/, and per-session registerPython stays on runtimeDataSourceBuilders. The first lookup still puts the triggering session's SparkFiles dir on sys.path, pickles every DefaultSource, and caches those classes in the companion object. Later Connect SQL (or Hive Thrift SQL) read.format(name) / USING name runs that class as the Spark UID without sending Python proto.

The planter is a Connect Python tenant who is already allowed Python. The victim is a SQL-only tenant on the same Connect driver. Java aliases (parquet, csv, json, orc, jdbc) stay on the Java ServiceLoader path. Later plants after the map is filled do not refresh. AC:H is first-lookup plus a name that is not a Java alias.

ID	no CVE yet
Class	Privilege leftover (cross-session Python Data Source as Spark UID; not unauth job submit)
Reach	Remote (shared Spark Connect driver)
CWE	CWE-269, CWE-502
CVSS	High: 8.8 CVSS:3.1/AV:N/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:H
Product	Apache Spark
Affected	4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8) Python Data Source static registry
Auth	Connect Python tenant (planter) plus a later SQL-only Connect / Hive Thrift SQL user (victim)
License	GNU Affero GPL v3.0
Lab	127.0.0.1:18970 only. Witness file. Gadget of the companion map.

What an attacker can do

Hold a Spark Connect Python session on a shared driver. Plant pyfiles/pyspark_<name>.py whose DefaultSource.name() is an absent connector short name (delta, iceberg, hudi, ...). Trigger the first Python Data Source lookup on that session. Every later session in the same JVM that read.format(name) or SQL USING name instantiates the pickled class as the Spark UID on the driver planner worker and on executors.

That is the full privilege set. The planter already runs Python. The leftover is that a SQL-only tenant, who never sent REGISTER_DATA_SOURCE / Python UDF proto, still executes that class.

Integrity (lab-proven)

Method	Sink	Privilege
Planter: AddArtifacts name pyfiles/pyspark_<name>.py then first format(name) / lookup	companion DataSourceManager.dataSourceBuilders (DataSourceManager.scala object) plus lookup_data_sources.py iter_modules of pyspark_*	Cache the triggering session's pickled DefaultSource JVM-wide. Lab: layer='static' trigger='attacker' first-wins=yes.
Victim: SQL USING <name> / spark.read.format(name) on a session that never planted	PythonDataSourceV2 createDataSourceInPython	Run that class as Spark UID. Lab: planted_by='attacker' witness_written=True static-cross-session=yes. Oracle file holds SPARK-PYTHON-DS-STATIC-WITNESS.

Closed on this walk (lab-proven)

Method	Sink	Result
Later plant of a second name after the map filled	same static map	Closed. First-wins. Lab: later-lookup found=False, companion still only delta.
Per-session registerPython("alpha")	runtimeDataSourceBuilders	Isolated. Clone copies runtime only. Lab: runtime-isolated=yes. Victim read of alpha is found=False.
Java alias parquet	Java DataSourceRegister ServiceLoader	Closed. Lab: java-alias-closed=yes layer='java'. Same for csv/json/orc/jdbc.

Inferred (same sinks, not the oracle)

Method	Sink	Privilege
Victim USING after plant	driver + executor Python workers	Cluster-wide code as Spark UID. The lab writes a witness file instead of a payload.
Same path	worker crash / spin	Availability as Spark UID. Not the High. The High is the code exec.
Hive Thrift SQL USING on a shared SparkSession	same static map	Same leftover if Thrift and Connect share the driver JVM. Not exercised.

What this leftover does not grant

• Unauthenticated job submit on an open cluster. SECURITY.md FAQ. By design.
• Java parquet / csv / json / orc / jdbc hijack. ServiceLoader wins.
• Artifact path escape (concatenatePaths rejects .. and absolute names).
• Client-supplied pythonExec. Connect planner uses server PYSPARK_PYTHON / python3.
• Cross-session leak of registerPython. Runtime map is per DataSourceManager instance.
• Single-user local pyspark with one session. There is no other tenant.
• A miss-first lookup that fills an empty map. Later plants do not refresh (AC:H).

How I found it

I pinned apache/spark v4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8). Twenty-eight source hunts. Job submit executing user code is by design. History Jackson, UI shell, PySpark local impersonation, and standalone RPC receive-path are closed on this peel. Connect artifacts isolate pyfiles under a session UUID. Runtime registerPython is per session. The companion object is not.

SPARK-45917 auto-registers pyspark_* packages from PYTHONPATH. That is intended for operator-installed connectors. The leftover is that the lookup worker also sys.path.inserts the triggering session's SparkFiles dir (worker_util.setup_spark_files), then pickles whatever pyspark_* DefaultSource it finds into a process-global map that never refreshes.

Wrong turn: treating registerPython isolation as the whole tenant bar. Runtime is isolated. Static is not.

Lab

cd lab
./run.sh

Loopback 127.0.0.1:18970. Compose project spark-python-ds-static. Witness SPARK-PYTHON-DS-STATIC-WITNESS. Python gadget of object DataSourceManager plus per-session runtime maps. Plant pyspark_delta.py only in attacker SparkFiles. Attacker lookup fills the companion. Later plant of iceberg does not refresh. Victim read of delta writes the oracle. registerPython("alpha") does not leak. Java parquet stays Java.

Last line:

SUCCESS SPARK-PYTHON-DS-STATIC static-cross-session=yes runtime-isolated=yes first-wins=yes java-alias-closed=yes SPARK-PYTHON-DS-STATIC-WITNESS

The fix

Do not put session SparkFiles on the lookup worker sys.path when filling the static map. Load operator PYTHONPATH packages only (SPARK-45917 as designed). Or key the static map by session / JobArtifactSet UUID, the way artifacts already are. First-wins empty fill should not pin another tenant's classes for the life of the driver.

Until then, do not share a Connect driver between a Python tenant and a SQL-only tenant you would not already trust with driver Python.

References