Engineering scan: notebooks, jobs, pipelines and DDL in Databricks and Snowflake

Data quality depends on the code that produces the data. On a Databricks or Snowflake connection, BeryX can also read the notebooks, jobs, pipelines, ML models and the DDL of views, tasks, streams and procedures, and look for risky practices in them: credentials written in code, SELECT *, scheduled jobs without retries, a timeout or an alert, notebooks that write tables with no job at all, writes to the Gold layer straight from Bronze, suspended tasks, stale streams and warehouses without auto-suspend.

Each heuristic that finds something opens one issue of the «Engineering» type, with the list of affected assets. A workspace with 300 notebooks using SELECT * has one practice issue, not 300.

The scan is opt-in per connection and read-only. On Databricks it only makes GET requests to the official APIs; on Snowflake it only runs catalog SHOW and SELECT statements. The code it reads is not stored: issues keep the finding type, the asset name and the line, and a credential's value is never shown or saved.

What it reads

The scan uses the connection's own credentials. Each collection is read separately: if the token or the role has no access to one of them, that collection is left out, the scan carries on and the panel lists it under «Partial coverage».

Databricks

With the connection's host and token, GET requests only, through the same read-only HTTP client and the same anti-SSRF guard as the other connectors:

The token needs read access to the workspace folders, jobs, pipelines and Model Registry you want analysed. Whatever it cannot read goes under «Partial coverage».

Snowflake

With the connection's user and role, catalog commands only. Before reaching the driver, each command goes through a guard that refuses anything other than a single SHOW, SELECT or DESCRIBE:

The role only sees the objects it holds some privilege on. There is never a SELECT on table data.

What it looks for

One issue per heuristic and per connection, with up to 50 assets listed (the count is the full one):

On each scan, an issue that persists is updated (an «acknowledged» one stays acknowledged), one that disappears becomes resolved and a new one is created. Turning the scan off resolves the connection's open engineering issues.

The same reading of the notebooks also proposes candidate rules: assertions written in the code (assert, checks with raise, dropna, dropDuplicates) become rule suggestions for the tables BeryX already knows, with the anonymised snippet as evidence. They show up in Suggestions with the «Mined from notebooks» origin.

Turning it on and requesting it in the app

1. In Connections, the card of a Databricks or Snowflake connection has the «Engineering scan» line, with the state (on or off) and the number of open findings. Click Open.
2. In the panel, an admin clicks Turn on scan.
3. With the scan on, click Request scan. If you want, tick Add an AI summary first (see What goes to the AI).
4. The scan runs in the background. The panel reads the state again until it finishes and shows the open findings per heuristic, what was read (notebooks, jobs, views…) and what was left out.
5. See the engineering findings in Issues opens /problemas?type=engineering, with the «Engineering (notebooks, jobs, DDL)» filter already applied. Each finding in the panel also leads to its issue.

A refused API or command only leaves that collection out. A scan that fails, for example with no connection to the source, leaves no record in the panel: test the connection and request it again.

Through the API

Three endpoints on the connection:

# State (a read-only dvk_* token is enough)
curl -s "$DATAVAULT_API_URL/api/v1/connections/$CONNECTION_ID/engineering" \
  -H "Authorization: Bearer $DATAVAULT_TOKEN"

# Turn it on and request the scan (an admin's session)
curl -s -X PATCH "$DATAVAULT_API_URL/api/v1/connections/$CONNECTION_ID/engineering" \
  -H "Authorization: Bearer $SESSION" -H "Content-Type: application/json" -d '{"enabled": true}'
curl -s -X POST "$DATAVAULT_API_URL/api/v1/connections/$CONNECTION_ID/engineering/scan" \
  -H "Authorization: Bearer $SESSION" -H "Content-Type: application/json" -d '{"llm_summary": false}'

• dvk_* API tokens are read-only: they work for the GET. The PATCH and the POST are done in the app or with a user's session.
• The POST answers 409 when the scan is off or inside one of the connection's blackout windows («DBA settings»). The PATCH answers 400 when turning it on for a connection that is neither Databricks nor Snowflake.
• In production, the POST queues the scan and answers {"status": "queued"}. Without a queue (a development install), it runs straight away and answers {"status": "completed", …} with the findings.
• Turning it on and off is recorded in the workspace audit log.

What is read-only and what is stored

• At the source: GET requests only (Databricks) and catalog SHOW/SELECT/DESCRIBE only (Snowflake). Nothing is created, changed or run at the source.
• In BeryX: the code is analysed in memory and discarded at the end of the scan. Issues keep each asset's name (notebook path, job name, schema.objeto) and a short detail («SELECT * na linha 12», «token Databricks (linha 4)»). A name that looks like a credential is also replaced before it is saved.
• Telemetry: the number of requests, the bytes read and the duration are kept in the scan record, next to the limits used.

What goes to the AI

Only when you ask for Add an AI summary (llm_summary: true), and only if there are findings.

• What it gets: the source (Databricks or Snowflake), the inventory counts and, for at most 15 heuristics, the name, severity, title, count and up to five examples (type, name and detail of the asset). All of it goes through credential and email removal first.
• What it never gets: the code, the notebooks' content and a credential's value.
• Personal folders: on Databricks, a personal notebook's path holds its owner's email (/Users/<email>/…). It goes as /Users/<utilizador>/…, and any other email as <email>.
• Data class: metadata_only. The task is called «Engineering scan» in the workspace AI policy: with AI turned off there is no summary, and the allowed models, your own key and the BYOK routes apply to it too. By default it goes to Claude Haiku by Anthropic.
• Cost: it counts towards the workspace AI budget, like the other tasks.
• With no provider available, the budget used up or AI turned off, the scan still runs and the panel says why there is no summary. The summary carries the «AI-generated content» label.

Who can do what

There is no plan limit: the scan is on every plan and does not count towards the scan quota. The AI summary counts towards the plan's AI budget.

Limits

Each scan has caps, so it does not weigh on the source or hold the worker. A connection can lower or raise each one with engineering_limits in the config sent through the API when the connection is created, never above the ceiling. The app has no field for this.

• A cap that is hit shows under «Partial coverage» and leaves the collection incomplete. With jobs or pipelines incomplete, the orphan_notebook heuristic is not assessed.
• On reaching timeout_s or max_requests, the scan stops and keeps what it has read. A hung request that goes past timeout_s plus 60 seconds fails the scan without saving anything.
• SELECT *, secrets and the Gold/Bronze layer are read from text patterns, without running the code: a name built at run time slips through, and a false positive is possible.
• The layer comes from names only. A lakehouse with no bronze, silver or gold (or equivalents) in its names has no gold_direct_write findings.