Project Nectarine

NES: Next Edit Suggestions

Evaluating how well AI code editors predict your next move

Presented by Shubhangi Thakur

Project Nectarine

The Road to Success in NES

A trained developer performs one real coding action, like a rename, a bug fix, or a refactor, in two AI-assisted editors side by side, then documents exactly what each editor’s AI suggested, whether it was correct, and how fast it appeared.

What we evaluate

Suggestion QualityScale: 1–5

LatencyScale: 1–5

Contextual UnderstandingScale: 1–5

Interruption FrequencyScale: 1–5

ImpactWhat our data unlocked

Freed up 1200 GPUs previously hosting cursor jump models
Decreased output tokens by 59%
Decreased prompt tokens by 5%
Reduced suggestion time to show by 7%
Reduced suggestion time to produce by 6%
Achieved an overall 57% cost savings compared to control

Phase 1The starting point

Model: The model made short, single edit guesses and often missed the developer’s intent.

Our data: 2,010 graded tasks mapped exactly where it failed.

Phase 2First head-to-head

Model: Two named model builds were tested side by side for the first time.

Our data: 1,000 comparison tasks, and NES Verifier was born.

Phase 3The leap to full patches

Model: From one small edit to proposing complete, multi step patches.

Our data: 1,000 diff patch tasks taught it to plan the whole change, not just the next line.

Phase 4Sharper precision

Model: Four candidate models refined in parallel, with clearer priorities.

Our data: 2,000 tasks with per patch priorities sharpened its precision.

Phase 5Where we are today

Model: Three tuned models now suggest precise, multi file changes as you type.

Our data: 500 tasks and counting keep raising the bar.

Task categorieshover

RenamingRepeated code detectionNew methodsCode rewritesBehavior changesBug detectionImportsMulti-file editsResponse speedCode navigation

Task workflowhover

01 Claim task
02 Write ground truth
03 Capture: VS Code
04 Capture: Cursor
05 Extract telemetry
06 NES Verifier (taxonomy fields)
07 Submit

Built our own toolhover

− Everything run directly through the Annotation Platform

+ NES Verifier, built in-house by the team’s own QMs

One tool for everything

Experts fill the full task taxonomy
Reviewers run quality control on every task
A reviewer sheet logs every verdict for double and triple checks
QMs manage operations, including payments

Human in the loop

Recurring Office HoursLive sessions with the expert pool every phase

Human review on every taskA reviewer checks each task and both recordings

Targeted spot checksExtra sampled audits logged in a dedicated sheet

Always on AI judgeAn automated eval reviews every task around the clock

Project Nectarine

What one task delivers

6,510tasks

delivered across five phases

225 GB+data shipped

59,000+ files

290humans in the loop

240 annotators and 50 reviewers

3,516automated eval verdicts

conducted on top of human review

Delivered by phasePhase 1: 2,010·Phase 2: 1,000·Phase 3: 1,000·Phase 4: 2,000·Phase 5: 500 and counting

NES Verifier

One NES task, packed and delivered

NES Verifier

telemetry.logevents.jsonlrecording.mp4task.json

ConsistencyQualityOn time

NES Verifier · task_0241

1 Claim2 Truth3 VS Code4 Cursor5 Telemetry6 NES Verifier form7 Submit

task_0241 · claimed ✓

Categorybug_detection

LanguagePython

Repositoryacme/payment-service

Entry filesrc/streams/writer.py · line 142

Modelpandia-4

Expected Patches GROUND TRUTH

p11 · Release-gating

- raise ValueError("bad stream")

+ raise InvalidArgument("Stream id not open")

Written before any AI sees the task

REC

VS Code Insiders

def write_stream(stream_id, buf):
check_range(stream_id)
raise InvalidArgument("Stream id not open")
raise InvalidArgument("Stream id not open")

Tab · suggestion accepted

REC

Cursor

def write_stream(stream_id, buf):
check_range(stream_id)
raise InvalidArgument("Stream id not open")
raise InvalidArgument("Stream id not open")

Same task, second editor

$ python extract_telemetry.py task_0241

━━━ task_0241 ━━━

VS Code: task_0241_vscode-insiders/

Cursor: task_0241_cursor/

stats: 15 opp / 9 shown / 5 accepted / 380ms avg

→ task_0241_telemetry.md ✓

1Task Identification✓

2Code Context✓

3Expected Patches✓

4Steps✓

5Suggestions by Editor✓

6Editor Metrics✓

7Overall Preference✓

8Session Files✓

task_0241-vscode-insiders/ · 6 files ✓ · task_0241-cursor/ · 4 files ✓

Submit task

✓ Task submitted, ready for review

Ground truth first: the ideal patches are written before any AI runs

The same task runs in both editors, recorded suggestion by suggestion

One script turns the raw logs into a clean telemetry report

Every section of the NES Verifier form gets filled and checked

One click and the task ships for review and delivery

First the task gets packed and delivered, then a guided look at how the taxonomy is completed in NES Verifier.

Output of one task

VS Code Insiders/

+telemetry-1.logModel events, accept/reject counts

+events-1.jsonlGhost-text & suggest-preview DOM events

+workspaceRecording.jsonlRaw suggestion stream, every insertText

+screenshot-1.pngWorkspace snapshot at capture stop

+recording-1.mp4Full screen recording, required for QC

+state.jsonCapture session metadata

Cursor/

+telemetry-1.logModel outputs, accept/reject counts

+events-1.jsonlInline suggestions & Tab presses

+screenshot-1.pngWorkspace snapshot at capture stop

+recording-1.mp4Source of truth for Cursor, no structured log

✓

Full taxonomy

predictions_by_step, every metric, and reasoning, submitted via NES Verifier to close the task

Humans in the loop

All of this is only possible thanks to the team