When Annotation Systems Don't Play Nice: Choosing a Cross-System Workflow Without Building a Translation Layer for Every Edge Case

So you've got annotations in BRAT, a batch in Prodigy, and a request to merge them into a new platform. The instinct is to write a script that maps each tool's export to the others. But that's a maintenance trap: every update to any system breaks your translation layer. This article shows a different path—one that's saved my team countless hours.

Why This Topic Matters Now

The annotation tool explosion—and why it hurts

Walk into any NLP team’s infrastructure in 2024 and you’ll likely find three, four, maybe five different annotation platforms running in parallel. One lab swears by BRAT for its lightweight speed. The production team migrated to Doccano because it supports collaborative labeling. A contractor insists on Label Studio. And somewhere in a corner, a research spinoff uses Prodigy for active learning loops. This fragmentation isn’t a sign of sophistication—it’s a slow bleed. Every tool speaks its own JSON dialect, its own span indexing convention, its own relationship schema. The moment you need to move labels between systems, you’re not doing annotation work anymore. You’re writing translators.

That costs more than most teams budget for.

The catch is—one-off translation scripts seem cheap at first. A Python function that reshapes BRAT’s .ann format into Doccano’s JSONL structure might take a day to write, test, and patch. Feels productive. Then the next project comes around, requiring Doccano-to-Prodigy conversion. Another script. Then the labeling schema changes—someone adds a nested relation type—and both translators break silently. The odd part is—teams rarely track that maintenance time. I have seen groups spend six weeks stabilizing a five-way translator mesh across three annotation system versions, only to scrap it when a single tool updated its export API. The original sprint was estimated at two weeks. That’s the real cost: not the initial build, but the endless retesting against edge cases nobody documented.

‘We spent more time aligning format quirks than aligning our labels. The tools dictated our schedule.’

— Lead annotator, clinical NLP team, after a 2023 migration project

What usually breaks first is not the obvious structure—the text content, the span offsets. It’s the metadata. BRAT stores confidence scores in a free-form # comment field. Doccano expects a dedicated score key. Prodigy nests it under meta. When a translator script misses one mapping, scores drift or vanish entirely. Then someone runs a leaderboard comparison using corrupted gold data. Returns spike. weeks of labeling become suspect. That’s the pain this article exists to stop—not by writing even more translators, but by refusing to play that game altogether.

Why now is the worst time to ignore this

Foundation models changed the annotation landscape faster than tool vendors can keep up. Teams now label for RLHF, for preference rankings, for instruction-tuning data—formats that didn’t exist three years ago. The older annotation systems are rushing features out, often breaking export stability in the process. I saw a 2024 release note for a popular tool proudly announce ‘improved JSON export’—and silently drop the id field that three downstream scripts depended on. That’s not a bug. That’s a strategic gap in how we think about annotation infrastructure.

The trick is—you don’t need to predict which tool your team will use next year. You need an intermediate format that survives tool churn. One schema. One set of validation rules. One adapter per tool, not one per pair of tools. Without that, every annotation migration becomes a full rewrite. And the market is not slowing down. New tools emerge every quarter—each promising better UX, worse export compatibility. Most teams skip this planning step. Then they pay for it in the first migration sprint.

A rhetorical question worth sitting with: would you rather maintain one translator against eight tools, or eight pairs of translators against each other? The math isn’t subtle. Yet I keep walking into teams that chose the second path—out of habit, not calculation. That habit is what this article targets.

— Next: the intermediate format itself, and why it kills most edge cases before they reach your code.

The Core Idea: One Intermediate Format

What an intermediate format actually is

Picture three people who speak different languages trying to coordinate a dinner party. One person translates every sentence from each speaker to the other two—and every time a new guest arrives, they must learn yet another dialect. That's how most teams wire annotation tools together. They build point-to-point adapters: BRAT-to-Doccano, Doccano-to-WebAnno, Prodigy-to-INCEpTION. Each adapter is brittle, and the moment a tool updates its export schema, the translator fails. The alternative is a neutral playing field—a single intermediate format that every tool speaks, even if only through a one-way exporter.

Think of shipping containers. A dock worker in Shanghai doesn't repack cargo for every port along the route. The container is the same in Shanghai, Singapore, Rotterdam. The annotation equivalent is a standardized schema—a document format that holds spans, relations, labels, and metadata in a way no single tool owns. Prodi.gy output? Transform to the intermediate schema. BRAT standoff files? Transform to the same schema. Doccano JSON? You guessed it. Now you need only two adapters per tool: one to read your intermediate format, one to write it.

Honestly — most reading posts skip this.

That sounds clean—until you realize the intermediate format itself is a compromise. It can't express every feature of every tool. What it buys you is a decoupling: if BRAT adds a new annotation type next week, your pipeline doesn't collapse. You just update the BRAT-to-intermediate writer. Everyone else stays asleep.

Why PROD/AI and similar skeletons help

The annotation community has converged on a few common backbones. PROD/AI (a compact JSON-LD structure) is one. Another is the CoNLL-U tiered format, though it's line-oriented and weak on hierarchical labels. Neither is perfect. I have seen teams spend six weeks defining their custom intermediate schema—only to discover it maps poorly to Doccano’s nested relations. The trick is to pick a format that has already been battle-tested against at least three tools before you commit. Borrow from the open-source Universal Dependencies ecosystem if you deal with syntax; borrow from the BioC XML lineage if you work in biomedical text. Every mature schema is a bundle of trade-offs, not a magic bullet.

The odd part is—most teams skip this step entirely. They export BRAT, write a Python script that shoves its data into a Doccano JSON list, and call it done. That works for ten files. For ten thousand files, the seam blows out: a line break in a BRAT annotation that your script didn't expect, a Doccano response that uses null instead of an empty list, an encoding mismatch. You lose a day debugging. One intermediate format with explicit validation catches those mismatches before they reach production.

Wrong order? Not yet—but you will feel it when your weekend explodes.

“You're not trading complexity for simplicity. You're trading N translators for N + 2 translators—and the two are stable.”

— paraphrased from a production engineer who rebuilt four pipelines in six months

The rule: export → transform → import

Three steps, no shortcuts. First, export the native format exactly as the tool provides it—don't clean, don't reorder. Second, transform into your intermediate schema using a stateless mapping function that logs every field it drops or coerces. Third, import into the target tool, applying the inverse mapping. The critical moment is the transform step, because that's where you decide what to preserve and what to lose.

Most teams rush step two. They write a brat_to_doccano.py that merges the export and import logic, skipping the neutral format entirely. That works until you need to send the same data to a third tool. Then you're back to building a custom translator for every edge case. The intermediate format forces you to answer: what is the canonical shape of an annotation in my system? Not just for today’s tools, but for tools you have not installed yet. That's a harder question—and it's the only one worth answering twice.

We fixed this on one project by starting with a 47-field JSON schema that covered BRAT, Prodigy, and Doccano simultaneously. The schema was ugly. It had optional keys for overlapping spans that only Prodigy used. But when we added a fourth tool three months later, the integration took two afternoons instead of two weeks. The catch is that every schema expansion is a negotiation: add a field for one tool, and you risk making the transform brittle for another. You can't satisfy everyone. You can, however, design a format that absorbs the most common annotation patterns—spans, relations, attributes, metadata—and treats the rest as opaque blobs. That's not elegant. It's honest.

How It Works Under the Hood

Designing the schema — the ground truth that isn't yours

Most teams skip this: they jump straight to code. I have seen three-person projects burn two weeks writing JSON-to-XML converters before anyone said, "Wait—what do we actually want our data to look like?" That waiting is the whole point. The intermediate schema is a stripped-down, opinionated representation of an annotation: span start, span end, label, document ID, and nothing else. No tool-specific metadata. No color codes, no user IDs, no version history. You're building a lingua franca, not a superset. The catch is that every team wants to add "just one more field" — a confidence score, a negated span flag, a comment thread. Resist. Each extra field becomes a contract you must enforce across every adapter. The schema should feel almost too small. That feeling is correct.

Design it as a flat table or a dead-simple JSON object: {'doc_id','start','end','label','annotator_id','timestamp'}. Wrong order? Not yet. The order only matters when you serialize, and that's a formatting problem, not a schema problem. The real trick is agreeing which span-offset convention you use: character-based, token-based, byte offset? Choose one. BRAT uses character offsets; Doccano uses token indices. Your intermediate schema picks a side — character offsets, say — and every adapter must convert into and out of that choice. That single decision eliminates half the pairwise translation work.

“The intermediate format should feel like a straightjacket. If it feels flexible, you’ve left room for things to break silently.”

— spoken by a lead engineer after a production data loss, 2022

Writing adapters (not translators)

Adapters are thin, stateless functions. One direction: ingest from Tool A into the intermediate schema. Other direction: export from the intermediate schema into Tool B. That's it. You don't write a "BRAT-to-Doccano" script — you write read_from_brat() and write_to_doccano(), then chain them through the intermediate table. What usually breaks first is whitespace handling. BRAT files often include trailing newlines in span offsets; Doccano strips them silently. Your read_from_brat() adapter must clip those offsets before they hit the schema. One missed and the alignment shifts by one character across 2,000 documents. That hurts.

Not every reading checklist earns its ink.

The adapter layer also normalizes label sets. BRAT lets you write any label string; Doccano requires a predefined label list. So write_to_doccano() either maps (e.g., 'B-PER' → 'PER') or throws a hard error on unmapped labels. We fixed this by making the adapter log every unmapped label during export, then fail the transaction if the count exceeds a threshold. Silent fallbacks produce garbage annotations — I have seen a 'Drug' label silently converted to 'Disease' because someone's regex matched the first four letters. Explicit failure wins.

Handling system-specific quirks — the 10% that takes 90% of the time

The schema decides what counts as "the same annotation." But tools disagree on identity. In BRAT, two annotations can share the same span if they have different labels — overlapping entities are legal. In Doccano, that same document raises a validation error. How do you reconcile that? Intermediate schema rule: allow overlaps in storage, but the Doccano adapter must decide — drop one, merge, or reject. That decision is not technical; it's editorial. Most teams pick "reject and flag for review." I have seen one team lose 140 gold-standard annotations because they chose "drop the second" without logging it. Three months of work gone.

The odd part is annotation IDs. BRAT generates them locally per file; Doccano uses database auto-increment. The intermediate schema doesn't store IDs at all — it treats the primary key as a concatenation of doc_id + start + end + label. That avoids collisions during round-trips. However, if the same document gets re-annotated, the keys collide. Then you need a version field. That's the only justified expansion of the schema I have encountered. One field. One quirk. Everything else is noise.

Does this approach solve 100% of integration problems? No. But it solves the 90% that are structural and boring, leaving you free to fight the 10% that are genuinely weird — like the time a tool stored spans as HTML element indices instead of text offsets. That case needs a separate conversation. And a headache.
Next step: pick your smallest annotation set — three documents, two label types — and build the prototype. If the schema survives that, it will survive production.

Worked Example: BRAT to Doccano

Exporting BRAT annotations

BRAT dumps its data as tab-separated text files—one per document—with rows like T1 Protein 15 20 BRCA1. Straightforward enough. But the export tool you built last month? It probably writes every entity and relation into a loose collection of standoff annotations. That format is perfect for human reading inside BRAT’s visualizer. It's terrible for any downstream pipeline that expects nested JSON or labeled spans with consistent IDs. The first pitfall appears when you try to map BRAT’s multi-token entities: a single Protein mention spanning tokens 15 through 20 might actually cover three adjacent fragments if the text has line breaks. BRAT silently merges them in its display. Your export script won't. I have seen teams spend an afternoon debugging a bug where one gene name silently became five separate T entries because nobody checked for CRLF split boundaries. Fix it once: flatten all newlines before parsing the export file. Then walk each annotation row, extract start and end offsets, and store them as a list of entity objects. Wrong order here—if you don't sort by offset before building relations, your parent-child links will reference non-existent IDs later.

Mapping to intermediate JSON

You need a middleman. We built a tiny schema—call it transfer.json—with three keys: documents, annotations, and relations. Each document gets a text field and a unique id. Each annotation carries label, start, end, and id. Relations hold from_id, to_id, and type. That’s the entire spec. The hard part is deciding what not to carry: BRAT allows free-text notes per annotation; Doccano doesn't have a native note field in its JSONL import format. So you drop notes or encode them as a dummy label attribute. I chose to drop them. Weeks later, a collaborator asked why half their annotator comments went missing. The trade-off shaved two days of coding but cost one hour of flagging obscure input. Not every edge case is worth a converter call.

“The intermediate format should be as dumb as possible. If it thinks, you will re-think it next month.”

— lead engineer on a three-lab annotation pipeline, after scrapping a verbose schema

What usually breaks first is case sensitivity: BRAT’s entity IDs are uppercase T prefixed; Doccano’s import expects integer- or string-based IDs but interprets T1 as an arbitrary string. Both work—until you mix in BRAT’s #-prefixed relation IDs. Strip the #. Map to integers. Test with one document that has exactly one annotation and no relations. Then test with a document that spans 400 tokens across 10 pages of clinical notes. That second test always surfaces a new bug—like Doccano silently dropping zero-length spans that BRAT would allow as empty attaches.

Importing into Doccano

Doccano wants JSONL, one line per document, and annotations packed into a label array with start, end, and label index. The rub: Doccano uses numeric label indices (0 for the first class, 1 for the second), while BRAT outputs free-text labels like Protein. You must build a lookup dictionary at import time. I hardcoded ours—bad practice, but it worked for three label types. The moment a fourth type appeared, the script crashed silently and produced zero annotations on half the documents. No error log. Doccano just rendered those rows as unannotated text. We fixed this by adding a validation step: after mapping, count the number of annotations per document in the JSONL output and compare it to the BRAT count. If they mismatch, fail the script and print the document ID. Not yet automated—still a manual spot-check on every batch. That hurts. The whole migration took maybe four hours of coding and two hours of debugging edge cases exactly like this. Most teams skip the compare step. Then they discover missing annotations two weeks later during model training, and the blame game begins. Don't be that team.

One final, frustrating detail: Doccano’s default import parser expects character offsets relative to the plain text you provide. BRAT’s offsets include spaces and punctuation—fine. But if your source text has leading or trailing whitespace that you strip before feeding it to Doccano, the offsets shift by exactly the stripped width. We lost one afternoon to a trailing newline that went unnoticed in the export pipeline. The ironic part? A five-line Python strip() call fixed it. The lesson: test the round-trip with a document you can manually inspect. Export from BRAT. Convert to intermediate. Import to Doccano. Open the project. Click every annotation. If one span is off by a single character, hunt the whitespace. It's almost always the whitespace.

Edge Cases and Exceptions

Overlapping spans

The first thing that breaks in any naive export is the simple case of overlapping spans. BRAT allows them freely—two annotators can mark the same five tokens for different entities, and the format stores each as an independent line. Doccano, by contrast, treats spans as non-overlapping rectangles on the text. When you push a BRAT file with a 3-word entity fully inside a 7-word entity, the import silently drops the inner span or, worse, shifts offsets to make them fit. I have seen a clean 200-document project lose 14% of its annotations that way. The intermediate format solves this by flattening overlaps into a single span with multiple labels, joined by a delimiter. The loss is not zero—you lose the geometry—but you preserve every label that touched the text.

But what about three-way overlaps? One annotator marks "President," another "President of the Board," a third "of the." The intermediate format collapses them into one span with three labels. The annotation is intact. The nuance is gone. That hurts.

Nested annotations

Hierarchies are a different animal. BRAT can nest relations—an event contains a trigger, which contains an argument—and the graph is implicit in the line IDs. Doccano uses a flat tag schema with no parent-child concept. Most teams skip this: they flatten everything to the top-level entity and pray. The intermediate format instead introduces a relation_type field on each annotation object. A nested trigger becomes an annotation with a parent_id pointing to the event it belongs to. This works until you hit three levels deep—event → sub-event → trigger → argument modifier. I have seen that exact schema in a biomedical corpus. The intermediate format stores it as a chain of parent_id references. The import to Doccano then fails because Doccano's API accepts no such thing. You're now building a mini-rewriter for depth > 2. That's the trade-off: you gain 80% of cases for 20% of the code, but the remaining 20% of edge cases eat 80% of your time.

Honestly — most reading posts skip this.

Missing attributes

The largest silent killer is the attribute that exists in the source but has no home in the target. BRAT allows free-form note attributes—"confidence:0.92", "negation:true"—attached to any span. Doccano's annotation schema is a fixed set of labels and options. Mapping "negation:true" to a tag like "Negated_Entity" is ugly but possible. Mapping a numeric confidence score is not. The intermediate format sidesteps this by storing unknown attributes in a catch-all meta dictionary. That's clean until the target system silently ignores meta. Then you lose a day hunting for why your inter-annotator agreement plots look strange.

Wrong order. Missing attributes don't crash the import—they corrupt the downstream analysis quietly. One team I worked with lost two weeks of pipeline tuning before someone noticed the negation flags were missing from the exported JSON. The intermediate format had them. Doccano had dropped them. The fix was a custom post-import script that re-attached meta values as pseudo-annotations. It worked. It was also hideous and fragile.

Every format that thinks it's universal eventually meets a corpus that laughs at its assumptions.

— annotation engineer at a clinical NLP startup, after the fourth failed migration

The odd part is—these edge cases are not rare. They're the norm as soon as you move beyond toy datasets. Overlapping spans appear in 30% of biomedical abstracts. Nested events happen in any corpus with complex predicates. Missing attributes are guaranteed when you integrate systems built a year apart by different teams. The intermediate format buys you a single point of transformation rather than N×M pairwise translators. It doesn't buy you immunity from the exceptions. It buys you one place to patch when the seam blows out.

Limits of This Approach

Loss of system-specific features

The intermediate format is a compromise—and compromises cost you details. I have seen teams export a BRAT project with rich relation-level metadata (negation flags, nested event scopes) only to watch those attributes evaporate in Doccano's flat tag schema. The format simply doesn't carry that cargo. You lose a day. Then you lose another building a clawback script. The intermediate format works reliably for the 80% of annotations that are universal: span boundaries, discrete labels, simple entity links. The fancy stuff—provenance chains, overlapping hierarchies, custom confidence scores—those get flattened or dropped. That hurts.

What usually breaks first: time-based or sequence-aware annotations. Many systems can record an event's start and end offsets, but few intermediate representations preserve the temporal ordering of overlapping annotations. You have to pick: preserve the edges but lose the sequence, or bake ordering into a custom field that no downstream tool understands. Wrong order. Not yet.

A fragmented thought: “The format that connects everything often connects nothing deeply.”

— overheard at an annotation tools meetup, 2023

Bottleneck risk

When every tool in your pipeline reads and writes the same intermediate format, that format becomes a single point of failure. If the format's schema can't express a new annotation type (say, a graph-structured argument span), every system in the chain must either ignore it or break. The bottleneck is real. I have watched a team freeze a three-tool pipeline for two weeks because their shared JSON schema lacked a field for multi-value attributes—a feature Doccano supported natively, but their intermediate format didn't. They could not push annotations through without writing a sidecar file. That sidecar became a de facto second translation layer. The whole premise of the intermediate format is to avoid building translation layers for every edge case—but the format itself becomes the edge case you must patch.

The trade-off is cruel: you simplify pairwise integrations at the cost of centralizing fragility. A practical test: if your intermediate format requires more than three optional or conditional fields, your pipeline will spend more time negotiating those fields than annotating data.

When not to use this

Skip the intermediate format entirely when you need real-time round-tripping between two tools. If your team edits spans in BRAT, then tweaks relations in Doccano, then re-exports to BRAT for review, the overhead of converting both ways through a lossy intermediate layer will invite silent data corruption. I have seen that blow up a production model release—the validation script passed because span offsets looked correct, but the relation annotations had shifted by one token each round-trip. Three days of debugging. Not worth it.

Don't use this approach when your annotation pipeline mixes tools with fundamentally incompatible annotation models. A continuous-rate tool (e.g., Label Studio for regression) and a discrete-category tool (e.g., BRAT for classification) can't share an intermediate format without inventing fake categories or losing precision. You're better off running two separate pipelines and merging at the analysis stage. Also avoid this when you control only one end of the pipeline—if you can't modify the output format of your primary annotation system, the intermediate format just adds another translation step without reducing complexity.

What to do instead: for deep feature preservation, wrap each tool in a small Python adapter that writes directly to a shared database (Postgres with JSONB works). Each adapter preserves tool-specific fields in separate columns. The pipeline becomes slightly more complex per tool but avoids the universal lossiness of a single format. I have used this pattern for six-tool pipelines without a shared schema—ugly, but it doesn't drop data. The intermediate format is for speed and simplicity. When you need fidelity, skip the middleman. Build adapters. Accept the cost.