I accidentally found myself inside a shadow data-labelling operation which distills frontier LLMs (Opus 4.8, GPT-5.5) to package post-training data for other labs.

Glossary

Model distillation: Distillation allows you to take knowledge present in a larger model and transfer it to a smaller model. This results in a smaller model with the same quality as the larger model on a subset of relevant tasks.

How it started

It all started with a vaguepost whatsapp message:

Looking for cracked people willing to work on frontier ai models and benchmark suites | Remote/sf | Paid(well) | need to be well versed with functioning of llms and data

The ever-curious engineer in me reached out, got interviewed on discord, hired (within the hour!), onboarded (within the next hour!) onto github, AWS, and google sheets. I had my suspicions right from the interview, but decided to play along to find out more.

I was then told I'd be "reviewing" tasks for the first week, post which I could be promoted to "pipelining".

During my onboarding, I was also told what I would have to do. I had to review PRs on github which were essentially agent runs by claude code/codex on different types of tasks (GIMP photo editing, audacity audio editing, etc) and look out for:

1. Certain metrics which they wanted averaged across the agent runs
2. Overall pass/fail rate
3. The traces of these agent runs, to ensure "[[fairness]]" of task

Fairness was a function of the inputs given to the model and the expectations from it. A task asking the model to recreate an audio from scratch with no details would fail the fairness check. A task asking it to remove background noise from an audio in audacity would pass.

How it worked

I was also onboarded onto “Launchpad”, their internal tool which gave me access to their binaries of Claude Code (“claud”) and Codex (“codx”), which cycled through hundreds of subscriptions to save on API pricing. I was told to use the best models with the highest effort on either of them while reviewing the PRs.

How the github looked

Why are these PRs valuable?

Every trace contains:

• prompts
• intermediate reasoning traces
• tool calls
• edits
• metadata

That's almost exactly the kind of high-quality data needed for post-training an LLM for better agentic capabilities.

The entire operation was super organized. There were ~2500 open PRs, each of different “task type”. I found the list of all task types for this particular project on google sheets:

After my onboarding, I had an informal chat with one of the folks on the discord who told me it was an “open secret” which lab this project was for.

I kept digging and came across several internal documents worth looking at.

Requirements of the “client”

Annotated transcript between the datafarm and “client” representative

The above screenshot is the first place I saw explicit references tying the project to Meta.

Discussion wrt LLMs for the agent runs

The sheets also documented their milestones and goals:

Here’s the requirements they had internally:

And probably the most damning of them all, here’s an internal document where they remark on what could’ve been better, but also pat themselves on the back and namedrop Meta’s Muse family of models. The document states how their data is “almost certainly going into muse and training it”.

The working conditions

The operation also relied heavily on inexpensive contract labor. Most everyone I interacted with on the discord were fellow Indians. The offered pay for this role was 250 USD for the work trial and 1.25k USD per month for full-time. I was informed during the interview that I would be required to work for ~16 hours a day, and I confirmed this number with the people on the discord.

The discord server had the stench of power-drunk, exploitative, self-styled “overlords” who got people to work themselves to the bone. Language like “slave them harder” was a common sight.

Some thoughts and questions

• Meta’s Superintelligence Lab clearly has the talent and resources to do this on their own, did they choose to outsource it because of legality?
• Is post-training the next scaling law for LLMs?
• Cycling through subscriptions to get subsidized inference instead of using the API is unethical and would likely violate OpenAI/Anthropic’s ToS.
• What kind of strategic partnerships allow the datafarm to extract more out of OpenAI’s models? (I’m guessing its just cheaper because of their relatively generous limits)
• What kind of infra is required to enable LLM-as-a-judge RL for a lab? Are verifiable rewards just easier because they can be deterministically and programmatically verified?
• Alexandr Wang just announced the next update to Muse Spark and claimed big improvements in “coding and agentic capabilities to be more competitive with other leading models”.

What I know

• I was onboarded.
• I saw PRs with agent runs, traces, and metrics.
• I saw internal documentation.
• I saw references to Meta.

What I don't know

• I don't know whether the data ultimately entered training.
• I don't know the contractual relationship.
• I don't know whether every project followed the same workflow.

Conclusion

The documentation I saw strongly suggested this project was being run for Meta. Separately, the workflow involved generating large quantities of high-quality outputs from frontier models and harnesses on a variety of tasks. That combination raises obvious questions about whether the resulting data was intended for post-training.

Whether or not this specific data ultimately made its way into a production model, one thing became obvious to me: post-training data has quietly become an industry of its own. The frontier race isn't just about bigger models with trillions of parameters anymore — it's increasingly about who can build and control the plumbing around the highest-quality training signals.