Product judgment for coding agents.
Agents can build the app. Senior PMs and designers tell them what to build, how it should work, and whether a real team would ship it.
Each task is a brief, a rubric of what shippable means, and graded attempts with a reason for every score.
What a task looks like
Task 7 of 25. Shared grocery list
Our grocery list app is single user. Add sharing so two people in one home can use the same list. Most of our users shop with a partner, and today they text each other screenshots. It has to work when one person is in the store with no signal. Out of scope: recipes and payments.
The rubric, 8 of 16 checks
- Inviting someone takes three steps or fewer from the listPass
- Each item shows who added itPass
- A checked item shows up on the other phone within secondsPass
- Changes made offline sync later without losing anythingMiss
- If both people edit the same item, neither change silently disappearsMiss
- The invited person can leave the listMiss
- An empty shared list says what to do firstPass
- Nothing out of scope was addedPass
Attempt 2 of 3: would not ship.
Why. It works in a demo with two phones online, but in a store with no signal the second person's checked items come back unchecked, which is the exact moment this feature exists for.
How labs use them
The rubric is the reward. Your agent learns what a senior PM checks before saying a feature is done.
The score is the eval. Track how often each model version ships product a real team would accept.
Misses are tagged by type, so you see where your agent falls short: edge cases, flows, states, scope, or design.
Run the same tasks through other agents and see where yours wins and loses, with reasons.
What you get each month
- 25 product tasks in the areas you choose, each with a brief and a rubric of 12 to 20 checks.
- 3 attempts per task from your agent, each graded by two senior PMs or designers. A third settles disagreements.
- A would-ship call and one sentence why for every attempt.
- The 10 misses your agent makes most, ranked, with examples.
- Everything as data files, ready for training or evals. It's yours.
- First delivery in 3 weeks. Cancel any month.
How we keep it good
- Real work. Briefs come from features writers have actually shipped, not puzzles.
- Checks anyone can verify. Every rubric line is pass or miss, with no vague scores.
- Two graders, then a third. Each attempt is graded twice on its own. A third settles disagreements.
- Known answer checks. Attempts with agreed scores are mixed into every batch. Graders who miss them are dropped.
- One sentence for every call. At least a dozen words on why. Vague sentences go back.
- Our agreement rate is in every delivery. You see how often the two graders matched.
Who writes the tasks
Senior product managers and product designers with 7 or more years shipping software. Every writer signs an NDA, passes a writing and grading test to join, and is scored on every batch. PMs and designers from Google, Adobe, Meta, Figma, Microsoft, Airbnb, Salesforce, and Stripe have attended our summits.
I'm Farid Ali. I've organized product and design summits for over 10 years: 49 editions, 7,300 tickets, people from 2,800 companies. The writers are people I know from them. I read every delivery before it goes out.
Farid Ali, founder, CoFi LabsPricing
Monthly, for labs
- Brief and rubric for each task
- 3 graded attempts per task
- Your 10 most common misses
- All of it as data files
- Paid by invoice
Volume, for labs
- Tasks in the areas you choose
- Your rubric format and file format
- Graded attempts as needed
- Weekly deliveries
Agent scorecard
- 20 tasks run through your agent and two competitors
- Every attempt graded twice
- Where you win and lose, with reasons
- 30 minute readout every month
Not sure yet? Ask for 3 sample tasks and try them on your agent first. Rankings are not for sale. Any public ranking we publish is on merit only.
Questions
New features, whole small apps, flows, and fixes to existing products. Web and mobile. You choose the areas.
For graded attempts, yes, or you send us the attempts. Tasks and rubrics alone need nothing from you.
You do. Tasks written for you are never sold to anyone else.
Yes. Every writer signs an NDA. We delete your outputs after delivery unless you ask us to keep them.
About 100 tasks a month to start, growing with you. Our writers come from a community of tens of thousands of PMs and designers.
Yes. Rubrics include how the result looks and reads, graded by senior designers.