“Product engineer” used to mean you built the feature and a growth or data team decided how to test it. On most small teams that split doesn’t exist anymore. You write the code, you decide the split, you watch the numbers, and you decide whether to ship, kill, or iterate — usually before lunch.
That’s not a job description change so much as a tooling gap. Here’s the toolkit that gap actually calls for, and where it tends to break down.
What “day to day” looks like
It’s rarely one big quarterly experiment. It’s a stream of small ones, stacked on top of each other:
- A new onboarding flow, tested against the current one for two weeks.
- A pricing page layout change, watched at 10% before going wider.
- A ranking or recommendation tweak, split by user cohort.
- A checkout redesign, rolled back within the hour if drop-off spikes.
None of these justify spinning up a dedicated experimentation platform on their own. All of them need the same four things.
The four things you actually reach for
- An exposure mechanism. Something that decides, per request, which variant a user sees — and keeps deciding the same thing for the same user.
- A ramp. You don’t ship a test to 100% of traffic on day one. You ship it to 5%, watch it, then 25%, then wider.
- A readout. Some way to tie “user saw variant B” to “user converted,” without gluing that together by hand every time.
- A kill switch. Most experiments are bad ideas — that’s the point of testing them. You need to be able to turn one off as fast as you turned it on.
Where the homemade version breaks
The tempting shortcut is a hash of the user ID in an if statement, plus a homemade event in your analytics tool. It works, once. It stops working the moment you need a second variant, a different split for a specific segment, or to change the percentage without a deploy. We’ve written about exactly where that line is in Feature Flags vs Environment Variables— the short version is that consistent bucketing is easy to get approximately right and easy to get subtly wrong, and you don’t want to be debugging your own hashing function during an incident.
What this looks like with ToggleTown
A test starts as a plain string flag — one flag key, one value per variant:
import { ToggleTownClient } from '@toggletown/sdk-node';
const client = new ToggleTownClient('tt_live_your_api_key');
await client.initialize();
const variant = client.getStringFlag('checkout-layout', 'control', {
userId: user.id,
});
return variant === 'treatment' ? renderNewCheckout() : renderOldCheckout();The rollout percentage controls the ramp — 5% today, 25% tomorrow, with the same users staying on the same side as you widen it. When something looks wrong, the kill switch is the same toggle you used to turn it on.
For the readout, you send one event per conversion:
await fetch('https://api.toggletown.com/sdk/experiments/track', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-API-Key': apiKey },
body: JSON.stringify({
userId: user.id,
flagKey: 'checkout-layout',
eventName: 'checkout_completed',
}),
});Rollout vs. a real experiment
Not every test needs a full readout — most day-to-day ones, you can eyeball in whatever analytics tool you already have, segmented by flag value. When one actually matters — a pricing change, a checkout redesign — that’s when it’s worth creating a proper experiment: a hypothesis, named variants with explicit traffic allocation, a primary metric, and a start/pause/complete lifecycle instead of a flag you remember to turn off yourself.
{
"flagId": "...",
"environmentId": "...",
"name": "Checkout Layout Test",
"hypothesis": "The single-page checkout increases completion rate",
"primaryMetric": "checkout_completed",
"variants": [
{ "name": "control", "value": "control", "trafficAllocation": 50 },
{ "name": "treatment", "value": "treatment", "trafficAllocation": 50 }
]
}Rollout percentages and targeting rules are on every plan, including free. Formal experiments — with named variants, a primary metric, and a results readout — are a Pro-plan feature, because that’s the point where you’re past “quick test” and into something you’ll actually report on.
The point
You don’t need a growth team to run an A/B test responsibly. You need a flag that can ramp, target, and roll back without a deploy, and a place to point the results. That’s it. If you’re doing the same thing on the marketing side of the funnel, see A/B Testing for Growth and Marketing Engineers.