A/B tests — compare widget variants

Instead of guessing which greeting text converts better, let your visitors decide. Two widget configurations run in parallel; half of the visitors see variant A, the other sees variant B. After a few weeks, the dashboard shows which variant statistically significantly wins.

How an Experiment Works

1. You create two (or more) widgets — for example, a clone of your standard widget with a different greeting text. Other config values (colors, position) remain the same.
2. In the tab Settings → A/B Tests, you create an experiment: name + which two widgets are compared + distribution (e.g., 50/50).
3. Click on Start → from then on, each new visitor is assigned a variant via deterministic hash. The same visitor lands in the same variant on the next visit.
4. Per conversation, SilentChat persists the variant_assignment ({group_id, label} JSONB). This ensures the statistics remain attributable, even if the experiment ends later.

Who Lands in Which Variant?

The assignment is deterministic: sha256(visitor_fingerprint + group_id) mod 100 results in a bucket 0-99. With 50/50, visitors with bucket 0-49 land in variant A, 50-99 in variant B.

Properties of this approach:

• Same visitor → same variant across page reloads and even between sessions (as long as the fingerprint is stable).
• Group-ID as salt isolates experiments from each other: a visitor who landed in "A" in experiment X has an independent 50/50 chance of "A" or "B" in experiment Y. No bias leak.
• As an additional safeguard, the widget stores the result in localStorage (sc_variant_<public_key>, value {group_id, label}). During initialization, the widget sends this value; the server prefers it over a new hash pick. This protects against fingerprint drift (browser update, privacy extension).

Statistics & Winner Suggestion

The dashboard shows per variant:

• Conversations — how many visitors landed in this variant
• Messages — engagement signal (chatty visitors)
• Conversions — conversations that (a) have at least one visitor message AND (b) have reached a status of closed/resolved/assigned or have been assigned to an agent. This conversion signal covers lead capture, ticket creation, and live handoff.
• Rate — conversions / conversations
• ≥ Confidence (95%) — the lower bound of the Wilson 95% confidence interval. This is the number you look at when declaring a winner — not the raw rate (raw rates lie with small N).

Winner badge only appears if both conditions are met:

1. Minimum 200 visitors per variant (sample-size-floor)
2. Confidence-lower of the leader > Confidence-upper of the runner-up (non-overlapping intervals — stricter than "rate is higher")

This protects against premature decisions. If the rate for A is, for example, dramatically higher than B at N=50, it could be pure noise — Wilson recognizes this and does not output a winner.

When to End?

The tenant decides manually — no auto-stop. Once the winner card is green, you click End. The status changes to completed, the picker stops running, and new visitors receive the standard widget again.

If you want to have the winner variant as the default after ending, copy the config values (greeting, etc.) into the original widget — this is still a manual step in v1; a one-click promote is on the roadmap.

GDPR

The widget writes a localStorage entry in the visitor's browser:

• Key: sc_variant_<public_key>
• Value: { "group_id": "<uuid>", "label": "A" } (no personal data — only group ID and letter)
• Storage duration: until the visitor clears the browser storage or the experiment ends

What you should include in your privacy policy:

We use a local storage entry (sc_variant_<id>) to keep visitors consistently in the same test variant. The entry contains exclusively a test ID and a variant name (e.g., "A"). Purpose: correct functioning of our A/B test; legal basis: legitimate interest in optimizing our website (Art. 6 (1) lit. f GDPR).

The hash picker itself uses the fingerprint — this is already covered by your existing visitor tracking section.

Frequently Asked Questions

More than two variants? In v1 only classic A/B. Multi-variant (A/B/C/D) is planned as follow-up plan #167b — waiting for tenant demand.

What happens to ongoing conversations when I end? They retain their variant_assignment stamp and are counted in the final statistics — even if the visitor returns days later.

Picker is deterministic — why then localStorage? Layered defense: the hash remains stable as long as the fingerprint is stable. localStorage steps in if the browser changes the fingerprint (update, privacy extension) — otherwise, a visitor would flip to the "other" variant in this case.

Do I need consent for localStorage? This depends on your tenant's cookie banner strategy. Since the value does not contain personal data and serves exclusively the test functionality, legitimate interest (Art. 6 (1) lit. f GDPR) is usually sufficient. Consult your data protection officer if in doubt.

API Reference

The picker logic runs on every POST /api/v1/widget/init — the visitor sees 0 additional latency because the pick is an indexed DB lookup + hash.