A/B tests

An A/B test shows one of two or more variants to each user and compares a conversion rate per variant. In Grain, a remote configuration value picks the variant. Two events measure it: one when the user sees the variant, one when the user converts. A/B tests use Analytics Web (@grainql/analytics-web). Grain Tag has no remote configuration. Read Remote configuration for the cache-first flow.

The pattern

A test has four parts.

Part	Where	What it is
The variant value	Dashboard, Dashboard > Remote Config	One key, for example `hero_variant`, with a rule that returns `A` to 50% of users and `B` to the rest.
The variant code	Your app	Reads the key and shows the matching content.
The exposure event	Your app	One event when the user sees the variant.
The conversion event	Your app	One event when the user completes the goal.

Both events carry the same test_name and variant properties. Without the variant on both events, the result cannot be split per variant.

The variant value

Create the key in the dashboard and add a rule that splits users into variants. Set the control variant as the default value in defaultConfigurations, so that a client with no response yet shows the control.

<GrainProvider
  config={{
    tenantId: 'your-tenant-id',
    defaultConfigurations: { hero_variant: 'A' }
  }}
>
  <App />
</GrainProvider>

The value is a string. A test with more than two variants uses more values, for example control, price_24, and price_27. Read Remote configuration to create the key.

The exposure event

Send the exposure event one time, when the component that shows the variant mounts. In React, send it from an effect that depends on the variant.

import { useEffect } from 'react';
import { useConfig, useTrack } from '@grainql/analytics-web/react';

const content = {
  A: { title: 'Welcome to our app', button: 'Get started' },
  B: { title: 'Do more in less time', button: 'Start a free trial' }
};

function Hero() {
  const { value } = useConfig('hero_variant');
  const variant = value === 'B' ? 'B' : 'A';
  const track = useTrack();

  useEffect(() => {
    track('test_viewed', { test_name: 'hero_cta', variant });
  }, [variant, track]);

  return (
    <div>
      <h1>{content[variant].title}</h1>
      <button onClick={() => track('test_converted', { test_name: 'hero_cta', variant })}>
        {content[variant].button}
      </button>
    </div>
  );
}

The code maps every unknown value to A. Then a typo in the dashboard shows the control instead of an empty page.

The conversion event

The conversion event is the goal of the test: a click, a signup, a purchase. Send it with the same test_name and variant as the exposure event. When the goal is on another page, read the variant again on that page with useConfig. The cache returns the same value for the same user.

function SignupForm() {
  const { value } = useConfig('hero_variant');
  const track = useTrack();

  const onSubmit = async () => {
    await createAccount();
    track('test_converted', { test_name: 'hero_cta', variant: value === 'B' ? 'B' : 'A' });
  };

  return <form onSubmit={onSubmit}>...</form>;
}

When the goal is a template event such as a signup or a purchase, add the variant property to that event instead of a second event. Read Template events.

Read the results

The conversion rate of a variant is the number of users with test_converted divided by the number of users with test_viewed. Both counts are filtered on the same variant. In the dashboard, build a track from test_viewed to test_converted and split it by the variant property. Read Tracks. To compute the rate outside the dashboard, count both events per variant with the Query API. Read Export and query data. Rules for a test:

Change one element per test. Then the difference has one cause.
Run the test for at least one full week, so that every weekday is in the sample.
Do not read the result before each variant has about 1000 users.
When a variant wins, move its share to 100% in the dashboard, then replace the code with the winning variant and delete the key.

Start here

Install without code

Install with code

Track events

Dashboard

Remote configuration

Guides

React

Advanced

The pattern

The variant value

The exposure event

The conversion event

Read the results

​The pattern

​The variant value

​The exposure event

​The conversion event

​Read the results

The pattern

The variant value

The exposure event

The conversion event

Read the results