A/B Testing

What is A/B Testing?

A/B testing shows different versions of your app to different users, then measures which performs better. Version A might have a blue button, version B a green button - data shows which gets more clicks. Why A/B test?

Remove guesswork from design decisions
Improve conversion rates with data
Validate changes before rolling out to everyone

Simple A/B Test

Start with a basic two-variant test:

import { useConfig, useTrack } from '@grainql/analytics-web/react';

function Hero() {
  const { value: variant } = useConfig('hero_variant');
  const track = useTrack();
  
  const variantA = {
    title: 'Welcome to Our App',
    subtitle: 'The best tool for productivity',
    buttonText: 'Get Started'
  };
  
  const variantB = {
    title: 'Boost Your Productivity',
    subtitle: 'Join thousands of happy users',
    buttonText: 'Start Free Trial'
  };
  
  const content = variant === 'B' ? variantB : variantA;
  
  const handleClick = () => {
    track('cta_clicked', { variant: variant || 'A' });
  };
  
  return (
    <div>
      <h1>{content.title}</h1>
      <p>{content.subtitle}</p>
      <button onClick={handleClick}>{content.buttonText}</button>
    </div>
  );
}

Set up in Grain Dashboard:

Create config key: hero_variant
Set values: A for 50% of users, B for the other 50%
Track which variant converts better

Multi-Variant Testing

Test more than two options:

function PricingCard() {
  const { value: pricingVariant } = useConfig('pricing_variant');
  
  const variants = {
    control: { price: 29, label: 'Pro Plan' },
    test_1: { price: 24, label: 'Pro Plan - Special' },
    test_2: { price: 29, label: 'Professional' },
    test_3: { price: 27, label: 'Pro Plan - Limited' }
  };
  
  const { price, label } = variants[pricingVariant || 'control'];
  
  return (
    <div>
      <h3>{label}</h3>
      <p>${price}/month</p>
    </div>
  );
}

Test multiple variants to find the optimal price and messaging.

Component-Based Testing

Swap entire components:

import { useConfig } from '@grainql/analytics-web/react';

function LandingPage() {
  const { value: heroVersion } = useConfig('hero_version');
  
  const HeroComponent = {
    'minimal': HeroMinimal,
    'detailed': HeroDetailed,
    'video': HeroWithVideo
  }[heroVersion || 'minimal'];
  
  return (
    <>
      <HeroComponent />
      <Features />
      <Pricing />
    </>
  );
}

Test completely different design approaches.

Tracking Test Results

Track when users see and interact with variants:

function ABTest() {
  const { value: variant } = useConfig('test_variant');
  const track = useTrack();
  
  // Track impression
  useEffect(() => {
    track('test_viewed', {
      test_name: 'hero_cta',
      variant: variant || 'A'
    });
  }, [variant, track]);
  
  // Track conversion
  const handleConversion = () => {
    track('test_converted', {
      test_name: 'hero_cta',
      variant: variant || 'A'
    });
    
    // Conversion action (signup, purchase, etc.)
  };
  
  return <button onClick={handleConversion}>Convert</button>;
}

Now you can compare conversion rates: test_converted / test_viewed for each variant.

Personalized Testing

Combine A/B testing with user segmentation:

function PersonalizedHero() {
  const { value: variant } = useConfig('hero_variant', {
    properties: {
      plan: userPlan,
      signup_date: userSignupDate
    }
  });
  
  // New users see variant B, existing users see variant A
  return <HeroComponent variant={variant} />;
}

Set rules in the dashboard to show different variants based on user properties.

Button Style Test

Test button designs:

function CTAButton() {
  const { value: buttonStyle } = useConfig('button_style');
  const track = useTrack();
  
  const styles = {
    default: {
      backgroundColor: '#007bff',
      borderRadius: '4px',
      padding: '12px 24px'
    },
    bold: {
      backgroundColor: '#ff4500',
      borderRadius: '8px',
      padding: '16px 32px',
      fontWeight: 'bold'
    },
    minimal: {
      backgroundColor: 'transparent',
      border: '2px solid #007bff',
      borderRadius: '4px',
      padding: '12px 24px'
    }
  };
  
  const style = styles[buttonStyle || 'default'];
  
  const handleClick = () => {
    track('cta_button_clicked', { button_style: buttonStyle || 'default' });
  };
  
  return (
    <button style={style} onClick={handleClick}>
      Sign Up
    </button>
  );
}

Feature Launch Testing

Test new features with a subset of users:

function Dashboard() {
  const { value: newDashboard } = useConfig('new_dashboard_enabled');
  
  if (newDashboard === 'true') {
    return <NewDashboard />;
  }
  
  return <LegacyDashboard />;
}

Start with 10% of users, gradually increase if metrics improve.

Best Practices

1. Test One Thing: Change one element at a time to know what caused the difference.

// ✅ Good: Testing one element
const buttonText = variant === 'B' ? 'Start Free' : 'Get Started';

// ❌ Bad: Too many changes
const content = variant === 'B' 
  ? { text: 'Start Free', color: 'red', size: 'large' }
  : { text: 'Get Started', color: 'blue', size: 'medium' };

2. Track Both Impressions and Conversions: Know how many saw each variant and how many converted. 3. Run Tests Long Enough: At least 1-2 weeks to account for day-of-week variations. 4. Significant Sample Size: Need enough users to draw conclusions (typically 1000+ per variant). 5. Document Tests: Track what you’re testing and why in your dashboard.

Analyzing Results

After running your test, compare metrics in the Grain Dashboard:

Impressions: How many saw each variant
Conversions: How many completed the goal action
Conversion Rate: Conversions / Impressions
Statistical Significance: Is the difference real or luck?

Gradual Rollout

Once you have a winner, roll it out gradually:

// Dashboard config progression:
// Week 1: 50% A, 50% B (testing)
// Week 2: 25% A, 75% B (B is winning)
// Week 3: 10% A, 90% B (confirming)
// Week 4: 0% A, 100% B (full rollout)

This catches any issues before full deployment.

Next Steps

Remote Config

Learn about configuration management

Feature Flags

Control features with flags

Analytics Dashboard

Build analytics dashboards

Getting Started

Integrations

Tracking & Events

Dashboard

MCP Server

Guides

React SDK

Advanced

What is A/B Testing?

Simple A/B Test

Multi-Variant Testing

Component-Based Testing

Tracking Test Results

Personalized Testing

Button Style Test

Feature Launch Testing

Best Practices

Analyzing Results

Gradual Rollout

Next Steps

Remote Config

Feature Flags

Analytics Dashboard

​What is A/B Testing?

​Simple A/B Test

​Multi-Variant Testing

​Component-Based Testing

​Tracking Test Results

​Personalized Testing

​Button Style Test

​Feature Launch Testing

​Best Practices

​Analyzing Results

​Gradual Rollout

​Next Steps

Remote Config

Feature Flags

Analytics Dashboard

What is A/B Testing?

Simple A/B Test

Multi-Variant Testing

Component-Based Testing

Tracking Test Results

Personalized Testing

Button Style Test

Feature Launch Testing

Best Practices

Analyzing Results

Gradual Rollout

Next Steps