Configuration Options

Case limits, seeding, complexity limits, and shrinkage control

Table of contents

1. Beyond Simple Limits
2. Case Limit Strategies
   1. Time-Based Limits
   2. Count-Based Limits with Starvation Tolerance
3. Understanding Cycles
4. Seeding for Reproducibility
   1. Default Behavior: Repeatable
   2. Explicit Seeds
   3. Non-Deterministic Mode
5. Complexity Limits
   1. Subtlety: Fixed-Size Collections
6. Shrinkage Control
   1. Limiting Shrinkage Attempts
   2. Advanced: Shrinkage Stop Callback
7. Configuration Chaining
8. Practical Recommendations
   1. For Local Development
   2. For CI/CD
   3. For Performance Testing
   4. For Debugging

---

Beyond Simple Limits

So far we’ve used .withLimit() to control how many test cases to generate. But Americium offers much more sophisticated configuration for different testing scenarios.

Let’s explore the full range of options.

---

Case Limit Strategies

Time-Based Limits

Instead of a fixed number of cases, you can specify a time budget:

import java.time.Duration;

trials
    .withStrategy(cycle -> 
        CasesLimitStrategy.timed(Duration.ofSeconds(5)))
    .supplyTo(testCase -> {
        // Test runs for approximately 5 seconds
    });

This is useful for:

• Performance testing - Run as many cases as possible in a fixed time
• CI/CD - Limit test duration regardless of complexity
• Exploratory testing - Spend fixed time exploring the input space

The cycle parameter represents the current shrinkage cycle - we’ll explain this shortly.

---

Count-Based Limits with Starvation Tolerance

First, what do we mean by ‘starvation’? When Americium supplies a test case to a trial, this doesn’t always result in the trial being run.

Reasons include:

• A call to filter blocking the synthesis of a test case prior to it being supplied.
• Duplicates of test cases already supplied are suppressed.
• The test case is too complex. (We’ll cover this later in the Complexity Budgeting section under Advanced Techniques).
• Rejection of a trial by the test itself.

This blocking of supply is what we mean by starvation; we need to heed this because we don’t want Americium to keep fruitlessly retrying the supply in an infinite loop.

Fortunately, the withLimit() method is actually shorthand for a built-in strategy that handles starvation automatically. If you write a trials instance that keeps starving its test, this built-in strategy will eventually decide to terminate the trials.

We can exert finer control though…

// When starvation doesn't happen, these are equivalent.
trials.withLimit(100)

trials.withStrategy(cycle -> 
    CasesLimitStrategy.counted(100, 0.0))

The second parameter of withStrategy is the starvation ratio. It controls how much filtering/rejection is acceptable:

trials.withStrategy(cycle -> 
    CasesLimitStrategy.counted(
        100,    // Maximum cases
        0.2))   // Allow 20% rejection rate

If more than 20% of attempted test cases cause starvation, the trials are terminated.

High rejection rates suggest your trial specification is inefficient. Consider redesigning to generate valid cases directly rather than filtering.

---

Understanding Cycles

When you see cycle -> CasesLimitStrategy..., what’s that about?

Cycles represent phases of the testing process:

• Initial exploration: finding a failing case
• Subsequent shrinkage attempts: finding simpler failing cases

You can configure different limits for each cycle:

trials.withStrategy(cycle -> {
    if (cycle.isInitial) {
        // Exploration: try many cases, accept a large degree of starvation.
        return CasesLimitStrategy.counted(1000, 1);
    } else {
        // Shrinkage: don't try too hard, further shrinkage is just a 'nice to have'.
        return CasesLimitStrategy.counted(500, 0.2);
    }
});

Or use time budgets that vary:

trials.withStrategy(cycle -> {
    // Exploration: want a quick test and don't want to look at everything.
    // Shrinkage: if we do see a failed test case, work hard to shrink it down.
    final Duration timeBudget = Duration.ofSeconds(cycle.isInitial ? 2 : 10)

    return CasesLimitStrategy.timed(timeBudget);
});

---

Seeding for Reproducibility

Default Behavior: Repeatable

By default, Americium uses a fixed internal seed, so runs are deterministic:

trials.withLimit(10).supplyTo(System.out::println);
// Always produces the same sequence!

This is great for local development - same failures reproduce reliably.

---

Explicit Seeds

You can control the seed explicitly:

trials
    .withLimit(100)
    .withSeed(42L)  // Use specific seed
    .supplyTo(testCase -> {
        // Deterministic with seed 42
    });

This is useful for:

• Reproducing specific runs - Share the seed value
• Parameterized testing - Run same test with different seeds

---

Non-Deterministic Mode

For CI/CD, you might want different cases on each run to explore the input space more thoroughly:

-Dtrials.nondeterministic=true

Now each test run uses a random seed, generating different test cases:

final Trials<Integer> trials = api().integers(1, 40);
      
trials.withLimit(5).supplyTo(System.out::println);

// Run 1: test cases 2, 14, 12, 4, 5
// Run 2: test cases 8, 18, 26, 21, 30
// Run 3: test cases 23, 29, 28, 7, 2

Over many CI runs, you’ll cover much more of the input space!

Best practice: Use deterministic mode locally, non-deterministic in CI.

---

Complexity Limits

Remember complexity shrinkage? You can limit how complex your test cases get:

trials
    .withLimit(100)
    .withComplexityLimit(50)
    .supplyTo(testCase -> {
        // Test cases limited to 50 degrees of freedom
    });

This prevents Americium from generating:

• Deeply nested structures
• Very long lists
• Highly complex recursive values

---

Subtlety: Fixed-Size Collections

Here’s an important detail - fixed-size collections are exempt from complexity limits:

api().integers()
    .immutableListsOfSize(100)  // Always 100 elements!
    .withLimit(10)
    .withComplexityLimit(1)    // Doesn't affect list size
    .supplyTo(list -> {
        assert list.size() == 100;  // ✓ Always true
    });

Why? You explicitly requested 100 elements. Americium respects your specification.

But varying-size collections respect the limit:

api().integers()
     .immutableLists() // Varying size
     .withLimit(10)
     .withComplexityLimit(5) // Restricts maximum size
     .supplyTo(System.out::println);

// [-1975667456, -54163312]
// []
// [1610305737]
// [1987104774, 1272878179]
// [1327605531, 633033882]
// [-685852310, -2071825701]
// [-1545019583]
// [-1589526881]
// [637450267]
// [499196904, -440688866]

The complexity limit does not directly count the number of items in the list. Think of it as being a guide heuristic; the smaller the limit, the shorter the lists in this example.

---

Shrinkage Control

Limiting Shrinkage Attempts

You can control how hard Americium tries to shrink:

trials
    .withLimit(100)
    .withShrinkageAttemptsLimit(10)
    .supplyTo(testCase -> {
        // Maximum 10 shrinkage attempts after initial failure
    });

Setting this to 0 disables shrinkage entirely:

trials
    .withShrinkageAttemptsLimit(0)  // No shrinkage!
    .supplyTo(testCase -> {
        // Test fails with initial failing case
    });

This is useful for:

• Debugging shrinkage itself - See the initial failure
• Performance - Skip shrinkage when you just want to know “does it fail?”

---

Advanced: Shrinkage Stop Callback

For ultimate control, provide a callback that decides when to stop:

import com.sageserpent.americium.generation.ShrinkageStop;

trials
    .withLimit(100)
    .withShrinkageStop(new ShrinkageStop() {
        private int trialsCount = 0;
        
        @Override
        public boolean test(int testCase) {
            // Custom logic here: regardless of cycle,
            // give up after doing 150 trials overall.
            return 150 == trialsCount++;
        }
    })
    .supplyTo(testCase -> {
        // Uses custom shrinkage stopping logic
    });

Parameters:

• shrinkageAttempts - Total shrinkage attempts so far
• consecutiveFailures - Failures since last improvement

This allows sophisticated strategies like:

• “Stop if no improvement in last 10 attempts”
• “Stop if we’ve been shrinking for 30 seconds”
• “Stop if the test case is ‘simple enough’”

---

Configuration Chaining

All configuration methods can be chained:

trials
    .withLimit(200)
    .withSeed(12345L)
    .withComplexityLimit(75)
    .withShrinkageAttemptsLimit(30)
    .supplyTo(testCase -> {
        // Fully configured!
    });

Or using the strategy approach:

trials
    .withStrategy(cycle -> 
        cycle == 0 
            ? CasesLimitStrategy.timed(Duration.ofSeconds(10))
            : CasesLimitStrategy.counted(100, 0.15))
    .withSeed(42L)
    .withComplexityLimit(50)
    .withShrinkageStop(myCustomStop)
    .supplyTo(testCase -> {
        // Even more configured!
    });

---

Practical Recommendations

For Local Development

trials
    .withLimit(100)              // Quick feedback
    .withShrinkageAttemptsLimit(20)  // Good shrinkage
    // Default seed (deterministic)

For CI/CD

# Run with:
-Dtrials.nondeterministic=true

trials
    .withStrategy(cycle -> 
        CasesLimitStrategy.timed(Duration.ofSeconds(30)))
    .withComplexityLimit(100)
    .withShrinkageAttemptsLimit(50)

For Performance Testing

trials
    .withStrategy(cycle -> 
        CasesLimitStrategy.timed(Duration.ofMinutes(5)))
    .withShrinkageAttemptsLimit(0)  // Skip shrinkage

For Debugging

trials
    .withLimit(10)
    .withShrinkageAttemptsLimit(0)  // See raw failures
    .withSeed(specificSeed)         // Reproduce specific run

---

Key Takeaways

• Two limit strategies: Time-based and count-based
• Count-based supports starvation ratio for rejection tolerance
• Cycles: Cycle 0 = exploration, Cycles 1+ = shrinkage
• Seeds: Fixed by default (repeatable), random with -Dtrials.nondeterministic=true
• Complexity limits control test case complexity (but exempt fixed-size collections)
• Shrinkage control: Limit attempts or use custom stop callbacks
• Configuration chains - combine multiple settings
• Different configurations for different scenarios (local vs CI vs debugging)