Smart Failure Handling in HCL Commerce with Circuit Breakers

In modern enterprise systems, stability and fault tolerance are not optional; they are essential. One proven approach to ensure robustness is the Circuit Breaker pattern, widely used in API development to prevent cascading failures. HCL Commerce takes this principle further by embedding circuit breakers into its HCL Cache to effectively manage Redis failures.

 What Is a Circuit Breaker?
The Circuit Breaker is a design pattern commonly used in API development to stop continuous requests to a service that is currently failing, thereby protecting the system from further issues. It helps maintain system stability by detecting failures and stopping the flow of requests until the issue is resolved.

The circuit breaker typically operates in three main (or “normal”) states. These are part of the standard global pattern of Circuit Breaker design.

Normal States:

1. CLOSED:

• At the start, the circuit breaker allows all outbound requests to external services without restrictions.
• It monitors the success and failure of these calls.

2. OPEN:

• The circuit breaker rejects all external calls.
• This state is triggered when the failure threshold is reached (e.g., 50% failure rate).
• It remains in this state for a specified duration (e.g., 60 seconds).

3. HALF_OPEN:

• After the wait duration in the OPEN state, the circuit breaker transitions to HALF_OPEN.
• It allows a limited number of calls to check if the external service has recovered.
• If these calls succeed (e.g., receive a 200 status), the circuit breaker transitions back to  CLOSED.
• If the error rate continues to be high, the circuit breaker reverts to the OPEN state.

Circuit breaker pattern with normal states

Special States:

1. FORCED_OPEN:

• The circuit breaker is manually set to reject all external calls.
• No calls are allowed, regardless of the external service’s status.

2. DISABLED:

• The circuit breaker is manually set to allow all external calls.
• It does not monitor or track the success or failure of these calls.

Circuit breaker pattern with special states

Circuit Breaker in HCL Cache (for Redis)

In HCL Commerce, the HCL Cache layer interacts with Redis for remote coaching. But what if Redis becomes unavailable or slow? HCL Cache uses circuit breakers to detect issues and temporarily stop calls to Redis, thus protecting the rest of the system from being affected.

Behavior Overview:

• If 20 consecutive failures occur in 10 seconds, the Redis connection is cut off.
• The circuit remains open for 60 seconds.
• At this stage, the circuit enters a HALF_OPEN state, where it sends limited test requests to evaluate if the external service has recovered.
• If even 2 of these test calls fail, the circuit reopens for another 60 seconds.

Configuration Snapshot

To manage Redis outages effectively, HCL Commerce provides fine-grained configuration settings for both Redis client behavior and circuit breaker logic. These settings are defined in the Cache YAML file, allowing teams to tailor fault-handling based on their system’s performance and resilience needs.

 Redis Request Timeout Configuration

Slow Redis responses are not treated as failures unless they exceed the defined timeout threshold. The Redis client in HCL Cache supports timeout and retry configurations to control how persistent the system should be before declaring a failure:

timeout: 3000           # Max time (in ms) to wait for a Redis response
retryAttempts: 3        # Number of retry attempts on failure
retryInterval: 1500    # Specifies the delay (in milliseconds) between each retry attempt.

With the above configuration, the system will spend up to 16.5 seconds (3000 + 3 × (3000 + 1500)) trying to get a response before returning a failure. While these settings offer robustness, overly long retries can result in delayed user responses or log flooding, so tuning is essential.

Circuit Breaker Configuration

Circuit breakers are configured under the redis.circuitBreaker section of the Cache YAML file. Here’s an example configuration:

redis:
  circuitBreaker:
    scope: auto
    retryWaitTimeMs: 60000
    minimumFailureTimeMs: 10000
    minimumConsecutiveFailures: 20
    minimumConsecutiveFailuresResumeOutage: 2 
cacheConfigs:
  defaultCacheConfig:
    localCache:
      enabled: true
      maxTimeToLiveWithRemoteOutage: 300

Explanation of Key Fields:

• scope: auto: Automatically determines whether the circuit breaker operates at the client or cache/shard level, depending on the topology.
• retryWaitTimeMs (Default: 60000): Time to wait before attempting Redis connections after circuit breaker is triggered.
• minimumFailureTimeMs (Default: 10000): Minimum duration during which consecutive failures must occur before opening the circuit.
• minimumConsecutiveFailures (Default: 20): Number of continuous failures required to trigger outage mode.
• minimumConsecutiveFailuresResumeOutage (Default: 2): Number of failures after retrying that will put the system back into outage mode.
• maxTimeToLiveWithRemoteOutage: During Redis outages, local cache entries use this TTL value (in seconds) to serve data without invalidation messages.

Real-world Analogy

Imagine you have a web service that fetches data from an external API. Here’s how the circuit breaker would work:

1. CLOSED: The service makes calls to the API and monitors the responses.
2. OPEN: If the API fails too often (e.g., 50% of the time), the circuit breaker stops making calls for 60 seconds.
3. HALF_OPEN: After 60 seconds, the circuit breaker allows a few calls to the API to see if it’s working again.
4. CLOSED: If the API responds successfully, the circuit breaker resumes normal operation.
5. OPEN: If the API still fails, the circuit breaker stops making calls again and waits.

Final Thought

By combining the classic circuit breaker pattern with HCL Cache’s advanced configuration, HCL Commerce ensures graceful degradation during Redis outages. It’s not just about availability—it’s about intelligent fault recovery.

For more detailed information, you can refer to the official documentation here:
HCL Commerce Circuit Breakers – Official Docs

Source: Read More