Don’t Believe the “Batteries Included” Hype
Last year I took over a data center migration project spanning 5000+ nodes. The previous team’s Redfish scripts? They could brick half a rack in one run. I’m not exaggerating—a single batch firmware update, lacking proper idempotency checks, turned the entire cluster’s BMCs into paperweights. The ops director’s face was darker than a power outage in the server room.
Redfish API looks simple on the surface—just RESTful + JSON, right? But throw it into production, and the rabbit hole goes deeper than you’d think. This post is the raw, unfiltered collection of every pitfall my team and I have hit over the past two years.
1. Connection Management: Don’t Let Sessions Become Time Bombs
1.1 Session Lifecycle Management
This is the most basic—and most frequently screwed up—part. Redfish’s session mechanism relies on X-Auth-Token, but I’ve seen countless beginners hardcode Basic Auth into every single request.
import requests
import time
# Wrong way: re-authenticate every request
def bad_way(bmc_ip, user, pwd):
session = requests.Session()
session.auth = (user, pwd)
# Every request carries plaintext credentials—exposed in logs
resp = session.get(f'https://{bmc_ip}/redfish/v1/Systems/1')
return resp.json()
# Right way: create a session and reuse it
def good_way(bmc_ip, user, pwd):
session = requests.Session()
session.verify = False # Use proper certs in production
# Create session
login_payload = {
"UserName": user,
"Password": pwd
}
login_resp = session.post(
f'https://{bmc_ip}/redfish/v1/SessionService/Sessions',
json=login_payload
)
if login_resp.status_code == 201:
token = login_resp.headers['X-Auth-Token']
session.headers.update({'X-Auth-Token': token})
# Set timeouts and retries
session.timeout = (5, 30) # connect timeout, read timeout
return session
else:
raise Exception(f"Login failed: {login_resp.status_code}")
Key lesson: Sessions have TTLs, usually 30 minutes by default. We’ve hit cases where sessions expired mid-batch operation, causing partial failures. Solution? Catch 401 status codes and automatically reconnect.
1.2 Connection Pooling
We started with single-threaded queries—40 minutes to poll 5000 nodes. Then we switched to ThreadPoolExecutor and promptly DDoSed our own BMCs. Many low-end BMCs max out at 8 concurrent connections.
from concurrent.futures import ThreadPoolExecutor, as_completed
import threading
# Semaphore to control concurrency
semaphore = threading.Semaphore(6) # Leave some headroom
def safe_query(bmc_ip, session):
with semaphore:
try:
resp = session.get(
f'https://{bmc_ip}/redfish/v1/Systems/1',
timeout=10
)
return resp.json()
except Exception as e:
log.error(f"{bmc_ip} query failed: {e}")
return None
# Process in batches
batch_size = 100
for i in range(0, len(bmc_list), batch_size):
batch = bmc_list[i:i+batch_size]
with ThreadPoolExecutor(max_workers=8) as executor:
futures = {executor.submit(safe_query, ip, session): ip
for ip in batch}
for future in as_completed(futures):
result = future.result()
# Process result
2. Idempotency and State Machines: Don’t Repeat Operations
There’s a Reddit thread that really resonated with me—someone asked “how to use Redfish without knowing the server model.” This exposes a core problem: too many people treat Redfish like a simple REST API, ignoring the state machine model underneath.
2.1 Check State Before Operations
def safe_reboot(bmc_ip, session):
# Get current state
system_resp = session.get(
f'https://{bmc_ip}/redfish/v1/Systems/1'
)
system = system_resp.json()
current_state = system.get('PowerState')
if current_state == 'On':
# Check for ongoing operations
if system.get('Status', {}).get('State') == 'Starting':
log.warning(f"{bmc_ip}: System is starting, skip reboot")
return False
# Execute reboot
resp = session.post(
f'https://{bmc_ip}/redfish/v1/Systems/1/Actions/ComputerSystem.Reset',
json={"ResetType": "ForceRestart"}
)
if resp.status_code == 204:
# Poll for completion
return wait_for_completion(bmc_ip, session, target_state='On')
else:
log.info(f"{bmc_ip}: Current state {current_state}, no reboot needed")
return False
2.2 Task Polling Done Right
Redfish async operations return Task resources. But vendor implementations vary wildly—Dell’s iDRAC can have a 30-second delay in Task status updates.
import time
import backoff
@backoff.on_exception(
backoff.expo,
(requests.exceptions.Timeout, requests.exceptions.ConnectionError),
max_tries=5
)
def wait_for_task(bmc_ip, session, task_url, timeout=300):
start = time.time()
while time.time() - start < timeout:
resp = session.get(f'https://{bmc_ip}{task_url}')
task = resp.json()
state = task.get('TaskState')
percent = task.get('PercentComplete', 0)
if state == 'Completed':
log.info(f"{bmc_ip}: Task completed in {time.time()-start:.1f}s")
return True
elif state == 'Failed':
error = task.get('Messages', [{}])[0].get('Message', 'Unknown')
raise Exception(f"Task failed: {error}")
elif state == 'Cancelled':
raise Exception("Task was cancelled")
# Dynamic polling interval
if percent > 50:
time.sleep(2)
else:
time.sleep(5)
raise TimeoutError(f"Task did not complete within {timeout}s")
3. Error Handling: Don’t Trust What the BMC Tells You
This is the part that’s driven me up the wall. Different vendors’ Redfish implementations vary so much it’s painful. Here’s our battle-tested summary:
| Vendor | Redfish Version | Consistency | Known Issues | Our Strategy |
|---|---|---|---|---|
| Dell iDRAC 9 | 1.6-1.12 | Good | Task status update delay | Add 5s initial wait |
| HPE iLO 5 | 1.0-1.10 | Moderate | Some properties return empty strings | Add null/empty checks |
| Supermicro X11 | 1.0-1.5 | Poor | No Redfish Event subscription support | Fall back to polling |
| Inspur | 1.6-1.8 | Moderate | Some ops return 500 but succeed | Verify actual operation outcome |
| Lenovo XClarity | 1.8-1.12 | Good | BMC reboot after firmware update takes forever | Set 600s timeout |
3.1 Defensive Parsing
def safe_get_property(obj, path, default=None):
"""Safely access nested properties"""
keys = path.split('.')
current = obj
for key in keys:
if isinstance(current, dict):
current = current.get(key)
if current is None:
return default
else:
return default
return current
# Usage
system_info = resp.json()
# Don't directly access potentially missing properties
serial = safe_get_property(system_info, 'SerialNumber', 'UNKNOWN')
model = safe_get_property(system_info, 'Model', 'UNKNOWN')
# Some vendors put MemorySummary in different places
memory_gb = safe_get_property(system_info, 'MemorySummary.TotalSystemMemoryGiB', 0)
4. The Art of Batch Operations
4.1 Batching and Retry Strategies
We once tried to push BIOS config changes to 2000 nodes simultaneously. It took down the entire datacenter’s DHCP service—all BMCs rebooted at once, creating a DHCP request storm.
class BatchRedfishManager:
def __init__(self, bmc_list, max_concurrent=6, batch_size=50):
self.bmc_list = bmc_list
self.max_concurrent = max_concurrent
self.batch_size = batch_size
self.results = {}
def execute_with_retry(self, operation_func, max_retries=3):
for batch_num, batch in enumerate(self._batches()):
log.info(f"Processing batch {batch_num+1}, servers {batch[0]} to {batch[-1]}")
with ThreadPoolExecutor(max_workers=self.max_concurrent) as executor:
futures = {}
for bmc in batch:
future = executor.submit(
self._retry_wrapper,
operation_func,
bmc,
max_retries
)
futures[future] = bmc
for future in as_completed(futures):
bmc = futures[future]
try:
result = future.result()
self.results[bmc] = {'status': 'success', 'data': result}
except Exception as e:
self.results[bmc] = {'status': 'failed', 'error': str(e)}
# Wait between batches to prevent BMC overload
if batch_num < len(self.bmc_list) // self.batch_size:
time.sleep(10)
return self.results
def _retry_wrapper(self, func, bmc, max_retries):
for attempt in range(max_retries):
try:
return func(bmc)
except (requests.exceptions.ConnectionError,
requests.exceptions.Timeout) as e:
if attempt == max_retries - 1:
raise
wait = 2 ** attempt # Exponential backoff
log.warning(f"{bmc}: Attempt {attempt+1} failed, retrying in {wait}s")
time.sleep(wait)
5. Security Best Practices
5.1 Credential Management
There’s a Reddit thread asking about “Secret Storage Best Practices.” In the Redfish context, this is critical. BMC passwords are the crown jewels—leak them, and you’ve handed over physical server control.
# Don't do this
BMC_PASSWORD = "admin123" # Hardcoded in script
# Use environment variables (still not ideal)
import os
password = os.environ.get('BMC_PASSWORD')
# Recommended: Use HashiCorp Vault or Infisical
import hvac
def get_bmc_credentials(bmc_ip):
client = hvac.Client(url='https://vault.example.com:8200')
client.token = os.environ['VAULT_TOKEN']
secret = client.secrets.kv.v2.read_secret_version(
path=f'bmc/{bmc_ip}',
mount_point='infra'
)
return {
'username': secret['data']['data']['username'],
'password': secret['data']['data']['password']
}
5.2 Certificate Verification
I know 99% of people still use verify=False in production. But at minimum:
# Minimum security config
session.verify = '/path/to/ca-bundle.crt' # Use custom CA bundle
# Or use certificate fingerprint verification (good for no-CA scenarios)
import hashlib
CERT_FINGERPRINTS = {
'bmc-001.example.com': 'sha256$ABC123...',
'bmc-002.example.com': 'sha256$DEF456...',
}
def verify_bmc(bmc_ip, cert_der):
fingerprint = hashlib.sha256(cert_der).hexdigest()
expected = CERT_FINGERPRINTS.get(bmc_ip)
if expected and fingerprint != expected:
raise Exception(f"Certificate mismatch for {bmc_ip}")
return True
6. Monitoring and Alerting
6.1 Key Metrics
Here’s what we ended up monitoring:
| Metric | Redfish Path | Alert Threshold | Notes |
|---|---|---|---|
| CPU Temp | /Chassis/{id}/Thermal#/Temperatures | >85°C | Emergency throttling threshold |
| Fan Speed | /Chassis/{id}/Thermal#/Fans | <30% RPM | Insufficient cooling |
| Power Supply | /Chassis/{id}/Power#/PowerSupplies | Redundancy lost | Alert immediately |
| Disk Health | /Systems/{id}/Storage#/Drives | Predicted Failure | Handle within 24h |
| Memory Errors | /Systems/{id}/Memory#/MemoryMetrics | CorrectedECC > 100/hr | Memory about to fail |
7. FAQ
Q: How do you handle Redfish API version compatibility?
A: First, GET /redfish/v1 and check the RedfishVersion field. Different versions support different properties. Use @odata.context for dynamic resource discovery. We maintain a version mapping table to decide which properties and endpoints to use based on the version number.
Q: How to prevent BMC overload during batch operations? A: Use a semaphore to cap concurrency (recommend ≤8), add 10-second delays between batches. If you hit ConnectionErrors, immediately reduce concurrency and increase delays. We have an automatic backoff mechanism—3 consecutive errors halves the concurrency.
Q: How do you handle different vendor Redfish implementations?
A: Abstract an adapter layer. Each vendor gets a concrete adapter that handles vendor-specific behaviors (Dell’s firmware upgrade flow is completely different from HPE’s). Use a factory pattern to select the right adapter based on the Manufacturer field.
Q: Is Redfish Event subscription reliable? A: No. In our environment, SuperMicro and Inspur BMCs frequently drop events. We use a hybrid approach: Events as a speed-up mechanism, with regular polling as a safety net.
8. Summary
Redfish API looks simple, but production deployment is a minefield. Core lessons:
- Never trust BMC responses—always do defensive checks
- Idempotency is rule #1—check state before operations, verify results after
- Concurrency control is non-negotiable—BMCs are more fragile than you think
- Vendor differences are huge—an abstraction adapter layer is mandatory
- Credential management must be strict—BMC password leak = server compromise
References & Community Insights
The following authoritative resources were referenced for architectural best practices and specifications:
