Some devices experience connection instability

Incident Report for signageOS

Postmortem

Date

August 28, 2025

Authors

signageOS Engineering Team

Summary

On August 28, the signageOS platform experienced an incident that temporarily affected device connectivity reporting and real-time system responsiveness. While no deployed devices or their content playback were impacted, some customers observed devices incorrectly marked as offline and received false alerts.

The incident was linked to a sudden surge of device reconnections that temporarily stressed parts of the platform’s messaging and monitoring services. This resulted in partial delays and intermittent unavailability in device status reporting.

The core platform services remained resilient, and playback of customer content was not disrupted.

Impact

  • No interruption to deployed devices or content playback.

  • Some devices were displayed as offline in the dashboard and API.

  • A number of customers received false downtime alerts.

  • Device status reporting and system logs were processed with temporary delays.

  • No loss of persistent data occurred.

Detection

The issue was identified through a combination of automated monitoring and engineering team observation.

Contributing Factors

  • An unusual spike of device reconnections placed unexpected load on platform services.

  • Certain background processes consumed more resources than anticipated under peak conditions.

Mitigation & Resolution

The engineering team applied a series of mitigation steps, including scaling adjustments and rebalancing of platform services. Non-critical processes were deprioritized to stabilize essential functions.

System performance was gradually restored, and normal operations resumed the same day.

Posted Aug 29, 2025 - 20:49 CEST

Resolved

This incident has been resolved.
Posted Aug 28, 2025 - 22:46 CEST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Aug 28, 2025 - 17:44 CEST

Update

We are continuing to work on this issue. Device functionality and content/applet playback remain unaffected. All devices continue to run without issue.
Posted Aug 28, 2025 - 15:21 CEST

Identified

Some devices experience connection instability. We identified the issue and are working on a fix.
Posted Aug 28, 2025 - 13:53 CEST
This incident affected: Platform and Screenshots.