Live Status\Incident History

December 22, 2024

Site Unreachable

Outage

Opened: Dec 22, 2024, 6:33 AM UTC

Duration: 8m 38s

Affected services:Mastodon Instance Frontend


  • Opened
    Dec 22, 2024, 6:33 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Dec 22, 2024, 6:42 AM UTC

    Issue resolved on monitor recovery.

Site Unreachable

Outage

Opened: Dec 22, 2024, 4:16 AM UTC

Duration: 14m 20s

Affected services:Mastodon Instance Frontend


  • Opened
    Dec 22, 2024, 4:16 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Dec 22, 2024, 4:30 AM UTC

    Issue resolved on monitor recovery.

Site Unreachable

Outage

Opened: Dec 22, 2024, 2:01 AM UTC

Duration: 9m 46s

Affected services:Static Content Server and Cache


  • Opened
    Dec 22, 2024, 2:01 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Dec 22, 2024, 2:11 AM UTC

    Issue resolved on monitor recovery.

September 5, 2024

Site Unreachable

Outage

Opened: Sep 5, 2024, 1:41 AM UTC

Duration: 9m 44s

Affected services:Mastodon Instance Frontend


  • Opened
    Sep 5, 2024, 1:41 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Sep 5, 2024, 1:50 AM UTC

    Issue resolved on monitor recovery.

August 17, 2024

Emergency Database Maintenance

Degraded Performance

Opened: Aug 17, 2024, 6:18 PM UTC

Duration: 4m 21s

Affected services:Database System


  • Monitoring
    Aug 17, 2024, 6:18 PM UTC

    Database maintenance is being performed on the instance to restore write performance due to conditions preventing routine, scheduled maintenance operations from running successfully. Performance of the site may be severely degraded as a result.

  • Resolved
    Aug 17, 2024, 6:22 PM UTC

    Maintenance operation has completed, write performance restored.

May 3, 2024

Site Unreachable

Outage

Opened: May 3, 2024, 9:28 AM UTC

Duration: 5m 10s

Affected services:Static Content Server and Cache


  • Opened
    May 3, 2024, 9:28 AM UTC

    Issue opened on monitor failure.

  • Resolved
    May 3, 2024, 9:33 AM UTC

    Issue resolved on monitor recovery.

April 12, 2024

Healthcheck Assertion Failed

Outage

Opened: Apr 12, 2024, 11:05 PM UTC

Duration: 4m 34s

Affected services:Static Content Server and Cache


  • Opened
    Apr 12, 2024, 11:05 PM UTC

    Issue opened on monitor failure.

  • Identified
    Apr 12, 2024, 11:07 PM UTC

    False alarm due to monitoring configuration

  • Resolved
    Apr 12, 2024, 11:10 PM UTC

    Monitoring hook has been updated, no actual downtime on static content server.

March 14, 2024

Site Unreachable

Outage

Opened: Mar 14, 2024, 6:12 PM UTC

Duration: 9m 21s

Affected services:Static Content Server and Cache


  • Opened
    Mar 14, 2024, 6:12 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Mar 14, 2024, 6:21 PM UTC

    Issue resolved on monitor recovery.

October 7, 2023

Site Unreachable

Outage

Opened: Oct 7, 2023, 6:44 PM UTC

Duration: 9m 36s

Affected services:Mastodon Instance Frontend


  • Opened
    Oct 7, 2023, 6:44 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Oct 7, 2023, 6:53 PM UTC

    Issue resolved on monitor recovery.

Site Unreachable

Outage

Opened: Oct 7, 2023, 6:41 PM UTC

Duration: 14m 29s

Affected services:Static Content Server and Cache


  • Opened
    Oct 7, 2023, 6:41 PM UTC

    Issue opened on monitor failure.

  • Identified
    Oct 7, 2023, 6:48 PM UTC

    Storage migration in progress, performing validation checks before toggling over

  • Resolved
    Oct 7, 2023, 6:56 PM UTC

    Issue resolved on monitor recovery.

October 5, 2023

Site Unreachable

Outage

Opened: Oct 5, 2023, 7:42 AM UTC

Duration: 9m 28s

Affected services:Static Content Server and Cache


  • Opened
    Oct 5, 2023, 7:42 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Oct 5, 2023, 7:51 AM UTC

    Issue resolved on monitor recovery.

September 24, 2023

Database Upgrade in Progress

Maintenance

Posted: Sep 24, 2023, 12:44 PM UTC

Affected services:Database System


  • Opened
    Sep 24, 2023, 12:44 PM UTC

    The system will be unavailable while the database is being upgraded

  • Identified
    Sep 24, 2023, 12:45 PM UTC

    Cluster upgrade in progress

  • Monitoring
    Sep 24, 2023, 12:51 PM UTC

    Upgrade complete, validating state

  • Resolved
    Sep 24, 2023, 1:30 PM UTC

    Upgrade performs well, maintenance over

July 2, 2023

Heavy queue load and increased processing times

Degraded Performance

Opened: Jul 2, 2023, 6:11 PM UTC

Duration: 2m 12s

Affected services:Database SystemActivityPub Workers


  • Monitoring
    Jul 2, 2023, 6:09 PM UTC

    We somehow touch a large chunk of the twitter migration, some tuning has been done to accomodate. We are currently monitoring the systems for bottlenecks.

  • Resolved
    Jul 2, 2023, 6:13 PM UTC

    Database has been cleaned up, a problematic fakerelay endpoint was putting undue pressure on the system. Situation normal.

June 1, 2023

Site Unreachable

Outage

Opened: Jun 1, 2023, 11:14 PM UTC

Duration: 14m 18s

Affected services:Mastodon Instance Frontend


  • Opened
    Jun 1, 2023, 11:14 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jun 1, 2023, 11:28 PM UTC

    Issue resolved on monitor recovery.

Site Unreachable

Outage

Opened: Jun 1, 2023, 11:14 PM UTC

Duration: 14m 48s

Affected services:Static Content Server and Cache


  • Opened
    Jun 1, 2023, 11:14 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jun 1, 2023, 11:29 PM UTC

    Issue resolved on monitor recovery.