[Status] Hosted Exchange downtime issue
Incident Report for CloudNordic
Resolved
Hosted Exchange virker igen og vidne serveren (witness) har forstået hvilken server er primær i driftsmiljøet, der spænder over flere samtidige datacentre.

Årsagen til fejlen var at netop "witness" serveren forvirrede den primære Exchange dritsnode, så den ikke kunne finde ud af at mounte databaser.

Selvom OWA, Outlook og IMAP var tilgængelig under fejlen, kunne disse klienter ikke få adgang til mail databaser af samme årsag og meldte derfor fejl.

Løsning: Hosted Exchange OWA og Outlook

Start: 18-08-2021 @ 13:30 CET
Slut: 18-08-2021 @ 15:40 CET

Vi beklager på det kraftigste driftsforstyrrelsen og de gener dette måtte have medført.

De allerbedste hilsner,

The Cloudnordic Helpdesk & Support Team

-----

The Hosted Exchange service once again up and running normally.

Root cause of the error was due to an error in the understanding on the Exchange witness server, of which Exchange server was the primary mailserver, enforcing the primary Exchange server not to be able to mount mail databases and thus causing outage in service.

Even though OWA, IMAP and Outlook services were online during the incident, they were unable to attach to mail storage databases, which caused downtime for client access.

Solution: Hosted Exchange OWA and Outlook

Start: 18-08-2021 @ 13:30 CET
End: 18-08-2021 @ 15:40 CET (expected)

We apologize for any inconvenience this incident may have caused.

Our very best regards,

The Cloudnordic Helpdesk & Support Team
Posted Aug 18, 2021 - 16:00 CEST
Update
We are continuing to monitor for any further issues.
Posted Aug 18, 2021 - 15:47 CEST
Update
Hosted Exchange er i øjeblikket nede og vi er igang med at remounte databaser til det primære driftsmiljø, idet det redundante setup er i uoverensstemmelse om hvem der er den primære Exchange løsning.

Vi er derfor ved at tvinge de redundante systemer til at forstå, at de ikke er primære, så det primære system igen kan starte databaser op og gå i drift igen.

Løsning: Hosted Exchange OWA og Outlook

Start: 18-08-2021 @ 13:30 CET
Slut: 18-08-2021 @ 15:30 CET (forventet)

Vi beklager på det kraftigste driftsforstyrrelsen og de gener dette måtte have medført.

De allerbedste hilsner,

The Cloudnordic Helpdesk & Support Team

-----

The Hosted Exchange service is currently non functional as the Exchange redundancy is out of sync and the Exchange environment is in disagreement of which server is currently primary.

We are therefore forcing the secondary environments to understand that they are not primary, so that databases can be started on the primary system once again.

Solution: Hosted Exchange OWA and Outlook

Start: 18-08-2021 @ 13:30 CET
End: 18-08-2021 @ 15:30 CET (expected)

We apologize for any inconvenience this incident may cause.

Our very best regards,

The Cloudnordic Helpdesk & Support Team
Posted Aug 18, 2021 - 14:34 CEST
Update
We are continuing to monitor for any further issues.
Posted Aug 18, 2021 - 14:33 CEST
Update
Alle Hosted Exchange mail databaser er dismounted og vi er igang med at sørge for at disse bliver remounted - og vender tilbage hurtigst muligt.

Løsning: Hosted Exchange OWA og Outlook

Start: 18-08-2021 @ 13:30 CET
Slut: 18-08-2021 @ 14:30 CET (forventet)

Vi beklager på det kraftigste driftsforstyrrelsen og de gener dette måtte have medført.

De allerbedste hilsner,

CloudNordic Helpdesk & Support Team

-----

All Hosted Exchange mail databases are dismounted and we are trying to remount the databases - and will return soonest possible with more information.

Solution: Hosted Exchange OWA and Outlook

Start: 18-08-2021 @ 13:30 CET
End: 18-08-2021 @ 14:30 CET (expected)

We apologize for any inconvenience this incident may cause.

Our very best regards,

CloudNordic Helpdesk & Support Team
Posted Aug 18, 2021 - 13:59 CEST
Monitoring
Hosted Exchange er i øjeblikket nede og vi er igang med at undersøge årsagen - og vender tilbage hurtigst muligt.

Løsning: Hosted Exchange OWA og Outlook

Start: 18-08-2021 @ 13:30 CET
Slut: 18-08-2021 @ 14:30 CET (forventet)

Vi beklager på det kraftigste driftsforstyrrelsen og de gener dette måtte have medført.

De allerbedste hilsner,

CloudNordic Helpdesk & Support Team

-----

We are experiencing that Hosted Exchange is non functional and we are looking into the matter - and will return soonest possible with more information.

Solution: Hosted Exchange OWA and Outlook

Start: 18-08-2021 @ 13:30 CET
End: 18-08-2021 @ 14:30 CET (expected)

We apologize for any inconvenience this incident may cause.

Our very best regards,

CloudNordic Helpdesk & Support Team
Posted Aug 18, 2021 - 13:38 CEST
This incident affected: CloudNordic services (Cloud Control Panel, OX AppSuite + Dovecot Pro, Hosted Exchange).