The issue with email notifications has been mitigated and all the notification emails from the last 24 hours has been sent. We are now working to fully resolve it by requesting an upgrade in email limit from our hosting provider which prevents this issue in the future.
Posted Apr 09, 2019 - 14:14 CEST
Identified
Since this morning we are facing issues with notification emails again. No notifications are being sent at the moment; this has been caused by our mail hosting provider having shut down the service due to exceeded amount of bounced email. It will unfortunately take 24 hours to restore this functionality by our hosting provider. We will send an update when the email notifications are up and running again.
Posted Apr 08, 2019 - 12:00 CEST
Update
Thanks to everyone who let us know their thoughts in the inSpired community thread on this issue. We have decided not to send the emails now due to the following reasons:
1. The oldest notifications are from Sunday afternoon. Users will mostly be confused when receiving them this late
2. Registration-emails sent to users have been expired, so they cannot activate their account with this email any more. Users that just registered (on communities not using SSO login) can re-confirm their account by logging in to the community and clicking 'confirm email' banner on the homepage after login.
3. Newly registered users who have asked a question / started a topic will be confused by this mail as well, as they do not expect that notifications will be sent. Receiving them many days later will not be good for their experience.
Posted Apr 05, 2019 - 16:02 CEST
Update
A further update regarding this issue:
The source of the shutdown was our hosting provider shutting down our email service without any form of warning. They claimed that our bounce rate for non existent emails was too high - the cause of this was a very large spam attack on one of our customer forums (an SSO only community - a case in which we cannot stop fake emails from being registered as we don't control the registration mechanism).
To mitigate this from happening again in the future we have made an alteration to the platform where if an email bounces for a certain email address, we now store this email and will no longer continue to attempt to send to a none existent mail address.
We are also in the process of procuring a new 3rd party email check service which will check all (even the SSO) registered email addresses before ever attempting to send mail to the email address.
There are 2 million emails currently in the queue waiting to be resent - no email has been lost. 2 million emails would also take approximately 3 days to fully send - this would add even more of a delay to when recipients received their expected emails.
We realise that releasing the email queue is not a simple straightforward process and not something we wanted to just initiate without getting some feedback from our customers and users first. As doing so could lead to the following happening: 1. Users getting emails for events that occurred nearly 5 days ago - this could cause confusion. 2. As so many emails are backlogged we have to be extra careful that we do not exceed our allowed sent mail quotas as doing so would cause emails to stop working again and send us right back to square one.
Therefore we are currently investigating the idea of not sending the email queue at all - we are curious as to what our users' sentiment on this is? Critical emails like new registrations are able to re-trigger the email confirmation message again by logging into the community front end once more.
Email service is back to working order now. All platform emails should be sending as normal - we are continuing to closely monitor the situation.
A further update will made later on today and a more detailed incident report will also be published here.
Posted Apr 05, 2019 - 09:37 CEST
Update
The issue is progressing well - we have had confirmation of email restoration from our hosting provider now.
There are approximately 1 million queued emails that are being released from the message queue, these will need to catch up before normal service is restored. All queued emails will be sent although the resolution process for all emails to catch up could take approximately 2 days at current volumes.
Posted Apr 03, 2019 - 12:57 CEST
Identified
We are aware of an issue regarding platform Email notifications - at the moment no platform email notifications for branded Communities are being sent.
The issue (an upper limit by our hosting on emails rather an a platform issue) has been identified and a fix is being implemented, emails that have failed to send have been stored to a message queue and will be resent once the issue is fully resolved - No emails will be lost.
Affected time: 23:42 (CET) Saturday 30th March - Now
Posted Apr 01, 2019 - 15:01 CEST
This incident affected: Status of our EMEA Community Infrastructure (Status of our EMEA Community Infrastructure) and Status of our US Community Infrastructure (Status of our US Community Infrastructure).