Tuesday, October 31, 2017

When good Flows go rogue

At about 2.15am local time this morning, two Microsoft Flows in my Office 365 tenant went rogue and started blasting select email addresses with continual emails.

The two Flows in question I used to handle registrations for my regular monthly webinars. You can read more about how I created these here:

Using Microsoft Flow for event confirmations

Basically, they are triggered by a submission from Typeform. They then send the registrant a confirmation email as well as writing the details to a SharePoint list. These Flows are linear and incorporate no looping. These Flows had run successfully for over 12 months and had not been edited, changed or even viewed in a few weeks.

However, at approximately 2.15am local time, both of these Flows started to execute repeatedly sending hundreds of emails to a select group of people who had previously registered for the webinars.

image

The above shows a very small sample of the the sent items from the mailbox in question.

The mailbox sending out the emails from the rogue Flows was not my production mailbox so when I checked my production inbox just before 6 am local time when I awoke, I was quickly made aware of the issue from various people.

I immediately logged into the tenant with the rogue Flows and disabled the Flows but emails continued to be sent. I then went in and deleted the Flows but email continued to be sent. I therefore went in and created an Exchange transport rule to prevent that mailbox from sending anything further.

At that point the emails stopped being sent. In hindsight, that could have been from exhaustion of emails queued to be sent upon disabling the Flows. Whatever the reason, outbound emails had apparently stopped.

I immediately then logged a support request with Microsoft to confirm that the rogue Flows where not still running in the background, even though I had deleted them.

My request was escalated to the SharePoint Team who look after Flow. All the details of my situation were recorded and verified via a screen sharing session.

With the Exchange transport rule still in place I looked at the Flow Admin and found:

clip_image001

I then downloaded the CSV file to get more details and found:

clip_image001[5]

The two rogue Flows had each run almost 5,000 times. Clearly an issue.

At this stage Microsoft is still investigating the issue behind the scenes and I have removed the Exchange transport rule and confirmed emails are not being sent. Thus, it appears the rogue Flows have ceased.

What is interesting here is that the Flows that went rogue were only designed to run once someone completed the online Typeform. However, overnight they decided to run over and over again obviously caught in some sort of loop.

My guess as to the cause is that the Typeform connector used with Microsoft Flow received some type of update causing it to replay previous registrations over and over. The strange part is the fact that it kept repeating even though it was never designed to loop.

I am sorry to those people who received over 600 emails from me due to this issue and if it keeps happening or reoccurs please contact me asap and let me know.

With both Flows now deleted I am going to have to rebuild them but the question is how (can?) I prevent something like this happening again?

My current thinking is that I move the registrations to their own dedicated email box that I can, in the worst situation, completely delete if needed. I also need to work out some sort of rule that prevents constant email being sent if they exceed a threshold (say 10 emails in 10 seconds) and take appropriate action.

I’ll have to have a think about how (or if) I can do this and how I go about creating and monitoring any new Flows I create. I welcome any suggestions people might have on how I can prevent a recurrence.

A painful example of what happens when automation breaks.