Developer forum

Forum » CMS - Standard features » MergeAndDeliver failed

MergeAndDeliver failed

E-mail notifications
Tomas Gomez
Reply
Posted on 01/09/2021 11:58:55

Hi,

We have an error sending an Email marketing campaign to 5000+ users. Some mails are sent and some others are not sent.

The task is timed out, and in Monitoring appear and error for each mail not sent:

Url: /Admin/Module/OMC/Emails/Statistics.aspx?OpType=RetryEmail&newsletterID=208
Description[199]: MergeAndDeliver failed. RecipientId: '111159'.
Exception type: ArgumentException
2021-09-01 10:53:38.024: [199]: MergeAndDeliver failed. RecipientId: '111159'.
System.ArgumentException: Value does not fall within the expected range.
at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
at System.Web.Hosting.IIS7WorkerRequest.SetUnknownResponseHeader(String name, String value, Boolean replace)
at System.Web.Hosting.IIS7WorkerRequest.SetKnownResponseHeader(Int32 index, String value, Boolean replace)
at System.Web.HttpHeaderCollection.Remove(String name)
at Dynamicweb.Controls.Legacy.HttpHelper.SetSecurityHeaders()
at Dynamicweb.Frontend.PageView.SetSecurityHeaders()
at Dynamicweb.Frontend.PageView.Output()
at Dynamicweb.EmailMarketing.Renderer.BuildPageHtml(Int32 intPageID, String template, PageViewContext pageviewContext)
at Dynamicweb.EmailMarketing.EmailContentProvider.GetContent(Message message, Recipient recipient)
at Dynamicweb.Mailing.MessagingHandler.MessageHandler.MergeAndDeliverInternal()
at Dynamicweb.Mailing.MessagingHandler.MessageHandler.MergeAndDeliver(RecipientQueueItem rqi)
at Dynamicweb.Mailing.MessagingHandler.QueueWorker.<>c__DisplayClass7_1.<StartMergeAndDeliver>b__0()
2021-09-01 10:53:38.040: Request headers: [   "Connection",   "Accept",   "Accept-Encoding",   "Accept-Language",   "Cookie",   "Host",   "Referer",   "User-Agent",   "sec-ch-ua",   "sec-ch-ua-mobile",   "upgrade-insecure-requests",   "sec-fetch-site",   "sec-fetch-mode",   "sec-fetch-user",   "sec-fetch-dest" ]
2021-09-01 10:53:38.040: Request item keys: [   "UserManagement.GetUserById.81836",   "UserManagement.GetUserById.81806",   "UserManagement.GetUserById.81816", [... MORE IDS ...] "UserManagement.GetUserById.82983" ]
2021-09-01 10:53:38.040: Request form: []

The only information about this "MergeAndDeliver failed" error inthe forum is at this post, biut it has a different cause (An item with the same key has already been added)  

Does somebody know the cause of this error and how to fix it?

Regards,
Tomas


Replies

 
Tomas Gomez
Posted on 14/09/2021 14:17:14
Reply

Hi,

We sent another newsletter and has the same issue, some mails are sent and some not. There are thousands of errors in the Monitoring log.

Which parameteres or config should I check to search for the cause of this error?

Regards,
Tomas

 
Scott Forsyth Dynamicweb Employee
Scott Forsyth
Posted on 14/09/2021 20:39:16
Reply

Hi Tomas,

I don't know if this will help with your first question above, but check out the MailRecipient table. The following query is helpful: select * from EmailRecipient order by RecipientSentTime desc

Each send to a user is stored there, including the error message per recipient if there is a failure. That may uncover some clues.

The first error suggests that it's something with generating the email itself (and not the sending of the email). Some field that it needs is probably empty or null or something. If you can narrow it down to a particular user, you can compare that user to a known good one to see if anything stands out as missing.

Scott

 
Tomas Gomez
Posted on 15/09/2021 11:48:17
Reply

Hi Scott,

Thanks for your suggestion.

Searching the DB, the RECIPIENTERRORMESSAGE is empty and the RECIPIENTERRORTIME is null for all the emails. This reinforces your remark that it's something with generating the email itself (and not the sending of the email)

As well, there is not an empty data because the failed mails are sent when re-running the mailing task. That is, we run the task and some mails are sent, so we have to run it again several times until all mails are sent.

Is there ny other place where I can get more info about this error?

Regards,
Tomas

 
Scott Forsyth Dynamicweb Employee
Scott Forsyth
Posted on 15/09/2021 15:26:37
Reply

Hi Tomas,

What may have happened is that one of the emails failed, and then it stops on that problem email and doesn't continue on. When you run into that again, look at the results in the database again and you'll see that the most recent ones are still waiting to send (with nulls in many of the fields). Look for the most recent one that looks different. In other words, find the most recent message that failed. That will show you details on the message that it got hung up on.

btw, I'm giving most details by memory without a current failed situation to refer to. It may be that RecipientSentTime is also null until it's sent. In that case, it's probably RecipientId that you should order by instead.

Regarding the error message, I don't know that it's clear enough which field is the problem field. However, you do have a good clue to work with. That's RecipientId = 111159. Check out select * from EmailRecipient where RecipientId = 111159. That will show which user it got hung up on. You may find that something stands out with that data that will help track down the cause. Look for anything out of the ordinary like a blank date (it expects a real date) or a null or negative integer or possible an empty field that would typically be filled in.

Scott

 
Tomas Gomez
Posted on 17/09/2021 13:05:44
Reply

Thanks for the tips, Scott.

I will look for them on the next newsletter to our users

Regards,
Tomas

 
Tomas Gomez
Posted on 22/09/2021 17:48:31
Reply

Hi, Scott

I just sent the newsletter to 5800 users without any issue. Maybe the problematic recipient has unsuscribed or changed the data, I don't. know.

Anyway thanks for your supportt. i'll write again if I detect any other issue.

Regards,
Tomas

 
Scott Forsyth Dynamicweb Employee
Scott Forsyth
Posted on 23/09/2021 19:41:27
Reply

Hi Tomas,

Glad that it worked this time. As you say, it could be a problematic recipient, or some intermittent issue like a timeout. I'm sure it will come up again in the future, but nice to see it work well this time.

Scott

 
Tomas Gomez
Posted on 29/09/2021 13:29:12
Reply

Hi Scott,

Today the newsletter has errors again. 

  • Searching in the DB the query "select * from EmailRecipient order by RecipientSentTime desc", there are records that has NULL data, I copy two examples below.
    • There is not null data in the record of the RecipientId with the error (134410)
    • All the records have sent date. i couldn't find any difference between the last record sent before the error (supposedly the mail with the error) and the other records.
  • I copy below the log in EmailMarketing > ScheduleHandler > Errors, just in case it can offer any clue.

The main difference between last week (no error) and this week is that last week the check "Render content for each recipient" was unchecked last week, and checked this week. Could it be a bug in this feature?

Regards,
Tomas

Log in EmailMarketing > ScheduleHandler > Errors

2021-09-29 09:22:12.6465|ERROR|EmailMarketing/ScheduleHandler|Send Email Error - Email ID '221'
System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Threading.Thread.SleepInternal(Int32 millisecondsTimeout)
   at System.Threading.Thread.Sleep(Int32 millisecondsTimeout)
   at Dynamicweb.EmailMarketing.Email.KeepAlive(Email email)
   at Dynamicweb.EmailMarketing.Email.SendEmail(Email email)
   at Dynamicweb.EmailMarketing.EmailScheduleHandler.ExecuteInternal(Email email)

Examples of the query "select * from EmailRecipient order by RecipientSentTime desc

 

133612 82184 MyName   MyName@gmail.com 212 29/09/2021 12:22:51 NULL
{ "Email:User.UserName": "MyName@gmail.com", "Email:User.Name": "MyName", "Email:User.FirstName": "", "Email:User.LastName": "", 
"Email:User.Address": "MyAddress", "Email:User.Address2": "", "Email:User.ZipCode": "03009", 
"Email:User.City": "Alicante", "Email:User.Country": "", "Email:User.Phone": "123456789", 
"Email:User.PhonePrivate": "", "Email:User.PhoneMobile": "123456789", "Email:User.Fax": "", 
"Email:User.CustomerNumber": "", "Email:User.Currency": "", "Email:User.Image": "", 
"Email:User.Company": "", "Email:User.Department": "", "Email:User.JobTitle": "", "Email:User.PhoneBusiness": "MyName", 
"Email:User.LoginToken": "2885a317df5782ebe9378106c176668c2edaef6009e8a514a03d244e1303979e", "Email:User.AccessUser_LinkedIn": null, 
"Email:User.AccessUser_UserIsAdmin": null, "Email:User.AccessUser_CanWriteOwnAddress": null, "Email:User.AccessUser_CIF_NIF": "123456", 
"Email:User.AccessUser_CemevisaUserId": 27710.0, "Email:User.AccessUser_Tag_mailchimp": null, "Email:User.AccessUser_Member_rating_mailchimp": "2", 
"Email:User.AccessUser_ERPAddressCode": null } 799d3c7d-fb57-420e-a0d7-aa2ef97f3b97  
 
134410 100942 MyName@gmail.com 212 29/09/2021 12:40:18 NULL
{ "Email:User.UserName": "MyName@gmail.com", "Email:User.Name": "", "Email:User.FirstName": "", "Email:User.LastName": "", 
"Email:User.Address": "", "Email:User.Address2": "", "Email:User.ZipCode": "", "Email:User.City": "", "Email:User.Country": "", 
"Email:User.Phone": "", "Email:User.PhonePrivate": "", "Email:User.PhoneMobile": "", "Email:User.Fax": "", "Email:User.CustomerNumber": "", 
"Email:User.Currency": "", "Email:User.Image": "", "Email:User.Company": "", "Email:User.Department": "", "Email:User.JobTitle": "", 
"Email:User.PhoneBusiness": "", "Email:User.LoginToken": "1e90d05d835d1c424761016552ed776b8f97b3666aaf8a7d02821283470a8746", 
"Email:User.AccessUser_LinkedIn": "", "Email:User.AccessUser_UserIsAdmin": false, "Email:User.AccessUser_CanWriteOwnAddress": false, 
"Email:User.AccessUser_CIF_NIF": "", "Email:User.AccessUser_CemevisaUserId": 0.0, "Email:User.AccessUser_Tag_mailchimp": "", 
"Email:User.AccessUser_Member_rating_mailchimp": "", "Email:User.AccessUser_ERPAddressCode": 0.0 } fa91a79e-e23a-4d99-9bcb-aac38e024482  
 
Scott Forsyth Dynamicweb Employee
Scott Forsyth
Posted on 29/09/2021 15:17:39
Reply
This post has been marked as an answer

Hi Thomas,

That does provide some more clues.

The "Render content for each recipient" setting does mean that every email needs to have a webpage rendered for them, so it's much more intensive on the server and has more opportunity for errors. 

As you say, that particular user doesn't seem to be the cause. It appears that it just happened while on that user. If you want to confirm, you could edit the user in EmailRecipient so that it's set again, and see if it works the next time. If so, then you will have confirmed that the issue isn't related to the recipient itself. It's highly likely that it will work.

That error message is occuring in Email.KeepAlive. You would think that would be a good safe and simple call. Possibly the web server is over worked due to rendering so many emails (aka webpages) and something causes that failure.

If anyone on the platform team is listening on this thread still, this would be worthwhile to look into. We had a similar issue last week sending a large welcome email for a new account. We had to manually restart/continue it about 20 times before it sent them all. We get that on a lot of sites, so it's not usual. 

Ideas for improvements are:

  • Look into this 'Thread was being aborted' in Email.KeepAlive. What would cause that.
  • Test with 100K rendered emails. It will likely fail a few times
  • It would be amazing to have a governor to keep CPU and resources done during a large mailing (this is very important to us). For example, have a X millisecond pause because each email or every X emails. Even better would be if the site can tell its own CPU (I don't know if the worker process has enough permissions typically) and it can throttle to a set CPU level. It terrifies us to send out 100K welcome emails because it's just when we are generating a lot of interest for the site that we bog down the server by generating 100K emails, so it's a catch-22. If we could even out the CPU and drag out the sending a bit longer, we can keep the CPU and site performance good during these times. The same applies to regular mailings too.

Thanks,

Scott

Votes for this answer: 1
 
Tomas Gomez
Posted on 30/09/2021 12:10:54
Reply

Hi Scott,

Total agreement with your diagnosis. It is not an issue of the users but of the server, your "therapy" looks accurate.

Regards,
Tomas

 
Scott Forsyth Dynamicweb Employee
Scott Forsyth
Posted on 06/10/2021 05:18:32
Reply
This post has been marked as an answer

Hi Tomas,

I created a feature request for the throttle so that it doesn't get missed in this discussion. The throttle is one of 2 things that we discussed. The other is why large mailing seem to just abort sometimes. 

Here's the feature request post: http://doc.dynamicweb.com/forum?ThreadID=78200&PID=1605

Best Regards,

Scott

Votes for this answer: 1
 
Tomas Gomez
Posted on 06/10/2021 09:40:03
Reply

Good request :-)

 

You must be logged in to post in the forum