We have resolved all the issues and completed all migrations and cleanup. Everyone is now on the new tech and everything is working correctly.
If you moved away from ExchangeDefender and still need some data, it will be live for another 60 days on our legacy platform, here are retrieval instructions.
If you would like our assistance, please let us or your IT provider know.
Update 06/19/2020 20:43 PM 1 days
We wanted to offer one final update before we close the ExchangeDefender NOC covering our Exchange migration.
The past few days have been largely consumed with cleanup and misc configuration requests already covered here. By far the biggest issue has been reseeding and legacy copies of mailboxes exceeding 25GB using nearly all internal, Microsoft/powershell, and third party tools there seems to be no predictable, foolproof, failsafe way to migrate a mailbox. The larger mailbox gets, the more difficult it seems to port (one particular user has been waiting on their mail for 2 weeks - they have a 70 GB mailbox and it's taken dozens of attempts of repair/recheck/export/move/seed/verify) and it has been the greatest source of frustration for us and for our clients, largely because the progress indicators are unreliable and process very prone to failure the larger the mailbox gets. This is why when we started offering 2016 years ago we set up the 50GB quota with 15GB realtime and 35GB in place archive setup so we can deliver on both service restoration and disaster recovery.
We are continuing to assist our partners in the following areas:
At this point everyone can connect, mail delivery and legacy reseeding are in progress, all systems for Exchange, ExchangeDefender, and
LiveArchive are working normally.
We're looking forward to closing this ugly chapter. We have done everything in our power, and we couldn't be more thankful for our partners who have helped us with the cleanup of the Microsoft disaster. Thank you. We are sorry that so many clients were inconvenienced with this, we planned and managed every step of this migration by the book with thousands of other successful migrations that happened from 2016 Aug 2019, but when your vendor pulls the rug underneath you and damages hundreds of mailboxes unannounced many of us will soon be enjoying the first day off in June. The only good news is, you will not have to go through this process again.
Update 06/16/2020 19:18 PM 4 days
We wanted to offer a major update on the migration, specifically covering the major issues we have addressed for some clients during the cleanup phase.
Distribution groups, forwarding - We have received reports from several organizations regarding issues covering distribution groups, group members, forwarding account directions (forward vs. store & forward). If any objects failed to import due to configuration/contents/policy/etc it is in the retry queue and will be published shortly.
Add / Delete Mailbox - We have addressed a bug in the add/remove process that was prohibiting certain organizations to add/remove accounts. Originally, as noted on anythingdown.com NOC, we blocked this function entirely because users were looking at an empty list and creating mailboxes (that would cause a collision when the new mailbox was migrated from the source). This problem is fixed, if you encounter an issue please open a ticket with a screenshot and as much info as you can provide.
Add / Delete Organization/Domain - At this moment it is not possible to add/remove organizations, or those that were in the system recently. In order to finalize the migration, the routing policies are locked down (meaning if you deleted a domain, ExchangeDefender will still treat them as local). We look forward to wrapping this up shortly.
Password / Login issues - This is by far the biggest ticket group category, we are still processing double digit requests for credentials, credential resets, and credential tests. Similar to the next group:
Outlook issues - We are still spending a lot of time going through the basic Outlook configuration steps. For an overwhelming majority, this transition has been transparent. Those that did not and had to take a manual configuration route, the process has been described at anythingdown.com 1) Make sure you have an autodiscover record 2) Make sure it propagates, then run the autodiscoverregistryhacks.zip 3) If you don't control your DNS, make adjustments to your local systems hosts file 4) Setup Outlook with autodiscover, the UPN must be used as your login address if you've changed it from your primary SMTP address.
Missing & Syncing Emails - Every mailbox that has been reconnected has either had all it's mail delivered directly, delivered in a Catchall account - user@domain.com. Some users are confusing items they see in their Inbox in LiveArchive but not in their Outlook/OWA (but after extensive searching we keep on finding missing messages in folders, Deleted Items, etc). If something is missing and absolutely critical in LiveArchive just click on the message and click Forward to your email address and the message will be forwarded to your Outlook/OWA.
Store & Forward - Several users were also unfortunately caught up in a custom policy that did not get migrated to the new Exchange. These are more legacy configs we did for some users in AUDC, things like renaming the OU or primary domains. For some of those accounts, the store and forward rule because a forwarding only rule, skipping the Inbox and going straight to the person that it's being copied to. We have fixed this issue and it should not be happening again.
Autodiscover - We have gotten several complaints about autodiscover. Microsoft has removed manual configurations in 2013 and no modern version of Microsoft Exchange supports a manual server setup. However, this is something that could be easily rectified even with minimal technical skills by modifying the local hosts file if you don't have the credentials to do it properly by modifying the DNS. Absolutely everything in the new infrastructure relies on the autodiscover record!
iPhone / Android Setup - For the most part, we are just confirming that all mobile devices should work fine with owa.xd.email as the server name, ditto for EWS integrated applications, we have not received more than an inquiry for the server name. For Android, things get sketchier when you consider all the different vendors, apps, and configurations. Again, so long as autodiscover is present and configured properly and your device is using a modern client, it should just work. When it doesn't, recreating it takes a few minutes.
NDR - Non delivery receipts and errors are always of high interest to our NOC team as we continue to go through cleanup and audit all the tickets and users.
These are the issues we are currently working on, in 3 shifts, and sorting them all out as fast as possible. I know that for many of our clients this transition has been messy, but you are on such a better and more secure platform that won't require you go through this process again. While modern platforms are more secure, their recovery from a disaster or issue (as some of you unfortunately went through) is extensive and at times unpredictably slow - so you have this much of a committment for us, we will make sure LiveArchive is able to step in on a whole new level when things like this happen.
Update 06/16/2020 07:53 AM 4 days
We're in the final stretches of importing public folder data from the SCROOGE, LOUIE and GLADSTONE cluster. Folders from DARKWING were successfully imported. Public folders from ROCKERDUCK will be exported in the next batch along with public folders with more than 5000 items.
Update 06/15/2020 10:11 AM 5 days
We are near finished with completing the first batch of data imports for public folders. We will be remapping identities and public folder permissions once the import completes (estimated around 730-8AM Eastern). We are targeting making the first batch available by 10AM Eastern
Update 06/15/2020 06:04 AM 5 days
Public Folder Update
We're in the progress of migrating public folder data. We fully intend on restoring all public folders with less than 2000 items by the start of business on Monday June 15th. However, we are not imposing the item limit to aptly named company calendar and contacts folders. Unfortunately there are a lot of public folders that were previously renamed in Outlook (on the legacy clusters) which is making remapping of the new identities difficult.
Public folders with more than 2000 items will be imported after the initial batch import completes.
Public folders will automatically become visible as additional mailboxes in Outlook and can be mapped in OWA by adding the public folder to the favorites. Public folders will be renamed by prefixing the org to the public folder name, removing the MSP name and the customer domain. For example, the previous public folder "demodomain.com_public" for the org demo will become "demo_public". We understand renaming the public folders may require some minor changes in LOB software, however, the renaming is required to ensure uniqueness of the public folder name.
Partners should expect to see public folder permission management inside of service manager by the end of the week.
We are taking requests to prioritize public folder imports. If your customer does not see their public folder by the start of business, please open a support request titled "Public Folder: expedite" . Inside the service request we will require the domain name of the customer requiring public folder data. If you happen to know the name of the specific public folders to expedite, they will be accommodated in the request. Customers with more than 2000 items requesting expedited access to public folders will have their expedited request filtered to mail within the last 90 days and the rest of their data will be imported in a subsequent batch.
Update 06/10/2020 15:45 PM 10 days
11 AM Update
Mailbox Data Reseeding
At 10PM last night we picked up the pace of email delivery for users whose automated Exchange migration failed. If you had a user in that situation for days, we have created an empty mailbox for them so they can start working, and their old mailbox is being imported in the background. Within a day, it will look exactly the same as the old thing except much faster, safer, more secure, and on the new generation of Exchange.
Account Connections
We are still assisting clients with password resets, account reconnections, misc Outlook issues and mobile. At this time there is nothing else to report on this front as all the issues are addressed following the instructions already mentioned in this NOC advisory. Outlook, mobile, OWA, EWS, printer/CRM connections, everything works. This is by no means a new system for us, remember that we started offering Exchange 2016 in 2016 and have been facilitating migrations to it ever since. While the disaster has been in the migration failure for so many, the experience on the new platform is not raising any issues. We're even slightly hopeful because many of the reported issues clients had in Exchange legacy (connectivity, timeouts, email delays, etc) have not been a problem on the next generation so far.
Next Steps
We will likely spend bulk of Wednesday and early Thursday completing mailbox reseeding, redelivery of mail from 6/1 - 6/9, and getting everyone connected. Thankfully LiveArchive has been a life saver and people have been able to work from there, we want to make sure everyone is back in their Outlook, OWA, mobile experience and we can move forward.
Public Folder data sync and reconnection will happen closer to the weekend, at this time we do not have a firm time frame but the process has been scripted, tested, and certified for months so we do not expect issues there.
We will keep you up to date here but PLEASE if you still are not able to get into your Outlook Web Access at https://owa.xd.email/owa open a ticket with our team and we'll help you get reconnected and going.
Update 06/10/2020 03:07 AM 10 days
11 PM Update
We are starting to replay more mailboxes faster and get everything in sync.
At this point the next stage of migration recovery is largely automated and one-on-one for weird Outlook/Sync problems. We will not have further updates from the migration team on this ticket until 11AM.
Update 06/09/2020 18:28 PM 11 days
We are still working on password resets, last week mail sync, and old mailbox sync for those that failed Microsoft's automated migration.
If you are at this point not able to get into Outlook Web Access, please open a ticket and list which email addresses that are not able to login to OWA (one per org)
Update 06/09/2020 07:40 AM 11 days
3:30AM
We are still wrapping up tickets from the day, a rather busy Monday getting everyone sorted out with Outlook cleanup and resync as required.
If you encounter any problems and are updating tickets, PLEASE provide screenshots and/or detailed error/log reports if you possibly can. Our support front line typically collects the information, double checks it, passes it on back to NOC or network admin on duty for resolution. If the error is vague the wait time and delay can depend on how long it takes support to actually determine the issue. So for example, if you're having login problems please specify your credentials and the site along with the error that you get. If you are getting mail delivery errors, post the NDR (non delivery receipt) or anything that will help us get your account and issue sorted out faster.
Update 06/08/2020 23:36 PM 12 days
7:35 PM
We are still replaying last weeks email into all affected mailboxes. You will see a "CatchAll-Import user@domain" folder with messages that were delivered before your mailbox could successfully go online.
Update 06/08/2020 23:13 PM 12 days
Monday
Overall, things are going in the right direction and we're going through typical Outlook cleanup, some reseeding, etc.
We have spent the day resetting passwords and helping clients reconnect their Microsoft Outlook, etc. Here is what we're seeing and hearing:
1. Many accounts need password resets. About half the work we've done today in support has been a simple password reset, so if your account is not letting you in with credentials you know, please reset it and give it 5 minutes to sync up. If that doesn't work, open a ticket immediately with the email address as the subject. If your mailbox is stuck in the password reset request, likewise, open a ticket.
2. Questions about Exchange autodiscover. No, you do not need to change your autodiscover if you've already got one. Yes, you absolutely need to have an autodiscover record for 2016 and above there is no "server name" because Outlook will not resolve it for configuration (it's outlook.xd.email). For mobile devices and anything relying on Exchange Web Services you can use owa.xd.email.
2a. Problems with Outlook. Please read https://www.exchangedefender.com/blog/2020/06/exchangedefender-exchange-setup/ almost all Outlook tickets are addressed with this. If you can login with OWA, if you have run autodiscoverregistryhacks.zip batch file as Admin, if you have Outlook 2010 with latest patches or newer - you should have 0 problems connecting to OWA.
3. Empty mailboxes, missing last week, etc. There is no reason to worry, we have mailboxes archived 3 different ways so if you can get into LiveArchive or had a domain org, we've got your email. We're replaying it but it is moving slooooooow. Essentially the process searches a temporary mailbox, locates messages sent to a specific primary email address, and then pushes those messages up. For blank mailboxes, described yesterday, they will be reseeded as well. Everything is moving in the right direction, we just need time. We're pushing the system to the limit in terms of traffic.
4. OWA "Lite" mode. Please do not use popup blockers on Outlook Web Access. If you do, it will block several critical components OWA has and it will launch in a light mode that looks like OWA from Exchange 2007. Just add it to the trusted sites or disable popup blocking on it. The theme can be managed through Outlook settings.
5. We have seen some NDRs that we are still investigating. So far we can explain away all of them (DKIM errors from LiveArchive, missing aliases, old organizations previously removed from Exchange org recreated because the migration has been staging since November!)
6. We have seen a few mobile device reports (calendar sync) that we are still investigating. We as of yet do not know if this is an issue, if the Outlook app is reprovisioned/reconnected properly, etc. In general, all mobile devices should use the owa.xd.email as the "server" and their UPN (login) and password.
Update 06/08/2020 00:47 AM 13 days
9 PM Update EST:
At this point we have moved, created, troubleshooted, and sync'd every clean mailbox in our enterprise and all accounts are active and working.
Any mailbox that we could not successfully autoreconnect to the new organization is being created from 9PM to 11PM and mail will be replicated into it manually.
This move assures that access to email is restored for everyone on our platform, regardless of Outlook/Exchange instrumentation problems. It also puts every inbox into the service manager and gives our clients the ability to manage their mailbox immediately. If you're having issues, create a new profile, reconnect the mailbox and all your data will sync down either immediately for active accounts and shortly for accounts that have been reported as inaccessible via OWA. We are doing this so that everyone can use OWA and Outlook immediately.
If you choose to setup a new profile, you're set. If you do not create a new profile, your Outlook will start up and give you two options "Use Temporary Mailbox" or "Use Old Data" - please click on "Use Temporary Mailbox". Here is a pic of the screen for reference:
Update 06/07/2020 21:27 PM 13 days
5:30 PM EST:
We are continuing to go through accounts of everyone that reported an issue. IF you reported an issue OR IF we are unable to login with any legacy/2016 credentials, we are continuing to work through those accounts and your mailboxes WILL be online tonight.
In terms of setup, the servers are owa.xd.email and outlook.xd.email, but neither is neccessary if your autodiscover is set. If you have/had an autodiscover, it does not need to be modified, all the autodiscover instances point to the new infrastructure. Make sure you've run autodiscoverregistryhacks.zip as the Administrator on the workstation, reboot and Outlook will restart. If Outlook was open the whole time during the move, you may get the prompt saying "The Microsoft Exchange administrator has made a change that requires you to quit and restart Outlook." but the result will be the same, Outlook will restart on the new infrastructure with your folders and new email in the Inbox.
Update 06/07/2020 17:05 PM 13 days
Noon:
We are continuing to go through all the tickets and confirm / double check every account. We've been following up with clients that have contacted us via tickets, Facebook, SMS, email and providing passwords, reconciling the view in the Service Manager.
We are anticipating that this process of double-checking every single account will be completed by 3-5PM EST today at which point we would have confirmed everyone that has raised an issue is online (and many others that have not).
Mail flow to everyone is resumed as soon as their mailbox is online and redeliveries are happening so we're confident that as soon as we clear the list above we can say everyone is online and running without issues.
Update 06/06/2020 14:34 PM 14 days
10AM EST
We are continuing to work on the tickets and reports of users that have not been able to login, many of these have been resolved already by our partners (by changing the password) but we will not leave a single ticket alone until we have been able to confirm everything is operating normally. This is an extremely important step in a migration because small issues in migration can cause larger issues down the road. (many of the accounts that did have problems had them because of a very custom / complicated configuration such as having multiple domains in the same organizations randomly used for authentication, clients opening 5 mailboxes in a same Outlook profile and typing in the wrong password, odd permissions, etc.) This is not to place blame on anyone that had such a configuration, we understand that every business has legitimate business process and operations needs, but when Microsoft & support tools we have at our disposal decide to stop supporting such configurations we have no easy means to recover them and it becomes a hours long ordeal or surgery.
We are still going through the tickets at this hour, we are waiting for the last few mailboxes to complete retry/reseed/retransfer/redo before we apply recipient policies so we can redeliver email that went into catchall mailboxes, intermedia failover authorized relay queues (so we don't bounce stuff that can be delivered), etc. None of these "missing" items are missing from LiveArchive, so if you're missing anything keep in mind you've had a LiveArchive mailbox since the day you signed up for ExchangeDefender for occasions exactly like this one. We have even deployed free LiveArchive for clients that purchased our barebones entry level Essentials products.
We are continuing to work on this and will provide updates as well as the timeline because the next steps are not reliant on using Microsoft tools (no PowerShell scripts with no progress or status indicators).
We have been heistant to introduce any other workarounds during this window as we were completely blindsided and needed to recover full access to as many people as possible. This will never happen again. Over the weekend we are working on some solutions that will greately expediate the self management of these mailboxes and authentication/ID processes.
Update 06/06/2020 06:55 AM 14 days
Midnight
Most of us are still here going through tickets and double-checking every failed login, mail bounce, and recipient policy. This part is currently underway and we anticipate completing it by 8 AM EST when the next post will be published. This step is critical in applying all the missing distribution group, alias, security policies, public folder permissions, folder mapping, etc. At that point, all "missing" mail that was not delivered to those aliases/dgs will be replayed and delivered.
We are still monitoring a few mailboxes that failed import/connection to the appropriate Exchange organization. Exchange mailbox import can fail after a certain threshold of bad items in the mailbox (we started with 50 and are currently at 100). Because all of these mailboxes currently have LiveArchive and can work, we're resyncing some of them.
We continue to see tickets regarding Outlook connectivity. If it works in OWA, it will work in Outlook. If it doesn't connect in Outlook 1) Download autodiscoverregistryhacks.zip, run "cmd" as Administrator, reboot 2) Confirm the workstation can resolve autodiscover.YOURDOMAIN.COM 3) Check OWA at https://owa.xd.email and confirm that you have the right credentials 4) Open your Outlook (2010 with all service packs or newer) 5) Start Outlook. You will be prompted to accept new configuration, or "repair/rebuild mailbox", or "follow mailbox to the new server" and after accepting all of that... your Outlook will open up, sync up all your folders and you're back where we started just with the most up to date version of Exchange possible.
The next update will follow at 8 AM EST. We are continuing to work through the weekend and hope to be able to sleep soon. Thank you for your patience and kind words through this process, we realize this is a giant inconvenience and a business interruption.
Update 06/05/2020 22:37 PM 15 days
Update 6:30 PM EST:
We have spent the entire day troubleshooting mailboxes across domains. The most common issue is the login mismatch, where users are using the wrong UPN or wrong username. We have been troubleshooting some tickets with missing aliases / distribution groups (those will be reapplied and mail redelivered), etc
We have been updating clients throughtout the day so if you opened a ticket this week you will get an update on your user/domain as soon as we confirm that they can access Outlook / OWA (this has been our protocol throughout the week).
Next update will be tonight at midnight, we are currently working on mail routing reports, troubleshooting reported failed OWA/Outlook logins, PF/DG/AL outstanding imports/issues/debugs.
Thank you for your patience. We are moving through this slow and tedious detailed work as fast as possible, every issue in this migration will be addressed. If anyone has ever unplugged a cable during a migration, PC update, or similar that's the disaster we had to clean up (but with certificates, authentication, routing, etc) that was crash dumped into our lap.
Update 06/05/2020 17:07 PM 15 days
Update as of 1PM:
The only issues we are still seeing is related to authentication, password failures and resets. Team is moving through them quickly.
Several Outlook tickets came up, please follow directions in this post (https://www.exchangedefender.com/blog/2020/06/exchangedefender-exchange-setup/). If they do not have Outlook 2010 or newer, if they do not run the autodiscoverregistryhacks.zip file on their PC with administrative privileges, if they do not reboot, Outlook will not connect to Exchange.
Update 06/05/2020 12:01 PM 15 days
7AM EST
As of 6:23am all mailboxes have been reconnected in the new environment. Everyone should be able to access their mail via Microsoft Outlook, Outlook Web Access, LiveArchive, etc. The instructions on how to handle misc issues is in this NOC post.
As of 5:30am all mail routing issues have been addressed and ExchangeDefender is delivering to all platforms as usual.
All access is fully restored to Microsoft Outlook, Outlook Web Access, and recovery systems.
We will be spending most of the day working wth clients and partners with login issues. Most people do not know their Outlook password, and misc issues surrounding Azure AD and AD congestion have slowed down credential resets earlier in the week. Today our final priority is helping the users with these authentication issues.
Update 06/04/2020 22:28 PM 16 days
6PM Update
We are continuing to go through misc Exchange issues and mailbox reconnections. We believe we have at this point sorted out all but the following service items which we are working on at the moment. The biggest issue today was continued latency in Active Directory, something that we've addressed and solved as of 4:30 PM. Here is a summary of what we have left:
Mail routing and delivery
At 6PM we solved the issue of roundabout mail routing for mailboxes that have not been connected yet. Because the rug was pulled under us on Sunday we have been exceptionally aggressive towards archiving - so any mail that couldn't be delivered to the primary email address would be routed to LiveArchive, to catch-all domain mailbox, and a failover node.
Items in LiveArchive that are not in OWA/Outlook
This is related to the routing/delivery of the mail between various failover systems. Good news is because we have written LiveArchive for this very reason, we will be able to sync Sent Items right into their Exchange mailbox.
Mailbox moves and remounts
We are doing the final sweep of mailboxes that have failed to mount. We still have a few partners that are affected.
Update 06/04/2020 13:55 PM 16 days
Important:
0. Download and run the registry files as the administrator, regardless of whether you already have autodiscover or not. The file is here: https://www.exchangedefender.com/media/autodiscoverregistryhacks.zip
1. If you already have an autodiscover record DO NOT CHANGE IT. We are
2. If you do not already have an autodiscover record, YOU DO NEED TO CREATE ONE. autodiscover cname autodiscover.xd.email.
The trailing dot is important, depending on your DNS software. You can tell if you've done it correcltly if your DNS lookup looks somewhat like this (from Windows 10 command prompt):
nslookup autodiscover.ownwebnow.com
Server: 65.99.255.161
Address: 65.99.255.161#53
autodiscover.ownwebnow.com canonical name = autodiscover.xd.email.
autodiscover.xd.email canonical name = autodiscover.dal.xdmail.online.
Name: autodiscover.dal.xdmail.online
Address: 72.249.54.208
If the autodiscover is missing, you will get an error instead:
nslookup autodiscover.xdref.com
Server: 8.8.8.8
Address: 8.8.8.8#53
** server can't find autodiscover.xdref.com: NXDOMAIN
----
To figure out if your workstation can detect the autodiscover record, Start > cmd > nslookup autodiscover.YOURDOMAIN.COM (substituting your domain name for it)
Update 06/04/2020 13:30 PM 16 days
9AM Update:
We are moving through our tickets, resolving any outstanding issues, and keeping people in the loop here and support.ownwebnow.com
1. One of the things we are noticing more and more is that some people do not have autodiscover setup. If you can login to Outlook Web App (https://owa.xd.email/owa/) or NGE, then your credentials are good and your mailbox is online; so if Outlook continuously prompts for a password or takes forever to start up a session, confirm that you have an autodiscover record and that you have run the registry hacks. The process is described here: https://www.exchangedefender.com/media/ExchangeDefenderExchange.docx
2. Overnight we added some more horsepower to our LiveArchive NGE, it's actually moving faster than Exchange at this point so we hope the improvements make it easier to work.
3. We were able to identify and fix the internal routing issues, some clients reported that they saw some messages in LiveArchive that did not show up on the Exchange side. The same issue affected many login failures (from time to time, not completely), some mail delivery, (it's in the catchall), etc. We believe the changes that were made overnight will allow user logins to proceed without an issue.
4. If you have an Office365 version of Outlook, or if you setup your outlook 5-10 years ago with autodiscover but are now running on the latest one, you will still need to get and run as administrator (and reboot) in order to enable non-Microsoft Exchange servers. If you HAVE autodiscover and you are still getting repeated password problems, please follow the instructions in our guide on how to download and execute the patch, reboot, and after a minute or two it should be fine.
5. If you've never had Autodiscover, manual server configuration has been decomissioned by Microsoft in Exchange since 2013. We recommend setting up a new mail profile (Start > Control Panel > (Click View > Small Icons) > Mail (Outlook 20xx) > Show Profiles > Add) and you'll be able to setup a new autodiscover profile without destroying your existing one.
6. We are preparing to roll out swing mailboxes for the few mailboxes that we still have not been able to attach.
7. Staff will be spending most of the day following up. If you can, please keep all ticket updates on the same ticket, we are going through issues domain by domain.
8. Please use OWA if possible, please use NGE as possible.
Next update will follow later today but
Update 06/04/2020 07:06 AM 16 days
3AM Update:
We are continuing to work with our partners and clients on getting users to the modern Exchange standards. Microsoft no longer supports "manual" server configurations as of Outlook/Exchange 2013 so if you do not have an autodiscover record you need to create one immediately (simply create an autodiscover CNAME record in your domain with the value "autodiscover.xd.email"). We know that many of our partners and clients aren't highly skilled in obscure Microsoft Exchange settings and configurations so we've written up a quick guide that just about anyone with Admin access to their PC or mobile device can follow:
https://www.exchangedefender.com/media/ExchangeDefenderExchange.docx
Please feel free to distribute the document or customize/brand it to your liking. The process is very simple, as long as you can login to Outlook Web App (https://owa.xd.email/owa/) those same credentials will get you into the new version of Exchange.
If you have autodiscover configured already (you should, without it even 2013 and earlier barely worked consistently) your Outlook will prompt you for your password. You may get an error or warning and within a few minutes a screen prompting you to accept new settings from "outlook.xd.email" will show up. Click on accept, Outlook will open your mailbox and everything is done. If you experience a problem in this stage, start Outlook in debug mode (outlook /rpcdiag) and observe the servers it connects to - if it's Office365 you will need to follow the same steps in the guide for running Outlook Registry Tools.
In terms of backend work, we are still helping partners mainly with authentication and mail flow problems. We still have a few users that are failing auth because of all we've discussed before, we're confident these last few changes will make things move smoother.
The most difficult issue for us remains communication and ability to update so many clients and partners, and troubleshoot smaller issues with individual mailboxes and organizations. We remain confident that even though we've been dealt this setback, we will have everyone on the new platform and it is well worth it. Next update will be provided at 9am.
Update 06/03/2020 20:02 PM 17 days
4PM Update
We are continuing to see the NGE environment normalize and people return to work. There are still some issues that we're trying to get to the bottom of (already covered in previous NOC posts, Outlook/autodiscover/password resets just not being as consistent; we have been troubleshooting these on the backend). For the most part, everything is working - and we have a number of users that are still not online that we're going above and beyond to restore the service where Microsoft tech is simply incapable of doing so (see below under IMPORTANT - PLEASE READ if you're still down)
The biggest issue at the moment is on the staffing and technology side. None of us have left since Sunday and our original migration and transition plan had to be compressed down and executed quickly. Our support is simply not capable of handling tens of thousands of tickets so we have been consolidating them per MSP and per domain and trying to get as many up and running as fast possible (see TECH DETAILS section) but some clients are understandably frustrated with the outage and creating dozens of tickets which is only slowing us down. Requests for a callback, to recite what we have in these NOC updates, is also slowing us down. We are on your team, we have your best interest in heart and it does not help us at all if your clients leave you. We take our responsiblity to our clients seriously and are doing everything we can to get everyone up and running - but we're overwhelmed and exausted and beaten and I'm sure everyone in IT can relate to that. We're working on it. Below is a brief technical summary for the outstanding clients that are down, an explanation, and a workaround.
TECHNICAL DETAILS - IF YOU ARE STILL NOT ABLE TO CONNECT
The remaining mailboxes that are still not online fall into two technical categories.
1) They have moved/moving and the O365 side has failed to reconnect the mailbox to the domain or we are attempting to move a mailbox and it errors out and fails after it reaches a failed item treshhold. The reconnection process and seeding process fails and we go through a manual attempt through PowerShell to map the mailbox to it's proper location so Outlook/OWA/EWS(mobile) can connect to it. There are tons of different technical issues here (from not meeting basic password complexity, one user had P@ssw0rd as a password - to bad X400 address / primary address mismatch / etc). Rinse, then repeat, over 700 times since Sunday. We're down to only a couple that are in this stage.
2) They are still in process of moving - These are mailboxes that were never flagged for move because they were renamed, had weird / circular permissions or other misconfiguration, OR, they got disconnected from the move when the source was disconnected. For these, we are moving them to production but some users have 30-40GB mailboxes and the process of migrating is slow. Once they are moved, we go back to step #1 and confirm that they can login. Then someone in support has to dig up the correct ticket and update it.
We are currently working with partners over the phone, over our NOC, over Facebook video/chat, over txt, and we're doing everything in our power to connect the remaining accounts. This is brutal and difficult and we understand how bad it is - and we are here working for you and trying everything we can to get everyone online as fast as possible. This whole episode was not our plan but Microsoft pulled the rug from under us and we're doing insane stuff just to get everyone working. See below:
IMPORTANT - PLEASE READ - IF YOU'RE STILL DOWN AND WANT AN IMMEDIATE WORKAROUND
If you do not want to rely on LiveArchive or NGE, or wait for the mailbox to be repaired/restored/reconnected/etc we can bring you to operations quickly - but fair disclosure, this is ugly yet functional. The following process can be executed by our NOC to create a temporary mailbox that we can then merge together when the original mailbox is ready.
Create a new mailbox with a unique email address swing.USERNAME@domain.com. On the edge transport servers we will create a rewrite rule which says mail from the swing address gets rewrote to the real address and mail to the real address is rewrote to the swing address.
The user will have the same display name with an extra space so Vlad Mazek => Vlad Mazek
We create a transport rule for intra company mail to redirect to the swing user.
At this point, all their mail flow is working fine and they can work out of OWA.
After completion
Once the migration is completed, they’ll be closed out of their swing mailbox (or we can give it full access permission to the real mailbox so it automagically shows up in owa/outlook). The real account is then active for login and we disable the rewrite rules and import the data from the swing mailbox to the real mailbox.
Update 06/03/2020 14:31 PM 17 days
With almost all mailboxes reconnected, we are now addressing mailboxes that are in a failed/disabled state. These mailboxes are disabled for a variety of reasons (account or domain mismatch, account deleted, not in the right security group, etc) and we're enabling them, clearing any errors, reconnecting.
We already know which mailboxes are in this state and we're working through the list, in the meantime please stay in LiveArchive if your mailbox is affected by this and we will update the existing tickets as we go along.
I know many of our partners are asking for an update and specifics to pass to their clients and we'll offer a detailed writeup by noon on www.exchangedefender.com/blog
Update 06/03/2020 08:33 AM 17 days
4:30 AM EST - The mailbox process is nearly complete, we have resolved a number of reported issues overnight as well. Our next step is to deliver PF and catchall mailbox access.
Update 06/02/2020 21:38 PM 18 days
We are at the tail end of getting everyone off the legacy platform, and we have restored access and mail flow to just about everyone. The last few mailboxes are exceptionally difficult and we are doing everything in our power to get them completed before midnight, but a few may slip. Here is a summary of work and issues we've resolved throughout the day in case you expereinced any of these problems we hope you can check again:
1. LiveArchive NGE DKIM signature - our infrastructure was not signing DKIM messages, in interest of expediency we've routed outbound mail via ExchangeDefender instead of implementing DKIM on the platform for all of our client domains.
2. Occasional 504 gateway timeouts - these typically happen when our backend docker services do not respond in time to the frontend proxy. We've bumped the resources and timeout settings there throughout the day and it's performing well considering exceptionally high usage.
3. "Report Issues" button in Service Manager - several of you have found it, it's something we've been trying to sort out throught the day so we can avoid doing individual service requests through support tickets. We can automate a lot of the discovery and analysis through this and get users problems autofixed going forward. Please note that this is not going to help mailboxes that aren't already mounted, this is more for the ongoing support.
4. Active Directory resiliance - we are adding more domain controllers to the mix. These will not go online till midnight for our clusters located in USA, and until tomorrow for EMEA/AUS.
5. We are sorry about this, we understand the frustration and we're doing all we can as fast as we can to get everyone rolling in the right direction. The rug was simply pulled out from underneath us with no warning and our actual migration checkup plan that was supposed to last 2 weeks on 6/15 had to be executed in 1 day.
Next update will follow at 9:30 PM EST.
Update 06/02/2020 16:37 PM 18 days
Noon update: We are in the final stages of reconnecting and disconnecting all the affected mailboxes (some larger mailboxes are requiring a reseed so if you haven't gotten your users and mail enabled public folders onto LiveArchive, this update is specifically for you).
https://www.exchangedefender.com/blog/2020/06/update-exchange-migration/
We have also addressed an issue where catchall mailboxes (catching/caching mail for inbound) are accepting messages but still issuing an NDR. Followup post will explain how to access that mail shortly.
We are optimistic about the current progress and believe we've addressed all the unexpected issues that have come up since Sunday. We will keep you up to date on our progress on the NOC site at www.anythingdown.com.
Update 06/02/2020 13:04 PM 18 days
9AM
We have largely restored access to practically every mailbox that had a simple fix and are now making additional passes over mailboxes and organizations that had custom configurations or other errors preventing them from reconnecting manually. The process of updating all tickets, clients, and mailbox configs is manual and time consuming so we've been working on a few things that will be announced ~10AM EST today.
1. We are prioritizing getting LiveArchive NGE (launched last month) access for users that are still affected. We will have a quick walkthrough on how to create accounts for public folders/etc so at least people that are still affected can get email.
2. The manual reconcile process is moving forward.
3. No additional issues have been reported, which is the only good news to report since Sunday, and our goal is to make sure EVERYONE can quickly recover
4. All tickets are being consolidated per company so we can keep you up to date better and we're discussing our update policy. We're all in the same boat here, while we are working as fast as possible and realize that everyone has a critical issue, everyone is upset or livid (as are we) --but we can either work on getting you online or we can deal with "ETA", "What do I tell my client", "This is still not working" ticket updates that are just putting all of us at a disadvantage. Priority is getting people email, there will be plenty of time to make everything right and reconcile things.
5. New UI and LiveArchive tweaks are being put in to allow people to manually address LiveArchive login/etc separate from Microsoft Exchange, so that mailboxes that are still affected and so that users that live in Public Folders or have mail flow interrupted for whatever reason, can continue to work.
We thank you for your patience and are working as hard and as fast as possible.
Update 06/02/2020 04:11 AM 18 days
Midnight update:
We are still going through cases and repairing/activating in the debug queue. More accounts are reconnected and verified, we are working through the queue. If you've reported the issue, or if we identified an issue with an account, we are working on it (and tickets will not be updated until either there is a new NOC update with our progress or we have verified we can login and manage all accounts in the domain).
We hope to have all issues (except Public Folders) addressed shortly, we will have a new process for reporting outstanding issues in the AM for users that are still reporting problems.
Update 06/02/2020 00:10 AM 19 days
Update 8PM EST:
We are currently working through a project queue that our partners and support team setup throughout the day to address any outstanding accounts that are having login issues. We are prioritizing authentication requests at this time and will be working through the night as long as it takes to get everyone into their Inbox. At the moment, the second biggest issue we are working on is related to reattaching Public Folder infrastructure to some organizations. We will be updating support requests as we reconnect entire organizations, and will kickstart PF reconnection afterwards.
More details: Remaining mailboxes that are currently inaccessible and password reset does not resolve the issue did not successfully move to the new organization and disconnect from the source (problem causes involve Active Directory errors, Exchange errors, password complexity) and the recovery process is to simply check the error, make required adjustments, and attempt to reconnect the mailbox. Because each issue is separate and caused by different factors, this is a slow manual process. We are working on it and anticipate everyone will be reconnected/restored tonight. There are several mailboxes in the move that were never picked up by the Migration Engine and we're manually moving those accounts too but some of the larger mailboxes are taking time, thankfully there are rather few of them and we have tons of resouces.
We will keep you up to date throughout the night and we are very sorry about the inconvenience this has caused our users. We're doing our best to get everyone up to speed and we realize that everyone has an urgent need to get back to their Outlook, we are here for you. In the meantime, we hope you can continue to work out of LiveArchive at https://nge.exchangedefender.com as we get this up and running smoothly, the level of complexity and issues with the legacy Exchange platforms has been significant and we have planned this process for months and executed countless test runs with very few issues. We realize that it doesn't matter how thought through and designed the process is if you can't login to your mailbox right now and we're continuing to work on this.
Update 06/01/2020 19:43 PM 19 days
Update as of 3:30 PM EST:
All but ~200 mailboxes have been restored, mail flow has been showing as nominal since it was activated this morning. The mailboxes still in manual debug are on organizations that have changed UPN or had some more complex settings that we are reconciling and activating manually.
1. The biggest update since the last NOC is that all organizations have been moved to our next gen Service Manager. If you see your organization in there you should be all set and all functionality (password changes, etc) is functional. ***IF*** you changed a password during any time and were not able to login to our new https://owa.xd.email/owa/ site, the password you set is likely in queue waiting to be applied to the mailbox. Once you can confirm you're good with OWA, Outlook should automatically offer to repair a profile and restart in the new one (no mail download, everything just starts working)
2. We are still working on reconnecting some PF for several organizations, this is done automatically on the backend.
3. If you experience this issue with Outlook 2010:
Microsoft Outlook: There is a problem with the proxy server's security certificate.
Outlook is unable to connect to the proxy server cas.darkwing.exchangedefender.com. (Error Code 8000000)
Start > Control Panel > Mail > Show Profiles > Properties > Click on Repair
That will force Outlook to go out, check with Autodiscover, download the certificate, and apply it correctly.
This only works on Outlook 2010. After that, the profiles automatically repair/redownload.
Update 06/01/2020 16:41 PM 19 days
Update 06/01/2020 15:07 PM 19 days
Mail flow to all our Exchange clients has been restored, along with distribution groups, shared mailboxes, etc. We have a few more tasks but all our Exchange clients should be in production.
We have received reports from some users that their passwords aren't working in Outlook. If you are experiencing this problem, please update the ticket with the list of email accounts (UPN) that are having an issue and we will start troubleshooting them. For the time being, if you are experiencing login issues please keep users in LiveArchive at https://nge.exchangedefender.com and OWA at https://owa.xd.email/owa/ (the trailing slash at the end is required, just typing in owa.xd.email will not work for now)
P.S. NO configuration (DNS/etc) change is required, this process is largely transparent. At most, you will have to click on OK/Rebuild/Restart when Outlook attempts to open the mailbox on the new infrastructure.
Update 06/01/2020 13:22 PM 19 days
We are in the final stage of the checkdown, reviewing the Public Folder and Distribution group transition. We're working as quickly as possible and look forward to restoring Outlook service momentarily, please keep your users in LiveArchive for the time being (it works on mobile phones too)
Update 06/01/2020 11:14 AM 19 days
As of 7AM the migration has been completed and we are nearing the end of all the error checks with the hope of restoring mail flow and Outlook connectivity shortly. We are still working on a few items (public folders, changed UPN accounts).
Update 05/31/2020 23:12 PM 20 days
We are currently in the final stage of finalizing moves from our Exchange legacy platforms/clusters to the new 365/2016/2019 SKU. The work is expected to be completed before midnight, May 31st, 2020. While we do not anticipate any issues and have tested everything thorouhgly, things in IT (and legacy apps) can cause problems and we've stepped up our staffing to help our partners and clients work through them all.
In the meantime, if your mail flow or access to Outlook gets interrupted, you are not down. There are two failover systems at ExchangeDefender you can rely on to continue working:
ExchangeDefender LiveArchive: https://nge.exchangedefender.com
Outlook Web App: https://owa.xd.email/owa
We will keep you updated on our progress here at www.AnythingDown.com and at https://support.ownwebnow.com