Configuration corruption

This is the Mailtraq Peer Support forum. Get assistance using and managing Mailtraq, and help others solve problems too.

Configuration corruption

Postby Tim Walter » Tue Jan 07th, 2014 11:13am

Mailtraq Version: 2.17.4.3256

Hi, Over the last 1-2 months my config repeatedly corrupts, such that I can't log in. Restoring from online backup sorts it each time. The main evidence is that User/Properties/Owner changes from "Tim" to User1 or User 2 etc.

I suspect it is something else running on the server (WIndows 2003 R2) that is mangling things but not sure quite where to look.

I know I am on an old version of Mailtraq and happy to upgrade if it will sort it but suspect the issue lies elsewhere. Not expecting a fix, but more a pointer of how to investigate/where to look

Thanks
Tim Walter
 
Posts: 97
Joined: Mon Sep 04th, 2006 6:20pm

Re: Configuration corruption

Postby Elric Pedder » Tue Jan 07th, 2014 3:38pm

There have been updates since the build you have that improve reliability with the configuration files but I can't be sure exactly what would be causing the problem you are seeing. Does this match any hardware events that you can think of such as shutdowns?
Mailtraq Development and Escalation Support
Novitraq Incorporated
User avatar
Elric Pedder
Mailtraq Escalation Support
 
Posts: 2675
Joined: Tue Nov 23rd, 2004 1:16pm
Location: Montreal, Canada

Re: Configuration corruption

Postby Martin Clayton » Wed Jan 08th, 2014 8:34am

Tim Walter wrote:Over the last 1-2 months my config repeatedly corrupts, such that I can't log in. Restoring from online backup sorts it each time. The main evidence is that User/Properties/Owner changes from "Tim" to User1 or User 2 etc.

I've been seeing similar for a long time, probably first reported here. With .3294 multi-domain things might look a bit different but I'd search the logs for:

  • Configuration Change (System): Detected, reloading...
  • Configuration Change (User:
  • Mailslot * owner changed from * to *
  • configuration: file does not yet exist

...if this event is anything to go by:

Code: Select all
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (System): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user1@subdomain2): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user1@subdomain2): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user2@subdomain2): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user2@subdomain2): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user3@subdomain1): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user3@subdomain1): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user4@subdomain1): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user4@subdomain1): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user5@subdomain3): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user5@subdomain3): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user7@subdomain1): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user7@subdomain1@subdomain1): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user0): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user0): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user8@subdomain1): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user8@subdomain1): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user9): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (User: user9): Reload complete
00000400 0000021C 27/10/2013 01:00:11 Mailslot user2 owner changed from 126237152 to 0
00000400 0000021C 27/10/2013 01:00:11 Mailslot user3 owner changed from 119094374 to 0
00000400 0000021C 27/10/2013 01:00:11 Mailslot user4 owner changed from 228903555 to 0
00000400 0000021C 27/10/2013 01:00:11 Mailslot user5 owner changed from 249861453 to 0
00000400 0000021C 27/10/2013 01:00:11 Mailslot user6 owner changed from 249861453 to 0
00000400 0000021C 27/10/2013 01:00:11 Mailslot user7 owner changed from 156678721 to 0
00000400 0000021C 27/10/2013 01:00:11 Mailslot user8 owner changed from 254758832 to 0
00000400 0000021C 27/10/2013 01:00:11 Mailslot non-primary4 owner changed from 254758832 to 0
00000400 0000021C 27/10/2013 01:00:11 Mailslot non-primary5 owner changed from 254758832 to 0
00000400 0000021C 27/10/2013 01:00:11 Mailslot non-primary6 owner changed from 254758832 to 0
00000400 0000021C 27/10/2013 01:00:11 Mailslot non-primary7 owner changed from 254758832 to 0
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (System): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user1): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user1): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user2): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user2): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user3): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user3): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user4): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user4): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary1): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary1): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary2): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary2): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user5): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user5): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user6): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Loading mailslot (user6) configuration: file does not yet exist
00008000 0000021C 27/10/2013 01:00:11 Loading mailslot (user6) configuration: file data missing (83/)
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user6): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary3): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary3): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: mailserver): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: mailserver): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user7): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user7): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user0): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user0): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user8): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: user8): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary4): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary4): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary5): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary5): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary6): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary6): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary7): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary7): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary8): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary8): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary9): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary9): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary10): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary10): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: newsserver): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: newsserver): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: postmaster): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: postmaster): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary11): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary11): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary12): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary12): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary13): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary13): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary14): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary14): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary15): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary15): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary16): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary16): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary17): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary17): Reload complete
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary18): Detected, reloading...
00008000 0000021C 27/10/2013 01:00:11 Configuration Change (Mailslot: non-primary18): Reload complete
User avatar
Martin Clayton
Expert User
 
Posts: 529
Joined: Sat Jan 15th, 2005 8:20am
Location: London, UK

Re: Configuration corruption

Postby Elric Pedder » Fri Jan 17th, 2014 11:13am

While the root cause of the problem remains unknown the reason this occurs is because Mailtraq reads the configuration file from disk and receives an early end-of-file from Windows. I've suspected that some sort of process in Windows occasionally blocks access, perhaps to scan the file, but it is so rare it can't be reproduced. Because Mailtraq reads the configuration file often, even an incredibly rare event can cause problems. In a more recent build we have markers at the end of the file and if Mailtraq doesn't see the marker it won't accept the data and this has addressed the problem of reloads while running but it can still happen on initial startup. That case is of course even more rare but it may be related to the high load a server machine experiences during the startup stage. Mailtraq retries the file loading so it is hard to say what might trigger it. Customers we've spoken to have often found the resolution to be reconfiguring anti-virus or backup software.
Mailtraq Development and Escalation Support
Novitraq Incorporated
User avatar
Elric Pedder
Mailtraq Escalation Support
 
Posts: 2675
Joined: Tue Nov 23rd, 2004 1:16pm
Location: Montreal, Canada

Re: Configuration corruption

Postby Tim Walter » Tue Jan 21st, 2014 3:27pm

Following on from this I have disabled Volume Shadow Copy on the server and so far it has not corrupted again since then, so early days but looks like it may have been the issue (so far so good)

Cheers
Tim Walter
 
Posts: 97
Joined: Mon Sep 04th, 2006 6:20pm

Re: Configuration corruption

Postby Tim Walter » Sat Jan 25th, 2014 3:45am

Yep, spoke too soon. Same thing Thursday night.

Noticed in Windows Event log that many times around 1 min past midnight mailtraq service repeatedly stops and starts, though can't see any concurrent activity elsewhere... (And although I've tried adding periodic auto-restart to Mailtraq just now, it hasn't been enabled til now)

FWIW this is the detail of the event log (though not sure it says anything to me!!!)
BW TIm

Code: Select all
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
  <EventID Qualifiers="16384">7036</EventID>
  <Version>0</Version>
  <Level>4</Level>
  <Task>0</Task>
  <Opcode>0</Opcode>
  <Keywords>0x8080000000000000</Keywords>
  <TimeCreated SystemTime="2014-01-25T00:38:11.613738100Z" />
  <EventRecordID>248230</EventRecordID>
  <Correlation />
  <Execution ProcessID="488" ThreadID="3104" />
  <Channel>System</Channel>
  <Computer>Server</Computer>
  <Security />
  </System>
- <EventData>
  <Data Name="param1">Mailtraq</Data>
  <Data Name="param2">stopped</Data>
  <Binary>4D00610069006C0074007200610071005300650072007600650072002F0031000000</Binary>
  </EventData>
  </Event>

Tim Walter
 
Posts: 97
Joined: Mon Sep 04th, 2006 6:20pm

Re: Configuration corruption

Postby Elric Pedder » Mon Jan 27th, 2014 11:24am

Is anything scheduled to happen at midnight?
Mailtraq Development and Escalation Support
Novitraq Incorporated
User avatar
Elric Pedder
Mailtraq Escalation Support
 
Posts: 2675
Joined: Tue Nov 23rd, 2004 1:16pm
Location: Montreal, Canada

Re: Configuration corruption

Postby Martin Clayton » Mon Jan 27th, 2014 12:04pm

Tim, I hope I'm not pulling your topic off course but looking at the multi-domain logs for "Configuration Change" there are 10 or so events which sometimes cascade into "changed owner" events. Single user instances seem to happen at random times of day whereas the larger events look more scheduled:

Code: Select all
28/10/2012 01:00:21 Configuration Change (System)
11/01/2013 08:07:04 Configuration Change (User) x1
15/02/2013 13:50:24 Configuration Change (User) x1
31/03/2013 02:00:51 Configuration Change (User) x9
29/04/2013 17:02:30 Configuration Change (User) x1
02/06/2013 13:09:02 Configuration Change (User) x1 (domain corruption)
22/07/2013 16:38:02 Configuration Change (User) x1 (domain corruption)
31/07/2013 16:53:00 Configuration Change (User) x1 (domain corruption)
27/08/2013 19:14:36 Configuration Change (User) x1 (domain corruption)
27/10/2013 01:00:11 Configuration Change (System)

All of the above is under Win2k SP4.

Scheduling wise, every day there's:
  • 00:17 Jim's perl log analysis script via Windows Task Scheduler
  • 02:00-05:00 Mtq maintenance period (02:00-07:00 at weekends)
  • 02:50 xcopy Mtq config files
  • 05:05 E: drive defrag (Mtq logs) via Task Scheduler
  • 05:06 Mtq online backup
plus monthly
  • 17:41 26th of month F: drive defrag (Mtq backups)
There's no av other than the Mailtraq managed clamd.exe
freshclam runs as a service checking every 30 minutes.
Simple DNS Plus and TightVNC server are also active.
I'd forgotten about this one but there was an ntp daemon which was synching the system clock with *.uk.pool.ntp.org every 30 minutes (now uninstalled).
User avatar
Martin Clayton
Expert User
 
Posts: 529
Joined: Sat Jan 15th, 2005 8:20am
Location: London, UK

Re: Configuration corruption

Postby Elric Pedder » Tue Jan 28th, 2014 1:19pm

Martin Clayton wrote:...if this event is anything to go by:

Is it possible to send me the original event log covering this period?
Mailtraq Development and Escalation Support
Novitraq Incorporated
User avatar
Elric Pedder
Mailtraq Escalation Support
 
Posts: 2675
Joined: Tue Nov 23rd, 2004 1:16pm
Location: Montreal, Canada

Re: Configuration corruption

Postby Martin Clayton » Tue Jan 28th, 2014 1:56pm

Elric Pedder wrote:Is it possible to send me the original event log covering this period?

Done!
User avatar
Martin Clayton
Expert User
 
Posts: 529
Joined: Sat Jan 15th, 2005 8:20am
Location: London, UK


Return to Mailtraq Support

Who is online

Users browsing this forum: No registered users and 3 guests