Host freeze-up issues
The MindTouch host sometimes ceases to respond - this is related to increasing memory usage by Mono (due to their faulty GC). The workaround for this issue is to install and run the checkdeki script - it will track memory usage and restart the host once a certain amount of memory has been consumed.
Service startup errors
Bug
reference to undefined name 'mantis' Exception of type 'MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException' was thrown. (click for details)Callstack:
at User:PeteE/MDC_issues
MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException: reference to undefined name 'mantis' Exception of type 'MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException' was thrown.
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptVar expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Expr.DekiScriptVar.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Evaluate (MindTouch.Deki.Script.Expr.DekiScriptAccess expr, DekiScriptExpressionEvaluationState state, Boolean evaluateProperties) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptAccess expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Expr.DekiScriptAccess.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptCall expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Expr.DekiScriptCall.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptReturnScope expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
Target Release: 9.08.1 (fixed)
When services are starting up and the server is under load, we often get the following exception:
2009-05-28 23:10:01,852 [1] ERROR MindTouch.Dream.DreamFeatureChain - handler for POST:local://c0c72c6af32646efa304048b040898e9/host/services failed (MindTouch.Dream.DreamHostService!PostServices)()
System.IndexOutOfRangeException: Array index is out of range.
at System.Xml.NameTable.Add (System.String key) [0x00000]
at System.Xml.XmlDocument.AddDefaultNameTableKeys () [0x00000]
at System.Xml.XmlDocument..ctor (System.Xml.XmlImplementation impl, System.Xml.XmlNameTable nt) [0x00000]
at System.Xml.XmlDocument..ctor (System.Xml.XmlNameTable nt) [0x00000]
at MindTouch.Dream.XDoc.NewXmlDocument () [0x00000]
at MindTouch.Dream.XDoc..ctor (System.String prefix, System.String tag, System.String ns) [0x00000]
at MindTouch.Dream.XDoc..ctor (System.String tag, System.String ns) [0x00000]
at MindTouch.Dream.XDoc..ctor (System.String tag) [0x00000]
at MindTouch.Dream.DreamCookie.get_AsSetCookieDocument () [0x00000]
at MindTouch.Dream.DreamHostService+<StartService>d__4f.MoveNext () [0x00000]
at MindTouch.Dream.Coroutine.Iterate_Helper (IEnumerator`1 coroutine, MindTouch.Dream.TaskBehavior behavior) [0x00000]
at System.VoidClosure`2[System.Collections.Generic.IEnumerator`1[MindTouch.Dream.IYield],MindTouch.Dream.TaskBehavior].Closure () [0x00000]
at MindTouch.Dream.Task.Execute (System.VoidHandler handler, MindTouch.Dream.TaskBehavior behavior) [0x00000]
2009-05-28 23:10:01,899 [2] ERROR DekiWikiService_developer.mozilla.org - StartService(Unable to start local service id '10' with SID 'sid://mindtouch.com/2007/12/dekiscript' Error: 'unable to initialize service (Array index is out of range.)')
MindTouch.Dream.ResultAsyncException: async exception thrown ---> System.Exception: unable to initialize service (Array index is out of range.)
at MindTouch.Dream.DreamService+<CreateServiceAsync_Helper>d__19.MoveNext () [0x00000]
at MindTouch.Dream.Coroutine.Iterate_Helper (IEnumerator`1 coroutine, MindTouch.Dream.TaskBehavior behavior) [0x00000]
at System.VoidClosure`2[System.Collections.Generic.IEnumerator`1[MindTouch.Dream.IYield],MindTouch.Dream.TaskBehavior].Closure () [0x00000]
at MindTouch.Dream.Task.Execute (System.VoidHandler handler, MindTouch.Dream.TaskBehavior behavior) [0x00000] --- End of inner exception stack trace ---
at MindTouch.Dream.ResultBase.Confirm () [0x00000]
at MindTouch.Dream.ResultBase.Wait () [0x00000]
at MindTouch.Dream.Result`1[MindTouch.Dream.Plug].Wait () [0x00000]
at MindTouch.Dream.DreamService.CreateService (System.String path, System.String sid, MindTouch.Dream.XDoc config) [0x00000]
at MindTouch.Deki.Logic.ServiceBL.StartService (MindTouch.Deki.ServiceBE service, Boolean forceRefresh, Boolean disableOnFailure, Boolean dekiStartup) [0x00000]
This appears to be due to the fact that XmlNameTable is not thread safe.
Pubsub Initialization
Bug
reference to undefined name 'mantis' Exception of type 'MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException' was thrown. (click for details)Callstack:
at User:PeteE/MDC_issues
MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException: reference to undefined name 'mantis' Exception of type 'MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException' was thrown.
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptVar expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Expr.DekiScriptVar.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Evaluate (MindTouch.Deki.Script.Expr.DekiScriptAccess expr, DekiScriptExpressionEvaluationState state, Boolean evaluateProperties) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptAccess expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Expr.DekiScriptAccess.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptCall expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Expr.DekiScriptCall.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptReturnScope expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
Target Release: Noatak
When MindTouch starts up (again under heavy load) in some cases the pubsub service isn't wired up to lucene properly. According to Arne, it appears that DekiPubSub sends the indexing event to the main PubSub bus, but the main PubSub bus is not able to dispatch the event to the LuceneService. This causes all indexing operations to fail.
petee 21 October 2009 We are working to reproduce this race condition on the MDC dev server and will have a fix in Noatak (possibly a hotfix for 9.08.1 if we can identify the race condition in time).
Index rebuild thread dying
Bug
reference to undefined name 'mantis' Exception of type 'MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException' was thrown. (click for details)Callstack:
at User:PeteE/MDC_issues
MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException: reference to undefined name 'mantis' Exception of type 'MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException' was thrown.
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptVar expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Expr.DekiScriptVar.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Evaluate (MindTouch.Deki.Script.Expr.DekiScriptAccess expr, DekiScriptExpressionEvaluationState state, Boolean evaluateProperties) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptAccess expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Expr.DekiScriptAccess.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptCall expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Expr.DekiScriptCall.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptReturnScope expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
Target Release: 9.08.1
When attempting to rebuild the lucene index on a large wiki, the indexer thread seems to disappear without throwing any kind of exception. This makes the rebuild process fail.
petee 21 Oct 2009 This can be avoided by switching to the legacy threadpool
arnec identified another possible cause for the index thread dying. If an exception occurs during the queuing operation (XmlNameTable exception for example) and is not caught, it will kill the background thread silently. The fix is twofold 1) fix xmlnametable (done) 2) wrap that code block in a try/catch so one failure doesn't kill the thread.
Errors when Lucene on NFS
Bug
reference to undefined name 'mantis' Exception of type 'MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException' was thrown. (click for details)Callstack:
at User:PeteE/MDC_issues
MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException: reference to undefined name 'mantis' Exception of type 'MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException' was thrown.
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptVar expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Expr.DekiScriptVar.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Evaluate (MindTouch.Deki.Script.Expr.DekiScriptAccess expr, DekiScriptExpressionEvaluationState state, Boolean evaluateProperties) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptAccess expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Expr.DekiScriptAccess.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptCall expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Expr.DekiScriptCall.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptReturnScope expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0
Target Release: Noatak
If the lucene index is being shared via an NFS mount, a read operation (search) on one server may fail while a write operations is taking place on another server. This happens because the file that the read operations is attempting to read is no longer available due to the write operation merging segments in the index into a different file. This generates an IOException (unable to access file). This is less likely to occur during "normal" conditions when the index isn't being written to at such a high rate (as it is during index rebuild).
The solution to this is to use a single "master" lucene service which each server sends indexing requests to. This would eliminate the locking problems inherent with NFS.
