Silo can't start in docker #4680
Description
I have strange error that can be reproduced in docker container only. When I run my application as regular application(not in the docker) all works as expected.
What is interesting that the silo terminating in case when there is a startup task and orphan records in OrleansMembershipTable table, I use ado.net membership.
If I remove orphan records from OrleansMembershipTable table or comment my startup task, silo able to start and works.
Startup task activate a grain with reminder and timer for several background staff.
Below the full log of orleans from docker.
I've skipped several sensitive info from the log
I use .NET 462, Orleans 2.0.3
Full Orleans Log
host.server_1 | [12:45:10 WRN] No implementation of IHostEnvironmentStatistics was found. Load shedding will not work yet host.server_1 | [12:45:10 INF] Starting OrleansTaskScheduler with 4 Max Active application Threads and 1 system thread. host.server_1 | [12:45:10 INF] Silo starting with GC settings: ServerGC=True GCLatencyMode=Interactive host.server_1 | [12:45:10 INF] -------------- Initializing silo on host 5a5cb449241b MachineName 5A5CB449241B at 172.26.83.96:11111, gen 266492710 -------------- host.server_1 | [12:45:10 INF] Starting silo Silo_d97f1 host.server_1 | [12:45:10 INF] Opened a listening socket at address 0.0.0.0:11111 host.server_1 | [12:45:10 INF] Opened a listening socket at address 0.0.0.0:30000 host.server_1 | [12:45:10 INF] Starting VirtualBucketsRingProvider on silo S172.26.83.96:11111:266492710/x28220225. host.server_1 | [12:45:10 INF] Added Server S172.26.83.96:11111:266492710/x28220225. Current view: [S172.26.83.96:11111:266492710 -> ] host.server_1 | [12:45:10 INF] -NotifyLocalRangeSubscribers about old <(0 0], Size=x100000000, %Ring=100.000%> new increased? True host.server_1 | [12:45:11 INF] Skip multicluster oracle creation (no multicluster network configured) host.server_1 | [12:45:11 INF] -------------- Started silo S172.26.83.96:11111:266492710, ConsistentHashCode 28220225 -------------- host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.Scheduler.WorkerPoolThread/System.5 on managed thread 7 host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.Scheduler.WorkerPoolThread/0 on managed thread 8 host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.Scheduler.WorkerPoolThread/1 on managed thread 9 host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.Scheduler.WorkerPoolThread/2 on managed thread 10 host.server_1 | [12:45:11 INF] Start Scheduler took 11 Milliseconds to finish host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.Scheduler.WorkerPoolThread/3 on managed thread 11 host.server_1 | [12:45:11 INF] Stage 2000: Orleans.Runtime.InsideRuntimeClient, Orleans.Runtime.Silo host.server_1 | [12:45:11 INF] Stage 4000: Orleans.Runtime.Silo host.server_1 | [12:45:11 INF] Stage 8000: Orleans.Runtime.Silo host.server_1 | [12:45:11 INF] Stage 10000: Orleans.Runtime.Versions.GrainVersionStore, Orleans.Storage.AdoNetGrainStorage-Default, Orleans.Storage.AdoNetGrainStorage-OrleansStorage host.server_1 | [12:45:11 INF] Stage 19999: Orleans.Runtime.Silo host.server_1 | [12:45:11 INF] Stage 20000: Orleans.Hosting.SiloBuilderStartupExtensions+StartupTask, Orleans.Hosting.SiloBuilderStartupExtensions+StartupTask, Orleans.Hosting.SiloBuilderStartupExtensions+StartupTask, Orleans.Runtime.Silo host.server_1 | [12:45:11 INF] Start InsideRuntimeClient took 9 Milliseconds host.server_1 | [12:45:11 INF] Lifecycle observer Orleans.Runtime.InsideRuntimeClient started in stage 2000 which took 10 Milliseconds. host.server_1 | [12:45:11 INF] Silo Start() host.server_1 | [12:45:11 INF] Configured ThreadPool.SetMinThreads() to values: 200,200. Previous values are: 2,2. host.server_1 | [12:45:11 INF] Configured .NET ServicePointManager to Expect100Continue=False, DefaultConnectionLimit=200, UseNagleAlgorithm=False to improve Azure storage performance. host.server_1 | [12:45:11 INF] ConfigureThreadPoolAndServicePointSettings took 4 Milliseconds to finish host.server_1 | [12:45:11 INF] Lifecycle observer Orleans.Runtime.Silo started in stage 2000 which took 10 Milliseconds. host.server_1 | [12:45:11 INF] Starting lifecycle stage 2000 took 24 Milliseconds host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.Messaging.IncomingMessageAcceptor on managed thread 12 host.server_1 | [12:45:11 INF] Stopping AsyncAgent Runtime.Messaging.IncomingMessageAcceptor that runs on managed thread 12 host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.Messaging.SiloMessageSender/PingSender on managed thread 13 host.server_1 | [12:45:11 INF] Start Message center took 3 Milliseconds to finish host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.Messaging.SiloMessageSender/SystemSender on managed thread 15 host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.Messaging.IncomingMessageAgent/Ping on managed thread 16 host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.Messaging.IncomingMessageAgent/System on managed thread 17 host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.Messaging.IncomingMessageAgent/Application on managed thread 18 host.server_1 | [12:45:11 INF] Start Incoming message agents took 2 Milliseconds to finish host.server_1 | [12:45:11 INF] Start host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer`1 on managed thread 19 host.server_1 | [12:45:11 INF] Starting AsyncAgent Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer on managed thread 20 host.server_1 | [12:45:11 INF] Stopping AsyncAgent Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer that runs on managed thread 20 host.server_1 | [12:45:11 INF] Start local grain directory took 7 Milliseconds to finish host.server_1 | [12:45:11 INF] Init implicit stream subscribe table took 5 Milliseconds to finish host.server_1 | [12:45:11 INF] Creating reminder grain service for type=Orleans.Runtime.ReminderService.AdoNetReminderTable host.server_1 | [12:45:11 INF] Create system targets and inject dependencies took 42 Milliseconds to finish host.server_1 | [12:45:11 INF] Lifecycle observer Orleans.Runtime.Silo started in stage 4000 which took 65 Milliseconds. host.server_1 | [12:45:11 INF] Starting lifecycle stage 4000 took 65 Milliseconds host.server_1 | [12:45:11 INF] Init transaction agent took 3 Milliseconds to finish host.server_1 | [12:45:11 INF] Before collection#1: memory=14MB, #activations=0, collector=<#Activations=0, #Buckets=0, buckets=[]>. host.server_1 | [12:45:11 INF] Init grain services took 2 Milliseconds to finish host.server_1 | [12:45:11 INF] After collection#1: memory=14MB, #activations=0, collected 0 activations, collector=<#Activations=0, #Buckets=0, buckets=[]>, collection time=00:00:00.0070010. host.server_1 | [12:45:11 INF] MembershipOracle starting on host = 5a5cb449241b address = S172.26.83.96:11111:266492710 at 2018-06-12 09:45:10.206 GMT, backOffMax = 00:00:20 host.server_1 | [12:45:14 INF] -CleanupTable called on silo startup. Membership table 2 silos, 1 are Active, 1 are Dead, Version=<26, 26>. All silos: [SiloAddress=S172.26.95.237:11111:266481757 SiloName=Silo_d57d4 Status=Active, SiloAddress=S172.26.87.1:11111:266489784 SiloName=Silo_34c84 Status=Dead] host.server_1 | [12:45:14 INF] Starting IAmAliveUpdateTimer. host.server_1 | [12:45:14 WRN] Noticed that silo S172.26.95.237:11111:266481757 has not updated it's IAmAliveTime table column recently. Last update was at 6/12/2018 6:42:48 AM, now is 6/12/2018 9:45:14 AM, no update for 03:02:26.0940000, which is more than 00:10:00. host.server_1 | [12:45:14 INF] Starting local silo status oracle took 2980 Milliseconds to finish host.server_1 | [12:45:14 INF] OnRefreshClusterMapTimer: refresh start host.server_1 | [12:45:14 INF] Init type manager took 13 Milliseconds to finish host.server_1 | [12:45:14 INF] Starting Windows perf counter stats collection with frequency=00:00:30 host.server_1 | [12:45:14 INF] Start silo statistics took 1 Milliseconds to finish host.server_1 | [12:45:14 INF] Starting DeploymentLoadPublisher. host.server_1 | [12:45:14 INF] Started DeploymentLoadPublisher. host.server_1 | [12:45:14 INF] Start deployment load collector took 16 Milliseconds to finish host.server_1 | [12:45:14 INF] Starting Silo Watchdog. host.server_1 | [12:45:14 INF] Lifecycle observer Orleans.Runtime.Silo started in stage 8000 which took 3026 Milliseconds. host.server_1 | [12:45:14 INF] Starting lifecycle stage 8000 took 3026 Milliseconds host.server_1 | [12:45:14 INF] Starting AsyncAgent Runtime.Watchdog on managed thread 35 host.server_1 | [12:45:14 INF] Lifecycle observer Orleans.Runtime.Versions.GrainVersionStore started in stage 10000 which took 0 Milliseconds. host.server_1 | [12:45:14 INF] Initialized storage provider: ServiceId=ServiceId-123 ProviderName=Default Invariant=System.Data.SqlClient ConnectionString=[skiped]. host.server_1 | [12:45:14 INF] Initialized storage provider: ServiceId=ServiceId-123 ProviderName=OrleansStorage Invariant=System.Data.SqlClient ConnectionString=[skiped]. host.server_1 | [12:45:14 INF] Lifecycle observer Orleans.Storage.AdoNetGrainStorage-Default started in stage 10000 which took 14 Milliseconds. host.server_1 | [12:45:14 INF] Lifecycle observer Orleans.Storage.AdoNetGrainStorage-OrleansStorage started in stage 10000 which took 40 Milliseconds. host.server_1 | [12:45:14 INF] Starting lifecycle stage 10000 took 40 Milliseconds host.server_1 | [12:45:14 INF] Starting AsyncAgent Runtime.Messaging.GatewayAcceptor on managed thread 38 host.server_1 | [12:45:14 INF] Starting AsyncAgent Runtime.Messaging.GatewayClientCleanupAgent on managed thread 39 host.server_1 | [12:45:14 INF] Stopping AsyncAgent Runtime.Messaging.GatewayAcceptor that runs on managed thread 38 host.server_1 | [12:45:14 INF] Start gateway took 3 Milliseconds to finish host.server_1 | [12:45:14 INF] -BecomeActive host.server_1 | [12:45:16 WRN] Noticed that silo S172.26.95.237:11111:266481757 has not updated it's IAmAliveTime table column recently. Last update was at 6/12/2018 6:42:48 AM, now is 6/12/2018 9:45:16 AM, no update for 03:02:28.1730000, which is more than 00:10:00. host.server_1 | [12:45:16 INF] About to send pings to 0 nodes in order to validate communication in the Joining state. Pinged nodes = [] host.server_1 | [12:45:16 INF] -ReadAll (called from BecomeActive) Membership table 3 silos, 2 are Active, 1 are Dead, Version=<28, 28>. All silos: [SiloAddress=S172.26.95.237:11111:266481757 SiloName=Silo_d57d4 Status=Active, SiloAddress=S172.26.83.96:11111:266492710 SiloName=Silo_d97f1 Status=Active, SiloAddress=S172.26.87.1:11111:266489784 SiloName=Silo_34c84 Status=Dead] host.server_1 | [12:45:16 WRN] Noticed that silo S172.26.95.237:11111:266481757 has not updated it's IAmAliveTime table column recently. Last update was at 6/12/2018 6:42:48 AM, now is 6/12/2018 9:45:16 AM, no update for 03:02:28.2200000, which is more than 00:10:00. host.server_1 | [12:45:16 INF] Added Server S172.26.95.237:11111:266481757/xA47E0772. Current view: [S172.26.83.96:11111:266492710 -> , S172.26.95.237:11111:266481757 -> ] host.server_1 | [12:45:16 INF] -NotifyLocalRangeSubscribers about old new increased? True host.server_1 | [12:45:16 INF] Will watch (actively ping) 1 silos: [S172.26.95.237:11111:266481757] host.server_1 | [12:45:16 INF] -ReadAll (called from BecomeActive, after local view changed, with removed duplicate deads) Membership table: 3 silos, 2 are Active, 1 are Dead, Version=<28, 28>. All silos: [SiloAddress=S172.26.95.237:11111:266481757 SiloName=Silo_d57d4 Status=Active, SiloAddress=S172.26.83.96:11111:266492710 SiloName=Silo_d97f1 Status=Active, SiloAddress=S172.26.87.1:11111:266489784 SiloName=Silo_34c84 Status=Dead] host.server_1 | [12:45:16 INF] -Finished BecomeActive. host.server_1 | [12:45:16 INF] Starting local silo status oracle took 2097 Milliseconds to finish host.server_1 | [12:45:16 INF] Lifecycle observer Orleans.Runtime.Silo started in stage 19999 which took 2102 Milliseconds. host.server_1 | [12:45:16 INF] Starting lifecycle stage 19999 took 2102 Milliseconds host.server_1 | [12:45:16 INF] Starting Orleans.Runtime.ReminderService.LocalReminderService grain service on: S172.26.83.96:11111:266492710 x28220225, with range host.server_1 | [12:45:16 INF] Start reminder service took 93 Milliseconds to finish host.server_1 | [12:45:16 INF] Lifecycle observer Orleans.Runtime.Silo started in stage 20000 which took 102 Milliseconds. host.server_1 | [12:45:16 INF] Reminder EVENT_LOG_REMOVE_HISTORY is disabled host.server_1 | [12:45:16 INF] Starting lifecycle stage 2000 took 2 Milliseconds host.server_1 | [12:45:16 INF] Lifecycle observer Orleans.Hosting.SiloBuilderStartupExtensions+StartupTask started in stage 20000 which took 134 Milliseconds. host.server_1 | [12:45:21 WRN] Exception getting a sending socket to endpoint S172.26.95.237:11111:266481757 host.server_1 | System.TimeoutException: Connection to 172.26.95.237:11111 could not be established in 00:00:05 host.server_1 | at Orleans.Runtime.SocketManager.Connect(Socket s, IPEndPoint endPoint, TimeSpan connectionTimeout) host.server_1 | at Orleans.Runtime.SocketManager.SendingSocketCreator(IPEndPoint target) host.server_1 | at Orleans.Runtime.LRU`2.Get(TKey key) host.server_1 | at Orleans.Runtime.Messaging.SiloMessageSender.GetSendingSocket(Message msg, Socket& socket, SiloAddress& targetSilo, String& error) host.server_1 | [12:45:21 ERR] SiloStatusChangeNotification failed host.server_1 | System.AggregateException: One or more errors occurred. ---> Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/15/0000000f@S0000000f->S172.26.95.237:11111:266481757*stg/15/0000000f@S0000000f #2: . Reason = Exception getting a sending socket to endpoint S172.26.95.237:11111:266481757 host.server_1 | --- End of inner exception stack trace --- host.server_1 | ---> (Inner Exception #0) Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/15/0000000f@S0000000f->S172.26.95.237:11111:266481757*stg/15/0000000f@S0000000f #2: . Reason = Exception getting a sending socket to endpoint S172.26.95.237:11111:266481757<--- host.server_1 | [12:45:21 WRN] -Did not get ping response for ping #0 from S172.26.95.237:11111:266481757. Reason = Original Exc Type: Orleans.Runtime.OrleansMessageRejectionException Message:Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/15/0000000f@S0000000f->S172.26.95.237:11111:266481757*stg/15/0000000f@S0000000f #10: . Reason = Recent (00:00:00.0155236 ago, at 2018-06-12 09:45:21.322 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 10 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | [12:45:21 WRN] Exception getting a sending socket to endpoint S172.26.95.237:11111:266481757 host.server_1 | System.TimeoutException: Connection to 172.26.95.237:11111 could not be established in 00:00:05 host.server_1 | at Orleans.Runtime.SocketManager.Connect(Socket s, IPEndPoint endPoint, TimeSpan connectionTimeout) host.server_1 | at Orleans.Runtime.SocketManager.SendingSocketCreator(IPEndPoint target) host.server_1 | at Orleans.Runtime.LRU`2.Get(TKey key) host.server_1 | at Orleans.Runtime.Messaging.SiloMessageSender.GetSendingSocket(Message msg, Socket& socket, SiloAddress& targetSilo, String& error) host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #5: . Reason = Exception getting a sending socket to endpoint S172.26.95.237:11111:266481757 host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #5: . Reason = Exception getting a sending socket to endpoint S172.26.95.237:11111:266481757 host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #7: . Reason = Recent (00:00:00 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 7 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #7: . Reason = Recent (00:00:00 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 7 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/22/00000016@S00000016->S172.26.95.237:11111:266481757*stg/22/00000016@S00000016 #8: . Reason = Recent (00:00:00 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 8 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.DeploymentLoadPublisher.d__19.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #12: . Reason = Recent (00:00:00 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 12 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #12: . Reason = Recent (00:00:00 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 12 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #14: . Reason = Recent (00:00:00 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 14 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #14: . Reason = Recent (00:00:00 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 14 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #16: . Reason = Recent (00:00:00 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 16 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #16: . Reason = Recent (00:00:00 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 16 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #18: . Reason = Recent (00:00:00 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 18 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #18: . Reason = Recent (00:00:00 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 18 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #20: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 20 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #20: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 20 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #22: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 22 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #22: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 22 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #24: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 24 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #24: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 24 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #26: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 26 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #26: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 26 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #28: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 28 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #28: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 28 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #30: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 30 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #30: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 30 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #32: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 32 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #32: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 32 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #34: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 34 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #34: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 34 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #36: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 36 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #36: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 36 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #38: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 38 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #38: . Reason = Recent (00:00:00.0155907 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 38 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #40: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 40 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #40: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 40 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #42: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 42 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #42: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 42 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #45: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 45 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #45: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 45 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] SelectTarget failed with Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #46: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 46 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #46: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 46 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | [12:45:21 ERR] Lifecycle observer Orleans.Hosting.SiloBuilderStartupExtensions+StartupTask failed to start due to errors at stage 20000. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #45: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 45 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.GetResult() host.server_1 | at host.Server.Common.Bootstrap`1.d__3.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.d__9.MoveNext() host.server_1 | [12:45:21 ERR] Lifecycle observer Orleans.Hosting.SiloBuilderStartupExtensions+StartupTask failed to start due to errors at stage 20000. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #46: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 46 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.GetResult() host.server_1 | at host.Server.Common.Bootstrap`1.d__3.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.d__9.MoveNext() host.server_1 | [12:45:21 ERR] Lifecycle start canceled due to errors at stage 20000 host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #45: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 45 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.GetResult() host.server_1 | at host.Server.Common.Bootstrap`1.d__3.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.d__9.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.LifecycleSubject.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.LifecycleSubject.d__4.MoveNext() host.server_1 | [12:45:21 ERR] Exception during Silo.Start host.server_1 | Orleans.Runtime.OrleansLifecycleCanceledException: Lifecycle start canceled due to errors at stage 20000 ---> Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #45: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 45 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.GetResult() host.server_1 | at host.Server.Common.Bootstrap`1.d__3.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.d__9.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.LifecycleSubject.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.LifecycleSubject.d__4.MoveNext() host.server_1 | --- End of inner exception stack trace --- host.server_1 | at Orleans.LifecycleSubject.d__4.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Silo.d__64.MoveNext() host.server_1 | [12:45:21 ERR] One or more errors occurred. host.server_1 | System.AggregateException: One or more errors occurred. ---> Orleans.Runtime.OrleansLifecycleCanceledException: Lifecycle start canceled due to errors at stage 20000 ---> Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #45: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 45 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.GetResult() host.server_1 | at host.Server.Common.Bootstrap`1.d__3.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.d__9.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.LifecycleSubject.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.LifecycleSubject.d__4.MoveNext() host.server_1 | --- End of inner exception stack trace --- host.server_1 | at Orleans.LifecycleSubject.d__4.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Silo.d__64.MoveNext() host.server_1 | --- End of inner exception stack trace --- host.server_1 | at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) host.server_1 | at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) host.server_1 | at System.Threading.Tasks.Task.Wait() host.server_1 | at host.Server.Host.Program.Main(String[] args) host.server_1 | ---> (Inner Exception #0) Orleans.Runtime.OrleansLifecycleCanceledException: Lifecycle start canceled due to errors at stage 20000 ---> Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #45: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 45 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.GetResult() host.server_1 | at host.Server.Common.Bootstrap`1.d__3.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.d__9.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.LifecycleSubject.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.LifecycleSubject.d__4.MoveNext() host.server_1 | --- End of inner exception stack trace --- host.server_1 | at Orleans.LifecycleSubject.d__4.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Silo.d__64.MoveNext()<--- host.server_1 | [12:45:21 INF] Silo Stopping... host.server_1 | [12:45:21 INF] Lifecycle observer Orleans.Hosting.SiloBuilderStartupExtensions+StartupTask stopped in stage 20000 which took 0 Milliseconds. host.server_1 | [12:45:21 INF] Lifecycle observer Orleans.Hosting.SiloBuilderStartupExtensions+StartupTask stopped in stage 20000 which took 0 Milliseconds. host.server_1 | [12:45:21 INF] Lifecycle observer Orleans.Hosting.SiloBuilderStartupExtensions+StartupTask stopped in stage 20000 which took 0 Milliseconds. host.server_1 | [12:45:21 INF] Stopping Orleans.Runtime.ReminderService.LocalReminderService grain service host.server_1 | [12:45:21 INF] Lifecycle observer Orleans.Runtime.Silo stopped in stage 20000 which took 8 Milliseconds. host.server_1 | [12:45:21 INF] Stopping lifecycle stage 20000 took 10 Milliseconds host.server_1 | [12:45:21 INF] Silo starting to Shutdown() host.server_1 | [12:45:21 INF] -ShutDown host.server_1 | [12:45:21 ERR] SiloStatusChangeNotification failed host.server_1 | System.AggregateException: One or more errors occurred. ---> Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/15/0000000f@S0000000f->S172.26.95.237:11111:266481757*stg/15/0000000f@S0000000f #47: . Reason = Recent (00:00:00.1092715 ago, at 2018-06-12 09:45:21.322 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 47 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | --- End of inner exception stack trace --- host.server_1 | ---> (Inner Exception #0) Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/15/0000000f@S0000000f->S172.26.95.237:11111:266481757*stg/15/0000000f@S0000000f #47: . Reason = Recent (00:00:00.1092715 ago, at 2018-06-12 09:45:21.322 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 47 without sending. CONNECTION_RETRY_DELAY = 00:00:01.<--- host.server_1 | [12:45:21 INF] DeactivateAllActivations. host.server_1 | [12:45:21 INF] DeactivateActivations: total 1 to shutdown, out of them 1 promptly, 0 later when become idle and 0 are already being destroyed or invalid. host.server_1 | [12:45:21 INF] Starting DestroyActivations #0 of 1 activations host.server_1 | [12:45:21 INF] Stopping lifecycle stage 2000 took 0 Milliseconds host.server_1 | [12:45:21 WRN] UnregisterManyAsync 1 failed. host.server_1 | Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #49: . Reason = Recent (00:00:00.1249664 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 49 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__107.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Catalog.d__76.MoveNext() host.server_1 | [12:45:21 INF] Done FinishDestroyActivations #0 - Destroyed 1 Activations. host.server_1 | [12:45:21 INF] Lifecycle observer Orleans.Runtime.Silo stopped in stage 19999 which took 69 Milliseconds. host.server_1 | [12:45:21 INF] Stopping lifecycle stage 19999 took 69 Milliseconds host.server_1 | [12:45:21 INF] Lifecycle observer Orleans.Runtime.Versions.GrainVersionStore stopped in stage 10000 which took 0 Milliseconds. host.server_1 | [12:45:21 INF] Lifecycle observer Orleans.Storage.AdoNetGrainStorage-OrleansStorage stopped in stage 10000 which took 0 Milliseconds. host.server_1 | [12:45:21 INF] Lifecycle observer Orleans.Storage.AdoNetGrainStorage-Default stopped in stage 10000 which took 0 Milliseconds. host.server_1 | [12:45:21 INF] Stopping lifecycle stage 10000 took 0 Milliseconds host.server_1 | [12:45:21 INF] Lifecycle observer Orleans.Runtime.Silo stopped in stage 8000 which took 0 Milliseconds. host.server_1 | [12:45:21 INF] Stopping lifecycle stage 8000 took 0 Milliseconds host.server_1 | [12:45:21 WRN] Ignoring Orleans.Runtime.OrleansMessageRejectionException exception thrown from an action called by Silo.Stop. host.server_1 | System.AggregateException: One or more errors occurred. ---> Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/10/0000000a@S0000000a->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #48: . Reason = Recent (00:00:00.0937146 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 48 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.d__11.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.d__14.MoveNext() host.server_1 | --- End of inner exception stack trace --- host.server_1 | at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) host.server_1 | at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) host.server_1 | at System.Threading.Tasks.Task.Wait(TimeSpan timeout) host.server_1 | at Orleans.OrleansTaskExtentions.WaitWithThrow(Task task, TimeSpan timeout) host.server_1 | at Orleans.Runtime.Utils.SafeExecute(Action action, ILogger logger, Func`1 callerGetter) host.server_1 | ---> (Inner Exception #0) Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/10/0000000a@S0000000a->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #48: . Reason = Recent (00:00:00.0937146 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 48 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.d__11.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.d__14.MoveNext()<--- host.server_1 | [12:45:21 INF] Lifecycle observer Orleans.Runtime.Silo stopped in stage 4000 which took 4 Milliseconds. host.server_1 | [12:45:21 INF] Stopping lifecycle stage 4000 took 5 Milliseconds host.server_1 | [12:45:21 INF] Lifecycle observer Orleans.Runtime.InsideRuntimeClient stopped in stage 2000 which took 0 Milliseconds. host.server_1 | [12:45:21 INF] Silo is Stopped() host.server_1 | [12:45:21 INF] -KillMyself host.server_1 | [12:45:21 INF] ActivationDirectory.PrintActivationDirectory(): Size = 0, Directory: host.server_1 | [] host.server_1 | [12:45:21 ERR] SiloStatusChangeNotification failed host.server_1 | System.AggregateException: One or more errors occurred. ---> Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/15/0000000f@S0000000f->S172.26.95.237:11111:266481757*stg/15/0000000f@S0000000f #50: . Reason = Recent (00:00:00.1874086 ago, at 2018-06-12 09:45:21.322 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 50 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | --- End of inner exception stack trace --- host.server_1 | ---> (Inner Exception #0) Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/15/0000000f@S0000000f->S172.26.95.237:11111:266481757*stg/15/0000000f@S0000000f #50: . Reason = Recent (00:00:00.1874086 ago, at 2018-06-12 09:45:21.322 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 50 without sending. CONNECTION_RETRY_DELAY = 00:00:01.<--- host.server_1 | [12:45:21 INF] Stop request processed host.server_1 | [12:45:21 INF] Stopping AsyncAgent Runtime.Messaging.SiloMessageSender/SystemSender that runs on managed thread 15 host.server_1 | [12:45:21 INF] Stopping Windows perf counter stats collection host.server_1 | [12:45:21 INF] Stop request processed host.server_1 | [12:45:21 INF] Stopping AsyncAgent Runtime.Messaging.SiloMessageSender/PingSender that runs on managed thread 13 host.server_1 | [12:45:21 INF] Statistics: ^^^ host.server_1 | App.Requests.Latency.Average.Millis=677.000 host.server_1 | App.Requests.LatencyHistogram.Millis=[0.1024:0.2047]=3, [0.2048:0.4095]=4, [0.4096:0.8191]=7, [0.8192:1.6383]=5, [1.6384:3.2767]=14, [3.2768:6.5535]=7, [6.5536:13.1071]=1, [104.8576:209.7151]=1, [1677.7216:3355.4431]=1, [3355.4432:6710.8863]=6, host.server_1 | App.Requests.TimedOut.Current=0 host.server_1 | App.Requests.Total.Requests.Current=49 host.server_1 | Catalog.Activation.Collection.NumberOfCollections.Current=1 host.server_1 | Catalog.Activation.Created.Current=1 host.server_1 | Catalog.Activation.CurrentCount=0 host.server_1 | Catalog.Activation.Destroyed.Current=1 host.server_1 | Catalog.Activation.FailedToActivate.Current=0 host.server_1 | Catalog.Activation.Shutdown.ViaDirectShutdown.Current=1 host.server_1 | ConsistentRing.AverageRangePercentage=50.000 host.server_1 | ConsistentRing.MyRange.RingDistance=x6A4EB65E host.server_1 | ConsistentRing.MyRange.RingPercentage=41.526 host.server_1 | ConsistentRing.Ring=[S172.26.83.96:11111:266492710 -> , S172.26.95.237:11111:266481757 -> ] host.server_1 | ConsistentRing.RingSize=2 host.server_1 | Directory.CacheSize=0 host.server_1 | Directory.Lookups.Cache.HitRatio=0, Delta=0 host.server_1 | Directory.Lookups.Cache.Issued.Current=20 host.server_1 | Directory.Lookups.Cache.Successes.Current=0 host.server_1 | Directory.Lookups.Full.Issued.Current=21 host.server_1 | Directory.Lookups.Local.Issued.Current=21 host.server_1 | Directory.Lookups.Local.Successes.Current=0 host.server_1 | Directory.Lookups.LocalDirectory.Issued.Current=2 host.server_1 | Directory.Lookups.LocalDirectory.Successes.Current=0 host.server_1 | Directory.Lookups.Remote.Received.Current=0 host.server_1 | Directory.Lookups.Remote.Sent.Current=20 host.server_1 | Directory.PartitionSize=1 host.server_1 | Directory.Registrations.Issued.Current=0 host.server_1 | Directory.Registrations.Local.Current=0 host.server_1 | Directory.Registrations.Remote.Received.Current=0 host.server_1 | Directory.Registrations.Remote.Sent.Current=0 host.server_1 | Directory.Registrations.SingleAct.Issued.Current=1 host.server_1 | Directory.Registrations.SingleAct.Local.Current=1 host.server_1 | Directory.Registrations.SingleAct.Remote.Received.Current=0 host.server_1 | Directory.Registrations.SingleAct.Remote.Sent.Current=0 host.server_1 | Directory.Ring=[S172.26.95.237:11111:266481757/A47E0772, S172.26.83.96:11111:266492710/28220225] host.server_1 | Directory.Ring.MyPortion.AverageRingPercentage=50.000 host.server_1 | Directory.Ring.MyPortion.RingDistance=2086405451 host.server_1 | Directory.Ring.MyPortion.RingPercentage=48.578 host.server_1 | Directory.Ring.MyPredecessors=[S172.26.95.237:11111:266481757/A47E0772] host.server_1 | Directory.Ring.MySuccessors=[S172.26.95.237:11111:266481757/A47E0772] host.server_1 | Directory.Ring.RingSize=2 host.server_1 | Directory.UnRegistrations.Issued.Current=0 host.server_1 | Directory.UnRegistrations.Local.Current=0 host.server_1 | Directory.UnRegistrations.Remote.Received.Current=0 host.server_1 | Directory.UnRegistrations.Remote.Sent.Current=0 host.server_1 | Directory.UnRegistrationsMany.Issued.Current=1 host.server_1 | Directory.UnRegistrationsMany.Remote.Received.Current=0 host.server_1 | Directory.UnRegistrationsMany.Remote.Sent.Current=1 host.server_1 | Directory.Validations.Cache.Received.Current=0 host.server_1 | Directory.Validations.Cache.Sent.Current=0 host.server_1 | Dispatcher.NewPlacement.Current=1 host.server_1 | Gateway.ConnectedClients.Current=0 host.server_1 | Gateway.LoadShedding.Current=0 host.server_1 | Gateway.Received.Current=0 host.server_1 | Grain.host.Server.Grains.EventLogMaintenanceGrain.Current=0 host.server_1 | Membership.ActiveCluster=[S172.26.95.237:11111:266481757] host.server_1 | Membership.ActiveClusterSize=1 host.server_1 | MessageCenter.ReceiveQueueLength=0 host.server_1 | MessageCenter.SendQueueLength=0 host.server_1 | Messaging.Acceptor.CheckedInSocketEventArgs.Current=0 host.server_1 | Messaging.Acceptor.CheckedOutSocketEventArgs.Current=0 host.server_1 | Messaging.Acceptor.InUseSocketEventArgs=0 host.server_1 | Messaging.Expired.AtDispatch.Current=0 host.server_1 | Messaging.Expired.AtInvoke.Current=0 host.server_1 | Messaging.Expired.AtReceive.Current=0 host.server_1 | Messaging.Expired.AtRespond.Current=0 host.server_1 | Messaging.Expired.AtSend.Current=0 host.server_1 | Messaging.Pings.ReplyMissed.S172.26.95.237:11111:266481757.Current=1 host.server_1 | Messaging.Pings.Sent.S172.26.95.237:11111:266481757.Current=1 host.server_1 | Messaging.Processing.ActivationData.All=0 host.server_1 | Messaging.Processing.Dispatcher.Processed.Errors.Direction.OneWay.Current=0 host.server_1 | Messaging.Processing.Dispatcher.Processed.Errors.Direction.Request.Current=20 host.server_1 | Messaging.Processing.Dispatcher.Processed.Errors.Direction.Response.Current=0 host.server_1 | Messaging.Processing.Dispatcher.Processed.Ok.Direction.OneWay.Current=0 host.server_1 | Messaging.Processing.Dispatcher.Processed.Ok.Direction.Request.Current=1 host.server_1 | Messaging.Processing.Dispatcher.Processed.Ok.Direction.Response.Current=0 host.server_1 | Messaging.Processing.Dispatcher.Processed.ReRoute.Direction.OneWay.Current=0 host.server_1 | Messaging.Processing.Dispatcher.Processed.ReRoute.Direction.Request.Current=0 host.server_1 | Messaging.Processing.Dispatcher.Processed.ReRoute.Direction.Response.Current=0 host.server_1 | Messaging.Processing.Dispatcher.Processed.Total.Current=21 host.server_1 | Messaging.Processing.Dispatcher.Received.Direction.OneWay.Current=0 host.server_1 | Messaging.Processing.Dispatcher.Received.Direction.Request.Current=1 host.server_1 | Messaging.Processing.Dispatcher.Received.Direction.Response.Current=0 host.server_1 | Messaging.Processing.Dispatcher.Received.OnActivationContext.Current=0 host.server_1 | Messaging.Processing.Dispatcher.Received.OnNullContext.Current=1 host.server_1 | Messaging.Processing.Dispatcher.Received.Total.Current=1 host.server_1 | Messaging.Processing.IGC.Forwarded.Current=0 host.server_1 | Messaging.Processing.IGC.ReRoute.Current=0 host.server_1 | Messaging.Processing.IGC.Resent.Current=0 host.server_1 | Messaging.Processing.IMA.Enqueued.ToActivationContex.Current=0 host.server_1 | Messaging.Processing.IMA.Enqueued.ToNullContex.Current=1 host.server_1 | Messaging.Processing.IMA.Enqueued.ToSystemTargetContex.Current=50 host.server_1 | Messaging.Processing.IMA.Received.Current=51 host.server_1 | Messaging.Received.BatchSize.PerSocketDirection.GatewayToClient=0.000 host.server_1 | Messaging.Received.BatchSize.PerSocketDirection.SiloToSilo=0.000 host.server_1 | Messaging.Received.BatchSizeBytesHistogram.Bytes.PerSocketDirection.GatewayToClient= host.server_1 | Messaging.Received.BatchSizeBytesHistogram.Bytes.PerSocketDirection.SiloToSilo= host.server_1 | Messaging.Received.Bytes.Header.Current=0 host.server_1 | Messaging.Received.Bytes.Total.Current=0 host.server_1 | Messaging.Received.Direction.OneWay.Current=0 host.server_1 | Messaging.Received.Direction.Request.Current=0 host.server_1 | Messaging.Received.Direction.Response.Current=0 host.server_1 | Messaging.Received.Messages.Total.Current=0 host.server_1 | Messaging.Received.MessageSizeHistogram.Bytes= host.server_1 | Messaging.Rejected.Request.Current=47 host.server_1 | Messaging.Rerouted.OneWay.Current=0 host.server_1 | Messaging.Rerouted.Request.Current=0 host.server_1 | Messaging.Rerouted.Response.Current=0 host.server_1 | Messaging.Sent.BatchSize.PerSocketDirection.GatewayToClient=0.000 host.server_1 | Messaging.Sent.BatchSize.PerSocketDirection.SiloToSilo=0.000 host.server_1 | Messaging.Sent.BatchSizeBytesHistogram.Bytes.PerSocketDirection.GatewayToClient= host.server_1 | Messaging.Sent.BatchSizeBytesHistogram.Bytes.PerSocketDirection.SiloToSilo= host.server_1 | Messaging.Sent.Bytes.Header.Current=0 host.server_1 | Messaging.Sent.Bytes.Total.Current=0 host.server_1 | Messaging.Sent.Direction.OneWay.Current=0 host.server_1 | Messaging.Sent.Direction.Request.Current=0 host.server_1 | Messaging.Sent.Direction.Response.Current=0 host.server_1 | Messaging.Sent.Failed.Request.Current=27 host.server_1 | Messaging.Sent.LocalMessages.Current=24 host.server_1 | Messaging.Sent.Messages.Total.Current=0 host.server_1 | Messaging.Sent.MessageSizeHistogram.Bytes= host.server_1 | Networking.Sockets.GatewayToClient.Duplex.Closed.Current=0 host.server_1 | Networking.Sockets.GatewayToClient.Duplex.Opened.Current=0 host.server_1 | Networking.Sockets.Silo.Receiving.Closed.Current=0 host.server_1 | Networking.Sockets.Silo.Receiving.Opened.Current=0 host.server_1 | Networking.Sockets.Silo.Sending.Closed.Current=0 host.server_1 | Networking.Sockets.Silo.Sending.Opened.Current=0 host.server_1 | Queues.QueueSize.Instantaneous.Scheduler.LevelOne=0 host.server_1 | Reminders.AverageTardiness.Seconds=0 Secs host.server_1 | Reminders.NumberOfActiveReminders=1 host.server_1 | Reminders.TicksDelivered.Current=0 host.server_1 | Runtime.IsOverloaded=False host.server_1 | Runtime.Threads.AsynchAgent.Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer`1.Current=1 host.server_1 | Runtime.Threads.AsynchAgent.Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer.Current=0 host.server_1 | Runtime.Threads.AsynchAgent.Runtime.Messaging.GatewayAcceptor.Current=0 host.server_1 | Runtime.Threads.AsynchAgent.Runtime.Messaging.GatewayClientCleanupAgent.Current=1 host.server_1 | Runtime.Threads.AsynchAgent.Runtime.Messaging.IncomingMessageAcceptor.Current=0 host.server_1 | Runtime.Threads.AsynchAgent.Runtime.Messaging.IncomingMessageAgent.Current=3 host.server_1 | Runtime.Threads.AsynchAgent.Runtime.Messaging.SiloMessageSender.Current=0 host.server_1 | Runtime.Threads.AsynchAgent.Runtime.Scheduler.WorkerPoolThread.Current=5 host.server_1 | Runtime.Threads.AsynchAgent.Runtime.Watchdog.Current=1 host.server_1 | Runtime.Threads.AsynchAgent.TotalThreadsCreated.Current=16 host.server_1 | Scheduler.NumLongQueueWaitTimes.Current=0 host.server_1 | Scheduler.NumLongRunningTurns.Current=0 host.server_1 | Scheduler.WorkItemGroupCount=14 host.server_1 | Serialization.BufferPool.AllocatedBuffers.Current=250 host.server_1 | Serialization.BufferPool.BuffersInPool=250 host.server_1 | Serialization.BufferPool.CheckedInBuffers.Current=250 host.server_1 | Serialization.BufferPool.CheckedOutBuffers.Current=250 host.server_1 | Serialization.BufferPool.DroppedBuffers.Current=0 host.server_1 | Serialization.BufferPool.DroppedTooLargeBuffers.Current=0 host.server_1 | Serialization.BufferPool.InUse.AllocatedAndNotInPool_Buffers=0 host.server_1 | Serialization.BufferPool.InUse.CheckedOutAndNotCheckedIn_Buffers=0 host.server_1 | Silo.StartTime=2018-06-12 09:45:10.221 GMT host.server_1 | Storage.Activate.Errors.Current=0 host.server_1 | Storage.Activate.Total.Current=0 host.server_1 | Storage.Clear.Errors.Current=0 host.server_1 | Storage.Clear.Latency=0 Secs host.server_1 | Storage.Clear.Total.Current=0 host.server_1 | Storage.Read.Errors.Current=0 host.server_1 | Storage.Read.Latency=0 Secs host.server_1 | Storage.Read.Total.Current=0 host.server_1 | Storage.Write.Errors.Current=0 host.server_1 | Storage.Write.Latency=0 Secs host.server_1 | Storage.Write.Total.Current=0 host.server_1 | SystemTarget..Current=4 host.server_1 | Transactions.Abort.Total.Current=0 host.server_1 | Transactions.Commit.Completed.Current=0 host.server_1 | Transactions.Commit.InDoubt.Current=0 host.server_1 | Transactions.Commit.QueueLength.Current=0 host.server_1 | Transactions.Commit.Request.Current=0 host.server_1 | Transactions.Start.Completed.Current=0 host.server_1 | Transactions.Start.QueueLength.Current=0 host.server_1 | Transactions.Start.Request.Current=0 host.server_1 | Watchdog.NumHealthChecks.Current=0 host.server_1 | host.server_1 | [12:45:21 INF] Lifecycle observer Orleans.Runtime.Silo stopped in stage 2000 which took 44 Milliseconds. host.server_1 | [12:45:21 INF] Stopping lifecycle stage 2000 took 45 Milliseconds host.server_1 | [12:45:21 INF] Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/2 that runs on managed thread 10 host.server_1 | [12:45:21 INF] Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/1 that runs on managed thread 9 host.server_1 | [12:45:21 INF] Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/3 that runs on managed thread 11 host.server_1 | [12:45:21 INF] Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/System.5 that runs on managed thread 7 host.server_1 | [12:45:21 INF] Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/0 that runs on managed thread 8 host.server_1 | [12:45:21 INF] OrleansTaskScheduler.PrintStatistics(): RunQueue=0, WorkItems=14, Directory: host.server_1 | [--System*WorkItemGroup:Name=*stg/0/00000000,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=33; Total processed=33; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-12:Queued=0; Detailed SchedulingContext=<[LowPrioritySystemTarget: S172.26.83.96:11111:266492710*stg/0/00000000@S00000000] CurrentlyExecuting=null> host.server_1 | --System*WorkItemGroup:Name=*stg/10/0000000a,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=3; Total processed=3; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-5:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/10/0000000a@S0000000a] CurrentlyExecuting=Transient Rejection (info: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/10/0000000a@S0000000a->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #48: . Reason = Recent (00:00:00.0937146 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 48 without sending. CONNECTION_RETRY_DELAY = 00:00:01.) Response S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a->S172.26.83.96:11111:266492710*stg/10/0000000a@S0000000a #48: > host.server_1 | --System*WorkItemGroup:Name=*stg/11/0000000b,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=4; Total processed=4; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-6:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/11/0000000b@S0000000b] CurrentlyExecuting=null> host.server_1 | --System*WorkItemGroup:Name=*stg/12/0000000c,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-2:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/12/0000000c@S0000000c] CurrentlyExecuting=null> host.server_1 | --System*WorkItemGroup:Name=*stg/13/0000000d,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=2; Total processed=2; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-8:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/13/0000000d@S0000000d] CurrentlyExecuting=null> host.server_1 | --System*WorkItemGroup:Name=*stg/14/0000000e,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=67; Total processed=67; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-13:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e] CurrentlyExecuting=Transient Rejection (info: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #49: . Reason = Recent (00:00:00.1249664 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 49 without sending. CONNECTION_RETRY_DELAY = 00:00:01.) Response S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a->S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e #49: > host.server_1 | --System*WorkItemGroup:Name=*stg/15/0000000f,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=34; Total processed=34; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-10:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/15/0000000f@S0000000f] CurrentlyExecuting=Transient Rejection (info: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/15/0000000f@S0000000f->S172.26.95.237:11111:266481757*stg/15/0000000f@S0000000f #50: . Reason = Recent (00:00:00.1874086 ago, at 2018-06-12 09:45:21.322 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 50 without sending. CONNECTION_RETRY_DELAY = 00:00:01.) Response S172.26.95.237:11111:266481757*stg/15/0000000f@S0000000f->S172.26.83.96:11111:266492710*stg/15/0000000f@S0000000f #50: > host.server_1 | --System*WorkItemGroup:Name=*stg/17/00000011,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=1; Total processed=1; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-9:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/17/00000011@S00000011] CurrentlyExecuting=null> host.server_1 | --System*WorkItemGroup:Name=*stg/19/00000013,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-14:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/19/00000013@S00000013] CurrentlyExecuting=null> host.server_1 | --System*WorkItemGroup:Name=*stg/20/00000014,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=76; Total processed=76; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-1:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/20/00000014@S00000014] CurrentlyExecuting=Unrecoverable Rejection (info: Orleans.Runtime.OrleansMessageRejectionException: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/14/0000000e@S0000000e->S172.26.95.237:11111:266481757*stg/10/0000000a@S0000000a #46: . Reason = Recent (00:00:00.0312286 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 46 without sending. CONNECTION_RETRY_DELAY = 00:00:01. host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.OrleansTaskExtentions.<g__ConvertAsync4_0>d`1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__112.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.d__8.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Placement.PlacementDirectorsManager.d__10.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.d__41.MoveNext() host.server_1 | --- End of stack trace from previous location where exception was thrown --- host.server_1 | at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() host.server_1 | at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) host.server_1 | at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<b__1>d.MoveNext()) Response S172.26.83.96:11111:266492710*grn/B3628901/00000000+Default->S172.26.83.96:11111:266492710*stg/20/00000014@S00000014 #44: > host.server_1 | --System*WorkItemGroup:Name=*stg/22/00000016,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=14; Total processed=14; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-4:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/22/00000016@S00000016] CurrentlyExecuting=Transient Rejection (info: Silo S172.26.83.96:11111:266492710 is rejecting message: Request S172.26.83.96:11111:266492710*stg/22/00000016@S00000016->S172.26.95.237:11111:266481757*stg/22/00000016@S00000016 #8: . Reason = Recent (00:00:00 ago, at 2018-06-12 09:45:21.353 GMT) connection failure trying to reach target silo S172.26.95.237:11111:266481757. Going to drop Request msg 8 without sending. CONNECTION_RETRY_DELAY = 00:00:01.) Response S172.26.95.237:11111:266481757*stg/22/00000016@S00000016->S172.26.83.96:11111:266492710*stg/22/00000016@S00000016 #8: > host.server_1 | --System*WorkItemGroup:Name=*stg/24/00000018,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-7:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/24/00000018@S00000018] CurrentlyExecuting=null> host.server_1 | --System*WorkItemGroup:Name=*stg/27/0000001b,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-3:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/27/0000001b@S0000001b] CurrentlyExecuting=null> host.server_1 | --System*WorkItemGroup:Name=*stg/28/0000001c,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=656; Total processed=656; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-11:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.26.83.96:11111:266492710*stg/28/0000001c@S0000001c] CurrentlyExecuting=null>] host.server_1 | [12:45:21 INF] Silo Stopped host.server_1 | [12:45:21 WRN] Process is exiting host.server_1 | [12:45:21 INF] Silo termination is in progress - Will wait for it to finish host.server_1 exited with code 1