13:29:36 - Updating FEB temperature widget...
13:29:44 - Starting task "Switch HVPA voltage state"
13:29:44 - Setting HVPA current limit to 12489.75 uA
13:29:44 - Setting Hvpa.Hvpa0.CurrentTrip in all drawers
13:29:54 - Setting Hvpa.Hvpa1.CurrentTrip in all drawers
13:30:04 - Setting Hvpa.Hvpa2.CurrentTrip in all drawers
13:30:13 - Setting Hvpa.Hvpa3.CurrentTrip in all drawers
13:30:23 - Setting Hvpa.Hvpa4.CurrentTrip in all drawers
13:30:33 - Setting Hvpa.Hvpa5.CurrentTrip in all drawers
13:30:43 - Setting Hvpa.Hvpa6.CurrentTrip in all drawers
13:30:53 - Setting HVPA ramp speed to 25 V/s
13:31:04 - Switching HVPA to state Nominal
13:31:23 - Setting HVPA current limits...
13:32:44 - Finished task "Switch HVPA voltage state"
13:49:14 - Updating voltage widget...
13:49:25 - Could not update voltage widget: Could not invoke the Read service: BadTimeout
13:49:29 - Updating voltage widget...
13:49:40 - Could not update voltage widget: Could not invoke the Read service: BadTimeout
13:49:44 - Updating FEB temperature widget...
13:49:52 - Updating voltage widget...
13:50:17 - High voltage locked channels will be checked every 60 s
13:51:18 - Checking for high voltage locked pixels...
13:51:20 - Module 91 has 1 pixel(s) locked (HVPA status of 0x3f)!
13:51:21 - Module 170 has 1 pixel(s) locked (HVPA status of 0x6f)!
13:51:22 - Module 197 has 1 pixel(s) locked (HVPA status of 0x7b)!
13:51:32 - HVPA channel 2 of drawer 197 has status 0x8003!
13:51:32 - HVPA channel 2 in module 197 is locked!
13:51:32 - HVPA channel 2 of drawer 231 has status 0x0007!
13:51:32 - HVPA channel 2 in module 231 is in rise default!
13:51:38 - HVPA channel 4 of drawer 170 has status 0x8003!
13:51:38 - HVPA channel 4 in module 170 is locked!
13:51:45 - HVPA channel 6 of drawer 91 has status 0x8003!
13:51:45 - HVPA channel 6 in module 91 is locked!
13:51:45 - HV is locked in 3 channel(s), was in rise default in 1 channel(s) and was still in progress in 0 channel(s)
13:51:45 - Unlocking 4 channel(s)...
13:51:45 - ("NmcError: Failed to write 'NectarFrontEnd_197.Hvpa.Hvpa2.Disable': status Bad statuses are present, code 0xfffffffb, status NmcStatus.Mismatch", 4294967291)
13:52:18 - Checking for high voltage locked pixels...
13:52:20 - Module 91 has 1 pixel(s) locked (HVPA status of 0x3f)!
13:52:21 - Module 170 has 1 pixel(s) locked (HVPA status of 0x6f)!
13:52:22 - Module 197 has 1 pixel(s) locked (HVPA status of 0x7b)!
13:52:32 - HVPA channel 2 of drawer 197 has status 0x8003!
13:52:32 - HVPA channel 2 in module 197 is locked!
13:52:32 - HVPA channel 2 of drawer 231 has status 0x0007!
13:52:32 - HVPA channel 2 in module 231 is in rise default!
13:52:38 - HVPA channel 4 of drawer 170 has status 0x8003!
13:52:38 - HVPA channel 4 in module 170 is locked!
13:52:45 - HVPA channel 6 of drawer 91 has status 0x8003!
13:52:45 - HVPA channel 6 in module 91 is locked!
13:52:45 - HV is locked in 3 channel(s), was in rise default in 1 channel(s) and was still in progress in 0 channel(s)
13:52:45 - Unlocking 4 channel(s)...
13:53:18 - Checking for high voltage locked pixels...
13:53:32 - HVPA channel 2 of drawer 231 has status 0x0007!
13:53:32 - HVPA channel 2 in module 231 is in rise default!
13:53:44 - HV is locked in 0 channel(s), was in rise default in 1 channel(s) and was still in progress in 1 channel(s)
13:53:44 - Unlocking 1 channel(s)...
13:54:18 - Checking for high voltage locked pixels...
13:54:32 - HVPA channel 2 of drawer 231 has status 0x0007!
13:54:32 - HVPA channel 2 in module 231 is in rise default!
13:54:44 - HV is locked in 0 channel(s), was in rise default in 1 channel(s) and was still in progress in 1 channel(s)
13:54:44 - Unlocking 1 channel(s)...
13:55:18 - Checking for high voltage locked pixels...
13:55:32 - HVPA channel 2 of drawer 231 has status 0x0007!
13:55:32 - HVPA channel 2 in module 231 is in rise default!
13:55:44 - HV is locked in 0 channel(s), was in rise default in 1 channel(s) and was still in progress in 2 channel(s)
13:55:45 - Unlocking 1 channel(s)...
13:55:46 - Stopped periodic check of locked high voltages
13:55:46 - High voltage locked channels will be checked every 60 s
13:55:46 - Stopped periodic check of locked high voltages
13:56:05 - Starting NMC
13:56:05 - Clearing PPS error counters
13:56:18 - Resetting FEB counters...
13:56:22 - Starting all configured modules...
13:57:48 - Calling method FSM.gotoObserving in object FSM
13:57:48 - Starting run 2599
13:59:25 - Checking UCTS clock...
13:59:25 - UCTS clock is late by 696949 seconds!
13:59:25 - Configuring UCTS with (10.200.7.250, 10.200.4.100, 44:a8:42:44:32:c9)
13:59:25 - Configuring TIB...
13:59:25 - Setting TIB forced mode to False
13:59:25 - Writing False to Camera.TIB.Forced.ForcedMode.ForcedMode_v
13:59:25 - Status: TibState.PpsStopped
13:59:26 - Alarms: AlarmVector.ReadError
13:59:26 - Connection status: False TXclient_alive: False
13:59:26 - Entering set_external_delay
13:59:26 - Setting ExternalDelayLocal to 27.000 ns
13:59:26 - Writing 27.0 to Camera.TIB.ExternalDelay.ExternalDelayLocal.ExternalDelayLocal_v
13:59:26 - Entering set_external_delay
13:59:26 - Setting ExternalDelayCalibration to 58.000 ns
13:59:26 - Writing 58.0 to Camera.TIB.ExternalDelay.ExternalDelayCalibration.ExternalDelayCalibration_v
13:59:26 - Entering set_external_delay
13:59:26 - Setting ExternalDelaySphe to 110.000 ns
13:59:26 - Writing 110.0 to Camera.TIB.ExternalDelay.ExternalDelaySphe.ExternalDelaySphe_v
13:59:26 - Setting TIB trigger mode to 0x70 = TriggerMask.SlowControl|Pedestal|Software
13:59:26 - Writing 112 to Camera.TIB.TriggerMode.ModeMask.ModeMask_v
13:59:26 - Writing 1 to Camera.TIB.InputsEnable.InputMask.InputMask_v
13:59:26 - Writing False to Camera.TIB.BUSY.DisableBusy.DisableBusy_v
13:59:26 - Writing 0.0 to Camera.TIB.BUSY.ExtraDelayBusy.ExtraDelayBusy_v
13:59:26 - Writing True to Camera.TIB.Throttler.DisableThrottler.DisableThrottler_v
13:59:26 - Writing 255 to Camera.TIB.Throttler.MaxTriggers50us.MaxTriggers50us_v
13:59:26 - Writing 255 to Camera.TIB.Throttler.MaxTriggers1ms.MaxTriggers1ms_v
13:59:26 - Status: TibState.PpsStopped
13:59:26 - Alarms: AlarmVector.ReadError
13:59:26 - Connection status: False TXclient_alive: False
13:59:26 - Writing False to Camera.TIB.Pedestal.PeriodicPedestalEnable.PeriodicPedestalEnable_v
13:59:26 - Writing 100 to Camera.TIB.Pedestal.PeriodicPedestalFrequency.PeriodicPedestalFrequency_v
13:59:26 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
13:59:26 - Writing False to Camera.TIB.Pedestal.DigitalPedestalEnable.DigitalPedestalEnable_v
13:59:26 - Writing 100 to Camera.TIB.Pedestal.DigitalPedestal_config_Rate.DigitalPedestal_config_Rate_v
13:59:26 - Writing 375764 to Camera.TIB.Pedestal.DigitalPedestalSeed.DigitalPedestalSeed_v
13:59:26 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
13:59:26 - Writing False to Camera.TIB.Pedestal.AnalogPedestalEnable.AnalogPedestalEnable_v
13:59:26 - Writing 3000.0 to Camera.TIB.Pedestal.AnalogPedestalThreshold.AnalogPedestalThreshold_v
13:59:26 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
13:59:30 - Connecting TIB to camera server at 10.10.4.12:222
13:59:30 - Starting TIB...
13:59:30 - Status: TibState.PpsStopped
13:59:30 - Alarms: AlarmVector.ReadError
13:59:30 - Connection status: False TXclient_alive: True
13:59:30 - Resetting run
13:59:30 - Wait for UCTS external trigger before enabling trigger...
13:59:30 - Status: TibState.Standby
13:59:30 - Alarms: AlarmVector.NoAlarm
13:59:30 - Connection status: False TXclient_alive: True
13:59:30 - Status: TibState.Standby
13:59:30 - Alarms: AlarmVector.NoAlarm
13:59:30 - Connection status: False TXclient_alive: True
13:59:31 - Status: TibState.CoutingPps
13:59:31 - Alarms: AlarmVector.NoAlarm
13:59:31 - Connection status: True TXclient_alive: True
13:59:35 - Starting TIB triggers...
13:59:35 - Enabling trigger
13:59:35 - Status: TibState.Running
13:59:35 - Alarms: AlarmVector.NoAlarm
13:59:35 - Connection status: True TXclient_alive: True
13:59:45 - Status: TibState.Running ConnectionStatus: True TXclient_alive: True
13:59:46 - Alarms: AlarmVector.NoAlarm
13:59:46 - Temperature: 44.38
13:59:46 - Local camera rate: 0.0 Hz
13:59:46 - Stereo trigger rate: 0.0 Hz
13:59:46 - SPE trigger rate: 0.0 Hz
13:59:46 - Calibration trigger rate: 0.0 Hz
13:59:46 - Software trigger rate: 0.0 Hz
13:59:46 - Pedestal trigger rate: 0.0 Hz
13:59:46 - Collected trigger rate: 0.0 Hz
13:59:46 - Busy rate: 0.0 Hz
13:59:46 - Camera trigger rate: 0.0 Hz
14:00:02 - Stopping TIB...
14:00:02 - Status: TibState.Running
14:00:02 - Alarms: AlarmVector.NoAlarm
14:00:02 - Connection status: True TXclient_alive: True
14:00:02 - Disabling trigger
14:00:02 - Disabling PPS
14:00:03 - Status: TibState.PpsStopped
14:00:03 - Alarms: AlarmVector.NoAlarm
14:00:03 - Connection status: True TXclient_alive: True
14:00:04 - Closing TIB connection to camera server
14:00:40 - Checking UCTS clock...
14:00:40 - UCTS clock is late by 696950 seconds!
14:00:40 - Configuring UCTS with (10.200.7.250, 10.200.4.100, 44:a8:42:44:32:c9)
14:00:40 - Configuring TIB...
14:00:40 - Setting TIB forced mode to False
14:00:40 - Writing False to Camera.TIB.Forced.ForcedMode.ForcedMode_v
14:00:40 - Status: TibState.PpsStopped
14:00:40 - Alarms: AlarmVector.NoAlarm
14:00:40 - Connection status: False TXclient_alive: False
14:00:40 - Entering set_external_delay
14:00:40 - Setting ExternalDelayLocal to 27.000 ns
14:00:40 - Writing 27.0 to Camera.TIB.ExternalDelay.ExternalDelayLocal.ExternalDelayLocal_v
14:00:40 - Entering set_external_delay
14:00:40 - Setting ExternalDelayCalibration to 58.000 ns
14:00:40 - Writing 58.0 to Camera.TIB.ExternalDelay.ExternalDelayCalibration.ExternalDelayCalibration_v
14:00:40 - Entering set_external_delay
14:00:40 - Setting ExternalDelaySphe to 110.000 ns
14:00:40 - Writing 110.0 to Camera.TIB.ExternalDelay.ExternalDelaySphe.ExternalDelaySphe_v
14:00:40 - Setting TIB trigger mode to 0x70 = TriggerMask.SlowControl|Pedestal|Software
14:00:40 - Writing 112 to Camera.TIB.TriggerMode.ModeMask.ModeMask_v
14:00:40 - Writing 1 to Camera.TIB.InputsEnable.InputMask.InputMask_v
14:00:41 - Writing False to Camera.TIB.BUSY.DisableBusy.DisableBusy_v
14:00:41 - Writing 0.0 to Camera.TIB.BUSY.ExtraDelayBusy.ExtraDelayBusy_v
14:00:41 - Writing True to Camera.TIB.Throttler.DisableThrottler.DisableThrottler_v
14:00:41 - Writing 255 to Camera.TIB.Throttler.MaxTriggers50us.MaxTriggers50us_v
14:00:41 - Writing 255 to Camera.TIB.Throttler.MaxTriggers1ms.MaxTriggers1ms_v
14:00:41 - Status: TibState.PpsStopped
14:00:41 - Alarms: AlarmVector.NoAlarm
14:00:41 - Connection status: False TXclient_alive: False
14:00:41 - Writing True to Camera.TIB.Pedestal.PeriodicPedestalEnable.PeriodicPedestalEnable_v
14:00:41 - Writing 1000 to Camera.TIB.Pedestal.PeriodicPedestalFrequency.PeriodicPedestalFrequency_v
14:00:41 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
14:00:41 - Writing False to Camera.TIB.Pedestal.DigitalPedestalEnable.DigitalPedestalEnable_v
14:00:41 - Writing 1000 to Camera.TIB.Pedestal.DigitalPedestal_config_Rate.DigitalPedestal_config_Rate_v
14:00:41 - Writing 375764 to Camera.TIB.Pedestal.DigitalPedestalSeed.DigitalPedestalSeed_v
14:00:41 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
14:00:41 - Writing False to Camera.TIB.Pedestal.AnalogPedestalEnable.AnalogPedestalEnable_v
14:00:41 - Writing 3000.0 to Camera.TIB.Pedestal.AnalogPedestalThreshold.AnalogPedestalThreshold_v
14:00:41 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
14:00:42 - Connecting TIB to camera server at 10.10.4.12:222
14:00:42 - Starting TIB...
14:00:42 - Status: TibState.PpsStopped
14:00:42 - Alarms: AlarmVector.NoAlarm
14:00:42 - Connection status: False TXclient_alive: True
14:00:42 - Resetting run
14:00:42 - Wait for UCTS external trigger before enabling trigger...
14:00:42 - Status: TibState.Standby
14:00:42 - Alarms: AlarmVector.NoAlarm
14:00:42 - Connection status: False TXclient_alive: True
14:00:42 - Status: TibState.Standby
14:00:42 - Alarms: AlarmVector.NoAlarm
14:00:42 - Connection status: False TXclient_alive: True
14:00:43 - Status: TibState.Armed
14:00:43 - Alarms: AlarmVector.NoAlarm
14:00:43 - Connection status: True TXclient_alive: True
14:00:46 - Starting TIB triggers...
14:00:46 - Enabling trigger
14:00:46 - Status: TibState.Running
14:00:46 - Alarms: AlarmVector.ReadError
14:00:46 - Connection status: True TXclient_alive: True
14:01:09 - Stopping TIB...
14:01:09 - Status: TibState.Running
14:01:09 - Alarms: AlarmVector.ReadError
14:01:09 - Connection status: True TXclient_alive: True
14:01:09 - Disabling trigger
14:01:10 - Disabling PPS
14:01:11 - Status: TibState.PpsStopped
14:01:11 - Alarms: AlarmVector.ReadError
14:01:11 - Connection status: True TXclient_alive: True
14:01:12 - Closing TIB connection to camera server
14:01:13 - Status: TibState.PpsStopped
14:01:13 - Alarms: AlarmVector.ReadError
14:01:13 - Connection status: False TXclient_alive: False
14:01:13 - Run duration: 0:00:26
14:01:13 - Local camera triggers: 0
14:01:13 - Slow control triggers: 0
14:01:13 - Single photo-electron triggers: 0
14:01:13 - Calibration triggers: 0
14:01:13 - Software triggers from UCTS: 0
14:01:13 - Pedestal triggers: 25580
14:01:13 - Collected input triggers: 23528
14:01:13 - Busy triggers: 0
14:01:13 - Camera event triggers sent to clusters: 23528