On SCT shift

Welcome to the ACR

On this twiki you will find a guideline through the SCT operations shift. Some information on this twiki is kindly stolen from Jose's ACR Tutorial. For more information on how to perform your tasks, please refer to:
  • The ACR tutorial itself, you can find it here
  • The manuals for DCS, DAQ and DQM, also found from here
  • Please do not hesitate to ask if you are not sure about something, but first consult the operations database in case its covered there (hint: enter a keyword and click 'search')

At the beginning of every shift

  • Come to the ACR at least 15 minutes before your shift and consult with the previous shifter
  • Check:
    • Any particular item under watch
    • The run plan
    • LHC status and plan for the coming shift
  • Read the SCT whiteboard
  • Log into the RunCommTool, which can be started from the online machine desktop panel click the General button and select RunComm. After loggin in, click "ready" to open up the check list. After filling out the first checklist, it will pop up every hour.
  • Log-in into e-log under your point-1 account, read e-log messages related to monitoring from the previous day or shift and ask the previous shifter (if available) on status, plans etc.
  • Log into the DCS, FSM Alarms (as documented here)and the ATLAS elog.

During the shift

  • Do the checks requested in the Check-List
  • Monitor DCS alarms on the DCS Alarm Panel
  • Watch for ERROR and FATAL DAQ messages in the MRS window coming from the SCT, document them in your elog.
  • React to DAQ problems, eg BUSY
  • Check Monitoring Histograms and DQM status flags
  • Check beam conditions are stable - see ROS plots
  • Monitor Cooling status and alarms
  • Document in the e-log (SCT and IDGen) problems and happenings
  • In case of problems contact relevant experts and/or Run Coordinator

At the beginning of a run

It is important to be specially vigilant at the beginning of a run. Problems most likely occur at this point. Be aware of:
  • FATAL / ERRORS from the SCT appearing in the MRS
  • Check BUSY status (and TTC Status). If no triggers are coming through one of the systems could be busy (it could be SCT!)
  • Stop-less removal could happen at this stage, the run control shifter should tell you if the SCT goes busy and may disable one or more of our RODs. Follow the instructions in the operations database if this happens.
  • Always write Information in SctGui Run comment window (pop-ups at the start and end of the run)

During the run

  • Keep checking the number of link errors in OHP. If the run control shifter did not tell you he/she disabled a ROD, you would see a big increase in link errors.
  • Check ROS Monitoring every hour. Look for errors and anomalous values in the results
  • Continue to check for errors in MRS
  • Document everything throughout your shift and upload a single elog entry at the end.

Before (or during) beam injection

Before and during beam injection we need to be in the STANDBY state, ie at reduced HV (50V instead of 150V)
  • Make sure that in the "Injection Permits" section of the "Statistics" tab or the SCT top-level secondary panel does not say MASKED. If it does, immediately tell to the shift leader.
  • If not already in STANDBY, Issue GOTO_STANDBY from the top level SCT node in the FSM. Recover any modules as necessary. If you can't get all branches to be in the STANDBY state, call the SCT DCS on-call expert.
  • Make sure that SCT Beam Protection is enabled. This is visible on the "Statistics" tab of the SCT top-level FSM panel. If it is not enabled, enable it.
  • Afterwards check that the SCT is fully in standby with SCT-> Tab "Statistics." The boxes in the middle should all read "SAFE." If any do not, assume it is not safe to inject beam, and tell the shift leader. Call the SCT DCS on-call expert.

Before stable beam (ie during ADJUST)

  • Open "IP1 Collimator Check" tab in the IDE panel and monitor the beam background, BCM activity, SCT occupancy with ROS monitoring and overall beam rates. You need to do these checks continuously and very closely, and check the trends in the rates. If the rates are too high (indicators are RED) and the trend is increasing, its unlikely we will be ready to switch in when stable beam is declared.

At stable beam

When STABLE BEAM Flag becomes TRUE, do the following:
  • Check "IP1 Collimator Check" tab in the IDE panel, beam background, BCM activity, SCT occupancy with ROS monitoring and overall beam rates.
  • Inform the ATLAS shift leader that the SCT is preparing to go 'READY' and will go on after stable beam + checks. Talk to PIXEL people on the left.
  • After stable beam was declared make on last overall check of "IP1 Collimator Check" plus collimator position (Cleaning Collimator Check tab in IDE fsm) for about 1 minute - if these have already been green for sometime before stable beams, you do not need to wait.
  • Inform shift leader that SCT is about to go on (might be skipped if shift leader has been informed already)
  • If all is OK (BCM quiet, SCT shows nominal occupancy, collimator positions are green) turn SCT 'ON'. This is done by issuing GOTO_READY from the top level SCT node,
  • Inform shift leader and PIXEL people that SCT is 'READY'.

Beam dump or beam adjustments

Planned Dump :
  • LHC send an ADJUST/ DUMP warning via the hand-shake
  • SCT will go to Standby automatically when BEAM DUMP state goes to WARNING * If the SCT tree remains in the READY state for more than 20 seconds after the beam dump handshake, issue GOTO_STANDBY from the top level SCT node in the FSM. Recover any modules as necessary. You can also issue additional GOTO_STANDBY commands to parts of the tree, if necessary. If you can't get the whole detector to be in the STANDBY state, call the SCT DCS on-call expert.
  • Afterwards check that the SCT is fully in standby with SCT-> Tab "Statistics." The boxes in the middle should all read "SAFE." If any do not, assume it is not safe to dump the beam or do beam manipulations, and tell the shift leader. Call the SCT DCS on-call expert.

Non planned Dump :

  • Ask (and wait) for a warm stop.
  • Send system to Standby if not already (see above for guidance)
  • Ask SL for an explanation and post in the e-log as much information as you can gather. Do not wait up to the end of your shift to post it!

At the end of the shift

  • Submit e-log entries SCT and IDGen and log out of the e-log
  • Inform the next shifter on the current situation of the running and about any special conditions or plans.
  • Log out of the RunComm Tool (= Check-List), DCS and FSM Alarms
  • Do not leave if the next shifter did not arrive (after 30 minutes call Run Coordinator if shifter has not arrived).
  • Sleep well!

-- Main.denooij - 2010-05-11

Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r6 - 2011-04-01 - davidr
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding ATLAS?Please contact the page author (see Topic revision above) or the Run Coordinator of the specific system.
Contact SysAdmins support only for technical issues