Request manager's corner
In this page, we collect the relevant information for request managers
How to increase the role of a user
All users register in McM have a designated role like user, generator_contact, request_manager etc.
To raise the role of a user, go to the user page: https://cms-pdmv.cern.ch/mcm/users?page=-1&shown=51
Find the user by its username or email, show the actions
And raise/decrease the role
Edit the user to add or remove a pwg to a user by using the drop down menu
The available values of "pwg" is a name of the PAG and/or POG like BTV, EGM, HIG, HIN, etc.
How to create a flow?
The flow objects characterize how to go from one campaign to another
Click on "create new flow" or edit an existing one
Edit the parameters of the flow
The core of the Flows is the "request parameters" which determined what is going to be applied on top the request and the campaign parameter in going to the next step.
The "sequences" parameter is created automatically with the proper formatting according to the "next release".
Nested in it, there can be any parameter in the request schema
The "time_event" and "size_event" need to be filled where applicable (i.e. going into a release for which no-one will fill it up by hand)
How to create a chained campaign?
Campaigns that are connected with a flow are gathered in chained campaign
When a flow is created, the chained campaigns are not created automatically
Click "Show possible Chained campaigns" (a list icon)
This will open chained_campaigns page where all not_created chained campaigns using/connecting to the selected flow are displayed
Click on "create" to add the corresponding chained campaign
Daily actions for a request manager
The actions that need to be taken for each requests are stored in McM. Upon setting an action, verify that it has not yet been set.
Get the ticket to be operated
Verify the current chaining situation of the requests by clicking on "view all requests" and check that member_of_chain is empty or contains known existing chains to which something need to be added by enacting the ticket.
Verify that there is a priority block number or a deadline provided
Click on "Generate chains"
"Generate chains and reserve" can be clicked instead if one wants to create all pertaining requests (we should get into the habit of doing so)
Verify the chaining situation of the requests by clicking on "view all requests" and check that member_of_chain is in accordance to expectations
Setting action on single request can be done through the priority change page directly, although using the mccm document is better for tracking
Select one of the tabs, either requests, chained requests or chained campaigns
Search for the ones you want by using the available parameters
Set the desired block, staged, threshold or flag (For requests, only block number) and click the checkbox on the left (same row).
Save the changes by clicking "Submit" button in the bottom right of your screen.
Key points:
Propagation: chained campaigns ---> chained requests ---> requests
If you modify the block number for chained requests, it will be propagated to all the requests in the chain
Chained campaign changes won't be propagated, but staged and threshold values will be used when you generate chained requests and requests in a mccm ticket, in case you didn't set them up in the ticket.
Repeat the operation of setting the action and create the chained requests as many times as necessary.
How to flow a chained request?
The import of a request into a new campaign is now fully performed with flows, although further manipulation of the requests are possible for full customisation.
Select the chained request with last status in done (meaning that the last request in the chain has status done)
Multiple select or single click on "Flow"
Depending on the level of approval of different components, the request will be imported and set to be "submit" approval (*)
How to force flow a chained request?
Due to mis-configuration of a request, it could end up with less statistics than expected. Notifications are send. One can then force flow if the available stat is OK
Select the chained request which flow is prevented due to missing statistics or because the statistics are good enough and the Analyzers need the request
Choose one of the following
Take a look at the details and figure out whether really something is not correct
Rewind the chain if the stat is not OK to the requester
Click "force flow" for the chain to prevent the statistics check to limit the flowing
How to submit a request?
Upon toggling "submit" approval, injection are performed and collected in batches
Have the "actions" displayed by clicking "Show" and ticking "Actions"
Toggling the "submit" approval will perform the injection in a separate thread
For requests in approval "approved", single or multiple select and click "next step"
In case the submission has failed (the request is still in approval "submit" and status "approved"
Single or multiple select and click "submit"
Upon completion (or failure) a notification is send and the status is changed to "submitted" (if success)
Request have to be part of a chain to be toggled into submit approval, report at the next MccM or browse for missing actions in HN or previous MccM minutes
How to resubmit a request?
It could happen that a request needs only to be resubmitted, and it does not have to be hard reset
Have the "actions" displayed by clicking "Show" and ticking "Actions"
Click on "soft reset" button to have the request return to the previous approval and status
The request will be removed from any "not announced" or done" batch
The request in request manager should be either rejected by hand, or use the invalidation procedure
The request can now be retoggled to submit before the system does pick it up for automatic submission
How to reserve a chain?
Instead of having request id being created as a chain items get completed, it is possible to reserve the next requests within a chained request
Click on the "Reserve chain" button to have all next requests created
Instead of clicking on "Generate chains for ...", click on "Generate chains and reserve for ..."
How to submit a chain?
Find the first request of a chain that needs to be submitted as a chain of request
Toggle the submit approval of the request
In case of failures with submit, please contact an admin
Chain Re-Submit
In case of failures, or the wrong request being put in request manager, a soft-reset of the chain should be done
Click on the "soft reset the flow" to have all request properly sof-reset
The request will be removed from any "not announced" or done" batch
The request in request manager should be either rejected by hand, or use the invalidation procedure
The requests in the chain can now be retoggled to submit before the system does pick it up for automatic submission
Injection Log
For requests in approval "submit" one can access the injection status and logs
If a "facetime" icon is present in the actions of a request, one can single or multiple click "see injection logs" and be redirected to a page with the logs and status
The injection status page that one is redirected at after submission can be closed at anytime, the injection process is not tied to the page.
How to announce a batch?
Have the "actions" displayed by clicking "Show" and ticking "Actions"
For batches in the new status, click on the "Announce the batch to data Ops"
Provide any further details of the submission batch, in addition to the "Notes" of the batch.
Notification
Notifications can be send automatically through McM to users registered to a given request
Multiple select or single click on "Notify"
Enter the text for notification and click "Send"
Request Reset
To reset a request after submission, given that usually a whole batch is affected, it is probably more efficient to reset the full batch : see below
Navigate to the batch the request is in by clicking "view batches containing ..." to navigate to the batch
Verify that indeed all requests in that batch are affected
If yes, proceed with batch reset.
If not, reset requests one by one.
Rewind
When a chained request needs to be invalidated, it has to berewinded
Click on rewind (this might not work on first click : to be fixed)
this will reset and invalidate the last request of the chain
Repeat as many times as necessary to invalidate requests in the chain, and to put the "currently processing" asterix next to the proper request.
Navigate to the request from which the chain should restart in order to reset it and operate on it
Batch Holding
It might happen that a batch of already submitted requests, but not announced has to be put on hold from being announced
Have the "actions" displayed by clicking "Show" and ticking "Actions"
Click on "hold/unhold" button to set the status to "hold"
Once the batch can eventually be announced, click again on the "hold/unhold" button to set the status back to "new"
Batch Reset
In the unlucky case that a whole batch need to be reset, instead of resetting requests one-by-one.
Have the "actions" displayed by clicking "Show" and ticking "Actions"
Click on the "reset batch" button.
Invalidations
For requests already in production, the output dataset and requests need to be invalidated
When a request is reset, its components are put in the invalidation DB
Browse the datasets and request that need invalidation
Navigate to https://cms-pdmv.cern.ch/mcm/restapi/invalidations/inspect/
You can announce it to dataops with https://cms-pdmv.cern.ch/mcm/restapi/invalidations/inspect/announced
Remove Request
Some request might have to be removed if not necessary anymore.
Find the request that needs to be removed.
It has to be in "new" status
It should not be the current "end point" of any chain. (rewind the chain if necessary)
Click on "Delete request"
If the corresponding chain has to be removed, refer to the corresponding section.
Remove Chain
Some chained requests might have been created by mistake and need to be removed.
Find the chain that need to be removed.
It's corresponding "action" needs to be disabled.
Its deletion cannot be leaving a request unchained, unless that request is new.
Click on "Delete chained request"
If end points request need to be removed first, please refer to the corresponding section.
Manage Step by Step
Step by Step for managing requests in McM
Once approved submit the requests by either
Letting the regular inspection do the submission
Toggling "submit" approval (and subsequently click submit upon hypothetical failures)
Upon completion of a request, either
Let the routine inspection perform the flowing
Navigate to the corresponding chained request and click "flow"
If the flow's approval is sufficient, the flowing will be done. Otherwise, if the chained request's approval is sufficient, the flowing will be done, otherwise nothing will happen
Upon successful flowing new requests will be created and put in the approval & status according to the flow
New requests can be submitted or further modified
====================================================
How to create new flows and chained campaigns
(0) Before creating new flows/chained campaigns, please check if there are existing flows which can be used. all flows can be found here:
What are most important are: "Allowed Campaigns", "Next Campaign", and "Request parameters".
(1) if a new flow is indeed needed, go to page
click "Create new flow"
put the name: (use similar name from similar existing flows)
[a new flow should appear on the page ]
in the action, click, "Edit details"
(2) to modify relevant information. (the best way is to copy and paste a similar flows and modify from that)
Allowed campaings:
Next campaign:
Request parameters : here is the place we update information like: release, global tag, pileup, etc. note that "process_string" should be unique for each flow. So be sure to pick up a different string for this value. (again, best way is to update from existing similar ones)
add "some notes" about this flow, though this is not absolutely needed.
Then click "Commit" in the left, Then click "View". Now in the approval column, it shows "None" In the Actions column, click "Next Step", approval column --> flow
(3) once a new flow is created. go to the "chained campaigns" page
click "Crate chained campaigns" in the bottom left,
should be on this page
Search for the flow name which you just created.
in the action columns, click "Create chain campaign" (the middle one)
in the "Alias" field, put something meaningful (again, the best way is to look up existing similar chained campaigns), though it is also fine to leave it empty.
click "Commit"
Go to this page again
Normally need to repeat the above for wmLHE, pLHE and non-LHE based.
(4) after the chained campaign is created, you should be able find the new "chains" when you create the ticket. It is either the "Alias" which you put in the chained campaign, or the full name (the PrepId in the first column of the chained campaign).
(4a) After clicking the "generate chains" button in the ticket page, click the "View chains from ..." in the Requests column, it will link to a page like:
On this page, you should find one or multiple PrepId, each of them refers to a different chain. You should be able to find the new chain which you just generated. And you will notice that in the "Chain" column, there is only one prepID (which is the root request in your ticket ).
Now, in the Action column, you should click the "Flow" button to move forward. After this, you will see a new request (with itsprepID) appeared. This request should be a DR request, if it is GS request (which means the root request is a LHE request), click again the flow. Here we assume the GS request is ready (which means in "done" status"). This is normally the case, if the GS request is not ready, we can not do DR at all.
Now you have the new DR request created. Now click on the DR request, it will link to your the DR request page, on this page, you can double check the "Sequences", if it contains the new information you added in the Flow, for example, if you updated the global tag, or the pileup, you should be able to see it there.
(4b) Here we might want to update the information of the size/event and time/event. Usually, if this flow is general flows (for example, the ones we used for Phys14DR, or the ones which we will have for RunII DR campaign), it is good to run the test command of the Request, which also test if everything is fine. For these flows, we usually run a TTbar sample to represent the all samples.
To run a test command, go to the request page,
click "Get test command" in the Action column.it should go to this page, for example
Then, login to lxplus,
open this file, make sure the number of events are at least 50, if not, put 50. it is the "-n " option in the cmsDriver command. Note that there can be multiple cmsDriver commands. you need to update all of them to be the same.
echo "ls -l" >> BPH-Phys14DR-00001
voms-proxy-init -voms cms -valid 999:00
Note here the XXXX refers to your ID (which is unique for each user), do not copy and paste directly below.
cp /tmp/x509up_uXXXX $HOME/private/personal/voms_proxy.cert
[ more here, https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookXrootdService#DownCmd]
bsub -q 8nh -J test < BPH-Phys14DR-00001
** After the "bsub" is done, in the LSFJOB_*/STDOUT file, add the values (can be more than one) of "AvgEventCPU", and add the size of the root files produced, given by the "ls -l", (and divided by TotalEvents and by 1024 to get kB). And put back these numbers in the Flow page, see (2) above in the "Request parameters"
Note that, you can also directly add these time and size information in the Request, to do that, go to the request page, click the "Edit details" button, you can fill these information as well. Normally we do this if we want to update these information for individual request.
(4c) Once the DR request is ready to go, you can click the "Next step" in the "Actions" column, to approve it, and click again, to submit. If submission is successful, the "Approval" column will be "submit" and the "Status" column will be "submitted". Otherwise there might be some problem of injection. In this case, you should also receive an email from the McM system (with part of the injection log sent to you). In this case, contact experts.
How to fix "configuration upload fail" error for request
Sometimes a request can be approved "by hand" twice in the DR step by mistake before its submission (a short period of time between the approvals) and that causes its configuration to fail upload. To fix this, here is an example for request HIN-pAWinter13DR53X-00084:
(1) Reset the request (it will change its Approval/Status from submit/approved to none/new);
(2) In the "Actions" column, click on the "Option Reset" button (this will change its "--datatier" parameter value in step 2 from GEN-SIM-RECO to GEN-SIM-RECO,DQM);
(3) In the "Actions" column, click on the "Next step" (this will change its Approval/Status from none/new to approve/approved).
How to change the priority block?
Sometimes, we need to update the priority for some request. To do this, first we need to find the root request, then go to the request page, and, then, click the button of "View action on this request" in the Actions column, find the chain you want to update,and update the block number.
Example:
(3) now, in this column, pLHE12DR53, click the second "tool button" , it will show "5", now you can change that to 6 , you can save the update by clicking the button (next the History) in the Actions field , the button is a "triangle with a exclamation mark inside"
How to select a range of request?
How to reset a request?
Sometimes, Gen contacts ask us to reset a request so they can modify it, first we need to find the request, and in the "Action" field, click the reset button
Reset the request
Instructions:
go to the chained request which contains the request;
rewind the flow until the asterix stays after the request to be reseted;
reset the request;
Basic instructions to submit a ticket
The following shows the instructions to approve a ticket (which is generally reported about in the MCCM Wednesday's meeting) and to submit the corresponding requests included in the ticket.
- Go to the page of the ticket to submit
- Check the block of the requests, the status of the requests (they need to be all approved by the GEN conveners), the flag of keep output and the name of the dataset (if it makes sense)
- Click the button "approve and reserve chains"
- Wait until the browser says "Everything went fine"
- Refresh the page and look at the "generated chains"
- Click "approve all requests"
- You are (almost) done!
- After some time (might be minutes or hours), you should get a mail saying "Injection succeded for...": this means that the status of the request changed from "approved" to "submitted"
For submitting a request you need to announce to data-ops the corresponding batch
- You need to go the batch page which by default shows the non-announced batches
- Click on actions
- Identify the corresponding batch, which should be listed in the page
- Click on the megaphone button, which automatically announces the batch
- You are done!
How to handle a pLHE request
- MC contact presents a pLHE request with the full chain up to NanoAOD in the ticket
- Req Manager approves the ticket and reserve up to GS step. This does not reserve all the other steps of the chain but only pLHE and GS requests will be in place.
We ask MC contacts to validate the two requests together ("star" button in the chained request page). When it's validated, GEN approves.
- Req Manager submits the two steps (pLHE+GS).
- When it’s done, the request will flow automatically to DR, Mini and NanoAOD. No additional action is needed for the whole cahin (except the announcement of the corresponding batch).
Last updated
