The solution to this problem is job profiling. Job profiling is a where a small helper program tracks how much CPU and memory your job uses and provides you with statistics. Using these, you can submit a single job with your best guesses, look at the output, and then have a reasonable idea of what to use for subsequent jobs of a similar type. Once you get the used to it, profiling can also be used to monitor, optimise and debug your jobs by additionally providing information on disk usage, threads and open files.
If you don't have any idea how much time your job will need, then as a starting point you should submit a single job which requests 7 days (or more). From this, you'll be able to get a better idea of how long your jobs are likely to take and will be able to make an informed decision for subsequent jobs. Please be aware that you cannot increase the run time of your job once you have submitted it, so it is better to overestimate than underestimate.
REQUESTED ******* TOTAL #######
CPU (cores)
8 +-----------------------------------------------------------------------+
| + + + + + + + + + |
| |
7 |-+ +-|
| # #### |
6 |-+ ##### # ## +-|
| # #### ## |
| #### #### |
5 |-+ ## #### +-|
| ## ##### |
| # ##### |
4 |-+ # ###### +-|
| # ########## |
| # ## |
3 |-+ # # +-|
| # # |
2 |-+ # # +-|
| # # |
| # # |
1 |-+ # # +-|
|#### # |
|# + + + + + + + + + # |
0 +-----------------------------------------------------------------------+
0 10 20 30 40 50 60 70 80 90 100
Job profiling step
This is a fairly simple graph of total aggregated CPU use over time. For this STAR run you can see that the bulk of the CPU use was at the start of the job and then there was significantly less after that. If you're wondering where the line is for requested CPU cores, you can't see it here because it's exactly on the maximum Y-axis: 8.
Of the 8 CPUs that this job requested, only around half of them were fully utilised over the whole runtime. This is an example of why it’s often only worth adding more CPUs to your job if the CPU graph is mostly maxed out for the whole job.
A good starting point for your CPU estimate is either the number in the documentation for your program or 1 CPU. It’s also usually better to use less cores all of the time than have more cores sitting idle, because overall you get more done (you can run more jobs at once) and because it’s more friendly to everyone else who might have needed those unused cores. It’s also important that you set the relevant number of CPU cores in both the Slurm options and in the program options - you don't want to reserve a lot of cores only for the program to not know that they're there and only use one.
Advanced tip: If your CPU profile has significant idle gaps, consider whether you can split your job up so that each stage uses all of its cores.
REQUESTED ******* TOTAL #######
Memory (kilobytes)
1.2e+07 +-----------------------------------------------------------------+
| + + + + + + + + + |
| |
|************************************************************* |
1e+07 |-+ +-|
| |
| |
8e+06 |-+ # +-|
| #### #################### |
| # # |
| # # |
6e+06 |-+ # # +-|
| # # |
| # # |
| # # |
4e+06 |-+ # # +-|
| ########################## # |
| ### # # |
2e+06 |-+# # ## +-|
| # # |
|# # |
|# + + + + + + + + + # |
0 +-----------------------------------------------------------------+
0 10 20 30 40 50 60 70 80 90 100
Job profiling step
REQUESTED ******* TOTAL #######
Here, I told STAR to use 10GB on the command line and it used most of that. Because of the way STAR works, it used more memory in the second half of the job. That's not great, but it could be tricky to fix because STAR does need the amount of memory it uses in the second half. This job also clearly shows the requested memory line in the plot.
The best way to estimate your memory requirements is to profile a single job. There are three key ways to do this; from most- to least-preferred they are:
1. Submit one job with a moderate memory requirement - if it doesn't have enough it'll run out and die, and you can try again with more
2. Submit one job with a large memory requirement but a short time limit, say an hour
3. Submit one job with a large memory requirement and let it finish
Please don’t submit multiple jobs asking for lots of memory when you don’t know how much they’re going to use. It could end up leaving a lot of resources idle that could have been used by other people.
Once you have a figure for how much memory to use, add a small amount of headroom. 25% - 50% is likely to be sufficient, so don’t overdo it! Remember that you can always increase it again if it turns out to be too little and you can always re-run jobs which ran out without having to re-run jobs which didn’t. Conversely, if you find that the first one was using more than typical, reduce it again.
Advanced tips: Memory is in high demand; if you can run more jobs with each using less memory this may be worth doing. Also, if you can split a single job with varying memory requirements into multiple, more predictable jobs, you can run more jobs at once.
REQUESTED ******* TOTAL #######
Data read (bytes)
8e+09 +-------------------------------------------------------------------+
| + + + + + + + + + |
| # |
7e+09 |-+ ## +-|
| ########################## # |
6e+09 |-+ # # +-|
| # # |
| # # |
5e+09 |-+ # # +-|
| # # |
| # # |
4e+09 |-+ # # +-|
| # # |
| # # |
3e+09 |-+ # # +-|
| # # |
2e+09 |-+ # # +-|
| ######### # # |
| ################# #### # |
1e+09 |-# # +-|
| # # |
|# + + + + + + + + + # |
0 +-------------------------------------------------------------------+
0 10 20 30 40 50 60 70 80 90 100
Job profiling step
REQUESTED ******* TOTAL #######
Data written (bytes)
7e+09 +-------------------------------------------------------------------+
| + + + + + + + + + |
| |
6e+09 |-+ ## +-|
| ########## # |
| ####################### # |
5e+09 |-+ ### # +-|
| ## # |
| ## # |
4e+09 |-+ ## # +-|
| # # |
| ## # |
| ### # |
3e+09 |-+ ## # +-|
| # # |
| ## # |
2e+09 |-+ ### # +-|
| ## # |
| # # |
1e+09 |-+ ## # +-|
| # # |
| #+ + + + + + + + + # |
0 +-------------------------------------------------------------------+
0 10 20 30 40 50 60 70 80 90 100
Job profiling step
In general, you won't necessarily need these. However, as you get more proficient with Slurm and particularly as you start to optimse and troubleshoot your jobs, this may become useful.
The archive can be extracted using tar(1) or the graphical interface of your choice. Inside you'll find the following:
- sps-JOBID.log: A log file containing overview information about the profiling process
- sps-JOBID-[cpu|mem|read|write].tsv: Raw per-process profiling data that you can re-analyse at any time
- sps-JOBID-[cpu|mem|read|write].tsv.ascii: ASCII plots of the full per-process profiling data
- sps-JOBID-[cpu|mem|read|write].tsv.png: PNG graphical plots of the full per-process profiling data; this can be very useful
- sps-JOBID-[cpu|mem|read|write].tsv.sum: Raw summarised profiling data that you can re-analyse at any time
- sps-JOBID-[cpu|mem|read|write].tsv.sum.ascii: ASCII plots of the summarised profiling data; these are the plots which appear in the Slurm log file
- sps-JOBID-[cpu|mem|read|write].tsv.sum.png: PNG graphical plots of the summarised profiling data
It's worth taking some time to look at these from time to time, as they can be quite revealing - in particular the full per-process PNG plots.