How to find out what a CPU is idle for?

How to find out what a CPU is idle for?

We are having a multithreaded application. We have measured the total time
when the application runs single threaded as well as running on two
threads(in both cases application uses only 2 cores).
In both cases the same amount of job takes around 5 minutes.
What we see using top command (or basic usage of perf tool) shows that our
CPU usage doesn't increase much(if at all).
I'd like to know if there is any tool that can tell us on which part(s) of
our application the cpu has nothing to do?(and possibly tell us why?)
thank you for your valuable comments
P.S: if you need more information to get a better picture, the application
runs in cycles (separated by barriers). in each cycle it has to invoke the
update() method of around 100,000 objects of different types(we call them
agents).