Categories module

I noticed that Categories module (the one that shows all categories with number of items in it), when published, slows server response for some unknown reason. It takes 6-8 seconds of "waiting for server" before site loads.
When module is unpublished E-shop opens lightning fast.

I used differend diagnostic tools, and all gave the same result.
Attached is graph from pingdom.com. Notoce the long yellow strip at the beginning, which represents waiting for server response. When Categories module is unpublished the whole site loads in les then 2 seconds (average 1,7 to be precise)

Does anyone else have the same issue with Categories module?
Thanks

Hello Ivan,

How many categories and products that you are having on your site?

Sincerely, Giang