Very true indeed. I was also randomly changing user-agents (Mozilla, Safari, Chrome, IE). I thought that this will be harder to tell whether there is a lot of traffic from the same network or someone is just intensively crawling the site.
For me, it was more a proof of how efficient and fast a crawler can be.
Also, a response from IMDB was very fast in less than 0.4 seconds, so not that much time was lost there.
For me, it was more a proof of how efficient and fast a crawler can be. Also, a response from IMDB was very fast in less than 0.4 seconds, so not that much time was lost there.