Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Very true indeed. I was also randomly changing user-agents (Mozilla, Safari, Chrome, IE). I thought that this will be harder to tell whether there is a lot of traffic from the same network or someone is just intensively crawling the site.

For me, it was more a proof of how efficient and fast a crawler can be. Also, a response from IMDB was very fast in less than 0.4 seconds, so not that much time was lost there.



Gray hat question out of curiosity and possible experience: did you also use proxies or perhaps even Tor?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: