Why you should consider IP rotation
When scraping or crawling a website, requests are sent much faster than a human from the same IP. Websites can easily flag the IP being used as a bot or crawler and send misleading information or block you completely. Sites employ these rate limitations to protect themselves from attacks or unwanted behavior. By periodically changing, or rotating your IP address you can avoid rate limitations and fly under the radar. This ensures target sites will be unable to identify your IP as a crawler. Rotating proxies can increase your success rate and get you the data you desire.
How do I rotate my IP address?
With our easy-to-use, open-source Proxy Manager, Bright Data allows you:
- Number of requests permitted before changing the IP
- Amount of time between IP changes
- The quantity of IPs in your rotation pool
Guidelines for automatically rotating your IP address with Bright Data’s Proxy Manager
- Download the Proxy Manager
- Click ‘Add new proxy’ port
- Choose the type of IP you want and click ‘Save’
- Go to the ‘IP control’ tab
- Choose your ‘Pool size’ or number of IPs you want to rotate through
- To change the IP after a certain number of requests use the ‘Max Requests’ field
- To rotate the IP after a specific time frame use ‘Session Duration’ field
Automated IP rotation using Proxy Rules
- Choose your trigger: a specific URL, Status code, response body or request time frame that when received you want the IP to automatically rotate
- Beside ‘Action’ choose ‘Retry with new IP’ or ‘Retry with New Proxy Port (waterfall)’
- If you would like to retry the request before rotating the IP choose a number beside ‘Retries’ or input your own
If you are still facing issues or wish to move beyond manual rotation, Bright Data can offer your company a targeted solution. By using our Rotating Residential network you will gain access to millions of user devices from across the world. You can then route your data collection traffic from a user perspective, retrieving the most accurate open source data points.
Using Residential proxies together with the Proxy manager will help you take advantage of the ‘waterfall feature’ in which requests are routed through different proxy networks using customized and automated rules.