Automating Proxy Rotation With Bash And Cron

Managing web scrapers or automated requests often requires rotating proxy servers to avoid IP bans and detection.



You can achieve seamless proxy rotation without heavy frameworks by leveraging bash and cron.



It avoids the complexity of third-party libraries, perfect for minimalistic environments or advanced users seeking granular control.



Begin with a simple text file containing your proxy endpoints.



Each line should contain one proxy in the format ipaddress port or ipaddress port username password if authentication is needed.



Your proxies.txt could include lines such as 10.0.0.5 3128 or 172.16.0.100 8080 admin secret.



Regularly refresh your proxy list to remove inactive or banned entries.



Create a bash script that picks a random proxy and writes it to a file consumed by your scraping tool.



The script will read more on hackmd.io the proxies.txt file, count the number of lines, pick one at random, and write it to a file called current_proxy.txt.



Below is a working script template:.



bin.



PROXY_FILE=.



tmp.



if ! -f "$PROXY_FILE" ; then.



echo "Proxy list not found".



exit 1.



fi.



LINE_COUNT=$(wc -l .



if $LINE_COUNT -eq 0 ; then.



echo "Proxy file contains no entries".



exit 1.



fi.



RANDOM_LINE=$((RANDOM % LINE_COUNT + 1)).



sed -n "PROXY_FILE" > "$OUTPUT_FILE".



Make the script executable with chmod +x rotate_proxy.sh.



Test it manually by running..



Configure a cron job to trigger the proxy rotation script periodically.



Edit your crontab via crontab -e and insert: 0 .



This will rotate the proxy every hour.



Tune the cron timing to match your scraping rate: 0,30 for half-hourly, or .



Be mindful not to rotate too frequently if your scraper runs continuously, as it may cause connection drops.



Ensure your scraping tool pulls the current proxy from current_proxy.txt before every HTTP call.



Most tools like curl or wget can use the proxy by reading from a file or environment variable.



to.



Continuously validate your proxies and remove those that fail to respond.



Modify the script to send a lightweight HTTP request to each proxy and purge those that time out.



rotation_log.txt to track changes over time.



This technique is lightweight, dependable, and easily expandable.



It avoids the overhead of complex proxy management libraries and works well on any Unixlike system.



You can achieve persistent, low-profile automation using only shell scripting and system scheduling—no external tools required.