Wget failing to download multiple files due to naming

Featured on Meta. Now live: A fully responsive profile. Reducing the weight of our footer. Related 0. Hot Network Questions. Question feed. Accept all cookies Customize settings. Active 5 years ago. Viewed 1k times. I think this would work:! Improve this question. InverseTelecine InverseTelecine 1 1 silver badge 8 8 bronze badges.

Add a comment. Active Oldest Votes. Improve this answer. Timothy Truckle Timothy Truckle 2 2 silver badges 7 7 bronze badges. I could approach the problem from the angle of parsing the log file, but that seemed like the wrong way to go about it, and figuring out how to parse out just the failed URLs would take me a long time.

Want to know which application is best for the job? Post your question in this forum. View Public Profile. View Review Entries. Find More Posts by powah. Find More Posts by jschiwal. Quote: Originally Posted by jschiwal The robots. Posting Rules. Similar Threads. As this can be a complicated task there are other options you may need to use such as -p , -P , --convert-links , --reject and --user-agent.

It is always best to ask permission before downloading a site belonging to someone else and even if you have permission it is always good to play nice with their server.

If you want to download a file via FTP and a username and password is required, then you will need to use the --ftp-user and --ftp-password options. If you are getting failures during a download, you can use the -t option to set the number of retries. Such a command may look like this:. If you want to get only the first level of a website, then you would use the -r option combined with the -l option.

It has many more options and multiple combinations to achieve a specific task. You can also find the wget manual here in webpage format. Redirecting Output The -O option sets the output file name. As for the second question, this is just an example, you can simply change the logic of for , or just make a list which contains url you want and traversing it.

Just giving different url to pool. First question working perfect, and seems to be a bit faster! Second and third question I'm lost Show 4 more comments. Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password.