Automated File Downloading with Azureus & RSS

As broadband internet has become more prevalent, so have peer-to-peer file sharing networks.  And while most come and go it mediocrity, one peer-to-peer network has stepped forward and truly evolved with the internet.  That network is known simply as BitTorrent.  The how’s and why’s of BitTorrent are beyond the scope of this story, but with only a basic understanding of the technology, anyone can use it to its full potential.

For now, lets assume that you are at least familiar with BitTorrent.  We are going to look at how to use the torrent client Azureus and RSS (Real Simple Syndication) feeds to automate the download of content.

Start by downloading and installing Azureus.  It is a java based client so it is inherently cross platform.  Once the client has been downloaded and installed, simply double click on the icon to launch it.  You will need to go through a small setup wizard before you will be ready to begin.

Once you have completed the setup, you should see a screen similar to the one below:

Simply select Installation Wizard from the Plugins menu.

Choose the SourceForge.net option and click next.

Check the box next to RSS Feed Scanner and click next.

I recommend installing the plugin only for the current user.  If there are problems with Azureus down the line, it is easier to isolate the issue using a secondary account that does not have other plug-ins installed.  Its not essential, but it is a good trouble shooting plan.

Finally, click the install button and wait while the plugin downloads and installs.  The installation is complete, just click the close button and exit the wizard.

Now select RSSFeed from the Plug-ins menu.  This will open an new panel called RSSFeed Scanner in the main window of Azureus.

Now we need to set up the RSS feed in Azureus and setup a filter for the files we are interested in.  This is done in the Options tab.

The trick in this screen is to use the scroll bar at the bottom of the screen and slide it all the way to the right.  This is the only way to see the + and – buttons that let us add the RSS feeds and filters.

There seems to be some sort of bug in the Java implementation on the Mac.  You must scroll to the right in order to see the + and –.

Now click the upper most + symbol to add the RSS feed information.

In this example, I have use TorrentSpy as the source for my RSS feed.  Many Trackers now use RSS to publish a list of their most recently added torrent files.  Bare in mind that not all RSS feeds will work.  In my testing, I found one feed that simply would not work in Azureus.

It appears that the RSS pages on some tracker web sites require specific data to be passed by the client viewing the feed.  To this end, Azureus has the Pass Cookie’s checkbox.  Rather than mess with the complexities of sites that require the cookies to be passed, I suggest avoiding these sites.  Since trackers are not hard to find, simply select another tracker with a less imposing RSS feed.

Setup the tracker as indicated above.  For the Directory field, simply select a directory on your hard drive where you would like the downloaded files kept.  Don’t forget to check the Enabled box before you click save.

Now its time to set up the filter.  This is what tells Azureus what to watch for in the RSS feed, and what to do with it when it finds it.  Click the lower of the two green + symbols to continue.

Name the filter whatever you like.

Checking the Filter is a Regular Expression box and specifying a correctly formatted filter sting is key.  In all honesty, a full grasp of Regular Expression is still beyond me at this point, but this filter does the job correctly.  Put simply, the filter does a word search on the torrent’s title containing the words Prison, Break, HDTV, XViD, and LOL, in any order.  If the torrent contains all of those key words, in any order, then the file is automatically added to the download queue in Azureus.

In my example, I wanted the HDTV capture in XViD format.  In addition to that, I like the format that the user with the handle LOL puts out, and I know that he always puts his handle in the title.  All of this criteria combined leads to a very specific file and helps to ensure that I don’t have 5 copies of Prison Break downloading every week.

In addition to this, the RSSFeed Scanner plug-in for Azureus has some more advanced features.  It keeps track of which files it has downloaded in the past.  That way, should the same file somehow make its way back into the RSS feed later, it will skip it rather than download it again.

The plug-in also has a way of recognizing season and episode information in the torrent’s title.  In the case of TV shows, there is usually some notation specifying the season and episode number of the file.  Azureus let you use this info to specify a range, if you only want to download certain episodes.

Beyond that, fill in the rest of the fields as indicated.  The info did not fit into one screen, so the rest of the setting are listed below:

 

The pretty much covers the configuration.  Since only newly posted files appear in the RSS feeds, only the latest files should be run past your filters.  It is a good idea to get an idea of the formatting of the files you will be letting Azureus search for.  In my case, I manually searched TorrentSpy first to get an idea of how the files were formatted and get an idea of what users posted files most regularly.  This is vital to setting up a reliable filter.

If you would like to see what files are currently listed in the RSS feed, click on the Status tab.  You will see how long until the feed is downloaded again, and you will see all files currently in the feed.

Keep in mind that you can use the info in the Status window to set up new filters, but the filters will not be compared with that data until the feed is refreshed.  This can be done at any time by right clicking on the feed title and selecting Refresh.

Once the Scanner finds a file to download, it will be added to the My Torrents tab in the main window.  The file is then downloaded as if you had added it to Azureus yourself.

 

There are a few things you can do to keep Azureus from aversely effecting your entire network.  By default, Azureus will attempt to use your entire upstream and downstream to share files.  Since broadband up-streams are normally much smaller than down-streams, it is important to limit the upstream.  If you do not, the upstream becomes completely saturated and it actually becomes difficult to surf the web, check email, or use advanced services such as Voice Over IP.

In order to cap the upload and download capabilities of Azureus, simply select Preferences from the Azureus menu.

Here, I have limited my max upload to 40KB.  I know that my upstream is limited by Comcast to 765Kb, so this will leave me plenty of bandwidth for normal activities.

Why is this so important?  I use a VoIP phone system for my home phone line.  If I don’t limit the upstream usage in Azureus I will have phone problems.  This is an easy way to insure reliable phone service when I leave Azureus open to upload and download torrent files.

While you are at it, click on Connection on the left.  I like to change the Incoming TCP Listen Port to something non-standard.  I can’t prove it, but I think Comcast messes with users who have traffic on this port.  Nearly every time I have Azureus open, my internet connection seem to be unstable.  As soon as I changed this setting to a non-standard port, my connection became reliable and I have not had any issues since.

In truth, I am not sure what the repercussions of changing this port might be.  From what I can tell, I am still able to upload and download files just as before.  Changing the port number just seems to bypass whatever filter Comcast uses on its subscribers.

If anyone knows differently, please post a comment below.  I will update this article with any new information.

In any case, there you have it.  Harnessing the power of BitTorrent and RSS, you can automate your peer-to-peer file transfers and simply your life.  This is positive proof that the internet is evolving.  Now, if we could just do something about the SPAM…

(Visited 15,562 times, 1 visits today)
8 Responses to Automated File Downloading with Azureus & RSS
  1. Ehab zarad Reply

    thxs so much for ur nice help

  2. Anonymous Reply

    You really should set your upload higher than that, generally to 70-90% of the max. (with faster connections like yours 90% is fine) but if your using VOIP throttle it back until you have no problems but you are being a bit mean with only 10%!

    You are meant to limit it to 90% not have 90% unused.

    Experiment but VOIP doesn’t require near that much upload!

  3. smanke Reply

    Anonymous assumes that I only have VoIP traffic. I didn’t feel the need to list each of my VPN tunnels, my home Intermapper server, or all of the FTP, RDC, and ARD work that I do.

    My point was simply that users need to be mindful of their upstream requirements when they use torrent clients.

    His point is well taken. Most users could allocate more of their upstream. Since I do a lot of work out of the house, I cannot.

  4. Anonymous Reply

  5. Shumbora Reply

    cool tutorial brother , i was having some issues with the filter but after applying your suggestions , it worked like a charm , good looking out bro :)

  6. Kryz Reply

    Good tutorial, although it would be nice if you could elaborate on regexp a bit more, that could help.

    One little note about your upload speed though: you say you have an upstream speed of 384KB/s, so you put your upload limit to 40KB/s. I’m not sure what kind of connection you have, but I bet it’s DSL or something, leading me to believe you have 384 Kilo _bits_ per second as upstream speed.

    That means that 40 Kilo _Bytes_ per second as upload limit in Azureus will almost completely eat up your upstream (40×8 = 320, and that’s without ATM & TCP/IP overhead, which should also be a factor 1.15, resulting in 368 Kb/s (note the small b), which is almost 384 Kb/s). In other words: you’re using quite a bit more than you should.

    Or, you have fiber to the home or VDSL, but I couldn’t find any reports on ComCast actually providing any of those services.

  7. smanke Reply

    Kryz,

    You caught me in a goof! It always tweaks me when people aren’t careful with the KB vs the Kb thing. In my case, my Comcast is 768Kb up rather than 384Kb. So my setting did leave me room for other apps, but only when you take into account my correct upstream.

    FWIW, I’m paying Comcast a little extra to get the increased upstream, but it makes my work a lot easier.

    That for pointing out my misuse of KB/Kb and helping me realize that I had specified the incorrect upstream pipe.

    As for RegEx, I invite any reader to provide more info that I can post in that area. Every time I touch regular expression, I end up with a migraine. Every time I think I have the expression exactly how I need it, it fails to work the way I intended. :-)

  8. Anonymous Reply

    thanks for the great post!!!

Leave a Reply

Your email address will not be published. Please enter your name, email and a comment.