Copying remote torrent downloads

Information and Tutorials on features in Mylar and how to use it
goin3d
Posts: 21
Joined: Sun Dec 08, 2019 5:34 am

Copying remote torrent downloads

Post by goin3d »

In my current setup i have mylar3 running locally on a QNAP (linux variant) and as needed passing 32p torrents up to my seedbox for downloading and further seeding. Apologies if this was covered in a post i couldn't find, but do any of the config.ini options allow for downloading of those remote torrent files upon completion? I could have sworn at one point there was a post saying the below was possible in running mylar on linux.
IE:
-mylar sends torrent to seedbox to download into a dedicated "mylar" folder
-periodically a TLS connection is made and downloads the folder into your local watch directory
-post processing happens from there

If not any recommendations on how to automate the transfer back down to mylar?

Thanks!
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Copying remote torrent downloads

Post by evilhero »

There is a built-in option within Mylar that can be used to download items once they've completed and then run them against post-processing (it doesn't download the entire folder at a scheduled time, but one-off's as required upon completion). I'll have to dig up the docs for it, as you can follow the instructions included within Mylar (root of mylar directory / post-processing folder / torrent-auto-snatch folder / read.me)

You could also use a program like harpoon that will monitor your rtorrent instance for completed downloads that were initiated by one of your auto-downloaders (ie. sonarr, radarr, lidarr, mylar, lazylibrarian) and then download them from the seedbox to your local machine and initiate a post-processing call to get that item into the respective library. Unfortunately both of these methods rely on the rtorrent client being remote (ie. seedbox), and your mylar install being local and both using a linux-based OS due to the usage of lftp. There several users that use these methods (not both obviously) and haven't had any real issues in doing so.

If you're using Windows, your best bet is to use rsync or something similar to copy back down the files every X mins/hrs - and point your folder monitor within Mylar to monitor that folder for new downloads every 5-10 minutes (we generally tell users don't go less than 5 minutes as there's been some wanky problems when dong so). Since lftp doesn't exist on Windows, harpoon or the mylar internal version won't work in that aspect.
goin3d
Posts: 21
Joined: Sun Dec 08, 2019 5:34 am

Re: Copying remote torrent downloads

Post by goin3d »

Thanks!!! I had just started reading through the work you did on Harpoon and will continue to do so. Also thanks for the heads up on the auto snatch folder, hadn't seen that before, but i'll start going through that as well.
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Copying remote torrent downloads

Post by evilhero »

[The torrent-auto-snatch stuff within Mylar is the base from which harpoon was built on. Harpoon came after Mylar, but it was also more encompassing than just dealing with comics as well as I wanted to have the same ability but with the other auto-downloaders.

One of the regular users of harpoon also has forked harpoon in order to have a version with a GUI - and from what I understand from the guys I know that use it it works very well, and is prettier!
goin3d
Posts: 21
Joined: Sun Dec 08, 2019 5:34 am

Re: Copying remote torrent downloads

Post by goin3d »

I've been trying to set this up but hit a snag. should i be just populating the .conf file with the connection information, or should that also go in the config.ini? i got confused because the config.ini has a place for the full script location, as well as all of the ssh connection fields as well.
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Copying remote torrent downloads

Post by evilhero »

goin3d wrote: Tue Apr 21, 2020 12:35 am I've been trying to set this up but hit a snag. should i be just populating the .conf file with the connection information, or should that also go in the config.ini? i got confused because the config.ini has a place for the full script location, as well as all of the ssh connection fields as well.
The get.conf has been depreciated from what I can remember - it's there, but it's not being used. Everything should be done in the config.ini only (I'm going from memory here, but that's the way I set it up to keep things less confusing for users).

I'll add this here just in case you need some further information, as the only section in the config.ini you need to worry about is the [AutoSnatch] section, which should look similar to this:

Code: Select all

[AutoSnatch]
auto_snatch_script = /opt/mylar/post-processing/torrent-auto-snatch/getlftp.sh
pp_sshhost = ultrasecret.site
auto_snatch = True
pp_sshuser = evilhero
pp_sshlocalcd = /Incoming/Torrents/comics
pp_sshport = 22
pp_sshkeyfile = /home/evilhero/.ssh/id_rsa
pp_sshpasswd = None
Here's the breakdown:
auto_snatch [True/False ] => enabling or disabling the usage of the auto-snatch scripts.
auto_snatch_script [location] => the full path location to where the getlftp.sh script is located.
pp_sshhost [location] => the domain name/ip of your remote instance (no port/paths)
pp_sshport [port#] => the port that your ssh runs on the remote instance (ie. not 80/443)
pp_sshuser [username] => the username you would use to ssh into the shell of your remote instance.
pp_sshpasswd [password] => the password for your remote instance (ssh)
pp_sshkeyfile [location] => secret key location if you key authentication is enabled on your remote instance
pp_sshlocalcd [location] => the local path location where you want the auto-snatch script to download to.

Note that if pp_sshkeyfile is set, the pp_sshpasswd should be set to None and/or left blank.
goin3d
Posts: 21
Joined: Sun Dec 08, 2019 5:34 am

Re: Copying remote torrent downloads

Post by goin3d »

ty so much for this breakdown!!!! and here i was trying to fight the .conf file LOL. Hopefully your detailed write-up helps other folks as well.

I was really hoping this would fix me up but i'm still getting the below: The files are going up to the torrent client just fine, they're downloading, but fail when trying to transfer back. As far as i can tell i'm able to manually ssh in with the same settings in the config file so not sure what's going on. Is there anything in the below that could point me in the next direction to search?

2020-04-23 21:43:23 ERROR Uncaught exception: Traceback (most recent call last):
File "/share/CACHEDEV1_DATA/Web/Mylar/mylar/logger.py", line 337, in new_run
old_run(*args, **kwargs)
File "/share/CACHEDEV1_DATA/.qpkg/QPython3/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/share/CACHEDEV1_DATA/Web/Mylar/mylar/helpers.py", line 3132, in worker_main
snstat = torrentinfo(torrent_hash=item, download=True)
File "/share/CACHEDEV1_DATA/Web/Mylar/mylar/helpers.py", line 2794, in torrentinfo
autosnatch_env['downlocation'] = re.sub("'", "\\'",downlocation)
File "/share/CACHEDEV1_DATA/.qpkg/QPython3/lib/python3.8/re.py", line 208, in sub
return _compile(pattern, flags).sub(repl, string, count)
TypeError: cannot use a string pattern on a bytes-like object
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Copying remote torrent downloads

Post by evilhero »

Looks like a conversion problem going from python2 to python3 that was never brought up (not many people use this option I'm guessing). Give me a few and I'll have a fix in the python3-dev branch that will get it straightened out.
goin3d
Posts: 21
Joined: Sun Dec 08, 2019 5:34 am

Re: Copying remote torrent downloads

Post by goin3d »

i appreciate you getting the fix out there so quickly! i'm now able to send the torrents up, and pull them back down into my incoming directory..so that part is all squared away!! It would now appear the last issue i'm having is during the post-processing phase for these transfers:

-I am not using completed download handling.
-I am renaming files during post processing.
-in this configuration i get the below error:

25-Apr-2020 19:54:13 - WARNING :: mylar.Process.554 : ThreadPoolExecutor-0_0 : GLOW [2019] is either Paused or in an Ended status with 100% completion. Ignoring for match.
25-Apr-2020 19:54:13 - INFO :: mylar.Process.1624 : ThreadPoolExecutor-0_0 : [POST-PROCESSING] No matches for Manual Run ... exiting.

When i turn off the option to rename it appears everything works ok. From searching about the forums i don't think this is related to the torrent process at all, but it's a new error for me.

Have you seen this before?
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Copying remote torrent downloads

Post by evilhero »

CDH is kinda inherited with the whole auto-snatch aspect - it doesn't use CDH, but it does a 'manual post-processing directly against that specific folder where the torrent was downloaded to. If the series you're attempting to download an issue for is 100% complete, then Mylar will assume it's not meant for that specific series and try to match up against another similar series in your watchlist if one exists, or if not - just continue along without doing anything.

Now the fact that you're renaming shouldn't affect anything unless it's encountering some sort of mismatch/error along the way. It hasn't been brought up before, so this possibly could be a new issue or some other aspect is forcing it to behave this way. Can you provide either a carepackage or a debug log snippet showing a unsuccessful post-processing and a successful one (the carepackage would be the easiest method to log both at once).
Post Reply