Post processing [SABNZBD] not working

Post any problems / bugs / issues that are Mylar-related in here.
Post Reply
crazyleo
Posts: 30
Joined: Wed Mar 04, 2015 7:45 am

Post processing [SABNZBD] not working

Post by crazyleo »

Hi there,

I've recently setup mylar in docker on my QNAP (TVS-871). However, my post processing integration is not working properly. I'm running the linuxio container image. All volumes are mounted to local folders on NAS. Mylar triggers the download, however, during post processing SABNZBD client cannot run the ComicRun.py and autoprocesscomic.py scripts. It just hangs and stays there till I restart the service. This is the error I'm getting on SABNZBD:

***************************************************************************************************************************************************
Requests module not found on system. I'll revert so this will work, but you probably should install
requests to bypass this in the future (ie. pip install requests)
Loading config from /config/scripts/autoProcessComics.cfg
Opening URL: http://x.x.x.x:8090/api?apikey=d08c1551 ... e&failed=0
Traceback (most recent call last):
File "/config/scripts/ComicRN.py", line 22, in <module>
sys.exit(autoProcessComics.processIssue(sys.argv[1], sys.argv[3], sys.argv[7], comicrn_version=comicrn_version))
File "/config/scripts/autoProcessComics.py", line 87, in processIssue
urlObj = urllib2.urlopen(url)
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1228, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
r = h.getresponse(buffering=True)
File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 453, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 417, in _read_status
raise BadStatusLine(line)
httplib.BadStatusLine: ''
******************************************************************************************************************************************************
I was also getting some "OutOfBound" message on mylar console.
I tried to update to the latest version but no I'm getting another weird error that says "Failed to start on port: 8090 - Is something else running?"

not sure what I'm doing wrong but any help would be highly appreciated.

Cheers
Last edited by crazyleo on Sat Jan 26, 2019 2:08 pm, edited 1 time in total.
crazyleo
Posts: 30
Joined: Wed Mar 04, 2015 7:45 am

Re: Post processing [SABNZBD] not working

Post by crazyleo »

so, I rebooted my container and tried again with a new download. The "Failed to start on port : 8090" error disappeared, however, sabnzbd still getting stuck trying to run ComicRN.py script. I also see the below error message on mylar console:

*******************************************************************************************************************
Traceback (most recent call last):
File "/app/mylar/mylar/logger.py", line 337, in new_run
old_run(*args, **kwargs)
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/app/mylar/mylar/PostProcessor.py", line 1370, in Process
self.nzb_or_oneoff_pp(tinfo=pp)
File "/app/mylar/mylar/PostProcessor.py", line 1793, in nzb_or_oneoff_pp
if dupthis['action'] == 'dupe_src' or dupthis['action'] == 'dupe_file':
TypeError: 'NoneType' object has no attribute '__getitem__'
*******************************************************************************************************************

I'm quite sure I've done something wrong but I can't pinpoint it.
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Post processing [SABNZBD] not working

Post by evilhero »

Do you have Completed Download Handling enabled for SABnzbd within Mylar as well as using the ComicRN script? You can only have one or the other running, as having both running will mean that they're both competing to post-process the same item and you'll end up getting errors due to race conditions.

The requests error is saying that the requests module can't be found in your sabnzbd container - the ComicRN script uses the local python to run which then tries to load the modules inherited on the system (since I can't package the requests module with SAB for obvious reasons). The easiest way to solve this particular problem is from the SABnzbd docker - do a pip install requests in order to install the requests module and then you won't be getting that warning message any longer.

You don't mention which commit/branch you're running - might be useful information as well.
crazyleo
Posts: 30
Joined: Wed Mar 04, 2015 7:45 am

Re: Post processing [SABNZBD] not working

Post by crazyleo »

Hi evilhero,
no, I don't have the "Completed Download Handling" option enabled for SABNZBD from within mylar. I'm only using post processing using ComcRN.py script from SABNZBD container. I tried doing a "pip install requests" but ran into further issues with my container. I'm getting "ImportError: cannot find name main" error every time I try to run pip command. I think something is broken here which is beyond my very limited knowledge of *nix platforms.

I do have some other questions though:
even though i have mylar and sabnzbd running as separate container with their own IPs. I am mounting share on NAS mounted as /downloads to both of them. Do I still need to check option box for "Are mylar/sabnzbd on separate machines" and specify the folder with unc rather than just saying /downloads? It is a locally mounted volume after all.
I am monitoring /downloads folder under post processing tab. Do I still need to check the "Completed Download Handling" if I want mylar to handle post processing? How is this different to monitoring /downloads folder on the post processing tab? Please explain.

Cheers
crazyleo
Posts: 30
Joined: Wed Mar 04, 2015 7:45 am

Re: Post processing [SABNZBD] not working

Post by crazyleo »

ok, so I managed to successfully install requests module via pip. I tried another download. Now, there are no errors on SABNZBD but the post-processing script (ComicRN.py) seems be stuck in a loop and never finishes; it just says running script "ComicRN.py". Not sure what is wrong here.
I check mylar container immediately and the "Failed to start on port : 8090" error came back and getting logged every second on the terminal screen. There were more errors in mylar logs:

***********************************************************************************************************************************************************
2019-01-27 09:24:29 WARNING There were no files located - check the debugging logs if you think this is in error.
2019-01-27 09:19:29 WARNING There were no files located - check the debugging logs if you think this is in error.
2019-01-27 09:14:29 WARNING There were no files located - check the debugging logs if you think this is in error.
2019-01-27 09:11:12 ERROR Uncaught exception: Traceback (most recent call last):
File "/app/mylar/mylar/logger.py", line 337, in new_run
old_run(*args, **kwargs)
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/app/mylar/mylar/PostProcessor.py", line 1370, in Process
self.nzb_or_oneoff_pp(tinfo=pp)
File "/app/mylar/mylar/PostProcessor.py", line 1793, in nzb_or_oneoff_pp
if dupthis['action'] == 'dupe_src' or dupthis['action'] == 'dupe_file':
TypeError: 'NoneType' object has no attribute '__getitem__'
2019-01-27 09:11:12 WARNING [DUPECHECK] Make sure if you are using ComicRN, you do not have Completed Download Handling enabled (or vice-versa). Aborting
2019-01-27 09:11:12 WARNING [DUPECHECK] File cannot be located in location specified. Something has moved or altered the name.
2019-01-27 09:09:29 WARNING No branch specified within config - will attempt to poll version from mylar
2019-01-27 09:09:29 ERROR Could not locate exceptions.csv file. Make sure it is in datadir: /config/mylar

***********************************************************************************************************************************************************

BTW, I have completely disabled post processing on mylar, i.e., from "downloads" tab and also from "Quality & Post Processing" tab. Not sure why I'm getting the DUPECHECK error. Also, how can I find what version I'm on. The setting page only tells me that Mylar version is "Master" and "--source build None".

Please help.

Cheers
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Post processing [SABNZBD] not working

Post by evilhero »

Ok, so it sounds like you're having a great time with all of this, so let's try and help you out as best we can.

If you have mylar & sabnzbd in separate containers/dockers, then you need to use the separate machines option within Mylar, as is normal with containers they are environments contained to themselves and cannot normally communicate with each other as per local addresses. Whichever path you specify for your download directory it has to be identical in both containers, so in your case odds are the /downloads 'should' work, but you may have to try with the UNC path in case it doesn't.

Do not monitor the same folder that your sabnzbd downloads go to if you're using either Completed Download Handling or the ComicRN script. The folder monitor will fire off every X minutes, and if it co-incides with a sabnzbd download occuring you'll have a race condition and either it will process fine, but error out on the 2nd attempt - or they'll both lock each other out.

Completed Download Handling (CDH) & ComicRN are the 2 methods available for post-processing. CDH will monitor your SABnzbd client for completed downloads and once they're completed it will automatically post-process the given items. ComicRN is a script that will run after a download completes in your SABnzbd client, and is an external call. If you can, CDH would be the better/easier method to use in most cases and it has the added benefit of being able to directly post-process to a specific ID instead of ComicRN which has to base everything off of the nzbname for matching (which works, but CDH would be more reliable). And of course, ComicRN requires the autoProcessComics.cfg file be configured for things to work, which is an extra step.

Folder monitoring will just monitor a given folder every X minutes and perform a manual post-processing action on the specified folder. The actual post-processing is the same regardless of the method being used, however both the manual post-processing and ComicRN rely on name parsing to match items to your watchlist, whereas the CDH already knows the specific ComicID/IssueID an item is tied to and post-processes directly against that without the need to parse anything.

Without the "post-processing" option enabled within Mylar, you won't get anything moved to the series directories. Renaming, metatagging etc are all part of the post-processing action (which are optional), but in order for them to be used - post-processing has to be enabled first.

Check your config.ini file. Make sure your git_branch = master and git_user = evilhero. It sounds like you either have more than one instance of Mylar firing off, or there's an error in your config.ini which is causing things to fail on startup (which really, the failing to start on port would indicate that there's something locking that port already - perhaps a failed post-processing job is locking the process?)
crazyleo
Posts: 30
Joined: Wed Mar 04, 2015 7:45 am

Re: Post processing [SABNZBD] not working

Post by crazyleo »

evilhero wrote: Sun Jan 27, 2019 5:05 pm t sounds like you're having a great time with all of this, so let's try and help you out as best we can.
Thanks buddy. I really appreciate this.
evilhero wrote: Sun Jan 27, 2019 5:05 pm so in your case odds are the /downloads 'should' work
So, this is my setup:
*****************************************************************************************************************
- "Download Settings" Tab has "Mylar/SABNZBD on separate machine" set to checked.
- SABNZBD category is set to "comics"
- "SABNZBD Download Directory" is set to "/downloads" on same tab. I'm guessing it means the downloads folder local to SABNZBD (which is the "complete" folder) as mylar is accessing SABNZBD container via full api access (I've also specified user login and password for SABNZBD as well).
- "Enable Completed Download Handling" option is set to yes on the same tab.

- On "Quality & Post Processing" Tab the "Enable Post Processing" option is set to yes. However, I'm not monitoring any folder here as per your suggestion.

- On SABNZBD there is a "Comics" category setup. However, I'm not using the "ComicRN.py" script with it.
- No post processing in SABNZBD as per your suggestion.
*****************************************************************************************************************

So, as per your suggestion I'm only using mylar to do post processing via "Completed Download Handling" option. However, I'm still not having any luck with this. SABNZBD does it's part and files are parked in "/downloads" (Completed folder).
Mylar is generating these error messages in the log file:

*****************************************************************************************************************
2019-01-27 21:31:21 ERROR Uncaught exception: Traceback (most recent call last):
File "/app/mylar/mylar/logger.py", line 337, in new_run
old_run(*args, **kwargs)
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/app/mylar/mylar/PostProcessor.py", line 1409, in Process
if dupthis['action'] == 'dupe_src' or dupthis['action'] == 'dupe_file':
TypeError: 'NoneType' object has no attribute '__getitem__'
2019-01-27 21:31:21 WARNING [DUPECHECK] Make sure if you are using ComicRN, you do not have Completed Download Handling enabled (or vice-versa). Aborting
2019-01-27 21:31:21 WARNING [DUPECHECK] File cannot be located in location specified. Something has moved or altered the name.
2019-01-27 21:27:50 WARNING No branch specified within config - will attempt to poll version from mylar
2019-01-27 21:27:50 ERROR Could not locate exceptions.csv file. Make sure it is in datadir: /config/mylar
*****************************************************************************************************************

Then on the container console of Mylar I'm getting this message:
*****************************************************************************************************************
Traceback (most recent call last):
File "/app/mylar/mylar/logger.py", line 337, in new_run
old_run(*args, **kwargs)
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/app/mylar/mylar/PostProcessor.py", line 1409, in Process
if dupthis['action'] == 'dupe_src' or dupthis['action'] == 'dupe_file':
TypeError: 'NoneType' object has no attribute '__getitem__'
*****************************************************************************************************************

It's complaining about ComicRN script and "Completed Download Handling" being enabled at the same time but that's not the case.
evilhero wrote: Sun Jan 27, 2019 5:05 pm Make sure your git_branch = master and git_user = evilhero.
My git_user is set to evilhero but git_branch is set to none. Should I just change it in the config file?
evilhero wrote: Sun Jan 27, 2019 5:05 pm failed post-processing job is locking the process?
This ^^. After all these changes and disabling post processing on SABNZBD that error message disappeared.

Let me know if you need further information to help me.
Thanks again.
crazyleo
Posts: 30
Joined: Wed Mar 04, 2015 7:45 am

Re: Post processing [SABNZBD] not working

Post by crazyleo »

crazyleo wrote: Mon Jan 28, 2019 1:28 am evilhero wrote: ↑Mon Jan 28, 2019 3:05 am
Make sure your git_branch = master and git_user = evilhero.
My git_user is set to evilhero but git_branch is set to none. Should I just change it in the config file?
I changed the git_branch to master and tried to update but nothing happened.
crazyleo wrote: Mon Jan 28, 2019 1:28 am evilhero wrote: ↑Mon Jan 28, 2019 3:05 am
failed post-processing job is locking the process?
Actually, I just noticed that it only generates the "is something running on port 8090" error when I restart mylar from web interface and I then have to reboot the container.

I'm also sometimes getting "Manual post processing is unable to run at this time" error message. I have to reboot the container after that. Please help.

Thanks.
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Post processing [SABNZBD] not working

Post by evilhero »

Your container is setting your log level to a level where only errors/warnings are logged - which isn't helpful in the least when it comes to debugging. You need to go into the Logs tab, and enable the verbose logging option in the upper right corner of that page. Then try and snatch something and get the logs from that. The debug logs will provide exact information as to what paths/filenames/etc that is being passed around and we can see more clearly what's happening.
"SABNZBD Download Directory" is set to "/downloads" on same tab. I'm guessing it means the downloads folder local to SABNZBD (which is the "complete" folder) as mylar is accessing SABNZBD container via full api access (I've also specified user login and password for SABNZBD as well).
The folder specified is the location relative to Mylar's where the downloads are located. It is accessing the SAB container via the API for general information (sending the nzb, monitoring for completion, but that's it). So you need to make sure that the path you put in Mylar is accessible from within the mylar container AND that it's pointing to the /downloads folder in sab with the required permissions (the user running mylar must have full authority to that folder in order to perform post-processing properly). Have you tried putting in the complete path to your sabnzbd download location ?

The errors you're encountering with the duplicate checker are due to Mylar not being able to properly locate the file given the path it's being passed back from SAB. So what happens Mylar sends the nzb to SAB, your SAB client downloads it. Once it's downloaded, SAB sends back information to Mylar as to what it downloaded and Mylar uses that information to post-process. The information SAB sends back is relative to the SABnzbd container cause that's all the information it knows.

As far as the update not doing anything - it's possible you're already up-to-date. If you have the verbose logs enabled, it will provide more information when you attempt to do an update as well.
Post Reply