Post Processing - Files not being moved (updated)

Post any problems / bugs / issues that are Mylar-related in here.
zeus163
Posts: 15
Joined: Mon Feb 01, 2021 5:18 am

Post Processing - Files not being moved (updated)

Post by zeus163 »

First off, thanks for this great tool. While it took me some time setting up (mostly because I'm not familiar with Python), I think it will be well worth it.

I think I have most things set up correctly, but for the life of me I can't get the post-processing of files to move to their directories. Mylar has correctly set up folders, but they remain empty and the files stay in my completed Sabnzbd folder. I followed this step: Configuration --> Quality & Post-Processing --> Enable Post-Processing) and selected move and then enabled folder monitoring and set the folder location. I then went to the download options and enabled Completed Download Handling option, which says it is the preferred method. Sabnzbd tests fine. But the files don't move.

In the meantime, what I've done is turned off Newznab (Nzbgeek) and just turned on Enable DDL (Getcomics) which seems to be doing the job. Except it didn't download four issues of Forty Seconds which I manually grabbed from there and then forced a check. Which seems to have worked.

I'd like to add Sanzbd back into this (if I can get it working correctly), but have Mylar run DDL in the search order first and then Sabnzbd second. Is that possible? I'll explore that more. I got DDL as the first order of search now. Now I just need to get Sabnzbd working correctly.

I'm sure it is something I'm doing wrong, but I'd love to get this part working correctly.

Oops...I missed out on some required information:
Mylar version: master v0.4.9.4
OS: Microsoft Windows 10 Pro/OS Version: 10.0.19041 N/A Build 19041

Since I've restarted Mylar and turned off newznab, I'm not sure my logs are helpful, plus Mylar seems to stop a couple of times today and I have to start it up from the command prompt again. I'm not sure what is happening there. I renamed mylar.py to mylar.pyw and copied it into the start folder to start Mylar on start up. Which seems to work, for the most part.

If I need to upload a carepackage or something, please let me know. I'll have to figure out how to attach it.

Thanks and sorry for babbling!
User avatar
evilhero
Site Admin
Posts: 2712
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Post Processing - Files not being moved (updated)

Post by evilhero »

If you happened to set the Folder Monitor location to the same location that SABnzbd downloads too - you're going to end up getting race conditions as the CDH may be post-processing an item when the Folder Monitor kicks off and ends up deadlocking both items. The only reason to enable the Folder Monitor is if you have another directory that constantly gets issues put into it and you want Mylar to post-process those items as they happen (ie. with Torrents). So if you're Folder Monitor is set to the same location as your SABnzbd completed folder, turn off Folder Monitoring.

You can specify provider order, by editing the config.ini file (with Mylar not running). Find the line in the .ini that's shown as:

Code: Select all

provider_order = 0, ddl
Then just add in the option in a numeric, provider_name detail - so as an example and assuming you named your provider 'Nzbgeek':

Code: Select all

provider_order = 0, ddl, 1, Nzbgeek
Note that sometimes it's easier to let Mylar generate the provider_order line based on your settings - so if you enable all the relevant providers you want, save the config and then shutdown Mylar and then edit the ini, you'll see the above with maybe the order switched around. Then it's just a matter of modifying the names in the right order (note that it is case-sensitive, so whatever provider name you enter has to match exactly to what Mylar says it was).

Now if you're still not getting items post-processing, you'll have to generate a carepackage file and attach the link somehow (mega/dropbox/something) so that we can see what's going on. Note that prior to doing this you should make sure Mylar is in debug mode - which can be with any or all of:
- the -v switch on the startup command
- setting the log_level = 2 in the config.ini
- going to the History tab / View Logs button / Toggle Verbose Logging on option
Once it's in debug mode, then try to perform a search & download so Mylar can log the full debug information of a run. Once it's failed, then generate the carepackage and attach it.

Hope that helps!
zeus163
Posts: 15
Joined: Mon Feb 01, 2021 5:18 am

Re: Post Processing - Files not being moved (updated)

Post by zeus163 »

Thanks for your reply. I really appreciate it!

The folder location is located as a sub-folder in my Sabnzbd downloads directories. However, it is not under Complete, it is under comics. It looks like this: L:\Sabnzbd\comics\mylar. The files are going to L:\Sabnzbd\Complete and then not moving out of there. I disabled Folder Monitor and the files still stayed in L:\Sabnzbd\Complete. I have a "comics" category in Sabnzbd, which is the Sabnzbd category I originally selected in Download Settings>SABnzbd Category, but since the files end up in the Complete folder, I changed "comics" to "Complete" under Download Settings>SABnzbd Category.

I then noticed this under logs:
2021-02-03 16:36:27 WARNING NZB seems to have been removed from queue: SABnzbd_nzo_qbjf_4y8
2021-02-03 16:36:22 ERROR Cannot find nzb SABnzbd_nzo_qbjf_4y8 in the queue. Was it removed?
2021-02-03 16:36:17 ERROR Cannot find nzb SABnzbd_nzo_qbjf_4y8 in the queue. Was it removed?
2021-02-03 16:36:17 INFO File has now downloaded!

So, I completely moved the comics folder out of my the Sabnzbd directories and created one called: L:\Comics - Mylar. I changed the settings in Mylar and restarted. When I tried a file again, here is what the logs gave me when I tried to download a file:
2021-02-03 16:30:50 WARNING Unable to complete post-processing call due to not finding file in the location provided. [{'status': True, 'nzo_id': 'SABnzbd_nzo_ajwi2yhr', 'queue': {'mode': 'queue', 'search': 'SABnzbd_nzo_ajwi2yhr', 'output': 'json', 'apikey': '977a3ab85e014c068385e0fa245fdbe0'}, 'comicid': '131723', 'issueid': '816304', 'apicall': True}]
2021-02-03 16:30:50 ERROR no file found where it should be @ - is there another script that moves things after completion ?
2021-02-03 16:30:50 INFO found matching completed item in history. Job has a status of Completed

The file downloads and stays in L:\Sabnzbd\Complete. So, yeah, I don't know. Then when I was exploring the config.ini file I noticed something and I'm wondering if it has anything to do with the problem? I doubt it, but I noticed it. Does that sab_to_mylar = False mean anything? sab_directory = None?
sab_category = Complete
sab_priority = Default
sab_to_mylar = False
sab_directory = None
sab_version = 3.1.1
sab_client_post_processing = True

My provider_order appears to be exactly as it should be: provider_order = 0, DDL, 1, NZBgeek. So, I'm happy about that.

I'm attaching the carepackage--hopefully I'm doing it right. I toggled verbose logging on option.

I really appreciate your help and hope it is something that I've totally missed--which is completely possible.

https://mega.nz/file/SQFi2RxK#egAWELtwU ... y59Gkcfy08
User avatar
evilhero
Site Admin
Posts: 2712
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Post Processing - Files not being moved (updated)

Post by evilhero »

So I looked at the carepackage, turns out you turned on debug just in time for DDL to grab the issue you were searching for, and then when you tried it again - the log stopped ;)

I think the problem might be your categories in SABnzbd and how you're thinking they relate to where Mylar is looking. The categories are just tags within SABnzbd - Mylar uses the tag to speed up monitoring by only looking for files being queued/downloaded that match that specific tag.

So basically, the category name in SABnzbd has to match exactly to what you put in Mylar - even if the directory hierarchy doesn't match to what you're thinking. Once Mylar finds the match in your recently downloaded items within SABnzbd, it knows exactly where to look for the finished file so it being in Completed vs Completed/comics doesn't really matter as far as the system goes (if that makes sense).

Note that this also affects Mylar sending files to SABnzbd as well - so if you have the category wrong within Mylar, it will send it to that category regardless. If the category doesn't exist within SABnzbd, it will send it to your base completed folder - which is what it sounds like it's doing, and then the post-processing wouldn't be able to find anything.

But yeah, change the category for sabnzbd to comics in Mylar. Save the config and then try a search again (with debug enabled of course prior). With any luck, you'll see it snatch and post-process accordingly. If not, you'll have to generate a carepackage after the failure to post-process and attach it again and hopefully it'll capture the full output.
zeus163
Posts: 15
Joined: Mon Feb 01, 2021 5:18 am

Re: Post Processing - Files not being moved (updated)

Post by zeus163 »

Awesome sauce! It worked! I swear I originally had that category set to "comics", but setting it back to "comics" and double-checking the .ini file to ensure it was set to that seems to have worked. To be honest, I don't really understand the Sabnzbd categories all that well. It seems like almost everything I grab seems to end up in "Complete" and not under the categories. I'm just glad this works.

A couple of questions if you don't mind. Since I have my provider_order to check DDL first, I've noticed that the files are on GetComics, but the API doesn't seem to find them and sometimes they aren't in Nzbgeek. Does the DDL API only search a few of the download links on GetComics? I'm just curious about that. I can grab them manually. It's not a big deal.

Also, since comics usually come out on Wednesday, how long does it take for pulls to start? It appears that nothing has started for the day yet. I'm just hoping I didn't mess something else up!

Once again. Thanks for all of your help. This is a great tool and I'm just bummed that it took me so long to come to it. Hopefully you didn't mind my questions.
User avatar
evilhero
Site Admin
Posts: 2712
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Post Processing - Files not being moved (updated)

Post by evilhero »

Glad you got it working!

The SABnzbd categories only come into play when the programs you use, utilise them. If you're just grabbing nzb's manually from an nzbprovider and loading them into SABnzbd, they get sent to the default category by default (or your base completed directory). Once the programs use them and and/or you manually change the categories on manual downloads, then your downloads should be put in the locations you specify within SABnbzd.

The DDL aspect doesn't have an API, so it just scrapes the given results page. The problem happens when you have more than one page of results - Mylar doesn't parse more than the 1st page (cause you could have like ~70 pages in results for some queries), so atm at least, if it's not on that first page Mylar will say it can't locate it for DDL. As far as the download links, Mylar will only use the Main Server option - if there isn't one listed, then it won't be able to grab it. This is due the variety of links provided requiring their own client to connect and/or one hasn't been made to do it in Python as of yet.

I say 'atm', as I'm going to be improving the search aspect abit for DDL, so that hopefully you'll be able to get what you're looking for on the 1st page(s) of a query as opposed to something else. I also have a way for it to grab mega & dropapk links too - so that should be coming soon-ish ?

Your best bet to make sure you have the greatest amount of coverage, is to utilise the DDL option with an nzb provider and even with the Experimental option (which searches nzbindex, a free indexer that can have items that aren't on the other providers).

As far as new comics, Wednesday's are the typical day of release you are correct. However, Mylar gets it's information from CV - so until CV updates their information on a given issue, Mylar won't be able to search/download/post-process it as it needs the information therein to be able to verify that the issue is what it says it is. Normally this used to be on Wednesdays, however in the last year-ish the updates occur between Wednesday - Friday.

You'll notice that the weekly pull list will show an issue as Wanted, however if you view the series within Mylar that particular issue doesn't show up in the issue listing (cause CV hasn't updated their info yet). So Mylar then puts all the issues that it knows should be available, but doesn't have any information yet into the Upcoming section of the Wanted tab (which is located near the bottom of the Wanted page). Once an issue has information on CV, and Mylar realises this, then it moves it from the Upcoming section into the Wanted section and searching/downloading/post-processing can be done.

So the good news in all of this is you shouldn't have to do anything Mylar to accommodate this - it's all handled by Mylar. Once the issue information is put on CV, Mylar will recognise this (during it's check that occurs every 4hrs), and update the information it has. At that point the given issue will appear in the series detail listing, which means that it's available to be searched /downloaded. The bad news is that you're kinda at the mercy of CV and their update schedule - and they tend to update by publishers in bulk, with the bigger names going later (ie. Marvel might not be updated until Thursday afternoon). They should start trickling down tho from Wedneday - Friday in most cases (DC of course I think still release some on Mondays).

Hope that helps!
zeus163
Posts: 15
Joined: Mon Feb 01, 2021 5:18 am

Re: Post Processing - Files not being moved (updated)

Post by zeus163 »

Wow. Thanks for taking the time to explain all of this for me. Now I understand the SABnzbd categories a bit better and tt sounds like mine is working correctly then.

I'm glad you know what you are doing with this because I definitely don't. Thanks for devoting your time to make things easier on me. I'm just bummed that it took me so long to discover your awesome tool.

I'll be anxiously awaiting your improved search aspect for DDL. I just need to make sure I read up on upgrading Mylar when it does happen. In the meantime, like you suggested, I added the Experimental option.

It sounds like I just need to hold tight then and let Mylar do it's thing since it is awaiting info from CV. So far, it doesn't seem like I've gotten any of the latest pulls, but now that I have a better understanding of what is going on, I'm just going to wait until it happens. I did notice that about the weekly pull list and the issue showing as wanted. I figured I just needed to be more patient.

Once again, thanks a ton. I really appreciate it. I'm sure I'll have more questions going forward, but I really do appreciate you taking the time to explain it to me.

Do you take donations other than Bitcoin?
User avatar
evilhero
Site Admin
Posts: 2712
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Post Processing - Files not being moved (updated)

Post by evilhero »

Hey, we all have to start somewhere - everyone went thru the same learning curve at some point, as you're not born with the knowledge.

As far as upgrading Mylar, as long as you have a normal type of installation (meaning you either git cloned / downloaded the source ) you should be able to do it all within the Mylar GUI when the update notification pops up (docker obviously has it's own update method). We try to recommend git installations at a minimum as they're generally easier as far as the upgrade path goes, but nothing wrong with docker, and I believe source installations should update fine as well.

The donation options available are listed on the 1st tab in the configuration (so BC and PP). Not necessary obviously, but many thanks if you choose to support the project.
zeus163
Posts: 15
Joined: Mon Feb 01, 2021 5:18 am

Re: Post Processing - Files not being moved (updated)

Post by zeus163 »

Ha! I'm such a nut. I didn't realize the "Donate" button was a link. I thought it was basically saying donate by clicking on the Bitcoin button/link. I know it's not necessary, but this will be super helpful for me in the long run. I do wish CV was a little faster, but that's not on you. I just needed to be more patient and figure it out.

Everything worked exactly like you said it would, all my weekly pulls came through and they ended up exactly where they were supposed to go. I just need to load them on my tablet now for the week. Awesome sauce.

I've now started the cumbersome process of indexing/collecting all my comics (from many different hard drives and places). But I love the interface and how I get information about the series and names of the issues. This project will take me some time, but at least it will start to help me organize my folders and folders of 0-Days from newsgroups.

I'm hoping you can help me with an issue that I am having though and it deals with TPB/Volumes. I have several TPB/Volumes I find on CV and then add them to Mylar. I then copy the version I have over to the folder Mylar created. When I have Mylar recheck the folder, it doesn't find/recognize the file. I think this might have to do with the name of the file. I'm just guessing though. Some of these files and have worked and some don't. For example, I added Star Wars: Poe Dameron: The Gathering Storm (2017) into Mylar and added it. I then put Star Wars - Poe Dameron v02 - The Gathering Storm (2017) GetComics.INFO.cbr into the folder that Mylar created and did a recheck. It didn't find it. So, a lot of my folders for these end up 0/1. Do you have any suggestions? Or can you help me so that they actually don't show as if I have missing issues?

I think I initially tried the git cloned, but had issues because of the version of Python I was using. I think using a newer version of Python was causing that and I ended up copying over the downloaded master source and ran it again after downgrading Python to 3.8.1. That worked. So, it sounds like I should just be able to upgrade from the GUI then. Fingers crossed. I won't have to do anything to any files, hopefully.

Once again. Thanks a ton. This was supposed to be just a quick thanks email and asking for a little tip on the TPB issues I'm having and it got longer than expected.
User avatar
evilhero
Site Admin
Posts: 2712
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Post Processing - Files not being moved (updated)

Post by evilhero »

The problem with TPB's is that they never, and I mean never, follow any kind of consistent naming scheme - especially when comparing to what scanners name them, and what CV names them.

In the example you posted, the reason why it's not matching is due to the actual naming of the issue vs what Mylar thinks the name is. So if the series is named Star Wars: Poe Dameron: The Gathering Storm (2017) and you try to get it to scan in Star Wars - Poe Dameron v02 - The Gathering Storm (2017) GetComics.INFO.cbr, the 2 aren't identical - not because of the GetComics.INFO at the end of the file, nor because of the use of '-' instead of ':'.

It's due to the placement of the v02 believe it or not. So Mylar is basically looking for Star Wars - Poe Dameron - The Gathering Storm, as that's what CV has it as. The filename tho is named as if it were Star Wars - Poe Dameron v02 with The Gathering Storm as the name of the TPB colleciton. You'll notice that the issue within Mylar (and thereby from CV) is named as issue #1, with an issue name of Volume 2.

So yeah, it gets confusing trying to mish-mash everything together so they can work. But you have options:
  • changing the file: rename the file to Star Wars - Poe Dameron - The Gathering Storm and that should check in fine.Leaving the volume as is within Mylar should be fine. Tested using ReCheck Files and it worked.
  • leaving the file as is: add the Star Wars - Poe Dameron to the Alternate Search Field in the Edit Settings tab within the series - and make sure the volume is set to v2. Tested using ReCheck Files and it worked.
Now the last one might not work as intended if you're post-processing issues against that particular issue or if you also happen to have the actual print issue series on your watchlist, as if you were to just name it Star Wars - Poe Dameron with no volume indentifer (within Mylar) it might get picked up as the normal print run series of issues, which would cause more confusion, especially if there's a v2 of that particular series that's not a TPB (this is if you were post-processing the issue - ReCheck Files/Refresh Series should just say it couldn't find any files)

TLDR; Basically try to go from the naming convention of CV if at all possible, failing that you can use the Alternate Search Name field to tell Mylar what the series should be named / looked for.
Post Reply