Cache and Config Issue

Post any problems / bugs / issues that are Mylar-related in here.
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Cache and Config Issue

Post by evilhero »

I'm looking at this atm - well trying to while it's quiet at least here...

One question - what are the permissions on the torrent folder you're trying to manually post-process from?

The reason I'm asking is it looks like it's failing right at the start of the tagging, which would mean it's either failing to create the temporary directory in your cache folder (which you said it was full, so totally understandable), however it's trying to move the file from the torrent folder into the cache folder - it might not be able to compete the move due to not having permissions to the file (or it might be locked in usage by the torrent client I guess, which is totally plausible given that it's windows). If you're wanting to seed the torrents, the 'copy' option is probably what you're wanting - atho that might not solve this cache-fillup problem as even if it was trying to move the file it shouldn't be redownloading if it's in a Snatched status.

If I pass you a cmtagmylar.py file to replace yours with - would you be able to run a manual post-process against the torrent folder you're monitoring (or whatever, just something that has the files you're trying to pp and it's not working)? It'll have more debugging information in it so it might help in figuring out what's going on and where it's messing up...
User avatar
MIghtyThor460
Posts: 32
Joined: Mon Apr 20, 2015 1:26 pm

Re: Cache and Config Issue

Post by MIghtyThor460 »

Well I did just realize that Deluge could be causing the post process failures. Once I cleared them out of Deluge they processed through fine. That still doesn't explain why Mylar would continually create the temp files though... If you wan't to send through a custom .py feel free! I'd be gladd to help solve that mystery!
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Cache and Config Issue

Post by evilhero »

Ok, here's the .py file that will log more details about each process in the creation of the temporary directory & moving/copying the files, as well as getting the exact error that's occuring (I'm hoping).

https://gist.github.com/evilhero/8b242f ... 0a3f46b63a

Just replace the cmtagmylar.py file you have with this one (the one located in your mylar/ subdirectory) - and also delete the .pyc file as that will cause problems when loading a new file (obviously make sure Mylar isn't running when you do this).

Then you can just do a manual pp of something even, just to get the log to see if it errors out. I'm wondering more about this that the shutil is trying to enforce some permissions after the copy/move and it's not letting it due to either the client locking it (deluge), or windows itself even. It could be anything really, but those two things made me thing abit....

Let me know how it goes, and thanks for testing ;)
User avatar
MIghtyThor460
Posts: 32
Joined: Mon Apr 20, 2015 1:26 pm

Re: Cache and Config Issue

Post by MIghtyThor460 »

Just saw this now. I will snatch a new file in the morning and try this with the torrent still seeding and without. I know I can't delete a seeded file from another application if it has any active connection associated with it. I'm wondering if that somehow translates to the temp file made in the cache. When moving, is the temp file a brand new file or a symbolic link?
User avatar
MIghtyThor460
Posts: 32
Joined: Mon Apr 20, 2015 1:26 pm

Re: Cache and Config Issue

Post by MIghtyThor460 »

So I haven't been able to get it to repeat the problem. It's like it suddenly stopped. If I notice it happen again I will update with the log.

So this is an example of it working:
2017-05-16 12:14:17 DEBUG [MANUAL META-TAGGING] Tidying up. Deleting temporary cache directory : c:\Mylar\cache\mylar_hxejr4
2017-05-16 12:14:17 INFO [MANUAL META-TAGGING] Sucessfully wrote metadata to .cbz (Mighty Thor 12 (2016).cbz) - Continuing..
2017-05-16 12:14:16 INFO [META-TAGGER][COMIC-TAGGER] Successfully wrote ComicRack tagging [c:\Mylar\cache\mylar_hxejr4\Mighty Thor 12 (2016).cbz]
2017-05-16 12:14:08 DEBUG [META-TAGGER] Absolute path to script: c:\Python27\python.exe
2017-05-16 12:14:08 DEBUG [META-TAGGER] Executing command: ['c:\\Python27\\python.exe', 'c:\\Mylar\\comictagger.py', '-s', '-m', 'volume=2', '--cv-api-key', u'REDACTED', '-o', '--id', u'553961', '--type', 'cr', u'c:\\Mylar\\cache\\mylar_hxejr4\\Mighty Thor 12 (2016).cbz']
2017-05-16 12:14:08 DEBUG [META-TAGGER] Enabling ComicTagger script: ['c:\\Python27\\python.exe', 'c:\\Mylar\\comictagger.py'] with options: ['-s', '-m', 'volume=2', '--cv-api-key', u'REDACTED', '-o', '--id', u'553961', '--type', 'cr', u'c:\\Mylar\\cache\\mylar_hxejr4\\Mighty Thor 12 (2016).cbz']
2017-05-16 12:14:08 INFO [META-TAGGER] ComicRack tagging meta-tagging processing started.
2017-05-16 12:14:08 DEBUG [META-TAGGER][COMIC-TAGGER][CBR-TO-CBZ] New filename: c:\Mylar\cache\mylar_hxejr4\Mighty Thor 12 (2016).cbz
2017-05-16 12:14:08 DEBUG [META-TAGGER][COMIC-TAGGER] : Archive exported successfully to: Mighty Thor 12 (2016).cbz (Original deleted)
2017-05-16 12:14:05 DEBUG [META-TAGGER] Absolute path to script: c:\Python27\python.exe
2017-05-16 12:14:05 DEBUG [META-TAGGER] Executing command: ['c:\\Python27\\python.exe', 'c:\\Mylar\\comictagger.py', '-e', '--delete-rar', u'c:\\Mylar\\cache\\mylar_hxejr4\\Mighty Thor 12 (2016).cbr']
2017-05-16 12:14:05 DEBUG [META-TAGGER] Enabling ComicTagger script: ['c:\\Python27\\python.exe', 'c:\\Mylar\\comictagger.py'] with options: ['-e', '--delete-rar', u'c:\\Mylar\\cache\\mylar_hxejr4\\Mighty Thor 12 (2016).cbr']
2017-05-16 12:14:05 DEBUG [META-TAGGER] CR Tagging enabled.
2017-05-16 12:14:05 DEBUG [META-TAGGER] ComicTagger 1.20.1 [ninjas.walk.alone / SHURIKEN] being used - using personal ComicVine API key supplied via mylar.
2017-05-16 12:14:05 DEBUG [META-TAGGER] Running the ComicTagger Add-on for Mylar
2017-05-16 12:14:05 DEBUG [META-TAGGER] comicpath : c:\Mylar\cache\mylar_hxejr4
2017-05-16 12:14:05 DEBUG [META-TAGGER] sabnzbdscriptpath : c:\Mylar
2017-05-16 12:14:05 DEBUG [META-TAGGER] downloadpath : \\WORLDMIND\Comics\Marvel\Mighty Thor (2015)
2017-05-16 12:14:05 DEBUG [META-TAGGER] scriptname : Mylar.py
2017-05-16 12:14:05 DEBUG [META-TAGGER] Paths / Locations:
2017-05-16 12:14:04 DEBUG [META-TAGGER] Filename is : \\WORLDMIND\Comics\Marvel/Mighty Thor (2015)\Mighty Thor 12 (2016).cbr
2017-05-16 12:14:04 DEBUG ComicTagger Path location for internal comictagger.py set to : c:\Mylar\comictagger.py
2017-05-16 12:14:04 DEBUG [META-TAGGER] dirName:\\WORLDMIND\Comics\Marvel/Mighty Thor (2015)
User avatar
MIghtyThor460
Posts: 32
Joined: Mon Apr 20, 2015 1:26 pm

Re: Cache and Config Issue

Post by MIghtyThor460 »

Success! (Failure?)
2017-05-16 14:32:25 INFO [POST-PROCESSING] Manual post-processing completed for 1 issues.
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] Post-Processing ABORTED.
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] Failed to move D:\ServerFolders\Documents\Torrents\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbr - check directories and manually re-run.
2017-05-16 14:32:25 DEBUG [POST-PROCESSING][move] D:\ServerFolders\Documents\Torrents\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbr ... to ... D:\ServerFolders\Comics\DC Comics\Deathstroke (2016)\Deathstroke 18 (2017).cbr
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] odir src : D:\ServerFolders\Documents\Torrents\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbr
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] Renaming D:\ServerFolders\Documents\Torrents\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbr
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] Destination: D:\ServerFolders\Comics\DC Comics\Deathstroke (2016)\Deathstroke 18 (2017).cbr
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] Source: D:\ServerFolders\Documents\Torrents\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbr
2017-05-16 14:32:25 INFO [POST-PROCESSING][DIRECTORY-CHECK] Found comic directory: D:\ServerFolders\Comics\DC Comics\Deathstroke (2016)
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] New Filename: Deathstroke 18 (2017)
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] Original Extension: .cbr
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] Original Filename: Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbr
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] ext:.cbr
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] ofilename: Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbr
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] odir: D:\ServerFolders\Documents\Torrents
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] ofilename:Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbr
2017-05-16 14:32:25 DEBUG [POST-PROCESSING] Unable to write metadata successfully - check mylar.log file. Attempting to continue without tagging...
2017-05-16 14:32:25 WARNING [META-TAGGER] Unable to create temporary directory to perform meta-tagging. Processing without metatagging.
2017-05-16 14:32:24 DEBUG [META-TAGGER] Filename is : D:\ServerFolders\Documents\Torrents\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbr
2017-05-16 14:32:24 DEBUG ComicTagger Path location for internal comictagger.py set to : c:\Mylar\comictagger.py
2017-05-16 14:32:24 DEBUG [META-TAGGER] dirName:D:\ServerFolders\Documents\Torrents
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] Metatagging enabled - proceeding...
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] New format: $Series $Issue ($Year)
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] Not an annual - removing from filename parameters
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] Comic Version: v4
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] Comic Location: D:\ServerFolders\Comics\DC Comics\Deathstroke (2016)
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] Year: 2016
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] Series: Deathstroke
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] Publisher: DC Comics
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] Issue Year : 2017
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] Pretty Comic Issue is : 18
2017-05-16 14:32:24 DEBUG Zero level supplement set to none.Issue will be set as : 18
2017-05-16 14:32:24 DEBUG issue detected greater than 10, but less than 100
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] Zero Suppression set to : none
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] issuenumOG: 18
2017-05-16 14:32:24 DEBUG [POST-PROCESSING] issueid: 594085
User avatar
MIghtyThor460
Posts: 32
Joined: Mon Apr 20, 2015 1:26 pm

Re: Cache and Config Issue

Post by MIghtyThor460 »

So a few minutes later this successfully completed.
2017-05-16 14:37:30 DEBUG [POST-PROCESSING] Tidying up. Deleting temporary cache directory : c:\Mylar\cache\mylar_gb3gbp
2017-05-16 14:37:30 DEBUG odir: c:\Mylar\cache\mylar_gb3gbp [self.nzb_folder: D:\ServerFolders\Documents\Torrents]
2017-05-16 14:37:30 DEBUG File Option: move [META-ENABLED: True]
2017-05-16 14:37:30 INFO [POST-PROCESSING] move successful to : D:\ServerFolders\Comics\DC Comics\Deathstroke (2016)\Deathstroke 18 (2017).cbz
2017-05-16 14:37:29 DEBUG [POST-PROCESSING][move] c:\Mylar\cache\mylar_gb3gbp\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz ... to ... D:\ServerFolders\Comics\DC Comics\Deathstroke (2016)\Deathstroke 18 (2017).cbz
2017-05-16 14:37:29 DEBUG [POST-PROCESSING] odir src : c:\Mylar\cache\mylar_gb3gbp\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz
2017-05-16 14:37:29 DEBUG [POST-PROCESSING] Renaming c:\Mylar\cache\mylar_gb3gbp\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz
2017-05-16 14:37:29 DEBUG [POST-PROCESSING] Destination: D:\ServerFolders\Comics\DC Comics\Deathstroke (2016)\Deathstroke 18 (2017).cbz
2017-05-16 14:37:29 DEBUG [POST-PROCESSING] Source: c:\Mylar\cache\mylar_gb3gbp\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz
2017-05-16 14:37:29 INFO [POST-PROCESSING][DIRECTORY-CHECK] Found comic directory: D:\ServerFolders\Comics\DC Comics\Deathstroke (2016)
2017-05-16 14:37:29 DEBUG [POST-PROCESSING] New Filename: Deathstroke 18 (2017)
2017-05-16 14:37:29 DEBUG [POST-PROCESSING] Original Extension: .cbz
2017-05-16 14:37:29 DEBUG [POST-PROCESSING] Original Filename: Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz
2017-05-16 14:37:29 DEBUG [POST-PROCESSING] ext:.cbz
2017-05-16 14:37:29 DEBUG [POST-PROCESSING] ofilename: Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz
2017-05-16 14:37:29 DEBUG [POST-PROCESSING] odir: c:\Mylar\cache\mylar_gb3gbp
2017-05-16 14:37:29 DEBUG [POST-PROCESSING] ofilename:Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz
2017-05-16 14:37:29 INFO [POST-PROCESSING] Sucessfully wrote metadata to .cbz (Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz) - Continuing..
2017-05-16 14:37:29 INFO [META-TAGGER][COMIC-TAGGER] Successfully wrote ComicRack tagging [c:\Mylar\cache\mylar_gb3gbp\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz]
2017-05-16 14:37:21 DEBUG [META-TAGGER] Absolute path to script: c:\Python27\python.exe
2017-05-16 14:37:21 DEBUG [META-TAGGER] Executing command: ['c:\\Python27\\python.exe', 'c:\\Mylar\\comictagger.py', '-s', '-m', 'volume=4', '--cv-api-key', u'REDACTED', '-o', '--id', u'594085', '--type', 'cr', u'c:\\Mylar\\cache\\mylar_gb3gbp\\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz']
2017-05-16 14:37:21 DEBUG [META-TAGGER] Enabling ComicTagger script: ['c:\\Python27\\python.exe', 'c:\\Mylar\\comictagger.py'] with options: ['-s', '-m', 'volume=4', '--cv-api-key', u'REDACTED', '-o', '--id', u'594085', '--type', 'cr', u'c:\\Mylar\\cache\\mylar_gb3gbp\\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz']
2017-05-16 14:37:21 INFO [META-TAGGER] ComicRack tagging meta-tagging processing started.
2017-05-16 14:37:21 DEBUG [META-TAGGER][COMIC-TAGGER][CBR-TO-CBZ] New filename: c:\Mylar\cache\mylar_gb3gbp\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz
2017-05-16 14:37:21 DEBUG [META-TAGGER][COMIC-TAGGER] : Archive exported successfully to: Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbz (Original deleted)
2017-05-16 14:37:19 DEBUG [META-TAGGER] Absolute path to script: c:\Python27\python.exe
2017-05-16 14:37:19 DEBUG [META-TAGGER] Executing command: ['c:\\Python27\\python.exe', 'c:\\Mylar\\comictagger.py', '-e', '--delete-rar', u'c:\\Mylar\\cache\\mylar_gb3gbp\\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbr']
2017-05-16 14:37:19 DEBUG [META-TAGGER] Enabling ComicTagger script: ['c:\\Python27\\python.exe', 'c:\\Mylar\\comictagger.py'] with options: ['-e', '--delete-rar', u'c:\\Mylar\\cache\\mylar_gb3gbp\\Deathstroke 018 (2017) (2 covers) (digital) (Minutemen-Midas).cbr']
2017-05-16 14:37:19 DEBUG [META-TAGGER] CR Tagging enabled.
2017-05-16 14:37:19 DEBUG [META-TAGGER] ComicTagger 1.20.1 [ninjas.walk.alone / SHURIKEN] being used - using personal ComicVine API key supplied via mylar.
2017-05-16 14:37:19 DEBUG [META-TAGGER] Running the ComicTagger Add-on for Mylar
2017-05-16 14:37:19 DEBUG [META-TAGGER] comicpath : c:\Mylar\cache\mylar_gb3gbp
2017-05-16 14:37:19 DEBUG [META-TAGGER] sabnzbdscriptpath : c:\Mylar
2017-05-16 14:37:19 DEBUG [META-TAGGER] downloadpath : D:\ServerFolders\Documents\Torrents
It looks like my hypothesis was right. When Deluge is actively using the snatched file it messes with the tagging process and it aborts before the cleanup phase. Is the tagger supposed to delete on abort?
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Cache and Config Issue

Post by evilhero »

Well you can't move a file when it's in use.. That's a limitation of windows. But it looks like you have the post-processing action set to move the files into your comic location instead of perhaps it should be copy (so that you can continue seeding).

As you can't move the file, tag it and then continue to seed it (as the metatagging will change the hash of the file), that would cause an initial error on the move due to the lock by deluge - I'm not quite sure of the options you have enabled however.

And yes it should be cleaning up the cache folder stuff on a failed tagging, so I'll have to figure that part out as if move is enabled deleting the cache will delete the file, whereas it could (in theory) move the file back to it's original location and continue post-processing without metatagging.
User avatar
MIghtyThor460
Posts: 32
Joined: Mon Apr 20, 2015 1:26 pm

Re: Cache and Config Issue

Post by MIghtyThor460 »

I have already changed my settings to Copy. I put them back to move to replicate the issue I was having. Would you like me to paste in my configure settings when I get home? And thank you again for helping to dig into this! This is above and beyond!
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Cache and Config Issue

Post by evilhero »

I don't think there's a need to at this point - I did some quick reading last night, and it looks like this is actually normal behaviour for deluge or any torrent client (in windows at least).

Apparently the 'fix' if you will is to pause the torrent upon completion, copy the file and then unpause it so that things can continue from both fronts equally without any problems. Unfortunately the monitor / pause stuff isn't built into Mylar fully so I'll have to code that stuff so that once it's completed, it will pause, copy, and then unpause for continued seeding. I'll also look into how Mylar can use the original file when post-processing fails during meta-tagging so that cleanup happens regardless due to the failure, but normal post-processing of the file can continue just without metatagging...

Thanks for all the info ;)
Post Reply