ReleaseDate Processing in manual postprocessing

Post any problems / bugs / issues that are Mylar-related in here.
Post Reply
bmfrosty
Posts: 14
Joined: Wed May 07, 2014 2:12 am

ReleaseDate Processing in manual postprocessing

Post by bmfrosty »

I have a file called: Animal Man 004 (2012) (digital) (Megan-Empire).cbr

I was trying to work out why it wasn't being processed in, so I started adding extra logging to filechecker.py, and after I worked out how it operated, I moved the file to it's own directory to scan. Here's the output I get:

12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] ***** Starting Compare /mnt/user/Primary/test
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] ***** Looking for comic: Animal Man
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] Looking in: /mnt/user/Primary/test
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] there are 1 files.
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] numberinseries: False
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] decimalinseries: False
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] subnm len : 7
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] filename is: Animal Man 004 (2012) (digital) (Megan-Empire).cbr
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] subname no brackets: Animal Man 004
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] watchcomic:Animal Man ..comparing to found file: Animal Man 004
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] Animal Man - watchlist match on : /mnt/user/Primary/test/Animal Man 004 (2012) (digital) (Megan-Empire).cbr
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] we should remove 0 characters
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] nonocount [0] cchk [Animal Man] length [10]
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] final jtd_len to prune [10]
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] before title removed from FILENAME [Animal Man 004 (2012) (digital) (Megan-Empire).cbr]
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] after title removed from FILENAME [ 004 (2012) (digital) (Megan-Empire).cbr]
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] creating just the digits using SUBNAME, pruning first [10] chars from [Animal Man 004]
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] after title removed from SUBNAME [004]
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] JUSTTHEDIGITS [004]
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] final justthedigits [004]
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] cchk = animal man
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] modwatchcomic = animal man
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] subname = animal man 004
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] SeriesYear: 2011
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] IssueTotal: 30
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] Comic Volume: None
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] volume detected: None
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] 0 Bracket Word: Animal Man 004
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] 1 Bracket Word: 2012
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] year detected: 2012
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] 2012 is within the series range of 2011-2014
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] Matched on versions, not on year - continuing.
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] poss_alpha: ['004']
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] lenalpha: 3
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] sub_removed: Animal Man 004
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] split_sub: ['Animal', 'Man']
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] split_mod: ['Animal', 'Man']
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : Man comparing to Man
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] word match exact. 1/2
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : Animal comparing to Animal
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] word match exact. 2/2
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] scnt:2
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] split_mod length:2
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] we got 100 percent.
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] this should be considered an exact match.Justthedigits:004
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] you have a total of 1 Animal Man comics
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : [FILECHECKER] ***** Finished Checking Animal Man comics /mnt/user/Primary/test
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : 2011-12-07 is before the issue year of 2012 that was discovered in the filename
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : 2012 comparing to 2012rechecking by month-check versus year.
12-May-2014 21:31:32 - DEBUG :: CP Server Thread-5 : Incorrect series - not populating..continuing post-processing

finally, after filechecker.py determines that the file is good, it seems to pass it back to postprocessor.py, which determines that it's the wrong year. I checked the database for the issue information, and I find that yes, I already have the issue, but it has two dates, one of which matches the what we're checking against, and the other which is something else - I'm guessing the issue date since the cover shows "Feb 2012".

It reads that the scanner labeled the issue with the cover date, and the postprocessor was checking the actual release date, and if I'm reading the logs correctly rejecting the file based on that mismatch.

And in addition to this, when I look in the GUI for the issue, it lists it as 2012-02-01.

305593|Animal Man|the hunt, part four: the rot|4||Downloaded||42600||2011-12-07|Animal Man 04 (2011).cbr|2012-02-01|4000|18346316||

I suspect that if the issue is checked to be in "either" year, and given a pass if it matches either, it might have moved forward.

Hope this helps,

-Ben
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: ReleaseDate Processing in manual postprocessing

Post by evilhero »

Yep, you pretty much figure out how it works ;)

The 2012-02-01 is the publication date, which is what is shown in the GUI as well.

The 2011-12-07 is the issue date (or store date).

I actually already came across this awhile ago, and thought I had pushed up the commit - but it would appear I didn't obviously.

There's 2 key things missing from the algorithims:

Code: Select all

                                         if int(issuechk['IssueDate'][5:7]) == 11 or issuechk['IssueDate'][5:7] == 12:
-                                            issyr = int(issuechk['IssueDate'][:4])
+                                            issyr = int(issuechk['IssueDate'][:4]) + 1
                                         elif int(issuechk['IssueDate'][5:7]) == 1 or int(issuechk['IssueDate'][5:7]) == 2:
-                                            issyr = int(issuechk['IssueDate'][:4])
+                                            issyr = int(issuechk['IssueDate'][:4]) - 1
Basically if the issuedate is in the last part of the year, or the first part of the subsequent year to either increment the year or decrease it accordingly in order to attempt to accomodate series' that have incorrect years tagged.

So it does the comparison of checking the found issue year (2012) against the issue date (2011). It's incorrect so it gets passed as being a False. So the line where it says '2012 comparing to 2012rechecking by month-check versus year',(was actually checking 2011 against 2012) with the new above logic in place, would actually be '2012 (2011+1 since it 's a November release) comparing to 2012...'

I should be able to push this out...I have a whack of things in the next commit, but they're all intertwined and I'm kinda abit hesitant to throw them all out since it could break for a few people - but I might not have a choice.
bmfrosty
Posts: 14
Joined: Wed May 07, 2014 2:12 am

Re: ReleaseDate Processing in manual postprocessing

Post by bmfrosty »

evilhero wrote:I should be able to push this out...I have a whack of things in the next commit, but they're all intertwined and I'm kinda abit hesitant to throw them all out since it could break for a few people - but I might not have a choice.
Ah. Will this be the refactor to get rid of the database locking issue? I'm *very* looking forward to that. That should stop a lot of my manual manipulation.
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: ReleaseDate Processing in manual postprocessing

Post by evilhero »

Yep, you got it - a queue for the all database writes, plus some new scheduling and thread controls. Coupled with an updated logger that will now log traceback errors (so that when run in daemon mode the errors will get captured). That and being able to dictate the provider searching order are all big updates. There's a bunch of other things as well, one being the error you mentioned in this thread, but yeah, it's all intertwined.

I'm not 100% happy with the queue-control, and it's going to need some more fine-tuning - but I have had it running on mine for over a week now, and I've been fixing the errors as they pop up, so hopefully I've gotten most of them.
bmfrosty
Posts: 14
Joined: Wed May 07, 2014 2:12 am

Re: ReleaseDate Processing in manual postprocessing

Post by bmfrosty »

evilhero wrote:Yep, you got it - a queue for the all database writes, plus some new scheduling and thread controls. Coupled with an updated logger that will now log traceback errors (so that when run in daemon mode the errors will get captured). That and being able to dictate the provider searching order are all big updates. There's a bunch of other things as well, one being the error you mentioned in this thread, but yeah, it's all intertwined.

I'm not 100% happy with the queue-control, and it's going to need some more fine-tuning - but I have had it running on mine for over a week now, and I've been fixing the errors as they pop up, so hopefully I've gotten most of them.
I should probably wait for that before I start exploring the source again. I don't program in python, but I'm slowly getting better at reading it.
Post Reply