Not finding issues

[Aimhere]

Hi,

Mylar master, commit 8082a61f17100f022a3daa754c9f7a764ecf7a38, on Synology DS413j NAS (OS: DSM 4.3).

I recently added a bunch of comics I wanted to start reading... first, I would search for stuff with the search box at the top of the page, then click the "add this comic" link. Mylar would load up the series info as expected. But as the days and weeks have gone by, Mylar has barely found any issues of these comics (no issues at all in most cases), even though an interactive search at my indexers' websites turns them up just fine.

For example, the Marvel title "S.H.I.E.L.D. (2015)", has 10 issues available, but Mylar hasn't found any. If I go to the comic details page and try to start a manual search for, say, issue 1, I get the following in the log:

Code: Select all

10-May-2016 19:36:09 - INFO    :: Thread-13 : Initiating manual search for S.H.I.E.L.D. issue: 1
10-May-2016 19:36:09 - INFO    :: Thread-13 : Issue Title given as : active mission: perfect bullets
10-May-2016 19:36:09 - INFO    :: Thread-13 : search provider order is ['Experimental', 'newznab:Usenet-Crawler', 'newznab:OZNZB.com']
10-May-2016 19:36:09 - INFO    :: Thread-13 : Shhh be very quiet...I'm looking for S.H.I.E.L.D. issue: 1 (2015) using experimental [RSS]
10-May-2016 19:36:22 - INFO    :: Thread-13 : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
10-May-2016 19:36:30 - INFO    :: Thread-13 : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
10-May-2016 19:36:30 - INFO    :: Thread-13 : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
10-May-2016 19:36:30 - INFO    :: Thread-13 : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
10-May-2016 19:36:30 - INFO    :: Thread-13 : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
10-May-2016 19:36:30 - INFO    :: Thread-13 : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
10-May-2016 19:36:30 - INFO    :: Thread-13 : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
10-May-2016 19:36:32 - INFO    :: Thread-13 : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

And that's all. After a good 15-minute wait, there are no further messages regarding the search. I am using three search providers (shown above), so I don't know why it never progressed beyond 'Experimental'.

So I tried another comic, "Hulk (2014)", issue 1. This time, a manual search found the issue almost immediately, and downloaded it as expected:

Code: Select all

10-May-2016 20:03:25 - INFO    :: Thread-14 : Initiating manual search for Hulk issue: 1
10-May-2016 20:03:25 - INFO    :: Thread-14 : Issue Title given as : who shot the hulk part 1
10-May-2016 20:03:25 - INFO    :: Thread-14 : search provider order is ['Experimental', 'newznab:Usenet-Crawler', 'newznab:OZNZB.com']
10-May-2016 20:03:26 - INFO    :: Thread-14 : Shhh be very quiet...I'm looking for Hulk issue: 1 (2014) using experimental [RSS]
10-May-2016 20:04:01 - INFO    :: Thread-14 : Could not find Issue 1 of Hulk(2014) using experimental [rss]
10-May-2016 20:04:01 - INFO    :: Thread-14 : Shhh be very quiet...I'm looking for Hulk issue: 1 (2014) using Usenet-Crawler(newznab) [RSS]
10-May-2016 20:04:04 - INFO    :: Thread-14 : Could not find Issue 1 of Hulk(2014) using Usenet-Crawler [rss]
10-May-2016 20:04:04 - INFO    :: Thread-14 : Shhh be very quiet...I'm looking for Hulk issue: 1 (2014) using OZNZB.com(newznab) [RSS]
10-May-2016 20:04:07 - INFO    :: Thread-14 : Could not find Issue 1 of Hulk(2014) using OZNZB.com [rss]
10-May-2016 20:04:07 - INFO    :: Thread-14 : Shhh be very quiet...I'm looking for Hulk issue: 1 (2014) using experimental
10-May-2016 20:04:21 - INFO    :: Thread-14 : Could not find Issue 1 of Hulk(2014) using experimental [api]
10-May-2016 20:04:21 - INFO    :: Thread-14 : Shhh be very quiet...I'm looking for Hulk issue: 1 (2014) using Usenet-Crawler (newznab)
10-May-2016 20:04:21 - INFO    :: Thread-14 : pausing for 60 seconds before continuing to avoid hammering
10-May-2016 20:05:26 - INFO    :: Thread-14 : status code: 200
10-May-2016 20:05:55 - INFO    :: Thread-14 : Download URL: https://www.usenet-crawler.com/api?apikey=6213e8509f089c27665d248aa1edf45a&t=get&id=5783e5ec4bc7a24487617cf6920f454d [VerifySSL:True]
10-May-2016 20:06:00 - INFO    :: Thread-14 : prov  : Usenet-Crawler (newznab)[5783e5ec4bc7a24487617cf6920f454d]
10-May-2016 20:06:00 - INFO    :: Thread-14 : [FAILED_DOWNLOAD_CHECKER] Successfully marked this download as Good for downloadable content
10-May-2016 20:06:00 - INFO    :: Thread-14 : Found Hulk (2014) issue: 1 using Usenet-Crawler (newznab)
10-May-2016 20:06:01 - INFO    :: Thread-14 : Successfully sent nzb to NZBGet!
10-May-2016 20:06:01 - INFO    :: Thread-14 : setting the nzbid for this download grabbed by Usenet-Crawler (newznab) in the nzblog to : 5783e5ec4bc7a24487617cf6920f454d
10-May-2016 20:06:01 - INFO    :: Thread-14 : passing to updater.
10-May-2016 20:06:02 - INFO    :: Thread-14 : [UPDATER] Updating status to snatched
10-May-2016 20:06:02 - INFO    :: Thread-14 : [UPDATER] Updated the status (Snatched) complete for Hulk Issue: 1
10-May-2016 20:06:23 - INFO    :: CP Server Thread-8 : ComicRN.py version: 1.01 -- autoProcessComics.py version: 1.0
10-May-2016 20:06:23 - INFO    :: CP Server Thread-8 : Starting postprocessing for : Hulk.001.2014.c2c.Digi-Hybrid.TLK-EMPIRE-HD
10-May-2016 20:06:23 - INFO    :: Post-Processing : [POST-PROCESSING] issuenzb found.
10-May-2016 20:06:23 - INFO    :: Post-Processing : [DUPECHECK] Duplicate check for /volume1/Downloads/dest/Comics/Hulk.001.2014.c2c.Digi-Hybrid.TLK-EMPIRE-HD
10-May-2016 20:06:23 - INFO    :: Post-Processing : [DUPECHECK] Duplication detection returned no hits. This is not a duplicate of anything that I have scanned in as of yet.
10-May-2016 20:06:24 - INFO    :: Post-Processing : [POST-PROCESSING] [1/1] Starting Post-Processing for Hulk issue: 1
10-May-2016 20:06:24 - INFO    :: Post-Processing : [POST-PROCESSING][DIRECTORY-CHECK] Found comic directory: /volume1/Comics/Marvel/Hulk (2014)
10-May-2016 20:06:24 - INFO    :: Post-Processing : [POST-PROCESSING][UPDATER] Setting status to Downloaded in history.
10-May-2016 20:06:27 - INFO    :: Post-Processing : [POST-PROCESSING][UPDATER] Updating Status (Downloaded) now complete for Hulk issue: 1
10-May-2016 20:06:27 - INFO    :: Post-Processing : [POST-PROCESSING][FILE-RESCAN] Now checking files for Hulk (2014) in /volume1/Comics/Marvel/Hulk (2014)
10-May-2016 20:06:27 - INFO    :: Post-Processing : [POST-PROCESSING][FILE-RESCAN] Now checking files for Hulk (2014) in :Hulk (2014)
10-May-2016 20:06:30 - INFO    :: Post-Processing : [POST-PROCESSING][FILE-RESCAN] Updated the status of 13 issues for Hulk (2014) that were not found.
10-May-2016 20:06:30 - INFO    :: Post-Processing : [POST-PROCESSING][FILE-RESCAN] Total files located: 4
10-May-2016 20:06:30 - ERROR   :: Post-Processing : Could not change permissions : /volume1/Comics/Marvel/Hulk (2014). Exiting...
10-May-2016 20:06:30 - INFO    :: Post-Processing : [POST-PROCESSING][FILE-RESCAN] I have physically found 4 issues, ignored 0 issues, snatched 0 issues, and accounted for 0 in an Archived state [ Total Issue Count: 4 / 17 ]
10-May-2016 20:06:30 - INFO    :: Post-Processing : [POST-PROCESSING] Post-Processing completed for: Hulk issue: 1

But this series has been in my library for WEEKS, and Mylar had only downloaded two other issues (4 and 10) on its own. Issue 1 has been on Usenet for a couple of years! In fact, issues 1-16 are all available.

These are hardly the only comics I'm having troubles with. Sure, in some cases an issue just isn't available, but when it IS available (again, interactive searches on the index sites find it), why isn't Mylar downloading it?

I tried going to the "Wanted" page and clicking "Force check", but that didn't seem to cause Mylar to do much of anything with the missing issues.

I really don't want to have to sit here and laboriously click on the "manual search" button for every issue I'm missing, across a few dozen comics. What am I overlooking here?

Aimhere

[evilhero]

In your particular case, there is more than one S.H.I.E.L.D. series that exist (3 actually). The volume that you're trying to search for is Volume 3, and is tagged as such within Mylar as being Volume 3. I checked the indexers, NONE of them have a Volume 3 label anywhere in the filenames. However, failing the volume match, Mylar does do a comparison check against the actual posting-date of the file to usenet, against what it knows the actual store date of the issue is. As long as the posting date of the issue to usenet is equal to or greater than the store date of the particular issue, Mylar will accept it as a match,

Note that the problems with S.H.I.E.L.D. are also indicative of your problems with Hulk (which also is a Volume 3).

If I were to take a guess, I would guess that you have a fairly large Wanted list (> ~ 100 items?) - this entire list gets 'Force Search'ed every 6 hours by default, which if you have not changed the default time setting, and based on your indexers - it would take a good chunk of time to get through the list entirely. Because it has to do every iteration + Alternate Search Naming for EACH issue in the list (1 issue x 3 issue number variations x 3 providers = ~10 minutes for 1 issue to get properly searched against the RSS + API). If it's not making it through your list, it's probably erroring out in the background or something is locking it up during the full search that is causing it to halt the search. That's why when you do the Manual Search it does an instant search for that one particular issue, whereas with the Force Search that particular issue could be in the queue to be searched and be in the 75th spot out of 300 (as an example).

Also take mind that the Experimental search is just that - experimental. Try a search one time and you'll get proper results, try the same search a few seconds later and you won't get any results. This is due to the indexer either being overloaded, or it has some sort of buffering problem that causes it - but I've noticed it on several occasions where it works fine, and then it doesn't work at all.

In the next few weeks there will be a pretty big change to the intervals that Mylar does searches. ALOT of it will be based on whether or not Mylar is running 24/7. If it is running 24/7, then it will rely only on RSS Feeds to monitor releases after doing an overall API search. When an issue is tagged into Wanted status, it will do a 'Manual Search' on the one issue and then just monitor the feeds for any changes. This is basically how Sonarr does it's searching - there's no point in hammering a site every 6 hrs for a particular issue, when the last API search returned no results. Continually sending the request to search will not yield anything different, since nothing has changed (save for RSS Feed inclusions, but that's why it would monitor those). If Mylar is set to run 24/7, but suffers some downtime for whatever reason, when it comes back up it will take the timestamp of the last RSS run and then check the timestamps on the rss feeds to ensure that there is continual coverage. If it's not, then it will attempt to do a backlog search against it.

If you want to test this theory - cull your wanted list to just have either the SHIELD series or the HULK series (mainly because there's < 16 issues total in each series), and then do a 'Force Search'. I'm betting that you'll get alot more hits, probably even more if you disable the 'Experimental' search for this test.

You might also want to run Mylar in verbose/debuggin mode so that you can actually see the full logging (-v on startup, or toggle Verbose ON in the Log GUI tab within Mylar).

[Aimhere]

I was wondering whether you think some of my issues might be fixed by switching to the development branch?

[evilhero]

Development has alot more fixes/improvements going into it a quicker rate than the Master branch - just because the workflow is that it goes into development first, gets tested there, and once the test is complete ok, then it goes into Master (usually about a month later because alot of the dev commits are interwoven so it's not a piecemeal type of update).

Updating to the development branch using the synopackage though is very troublesome from my understanding of problems that have occured in the past.

[Aimhere]

Actually, I only used the Synocommunity package for my initial install, way back when. Since then, I found instructions to change it to use git instead, so updates have been going okay.

[Aimhere]

So I disabled Experimental Search, "paused" the comics whose run has ended (and for which I have all issues) so Mylar would skip them entirely, and I also noticed there was an update available for the Master branch, so I went ahead and updated. And now, Mylar has downloaded a large batch of issues which it had previously not found. I call that an improvement.

[Aimhere]

Still seeing some failures finding issues that exist on Usenet. For example, I have "Deadpool Team-Up (2009)" in my library, and Mylar's not finding anything. If I start a manual search for issue 899, here's what I get in debug log (ignore the stuff about Avengers, I didn't want to leave anything out):

Code: Select all

15-May-2016 10:09:40 - DEBUG   :: Thread-76 : ComicID:29329
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : mode:want
15-May-2016 10:09:40 - INFO    :: Thread-76 : Initiating manual search for Deadpool Team-Up issue: 899
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : Publisher is : Marvel
15-May-2016 10:09:40 - INFO    :: Thread-76 : Issue Title given as : merc with a myth
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : Checking for torrent enabled.
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : newznab name:Usenet-Crawler @ https://www.usenet-crawler.com
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : newznab name:OZNZB.com @ https://www.oznzb.com
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : nzbprovider(s): ['newznab:Usenet-Crawler', 'newznab:OZNZB.com']
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : there are : 0 torrent providers you have selected.
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : there are : 2 nzb providers you have selected.
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : Usenet Retention : 2350 days
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : found provider in existing enabled providers.
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : checking against nzb provider: newznab:Usenet-Crawler
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : newznab match against: newznab:Usenet-Crawler
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : sucessfully matched - appending to provider.order sequence
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : checking against nzb provider: newznab:OZNZB.com
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : provider order sequence is now to start with Usenet-Crawler at spot #1
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : found provider in existing enabled providers.
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : checking against nzb provider: newznab:Usenet-Crawler
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : checking against nzb provider: newznab:OZNZB.com
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : newznab match against: newznab:OZNZB.com
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : sucessfully matched - appending to provider.order sequence
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : provider order sequence is now to start with OZNZB.com at spot #2
15-May-2016 10:09:40 - INFO    :: Thread-76 : search provider order is ['newznab:Usenet-Crawler', 'newznab:OZNZB.com']
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : Initiating Search via : rss
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : using Newznab host of : Usenet-Crawler
15-May-2016 10:09:40 - INFO    :: Thread-76 : Shhh be very quiet...I'm looking for Deadpool Team-Up issue: 899 (2010) using Usenet-Crawler(newznab) [RSS]
15-May-2016 10:09:40 - DEBUG   :: Thread-76 : calpha/cnumber: None / None
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : RSS Check: yes
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : nzbprov: newznab
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : comicid: 29329
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : Sending request to RSS for Deadpool Team-Up : 899 (2010)
15-May-2016 10:09:41 - INFO    :: Thread-76 : Could not find Issue 899 of Deadpool Team-Up(2009) using Usenet-Crawler [rss]
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : using Newznab host of : OZNZB.com
15-May-2016 10:09:41 - INFO    :: Thread-76 : Shhh be very quiet...I'm looking for Deadpool Team-Up issue: 899 (2010) using OZNZB.com(newznab) [RSS]
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : calpha/cnumber: None / None
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : RSS Check: yes
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : nzbprov: newznab
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : comicid: 29329
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : Sending request to RSS for Deadpool Team-Up : 899 (2010)
15-May-2016 10:09:41 - INFO    :: Thread-76 : Could not find Issue 899 of Deadpool Team-Up(2009) using OZNZB.com [rss]
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : Could not find issue doing a manual search via : rss
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : Initiating Search via : api
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : using Newznab host of : Usenet-Crawler
15-May-2016 10:09:41 - INFO    :: Thread-76 : Shhh be very quiet...I'm looking for Deadpool Team-Up issue: 899 (2010) using Usenet-Crawler (newznab)
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : calpha/cnumber: None / None
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : RSS Check: None
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : nzbprov: newznab
15-May-2016 10:09:41 - DEBUG   :: Thread-76 : comicid: 29329
15-May-2016 10:09:41 - INFO    :: Thread-76 : pausing for 60 seconds before continuing to avoid hammering
15-May-2016 10:10:35 - DEBUG   :: SEARCH : [SSL: True] Search URL: https://www.usenet-crawler.com/api?t=search&q=avengers%20earths%20mightiest%20heroes%2006&o=xml&cat=7030&apikey=xUDONTNEEDTOKNOWTHISx
15-May-2016 10:10:37 - DEBUG   :: SEARCH : RSS Check: None
15-May-2016 10:10:37 - DEBUG   :: SEARCH : nzbprov: newznab
15-May-2016 10:10:37 - DEBUG   :: SEARCH : comicid: 47809
15-May-2016 10:10:37 - INFO    :: SEARCH : pausing for 60 seconds before continuing to avoid hammering
15-May-2016 10:10:42 - DEBUG   :: Thread-76 : [SSL: True] Search URL: https://www.usenet-crawler.com/api?t=search&q=deadpool%20team-up%20899&o=xml&cat=7030&apikey=xUDONTNEEDTOKNOWTHISx
15-May-2016 10:10:43 - INFO    :: Thread-76 : Could not find Issue 899 of Deadpool Team-Up(2009) using Usenet-Crawler [api]
15-May-2016 10:10:43 - DEBUG   :: Thread-76 : using Newznab host of : OZNZB.com
15-May-2016 10:10:43 - INFO    :: Thread-76 : Shhh be very quiet...I'm looking for Deadpool Team-Up issue: 899 (2010) using OZNZB.com (newznab)
15-May-2016 10:10:43 - DEBUG   :: Thread-76 : calpha/cnumber: None / None
15-May-2016 10:10:43 - DEBUG   :: Thread-76 : RSS Check: None
15-May-2016 10:10:43 - DEBUG   :: Thread-76 : nzbprov: newznab
15-May-2016 10:10:43 - DEBUG   :: Thread-76 : comicid: 29329
15-May-2016 10:10:43 - INFO    :: Thread-76 : pausing for 60 seconds before continuing to avoid hammering
15-May-2016 10:11:43 - DEBUG   :: Thread-76 : [SSL: True] Search URL: https://www.oznzb.com/api?t=search&q=deadpool%20team-up%20899&o=xml&cat=7030&apikey=xUDONTNEEDTOKNOWTHISx
15-May-2016 10:11:45 - INFO    :: Thread-76 : Could not find Issue 899 of Deadpool Team-Up(2009) using OZNZB.com [api]
15-May-2016 10:11:45 - DEBUG   :: Thread-76 : Could not find issue doing a manual search via : api

Thing is, if I open a browser and manually enter the url "https://www.usenet-crawler.com/api?t=se ... OKNOWTHISx" (substituting my own API key, of course), Usenet-Crawler actually DOES find the issue:

[copied from U-C HTML]

Code: Select all

usenet-crawler
usenet-crawler Feed
Deadpool Team-Up 899 2010 c2c Legion-CPS
Tuesday, December 18, 2012 11:58 AM
Deadpool Team-Up 899 2010 c2c Legion-CPS
Media files
6c8ca08ad298c01443ba1fd16948c923.nzb&i=15490&r=6213e8509f089c27665d248aa1edf45a (Alt.Binz NZB file)

I notice that in the debug log, the failure message is worded as "Could not find Issue 899 of Deadpool Team-Up(2009) using Usenet-Crawler [api]". Note the lack of a space between the title and the year. Might Mylar be parsing the search results incorrectly? Even the debug log doesn't go into any detail about what Mylar is ACTUALLY looking for...

[evilhero]

I notice that in the debug log, the failure message is worded as "Could not find Issue 899 of Deadpool Team-Up(2009) using Usenet-Crawler [api]". Note the lack of a space between the title and the year. Might Mylar be parsing the search results incorrectly? Even the debug log doesn't go into any detail about what Mylar is ACTUALLY looking for...

The 'Deadpool Team-Up(2009)' is just a spacing issue when it's recorded to the log - it doesn't actually do the search specifically against that term

The debug log doesn't show the search results because then you'd be getting a massive amount of spam within the logs (and I mean massive even by Mylar's logging standards).

I'll check into this and see what's going on - it might have something to do with the way the filename is being parsed from the site that it's not considering it a match.