Torrent questions

Post any problems / bugs / issues that are Mylar-related in here.
Post Reply
herjar
Posts: 13
Joined: Sat Apr 12, 2014 11:55 am

Torrent questions

Post by herjar »

Hi.

I have some questions regarding torrents in Mylar.

I have set up Mylar to use torrents on my seedbox. It is working and is downloading fine.
The problem is that it's not finding a lot of files even if they are on CBT. I think I read somewhere that it can't search and is just using RSS?
If so, which RSS is it using? My personal follow list rss? Or just the latest uploaded files on CBT?

In my short experience it seems to find much more on KAT than CBT. I much prefer to use CBT though.
If I choose both CBT and KAT does it search both? From the logs it looks like it just searches CBT. I have to disable CBT to get results from KAT.
Is it possible to make it search CBT first, and if it can't find it then look at KAT?

In short I would love a quick explanation of how the torrent search works.

I have also seen some parsing issues even if the files exist on CBT. An example here where "Swamp thing v5 029" exists, but can't be downloaded. It says "Version detected as v5". What does version mean in this context? Is it the same as volume, or does it not understand that it's the volume? It downloaded fine on KAT, without V5 in the name though.

12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : checking search result: Swamp Thing v5 29 - ( mixed type ) ( Digital ) ( JK-Empire)
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : Wed, 5 Mar 2014 17:21:48 +0000 is after store date of 2014-01-08
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : Entry: Swamp Thing v5 29 - ( mixed type ) ( Digital ) ( JK-Empire)
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : Cleantitle: swamp thing v5 29 - ( mixed type ) ( digital ) ( jk-empire)
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : possible versioning..checking
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : Version detected as v5
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : Cleantitle: swamp thing v5 29 - ( mixed type ) ( digital ) ( jk-empire)
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : 0. Bracket Word: swamp thing v5 29 -
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : Comic: swamp thing v5 29 -
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : UseFuzzy is : None
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : ComVersChk : 0
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : Year not given properly formatted but Version detected.Bypassing Year Match.
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : 1. Bracket Word: mixed type
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : 3. Bracket Word: digital
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : 5. Bracket Word: jk-empire
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : Scanner detected: jk-empire
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : original nzb comic and issue: swamp thing v5 29 -
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : adjusted nzb comic and issue: swamp thing v5 29
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : adjusting from: swamp thing v5 29 to: 29
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : ['swamp', 'thing', 'v5', '29'] nzb series word count: 3
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : ['swamp', 'thing'] watchlist word count: 2
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : this has a version #...let's adjust
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : Version detected as v5
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : volume detection commencing - adjusting length.
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : splitst : 2
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : len-watchcomic : 2
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : length match..proceeding
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : search-length: 2
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : Watchlist-length: 2
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : splitit: swamp
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : scount : 0
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : 0 Comparing: swamp .to. swamp
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : word matched on : swamp
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : splitit: thing
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : scount : 1
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : 1 Comparing: thing .to. thing
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : word matched on : thing
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : splitit: v5
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : scount : 2
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : possible versioning..checking
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : watch comicversion is None
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : version found: v5
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : vers4year: no
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : vers4vol: v5
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : FCVersion: 5
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : DCVersion: 1
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : SCVersion: 2011
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : Versions wrong. Ignoring possible match.
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : scount:0
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : splitit-len:3
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : we got 0.0 percent.
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : failure - we only got 0.0% right!
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : couldn't find a matching comic using CBT [RSS]
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Torrent questions

Post by evilhero »

herjar wrote: The problem is that it's not finding a lot of files even if they are on CBT. I think I read somewhere that it can't search and is just using RSS?
If so, which RSS is it using? My personal follow list rss? Or just the latest uploaded files on CBT?
Yes, the problem with using CBT as opposed to KAT is that CBT will not allow the search parameter through their api - what it means is that you can't search for anything via a URL, whereas with KAT you can. The end result is that Mylar has to use your follow-list AND the lastest uploaded files on CBT in order to find anything, the catch though is that Mylar caches every single result into it's db. So the longer you leave Mylar running (with CBT enabled and RSS Feeds checked off), it will cache every result every X minutes the RSS is polled and store that information in it's db. It does this for CBT, KAT, and every nzb provider as well. When Mylar initiates a search for an issue (whether it's scheduled or manually initiated), it first polls the RSS Feed for releases, then it checks the db for any matches for any provider that you have enabled at that time, and then if both of those fails it will proceed to do either URL searches (KAT), or api searches (nzb providers). If the RSS isn't enabled, but CBT/KAT are, it will still check the RSS for CBT - but it won't cache any results to db which would mean that you wouldn't get alot of hits if you're back-searching stuff or it's primarily not a 0-day release.
If I choose both CBT and KAT does it search both? From the logs it looks like it just searches CBT. I have to disable CBT to get results from KAT.
Is it possible to make it search CBT first, and if it can't find it then look at KAT?
I just came across this myself, where if both were enabled it would only use KAT - suffice to say it will be fixed in the next development commit, but I believe it would still go in the order of KAT, CBT, then nzb providers. I have the framework in place (not entirely but it's getting there) for users to be able to say the order in which things get searched, and would allow for jumping back and forth between torrents and nzb providers - so you could have it search in this order: CBT -- dognzb -- KAT -- newznab -- experimental.

I also came across some interesting findings this past week with regards to how stuff flows on KAT - when checking on KAT, alot of comics that were on there that were 0-day were thrown in the 'Other' category, not the 'Comics' category. Eventually they got moved over to the correct category, but there were a good handful still in 'Other' at the end of the day. I've added an extra couple polls for the RSS for KAT now so that it will grab the feeds for both the Comics and Other category (limiting it to cbr files since that's how the vast majority are sent) and cache/search those results which increases the percentage hits greatly. This will be available in the next development commit as well.
I have also seen some parsing issues even if the files exist on CBT. An example here where "Swamp thing v5 029" exists, but can't be downloaded. It says "Version detected as v5". What does version mean in this context? Is it the same as volume, or does it not understand that it's the volume? It downloaded fine on KAT, without V5 in the name though.

Yep, that's how it's supposed to be actually. The version/volume thing unfortunately are interchangeable within Mylar, they mean the same thing - however at some point (soon), I'll be removing all references to the Version and be replacing it with Volume for more consistency within the program.

The 'Version detected as v5' is Mylar indicating that it found the volume in the filename title, however from your log it would appear that your Swamp Thing doesn't have any volume labels attached to it, so in that case Mylar assumes it's a 'v1' - as in the only series in existence with that exact title (which is obviously wrong in this case).
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : possible versioning..checking
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : watch comicversion is None
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : version found: v5
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : vers4year: no
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : vers4vol: v5
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : FCVersion: 5
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : DCVersion: 1
12-Apr-2014 13:12:35 - DEBUG :: Thread-13 : SCVersion: 2011
So for further information, this is the breakdown of that log:
Watch comicversion -- the version that's attached to the given series (whatever is displayed on your comicdetails page for the given series) ...as an aside it should say 'watchcomic version'
version found -- the version that's found within the filename that's attempting to be matched
vers4year -- if the 'version found' was in the format of Vxxxx, where xxxx is the series start year.
vers4vol -- if the 'version found' was in the format of Vx where x is the volume numbering.
FCVersion -- the numeric for 'version found'
DCVersion -- the numeric for 'watch comicversion'
SCVersion -- the series year for the given series (watch comic)

So when it does the compare it compares v5 to v1 and obviously they don't match so it rejects the result. If the series doesn't have a volume label (V5), in most cases it will have the issue year attached (ie. "Swamp Thing 029 (2014)" ) which Mylar will match against and ignore the requirements for a Volume match. Mylar will pick either the year match, or the volume match depending on if the given data is provided for in the filename.

You can add your own Volume label to each series, by (when on the comic details page for the given series) clicking on the Edit tab and then entering in the Volume label (it can be in either the V1 format, or the V2012 format). Now by default, Mylar will attempt to populate the information itself when you add a series initially, or when it does a refresh of a series (either manually, or scheduled via the 6hour global refresh, or when the weekly pull list gets refreshed and there's a match as mylar refreshes the given series that matches in order to ensure that the issue data is as up to date as possible).

Hopefully that helps out abit in how the torrenting and matching works within Mylar - I personally use CBT and nzb (with KAT ocassionally but moreso for testing than anything), so it's good to hear that the seedbox sending-portion is working for someone else other than just myself ;)

If you have any other questions or feel things need to be addressed/fixed , feel free to ask away and I'll do my best to answer and or even fix things if things aren't working as they should.
herjar
Posts: 13
Joined: Sat Apr 12, 2014 11:55 am

Re: Torrent questions

Post by herjar »

Thank you for a very thorough explanation.

I didn't have the RSS box ticked. I thought it was for nzb :)
The text could maybe be changed to "Enable RSS Feed searches for nzb and torrents"?
Just looks like it's located with the nzb stuff now.

Choosing the order for downloading sounds great. Looking forward to the next commit.

Yea it's working fine on my seedbox, but I'm not using the "seedbox client" in Mylar. I actually have two copies of Mylar.
One on the seedbox that does all the downloading, with a script that sends the comics to a local folder. I then have a local copy that sorts the comics to the right folders.
A bit extra work to set up as I have to add comics twice, but it's the same thing I do with sickbeard and it works great once it's set up. This way I don't need to have my local PC on all the time.
One request i could add here is the possibility to choose which folder to put the comic to when you add it (similar to sickbeard). Currently it always creates a new folder that I have to delete :)
Not a big deal though, it works fine.

This is the comic that failed:
http://www.comicvine.com/swamp-thing/4050-42599/
It says in the text that it's the fifth volume, but where do I see which volume is actually stored on comicvine?
Thanks for pointing me to where I can add it manually. That should fix some searches.

BTW, I joined in lobbying for Mylar here:
http://nzb360.com/forums/viewtopic.php? ... 5472f75202
Don't know if you've seen it. It's a great Android program. Even if I don't use any of the nzb stuff it's great for controlling SB, CP and HP.
Hope he gets around to adding Mylar.
herjar
Posts: 13
Joined: Sat Apr 12, 2014 11:55 am

Re: Torrent questions

Post by herjar »

Should I keep posting the issues I find with parsing comic names?
Or is there a better place for that?
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Torrent questions

Post by evilhero »

herjar wrote: I didn't have the RSS box ticked. I thought it was for nzb :)
The text could maybe be changed to "Enable RSS Feed searches for nzb and torrents"?
Just looks like it's located with the nzb stuff now.
Yes, you're not the first to mention that - I'll probably add in the change to the next commit, and over the course of the next few commits I'll be re-organizing the options within the configuration GUI as it's gotten abit of a mess atm.
One request i could add here is the possibility to choose which folder to put the comic to when you add it (similar to sickbeard). Currently it always creates a new folder that I have to delete :)
I've never used an option like that in sickbeard - or even knew it existed tbh. Are you wanting the option to be on a series add ? I'm not sure how easy that would work given the way that the add series works - are you looking for just a general place to download all the 'new' comics you get ? (the pull-list has an option to 'Download just the week's pull to a given week folder off of your comic location root). I guess I'm just unsure as to how you're using Mylar - to track your series, or to just download them and then forget about them (and thus deleting the series directories).
This is the comic that failed:
http://www.comicvine.com/swamp-thing/4050-42599/
It says in the text that it's the fifth volume, but where do I see which volume is actually stored on comicvine?
Well there isn't any field within ComicVine that actually lists specifically what volume a series is. There is one that's included in the API, but it rarely gets used. However the volume is most often within the description field. By default, when you add a series (or Refresh if a volume label isn't present within Mylar for the given series) Mylar will automatically attempt to locate the information within the description field. The problem with the Swamp Thing series that you've mentioned is that nowhere in the description does it say the word 'volume' and because of that Mylar won't grab the information ("A New 52 series, starring Swamp Thing. This is the fifth Swamp Thing series and the first not under the Vertigo imprint since the second series switched over at Vertigo's founding in 1993.") Now if the description said "fifth volume of Swamp Thing" or something along those lines, Mylar would have auto-populated the volume and you probably wouldn't have looked into it further ;)
BTW, I joined in lobbying for Mylar here:
http://nzb360.com/forums/viewtopic.php? ... 5472f75202
Don't know if you've seen it. It's a great Android program. Even if I don't use any of the nzb stuff it's great for controlling SB, CP and HP.
Hope he gets around to adding Mylar.
I actually do use nzb360 on my both of my android devices (have for awhile now), and it's by far and away the best mobile management/control application that's out there for those applications. I don't think the API that Mylar has in place atm would be robust enough for him to be able to pull anything out of, but hey - if anyone can do it, I would bet on him. That being said, if he happens to point out some things that the API is currently missing or may need to get fixed/added, I'd be more than happy to work with him in order to get a functional version of Mylar working with his app.
Should I keep posting the issues I find with parsing comic names?
Or is there a better place for that?
There isn't a problem with posting the problems with parsing names within the forums - maybe start a new thread though if you happen to come across any more (that way it's easier to find instead of being within the confines of a torrent-related question). You can also post on github - either or both works for myself, as with technology nowadays I'll get notified either way of a new post and hopefully respond accordingly within a reasonable amount of time ;)
Post Reply