Different site for list of new releases?

Enhancement requests or feature requests for future Mylar releases.
leaderdog
Posts: 218
Joined: Sun Apr 26, 2015 1:52 pm

Different site for list of new releases?

Post by leaderdog » Sat May 07, 2016 1:12 pm

Hi Evilhero,

This past week pretty much none of my comics were downloaded. Most of them didn't "refresh" so even when I grabbed all the new releases via newsleecher most of the books just sat there. When I went to look the refresh never took place so I did a refresh for some of them then the manual search did its thing.

I'm pretty sure you're using diamonds list for the new releases which doesn't really give much info other than the name. Thus most of my "weekly" don't update at all since the name often varies between weeks as to what they short formed the names for.

Would there be any benefit to a site like this?
hxxp://leagueofcomicgeeks.com/comics/new-comics (they have an app too, not sure if that would give all the info needed)
or
hxxps://www.mycomicshop.com/newreleases?dw=-1
or
hxxp://freshcomics.us/comics/2016-05-04/html (also have app)

I don't know if a simple scraper (I say simple but I've no knowledge of coding) to scrape the comic name, year, publisher, issue, to have a more accurate transition into mylar? That way the weekly list might be more accurate and able to refresh the books that it sees. I know comicvine is to blame for some downloads being ignored since the leave published date blank half the time, but with a more accurate weekly list to mylar connection this could be solved with a command to refresh zero dates.

Even when I've tried doing the name change/association in the mylar folder, it doesn't always work as again, diamond isn't always consistent with how they name the issue for that week, or for whatever reason the name change thing just doesn't care what I put in. And of course that is a bit tedious as well.

Thoughts? Am I out to lunch on this or this wouldn't make any difference?

Thanks, love the program.

User avatar
evilhero
Site Admin
Posts: 2263
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Different site for list of new releases?

Post by evilhero » Sat May 07, 2016 9:18 pm

Yes, Mylar uses diamond's list - and the reason being that initially it avoided me having to parse the website, and in doing so add some inconsistencies to the capturing of the data. At diamond, every Monday a .csv file can be retrieved showing the weekly pull-list of comics, which is the way I went. Of course, a few months ago I added an ALT_PULL option to the config which allows the user to either retrieve the .csv file as normal, or actually parse the webpage.

I'm not fond of the current way that Mylar pulls the pull-list down and more-so the inconsistency of the data itself (if you're ever on irc, you'd hear the rumblings amongst those on there usually). Using Diamond is the reason why the substitutes.csv file exists, as well as the Alternate Search Name (well partially at least). The blank dates on CV are actually handled by Mylar fairly well from what I've seen thus far - if a series is on the pullist, and it's a valid issue via numbering sequence it gets marked as Wanted. It then does a check against CV for the issue - if the issue doesn't exist for the series on CV, the series gets refreshed to pull down any new issue data. If it's still not pulled down, the issue is populated on the series detail page as having the pull-list date for a publication date, and it's thrown into the Upcoming section as being Wanted - but there's no link between the Series & Issue at this point. Mylar will then refresh the pull-list every 6 hrs until CV has the issue information available and once that occurs it gets populated into a Wanted status.

Based on the subset list you gave of possible replacements (which, btw thanks - it's always good to see what's available), I'm leaning towards the leagueofcomicgeeks just because it seems like it might be abit easier to parse. Of course there are other factors to consider in all of this, but that's definitely the one I think might work the best.

On the same vein, in an upcoming dev commit I have the weekly pull-list using the same dynamic naming matching as what's used during post-processing - which should help match things abit better, but with the way it currently is, it won't alleviate the problem with short-naming and such unless you're using the substitutes.csv file.

I'll update this when I have more details, but I might try and start changing things over to a new website for information in the next day or so...

leaderdog
Posts: 218
Joined: Sun Apr 26, 2015 1:52 pm

Re: Different site for list of new releases?

Post by leaderdog » Sat May 07, 2016 11:10 pm

Sounds great.

The only true benefit of diamond is, it's probably not going anywhere. The leagueofcomicgeeks looks pretty well used too tho, so hopefully it will be around for many a year to come.

But I agree, that one seems to have all the info pretty handy. I'm not an "apps" person, so I don't know what their app is like or if that would hand the info over fairly easily by itself as well. or if that's a whole other beast all together.

I look forward to seeing what you come up with.

Thanks Evilhero.

User avatar
evilhero
Site Admin
Posts: 2263
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Different site for list of new releases?

Post by evilhero » Tue May 10, 2016 5:01 pm

Ok, here's an update:

It's done and works pretty good.
:)

I just want to change the frequency in which Mylar accesses the site, so it doesn't end up hammering it - so once I figure that out, it will be in the subsequent development commit.

leaderdog
Posts: 218
Joined: Sun Apr 26, 2015 1:52 pm

Re: Different site for list of new releases?

Post by leaderdog » Wed May 11, 2016 3:51 am

Hi Evilhero,

That's great news! Can't wait to see this in action.

Does each person running mylar tag the site to get new release info, or do you have one location (like github or something) where it downloads to and pushes that info to us?

rubs hands together. can't wait! :)

Destron
Posts: 59
Joined: Thu Apr 28, 2016 7:37 pm

Re: Different site for list of new releases?

Post by Destron » Thu May 12, 2016 3:13 am

I'm very curious about this. Can't wait to see what you've come up with. Did any of those sites have an RSS feed for the new releases?

Destron
Posts: 59
Joined: Thu Apr 28, 2016 7:37 pm

Re: Different site for list of new releases?

Post by Destron » Thu May 12, 2016 3:26 am

Also curious: Is there a drawback to using the ComicVine API to pull a list of weekly comics by store_date?

User avatar
evilhero
Site Admin
Posts: 2263
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: Different site for list of new releases?

Post by evilhero » Thu May 12, 2016 1:50 pm

leaderdog wrote:Hi Evilhero,

That's great news! Can't wait to see this in action.

Does each person running mylar tag the site to get new release info, or do you have one location (like github or something) where it downloads to and pushes that info to us?

rubs hands together. can't wait! :)
I'm not about to release this until I can get the 'middle-man' scenario working - so basically, there will be another site that will maintain/update/hold the pull-list data and handle all the retrievals from the web-site(s). Mylar will then request the pull-list data for a given date from this site, and it will send it back in a format Mylar can understand. This way, the website that we're pulling the metadata from doesn't get bombarded by requests that originate from Mylar, and I can clean/manipulate the data in order to make it work best from Mylar at that level (instead of patching Mylar itself, which should help with some problems).
Destron wrote:I'm very curious about this. Can't wait to see what you've come up with. Did any of those sites have an RSS feed for the new releases?
No, unfortunately no RSS on any of the sites - probably for the reason that they want to drive users to their website for the information (if I were to guess).
Destron wrote:Also curious: Is there a drawback to using the ComicVine API to pull a list of weekly comics by store_date?
The CV API is good, but it's not that good unfortunately. Because the information for new releases doesn't get populated until Monday (and even then the store dates are all 0000-00-00), the actual information doesn't get pushed into the CV db until Tuesday, sometimes Wednesday. In theory, Mylar could use the information from then, but then you wouldn't know what's actually released until the Wednesday in most cases (or late Tuesday), and it would be constantly updating in the background in order to see what was new. They have that in a request on their forums that asks for api access to the new releases / week, but CV isn't especially quick at implementing things (just limiting moreso lately).

Destron
Posts: 59
Joined: Thu Apr 28, 2016 7:37 pm

Re: Different site for list of new releases?

Post by Destron » Thu May 12, 2016 2:38 pm

I'm not about to release this until I can get the 'middle-man' scenario working - so basically, there will be another site that will maintain/update/hold the pull-list data and handle all the retrievals from the web-site(s). Mylar will then request the pull-list data for a given date from this site, and it will send it back in a format Mylar can understand. This way, the website that we're pulling the metadata from doesn't get bombarded by requests that originate from Mylar, and I can clean/manipulate the data in order to make it work best from Mylar at that level (instead of patching Mylar itself, which should help with some problems).
This makes a lot of sense. Have you decided which one you want to use? Is there anything I can do to help?

Destron
Posts: 59
Joined: Thu Apr 28, 2016 7:37 pm

Re: Different site for list of new releases?

Post by Destron » Thu May 12, 2016 2:44 pm

Actually... check your messages :)

Post Reply