SFTP for Android Tablet in Reading List Management Issue

Information and Tutorials on features in Mylar and how to use it
Post Reply
spoonefl1
Posts: 7
Joined: Mon Nov 16, 2015 5:35 pm

SFTP for Android Tablet in Reading List Management Issue

Post by spoonefl1 »

First, I'd just like to say that I really like Mylar. I haven't been using it very long (only started playing with it last week), but my collection is so much more organized and easy to manage already. Thanks for that. I'll be making a donation soon to buy you a beer. A good beer, at that.
Last night I began the process of trying to set up Reading List Management. I read with an Android tablet, so the sFTP feature is really appealing to me. I setup an SFTP server on my tablet, grabbed its IP on my LAN and its port. Mylar runs on the same LAN. There are no software firewalls. I've verified that the tablet is reachable via the same IP and port that I entered into Mylar (as IP:PORT) with the same credentials I entered as well. The SFTP server is reachable and working. But if I click "Sync" in Mylar, the logs always show *IP is not reachable at this time."

I'm running master, and it's updated. I'm on Windows using Python 2.7.7, and I run Mylar as administrator.

What should I be looking for to solve this? I can connect from the machine running mylar to the tablet via sFTP.

Code: Select all

2015-11-16 15:02:26	INFO	[READLIST-TRANSFER] The host 192.168.1.7 is not Reachable at this time.
2015-11-16 15:02:26	INFO	[READLIST-TRANSFER] 17 issues will be sent to your reading device.
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: SFTP for Android Tablet in Reading List Management Issu

Post by evilhero »

Well I'm glad someone else will be using the SFTP-Readlinglist option aside from just me (which is why I initially created it, then just merged it in because I thought someone else might find it useful).

So from the error, and from looking at some of the code - it looks like Mylar is having problems performing a successful ping attempt to the tablet IP provided. My tablet isn't with me, so I can't test to the android atm - but you might want to verify that the IP you put into the SFTP options within mylar (the 192.168.2.7 in the log) is the correct IP. I know unless you have your router configured to assign specific IPs to particular machines, it will dynamically assign a new IP on starting up that machine.

Aside from that, I'm not sure at the moment as the logging in that particular module is pretty sparse (by design belief it or not, too much logging in some other modules). I'll track down my tablet and try to do some sends from my Windows laptop to my tablet and see if I can duplicate the problems at hand ;)
spoonefl1
Posts: 7
Joined: Mon Nov 16, 2015 5:35 pm

Re: SFTP for Android Tablet in Reading List Management Issu

Post by spoonefl1 »

Well, thank you for getting back to me! I think it's a killer feature, personally. I know of no other more convenient way to get comics to my tablet, which is where I do all of my reading. Until now I've just been using Dropbox and copy pasting folders on Android using Astro File Manager. It works, but it's nothing like as automatic as this seems.

I installed Filezilla on the machine running mylar, and was able to successfully SFTP to the tablet using the IP and credentials in mylar. But I think you're right: mylar isn't able to see the server for some reason. IP , port and creds are definitely right.

I'm afraid I can't help from the development side, but if I can ever be useful for testing, please let me know. And thank you.
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: SFTP for Android Tablet in Reading List Management Issu

Post by evilhero »

I finally got a chance to look at some of the code from my windows box.

Turns out linux ping options, are different than Windows - which caused the response to always be that the IP cannot be reached. From a linux machine (and the way mylar currently is), it issues a 'ping -c1 <ipaddress>' in order to see if the android is running. From windows, it has to be 'ping -n 1 <ipaddress>' - really stupid of me to miss it, as it's really simple.

I'll commit this up shortly, and you can try it to see if it's working for you. Provided you have the Paramiko library installed - you shouldn't have any further problems I hope (I was able to use the Sync option and have it send like 60 comics to my tablet last night). I do believe that the select an issue and choose 'Send' from the dropdown box isn't functioning atm for some reason, so just a word of warning if you're going to utilize that. I have to dig into that one abit more to see what's going on. I'll edit this post when the commit is live in the development build with this fix ;)

EDIT. Fix is now live in the latest development commit (a32264e)
spoonefl1
Posts: 7
Joined: Mon Nov 16, 2015 5:35 pm

Re: SFTP for Android Tablet in Reading List Management Issu

Post by spoonefl1 »

Thanks very much for that! Sorry for the delayed reply. I just got back in town and had a chance to switch branches and test. I'm closer, but not quite there. Relevant error:

Code: Select all

2015-11-24 19:46:28	ERROR	Uncaught exception: Traceback (most recent call last):
File "c:\Program Files (x86)\mylar-master\mylar\logger.py", line 159, in new_run
old_run(*args, **kwargs)
File "C:\Python27\lib\threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "c:\Program Files (x86)\mylar-master\mylar\readinglist.py", line 233, in syncreading
success = mylar.ftpsshup.sendfiles(sendlist)
File "c:\Program Files (x86)\mylar-master\mylar\ftpsshup.py", line 101, in sendfiles
return sendtohome(sftp, remotepath, filelist, transport)
File "c:\Program Files (x86)\mylar-master\mylar\ftpsshup.py", line 139, in sendtohome
sftp.stat(remotesend)
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 413, in stat
t, msg = self._request(CMD_STAT, path)
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 730, in _request
return self._read_response(num)
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 762, in _read_response
raise SSHException('Server connection dropped: %s' % str(e))
SSHException: Server connection dropped: 
I think what is happening has to do with the SFTP host key not getting accepted automatically. I totally admit that I'm guessing, but I'm not getting asked for it, even though I see in the logs:

Code: Select all

2015-11-24 19:46:28	INFO	Initiating send for 17 files...
2015-11-24 19:46:28	INFO	[READLIST-TRANSFER] The host 192.168.1.7 is Reachable. Preparing to send files.
Again, I'm guessing, but mylar/ftpsshup.py doesn't seem to have anything about auto-adding keys, or any way to ask me to accept them. Is that correct?

These guesses are based on the fact that after receiving a failure as above, I connected via Filezilla and was asked (as expected) to accept the key. Then I realized that I hadn't seen anything like that in Mylar and started poking around:

https://stackoverflow.com/questions/138 ... key-policy
https://stackoverflow.com/questions/285 ... g-host-key
#http://jessenoller.com/blog/2009/02/05/ ... -different

If I understood Python better, I'd try to fix it myself with that snippet from the first link, but I know enough to know how little I know.
When I get a chance, I'm going to see if I can find the known_hosts from Filezilla, figure out where paramiko keeps the same thing, and copy it in. If that were to work, then it would prove my wild, outlandish guess.
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: SFTP for Android Tablet in Reading List Management Issu

Post by evilhero »

I think the problem might be that you don't have password authentication enabled as a fallback in case the keys don't get accepted. I know on some sftp servers you can set the level of authentication and if keys are required to be passed. It's also possible that I know nothing of whatibtalk about on this - and the error might be related to something else entirely ;)

Right now mylar only does basic password authentication and doesn't even try for key authentication. But I'm pretty positive depending on your sftp server that you're using on your tablet, you can configure the failure portion. I'm not sure what sftp server software I'm running on my tablet - I'd be interested in knowing what you're running on yours so then I could try to duplicate / work through the problem to try and fix things.

Can you confirm what version of paramiko you're currently dining on your mylar machine as well?
spoonefl1
Posts: 7
Joined: Mon Nov 16, 2015 5:35 pm

Re: SFTP for Android Tablet in Reading List Management Issu

Post by spoonefl1 »

I'm miscommunicating. I'm not talking about logging into the SFTP server with public/private keys. I only have user/pass authentication set up. What I'm talking about is the RSA key message you get the first time you ssh into a new computer. On Linux, once you SSH to a new IP , you're asked if you want to make the connection, and if you say yes, then it adds that key to the known_hosts file, which I believe is stored in ~/.ssh. I brought my tablet into work today, so the IP in the below is incorrect, and I'm using SSH on linux just to demonstrate:

Code: Select all

$ ssh [email protected]
The authenticity of host '10.0.0.151 (10.0.0.151)' can't be established.
RSA key fingerprint is 17:60:bb:44:2f:36:d8:df:6b:98:fb:63:7f:52:a7:a1.
Are you sure you want to continue connecting (yes/no)? 
I'm not getting a similar message when I hit Sync from Mylar.

I had hoped to find the rsa key for my tablet from Filezilla, but my googling has steered me wrong. Everything I can find says that they key should be in the registry, but it isn't. I don't know if Filezilla is no longer based on PuTTY, but I'm unable to find anything by searching it. And there's no known_hosts file that I can find either.

I tried a number of different FTP servers on my tablet. In the Play store there are a ton of them, but a lot are the exact same (or very similar) apps made by the same publisher, so of the six or seven I tried, only three were actually different. All of them have the same basic setup. I chose a user/pass authentication and used the same port set in Mylar. The logs of each of the apps are pretty thin, but each suggests that my mylar server is connecting (the IP shows in the logs), but then almost immediately disconnects. I re-entered user/pass on both the servers and in mylar to make sure that wasn't the issue.

I installed Paramiko via Pip yesterday: paramiko (1.16.0).

Later on I'll have a Linux box at home to mess with, so I can see if I can use it to connect and get the rsa host key. The problem then becomes what to do with it. Paramiko's docs state:
load(filename)
Read a file of known SSH host keys, in the format used by OpenSSH. This type of file unfortunately doesn’t exist on Windows, but on posix, it will usually be stored in os.path.expanduser("~/.ssh/known_hosts").
http://docs.paramiko.org/en/1.16/api/hostkeys.html

I really don't know what to do about that, if it just won't use one. The only thing I can think of (super limited knowledge over here) is to auto accept the host's rsa key. Even if it doesn't ever get stored, it should work, just because it always gets accepted.
User avatar
evilhero
Site Admin
Posts: 2883
Joined: Sat Apr 20, 2013 3:43 pm
Contact:

Re: SFTP for Android Tablet in Reading List Management Issu

Post by evilhero »

This is partially my fault - when I tested it prior to the last commit I had only tested it on a Windows machine running against Mylar being on an Ubuntu machine and the android tablet (dunno what I was thinking, it was like 3am).

So today, I rolled out my android - started up my SSH server on it (it's called SSH Server by Ice Cold Apps, but have it set for SFTP and SCP modes). Tried to run through Mylar syncing a few comics, and it ran into some basic errors (didn't have paramiko installed on my windows laptop, and then a failure handler message). Once those errors were fixed (mainly my problem, not mylar) I hit sync, and (note that the IPs are messed up on purpose) :

Code: Select all

25-Nov-2015 14:35:29 - INFO    :: Thread-12 : [READLIST-TRANSFER] 1 issues will be sent to your reading device.
25-Nov-2015 14:35:29 - INFO    :: Thread-12 : [READLIST-TRANSFER] The host 215.168.1.257 is Reachable. Preparing to send files.
25-Nov-2015 14:35:29 - DEBUG   :: Thread-12 : Destination: 215.168.1.257
25-Nov-2015 14:35:29 - DEBUG   :: Thread-12 : Using SSH port : 18281
25-Nov-2015 14:35:30 - DEBUG   :: Thread-12 : remote path set to /storage/emulated/legacy/Comics
25-Nov-2015 14:35:30 - INFO    :: Thread-12 : Initiating send for 1 files...
25-Nov-2015 14:35:30 - DEBUG   :: Thread-12 : Checking filename for problematic characters: Animal Man 000 (2012) (Digital) (Zone-Empire).cbr
25-Nov-2015 14:35:30 - INFO    :: Thread-12 : Sending : C:\Temp\Comics\Animal Man (2011)\Animal Man 000 (2012) (Digital) (Zone-Empire).cbr
25-Nov-2015 14:35:30 - INFO    :: Thread-12 : To : /storage/emulated/legacy/Comics\Animal Man 000 (2012) (Digital) (Zone-Empire).cbr
25-Nov-2015 14:35:39 - INFO    :: Thread-12 : Marking as being successfully Downloaded to 3rd party device (Queuing to change Read Status to Downloaded)
25-Nov-2015 14:35:39 - DEBUG   :: Thread-12 : Upload of readlist complete.
Now this was on a Windows 10 machine, with a newly updated mylar (to the most recent commit), recent Paramiko (1.16.0) and had never made a connection to the SSH Server on the tablet ever (so no possibility of any key knowledge prior if they existed). I never got prompted for any acceptance of any keys, or any messages in either the windows console or the tablet console. It just 'worked' *shrug*

Maybe if you try running the same ssh server that I have? There's something amiss somewhere that's causing an issue the connection to your tablet - right now, the only difference might be the server software being used. I'm also not 100% sure if using the SFTP protocol (which mylar employs via paramiko), if it actuallys sends it via the SSH protocl or if it's an actual FTP layered onto the SSH - meaning I don't know if keys are required to be exchanged for a proper connection, although in my case above, they weren't.
spoonefl1
Posts: 7
Joined: Mon Nov 16, 2015 5:35 pm

Re: SFTP for Android Tablet in Reading List Management Issu

Post by spoonefl1 »

Well, you're right. And I'm right. But more wrong than right. Way, way more wrong.

So the issue didn't lie with Mylar, paramiko, or rsa keys, either.

And it also didn't lie with the SFTP server (yours is one of the ones I have installed and identically configured).

It turns out that if you set an improper path in the Reading List Management SFTP settings, you'll get what I was getting last night. In other words, I screwed up. I had a path set there (/) that was above the path I set in the SFTP server (/mnt/extSDcard/Comics, or something like that). I guess when it connected and attempted to read that directory, it either disconnected itself or got booted. Currently I'm happily uploading comics to my tablet with the path set to blank.

Thanks so much for all of your help, and I apologize for wasting your time for the last couple days with this.
Post Reply