Using smart_open for reading gzip file with sftp

[dineshbvadhia]

I want to read a remote gzip file line by line using sftp. I have a working solution using an sftp package but want to make it work with smart_open but the conversion is not working :(

The sftp code to convert is:

    with (
        open(remotepath, 'r', host, user, password, port, transport_params) as s,
        gzip.open(s.open(remotepath, mode="r"), "rt", encoding="utf-8") as g,
    ):
	while line := g.readline():

My attempt is:

    from smart_open import open

    with open(remotepath, 'r', host, user, password, port, transport_params) as s:
        with gzip.open(s.open(remotepath, mode="r"), "rt", encoding="utf-8") as g:
            while line := g.readline():

Which generates the Traceback:

    with open(remotepath, 'r', host, user, password, port, transport_params) as s:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ...\smart_open\smart_open_lib.py", line 228, in open
    decoded = _encoding_wrapper(
              ^^^^^^^^^^^^^^^^^^
  ...\smart_open\smart_open_lib.py", line 436, in _encoding_wrapper
    fileobj = so_utils.TextIOWrapper(
              ^^^^^^^^^^^^^^^^^^^^^^^
ValueError: illegal newline value: 3373

What am I doing wrong?

[ddelange]

if your filename ends on .gz, smart_open will transparently decompress using gzip for you. if it does not, you can still force gzip (de)compression with the compression parameter:

https://github.com/piskvorky/smart_open?tab=readme-ov-file#compression-handling

>>> from smart_open import open
>>> with open('sftp://smart_open/tests/test_data/1984.txt.gzip', compression='.gz') as fin:
...     print(fin.read(32))
It was a bright cold day in Apri

[ddelange]

aha, i see. if you're making use of the top level smart_open.open (and not smart_open.ssh.open), host and port etc should be embedded in the uri.

top level open will use smart_open.ssh.open_uri:

smart_open/smart_open/smart_open_lib.py

Line 366 in 5a82613

def _open_binary_stream(uri, mode, transport_params):

which in turn parses host and port etc out of the uri, and propagates to smart_open.ssh.open:

smart_open/smart_open/ssh.py

Line 113 in 5a82613

def open_uri(uri, mode, transport_params):

[dineshbvadhia]

I'm using a cli to kickoff a program which calls smart_open eg:

python myprogram.py ... --remotepath test.gz --host 127.0.0.1 --user admin --password admin --port 3373

Is smart_open.open or smart_open.ssh.open appropriate?

[mpenkov]

Please see the documentation.

help(smart_open)

smart_open.open is the function that you want.

[ddelange]

>>> from smart_open import open
>>> with open('sftp://username:password@host:port/path/file.gz') as fin:
...     print(fin.read(32))
It was a bright cold day in Apri

[ddelange]

You can also use an SSH key file instead of embedding a password in the uri: https://github.com/piskvorky/smart_open/pull/522/files

[dineshbvadhia]

Ok, but still having a problem. As before, this works:

    from smart_open import open
    
    with open(remotepath, 'r', transport_params) as g:

remotepath is a fully qualified posix Path:

This doesn't work:
with open(uri, 'r', transport_params) as g:

I'm developing on Windows 11 Pro (with latest updates). Does smart_open expect a Windows path or uri in a specific format and/or pre-processed?

I'm happy to use the working solution above but it would be good to identify why the uri method doesn't work. It could just be Windows hell :(

[ddelange]

the uri should be formatted as in my last code snippet above. that will trigger the uri parsing mechanic as described in my comment before

[dineshbvadhia]

Ok, it works. The path wasn't prescribed relative to where the sftpserver was running. Phew. Thank-you for your patience :)

About the use of an SSH key file: My software is for general use by developers. Wrt sftp, the user name and password are mandatory, a keyfile and passphrase are optional. Currently, the setup is a CLI as shown earlier #827 (reply in thread). So, what is a secure way to start the program which includes sftp and non-sftp arguments?