python - Access HDF files stored on s3 in pandas -


i'm storing pandas data frames dumped in hdf format on s3. i'm pretty stuck can't pass file pointer, url, s3 url or stringio object read_hdf. if understand correctly file must present on filesystem.

source: https://github.com/pydata/pandas/blob/master/pandas/io/pytables.py#l315

it looks it's implemented csv not hdf. there better way open hdf files copy them filesystem?

for record, these hdf files being handled on web server, that's why don't want local copy.

if need stick local file: there way emulate file on filesystem (with real path) can destroyed after reading done?

i'm using python 2.7 django 1.9 , pandas 0.18.1.


-

Comments

Post a Comment

Popular posts from this blog

cron - how to properly run Python script with crontab on every system startup -

i have python script should open linux terminal, browser, file manager , text editor on system startup. decided crontab suitable way automatically run script. unfortunately, doesn't went well, nothing happened when reboot laptop. so, captured output of script file in order clues. seems script partially executed. use debian 8 (jessie), , here's python script: #!/usr/bin/env python3 import subprocess import webbrowser def action(): subprocess.call('gnome-terminal') subprocess.call('subl') subprocess.call(('xdg-open', '/home/fin/documents/learning')) webbrowser.open('https://reddit.com/r/python') if __name__ == '__main__': action() here's entry in crontab file: @reboot python3 /home/fin/labs/my-cheatcodes/src/dsktp_startup_script/dsktp_startup_script.py > capture_report.txt here's content of capture_report.txt file (i trim several lines, since long, prints folder structures. seems came ...
Read more

elasticsearch - Is the order of operations guaranteed in a bulk update? -

i sending delete , index requests elasticsearch in bulk (the example adapted from docs ): { "delete" : { "_index" : "test", "_type" : "type1", "_id" : "1" } } { "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } } { "field1" : "value1" } the sequence above intended first delete possible document _id=1 , index new document same _id=1 . is order of actions guaranteed? in other words, example above, can sure delete not touch document index ed afterwards (because order not respected reason or another)? the delete operation useless in scenario, if index document same id, automatically , implicitly delete/replace previous document same id. so if document id=1 exists, sending below command replace (read delete , re-index it) { "index" : { "_index" : "test", ...
Read more

Slow performance first queries on SQL Azure -

i have small database (50mb) , i'm on basic plan. there single user, need create many databases (always 1 per user) since used training purposes. each database created doing following statement: create database training1 copy of modeldatabase1 we seem getting very slow performance when first query database, afterwards seems acceptable. to give idea: have sp: startupevents runs when application started. query takes 25 seconds run first time. seems incredible since database small, , tables query calls don't contain many records. if run procedure afterwards executes immediately... how can avoid this?
Read more