hadoop - Apace Drill reading gz and snappy performance -


i'm using apache drill 1.8. , test porpoise made .csv 2 parquet files. csv 4gb big, parquet gz codec 120mb , second parquet snappy codec 250gb big.

as spark using snappy default codec, , snappy should performance faster face 1 problem.

this files block size , etc on hadoop:

  1. with snappy codec: enter image description here

  2. with gz codec: enter image description here

time when i'm trying query in drill (which have default snappy codec) parquet files on snappy codec around 18seconds. time when i'm trying query in drill parquet files on gz codec same query around 8seconds.

(it's simple query select 5 columns, ordering 1 , limiting on one)

i'm little confused now. isn't snappy more efficient i/o? making mistake somewhere or how works. if explain me super grateful because couln't find useful on net. thank once more!


-

Comments

Post a Comment

Popular posts from this blog

CS-Cart - Change login URL in notification email -

i testing user account registration system. when new users sign accounts, sent notification email. email, default, includes line: login url: http://mystore.com/ the problem is, that's not login url. login url http://mystore.com/login/ the url seems generated line in email template (profiles_info.tpl): {if $user_data.company_id}{"?company_id=`$user_data.company_id`"|fn_url:'c':'http'}{else}{""|fn_url:'c':'http'}{/if} but don't understand how fn_url works (there doesn't seem documentation). how can fix login url - or best way so? this function define in /app/functions/fn.common.php /** * prepares parameters before building url * * @param string $url urn (uniform resource name or query string) * @param string $area area ('a' - admin or 'c' - client) * @param string $protocol output url protocol (protocol://). if equals 'rel', no protocol in...
Read more

cron - how to properly run Python script with crontab on every system startup -

i have python script should open linux terminal, browser, file manager , text editor on system startup. decided crontab suitable way automatically run script. unfortunately, doesn't went well, nothing happened when reboot laptop. so, captured output of script file in order clues. seems script partially executed. use debian 8 (jessie), , here's python script: #!/usr/bin/env python3 import subprocess import webbrowser def action(): subprocess.call('gnome-terminal') subprocess.call('subl') subprocess.call(('xdg-open', '/home/fin/documents/learning')) webbrowser.open('https://reddit.com/r/python') if __name__ == '__main__': action() here's entry in crontab file: @reboot python3 /home/fin/labs/my-cheatcodes/src/dsktp_startup_script/dsktp_startup_script.py > capture_report.txt here's content of capture_report.txt file (i trim several lines, since long, prints folder structures. seems came ...
Read more

elasticsearch - Is the order of operations guaranteed in a bulk update? -

i sending delete , index requests elasticsearch in bulk (the example adapted from docs ): { "delete" : { "_index" : "test", "_type" : "type1", "_id" : "1" } } { "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } } { "field1" : "value1" } the sequence above intended first delete possible document _id=1 , index new document same _id=1 . is order of actions guaranteed? in other words, example above, can sure delete not touch document index ed afterwards (because order not respected reason or another)? the delete operation useless in scenario, if index document same id, automatically , implicitly delete/replace previous document same id. so if document id=1 exists, sending below command replace (read delete , re-index it) { "index" : { "_index" : "test", ...
Read more