scala - Laziness of spark pipline -


i have question laziness of apache spark pipelines. understand spark transformations , actions.

let's take example:

the following snippet transformation returns rdd of filenames in folder. 

val filenames = sc.wholetextfiles("pathtodirectory").map( _ match { case (filename, content) =>  filename })

if execute action :

filenames.count

spark executing defined transformations including loading 'filename' rdd , 'content' rdd memory if 'content' not used

is there better way write spark transformations such as: not used code (like 'content' rdd in previous example) not evaluated while executing actions?

thank feedback.


-

Comments

Post a Comment

Popular posts from this blog

cron - how to properly run Python script with crontab on every system startup -

i have python script should open linux terminal, browser, file manager , text editor on system startup. decided crontab suitable way automatically run script. unfortunately, doesn't went well, nothing happened when reboot laptop. so, captured output of script file in order clues. seems script partially executed. use debian 8 (jessie), , here's python script: #!/usr/bin/env python3 import subprocess import webbrowser def action(): subprocess.call('gnome-terminal') subprocess.call('subl') subprocess.call(('xdg-open', '/home/fin/documents/learning')) webbrowser.open('https://reddit.com/r/python') if __name__ == '__main__': action() here's entry in crontab file: @reboot python3 /home/fin/labs/my-cheatcodes/src/dsktp_startup_script/dsktp_startup_script.py > capture_report.txt here's content of capture_report.txt file (i trim several lines, since long, prints folder structures. seems came ...
Read more

elasticsearch - Is the order of operations guaranteed in a bulk update? -

i sending delete , index requests elasticsearch in bulk (the example adapted from docs ): { "delete" : { "_index" : "test", "_type" : "type1", "_id" : "1" } } { "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } } { "field1" : "value1" } the sequence above intended first delete possible document _id=1 , index new document same _id=1 . is order of actions guaranteed? in other words, example above, can sure delete not touch document index ed afterwards (because order not respected reason or another)? the delete operation useless in scenario, if index document same id, automatically , implicitly delete/replace previous document same id. so if document id=1 exists, sending below command replace (read delete , re-index it) { "index" : { "_index" : "test", ...
Read more

Slow performance first queries on SQL Azure -

i have small database (50mb) , i'm on basic plan. there single user, need create many databases (always 1 per user) since used training purposes. each database created doing following statement: create database training1 copy of modeldatabase1 we seem getting very slow performance when first query database, afterwards seems acceptable. to give idea: have sp: startupevents runs when application started. query takes 25 seconds run first time. seems incredible since database small, , tables query calls don't contain many records. if run procedure afterwards executes immediately... how can avoid this?
Read more