scala - Laziness of spark pipline -


i have question laziness of apache spark pipelines. understand spark transformations , actions.

let's take example:

the following snippet transformation returns rdd of filenames in folder. 

val filenames = sc.wholetextfiles("pathtodirectory").map( _ match { case (filename, content) =>  filename })

if execute action :

filenames.count

spark executing defined transformations including loading 'filename' rdd , 'content' rdd memory if 'content' not used

is there better way write spark transformations such as: not used code (like 'content' rdd in previous example) not evaluated while executing actions?

thank feedback.


-

Comments

Post a Comment

Popular posts from this blog

many to many - Django Rest Framework ManyToMany filter multiple values -

i have 2 models, 1 defining users, other defining labels on these users. using django rest framework create api. able query users @ least labels ids 1 , 2. for instance users' labels are: [(1,2), (1,2,3), (2,3), (1,3)] want query return [(1,2), (1,2,3)] . so far, i've managed query users given label (let's id=1) doing: /api/users/?labels=1 , unable query users labels 1 , 2. i've tried /api/users/?labels=1,2 or /api/users/?labels=1&labels=2 return invalid users, i.e. users without labels 1 or 2... any welcome. thanks, dimitry github test repo: https://github.com/thedimlebowski/drf-m2m-filter code: models.py class label(models.model): name = models.charfield(max_length = 60) class user(models.model): labels = models.manytomanyfield(label) filters.py class userfilter(django_filters.filterset): labels = django_filters.filters.baseinfilter( name='labels', lookup_type='in', ) class meta:...
Read more

java - Jasper subreport showing only one entry from the JSON data source when embedded in the Title band -

Image
i having issues populating data in subreport in title band of main report. data populated correctly in subreport in detail band of main report. trying find wrong in json query. appreciated. json sample data source (expenses.json) { "expenses": { "date": "8 sep 2016", "accounts": [ { "title": "xyz corp (111)", "accountname": "xyz corp", "accountnumber": "111", "transactions": [ { "date": "21 jun 2016", "name": "gas", "price": "17.50" }, { "date": "12 may 2016", "name": "shopping", "price": "111.99" }, { "date": "30 apr 2016", "name...
Read more

Java Entity Manager - JSON reader was expecting a value but found 'db' -

i have code working: entitymanager entitymanager = getentitymanager(); entitymanager.gettransaction().begin(); string query = "db.band.find({})"; list<band> list = (list<band>) entitymanager.createnativequery(query, band.class).getresultlist(); entitymanager.close(); return list; it returns list no problem. want sort list date : entitymanager entitymanager = getentitymanager(); entitymanager.gettransaction().begin(); string query = "db.band.find({something__id:objectid(\"" + myid + "\")}).sort({\"somethingelse.date\":-1})"; list<band> list = (list<band>) entitymanager.createnativequery(query, band.class).getresultlist(); entitymanager.close(); return list; my console gives me message: org.bson.json.jsonparseexception: json reader expecting value found 'db' i checked string working in mongo console , did. ideas? edit: tried putting "something__id" between quotes because would...
Read more