hadoop - aggregate ordered rows in hive table -


i have table in hive 4 columns this:

row_id| user_id|product_id| duration 1       1       product1    3 2       1       product1    1 3       1       product2    6 4       1       product3    2 5       1       product1    4 6       1       product4    3 7       1       product4    5 8       1       product4    7 9       2       product4    3 10      2       product4    6

i want aggregate rows of same product each user, sum duration , count clicks if consequent in order

row_id| user_id|product_id |duration_per_product |clicks_per_product 1       1       product1    4                       2 2       1       product2    6                       1 3       1       product3    2                       1 4       1       product1    4                       1 5       1       product4    15                      3 6       2       product4    9                       2

any ideas how in hive 1.1.0?

group doesn't work because don't want group products if consequent , have tried case,lag , lead didn't work!

thanks!

first off, want in loop, hive not suitable these kind of problems.

that being said, here approach should work:

suppose our dataset

1       1       product1    3 2       1       product1    1 3       1       product2    6 4       1       product1    4

identify starter rows: 1,3,4 can done doing left join on id=id+1 , seeing whether user , product match.

join onto these starters user , product:

1 1 1 2 1 4 3 3 4 1 4 2 4 4

filter out things in wrong order (starter after row), remaining are: 

1 1 1 2 1 4 3 3 4 4

group find maximum valid starter each row, remaining be:

1 1  1 2 3 3 4 4

now join reattach relevant dimensions

1 1 3 1 2 1 3 3 6 4 4 4

now can results grouping on starter id.

1 4 3 6 4 4

of course can choose use join attach name of product.

1 product1 4 3 product2 6 4 product1 4

and all!


-

Comments

Post a Comment

Popular posts from this blog

many to many - Django Rest Framework ManyToMany filter multiple values -

i have 2 models, 1 defining users, other defining labels on these users. using django rest framework create api. able query users @ least labels ids 1 , 2. for instance users' labels are: [(1,2), (1,2,3), (2,3), (1,3)] want query return [(1,2), (1,2,3)] . so far, i've managed query users given label (let's id=1) doing: /api/users/?labels=1 , unable query users labels 1 , 2. i've tried /api/users/?labels=1,2 or /api/users/?labels=1&labels=2 return invalid users, i.e. users without labels 1 or 2... any welcome. thanks, dimitry github test repo: https://github.com/thedimlebowski/drf-m2m-filter code: models.py class label(models.model): name = models.charfield(max_length = 60) class user(models.model): labels = models.manytomanyfield(label) filters.py class userfilter(django_filters.filterset): labels = django_filters.filters.baseinfilter( name='labels', lookup_type='in', ) class meta:...
Read more

java - Jasper subreport showing only one entry from the JSON data source when embedded in the Title band -

Image
i having issues populating data in subreport in title band of main report. data populated correctly in subreport in detail band of main report. trying find wrong in json query. appreciated. json sample data source (expenses.json) { "expenses": { "date": "8 sep 2016", "accounts": [ { "title": "xyz corp (111)", "accountname": "xyz corp", "accountnumber": "111", "transactions": [ { "date": "21 jun 2016", "name": "gas", "price": "17.50" }, { "date": "12 may 2016", "name": "shopping", "price": "111.99" }, { "date": "30 apr 2016", "name...
Read more

Java Entity Manager - JSON reader was expecting a value but found 'db' -

i have code working: entitymanager entitymanager = getentitymanager(); entitymanager.gettransaction().begin(); string query = "db.band.find({})"; list<band> list = (list<band>) entitymanager.createnativequery(query, band.class).getresultlist(); entitymanager.close(); return list; it returns list no problem. want sort list date : entitymanager entitymanager = getentitymanager(); entitymanager.gettransaction().begin(); string query = "db.band.find({something__id:objectid(\"" + myid + "\")}).sort({\"somethingelse.date\":-1})"; list<band> list = (list<band>) entitymanager.createnativequery(query, band.class).getresultlist(); entitymanager.close(); return list; my console gives me message: org.bson.json.jsonparseexception: json reader expecting value found 'db' i checked string working in mongo console , did. ideas? edit: tried putting "something__id" between quotes because would...
Read more