python - SQL / Pandas equivalent -


what pandas equivalent sql query :

select  column1,          sum(column2) a,          count(distinct column3) b,          sum(column2) / count(distinct column3) c     table1 group column1

thanks on that!!

i'm not sure sum(column2) / count(distinct column3) c part can done in same single step, can in 2 steps:

demo:

in [47]: df = pd.dataframe(np.random.randint(0,5,size=(15, 3)), columns=['c1','c2','c3']) in [48]: df out[48]:     c1  c2  c3 0    4   0   3 1    2   3   2 2    1   2   3 3    3   3   0 4    1   0   4 5    1   1   1 6    2   3   3 7    2   2   2 8    4   0   0 9    1   1   0 10   1   3   0 11   4   3   1 12   0   0   3 13   3   1   0 14   4   3   1  in [49]: x = df.groupby('c1').agg({'c2':'sum', 'c3': 'nunique'}).reset_index().rename(columns={'c2':'a', 'c3':'b'})  in [50]: x out[50]:    c1   b 0   0  0  1 1   1  7  4 2   2  8  2 3   3  4  1 4   4  6  3  in [51]: x['c'] = x.a / x.b  in [52]: x out[52]:    c1   b     c 0   0  0  1  0.00 1   1  7  4  1.75 2   2  8  2  4.00 3   3  4  1  4.00 4   4  6  3  2.00

-

Comments

Post a Comment

Popular posts from this blog

java - Jasper subreport showing only one entry from the JSON data source when embedded in the Title band -

Image
i having issues populating data in subreport in title band of main report. data populated correctly in subreport in detail band of main report. trying find wrong in json query. appreciated. json sample data source (expenses.json) { "expenses": { "date": "8 sep 2016", "accounts": [ { "title": "xyz corp (111)", "accountname": "xyz corp", "accountnumber": "111", "transactions": [ { "date": "21 jun 2016", "name": "gas", "price": "17.50" }, { "date": "12 may 2016", "name": "shopping", "price": "111.99" }, { "date": "30 apr 2016", "name"
Read more

mapreduce - Resource manager does not transit to active state from standby -

one spark job running more 23 days , caused resource manager crash. after restarting resource manager istance (there 2 of them in our cluster) both of them stayed in standby state. and getting error: error org.apache.hadoop.yarn.server.resourcemanager.resourcemanager failed load/recover state org.apache.hadoop.yarn.exceptions.yarnexception: application id application_1470300000724_40101 present! cannot add duplicate! we not kill 'application_1470300000724_40101' yarn resource manager not working. killed instances unix level on nodes dint work. have tried rebooting nodes , still same. somewhere 1 entry of job still there , preventing resource manager elected active. using cloudera 5.3.0 , can see issue has been addressed , resolved in cloudera 5.3.3. @ moment need workaround past now. to resolve issue can format rmstatestore executing below command: yarn resourcemanager -format-state-store but careful clear application history executed befor
Read more

serialization - Convert Any type in scala to Array[Byte] and back -

i have following question: i have variable value in program declared value. i want convert value byte array.. how can serialize byte array , back? found examples related other types such double or int, not any. this should need. it's pretty similar how 1 in java. import java.io.{bytearrayinputstream, bytearrayoutputstream, objectinputstream, objectoutputstream} object serialization extends app { def serialise(value: any): array[byte] = { val stream: bytearrayoutputstream = new bytearrayoutputstream() val oos = new objectoutputstream(stream) oos.writeobject(value) oos.close stream.tobytearray } def deserialise(bytes: array[byte]): = { val ois = new objectinputstream(new bytearrayinputstream(bytes)) val value = ois.readobject ois.close value } println(deserialise(serialise("my test"))) println(deserialise(serialise(list(1)))) println(deserialise(serialise(map(1 -> 2)))) println(deserialise(seri
Read more