Spark Streaming with MongoDB backend -


i have requirement read csv files pumped telemetry equipment location on cloud , store relevant data mongodb store. using spark streaming read new files (they arrive every minute, more frequent) , using momgodb-spark connector.the problem data not being loaded momgodb. have added dataframe's show() steps in code , being displayed @ console correctly, means streaming application reading , processing files expected. final step of saving mongodb not happening. code looks follows

reqdata.foreachrdd { edata =>     import sqlcontext.implicits._     val loaddata = edata.map(w => energydata(w(0).tostring,w(1).tostring,w(2).tostring)).todf()     loaddata.show()     loaddata.printschema();      mongospark.save(loaddata.write.option("uri","mongodb://127.0.0.1:27017/storedata.energydata").mode("overwrite")) }  ssc.start()

the loaddata.show() function displaying data fine.

i have checked mongodb logs , found few strange lines like

"2016-09-07t08:12:30.109-0700 network [initandlisten] connection accepted 127.0.0.1:55694 #212 (3 connections open) 2016-09-07t08:12:30.111-0700 command [conn212] cmd: drop storedata.energydata"

now, don't understand why mongo drop collection @ all. highly appreciated

i solved myself changing saving mode append:

mongospark.save(loaddata.write.option("uri","mongodb://127.0.0.1:27017/storedata.energydata").mode("append"))

-

Comments

Post a Comment

Popular posts from this blog

java - Jasper subreport showing only one entry from the JSON data source when embedded in the Title band -

Image
i having issues populating data in subreport in title band of main report. data populated correctly in subreport in detail band of main report. trying find wrong in json query. appreciated. json sample data source (expenses.json) { "expenses": { "date": "8 sep 2016", "accounts": [ { "title": "xyz corp (111)", "accountname": "xyz corp", "accountnumber": "111", "transactions": [ { "date": "21 jun 2016", "name": "gas", "price": "17.50" }, { "date": "12 may 2016", "name": "shopping", "price": "111.99" }, { "date": "30 apr 2016", "name...
Read more

serialization - Convert Any type in scala to Array[Byte] and back -

i have following question: i have variable value in program declared value. i want convert value byte array.. how can serialize byte array , back? found examples related other types such double or int, not any. this should need. it's pretty similar how 1 in java. import java.io.{bytearrayinputstream, bytearrayoutputstream, objectinputstream, objectoutputstream} object serialization extends app { def serialise(value: any): array[byte] = { val stream: bytearrayoutputstream = new bytearrayoutputstream() val oos = new objectoutputstream(stream) oos.writeobject(value) oos.close stream.tobytearray } def deserialise(bytes: array[byte]): = { val ois = new objectinputstream(new bytearrayinputstream(bytes)) val value = ois.readobject ois.close value } println(deserialise(serialise("my test"))) println(deserialise(serialise(list(1)))) println(deserialise(serialise(map(1 -> 2)))) println(deserialise(seri...
Read more

SonarQube Plugin for Jenkins does not find SonarQube Scanner executable -

Image
i trying run sonarqube scanner within jenkins post-build step. however, keep getting error message below: ------------------------------------------------------------------------ sonar analysis failed ------------------------------------------------------------------------ fatal: sonarqube scanner executable not found sonarqube build step 'execute sonarqube scanner' marked build failure from similar questions on stackoverflow read 1 should choose "install automatically" sonarqube scanner, have done. my configurations follows: sonarqube 6.0 jenkins 1.609.3 sonarqube plugin 2.4.4 sonarqube servers sonarqube scanner build-step when config sonarqube scanner in manage jenkins --> global tool configuration --> sonarqube scanner, besides select "install automatically", need add installer. see screenshot here: sonarqube scanner --> add installer
Read more