SPARK SQL update table based on condition

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

SPARK SQL update table based on condition

Suresh Ayyavoo
Hi All,

Senario: update existing row based on column value in a table, this is how is do in SPARK SQL, i'm i doing right? is there any better way to do it in spark sql?

1. Table (zeppelin-report) row should be updated based on column value

1. select * from table1 where id not in(select id from table1)
(filter the row that need to be updated)
SaveMode.Append (tableB) temp table

2. read new row and append to above table
SaveMode.Append (tableB) temp table

3. SaveMode.Overwrite(zeppelin-report)
This zeppelin report table data is dropped

4.save tableB as Zeppelin-report table

5.SaveMode.Overwrite(tableB)
drop the value of temp table for reuse


Suresh Ayyavoo

Solution Architect / R&D Lead

iAPPS Pte. Ltd.

3 Fusionopolis Way, Symbiosis #13-25 S(138633)

[O] 64631795   [F] 6778 5300 [M] 91540224

Website: www.iappsasia.com  

Facebook: www.facebook.com/iappsasia 

Youtube: www.youtube.com/user/iAPPSasia

Reply | Threaded
Open this post in threaded view
|

Fwd: SPARK SQL update table based on condition

Suresh Ayyavoo
any comments?


---------- Forwarded message ----------
From: Suresh Ayyavoo <[hidden email]>
Date: Tue, Oct 17, 2017 at 4:40 PM
Subject: SPARK SQL update table based on condition
To: [hidden email]


Hi All,

Senario: update existing row based on column value in a table, this is how it's done in SPARK SQL, i'm i doing right? is there any better way to do it in spark sql?

1. Table (zeppelin-report) row should be updated based on column value

1. select * from table1 where id not in(select id from table1)
(filter the row that need to be updated)
SaveMode.Append (tableB) temp table

2. read new row and append to above table
SaveMode.Append (tableB) temp table

3. SaveMode.Overwrite(zeppelin-report)
This zeppelin report table data is dropped

4.save tableB as Zeppelin-report table

5.SaveMode.Overwrite(tableB)
drop the value of temp table for reuse