comcast1
I will start to memo what I have been doing at Comcast. The journey is still going on. There have been a lot of fun projects. Recently it is dwindling down due to significant market change happening.
The first project I like to describe is one of the current projects. It is about processing some personal informations. The data processing has been done in On Prem MSSQL servers. We are in the process of moving the whole operation into Databricks. Years ago, the team of On Prem tried to do something with AWS but couldn’t do it much. Now with Databricks the team felt a bit more comfortable.
What I have been doing is to help the process move to Databricks. And also helping the team to find faster processing solution.
For example, recently, I made 6 hours taking process into 1 hour proces via clever use of Spark and Futures. Seemingly obvious .. but people from On Prem don’t know how Spark and parallel process could work. Knowing the Spark architecture a bit helps one to figure out this clever implementation. It took 2 days fo identify and make it fully operational.