comcast3-wrongway

While I had done some interesting projects, there also have been not the best decisions I have seen.

The organization I worked is relatively small organization under Comcast umbrella.

There was a big divide in technology used in this org, even if it is a very small org. One side is linux, hadoop and AWS. The other side is Microsoft based. One side uses Postgres, shell, java and scripts. The other side uses MSSQL, visual studio and SSIS. One side is quite automated and the other side is somewhat manual.

We were moving fast at that time, we just adopted hadoop technology and used MapR Hadoop. The reason for MapR Hadoop is its offering NFS mount. That eases some users who are more accustomed to explorej file system via basic ’ls’ command.

Then, our team decided move to AWS . Hadoop is just too expensive to justify.

As we were moving to AWS, we were trying to help the other side Microsoft crowd.

When we were chatting with the crowd, one of them proudly said, he extracted some strings from each line of a file by

  1. load the file into MSSQL table
  2. split each row using split function
  3. extract strings from each row
  4. dump the resulting table into file

When you can do all these in one bash line…. we couldn’t say anything.

The stupidity has no bound