সময়ের আঁকে বাঁকে

সেপ্টেম্বর মাসের ঝলমলে রোদে মার্কিন যুক্ত রাষ্ট্রের পশ্চিম প্রান্তে হারিয়ে যাওয়ার স্বপ্ন দেখলাম একদিন । কিছু বৈশিষ্ট আছে পশ্চিম দিকের লাল মাটির দেশে যা দেখার চেও বেশি অনুভব করা যায় শুধু । ভাগ্যক্রমে ঠান্ডা পড়ার আগেই ঐতিহাসিক সব যুদ্ধের সেই পীঠস্থানে যাওয়া হলো । গাছ পশু প্রাণী সব আলাদা ... অবাক রকমের বৈপরীত্ব জাস্ট কয়েকশো মাইল এদিক ওদিকের মধ্যে ... শুকনো লাল মাটির দেশে সব কিছুই অন্য রকম লাগে । পথের ধারের বিশাল বিশাল লাল পাথর থেকে শুরু করে গাড়ি থেকে নেমে গ্যাস ভরার সময় যত দূরে চোখ যায় ততদূর রোমাঞ্চকর উঁচুনিচু ল্যান্ডস্ক্যাপের দিকে তাকিয়ে চোখের সামনে ভেসে ওঠে কয়েকশো বছর আগের সিউক্স, আপাচে ও নাভাজো উপজাতিদের জীবিকা । বহু জায়গাতে তাদের দোকান ও ষ্টল ও মিউজিয়াম এ গিয়ে কথা বলে জানা যায় মানুষ আজ যেমন তখনও তেমনি ছিল ... পার্থক্য হচ্ছে শুধু আজ রয়েছে ইউনাইটেড নেশনস আর জি৮ সামিটের শৃঙ্খল আর তখন ছিল শুধু যার যত বড় আগ্নেয়াস্ত্র । যুগ যুগ ধরে যারা দাপিয়ে বিচরণ করেছিল উত্তর থেকে দক্ষিণ, যারা ঘোড়ার পিঠে চড়ে অভ্রান্ত নিশানায় করতো বাইসন শিকার, যাদের বীরত্বের প্রতীক নিউ মক্সিকো থেকে শুরু করে অ্যারিজোনা হয়ে উটাহ থেকে ডাকোটা অবধি পাওয়া যায়, যাদের ইনস্টাগ্রাম পেজ এ আজও ফুটে ওঠে গভীর যন্ত্রণার আর্তনাদ, যাদের আত্মসমর্পনের হাড় হিম করে দেওয়া কাহিনী বরাবরের জন্য রাঙিয়ে দিয়েছে মার্কিন ইতিহাস - সেই তাদের পদধ্বনি অসম লড়াইয়ে নীরব থেকে নিস্তব্ধ হয়ে তাদের নিষ্প্রাণ অশ্রূ রয়ে গেলো বিশ্বাসঘাতকতার দৃষ্টান্ত ট্রেইল অফ টিয়ারস নামে বিখ্যাত হয়ে ।

লেক পাওয়েল নামে পরিচিত সুগম ও ইতিহাসের পাতা থেকে উঠে আশা এই বিস্ময় উপলব্ধি করতে হলে অবগত হতে হয় গত সহস্র বছরের ইতিহাসের সাথে - বিশেষ করে ১৯৫৬ - ৬৬ সময়টি গ্লেন ক্যানিয়ন বাঁধের আবির্ভাব এবং আপার বেসিনে জল সঞ্চয়ের মনুষ্যসৃষ্ট এই অভিনব লেকের গল্প । গ্রান্ড ক্যানিয়নের প্রকাণ্ডতা ও রোমাঞ্চে ভরা রোদ আর ছায়ার রেশ কাটার আগেই ছোট্ট শহর পেজ থেকে একটু দূরে নাভাজো স্যান্ডস্টোনের রঙিন প্রদেশে গাঢ় নীল আহ্বানে ভেসে পড়লাম। মোটর চালিত নৌকার ছাদের থেকে প্রকৃতির নিঃস্বার্থ আবেগের প্রলেপে মিশে গিয়ে অনুভব করা যায় তীব্র এক জীবিত উপত্যকার গভীর কাহানি - যেখানে জলের থেকে শুরু করে দুদিকের পাথররাও জানাতে চায় তাদের বছরের পর বছরের অবক্ষয়ের ও সহিষ্ণুতার অমর শিলালিপি । গ্লেন ক্যানিয়নের সান্নিধ্যে আর অভিভাবত্বে প্রতিনিয়ত নিজেকে আবিষ্কার করে চলেছে এন্টিলোপ ক্যানিয়নের ভেতর দিয়ে প্রবাহমান ৫০০ ফুট গভীর মেঘের রঙে তাল মিলিয়ে নীল উন্মাদনা । স্যান্ডস্টোনের ঝলমলে সময়ের রঙের বহির্প্রকাশ দেখে ভুলে যাই কোন দিকে তাকাবো আর কিসের ছবি তুলবো - প্রাচীনত্বের সুরে ভেসে চলার তীক্ষ্ন চমক ।

Read More »

Unix find Lines containing specific string, then Cut and Insert these Lines after specific Lines containing another specific string

The world of data integration and transformation gets more and more exciting every day. Where we have data fuelling intelligent applications, and transformations paving the way to create cleaner and leaner data. Let's dive in to some sorting and ordering transformations using shell scripting which will subsequently be triggered via ODI. We now have an interesting scenario where we have to identify all the records having the string "Parmesan Cheese", and after identifying the unique identifier, we have to cut this record and paste it under the immediate next record having the string "Filet Mignon" with the same unique identifier. Let's see a quick example below:

Base Dataset:
"X1","Y1","Z1","Parmesan Cheese","Yummy","Delicious"
"X1","Y1","Z1","Grilled Salmon","Amazing","Tender"
"X1","Y1","Z1","Filet Mignon","Juicy","Exquisite"
"X2","Y2","Z2","Parmesan Cheese","Yummy","Delicious"
"X2","Y2","Z2","Grilled Salmon","Amazing","Tender"
"X2","Y2","Z2","Filet Mignon","Juicy","Exquisite"

Required Dataset:
"X1","Y1","Z1","Grilled Salmon","Amazing","Tender"
"X1","Y1","Z1","Filet Mignon","Juicy","Exquisite"
"X1","Y1","Z1","Parmesan Cheese","Yummy","Delicious"
"X2","Y2","Z2","Grilled Salmon","Amazing","Tender"
"X2","Y2","Z2","Filet Mignon","Juicy","Exquisite"

"X2","Y2","Z2","Parmesan Cheese","Yummy","Delicious"

The below Unix script will process the data as per our required logic. First it will create a lookup file lookup.txt containing all the records having "Parmesan Cheese".

lookup.txt
"X1","Y1","Z1","Parmesan Cheese","Yummy","Delicious"
"X2","Y2","Z2","Parmesan Cheese","Yummy","Delicious"

In summary, for each record being read in getEntireRecord from this lookup file, it will take the getUniqueRecIdentifier ("X1","Y1","Z1") and find the line number lineNumOfFiletMignon (3) of the "Filet Mignon" record having same identifier ("X1","Y1","Z1"). Now we know where to insert the "Parmesan Cheese" record getEntireRecord - the line number will be lineNumToInsertParmesanCheese which is the next line, so add one (3+1=4).

Full logic:

filename="file.txt"
grep 'Parmesan Cheese' $filename > lookup.txt
lkpfilename="lookup.txt"

while read -r line
do
readLine=$line
getUniqueRecIdentifier="$(cut -c1-8 <<<"$readLine")"
getEntireRecord="$(cut -c1-100 <<<"$readLine")"
generateSameIdFiletMignon=$getUniqueRecIdentifier""',"Filet Mignon"'""
lineNumOfFiletMignon="$(grep -n "$generateSameIdFiletMignon" $filename | head -n 1 | cut -d: -f1)"
lineNumToInsertParmesanCheese=$((lineNumOfFiletMignon + 1))
sed -i ''"$lineNumToInsertParmesanCheese"'i '"$getEntireRecord"'' file.txt
lineNumToBeDeleted="$(grep -n "$getEntireRecord" $filename | head -n 1 | cut -d: -f1)"
sed -i ''"$lineNumToBeDeleted"'d' file.txt
done < "$lkpfilename"

rm $lkpfilename

Detailed Explanation: To identify lineNumOfFiletMignon we are using grep -n as seen below, with head -n 1 to get the first record for the specific combination, even though we know it will give only one record in our case. Then we have cut -d: -f1 to get the first column as the Unix line number.

lineNumOfFiletMignon="$(grep -n "$generateSameIdFiletMignon" $filename | head -n 1 | cut -d: -f1)"

Now we are adding one to lineNumOfFiletMignon to get lineNumToInsertParmesanCheese.

lineNumToInsertParmesanCheese=$((lineNumOfFiletMignon + 1))

Once we have identified lineNumToInsertParmesanCheese we can use sed -i then the line number where we want to insert our record followed by the record string and file name. Since we are iteratively storing the entire records iteratively in getEntireRecord from lookup.txt, we are using the same in the sed -i command.

sed -i ''"$lineNumToInsertParmesanCheese"'i '"$getEntireRecord"'' file.txt

After we do the above, we are going to have a duplicate original record of "Parmesan Cheese" that has to be deleted, this is calculated in lineNumToBeDeleted by using the entire record string getEntireRecord which was retrieved from lookup.txt.

lineNumToBeDeleted="$(grep -n "$getEntireRecord" $filename | head -n 1 | cut -d: -f1)"

The duplicate original line will be removed by the below sed -i command where we are providing the line number lineNumToBeDeleted to be deleted with d at the end for deletion, followed by the file name.

sed -i ''"$lineNumToBeDeleted"'d' file.txt

Then at the end we can safely delete our lookup file, which was happily storing all the "Parmesan Cheese" for us until now!

The above activity can also be done in Excel macro, but considering the amount of maintenance and scalability factors, we are clear which option to choose now.

Read More »

ODI BI Apps Machine Learning Power-Up

Like many millenials, one of my favorite video game was Mario. The most satisfying thing about Super Mario were the power-up mushrooms - that gave it added abilities, to keep conquering world after world. Similarly, today when data and information are created at an accelerating rate, outstripping the ability of humans to keep up - it becomes imperative for enterprise operations to enable a digital workforce to achieve demonstrable gains in efficiency and productivity.

Operational Analytics, with subjective experience, is indeed very much useful - but it's more inclined towards Descriptive Analytics, based on what has already happened in the past. With Predictive Analytics, it opens up a whole new world where we can design algorithms to detect complex pattern - and provide powerful insights to predict the future. The more powerful our mathematical algorithm, and the more robust our datasets, the better we get with our statistical and strategic inferences.

With a journey that started few months back with analyzing and synthesizing vast amounts of logs generated by Oracle BI Apps, Oracle Hyperion Essbase, and Oracle Data Integrator, it's fascinating to see how today unprecedented levels of efficiency and quality can be achieved by transcending conventional performance tradeoffs. Let's coin in the term Intelligent Process Automation here - since it will not be fair to navigate this picturesque landscape without getting a deep feel of the next-gen tools forming the core of this cognitive technical process.

How does IPA fit in our ODI BI Apps Power-Up? Well, wait for it, let's put it out there in as much crispy and munchy (reminding me of chocolate chip cookies...umm..) way as possible. We get to know the answers to all the following questions today, in near real-time. When does the application encounter "ODI-10188: Error while login from OPSS" due to Authentication issues which causes critical Production ODI jobs to fail? When does the application face errors due to "Unable to create connection to LDAP" which creates fatal scenarios in complex running processes? When does the application face errors like "LDAP response read timed out" which causes ODI jobs or online OBIEE reports to error out? Can our IPA model figure out what went wrong by itself and let me know?

Now, let's see what happens when we create a model that will continuously "teach" our "agent" to "learn" from the stream of situational data, analyze the same, and respond to complex queries. What happens when we inject decision-making capabilities to enhance our "agent", such that it is able to learn and adapt with time? We start getting answers to all the following questions - how stable does the system look? Since applications and jobs running fine does not necessarily indicate everything is fine, should we be aware of any "indicators" that can serve as giving us predictive information of the future state? Why is the application or system behavior the way it is now? Which teams need to be involved right away when the system behaves in a specific pattern? Can the system auto-heal given a specific scenario and then share that information? When can we anticipate a specific good news scenario that happened in the past? How can we predict a major upcoming issue that has happened in the past? How close are we to reaching our specific target figures?

Thus with the interplay of concepts, technologies, it's fascinating to see how we are able to create strategic assets, helping us achieve unprecedented levels of efficiency, control, quality, and most importantly, speed - which is definitely poised to transform the existing workforce, with radically enhanced response times, and ofcourse, reduced operational risks.

Read More »

SNP_SESSION - The Data Analyst’s Dream Table of Oracle Data Integrator

The Oracle Business Intelligence Applications stack provides an array of tools during the implementation, and each of them comes with its rich set of features. The awesomeness comes when we get to experience several business use-cases and scenarios, analyze the metrics and data, interpret them along the lines of the business process, and ultimately when also we encounter product limitations - and discover amazing ideas to make our lives easier. Happiness...as deep the word sounds, becomes almost synonymous in such cases, as we make breakthroughs through innovative complex ideas.

All of us are aware of this repository table SNP_SESSION in ODI, the unattractive component that shows lots of rows and numbers and dates, often just helps only to find a specific information and then we are done with it. In an environment where overnight several incremental loads consume around 6 hours daily, it generates lots of logs and data and writes to all of the repository tables, including session level details in SNP_SESSION. All information of every session like rows processed, rows inserted, rows updated, period and filter variables used, duration taken, start time, end time - are logged in SNP_SESSION.

To understand the prowess of SNP_SESSION, we need to get to a few questions first, and then the train of thoughts and discoveries can follow. For a session am interested in, what is the behavior of this session over the last 4 months? Does it have any pattern during specific periods? Does it have any relation with other sessions’ attributes? Does a data volume of another session or duration of a different session influence this? Since rows processed do not always proportionately impact session durations, does a % variance of a different session impact the session am interested in to an extent? Say in my today’s load, can I find which scenarios from the past repeated today, say with similar data volume or % variance in duration? Can I foresee untold information or what is going to happen as the loads progress by real-time analysis of the data?

It’s been very exciting to know over the last few weeks that all of the above questions can be answered, tremendously by using SNP_SESSION, and with some help from SNP_LP_INST and SNP_LPI_STEP. We have implemented a solution which is now in its final testing phase with live data, and will heavily complement manual human monitoring activities - by providing root causes before the impact happens, and providing additional insights into the application which otherwise often gets overlooked due to the vastness of the system.

We have calculated the weighted average of each session duration, load plan wise, over a period of last several months, with an algorithm that took a long time to develop after a lot of brainstorming. Then came the perilous task of calculating the standard deviation of the weight samples so as to help calculate the accuracy of our analysis, but it finally happened! Next came analyzing the data volume - with NB_ROW, NB_INS, NB_UPD already available in SNP_SESSION and waiting for us. Comparing today’s volume with the weighted average for the corresponding session itself started giving insights, but we wanted more. We asked what next, what if, why now, what then, and each metric opened up new paths before us to explore.

Each field of SNP_SESSION gave rise to almost 3-4 metrics of it’s own, giving rise to real-time daily calculation and analysis of the datasets during executions, and the impact it causes to the parent load plan. During the execution of each load, we are able to get insightful emails consisting of detailed analysis of the load - and a similar day in history if today’s scenario matched, and what happened then, and how the day went with the consequent activities.

But again, we need a single indicating factor, giving rising to the calculation of probability - as a single point of figure to indicate the possibility of actual realization of our real-time predictions, for each prediction. Hence more brainstormings followed, and finally now every email gets tagged by a probability that makes it so much more meaningful. Thus it becomes so true in today’s world, data is only useful when we know how to process it to our benefit - and it will truly continue to become more and more the case in future!

Read More »

Sensational Sequoia

The exquisite Sierra Nevada, in the magnificent slopes of California, is the home to some of the oldest living things on earth – the dazzling Giant Sequoia trees, some as old as 3500 years. The stunning facts of these exquisite trees were as overwhelming to me as a 10-year old, as it is today – and the long dream of experiencing these breathtaking wildernesses came true few days back.

Driving via Three Rivers Visalia through the wild fascinating roads uphill to the Giant Forest is an experience in itself that takes some time to sink in. The gradual change in the landscape around us as we approach is magnificent, the appearance of the trees, the heights, the feeling of suddenly shrinking is so electrifying – one just gets dazed.

The role of forest fires on the lifecycle of the Sequoia trees is interesting. Due to reduction in the number of natural forest fires today, the Park Service executes controlled fires to remove competing vegetation to allow Sequoia seedlings to germinate - which has difficulty otherwise. The Giant Forest Museum is a storehouse of some gripping figures and facts – the Sentinel tree greets all the visitors as we enter the museum, a statuesque of over 250 feet high.

As we went down from the parking lot via the Sherman Tree Trail, we could feel the energy and excitement around us – the weather felt perfect for visitors, not a surprise given that the Sequoias thrive best in humid climates. When we reached, standing in front of the General Sherman Tree – the magnificence and awe was too much to absorb, and took an hour of lingering around; to really believe it's happening for me. A part of me felt complete. For me, it was always more than a tree. That day I realized it for real. The largest living tree – the all so familiar impressive artistry that we all have craved to see for so long after reading about it in books – right before us.

The enchanting walks around Sequoia forest, the rousing emotions, the captivating views, the grace and elegance of surviving thousands of years withstanding fires and storms, converted it into a spellbinding magical forest – resulting in an indelible weekend.

The wildlife comprises of hundreds of black bears, rattlesnakes, mountain lions, and some friendly rodents. The visitor guide below is really well versed with all the different situations that our curious minds wandered on, and it was interesting to read.

Read More »