Data Aggregation or Analytics? Think Columns

A Short Primer On Columnar Approach To Data

In RDBMS, data is represented in the form of tuples or rows, while a NoSQL form of database often stores data in the form of columns. Column-centric databases like Cassandra or HBase is the talk of the town today.

With traditional relational model, the data records or entries are represented as rows. It is best suited for typical online transactional systems or application that needs rich transaction ACID properties for its data. In column form of database, the unstructured or semi-structured data or information is aggregated into number of columns or family of columns for further analysis. Column based data store is more suited for data analytics and offers highest performance as there are no strict constraints of data normalization rules.

The concept of analytics is about applying intelligence to the dormant read-only data and therefore collecting such legacy data and persisting it into organized set of fewer columns makes it easier to query and process. These columns are stored as rows with each column having a certain key-value pair of attributes often nested in nature. This type of data organization helps in processing analytical queries faster. A good small example can be of e-commerce domain when dealing with certain category of merchandise, say footwear. Footwear category data can be aggregated and columnized with attributes like brand, type, size, color, style, weather-prone,etc. It becomes simple and fast to analyse category of information in a columnar way. The volume of data can itself reach in thousands or even millions.

Column based database also provides the concept of TTL or time to live for its column data. Products in e-commerce tend to get obsolete or outdated very fast. One can set the TTL on such columns and it will be automatically discarded after the specified time has elapsed. Another feature column based database provides is its ability to compress data that can increase the capacity of its storage. Compression can be configured to apply on closely related segments of data so that the reads and writes are fast. Modern NoSQL databases uses advanced compression techniques and algorithms that can greatly reduce disk IO.

One more important aspect of columns based database is that certain attributes of single family of columns (represented as rows) may vary or be omitted. For example, a footwear column type ‘formals’ may have more attributes than say type ‘flip-flops’. In short the schema of single family of columns may not be consistent. This kind of flexible schema or data model allows for complex data structures to be designed with ease. The design takes an application perspective than a schema perspective. And therefore, you do not have to break your head on applying different normalization rules.

A typical column based DB has the following characteristics.

  • It works on large volume of unstructured or semi-structured data
  • The design model has a more flexible and design is application driven than schema driven
  • Column values are typically stored as key-value attributes that can be nested to create a hierarchy
  • Data model can be or is usually multi-dimensional
  • Column is a unit of storage
  • It is best suited for aggregation and analytics use cases
  • Columns may have a time to live constraint
  • Columns relationships are represented using aggregation

So when you want to perform data aggregation or analytics, think a column based NoSQL database approach!

Trigent Heroes

Dear Colleagues,

Battered by floods, the residents of Cuddalore District in V, over the last two weeks, have watched flash floods and torrential rains sweep away families, belongings and everything they had identified as their own.

In the Trigent office at Bangalore, six people decided to make a difference and planned to visit Cuddalore to participate in the relief activities and contribute whatever they could to lessen the pain and hardship of the people. Vijay Sam, Senthil Kumar, Ramesh. S. Rajesh Kanna, Barathkumar and Sham were soon joined in their enthusiasm by over 70 people from the Bangalore office who donated money, food and supplies.

The six-member team, ‘Cuddalore Relief Fund’, went to the rain-torn and worst affected areas within Cuddalore on December 6, 2015, to distribute relief material including water, medicines, food and blankets.

Summing up the experience, Vijay Sam says, “It was a collaborative effort. Multiple people, including the leadership team, helped in organizing funds, coordination and planning, transportation, loading, packing and delivery. I am truly glad to have been a part of this team.”

For Senthil it was not as much about personal safety as about what he was required to do. “We received information that a lot of relief materials are distributed in Chennai while Cuddalore was being sort of overlooked. We thus chose to distribute relief material in Cuddalore.” He says, matter-of-factly.

We, all at Trigent, are extremely proud of this overwhelming show of solidarity. Trigent as a company would also like to participate by making a matching contribution toward this initiative.

Thank you all for taking up this challenge and working toward a good cause in a timely manner.

Embed Software Product Testing in your DNA

As software product development is getting complex with the demand for breakthrough features and functionalities, software testing techniques are getting even more complex. The introduction of new features opens the door for many test case scenarios. Besides, there are various combinations of platforms, browsers, and devices that need to be tested for each scenario.

We know software product testing is not a new fad; it has seen its fair share of transition from manual testing to test automation and likewise testing tools have also evolved. However, as the market is abuzz with innovative products and platforms, businesses are constantly slogging to be front runners in efficiency and customer experience. Testing is no longer a ritual but a market readiness strategy for any respectable product or application.

New age start-ups have taken the market by storm, as the demand to release quality products faster has become increasingly important. Delay in launch of a product and you succumb to laggard status, while a minor defect and you invite customers and social media scalpel.

Given the demand and shift towards high-quality standards, a large number of software testing companies in India are investing time and money to improve customer experience and satisfaction. Be it a cloud platform or on-premise platform, Software product testing companies make use of the latest technologies and tools to get insights and feedback about the quality of their product. You can embed product testing either as an extension to your internal team or engage an independent validator whose approach would be structured and unbiased.

Outsourcing software product testing

The best strategy to address software product testing is to engage an offshore independent testing vendor. It would not only provide an independent eye but also reduce investments in terms of resource utilization. Other benefits would include greater return on quality assurance, augmented efficiency, flexibility, and minimized revenue cycle.

Before outsourcing your projects to software performance testing companies, do thorough research to see if the company has the required skills, experience, and expertise. It is important to find the right vendor with the right profiling in the performance testing strategy, as it would help your company optimize products and meet the high-quality standards of today’s demanding customers.

.Net Programming – Padma Krishnan Speaks!

Senior .NET Project Manager, Mrs. Padma Krishnan speaks about her experience and a .NET project she is currently handling at Trigent. Mrs. Padma Krishnan is an MBA from University of Texas at Austin with specialization in Information Management. She has more than 20 years of experience in Information Technology(IT) and has spent considerable amount of her career serving clients in the U.S. She handles a bunch of .NET projects and helps client get best in class customized solutions that meet their business goals. Continue reading .Net Programming – Padma Krishnan Speaks!