Organizations today have a lot of data. What they don't have is what comes after the data. We take your data from any source as input and deliver intelligence, recommended decisions and actions as output.

Our cloud-native, data-agnostic platform is currently:

connected to 110+ data sources and counting

processing 114M alerts and events per day

Where do we store collected processed data?

 as Cloud agnostic Highly Available Object Storage for our Platform that can be Scaled Up or Scaled Out on demand to store the Processed logs. MinIO Object storage acts a Data lake layer of the Human Managed Platform. Life Cycle management policies are applied in MinIO such that only the fast access to hot tier data lives in MinIO and the rest are further extended to the Cloud’s native Object storage service like S3, Cloud Storage etc.


We created custom Cloud agnostic Helm chart and templates to deploy the appropriates that are needed first for MinIO. MinIO Tenants are then deployed on respective Kubernetes Clusters in various Clouds and are replicated amongst each other. Hence even if a whole Cloud solution fails, HM Platform’s Object storage solution would still function as they are replicated across other Tenants in other Clouds. This supports seamless continuation of service consumption. Each MinIO Tenant is separately monitored by Monitoring team using Prometheus metrics and Grafana dashboards. Human Managed uses Istio Ingress Gateway to effectively utilize the Cloud Load Balancers to support ingress to the Object storage.

Provisioning Object storage solution in Human Managed is automated and would take approximately 30 minutes to come to live.

 All product names, logos and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos and brands does not imply endorsement.

To get the latest news from Human Managed, follow us on 

Want to discuss how we can help solve your cyber, digital, or risk operations through data?

Please contact us at hello@humanmanaged.com.

How we scale cloud agnostic object storage with MinIO

Scaling cloud agnostic object storage

To get anything out of big data, you first need to ingest data from multiple sources. Big data collection is notoriously difficult. But does it have to be?

Imagine having one smartphone that can only handle one mobile application at a time. It would be manageable if you only use 1 or 2 applications, but what if you have 50 mobile applications: One smartphone for Google News, another one for Facebook, another one for Instagram, and so on. Sounds awful, right?!

That is one of the main problems we’ve encountered in trying to develop the data ingestion for our Collect Microservice at Human Managed. Our platform generates intelligence and action from customers’ data, which means we work with many data sources as our input for analysis(currently standing at 30+ data sources and counting!)

Initially, we developed numerous bash and python scripts running via cron job or AWS lambda just to automatically fetch the data from multiple APIs and data sources. However, this became difficult for us to manage once we tried to scale up and replicate for each data source and customer that we have.

We had to find a way to address this with something that we can 

, an open-source data integration platform that syncs data from applications, APIs, and databases and transfer them to the destination that you want such as database and object storage. Remember the analogy I gave earlier? Airbyte is 

the type of smartphone that can handle multiple mobile applications, which makes data integration simple, secure, and extensible.

We utilized existing Airbyte connectors that fits into our use cases and also developed our own connectors when necessary. Since Airbyte is an open-source platform, the connectors available are developed by the community where you can also 

(We plan to contribute our custom connectors and improvements in existing ones so that we could also give back to the community, so stay tuned!)

Once we converted our automation scripts into Airbyte connectors, we were able to easily implement and organize the batch collection that happens within our Collect microservice. In replicating the same connection but for another customer, we just set the configuration in the existing connector, set the destination, and that’s it! We were able to integrate the another data source into our platform in just minutes.

Referring back to the analogy of having multiple smartphone, with Airbyte, we have now a 

one single powerful smartphone that can handle and receive data from multiple mobile applications

). This solved the problem of having to maintain, and organize the automation of ingesting data which can also be customized, scaled, and managed however suits our needs.

There will always be a setback in using any other tool, even if it’s Airbyte. The first thing that comes into mind is the learning curve required to develop and maintain custom airbyte connectors. If you are not familiar with the concepts, you’ll have to figure out a lot of these essential concepts to grasp in order to properly understand and implement an Airbyte connector. But once figured out, most of the development will be easier than before because of the 

As for its limitation, the current lack of support for data backup of raw data in-between the Extract (E) and Load (L) is not yet possible. This conflicts with our architecture of storing the raw logs without the metadata added by Airbyte.

Even though I’ve also mentioned the cons of implementing Airbyte. The benefits far outweigh the disadvantages and gives you the following capabilities:

Open source and free for all to contribute

Customizable, Modular, Scalable, Secured, and Compliant

Set Connection Syncs and Incremental Stream State

The capacity and features of Airbyte to unify your data integration pipelines under one fully managed platform is fit to develop our Collect microservice since it enables us to easily integrate data sources, develop custom connectors when necessary, and manage these connections without worrying too much on security and compliance.

With a fast growing community and contributions, more and more connectors will be made and improved in the future which will continue to innovate and push to evolve the data ingestion capabilities of Airbyte.

You could learn more about Airbyte and get involved in the open-source platform by visiting the 

Note: This blog was originally published in August 2022 and has been updated for accuracy. 

Categories

DataOps

Scaling cloud agnostic object storage

Orchestrating data collection with Airbyte