The Human Managed platform operates on the edges of technology-forward enterprises in the essential services sectors. Some questions at the forefront of every decision-makers’ mind are:

How do I get more value from data using AI?

How do I empower and scale my team’s decision-making with AI?

How do I improve business resiliency with AI?

These are complex questions to answer. But a great silver lining is that you can solve all these problems and more with distributed and scalable data operations. We pick the three ops decisions with the highest impact and value for businesses of all sizes and industries as we enter the age of AI headfirst.

In today’s digital world, data directly impacts your organisation’s top and bottom lines. The last few years have seen explosive growth of AI tools and services accessible to players of all sizes and sectors. Within this fast-changing environment, businesses should focus on the value they can generate from data. The faster, more agile and scalable your dataops is, the bigger your AI-powered opportunities are. For enterprises, this means operationalising any data from anywhere.

Challenges: Now you have the data, what do you do with it?

Businesses are usually adept at identifying the business problems they want to solve, what data is essential for the problem, and where it will come from. The real challenge comes after.

For example, say your cyber goal is to control exposures impacting your most critical service, such as a banking app server. Some key sources that generate data related to assets, violations, and vulnerabilities are managed endpoint detection tools, vulnerability management SaaS, and external threat databases. These data sources could be sensors deployed on-premise, in the public cloud, or in your software provider’s cloud. The data from these sources vary in volume, format, schema, integration methods, etc.

Getting all the relevant data from multiple sources, synthesising them, and processing them through a unified pipeline is infamously challenging to build and scale across numerous use cases. This limits the depth of analysis businesses can run on their data and drives siloed or tool-driven approaches to problem-solving.

Opportunities: Distributed data engineering as input for AI models

The Human Managed DataOps platform ensures that whatever data you want to analyse is continuously collected, processed, and stored to be AI-ready.

One of our customers, a leading ASEAN conglomerate, approached us with a widely shared problem in cyber operations: effective prioritisation. They had struggled with siloed asset databases for 20+ years and managing disparate cybersecurity tools across the public cloud, software vendor cloud, and on-premise. This resulted in manual and slow cyber operations, where many issues slipped through.

The goal was to automatically contextualise and prioritise our customer’s cybersecurity issues as and when the alerts are generated. The customer’s job was completed when they chose 10 data sources to provide us with the required input (alerts, logs, metrics from SaaS and on-prem systems) and context (asset databases, strategies, and business logic).

The HM platform onboarded the customer’s data for continuous cyber operations in less than a month. We catalogued their assets, controls and attributes and structured their cybersecurity alerts, logs and metrics under one data schema and model.

Our solution meant that the variable components — the ingredients — for any analysis by any kind of computing, whether rule-based programming or AI/ML models, were ready.


MLOps: Tuning AI models personalised for your business

Although AI, machine learning, and neural networks are not new, what drove the now-familiar explosion of new AI-powered capabilities is the underlying pre-trained models that picked up speed coined in 2021 as foundation models. Foundation models (sometimes called general-purpose AI or GPAI) are AI neural networks trained on massive raw, unstructured data, often with unsupervised learning, that can be adapted to perform a wide range of general tasks without human intervention.

Foundation model, including generative AI-powered apps beyond our wildest imaginations like ChatGPT, can understand language, generate text and images, and converse in natural language. As AI’s capabilities continue to capture consumers’ hearts and minds, businesses are in a race to decide how best they can adapt AI models in their operations.

Challenges: Now you have the AI models, how do you make it work for your business context?

Foundation models are called ‘foundation’ because they act as the base to build apps that solve different problems. Open and commercial AI models are trained on generalised and unlabelled data, such as data scraped from the Internet or untraceable databases.

Generic AI models, no matter how advanced, will not magically produce accurate and precise outputs suited for your unique business context. For machine learning to be operational, AI models must be trained, tuned, and improved with data, logic, and patterns unique to your business.

Opportunities: Decision models combining tribal knowledge and trending knowledge

The Human Managed MLOps platform tunes foundation AI models with a knowledge base that is unique to each customer’s business context and incorporates a human-in-the-loop feedback cycle to continuously improve and measure the performance of the customer’s personalised AI model. This way, our customers get the best of both worlds: trending knowledge that builds foundation models and tribal knowledge that builds their own context models.

One of the business-critical use cases we apply MLOps for is fraud detection and management for a global banking customer, where we continuously tune AI models with customers’ personalised data to build their own contextualised fraud model that classifies, predicts, or indicates suspicious fraudulent activities.

Over decades of operations, our customer has amassed tribal knowledge and experiences on their fraud landscape and wanted an automated and scalable solution to increase their detection accuracy and decrease their response time.

Examples of their tribal knowledge include indicators of different types of fraud (e.g. repeated withdrawals from the same account on the same day), when a detection alert should be triggered (e.g. withdrawal amount is higher than US$50,000), and what needs to be done when there is a detection (e.g. send a critical alert to fraud investigation unit to investigate suspicious transaction).

All these data points form the unique context of our customer, such as their business logics, prioritised assets, and historical patterns. The Human Managed MLOps platform transforms these data points into structured data and code to form features and labels that tune the customer’s contextualised fraud model.

Once the input data, AI/ML processes, and desired output are aligned, it’s a virtuous circle of human and machine collaboration because ML models improve with more training and feedback. Our customer continues to add more datasets, rules and conditions while the platform continues to learn from data to improve the accuracy of their ransomware model.

IntelOps: Applying AI for better and faster decisions and actions

In today’s turbulent digital world, where speed and agility have become a necessity rather than an aspiration, leaders should pay attention to how they can make their businesses resilient. Business resiliency is about sustaining during unknown conditions and improving and coming out stronger from VUCA (volatile, uncertain, complex, ambiguous) conditions.

Resiliency, when executed right, can turn challenges into opportunities to protect their assets better, create more revenue, and make customers happier. Achieving business resiliency through data is highly impactful for many reasons, not least to understand the changing environment in which you operate, as well as generate intel to make better and faster decisions and actions.

Challenges: Now you have the intel, how do you apply it with the right priorities?

By 2025, it has been predicted that data will be embedded in every decision, interaction, and process (Source: McKinsey, The data-driven enterprise of 2025).

We’ve seen how context can be built through data and models. DataOps prepares the data to generate valuable intel, and MLOps improves AI models to learn from contextualised data.

Getting the proper intel consistently is a challenging feat. However, the benefits of the right intel are limited if you do not apply it to the right problem at the right time. How do businesses ensure this? Even with the best intel made available through DataOps and MLOps, if it is not served to the right audience at the right time, the value of that intel is not realised, and the window of opportunity closes. The challenge here is to bring DataOps and MLOps processes together in a consistent and scalable operational cycle across the business, which we call IntelOps.

Opportunities: Data-driven resiliency with I.DE.A.

Acting with speed and accurate prioritisation is critical to business resiliency. The Human Managed platform services are designed to empower the end-to-end decision-making process, from generating personalised intel to ranked decisions and prescriptive actions. We call this the IDEA Model (Intelligence Decision Action).

Some of our most subscribed services are data-driven asset management, attack surface management, fraud, and security posture management. For every service, our DataOps platform ensures that all of our customers’ data gets processed and analysed to create contextualised insights, which then act as input for the MLOps platform to continuously tune the AI model for more accurate and precise intel. Finally, our IntelOps platform takes contextualised intel to generate ranked decisions and prescriptive actions based on calculated priorities and impact.

One of our customers subscribed to our network security posture management service to move the needle on fixing 40,000 violations that have been open and unresolved for over two years. Contextualised intel on the violations — no matter how organised and easy to understand — did not get their operations team to decide on the next steps because the number of issues was so high.

To make progress on the cyber posture issue, the customer asked for decisions and actions that would have the “biggest bang for the buck”, which is precisely what we delivered. Our IntelOps platform ran computation on 40,000 reported violations on 1000 network segments, protecting around 50,000 assets accessed by 40,000 employees. As an output, we generated four ranked decisions and 16 prescriptive actions that would remove 40 per cent of all violations and improve 100 per cent of customer’s critical assets. The prescriptive actions were delivered through a nudge-based dispatch system to the users who could affect the change.

Get the data right, apply to unique business context, build business resiliency

In conclusion, a business can harness the potential of AI when it has a complete understanding and control of its enterprise data across all sources. Data must be continuously collected, processed and stored to be AI-ready.

For machine learning to be operational, AI models must be trained, tuned, and improved with data, logic, and patterns unique to your business. Finally, applying AI for better and faster decisions means ensuring the company stays resilient against changing VUCA (volatile, uncertain, complex, ambiguous) conditions.

A business can harness the potential of AI when it has a complete understanding and control of its enterprise data across all sources

3 data decisions to make in 2024 for businesses to become AI-native

Organizations today have a lot of data. What they don't have is what comes after the data. We take your data from any source as input and deliver intelligence, recommended decisions and actions as output.

Our cloud-native, data-agnostic platform is currently:

connected to 110+ data sources and counting

processing 114M alerts and events per day

Where do we store collected processed data?

 as Cloud agnostic Highly Available Object Storage for our Platform that can be Scaled Up or Scaled Out on demand to store the Processed logs. MinIO Object storage acts a Data lake layer of the Human Managed Platform. Life Cycle management policies are applied in MinIO such that only the fast access to hot tier data lives in MinIO and the rest are further extended to the Cloud’s native Object storage service like S3, Cloud Storage etc.


We created custom Cloud agnostic Helm chart and templates to deploy the appropriates that are needed first for MinIO. MinIO Tenants are then deployed on respective Kubernetes Clusters in various Clouds and are replicated amongst each other. Hence even if a whole Cloud solution fails, HM Platform’s Object storage solution would still function as they are replicated across other Tenants in other Clouds. This supports seamless continuation of service consumption. Each MinIO Tenant is separately monitored by Monitoring team using Prometheus metrics and Grafana dashboards. Human Managed uses Istio Ingress Gateway to effectively utilize the Cloud Load Balancers to support ingress to the Object storage.

Provisioning Object storage solution in Human Managed is automated and would take approximately 30 minutes to come to live.

 All product names, logos and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos and brands does not imply endorsement.

To get the latest news from Human Managed, follow us on 

Want to discuss how we can help solve your cyber, digital, or risk operations through data?

Please contact us at hello@humanmanaged.com.

How we scale cloud agnostic object storage with MinIO

Scaling cloud agnostic object storage

To get anything out of big data, you first need to ingest data from multiple sources. Big data collection is notoriously difficult. But does it have to be?

Imagine having one smartphone that can only handle one mobile application at a time. It would be manageable if you only use 1 or 2 applications, but what if you have 50 mobile applications: One smartphone for Google News, another one for Facebook, another one for Instagram, and so on. Sounds awful, right?!

That is one of the main problems we’ve encountered in trying to develop the data ingestion for our Collect Microservice at Human Managed. Our platform generates intelligence and action from customers’ data, which means we work with many data sources as our input for analysis(currently standing at 30+ data sources and counting!)

Initially, we developed numerous bash and python scripts running via cron job or AWS lambda just to automatically fetch the data from multiple APIs and data sources. However, this became difficult for us to manage once we tried to scale up and replicate for each data source and customer that we have.

We had to find a way to address this with something that we can 

, an open-source data integration platform that syncs data from applications, APIs, and databases and transfer them to the destination that you want such as database and object storage. Remember the analogy I gave earlier? Airbyte is 

the type of smartphone that can handle multiple mobile applications, which makes data integration simple, secure, and extensible.

We utilized existing Airbyte connectors that fits into our use cases and also developed our own connectors when necessary. Since Airbyte is an open-source platform, the connectors available are developed by the community where you can also 

(We plan to contribute our custom connectors and improvements in existing ones so that we could also give back to the community, so stay tuned!)

Once we converted our automation scripts into Airbyte connectors, we were able to easily implement and organize the batch collection that happens within our Collect microservice. In replicating the same connection but for another customer, we just set the configuration in the existing connector, set the destination, and that’s it! We were able to integrate the another data source into our platform in just minutes.

Referring back to the analogy of having multiple smartphone, with Airbyte, we have now a 

one single powerful smartphone that can handle and receive data from multiple mobile applications

). This solved the problem of having to maintain, and organize the automation of ingesting data which can also be customized, scaled, and managed however suits our needs.

There will always be a setback in using any other tool, even if it’s Airbyte. The first thing that comes into mind is the learning curve required to develop and maintain custom airbyte connectors. If you are not familiar with the concepts, you’ll have to figure out a lot of these essential concepts to grasp in order to properly understand and implement an Airbyte connector. But once figured out, most of the development will be easier than before because of the 

As for its limitation, the current lack of support for data backup of raw data in-between the Extract (E) and Load (L) is not yet possible. This conflicts with our architecture of storing the raw logs without the metadata added by Airbyte.

Even though I’ve also mentioned the cons of implementing Airbyte. The benefits far outweigh the disadvantages and gives you the following capabilities:

Open source and free for all to contribute

Customizable, Modular, Scalable, Secured, and Compliant

Set Connection Syncs and Incremental Stream State

The capacity and features of Airbyte to unify your data integration pipelines under one fully managed platform is fit to develop our Collect microservice since it enables us to easily integrate data sources, develop custom connectors when necessary, and manage these connections without worrying too much on security and compliance.

With a fast growing community and contributions, more and more connectors will be made and improved in the future which will continue to innovate and push to evolve the data ingestion capabilities of Airbyte.

You could learn more about Airbyte and get involved in the open-source platform by visiting the 

Note: This blog was originally published in August 2022 and has been updated for accuracy. 

Categories

DataOps

3 data decisions to make in 2024 for businesses to become AI-native

Scaling cloud agnostic object storage

Orchestrating data collection with Airbyte