I wrote about how we integrate Okta and Auth0

 using Auth0’s Social Connector (it runs OAuth2 behind the scenes). Today I will get into a bit more detail about how it’s done behind the scenes.

Our applications are registered directly with Auth0 as a “Regular Web Application”. This first layer of registration allows us to control 

. In other words, only applications authorized by our platform can communicate with our backend.

The second level authentication and authorization that we do is at the 

. This is where the Okta + Auth0 integration comes into play (there’s also 

 but we’ll get into that as a separate topic). The user is redirected to Okta to complete sign-in (enter your username, MFA, etc.) and then returned to Auth0 to continue with the login flow.

 script for that Social connector we configured to talk to Okta. In this script, the Auth0 runtime populates a number of parameters with useful return information from Okta. You get 3 things:

a ctx or context object which gives you access to the id token

and a callback, cb, for you to return a profile object to Auth0 that we use to construct the JWT

The access token is specific to the user who just completed the logon in Okta. We use this access token as part of the Authorization Bearer scheme to call the Okta OIDC UserInfo API (

https://developer.okta.com/docs/reference/api/oidc/#userinfo

). This allows us to fetch additional attributes about the user.

It is highly recommended to use an Okta custom authorization server for this purpose (

https://developer.okta.com/docs/guides/customize-authz-server/main/

). Why? Because the custom authorization server allows more control over the JWT that Okta mints (

). It’s also possible to move some of the custom claims being generated in Auth0’s 

 script to an Okta custom authorization server (it’s really up to you, but we prefer Auth0’s approach because Okta’s expression language pales in comparison to raw Javascript :) )

The return JSON from the Okta API call makes it very easy to parse properties and do additional processing. For example, here is part of our 

 object is what we return to Auth0 as part of our 

 script. This object is used by Auth0 to create the user profile and issue a JWT. This can be done like so at the end of the script:

To add a new claim to the JWT that Auth0 mints, all you have to do is to modify the 

 object to your heart’s content. For example:

 then? That is simply the user’s OIDC token and we can get that from one of the function parameter that Auth0 passes to us i.e. the 

Now that you have both the access token and id token, you can manipulate the final JWT that gets issued! Oh, one more thing — how do you remove a claim? Easy as pie:

When you view your the JIT created user in Auth0, you will see something similar to the image posted below where each field is generated from the 

 object that you manipulate. These fields can be used for 2 purposes — for auditing (you have a snapshot of the user the last time they logged on, so it’ really up to you what fields you want to populate for auditing purposes), and finally for JWT generation.

To get these fields added to the JWT token that Auth0 mints, you use another wonderful Auth0 feature called the login action flow (

https://auth0.com/docs/customize/actions/flows-and-triggers/login-flow

Here’s a sample of how you can do that:

If you need more information, feel free to reach out to us at 

Minting JSON Web Tokens (JWTs) for user level authentication and authorization 

Minting JSON Web Tokens (JWTs)

To get anything out of big data, you first need to ingest data from multiple sources. Big data collection is notoriously difficult. But does it have to be?

Imagine having one smartphone that can only handle one mobile application at a time. It would be manageable if you only use 1 or 2 applications, but what if you have 50 mobile applications: One smartphone for Google News, another one for Facebook, another one for Instagram, and so on. Sounds awful, right?!

That is one of the main problems we’ve encountered in trying to develop the data ingestion for our Collect Microservice at Human Managed. Our platform generates intelligence and action from customers’ data, which means we work with many data sources as our input for analysis(currently standing at 30+ data sources and counting!)

Initially, we developed numerous bash and python scripts running via cron job or AWS lambda just to automatically fetch the data from multiple APIs and data sources. However, this became difficult for us to manage once we tried to scale up and replicate for each data source and customer that we have.

We had to find a way to address this with something that we can 

, an open-source data integration platform that syncs data from applications, APIs, and databases and transfer them to the destination that you want such as database and object storage. Remember the analogy I gave earlier? Airbyte is 

the type of smartphone that can handle multiple mobile applications, which makes data integration simple, secure, and extensible.

We utilized existing Airbyte connectors that fits into our use cases and also developed our own connectors when necessary. Since Airbyte is an open-source platform, the connectors available are developed by the community where you can also 

(We plan to contribute our custom connectors and improvements in existing ones so that we could also give back to the community, so stay tuned!)

Once we converted our automation scripts into Airbyte connectors, we were able to easily implement and organize the batch collection that happens within our Collect microservice. In replicating the same connection but for another customer, we just set the configuration in the existing connector, set the destination, and that’s it! We were able to integrate the another data source into our platform in just minutes.

Referring back to the analogy of having multiple smartphone, with Airbyte, we have now a 

one single powerful smartphone that can handle and receive data from multiple mobile applications

). This solved the problem of having to maintain, and organize the automation of ingesting data which can also be customized, scaled, and managed however suits our needs.

There will always be a setback in using any other tool, even if it’s Airbyte. The first thing that comes into mind is the learning curve required to develop and maintain custom airbyte connectors. If you are not familiar with the concepts, you’ll have to figure out a lot of these essential concepts to grasp in order to properly understand and implement an Airbyte connector. But once figured out, most of the development will be easier than before because of the 

As for its limitation, the current lack of support for data backup of raw data in-between the Extract (E) and Load (L) is not yet possible. This conflicts with our architecture of storing the raw logs without the metadata added by Airbyte.

Even though I’ve also mentioned the cons of implementing Airbyte. The benefits far outweigh the disadvantages and gives you the following capabilities:

Open source and free for all to contribute

Customizable, Modular, Scalable, Secured, and Compliant

Set Connection Syncs and Incremental Stream State

The capacity and features of Airbyte to unify your data integration pipelines under one fully managed platform is fit to develop our Collect microservice since it enables us to easily integrate data sources, develop custom connectors when necessary, and manage these connections without worrying too much on security and compliance.

With a fast growing community and contributions, more and more connectors will be made and improved in the future which will continue to innovate and push to evolve the data ingestion capabilities of Airbyte.

You could learn more about Airbyte and get involved in the open-source platform by visiting the 

Note: This blog was originally published in August 2022 and has been updated for accuracy. 

Orchestrating data collection with Airbyte

I’ve joined Human Managed — a data company that empowers decision — in late December 2021 as a front end developer. In a short amount of time, I went through a steep learning curve, working with technologies and processes that I have never worked with before.

One of those is Git, a distributed, open-source version control (VCS) that enables you to:

revert to earlier code version when needed

Back in my college days, I used to code without any version control tool like Git. What I did was copy and paste source code folders and rename it with a version number. My reasoning back then is, since I do code alone, then there’s no point in using a version control system. Lo and behold, this idea backfired to me in my thesis project. There was a time when I needed to update a block of code for a fix, and it took me hours to do so.

I used to agree with the guy at the bottom😂

I knew that I needed to learn git when I joined HM. I won’t be coding alone anymore since I’ll be collaborating with my colleagues. When I was trying to learn it during my first few weeks, I initially thought it was really complicated since there are a lot of commands that you need to understand. But when I started to contribute in our internal codebase, I (also with the help of my colleagues) managed to learn it along the way. I have lots of realizations on how this tool can benefit myself before and could’ve made me more productive back then, how I wish I learned about it sooner!

 to store our source code, keep track of our changes for every apps per microservices and run a CI/CD pipeline to ensure that stages are automated and done correctly.

Through git, we have a commit ID which keeps track of our changes. We can always go back to whichever version we have, in any cases that we wish to.

For example, imagine you are working on a feature within a webapp and made around six commits already. You tested the application, then found that there is a bug in your app. Upon checking, you realized that the bug was caused by the initial commit you had in the source code. With git, you can revert a change and undo it, so you can go back to the original state of the source code.

In our applications, we use GitLab as our git-based web platform environment for version control and source code management. Every day, there are changes to be made in the codebase. A developer can work concurrently with the other one through git since we utilize its branching functionality and it being a decentralized version control system. We create a branch for a specific update in our webapp, and merge it to the main ‘production’ source code once it’s finished. This way, we are not dependent on each other if we want to make changes to our applications. We can do things seamlessly without the worry of having any conflicts with what we are doing.

 is used as our DevOps platform. Since it is git-based, we use GitLab as our remote repository to store our source code. An event is triggered via git push/commit to the remote which will execute a CI/CD pipeline through GitLab. We use it to run automated build, test, and deployment. Through gitlab, we can deploy apps immediately without the need to wait for someone to finish something, since steps are already automated.

Every action we do within our internal repositories are tracked via webhook which is configured in a CI channel wherein relevant individuals are added. We use this as our collaboration method to notify and make everyone aware on what’s happening to our apps in microservices.

Using GitLab enables us to automate majority of version control and source code management. It’s beneficial for software engineering since manual steps are already taken out of the picture through its automated devops functionalities.

Imagine doing manual build, test, and deployment for your applications. You’ll go back to the days where it’s not as efficient as compared to the automated stages that a devops tool like GitLab can offer. For sure it must have brought a lot of headaches to every developer before! Lucky enough, we live in an advanced tech-era now where everything can be automated already.

. In layman’s term, this is the set of instructions that you need to configure to automate all the actions/stages that you’ll be automating in your application.

Someone just needs to understand how to write this CI file, and all the end-to-end stages for CI/CD will be automated already. That’s how handy and powerful this file is!

This is just the beginning of our code automation story. Just like git, this narrative can be further 

 to areas like application security, automated deployments, and gitops. But that would be for another blog! 😊

Continuous integration at Human Managed

Automating and securing code management with Git

’s event-driven API story. In this part, we cover the high-level decisions that we made to build this part of our platform. In the second part, we explore how we translated these high-level decisions into run-time components.

All product names, logos, and brands are property of their respective owners

. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

We had to build an event-driven modular platform. What does that even mean? There are two levels of conversation when you define something like this: the 50,000-feet-in-the-cloud view, and the sea-level view.

I’ll break it down into its constituents for you:

 — what is an event? I believe there are only 3 things you need to remember: there is a time dimension involved, it generates data, it is of interest to you.


 — think LEGO, LEGO, LEGO. The modular architecture needs a few things to be come together: loose coupling, reusability, and orchestration.


 — there are many definitions of this, but I’ll settle on this one: it facilitates a broad set of interactions across a large number of people, processes, and technologies.

The challenge was to translate these lofty goals into engineering work. But, if you really think about it, all 3 come together neatly. The modular component is the logical building block of a platform. In a platform, I’d rather have the ability to build a house than to have a single instance of a house. How do you orchestrate all the modular components that interact in a platform? You use an event-driven approach!

Automate — how we let machines do the work

Orchestrate — how we curate and control all the moving parts

Present — how we let people know the important stuff

An API layer was an obvious choice we had to build to address the 4 stacks, and the 50,000 feet vision. It has one main job to do — it connects, automates, orchestrates, and presents event-driven intelligence to our customers and our community.

If you have a clean-slate, and you can pick any technology to build with, where would you start? That was the challenge we faced 1 year ago when deciding on our API strategy.

Sure, we had some guiding principles and a few rules to adhere by:

It has to run in a public cloud provider infrastructure.

It has to adhere to a microservices architecture.

Since we knew it had to be stateless and it had to use a microservices architecture, the obvious choice would be to pick from one of the “serverless functions” (I use a quote because there really are servers in the background, but it’s taken care of by the cloud provider) available in the cloud.

 functions, but realized quickly it had a performance hit. AWS would automatically shutdown the instance running your lambda function, and spin it up on-demand. Great for cost management, but not so great for performance. AWS does not even guarantee the compute instance that was used to run your lambda function previously would be the same one it would spin-up in again (although, if you really think about it, it only makes sense for AWS to re-use the same underlying compute instance because it’s more efficient for them).

Based on some initial benchmark tests, we finally settled for 

 (in hind sight, the comparison wasn’t apples to apples because the equivalent in AWS is 

There were also a number of API web frameworks we could use, but the choice here was very quickly obvious to us. First, we have Python skillsets, second, we wanted something fast, third, we wanted to support the 

FastAPI is really fast (yes, that means performant!)

This was a mature project with regular updates being made. The learning curve was minimal as the author had provided a very good set of documentation: he had documented steps to solve common “how do I do this?” questions! That was a great boon to us in bridging the learning gap.

 would be the preferred choice for us for many reasons. It’s industry standard, it’s really secure, it allows users to grant / deny permissions before resources are accessed, and it’s pretty extensible (which was an important point for us — more on this later).

One of the key strengths of OAuth 2.0 is the ability to carry your authentication state around in a 

. The JWT is self-containing, but we decided to extend JWTs by adopting the 

 design pattern. The switch to an opaque token was only made after the initial implementation where we had sensitive information embedded in the JWT. The drive to an opaque design was triggered by a need to enrich the JWT without bloating the token with claims unnecessarily. The need to keep sensitive data from prying eyes was a secondary decision (but a right one!).

There was a key distinction between these two functions on the platform. Identity management would control user identities, while authorization management would control machine identities. Both can be combined, or identity management can be removed from the equation:

If a machine needs to access our APIs, only authorization management would be involved.

If a user needs to access our APIs, access would always be via an application. Identity management would take care of the user, while authorization management would take care of the application used by the user.

 for Authorization Management — playing to the strength of their individual portfolios (note: Okta eventually 

We wanted a concept of pluggable data access providers. To this end, we encapsulated the data access component as re-usable Python classes in our App Engine run-time. Each class has a standard interface for talking to different database providers that are defined by the type of database technology it encapsulates i.e. RDBMS, document DB, etc.

We currently have 2 underlying database technologies in use — 

. There will be a lot more in the future, and integrating them into our modular design will be easy.

Caching was introduced late into the game. This is implemented using a custom developed Python class that wraps around a 

 instance. The caching strategy needed to be adjusted a number of times, but it has brought about a significant improvement in API response times. The first version of our caching strategy saw a 50% improvement in response times. The final version saw response times drop from 2–3s to < 1000ms (some APIs can now respond within sub-100ms).

It turns out the biggest hit to API performance was the calls to external systems — due to network latency, and compute requirements of the external system. For example: calls to an RDBMS to retrieve data, calls to a decryption service, calls to a secret vault manager, etc.

When caching a result in REDIS, we need a unique key to define the cache entry. To achieve this, we needed to use a fast hashing algorithm that has good dispersion, randomness, and minimal collisions in the quality of hashes generated. Yes, we use calculated hashes for all our keys rather than storing the keys in their underlying native format (don’t worry, this is really easy to understand when you read Part 2 of this series, I promise!).

This turned out to be the easiest decision. We needed an API Gateway! The gateway acts as a “firewall” for our APIs, among other things. It controls access, versioning, rate-limiting, and monitoring of our API stack. Since our APIs are hosted in Google App Engine, it was important for us to ensure that only the API Gateway is authorized to communicate with our App Engine instance. We achieved this by simply implementing IP whitelisting on App Engine’s firewall, and reserving a static public IP for our API Gateway’s egress traffic.

 (by the way, they have a really adorable mascot) as our API Gateway due to it’s excellent performance, ease of use, and extensibility. Plus, it runs out of Kubernetes, which we decided would be our run-time orchestration engine of choice for many of our platform components.

The choice of using Kubernetes will eventually drive the conversion of our App Engine instance into a Kubernetes workload. But that’s a story for another time.

Our opaque token design is simple. The opaque key is generated in an Okta workflow, using a combination of a random UUIDv4 number and an Okta group-id. This opaque key is then inserted into a custom claim that is part of a token minted by Okta. The final JWT that a user possess for his / her logon session is actually minted by Auth0. More on this later!

We establish a store for customer metadata in MongoDB. A document DB was the logical choice because of the malleability of the underlying “schema” of the document.

The metadata is read at run-time by the APIs.

With those pieces defined, it was time to start work. Stay tuned for Part 2, where we cover how we engineered these components.

Note: This blog was originally published in July 2022 and has been updated for accuracy. 

Categories

Engineering

Minting JSON Web Tokens (JWTs)

Orchestrating data collection with Airbyte

Automating and securing code management with Git

Building event-driven APIs for a modular microservices platform (part 1)