- Performing analytics or business intelligence on structured API data
- Creating structured data exports to import into external tools
- Decoupling real-time webhook handling from downstream ETL
- Scaling data pipelines efficiently
Setup
1
Create Credentials
Create a client ID and client secret in the Sohar dashboard.
2
Configure Webhooks
Configure a webhook endpoint to listen for events. Subscribe your endpoint to the
verification.updated event.3
Configure Amazon S3
Configure an Amazon S3 bucket with programmatic write permissions via IAM user or role.
Workflow
The diagram below outlines the workflow we will build to transfer Verification data from Sohar to your data warehouse.
1
Receive a webhook
2
Call the Get Verification API to fetch JSON data
3
Store that data in Amazon S3
4
Use your ETL/ELT tool to load the data into your warehouse
Handle Webhook Events
Here’s a basic Express route to receive webhook events from Sohar and save the Verification response to Amazon S3:Load to the Data Warehouse
Now that data is in S3, you can:- Use AWS Glue to crawl and ETL the data into Redshift or Athena
- Use dbt + dbt-s3 to ELT from S3 into Snowflake or BigQuery
- Use Airbyte, Fivetran, or custom scripts for downstream ingestion