Secure Data Connections

Swift supports multiple secure data connection methods to integrate with your existing infrastructure. Our platform is designed to handle both real-time streaming and batch processing workflows.

Apache Kafka

Swift integrates seamlessly with Apache Kafka for real-time event streaming. Our Kafka connectors support:

Secure authentication using SASL/SCRAM
TLS encryption for data in transit
Configurable consumer groups
Automatic offset management
Dead letter queues for error handling
Schema validation and evolution

Connecting Apache Kafka with Swift’s API

To connect Apache Kafka with Swift’s API at api.joinswift.app, follow these steps:

Set Up Kafka Connect:
- Ensure you have Kafka Connect installed and running in your environment.
- Add the necessary connectors for HTTP Sink to your Kafka Connect setup.

Configure the HTTP Sink Connector:

Create a configuration file for the HTTP Sink Connector with the following settings:

{
  "name": "http-sink-connector",
  "config": {
    "connector.class": "io.confluent.connect.http.HttpSinkConnector",
    "tasks.max": "1",
    "topics": "your-kafka-topic",
    "http.api.url": "https://api.joinswift.app/v1/data/ingest",
    "http.headers.content.type": "application/json",
    "http.headers.authorization": "Bearer <your-api-key>",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
  }
}

Replace your-kafka-topic with the name of your Kafka topic.
Replace <your-api-key> with your actual API key for authentication.

Deploy the Connector:
- Deploy the HTTP Sink Connector to your Kafka Connect cluster using the REST API:
  curl -X POST -H "Content-Type: application/json" --data @http-sink-connector.json http://<kafka-connect-host>:<kafka-connect-port>/connectors
- Replace <kafka-connect-host> and <kafka-connect-port> with the appropriate values for your Kafka Connect setup.
Verify the Connection:
- Monitor the Kafka Connect logs to ensure the connector is running without errors.
- Check the Swift API logs to verify that data is being ingested correctly.

Amazon Kinesis

For AWS-native architectures, Swift provides full Amazon Kinesis support including:

Kinesis Data Streams for real-time data ingestion
Kinesis Data Firehose for data transformation and loading
Enhanced fan-out consumers for high-throughput processing
Server-side encryption with AWS KMS
IAM role-based access control
Automatic scaling and resharding

Setting Up Amazon Kinesis with Swift’s API

To connect Amazon Kinesis with Swift’s API at api.joinswift.app, follow these steps:

Set Up Kinesis Data Stream:
- Ensure you have an Amazon Kinesis Data Stream created in your AWS account.
- Note the name of your Kinesis Data Stream.
Configure Kinesis Data Firehose:
- Create a Kinesis Data Firehose delivery stream with the following settings:
  - Source: Kinesis Data Stream
  - Destination: HTTP Endpoint
  - HTTP Endpoint URL: https://api.joinswift.app/v1/data/ingest
  - Content Type: application/json
  - HTTP Endpoint Access Key: <your-api-key>

Set Up IAM Role:

Create an IAM role with the necessary permissions to allow Kinesis Data Firehose to write to the HTTP endpoint.

Attach the following policy to the IAM role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:PutRecord",
        "firehose:PutRecordBatch"
      ],
      "Resource": "arn:aws:firehose:region:account-id:deliverystream/your-delivery-stream-name"
    }
  ]
}

Replace region, account-id, and your-delivery-stream-name with the appropriate values.

Deploy the Delivery Stream:
- Deploy the Kinesis Data Firehose delivery stream and start sending data to your Kinesis Data Stream.
- Ensure that the data is being forwarded to the Swift API endpoint.
Verify the Connection:
- Monitor the Kinesis Data Firehose logs to ensure the delivery stream is running without errors.
- Check the Swift API logs to verify that data is being ingested correctly.

Batch Processing Connections

AWS Glue

Swift integrates with AWS Glue for ETL workloads:

Automated data catalog synchronization
Support for both scheduled and event-driven jobs
Custom transformation scripts
Data quality validation rules
Incremental processing capabilities
Job monitoring and alerting

Setting Up Batch Processing with AWS Glue

To set up batch processing for AWS Glue with Swift’s API at api.joinswift.app, follow these steps:

Create an AWS Glue Job:
- Navigate to the AWS Glue console and create a new Glue job.
- Choose a name for your job and select an IAM role with the necessary permissions to access your data sources and the Swift API.
Configure the Data Source:
- Define your data source in the AWS Glue Data Catalog. This could be an S3 bucket, a database, or another supported data source.
- Ensure that the data source schema is properly defined in the Data Catalog.

Write the ETL Script:

Write a custom ETL script in Python or Scala to transform and process your data. Below is an example Python script that reads data from an S3 bucket, processes it, and sends it to the Swift API:

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
import requests
import json

args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# Read data from S3
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "your-database", table_name = "your-table", transformation_ctx = "datasource0")

# Process data (example transformation)
applymapping1 = ApplyMapping.apply(frame = datasource0, mappings = [("column1", "string", "column1", "string"), ("column2", "int", "column2", "int")], transformation_ctx = "applymapping1")

# Convert to DataFrame
df = applymapping1.toDF()

# Send data to Swift API
url = "https://api.joinswift.app/v1/data/ingest"
headers = {
    "Authorization": "Bearer <your-api-key>",
    "Content-Type": "application/json"
}

for row in df.collect():
    data = {
        "column1": row["column1"],
        "column2": row["column2"]
    }
    response = requests.post(url, headers=headers, data=json.dumps(data))
    if response.status_code != 200:
        print(f"Failed to send data: {response.text}")

job.commit()

Configure Job Triggers:
- Set up triggers for your Glue job. You can schedule the job to run at specific intervals or configure it to run based on events (e.g., new data arrival in S3).
Monitor and Debug:
- Use the AWS Glue console to monitor the job execution. Check the logs for any errors and debug as necessary.
- Ensure that the data is being successfully sent to the Swift API and processed correctly.

Apache Spark

Our platform works with Apache Spark for large-scale data processing:

Structured Streaming support
Delta Lake integration
Custom UDFs for data transformation
Resource management and optimization
Job scheduling and monitoring
Fault tolerance and recovery

Batch Processing with Apache Spark or Databricks

To set up batch processing for Apache Spark or Databricks with the Swift API, follow these steps:

Set Up Your Environment:
- Ensure you have Apache Spark or Databricks environment ready.
- Install necessary libraries such as requests for API calls and pyspark for Spark operations.

Initialize Spark Session:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("SwiftBatchProcessing") \
    .getOrCreate()

Read Data:
- Load your data into a Spark DataFrame. This can be from various sources such as S3, HDFS, or local files.
```
df = spark.read.csv("s3://your-bucket/your-data.csv", header=True, inferSchema=True)
```

Process Data:

Perform necessary transformations on your DataFrame.

processed_df = df.select("column1", "column2") \
    .withColumnRenamed("column1", "new_column1") \
    .filter(df["column2"] > 100)

Send Data to Swift API:

Convert the DataFrame to Pandas for easier row-wise operations.

import requests
import json

pandas_df = processed_df.toPandas()

url = "https://api.joinswift.app/v1/data/ingest"
headers = {
    "Authorization": "Bearer <your-api-key>",
    "Content-Type": "application/json"
}

for index, row in pandas_df.iterrows():
    data = {
        "new_column1": row["new_column1"],
        "column2": row["column2"]
    }
    response = requests.post(url, headers=headers, data=json.dumps(data))
    if response.status_code != 200:
        print(f"Failed to send data: {response.text}")

Schedule the Job:
- Use Databricks job scheduling or Apache Spark’s job scheduling capabilities to run the batch processing at regular intervals.
- For Databricks, you can use the Databricks Jobs UI to create and schedule jobs.
- For Apache Spark, you can use tools like Apache Airflow or cron jobs to schedule your Spark job.
Monitor and Debug:
- Monitor the job execution through Spark UI or Databricks workspace.
- Check logs for any errors and debug as necessary.
- Ensure that the data is being successfully sent to the Swift API and processed correctly.

By following these steps, you can set up efficient batch processing for your data using Apache Spark or Databricks and integrate it seamlessly with the Swift API.

API Categories

Swift organizes its data connections into distinct API categories:

Transaction APIs

Real-time transaction processing
Batch transaction uploads
Transaction enrichment
Reconciliation endpoints
Settlement processing
Fee calculation

Customer APIs

Account management
Profile updates
KYC/AML data processing
Customer segmentation
Interaction history
Preference management

Analytics APIs

Risk scoring
Fraud detection
Deposit trend analysis
Customer behavior insights
Performance metrics
Custom report generation

Administrative APIs

User management
Role-based access control
Audit logging
System configuration
Alert management
Integration settings

Security Standards

All data connections implement industry-leading security measures:

Authentication

OAuth 2.0 / OpenID Connect
Mutual TLS (mTLS)
API key management
JWT token validation
SAML 2.0 support
Multi-factor authentication

Encryption

TLS 1.3 for data in transit
AES-256 for data at rest
Hardware security module support
Key rotation policies
Certificate management
End-to-end encryption options

Monitoring and Observability

Comprehensive monitoring is available for all connection types. Please contact [email protected] for more information.

About Swift

Getting Started

Guides

Apache Kafka

Connecting Apache Kafka with Swift’s API

Amazon Kinesis

Setting Up Amazon Kinesis with Swift’s API

Batch Processing Connections

AWS Glue

Setting Up Batch Processing with AWS Glue

Apache Spark

Batch Processing with Apache Spark or Databricks

API Categories

Transaction APIs

Customer APIs

Analytics APIs

Administrative APIs

Security Standards

Authentication

Encryption

Monitoring and Observability

About Swift

Getting Started

Guides

​Apache Kafka

​Connecting Apache Kafka with Swift’s API

​Amazon Kinesis

​Setting Up Amazon Kinesis with Swift’s API

​Batch Processing Connections

​AWS Glue

​Setting Up Batch Processing with AWS Glue

​Apache Spark

​Batch Processing with Apache Spark or Databricks

​API Categories

​Transaction APIs

​Customer APIs

​Analytics APIs

​Administrative APIs

​Security Standards

​Authentication

​Encryption

​Monitoring and Observability

Apache Kafka

Connecting Apache Kafka with Swift’s API

Amazon Kinesis

Setting Up Amazon Kinesis with Swift’s API

Batch Processing Connections

AWS Glue

Setting Up Batch Processing with AWS Glue

Apache Spark

Batch Processing with Apache Spark or Databricks

API Categories

Transaction APIs

Customer APIs

Analytics APIs

Administrative APIs

Security Standards

Authentication

Encryption

Monitoring and Observability