Implementing Custom Connectors

Fusebit adds ConnectorConnector - A connector is the package from Fusebit that manages the relationship between one or more integrations and a specific service. One of the most common types of connector is an OAuth connector, which takes care of the OAuth negotiation between your customers and the service you're integrating, so that you don't have to!s (reference) at a regular clip, but occasionally there will be a Connector that isn't yet part of the officially supported package. This might be an internal Connector, pointing at your API endpoints, or it might be a Connector needed to support a specific SaaS peer that your customer needs.

After checking in with us to make sure it's not already on our roadmap or in private release, you can follow this document to create your own Connector.

👍
Estimated level of effort: 1-3 days

Scenario: Integrate with NewCo

Congratulations on a new sale! A customer has eagerly signed up for your product - but before the ink is even dry, they're demanding that you integrate with their other favorite software company: NewCo. NewCo is a fairly new SaaS that you've never heard of before, but this deal is worth a lot and you eagerly promise that the integration will be complete right away!

Fortunately, you have Fusebit in your corner! Let's build a Connector for NewCo that will keep your customer happy, and your developers happier.

Setting Up the Environment

Before we get started, let's make sure that we have the right tools in place.

Install the Fusebit CLI via npm install -g @fusebit/cli
Sign in to your Fusebit account, or create a new account, by running fuse init
Your favorite editor - we use Visual Studio Code with various Javascript extensions installed, but any editor will do (Benn uses vim, the poor fellow)
(Optional) Add the Fusebit internal npm registry to your npm search path via fuse npm login

Collecting Information

Like any good project, we'll start this one off by collecting a few important pieces of information to help speed the process along later:

Naming the Connector - sometimes you can get away with newco, but some SaaS (looking at you, GitHub) require special designations.
Identify if there is a standard package published on npm that you would expect to use to invoke NewCo's API. If there isn't one, the next best option is to use our built in ApiClient to provide a friendly HTTP method-based interface, please reach out to us via the Intercom messenger on this page for help.
Find the OAuth documentation for NewCo. Hopefully they published an easy-to-find page that includes the necessary URLs, but sometimes that page is distressingly difficult to find. If NewCo doesn't support OAuth, let us know and we'll help you out with the details.

Starting the Project

What Are We Building?

Let's talk about what we're going to build, and what building a Connector actually means for Fusebit. Fusebit separates out the relationship with a service like NewCo into two different pieces: a Connector, and a Provider.

A Connector handles OAuth, credential revocation, security issues, and other related secret management tasks. It runs in a separate context with different permissions than the IntegrationIntegration - An integration is the place the you write your code to get things done! Written in NodeJS, an integration runs in Fusebit's secure and scalable environment to translate between the needs of your backend application and the remote service or services you're connecting to. where you and your coworkers will write the meat of the functionality.

The Provider, on the other hand, acts as the local-to-the-Integration half of the Connector pair. It takes the security credentials for specific requested IdentityIdentity - An identity is a unique relationship one of your customers has with a service. An identity can be used by multiple integrations to act on that service on behalf of your customer. objects and converts them into a convenient, already configured SDK object. When you use code in the Integration that looks like this...

const sdk = await integration.service.getSdk(ctx, connectorName, ctx.params.installId);

... you're actually leveraging the code in the Provider to request the authorization token - usually a JWT - from the remote Connector, and then provision that into a SDK.

Steps To Build

There's a handful of steps that we're going to follow here to build out the Connector and the Provider:

Create the Connector project
Create a Provider project
Adding the SDK
Publishing the Connector and Provider to your private Fusebit npm Registry
Deploying a new Integration and Connector!

We'll go through each of those steps in detail.

Let's Build!

Creating the Project Directories

Fusebit is built around NodeJS, so let's start by creating two new projects for the Provider and the Connector. We'll call them newco-connector and newco-provider for now.

📘
Using the Private Fusebit npm Registry
If you want to publish your packages to the public npmjs.org registry, great! Many other people will be sure to benefit, and it's a great way to promote your support for NewCo to other NewCo customers!
But sometimes, you want to start with keeping a package private until it meets your bar for releasing into the wild. That's okay, we totally understand! We do the same thing ourselves, which is why we have a private npm Registry baked into your Fusebit Account.
You'll need to configure it to use a scope that's unique to you, @mycompany is a good choice. You can set that scope by using the Fusebit CLI we installed earlier:
fuse npm registry scope set @mycompany
# Then log in again to make sure your ~/.npmrc file is up to date
fuse npm login
Once done, make sure to edit the package.json for each of the two projects and specify the name correctly:
{ "name": "@mycompany/newco-connector" } and
{ "name": "@mycompany/newco-provider" }.
This will allow for each of the two projects to use npm publish to be published into the private Fusebit npm Registry, and be available to any Integrations that are being deployed in your Account.

Each of the projects can be initialized using the standard npm init command:

$ mkdir newco-connector newco-provider
$ cd newco-connector
$ npm init
...
package name: (newco-connector) newco-connector
...

$ cd ../newco-provider
$ npm init
...
package name: (newco-provider) newco-provider
...

Initializing TypeScript and npm

Here's the starter project files we use, including Typescript, and a basic starter index.ts file:

import { Connector } from '@fusebit-int/framework';
import { OAuthConnector } from '@fusebit-int/oauth-connector';

const TOKEN_URL = 'https://newco.com/oauth/access_token';
const AUTHORIZATION_URL = 'https://newco.com/oauth';
const REVOCATION_URL = 'https://newco.com/oauth/revoke';
const SERVICE_NAME = 'NewCo';

class ServiceConnector extends OAuthConnector {
  protected addUrlConfigurationAdjustment(): Connector.Types.Handler {
    return this.adjustUrlConfiguration(TOKEN_URL, AUTHORIZATION_URL, SERVICE_NAME.toLowerCase());
  }
}

const connector = new ServiceConnector();

export default connector;
export { ServiceConnector };

{
  "name": "newco-connector",
  "version": "1.0.0",
  "main": "libc/index.js",
  "files": [
    "libc/**/*.js",
    "libc/**/*.d.ts",
    "libc/**/*.json"
  ],
  "scripts": {
    "build": "tsc -b --pretty",
    "lint:check": "eslint . --ext .ts --color",
    "lint:fix": "eslint . --ext .ts --color --fix"
  },
  "dependencies": {
    "@fusebit-int/oauth-connector": ">=7.24.1",
    "superagent": "6.1.0"
  },
  "devDependencies": {
    "@fusebit-int/framework": ">=7.24.1",
    "@types/superagent": "^4.1.12",
    "@typescript-eslint/eslint-plugin": "^4.31.0",
    "@typescript-eslint/parser": "^4.31.0",
    "eslint": "^7.32.0",
    "eslint-config-prettier": "^8.3.0",
    "eslint-plugin-import": "^2.24.2",
    "eslint-plugin-security": "^1.4.0",
    "typescript": "^4.4.3"
  },
  "peerDependencies": {
    "@fusebit-int/framework": "*"
  }
}

{
  "compilerOptions": {
    "target": "es2017",
    "module": "CommonJS",
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true,
    "strict": true,
    "composite": true,
    "esModuleInterop": true,
    "moduleResolution": "node",
    "resolveJsonModule": true,
    "jsx": "react",
    "forceConsistentCasingInFileNames": true,
    "rootDir": "src",
    "outDir": "libc"
  },
  "include": ["src", "src/**/*.json"],
  "exclude": ["node_modules"]
}

And here is the Providers starting files:

import { Internal } from '@fusebit-int/framework';
import { Client } from '@newco/sdk';

type FusebitNewCoClient = Client & { fusebit?: any };

export default class NewCoProvider extends Internal.Provider.Activator<FusebitNewCoClient> {
  /*
   * This function will create an authorized wrapper of the NewCo SDK client.
   */
  public async instantiate(
    ctx: Internal.Types.Context,
    lookupKey: string): Promise<FusebitNewCoClient>
  {
    const credentials = await this.requestConnectorToken({ ctx, lookupKey });
    const client: FusebitNewCoClient = new Client({ auth: credentials.access_token });

    return client;
  }
}

{
  "name": "newco-provider",
  "version": "1.0.0",
  "main": "libc/index.js",
  "files": [
    "libc/*.js",
    "libc/*.d.ts"
  ],
  "scripts": {
    "build": "tsc -b --pretty",
    "lint:check": "eslint . --ext .ts --color",
    "lint:fix": "eslint . --ext .ts --color --fix",
  },
  "peerDependencies": {
    "@fusebit-int/framework": "*"
  },
  "devDependencies": {
    "@fusebit-int/framework": ">=7.24.1",
    "@types/node": "^16.9.2",
    "@types/request": "^2.48.7",
    "@typescript-eslint/eslint-plugin": "^4.31.0",
    "@typescript-eslint/parser": "^4.31.0",
    "eslint": "^7.32.0",
    "eslint-config-prettier": "^8.3.0",
    "eslint-plugin-import": "^2.24.2",
    "eslint-plugin-security": "^1.4.0",
    "typescript": "^4.4.3"
  },
  "dependencies": {
    "@newco/sdk": "1.0.0"
  }
}

{
  "compilerOptions": {
    "target": "es2017",
    "module": "CommonJS",
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true,
    "strict": true,
    "composite": true,
    "esModuleInterop": true,
    "moduleResolution": "node",
    "resolveJsonModule": true,
    "jsx": "react",
    "forceConsistentCasingInFileNames": true
    "rootDir": "src",
    "outDir": "libc"
  },
  "include": ["src", "src/**/*.json"],
  "exclude": ["node_modules"]
}

❗️
Use peerDependencies for @fusebit-int/framework
Most of the package.json and tsconfig.json files are pretty boilerplate - there shouldn't be anything surprising to a Javascript or TypeScript developer there. There is one piece that is worth highlighting in the package.json files however:
  "peerDependencies": {
    "@fusebit-int/framework": "*"
  }
The peerDependencies section of package.json is not commonly used, but in this case it is necessary to avoid proliferation of the @fusebit-int/framework package, which is provided as a default part of the Connector and the Provider. Strange bugs can occur when this is not in the peerDependencies section!

Let's take a look at each of the index.ts files...

newco-connector/src/index.ts

The newco-connector package should export an instance of an object derived from the @fusebit-int/framework/Connector class. This conforms to the specification that the system expects - and, incidentally, is almost identical to the contract that the Integration object provides in the integration.js you might have already written.

We've provided an OAuthConnector object that does all of the nitty gritty mechanics of a "normal" OAuth handshake, including token rotation and various other capabilities. If the implementation that NewCo provides is not conventional, please let us know so we can help!

Let's start by importing in the Connector and OAuthConnector classes:

import { Connector } from '@fusebit-int/framework';
import { OAuthConnector } from '@fusebit-int/oauth-connector';

With those two objects in hand, let's define some constants that we picked up from the documentation we identified earlier:

const TOKEN_URL = 'https://newco.com/oauth/access_token';
const AUTHORIZATION_URL = 'https://newco.com/oauth';
const REVOCATION_URL = 'https://newco.com/oauth/revoke';
const SERVICE_NAME = 'NewCo';

We declare these at the top of the file so that they are easily and clearly identified - more than a few bugs have been created because those URLs were slightly off! And, for good measure, we include the name of the service we are supporting: NewCo.

class ServiceConnector extends OAuthConnector {
  protected addUrlConfigurationAdjustment(): Connector.Types.Handler {
    return this.adjustUrlConfiguration(TOKEN_URL, AUTHORIZATION_URL, SERVICE_NAME.toLowerCase());
  }
}

Now we've done the hard part of creating a Connector - calling adjustUrlConfiguration so that the configuration shows up properly in the management portal!

We also created a new class ServiceConnector that derives from the OAuthConnector base class. This lets us inherit a whole bunch of work necessary to support an OAuth protocol. If NewCo isn't OAuth compatible, then reach out via Intercom for more details on how best to interface with them. We support a variety of interfaces for Connectors, depending on the mechanisms, security, and privacy considerations that are important.

Once we've created the ServiceConnector class, let's return a new instance of it (and the class object, for good measure!) from this module.

const connector = new ServiceConnector();

export default connector;
export { ServiceConnector };

newco-provider/src/index.ts

The Provider acts to exchange the credentials supplied by the Connector and turn them into a viable client that the end developer can use. Here at Fusebit we try to make sure that the client returned matches both the public documentation as well as any sources and other tutorials that might be out there for the public SDK already. This allows the end developer - yourself, or your coworkers - to rapidly identify the "what" that they're trying to do, since they no longer have to worry about the "how".

We start the Provider off by importing in the Internal module from @fusebit-int/framework, as well as the published SDK from NewCo in the form of @newco/sdk. NewCo, happily for us, returns a Client object that contains their SDK, so we can focus our efforts there.

import { Internal } from '@fusebit-int/framework';
import { Client } from '@newco/sdk';

With the Client in hand, we declare a utility type so that your editor will recognize the typings from TypeScript...

type FusebitNewCoClient = Client & { fusebit?: any };

... and then declare a new class NewCoProvider.

export default class NewCoProvider extends Internal.Provider.Activator<FusebitNewCoClient> {

This derives from Internal.Provider.Activator, which expects the implementation to provide an instantiate function.

  public async instantiate(
    ctx: Internal.Types.Context,
    lookupKey: string): Promise<FusebitNewCoClient>
  {
    const credentials = await this.requestConnectorToken({ ctx, lookupKey });
    const client: FusebitNewCoClient = new Client({ auth: credentials.access_token });

    return client;
  }
}

The instantiate function takes as its parameters a Fusebit context object - this is the same object that is supplied to the routes in your integration.js handlers - as well as a lookupKey. This lookupKey is often, but not always, a Identity object id, and is used by the Connector (in code that's part of the OAuthConnector base class) to find the matching OAuth credentials.

This request returns a credentials object which includes a valid access_token. Happily, the Client from @newco/sdk takes that access_token as a constructor parameter, and we're then able to return the client. The client will then be presented back to the caller of the getSdk, getSdkByTenant, or similar functions for use in your Integration.

Using the New Connector!

At this point, you've successfully created a new Connector leveraging the NewCo published SDK and their OAuth endpoints. Let's make use of it!

Start by compiling the projects:

$ cd newco-connector && npm run build
$ cd ../newco-provider && npm run build

Once they finish compiling, you can publish them to npm - either the Fusebit Private Registry, or the public npm registry, using npm publish.

📘
If you publish to the public registry, you will have to increase the version number each time you need to publish a new version. The Fusebit Private Registry is less concerned about supply-chain attacks since each registry is private to a specific customer.

Now you can use the Provider by specifying that package in the Integration's component list and package.json:

{
  "id": "example-integration",
  "tags": {},
  "handler": "./integration",
  "components": [
    {
      "name": "newcoConnector",
      "entityId": "newco-example-connector",
      "provider": "newco-provider",
      "entityType": "connector"
    }
  ],
  "componentTags": {},
  "configuration": {}
}

{
  "scripts": {
    "deploy": "fuse integration deploy example-newco-integration -d .",
    "get": "fuse integration get example-newco-integration -d ."
  },
  "dependencies": {
    "superagent": "*",
    "@fusebit-int/framework": ">5.2.0",
    "newco-provider": "*"
  },
  "files": [
    "./integration.js"
  ]
}

const { Integration } = require('@fusebit-int/framework');
const integration = new Integration();

integration.router.get('/api/:tenantId/test', async (ctx) => {
  const sdk = await integration.service.getSdk(ctx, 'newcoConnector', ctx.params.tenantId);

  // Use the fully authorized and operational NewCo SDK
  const thing = await sdk.getAThing();

  ctx.body = thing;
});

module.exports = integration;

This will make the Provider available when getSdk is called with newcoConnector as the function's parameter, and the NewCoProvider will use the Connector located at newco-example-connector in Fusebit for the class's authorization needs.

On the Connector side, we can create a new Connector called newco-example-connector that makes use of our newco-connector package:

{
  "id": "example-newco-connector",
  "tags": {},
  "handler": "newco-connector",
  "configuration": {
    "mode": {
      "useProduction": true
    },
    "scope": "offline_access api",
    "clientId": "eY...Fd",
    "clientSecret": "Ey...fD"
  }
}

{
  "scripts": {
    "deploy": "fuse connector deploy example-newco-connector -d .",
    "get": "fuse connector get example-newco-connector -d ."
  },
  "dependencies": {
    "@fusebit-int/oauth-connector": ">5.2.0",
    "@fusebit-int/framework": ">5.2.0",
    "newco-connector": "*"
  }
}

The handler field in the fusebit.json refers to the package name that the Connector is published under - so if you're using the registry, make sure it includes the scope from the package name!

What's Next?

At this point, you've successfully created and, hopefully, executed your new Integration, Connector, and Provider! Congratulations! But wait, there's more!

Fusebit also supports:

Webhooks
CRON and Scheduled events
Pagination and queueing

Reach out to us via Intercom if you'd like support in building any of these capabilities into your new Connector!

Implementing Custom Connectors

👍
Estimated level of effort: 1-3 days

Setting Up the Environment

Collecting Information

Starting the Project

What Are We Building?

Steps To Build

Let's Build!

Creating the Project Directories

📘
Using the Private Fusebit npm Registry

Initializing TypeScript and npm

❗️
Use `peerDependencies` for `@fusebit-int/framework`

newco-connector/src/index.ts

newco-provider/src/index.ts

Using the New Connector!

📘

What's Next?

👍Estimated level of effort: 1-3 days

Setting Up the Environment

Collecting Information

Starting the Project

What Are We Building?

Steps To Build

Let's Build!

Creating the Project Directories

📘Using the Private Fusebit npm Registry

Initializing TypeScript and npm

❗️Use peerDependencies for @fusebit-int/framework

newco-connector/src/index.ts

newco-provider/src/index.ts

Using the New Connector!

📘

What's Next?

👍
Estimated level of effort: 1-3 days

📘
Using the Private Fusebit npm Registry

❗️
Use `peerDependencies` for `@fusebit-int/framework`