Extending Sharetribe Flex with backend capabilities

Sharetribe Flex is a complex, production-ready system for building an online marketplace. If your project needs a lot of customization and flexibility (pun intended), but you don’t have time to build everything from scratch and time to market is very important, then it’s a great choice. However even in Flex (which is a more advanced version of its cousin „Go”) there is a limit of default customization that you can do. In this article we will explore how to push past those limits. Extending backend capabilities can be a great way to augment your marketplace without losing anything that you’ve already built. 
‍
Why is this important? Using the recipes below we were able to create a lot of advanced functionalities in our marketplace application, some of them are:

• Multiple listing categories that serve different purposes: project listings, expert profiles, services, support tickets and even exams that enable us to carry out our own experts certification. Those listings are then modified based on different triggers coming from our platform. We have also prepared endpoints for group editing of those listings based on different criteria.
• Frontend presentation of analytical data generated by our backend server, for instance number of offers and median offer value for our bidding system where our experts can provide multiple offers for each project posted by our clients.
• Sending custom emails that are not directly triggered by transaction state transitions (so there is a lot more flexibility) and can be easily modified without usage of Flex CLI by our non-technical team.
• Custom inbox system that introduced group messages, including attachments in messages, reading receipts that track and display if messages were opened and ability to trigger those messages from other users actions.
• Creating CSV files that exports merged data from multiple listings and user accounts that enable our marketing team to easily track the global state of our data and keep track of all user activities.
• Custom notifications system that allows us to specify our own triggers, send group notifications to different types of users, track notification open status and hide selected notifications after opening.
• File management system that allows us to store large files provided by users and expose them on the platform in form of convenient links.
• Predefined customer paths that include scheduled, automated emails and in-app notifications, that allows us to remind users about different types of actions that they can perform.
• Custom Stripe integration with balance management that enabled us to create deposit mechanism for our clients.
• Server-side PDF offers generation triggered by users action in a web application, accessible both from the app as well as delivered via email.
  

In this article you will find how to:

1. Set up Integration API with some practical suggestions on API keys security, keeping a unified project structure with the same deployment process, as well as creating your own Node.js REST API for your custom endpoints.
2. Connect those backend endpoints to the frontend React application using already built request-related logic from Sharetribes’ FTW template.
3. Tips for running a project in development mode with a CRA proxy without a need to rebuild it every time (hot-reloading for both frontend and backend).
4. Set up simple token-based verification middleware for your new API with additional notes on developing more secure infrastructure with time-limited tokens and permission validation.

Tool selection and prerequisites:

In order to achieve all of the things listed above we will have to make some arbitrary decisions on tool selection, however you can change those tools if you have your own preferences and almost all of the presented guidelines will still be applicable and should provide a lot of value. Tools that we are going to use here are MySQL installed on NGINX server, serving as our database and Sequelize as database ORM. We will also assume that you are using Sharetribe Web Template („FTW-daily” or „FTW-hourly”) when speaking about Node.js setup as it is a nice background that will both speed up your initial development process as well as enable us to make this article a bit shorter :) Some prerequisites are also basic knowledge of JavaScript and Node.js.

1. Setting up Integration API

There are two main APIs that are provided by Sharetribe. Marketplace API is something that you probably already know and use as it is integrated into FTW templates. It's a great tool for building initial UI but it’s also designed with some limitations which are derived from the assumption that all Marketplace API calls are used as the result of users actions directly from their devices. All of these calls are made in the names of authenticated users and you will not be able to access some of the marketplace data from them, for instance other users’ private data, transactions or messages. 

What if you would like to build some entities that share data between multiple users? Or maybe create some custom data analytics that will group data from multiple users and listings and export them as CSV files? This is where Integration API is needed. Some of the additional capabilities of integration API are: ability to query multiple users at one request, advanced transaction queries, updating/closing/opening/approving listings, and much more.

According to Sharetribe docs there are 3 different ways to use Integration API: building on top of existing Flex Template for Web (FTW), fork their Integration API example and build on top of that or create completely new application. First method has several benefits that includes faster development time (you are working on an already built server), uniform code base (you can keep frontend and backend in the same github repository) and unmodified deployment process (after making changes to backend you can deploy your app without any changes to underlying process), so it became our choice that will be described below.

Why not using Integration API directly from the frontend app? Because it would expose Integration API secret keys and put at risk your marketplace data security. Flex Template is based on the Create React App tool which is embedding all environment variables in build time.

First step to start is to create Integration API application in Flex Console which is described in details in Sharetribe docs. You don’t have to follow all the steps that includes cloning example Git repository, only setup app in Flex Console and save credentials in your .env file with these variable names:

FLEX_INTEGRATION_CLIENT_ID

FLEX_INTEGRATION_CLIENT_SECRET

If you use two different environments for test and production make sure to create Integration API applications in both of them and include them in your env.development and .env files accordingly.

Now it’s time to set up the backend server. FTW template is designed with Server Side Rendering (SSR) in mind so we have our Node.js server out of the box. We will now create new folder structure for our Integration API endpoints like this:

Some of the endpoints like “batch” jobs to update multiple listings in marketplace data will be only used by us so we’ve grouped them in /internal directory, others will be exposed outside for our frontend app to use, so they will be stored in /external folder. In /api-integration directory we will create a separate file called integrationSdk.js that will encapsulate logic related to Integration API authentication via our API keys:

We can now set up our first endpoint that will use Integration API. For test purposes we will define an endpoint that will receive some user id and return data about this user in the response body.

When the endpoint is ready we will also have to connect it to our router. We will use an already existing API router that puts all endpoints behind /api route. In server/apiRouter.js which should be already present in your project (part of FTW template) at the top of the file add a line which import our test endpoint function, then find part of the file where router middleware is defined and paste second part of code which connects router to our function after that.

To test your new server side Integration API endpoint you don’t have to run entire project with sharetribe-scripts, you can only run Node.js server with node command like this:

If everything has started correctly we can now move to Postman, which is a software used by many developers to test APIs. We will create new request, and setup our test data by extracting some example userId from existing marketplace data:

Before running request one more important thing is to set Content-Type in Headers section to “application/transit+json” to ensure correct parsing of our request body:

After running our example request we see that everything went fine and we can even have a glimpse at users’ attributes which were cut in the middle by author because whole response was very long:

2. Setting up frontend helper function to run all the requests

We have now established basic example of server side endpoint that connects to Integration API and in this way we can explore whole API further to get some useful data and ability to write this data into Flex database as well. One of the most used resources in our development team is Sharetribes’ API Reference.

What is probably missing is how to run those requests from our frontend React application and we will explore it now. In our FTW template there is already established request-related logic separation that resides in src/util/api.js file. The only thing that we have to do to extend it is to export our own custom function at the bottom of the file:

Now we can import it and use it in any container or component that is present in our React application like this:

Before deployment of your new enhanced marketplace application please remember to update your environment variables (integration client id and secret obtained from Flex Console).

3. Enable hot-reloading for both frontend and backend

Hot-reloading is a very nice and useful feature that speeds up the development process and enhances developer experience. You are most likely aware how it works on the front-end side when you are running “yarn run dev” command from sharetribe-scripts. However what you will notice is that changes made to backend endpoints will not be refreshed in this mode. What we need to do in order to see our changes is to run a “yarn run dev-server” script that will rebuild our backend. If you have a little experience with running those scripts you will certainly notice that the dev-server script is a bit slow, because it’s running the build process under the hood. If we want to speed things up in development mode we can follow these steps. First we have to install nodemon which is Node.js tool that automatically detects any changes saved in our source code and restarts server. We can do it by running command:

Then we have to set up a Create React App proxy in our package.json file. It will tell our development server to proxy all requests from frontend app running in development mode on default port 3000 to our backend server that we will run simultaneously on port 4000. 

You can find more details about proxying in Create React App docs.

Right now, last step that we have to make is to open two separate terminals, for instance in our Visual Studio Code IDE or other tool that we are using to run sharetribe-scripts, and run our frontend app and backend app simultaneously:

Voila! We have both our frontend and backend running and they will automatically reload in case of any changes made to any of them.

4. Token based verification

Our backend endpoints are running, but we might not want to expose all of them for anyone to access. In this case one of the best options is to create some token based verification, that we will in fact implement. We will use very popular JWT tokens, create functions for their generation and explore how to link them easily to our endpoints as convenient middleware.

First we have to install the ‘jsonwebtoken’ package from npm. Then we have to generate our private key for token generation (separate for test and production environments) and save it in .env.development and .env files accordingly. It should be string with alphanumeric and special characters, I’d suggest a 128-bit string that you can generate with any tool found in google but you can use longer as well. We will save it under JWT_PRIVATE_KEY variable name. Now, in the /api-integration directory we will create a jwt.js file which will export functions responsible for generating and checking our token. Our token generation is based on users UUID that in production code will be received from frontend requests. In that way we will have consistency between our tokens, users and marketplace data. We will also be able to easily verify users' permission based on JWT token, because after decoding it we could decode users id from them and check if there is a match. 

Now we can add authenticateToken() function as middleware to our testIntegrationAPI endpoint. It will intercept our request, check headers in search for a token and then validate this token with our private key. In case no token is present we will send 401 HTTP code status, and in case token is not valid we will send 403 code. When the token is correct, control will be passed over to the original function that was requested, in our case testIntegrationAPI().

To test our endpoint we have to generate our token for test purposes, in order to do that we will import generateToken into our apiRouter.js (this should not be done in production, it’s only our testing method) and run it for some arbitrary userId taken from our Flex Console. 
‍

After running our backend server we will get output like this:
‍

Now we will copy this token and insert it into our Postman request in the headers section. Don’t forget the space between ‘Bearer’ string and token value.

We can test our /testIntegrationAPI endpoint once more. If we turn off our ‘authorization’ field in headers it should return 401 status. If we provide a token with some mismatching value (for instance delete last letter) it should return 403 status, and in case where we will provide a valid token it should run our testIntegrationAPI() function.

One other thing to mention is that in our testIntegrationAPI() we can actually access the value stored in our JWT token if it’s valid. We can do it like this:

After testing our endpoint again we should see our user object with an id stored inside that should match the id of the user that we provided when the token was generated. In this way we can check who is the owner of the token and based on that we can allow or deny access to some of the resources provided by our endpoints.

Code above was only a short example of how to implement a convenient and easy way to secure our endpoints with token verification. In production-ready implementation we would have to implement functions for generating token for every user which would be called upon registration by trigger from frontend application. Suggested place to store generated tokens is some field like userToken created in privateDate in marketplace data (Flex Console). We would also have to implement a function that updates every existing user with generated token to enable them to use our token-secure endpoints.

Final Notes

In this article we explored how to further extend Sharetribes’ FTW template for building advanced marketplace applications. We’ve learned how to safely connect to Integration API and how to use it from our React application. There are of course other things that can help you further extend your backend capabilities and will nicely integrate with solutions established above. At some point in more advanced projects (like our own SoftwareSupp.com marketplace) you can also think about setting up your own database and integrating it with your backend server. From our experience we can hint that it is useful to connect your data entities by UUIDs from Sharetribe to make your data model consistent with marketplace data. You can also think about creating your own file management server that you can easily connect to this database and server-side endpoints as well as setup custom email automation with scheduler to trigger them at predefined date and time. We have done all of those in our application and we can attest that it is certainly doable but it did not fit into the scope of this article. If you are looking for help with your Sharetribe development process you can find experts with us.