A Small breakdown of our backend systems and how our microservices work:
At the heart of our platform is a suite of over 20 microservices and applications built on NodeJS that enable users to effortlessly generate and utilize Linux containers within their preferred chatroom. Our platform currently integrates with Discord, Revolt.Chat, and Guilded.gg.
To ensure scalability and reliability, all of these microservices are hosted on a single node on Oracle Cloud, leveraging the ARM architecture which can flex in resources as needed.
In the future as we grow, we can leverage the below methods and services to link multiple nodes into this single service.
The entire system is orchestrated through a well-defined container generation workflow.
Container Generation Workflow:
When a user submits a generation command to one of our platform bots, the process begins. The bot first checks the MySQL database server for the existence of the user’s platform ID. If the ID is not found, a new one is generated, along with some default information such as a creation date, expiration date, and a randomly assigned SSH port. This information is held in memory for the final docker run command.
The container generation service then uses this information, to create a new container using the internal docker API socket assigning one of 50 internal docker networks, container names and hostnames are assigned to be the users unique platform ID. This allows the container to be accessed from within any platforms bot using “Account Binding” where we keep track of the users Chat Platform IDs and relate them within MySQL.
The generation work flow also will check the MySQL users table to see if the user has an active key from our web store, if there is a key found and it is valid, the service will automatically reflect the upgraded resources on generation.
The next step is to scan a set range of networking ports and randomly select ports out of this list, we choose ports at several different port ranges at the same time to ensure there is no random port overlapping. This allows us to assign ports during the containers life cycle and allow users to start and stop their containers with a very low likelihood of failure due to double assigned ports. Once ports are detected and assigned, the container is started via “docker run”.
Once the container is up and running, the last and finishing touch is for the system to configure Portainer access for the end user via API requests to our Portainer instance adding a team membership and setting up direct ownership of the container.
The user is notified of the completed action and provided with the necessary SSH and login details for their container as well as Portainer. The containers have a lifespan of 7 days, running in a “server” mode within the docker environment using the “tail -f /dev/null” trick to keep containers alive.
For security purposes, our platform bots communicate directly with the containers using “docker exec”, keeping the host OS secure.
Main Micro Services breakdown:
1) Expiration Service Verification
Our expiration service verifies all containers that have been generated with a specific prefix to guarantee that system service containers are not deleted. In the event that a container has not been paid for, or has expired it will be automatically removed to free up space for new users. The expiration service runs daily at 6 PM. Users can run a command within our bots to extend another 7 days or upgrade.
2) License Verification
The micro service named “LicExpire” performs a verification check on the MySQL database every hour to identify any expired keys. If a key is found to be expired, the system will automatically downgrade the associated container. Additionally, the service provides an additional 7 days of free service to prevent data loss and allow customers sufficient time to extend or purchase new upgrades.
3) Port Discovery Service
A common issue users faced was detecting open ports, hence we provide Portainer access to provide this information easier or in the case of platform issues and the user needs terminal access. However, users are unable to duplicate, edit, or modify any aspect of their containers through Portainer, as this access is mainly for container information and terminal access. Docker containers do not have information on exposed ports and it is not possible to retrieve this information from inside the container due to the unavailability of the socket. To address this issue, a port discovery service has been implemented. This service sends an HTTP query to Docker to request the exposed ports for the container’s IP address. The service then scans all ports and informs the user which ports are in use or available. A specific request can be made to return a single unused port, enabling automated software installations.
4) Message Overflow Service
When a user submits a command through the bot, the commands are executed non-interactively, which may result in a large output that may pose challenges on some platforms. To mitigate this, we have implemented a custom output storage service. The service is a middleware server built using Express, which listens for incoming requests. When a request is received with a 2048-byte header key and a designated user agent, the service stores a text document as a cache file within our web server to be served to the user. The bot automatically invokes this service when the output exceeds 1980 characters.
5) File Upload Service
We have created a custom file upload service that enables users to upload files in a user-friendly manner through their command line. The file upload service functions similarly to our output storage service, but includes an authentication key for security purposes which is stored in the users table in MySQL. The authentication key can be registered or modified within the bot platform. This key ensures that only valid files are uploaded to a specific container by their owner. We have an automated installer which asks users for their ID and their Key. Once provided the installer will configure a custom bash script as an upload client.
6) Alert Notification Service
To address the challenge of informing users about important actions taken by our system privately, providing timely warnings for service expiration, we have implemented an alert notification service. This service utilizes the Express server as a hybrid platform bot, consisting of three microservices. Each of the notification microservice listens for incoming requests. When a request is received, the unique user ID is used to retrieve the user’s platform ID, and a request is then sent to the relevant platform API to send a direct message to the user. The alert notification service can be triggered manually by calling the “notif” binary within the container or by using our API service. This service alerts users to possible bad SSH logins, as well as abusive WordPress activity and even when their container will expire as a warning to extend their service.
7) DNS Services
We utilize the PowerDNS Server along with its gMySQL plugin system to leverage the use of an in-system hosted DNS. This is done using a custom bot service which communicates directly within the PowerDNS database to administrate domain ownership and initial record data. Users register their domains within the bot itself and then the domains are able to be administered within a panel built off of PowerDNS Admin. If users do not own their own domain, we offer free subdomains.
8) Subdomain registration and Virtual Host Generation.
One of the primary uses of our platform is form web developers learning how to code or administrate websites. Due to this fact, we wanted a way to allow users to proxy their work within our main webserver. To do this, we leverage httpd’s configuration files, the bot has predefined configuration files that are customized to the users needs. When a user submits a request to the bot to generate a new vhost providing a hostname and a local port, the bot will write the needed configuration files, ensure that our wildcard SSL Certificate is valid and then gracefully restart the server. The bot then notifies the user their URL is ready.
9) Platform Protection via Custom Services
Included within our platform are several micro services which find and remove containers generated by bad actors. The techniques I won’t go into here, however, we do have the following protections included:
- Mining protection to remove abusive containers utilizing too much CPU Power due to mining.
- Port Spam / Port Flood Protection
- Malicious File Upload Detection
- Abusive Script Detection to detect internal hack attempts
- Fork Bomb Protection.
10) Notable Features:
1) We offer direct opening of git issues right within our service allowing users to easily report bugs or issues.
2) We offer container exports allowing all users the access to download their full file-system image to migrate.
3) Our system can automatically configure SSH services to only allow a specific public-key provided by the user.1
4) Our system provides care-free SSL certificate generation for hosted domains within PowerDNS.
5) Our Discord bot can automatically create other bots using provided token and application ID allowing all users instant access to a running discord bot they can start to learn with.
6) We offer our own custom API to interact with our service in code: https://docs.ssh.surf/en/api
7) Users can write files within any of our platforms chatroom and code directly in the channel.
8) Users can send files from their container right to the respective platform enabling easy file sharing.
In conclusion, our systems and services have been designed with speed efficiency and ease of use in mind for the end user. We leverage several technologies such as Docker, Portainer, NodeJS, MySQL, REST API’s, DNS, Webservers and other libraries to bring this wonderful learning experience to our users. I am even more excited to continue learning new methods and ways I can make this system better for the future and continue to assist developers in their learning adventures.