Introduction

I'm Lucas, 23 years old, born and raised in Brazil. I'm currently in the 11th period of my Bachelor's Degree in Computer Engineering at Universidade Federal de Goiás. As long as I can remember, I've always loved games, competition, and discovering and using patterns. These traits paved the way for multiple paths in my life, such as going from Football (Soccer) to online games, where some were cooperative and some competitive-cooperative, or the passion for patterns into a love for mathematics and physics.

At the end of 2014, after learning about Minecraft Servers, the natural path was to start my programming journey. So I followed it by creating plugins to add game modes and functionalities to play with friends. Since then, I've been keeping the habit of programming, going through some languages: Java, C, a bit of C++, JavaScript, TypeScript, Go, and Rust.

Learning how to program helped me improve my ability to shape patterns into words without the strictness of math. This skill permitted me to learn more, which proved helpful for getting into a University, as most universities in Brazil use a National Test (ENEM) to select applicants, where this test relies on speed and accuracy of the information.

Joining the University allowed me to work and experiment with multiple aspects of Software and Hardware development throughout the classes. Among all the projects, my favorite is a Compiler, as I developed it from scratch without using Regex or any other tools to build parts of the project. What makes this project special is that a Compiler, in essence, is a translator, which turns building it into a lovely giant puzzle where each piece is a different pattern.

Jobs

Hold the place for a summary of my career so far with timeline

At Tindin

In my time at Tindin throughout 2022, I had the opportunity to work on multiple products, first maintaining a legacy system written in pure JavaScript, then joining the team on the newest product, this time written in TypeScript.

For the first month, my only responsibility was to support the old product, where I solved multiple bugs and implemented some missing features while maintaining constant communication with the support team.

After the first month, I started the process of joining the team in the new project, where I was at first just doing some simple features and learning the system structure. I completed this process during the chaos of an event to announce the product, where I was able to help the team figure out where the bugs and the terrible performance were coming from.

Not long after the event, I was already responsible for some core system refactors, implementing impactful features, looking after the database and the application performance, integrating the system with marketing platforms, and assisting the team in their code. During this period, some of the tasks I got were to rewrite the Permission system to accommodate new requirements and find the reason for poor system performance, which led to this post.

TLDR

Throughout my time at TINDIN, I did this:

  • Supporting a legacy system;
  • Developed and maintained the features of the new company product;
  • Refactors in the structure;
  • Improve system performance;
  • Assisted colleagues with their code;
  • Integrate the system with other platforms.

Posts

  1. How to manage connections in a lambda environment
  2. Code Duplication is Bad, Right?!

How to control connections with a database from within a node lambda environment

Running out of database connections is a scary thing. Out of nowhere, multiple support requests pop out saying that the service is 'laggy', 'taking too much time, 'slow', then you start looking around and find that the problem is that you're topping the number of connections allowed for that database, you try to upgrade the database and goes to search on how to fix it.

Regarding this tutorial

  • It is focused on problems related to code or configuration, so cases related to a peak in the number of simultaneous lambdas instances that is bigger than the number of connections are not a topic
  • We're using javascript/typescript running on NodeJS, but it's possible to extend the reasoning behind these solutions to other languages and runners

Why it happens?

There are many reasons that it can happen, but the main ones that are related and can be solved by code or configuration are:

  1. Lambda functions create new connections without dropping them
  2. Handlers creating connections with a big connection pool

The first one can only happen if the lambda instance keeps the connection to the database. So let's say that you're expecting that 200 lambda instances are executing at the same time over the timespan that a lambda can live, but the DB accepts only 100 connections simultaneously, that means that 100 lambda instances won't be able to connect if each lambda keeps its connection and then the service stop working.

The second one mostly happens if there is no tunning in the configuration used to create a connection to the DB.

Identifying

Identifying each problem can be a burdening task, but here are some characteristics that are related to each problem

Creating new connections and not dropping them afterward

A problem like this will appear on a graph of the number of connections to the database over time. So if you have a lot of long straight lines on normal usage, it's usually a sign that you have a problem dropping the connections. But if you cross-reference with the number of lambda instances that were executing at the time, and both are stable, it's almost sure that you're having a problem keeping connections alive.

Big connection pool

This problem is usually easier to identify if you have a small number of lambda instances running at some point, let's say 10 lambda instances, and at the same time have a very high amount of connections, let's say 200 connections, then this is a very strong sign that the connection pool is not configured properly.

How to solve "Create new connections without dropping them"

There are two main ways to solve this problem, one consists of making a connection and then dropping it before returning a result, and the other one is to tweak the connection configuration to make it drop after some idle time if your driver and database have this feature.

Create a connection and drop it after use

Because of the assumption that the instance is keeping the connection, using this solution means a change in the structure, so be careful when choosing this solution

The solution will differ from synchronous and asynchronous handlers because of how those instances are handled.

Synchronous

In a synchronous handler, the first thing to do is to remove anything that allows the lambda return without dropping the connection, for NodeJS lambda functions this is the context.callbackWaitsForEmptyEventLoop = false. After this step, just adjusts the handler to create a connection and then drop it at the end. Because the handle is synchronous, there is no need to bother with a connection being stored globally, as no other requests are running at the same time.

An example of synchronous code can be:

let connection = null

const connect = () => {
  // create a connection and save it
}

const disconnect = () => {
  // disconnect the connection saved
}

const handler = (event, context, callback) => {
  connect()
  const response = doSomething(event, context)
  disconnect()
  callback(null, response)
}

Asynchronous

Here things start to get a bit complicated, first because of the way async lambda works, anything in the global scope is kept alive throughout the lifetime of the instance and second as handling connections like the synchronous case would fail and probably cause more problems as it would create race conditions over the connection and be possible to try to use a connection that was disconnected by another request.

So as storing the connection in a global scope would cause problems because of racing conditions, using this type of solution probably will mean a bigger refactor if you use mongoose with an 'export Model pattern' or related patterns.

All it's needed to use this solution is to create a connection inside the handler at the start of the request (Be careful with connection configuration and especially the connection pool), pass this connection to every place that uses a connection to the DB, and in the end close it.

An example of asynchronous code would be:

const connect = async () => {
  // create a connection and return it
}

const disconnect = async (connection) => {
  // close the connection
}

const handler = async (event, context) => {
  await connect()
  const response = await doSomething(connection, event, context)
  await disconnect()

  return response
}

Tweaking the connection configuration to auto disconnect after idle time

Most of the database and database drivers support a configuration like maxIdleTimeMS. If you're using one that supports it, at the moment of creation of the connection just set the config to something close to 10x the time of the slowest query that you have, it'll be probably something around 1000ms, after that is manual tweaking, but by my experience, it will be enough to keep in control

How to solve "Connections with big connection pool"

The problem is self-explanatory, so following the same steps as tweaking the max idle time, you can probably change the size of the connection pool and some drivers may offer the possibility to control even the range of the number of connections in the pool. For example, using mongoose (MongoDB), it's possible to control the max and the minimum size of the pool by setting maxPoolSize and minPoolSize respectively.

Code Duplication is Bad, Right?!

Code duplication is often considered a bad practice in programming. The DRY (Don't Repeat Yourself) principle advocates against it, and developers usually put in a lot of effort to identify and eliminate duplicated code when refactoring. However, is it always true that code duplication is harmful? I believe that there is more to this story.

Most of the new and semi-refactored code is in some way a duplicate. Then it gets refactored, hopefully reducing it to only the essentials. This is the natural process of code. Aiming to write the "perfect code" on the first try is the same as trying to go for a run without a proper warmup. Anyone who runs without warming up expects either an injury or low performance. Writing a far-from-perfect code full of duplications is the warmup needed to create the so-called "perfect code."

Making the "perfect code" is a process. Attempting to strictly adhere to a principle like DRY from the start can hinder the process and result in overly complex code at the wrong time, similar to premature optimization.

Example

An example can be (sorry, but its react):

    <form>
      {header && <h2>{header}</h2>}
      <label>
        Field 1:
        <input type="text" name="field1" value={formData.field1} onChange={handleChange} />
      </label>
      <label>
        Field 2:
        <input type="text" name="field2" value={formData.field2} onChange={handleChange} />
      </label>
      <label>
        Field 3:
        <input type="text" name="field3" value={formData.field3} onChange={handleChange} />
      </label>
      <div>
        <button onClick={handleSave}>Save</button>
        {onDelete && <button onClick={handleDelete}>Delete</button>}
      </div>
    </form>

This code has a lot of repetitions, but it's simple enough to be maintainable, and it's possible to use this as a first draft. An improvement to this could be:

    <form>
      {header && <h2>{header}</h2>}
      <FormField label="Field 1" name="field1" value={formData.field1} onChange={handleChange} />
      <FormField label="Field 2" name="field2" value={formData.field2} onChange={handleChange} />
      <FormField label="Field 3" name="field3" value={formData.field3} onChange={handleChange} />
      <div>
        <button onClick={handleSave}>Save</button>
        {onDelete && <button onClick={handleDelete}>Delete</button>}
      </div>
    </form>

This code has a few duplications, but it improved the maintainability by grouping each form field into a component, reducing the number of places to change when the time to change comes, and it didn't introduce much complexity.

In contrast, developing strictly from DRY, we could very easily get into a complex code such as:

    <form>
      {header && <h2>{header}</h2>}
      {formFields.map((field, index) => (
        <FormField
          key={index}
          label={field.label}
          name={field.name}
          value={field.value}
          onChange={(e) => handleChange(index, e.target.value)}
        />
      ))}
      <div>
        <button onClick={handleSave}>Save</button>
        {onDelete && <button onClick={handleDelete}>Delete</button>}
      </div>
    </form>

This type of code usually requires a lot of context to where to change, reducing readability, as multiple code hops are needed to understand what this is doing. Reducing readability hurts the maintainability, even though it was slightly improved, from the code perspective, as adding one more field is just a matter of editing an array. It's worth noting that this type of code can improve code readability in some cases, but in most cases, it can decrease it.

A possible better way

When it comes to identifying code duplication, it can be helpful to view it as a sign that indicates the need for refactoring. Duplications are essentially sections of code that share the same structure but differ in the input data. This is a clear indication of where changes can be made to improve the code. Refactored code typically has a better structure than from-the-start-generic code, as it has been modified based on the actual markers of duplication rather than just hypothetical ones.

Conclusion

In conclusion, code duplication is often detrimental. Principles like DRY aim to reduce it, but applying them prematurely can increase code complexity, hurting readability and maintainability. A better way to see code duplication is as markers for refactors.