- This module extends TypeORM capabilities to support distributed database environments with List and Range-based sharding strategies in a NestJS application.
The development of this library was driven by the need for robust sharding or multi-tenancy support in web applications. Traditional single-database approaches can limit scalability and isolation between tenants. Sharding offers a solution by partitioning data across multiple databases, thus enhancing performance, scalability, and data isolation—key components in multi-tenant architectures. This library aims to simplify the integration of sharding into NestJS applications, making it easier to manage large-scale, multi-tenant data models.
- Support for List and Range Sharding: Configure your shards based on simple list values or range segments.
- Transparent Repository Operations: Interact with your entities as if they were part of a non-sharded database.
- NestJS Integration: Fully integrated with NestJS modules and dependency injection system.
- NPM
$ npm install nest-typeorm-sharding-repository --save- Yarn
$ yarn add nest-typeorm-sharding-repository- Instead of setting the TypeORM, set the ShardingManager. Most of the settings are similar to TypeORM.
- In this example, the User entity has a key of number type. It is recommended to set the ShardingManager according to the rules of a compatible key or range of keys.
import { Module } from '@nestjs/common';
import { User } from '../entity/user';
import { ShardingType } from 'nest-typeorm-sharding-repository';
import { ShardingModule } from 'nest-typeorm-sharding-repository';
@Module({
imports: [
ShardingModule.forRoot({
shardingType: ShardingType.LIST,
type: 'sqlite',
synchronize: true,
logging: 'all',
entities: [User],
shards: [
{...dbConf, database: ':memory:', key: 'default', default: true },
{...dbConf, database: ':memory:', key: 'partner1' },
{...dbConf, database: ':memory:', key: 'partner2' },
{...dbConf, database: ':memory:', key: 'partner3' }
/**
* For Range based sharding
* {...dbConf, database: ':memory:', minKey: 0, maxKey: 10000 }
* */
],
}),
ShardingModule.forFeature([User])
],
})
export class AppModule {}- Use @ShardingEntity instead of @Entity decorator.
- Inherit ShardingBaseEntity instead of BaseEntity.
- New data will be inserted into the added database shard.
import { Column, Index, PrimaryGeneratedColumn } from 'typeorm';
import { ShardingEntity } from 'nest-typeorm-sharding-repository';
import { ShardingType } from 'nest-typeorm-sharding-repository';
import { ShardingBaseEntity } from 'nest-typeorm-sharding-repository';
@ShardingEntity<User, string>({
/**
* Declare the sharding mechanism
* RANGE and LIST based sharding are supported
* */
type: ShardingType.LIST,
/**
*
* Define a boolean function that will do sharding resolution given an entity and a sharding key.
* It will determine which shard will be picked.
* This will be used during save operations.
* If it doesn't resolve, the first default shard or if its not defined, the last shard will be picked.
* NB: To process one entity, it can be called as many as the number of shards
* */
findShard: (entity: Case4, shardingKey: string) => !!entity.partner && shardingKey === entity.partner,
})
export class User extends ShardingBaseEntity {
@PrimaryGeneratedColumn()
id!: number;
@Column()
@Index()
firstName!: string;
@Column()
lastName!: string;
@Column()
age!: number;
@Column()
partner!: string;
}- For Range based sharding follow the below example
@ShardingEntity<User, number>({
type: ShardingType.RANGE,
// Decide in which shard the entity will be processed. To process one entity, it can be called as many as the number of shards.
// If data is being inserted, the key may be empty.
// If false is returned in all cases, the last shard is chosen.
// For minKey and maxKey, the values defined in ShardingManager are delivered.
findShard: (entity: User, minKey, maxKey) => entity.id && minKey <= entity.id && entity.id < maxKey,
// Similar to findShard, but an ID value is passed instead of an entity.
// The type of ID is ambiguous. In this example, passed as number.
// Usually, ID values referenced in repository.findByIds() etc. are passed.
// You need to determine which shard the id resides on.
findShardById: (id: number, minKey, maxKey) => id && minKey <= id && id < maxKey,
})
export class User extends ShardingBaseEntity {
@PrimaryGeneratedColumn() id: number;
@Column() firstName: string;
@Column() lastName: string;
@Column() age: number;
}-
ShardingRepositoryServicecompatible with TypeORM.BaseEntity and ShardingBaseEntity. - Repository Interface
import { Injectable } from '@nestjs/common';
import { InjectRepository } from 'nest-typeorm-sharding-repository';
import { User } from '../entity/user';
import { ShardingRepositoryService } from 'nest-typeorm-sharding-repository';
@Injectable()
export class UserService {
constructor(
@InjectRepository(User)
private readonly userRepository: ShardingRepositoryService<User>,
) {}
async findAllCases(): Promise<User[]> {
return this.userRepository.find();
}
async saveCase(user: User): Promise<void> {
await this.userRepository.save(user)
}
}// Provides almost the same functionality as TypeORM BaseEntity.
const entity = await Case1.save({
firstName: 'Typeorm',
lastName: 'Sharding',
age: 10,
});
await entity.save();
await entity.remove();
await entity.softRemove();
await entity.recover();
await entity.reload();- An example of how to write end to end tests, assuming you have set up the main AppModule.
import { Test, TestingModule } from '@nestjs/testing';
import { INestApplication } from '@nestjs/common';
import { AppModule } from './module/your-app-module';
import { CaseService } from './service/case-service';
import { Case4 } from './entity/case4';
import { ShardingRepositoryService } from 'nest-typeorm-sharding-repository';
import { ShardingManager } from 'nest-typeorm-sharding-repository';
import { getRepositoryToken } from 'nest-typeorm-sharding-repository';
describe('E2E Tests Example', () => {
let app: INestApplication;
let caseService: CaseService;
let case4Repository: ShardingRepositoryService<Case4>;
let shardingManager: ShardingManager;
beforeAll(async () => {
const moduleFixture: TestingModule = await Test.createTestingModule({
imports: [AppModule],
providers: [CaseService],
}).compile();
app = moduleFixture.createNestApplication();
caseService = app.get<CaseService>(CaseService);
const repositoryToken = getRepositoryToken(Case4);
case4Repository = app.get(repositoryToken);
shardingManager = app.get(ShardingManager);
case4Repository.delete({});
await app.init();
});
beforeEach(async () => {
await case4Repository.delete({});
});
it('should ensure service is defined', async () => {
expect(caseService).toBeDefined();
})
it('should ensure repository is defined', () => {
expect(case4Repository).toBeDefined();
});
it('should retrieve a repository service and interact with the database', async () => {
const result = await caseService.findAllCases();
expect(result).toBeDefined();
});
it('should save entity in the db and be able to find it', async () => {
const case4: Case4 = new Case4()
case4.age = 4
case4.firstName = 'Foo';
case4.lastName = 'Bar';
case4.partner = 'test';
await caseService.saveCase(case4);
const cases = await caseService.findAllCases();
const count = await case4Repository.count();
expect(cases).toBeDefined();
expect(cases.length).toBe(1);
expect(count).toBe(1);
})
// Additional tests...
afterAll(async () => {
await app.close();
});
});Currently, most repository find methods in our library do not automatically determine which shard to query, particularly when dealing with List-based sharding where a specific field, such as a partner field, acts as the sharding key. For example, consider the following usage in a user service:
userRepository.findBy({ firstName: 'foo'})In scenarios like this, it's challenging to know which shard to use based on the query alone. At present, our approach involves conducting a full scan across all available shards using parallel execution. Although this method leverages parallelism, it may lead to potential performance drawbacks, especially when the number of shards increases significantly.
To address this, when using List-based sharding, we have enhanced the TypeORM BaseEntity methods findOneById and findByIds to support an optional field called shardingKey:
Repository<T>.findOneById(id: string | number | Date | ObjectID, shard, shardingKey?: string): Promise<T | null>
Repository<T>.findByIds(ids: any[], shardingKey?: string): Promise<T[]>These enhancements allow you to explicitly specify a sharding key along with the query, thereby improving query resolution performance by directly accessing the appropriate shard.
In future developments, we will consider extending all repository find methods with optional sharding key parameter. The downside is the change of the the TypeORM BaseEntity, which we intend to follow as much as we can for cross compatibility
Looking ahead, we are considering extending all repository necessary, finds and updates methods to include an optional sharding key parameter. This modification aims to streamline query processes across sharded environments by ensuring that queries are directed to the correct shard, thereby enhancing efficiency and performance.
However, one challenge we face with this approach is the modification of the TypeORM BaseEntity. We aim to maintain as much compatibility with the original TypeORM interfaces as possible to ensure that our library can be used seamlessly with existing TypeORM-based applications. We will continue to explore ways to integrate these features without significantly diverging from the established patterns of TypeORM.
This project is a fork of kibae/typeorm-sharding-repository, originally developed by Kibae Shin. Significant enhancements and adaptations have been made to support list based sharding and integrate this solution seamlessly with NestJS, focusing on transparency and ease of use in multi-tenant environments.