BLOGAtlas Vector Search voted most loved vector database in 2024 Retool State of AI report — Read more >>
presentation

KBank successfully migrate data from 16 databases with minimal downtime

presentation promo imageplaySign up to watch

 
MAKE, by KBank in Thailand, is a banking application featuring the categorization of transactions, receipt of monthly expenditure summaries, and a unique savings technology called "cloud pockets."

Nicha Rojsrikul, Innovation Engineer, KASIKORN Business Technology Group, presented at MongoDB.local Bangkok, sharing the app’s migration from self-hosted community edition of MongoDB database to MongoDB Atlas, the SaaS offering of MongoDB. A move prompted by the community edition reaching its end of life, and the manpower required to update and maintain it in-house. Atlas, by contrast, came with low effort upgrading, detailed metrics and a support team.

KBTG at MongoDB.local Bangkok

The team’s recommendations for migration were summed up as follows:
⁃ Use live migration  
⁃ test with copy of production data first  
⁃ Split or delay migration if necessary

Here’s the step-by-step guide:
1) Know your system
MAKE had 16 MongoDB databases to migrate, and most of them were less than 10 gigabytes. Only two databases were larger than 100 GB. As a banking app, security was key, meaning the migration would have to run within the VPC network.

2) Do your research
a. Migration tools
MAKE considered 3 tools:
mongodump and mongostore: a common-line tool for breaking up MongoDB could also be adapted to migrate data. Although simple, it involved downtime.  
mongoimport and mongoexport: generates human readable data, like JSON, CSB, or TSV, instead of binary backup. Simple to use, has an extra data filtering function, but is slower than mongostore.  
mongomirror: an official tool needing no downtime. If set up for VPC peering, either the peer's VPC CIDR block (or a subset) or the Security Group pf the peer VPC.

MAKE choose mongodump and mongostore.

b. Verification tools
dbHash : an application command for MongoDB that calculates checksum.  
GoDriver : having previously used GoServer, MAKE decided to update to the then latest GoDriver 1.10

3) Prepare for migration
a. Preparing target databases
MAKE had to create a VPC peering connection between MongoDB Atlas and Google Cloud Platform. After that, the MongoDB clusters – using higher than needed clusters for quicker migration.

b. Preparing the system
A simple upgrade of MongoDB Driver on MAKE’s system was followed by a regression test and performance test.

c. Preparing the scripts
To make the migration easier, a simple script was created that could run with different param files, enabling them to simply provide parameters such as file to read, database and number. This eliminates the need for 16 separate scripts, thus accelerating migration.

d. Testing the scripts
Nicha recommends testing with production data and gathering information like system metrics, CPU usage, memory usage, and storage usage for troubleshooting purposes along the way.

4) Migration and deployment
With 16 databases to migrate, and customers waiting to use the app, simplicity and minimum downtime were key. MAKE had two cloud engineers run migration on two separate databases simultaneously.

For large databases, such as the 126 GB log database and the 100GB transaction database, each was first duplicated on a higher performance node and then migrated from there to take advantage of the higher IOPS. Once all data had been migrated successfully, MAKE connected its service to the new database, tested the system, and proceeded to deployment.

Now, the MAKE banking app is running all its migrated services on the new MongoDB Atlas database.


playSign up to watch
View Presentation
  • IT Executive (CIO, CTO, VP Engineering, etc.)
  • Business Executive (CEO, COO, CMO, etc.)
  • Architect
  • Business Development / Alliance Manager
  • DBA
  • Technical Operations
  • Director / Development Manager
  • Product / Project Manager
  • Software Developer / Engineer
  • Mobile Developer
  • Business Analyst
  • Data Scientist
  • Student
  • Other
  • United States
  • Afghanistan
  • Åland Islands
  • Albania
  • Algeria
  • American Samoa
  • Andorra
  • Angola
  • Anguilla
  • Antarctica
  • Antigua and Barbuda
  • Argentina
  • Armenia
  • Aruba
  • Australia
  • Austria
  • Azerbaijan
  • Bahamas
  • Bahrain
  • Bangladesh
  • Barbados
  • Belarus
  • Belgium
  • Belize
  • Benin
  • Bermuda
  • Bhutan
  • Bolivia, Plurinational State of
  • Bonaire, Sint Eustatius and Saba
  • Bosnia and Herzegovina
  • Botswana
  • Bouvet Island
  • Brazil
  • British Indian Ocean Territory
  • Brunei Darussalam
  • Bulgaria
  • Burkina Faso
  • Burundi
  • Cambodia
  • Cameroon
  • Canada
  • Cape Verde
  • Cayman Islands
  • Central African Republic
  • Chad
  • Chile
  • China
  • Christmas Island
  • Cocos (Keeling) Islands
  • Colombia
  • Comoros
  • Congo
  • Congo, the Democratic Republic of the
  • Cook Islands
  • Costa Rica
  • Côte d'Ivoire
  • Croatia
  • Curaçao
  • Cyprus
  • Czech Republic
  • Denmark
  • Djibouti
  • Dominica
  • Dominican Republic
  • Ecuador
  • Egypt
  • El Salvador
  • Equatorial Guinea
  • Eritrea
  • Estonia
  • Ethiopia
  • Falkland Islands (Malvinas)
  • Faroe Islands
  • Fiji
  • Finland
  • France
  • French Guiana
  • French Polynesia
  • French Southern Territories
  • Gabon
  • Gambia
  • Georgia
  • Germany
  • Ghana
  • Gibraltar
  • Greece
  • Greenland
  • Grenada
  • Guadeloupe
  • Guam
  • Guatemala
  • Guernsey
  • Guinea
  • Guinea-Bissau
  • Guyana
  • Haiti
  • Heard Island and McDonald Islands
  • Holy See (Vatican City State)
  • Honduras
  • Hong Kong
  • Hungary
  • Iceland
  • India
  • Indonesia
  • Iraq
  • Ireland
  • Isle of Man
  • Israel
  • Italy
  • Jamaica
  • Japan
  • Jersey
  • Jordan
  • Kazakhstan
  • Kenya
  • Kiribati
  • Korea, Republic of
  • Kuwait
  • Kyrgyzstan
  • Lao People's Democratic Republic
  • Latvia
  • Lebanon
  • Lesotho
  • Liberia
  • Libya
  • Liechtenstein
  • Lithuania
  • Luxembourg
  • Macao
  • Macedonia, The Former Yugoslav Republic of
  • Madagascar
  • Malawi
  • Malaysia
  • Maldives
  • Mali
  • Malta
  • Marshall Islands
  • Martinique
  • Mauritania
  • Mauritius
  • Mayotte
  • Mexico
  • Micronesia, Federated States of
  • Moldova, Republic of
  • Monaco
  • Mongolia
  • Montenegro
  • Montserrat
  • Morocco
  • Mozambique
  • Myanmar
  • Namibia
  • Nauru
  • Nepal
  • Netherlands
  • New Caledonia
  • New Zealand
  • Nicaragua
  • Niger
  • Nigeria
  • Niue
  • Norfolk Island
  • Northern Mariana Islands
  • Norway
  • Oman
  • Pakistan
  • Palau
  • Palestinian Territory, Occupied
  • Panama
  • Papua New Guinea
  • Paraguay
  • Peru
  • Philippines
  • Pitcairn
  • Poland
  • Portugal
  • Puerto Rico
  • Qatar
  • Réunion
  • Romania
  • Russian Federation
  • Rwanda
  • Saint Barthélemy
  • Saint Helena, Ascension and Tristan da Cunha
  • Saint Kitts and Nevis
  • Saint Lucia
  • Saint Martin (French part)
  • Saint Pierre and Miquelon
  • Saint Vincent and the Grenadines
  • Samoa
  • San Marino
  • Sao Tome and Principe
  • Saudi Arabia
  • Senegal
  • Serbia
  • Seychelles
  • Sierra Leone
  • Singapore
  • Sint Maarten (Dutch part)
  • Slovakia
  • Slovenia
  • Solomon Islands
  • Somalia
  • South Africa
  • South Georgia and the South Sandwich Islands
  • Spain
  • Sri Lanka
  • Suriname
  • Svalbard and Jan Mayen
  • Swaziland
  • Sweden
  • Switzerland
  • Taiwan
  • Tajikistan
  • Tanzania, United Republic of
  • Thailand
  • Timor-Leste
  • Togo
  • Tokelau
  • Tonga
  • Trinidad and Tobago
  • Tunisia
  • Turkey
  • Turkmenistan
  • Turks and Caicos Islands
  • Tuvalu
  • Uganda
  • Ukraine
  • United Arab Emirates
  • United Kingdom
  • United States Minor Outlying Islands
  • Uruguay
  • Uzbekistan
  • Vanuatu
  • Venezuela, Bolivarian Republic of
  • Viet Nam
  • Virgin Islands, British
  • Virgin Islands, U.S.
  • Wallis and Futuna
  • Western Sahara
  • Yemen
  • Zambia
  • Zimbabwe