Introduction to Import Existing Deployment for Automation

MongoDB
February 23, 2015
#Cloud

Why Import for Automation?

Until today, it was only possible to manage new deployments with MMS Automation. Starting today, you can get the benefits of Automation on the vast majority of your pre-existing deployments. Automation allows you to turn long and complicated manual processes into single click operations with MMS.

We’ve tried to minimize the amount of work you’ll need to do in order to attach MMS to your pre-existing deployments. In some cases some re-configuration will be necessary, and some deployment types aren’t yet supported, but stay tuned.

Getting Started

To successfully import, please check your deployment meets the following requirements:

  1. You have a Monitoring Agent and your deployment is monitored. If you have no MMS Agents running:
    • Install an Automation Agent on every server in the deployment. Instructions for installing Automation Agents can be found at Administration -> Agents
    • Use MMS Automation to deploy a Monitoring Agent. See docs here.
  2. The fully qualified domain name (“hostname -f”) for each server in the deployment must be resolvable from every other server. For example, if you login to Server A, and “hostname -f” returns a.example.com, then this address must be resolvable from Server B, C and D as well.
  3. If you are using MongoDB authentication, you must configure a MongoDB user account that the Automation Agents will use to authenticate to your deployment. 
          use admin
      db.createUser({user: "mms-automation", pwd: "yourpassword", roles: ['clusterAdmin', 'dbAdminAnyDatabase', 'readWriteAnyDatabase', 'userAdminAnyDatabase','restore']})
    Please note that if you are importing a sharded cluster, this user must be created via the mongos, as well as on every shard. The same username and password must be used throughout.
  4. Your deployment makes use only of supported MongoDB configuration file options. See docs here.
  5. The MongoDB processes that are being imported must be running as the same system user as the Automation Agent. For example, if your MongoDB process is running as the “mongod” user, the Automation Agent must also be run as the “mongod” user.
  6. If you wish to import multiple replica sets or clusters that are using MongoDB authentication, they must all use the same keyFile.

How It Works

To start the process, click on the green Add button and choose Import Existing for Automation.

The first step is to choose the item you would like to import. You will be presented with a list of all the sharded clusters, replica sets and standalones that you are currently monitoring in MMS. In this example, we’re going to import a replica set called “leaf”.

If you are using MongoDB authentication, you will additionally be asked to enter the username and password for the MongoDB user account you prepared for the Automation Agent user.

Once you click Start Import, the Automation Agents will query each process for detailed information about its current state. The Automation Agents will also gather information about any users and custom roles defined in your deployment. This information is sent back to MMS, which uses this information to construct a detailed map of your deployment.

When this process is complete, you will be invited to view the draft deployment:

The draft deployment might look like this:

The final step is to “Review & Deploy” your changes. After clicking this you will be given one last preview, and then configuration will be broadcast to the Automation Agents.

When the Automation Agents receive the new configuration, they will initiate a rolling restart of the new processes. When this completes, your deployment will be in “Goal state”.

Uh-Oh, It Doesn’t Seem To Be Working

When the Automation Agents gather information about the state of your deployment, we also perform extensive validation to ensure that your deployment is suitable for import.

However, if we missed something, and things don’t seem to be going well, you can always Unmanage any item from your deployment. When you Unmanage the item, it is removed from the configuration for the Automation Agents. The Automation Agents will not shutdown the processes, they will simply stop interacting with them.

← Previous

MongoDB 3.0 features: Big Polygon

MongoDB has supported geospatial queries since MongoDB 1.4, and has added new geospatial features with most releases, including support for simple polygons in MongoDB 2.0, GeoJSON in MongoDB 2.4, and several new capabilities in MongoDB 2.6. In this post we'll cover a new feature in MongoDB 3.0 for searching polygons that are larger than half the earth's surface. I am using PHP for my examples, but of course you can do this from any other supported language too. Finding objects in a specific area (polygon) can be done like so: demo->points; $c->find( [ 'loc' => [ '$geoWithin' => [ '$geometry' => [ 'type' => 'Polygon', 'coordinates' => [ [ [ -0.91, 51.74 ], [ -0.91, 51.27 ], [ 0.67, 51.27 ], [ 0.67, 51.74 ], [ -0.91, 51.74 ] ] ] ] ] ] ] ); ?> The GeoJSON format, which we use in this query, defines a polygon as an array of linear rings, which themselves are a list of coordinate pairs beginning and ending with the same point to form a closed ring. The first linear ring is required and defines the exterior boundary of the polygon. Subsequent rings, if specified, would define interior boundaries (i.e. holes) within the polygon. As an image, the geometry in this query looks like: The GeoJSON format specification does not mention anything about the direction or winding of points. The application is left to determine if the overall geometry defines an inclusive or exclusive search area. Both of the following images are possible interpretations when just taking the GeoJSON specification into account: These queries would yield very different results! Because GeoJSON does not address winding, sometimes applications make wrong decisions . MongoDB deterministically chooses the area that is the "smallest of the two". For our query, it would consider the polygon as an inclusive search area, which is the first of the two interpretations shown above. This logic works for most applications, but it falls apart when you want to find objects within a polygon that spans more than half of the Earth's surface. In the small example of London, you would certainly expect the smallest area, the one that covers London to be the search area. But when we consider this much larger polygon (a circle centred around 25°N, 90°E with a diameter of 115°), then it is not so obvious: When the polygon is larger than half of the Earth's surface, the "smallest of the two" is actually the area we intended to exclude . Thankfully, MongoDB supports a special coordinate reference system (CRS), which can force geospatial queries to consider the direction of coordinates in deciding upon an inclusive or exclusive area. Consistent with KML , and WKT/WKB , counter-clockwise and clockwise winding will imply inclusive and exclusive behaviour, respectively. This results in the following two possibilities: Big Polygon support in MongoDB 3.0 addresses this. MongoDB's implementation of GeoJSON allows us to specify a custom coordinate reference system on any geometry object via a crs property. Currently MongoDB only supports our custom CRS, to specify strict winding. In the future, MongoDB is likely to support others , such as the US National Grid , UTM or the UK's Ordnance Survey National Grid as well. In the query, our custom CRS would appear as an extra option within the $geoWithin or $geoIntersects criteria: demo->points; $c->find( [ 'loc' => [ '$geoWithin' => [ '$geometry' => [ 'type' => 'Polygon', 'coordinates' => [ [ [ -0.91, 51.74 ], [ -0.91, 51.27 ], [ 0.67, 51.27 ], [ 0.67, 51.74 ], [ -0.91, 51.74 ] ] ], 'crs' => [ 'type' => 'name', 'properties' => [ 'name' => 'urn:x-mongodb:crs:strictwinding:EPSG:4326' ] ] ] ] ] ] ); ?> This example is only relevant when searching an area that is larger than half of the Earth's surface. In most cases, the default "smallest of the two" behavior will be sufficient and you will not need to specify a custom coordinate reference system. If you’re considering updating your version of MongoDB, take a look at our Major Version Upgrade consulting services: Upgrade with Confidence

February 23, 2015
Next →

Accelerating to T+1 - Have You Got the Speed and Agility Required to Meet the Deadline?

On May 28, 2024, the Securities and Exchange Commission (SEC) will implement a move to a T+1 settlement for standard securities trades , shortening the settlement period from 2 business days after the trade date to one business day. The change aims to address market volatility and reduce credit and settlement risk. The shortened T+1 settlement cycle can potentially decrease market risks, but most firms' current back-office operations cannot handle this change. This is due to several challenges with existing systems, including: Manual processes will be under pressure due to the shortened settlement cycle Batch data processing will not be feasible To prepare for T+1, firms should take urgent action to address these challenges: Automate manual processes to streamline them and improve operational efficiency Event-based real-time processing should replace batch processing for faster settlement In this blog, we will explore how MongoDB can be leveraged to accelerate manual process automation and replace batch processes to enable faster settlement. What is a T+1 and T+2 settlement? T+1 settlement refers to the practice of settling transactions executed before 4:30pm on the following trading day. For example, if a transaction is executed on Monday before 4:30 pm, the settlement will occur on Tuesday. This settlement process involves the transfer of securities and/or funds from the seller's account to the buyer's account. This contrasts with the T+2 settlement, where trades are settled two trading days after the trade date. According to SEC Chair Gary Gensler , “T+1 is designed to benefit investors and reduce the credit, market, and liquidity risks in securities transactions faced by market participants.” Overcoming T+1 transition challenges with MongoDB: Two unique solutions 1. The multi-cloud developer data platform accelerates manual process automation Legacy settlement systems may involve manual intervention for various tasks, including manual matching of trades, manual input of settlement instructions, allocation emails to brokers, reconciliation of trade and settlement details, and manual processing of paper-based documents. These manual processes can be time-consuming and prone to errors. MongoDB (Figure 1 below) can help accelerate developer productivity in several ways: Easy to use: MongoDB is designed to be easy to use, which can reduce the learning curve for developers who are new to the database. Flexible data model: Allows developers to store data in a way that makes sense for their application. This can help accelerate development by reducing the need for complex data transformations or ORM mapping. Scalability: MongoDB is highly scalable , which means it can handle large volumes of trade data and support high levels of concurrency. Rich query language: Allows developers to perform complex queries without writing much code. MongoDB's Apache Lucene-based search can also help screen large volumes of data against sanctions and watch lists in real-time. Figure 1: MongoDB's developer data platform Discover the developer productivity calculator . Developers spend 42% of their work week on maintenance and technical debt. How much does this cost your organization? Calculate how much you can save by working with MongoDB. 2. An operational trade store to replace slow batch processing Back-office technology teams face numerous challenges when consolidating transaction data due to the complexity of legacy batch ETL and integration jobs. Legacy databases have long been the industry standard but are not optimal for post-trade management due to limitations such as rigid schema, difficulty in horizontal scaling, and slow performance. For T+1 settlement, it is crucial to have real-time availability of consolidated positions across assets, geographies, and business lines. It is important to note that the end of the batch cycle will not meet this requirement. As a solution, MongoDB customers use an operational trade data store (ODS) to overcome these challenges for real-time data sharing. By using an ODS, financial firms can improve their operational efficiency by consolidating transaction data in real-time. This allows them to streamline their back-office operations, reduce the complexity of ETL and integration processes, and avoid the limitations of relational databases. As a result, firms can make faster, more informed decisions and gain a competitive edge in the market. Using MongoDB (Figure 2 below), trade desk data is copied into an ODS in real-time through change data capture (CDC), creating a centralized trade store that acts as a live source for downstream trade settlement and compliance systems. This enables faster settlement times, improves data quality and accuracy, and supports full transactionality. As the ODS evolves, it becomes a "system of record/golden source" for many back office and middle office applications, and powers AI/ML-based real-time fraud prevention applications and settlement risk failure systems. Figure 2: Centralized Trade Data Store (ODS) Managing trade settlement risk failure is critical in driving efficiency across the entire securities market ecosystem. Luckily, MongoDB integration capabilities (Figure 3 below) with modern AI and ML platforms enable banks to develop AI/ML models that make managing potential trade settlement fails much more efficient from a cost, time, and quality perspective. Additionally, predictive analytics allow firms to project availability and demand and optimize inventories for lending and borrowing. Figure 3: Event-driven application for real time monitoring Summary Financial institutions face significant challenges in reducing settlement duration from two business days (T+2) to one (T+1), particularly when it comes to addressing the existing back-office issues. However, it's crucial for them to achieve this goal within a year as required by the SEC. This blog highlights how MongoDB's developer data platform can help financial institutions automate manual processes and adopt a best practice approach to replace batch processes with a real-time data store repository (ODS). With the help of MongoDB's developer data platform and best practices, financial institutions can achieve operational excellence and meet the SEC's T+1 settlement deadline on May 28, 2024. In the event of T+0 settlement cycles becoming a reality, institutions with the most flexible data platform will be better equipped to adjust. Top banks in the industry are already adopting MongoDB's developer data platform to modernize their infrastructure, leading to reduced time-to-market, lower total cost of ownership, and improved developer productivity. Looking to learn more about how you can modernize or what MongoDB can do for you? Zero downtime migrations using MongoDB’s flexible schema Accelerate your digital transformation with these 5 Phases of Banking Modernization Reduce time-to-market for your customer lifecycle management applications MongoDB’s financial services hub

May 25, 2023