A Loader Balancer Application

A software load balancer written using Java Servlets

Required Tools/Softwares to successfully verify balancing

1. Java 8+ (ensure the JDK is properly installed and configured on system)
2. Apache Tomcat (version 10.0.8) (will be our server on which our servlets will be deployed)
3. Apache Jmeter (version 5.6.3) (to test our load balancer) whether its working correctly or not.
4. servlet-api.jar for compiling the java servlets found in the tomcat lib directory. Its not required to know its details but might be a case where we may have to explicitly provide in the class path to compile our servlets. I did not face any issue. Don't worry if there's any issue will provide the commands to compile the code.

Overview of the setup

Proxy Servers and Load Balancer

Have implemented a software load balancer (there are hardware load balancers as well not going into that) which will be acting as a reverse proxy and will be exposed to the internet. Anyone on the web trying to request a resource from my website will be interacting with this server.

Let's say my website is called thenextbigthing.com and my proxy server's address is 10.2.2.5 (just some random IP).

When anyone enters https://thenextbigthing.com in their browser post the DNS resolution the request would be directed to this server. First the enycrption keys are generated as per the HTTPS protocol and once this has been done every resource required by the user from our website is fetched from the application servers to which the load balancer routes the request.

Once the server to which the request will be forwarded is decided the required action is performed by it and the response is returned to the proxy server which then responds to the requester. Its a proxy as it never performed any business operation/logic and the user interacted only with the proxy it does not know about the actual application server.

Read more about proxy servers here or from any other online resource. Benefits of using a proxy server are listed below:

1. Load Balancing (what we will do) to prevent overloading a single server, allows horizontal scaling of the application server layer a new server can just be added to the available pool. As no business logic resides on the proxy server it just acts an orchestrator or mediator between the user and the application.
2. Caching static content (ie images/html/javascript/css that will be same throughout the application) at the Proxy server.
3. Enable rate limiting ensuring our actual application servers (which are also connected with our data persistence layer) are safe from DOS attacks.
4. SSL encryption only at the proxy reduces a lot of overhead.
5. Can be used as an API gateway as well.

Note: The proxy server should be capable of handling multiple requests, encryption, rate limiting and caching so must be having optimal compute power.

Repo Structure

The top level directories and what they contain are listed below

1. jMeterTestPlansAndResults: To test the functionality have added the required .jmx files along with the results screenshots.
2. servlets: All JSP servlets are in this directory. The three subdirectories are three servlets that will deployed on 3 different tomcats. As the name suggests the proxy servlet (our load balancer) in the sub-directory reverse-proxy, server1 and server2 are the application servlets that contain our application logic. Ideally this will be identical but just to show that both are being used have slightly modified the playload this will help us in testing and verifying our results as well.
3. setupScripts: Batch scripts that will be required for code compilation and simulating test scenarios
4. tomcatConfiguration: The configuration required at all the running Tomcats. Port details mostly.

Note: In an actual setup all this code will be deployed on separate machines connected over the network, for learning purpose all this can be deployed on the same machine at different ports.

Starting and using the project

Windows machine only

1. Prepare all the tomcat servers (i have taken three feel free to add as many as you like provided you have also gone through the code)

   

   We will not be changing any other configuration or file in the server, all properties will be managed through a script provided in the project. Ensure all the directory names are exactly same

2. The revere-proxy is our load balancer server, server1 and server2 are the application servers.

3. Naivigate to the directory /setupScripts it contains all the batch scripts that will help us in managing our code easily. All scripts must be excuted from this directory.

4. First execute replace_server_properties.bat this will replace the server properties for all the tomcats that are required. It will ask for 2 user inputs

   1. The number of servers in the setup (2).
   2. The path where all the tomcat servers are.

5. Now let's first execute the script compile.bat. It will ask for 2 user inputs

   1. The number of servers in the setup (2).
   2. The path where all the tomcat servers are. All our code has been compiled and moved to the corresponding server for deployment

6. We have to start the servers with the script, startup.bat. It will ask for 2 user inputs

   1. The number of servers in the setup (2).
   2. The path where all the tomcat servers are. Lets verify whether three servers are actually running or not using the netstat command

   netstat -a -b
   

   

   1. On ports 9000 and 9001 our application servers, 'server1' and 'server2' are listening.
   2. On port 9999 our proxy server is listening. Anyone who wants to access our application will be interacting with server listening on port 9999.
   3. These port configurations are in the /tomcatConfiguration directory and should not be changed because our code and test plans will also have to be changed for the same.

7. Verify by pasting this link in your browser (do this atleast 4-5 times, you will see the request has been handled by our application servers)

   http://localhost:9999/proxy/testing/test
   

   The code written for both the application servers server1 and server2 is exactly the same only thing different is the response they have are just giving out their server id 1 or 2.

8. Run the tests for jMeter.

   1. Load the test plans into your jMeter installation test plans are in the directory jMeterTestPlansAndResults/testPlans Directory structure Choose this file
   2. Do not change anything just simply click on run, have added step by step screenshots in the jMeterTestPlansAndResults/results/withTwoServers/ folder The Thread group configuration (1000 threads with a ramp up period of 1 second) The HttpRequest Sampler details of the proxy server are here Extractor for getting the payload from the proxy returned, basically we want to see which server has responded The assertion that we will be using to count requests distribution Requests handled by server 1 Requests handled by server 2 As can be seen both have handled 500 requests each that means our load balancer is working correctly it has distributed the load evenly to both the servers as per our scheduling algorithm (round robin scheduling)
   3. You can either manually stop the servers or can use the script shutdown.bat. It will be expecting 2 user inputs
      1. The number of servers in the setup (2).
      2. The path where all the tomcat servers are.

9. Scenario where one application server goes down. As discussed in the earlier section a major reason of having sych a setup is to ensure our system is highly available.

   1. Execute the script one_server_goes_down.bat.
   2. Run the tests in step 8, have added step by step screenshots in the jMeterTestPlansAndResults/results/whenOneServerIsDown/ folder.
      1. Verify using the netstat command
      2. The server listening on port 9000 is not there anymore
      3. Load distribution As expected no requests were handled by server 1 Server 2 was the only available server so it handled all the traffic

10. Have added this section to show the importance of the synchronized block in our LoadBalancerImplementation class.

    1. Switch to the branch synch-block-removed
    2. Repeat steps 5 to compile the code if your servers are already running then they will automatically detect any classes change and will redeploy all the servlets. If not then simply repeat step 6.
    3. Running the tests
       1. The code block we changed

          private synchronized String serverToBeUsed() {
              lastUsedID = (lastUsedID + 1) % serverPool;
              LOGGER.info(String.format("Server Number %d is used", lastUsedID));
              return servers.get(lastUsedID);
          }

          We haved removed the synchronized keyword and the volatile keyword from this block

              private static volatile int lastUsedID = -1;

          This will be leading to uneven distribution of the incoming requests as multiple threads might be trying to modify the static variable lastUsedID at the same time, the synchronized keyword ensured that the fuction serverToBeUsed() is executed by only thread at a time and rest all are waiting (essentially we had put a mutex lock on the variable lastUsedID in the previous implementation of the block).
       2. Run the tests 2-3 times and you will observe the distribution to the servers is random. Screenshots are provided in the folder jMeterTestPlansAndResults/results/withoutConsideringTheConcurrencyClassicRaceCondition/ Requests that were handled by Server 1 Requests that were handled by Server 2 Running the tests again
       3. You can either manually stop the servers or can use the script shutdown.bat. It will be expecting 2 user inputs
          1. The number of servers in the setup (2).
          2. The path where all the tomcat servers are.

Conclusion

Hope you enjoyed reading this article and were successfully able to reproduce the results. Do share your feedback.

Maybe we might need to deploy this on separate machines and then can then implement a weighted round robin balancing technique that will distribute load according to their weights (that may be decided based on the compute available on the servers).

I am trying to complete a series of projects that will involve a load balancer, a persistent datasource with querying capabilities and an in memory key value store.

The end goal is to run a complete website on these integrated systems.