redisearchStock

A stock ticker solution typeahead solution based on downloaded stock files. Uses Python redisearch for an API and jquery with bootstrap ajax typeahead plugin.

Outline

• Overview
• Initial Project Setup
• Important Links
• Instructions
  • Create Environment
  • Download the datafiles
  • Set Environment Variables
  • Multiple Deployment options
  • Docker Compose Python
    • Prepare the Load
  • Deploy Python on Linux
    • Create python environment
    • Prepare Ticker Load on Python
  • Kubernetes
    • Install Redis Enterprise
    • Add Redisinsights
    • Deploy application
    • Publish Docker Image
    • Add Memtier Benchmark
  • Use the Application
    • Create Index
    • Start Ticker Load
• Troubleshooting

Overview

Uses python to load stock market daily values from download flat files. Using a redisearch index, a web browser application written in jquery uses typeahead prioritizing stocks with the highest volume to provide type ahead suggestions. When a stock is selected, more detail is displayed on the stock history.

Important Links/Notes

• bootstrap ajax typeahead example
• Redis Stack
• Redis Search
• Redis Insight
• Stooq stock files
• Run Python on k8s
• k8s persistent volume claim
• GKE persistent volume claims

Bootstrap ajax typeahead

This plugin is in the repository so no setting is needed. To learn more-follow the directions in the bootstrap-ajax-typahead github

Instructions

Create environment

Clone the github

get clone https://github.com/jphaugla/redisearchStock.git

Download the datafiles

• Download stock files here Stooq stock files
• Once downloaded, move the file (it should be a directory called data) to the main redisearchStock directory
  • Can combine the various stooq files at the daily level by including world, us, etc under this daily directory
  • There is a separate file for each stock or currency with a long history of data. See instructions below for setting the environment variables to limit history load
  • To simplify things, best to remove the spaces in the file directory such as "nasdaq stocks". The spaces are dealt with in docker-compose but never cleaned this up in k8s. Just easier to eliminate the spaces.

Set environment variables

The docker compose file has the environment variables set for the redis connection and the location of the data files. In k8s, the environment is set in the configmap. This code uses redisearch. The redis database must have redisearch installed. In docker, the redis stack image contains all the modules. In k8s, redisearch is added in the database yaml file. Check the environment variables for appropriateness. Especially check the TICKER_DATA_LOCATION because loading all of the US tickers with all of the history can be a lot of data on a laptop. Here is an explanation of the environment variables. Modify these values in docker-compose.yml or in the configmap for k8s. If outside of a container, there is a file created with environment variable at scripts/app.env.

variable	Original Value	Desccription
REDIS_HOST	redis	The name of the redis docker container
REDIS_PASSWORD		Redis Password
REDIS_PORT	6379	redis port
REDIS_INDEX	Ticker	redis port
PROCESSES	6	On larger machines, increasing this will increase load speed
WRITE_JSON	false	flag to use JSON instead of Hash structures
PIPELINE	true	flag to use pipeline for each of the ticker files (only works on python)
OLDEST_VALUE	20220101	Skip any records older than this date
CURRENT_VALUE	20220414	Use this as current value. This also sets mostrecent flag
PROCESS_RECENTS	false	will set most recent flag for specified keys back to false (requires creation of specific redis set)
----------------------	-----------------	------------------------------------------------------------------------------------------------------

Multiple options for creating the environment:

• run with python docker-compose using a flask and redis container
• installing for mac os
• running on linux (probably in the cloud)
• running on kubernetes (example uses GKE)
• run using java instead of python This java implemention is in a subfolder

Docker Compose Python

Build just needs to be done initially. NOTE: if building a new image for k8s, ensure the Dockerfile is doing a copy of the src directory into the image and not relying on docker-compose mount of the src directory. Additionally, docker can be run with the Java application. See java readme

cd python
docker-compose build
docker-compose up -d 

Deploy Python

Can be issues with running flask on linux at the time of installing requirements files

create python environment

cd src
python3 -m venv venv
source venv/bin/activate

• Use an environment file for locations
• Need to make sure the data location variables are set correctly
• Can also set the number of concurrent processes for the client using the "PROCESSES" environment variable

source scripts/app.env

Prepare Ticker on Python

• execute python scripts from the src directory
• if trouble with installing requirements, here are some links Flask on AWS Linux 2 Flask on Ubuntu Flask on RedHat

cd src
pip install -r requirements.txt

Kubernetes

This example is showing GKE steps-adjust accordingly for other versions

Install Redis Enterprise k8s

• Get to K8 namespace directory

cd k8s

• Follow Redis Enterprise k8s installation instructions all the way through to step 4. Use the demo namespace as instructed.
• For Step 5
  • the admission controller steps are not necessary
  • but the webhook instructions are not necessary
• Don't do Step 6 as the databases for this github are in the k8s subdirectory of this github

Create redis enterprise database.

Verify cluster is ready

• check health using kubectl

kubectl get all
kubectl get rec
kubectl get pods

• check enterprise node cluster ui (optional steps just for demonstration purposes)

./getClusterUnPw.sh

• port forward so can access the redis cluster

kubectl port-forward service/rec-ui 8443

• https://localhost:8443

Create redis database

kubectl apply -f redis-enterprise-database.yml

verify database

• check for database in management ui
• check database yaml output
• get port and password for database
• port-forward to the database

kubectl get redb/redis-enterprise-database -o yaml
./getDatabasePw.sh
kubectl port-forward service/redis-enterprise-database 10740

• connect to redis-cli - use the password and port from the output of ./getDatabasePw.sh

redis-cli -p 16379 -a rhliu76

• log into redis enterprise node and use rladmin

kubectl exec -it rec-0 -- bash
rladmin status extra all

change database

• can make changes to redis database from controller and verify in either rladmin or in the GUI
• use https://localhost:8443 to view the changes
• modify redis-enterprise-database.yml to enable replication and to increase database size

edit redis-enterprise-database.yml
kubectl apply -f redis-enterprise-database.yml

• will see these changes in the management ui and in rladmin

Add redisinsights

These instructions are based on Install RedisInsights on k8s   The above instructions have two options for installing redisinights, this uses the second option to install without a service (avoids creating a load balancer)

• copy the yml file above into a file named redisinsight.yml (this file is already in the k8s directory)
• create redisinsights

kubectl apply -f redisinsight.yml
kubectl port-forward deployment/redisinsight 8001

• from chrome or firefox open the browser using http://localhost:8001
• Click "I already have a database"
• Click "Connect to Redis Database"
• Create Connection to target redis database with following parameter entries

Key	Value
host	redis-enterprise-database.demo
port	18154 (get from ./getDatabasepw.sh above)
name	TargetDB
Username	(leave blank)
Password	DrCh7J31 (from ./getDatabasepw.sh above)

• click ok

Deploy redis-searchStock on Kubernetes

• must log into docker to have access to the docker image

docker login

• modify, create the environmental variables by editing configmap.yml
  • can find the IP addresses and ports for each of the databases by running kubectl get services
  • In the example below the IP address for the REDIS_HOST in the configmap.yaml is 10.28.16.188
  • get the database password and port by running getDatabasePw. Put the returned password the configmap REDIS_PASSWORD and REDIS_PORT
• apply the configuration map

cd k8s
# change REDIS_PASSWORD and REDIS_PORT based on ./getDatabasePw
vi configmap.yaml
kubectl apply -f configmap.yaml

• deploy the redis-searchstock
• can switch between the java version and python version by changing image used in stock.yaml. Jedis version is commented out.

kubectl apply -f pvc.yaml
kubectl apply -f stock.yaml
kubectl get pods

• port forward and continue with testing of the APIs
  • NOTE: get exact name use kubectl get pods

kubectl port-forward redis-searchstock-c568d9b6b-5t8v6 5000

Run search stock

• get the data files from stooq
• copy the stooq files into place. Easiest to zip them before copying using tar -cvf all_tar.tgz data/daily
  • can be issue with hidden files caused by running tar on Mac and deploying on linux see troubleshooting

kubectl cp all_tar.tgz redis-searchstock-c568d9b6b-5t8v6:/data

Publish docker image

The docker image used in k8s/stock.yaml is jphaugla/redis-searchstock:latest If building own image, have docker login, change to personal docker account and change the stock.yaml To build different version of the docker image: docker-compose build, docker tag, docker publish

cd python
docker login
docker-compose build
docker image tag redis-searchstock:latest jphaugla/redis-searchstock:latest
docker image push jphaugla/redis-searchstock:latest

Memtier benchmark

Can also optionally deploy memtier benchmark as a pod in the cluster Before adding memtier.yml, must have extra node and then label the node

kubectl label nodes gke-jph-k8s-cluster-default-pool-6ecc6b17-zllk app=memtier
kubectl apply -f memtier.yml

Deploy redis-sql

Can deploy redis-sql to run SQL on top of redisearch

• Edit the URI string for connection to redisenterprise using ./getDatabasePw.sh
• deploy configmap and yaml for trino

cd k8s
kubectl apply -f trino-configmap.yaml
kubectl apply -f trino.yaml

Use redisinsights or the management UI to observe the benchmark results

Use the application

Create Index

There is python running in the flask container (appy.py) listening for API calls. One of the API calls will recreate the index. Use the following script to create the index

cd scripts
./redoIndex.sh

Prepare the load

* If PROCESS_RECENTS is set, set list of recent dates to specifically set the MostRecent flag to false
  * This is needed when loading the next set of values.  E.g.  Current data is 20220315 and want to ensure three previous dates are false for MostRecent.  
```bash
docker exec -it redis redis-cli
sadd remove_current 20220314 20220313 20220312 

Start Ticker Load

• verify the directory location specified in the script

cd scripts
./loadFile.sh

Can observe the load progress by watching the load for each file

docker exec -it redis redis-cli 
hgetall ticker_load

• THIS IS HOW to start flask app server
• However, it is already running as part of the flask container

docker exec -it flask bash -c "python appy.py"

Go to the stock type ahead page and find the desired stock

These are a group of sample redis-cli queries to see

redic-cli -f scripts/searchQueries.txt

There are scripts in the scripts directory in addition to loadFile.sh and redoIndex.sh. Check these out for additional API calls available such as adding a ticker, deleting a ticker, getting one ticker field, removing one ticker field, etc.

Troubleshooting

Moving the Stooq datafiles from apple (BSD) to Linux can cause an odd read errors in the loading file program delete all these erroneous files. I have now added some code to ignore hidden files in python version of code.

find data/daily -type f -iname "._*.*" -ls -delete