Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove references to 'arctic' from init scripts #1444

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions docs/data.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ An easy way to bulk download data from [Barchart](https://www.barchart.com) is t

Alternatively, if you are very patient, you can manually download the data from the Barchart historical data pages, such as [this one
for Cotton #2](https://www.barchart.com/futures/quotes/KG*0/historical-download).
Then, to read the data, you can use [this script](/sysinit/futures/barchart_futures_contract_prices.py), which in turn calls this [other more general script](/sysinit/futures/contract_prices_from_csv_to_arctic.py). Although it's very specific to Barchart, with some work you should be able to adapt it. You will need to call it with the directory where your Barchart .csv files are stored.
Then, to read the data, you can use [this script](/sysinit/futures/barchart_futures_contract_prices.py), which in turn calls this [other more general script](/sysinit/futures/contract_prices_from_csv_to_db.py). Although it's very specific to Barchart, with some work you should be able to adapt it. You will need to call it with the directory where your Barchart .csv files are stored.

The script does two things:

Expand Down Expand Up @@ -210,18 +210,18 @@ Here we can see that the barchart files have one initial row we can ignore, and
The actual reading and writing is done here:

```python
def init_arctic_with_csv_futures_contract_prices_for_code(instrument_code: str, datapath: str,
def init_db_with_csv_futures_contract_prices_for_code(instrument_code: str, datapath: str,
csv_config=arg_not_supplied):
print(instrument_code)
csv_prices = csvFuturesContractPriceData(datapath, config=csv_config)
arctic_prices = arcticFuturesContractPriceData()
db_prices = diag_prices.db_futures_contract_price_data

csv_price_dict = csv_prices.get_merged_prices_for_instrument(instrument_code)

for contract_date_str, prices_for_contract in csv_price_dict.items():
print(contract_date_str)
contract = futuresContract(instrument_code, contract_date_str)
arctic_prices.write_merged_prices_for_contract_object(contract, prices_for_contract, ignore_duplication=True)
db_prices.write_merged_prices_for_contract_object(contract, prices_for_contract, ignore_duplication=True)
```

The objects `csvFuturesContractPriceData` and `arcticFuturesContractPriceData` are 'data pipelines', which allow us to read and write a specific type of data (in this case OHLC price data for individual futures contracts). They have the same methods (and they inherit from a more generic object, futuresContractPriceData), which allows us to write code that abstracts the actual place and way the data is stored. We'll see much more of this kind of thing later.
Expand Down Expand Up @@ -260,7 +260,7 @@ Then the roll calendar, plus the individual futures contract prices, can be used
<a name="roll_calendars_from_approx"></a>
### Generate a roll calendar from actual futures prices

This is the method you'd use if you were really starting from scratch, and you'd just got some prices for each futures contract. The relevant script is [here](/sysinit/futures/rollcalendars_from_arcticprices_to_csv.py); you should call the function `build_and_write_roll_calendar`. It is only set up to run a single instrument at a time: creating roll calendars is careful craftsmanship, not suited to a batch process.
This is the method you'd use if you were really starting from scratch, and you'd just got some prices for each futures contract. The relevant script is [here](/sysinit/futures/rollcalendars_from_db_prices_to_csv.py); you should call the function `build_and_write_roll_calendar`. It is only set up to run a single instrument at a time: creating roll calendars is careful craftsmanship, not suited to a batch process.

In this script (which you should run for each instrument in turn):

Expand Down Expand Up @@ -321,7 +321,7 @@ A *valid* roll calendar will have current and next contract prices on the roll d

#### Manually editing roll calendars

Roll calendars are stored in .csv format [and here is an example](/data/futures/roll_calendars_csv/EDOLLAR.csv). Of course you could put these into Mongo DB, or Arctic, but I like the ability to hack them if required; plus we only use them when starting the system up from scratch. If you have to manually edit your .csv roll calendars, you can easily load them up and check they are monotonic and valid. The function [`check_saved_roll_calendar`](/sysinit/futures/rollcalendars_from_arcticprices_to_csv.py) is your friend. Just make sure you are using the right datapath.
Roll calendars are stored in .csv format [and here is an example](/data/futures/roll_calendars_csv/EDOLLAR.csv). Of course you could put these into Mongo DB, or Arctic, but I like the ability to hack them if required; plus we only use them when starting the system up from scratch. If you have to manually edit your .csv roll calendars, you can easily load them up and check they are monotonic and valid. The function [`check_saved_roll_calendar`](/sysinit/futures/rollcalendars_from_db_prices_to_csv.py) is your friend. Just make sure you are using the right datapath.


<a name="roll_calendars_from_multiple"></a>
Expand Down Expand Up @@ -367,7 +367,7 @@ Step 5 can sometimes throw up warnings or outright errors if things don't look r
<a name="mult_adj_csv_to_arctic"></a>
### Writing multiple prices from .csv to database

The use case here is you are happy to use the shipped .csv data, even though it's probably out of date, but you want to use a database for backtesting. You don't want to try and find and upload individual futures prices, or create roll calendars.... the good news is you don't have to. Instead you can just use [this script](/sysinit/futures/multiple_and_adjusted_from_csv_to_arctic.py) which will just copy from .csv (default ['shipping' directory](/data/futures/multiple_prices_csv)) to Arctic.
The use case here is you are happy to use the shipped .csv data, even though it's probably out of date, but you want to use a database for backtesting. You don't want to try and find and upload individual futures prices, or create roll calendars.... the good news is you don't have to. Instead you can just use [this script](/sysinit/futures/multiple_and_adjusted_from_csv_to_db.py) which will just copy from .csv (default ['shipping' directory](/data/futures/multiple_prices_csv)) to Arctic.

This will also copy adjusted prices, so you can now skip ahead to [creating FX data](#create_fx_data).

Expand Down Expand Up @@ -399,12 +399,12 @@ spliced_multiple_prices = os.path.join('data', 'futures', 'multiple_prices_csv_s
if not os.path.exists(spliced_multiple_prices):
os.makedirs(spliced_multiple_prices)

from sysinit.futures.rollcalendars_from_arcticprices_to_csv import build_and_write_roll_calendar
from sysinit.futures.rollcalendars_from_db_prices_to_csv import build_and_write_roll_calendar
instrument_code = 'GAS_US_mini' # for example
build_and_write_roll_calendar(instrument_code,
output_datapath=roll_calendars_from_arctic)
```
We use our updated prices and the roll calendar just built to [calculate multiple prices](#/sysinit/futures/multipleprices_from_arcticprices_and_csv_calendars_to_arctic):
We use our updated prices and the roll calendar just built to [calculate multiple prices](#/sysinit/futures/multipleprices_from_db_prices_and_csv_calendars_to_arctic):

```python
from sysinit.futures.multipleprices_from_db_prices_and_csv_calendars_to_db import
Expand Down Expand Up @@ -450,8 +450,8 @@ assert(supplied.iloc[-1].FORWARD_CONTRACT == generated.loc[last_supplied:].iloc[
spliced = pd.concat([supplied, generated])
spliced.to_csv(os.path.join(spliced_multiple_prices, instrument_code+'.csv'))

from sysinit.futures.multiple_and_adjusted_from_csv_to_arctic import init_arctic_with_csv_prices_for_code
init_arctic_with_csv_prices_for_code(instrument_code, multiple_price_datapath=spliced_multiple_prices)
from sysinit.futures.multiple_and_adjusted_from_csv_to_db import init_db_with_csv_prices_for_code
init_db_with_csv_prices_for_code(instrument_code, multiple_price_datapath=spliced_multiple_prices)
```

<a name="back_adjusted_prices"></a>
Expand Down
8 changes: 4 additions & 4 deletions sysinit/futures/adjustedprices_from_db_multiple_to_db.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,20 +45,20 @@ def process_adjusted_prices_single_instrument(
ADD_TO_CSV=False,
):
(
arctic_multiple_prices,
parquet_adjusted_prices,
db_multiple_prices,
db_adjusted_prices,
csv_adjusted_prices,
) = _get_data_inputs(csv_adj_data_path)
if multiple_prices is arg_not_supplied:
multiple_prices = arctic_multiple_prices.get_multiple_prices(instrument_code)
multiple_prices = db_multiple_prices.get_multiple_prices(instrument_code)
adjusted_prices = futuresAdjustedPrices.stitch_multiple_prices(
multiple_prices, forward_fill=True
)

print(adjusted_prices)

if ADD_TO_DB:
parquet_adjusted_prices.add_adjusted_prices(
db_adjusted_prices.add_adjusted_prices(
instrument_code, adjusted_prices, ignore_duplication=True
)
if ADD_TO_CSV:
Expand Down
6 changes: 3 additions & 3 deletions sysinit/futures/barchart_futures_contract_prices.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
)
from syscore.dateutils import month_from_contract_letter

from sysinit.futures.contract_prices_from_csv_to_arctic import (
from sysinit.futures.contract_prices_from_csv_to_db import (
init_db_with_csv_futures_contract_prices,
)

Expand Down Expand Up @@ -98,7 +98,7 @@ def strip_file_names(pathname):
)


def transfer_barchart_prices_to_arctic(datapath):
def transfer_barchart_prices_to_db(datapath):
strip_file_names(datapath)
init_db_with_csv_futures_contract_prices(datapath, csv_config=barchart_csv_config)

Expand All @@ -107,4 +107,4 @@ def transfer_barchart_prices_to_arctic(datapath):
input("Will overwrite existing prices are you sure?! CTL-C to abort")
# modify flags as required
datapath = "*** NEED TO DEFINE A DATAPATH ***"
transfer_barchart_prices_to_arctic(datapath)
transfer_barchart_prices_to_db(datapath)
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
diag_prices = diagPrices()


def init_arctic_with_csv_futures_contract_prices(
def init_db_with_csv_futures_contract_prices(
multiple_price_datapath=arg_not_supplied, adj_price_datapath=arg_not_supplied
):
csv_multiple_prices = csvFuturesMultiplePricesData(multiple_price_datapath)
Expand All @@ -19,29 +19,29 @@ def init_arctic_with_csv_futures_contract_prices(

instrument_codes = csv_multiple_prices.get_list_of_instruments()
for instrument_code in instrument_codes:
init_arctic_with_csv_prices_for_code(
init_db_with_csv_prices_for_code(
instrument_code,
multiple_price_datapath=multiple_price_datapath,
adj_price_datapath=adj_price_datapath,
)


def init_arctic_with_csv_prices_for_code(
def init_db_with_csv_prices_for_code(
instrument_code: str,
multiple_price_datapath=arg_not_supplied,
adj_price_datapath=arg_not_supplied,
):
print(instrument_code)
csv_mult_data = csvFuturesMultiplePricesData(multiple_price_datapath)
db_mult_data = diagPrices.db_futures_multiple_prices_data
db_mult_data = diag_prices.db_futures_multiple_prices_data

mult_prices = csv_mult_data.get_multiple_prices(instrument_code)
db_mult_data.add_multiple_prices(
instrument_code, mult_prices, ignore_duplication=True
)

csv_adj_data = csvFuturesAdjustedPricesData(adj_price_datapath)
db_adj_data = diagPrices.db_futures_adjusted_prices_data
db_adj_data = diag_prices.db_futures_adjusted_prices_data

adj_prices = csv_adj_data.get_adjusted_prices(instrument_code)
db_adj_data.add_adjusted_prices(
Expand All @@ -51,6 +51,6 @@ def init_arctic_with_csv_prices_for_code(

if __name__ == "__main__":
## modify datapaths if required
init_arctic_with_csv_futures_contract_prices(
init_db_with_csv_futures_contract_prices(
adj_price_datapath=arg_not_supplied, multiple_price_datapath=arg_not_supplied
)
2 changes: 1 addition & 1 deletion sysinit/futures/safely_modify_roll_parameters.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
get_input_from_user_and_convert_to_type,
)
from sysdata.data_blob import dataBlob
from sysinit.futures.rollcalendars_from_arcticprices_to_csv import (
from sysinit.futures.rollcalendars_from_db_prices_to_csv import (
build_and_write_roll_calendar,
)
from sysinit.futures.multipleprices_from_db_prices_and_csv_calendars_to_db import (
Expand Down
2 changes: 1 addition & 1 deletion sysinit/futures/tests/test_sysinit_futures.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import pytest
from sysinit.futures.rollcalendars_from_arcticprices_to_csv import (
from sysinit.futures.rollcalendars_from_db_prices_to_csv import (
build_and_write_roll_calendar,
check_saved_roll_calendar,
)
Expand Down