Design/status/roadmap on the clustered version?

[blurrcat]

It is mentioned in the FAQ that the clustered version is in active development. Can you share more details on the design, status, or roadmap? The technical paper doesn't seem to shed much light on this to me. Am I missing anything?

Anyway, great job TimescaleDB!

[akulkarni]

The technical paper describes both our single-node design as well as the clustered design. But if something is unclear, happy to answer any questions.

We haven't announced a timeline for the open-source clustered version yet, but it's on the order of months.

[archenroot]

@akulkarni Hi guys, I think you are quite busy, but do we have any news on scaling out timescaledb accross multiple nodes? In general the idea is if I can do something like with Cassandra, when the load is increasing, I just "plugin" into cluster new nodes, sync and go. If selected I plan to use the timescaledb in Mesos & Docker environment...

[akulkarni]

Hi @archenroot, Are you worried about ingest load or query load?

If ingest load, our single node instance regularly handles 100k inserts/sec (400k inserts/sec under certain configurations). (Which according to our benchmarks is equivalent to several Cassandra nodes.) But let me know if you need more than that.

If query load, then we support read-clustering using PG streaming replication. Then you would round-robin your queries across the various nodes to increase query throughput.

[archenroot]

@akulkarni
thanks for answer first of all.

So, lets imagine I am doing IoT project, where the ultimate sensor data (including images from cameras) will be in volume of unique records at ~60,000,000 per day (I count the target of course).
This represents data per:
1 year - 21,900,000,000
5 years - 109,500,000,000

Writes:
As you suggested timescaledb is capable of high writes (100k inserts/sec), which as number looks fine, but the question is also about hardware sizing. Let's look from service (backend-layer perspective). I will be deploying let's say initially 3 active-active instances, while the load of events comming from device will increase in time, I will simply deploy more containers (docker) if possible within Mesos (DCOS) infrastructure, so horizontal scalability is not a big deal. Based on monitoring load you can not only add instances, but also dynamically remove them to save money when the load is not high (Same apply for message broker network, imagine you are in AWS, etc.).

Writes Question:
But how to scale properly timescaledb. Is it even scalable in similar way as springboot based microservices are? Or it is better for me to directly buy some 4x socket powerfull servers which on the other hand won't be used fully from beginning of project.

Reads:
Good point here, in case of PGStreaming the slave instances are also timescaledb, right?

Multitenancy:
That is the way of how to design the data, I was working before on Oracle 11g Virtual Private Database framework to implement security not on application level (so application fetch all data which later filters), but purely on db level (the client id within the session is recognized by database and it will only return client's data. In Oracle you can do whatever you want, I mean it is not only "disabling data" on row or column, but on cell level. But it is on the other hand quite tricky and I understand this goes beyond timescalleDB, just pointing this out as this will be the way if possible in the design part. Is TimescaleDB compatible with any kind of tenant mechanism(either via integration?

But Multitenancy is out of topic question and there is nice discussion here:
https://stackoverflow.com/questions/44524364/postgresqls-schemas-for-multi-tenant-applications

In our case it more looks we have dozen of tenants which has relativelly small amount of data (gross numbers):

• 100,000 records per year
• 500,000 records per 5 years

Offtopic - can I combine timescaledb with CONTINUOUS VIEWS from pipeline DB?
Imagine I will create 1 database for all the writes based on timescaledb and 2nd database based on pipelineDB, it should be theoretically possible to create continuous views in pipelineDB where the source will be READ SLAVE nodes of the timescaledb, is it possible? Even suggestion here helps, I will test myself of course :-) This idea is mainly to provide real-time AGGREGATION updates during specific time period of time.

Last note, I expect none or minimum UPDATE statements in the system.

[akulkarni]

Hi @archenroot,

Apologies for the delay in responding. Reopening this issue so it doesn't get lost again.

Responses:

1. Storage capacity - Two things worth mentioning here: First, we support data retention policies that would allow you to aggregate data into lower granularities and age out (ie delete) the raw data. We also support moderate elastic scaling via network-attached disks.

2. Writes - We currently do not support horizontal scale out for writes. But according to my calculations, you are doing <1000 inserts/sec (60,000,000/day), so our single node performance should be more than enough. (If instead you are looking for clustering for redundancy, then we would suggest setting up a read slave for failover.)

3. Reads - Yes, the read/slave instances would have to be also TimescaleDB.

4. Multi-tenancy - We have not yet built any native functionality for multi-tenancy, outside of what may already be available in PostgreSQL. Currently our users who require this have been handling themselves at the application layer. There may however be multi-tenant add-on options for PostgreSQL, but we haven’t been able to research those yet.

5. PipelineDB - I’m not sure if you’re suggesting to make the read slave just PipelineDB, or TimescaleDB + PipelineDB. If the former, then I don’t know how it would handle time-series data - we have limited experience. If the latter, then it is possible it may work, but again needs to be tested. (But if you do try out yourself, please let us know how it goes.)

[jrevault]

Hi @akulkarni,
I would second @archenroot thoughts, being able to add seamlessly nodes would help in a case of important new data acquisition, in order to avoid migrating the server each time it would go beyond its capacity.
Today consequences for us is that we must 'pay' for a bigger server just because I cannot easily add new nodes.... if I pay only for the right dimensioned server, I know I'll have to migrate every 2 months (and I would really do something else than often migrating data 😅 ).. so we pay more 😞

And thanks for the wonderful work with Timescale 👍

[archenroot]

If you are aware of architecture like Mesos or DCOS, uzually you are running either storage or services and monitor how they are used, when you reach metrics treshold, you just deploy one or more instances and the system autoreconfigure cluster on the fly. Same happens when the load is falling down, you remove instances, as they are not needed anymore. That is the beauty of autoscalable systems.

[mfreed]

@jrevault One interesting aspect of even timescale's design today is that you can elastically add disks to a hypertable, so the storage capacity can scale with your storage needs. We've seen this applied particularly effectively with network-attached storage (e.g., EBS, Azure Storage) in the cloud.

@archenroot Yep, quite familiar with both of those cluster-management services. That said, scaling up/down a storage system like Timescale is quite a bit different than a stateless, shared-nothing service like your web front-end or API service, as membership changes necessitate application-level support to ensure proper consistency in the stateful storage tier. :)

[jrevault]

@archenroot and @mfreed (sorry for this late answer but we were going live this WE)

Our problem isn't adding disk space, it's a more a memory and CPU one ( well at least it's my understanding of it)
With users growth we will have to treat more and more data.
If each month during 2 years, we multiply by 5 the initial data acquisition rhythm, we need to regularly have more memory and CPU to handle it, meaning changing server size and adapting postgres conf (max_connections, shared_buffers, effective_cache_size, etc...) so I was thinking with horizontal scalability.
But maybe I don't take the problem with the good angle because at start we won't have many data to ingest.
To have an idea of quantity of data to ingest (it's not big), we store student's exercises answers.
In about 2 months we should ingest ~24 000 000 exercises per month meaning on users peak (mostly 4 hours in a day) ~40 inserts/sec (not really impressive) a small server can fit.
5 months later, we should have ~1 000 inserts/sec
5 months later, we should have ~2 000 inserts/sec
and so on...
until we reach a stabilization point not known yet.

[aleksve]

Thanks for interesting ideas! I am evaluating TimescaleDB for a fairly large data collection project. I expect a fairly moderate data inflow rate, maybe 20 000 entries per second. But we want to retain the historical data to the extend possible. 20 000 entires per year accumulate quickly, we expect about 100 Tb of data per year. @akulkarni mentioned something very relevant in the post of Oct 24, 2017: TimescaleDB supports network-allocated storage, which could of course be scaled up to several petabytes. Judging from the TimescaleDB API[1], storing some chunks on a NAS is doable. But is it possible to makeTimescaleDB differentiate between the tablespaces? Can TimescaleDB be configured to create new chunks on the local, RAID storage, and move them to NAS when the data gets older/colder?

Thanks!

[1] Hypertable management, attach_tablespace() https://docs.timescale.com/v0.9/api#hypertable-management

[murugesan70]

Hi @akulkarni @mfreed
we at Plume Designs (Cloud managed wifi company) are evaluating timescaleDB for our time-series DB and I have the same feature request as others.

1. Horizontally scale timescaleDB for our ever growing cloud infrastructure with 200+TB timeseries DB. Apart from vertical scaling, we do want to horizontally scale on number of servers (apart from active-active cluster failover)
   When do you actually plan to roll out horizontally scaling timescaleDB?

Thanks

[archenroot]

@murugesan70 - I am not sure, but looks like VoltDB can support partition horizontal split of tables. Take a look.

[dianasaur323]

Hi everyone! I'm the PM heading up clustering. I'm looking for a couple people to help us provide feedback as we move forward with this. If you'd like to be considered as part of that (we can only take a couple people), please let me know! I'm available in our Slack community as Diana Hsieh, or you can ping me at diana at timescale.com.

[dianasaur323]

Hello~~ Public facing documentation is out for our private beta of clustering (which is currently only in limited release). Please go ahead and take a look!

https://docs.timescale.com/clustering/introduction/architecture#timescaledb-clustering and
https://docs.timescale.com/clustering/getting-started/scaling-out